Return pandas.DataFrame

[xinbinhuang]

Hi team,

It seems that currently fire cannot return dataframe from function or method calls. I would to like make a contribution to this and create a PR if this would be good feature to fire.

Let me know how you thinks

Cheers
Bin

[nfultz]

That's odd, this worked in ~2018, I demoed it in a lightning talk at a pydata meetup - https://docs.google.com/presentation/d/1NHAYGN4Fx4YBVcaGAiJ4U3gEkyhhgE643r7Ka2MF2wI/edit?usp=sharing

EDIT:

My script still works, you just need to manually call to_string at the end

$ pd read_csv drive/aws_costs.csv - mean - round - to_string

[xinbinhuang]

Hmm I see. I didn't know about the to_string method for Dataframe.

My Fire component is actually a class which utilize pandas under the hood and it returns a dataframe for most of the methods.

I think it's not intuitive for end-user (i.e. my client and colleague) to know that they need to chain command with - to_string to render the output properly. Given that dataframe is such a popular object right now, do you think it's worth adding the - to_string as default behavior to Fire?

[nfultz]

If you look at the stack trace, it's because DataFrames fail on inspect, which is used by fire to generate the list of subcommands.

You reproduce it ouside of fire using:

>>> import pandas
>>> import inspect
>>> df = pandas.read_csv("drive/aws_costs.csv")
>>> inspect.getmembers(df)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/inspect.py", line 342, in getmembers
    value = getattr(object, key)
  File "/home/nfultz/.local/lib/python3.6/site-packages/pandas/core/frame.py", line 409, in _constructor_expanddim
    raise NotImplementedError("Not supported for DataFrames!")
NotImplementedError: Not supported for DataFrames!

So if you make pandas work with inspect, fire will be compatible accordingly, or at least not crash.

OTOH, I'm not sure generating a list of the data frame API is very helpful since it's quite large.

[dbieber]

Python Fire should work even when inspect.getmembers doesn't. I'll mark this as a bug.

[nfultz]

Fair enough. See also pandas-dev/pandas#31474 , they fixed it but haven't released it yet.

It's also been filed against inspect https://bugs.python.org/issue35108#msg361208

[xinbinhuang]

@nfultz I am not familiar with inspect.getmembers in general, but I do get impressed by how quickly you gather all these information! These information is pretty good for me probably other people as a reference.

Thank you so much for the help!

[dbieber]

Would you be able to provide a minimal example reproducing the issue?

[genos]

Would you be able to provide a minimal example reproducing the issue?

Please pardon me for barging in, but I use Pandas a lot and have really enjoyed using fire for CLIs.

Script

#!/usr/bin/env python3
import fire
import pandas as pd

def test(n: int) -> pd.DataFrame:
    return pd.DataFrame(data=range(n))

if __name__ == "__main__":
    fire.Fire(test)

Environment

Setting up a Python 3.8 virtual environment, activating it, installing dependencies:

~/tmp ∃ python3 -m venv env
~/tmp ∃ source ./env/bin/activate
(env) ~/tmp ∃ pip install -U pip fire jinja2 pandas -qqq
(env) ~/tmp ∃ python --version
Python 3.8.6
(env) ~/tmp ∃ pip freeze
fire==0.3.1
Jinja2==2.11.2
MarkupSafe==1.1.1
numpy==1.19.2
pandas==1.1.3
python-dateutil==2.8.1
pytz==2020.1
six==1.15.0
termcolor==1.1.0

Running

It looks like our version of Pandas fixes the inspect-related NotImplementedError mentioned above; when running the script inside our environment, we get a ValueError

(env) ~/tmp ∃ ./w.py
ERROR: The function received no value for the required argument: n
Usage: w.py N

For detailed information on this command, run:
  w.py --help
(env) ~/tmp ∃ ./w.py 10
/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/inspect.py:350: FutureWarning: _AXIS_NAMES has been deprecated.
  value = getattr(object, key)
/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/inspect.py:350: FutureWarning: _AXIS_NUMBERS has been deprecated.
  value = getattr(object, key)
Traceback (most recent call last):
  File "./w.py", line 9, in <module>
    fire.Fire(test)
  File "/Users/graham/tmp/env/lib/python3.8/site-packages/fire/core.py", line 164, in Fire
    _PrintResult(component_trace, verbose=component_trace.verbose)
  File "/Users/graham/tmp/env/lib/python3.8/site-packages/fire/core.py", line 266, in _PrintResult
    help_text = helptext.HelpText(
  File "/Users/graham/tmp/env/lib/python3.8/site-packages/fire/helptext.py", line 63, in HelpText
    actions_grouped_by_kind = _GetActionsGroupedByKind(component, verbose=verbose)
  File "/Users/graham/tmp/env/lib/python3.8/site-packages/fire/helptext.py", line 332, in _GetActionsGroupedByKind
    members = completion.VisibleMembers(component, verbose=verbose)
  File "/Users/graham/tmp/env/lib/python3.8/site-packages/fire/completion.py", line 365, in VisibleMembers
    return [
  File "/Users/graham/tmp/env/lib/python3.8/site-packages/fire/completion.py", line 367, in <listcomp>
    if MemberVisible(component, member_name, member, class_attrs=class_attrs,
  File "/Users/graham/tmp/env/lib/python3.8/site-packages/fire/completion.py", line 311, in MemberVisible
    if member in (absolute_import, division, print_function):
  File "/Users/graham/tmp/env/lib/python3.8/site-packages/pandas/core/generic.py", line 1329, in __nonzero__
    raise ValueError(
ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

[dbieber]

Thanks!

[genos]

Happy to contribute!

If we uninstall the released version of fire and instead install from the current master commit from GitHub:

(env) ~/tmp ∃ pip uninstall fire
Found existing installation: fire 0.3.1
Uninstalling fire-0.3.1:
  Would remove:
    /Users/graham/tmp/env/lib/python3.8/site-packages/fire-0.3.1-py3.8.egg-info
    /Users/graham/tmp/env/lib/python3.8/site-packages/fire/*
Proceed (y/n)? y
  Successfully uninstalled fire-0.3.1
(env) ~/tmp ∃ pip install git+https://github.com/google/python-fire.git@878b8d86f488ef2606cffdf58297dd2781708316 -qqq

it looks like we get dropped into the help information for pd.DataFrame:

(env) ~/tmp ∃ ./w.py 10
/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/inspect.py:350: FutureWarning: _AXIS_NAMES has been deprecated.
  value = getattr(object, key)
/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/inspect.py:350: FutureWarning: _AXIS_NUMBERS has been deprecated.
  value = getattr(object, key)
NAME
    w.py 10 - Two-dimensional, size-mutable, potentially heterogeneous tabular data.

SYNOPSIS
    w.py 10 GROUP | COMMAND | VALUE

DESCRIPTION
    Data structure also contains labeled axes (rows and columns).
    Arithmetic operations align on both row and column labels. Can be
    thought of as a dict-like container for Series objects. The primary
    pandas data structure.

GROUPS
    GROUP is one of the following:

     T
       Two-dimensional, size-mutable, potentially heterogeneous tabular data.

     at
       Access a single value for a row/column label pair.

     attrs

     axes

     columns
       Immutable Index implementing a monotonic integer range.

     dtypes
       One-dimensional ndarray with axis labels (including time series).

     iat
       Access a single value for a row/column pair by integer position.

     iloc
       Purely integer-location based indexing for selection by position.

     index
       Immutable Index implementing a monotonic integer range.

     loc
       Access a group of rows and columns by label(s) or a boolean array.

     plot
       Make plots of Series or DataFrame.

     shape

     style
       Helps style a DataFrame or Series according to the data with HTML and CSS.

COMMANDS
    COMMAND is one of the following:
     abs
       Return a Series/DataFrame with absolute numeric value of each element.

     add
       Get Addition of dataframe and other, element-wise (binary operator `add`).


# Several lines omitted

[genos]

Just tested against the latest commit (3be260e), still getting the same behavior.

[genos]

[ping] Just checked against the latest commit (bc43dca), still getting the same behavior.

[dbieber]

Thanks for the pings.
I can confirm the behavior you're seeing.

The issue now seems to stem from our implementation of "HasCustomStr".

python-fire/fire/value_types.py

Line 63 in bc43dca

def HasCustomStr(component):

HasCustomStr should return whether an object defines a custom "__str__" function. DataFrames clearly do, since str(df) prints the dataframe nicely. However, HasCustomStr(df) returns False rather than True.

Next step is to figure out why.

[dbieber]

Ah,

So if I'm understanding things correctly, here's what's happening.

DataFrames don't redefine __str__, they just redefine __repr__. The default implementation of __str__ is simply to call __repr__.

The fix will be to update our HasCustomStr method to also check for custom a __repr__ implementation. If either is present, we'll return True.

This will switch Fire's behavior from dropping you into the help screen to instead just printing out the DataFrame.