[ENH] Add null_flag function #501

[anzelpwj]

PR Description

Adds a null flag function, which creates a new column in a DataFrame to mark which rows had null values. You can choose a subset of columns to consider as needed.

This PR resolves #501.

PR Checklist

Please ensure that you have done the following:

1. PR in from a fork off your branch. Do not PR from <your_username>:master, but rather from <your_username>:<branch_name>.
2. If you're not on the contributors list, add yourself to AUTHORS.rst.
3. Add a line to CHANGELOG.rst under the latest version header (i.e. the one that is "on deck") describing the contribution.

Quick Check

To do a very quick check that everything is correct, follow these steps below:

• Run the command make check from pyjanitor's top-level directory. This will automatically run:
  • black formatting
  • pycodestyle checking
  • running the test suite
  • docs build

Once done, please check off the check-box above.

If make check does not work for you, you can execute the commands listed in the Makefile individually.

Code Changes

If you are adding code changes, please ensure the following:

• Ensure that you have added tests.
• Run all tests ($ pytest .) locally on your machine.
  • Check to ensure that test coverage covers the lines of code that you have added.
  • Ensure that all tests pass.

Relevant Reviewers

Please tag maintainers to review.

• @ericmjl
• @jk3587

[codecov]

Codecov Report

Merging #510 into dev will increase coverage by 0.03%.
The diff coverage is 95.45%.

@@            Coverage Diff             @@
##              dev     #510      +/-   ##
==========================================
+ Coverage   92.88%   92.92%   +0.03%     
==========================================
  Files          10       10              
  Lines         872      891      +19     
==========================================
+ Hits          810      828      +18     
- Misses         62       63       +1

[anzelpwj]

Just realized I probably need to add some files to the docs folder. Will do that tomorrow.

[ericmjl]

Thanks for the PR, @anzelpwj! I think this will be a great addition to the library. From this first-pass review, I have a few requested changes, mostly centering around docstrings + the tests, with the comments coming from a maintainability perspective. Hope you understand 😄.

[ericmjl]

Everything looks great, thanks @anzelpwj!

[hectormz]

@ericmjl, @anzelpwj I can move this conversation to an issue if more appropriate.

I rebased recently and included this PR. I'm getting some failing tests that were added with this PR:

• test_non_method_functional
• test_functional_on_some_columns
• test_rename_output_column
• test_functional_on_all_columns

The issue is that flag_nulls() is adding a column with dtype=int32, but the expected dataframe is adding a column that is int64. Not sure why that might only be happening for me, and passed tests for all of you and Azure.

Adding a line like:

expected = expected.astype({"null_flag": "int32"})

fixes the issues. Any thoughts?

I'm on Windows in a pyjanitor venv w/ Python 3.7.4, Pandas 0.24.2

[anzelpwj]

Hmmm, what if we explicitly set check_column_type='equiv' in the dataframes assert? Does that fix it?

[hectormz]

Hmm, I believe that is the default, and setting it explicitly didn't change for me. So that's even stranger that this is happening.

[anzelpwj]

It is default, but thought we'd check anyway. I'd be fine with this fix, but TBH we should open up a ticket on the Pandas project to list this as a bug with their assert method.

[Ram-N]

I am on Windows10 and getting the exact same 4 tests failing, reported by @hectormz above.

 assert_frame_equal(df, expected)
E       AssertionError: Attributes are different
E       
E       Attribute "dtype" are different
E       [left]:  int32
E       [right]: int64

The thing is, I am just getting the dev env set up. I haven't made a single change to the code or to the docs yet.

Is there any way to 'soften' these tests so that int32 and int64 are treated as equivalent dtypes?

[hectormz]

@anzelpwj it might be an issue with Pandas assert method. @Ram-N , are you using vanilla Python as well? Not anaconda?

[Ram-N]

@hectormz I am using Anaconda's Python.

Python 3.7.3 | packaged by conda-forge | (default, Jul  1 2019, 22:01:29) 
[MSC v.1900 64 bit (AMD64)] :: Anaconda, Inc. on win32

[anzelpwj]

Would y'all like me to make the extra PR for the expected = expected.astype({"null_flag": "int32"}) line?

[Ram-N]

@anzelpwj Yes, please. That will help me. I can rebase and continue.