Skip to content

Default to_* methods to compression='infer' #22011

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 40 commits into from
Aug 1, 2018
Merged
Changes from 1 commit
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
8689167
Default to_csv & to_json to compression='infer'
dhimmel Jul 21, 2018
3ccfb00
to_json compression=infer in pandas/core/generic.py
dhimmel Jul 21, 2018
648bf4d
Simplify CSVFormatter.save
dhimmel Jul 21, 2018
be724fa
Exploratory commit of what CSVFormatter.save should look like
dhimmel Jul 21, 2018
9fe27c9
fixup! Simplify CSVFormatter.save
dhimmel Jul 23, 2018
65f0689
"Revert changes not related to compression default
dhimmel Jul 23, 2018
868e671
TST: test to_csv infers compression by default
dhimmel Jul 23, 2018
c3b76ee
Debugging print statements
dhimmel Jul 23, 2018
cebc0d9
Debugging: use logging rather than print
dhimmel Jul 23, 2018
8411eb2
_infer_compression in CSVFormatter
dhimmel Jul 26, 2018
c098c8f
CSVFormatter: process encoding in init for consistency
dhimmel Jul 26, 2018
2f6601d
TST + DOC: test_compression_warning docstring
dhimmel Jul 26, 2018
eb7f9b5
fixup! CSVFormatter: process encoding in init for consistency
dhimmel Jul 26, 2018
d4a5c90
Tests passing: remove debugging
dhimmel Jul 26, 2018
abd19e3
Parametrized test for compression='infer' is default
dhimmel Jul 26, 2018
2f670fe
Default compression='infer' in series.to_csv
dhimmel Jul 26, 2018
aa9ce13
What's New Entry for v0.24.0
dhimmel Jul 26, 2018
a6aabad
Remove unused tmpdir fixture argument
dhimmel Jul 26, 2018
8a0c97e
Update to_json docstring
dhimmel Jul 26, 2018
6be808d
Change test docstrings to comments
dhimmel Jul 26, 2018
63e6591
Consolidate testing to a single parametrized test
dhimmel Jul 26, 2018
fadb943
Split test_compression_defaults_to_infer into Series & DataFrame tests
dhimmel Jul 26, 2018
0edffc7
Parametrize write_kwargs
dhimmel Jul 26, 2018
97f5de5
Fix kwargs in test_series_compression_defaults_to_infer
dhimmel Jul 26, 2018
83bc0a8
Attempt to fix CSV series roundtrip
dhimmel Jul 26, 2018
874a4bf
Fix test failure
dhimmel Jul 26, 2018
14c3945
Python 2 flake8 error
dhimmel Jul 26, 2018
9a4dc41
Reduce / remove comments
dhimmel Jul 27, 2018
25bdb4c
Merge master: fix zip-docs conflict
dhimmel Jul 29, 2018
1ba8f3a
DOC: versionchanged & tweaks
dhimmel Jul 30, 2018
24e051e
Update doc/source/io.rst as needed
dhimmel Jul 30, 2018
387d1d2
Move tests from tests/test_common.py to tests/io/test_common.py
dhimmel Jul 30, 2018
12f14e2
Organize / simplify pandas/tests/test_common.py imports
dhimmel Jul 30, 2018
6db23d9
Ignore flake error needed for test
dhimmel Jul 30, 2018
e3a0f56
fixup! Organize / simplify pandas/tests/test_common.py imports
dhimmel Jul 30, 2018
af8c137
change import: cmn to icom
dhimmel Jul 31, 2018
f8829a6
Blank lines after versionchanged
dhimmel Jul 31, 2018
918c0f8
Move compression tests to new file tests/io/test_compression.py
dhimmel Jul 31, 2018
eadf68e
blank lines before .. versionchanged
dhimmel Jul 31, 2018
cf5b62e
Remove comments and space after GH
dhimmel Aug 1, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Split test_compression_defaults_to_infer into Series & DataFrame tests
  • Loading branch information
dhimmel committed Jul 26, 2018
commit fadb943851e395cfd6e7dbe05ba86d15ea2ee6f2
41 changes: 28 additions & 13 deletions pandas/tests/test_common.py
Original file line number Diff line number Diff line change
Expand Up @@ -256,28 +256,43 @@ def test_compression_size_fh(obj, method, compression_only):
assert uncompressed > compressed


@pytest.mark.parametrize('input', [
DataFrame([[1.0, 0, -4.4],
[3.4, 5, 2.4]], columns=['X', 'Y', 'Z']),
Series([0, 1, 2, 4], name='X'),
])
@pytest.mark.parametrize('methods', [
@pytest.mark.parametrize('write_method, read_method', [
('to_csv', pandas.read_csv),
('to_json', pandas.read_json),
('to_pickle', pandas.read_pickle),
])
def test_compression_defaults_to_infer(input, methods, compression_only):
# Test that to_* methods default to inferring compression from paths.
# https://github.com/pandas-dev/pandas/pull/22011
write_method, read_method = methods
def test_dataframe_compression_defaults_to_infer(
write_method, read_method, compression_only):
# Test that DataFrame.to_* methods default to inferring compression from
# paths. https://github.com/pandas-dev/pandas/pull/22011
input = DataFrame([[1.0, 0, -4.4], [3.4, 5, 2.4]], columns=['X', 'Y', 'Z'])
extension = _compression_to_extension[compression_only]
kwargs = {}
if write_method == 'to_csv':
kwargs['index'] = False
with tm.ensure_clean('compressed' + extension) as path:
# assumes that compression='infer' is the default
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you can reduce the amount of comments that would be ideal. Citing this as a specific example it doesn't add anything that's not already known from the test name.

Generally we just go # GHXXXXX as a comment at the beginning of the test to reference the original issue

getattr(input, write_method)(path, **kwargs)
output = read_method(path, compression=compression_only)
tm.assert_frame_equal(output, input)


@pytest.mark.parametrize('write_method, read_method', [
('to_csv', pandas.Series.from_csv),
('to_json', pandas.read_json),
('to_pickle', pandas.read_pickle),
])
def test_series_compression_defaults_to_infer(
write_method, read_method, compression_only):
# Test that Series.to_* methods default to inferring compression from
# paths. https://github.com/pandas-dev/pandas/pull/22011
input = Series(100 * [0, 5, -2, 10])
extension = _compression_to_extension[compression_only]
with tm.ensure_clean('compressed' + extension) as path:
# assumes that compression='infer' is the default
getattr(input, write_method)(path)
output = read_method(path, compression=compression_only)
assert_equals = (tm.assert_frame_equal if isinstance(input, DataFrame)
else tm.assert_series_equal)
assert_equals(output, input)
tm.assert_series_equal(output, input)


def test_compression_warning(compression_only):
Expand Down