Skip to content

BUG: More Indicative Error when pd.melt with duplicate columns #61475

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
3 tasks done
James-Lee-998 opened this issue May 21, 2025 · 2 comments · May be fixed by #61484
Open
3 tasks done

BUG: More Indicative Error when pd.melt with duplicate columns #61475

James-Lee-998 opened this issue May 21, 2025 · 2 comments · May be fixed by #61484
Assignees
@ZanirP
Labels
Bug Error Reporting Incorrect or improved errors from pandas good first issue Reshaping Concat, Merge/Join, Stack/Unstack, Explode

Comments

@James-Lee-998
Copy link

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

x = pd.DataFrame([[1, 2, 3], [3, 4, 5]], columns=["A", "A", "B"])
pd.melt(x, id_vars=["A"], value_vars=["B"])

Issue Description

Error raised when melting on DataFrame with duplicate column headers

import pandas as pd

x = pd.DataFrame([[1, 2, 3], [3, 4, 5]], columns=["A", "A", "B"])
pd.melt(x, id_vars=["A"], value_vars=["B"])

Above raises:

File "pandas\core\reshape\melt.py", line 110, in melt
  if not isinstance(id_data.dtype, np.dtype):
                    ^^^^^^^^^^^^^
File "pandas\core\generic.py", line 6286, in __getattr__
  return object.__getattribute__(self, name)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'DataFrame' object has no attribute 'dtype'. Did you mean: 'dtypes'?

From pandas\core\reshape\melt.py:108 pop method causes id_data to be assigned to a DataFrame object rather causing above AttributeError

for col in id_vars:
  id_data = frame.pop(col)
  if not isinstance(id_data.dtype, np.dtype):

When having duplicate column headers in a dataframe it raises an AttributeError, should this instead indicate a hint about melting on duplicated column headers? Possibly implement a check prior to .dtype being called?

Expected Behavior

Error raised should be indicative of duplicated column headers.

for col in id_vars:
  id_data = frame.pop(col)
  if isinstance(id_data, pd.DataFrame):
    raise Exception(f"{col} is a duplicate column header")
  if not isinstance(id_data.dtype, np.dtype):

Installed Versions

INSTALLED VERSIONS

commit : 0691c5c
python : 3.11.4
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19045
machine : AMD64
processor : Intel64 Family 6 Model 154 Stepping 3, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United Kingdom.1252
pandas : 2.2.3
numpy : 1.26.4
pytz : 2024.2
dateutil : 2.9.0.post0
pip : 25.1.1
Cython : None
sphinx : None
IPython : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.13.4
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : 2024.9.0
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.6
lxml.etree : 5.3.1
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : 3.1.5
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : 17.0.0
pyreadstat : None
pytest : 8.3.3
python-calamine : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : 0.9.0
xarray : None
xlrd : 2.0.1
xlsxwriter : 3.2.0
zstandard : None
tzdata : 2024.2
qtpy : None
pyqt5 : None
@James-Lee-998 James-Lee-998 added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels May 21, 2025
@James-Lee-998 James-Lee-998 changed the title More Indicative Error when pd.melt with duplicate columns BUG: More Indicative Error when pd.melt with duplicate columns May 21, 2025
@rhshadrach
Copy link
Member

Thanks for the report, confirmed on main. Agreed we should check and raise a more understandable error when there are duplicate columns. PR to fix is welcome!

@rhshadrach rhshadrach added Reshaping Concat, Merge/Join, Stack/Unstack, Explode good first issue Error Reporting Incorrect or improved errors from pandas and removed Needs Triage Issue that has not been reviewed by a pandas team member labels May 23, 2025
@ZanirP
Copy link

ZanirP commented May 23, 2025

take

@ZanirP ZanirP linked a pull request May 23, 2025 that will close this issue
Open
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ZanirP ZanirP

Labels
Bug Error Reporting Incorrect or improved errors from pandas good first issue Reshaping Concat, Merge/Join, Stack/Unstack, Explode
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

BUG: Raise clear error for duplicate id_vars in melt (GH61475) ZanirP/pandas
3 participants
@rhshadrach @ZanirP @James-Lee-998