-
-
Notifications
You must be signed in to change notification settings - Fork 18.7k
Description
Pandas version checks
I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
I have confirmed this bug exists on the main branch of pandas.
To pick up a draggable item, press the space bar. While dragging, use the arrow keys to move the item. Press space again to drop the item in its new position, or press escape to cancel.
Reproducible Example
import pandas as pd
from pyarrow import string
df = pd.DataFrame([
[0,"X","A"],
[1,"X","A"],
[2,"X","A"],
[3,"X","B"],
[4,"X","B"],
[5,"X","B"],], columns = ["a","b","c"]).astype({"a":int,
"b":str,"c":pd.ArrowDtype(string())})
df.set_index("b").groupby("a").agg(lambda df: df.to_dict())
Issue Description
When applying groupby aggregate on a column with type defined using pd.ArrowDtype()
the pandas tries to cast the output into the original type, which can raise an error (e.g. pyarrow.lib.ArrowNotImplementedError: Unsupported cast from struct<location_abbreviation: string> to utf8 using function cast_string
for the example provided).
For example, if string[pyarrow]
is used, then this behaviour doesn't occur:
import pandas as pd
df = pd.DataFrame([
[0,"X","A"],
[1,"X","A"],
[2,"X","A"],
[3,"X","B"],
[4,"X","B"],
[5,"X","B"],], columns = ["a","b","c"]).astype({"a":int,
"b":str,"c":"string[pyarrow]"})
df.set_index("b").groupby("a").agg(lambda df: df.to_dict())
Or if the user-defined function also has *args
or **kwargs
, this coercion is not applied:
import pandas as pd
df = pd.DataFrame([
[0,"X","A"],
[1,"X","A"],
[2,"X","A"],
[3,"X","B"],
[4,"X","B"],
[5,"X","B"],], columns = ["a","b","c"]).astype({"a":int,
"b":str,"c":pd.ArrowDtype(string()})
df.set_index("b").groupby("a").agg(lambda df, _: df.to_dict(), [])
both returns:
a | c |
---|---|
0 | {'X': 'A'} |
1 | {'X': 'A'} |
2 | {'X': 'A'} |
3 | {'X': 'B'} |
4 | {'X': 'B'} |
5 | {'X': 'B'} |
Expected Behavior
I would expect the code from example to return:
a | c |
---|---|
0 | {'X': 'A'} |
1 | {'X': 'A'} |
2 | {'X': 'A'} |
3 | {'X': 'B'} |
4 | {'X': 'B'} |
5 | {'X': 'B'} |
Installed Versions
INSTALLED VERSIONS
commit : 2cc3762
python : 3.11.6
python-bits : 64
OS : Linux
OS-release : 5.10.223-211.872.amzn2.x86_64
Version : #1 SMP Mon Jul 29 19:52:29 UTC 2024
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 2.3.0
numpy : 1.26.4
pytz : 2025.2
dateutil : 2.9.0.post0
pip : 24.3.1
Cython : None
sphinx : None
IPython : 9.3.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.13.3
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : 2025.5.1
html5lib : None
hypothesis : 6.135.0
gcsfs : None
jinja2 : 3.1.6
lxml.etree : 5.4.0
matplotlib : 3.10.3
numba : None
numexpr : None
odfpy : None
openpyxl : 3.1.5
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : 18.1.0
pyreadstat : None
pytest : 7.4.4
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.14.1
sqlalchemy : None
tables : None
tabulate : 0.9.0
xarray : None
xlrd : None
xlsxwriter : None
zstandard : None
tzdata : 2025.2
qtpy : None
pyqt5 : None
Activity
heoh commentedon Jun 12, 2025
Thanks for describing the issue. I'd like to try work on it.
heoh commentedon Jun 12, 2025
take
BUG: Fix GroupBy aggregate coersion of outputs inconsistency for pyar…
BUG: Fix GroupBy aggregate coersion of outputs inconsistency for pyar…
BUG: Fix GroupBy aggregate coersion of outputs inconsistency for pyar…
Merge branch 'main' into fix/pandas-dev#61636
Merge branch 'main' into fix/pandas-dev#61636
Merge branch 'main' into fix/pandas-dev#61636
Merge branch 'main' into fix/pandas-dev#61636