Skip to content

BUG: Groupby aggregate coersion of outputs inconsistency for pyarrow dtypes #61636

@AndrejIring

Description

@AndrejIring

Pandas version checks

  • I have checked that this issue has not already been reported.

    I have confirmed this bug exists on the latest version of pandas.

    I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
from pyarrow import string

df = pd.DataFrame([
    [0,"X","A"],
    [1,"X","A"],
    [2,"X","A"],
    [3,"X","B"],
    [4,"X","B"],
    [5,"X","B"],], columns = ["a","b","c"]).astype({"a":int,
    "b":str,"c":pd.ArrowDtype(string())})

df.set_index("b").groupby("a").agg(lambda df: df.to_dict())

Issue Description

When applying groupby aggregate on a column with type defined using pd.ArrowDtype() the pandas tries to cast the output into the original type, which can raise an error (e.g. pyarrow.lib.ArrowNotImplementedError: Unsupported cast from struct<location_abbreviation: string> to utf8 using function cast_string for the example provided).

For example, if string[pyarrow] is used, then this behaviour doesn't occur:

import pandas as pd


df = pd.DataFrame([
    [0,"X","A"],
    [1,"X","A"],
    [2,"X","A"],
    [3,"X","B"],
    [4,"X","B"],
    [5,"X","B"],], columns = ["a","b","c"]).astype({"a":int,
    "b":str,"c":"string[pyarrow]"})

df.set_index("b").groupby("a").agg(lambda df: df.to_dict())

Or if the user-defined function also has *args or **kwargs, this coercion is not applied:

import pandas as pd


df = pd.DataFrame([
    [0,"X","A"],
    [1,"X","A"],
    [2,"X","A"],
    [3,"X","B"],
    [4,"X","B"],
    [5,"X","B"],], columns = ["a","b","c"]).astype({"a":int,
    "b":str,"c":pd.ArrowDtype(string()})

df.set_index("b").groupby("a").agg(lambda df, _: df.to_dict(), [])

both returns:

a c
0 {'X': 'A'}
1 {'X': 'A'}
2 {'X': 'A'}
3 {'X': 'B'}
4 {'X': 'B'}
5 {'X': 'B'}

Expected Behavior

I would expect the code from example to return:

a c
0 {'X': 'A'}
1 {'X': 'A'}
2 {'X': 'A'}
3 {'X': 'B'}
4 {'X': 'B'}
5 {'X': 'B'}

Installed Versions

INSTALLED VERSIONS

commit : 2cc3762
python : 3.11.6
python-bits : 64
OS : Linux
OS-release : 5.10.223-211.872.amzn2.x86_64
Version : #1 SMP Mon Jul 29 19:52:29 UTC 2024
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.3.0
numpy : 1.26.4
pytz : 2025.2
dateutil : 2.9.0.post0
pip : 24.3.1
Cython : None
sphinx : None
IPython : 9.3.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.13.3
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : 2025.5.1
html5lib : None
hypothesis : 6.135.0
gcsfs : None
jinja2 : 3.1.6
lxml.etree : 5.4.0
matplotlib : 3.10.3
numba : None
numexpr : None
odfpy : None
openpyxl : 3.1.5
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : 18.1.0
pyreadstat : None
pytest : 7.4.4
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.14.1
sqlalchemy : None
tables : None
tabulate : 0.9.0
xarray : None
xlrd : None
xlsxwriter : None
zstandard : None
tzdata : 2025.2
qtpy : None
pyqt5 : None

Activity

heoh

heoh commented on Jun 12, 2025

@heoh
Contributor

Thanks for describing the issue. I'd like to try work on it.

heoh

heoh commented on Jun 12, 2025

@heoh
Contributor

take

added and removed
Needs TriageIssue that has not been reviewed by a pandas team member
on Jul 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Participants

    @simonjayhawkins@heoh@AndrejIring@rhshadrach

    Issue actions

      BUG: Groupby aggregate coersion of outputs inconsistency for pyarrow dtypes · Issue #61636 · pandas-dev/pandas