Skip to content

BUG: Fix sort_values and factorize with Arrow null type (GH#54908) - #69463

Merged
rhshadrach merged 2 commits into
pandas-dev:mainfrom
bodapatisaikrishna:fix/54908-arrow-null-sort
Sep 28, 2026
Merged

rhshadrach merged 2 commits into
pandas-dev:mainfrom
bodapatisaikrishna:fix/54908-arrow-null-sort

Conversation

@bodapatisaikrishna

@bodapatisaikrishna bodapatisaikrishna commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Check exactly one of the following, per the automated contributions policy:

  • I did not use AI to develop this pull request.
  • I used AI to develop this pull request.

Description

Sorting a DataFrame containing columns with PyArrow null type (e.g. df.sort_values(list(df.columns))) raised ValueError: Categorical categories cannot be null.

Root Cause

When sorting multiple columns, DataFrame.sort_values uses lexsort_indexer, which factorizes columns via factorize(use_na_sentinel=True) and creates Categorical from the resulting codes and categories. For pa.null(), pa.chunked_array.dictionary_encode(null_encoding="mask") produces a dictionary of null type containing a null element ([<NA>]). Because use_na_sentinel=True was passed, returning null categories violated the CategoricalDtype invariant that categories cannot contain nulls. This also caused pd.Categorical(s) and s.astype("category") to fail on null[pyarrow] Series.

Fix

In ArrowExtensionArray.factorize, handle pa.types.is_null(data.type):

  • When use_na_sentinel=True or array is empty: return sentinel indices (-1) and an empty chunked array with pa.null() as uniques.
  • When use_na_sentinel=False: return 0 indices and uniques containing a single null scalar [None] with pa.null().

@github-actions github-actions Bot added the Awaiting Review PR in the maintainers' court; exempt from staleness label Sep 26, 2026

@rhshadrach rhshadrach left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR!

Comment thread pandas/tests/extension/test_arrow.py Outdated
tm.assert_frame_equal(result, expected)


def test_sort_values_null():

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you either parametrize this or split it up into individual tests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parametrized test_sort_values_null across ascending and split out the empty and Series cases into separate tests.

Comment thread pandas/tests/extension/test_arrow.py Outdated
ser = pd.Series([None, None], dtype="null[pyarrow]")
result = pd.Categorical(ser)
assert len(result.categories) == 0
assert result.isna().all()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you check the full result here directly and below.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to assert against the full expected Categorical and Series directly.

@github-actions github-actions Bot removed the Awaiting Review PR in the maintainers' court; exempt from staleness label Sep 28, 2026

@rhshadrach rhshadrach left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@rhshadrach rhshadrach added Bug Algos Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff Arrow pyarrow functionality labels Sep 28, 2026
@rhshadrach rhshadrach added this to the 3.1 milestone Sep 28, 2026
@rhshadrach
rhshadrach merged commit e0b6e97 into pandas-dev:main Sep 28, 2026
66 checks passed
@rhshadrach

Copy link
Copy Markdown
Member

Thanks @bodapatisaikrishna

bodapatisaikrishna added a commit to bodapatisaikrishna/bodapatisaikrishna that referenced this pull request Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Algos Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff Arrow pyarrow functionality Bug

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG: sort_values raises error about null categorical values when no categorical types are used

2 participants