Skip to content

fix(parquet): support non-primitive map keys in record reader - #11229

Merged
Jefffrey merged 3 commits into
apache:mainfrom
hsdfat:parquet-record-complex-map-keys
Oct 2, 2026
Merged

Jefffrey merged 3 commits into
apache:mainfrom
hsdfat:parquet-record-complex-map-keys

Conversation

@hsdfat

@hsdfat hsdfat commented Sep 27, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

The record API (RowIter / TreeBuilder) asserted that map keys are primitive, so reading a file with e.g. struct- or list-keyed maps panicked. The Parquet spec has no such restriction and the arrow reader already handles these files.

This picks up the fix from #7769 by @njaremko, which went stale waiting on a test file in apache/parquet-testing#87. Thanks @njaremko for the original fix.

What changes are included in this PR?

  • Remove the primitive-key assertion in TreeBuilder::reader_tree. The key reader was already built via reader_tree in the same way as the value reader, so no other reader changes are needed.
  • Map stores entries as Vec<(Field, Field)>, so complex keys need no Hash/Eq; Display and to_json_value already handle non-primitive keys.

Are these changes tested?

Yes. test_compound_map_key writes a file in memory with ArrowWriter containing a map with struct keys and a map with list keys (including an empty map and an empty-list key), then reads it back with the record API and checks the rows. Generating the file in the test avoids depending on apache/parquet-testing#87 and addresses the review feedback on #7769 (descriptive test name, only the relevant columns). The test panics without the fix.

Are there any user-facing changes?

Files with non-primitive map keys can now be read with the record API instead of panicking. No API changes.

AI usage: this change was prepared with the help of AI assistants (Claude, OpenAI models). I reviewed the change and the reader's key/value level handling, and verified the test fails before and passes after the fix.

@github-actions github-actions Bot added the parquet Changes to the parquet crate label Sep 27, 2026

@etseidl etseidl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

@etseidl

etseidl commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

doc failure should be fixed by #11305

@etseidl etseidl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me. Thanks for picking this up @hsdfat.

@Jefffrey Jefffrey added the enhancement New feature/addition label Oct 2, 2026
@Jefffrey
Jefffrey enabled auto-merge October 2, 2026 04:50
@Jefffrey
Jefffrey added this pull request to the merge queue Oct 2, 2026
@Jefffrey

Jefffrey commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

thanks @hsdfat & @etseidl

Merged via the queue into apache:main with commit 1514ea4 Oct 2, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature/addition parquet Changes to the parquet crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unable to parse parquet files with maps with complex keys

3 participants