Skip to content

Reading parquet with (pre-release) arrow fails with "out of order projection is not supported" #2543

Description

@alamb

Describe the bug

When testing using a pre-release version of arrow-rs with apache/arrow-rs#1682 from @tustvold which added stricter error checking two datafusion tests fail with "out of order projection is not supported"

---- physical_plan::file_format::parquet::tests::evolved_schema_filter stdout ----
thread 'physical_plan::file_format::parquet::tests::evolved_schema_filter' panicked at 'called `Result::unwrap()` on an `Err` value: ArrowError(ExternalError(ParquetError(General("out of order projection is not supported"))))', datafusion/core/src/physical_plan/file_format/parquet.rs:968:14

---- physical_plan::file_format::parquet::tests::evolved_schema_inconsistent_order stdout ----
thread 'physical_plan::file_format::parquet::tests::evolved_schema_inconsistent_order' panicked at 'called `Result::unwrap()` on an `Err` value: ArrowError(ExternalError(ParquetError(General("out of order projection is not supported"))))', datafusion/core/src/physical_plan/file_format/parquet.rs:819:14

To Reproduce
See #2530

Expected behavior
Tests should succeed

Additional context
@tustvold suggests there is some interplay of the parquet schema adapter logic that was silently masking a bug in arrow-rs. Now that arrow-rs has stricter error checking, the query is erroring #2530 (comment)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions