Migrate Parquet support to Mason package - #5510
Conversation
- Updated `test_checkpoint` to include `uint8` as a valid dtype and adjusted array creation accordingly. - Introduced `test_unknown_entry_type` to validate error handling for unknown checkpoint entry types. - Enhanced parquet tests in `io_test.py` to ensure proper handling of uint8 data types, including round-trip tests for various scenarios. - Removed a known failure case from `parquet_edge_test.py` related to concatenated gzip members due to Arrow v9 correctness issues.
jabraham17
left a comment
jabraham17
left a comment
There was a problem hiding this comment.
Review part 1, I have not yet looked at the Parquet server code changes
There was a problem hiding this comment.
I take it you have now implemented delta encoding? Nice!
There was a problem hiding this comment.
Delta encoding appears to work with the current Arrow, so it seems like expecting an error here is stale? It's somewhat unrelated to this PR though in my opinion, so should I remove this edit?
There was a problem hiding this comment.
this is dependent on arrow version? If you are confident it will work on all supported versions of arrow, I agree the comment is stale and you can leave this in. But if this is arrow version dependent I think you should leave it alone
There was a problem hiding this comment.
If the test suite is run with Arrow v9, this will flag as a failure, correct?
There was a problem hiding this comment.
Yeah, I thought the current version we're using though is 24 (which causes the test to fail since it'll have correct data); should I preserve compatibility with 9?
There was a problem hiding this comment.
The latest version that can be used is 24, but the test suite can (and is) run with older versions. I think the main system where this test runs on a nightly basis is with arrow 9. Sadly, this test does not yet run in the CI.
I would suggest reverting this, or guarding it with a version check
2 participants
Replaces Arkouda's Parquet implementation with the external Mason
Parquetpackage.Arkouda now retains only server-specific responsibilities: message handling, symbol-table conversion, client responses, and Arkouda-specific null processing. File, schema, reader, writer, and distributed Parquet mechanics are delegated to the package.
Changes
ParquetMsgfor existing server modules.Dependency
This PR depends on the companion Mason Parquet PR and should be merged after it.