Skip to content

Commit 6286361

Browse files
dfa1claude
andcommitted
fix(csv): disable globalDict by default to prevent OOM on large files
globalDict=true buffers every column's raw data across all chunks in memory until close() — for 1B rows that's O(total rows) heap regardless of chunkSize. CSV import is streaming so globalDict must be off by default. Per-chunk dict encoding still runs inside each chunk. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 0b6784b commit 6286361

1 file changed

Lines changed: 6 additions & 1 deletion

File tree

csv/src/main/java/io/github/dfa1/vortex/csv/ImportOptions.java

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,13 @@ public record ImportOptions(
1212
ProgressListener progressListener,
1313
WriteOptions writeOptions
1414
) {
15+
/// Default options.
16+
///
17+
/// Global dictionary is disabled because CSV import is streaming — enabling it
18+
/// would buffer every column's raw data across all chunks until close, consuming
19+
/// O(total rows) heap. Per-chunk dict encoding still applies inside each chunk.
1520
public static ImportOptions defaults() {
16-
return new ImportOptions(',', 65_536, true, null, null, WriteOptions.cascading(3));
21+
return new ImportOptions(',', 65_536, true, null, null, WriteOptions.cascading(3).withGlobalDict(false));
1722
}
1823

1924
/// Override the inferred schema. The struct's field names become column names;

0 commit comments

Comments
 (0)