Commit 6286361
fix(csv): disable globalDict by default to prevent OOM on large files
globalDict=true buffers every column's raw data across all chunks in
memory until close() — for 1B rows that's O(total rows) heap regardless
of chunkSize. CSV import is streaming so globalDict must be off by default.
Per-chunk dict encoding still runs inside each chunk.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>1 parent 0b6784b commit 6286361
1 file changed
Lines changed: 6 additions & 1 deletion
Lines changed: 6 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
15 | 20 | | |
16 | | - | |
| 21 | + | |
17 | 22 | | |
18 | 23 | | |
19 | 24 | | |
| |||
0 commit comments