You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
perf(images): consider a preloaded C allocator for the chtypes ingest path (after #712) #824
Not actionable yet. This is follow-up work for #712's move to chtypes v2. Measure it with WaveHouse's own ingest benchmark before deciding.
Context. After #712, WaveHouse's ingest CPU is dominated by the chtypes library call (chs_preview_batch). A profile of WaveHouse's ingest path on linux-arm64 put about 29% of the library's time in libc malloc/free/mem*, from per-row allocation inside the vendored ClickHouse reader.
Measured by chtypes on WaveHouse's request shape (100-row JSONEachRow calls; documents identical in every case):
Preloading a different C allocator into the process (LD_PRELOAD):
jemalloc
mimalloc
tcmalloc
arm64, 1 thread
−18%
−20%
−18%
amd64, 1 thread
−13%
−19%
−23%
arm64, 8 threads
−11%
−15%
−3%
No measurable cost with several ClickHouse-line libraries loaded at once.
GLIBC_TUNABLES=glibc.rtld.optional_static_tls=262144: arm64 −5% at 1 thread and about −2% at 8; nothing on x86-64.
Trade-offs to weigh:
LD_PRELOAD replaces the whole process's C allocations, not just the library's. In a Go process that is mostly chtypes, but any other cgo code is affected too.
It adds a shared library to the images (deployments/Dockerfile*), and a new dependency to track.
Activity