Repository navigation
bug(ingest): explicit null id_field slips past require_id and dedupes every row onto "<nil>" #370
Description
Area: ingest · api — bug (silent data loss / correctness) · found via PR review (#219 follow-up)
Expected: With dedupe on, a row whose id_field is present but null ({"event_id": null}) is treated like a missing id — counted by wavehouse_ingest_dedupe_missing_id_total and rejected under require_id: true (else published un-deduped).
Actual: internal/api/ingest.go processRecord does idVal, ok := data[h.IDField]; an explicit JSON null decodes to a present key, so ok == true and the record skips the missing-id branch. fmt.Sprint(nil) yields the literal "<nil>", and the dedup key is un-namespaced (internal/dedupe/embedded.go:29, key := []byte(eventID) — no table/scope prefix), so every null-id row across every table collides on the single key "<nil>": the first is ingested, the rest are dropped as duplicates. require_id never fires and the metric never counts them.
Impact: Silent data loss whenever dedupe is on and a producer emits null for the id (common when a serializer writes null for an unset field). The require_id tripwire — whose whole point is guaranteeing a usable id — is defeated, and the omission is invisible in metrics.
Scope: treat a nil value as missing (if !ok || idVal == nil) so null flows through the existing missing-id path; add tests for {"event_id": null} in both require_id modes. Decide in the same fix whether empty-string "" should also count as missing (same collision class). Separate latent issue: fmt.Sprint keys collide across types (123 vs "123").
Related: #219 (the missing-id handling this extends)
Metadata
Metadata
Assignees
Type
Projects
- Status
Activity
coderabbitai commentedon Jul 6, 2026
🔗 Related PRs
#122 - fix(ingest): break infinite-Nak loop on permanent delete errors [merged]
#182 - refactor: full api --> ingest --> clickhouse --> dlq refactor [closed]
🧪 Issue enrichment is currently in open beta.
You can configure auto-planning by selecting labels in the issue_enrichment configuration.
To disable automatic issue enrichment, add the following to your
.coderabbit.yaml:💬 Have feedback or questions? Drop into our discord!
EricAndrechek commentedon Jul 6, 2026
Cross-linking #222 — the cross-table dimension of this shares a root cause with it.
The global
"<nil>"collision described here happens because the dedup key is un-namespaced (internal/dedupe/embedded.go:29,key := []byte(eventID)— no table/scope prefix), which is exactly item 2 of #222 ("namespace the dedupe key by table"). So #222's keyspace fix resolves the cross-table half of this null collision on its own; what stays unique to this issue is then just treating a present-but-null(and missing)id_fieldas "no id" inprocessRecord(if !ok || idVal == nil).Flagging so the keyspace change gets coordinated across both rather than patched twice.