Skip to content

bug(api): NDJSON ingest silently drops every object after the first on a line #398

Open

Description

@taitelee

Summary

The NDJSON line reader decodes one object per line with a single dec.Decode and no trailing-token check, so {"a":1}{"a":2} on one line ingests the first object and silently discards the rest.

Detail

internal/api/record_reader.go:148-161 (lineReader.Next): per non-blank line it does one dec.Decode(&m) and returns, never calling dec.More() / a second decode. A client bug that drops a newline between two records produces one line with two objects; the batch response reports succeeded:1 and the second record is lost with a 200. The admin query handler (internal/api/query.go:181) explicitly guards this exact class ("a real risk if a buggy client double-encodes") — the ingest path doesn't.

Fix direction

After decoding, reject a line with trailing non-whitespace tokens (dec.More() or a follow-up decode expecting EOF) as a per-record syntax error, so the loss surfaces instead of passing silently.

Found in a repo-wide audit; verified by code trace.

Activity

  1. added
    bugSomething isn't working
    area/apiHTTP handlers, routing, middleware
    area/ingestIngest pipeline (Bento, batching, DLQ)
    on Jul 8, 2026
  2. coderabbitai commented on Jul 8, 2026

    @coderabbitai
    🔗 Related PRs

    #182 - refactor: full api --> ingest --> clickhouse --> dlq refactor [closed]


    🧪 Issue enrichment is currently in open beta.

    You can configure auto-planning by selecting labels in the issue_enrichment configuration.

    To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

    issue_enrichment:
      auto_enrich:
        enabled: false

    💬 Have feedback or questions? Drop into our discord!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/apiHTTP handlers, routing, middlewarearea/ingestIngest pipeline (Bento, batching, DLQ)bugSomething isn't working

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions