Skip to content

sdk: keyset pagination drops rows tying the page-boundary value (Go + TS) #452

Open
Open
sdk: keyset pagination drops rows tying the page-boundary value (Go + TS)#452
Labels
area/queryStructured query AST, SQL builderarea/sdkTypeScript SDK (clients/ts/)bugSomething isn't working

Description

@jfwoods
jfwoods
opened on Aug 11, 2026

Keyset pagination in both SDKs builds its cursor from a single column with a strict comparison, so rows that tie on the boundary value are silently dropped.

Problem

Page.Next filters on the first .OrderBy() column only, using gt (asc) or lt (desc) against the last row's value:

  • Go: fetchNextTyped, clients/go/query_builder.go — reads q.state.orderBy[0], emits a strict gt/lt filter.
  • TS: clients/ts/src/query-builder.ts — same shape.

If the last row of a page shares its cursor value with the first rows of the next page, those tied rows are skipped: the filter excludes everything <= (or >=) the boundary. Paginating on a low-cardinality column (page, status, a truncated timestamp) can drop an arbitrary number of rows with no error and no signal — HasMore and Next behave exactly as they do on a clean page.

There is also no Decode-path counterpart: when the cursor column is absent from the projection, pagination ends quietly by design (documented, TS parity). Ties are different — the caller asked for a valid cursor and silently lost rows.

Proposed fix

A composite cursor: filter on (col, tie_breaker) rather than col alone, using a lexicographic comparison. Requires:

  1. A tie-breaker column — the second .OrderBy() if present, otherwise a server-provided unique column.
  2. Wire-format support for a tuple comparison, or an (a > x) OR (a = x AND b > y) expansion in the structured-query filter grammar (internal/query/builder.go).
  3. Matching changes in both SDKs plus a shared wire_cases.json conformance case, so they can't drift.

This is a wire-format change shared by both clients, which is why it isn't being folded into the Go SDK PR.

Interim

Documented as a caveat in docs/src/content/docs/sdk/go/queries.md (and the TS twin): paginate on a column that is unique per row, or accept that ties at a page edge can be dropped.

Related

Surfaced during PR #434 review-thread triage. Same family as the untyped-path float64 precision ceiling already documented on both pages.

Activity

  1. github-project-automation
    added this to WaveHouse Task Boardon Aug 11, 2026
  2. github-project-automation
    moved this to Backlog in WaveHouse Task Boardon Aug 11, 2026

  3. coderabbitai commented on Aug 11, 2026

    @coderabbitai
    coderabbitaiBot
    on Aug 11, 2026 – with coderabbitai
    ⚠️ Possible Duplicate Issue(s)
    🔗 Related PRs

    #177 - feat: caching for local table/scope invalidation [closed]
    #335 - fix(query): enforce ClickHouse resource caps server-side [closed]
    #381 - fix(stream): apply policy row-filter per subscriber on SSE [open]
    #434 - feat(sdk): add Go client SDK with full API-tree parity [open]

    🧪 Issue enrichment is currently in open beta.

    You can configure auto-planning by selecting labels in the issue_enrichment configuration.

    To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

    issue_enrichment:
  auto_enrich:
    enabled: false

    💬 Have feedback or questions? Drop into our discord!

  4. jfwoods
    added 2 commits that reference this issue on Aug 11, 2026

    fix(sdk): clamp Retry-After before overflow; unalias insert-result po…

    0a46251

    docs: finish the TS side of the pagination parity sweep

    4cc229a

  5. EricAndrechek commented on Aug 25, 2026

    @EricAndrechek
    EricAndrechek
    on Aug 25, 2026
    Member

    Consolidating #401 into this issue. Same bug — keyset cursor built from a single column with a strict comparison, so rows tying the boundary value are dropped. This issue is the survivor because it is broader: it covers both SDKs, and it names the wire-format work (tuple comparison or (a > x) OR (a = x AND b > y) expansion in internal/query/builder.go) plus the shared wire_cases.json conformance case that keeps the two clients from drifting. #401 was TS-only.

    But #401 was not a strict subset, so closing it as a plain duplicate would have dropped a real defect. Two things carry over:

    1. A second defect on the same line — the cursor predicate accumulates.

    clients/ts/src/query-builder.ts:257, in _fetchNext:

    const nextBuilder = this._clone({
  filters: [...this._state.filters, cursorFilter],   // <- appends, never replaces
  orderBy: this._state.orderBy,
});

    Each next() keeps the previous page's bound and appends a new one, so page N sends N stacked event_ts < filters; only the tightest binds. Correctness holds — this one doesn't lose rows — but request body size and server-side parse cost grow linearly with pagination depth. Still live on b2eee9d.

    This belongs here rather than in its own issue because it is the same edit: the tie-breaker fix has to rewrite exactly this _clone({filters: …}) call. The requirement is that a fix replaces the cursor bound rather than accumulating it — get that wrong while adding the composite cursor and you stack two predicates per page instead of one.

    2. It has already bitten in production.

    From the WaveHouse-Stats dogfooding log (Stats #57, 2026-07-08): Stats skipped a same-second CI event on "Load more" over the live feed. That is the tie-drop failing in a real consumer, not a constructed case. Useful framing from #401 on when it matters: fine for a recent-activity feed, wrong for exhaustive or audit reads — which is the usage that will actually notice.

    Also from #401: this is the client-side keyset cursor, distinct from the backend-owned server-side cursor follow-up (#274).

    Related: #401 (closed as duplicate of this), #274, #434 (Go SDK, where the Go half of this lands).

    Consolidated by the pm-triage routine on 2026-08-25, on Eric's instruction; line refs verified against b2eee9d.

  6. EricAndrechek
    added
    bugSomething isn't working
    area/sdkTypeScript SDK (clients/ts/)
    area/queryStructured query AST, SQL builder
    on Aug 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/queryStructured query AST, SQL builderarea/sdkTypeScript SDK (clients/ts/)bugSomething isn't working

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions