Answer a filtered vector search's indexed conditions from the index, instead of decoding a record at every visited node

DESIGN.md

@@ -69,6 +69,7 @@ Index of the design notes for the harper core: one line per note, grouped by the
 - [HNSW layers above 0 are for routing only, and must be searched greedily](resources/indexes/DESIGN.md#hnsw-layers-above-0-are-for-routing-only-and-must-be-searched-greedily) — Layers above 0 are searched greedily (`ROUTING_EF`); a full-`ef` search there is a second linear-in-N term.
 - [`efConstruction` and the search-`ef` ceiling both auto-scale with the graph](resources/indexes/DESIGN.md#efconstruction-and-the-search-ef-ceiling-both-auto-scale-with-the-graph) — `efConstruction` and the search-`ef` ceiling scale with graph size; a constant erodes recall in a way no search-side setting recovers.
 - [An approximate index returns at most `ef` rows, so `limit` has to reach it](resources/indexes/DESIGN.md#an-approximate-index-returns-at-most-ef-rows-so-limit-has-to-reach-it) — An approximate index returns at most `ef` rows, so `searchByIndex` passes `offset + limit` as `minResults`.
+- [A filtered vector search may admit from an index key set only if the set holds every predicate match](resources/indexes/DESIGN.md#a-filtered-vector-search-may-admit-from-an-index-key-set-only-if-the-set-holds-every-predicate-match) — An indexed companion condition becomes an HNSW allow-set only when its record predicate rejects every value the index cannot hold.
 - [Derived-index runtime: committed-log delivery to native index backends (`resources/derivedIndexRuntime.ts`)](resources/indexes/DESIGN.md#derived-index-runtime-committed-log-delivery-to-native-index-backends-resourcesderivedindexruntimets) — The transaction log is the durable fact; a commit only wakes the lock-elected runner, which resumes every backend from an exact cursor.
 - [Native full-text backend](resources/indexes/DESIGN.md#derived-index-runtime-committed-log-delivery-to-native-index-backends-resourcesderivedindexruntimets) — The native full-text subsection documents Tantivy files as node-local derived state, with one elected writer advancing each index while per-worker readers use publication and reset fencing.
 - [Native HNSW plane: a file-primary mmap graph on the derived-index runtime (`resources/indexes/HierarchicalNavigableSmallWorld.ts`, `resources/indexes/hnswDerivedIndex.ts`, `resources/indexes/hnswPlaneBinding.ts`)](resources/indexes/DESIGN.md#native-hnsw-plane-a-file-primary-mmap-graph-on-the-derived-index-runtime-resourcesindexeshierarchicalnavigablesmallworldts-resourcesindexeshnswderivedindexts-resourcesindexeshnswplanebindingts) — The mmap file owned by `@harperfast/hnsw` is the index; it is the default for audited RocksDB tables, and `nativePlane: false` keeps the JS graph.

resources/DESIGN.md

@@ -113,7 +113,7 @@ One giant `makeTable()` factory that returns a `TableResource extends Resource`
 | How does a URL path map to a Resource?                                         | `Resources.ts → getMatch` (exact/prefix fast path) then `matchParamRoute` (parameterised routes); see "Path routing" below                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
 | What do chained conditions (`a=ge=X&=le=Y`) mean over array values?            | SAME-ELEMENT scoping: `prepareConditions` (`Table.ts → search`) collapses the chain into one range comparator (`gele`/`gtlt`/…) before execution, so the indexed path (per-element index entries, one range scan) and the unindexed path (`search.ts → attributeComparator`, per-element `some` over the collapsed predicate) agree. Repeating the attribute as two independent conditions is independently existential (different elements may satisfy different legs). Only a single `and`-chained leg is supported — `\|=` and a second `&=` are rejected. An indexed scan whose range spans more than one indexed value collapses to one result per record before paging (`search.ts → distinctRecords`, #2434), so `limit`/`offset` count records rather than index entries; element equality stays uncollapsed because `[indexedValue, primaryKey]` is already unique. Pinned by `unitTests/resources/query-array-scoping.test.js`; known gaps: chained-leg values are never type-coerced (#2433), an undeclared (untyped) indexed attribute holding an array still repeats per entry, error paths (#2435).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
 | How does HNSW keep the graph connected on delete?                              | `indexes/HierarchicalNavigableSmallWorld.ts → index()` delete path: zero-degree orphans reindexed via `needsReindexing`; severed multi-node islands detected and reconnected by `repairSeveredNeighbors` (#1712)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
-| How is a filter applied _during_ a vector search?                              | Predicate-aware traversal (#1241): `search.ts → executeConditions` composes companion AND conditions with request `vectorFilter` / `rowFilter` predicates into one `(primaryKey) => boolean` (`composeRecordFilter`) and passes it to `HierarchicalNavigableSmallWorld.search(cond, ctx, { filter })`. The filter gates result admission at layer 0 only (routing ignores it, ACORN-style); a visit budget (`filterExpansion`) bounds the under-filled/selective case. Very selective _condition_ filters are instead diverted to the exact brute-force path by the query planner's `estimateCountAsSort` ordering.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
+| How is a filter applied _during_ a vector search?                              | Predicate-aware traversal (#1241): `search.ts → executeConditions` composes companion AND conditions with request `vectorFilter` / `rowFilter` predicates into one `(primaryKey) => boolean` (`composeRecordFilter`) and passes it to `HierarchicalNavigableSmallWorld.search(cond, ctx, { filter })`. The filter gates result admission at layer 0 only (routing ignores it, ACORN-style); a visit budget (`filterExpansion`) bounds the under-filled/selective case. Companion conditions an index range answers exactly (equality, `gt`/`ge`, `between`, nested AND/OR over those, when the record predicate rejects every value the index cannot supply: a missing value, and a `0` that may have been written as `-0`) are additionally planned into a key-only scan (`planCandidateKeys`, #2688) and passed as `{ candidateKeys }`; the index admits from that key set instead of loading a record per visited node, and drops the predicate entirely when the set covers every pushed-down condition and no `rowFilter`/`vectorFilter` is in play. Very selective _condition_ filters are instead diverted to the exact brute-force path by the query planner's `estimateCountAsSort` ordering.                                                                                                                                                                                                                                                                                                                                                                                                                                   |
 | How does post-ordering resolve vector distances safely?                        | Each comparator owns its `Sort`, passes it directly to the custom-index resolver, and caches distances by that immutable per-query sort object.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
 | How is application row filtering applied?                                      | Authorization admission happens in the resource operation before query work. The legacy `allow*` hook, when armed by the protocol, is evaluated once with its historical receiver semantics; overriding it never changes its scope. An operation override may add indexed conditions and/or attach the JavaScript-only synchronous `target.rowFilter(record, context)`. `Table.search` composes it with query filters and rechecks the final materialized cache/source record. `SubscriptionRequest.rowFilter` covers full-row events; `eventFilter(event, context)` explicitly handles tombstones/messages/raw events. Prefer indexed conditions because an opaque predicate may inspect every admitted candidate and `limit` applies after filtering.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
 

resources/indexes/DESIGN.md

@@ -178,6 +178,33 @@ unit switch does not shift the estimate on its own. The ef term remains the conf
 not the runtime auto-scaled value, so the planner increasingly underestimates vector traversal cost
 as an automatically scaled graph grows.
 
+## A filtered vector search may admit from an index key set only if the set holds every predicate match
+
+`search.ts → planCandidateKeys` turns companion AND conditions a secondary index answers (equality,
+`gt`/`ge`/`lt`/`le`, the `between` family, nested AND/OR over those) into key-only range scans, and the
+HNSW index admits from the keys instead of loading a record per visited node (#2688). A set covering
+every pushed-down condition, with no `rowFilter`/`vectorFilter`, replaces the predicate; otherwise it
+gates the residual predicate. The post-filter re-checks every row, so the set may over-admit but must
+never omit: an omission is unrecoverable and makes results depend on whether the cost model built it.
+
+`planCandidateKeyScan` enforces that by running the condition's own `filterByType` predicate on the
+records an index cannot supply: one lacking the attribute, a null one when the index holds no nulls,
+and a `0`, because a number written as `-0` is indexed under an encoding that sorts past every number
+and decodes to a wrong primary key (ordered-binary; index-led queries miss that row too). A condition
+admitting any of them stays on the predicate path. `compareKeys` ranks undefined below every scalar, so
+this declines `lt`/`le` over a number, string or boolean bound, `ge null` and any range covering 0; the
+`lt`/`le` that remain (a BigInt bound) start at `null`, not `searchByIndex`'s `true`, so indexed nulls
+are read. Normalizing `-0` in the encoder or index writer would let zero-covering ranges plan again.
+
+Building the set is a cost decision: one index-entry read per match against a record load per visited
+node, where filling `ef` matches at selectivity `s` takes about `ef / s` visits. `KEYS_PER_PREDICATE_VISIT`
+is that ratio at the warm, regression-safe end of the measured range; `MAX_CANDIDATE_KEYS` and half the
+node count cap the synchronous scan. A complete set derives the visit budget from its selectivity,
+widening the `filterExpansion` budget by at most `ALLOW_SET_BUDGET_MAX_WIDENING` and never narrowing it;
+a configured `filterExpansion` stays authoritative. `CandidateKeySet` is the one identity rule for both
+the scan's intersection and admission. A node-id bitset for the native plane is not built: the
+pk → node-id lookup costs more than the record load it saves.
+
 ## Derived-index runtime: committed-log delivery to native index backends (`resources/derivedIndexRuntime.ts`)
 
 A derived index (the native HNSW or Tantivy full-text plane) is a materialized view