Skip to content

Goal: SDK call for market volume over a time period #1429

Closed
5 of 5 issues completed
Closed

Description

@vinarmani

Why this was created

The order book query surface covers the book, depth, and best prices, but not activity. get_order_book, get_market_depth, and get_best_prices all answer "what is resting right now". Nothing answers "how much traded".

The data is already on the node. ob_order_events records every fill with price, amount, outcome, participant, and block timestamp, so volume and unique traders are computable without new state. What is missing is an SDK call.

Outcome

Each SDK exposes get_market_activity(query_id, from_ts, to_ts), returning volume and unique traders for a single order book over a time window, in the same shape and at the same level as get_market_depth.

Current state

The metrics are reachable today only by writing the SQL yourself. Kwil nodes accept ad hoc read-only SQL through the user.query RPC and the mainnet gateway serves it, so the figure can be computed against a production node right now, but every consumer re-derives the rule by hand.

Scope

No node change. The SDKs ship one pinned SQL statement run through the ad hoc query RPC, with $query_id, $from_ts, $to_ts as parameters, returning one row per call. The statement is the definition and the three SDKs carry the same text. It has been run against mainnet and cross-checked per event type; the record is on the sub-issues.

If a get_market_activity node action is added later, each SDK swaps its method body to call it with no signature change. That action is node work tracked separately and is not required for this Goal. The index described under Future improvement below is likewise not required, though every sub-issue notes its absence.

Definition to hold fixed

Volume is filled orders, counted once per trade, in cents of that market's own collateral token.

Event Volume Counts as a trader
direct_buy_fill price * amount yes
direct_sell_fill no, it is the same trade yes
mint_fill, burn_fill 100 * amount, YES row only yes
split_placed no, it is a deposit no
placements, cancels, amends, settled no no
  • Mint and burn are volume. They are two-party matches with agreed complementary prices, settled by creating or destroying a pair rather than transferring shares, and they are the arbitrage boundaries that hold YES + NO near 100.
  • split_placed is not volume. It is unilateral: collateral in, pair out, no counterparty and no agreed price.
  • Use an allowlist over event types, never a denylist, so an event type added later is excluded until someone decides it is volume.
  • Volume is always per bridge and never summed across bridges. Base units differ between collateral tokens by orders of magnitude and this layer holds no price feed. Unique traders are comparable across bridges.

Returned fields

One object, because every field falls out of the same pass over the event rows: bridge, volume_cents, direct_cents, mint_burn_cents, unique_traders, fill_count, direct_fill_count, shares_traded, first_event_ts, last_event_ts, coverage_from_block, coverage_complete. The direct / mint_burn split is diagnostic: two markets with identical volume behave differently if one is counterparty trading and the other is one participant working the mint and burn boundary.

Retention

ob_order_events is trimmed once indexed, so the node holds a rolling window. A naive query returns 0 for a market whose events were trimmed, indistinguishable from a market with no activity. The statement therefore returns coverage_from_block, the earliest retained block, and coverage_complete, which compares it to the market's creation height. A zero with coverage_complete = false is truncation, not inactivity.

Sub-issues

In dependency order. Python wraps Go through its C bindings, so Go ships first and is tagged; JS is independent.

  1. Problem: GetMarketActivity in sdk-go sdk-go#205 Problem: GetMarketActivity in sdk-go
  2. Problem: getMarketActivity in sdk-js sdk-js#180 Problem: getMarketActivity in sdk-js
  3. Problem: get_market_activity in sdk-py sdk-py#139 Problem: get_market_activity in sdk-py

Future improvement: index ob_order_events for per-market time windows

Out of scope for this Goal, but worth recording here because all three sub-issues reference it and the SDK call is the first consumer that will feel its absence.

ob_order_events is indexed on id (for the indexer's sequential sync) and block_height (for trim_order_events), and on nothing else. The activity statement filters on query_id and a block_timestamp range, so every call is a sequential scan of the whole retained table. Measured on a synced mainnet node at ~120k rows, a seven-day windowed aggregate ran in about 35 ms, which is fine on its own. The problem is the shape: cost scales with total retained events, not with the window requested. A one-hour window costs the same as a one-year window, every call gets slower as retention grows, and a consumer charting several markets or polling repeatedly multiplies that scan.

The fix is a node migration adding an index on (query_id, block_timestamp). It turns the per-market read into an index range scan whose cost tracks the number of events in the window, and it also serves the existing per-market event queries in 050-order-book-event-queries.sql. Writes are not a concern: the table is append-only with periodic trims, and the index is small relative to the rows.

One decision to make before choosing the column order. (query_id, block_timestamp) serves the per-market case this Goal defines. If protocol-wide windowed reads are also wanted later, such as "all fills in the last hour across every market", a leading block_timestamp serves those range scans better, and the two access patterns want two indexes rather than a compromise on one. Decide which pattern the index is really for rather than guessing.

Because this is a public read that any caller can invoke and that every node executes, the index should land before the SDK call sees heavy use. It needs no change to the SDKs: the pinned statement is unchanged and simply gets faster.

Out of scope

  • Any node change: action, migration, or the index above.
  • Historical metrics beyond the retained window. That belongs to a service that keeps its own copy.
  • Protocol wide totals across bridges.
  • Charting endpoints and bucketed series.

Deadline

ETA: 19-Oct-2026 (Monday)

Activity

  1. self-assigned this
    on Sep 17, 2026
  2. added theissue type on Sep 17, 2026
  3. vinarmani commented on Sep 18, 2026

    @vinarmani
    ContributorAuthor

    Two findings from building the same aggregate outside the node

    Built this analysis as plain SQL against a synced mainnet node to see what the
    action would be doing. Two things came out that are worth folding into the spec.

    1. ob_order_events has no index on block_timestamp

    Indexes today are on id and block_height. Every time-windowed query is
    therefore a sequential scan:

    EXPLAIN ANALYZE
    SELECT query_id, count(*) FROM main.ob_order_events
    WHERE block_timestamp > extract(epoch FROM now())::INT8 - 86400*7 GROUP BY 1;
    
    ->  Parallel Seq Scan on ob_order_events   Execution Time: 34.8 ms   (120,226 rows)
    

    35 ms at 120k rows is nothing. The problem is that it scales with total
    events
    , not with the size of the window requested. A one-hour window costs the
    same as a one-year window, and both get slower as the table grows.

    That is the wrong shape for $from_ts/$to_ts parameters, and it is worse for
    the charting series, which issues many windowed reads.

    (query_id, block_timestamp) serves the per-market case. If protocol-wide
    windowed reads are also wanted, a leading block_timestamp index would serve
    range scans better. Worth deciding which access pattern the action is really
    for before choosing the column order.

    2. Streams are keyed by (data_provider, stream_id), and the collision is real

    main.streams is keyed on the composite, and 564 stream_ids are currently
    shared across more than one provider
    on mainnet:

    SELECT count(*) FROM (
      SELECT stream_id FROM main.streams GROUP BY stream_id HAVING count(*) > 1) x;
    -- 564

    Any implementation that resolves a stream by stream_id alone can silently read
    a different provider's data. It will not error. It will return a plausible
    number.

    Our own scan had exactly this bug. It happened to be correct only because no
    stream_id used by a live market currently collides, which is luck rather than
    design, and it would start returning wrong answers the day a market is created
    on a colliding id.

    Both halves of the key are already in query_components:

    data_provider : ABI word 0, address in the low 20 bytes -> hex chars 25..64
    stream_id     : ABI word 1, ASCII NUL-padded            -> hex chars 65..128
    

    Verified against contractsapi.DecodeMarketData on mainnet order book 1176:
    hex extraction gives 0x1566e5dad82127d2193a504100ec16c17407f3da, the SDK gives
    the same address checksummed.

    Why these two interact

    main.streams also has no index on stream_id, so each text lookup is a
    parallel sequential scan over ~260k rows, about 21 ms. Resolving the stream once
    per market rather than once overall made our scan O(markets x streams).

    Fixing the key and hoisting the resolution were the same change, and together
    took the scan from 2.6 s to 0.285 s, a 9x improvement on 167 live markets.
    The gain grows with market count, since the per-market term disappears.

    If the action resolves a stream at all, resolve the composite once and join on
    streams.id thereafter.

  4. MicBun commented on Oct 7, 2026

    @MicBun
    Contributor

    @holdex goal set-eta 19-Oct-2026

  5. MicBun commented on Oct 8, 2026

    @MicBun
    Contributor

    I'm moving the activity definition into a node migration, as a get_market_activity action, instead of carrying the SQL in each SDK.

    With the SQL in the SDKs, every change to what counts as volume means changing and releasing each SDK, and anyone on an older SDK version keeps getting the old numbers. That separate maintenance would get harder to manage over time. A node action keeps one definition in one place, the way get_market_depth and the other order book reads already work, and each SDK just calls it.

    It also makes a call one request instead of two, about 270 ms instead of 615 ms from my workstation against mainnet, and it works on nodes in Kwil private mode. The index from your future improvement section goes into the same migration.

    Your definition of volume stays exactly as written. The node work is tracked here:

    The sdk-go pull request will switch to the action, and the JS and Python calls will use it from the start. The SDK call works on mainnet once a node release carrying the migration is deployed.

  6. MicBun commented on Oct 16, 2026

    @MicBun
    Contributor

    All five sub-issues are closed, and the call is now in a release of every SDK:

    As I wrote above, the definition lives in the node action get_market_activity, so each SDK only calls it. I ran all three releases against mainnet today, and they report the same volume, traders and fills for the same market.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions