Problem
A metric has one definition, but there is often more than one physical table that could answer a given question about it. DJ already chooses between candidate tables (e.g., you request metrics and dimensions and routing picks the cheapest source that covers the request). Today, though, those candidates can only relate to each other in one way: as pre-aggregations of a common parent.
Pre-aggregations are best for covering rollups, where any derivation of an existing node can be plugged in and roll-ups can route to it. This proposal is addressing a different use case, where several tables may be at the same grain and have different dimensions, none derived from any other:
orders_daily → customer_dim
orders_enriched_daily → customer_dim, product_dim
orders_by_region_daily → customer_dim, region_dim
Here is nothing to nominate as the source-of-truth node that you would point the metric at. Declaring one and registering the rest as its pre-aggregations is assuming a derivation that does not exist, and picks arbitrarily between equals. Furthermore, a pre-aggregation's grain resolves through its parent's dimension graph, and dimension_columns only rebinds a dimension already listed. In short, pre-aggregations don't plug into the rest of the dimensions graph (they're not meant to be first-class semantic entities).
Proposal
Let a metric declare one query per source:
name: default.completed_orders
node_type: metric
queries:
- SELECT SUM(is_completed) FROM default.orders_daily
- SELECT SUM(completed_flag) FROM default.orders_enriched_daily
- SELECT SUM(CASE WHEN status = 'COMPLETED' THEN 1 ELSE 0 END)
FROM default.orders_by_region_dailyEach source points to an ordinary node, so its dimension links, availability, partitions, and its own pre-aggregations all come for free. Ties are broken by declaration order.
Semantics
Routing
For a given request, keep the sources whose dimension graph reaches every requested dimension and contain the required measures. Then choose among the survivors ordered by fewest joins, breaking ties with declaration order in the YAML.
Source selection happens above pre-aggregation matching. We pick a source and then let the existing matcher pick a pre-agg beneath it.
When no source qualifies, the message should say why each candidate was eliminated (dimension unreachable, measure absent).
Dimension Availability
Two dimensions can each be available on some source while no single source carries both. region and product in the example above are exactly this. The available-dimensions API must therefore answer per source rather than returning a flattened union, or it will report combinations that cannot be served.
Problem
A metric has one definition, but there is often more than one physical table that could answer a given question about it. DJ already chooses between candidate tables (e.g., you request metrics and dimensions and routing picks the cheapest source that covers the request). Today, though, those candidates can only relate to each other in one way: as pre-aggregations of a common parent.
Pre-aggregations are best for covering rollups, where any derivation of an existing node can be plugged in and roll-ups can route to it. This proposal is addressing a different use case, where several tables may be at the same grain and have different dimensions, none derived from any other:
Here is nothing to nominate as the source-of-truth node that you would point the metric at. Declaring one and registering the rest as its pre-aggregations is assuming a derivation that does not exist, and picks arbitrarily between equals. Furthermore, a pre-aggregation's grain resolves through its parent's dimension graph, and
dimension_columnsonly rebinds a dimension already listed. In short, pre-aggregations don't plug into the rest of the dimensions graph (they're not meant to be first-class semantic entities).Proposal
Let a metric declare one query per source:
Each source points to an ordinary node, so its dimension links, availability, partitions, and its own pre-aggregations all come for free. Ties are broken by declaration order.
Semantics
Routing
For a given request, keep the sources whose dimension graph reaches every requested dimension and contain the required measures. Then choose among the survivors ordered by fewest joins, breaking ties with declaration order in the YAML.
Source selection happens above pre-aggregation matching. We pick a source and then let the existing matcher pick a pre-agg beneath it.
When no source qualifies, the message should say why each candidate was eliminated (dimension unreachable, measure absent).
Dimension Availability
Two dimensions can each be available on some source while no single source carries both.
regionandproductin the example above are exactly this. The available-dimensions API must therefore answer per source rather than returning a flattened union, or it will report combinations that cannot be served.