Repository navigation
Ingest dbt manifest.json into Model/Source nodes + DEPENDS_ON lineage #576
Description
What problem does this solve?
dbt computes a resolved dependency graph into target/manifest.json, but it is never ingested, so a project's model and source lineage is invisible to the index even though dbt already calculated it. Model-to-source lineage in particular cannot be reconstructed from source files alone.
Public test bed: dbt-labs/jaffle_shop (run dbt compile to produce target/manifest.json).
Proposed solution
Add an explicit, opt-in MCP tool ingest_dbt_manifest { project, manifest_path } that runs after index_repository against an existing project store:
- Parse
manifest.jsonwith the vendoredyyjson. - Emit
model/seed/snapshotasModel/Seed/Snapshotnodes andsourceasSourcenodes, keyed by dbtunique_id. - Emit each node's
depends_on.nodesasDEPENDS_ONedges (model-to-model and model-to-source).
Zero schema change: reuses cbm_store_upsert_node / cbm_store_insert_edge and the existing DEPENDS_ON edge. Only lineage-bearing resources become nodes; test / analysis / operation are skipped. This is the authoritative dbt path, complementing source-level extraction (#575).
Alternatives considered
- Auto-running ingestion inside
index_repositorywhentarget/manifest.jsonis present: deferred so this adds only an explicit, opt-in tool and leaves default indexing untouched. - Ingesting
test/ analysis nodes by default: skipped to keep the graph lineage-only.
Confirmations
- I searched existing issues and this is not a duplicate.
Activity
test(dbt): regression fixture for ingest_dbt_manifest (DeusData#576)
DeusData commentedon Jun 23, 2026
Thanks — ingesting the resolved
manifest.jsonas the authoritative dbt lineage path (opt-iningest_dbt_manifesttool; Model / Source / Seed / Snapshot nodes +DEPENDS_ON; skipping tests / operations) is the right complement to the source-level extraction in #575.Implemented in your PR #583 — continuing there. Labeling
enhancement/parsing/quality.test(dbt): regression fixture for ingest_dbt_manifest (DeusData#576)
DeusData commentedon Aug 20, 2026
Closing this as not planned in the form proposed — ingesting
target/manifest.jsonthrough a dedicated MCP tool — while the underlying goal has been met from the source side.What shipped instead: #1756 (co-authored with @alexisperinger-ux) reads
{{ ref() }}and{{ source() }}out of the repo's own.sqlmodels during ordinary indexing, producingModelnodes and lineage edges. Combined with theTable/Viewnodes from #1749, "what feeds this model" and "what breaks if I change this" are now answerable in dbt repos without any extra step.Why not the manifest tool (detail on #583, which implemented it well): the indexing pipeline publishes a generation by building a staging database and renaming it over the live one, so anything written into the store from outside the indexer is erased by the next index run. Watched projects reindex automatically in the background, so ingested manifest nodes would disappear at an unpredictable moment with no error. The only externally-authored data that survives today is ADRs, and only because there is explicit capture-and-restore machinery for them.
What the manifest still offers that source parsing cannot, and why this may return: resolved database and schema names, materialization strategy, seeds and snapshots, and the authoritative DAG after macro expansion. If we do take it, the shape would be a pipeline pass that reads
target/manifest.jsonwhen one is present, so the data is rebuilt on every index like everything else in the graph — not a tool that injects into a store the next reindex will overwrite. That is a design conversation rather than a bug, so it belongs in a fresh proposal rather than as an open issue with no plan attached.Thank you @alexisperinger-ux for the issue and the implementation — the reasoning above is largely a product of reviewing your PR carefully.