Iceberg + REST Catalog support #2242

Activity

1. ihistand commented on Aug 21, 2026

   

   ihistand

   on Aug 21, 2026

   More actions

   The four-part naming is real, but for read-only sources there's a workaround that works today with no core change.

   resolveTarget wraps the whole dotted identifier in a single pair of backticks
   (core/compilation_sql/index.ts), and nothing validates dots inside the dataset part, so a
   declaration can carry catalog and namespace together:

   -- definitions/my_table.sqlx
   config {
     type: "declaration",
     database: "my-project",
     schema: "my-catalog.my-namespace",
     name: "my-table"
   }

   compiles to `my-project.my-catalog.my-namespace.my-table`, which is exactly the P.C.N.T
   form. Same idea in actions.yaml, where the keys are project and dataset, or via
   declare({ ... }) in a JS file.

   Caveats: the action's readable name is now a four-dot string, so --actions selectors and any
   --schema-suffix / schema override will treat it oddly. It's fine for sources you only ref();
   I wouldn't lean on it for anything else. (I read this off the compiler rather than running it
   against a live catalog, so worth a quick check on your side.)

   On the incremental-target half, I don't think that one is Dataform's to solve. BigQuery has
   read-only access to tables through the Iceberg REST catalog endpoint, so there's no DML or CTAS
   for Dataform to emit. Anything Dataform materializes has to go through an engine that can write.

   It's also worth separating the two things being recommended: the REST catalog endpoint is
   recommended over the older Custom Apache Iceberg catalog for BigQuery endpoint - catalog
   endpoint vs catalog endpoint. BigQuery-managed Iceberg tables aren't the deprecated side of that
   comparison, and they're the writable ones. So bigquery.iceberg isn't a dead path for tables you
   want Dataform to build; it's the only path where BigQuery can do the writing.
2. derekperkins commented on Aug 21, 2026

   

   derekperkins

   on Aug 21, 2026

   Author

   More actions

   Thanks for the response and the query workaround. BigQuery has supported DML in preview for the REST catalog since June. Since my primary use case for dataform is data pipelines DML, I'm hoping support can land. I'll try out your combo schema workaround to see if that works for that.

   

   https://docs.cloud.google.com/bigquery/docs/release-notes#June_25_2026