Repository navigation
Support setting BigQuery universeDomain - #3
Open
ybelleguic wants to merge 68 commits into
Open
ybelleguic wants to merge 68 commits into
ybelleguic wants to merge 68 commits into
Conversation
ybelleguic
marked this pull request as ready for review
September 18, 2026 12:54
ybelleguic
force-pushed
the
cursor/bigquery-universe-domain-clean-3190
branch
from
September 18, 2026 13:06
821f551 to
2f09b5c
Compare
* Update transitive deps * Remove vsce * Update js-beautify * Update eslint --------- Co-authored-by: Artur Piotr Izaak Laskowski <apilaskowski@google.com>
cursor
Bot
force-pushed
the
cursor/bigquery-universe-domain-clean-3190
branch
from
September 21, 2026 13:50
b4dc413 to
8bf34e9
Compare
…rm-co#2307) (dataform-co#2319) * style(cli): format execution_sql.ts and execution_sql_test.ts with prettier * fix(cli): hoist DECLARE statements to start of onSchemaChange procedure (dataform-co#2307)
* Remove unused git workflow * trigger CI --------- Co-authored-by: Rafal Hawrylak <hawrylak@google.com>
Runs `prettier --write '**/*.ts'` at the project root with the version pinned in package.json (3.9.4) and the existing `.prettierrc` (printWidth: 100). No behavior changes. Purpose: absorb accumulated formatting drift into a single NFC commit so future behavior PRs are not obscured by reformatting churn.
Ensure that backticks and dollar signs in sqlx sql comments are properly escaped even when preceded by a backslash.
* Bump fast-uri from 3.1.4 to 3.1.8 Bumps [fast-uri](https://github.com/fastify/fast-uri) from 3.1.4 to 3.1.8. - [Release notes](https://github.com/fastify/fast-uri/releases) - [Commits](fastify/fast-uri@v3.1.4...v3.1.8) --- updated-dependencies: - dependency-name: fast-uri dependency-version: 3.1.7 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * Update fast-uri to 3.1.6 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Marcin Biernacik <biernacik@google.com>
cursor
Bot
force-pushed
the
cursor/bigquery-universe-domain-clean-3190
branch
3 times, most recently
from
September 28, 2026 09:51
31a24e9 to
35b5f1a
Compare
BUG=566964693
|
No — the user-facing Cloud docs don’t mention optional |
Bumps [@grpc/grpc-js](https://github.com/grpc/grpc-node) from 1.14.4 to 1.14.5. - [Release notes](https://github.com/grpc/grpc-node/releases) - [Commits](https://github.com/grpc/grpc-node/compare/@grpc/grpc-js@1.14.4...@grpc/grpc-js@1.14.5) --- updated-dependencies: - dependency-name: "@grpc/grpc-js" dependency-version: 1.14.5 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Fixes GHSA-6j4f-fj2g-mc7p, GHSA-qhr7-859c-m2p7 and GHSA-q2hr-2g5m-vwhr (DoS via recursion / quadratic expansion in brace-expansion < 5.0.12). Resolves Dependabot alerts dataform-co#379, dataform-co#380, dataform-co#381.
…ng an empty lineage block (dataform-co#2382)
…test key to ADC (dataform-co#2362) * Migrate BigQuery integration tests from KMS-encrypted test key to ADC * Run integration tests as the impersonated dataform-testing service account * Ask for a service account to impersonate in init-creds * Address review comments
… datasets (dataform-co#2385) * Add randomized suffixes to dataset names; remove stale at test start * Fix JiT integration test tmp dir race: each test owns its resources --------- Co-authored-by: Marcin Biernacik <biernacik@google.com>
…m-co#2384) * feat(protos): add Lakehouse config protobufs and workflow settings mapping * feat(core): support 4-part Lakehouse target normalization and ref/resolve * feat(core): add Lakehouse config support and validations for actions * refactor(core): extract Lakehouse resolution and validation into core/actions/lakehouse.ts * fix(core): apply defaultLakehouseConfig in ref/resolve and reject incomplete Lakehouse refs * fix(core): tighten dependency target parsing and update ref error message * refactor(core): use non-null checks in resolvableAsActionConfigTarget and type ILegacyAssertionConfig.lakehouse
…orm run (dataform-co#2389) * test: Make bigquery search integration test ignore similarly named leftover datasets * Use defaultLocation for BigQuery jobs in dataform run Previously the runner never passed a location to BigQuery jobs, so they ran in the location from .df-credentials.json, ignoring the project's defaultLocation (e.g. 'dataset not found in region EU' for europe-west1 datasets). * fix(cli): Create datasets in defaultLocation and warn on location mismatch * fix(cli): Use defaultLocation for search() and lineage emitter
* Add canary test, building the Core bundle and using it to compile a sample project on GCP-based Dataform to ensure cloud backend does not regress
…form-co#2391) The Data Lineage API ProcessOpenLineageRunEvent endpoint does not ingest job.facets.sql (SQLJobFacet) or run.facets.errorMessage (ErrorMessageRunFacet); Knowledge Catalog instead resolves the executed SQL query and job status from BigQuery via run.facets.externalQuery. Removing these unused facets avoids sending raw SQL text and task error messages in OpenLineage payloads and aligns the CLI payload builder with Cloud Dataform.
cursor
Bot
force-pushed
the
cursor/bigquery-universe-domain-clean-3190
branch
from
October 20, 2026 13:29
5c41b18 to
4bbd7a6
Compare
cursor
Bot
force-pushed
the
cursor/bigquery-universe-domain-clean-3190
branch
2 times, most recently
from
October 20, 2026 14:58
abd2cb0 to
879b829
Compare
Add an optional universe_domain field to the BigQuery credentials proto and plumb it through to the BigQuery and data-lineage clients so the CLI can target non-default universes (e.g. Trusted Partner Cloud). The value is read from .df-credentials.json and can be entered during 'dataform init-creds'. When unset, clients keep defaulting to the Google Default Universe. Adds unit tests covering credential parsing and client construction.
Almost all users are on GDU; the proto field already lets TPC users set universeDomain by hand in .df-credentials.json.
TPC service-account keys include universe_domain. Copy it into .df-credentials.json so the BigQuery client and google-auth agree instead of BigQuery defaulting to googleapis.com.
@google-cloud/bigquery already falsy-checks options.universeDomain, so pass credentials.universeDomain || undefined. Keep the omit-empty guard in the lineage emitter, where gax uses ?? instead.
Protobufjs returns "" for unset strings; tighten the assertion so the test would fail if the value became undefined.
The suite does not exercise BigQueryDbAdapter, so keep it as a sibling rather than nesting it inside that suite.
googleapis.com is the default and would never be set here; give a non-default example instead.
Cloud Data Lineage is unavailable in GDC/TPC. Drop the ineffective LineageClient universeDomain option (apiEndpoint still targets GDU) and skip creating a LineageEmitter when credentials.universeDomain is set.
Keep the existing universe-domain commits, and teach the impersonated auth client from dataform-co#2362 to use the same universe so token exchange and BigQuery agree when both fields are set.
cursor
Bot
force-pushed
the
cursor/bigquery-universe-domain-clean-3190
branch
from
October 21, 2026 08:32
879b829 to
6cf0da3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds an optional
universeDomainon BigQuery credentials so the CLI can connect to a non-default universe (for example a Trusted Partner Cloud) instead ofgoogleapis.com.Rebased onto
dataform-co/dataformmainafter dataform-co#2362.impersonate_service_accountkeeps field 5;universe_domainis field 6.The review for this branch is dataform-co#2323.
Changes
protos/profiles.proto: optionalstring universe_domain = 6on theBigQuerycredentials message.getBigQueryClientOptions()forwardsuniverseDomainto the BigQuery client when it is set. An empty value stays ongoogleapis.com.impersonateServiceAccountis also set, that same universe is passed toGoogleAuthandImpersonated, so the token exchange usesiamcredentials.<universeDomain>.init-credsdoes not prompt for a universe. A JSON key'suniverse_domainis copied onto the credentials so BigQuery and google-auth agree. ADC users setuniverseDomainby hand in.df-credentials.json.skip_reason=unsupported_universewhenuniverseDomainis set. Cloud Data Lineage is not available outside the Google Default Universe, and the lineage client is not given a universe override.cli/README.mdandcontributing.mddocument the optional field.Testing
bazel test //cli:credentials_test //cli/api:dbadapters/bigquery_test //cli/api:lineage/emitter_factory_test //tests/api:credentials.spec