BigQuery

Google BigQuery: the bigquery source, from the bigquery package. Declare the package once in dependencies.yml; DRE installs it on demand:

dependencies.yml

plugins:
  - bigquery

Profile fields

Set in profiles.yml, under connections:, in a target with type: bigquery.

Field	Default	Notes
method	oauth	How to sign in: oauth (gcloud application-default credentials), service-account (a key file), service-account-json, oauth-secrets or external-oauth-wif.
project		The Google Cloud project to read from (dbt’s database also works); required, checked by the plugin.
dataset		Default dataset for unqualified table names (dbt’s schema also works)
location		Where jobs run, e.g. US, EU or europe-west2.
keyfile		Path to a service account key file, for method service-account. A file path; ~/ is your home directory.
keyfile_json		A service account key’s JSON, inline, for method service-account-json. Secret: use env_var().
token		An OAuth access token, for method oauth-secrets. Secret: use env_var().
refresh_token		An OAuth refresh token, for method oauth-secrets with client_id and client_secret. Secret: use env_var().
client_id		OAuth client ID, for method oauth-secrets with a refresh token.
client_secret		OAuth client secret, for method oauth-secrets with a refresh token. Secret: use env_var().
token_uri		Where a refresh token is exchanged (default https://oauth2.googleapis.com/token)
workload_pool_provider_path		The workload identity pool provider, for method external-oauth-wif.
token_endpoint		Where the external identity’s token comes from, for method external-oauth-wif: type, request_url, request_data.
service_account_impersonation_url		The service account to impersonate after workload identity federation.
impersonate_service_account		A service account email to run as, using the signed-in identity.
scopes		OAuth scopes (default: bigquery, cloud-platform and drive)
execution_project		The project jobs run and are billed in, when not project
quota_project		The project API quota is charged to.
priority		Interactive (default) or batch. One of interactive, batch.
maximum_bytes_billed		A job that would bill more bytes than this fails instead of running.
job_execution_timeout_seconds		Stop a query that runs longer than this.
job_creation_timeout_seconds		Give up starting a query after this long.
retries	3	How many times to try starting the session again after a temporary error (0: never)
job_retries		How many times a query that fails with a transient error is run again (default 1)
job_retry_deadline_seconds		Stop retrying a query after this long.
api_endpoint		A BigQuery API endpoint other than Google’s, e.g. Private Service Connect or an emulator.

Notes

An alpha: bigquery is published as 1.0.0 pre-releases (1.0.0-alpha.N), which dre deps installs while there’s no stable release. It is tested against the BigQuery emulator; sessions, load and dry runs need the real service. Field names and values are dbt-bigquery’s, so a dbt profile can be copied across; dbt fields that only matter for building models (threads, Dataproc, gcs_bucket, …) are accepted and ignored.

Field	Notes
method	oauth (default: gcloud’s application-default credentials, from gcloud auth application-default login), service-account, service-account-json, oauth-secrets or external-oauth-wif.
project	Required. The project tables are read from. dbt’s database works too.
dataset	Default dataset for unqualified table names. dbt’s schema works too.
location	Where jobs run, e.g. US, EU, europe-west2.
keyfile	For service-account: path to a key file.
keyfile_json	For service-account-json: the key’s JSON, as a YAML map or a string.
token	For oauth-secrets: an access token.
refresh_token, client_id, client_secret, token_uri	For oauth-secrets: a refresh token and its OAuth client.
workload_pool_provider_path, token_endpoint, service_account_impersonation_url	For external-oauth-wif (Microsoft Entra): token_endpoint has type: entra, request_url and request_data.
impersonate_service_account	Run as this service account, using the signed-in identity.
scopes	OAuth scopes. Default: BigQuery, Cloud Platform and Drive (for Sheets-backed tables).
execution_project	The project jobs run and are billed in, when not project.
quota_project	The project API quota is charged to.
priority	interactive (default) or batch.
maximum_bytes_billed	A job that would bill more fails instead of running. Set on every job.
job_execution_timeout_seconds, job_creation_timeout_seconds	Stop a query that runs, or takes to start, longer than this.
retries	How many times to try starting the session again after a server error, a rate limit or a dropped connection (default 3).
job_retries, job_retry_deadline_seconds	A query that fails with a server error or rate limit runs again, up to job_retries times (default 1) within the deadline.
api_endpoint	A BigQuery API endpoint other than Google’s (Private Service Connect, an emulator). Results are then read over REST only.

connections:
  bq:
    targets:
      dev:        # you, through gcloud
        type: bigquery
        project: my-project
        dataset: reporting
        location: EU
        maximum_bytes_billed: 10000000000
      prod:       # a service account, for the scheduler
        type: bigquery
        method: service-account
        keyfile: /secrets/reporting-sa.json
        project: my-project
        dataset: reporting
        location: EU

Capabilities: sessions, check, load. open starts a BigQuery session and every job of the Binding runs in it, so temp tables last for the whole Binding. check is a dry run: BigQuery validates the statement, and the bytes it would process are logged (-v shows them). load creates a temp table in the session from one SQL statement, with a warning.

Results come through the Storage Read API (Arrow) when the client library uses it, which it does for results larger than one page, and over the REST API otherwise or when the identity can’t create read sessions (bigquery.readsessions.create). Both give the same output.

The package is written in Go, on Google’s official BigQuery client. DRE stores nothing for BigQuery: sign-in goes through Google’s own libraries.

Test the billing and data projects separately

The dev profile above is a minimal application-credential connection with a billing cap. Use the tutorial structure with bigquery declared and a result query select 1 as connection_check. Run dre validate --live <report> before previewing. For business data, adapt the monthly finance workbook to BigQuery tables and dialect; replace any DuckDB-specific SQL.

Use execution_project when jobs should run in a different project from the data, and quota_project for API quota attribution. priority: batch suits scheduled work that can wait; interactive suits immediate previews. maximum_bytes_billed rejects an unexpectedly large scan rather than merely warning.

connection-failed can mean missing application credentials on the runner. query-failed can mean dataset location mismatch, data access denied, or a query beyond the billing cap. Choose the dataset’s location, grant data and job permissions to the actual runner identity, and narrow the SQL before raising a cap. See profiles, error codes and connection.