# Databricks > The databricks source and destination: SQL warehouses, Volumes and workspace files. Databricks SQL warehouses, Unity Catalog Volumes and workspace files: the `databricks` source and `databricks` destination, from the `databricks` package. Declare the package once in `dependencies.yml`; DRE installs it on demand: dependencies.yml ```yaml plugins: - databricks ``` ### `databricks` source: profile fields Set in `profiles.yml`, under `connections:`, in a target with `type: databricks`. Field Default Notes `host` (required) Workspace host, e.g. adb-123.4.azuredatabricks.net. `http_path` (required) The SQL warehouse’s HTTP path, e.g. /sql/1.0/warehouses/abc. `auth_type` `pat` Pat (a token) or oauth (browser sign-in; with client\_id and client\_secret, a service principal) One of `auto`, `pat`, `token`, `oauth`. `token` Personal access token, for auth\_type pat. Secret: use `env_var()`. `client_id` OAuth client; a service principal’s application ID (browser sign-in defaults to databricks-cli) `client_secret` Service principal OAuth secret. Secret: use `env_var()`. `catalog` Default catalog. `schema` Default schema. `profile` A ~/.databrickscfg profile to sign in with (auth\_type auto) `scopes` OAuth scopes. `redirect_port` The localhost port browser sign-in redirects to (default 8020) `retry_timeout` Seconds to keep waiting while a stopped warehouse starts (default 900) `retries` `3` How many times to try again after a temporary error (0: never): connecting, or a Volume or workspace upload. ### `databricks` destination: profile fields Set in `profiles.yml`, under `destinations:`, in a target with `type: databricks`. Field Default Notes `host` (required) Workspace host, e.g. adb-123.4.azuredatabricks.net. `auth_type` `pat` Pat (a token) or oauth (browser sign-in; with client\_id and client\_secret, a service principal) One of `auto`, `pat`, `token`, `oauth`. `token` Personal access token, for auth\_type pat. Secret: use `env_var()`. `client_id` OAuth client; a service principal’s application ID (browser sign-in defaults to databricks-cli) `client_secret` Service principal OAuth secret. Secret: use `env_var()`. `profile` A ~/.databrickscfg profile to sign in with (auth\_type auto) `scopes` OAuth scopes. `redirect_port` The localhost port browser sign-in redirects to (default 8020) `retries` `3` How many times to try again after a temporary error (0: never): connecting, or a Volume or workspace upload. ### `databricks` destination: options Set in the report’s `output.destination` entry. Option Type Default Notes `if_exists` string `overwrite` When a file is already at the path: `overwrite` it, fail with `error`, or `number` the new one. One of `overwrite`, `error`, `number`. ## As a source Field Notes `host` Workspace host. `http_path` The SQL warehouse’s HTTP path. `auth_type` `auto` (default), `pat` or `oauth`. `token` A personal access token, or any other bearer token. Optional with `auto`. `profile` A `~/.databrickscfg` profile to sign in with (for `auto`). `client_id` For `oauth`: the OAuth client. Browser sign-in defaults to `databricks-cli`, which every workspace has. For a service principal, its application ID. `client_secret` For `oauth`: a service principal’s OAuth secret. Without it, `oauth` signs you in through the browser. `scopes` For `oauth`: default `all-apis offline_access` for browser sign-in, `all-apis` for a service principal. `redirect_port` For browser sign-in: the localhost port the sign-in redirects to. Default 8020, which is what `databricks-cli` allows. `catalog`, `schema` Defaults for the session. `retries` How many times to try connecting again after a dropped connection, default 3 (see [Tries again](https://getdre.com/docs/plugins/#tries-again)). The destination uses it for uploads too. `retry_timeout` Seconds to keep waiting while a stopped warehouse starts. Default 900. While it waits, DRE says so every 30 seconds. A host that doesn’t resolve, or refuses the connection, fails at once. ```yaml connections: warehouse: targets: dev: # you, through the browser type: databricks host: dbc-123.cloud.databricks.com http_path: /sql/1.0/warehouses/abc auth_type: oauth prod: # a service principal, for the orchestrator type: databricks host: dbc-123.cloud.databricks.com http_path: /sql/1.0/warehouses/abc auth_type: oauth client_id: "{{ env_var('DATABRICKS_CLIENT_ID') }}" client_secret: "{{ env_var('DATABRICKS_CLIENT_SECRET') }}" ``` With `auth_type: auto` (the default) most setups need no sign-in fields at all. DRE uses, in order: 1. `token` in the profile, or `client_id` + `client_secret` (a service principal); 2. whatever Databricks’ own tools would use, through Databricks’ Go SDK: `DATABRICKS_TOKEN`, or `DATABRICKS_CLIENT_ID` + `DATABRICKS_CLIENT_SECRET`; a `~/.databrickscfg` profile (`profile:`, or `DATABRICKS_CONFIG_PROFILE`); a `databricks auth login` session; the VS Code extension; CI OIDC tokens (GitHub Actions, Azure DevOps); Azure and Google credentials; 3. DRE’s own saved sign-in, then a browser sign-in if a person is at the terminal. When no one is at the terminal (a scheduler, CI, a Databricks job), DRE never waits for a browser: it fails at once and lists what would work. In a Databricks job, give it `DATABRICKS_TOKEN` or a service principal. DRE signs in only when a report actually uses the profile: a connection when the first query on it runs, a destination when it delivers. Browser sign-in opens your browser the first time and saves the session in `~/.dre/oauth_sessions.json`, which only you can read. The file has one entry per workspace and OAuth client, so a report can read from one workspace and deliver to another, and the `databricks` source and destination share one sign-in per workspace. After that the refresh token renews the session, and the browser only opens again once the refresh token stops working. Delete the file (or its entry) to sign out. Set `DRE_NO_BROWSER=1` to only print the sign-in URL. Service principal tokens stay in memory. With either kind, the access token is renewed before it expires, so a long run keeps its session. Passwords, tokens and client secrets are never saved: put them in environment variables and use `env_var()`. Capabilities: `sessions`, `check` (via `EXPLAIN`), `load`. The plugin holds a real warehouse session, so temp views and `SET`s last for the whole Binding. The session runs in UTC. Warehouses have no read-only mode. The `databricks` package is written in Go, on Databricks’ official Go connector (`databricks-sql-go`): SQL warehouses only hold sessions for Databricks’ own clients, and the connector identifies itself with `dre` appended. One program serves as this source and the `databricks` destination. Set `DATABRICKS_LOG_LEVEL=debug` to see the connector’s own log. Databricks SQL reads backslashes as escapes in string literals and doesn’t read `''` as an escaped quote: `'O''Brien'` is two literals, `'O'` and `'Brien'`, which Databricks joins into `OBrien`. Jinja that builds literals from values should escape for Databricks (`'O\'Brien'`); `dre_utils` does this through `dispatch()`, and lookups are inlined portably. `VARIANT`, `STRUCT`, `ARRAY` and `MAP` columns arrive as compact JSON text (from `databricks` 1.2.0; before, `STRUCT`, `ARRAY` and `MAP` were passed on as nested Arrow), intervals and geography as text. See [Types from warehouses](https://getdre.com/docs/plugins/#types-from-warehouses). ## As a destination Unity Catalog Volumes and workspace files, chosen by the path. `host` and the same sign-in fields as the `databricks` source (`auth_type`, `token`, `client_id`, `client_secret`, `profile`, `scopes`, `redirect_port`), so one set of credentials, and one OAuth session per workspace, serves both. It’s the same program as the source. ```yaml destinations: lakehouse: targets: prod: {type: databricks, host: dbc-123.cloud.databricks.com} ``` - **`/Volumes////...`**: uploaded to the Volume through the Files API. Missing directories under the volume are created. Use it anywhere, for any size of file. - **`/Workspace/Users//...`, `/Workspace/Shared/...` or `/Workspace/Repos/...`** (the `/Workspace` prefix is optional): a workspace file, for outputs people open from the workspace browser, next to notebooks and dashboards. Missing folders are created, the file replaces one already at the path (unless `if_exists` says otherwise), and it’s always a plain file: a `.sql` or `.py` output isn’t turned into a notebook. Workspace files are meant for small files (the import API takes up to about 10 MB); use a Volume for large outputs. On Databricks compute, where `/Volumes` and `/Workspace` are mounted, the file is copied there directly instead: no API call and no sign-in, with the job’s own access. The same report works outside Databricks (a laptop, Airflow, CI), where it uploads, and in a Databricks job or cluster. Takes `if_exists` (see [A file already at the path](https://getdre.com/docs/plugins/#a-file-already-at-the-path)): the Files and Workspace APIs’ `overwrite=false` refuses a taken name in the same step. ## Verify a warehouse, then add report SQL The dev profile above is a minimal browser-authenticated warehouse connection. Copy the [tutorial](https://github.com/get-dre/dre/blob/v0.5.1/examples/tutorial/) structure, declare `databricks`, and start the result query with `select 1 as connection_check`. `dre validate --live ` validates its statement; `dre run --preview 5` checks the formatted output without uploading it. For a realistic workbook, adapt [monthly finance](https://github.com/get-dre/dre/blob/v0.5.1/examples/monthly-finance/) to existing warehouse tables and replace any DuckDB-specific SQL. A Volume destination on that workbook needs `destination: {profile: lakehouse, path: /Volumes/main/reporting/reports/monthly.xlsx}` under its output, and a matching destination target entry. `connection-failed` needs the workspace host, warehouse HTTP path and warehouse permission checked. A browser login cannot run unattended; configure the job’s supported identity. `delivery-failed` on a Volume needs access to that catalog, schema and volume, even when SQL warehouse queries succeed. `missing-target-entry` means the connection or destination lacks the selected target. See [profiles](https://getdre.com/docs/reference-profiles/), [destination keys](https://getdre.com/docs/reference-report/#outputdestination), [error codes](https://getdre.com/docs/reference-error-codes/), and [target](https://getdre.com/docs/glossary/#target). [Edit page](https://github.com/get-dre/dre/edit/master/docs/plugin-databricks.md) [Previous PostgreSQL](https://getdre.com/docs/plugin-postgres/)[Next BigQuery](https://getdre.com/docs/plugin-bigquery/)