Skip to content
You're reading the docs for DRE 0.12 (v0.12.2). This page for 0.13

Concepts

New to a term? The glossary defines DRE’s vocabulary and differences from dbt.

  • Report: one or more Jinja-templated SQL queries plus an output config, declared in YAML. A .sql file under reports/ that no YAML lists is an error, unless allow_unmanaged: true in dre_project.yml lets it run as an unmanaged report, for quick tests (breaking in 0.12).
  • Set and Binding: one report can run as many named variants (clients, regions, departments). A Binding is a report paired with a Set, with its own profile, variables, query subset and output.
  • Connections, destinations, sources and the target: a connection is what queries read from and a destination where output goes, both in profiles.yml. A source is a declared table (dbt’s sources:), read with {{ source('sales', 'orders') }}. A target is an environment (dev, prod): each profile has an entry per target and picks one per run (--target, DRE_TARGET, else its own target:, else dev). Each query runs on its own connection when it names one (profile: on the query, or a source’s), else on the report’s, so one workbook can hold DuckDB, Postgres and Databricks tabs. See connections and sources.
  • Jinja everywhere: SQL, paths and options render with var(), env_var(), run.* and your macros in macros/. connection.* (the query’s connection) and profile('name') read connection settings, so names can follow the environment: {{ connection.catalog }}.{{ connection.schema }}.orders. run.date is a date you can navigate (run.date.prev_month.start.date), in the run’s timezone (UTC unless you set one). run_query() and columns() let a macro query the query’s own connection while rendering. ref('file') reuses another .sql file as a subquery. See templates.
  • Lookups: mapping tables you maintain as files in lookups/ (csv, xlsx, xls, json, jsonl, yml) rather than in the database. ref('countries') makes one usable like a table: small ones are inlined into the SQL, larger ones (over 200 rows by default) are loaded into a temp table by the source plugin. lookup('countries') hands the rows to Jinja. Values are text unless a lookups/<name>.yml config gives columns types; see lookups.
  • Plugins: every source, format and destination is a plugin that speaks DRE’s plugin protocol. Plugins ship in packages, one per system: databricks is the Databricks source and destination, object_store is S3, GCS and Azure Blob. A project declares each package once and DRE installs it on demand.
  • Delivery: one output can go to several destinations in a single run, e.g. object storage (S3, GCS, Azure Blob), SFTP/FTP, Databricks Volumes or workspace files, an email with the file attached, or a Slack channel. See plugins.
  • Outputs and messages: a report can have several outputs (output: as a list), all made from one run of its queries, each from its own queries:. A message output turns the results into a short headline posted to Slack, Teams, Google Chat or an email body; when: sends an output only when the data calls for it. See messages.
  • Data tests: query-level checks return failure rows. dre run skips them, dre test runs fresh queries and checks without report formatting or delivery, and dre build checks before producing reports. Local checks inspect saved rows through core-owned embedded DuckDB; source checks inspect an explicit relation in the query’s existing database session. See data tests.
  • Logs: each dre run, dre test or dre build of a report and Set writes dre.log in its own run folder (target/run/<report>/<set>/runs/<run id>/), including the full SQL of each statement, the run’s settings and every step; it goes when the run is pruned (keep_runs). Other commands write to the console only. --log-format json prints the same events as JSON lines.
  • Verification: dre validate checks the project and compiles its SQL; with -s it also shows, per selected Binding, the compiled files, the target, each query’s connection, the output file and every destination (non-dev targets stand out). dre compile just renders the SQL into target/compiled/ and lists the files. dre validate --live checks every statement against the database. --preview and schema-drift detection check a report before it reaches anyone.
  • Generated files: everything DRE writes goes in the target path, target/ in the project by default: compiled SQL, each run’s output files, schema snapshots, run_results.json and the manifest. --target-path, DRE_TARGET_PATH or target_path: in dre_project.yml move it (see below).
  • Selecting: run, test, build, compile, validate and ls take report names, tag:<tag>, source:<source> or source:<source>.<table>, folder names or dotted folder paths, as arguments (dre run daily monthly) or with -s/--select. Several match any of them: -s daily monthly, -s daily,monthly, or repeated -s (a semicolon works too, quoted: -s "daily;monthly"). Since 0.12 every term must match a report, and a bare name that picks different reports as a report name, a tag and a folder is an error rather than a guess: write report:<name>, tag:<tag> or folder:<dotted path>. A report in a folder of its own name, alone, is one reading.