The manifest and run_results.json

DRE writes two kinds of JSON file into the target path (target/ unless you move it). They read as a pair:

manifest.json says what the project is: every report, Set, Binding, schedule and plugin, as DRE resolves them. One file for the whole project.
run/<report>/<binding>/runs/<run id>/run_results.json says what one run of one Binding did: its vars, rows, files, deliveries and status. A scoped CLI run also records discovered errors outside its selection; its manifest_checksum is null because it did not rebuild a manifest.

Changed in 0.2. Schema 2: a sources section, the run’s target, and each query’s connection and depends_on. Like dbt’s, the manifest is resolved for the run’s inputs. See Upgrading to 0.2.

The manifest (and dre ls --output json, which prints part of it) is DRE’s supported way for orchestrators, CI and other tools to read a project. Read it rather than DRE’s YAML: it has the layering (project < folders < report < Set) and schedule resolution already applied.

The project manifest at a glance

dre compile and dre validate write target/manifest.json: the whole project as DRE resolves it, whatever was selected. It lists every report with its Sets and Bindings (merged vars, queries with their connection and the sources they read, output, destinations), every schedule with the Bindings it runs, the declared sources and plugins, and checksums for change detection. It’s built offline, with no connection or profiles.yml, never contains secrets or connection settings, and is the same byte for byte for the same project and the same inputs (below). An orchestrator can generate one task per schedule from it (each runs dre run --schedule <name>), and CI can compare two manifests to find the reports a change touched. dre ls prints slices of it: dre ls -s tag:regulatory, dre ls --schedule close_monthly --output json. See its JSON Schema.

When it’s written

dre compile and dre validate write manifest.json right after the project loads, before plugin option checks and compilation. It always describes the whole project, whatever was selected. A later error still leaves that manifest.

From DRE 0.9, dre run loads its selection without building the whole manifest. It leaves an existing manifest unchanged, including when the project cannot load. Integrations should use dre validate --json or dre compile when they need a current full-project manifest. A manifest left by an earlier command can be stale; a run’s null manifest_checksum makes no claim about it.

A report with problems (a missing query file, a bad option, a query whose parse pass fails or whose connection doesn’t resolve) is listed with "valid": false and its errors; the manifest is still written. dre validate still fails. valid covers the checks made when the project loads; the ones that need plugins or rendering (option checks, compiling) come after the manifest is written and are reported by dre validate itself.
A project that can’t load at all (no dre_project.yml, unreadable YAML) writes no manifest and removes an old one, so a stale file is never taken as current.
The file is written to a temporary name and renamed, so a reader never sees half of it.
dre ls never writes it; dre clean removes it with the rest of the target folder.

It’s built offline: no database connection, no profiles.yml and no plugins are needed, so it works on a fresh CI runner with no credentials.

Resolved for the run’s inputs

Like dbt’s, the manifest records values resolved from the run’s inputs: the run’s target (--target, DRE_TARGET, else dev), --var and the project’s vars, environment variables, and run.* (DRE_RUN_DATE, DRE_RUN_AT, --timezone). Jinja in profile: values and in source fields decides each query’s connection, a destination’s profile and a source’s schema, so two targets can give two manifests. The same project with the same inputs gives the same bytes. CI that compares manifests should build both with the same target and vars.

What’s in it

{
  "schema_version": "dre/manifest/v3",
  "version": "0.2.0",
  "project": {"name": "acme_reports", "target": "dev", "default_profile": "warehouse", "timezone": "UTC", "checksum": "…"},
  "reports": {
    "monthly": {
      "name": "monthly",
      "managed": true,
      "file": "reports/finance/monthly/monthly.yml",
      "folder": ["finance", "monthly"],
      "tags": ["regulatory"],
      "default_set": "client_a",
      "queries": [{"query": "m", "file": "reports/finance/monthly/m.sql", "tab": true}],
      "depends_on": {"sources": ["sales.orders"]},
      "checksum": "…",
      "valid": true,
      "bindings": [
        {
          "set": "client_a",
          "profile": "warehouse",
          "vars": {"client": "client_a", "region": "emea"},
          "queries": [
            {"query": "m", "file": "reports/finance/monthly/m.sql", "tab": true,
             "connection": "warehouse", "depends_on": {"sources": ["sales.orders"]}}
          ],
          "output": {"format": "csv", "options": {}},
          "destinations": [{"profile": "inbox", "path": "out/monthly-{{ run.date.yyyymmdd }}.csv"}],
          "outputs": [{"format": "csv", "options": {},
                       "destinations": [{"profile": "inbox", "path": "out/monthly-{{ run.date.yyyymmdd }}.csv"}]}],
          "schedules": ["close_a"]
        }
      ]
    }
  },
  "schedules": {
    "close_a": {
      "name": "close_a",
      "report": "monthly",
      "set": "client_a",
      "schedule": {"cron": "0 6 1 * *"},
      "vars": {},
      "bindings": [{"report": "monthly", "set": "client_a"}]
    }
  },
  "sources": {
    "sales": {
      "name": "sales",
      "file": "sources/shop.yml",
      "profile": "warehouse",
      "schema": "main",
      "tags": [],
      "meta": {},
      "tables": {
        "orders": {
          "name": "orders", "identifier": "raw_orders", "tags": [], "meta": {},
          "quoting": {"database": false, "schema": false, "identifier": false},
          "columns": [{"name": "id", "data_type": "bigint"}],
          "used_by": ["monthly"]
        }
      }
    }
  },
  "plugins": [{"package": "duckdb", "version": "*", "source": {"type": "registry"}}]
}

schema_version: the format’s version (see Versioning). version: the DRE that wrote it.
project: its name, the run’s target (target.name, never a profile’s own default), the default connection (as written), timezone:, and the project-wide checksum.
reports, by name: whether it’s managed (declared in YAML) or a bare .sql, its defining file, folder segments, tags, timezone, default Set, queries (with tab settings and column options, and a query’s own profile: as written), every source any Binding reads (depends_on.sources), checksum, validity, and its Bindings.
Each Binding: its Set (null for a report without Sets), the inherited connection (rendered), fully merged vars, queries (each with the connection it runs on and depends_on.sources, from the parse pass), outputs in declared order (name, format, the queries it formats, when, options, extension, template file, and its destinations: rendered profile name and the path template, unrendered), and the schedules that run it. output repeats the first output and destinations lists every output’s destinations, as before 0.3.
schedules, by name: the report or selector and Set it targets, its timing (cron, every or rrule with starting, at, except and also; a shared timing’s fields, with its name in timing), whether it’s enabled, its vars and timezone, and the Bindings it runs, resolved the way dre run --schedule <name> resolves them. To know when each one fires, with the exact command for each firing, use dre schedule ls.
sources, by name: each declared source with its rendered profile, database and schema, description, tags and meta, and its tables (rendered identifier, the quoting that applies, declared columns, and used_by, the reports that read it; empty when unused). source() names a table as source.table everywhere in the manifest.
plugins: the declared packages, each with its version requirement and source (registry, github or local, plus a location unless it’s DRE’s own registry).

Paths are relative to the project root with forward slashes on every OS. Keys are sorted. There are no timestamps, host names or absolute paths, so the same project gives the same bytes on any machine, and moving the target path doesn’t change the file.

Generic integration context

DRE 0.4.1 adds report description, opaque meta, and meta_layers with the originating file and configuration key. Layers preserve project, folder and report evidence separately; external consumers choose their own interpretation. Core does not validate companion-specific meta.ai. Declared var_schema records types, descriptions, defaults, required inputs and allowed values. Existing projects keep working without those optional declarations.

An isolated dre run --no-deliver --output-json <dir> also exports typed rows per query and normal formatted outputs, with run results outside the scheduled report’s history. Beside each query’s typed <query>.json in <dir>/<report>/<binding>/ it saves the exact Arrow IPC stream DRE received (<query>.arrows: decimals, timezones and nulls as they were; it opens directly in DuckDB, Polars or pandas) and an index, results.json (dre/saved-results/v1: each query’s name, connection, rows, file and its SQL’s SHA-256). dre run --from-results <dir> replays them. Each compiled query has a SHA-256 companion file identifying exact SQL before secret masking. Displayed SQL may be masked; hashes and run provenance preserve the execution reference. Live catalogue metadata is obtained separately through configured connections and their upstream comments.

What’s left out

Anything that only exists after rendering: compiled SQL (that’s compiled/), rendered output and destination paths, connection.* values.
Connection settings and credentials. Profiles appear by name only.
Secrets: values of DRE_SECRET_* variables are masked as *****, as in run_results.json and the compiled SQL (mask_secrets: false in dre_project.yml turns that off everywhere).
Engine internals that might change meaning. A field is left out rather than shipped with a meaning that could shift.

Checksums

Every checksum is a SHA-256 in hex, over file contents only (not timestamps), in a fixed order.

A report’s checksum covers its defining file (YAML, or the .sql of an unmanaged report), each of its query files, and its template file.
The project’s checksum covers the shared inputs: dre_project.yml, folder config, schedules.yml, dependencies.yml and any other YAML that isn’t a report’s own, everything under macros/ and lookups/, and every .sql under reports/ that isn’t a declared query (the usual ref() targets, including unmanaged reports).

To find what a change touched, compare two manifests: if the project checksum differs, treat every report as changed; otherwise the reports whose checksum differs are the ones to preview or validate (dre validate -s a,b).

`dre ls`

dre ls prints the reports and Bindings a selection covers, with the same selectors as run:

dre ls                              # every Binding
dre ls -s tag:regulatory --set all  # a selection
dre ls --schedule close_monthly     # exactly what that schedule runs
dre ls --schedule close_monthly --output json

The default output is one Binding per line (report, Set, the connections its queries run on, format, destinations). dre ls -s source:sales.orders lists the reports that read a source table, and dre ls --resource-type source lists the declared sources, flagging unused ones (see Sources). --target and --var resolve them as a run would. --output json prints a document in the manifest’s shape holding only the matching reports and Bindings (and, with --schedule, that schedule). Data goes to stdout and messages to stderr; a selector or schedule that matches nothing exits 2 (see exit codes). dre ls needs no connection or profiles.yml and writes nothing.

dre validate --json includes the same document under "project".

`run_results.json`

Each Binding a dre run executes writes run/<report>/<set or default>/runs/<run id>/run_results.json in the target path (dre history <report> --latest --path prints the latest run’s folder; see the target path). It records its run_id, the report, Set, the inherited profile, the target, the connections its queries used, the schedule and its vars, every var the run used, the run date and timezone, the command’s parameters, the status (success, error, cancelled when Ctrl-C or a termination signal stopped it, timed_out when its timeout expired) and any error, each result set (rows, columns and the connection it came from), each output file (path, relative to the project root, or to the target path when that’s outside the project), each delivery (its profile, type, target and status: delivered, not_delivered for a {deliver: false} entry, or failed), schema drift, the resolved target_path, and manifest_checksum: the SHA-256 of manifest bytes written by the caller, when supplied. A scoped CLI run sets it to null and does not hash or rebuild a stale manifest.

project_errors, when non-empty, lists the first discovered error in each report outside the run’s selection: {"path": "reports/other/other.yml", "code": "unknown-key", "message": "..."}. Paths use forward slashes, and messages follow the normal secret masking rules. The same list is written to every Binding result, including a failed query’s result. Whole-project dre validate remains the full check; a run does not execute other reports’ plugin validators.

With several outputs, outputs (every file) and deliveries (every destination) cover all of them, and each file names its output. output_results has one entry per output, in declared order: its name, format, the queries it formatted, its status (delivered, kept when it stays in the target path, skipped when its when: was false, its message rendered empty or it had no data, or failed), any error, its files, delivery note and deliveries; when (true or false) when it has a when:; reason: no_data when on_empty skipped or failed it, no_data (its queries that returned no rows) and dropped (empty queries left out with on_empty: skip); for a per-row output, for_each with its driver query, key, row counts (rows, succeeded, skipped, failed, held, not_started, orphan_rows), failed_keys (each key, code, message) and the sidecar, <output>.rows.jsonl, with one line per row (its key, status, reason, files with their sha256, deliveries and error); and for a message output, message with the full rendered title and text that were sent. A report’s checksum covers a message output’s file: template, like an xlsx template.

Versioning

Both files are public contracts with a schema_version, and JSON Schemas generated from DRE’s own types: manifest.schema.json for the manifest (dre/manifest/v3) and run-results.schema.json for run_results.json (dre/run-results/v1).

Adding an optional field keeps the version. Ignore fields you don’t know.
Removing, renaming or re-typing a field, or changing what a field means, bumps it: in a minor DRE release before 1.0, a major one after. Check schema_version and refuse one you don’t support.

Changed in 0.4. The manifest’s "schema": 2 is replaced by "schema_version": "dre/manifest/v3", with nothing else changed. run_results.json gains schema_version (dre/run-results/v1), and a failed run’s error_code and error_kind (see the error codes reference).

Before 0.4 the manifest had "schema": 2 (DRE 0.2 and 0.3) or "schema": 1 (DRE 0.1, without sources, target or per-query connection).

The idea of a project manifest comes from dbt; the format and code are DRE’s own.

Inspect a concrete run

cd examples/tutorial
dre validate --json
dre run --preview 5
dre history --latest --path sales

sales is the report name shown by dre ls in that project. The complete tutorial example gives a small manifest and run-result pair to inspect. Use the history path rather than assuming a fixed run directory. project-file-missing leaves no current manifest; do not reuse a manifest saved from another directory or target. A consumer must inspect valid, status and delivery fields: a JSON file’s existence does not mean the report passed. See CLI reference, manifest and run results.