# DRE plugin protocol, version 1 | DRE > How a plugin talks to DRE, for writing a plugin in any language. You're reading the docs for DRE 0.4 (v0.4.4). [This page for 0.5](https://getdre.com/docs/protocol/) · [New in 0.5](https://getdre.com/docs/new-in-0.5/) # DRE plugin protocol, version 1 For a complete implementation, read the first-party [CSV format](https://github.com/get-dre/dre/blob/v0.4.4/plugins/csv/src/lib.rs) and its [entry point](https://github.com/get-dre/dre/blob/v0.4.4/plugins/csv/src/main.rs); its integration tests live in [`plugins/csv/tests`](https://github.com/get-dre/dre/blob/v0.4.4/plugins/csv/tests/). The example requests below are minimal wire messages, not a substitute for the framing and handshake. The [glossary](https://getdre.com/docs/v0.4/glossary/) defines plugin packages, formats and destinations. Every source, format and destination in DRE is a plugin, served by a separate executable that DRE core starts and talks to over stdin and stdout. Plugins ship in packages: one executable can serve several plugins (the `databricks` package is a source and a destination; `object_store` is three destinations), and core names the one it wants in the handshake. Plugins can be written in any language. This document is the contract between core and a plugin. Any change to it means a new protocol version. The reference implementation is the [`dre-protocol`](https://crates.io/crates/dre-protocol) crate (`cargo add dre-protocol`). It has its own version, apart from both DRE’s and this protocol’s. It has three parts: - the core side, `host`; - a plugin SDK, `plugin`, which gives Rust authors framing, the handshake and error handling; - a conformance suite, `conformance`, that any plugin binary can be checked against. ## The plugin interface Every plugin, whatever its kind, follows one interface. Core finds a plugin by its kind and name alone and knows nothing else about it, so a new source, format or destination needs no change to core. Every plugin: 1. **Answers the handshake** (`hello`) with its kind, name, version and capabilities. 2. **Describes itself** (`describe`): the connection fields a `profiles.yml` target of its type takes, and the options a report’s config block for it takes (`option_fields`). 3. **Validates a config block** (`validate`): checks one block of options and replies with every problem found, without connecting or writing anything. `dre validate` and `dre run` send every block the project gives the plugin before anything runs. 4. **Checks its connection settings** (`validate_connection`, sources and destinations): a profile entry’s settings, without connecting. `dre validate` sends it for every entry the run would use (every entry with `--all-targets`). 5. **Does its one job**: a source runs statements (`open`, `execute`, and optionally `check` and `load`); a format writes files (`write`); a destination delivers them (`deliver`). 6. **Closes cleanly** on `close` or at the end of its input. The config block a plugin owns: Kind Block Keys core owns (never sent) format a report’s `output:` map, plus the project’s `format_options.` `format`, `destination`, `template`, `extension` destination one entry of `output.destination` `profile`, `path` source none; its settings are the profile’s connection fields Rules for option messages. Each problem is one sentence that names the key in backticks (`` `quoting` must be one of `minimal`, `all`, `strings`, `none` ``). Core adds where it was found (the file, the report and Set, the destination profile). A plugin rejects keys it doesn’t declare, so a misspelt key is an error rather than silently ignored. In Rust, the `dre_protocol::plugin` SDK does all of this. A plugin implements one trait (`Source`, `Format` or `Destination`) and is served with `serve_source`, `serve_format` or `serve_destination`; a package of several passes them all to `serve_package`. A plugin declares its options as `OptionField`s (name, type, allowed values, bounds, default, description), and adds any rule a declaration can’t express in `validate()`. The SDK answers `describe` and `validate` from those declarations, advertises `validate`, and checks the options again before every `write` and `deliver`, so plugin code only sees options that passed. Connection fields declare a `kind` and `choices` the same way, and `validate_connection()` adds rules about several fields; the SDK runs those checks for `validate_connection` and again before `open` and every `deliver`, so `dre validate` and `dre run` give the same message. The Go module does the same (`Field.Kind`, `Field.Choices`, `Role.ValidateConnection`, `Role.Accepts`). The conformance suite checks every part of the interface. ### Delivery rules in the SDKs File destinations share a set of rules: what to do when a file is already at the path (`if_exists`: `overwrite`, `error`, `number`), uploading under a temporary name and renaming (`atomic`, `temp_dir`), trying again after a temporary error (`retries`), and client timeouts (`connect_timeout`, `timeout`). The SDKs implement them once, so a destination only implements a few primitives on its server: write a file (optionally failing if one is there), rename (with or without replacing), exists, delete, and flags for what it can do in one step. - Rust: `dre_protocol::delivery` (`Store`, `Rules`, `deliver`; `LocalStore` is the local file system, used by core’s `local` destination). - Go: `go/plugin` (`Store`, `Rules`, `Deliver`). A temporary file is named `..dre-part`, next to the final file or in `temp_dir`. With `number`, the new file is saved as `_2.`, then `_3`. Where a server can’t check and write in one step (FTP), the rules look first, then write. A file already there with `if_exists: error` fails with kind `delivery` and code `/file-exists`. ## Naming and location A plugin is identified by its kind and name, written `/`: - `kind` is `source`, `format` or `destination`. - `name` matches `[a-z0-9_]+`. It is the value used in `profiles.yml` (`type: duckdb`) and in `output.format` (`format: xlsx`). A package’s executable is named `dre-plugin-` (plus `.exe` on Windows), e.g. `dre-plugin-object_store`. An executable serving one plugin may instead be named `dre--`, e.g. `dre-source-duckdb`; it is then a package of that one plugin, called ``. Core looks in the project’s `dre_deps/plugins` (or `DRE_PLUGINS_DIR`), in two layouts: - `///`: versioned installs, side by side, as the plugin manager lays them out. `plugin.json` there names the executable and the plugins it provides: `{"executable": "dre-plugin-object_store", "provides": ["destination/s3", ...]}`. - `/dre-plugin-` or `/dre--`: placed by hand, for development. Core asks a `dre-plugin-` executable what it provides with a handshake. ## Streams Stream Direction Content stdin core → plugin frames stdout plugin → core frames, and nothing else stderr plugin → core free-form UTF-8 lines: what a plugin can’t send as a `log` message (a panic, a native driver’s output). Core keeps them in its log file at warn, shows them with `-v`, and quotes the last ones when a plugin crashes. Plugins log through [`log` messages](#logs-and-progress) instead. A plugin must never write anything but frames to stdout. Core sets `DRE_INTERACTIVE` in every plugin’s environment: `1` when a person is at core’s terminal, `0` otherwise (a scheduler, CI). A plugin that needs a person, such as for a browser sign-in, must fail with an explanation instead of waiting when it isn’t `1`. A value already in core’s environment is passed through unchanged. ## Frames A frame is a 4-byte **big-endian** unsigned length `N`, then `N` bytes of body. `N` is between 1 and 2^30. The first body byte is the frame type: Byte Type Rest of the body `0x4A` (`J`) control message one UTF-8 JSON object with a `type` field `0x41` (`A`) data one Arrow IPC **stream** (schema message, record batches, end-of-stream marker) Each data frame is self-contained: it carries its own schema, so it can be decoded alone. Large result sets are sent as many data frames, one or more batches each. A frame with an unknown type byte, a bad length or invalid JSON is malformed. The receiver reports it and stops. ## Handshake The first message core sends is `hello`, naming the plugin it wants served: ```json {"type": "hello", "min_version": 1, "max_version": 1, "core_version": "…", "plugin": "destination/s3"} ``` The plugin picks the highest protocol version both sides support and replies as that plugin: ```json {"type": "hello", "protocol_version": 1, "kind": "destination", "name": "s3", "version": "1.2.0", "capabilities": ["validate"], "provides": ["destination/s3", "destination/gcs", "destination/azure_blob"]} ``` - `plugin` is optional. Without it, an executable serves its first plugin (core leaves it out only to ask a hand-placed package what it provides). - An executable that doesn’t provide the plugin asked for replies `error` and exits non-zero. Core also refuses a reply whose `kind` and `name` aren’t the plugin it asked for. - `provides` lists every plugin the executable serves. It may be left out by an executable serving one plugin; that plugin is then its `kind` and `name`. - An executable serving one plugin may ignore `plugin`, so plugins written before packages keep working unchanged. If the ranges don’t overlap, it replies with its own range and exits non-zero: ```json {"type": "version_mismatch", "min_version": 1, "max_version": 2} ``` Core reports both ranges to the user. It waits 30 seconds for the hello reply (`DRE_PLUGIN_HANDSHAKE_TIMEOUT_MS` overrides this). A plugin that hasn’t answered by then is reported as not responding. Capabilities: Capability Meaning `sessions` Source: one session (connection) is held across every request until `close`, so temp tables and session settings persist. Core refuses to run a Binding with more than one statement on a source without it. `read_only` Source: honours `read_only: true` on `open`. `check` Source: supports `check` (verify a statement without executing it). `load` Source: supports `load` (rows into a temporary table on the session). `multi_file` Destination: takes every file of one output in a single `deliver` (`files`), e.g. one email carrying every attachment. `message` Destination: takes a message (`deliver` with `message`), such as a chat post or an email body. `message_only` Destination: takes only messages; core never sends it a file output. Implies `message`. `validate` Answers `validate`. Required: every first-party plugin advertises it (the Rust SDK does so itself). Core warns that a plugin without it predates option checks and should be updated. ## Requests and replies Every request gets exactly one reply. Core sends one request at a time and waits for its reply, except for `cancel`. ### Request ids Every message after the handshake carries an `id`: core numbers its requests (1, 2, …), and the reply, and every message that belongs to the request, carries the request’s id. That includes the messages core streams as part of a request (`result_set_end`, `finish`) and the ones the plugin streams back (`result`, `result_end`, `log`, `progress`). Arrow data frames have no id: they belong to the request whose messages surround them. ```json {"type": "execute", "id": 7, "sql": "select 1"} {"type": "result", "id": 7, "columns": ["1"]} ``` A `log` message sent between requests has no id. A reply whose id isn’t the running request’s is a protocol error. ### Errors Any request may be answered with an error: ```json {"type": "error", "id": 7, "message": "human-readable explanation", "kind": "auth", "code": "postgres/password-refused"} ``` - `kind` is one of `config`, `plugin`, `refused`, `connection`, `auth`, `query`, `delivery`, `internal`, `cancelled` and `timed_out`, the kinds of DRE’s [error codes](https://getdre.com/docs/v0.4/reference-error-codes/). It decides whether trying again can help. - `code` names the problem, namespaced by the plugin (`/`), and is recorded in `run_results.json`. - Both are optional. Without a `code`, core reports the failure under its own code for the step (`query-failed`, `delivery-failed`, …). A `code` without a `kind` counts as `internal`. After an error reply the plugin keeps serving. Unknown request types, and requests meant for another plugin kind, are answered with `error`. ### Cancel ```json {"type": "cancel", "id": 7} ``` `cancel` asks the plugin to stop request 7. It has no reply of its own: the request replies `error` with kind `cancelled`, or its usual reply if it finished first. A plugin reads stdin while a request runs so that it sees `cancel` at once, and stops what it can (a query on the server, a loop between batches). A cancel for a request that isn’t running, or has already been answered, is ignored. A request cancelled before the plugin started it is answered `cancelled` without running. End of input while a request runs means core has gone: the plugin stops the request too. ### Logs and progress ```json {"type": "log", "id": 7, "level": "info", "message": "waiting for the warehouse to start", "fields": {"elapsed_s": 30}} {"type": "progress", "id": 7, "message": "uploading", "done": 3, "total": 10} ``` A plugin may send `log` and `progress` messages at any time while a request runs, before its reply. `level` is `error`, `warn`, `info`, `debug` or `trace`. Core shows `info` to the person, `warn` and `error` as warnings, and keeps `debug` and `trace` for `-v` and the log file. `fields` (optional) is structured context; never put secrets in it. `progress` is optional: `message`, `done` and `total` are each optional, and core shows the latest. The SDKs bridge each language’s logging: in Rust the `log` crate (re-exported as `dre_protocol::log`; records from the plugin’s dependencies are kept for `-v`), in Go `log/slog`. ### All kinds Request Reply `{"type":"describe"}` `{"type":"describe","connection_fields":[{"name","description","required","secret","default","same_as_source","manual","kind","choices"}],"option_fields":[{"name","type","description","required","default","choices","min","max"}],"identifier_quote":"\""}` `{"type":"validate","options":{…}}` `{"type":"validated","errors":["…"]}` `{"type":"validate_connection","connection":{…},"unresolved":["…"]}` (sources, destinations) `{"type":"validated","errors":["…"],"warnings":["…"]}` `{"type":"close"}` `{"type":"ok"}`, then the plugin exits 0 `describe` lists the fields a `profiles.yml` target of this plugin’s type accepts. `dre init` uses it to prompt for connection details. By default it offers fields marked `secret` as `env_var()` references. A destination field with `"same_as_source": ""` defaults to the value entered for a connection profile of that source type (for example one Databricks host for both). `dre init` doesn’t ask for a field marked `"manual": true`: an alternative to a prompted field (a key’s text instead of its file) or a nested block (`ssh:`), set by hand. A `secret` field, prompted or not, is one templates can’t read. Format plugins return an empty list. `option_fields` lists the options the plugin takes (see [The plugin interface](#the-plugin-interface)). `type` is one of `string`, `char` (exactly one character), `boolean`, `integer`, `number`, `strings` (a string or a list of strings), `list`, `map` or `any`. `choices` limits a string to those values; `min` and `max` bound a number, inclusive. It may be omitted when the plugin takes no options. `identifier_quote` (sources only) is the character the database quotes identifiers with: `"` for DuckDB and Postgres, a backtick for Databricks. Core quotes the parts of a [source](https://getdre.com/docs/v0.4/sources/) that ask for it (`quoting:`) with this character, doubling it inside a name. Every source returns it; formats and destinations leave it out. Added in DRE 0.2 (duckdb and postgres 1.1.0, databricks 1.1.0); core 0.2 needs it only for a source with `quoting:`. `message_limit` (destinations advertising `message`) is the most characters a message may have in the service after translation. `dre run --preview` shows it next to each message’s length. It may be omitted. Added in DRE 0.3. `validate` checks one config block of options and replies `validated` with every problem found, each a sentence naming the key; `errors` is empty when the block is fine. The plugin doesn’t connect, read or write anything. A destination’s string values may still hold Jinja (`{{ … }}`, `{% … %}`), which core renders only before `deliver`: such a value is checked for presence only. `validate_connection` (protocol 1) checks a profile entry’s connection settings and replies `validated` without connecting, resolving names or opening files beyond checking one exists. First the generic checks from `connection_fields`: a `required` field missing (one with a `default` never is), a value of the wrong `kind` (`string`, `integer` (a number or a string of one), `boolean`, `path` (a file that exists; `~/` is the home directory), `duration`, `map`), a string not in `choices`. A key no field declares is a **warning**, with the nearest field’s name; it never stops a run. Then the plugin’s own rules about several fields. Keys listed in `unresolved` (their `env_var()` is unset, which core reports itself) arrive as `null`: they count as set, but aren’t checked. Messages name fields, never values, and the SDK redacts any `secret` field’s value from them. The same checks run before `open` and `deliver`: errors fail the request (kind `config`), warnings are sent as `log` messages. When stdin closes, the plugin exits. ### Source ```json {"type": "open", "connection": {…profile target fields…}, "read_only": false} ``` `open` starts the session. `connection` holds every field of the selected `profiles.yml` target except `type`, with `env_var()` already rendered by core. The reply is `ok`. ```json {"type": "execute", "sql": "select …", "row_limit": 100} ``` `execute` runs exactly one statement. Core splits files into statements itself. `row_limit` is optional; when present, the plugin returns at most that many rows. The reply is one of: - `{"type":"no_result","rows_affected":3}`: the statement returned no result set (DDL, DML, `SET`, …). `rows_affected` is optional. - `{"type":"result","columns":["a","b"]}`, followed by **one or more** data frames (the first one carries the schema, even for zero rows), followed by `{"type":"result_end","rows":42}`. An `error` may replace `result_end` if the stream fails part-way through. The report’s YAML, not the reply, decides what goes in the output: each query entry makes one tab (a sheet, or a file for single-table formats) from its file’s last statement unless it has `tab: false`. Core only checks the reply against that: a tab statement must reply `result` (zero rows still gives a tab with its column names), and results of other statements are discarded. `run_query()` in templates, `--preview` (with `row_limit`) and `dre validate --live` all use this same request. ```json {"type": "check", "sql": "select …"} ``` `check` verifies a statement without executing it, in the dialect’s own way (for example `EXPLAIN`). The reply is `ok`, or an `error` explaining what’s wrong. Only sent to plugins that advertise `check`. ```json {"type": "load", "name": "countries"} ``` `load` puts rows into a temporary table (or view) on the session, for a lookup too large to inline. Core then streams the rows as one or more data frames (the first carries the schema) and a `{"type":"result_set_end"}`. Column types are `Utf8`, `Int64`, `Float64`, `Boolean` and `Date32`. The plugin replies: ```json {"type": "loaded", "relation": "dre_lookup_countries", "rows": 5000, "warning": "…"} ``` `relation` is what core puts in the SQL wherever the lookup is referenced. Set `warning` when the database has no bulk path and the load went through ordinary SQL; core shows it to the user. First-party plugins: DuckDB uses its appender, Postgres `COPY`, and Databricks a temporary view built from one `VALUES` statement, with a warning. Only sent to plugins advertising `load`. ### Format ```json {"type": "write", "path": "/…/target/run/monthly/client_a/monthly.xlsx", "format": "xlsx", "options": {…format options…}, "result_sets": [{"name": "Summary", "query": "query_01", "result_index": 1, "anchor": "A1", "header": true, "columns": {"amount": {"format": "#,##0.00"}}}], "template": {…}} ``` `result_index` is always 1: each query contributes at most one result set. `anchor`, `header` and `columns` (the query entry’s per-column options: `format`, `formula`, `total`) are left out when not set. After `write`, core streams each result set in the order listed. Each one is sent as one or more data frames (the first carries the schema), then `{"type":"result_set_end"}`. After the last result set, core sends `{"type":"finish"}`. The plugin writes the file(s) and replies: ```json {"type": "written", "files": ["/…/monthly.xlsx"], "warnings": ["…"]} ``` `warnings` (optional) are shown to the person, e.g. a value the format had to write as text. If the write fails part-way, the plugin replies `{"type": "error", ...}` at once, without waiting for `finish`, and then reads (and discards) the rest of the stream up to `finish`, sending nothing more. Core stops sending data when it sees the error, so a large result isn’t streamed into a plugin that gave up. That error is the request’s only reply. `options` are the report’s resolved format options (see the YAML schema). `template` is present only for an xlsx template output: `{"file": "", "bindings": [...], "values": {"!": ""}}`, with single-cell `value`s already rendered by core. ### Destination ```json {"type": "deliver", "local_path": "/…/target/run/…/monthly.xlsx", "remote_path": "s3://bucket/monthly-20260125.xlsx", "connection": {…}, "options": {…}} ``` `deliver` copies the local file to `remote_path`. The path is already rendered, and may be absent when the destination profile alone says where. The reply is `{"type":"delivered","location":""}`, with `"attempts": n` when the plugin had to try more than once (protocol 1; core records it in `run_results.json`). The local file is never removed. It stays in `target/` whatever the outcome. `options` holds the destination entry’s plugin options: every key of the entry in `output.destination` other than `profile` and `path` (for example `to` and `subject` for email, `channel` and `message` for Slack). Core renders their Jinja before sending, so string values arrive final. It is `{}` when the entry has none. Core has already had them checked with `validate`, and the Rust SDK checks them again before calling the plugin. A destination that advertises the `multi_file` capability receives every file of one output in a single request, in place of `local_path`/`remote_path`: ```json {"type": "deliver", "files": [{"local_path": "/…/daily_orders.csv", "remote_path": "…/daily_orders.csv"}, {"local_path": "/…/daily_refunds.csv", "remote_path": "…/daily_refunds.csv"}], "connection": {…}, "options": {…}} ``` It replies with one `delivered` for the whole set. Exactly one of `local_path` or `files` is present. Core only sends `files` to a plugin that advertised `multi_file`; any other plugin gets one `deliver` per file. A single-file output is always sent in the `local_path` form. #### Messages A `message` output (DRE 0.3) renders a title and text. To a destination that advertises `message`, core sends it in one `deliver`: ```json {"type": "deliver", "message": {"title": "Daily revenue", "text": "Revenue **€12,340** (+4.1%)", "path": "/…/target/run/daily/default/daily.md"}, "files": [{"local_path": "/…/target/run/daily/default/detail.xlsx"}], "connection": {…}, "options": {…}} ``` - `title` is one line of plain text. `text` is the portable Markdown subset: `**bold**`, `*italic*` or `_italic_`, `` `code` ``, `[text](https://github.com/get-dre/dre/blob/v0.4.4/docs/url)`, `-` bullets and backslash escapes (`\*`); no headings or tables. Core has escaped every value the template printed, so the plugin translates the whole text to its service’s dialect (`dre_protocol::markdown` has the parser and translations to Slack, Google Chat, Teams, HTML and plain text). - `html`, when present, is an HTML body the plugin may use instead of converting `text`. Core 0.3 never sends it; it leaves room for a future HTML output. - `path` is the `.md` file core wrote (`# title`, a blank line, the text), for attaching the full message when the text is over the service’s limit. - `files` holds the files of other outputs the entry names in `attach:` (often none). Neither `local_path` nor `remote_path` is set. A destination that doesn’t advertise `message` gets a message output as its `.md` file, in the usual `local_path` form. Core never sends a file output to a destination that advertises `message_only`: `dre validate` refuses it, as it refuses `attach:` to a destination that takes only messages or only files. Older plugins, which advertise neither, keep working for files. ## Errors and failure - A plugin that exits, crashes or closes stdout unexpectedly is reported with its exit status and its last stderr lines. - A malformed frame from the plugin is reported as a protocol error. - A plugin that panics or hits an internal error should reply `error` before exiting. The SDK does this. ## Versioning Version 1 added request ids, `cancel`, `log`, `progress` and the error `kind` and `code`. Core 0.4 requires version 1; a plugin that only speaks version 0 is refused with a message naming `dre plugin update `. The SDKs speak versions 0 and 1, so a plugin built on them still works with core 0.3: on version 0 they send no ids and log to stderr with the prefixes core 0.3 shows (`info:` , `warning:` ). The protocol version is a single integer. Core and plugins release independently, so each side states the range it supports and the handshake picks the highest common version. Adding a new optional field to a message does not change the version; anything else does. ## Conformance `dre_protocol::conformance::run(path)` checks a plugin binary. It covers the handshake and identity, speaking versions 0 and 1, refusal of an unsupported version, request ids on replies, ignoring a `cancel` for a request that isn’t running, `describe` (and, for a source, that it gives a one-character `identifier_quote`), error replies for unknown and wrong-kind requests, `validate` (advertised, answered, and refusing an option the plugin doesn’t declare), behaviour on a malformed frame, and a clean exit on `close` and on end of input. For a destination it also sends a `deliver` carrying `options` and, when `multi_file` is advertised, a `files` delivery, and when `message` is advertised a message `deliver`, and expects a reply to each (`delivered` or `error`) with the plugin still serving. A `message_only` plugin must also advertise `message` and must refuse a file `deliver` with an `error`. `conformance::run_with_env` runs the suite with extra environment variables. Every first-party plugin runs the suite in its tests. [Previous Project configuration](https://getdre.com/docs/v0.4/project-configuration/)[Next dependencies.yml reference](https://getdre.com/docs/v0.4/reference-dependencies/)