DRE plugin protocol, version 1
For a complete implementation, read the first-party CSV format
and its entry point; its integration tests live in
plugins/csv/tests. The example requests below are minimal wire
messages, not a substitute for the framing and handshake. The
glossary defines plugin packages, formats and destinations.
Every source, format and destination in DRE is a plugin, served by a separate executable that
DRE core starts and talks to over stdin and stdout. Plugins ship in packages: one executable can
serve several plugins (the databricks package is a source and a destination; object_store
is three destinations), and core names the one it wants in the handshake. Plugins can be written
in any language. This document
is the contract between core and a plugin. Only an incompatible wire change means a new protocol
version; compatible additions don’t.
The reference implementation is the dre-protocol crate
(cargo add dre-protocol). It has its own version, apart from both DRE’s and this protocol’s.
It has three parts:
- the core side,
host; - a plugin SDK,
plugin, which gives Rust authors framing, the handshake and error handling; - a conformance suite,
conformance, that any plugin binary can be checked against (see Conformance: the cases and the message schema are data, for any language).
The plugin interface
Section titled “The plugin interface”Every plugin, whatever its kind, follows one interface. Core finds a plugin by its kind and name alone and knows nothing else about it, so a new source, format or destination needs no change to core.
Every plugin:
- Answers the handshake (
hello) with its kind, name, version and capabilities. - Describes itself (
describe): the connection fields aprofiles.ymltarget of its type takes, and the options a report’s config block for it takes (option_fields). - Validates a config block (
validate): checks one block of options and replies with every problem found, without connecting or writing anything.dre validateanddre runsend every block the project gives the plugin before anything runs. - Checks its connection settings (
validate_connection, sources and destinations): a profile entry’s settings, without connecting.dre validatesends it for every entry the run would use (every entry with--all-targets). - Does its one job: a source runs statements (
open,execute, and optionallycheckandload); a format writes files (write); a destination delivers them (deliver). - Closes cleanly on
closeor at the end of its input.
The config block a plugin owns:
| Kind | Block | Keys core owns (never sent) |
|---|---|---|
| format | a report’s output: map, plus the project’s format_options.<format> |
name, format, queries, when, destination, template, extension, on_empty, for_each, file_name |
| destination | one entry of output.destination |
profile, path, attach, hold, rate, burst, zip |
| source | none; its settings are the profile’s connection fields |
Rules for option messages. Each problem is one sentence that names the key in backticks
(`quoting` must be one of `minimal`, `all`, `strings`, `none`). Core adds where it was found
(the file, the report and Set, the destination profile). A plugin rejects keys it doesn’t
declare, so a misspelt key is an error rather than silently ignored.
In Rust, the dre_protocol::plugin SDK does all of this. A plugin implements one trait
(Source, Format or Destination) and is served with serve_source, serve_format or
serve_destination; a package of several passes them all to serve_package. A plugin declares its options as OptionFields (name, type,
allowed values, bounds, default, description), and adds any rule a declaration can’t express in
validate(). The SDK answers describe and validate from those declarations, advertises
validate, and checks the options again before every write and deliver, so plugin code only
sees options that passed. Connection fields declare a kind and choices the same way, and
validate_connection() adds rules about several fields; the SDK runs those checks for
validate_connection and again before open and every deliver, so dre validate and
dre run give the same message. The Go module does the same (Field.Kind, Field.Choices,
Role.ValidateConnection, Role.Accepts). The conformance suite checks every part of the
interface.
Delivery rules in the SDKs
Section titled “Delivery rules in the SDKs”File destinations share a set of rules: what to do when a file is already at the path
(if_exists: overwrite, error, number), uploading under a temporary name and renaming
(atomic, temp_dir), trying again after a temporary error (retries), and client timeouts
(connect_timeout, timeout). The SDKs implement them once, so a destination only implements
a few primitives on its server: write a file (optionally failing if one is there), rename (with
or without replacing), exists, delete, and flags for what it can do in one step.
- Rust:
dre_protocol::delivery(Store,Rules,deliver;LocalStoreis the local file system, used by core’slocaldestination). - Go:
go/plugin(Store,Rules,Deliver).
A temporary file is named .<name>.<unique>.dre-part, next to the final file or in temp_dir.
The token is fresh for every upload attempt, including retries, across processes and deliveries
sharing a staging directory. Both SDKs use a process ID, timestamp and process-local sequence.
Conformance operation logs replace these tokens with <temp-1>, <temp-2>, and so on, in write
order; the runner checks their shape and refuses reuse. With
number, the new file is saved as <name>_2.<ext>, then _3. Where a server can’t check and
write in one step (FTP), the rules look first, then write. A file already there with if_exists: error fails with kind delivery and code <plugin>/file-exists.
Naming and location
Section titled “Naming and location”A plugin is identified by its kind and name, written <kind>/<name>:
kindissource,formatordestination.namematches[a-z0-9_]+. It is the value used inprofiles.yml(type: duckdb) and inoutput.format(format: xlsx).
A package’s executable is named dre-plugin-<package> (plus .exe on Windows), e.g.
dre-plugin-object_store. An executable serving one plugin may instead be named
dre-<kind>-<name>, e.g. dre-source-duckdb; it is then a package of that one plugin, called
<name>.
Core looks in the project’s dre_deps/plugins (or DRE_PLUGINS_DIR), in two layouts:
<dir>/<package>/<version>/: versioned installs, side by side, as the plugin manager lays them out.plugin.jsonthere names the executable and the plugins it provides:{"executable": "dre-plugin-object_store", "provides": ["destination/s3", ...]}.<dir>/dre-plugin-<package>or<dir>/dre-<kind>-<name>: placed by hand, for development. Core asks adre-plugin-<package>executable what it provides with a handshake.
Streams
Section titled “Streams”| Stream | Direction | Content |
|---|---|---|
| stdin | core → plugin | frames |
| stdout | plugin → core | frames, and nothing else |
| stderr | plugin → core | free-form UTF-8 lines: what a plugin can’t send as a log message (a panic, a native driver’s output). Core keeps them in its log file at warn, shows them with -v, and quotes the last ones when a plugin crashes. Plugins log through log messages instead. |
A plugin must never write anything but frames to stdout.
Core sets DRE_INTERACTIVE in every plugin’s environment: 1 when a person is at core’s
terminal, 0 otherwise (a scheduler, CI). A plugin that needs a person, such as for a browser
sign-in, must fail with an explanation instead of waiting when it isn’t 1. A value already in
core’s environment is passed through unchanged.
Frames
Section titled “Frames”A frame is a 4-byte big-endian unsigned length N, then N bytes of body. N is between 1
and 2^30. The first body byte is the frame type:
| Byte | Type | Rest of the body |
|---|---|---|
0x4A (J) |
control message | one UTF-8 JSON object with a type field |
0x41 (A) |
data | one Arrow IPC stream (schema message, record batches, end-of-stream marker) |
Each data frame is self-contained: it carries its own schema, so it can be decoded alone. Large result sets are sent as many data frames, one or more batches each.
A frame with an unknown type byte, a bad length or invalid JSON is malformed. The receiver reports it and stops.
Handshake
Section titled “Handshake”The first message core sends is hello, naming the plugin it wants served:
{"type": "hello", "min_version": 1, "max_version": 1, "core_version": "…", "plugin": "destination/s3"}The plugin picks the highest protocol version both sides support and replies as that plugin:
{"type": "hello", "protocol_version": 1, "kind": "destination", "name": "s3", "version": "1.2.0", "capabilities": ["validate"], "provides": ["destination/s3", "destination/gcs", "destination/azure_blob"]}pluginis optional. Without it, an executable serves its first plugin (core leaves it out only to ask a hand-placed package what it provides).- An executable that doesn’t provide the plugin asked for replies
errorand exits non-zero. Core also refuses a reply whosekindandnamearen’t the plugin it asked for. provideslists every plugin the executable serves. It may be left out by an executable serving one plugin; that plugin is then itskindandname.- An executable serving one plugin may ignore
plugin, so plugins written before packages keep working unchanged.
If the ranges don’t overlap, it replies with its own range and exits non-zero:
{"type": "version_mismatch", "min_version": 1, "max_version": 2}Core reports both ranges to the user. It waits 30 seconds for the hello reply
(DRE_PLUGIN_HANDSHAKE_TIMEOUT_MS overrides this). A plugin that hasn’t answered by then is
reported as not responding.
Capabilities:
| Capability | Meaning |
|---|---|
sessions |
Source: one session (connection) is held across every request until close, so temp tables and session settings persist. Core refuses to run a Binding with more than one statement on a source without it. |
read_only |
Source: honours read_only: true on open. Used by dre query and catalogue discovery; see the session limits below. |
check |
Source: supports check (verify a statement without executing it). |
load |
Source: supports load (rows into a temporary table on the session). |
catalog |
Source: can discover accessible relations through native database metadata. |
multi_file |
Destination: takes every file of one output in a single deliver (files), e.g. one email carrying every attachment. |
message |
Destination: takes a message (deliver with message), such as a chat post or an email body. |
message_only |
Destination: takes only messages; core never sends it a file output. Implies message. |
validate |
Answers validate. Required: every first-party plugin advertises it (the Rust SDK does so itself). Core warns that a plugin without it predates option checks and should be updated. |
Requests and replies
Section titled “Requests and replies”Every request gets exactly one reply. Core sends one request at a time and waits for its reply,
except for cancel.
Request ids
Section titled “Request ids”Every message after the handshake carries an id: core numbers its requests (1, 2, …), and the
reply, and every message that belongs to the request, carries the request’s id. That includes the
messages core streams as part of a request (result_set_end, finish) and the ones the plugin
streams back (result, result_end, log, progress). Arrow data frames have no id: they
belong to the request whose messages surround them.
{"type": "execute", "id": 7, "sql": "select 1"}{"type": "result", "id": 7, "columns": ["1"]}A log message sent between requests has no id. A reply whose id isn’t the running request’s is
a protocol error.
Errors
Section titled “Errors”Any request may be answered with an error:
{"type": "error", "id": 7, "message": "human-readable explanation", "kind": "auth", "code": "postgres/password-refused"}kindis one ofconfig,plugin,refused,connection,auth,query,delivery,internal,cancelledandtimed_out, the kinds of DRE’s error codes. It decides whether trying again can help.codenames the problem, namespaced by the plugin (<plugin>/<slug>), and is recorded inrun_results.json.- Both are optional. Without a
code, core reports the failure under its own code for the step (query-failed,delivery-failed, …). Acodewithout akindcounts asinternal.
After an error reply the plugin keeps serving. Unknown request types, and requests meant for
another plugin kind, are answered with error.
Cancel
Section titled “Cancel”{"type": "cancel", "id": 7}cancel asks the plugin to stop request 7. It has no reply of its own: the request replies
error with kind cancelled, or its usual reply if it finished first. A plugin reads stdin while
a request runs so that it sees cancel at once, and stops what it can (a query on the server, a
loop between batches). A cancel for a request that isn’t running, or has already been answered,
is ignored. A request cancelled before the plugin started it is answered cancelled without
running. End of input while a request runs means core has gone: the plugin stops the request too.
Logs and progress
Section titled “Logs and progress”{"type": "log", "id": 7, "level": "info", "message": "waiting for the warehouse to start", "fields": {"elapsed_s": 30}}{"type": "progress", "id": 7, "message": "uploading", "done": 3, "total": 10}A plugin may send log and progress messages at any time while a request runs, before its
reply. level is error, warn, info, debug or trace. Core shows info to the person,
warn and error as warnings, and keeps debug and trace for -v and the log file. fields
(optional) is structured context; never put secrets in it. progress is optional: message,
done and total are each optional, and core shows the latest.
The SDKs bridge each language’s logging: in Rust the log crate (re-exported as
dre_protocol::log; a dependency’s info, warning and error records are kept for -v, its debug and trace records dropped), in Go log/slog.
All kinds
Section titled “All kinds”| Request | Reply |
|---|---|
{"type":"describe"} |
{"type":"describe","connection_fields":[{"name","description","required","secret","default","same_as_source","manual","kind","choices"}],"option_fields":[{"name","type","description","required","default","choices","min","max"}],"identifier_quote":"\"","string_literal":"backslash"} |
{"type":"validate","options":{…}} |
{"type":"validated","errors":["…"]} |
{"type":"validate_connection","connection":{…},"unresolved":["…"]} (sources, destinations) |
{"type":"validated","errors":["…"],"warnings":["…"]} |
{"type":"close"} |
{"type":"ok"}, then the plugin exits 0 |
describe lists the fields a profiles.yml target of this plugin’s type accepts. dre init
uses it to prompt for connection details. By default it offers fields marked secret as
env_var() references. A destination field with "same_as_source": "<source type>" defaults
to the value entered for a connection profile of that source type (for example one Databricks host for
both). dre init doesn’t ask for a field marked "manual": true: an alternative to a prompted
field (a key’s text instead of its file) or a nested block (ssh:), set by hand. A secret
field, prompted or not, is one templates can’t read. Format plugins return an empty list.
option_fields lists the options the plugin takes (see The plugin interface).
type is one of string, char (exactly one character), boolean, integer, number,
strings (a string or a list of strings), list, map or any. choices limits a string to
those values; min and max bound a number, inclusive. It may be omitted when the plugin
takes no options.
File-format metadata
Section titled “File-format metadata”A file format’s describe reply must supply format_properties for DRE 0.12:
{"type":"describe","id":1,"connection_fields":[],"option_fields":[], "format_properties":{"extension":"csv","layout":"single_table"}}extension is a bare suffix without any dot, /, \, whitespace or control character.
It also rejects the reserved Windows filename characters :, <, >, ", |, ? and *.
An empty string means no extension. These rules apply to the plugin’s metadata, before file
prediction; do not send a full filename or a leading dot. layout declares behavior
independently of the plugin’s name:
| Layout | Core behavior |
|---|---|
single_table |
A file holds one query result; several results split into separate files. |
multiple_tables |
One file receives all the output’s query results. |
workbook |
One workbook receives the results; workbook sheet/template-binding checks apply. |
document |
The renderer receives the document context; its template controls the sections shown. |
Core resolves these properties from the selected, pinned plugin before predicting files or
checking format-specific settings. It never guesses traits from a format name. Missing or
invalid metadata produces an options-unchecked warning and omits file predictions; file
rendering fails before source queries rather than using guessed behavior. Compilation and
data-test-only commands do not require file-format metadata. The built-in message output is
independent of this file-format contract.
Source SQL policy
Section titled “Source SQL policy”Sources enabling dre query supply sql_dialect in describe. The policy names a parser
(generic, duckdb, postgres or databricks), identifier_quote, identifier_case
(insensitive, lower_unless_quoted or upper_unless_quoted), literal_style (doubling or
hex), and explicit denied_functions and file_formats lists. Empty lists are explicit
policies; omitted safety fields and unknown policy keys are invalid. identifier_quote must
be a double quote or a backtick. Entries in denied_functions and file_formats must be
nonempty ASCII letters, digits or underscores. A hex policy supplies hex_literal with
exactly one {hex} placeholder; a doubling policy leaves it out. read_only_notice is optional
and, when present, must contain non-whitespace text; it documents the permission boundary
when the source cannot provide a database-enforced read-only session. Missing or invalid
policies refuse dre query; they never enable a permissive default. A separately advertised
identifier_quote must agree with the policy.
The SDKs transport these fields without changing protocol 1. The Rust SDK exposes
SqlDialect and SqlCatalog; source plugins own native catalogue queries and the catalog
exchange remains capability-gated. Core validates the returned dre/catalog/v1 document and
opens a read-only catalogue session when the source advertises read_only.
identifier_quote (sources only) is the character the database quotes identifiers with: " for
DuckDB and Postgres, a backtick for Databricks. Core quotes the parts of a
source that ask for it (quoting:) with this character, doubling it inside a name.
Every source returns it; formats and destinations leave it out. Added in DRE 0.2 (duckdb and
postgres 1.1.0, databricks 1.1.0); core 0.2 needs it only for a source with quoting:.
string_literal (sources only, optional) says how core spells a string holding ' or \ when
it writes one into SQL, as it does for an inlined lookup. Absent, or chr: the
characters are chr(39) and chr(92) joined with ||, which DuckDB, Postgres and Databricks
read the same way. backslash: \' and \\ inside the quotes, for a database without chr()
(ClickHouse). A core that doesn’t know the field ignores it, so a plugin can send it to any
core; core asks the plugin to update DRE when it names a style core doesn’t know. Added in DRE
0.5.1 (clickhouse 1.0.1). With backslash, core also reads a backslash inside quotes as an escape when it
splits that source’s SQL into statements (DRE 0.12).
message_limit (destinations advertising message) is the most characters a message may have
in the service after translation. dre run --preview shows it next to each message’s length.
It may be omitted. Added in DRE 0.3.
single_writer (sources only, optional) names a connection field that holds a file only one
process may write at a time: {"field": "path", "except": [":memory:"]} for DuckDB. On a
connection whose entry sets that field to a string other than the exceptions, core runs one
Binding at a time, whatever the entry’s threads:. Core asks only when threads: is above 1.
Added in DRE 0.9 (duckdb 1.3.0); an older DuckDB plugin doesn’t send it, so a DuckDB file with
threads: above 1 needs duckdb 1.3.0 or later.
validate may carry a template (DRE 0.6, formats only): for the html and pdf formats,
{"file": "<absolute path>" or null, "project_root", "queries": [...]}, so the plugin checks the
template too (syntax, includes, assets, results.<name>), replying template problems in errors
and warnings. A plugin that doesn’t know the field ignores it.
validate checks one config block of options and replies validated with every problem found,
each a sentence naming the key; errors is empty when the block is fine. The plugin doesn’t
connect, read or write anything. A destination’s string values may still hold Jinja
({{ … }}, {% … %}), which core renders only before deliver: such a value is checked for
presence only.
validate_connection (protocol 1) checks a profile entry’s connection settings and replies
validated without connecting, resolving names or opening files beyond checking one exists.
First the generic checks from connection_fields: a required field missing (one with a
default never is), a value of the wrong kind (string, integer (a number or a string of
one), boolean, path (a file that exists; ~/ is the home directory), duration, map), a
string not in choices. A key no field declares is a warning, with the nearest field’s
name; it never stops a run. Then the plugin’s own rules about several fields. Keys listed in
unresolved (their env_var() is unset, which core reports itself) arrive as null: they
count as set, but aren’t checked. Messages name fields, never values, and the SDK redacts any
secret field’s value from them. The same checks run before open and deliver: errors fail
the request (kind config), warnings are sent as log messages.
When stdin closes, the plugin exits.
Source
Section titled “Source”{"type": "open", "connection": {…profile target fields…}, "read_only": false, "sandbox": false}open starts the session. connection holds every field of the selected profiles.yml target
except type, with env_var() already rendered by core. The reply is ok.
read_only: true requests a read-only database session. A source advertising read_only
must enforce it, or fail open rather than silently opening a writable session.
sandbox is optional in protocol 1 and defaults to false. sandbox: true requests a
read-only session that also refuses access to files, programs and networks beyond the database,
where the engine can enforce those restrictions. For example, disable file readers, extension
loading and external execution. A source must refuse operations it cannot safely permit.
This is an engine restriction, not an operating-system sandbox. The Rust SDK’s
open_sandboxed defaults to read-only open; the Go SDK falls back to OpenReadOnly.
Authors must override those hooks for additional engine restrictions.
dre query requires a valid sql_dialect policy. When read_only is advertised it sends
both flags as true. A policy with an explicit read_only_notice may instead use a normal
session after the single-query checks; Databricks uses this path. Use a read-only database
role there: SQL validation does not replace database permissions. Sources with neither
read-only support nor an explicit policy notice are refused. Older plugins may ignore
sandbox, so database permissions remain the access boundary even when the source supports
read-only sessions.
{"type": "execute", "sql": "select …", "row_limit": 100}execute runs exactly one statement. Core splits files into statements itself. row_limit is
optional; when present, the plugin returns at most that many rows. The reply is one of:
{"type":"no_result","rows_affected":3}: the statement returned no result set (DDL, DML,SET, …).rows_affectedis optional.{"type":"result","columns":["a","b"]}, followed by one or more data frames (the first one carries the schema, even for zero rows), followed by{"type":"result_end","rows":42}. Anerrormay replaceresult_endif the stream fails part-way through.
The report’s YAML, not the reply, decides what goes in the output: each query entry makes one
tab (a sheet, or a file for single-table formats) from its file’s last statement unless it has
tab: false. Core only checks the reply against that: a tab statement must reply result (zero
rows still gives a tab with its column names), and results of other statements are discarded.
run_query() in templates, --preview (with row_limit) and dre validate --live all use this
same request.
{"type": "check", "sql": "select …"}check verifies a statement without executing it, in the dialect’s own way (for example
EXPLAIN). The reply is ok, or an error explaining what’s wrong. Only sent to plugins that
advertise check.
{"type": "load", "name": "countries"}load puts rows into a temporary table (or view) on the session, for a lookup too large to
inline. Core then streams the rows as one or more data frames (the first carries the schema) and
a {"type":"result_set_end"}. Column types are Utf8, Int64, Float64, Boolean and
Date32. The plugin replies:
{"type": "loaded", "relation": "dre_lookup_countries", "rows": 5000, "warning": "…"}relation is what core puts in the SQL wherever the lookup is referenced. Set warning when the
database has no bulk path and the load went through ordinary SQL; core shows it to the user.
First-party plugins: DuckDB uses its appender, Postgres COPY, and Databricks a temporary view
built from one VALUES statement, with a warning. Only sent to plugins advertising load.
{"type": "catalog", "tables": ["orders"], "schemas": []}catalog discovers schemas, tables/views, columns, types and available identifier metadata. It
replies {"type":"catalog","catalog":{…}}, where catalog is the existing dre/catalog/v1
document used by the CLI. Each table records the requested names that resolved to it, so core
can reject an absent requested object. Core owns the selected connection, target and source
type in the final document. Only sent to plugins advertising catalog.
Format
Section titled “Format”{"type": "write", "path": "/…/target/run/monthly/client_a/monthly.xlsx", "format": "xlsx", "options": {…format options…}, "result_sets": [{"name": "Summary", "query": "query_01", "result_index": 1, "anchor": "A1", "header": true, "columns": {"amount": {"format": "#,##0.00"}}}], "template": {…}}result_index is always 1: each query contributes at most one result set. Optional anchor,
header, autofit and style come from the query entry. columns carries per-column
format, formula, total, width and style; unset fields and an empty columns map are
omitted. For XLSX, query autofit and sheet style override output defaults, while column
width and cell style override those query settings. Their shapes are defined in the
message schema and report reference.
After write, core streams each result set in the order listed. Each one is sent as one or more
data frames (the first carries the schema), then {"type":"result_set_end"}. After the last
result set, core sends {"type":"finish"}. The plugin writes the file(s) and replies:
{"type": "written", "files": ["/…/monthly.xlsx"], "warnings": ["…"]}warnings (optional) are shown to the person, e.g. a value the format had to write as text.
A core that can resolve registry fonts sends "font_requests": true in write. Otherwise this
optional field is absent or false. A format must not request fonts without that opt-in, so a
new format still works with older cores; older formats ignore the extra field.
After consuming all result sets and finish, the format can reply before layout:
{"type":"need_fonts","id":1,"font":{"name":"noto_sans_jp","version":"2.4.0"}}Core downloads the exact font package through its registry, verifies its checksums, and replies with absolute local font paths, or an error. The reply retains the write request’s id:
{"type":"fonts","id":1,"paths":["/…/.dre/fonts/noto_sans_jp/2.4.0/regular.otf","/…/.dre/fonts/noto_sans_jp/2.4.0/bold.otf"],"error":null}Core answers each package once per run, shared by every process of the run: a font it couldn’t
supply is not tried again, and later replies for it carry the same error with
"repeated": true (absent means false). The person has then been told once, so a format warns
only on a reply without repeated; it can still fail the write on any error. In the Rust SDK,
ResultSets::request_fonts returns core’s reason as a FontError with message and
repeated.
The format then sends its usual written reply. fonts continues the existing write; it does
not start a new request. The wire protocol version remains 1: the exchange is additive and
only used when core advertises support. dre-protocol 0.10.0 adds the SDK helpers.
If the write fails part-way, the plugin replies {"type": "error", ...} at once, without waiting
for finish, and then reads (and discards) the rest of the stream up to finish, sending
nothing more. Core stops sending data when it sees the error, so a large result isn’t streamed
into a plugin that gave up. That error is the request’s only reply.
options are the report’s resolved format options (see the YAML schema). template is present
for an xlsx template output: {"file": "<absolute path>", "bindings": [...], "values": {"<sheet>!<cell>": "<rendered value>"}}, with single-cell values already rendered
by core. For the html and pdf formats it’s always present, as the document template’s
context (DRE 0.6): {"file": "<absolute path>" or null, "project_root", "run": {"report", "set", "target", "schedule", "date", "now", "scheduled_at", "timezone", "week_start", "week_numbering"}, "vars": {...}, "cli_vars": {...}, "locale"}. vars holds every resolved var
(the Binding’s, with the schedule’s on top) and cli_vars the --var values as typed, which
var() reads first. The plugin renders the template itself; dre-template’s document module builds the same
var(), run.*, date and number filters core’s templates have.
Destination
Section titled “Destination”{"type": "deliver", "local_path": "/…/target/run/…/monthly.xlsx", "remote_path": "s3://bucket/monthly-20260125.xlsx", "connection": {…}, "options": {…}}deliver copies the local file to remote_path. The path is already rendered, and may be absent
when the destination profile alone says where. The reply is
{"type":"delivered","location":"<where it landed>"}, with "attempts": n when the plugin had to
try more than once (protocol 1; core records it in run_results.json). The local file is never
removed. It stays in target/ whatever the outcome.
options holds the destination entry’s plugin options: every key of the entry in
output.destination other than the core-owned keys in the table above (for example to and subject for email,
channel and message for Slack). Core renders their Jinja before sending, so string values
arrive final. It is {} when the entry has none. Core has already had them checked with
validate, and the Rust SDK checks them again before calling the plugin.
A destination that advertises the multi_file capability receives every file of one output in a
single request, in place of local_path/remote_path:
{"type": "deliver", "files": [{"local_path": "/…/daily_orders.csv", "remote_path": "…/daily_orders.csv"}, {"local_path": "/…/daily_refunds.csv", "remote_path": "…/daily_refunds.csv"}], "connection": {…}, "options": {…}}It replies with one delivered for the whole set. Exactly one of local_path or files is
present. Core only sends files to a plugin that advertised multi_file; any other plugin gets
one deliver per file. A single-file output is always sent in the local_path form.
Messages
Section titled “Messages”A message output (DRE 0.3) renders a title and text. To a destination that advertises
message, core sends it in one deliver:
{"type": "deliver", "message": {"title": "Daily revenue", "text": "Revenue **€12,340** (+4.1%)", "path": "/…/target/run/daily/default/daily.md"}, "files": [{"local_path": "/…/target/run/daily/default/detail.xlsx"}], "connection": {…}, "options": {…}}titleis one line of plain text.textis the portable Markdown subset:**bold**,*italic*or_italic_,`code`,[text](https://github.com/get-dre/dre/blob/v0.13.2/docs/url),-bullets and backslash escapes (\*); no headings or tables. Core has escaped every value the template printed, so the plugin translates the whole text to its service’s dialect (dre_protocol::markdownhas the parser and translations to Slack, Google Chat, Teams, HTML and plain text).html, when present, is an HTML body the plugin may use instead of convertingtext. Core 0.3 never sends it; it leaves room for a future HTML output.pathis the.mdfile core wrote (# title, a blank line, the text), for attaching the full message when the text is over the service’s limit.filesholds the files of other outputs the entry names inattach:(often none). Neitherlocal_pathnorremote_pathis set.
A destination that doesn’t advertise message gets a message output as its .md file, in the
usual local_path form. Core never sends a file output to a destination that advertises
message_only: dre validate refuses it, as it refuses attach: to a destination that takes
only messages or only files. Older plugins, which advertise neither, keep working for files.
Errors and failure
Section titled “Errors and failure”- A plugin that exits, crashes or closes stdout unexpectedly is reported with its exit status and its last stderr lines.
- A malformed frame from the plugin is reported as a protocol error. A plugin’s stdout carries frames only: the Rust SDK keeps the original stdout for frames and points the process’s stdout at stderr, so a dependency that prints can’t corrupt the stream (dre-protocol 0.7).
- A plugin that panics or hits an internal error should reply
errorbefore exiting. The SDK does this.
Versioning
Section titled “Versioning”Version 1 added request ids, cancel, log, progress and the error kind and code. Core
Core requires version 1: a plugin that only speaks version 0 is refused with a message naming
dre plugin update <package>.
The protocol version is a single integer. Core and plugins release independently, so each side states the range it supports and the handshake picks the highest common version.
Within a wire version, optional fields may be added only when their absence preserves the old
behavior and peers ignore fields they do not recognize. New requests, replies and exchanges
must be gated by an advertised capability or an explicit request opt-in before either peer
sends them. For example, need_fonts is allowed only after write.font_requests: true.
A peer must continue to serve the existing exchange when the other peer does not opt in.
Removing or renaming messages or fields, adding required fields, changing existing meanings, or changing framing or Arrow encoding requires a new wire version. A plugin built for wire version 1 continues to work with hosts supporting version 1 under these rules.
The Rust SDK package version is separate from the wire version. Request and Response are
#[non_exhaustive]: downstream matches must include a wildcard. This permits Rust enum
additions; it does not make unknown wire messages valid or bypass capability negotiation.
dre-protocol contains the plugin SDK, frames, messages and language-neutral conformance
fixtures. Process management and the Rust conformance runner live in dre-plugin-host.
When upgrading from SDK 0.11 to 0.12, plugin tests add dre-plugin-host as a dev-dependency
and replace dre_protocol::host and dre_protocol::conformance imports with
dre_plugin_host::host and dre_plugin_host::conformance. Plugin implementations continue
to depend on dre-protocol; existing plugin binaries need no rebuild for this split.
dre-core, dre-plugin-host and dre-template (the template context and filters core and the
document plugin share) are implementation libraries for the matching CLI release;
their Rust APIs carry no stability guarantee. Integrations should use the CLI and published
JSON contracts, and plugins should use the protocol. Core’s template and run internals are
not an embedding API.
Conformance
Section titled “Conformance”The protocol is published as data, so a plugin in any language can be checked against it. Three
files in crates/dre-protocol/conformance are the
contract:
schema.json: a JSON Schema (draft 2020-12) for every message.$defs/Requestis any message core sends and$defs/Responseany message a plugin sends. Itsx-dreblock lists the protocol version range (min_version,max_version,core_min_version), every capability name and every errorkind. It is generated fromdre-protocol’s message types, and CI fails when the two differ; the Go SDK checks its constants against it.protocol.json: the conformance cases. Each is a request sequence and the properties the replies must have: the handshake and identity, versions 0 and 1, refusing an unsupported version, request ids,cancel,describe,validateandvalidate_connection, error replies (with a knownkindand a namespacedcode), malformed frames, a format’s write, a source’s session (streaming a result set,row_limit,no_result,check, a failed statement, cancelling a running statement), a destination’sdeliverforms, and exiting oncloseand at end of input. Every message a plugin sends is also checked against the schema.delivery.json: the delivery rules file destinations share (if_exists, temporary names andtemp_dir, retries,retry_wait,max_retry_wait,Retry-After, numbered names), as cases each SDK runs against its delivery code with an in-memory store.
Running the suite on your plugin
Section titled “Running the suite on your plugin”Build your plugin, then run the dre-conformance command from the dre-plugin-host crate
(cargo install dre-plugin-host) against the executable:
dre-conformance ./dre-plugin-mydb --session session.jsonIt prints every case (passed, failed with the reason, or skipped because it doesn’t apply to your plugin’s kind or capabilities) and exits 1 when one fails. The session cases run only for a source given a session file: a connection that opens, and a statement per role.
{ "connection": {"path": ":memory:"}, "statements": { "rows": "select * from range(10) t(n)", "empty": "select 1 as n where false", "no_result": "create temporary table t (n int)", "error": "select * from no_such_table", "slow": "a statement that runs until it's cancelled" }}rows returns at least 3 rows; slow runs for at least a few seconds. Leave out a role your
database can’t express; the cases that need it are skipped.
In Rust, dre_plugin_host::conformance::assert_conforms(path) (or assert_conforms_with with
conformance::Options, for a session or environment variables) fails a test on any failed
case. In Go, plugintest.Conformance(t, pkg, plugintest.Options{}) (module
github.com/get-dre/dre/go/plugin, which embeds a copy of the cases) runs the same cases against
a plugin.Package in-process. Every first-party plugin, Rust and Go, passes the suite in its
tests.
Writing a runner
Section titled “Writing a runner”A runner in another language reads protocol.json and, for each case that applies, starts the
plugin and runs the steps in order. A case applies when the plugin’s kind is in kinds (all
kinds when absent), it advertises every capability in requires, and the runner has a session
with every statement in session. A case with "scope": "package" runs once per package
executable rather than once per plugin.
Unless a case’s first step is a hello, the runner first sends a hello with its own version
range (naming the plugin, for a package) and expects a hello reply. Steps:
| Step | What the runner does |
|---|---|
{"hello": {...}} |
Send a hello, its fields overriding the default range (plugin too) |
{"send": {...}} |
Send the JSON message as given |
{"send_raw": "<hex>"} |
Write these bytes as they are |
{"send_arrow": {"columns": [{"name", "type", "values"}], "rows": n}} |
Send an Arrow frame of n rows: each column’s values (null for a null), or n nulls without them; types int64, float64, utf8, boolean |
{"result_set": {"id": n, "rows": m}} |
Read Arrow frames (at least one, its columns those of the last result reply), then a result_end with id n whose rows is the rows sent and matches m |
{"close_input": true} |
Close the plugin’s stdin |
{"exit": m} |
Wait for the process to exit; its exit code must match m |
A step with expect then reads the next message other than log and progress and matches it
against expect; with "or_exit": true, the plugin exiting instead also passes. A plugin has
20 seconds for each reply.
Matching: an object matches an object that has at least its keys, each matching; a list
matches a list item by item; any other value matches an equal value (numbers by value). An
object whose keys start with $ is a matcher:
| Matcher | Matches |
|---|---|
{"$absent": true} |
a missing (or null) value |
{"$any": true} |
any value that is there |
{"$one_of": [m, ...]} |
a value matching one of them |
{"$type": "string"} |
a value of that JSON type (integer too) |
{"$nonempty": true} |
a non-empty string or list |
{"$len": n} |
a string of n characters or a list of n items |
{"$contains": m} |
a string containing the text m, or a list with an item matching m |
{"$every": m} |
a list whose items all match m |
{"$in": [v, ...]} |
one of these values |
{"$min": n}, {"$lt": n} |
a number at least n, or below n |
{"$file_exists": true} |
a path to an existing file |
Before a case runs, every ${name} in it is replaced: a string that is only a placeholder
becomes the value, whatever its type; elsewhere it is spliced in as text. The names are kind,
name and plugin (<kind>/<name>), tmp (an empty directory for the case), min_version
and max_version (the runner’s range), far_version (a version no plugin speaks),
capabilities and error_kinds (from the schema), connection, and sql.<role> for each
session statement.