Extracts to S3, GCS or Azure Blob

Another team or system wants a typed extract in a bucket, partitioned by date, every day. Selecting the rows is the easy part; naming the file, typing the columns and getting it to the right bucket is the part DRE takes off your hands.

Recorded from a real run: one command, then the file in each of the three stores.

↓ Download the parquet file (regions.parquet)

reports/extracts/extracts.yml

queries: [extract_regions]
output:
  format: parquet
  destination:
    - {profile: s3_keys, path: "s3://dre-demo/extracts/{{ run.date.yyyy }}/{{ run.date.mm }}/regions.parquet"}
    - {profile: gcs, path: "extracts/regions-{{ run.date.yyyymmdd }}.parquet"}
    - {profile: azure_cs, path: "extracts/regions-{{ run.date.yyyymmdd }}.parquet"}

reports/extracts/extract_regions.sql

select * from {{ ref('delivery_sample') }}

terminal

dre run extracts