YAML Config Guide

📌 The exhaustive, always-current field reference is generated from the code: config-reference.md — rendered from rivet schema config (the schemars-derived JSON Schema), so it cannot drift and needs no manual verification. This page is the guide: the same options with defaults, rationale, and worked examples. For the guaranteed-current field/type/enum list, trust the generated reference.

The most-used options, grouped by section, with the why and examples.

Root

source

Connection approaches (mutually exclusive):

1. URL-based: provide exactly one of url, url_env, or url_file
2. Structured: provide host, user, database (+ optional port, password/password_env)

TLS

Example (production):

source:
  type: postgres
  url_env: DATABASE_URL
  tls:
    mode: verify-full
    ca_file: /etc/ssl/certs/rds-ca-2019-root.pem

Example (local dev only — no TLS):

source:
  type: mysql
  host: 127.0.0.1
  port: 3306
  user: dev
  password_env: DEV_PWD
  database: rivet
  tls: { mode: disable }       # explicit opt-out — silences the plaintext WARN

Example (SQL Server — sqlserver:// scheme, port 1433):

source:
  type: mssql
  url_env: MSSQL_URL           # sqlserver://user:pass@host:1433/database
  tls:
    ca_file: /etc/ssl/certs/your-sql-server-ca.pem   # private CA, or:
    # accept_invalid_certs: true                      # self-signed dev cert

SQL Server always encrypts the login handshake, so TLS is on regardless; the tls: block only controls how the server certificate is trusted. Supported export modes and types are listed in compatibility.md.

When tls: is omitted entirely, Rivet connects without TLS and emits a WARN so you notice. See reference/compatibility.md for which servers ship TLS-ready and Rivet’s dev-environment defaults.

Credentials & plan artifacts

A PlanArtifact (produced by rivet plan) is designed to be committed / reviewed; it must not carry plaintext credentials. Rivet enforces ADR-0005 PA9 (SourceConfig::redact_for_artifact):

• password: field → always stripped from the artifact (set to None).
• url: containing scheme://user:pass@… → userinfo rewritten to REDACTED.
• password_env / url_env / url_file → preserved as references so apply-time can re-resolve against the apply-environment.

When redaction runs, rivet plan logs:

WARN plan 'orders': plaintext credentials stripped from artifact —
     apply time must have equivalent env/file-based auth available

Recommendation: use password_env (or url_env) everywhere; only use plaintext password: for one-off local scripts. See ADR-0005 PA9.

exports[]

Each entry in the exports list defines one export job.

Compression profiles

compression_profile is the recommended way to pick a codec. It maps to a (codec, level) pair and takes precedence over any compression / compression_level fields.

exports:
  - name: events
    format: parquet
    compression_profile: balanced    # zstd level 3
    destination: { type: local, path: ./out }

If you need a specific codec that is not covered by the presets, use compression + compression_level directly and omit compression_profile.

Verification depth

verify controls how thoroughly --validate (and rivet validate) checks each part at the destination:

• size (default) — confirm each part exists at its recorded size_bytes, plus manifest self-consistency and _SUCCESS. Content is also MD5-checked for free whenever the store surfaces a checksum in its listing, but a part without one is accepted as size-only.
• content — require every part’s content MD5 to match the store’s listing checksum (no download). Any part that could only be size-verified fails validation with an actionable message.

How content verification works: Rivet computes each part’s MD5 before upload and records it in the manifest; GCS and Azure compute their own for a part uploaded as a single PUT and return it in object listings. --validate compares the two with no download. Parts large enough to stream as multipart / block-list get no checksum, and neither does S3 (its ETag is not an MD5 under SSE-KMS / SSE-C, so rivet does not trust it) or local FS. Under verify: content on GCS / Azure, set destination.oneshot_budget_mb (default 64 MB) comfortably above your part size: the budget is shared by concurrent uploads, so a part one-shots only if it fits what is free at that moment. S3 cannot meet verify: content.

The run report and rivet validate show coverage explicitly, e.g. 3 verified (2 md5, 1 size-only).

Parquet row group tuning

Parquet row groups affect memory usage during write, compression ratio, and downstream query performance (predicate pushdown, column skipping). When parquet: is omitted, Rivet uses the library default of 1,048,576 rows per group, which is optimal for narrow tables but can be large for wide tables.

exports:
  - name: events
    format: parquet
    parquet:
      row_group_strategy: auto          # auto | fixed_rows | fixed_memory
      target_row_group_mb: 128          # target Arrow buffer size per group (auto + fixed_memory)
      max_row_group_mb: 256             # optional upper bound (all strategies)

Examples:

# Auto-tune for a wide JSON table — groups sized to ~64 MB
parquet:
  row_group_strategy: auto
  target_row_group_mb: 64
  max_row_group_mb: 128

# Fixed row count — useful when downstream tooling requires exact group sizes
parquet:
  row_group_strategy: fixed_rows
  row_group_rows: 500000

Note: rivet plan shows the selected strategy and target in the Format section when parquet: is configured.

rivet init auto-generates this block for chunked exports and large full-mode tables, pre-selecting target_row_group_mb: 64 for wide schemas (≥ 5 text/JSON/bytea columns) and 128 for narrow ones.

exports[].on_schema_drift — schema drift policy

Controls what Rivet does when it detects a structural change in the output schema (column added, removed, or retyped) compared to the snapshot stored from the previous run.

fail is useful in CI pipelines where schema changes must be reviewed before the new shape is exported downstream.

exports:
  - name: orders
    on_schema_drift: fail

When fail triggers, behavior depends on the runner: in single, keyset, and parallel-Mongo modes the schema check runs post-extraction, so the output file has already been written to the destination (but no cursor advance or manifest commit occurs). In chunked mode the check runs pre-chunk from a scan-free type probe, so the run aborts before any chunk is written. Re-run after confirming the schema change is intentional, or switch to warn to accept it.

exports[].columns — per-column type overrides

Override the Arrow type Rivet infers for a specific column. Useful when:

• a NUMERIC / DECIMAL column has no explicit precision/scale in the source schema (beyond rivet init’s default decimal(38,18) placeholder), or
• you need a narrower precision for BigQuery NUMERIC compatibility.

columns:
  <column_name>: <type>          # applies to every captured table with the column
  "<table>.<column>": <type>     # applies to ONE table, wins over the bare key

Supported override types: decimal(p,s) / numeric(p,s) (both precision and scale required; precision > 38 produces Decimal256), integer widths (smallint/int/bigint/int16/int32/int64), floats (real/float4/double/float8), bool, text/string/varchar, uuid, json/jsonb, date, the timestamp* family (naive / tz / _ns variants), and binary (bytea/binary/varbinary/blob).

Overrides apply to batch and CDC identically (the same resolution surface). Key shapes: a bare column name applies to every captured table that has the column — on a multi-table CDC export that means ALL of them; a qualified "table.column" key targets one table and wins over the bare key there. A qualified key naming a table the export does not capture — or used on a query-shaped export — is a config error at load.

Example:

exports:
  - name: orders
    query: "SELECT id, amount, fee FROM orders"
    format: parquet
    destination:
      type: local
      path: ./out
    columns:
      amount: decimal(18,2)
      fee: decimal(18,6)

rivet init generates these automatically. When introspecting a table, rivet init reads numeric_precision and numeric_scale from information_schema.columns. If both are present, it emits a concrete override (decimal(p,s)). If the column is unbounded (NUMERIC without explicit precision), rivet init emits a working default decimal(38,18) plus a # REVIEW: YAML comment — the config header adds a # NOTE: line, and rivet init -o … prints a stderr reminder so you tighten precision when you know the real domain rules:

    columns:
      price: decimal(38,18)  # REVIEW: DDL has no numeric(p,s); edit to the real decimal(p,s) …

Type overrides are applied at export time and are reflected in rivet check --type-report output.

Some PostgreSQL types have no Arrow representation and cannot be exported directly. Rivet will report an error listing all unmappable columns before the run starts.

Use a SQL expression in your query field to work around any unsupported type:

exports:
  - name: locations
    query: >
      SELECT id, name, ST_AsText(geom) AS geom_wkt
      FROM locations
    format: parquet
    destination:
      type: local
      path: ./out

Rivet exports the WKT text as a Utf8 (string) column. Downstream tools (DuckDB, GeoPandas, QGIS) can reconstruct geometry from WKT.

Mode-specific fields

Incremental (mode: incremental):

Chunked (mode: chunked):

Time-window (mode: time_window):

exports[] — value-based partitioning

Splits a full, chunked or incremental export’s rows into Hive-style col=value/ destination sub-folders by a date column. See partitioning.md.

exports[].meta_columns

exports[].quality

Uniqueness tracking uses typed xxHash3-64 internally — numeric and binary columns are hashed directly from raw bytes without string formatting. unique_max_entries is the primary knob to control memory on very large tables.

Example:

quality:
  row_count_min: 100
  null_ratio_max:
    email: 0.05          # email must be <5% null
  unique_columns:
    - id
    - email
  unique_max_entries: 1000000   # stop after 1M unique values; warn if limit hit

Without unique_max_entries — tracking is unbounded. Safe for tables with hundreds of thousands of rows; may use significant RAM on tables with tens or hundreds of millions of distinct values.

With unique_max_entries — tracking stops at the limit and the run summary shows a warning. Duplicates found before the limit still fail the run; ones past it go unseen. Use when you want a best-effort uniqueness check without memory risk.

exports[].destination

The complete per-backend field list (local / s3 / gcs / azure / stdout) is in the generated config-reference.md (section exports[].destination). Per-backend setup, auth flows, and permissions: destinations/ — local · s3 · gcs · azure · stdout, plus the cloud auth matrix.

A relative path is anchored to the config file

A relative local path (./output/orders) names the directory beside the config file, whatever directory rivet is started from: rivet run -c /etc/rivet/rivet.yaml from a cron entry and from a shell in /home/op both deliver to /etc/rivet/output/orders, where the state (.rivet_state.db) and a cdc.checkpoint already live. Every command resolves it the same way, once, when the config is loaded: run, plan/apply, validate, reconcile, repair, state, doctor and load.

Before 0.32 the path was relative to the working directory, so only a run started outside the config’s directory lands elsewhere now; a cursor stored by such a run continues under the anchored path. While the working-directory place still holds the export’s files (its manifest.json) and the directory beside the config holds none, every command is refused with RIVET_DEST_RELATIVE_PATH_MOVED (exit 5) naming both directories: move the files beside the config, or make the path absolute. A path that should not move with the config is written absolute.

Path and prefix placeholders

The path (local) and prefix (S3 / GCS) fields support template placeholders, substituted when the export runs. A plan sealed by rivet plan keeps {date} as written and rivet apply resolves it to the UTC day it runs on; rivet plan --date YYYY-MM-DD freezes that day into the plan instead.

destination:
  type: s3
  bucket: my-data
  prefix: exports/{date}/{export}/
  region: us-east-1

With an export named orders running on 2026-05-14, this resolves to exports/2026-05-14/orders/.

For an export that continues from a stored cursor (incremental, keyset_incremental, MongoDB resume), the destination and the format are compared between runs. The destination is compared as written, with the placeholder in place: exports/{date}/{export}/ is the same destination on every day. Another bucket, path or prefix, another store or another format is refused (RIVET_STATE_CURSOR_DESTINATION_MISMATCH, exit 5), because the new location would hold only the rows past the cursor: restore the setting to continue, or run rivet state reset -c <config> --export <name> for a full pass delivered where the config points now. A value that comes from ${VAR} is compared after substitution, so a destination that should change from run to run is spelled with {date}.

A cursor is also tied to the source it was read from. A run whose source key (engine://host:port/database, credentials and query left out) holds no progress for the export, while another source key of the same export holds a cursor that delivers to the same destination, is refused (RIVET_STATE_CURSOR_SOURCE_RESPELLED, exit 5): it is the same server reached under another address, and the run would read the whole table again beside the rows the cursor delivered. rivet state accept -c <config> --export <name> moves the progress onto the source key the config names now; rivet state accept followed by rivet state reset -c <config> --export <name> starts the export over for two different servers that share a destination. rivet plan and rivet apply <plan> refuse the same way. A source with a destination of its own is never compared with another.

notifications

notifications.slack

Example:

notifications:
  slack:
    webhook_url_env: SLACK_WEBHOOK
    on: [failure, schema_change]

Environment variable interpolation

Any string value can reference environment variables:

source:
  url: "postgresql://${DB_USER}:${DB_PASS}@${DB_HOST}:5432/mydb"

Query parameters

Queries can use ${key} placeholders filled by --param key=value:

exports:
  - name: filtered
    query: "SELECT * FROM orders WHERE region = '${region}'"

rivet run --config export.yaml --param region=us-east

A placeholder is filled from --param first, then from the environment variable of the same name; one that neither supplies stops the command before anything runs.

For an export that stores a cursor (incremental, keyset), the query is compared between runs as written, with the placeholder in place: another value of ${region} continues from the stored cursor, and which rows that leaves undelivered is yours to decide. An edit of the query text around the placeholder is refused like any other edit (RIVET_STATE_CURSOR_STREAM_MISMATCH; see rivet state accept).