rivet init — config scaffolding

rivet init connects to PostgreSQL, MySQL, SQL Server, Oracle, or MongoDB, introspects tables (collections on MongoDB), and prints a YAML scaffold you can save and edit before running rivet check / rivet run.

Generated configs use url_env: DATABASE_URL so secrets are not embedded in the file. Set DATABASE_URL (or switch to url: / structured credentials) before running exports.

Modes

Single table

Provide --table (optionally schema-qualified: public.orders on PostgreSQL, dbo.orders on SQL Server).

export DATABASE_URL='postgresql://user:pass@localhost:5432/mydb'
rivet init --source "$DATABASE_URL" --table orders -o rivet.yaml

# Qualified name (PostgreSQL)
rivet init --source "$DATABASE_URL" --table analytics.facts -o rivet.yaml

export DATABASE_URL='mysql://user:pass@localhost:3306/mydb'
rivet init --source "$DATABASE_URL" --table orders -o rivet.yaml

Rivet emits one export block: SELECT of all columns, a suggested mode (full, incremental, or chunked) from row estimates and column types, plus chunk_* or cursor_column when applicable. Every scaffold uses format: parquet and, by default, meta_columns with exported_at: true and row_hash: true (lineage and row fingerprinting in the output — see exports[].meta_columns in config). When the heuristic picks chunked, the scaffold also includes chunk_checkpoint: true (resumable runs, rivet run --resume, and reconcile/repair — see chunked mode).

Whole PostgreSQL schema

Omit --table. All base tables and views in the target schema are introspected; the file contains one export per object, sorted by name.

• --schema — PostgreSQL schema name (default: public).

export DATABASE_URL='postgresql://user:pass@localhost:5432/mydb'
rivet init --source "$DATABASE_URL" --schema public -o rivet_all_public.yaml

# Non-default schema
rivet init --source "$DATABASE_URL" --schema analytics -o rivet_analytics.yaml

The database itself comes from the connection URL path (/mydb).

Whole MySQL database

Omit --table. All base tables and views in the database are listed from information_schema.

• If the URL already includes the database (mysql://.../mydb), that database is used.
• If the URL has no database path, pass --schema <database> (same flag name as for Postgres; on MySQL it selects the database name for listing).

rivet init --source 'mysql://user:pass@localhost:3306/rivet' -o rivet_mysql.yaml

# URL without database — name it explicitly
rivet init --source 'mysql://user:pass@localhost:3306/' --schema rivet -o rivet_mysql.yaml

Heuristics (suggested mode)

The table above applies to the SQL engines (PostgreSQL / MySQL / SQL Server). MongoDB is schemaless — rivet init introspects no columns, primary keys, or cursor / chunk candidates — so every collection scaffolds mode: full (one export per collection), regardless of document count. MongoDB’s only batch mode is full; use --mode cdc for change capture.

Chunked exports and checkpointing

When the suggested mode is chunked, the scaffold always includes chunk_checkpoint: true. That enables resumable chunk runs after crashes or transient errors (rivet run --resume), chunk state in rivet state chunks, and reconcile/repair workflows. Set it to false only if you intentionally do not want checkpoint state on disk.

Meta columns (defaults)

The YAML scaffold enables exported_at and row_hash for every export. Set either to false, or remove the meta_columns block entirely, to turn them off.

Row estimates are cheap metadata (pg_class.reltuples on PostgreSQL, information_schema.TABLES.TABLE_ROWS on MySQL), not exact COUNT(*).

Always run rivet check --config <file> and adjust modes, destinations, and tuning before production runs.

DECIMAL / NUMERIC column overrides

Rivet reads numeric_precision and numeric_scale from information_schema.columns during introspection. When a NUMERIC or DECIMAL column has explicit precision and scale, the scaffold automatically emits a columns: block with the correct decimal(p,s) override — so exports don’t fail at runtime with an “unsupported type” error:

exports:
  - name: payments
    query: >
      SELECT id, amount, fee
      FROM payments
    mode: chunked
    chunk_column: id
    chunk_size: 100000
    chunk_checkpoint: true
    format: parquet
    columns:
      amount: decimal(18,2)
      fee: decimal(18,6)
    destination:
      type: local
      path: ./output

If the column is declared as plain NUMERIC (no precision / scale in the DDL), rivet init still emits columns: so exports run: it uses decimal(38,18) as a wide default (Decimal128 in Arrow), prefixes the YAML header with a # NOTE: pointing at these lines, and adds # REVIEW: inline on each such column — plus a rivet: note line on stderr when you write rivet init -o <file>. Replace the defaults with precision/scale from your domain (or constrain the DDL) before trusting the export:

    columns:
      price: decimal(38,18)  # REVIEW: DDL has no numeric(p,s); edit to the real decimal(p,s) or change the column type — values outside this bound may truncate or fail export.

CDC stream objects (--mode cdc)

A CDC scaffold written with -o <file> names its stream objects after the file, so two configs on one cluster never read each other’s stream: the PostgreSQL slot is rivet_<stem>_<table> (rivet_<stem>_<schema>_cdc for a whole-schema stream), the stem being the file name in lowercase letters, digits and _ (-o staging.yaml -> rivet_staging_public_cdc), cut to PostgreSQL’s 63 bytes; the MySQL server_id is 1000000 plus a stable hash of the stem below 1000000, plus the export’s position among the config’s CDC exports. A scaffold printed to stdout keeps rivet_<table> and server_id: 4271 and says beside each that it is one per cluster. A CDC export of a table is named <table>_cdc and writes to exports/<table>_cdc/, beside the batch export of the table, never inside it.

The checkpoint and the CDC destination are named after the file too (since 0.32.2): -o rivet.yaml writes checkpoint: ./cdc/rivet_<table>.ckpt (./cdc/rivet_<schema>_cdc.ckpt for a whole-schema stream) and the destination ./output/rivet/<table>_cdc/ (exports/rivet/<table>_cdc/ in a bucket, rivet/cdc/ / ./output/rivet/cdc/ for a whole-schema stream). init refuses RIVET_INIT_CDC_NAME_TAKEN (exit 5), writing nothing, when that checkpoint already exists or another config in the directory names the same checkpoint or destination: write the config under another -o name. A run records which config and export own a checkpoint (<checkpoint>.owner) and refuses RIVET_SOURCE_CDC_CHECKPOINT_SHARED (exit 5) for another config while the owner still names it.

Flags (summary)

Avoiding credentials on the command line

Shell history, process listings (ps, /proc/<pid>/cmdline), and container inspect logs all capture --source "postgresql://user:pass@host/db" verbatim. For anything beyond local dev, use --source-env or --source-file:

# Recommended — env var resolved inside the process only.
export DATABASE_URL='postgresql://user:pass@host:5432/db'
rivet init --source-env DATABASE_URL --schema public -o cfg.yaml

# File-based — useful when the URL is managed by your secrets mount.
rivet init --source-file /run/secrets/database_url --table orders -o cfg.yaml

Exactly one of --source, --source-env, --source-file must be provided (enforced by clap’s ArgGroup).

Discovery artifact (--discover)

rivet init --discover runs the same introspection but emits a machine-readable JSON document (schema described in src/init/artifact.rs). Intended consumers: external orchestration tools, code review, and automated config generators.

rivet init --source "$PG_URL" --schema public --discover -o discovery.json
rivet init --source "$MY_URL" --table orders   --discover    # pipes JSON to stdout

Per-table fields (tables[]):

The artifact is advisory — same policy as plan prioritization (ADR-0006): no runtime effect, no auto-application.

Docker Compose in this repository

The repo root docker-compose.yaml defines Postgres and MySQL (rivet / rivet users, database rivet) with the same schema as dev/postgres/init.sql and dev/mysql/init.sql.

docker compose up -d postgres mysql
export PG_URL='postgresql://rivet:rivet@localhost:5432/rivet?sslmode=disable'
export MY_URL='mysql://rivet:rivet@localhost:3306/rivet'

# One table
rivet init --source "$PG_URL" --table orders -o rivet_orders.yaml

# Whole PostgreSQL schema public
rivet init --source "$PG_URL" --schema public -o rivet_public.yaml

# Whole MySQL database from URL
rivet init --source "$MY_URL" -o rivet_mysql.yaml

To refresh many files at once (per-table YAMLs plus combined schema snapshots), run python3 -m dev.pytools.dev_scripts regen-docker-configs from the repo root after the DBs are up (and optionally seeded).

Bucket destination: --gcs-bucket / --s3-bucket / --azure-container

Each flag writes the export’s destination: in that store, with the same prefix (exports/<table>/, cdc/ for a CDC stream); at most one of them, and none with --discover. No credential is written. GCS and S3 take their ambient credentials; an Azure scaffold names its account and the env var of the account key:

rivet init --source "$PG_URL" --table orders \
  --azure-container exports --azure-account myaccount -o rivet.yaml
export AZURE_STORAGE_KEY=...   # or put `sas_token_env: <VAR>` in place of account_key_env
rivet run -c rivet.yaml        # Parquet → az://exports/exports/orders/

    destination:
      type: azure
      bucket: exports
      prefix: exports/orders/
      account_name: myaccount
      account_key_env: AZURE_STORAGE_KEY

--azure-endpoint <URL> writes endpoint: for a store at another address, such as Azurite (http://127.0.0.1:10000/devstoreaccount1); without it the endpoint is derived from the account.

Warehouse scaffold: --bigquery-project / --bigquery-dataset / --gcs-bucket

With the three flags together, the generated config carries the warehouse half of the cycle — a top-level load: block (target: bigquery, pk: auto, cluster_by: auto, cleanup_source: true), a per-table partition: guess (the creation stamp — created_at / CreatedDate … — at granularity: day, never a mutation stamp, which would move a row between partitions on every update) and, for a mode that carries deltas (incremental, cdc), layout: base_buffer so rivet compact has a base to merge into (BigQuery; Snowflake takes it by default and the block writes no line). Every value is a guess from the catalog: review the block before the first load.

rivet init --source "$PG_URL" --table orders --mode incremental \
  --gcs-bucket my-bucket --bigquery-project my-proj --bigquery-dataset my_ds -o rivet.yaml
rivet run     -c rivet.yaml   # Parquet → gs://my-bucket/exports/orders/
rivet load    -c rivet.yaml   # → the base on the first pass, the buffer on later ones
rivet compact -c rivet.yaml   # MERGE the buffer into the base and drop it

--gcs-bucket is required with the BigQuery flags: rivet load reads GCS only, so a load: block over a local or S3 destination is a config its own next step refuses. For a whole-database CDC scaffold (one tables: stream with backfill: auto) the partition guesses are written on the stream’s load.tables.<table> blocks — the place the load reads them — not on the per-table recipes, which the load never reads.

Warehouse scaffold: --snowflake-* / --gcs-bucket

The same for Snowflake: --snowflake-account, --snowflake-user and --snowflake-private-key-path (the key-pair login; --snowflake-role is optional), --snowflake-warehouse, --snowflake-database, --snowflake-schema and --snowflake-storage-integration (a pre-created GCS STORAGE INTEGRATION) write a load: block with target: snowflake, pk: auto, cluster_by: auto, cleanup_source: true and a per-table partition: guess at granularity: day, which the loader turns into a leading DATE_TRUNC('DAY', …) clustering key (Snowflake has no partitions; a clustering key turns on billed Automatic Clustering, so drop the line to cluster by the key alone).

rivet init --source "$MYSQL_URL" --mode incremental --gcs-bucket my-bucket \
  --snowflake-account myorg-myaccount --snowflake-user RIVET_USER \
  --snowflake-private-key-path /path/to/rsa_key.p8 --snowflake-warehouse COMPUTE_WH \
  --snowflake-database ANALYTICS --snowflake-schema PUBLIC \
  --snowflake-storage-integration MY_GCS_INT -o rivet.yaml
rivet run  -c rivet.yaml   # Parquet → gs://my-bucket/exports/<table>/
rivet load -c rivet.yaml   # COPY → ANALYTICS.PUBLIC.<table>

All five flags are required together (the loader needs every value), and --gcs-bucket is required: Snowflake reads the export through a GCS external stage, so --s3-bucket is refused. One config loads into one warehouse, so the Snowflake flags are refused beside --bigquery-* or --clickhouse-*. On Snowflake an incremental or CDC export gets the default layout, which writes no layout: line (a physical <table> plus a <table>__changes buffer that rivet compact folds in); write layout: log_view to read a <table>__changes log through the deduplicated <table> view instead. A CDC stream with no baseline (a table named snapshot) is scaffolded with layout: log_view, because the default base would start empty and the config is refused.

Limitations

• Not a migration or DDL tool — only read-only introspection and YAML output.
• Views are included in schema-wide / database-wide runs; ensure each view is selectable for your user.
• Suggested modes are heuristics; large or sparse tables may need manual chunked / chunk_by_key / chunk_by_days tuning (see chunked mode).