Cloud Destinations

Single-page tour of every destination Rivet ships with: what they have in common, where they differ, and what guarantees you can build downstream infrastructure on top of.

For the per-backend deep dive, read the dedicated pages:

For the credential matrix (env vars, profile files, identity providers), see docs/cloud-auth.md.

The supported destination set is exactly: AWS S3, Google Cloud Storage, Azure Blob Storage, the local filesystem, stdout — plus their loopback dev emulators (MinIO for S3, fake-gcs-server for GCS, Azurite for Azure), which CI exercises. Other S3-compatible services (Cloudflare R2, Wasabi, Backblaze B2), sovereign clouds, and custom-DNS endpoints are untested and not supported; the config-load endpoint guard rejects them by design.

Common output contract

Every non-streaming destination (Local, S3, GCS, Azure) produces the same three artefacts at the resolved prefix on a clean run:

_SUCCESS and a success manifest say that the prefix holds a whole export. They do not say that the last run succeeded: a run that fails before its first write (the source cannot be reached, the password is wrong, the run is refused when it opens) leaves the previous _SUCCESS, manifest.json and every other object exactly as they were, and tells its failure through its exit code and the run journal (rivet metrics). Only on a prefix that holds no manifest yet does it leave its own failed manifest (0 parts, no _SUCCESS), so rivet validate exits 1 on a prefix whose only run failed. A run that put a part at the destination and then failed removes _SUCCESS and writes a failed manifest naming the parts it wrote, because the prefix is no longer the export the marker described. --resume therefore reads the run journal beside the marker: an export whose last journaled run failed is run again by rivet apply <config> --resume and is not refused by rivet run --resume, even though an earlier run’s _SUCCESS is still at its prefix.

The contract is atomic at write boundaries, not at the prefix: manifest.json is written before _SUCCESS, so an Airflow / CI sensor that polls for _SUCCESS never sees a half-built manifest. A HEAD _SUCCESS is cheap enough that downstream consumers should prefer it over GET manifest.json for the “data ready?” signal.

Schema and resume semantics:

• schema.json is reserved for a future release (per-run schema snapshot — see the roadmap). Today the schema fingerprint lives inside manifest.json under schema_fingerprint.
• _quarantine/<run_id>/ lands on resume when M9 finds an untracked or corrupt part. The original byte location is preserved under that subtree for forensics.

Authentication modes

Local

No credentials. Permissions come from the OS. Use path: (with optional {date}/{table}/{export}/{run_id} placeholders) to point at the output directory.

Amazon S3

region: is optional when the SDK can derive one from the profile or env vars; required otherwise. endpoint: overrides the resolved S3 endpoint. Only a loopback endpoint (MinIO) is a supported custom-endpoint path; any non-loopback endpoint (AWS GovCloud, Cloudflare R2, Wasabi, custom domains) is rejected at config load as an exfiltration guard. The one waiver is allow_anonymous: true — the anonymous-emulator escape, which sends no credentials at all; it is not an auth path for those services, which remain untested / not supported as Rivet destinations (see cloud-auth.md, “S3-compatible storage”).

Google Cloud Storage

bucket: is required. endpoint: overrides the default storage.googleapis.com.

Azure Blob Storage

account_key_env and sas_token_env are mutually exclusive — picking both is refused at config-load time with a message that names both fields. account_name is the prefix in <account>.blob.core.windows.net; an explicit endpoint: takes precedence over the derived URL, but only a loopback emulator endpoint (Azurite) is accepted alongside credentials — a non-loopback endpoint is rejected at config load unless allow_anonymous: true, which itself cannot be combined with credentials, so sovereign clouds are not reachable.

The Azure SAS-token body may be pasted with or without the leading ? — Rivet trims it transparently so sv=…&sig=… and ?sv=…&sig=… are both accepted.

Future Azure modes (Managed Identity, Service Principal, workload identity federation) are on the roadmap but not yet supported.

Manifest + _SUCCESS

The trust contract is the same on every cloud backend. See ADR-0012 for the formal invariants:

• M1: parts before manifest.
• M2: _SUCCESS carries the fingerprint of the exact manifest.json bytes — fingerprint drift means something else wrote that prefix.
• M5: with _SUCCESS present, every listed part exists at recorded size and content fingerprint.
• M6: legacy prefixes (no manifest.json) are surfaced as legacy_run: true; rivet validate returns success without certifying. A bucket or container that does not exist is not a legacy prefix: rivet validate exits 1 with RIVET_VERIFY_MANIFEST_READ_ERROR naming it.
• M8: --resume against a _SUCCESS-marked prefix is refused without --force (RIVET_DEST_ALREADY_COMPLETE, exit 5). --resume --force continues an interrupted run of the export if there is one, else runs as a plain run does (a full export is exported again, a delta export continues past its cursor): new parts land beside the old ones, nothing is overwritten, and manifest.json then describes only the new run.
• M9: untracked or corrupt parts encountered on resume are moved under _quarantine/<run_id>/ rather than deleted.

Resume behavior

rivet run --resume walks the destination prefix, cross-checks against the state DB and the prior manifest, and decides per-chunk:

• skip — chunk already committed.
• rewrite — chunk was in-progress (or its part is missing); re-export the same key range.
• quarantine — untracked or corrupt object at the chunk’s part path; move to _quarantine/<run_id>/ and re-export.

A resume writes manifest.json and _SUCCESS only when the run finishes cleanly. Over a prefix that already holds _SUCCESS (a --resume --force), a resume that fails before it puts a part at the destination leaves that marker and its manifest in place, so a sensor polling _SUCCESS still sees the last whole dataset; a resume that wrote a part and then failed removes _SUCCESS and leaves a failed manifest.

Validate behavior

rivet validate is the standalone counterpart to rivet run --validate (see docs/destinations/ for the per-backend nuance). It never queries the source: only HEAD / GET _SUCCESS / GET manifest.json against the resolved destination.

Validate flags:

• --date YYYY-MM-DD — resolve {date} against the given day instead of today. The flag a “did yesterday’s run land cleanly?” Airflow sensor needs.
• --run-id <RID> — substitute {run_id} in the destination template (composes with --date).
• --prefix <STRING> — bypass placeholder resolution and verify exactly that prefix. Refused with multiple exports — see the inline error.

The resolved physical prefix is surfaced in both --format pretty and --format json output (resolved_prefix) so it’s obvious which bytes were checked.

Reconcile and repair

Both honor the same placeholder resolver as run and validate.

Quarantine behavior

Path layout under the destination prefix:

<prefix>/
  part-000001.parquet
  part-000002.parquet
  manifest.json
  _SUCCESS
  _quarantine/
    <run_id>/
      part-XXXXXX.parquet   ← evicted object kept verbatim

Quarantine is best-effort: a successful copy followed by a failed delete leaves the object reachable at both paths. M9 re-trips on the next resume and the orphan eventually gets moved.

Support matrix

Known limitations

• rivet load --pool N opens files in proportion to N — every table loaded at once holds its own state connection, its own lock file and its own BigQuery connections. Its storage requests share one budget of 16 in flight for the whole process, so storage connections stay near 16 however large the pool is (a connection can outlive its request for a moment: 16 were counted on a release build, up to 22 on a debug build). Measured on macOS, GCS to BigQuery with a SQLite state: 26 open files at --pool 1, 50 at --pool 4, 117 at --pool 16. The default shell limit on macOS is 256 open files (ulimit -n); Linux shells and containers usually start at 1024 or more. Raise the limit or lower --pool when a load fails with Too many open files. Storage requests go over HTTP/1.1, one connection per request in flight; HTTP/2, which would carry them over one connection per host, is not enabled.
• Object lifecycle policies — Rivet does not configure retention, lifecycle transitions, encryption-at-rest, or replication rules on the destination. Manage those out-of-band (Terraform, console).
• Incomplete multipart uploads are not aborted on failure — when a streamed (large-part) upload fails mid-transfer, the multipart upload is left open: its already-uploaded parts are billed but invisible to listings, and Rivet currently has no abort call on that error path (a crash could never run one anyway). Configure an abort-incomplete-multipart-upload lifecycle rule on every destination bucket (S3: AbortIncompleteMultipartUpload, e.g. 7 days; GCS/Azure: the equivalent incomplete-upload cleanup) — this is load-bearing hygiene, not an optimization.
• Azure SAS expiry — SAS tokens are short-lived by design. Rivet reads the env var once at process start; a long-running export whose token expires mid-flight will fail at the next write. Pair short SAS validity with appropriately small exports, or use account-key auth.
• Eventual consistency on first list — S3 / GCS / Azure are read-after-write consistent for new objects but list operations can lag on some backends. This affects --resume only, which uses list_prefix; the validate path uses targeted HEAD requests and is consistent.
• Network egress costs — Rivet does not enforce a destination / source region affinity. Exporting a 100 GB table from us-east-1 Postgres to an eu-west-1 bucket goes the long way around at full egress price. Pin source and destination to the same region for any non-trivial workload.

Reporting trust-contract issues

A trust-contract violation is treated as a security-grade bug. See SECURITY.md for the disclosure channel. Examples:

• _SUCCESS present, but a part listed in manifest.json is missing.
• _SUCCESS fingerprint disagrees with the bytes of manifest.json.
• manifest.json references the wrong schema_fingerprint for the parts at the prefix.
• A credential ever appearing in any artefact (manifest.json, summary.json, journal events, log lines).