ADR-0005: Plan/Apply Contracts

Status: Accepted
Date: 2026-04
Context: Rivet implements a Terraform-style plan/apply workflow for data extracts. rivet plan generates a sealed PlanArtifact that captures the full execution intent at a point in time. rivet apply consumes the artifact and executes it. Because plan time and apply time are separated, the contracts between them must be explicit to ensure predictable, auditable execution.

Two kinds of plan (owner decision, 2026-10-16). A plan whose strategy reads no stored cursor (mode: full, chunked ranges, keyset without keyset_incremental, time window) is a point-in-time snapshot: it is applied once, it ages (PA3), and a replay of it after an exit-0 apply extracts nothing (PA11). An incremental plan (mode: incremental, keyset_incremental, Mongo resume) is a reusable description of the extraction: it freezes the query, the strategy and the chunk parameters, not the position; every apply reads the cursor from state as rivet run does and delivers what arrived since — new rows or none, never a duplicate. PlanArtifact::is_reusable (the strategy’s continues_stored_cursor) is the one place this is decided.

Problem

Separating planning from execution introduces a temporal gap between analysis and action:

• Data in the source may change between plan and apply.
• The cursor may advance (another incremental run completed).
• The config file may be edited.
• The artifact may be arbitrarily old.

Without explicit contracts, rivet apply cannot reason about whether the artifact is still valid to execute, and operators cannot reason about what guarantees the system provides.

Contracts

PA1 — Artifact Is the Communication Channel (ACC)

A PlanArtifact is the sole input to rivet apply. Apply does not re-read the config file, re-run preflight queries, or re-compute chunk boundaries. Everything needed for execution is embedded in the artifact.

Rationale: Decoupling apply from config re-parsing enables apply to work from a sealed, auditable snapshot. The artifact can be stored as a CI artifact, committed to a PR, or reviewed before execution.

Consequence: Any change to config, queries, or chunk boundaries after rivet plan requires regenerating the artifact with a new rivet plan invocation. Applying a stale artifact against a changed config is detected by staleness checks (PA3) and fingerprint logging (PA6), not by a config re-parse.

The one exception (amendment 2026-10-17 below): apply reads the clock once, for {date} in the destination; it still recomputes no chunk, query or cursor.

PA2 — Artifact Immutability (AI)

A PlanArtifact file must not be modified after it is written by rivet plan. rivet apply treats the artifact as a sealed, read-only input.

Rationale: The artifact’s plan_id and created_at field are set at plan time. Any post-generation modification breaks the audit trail and the staleness check. The artifact is a point-in-time snapshot, not a mutable config.

Current implementation: PlanArtifact is deserialized from the file at apply time. No write-back or in-place mutation occurs. The file on disk is never opened for writing by apply_cmd. [Update: immutability is now machine-enforced, not just an operator convention — see PA10.]

PA3 — Staleness Boundary (SB)

For a point-in-time plan, rivet apply enforces a maximum age between plan time and apply time (a reusable incremental plan has no age: it is checked by PA10 and PA4 instead):

• Age < 1 hour: apply proceeds silently.
• 1 hour ≤ age < 24 hours: apply emits a WARN log and proceeds.
• Age ≥ 24 hours: apply rejects the artifact with an error unless --force is passed.

Rationale: The value of a pre-computed plan degrades as the source drifts. An artifact that is hours old may describe chunk boundaries that are no longer representative of the current data distribution. The hard error threshold prevents accidentally applying a week-old plan that was forgotten in a directory.

Reusable plans: a 24 h expiry would forbid the use an incremental plan exists for (one reviewed plan applied by a schedule for weeks), so PA3 does not apply to it. What still invalidates one: PA10 (an edited artifact is refused), PA4 (another run moved the cursor), and the source’s own answer to the frozen query (a dropped column fails the query; the schema-drift policy judges a changed one). Proposed, not implemented here: refuse an incremental plan whose source config no longer matches (plan_fingerprint against the config at config_path when that file exists) or whose table’s column set changed since planning (a column-list fingerprint taken at plan time), each with a code and the remedy rivet plan.

Current implementation: PlanArtifact::staleness(warn_after, error_after) computes the age from created_at to Utc::now(). apply_cmd::run_apply_command calls this for a point-in-time plan and either warns, bails, or proceeds. The expires_at field in the artifact JSON provides the deadline in a human-readable form.

Test coverage: test staleness_fresh, test staleness_expired_artifact in plan/artifact.rs.

PA4 — Cursor Snapshot Integrity (CSI)

For Incremental exports, rivet apply verifies that the cursor value in StateStore at apply time equals the cursor value captured in ComputedPlanData.cursor_snapshot at plan time, or that the newest successful run of the export from the same source applied this same plan (run_status.plan_id). A cursor moved by an earlier apply of the same plan is the plan’s own progress, not drift. Any other difference rejects the artifact.

Rationale: the apply reads its starting cursor from state, never from the snapshot, so a moved cursor cannot make it re-extract rows; what the check guards is the review: a plan reviewed against one position must not run silently after another process (a rivet run, another plan) took the window it was reviewed for. Its own earlier applies took nothing from anyone else, so they do not drift it (owner decision 2026-10-16; before, the plan’s first apply moved the cursor and every later apply of the same artifact was refused RIVET_PLAN_ARTIFACT_STALE).

Failure mode: If cursor drift is detected and --force is not passed, apply exits with:

plan 'orders': cursor has drifted since plan was generated
  (plan snapshot: "2026-04-14T09:00:00Z", current: "2026-04-14T11:00:00Z")
Regenerate with `rivet plan` or pass --force to skip this check.

Scope: This check applies only to Incremental exports (where cursor_snapshot is non-None). Snapshot, TimeWindow, and Chunked exports have no cursor and are not subject to this check. Chunked exports instead have partition-level progression tracked by ADR-0008 (committed) and ADR-0009 (verified).

Cursor policy: How the snapshot string is produced (single column vs COALESCE progression) is defined in ADR-0007; PA4 compares opaque strings regardless of mode.

Current implementation: apply_cmd::run_apply_command calls PlanArtifact::cursor_matches(current), then StateStore::cursor_last_moved_by_plan. cursor_matches returns true when cursor_snapshot is None (all non-incremental strategies, and an incremental plan made before any cursor was stored).

Test coverage: test cursor_matches_none_snapshot, test cursor_matches_incremental in plan/artifact.rs.

PA5 — Chunk Range Monotonicity (CRM)

Chunk ranges stored in ComputedPlanData.chunk_ranges must satisfy:

1. Each range (start, end) satisfies start ≤ end.
2. Consecutive ranges (s1, e1) and (s2, e2) satisfy s2 = e1 + 1 (no gaps, no overlaps).

An empty chunk_ranges is valid for non-Chunked strategies.

Rationale: These invariants mirror generate_chunks output guarantees (see math.rs). At apply time, ranges are replayed as ChunkSource::Precomputed and passed directly to build_chunk_query_sql without re-validation. A non-monotonic or overlapping range would produce incorrect WHERE predicates.

Current implementation: detect_and_generate_chunks produces ranges via generate_chunks which guarantees monotonicity by construction. The artifact round-trips ranges through JSON without mutation.

Future: An explicit PlanArtifact::validate_chunk_ranges() method that enforces this invariant before execution is a candidate for a future hardening pass.

PA6 — Fingerprint Stability (FS)

For Chunked exports, the plan_fingerprint field is computed at plan time from (base_query, chunk_column, chunk_size, chunk_count, dense, by_days) via chunk_plan_fingerprint. This fingerprint is embedded in the artifact and displayed in rivet plan’s summary output.

Rationale: If the config changes between plan and apply (query rewritten, chunk_size changed), the fingerprint computed from the new config would differ from the artifact’s fingerprint. This mismatch is an operator signal that the artifact may no longer represent the intended extraction, even if apply can technically proceed.

Current behavior: The fingerprint is displayed only at plan time (PlanArtifact::print_summary); nothing at apply time reads, logs, or enforces it. The operator is responsible for regenerating the plan when the config changes.

Alignment with ADR-0001 I5: The chunk checkpoint system (chunk_run.plan_hash) enforces fingerprint matching at resume time. The plan artifact fingerprint is a complementary audit signal, not a resume gate.

PA7 — State Writes Unchanged (SWU)

rivet apply uses the same state persistence paths as rivet run. File manifest, cursor updates, and run metrics are written via StateStore using the same invariants defined in ADR-0001.

Rationale: The artifact replaces chunk boundary detection only. It does not change the semantics of state persistence. All ADR-0001 invariants (I1–I7) apply unchanged to an apply run.

Consequence: The StateStore file (.rivet_state.db) used by apply is resolved primarily from the directory of the artifact’s recorded config_path when that directory exists (F13, 0.7.5 audit — this keeps apply consistent with rivet run’s state location). Only if the config directory is gone, or the artifact predates 0.7.5 and recorded no config path, does apply fall back to the plan file’s own directory, emitting a WARN about the divergence.

PA8 — Diagnostics Are Advisory (DAA)

Preflight diagnostics embedded in PlanDiagnostics — verdict, warnings, recommended_profile — are advisory metadata captured at plan time. They do not gate execution at apply time. A plan with verdict: "Unsafe" can be applied; the verdict is preserved for auditability.

Rationale: The decision to apply a plan is the operator’s. A degraded verdict is information, not a veto. Vetoing at apply time would be surprising because the operator already reviewed the plan before deciding to apply.

Contrast with plan validation: validate_plan(&plan) (ADR-0003) is re-run at apply time with Rejected diagnostics as hard gates. Plan validation checks structural constraints (e.g., stdout + chunked). Preflight diagnostics check operational health (e.g., missing index). Only the former is enforced at apply time.

PA9 — Artifact Credential Redaction (ACR)

A PlanArtifact must not contain plaintext credentials. Before the resolved plan is embedded, PlanArtifact::new runs SourceConfig::redact_for_artifact:

• password → always stripped (set to None).
• url with scheme://user[:password]@… → userinfo replaced with REDACTED (host/port/path preserved).
• url_env, url_file, password_env — preserved. They are references (env var names, file paths) that apply needs to re-resolve credentials at runtime; not secrets themselves.
• host, port, user, database — preserved (infrastructure metadata, not credentials).

Rationale: Plan artifacts are designed to be stored, committed to PRs, or shared for review (PA1, PA2). Historic config patterns with inline password: or credentials-in-URL would leak into every artifact. Silently stripping plaintext (rather than failing loudly) keeps the plan/apply workflow operational while making artifacts safe-by-default.

Operator contract: When redaction runs, Rivet logs a WARN: plan '<name>': plaintext credentials stripped from artifact — apply time must have equivalent env/file-based auth available. Operators must ensure url_env / password_env / url_file equivalents are set in the apply environment. For existing YAML configs with plaintext password, migrate to password_env before relying on plan/apply.

Scope of protection: PA9 covers only the source-side credentials embedded in ResolvedRunPlan.source. Destination secrets are already ADR-0004-compliant (S3/GCS use env/file references via access_key_env, secret_key_env, credentials_file; no plaintext equivalents exist in the destination schema).

Current implementation: SourceConfig::redact_for_artifact in src/config/source.rs; called from PlanArtifact::new in src/plan/artifact.rs. URL parsing uses a path-aware @ scan so @ in a query string or path does not trigger false redaction.

Test coverage: redact_plaintext_password_stripped, redact_password_embedded_in_url, redact_url_without_userinfo_is_unchanged, redact_env_references_are_preserved, redact_does_not_confuse_at_in_path in src/config/tests/secops.rs; artifact_strips_plaintext_password_from_source, artifact_strips_credentials_from_url in src/plan/artifact.rs.

PA10 — Artifact Tamper-Evidence (ATE)

Before running any query, rivet apply calls PlanArtifact::verify_integrity() and rejects an artifact whose resolved_plan was edited after planning. Unlike staleness (PA3) and cursor drift (PA4), this gate is not bypassable by --force — a hand-edited execution contract is never something the operator can opt into; the only correct recovery is to re-run rivet plan.

Rationale: PA2 declares the artifact a sealed, read-only input; PA10 is its machine enforcement. Without it, a post-plan edit would execute silently with a plan_id/created_at audit trail that no longer describes what ran.

Current implementation: PlanArtifact::verify_integrity in src/plan/artifact.rs (integrity checksum over the resolved plan), called from apply_cmd::run_apply_command before any state or source access (finding #16).

PA11 — An Applied Point-in-Time Plan Answers Its Replay (APR)

The run that applies a plan to exit 0 records the plan’s plan_id on its run_status row (state schema v37). rivet apply of a point-in-time artifact (mode: full, chunked ranges, keyset without keyset_incremental, time window) whose plan_id a finished run of the export recorded exits 0 WITHOUT extracting and prints apply: plan <plan_id> of export '<export>' was already applied by run <run_id> (<status>, <rows> rows, <files> files, <bytes> bytes, finished <time>); nothing was extracted again. To extract new data, make a new plan with rivet plan. on stderr. It writes no run, no manifest and no metrics row, so rivet metrics --last 1 still names the run that applied the plan. --force extracts again (already_applied in the run report’s force_bypassed).

A reusable (incremental) artifact is never answered: each apply, a scheduler retry included, continues from the stored cursor and delivers what arrived since, new rows or none, never a duplicate. Answering it would leave those rows undelivered with exit 0 (measured on this PR’s first version: keyset_incremental applied twice with 50 rows inserted between, source 150, delivered 100).

Rationale: a scheduler retries a task whose process died after rivet apply exited 0, with the same sealed artifact. Before PA11 that retry extracted the whole table again (mode: full, chunked, keyset). A plan whose run failed, was interrupted, or died before exit 0 recorded nothing and replays as before.

Current implementation: StateStore::record_applied_plan (after an exit-0 run) and StateStore::run_that_applied_plan (before staleness and PA4, asked only when !PlanArtifact::is_reusable), both called from apply_cmd::run_apply_command.

Contract Summary Table

Interaction with Existing ADRs

Failure Point Map

Test Coverage

plan/artifact.rs tests: round_trip_json, round_trip_chunked, staleness_fresh, staleness_expired_artifact, cursor_matches_none_snapshot, cursor_matches_incremental.

PA5 structural coverage is provided by pipeline/chunked/math.rs tests (test_generate_chunks, test_generate_chunks_exact, test_generate_chunks_empty).

Amendment 2026-09-26: PA1 holds for a plan artifact only

rivet apply <config.yaml> is a second, artifact-free mode: it loads the config and runs every export live, wave by wave (or as a --pool N pool), so PA1–PA6 and PA10 do not apply to it. A JSON plan artifact still takes the sealed path — integrity, staleness and precomputed chunks — and --pool is refused for one.

Amendment 2026-10-17: PA1’s one clock read, {date} resolves at apply

A sealed plan keeps {date} in resolved_plan.destination as written, and rivet apply resolves it to its own UTC day through the one resolver rivet run resolves with (destination::placeholder::resolve_destination, before anything reads the destination), so the files, the manifest, the state rows and the load spec all land under the apply’s day. This is the one value an apply reads from the clock; it still recomputes no chunk, query or cursor. rivet plan --date <day> freezes that day into the artifact instead: the operator asked for it. An artifact sealed by an earlier release holds an already-resolved path, which apply leaves as it is. PA10 covers the destination as written in the artifact, so an apply on another day is not an integrity failure. Measured before the amendment (P-26): a plan made before UTC midnight and applied after it wrote under the planning day while the data and the part names were of the apply.