Source-safe under load

The first question a serious operator asks about an extraction tool is what does it do to my source database while it runs? A careless SELECT * holds a long-running transaction, pins a read snapshot, inflates temp space, and spikes p99 latency for every other query on the box. Rivet is built so that the honest answer is “almost nothing you’ll notice.”

What it holds open on the source

In mode: chunked (keyset and range) a batch export reads the source in bounded pages — one chunk / one page at a time — and flushes each to a Parquet part before asking for the next. The longest query Rivet holds open on the source in mode: chunked is a single page, not a full-table scan.

mode: full reads the table in one pass, and what stays open for that pass depends on the engine:

For a large table on a busy source, use mode: chunked: there PostgreSQL opens one transaction per page as well.

In the cross-tool benchmark (PostgreSQL → Parquet, measured under a concurrent OLTP workload), the longest single server-side query each tool held was:

“Longest source query” is the age of the statement in flight on PostgreSQL, sampled every 50 ms: for Rivet that is one FETCH. In mode: full (the first row) the transaction around those fetches is as long as the run.

Rivet is the only tool in this PostgreSQL field whose statements are all short. Everything else holds one server-side query open for the length of a full scan — 8 to 95 seconds here, and proportionally longer on a real table. That is the query your DBA sees in pg_stat_activity blocking autovacuum, or the one a pooler’s statement timeout kills at the worst moment.

It stays gentle on concurrent traffic

Under the same concurrent OLTP workload, the p99 latency multiplier (how much the export slows other queries) is mid-field for Rivet — 5.1× single-stream, 3.3× chunked — comparable to the rest, but achieved at 57 MB of RAM and zero long queries instead of hundreds of megabytes to multiple gigabytes with a scan pinned open. Rivet trades a little client-side CPU for a source that never sees a heavy query.

If the source is fragile, production-shared, or behind a pooler (pgBouncer, ProxySQL, MaxScale), that trade is the whole point. See MSSQL gentle extraction and resource-aware extraction for the per-engine knobs.

CDC is a passive reader too

Change-data-capture reads the transaction log, not your tables. Running a looping CDC drain against a live writer, the writer’s throughput barely moves — except on SQL Server, and that cost is inherent to the engine, not to Rivet:

The full per-engine numbers, harnesses, and contracts live in the performance & harm ledger. The one operational hazard that is per-engine — a CDC reader pinning source log retention — is covered on its own page: CDC cost, per engine.