Tuning for throughput
Since 0.6.5. This page shows how to measure what one engine and one PostgreSQL complete per
second on your hardware, and which settings move it, without giving up durability: every number
here was taken with fsync, synchronous_commit and full_page_writes on.
What limits a durable engine
An instance’s state is safe only once PostgreSQL has flushed its write-ahead log to disk, and a disk flushes a limited number of times per second. A consumer NVMe drive manages about 300; a server SSD with a power-loss-protected cache manages tens of thousands. Measure yours with PostgreSQL’s own tool:
/usr/lib/postgresql/18/bin/pg_test_fsync -s 2 -f /path/on/the/database/disk/testfile
What decides throughput is how many commits share each flush. The engine helps in three ways:
- it commits only what must be durable before anything is acknowledged: a step’s state, its aliases, its waits and its outbound messages, in one transaction. Bookkeeping that a crash cannot hurt — an ownership claim, the outbox’s row locks, the delete of an already delivered row — does not wait for its own flush, and the inbox dedup record rides the step’s own transaction;
- it delivers outbound messages on several workers at once (
sutra.outbox.workers), deletes delivered rows in batches, and handslocal://deliveries straight from the sender’s commit to a worker; - with several execution lanes (
sutra.engine.shards), several steps commit at the same time, and PostgreSQL flushes them together.
So on a slow disk, add lanes: each lane is one more commit that can share a flush.
What may skip its own flush, and why
The server keeps synchronous_commit = on. Three kinds of transaction set
synchronous_commit = off for themselves alone, and only because a crash that loses them can
neither lose an acknowledged state nor invent one:
| Transaction | What it writes | Why it may skip its flush |
|---|---|---|
| instance ownership claim | which engine is resuming an instance | a claim held by an engine that crashed is void anyway; the step that follows commits synchronously |
| outbox claim and its lease renewal | row locks, and a lease: the claimed rows’ due time moved 30 s ahead | a lost lease only makes the rows claimable again, so they are sent again: at-least-once, deduplicated on the idempotency key |
| delete of delivered outbox rows | removes rows already sent | if it is lost, the row is sent again: outbound delivery has always been at-least-once, and receivers deduplicate on the idempotency key |
The inbox dedup record of a delivery is not on the list: it is written in the same transaction as the step that processes the delivery, so it exists exactly when the step’s effects do.
Everything else — every step’s state, aliases, waits and outbound rows, every data-store write, every retry or poison mark — commits synchronously, as before. The tier-2 suite kills the engine, and separately the database, in the middle of a load and checks that every accepted request still completes exactly once.
The workload
The repository carries a workload that exercises the whole durable path, in
tools/sutra-load-test:
fixtures/relay/— two processes.relay-intaketakes an order over HTTP, validates it with a small.srlruleset, reads two rows from a data store, forwards it over a local channel torelay-worker, waits for the worker’s answer on a second local channel, and calls back an HTTP endpoint. Three durable hops per order.profiles/relay.js— k6 posts orders at a constant rate.sink/— a small HTTP server that receives the callbacks and measures each order’s latency from the moment k6 sent it.relay-bench.sh— starts a throwaway PostgreSQL cluster (durable settings) on the disk you run it from, the sink and the engine, warms up, runs k6, waits until every order has called back, and prints one line of results plus the busiest SQL statements.
It needs k6, jq, PostgreSQL’s server binaries (PG_BIN_DIR, default
/usr/lib/postgresql/18/bin) and an optimised build of the engine and the CLI. The runs below
used the faster-to-build release-it profile:
(cd rust && cargo build --profile release-it -p sutra-dist -p sutra-cli)
export SUTRA_ENGINE_BIN=$PWD/rust/target/release-it/sutra-engine
export SUTRA_CLI=$PWD/rust/target/release-it/sutra
The script refuses to put the database on tmpfs, where a flush costs nothing and the numbers
would mean nothing.
Step 1: the defaults
RATE=100 DURATION=60s tools/sutra-load-test/relay-bench.sh
[relay-bench] building the sink
[relay-bench] warm-up 10/s x 10s
[relay-bench] measured run 100/s x 60s
| offered | sent | settled | dup | drained | settles/s | settle p50 | settle p99 | intake p50 | intake p99 | commits/settle | WAL flushes/settle | WAL KB/settle | WAL fsync busy | engine CPU (cores) | PG CPU (cores) | CPU ms/settle (engine+PG) | engine env |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 100/s x 60s | 5167 | 5167 | 0 | yes | 70.1 | 6729 ms | 13395 ms | 6708 ms | 10001 ms | 16.0 | 3.01 | 10.2 | 63% | 0.24 | 0.25 | 7.09 | `defaults` |
Top statements by calls (calls | total ms | query):
72550 | 30 | BEGIN
72550 | 24 | COMMIT
46805 | 187 | SELECT set_config($1, $2, $3)
25745 | 140 | SELECT set_config($1, $2, $3), set_config($4, $5, $6)
15501 | 402 | INSERT INTO outbox_entry (entry_id, deployment_id, instance_id, body, content_type, destination, headers_json,
15396 | 332 | DELETE FROM outbox_entry WHERE deployment_id = $1 AND entry_id = ANY($2)
10334 | 118 | SELECT instance_id FROM alias_index WHERE deployment_id = $1 AND alias_name = $2 AND alias_value = $3 AND live
10334 | 144 | SELECT store_value, rev FROM data_store WHERE store_name = $1 AND store_key = $2
One lane executes one step at a time, so its commits never share a flush; on this disk that caps
the engine near 70 orders per second, and the rest of the offered load queues at
intake (intake p99). settled can exceed sent: k6 gives up on a request after 10 seconds
and does not count it, but the intake had already accepted it, and it completes like any other.
Step 2: lanes, and a pool to match
Each lane holds a database connection while it commits, so raise the main pool with the lanes:
RATE=100 DURATION=60s ENGINE_ENV="SUTRA_ENGINE_SHARDS=8" tools/sutra-load-test/relay-bench.sh
[relay-bench] warm-up 10/s x 10s
[relay-bench] measured run 100/s x 60s
| offered | sent | settled | dup | drained | settles/s | settle p50 | settle p99 | intake p50 | intake p99 | commits/settle | WAL flushes/settle | WAL KB/settle | WAL fsync busy | engine CPU (cores) | PG CPU (cores) | CPU ms/settle (engine+PG) | engine env |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 100/s x 60s | 6001 | 6001 | 0 | yes | 99.8 | 20 ms | 72 ms | 5 ms | 26 ms | 15.9 | 2.81 | 10.2 | 87% | 0.35 | 0.37 | 7.31 | `SUTRA_ENGINE_SHARDS=8` |
Then the target rate:
RATE=500 DURATION=60s \
ENGINE_ENV="SUTRA_ENGINE_SHARDS=32 SUTRA_DATASOURCE_MAX_CONNECTIONS=40" \
tools/sutra-load-test/relay-bench.sh
[relay-bench] warm-up 10/s x 10s
[relay-bench] measured run 500/s x 60s
| offered | sent | settled | dup | drained | settles/s | settle p50 | settle p99 | intake p50 | intake p99 | commits/settle | WAL flushes/settle | WAL KB/settle | WAL fsync busy | engine CPU (cores) | PG CPU (cores) | CPU ms/settle (engine+PG) | engine env |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 500/s x 60s | 30001 | 30001 | 0 | yes | 499.1 | 31 ms | 185 ms | 8 ms | 52 ms | 14.1 | 0.57 | 10.2 | 89% | 1.79 | 2.01 | 7.69 | `SUTRA_ENGINE_SHARDS=32 SUTRA_DATASOURCE_MAX_CONNECTIONS=40` |
Read the columns this way:
| Column | Meaning |
|---|---|
settles/s | orders that called back, per second, from the start of the run to the last callback |
settle p50 / p99 | time from k6 sending the order to the sink receiving the callback |
intake p50 / p99 | time for the intake to answer 202 |
commits/settle | database transactions per order, read-only ones included |
WAL flushes/settle | disk flushes per order — below 1 means commits share flushes |
WAL fsync busy | share of the run the disk spent flushing |
engine CPU | cores the engine used |
PG CPU | cores PostgreSQL’s backends used (since 0.6.6) |
CPU ms/settle | engine plus PostgreSQL CPU time per order, in milliseconds — the number to watch when the host, not the disk, is the limit (since 0.6.6) |
When WAL fsync busy is near 100% and adding lanes no longer raises settles/s, the disk is the
limit: a faster disk, or PostgreSQL’s commit_delay (which makes a flush wait a few microseconds
for more commits to join it, still durable), are what move it next. When the disk is not busy but
every core is, the CPU is the limit: CPU ms/settle times the rate you want is the core count it
takes. 0.6.6 cut it by about a fifth on this workload: with SUTRA_OUTBOX_WORKERS=32 SUTRA_OUTBOX_TICK_INTERVAL=PT0.01S added to the last run, 8.05 ms on 0.6.5 and 6.33 ms on 0.6.6.
The settings
| Setting | Default | Raise it when |
|---|---|---|
sutra.engine.shards | 1 | the disk is busy less than the time it takes to flush, and orders queue at intake. Read Execution lanes first: lanes run different instances in parallel. |
sutra.datasource.max-connections | 8 | you raise the lanes: keep it at least the lane count, plus a few |
sutra.datasource.dispatch-max-connections | 4 | rarely; the dispatcher’s claim and its batched delete each hold one connection, the rest serve the defers of failed sends |
sutra.outbox.workers | 32 | many destinations are slow at once: it bounds the sends in flight across all of them |
sutra.outbox.destination-concurrency | 32 | one destination is slow and can take more parallel requests; see Outbound bursts |
sutra.outbox.max-buffered | 4096 | sutra.outbox.buffered sits at the bound during a burst and memory allows |
sutra.outbox.batch-size | 50 | a deployment enqueues far more than 50 rows between claims |
sutra.outbox.tick-interval | PT5S | you want retries and rows from other replicas picked up sooner; local deliveries do not wait for it |
cache: on a data store | off | a store is read on every request and changed rarely — see Caching reads |
Outbound bursts
Each process instance’s outbound rows form one ordered chain: they leave one after another, in
the order they were enqueued, while different instances’ chains send in parallel. Two bounds
apply to a send, taken in this order: one per destination authority (scheme, host and port;
sutra.outbox.destination-concurrency) and one across every destination
(sutra.outbox.workers). A row waiting for a busy destination holds no global slot, so a slow
counterparty delays only its own rows. The claimer stops claiming only when
sutra.outbox.max-buffered rows are held in memory.
Three instruments show whether outbound delivery keeps up:
sutra.outbox.delivery.lag(histogram, milliseconds, byscheme): enqueue to delivery, once per delivered row. Its p99 is the number to watch.sutra.outbox.in-flight(gauge): atsutra.outbox.workersfor long, the global bound is the limit.sutra.outbox.buffered(gauge): atsutra.outbox.max-buffered, the claimer is waiting.
The defaults come from a burst bench: tools/sutra-load-test/relay-bench.sh BURST=3500 enqueues
3,500 rows in about 6 s across two HTTP destinations, one answering at once and one after
500 ms (30% of instances), on durable PostgreSQL. Lag is measured by the receiving end, against
the instant the engine built each request.
| Engine | workers / destination-concurrency | Drain | Fast lag p50 / p99 / max | Slow lag p99 | Slow rows/s |
|---|---|---|---|---|---|
| 0.8.2 | 8 workers by instance | 84.5 s | 35.9 s / 73.9 s / 78.9 s | 73.6 s | 12.5 |
| 0.8.3 | 8 / 8 | 65.5 s | 6.3 s / 18.8 s / 19.1 s | 59.1 s | 16.1 |
| 0.8.3 | 16 / 8 | 65.5 s | 5 ms / 7 ms / 10 ms | 59.8 s | 16.0 |
| 0.8.3 | 16 / 16 | 32.8 s | 172 ms / 666 ms / 864 ms | 27.6 s | 32.1 |
| 0.8.3 | 32 / 16 | 32.9 s | 5 ms / 11 ms / 17 ms | 27.5 s | 32.1 |
| 0.8.3 | 32 / 32 (default) | 16.5 s | 25 ms / 166 ms / 187 ms | 12.3 s | 64.2 |
| 0.8.3 | 64 / 16 | 32.9 s | 4 ms / 7 ms / 9 ms | 27.8 s | 32.0 |
| 0.8.3 | 64 / 32 | 16.5 s | 4 ms / 7 ms / 15 ms | 12.3 s | 64.2 |
| 0.8.3 | 128 / 32 | 16.5 s | 5 ms / 8 ms / 11 ms | 12.2 s | 64.2 |
Peak PostgreSQL connections stayed at 8–9 in every run.
The defaults are the smallest pair whose fast-destination p99 stays under 2 s and whose drain
time is within 10% of the best. The slow destination drains at about
destination-concurrency / its latency; the fast one is only held back once the slow one can take
every global slot (8 / 8 above).
Watch outbound lag under a burst
This walkthrough runs the burst bench twice against a local OpenTelemetry Collector, first with a
global bound so small that the slow destination takes all of it, then with the defaults, and
reads sutra.outbox.delivery.lag from the collector each time. It needs Docker, k6, PostgreSQL
18’s server binaries and a release build of the engine and CLI (cargo build --profile release-it -p sutra-dist -p sutra-cli).
Step 1: start a collector that receives OTLP and exposes the metrics for Prometheus:
cat > /tmp/otelcol.yaml <<'YAML'
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
exporters:
prometheus:
endpoint: 0.0.0.0:8889
service:
pipelines:
metrics:
receivers: [otlp]
exporters: [prometheus]
YAML
docker run -d --name walk-otelcol -p 127.0.0.1::4317 -p 127.0.0.1::8889 \
-v /tmp/otelcol.yaml:/etc/otelcol-contrib/config.yaml:ro otel/opentelemetry-collector-contrib
OTLP=$(docker port walk-otelcol 4317/tcp | head -1)
PROM=$(docker port walk-otelcol 8889/tcp | head -1)
Step 2: a burst with the slow destination holding the fast one. Eight sends in flight in total and eight per destination: the slow destination can take every slot.
export SUTRA_ENGINE_BIN=rust/target/release-it/sutra-engine SUTRA_CLI=rust/target/release-it/sutra
BURST=3500 ENGINE_ENV="SUTRA_TELEMETRY_OTLP_ENDPOINT=http://$OTLP \
SUTRA_TELEMETRY_METRICS_EXPORT_INTERVAL=1000 SUTRA_TELEMETRY_SERVICE_NAME=walk-8 \
SUTRA_OUTBOX_WORKERS=8 SUTRA_OUTBOX_DESTINATION_CONCURRENCY=8" \
tools/sutra-load-test/relay-bench.sh
curl -s "http://$PROM/metrics" | grep 'sutra_outbox_delivery_lag_milliseconds_bucket{job="walk-8"' \
| grep -E 'le="(100|1000|10000|\+Inf)"'
The bench’s block shows the fast destination waiting: on our run its lag was 6.5 s / 17.5 s / 17.7 s (p50 / p99 / max), and the histogram has 41 of the 3,500 rows within 100 ms, 231 within a second and 1,746 within ten.
Step 3: the same burst with the defaults (32 / 32):
BURST=3500 ENGINE_ENV="SUTRA_TELEMETRY_OTLP_ENDPOINT=http://$OTLP \
SUTRA_TELEMETRY_METRICS_EXPORT_INTERVAL=1000 SUTRA_TELEMETRY_SERVICE_NAME=walk-default" \
tools/sutra-load-test/relay-bench.sh
curl -s "http://$PROM/metrics" | grep 'sutra_outbox_delivery_lag_milliseconds_bucket{job="walk-default"' \
| grep -E 'le="(100|1000|10000|\+Inf)"'
The fast destination no longer waits: 26 ms / 165 ms / 184 ms, and 2,139 rows land within
100 ms. The rows still over a second are the slow destination’s own (1,048 of them, drained at
64 rows/s), and the whole burst drains in 16.5 s instead of 66.7 s. The histogram carries only
the scheme attribute, so both HTTP destinations share one series; the per-destination split
comes from the bench’s sink.
Step 4: clean up. docker rm -f walk-otelcol. The bench stops its own engine, sinks and
database.
PostgreSQL autovacuum
Several engine tables churn at the request rate:
outbox_entrygets one insert and one delete per delivery;instance_stateis rewritten on every step;waiting_event,alias_indexandinbox_seengain and lose rows as instances park, finish and age out.
Each delete or rewrite leaves a dead row version, and the engine’s hot queries step over those until a vacuum removes them. The claim of due outbox rows is the one that suffers most.
PostgreSQL’s defaults vacuum a table once 20% of it is dead and look for such tables once a minute. That is too late for a queue: it stays small while it produces thousands of dead rows a second. One measured run at about 1,700 requests a second used the defaults. Every two and a half minutes the outbox claim took 1–1.6 s, throughput fell below 100 a second for some 40 s, and p99 latency reached 38 s.
Since 0.6.8 the engine’s migrations set per-table values. You need do nothing for them:
| Table | Vacuum after | Cost throttle |
|---|---|---|
outbox_entry | 5,000 dead rows, whatever the table’s size (autovacuum_vacuum_scale_factor = 0, autovacuum_vacuum_threshold = 5000) | none (autovacuum_vacuum_cost_delay = 0); the table is small |
instance_state, waiting_event, alias_index, inbox_seen | 1% of the table plus 5,000 rows (0.01, 5000) | the server’s setting; these tables can be large |
Statistics are refreshed on the same kind of trigger. At low rates 5,000 dead rows take a long time to accumulate, so these settings cost nothing there.
One setting is server-wide: autovacuum_naptime. It sets how often autovacuum wakes to look,
and a table cannot override it. For a deployment running hundreds of requests a second or more,
lower it in postgresql.conf, or in your managed service’s database flags:
autovacuum_naptime = 1s # default 1min
autovacuum_vacuum_cost_limit = 2000 # default 200 (via vacuum_cost_limit); lets the larger tables keep up
The run above, repeated with naptime 1s and an unthrottled global autovacuum, went on for ten
minutes at 2,500 requests a second with p99 329 ms and no slow statements. A busy autovacuum worker
uses some CPU and I/O, so watch both after the change.
MySQL/MariaDB and SQL Server have no per-table equivalent. InnoDB purge and SQL Server’s ghost cleanup are server-wide, and the engine ships no setting for them.
Profiling
PERF_DIR=/some/dir records a CPU profile of the engine and of the database backends over the
measured run, and writes a flamegraph of each plus a list of the hottest functions. It needs
perf usable by your user (kernel.perf_event_paranoid of 1 or less) and
inferno (cargo install inferno). Build the engine with frame
pointers so the stacks are whole:
(cd rust && RUSTFLAGS="-C force-frame-pointers=yes" cargo build --profile release-it -p sutra-dist)
PERF_DIR=/tmp/relay-profile RATE=100 DURATION=30s tools/sutra-load-test/relay-bench.sh
On the run above the engine used well under one core; most of its time went to the database protocol, and the rest of the wall clock was the disk.