Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Tuning for throughput

Since 0.6.5. This page shows how to measure what one engine and one PostgreSQL complete per second on your hardware, and which settings move it, without giving up durability: every number here was taken with fsync, synchronous_commit and full_page_writes on.

What limits a durable engine

An instance’s state is safe only once PostgreSQL has flushed its write-ahead log to disk, and a disk flushes a limited number of times per second. A consumer NVMe drive manages about 300; a server SSD with a power-loss-protected cache manages tens of thousands. Measure yours with PostgreSQL’s own tool:

/usr/lib/postgresql/18/bin/pg_test_fsync -s 2 -f /path/on/the/database/disk/testfile

What decides throughput is how many commits share each flush. The engine helps in three ways:

  • it commits only what must be durable before anything is acknowledged: a step’s state, its aliases, its waits and its outbound messages, in one transaction. Bookkeeping that a crash cannot hurt — an ownership claim, the outbox’s row locks, the delete of an already delivered row — does not wait for its own flush, and the inbox dedup record rides the step’s own transaction;
  • it delivers outbound messages on several workers at once (sutra.outbox.workers), deletes delivered rows in batches, and hands local:// deliveries straight from the sender’s commit to a worker;
  • with several execution lanes (sutra.engine.shards), several steps commit at the same time, and PostgreSQL flushes them together.

So on a slow disk, add lanes: each lane is one more commit that can share a flush.

What may skip its own flush, and why

The server keeps synchronous_commit = on. Three kinds of transaction set synchronous_commit = off for themselves alone, and only because a crash that loses them can neither lose an acknowledged state nor invent one:

TransactionWhat it writesWhy it may skip its flush
instance ownership claimwhich engine is resuming an instancea claim held by an engine that crashed is void anyway; the step that follows commits synchronously
outbox claim and its lease renewalrow locks, and a lease: the claimed rows’ due time moved 30 s aheada lost lease only makes the rows claimable again, so they are sent again: at-least-once, deduplicated on the idempotency key
delete of delivered outbox rowsremoves rows already sentif it is lost, the row is sent again: outbound delivery has always been at-least-once, and receivers deduplicate on the idempotency key

The inbox dedup record of a delivery is not on the list: it is written in the same transaction as the step that processes the delivery, so it exists exactly when the step’s effects do.

Everything else — every step’s state, aliases, waits and outbound rows, every data-store write, every retry or poison mark — commits synchronously, as before. The tier-2 suite kills the engine, and separately the database, in the middle of a load and checks that every accepted request still completes exactly once.

The workload

The repository carries a workload that exercises the whole durable path, in tools/sutra-load-test:

  • fixtures/relay/ — two processes. relay-intake takes an order over HTTP, validates it with a small .srl ruleset, reads two rows from a data store, forwards it over a local channel to relay-worker, waits for the worker’s answer on a second local channel, and calls back an HTTP endpoint. Three durable hops per order.
  • profiles/relay.js — k6 posts orders at a constant rate.
  • sink/ — a small HTTP server that receives the callbacks and measures each order’s latency from the moment k6 sent it.
  • relay-bench.sh — starts a throwaway PostgreSQL cluster (durable settings) on the disk you run it from, the sink and the engine, warms up, runs k6, waits until every order has called back, and prints one line of results plus the busiest SQL statements.

It needs k6, jq, PostgreSQL’s server binaries (PG_BIN_DIR, default /usr/lib/postgresql/18/bin) and an optimised build of the engine and the CLI. The runs below used the faster-to-build release-it profile:

(cd rust && cargo build --profile release-it -p sutra-dist -p sutra-cli)
export SUTRA_ENGINE_BIN=$PWD/rust/target/release-it/sutra-engine
export SUTRA_CLI=$PWD/rust/target/release-it/sutra

The script refuses to put the database on tmpfs, where a flush costs nothing and the numbers would mean nothing.

Step 1: the defaults

RATE=100 DURATION=60s tools/sutra-load-test/relay-bench.sh
[relay-bench] building the sink
[relay-bench] warm-up 10/s x 10s
[relay-bench] measured run 100/s x 60s
| offered | sent | settled | dup | drained | settles/s | settle p50 | settle p99 | intake p50 | intake p99 | commits/settle | WAL flushes/settle | WAL KB/settle | WAL fsync busy | engine CPU (cores) | PG CPU (cores) | CPU ms/settle (engine+PG) | engine env |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 100/s x 60s | 5167 | 5167 | 0 | yes | 70.1 | 6729 ms | 13395 ms | 6708 ms | 10001 ms | 16.0 | 3.01 | 10.2 | 63% | 0.24 | 0.25 | 7.09 | `defaults` |
Top statements by calls (calls | total ms | query):
72550 | 30 | BEGIN
72550 | 24 | COMMIT
46805 | 187 | SELECT set_config($1, $2, $3)
25745 | 140 | SELECT set_config($1, $2, $3), set_config($4, $5, $6)
15501 | 402 | INSERT INTO outbox_entry (entry_id, deployment_id, instance_id, body, content_type, destination, headers_json,
15396 | 332 | DELETE FROM outbox_entry WHERE deployment_id = $1 AND entry_id = ANY($2)
10334 | 118 | SELECT instance_id FROM alias_index WHERE deployment_id = $1 AND alias_name = $2 AND alias_value = $3 AND live
10334 | 144 | SELECT store_value, rev FROM data_store WHERE store_name = $1 AND store_key = $2

One lane executes one step at a time, so its commits never share a flush; on this disk that caps the engine near 70 orders per second, and the rest of the offered load queues at intake (intake p99). settled can exceed sent: k6 gives up on a request after 10 seconds and does not count it, but the intake had already accepted it, and it completes like any other.

Step 2: lanes, and a pool to match

Each lane holds a database connection while it commits, so raise the main pool with the lanes:

RATE=100 DURATION=60s ENGINE_ENV="SUTRA_ENGINE_SHARDS=8" tools/sutra-load-test/relay-bench.sh
[relay-bench] warm-up 10/s x 10s
[relay-bench] measured run 100/s x 60s
| offered | sent | settled | dup | drained | settles/s | settle p50 | settle p99 | intake p50 | intake p99 | commits/settle | WAL flushes/settle | WAL KB/settle | WAL fsync busy | engine CPU (cores) | PG CPU (cores) | CPU ms/settle (engine+PG) | engine env |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 100/s x 60s | 6001 | 6001 | 0 | yes | 99.8 | 20 ms | 72 ms | 5 ms | 26 ms | 15.9 | 2.81 | 10.2 | 87% | 0.35 | 0.37 | 7.31 | `SUTRA_ENGINE_SHARDS=8` |

Then the target rate:

RATE=500 DURATION=60s \
  ENGINE_ENV="SUTRA_ENGINE_SHARDS=32 SUTRA_DATASOURCE_MAX_CONNECTIONS=40" \
  tools/sutra-load-test/relay-bench.sh
[relay-bench] warm-up 10/s x 10s
[relay-bench] measured run 500/s x 60s
| offered | sent | settled | dup | drained | settles/s | settle p50 | settle p99 | intake p50 | intake p99 | commits/settle | WAL flushes/settle | WAL KB/settle | WAL fsync busy | engine CPU (cores) | PG CPU (cores) | CPU ms/settle (engine+PG) | engine env |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 500/s x 60s | 30001 | 30001 | 0 | yes | 499.1 | 31 ms | 185 ms | 8 ms | 52 ms | 14.1 | 0.57 | 10.2 | 89% | 1.79 | 2.01 | 7.69 | `SUTRA_ENGINE_SHARDS=32 SUTRA_DATASOURCE_MAX_CONNECTIONS=40` |

Read the columns this way:

ColumnMeaning
settles/sorders that called back, per second, from the start of the run to the last callback
settle p50 / p99time from k6 sending the order to the sink receiving the callback
intake p50 / p99time for the intake to answer 202
commits/settledatabase transactions per order, read-only ones included
WAL flushes/settledisk flushes per order — below 1 means commits share flushes
WAL fsync busyshare of the run the disk spent flushing
engine CPUcores the engine used
PG CPUcores PostgreSQL’s backends used (since 0.6.6)
CPU ms/settleengine plus PostgreSQL CPU time per order, in milliseconds — the number to watch when the host, not the disk, is the limit (since 0.6.6)

When WAL fsync busy is near 100% and adding lanes no longer raises settles/s, the disk is the limit: a faster disk, or PostgreSQL’s commit_delay (which makes a flush wait a few microseconds for more commits to join it, still durable), are what move it next. When the disk is not busy but every core is, the CPU is the limit: CPU ms/settle times the rate you want is the core count it takes. 0.6.6 cut it by about a fifth on this workload: with SUTRA_OUTBOX_WORKERS=32 SUTRA_OUTBOX_TICK_INTERVAL=PT0.01S added to the last run, 8.05 ms on 0.6.5 and 6.33 ms on 0.6.6.

The settings

SettingDefaultRaise it when
sutra.engine.shards1the disk is busy less than the time it takes to flush, and orders queue at intake. Read Execution lanes first: lanes run different instances in parallel.
sutra.datasource.max-connections8you raise the lanes: keep it at least the lane count, plus a few
sutra.datasource.dispatch-max-connections4rarely; the dispatcher’s claim and its batched delete each hold one connection, the rest serve the defers of failed sends
sutra.outbox.workers32many destinations are slow at once: it bounds the sends in flight across all of them
sutra.outbox.destination-concurrency32one destination is slow and can take more parallel requests; see Outbound bursts
sutra.outbox.max-buffered4096sutra.outbox.buffered sits at the bound during a burst and memory allows
sutra.outbox.batch-size50a deployment enqueues far more than 50 rows between claims
sutra.outbox.tick-intervalPT5Syou want retries and rows from other replicas picked up sooner; local deliveries do not wait for it
cache: on a data storeoffa store is read on every request and changed rarely — see Caching reads

Outbound bursts

Each process instance’s outbound rows form one ordered chain: they leave one after another, in the order they were enqueued, while different instances’ chains send in parallel. Two bounds apply to a send, taken in this order: one per destination authority (scheme, host and port; sutra.outbox.destination-concurrency) and one across every destination (sutra.outbox.workers). A row waiting for a busy destination holds no global slot, so a slow counterparty delays only its own rows. The claimer stops claiming only when sutra.outbox.max-buffered rows are held in memory.

Three instruments show whether outbound delivery keeps up:

  • sutra.outbox.delivery.lag (histogram, milliseconds, by scheme): enqueue to delivery, once per delivered row. Its p99 is the number to watch.
  • sutra.outbox.in-flight (gauge): at sutra.outbox.workers for long, the global bound is the limit.
  • sutra.outbox.buffered (gauge): at sutra.outbox.max-buffered, the claimer is waiting.

The defaults come from a burst bench: tools/sutra-load-test/relay-bench.sh BURST=3500 enqueues 3,500 rows in about 6 s across two HTTP destinations, one answering at once and one after 500 ms (30% of instances), on durable PostgreSQL. Lag is measured by the receiving end, against the instant the engine built each request.

Engineworkers / destination-concurrencyDrainFast lag p50 / p99 / maxSlow lag p99Slow rows/s
0.8.28 workers by instance84.5 s35.9 s / 73.9 s / 78.9 s73.6 s12.5
0.8.38 / 865.5 s6.3 s / 18.8 s / 19.1 s59.1 s16.1
0.8.316 / 865.5 s5 ms / 7 ms / 10 ms59.8 s16.0
0.8.316 / 1632.8 s172 ms / 666 ms / 864 ms27.6 s32.1
0.8.332 / 1632.9 s5 ms / 11 ms / 17 ms27.5 s32.1
0.8.332 / 32 (default)16.5 s25 ms / 166 ms / 187 ms12.3 s64.2
0.8.364 / 1632.9 s4 ms / 7 ms / 9 ms27.8 s32.0
0.8.364 / 3216.5 s4 ms / 7 ms / 15 ms12.3 s64.2
0.8.3128 / 3216.5 s5 ms / 8 ms / 11 ms12.2 s64.2

Peak PostgreSQL connections stayed at 8–9 in every run.

The defaults are the smallest pair whose fast-destination p99 stays under 2 s and whose drain time is within 10% of the best. The slow destination drains at about destination-concurrency / its latency; the fast one is only held back once the slow one can take every global slot (8 / 8 above).

Watch outbound lag under a burst

This walkthrough runs the burst bench twice against a local OpenTelemetry Collector, first with a global bound so small that the slow destination takes all of it, then with the defaults, and reads sutra.outbox.delivery.lag from the collector each time. It needs Docker, k6, PostgreSQL 18’s server binaries and a release build of the engine and CLI (cargo build --profile release-it -p sutra-dist -p sutra-cli).

Step 1: start a collector that receives OTLP and exposes the metrics for Prometheus:

cat > /tmp/otelcol.yaml <<'YAML'
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
exporters:
  prometheus:
    endpoint: 0.0.0.0:8889
service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheus]
YAML
docker run -d --name walk-otelcol -p 127.0.0.1::4317 -p 127.0.0.1::8889 \
  -v /tmp/otelcol.yaml:/etc/otelcol-contrib/config.yaml:ro otel/opentelemetry-collector-contrib
OTLP=$(docker port walk-otelcol 4317/tcp | head -1)
PROM=$(docker port walk-otelcol 8889/tcp | head -1)

Step 2: a burst with the slow destination holding the fast one. Eight sends in flight in total and eight per destination: the slow destination can take every slot.

export SUTRA_ENGINE_BIN=rust/target/release-it/sutra-engine SUTRA_CLI=rust/target/release-it/sutra
BURST=3500 ENGINE_ENV="SUTRA_TELEMETRY_OTLP_ENDPOINT=http://$OTLP \
  SUTRA_TELEMETRY_METRICS_EXPORT_INTERVAL=1000 SUTRA_TELEMETRY_SERVICE_NAME=walk-8 \
  SUTRA_OUTBOX_WORKERS=8 SUTRA_OUTBOX_DESTINATION_CONCURRENCY=8" \
  tools/sutra-load-test/relay-bench.sh
curl -s "http://$PROM/metrics" | grep 'sutra_outbox_delivery_lag_milliseconds_bucket{job="walk-8"' \
  | grep -E 'le="(100|1000|10000|\+Inf)"'

The bench’s block shows the fast destination waiting: on our run its lag was 6.5 s / 17.5 s / 17.7 s (p50 / p99 / max), and the histogram has 41 of the 3,500 rows within 100 ms, 231 within a second and 1,746 within ten.

Step 3: the same burst with the defaults (32 / 32):

BURST=3500 ENGINE_ENV="SUTRA_TELEMETRY_OTLP_ENDPOINT=http://$OTLP \
  SUTRA_TELEMETRY_METRICS_EXPORT_INTERVAL=1000 SUTRA_TELEMETRY_SERVICE_NAME=walk-default" \
  tools/sutra-load-test/relay-bench.sh
curl -s "http://$PROM/metrics" | grep 'sutra_outbox_delivery_lag_milliseconds_bucket{job="walk-default"' \
  | grep -E 'le="(100|1000|10000|\+Inf)"'

The fast destination no longer waits: 26 ms / 165 ms / 184 ms, and 2,139 rows land within 100 ms. The rows still over a second are the slow destination’s own (1,048 of them, drained at 64 rows/s), and the whole burst drains in 16.5 s instead of 66.7 s. The histogram carries only the scheme attribute, so both HTTP destinations share one series; the per-destination split comes from the bench’s sink.

Step 4: clean up. docker rm -f walk-otelcol. The bench stops its own engine, sinks and database.

PostgreSQL autovacuum

Several engine tables churn at the request rate:

  • outbox_entry gets one insert and one delete per delivery;
  • instance_state is rewritten on every step;
  • waiting_event, alias_index and inbox_seen gain and lose rows as instances park, finish and age out.

Each delete or rewrite leaves a dead row version, and the engine’s hot queries step over those until a vacuum removes them. The claim of due outbox rows is the one that suffers most.

PostgreSQL’s defaults vacuum a table once 20% of it is dead and look for such tables once a minute. That is too late for a queue: it stays small while it produces thousands of dead rows a second. One measured run at about 1,700 requests a second used the defaults. Every two and a half minutes the outbox claim took 1–1.6 s, throughput fell below 100 a second for some 40 s, and p99 latency reached 38 s.

Since 0.6.8 the engine’s migrations set per-table values. You need do nothing for them:

TableVacuum afterCost throttle
outbox_entry5,000 dead rows, whatever the table’s size (autovacuum_vacuum_scale_factor = 0, autovacuum_vacuum_threshold = 5000)none (autovacuum_vacuum_cost_delay = 0); the table is small
instance_state, waiting_event, alias_index, inbox_seen1% of the table plus 5,000 rows (0.01, 5000)the server’s setting; these tables can be large

Statistics are refreshed on the same kind of trigger. At low rates 5,000 dead rows take a long time to accumulate, so these settings cost nothing there.

One setting is server-wide: autovacuum_naptime. It sets how often autovacuum wakes to look, and a table cannot override it. For a deployment running hundreds of requests a second or more, lower it in postgresql.conf, or in your managed service’s database flags:

autovacuum_naptime = 1s              # default 1min
autovacuum_vacuum_cost_limit = 2000  # default 200 (via vacuum_cost_limit); lets the larger tables keep up

The run above, repeated with naptime 1s and an unthrottled global autovacuum, went on for ten minutes at 2,500 requests a second with p99 329 ms and no slow statements. A busy autovacuum worker uses some CPU and I/O, so watch both after the change.

MySQL/MariaDB and SQL Server have no per-table equivalent. InnoDB purge and SQL Server’s ghost cleanup are server-wide, and the engine ships no setting for them.

Profiling

PERF_DIR=/some/dir records a CPU profile of the engine and of the database backends over the measured run, and writes a flamegraph of each plus a list of the hottest functions. It needs perf usable by your user (kernel.perf_event_paranoid of 1 or less) and inferno (cargo install inferno). Build the engine with frame pointers so the stacks are whole:

(cd rust && RUSTFLAGS="-C force-frame-pointers=yes" cargo build --profile release-it -p sutra-dist)
PERF_DIR=/tmp/relay-profile RATE=100 DURATION=30s tools/sutra-load-test/relay-bench.sh

On the run above the engine used well under one core; most of its time went to the database protocol, and the rest of the wall clock was the disk.