Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Introduction

Sutra

Sutra is a Rust-native, message-native workflow engine built on BPMN 2.0. It turns a standard BPMN process into a declarative way to consume typed, schema-validated messages off any channel, route and correlate them, pause for human decisions, and reply — as a single statically-compiled binary (no JVM, no GC) that is multi-tenant, content-addressed, and hot-deployable.

Most workflow engines treat the payload as an opaque blob and start a process with a REST call. Sutra makes the message a first-class, typed contract: a start event binds to a channel (HTTP or a broker) and a message type; the engine decodes the real wire format, validates it against its schema on the way in, and drives the process with the typed payload, surfacing violations as routable soft errors rather than exceptions. The engine itself ships the schema-less structural formats (JSON, XML, YAML, CSV, raw text and bytes); a typed contract comes either from the deployment package — drop your XSDs into it and the engine compiles them when the package deploys — or from an extension crate implementing the codec SPI, which is how a message standard (an industry wire format with its own envelope grammar and schema editions) is served.

flowchart LR
    W["Wire bytes<br/>HTTP · Kafka · SQS · …"] --> C["Channel<br/>transport + auth"]
    C --> D["Codec<br/>decode + schema-validate"]
    D -->|"typed payload"| P["Process<br/>BPMN 2.0 + DMN"]
    D -.->|"violation"| R["Reject path<br/>a branch you modelled"]
    P --> O["Reply · emit · park for a human"]

Validation sits in the doorway, not inside your process — which is why a schema violation is a branch you drew rather than an exception you catch.

The engine core is domain-neutral: no business vertical lives in the engine itself — it lives entirely in codecs, channels, and modules. It also collects no telemetry of its own — no usage statistics, no crash reports, nothing phones home, ever; the only data a running engine ever sends anywhere is what you explicitly configure it to export (see No telemetry, no phone-home) — worth stating plainly for an engine meant to carry sensitive, typed messages.

Why Sutra exists

Nearly every production-grade BPMN 2.0 engine is JVM-based; the production-grade native-language orchestrators (Temporal, Cadence, Argo) deliberately aren’t BPMN. Sutra is built for that gap — a native-compiled, standards-oriented BPMN engine that treats the message as the contract.

What Sutra ships

Sutra is four things, and this book covers all of them:

  1. The engine — a container image built from the sutra-dist composition root (docker build -f rust/Dockerfile rust/), the thing you actually run. It force-links the built-in formats, transports, redactors, and secret resolvers into one deployable binary.
  2. The sutra CLI — one binary (rust/crates/sutra-cli) that scaffolds, validates, packages, deploys, and inspects deployments. See Getting started and the CLI reference.
  3. Two standalone library crates — sutra-feel (the FEEL expression language) and sutra-dmn (the DMN 1.5 decision-table core). Both compile and run independently of the engine — embed FEEL or DMN evaluation in your own Rust project without pulling in BPMN, channels, or persistence.
  4. This book — Getting Started, Building BPMN Solutions, Architecture, Operating, Reference, and Contributing.

Where these run. The CLI is published for linux x86_64/arm64, macOS x86_64/arm64 and windows x86_64; the engine image is Linux only (linux/amd64 + linux/arm64). So a Mac or Windows developer authors, lints, packages and deploys natively, and the engine runs as a Linux container — the same way it runs in production. The two library crates are plain Rust and build wherever cargo does. Details in installation.

Honest capability summary

  • BPMN 2.0 coverage. Start/end/intermediate events, all four gateway kinds, embedded / transaction / ad-hoc / event sub-processes, service/script/business-rule/user tasks, data objects and data stores, link/escalation/error events, compensation, and path-coverage instrumentation for compliance reporting — see Coverage: declared routes as the compliance signal.
  • DMN + FEEL conformance — measured, not asserted. Sutra’s DMN/FEEL evaluator is checked against the real OMG DMN Technical Compatibility Kit. Current standing: compliance level 2, 100% (126/126); compliance level 3, 99.4% absolute (3365/3385), with 100% of attempted cases passing (0 semantic failures — the remaining gap is 20 enumerated assertions it deliberately doesn’t attempt: external Java function execution, and dates with nine-digit years). See DMN-TCK conformance for what that means and what is deliberately out of scope.
  • Channels and transports. HTTP, five brokers (Kafka, RabbitMQ, AWS SQS, Google Pub/Sub, AMQP 1.0), Dapr pub/sub, Knative Eventing, and an air-gapped file transport — nine in all, behind one neutral transport SPI — see Channels and transports.
  • Typed codecs — the machinery, not a catalog of standards. Decode and schema-validation are one step, and what a codec yields is a messageType, a walkable payload, and a shape every FEEL path in the process is checked against at load time. Six formats are built in (JSON, XML, YAML, CSV, raw text, raw bytes). A schema-bound codec comes from one of two places: the deployment package itself — XSDs under schemas/<name>/, compiled at deploy time, no Rust and no engine build — or an extension crate against the public codec SPI, which is what a message standard (an industry wire format with its own envelope grammar, wrapper profiles, and schema editions) needs and what a downstream distribution force-links for itself. The engine ships no domain codec of its own; see Domain neutrality and the SPI model.
  • Stateful execution. Wait states (userTask, intermediate message catch) suspend, persist, and rehydrate on PostgreSQL, correlated by a business key you name — see Wait states and human tasks. State shared across instances lives in a data store, which is key/value by default — one row per key, holding a value of any shape — and which can additionally project a flat record onto real typed columns, so the SQL tooling you already point at that database reads it directly.
  • What’s still thin. There is no operations console — the admin HTTP API is complete, but a UI over it is roadmap, not shipped. Worker helper libraries beyond the HTTP pull API are ecosystem work that hasn’t happened yet, and macOS has no published CLI binary (it builds from source). Where a chapter below is a stub, it says so explicitly rather than padding.

Where to go next

  • Installation — one line for the CLI, one pull for the engine.
  • Quickstart — scaffold an app and watch a typed message flow through: decode → validate → route → reply.
  • Concepts — the ideas that make Sutra different: typed message contracts, channels, wait-states, the q: vocabulary, and content-addressed deployment.

This book is a work in progress. Chapters are being filled in as the 1.0 release approaches.

What’s new in 0.8

Engine 0.8.0 adds one attribute to the q: namespace, so a flow can choose the status and headers of a synchronous HTTP answer. The addition is backward compatible: a package written for 0.7 or 0.6 loads and runs unchanged. What changes on upgrade is listed in Upgrading to 0.8.0. 0.8.0 is also the first engine image since 0.6.25: 0.7.0 published no image, so everything in What’s new in 0.7 arrives with it.

FeatureWhat it gives youTry it
Outbound delivery no longer queues behind a slow destination (0.8.3)Each instance’s deliveries form their own ordered chain, under a global bound (sutra.outbox.workers, now 32) and a per-destination bound (sutra.outbox.destination-concurrency, 32). On a 3,500-row burst against a fast and a slow destination, the fast one’s p99 delivery lag went from 73.9 s to 166 ms. New metrics sutra.outbox.delivery.lag, sutra.outbox.in-flight, sutra.outbox.buffered.Watch outbound lag under a burst
AMQP 1.0 channels process deliveries concurrently (0.8.3)max-in-flight (default 16) intake calls per binding; a message is still accepted only after it is persisted, and 1 keeps the previous one-at-a-time behaviour. prefetch-count now sets the link credit.Channels
FEEL scopes without deep copies (0.8.3)Filters, quantifiers, for loops, context literals and function calls no longer copy the whole variable scope per item; results are identical.Rules
Status and headers on a synchronous answer<q:reply status="201"> (digits or FEEL) and the reply’s <q:header> children set the HTTP status and response headers, so a flow can answer like a REST API or an auth check.Answering with a status and headers
Broker outages as messagesWith sutra.engine.connectivity-events.channel set, each outage start and end the AMQP transport sees on a deployment’s channel reaches that deployment as a JSON message.Broker outages as messages
FEEL durations keep fractional secondsduration("PT0.001S") is one millisecond, not zero, through parsing, string(), arithmetic, comparison and .seconds.Durations with fractional seconds
No accepted message lost across a database restart (0.8.2)A data task that cannot reach its database (a restart breaks every pooled connection) now refuses the delivery for a retry (HTTP 503, broker redelivery, outbox backoff) instead of dead-lettering it, and a step that has run rides out a short outage before giving up, so it is recorded rather than run again. New code SUTRA.RUNTIME.DATASTORE.UNAVAILABLE.Data stores
The AMQP 1.0 sink keeps its connections (0.8.2)Up to 8 connections per broker (SUTRA_SINK_AMQP_MAX_CONNECTIONS), each with one session and a link per target, instead of a connection per message: on a durable queue 22.3 → 227 messages/s one at a time. Delivery guarantee unchanged.IBM MQ
The SQL audit sink writes in batches (0.8.1)With sutra.audit.sql on, the audit writer commits once per batch of up to 256 events instead of once per event, so on a durable PostgreSQL the journal keeps up with a busy engine (on a 50 orders/s relay workload it had been 85,770 rows, about five minutes, behind). Still best-effort and behind execution; nothing to configure.Audit sinks

What’s new in 0.7

Engine 0.7.0 adds one attribute to the q: namespace, so a process can delete a data-store row. The addition is backward compatible: a package written for 0.6 loads and runs unchanged. What changes on upgrade is listed in Upgrading to 0.7.0. 0.7.0 published its CLI and crates but no engine image; the first image with these features is 0.8.0 (see Upgrading to 0.8.0).

FeatureWhat it gives youTry it
Delete a data-store row from a process<q:store key="…" delete="true"/> on a data task’s write removes the row; with expect="unchanged" it applies only if the row is unchanged since the instance read it. A row created again never reuses a revision, so no replica’s read cache serves the old one.Deleting a row
Answers no instance waits forA message that correlates to no live instance is logged and audited, and uncorrelated: { start: … } on its channel can send it to a process instead of refusing it.An answer no instance waits for
IBM MQ over AMQP 1.0The amqp transport works with an IBM MQ queue manager’s AMQP channel: queues and topics, plain MQSTR messages with no MQRFH2 header, TLS and SASL, and logged outage windows.Connecting the engine to IBM MQ
Codecs that seal what they sendPayloadCodec::seal and bind_refs let an extension codec sign outbound messages, template renders included, with keys named as references.Domain neutrality and the SPI model

What’s new in 0.6

Engine 0.6.0 lets a process work with sound, images and stored recordings, share a limited number of slots across deployments, and loop back to a wait as many times as it needs. Every feature below has a walkthrough you can run on your own machine with the bundled stub model server, so no provider key is needed.

FeatureWhat it gives youTry it
Audio in model callsaudio parts on a user turn, formats written as media types, audioOutput for a spoken answer, and audio / transcript / audioPreserved on the reply. require: gains audio-input, audio-output, audio-understanding and video-understanding.Audio, images and video
voice: pipelineSpeech-to-text, then a text model, then text-to-speech, through an llm.speech provider, for models that take no audio.The voice pipeline
Video and image understanding (preview)Video clips, sampled frames and a one-frame image part.Try it
Native gemini providerGemini’s own API, which carries audio, images and video.Providers
Structured answersA call with json: true gets the parsed answer under json, so a flow reads its fields directly.The reply
Media references and object storesNamed object stores in the engine configuration. A media part gives {ref, sha256} instead of inline data, and the channel reads it from llm.media-store.Recordings and media references
q:audit capture="media"Audit events carry media references, never the media.Recordings: the audit trail
Erasure and retention delete recordingsSubject erasure and terminal retention delete the referenced objects and amend the audit events.Recordings: erasure and retention
Capacity pools and session-end:Named, cluster-wide slot counts drawn from by inbound and outbound channels. A full pool answers 503 inbound and holds outbound requests in the outbox. A session can end before its instance does.Capacity pools and sessions
Re-enterable waitsA loop back to a message catch, user task, timer or channel call parks a new wait each time round.Waiting at the same node again
Timer intervals from FEEL (0.6.4)A timeDuration, timeDate or timeCycle, a q:timeout and the q:retry delays can each be a FEEL expression over the instance’s variables and config.<key>. Deployment properties: in package.yaml hold per-environment values.Timer intervals from FEEL
Durable throughput (0.6.5)A concurrent outbox dispatcher, in-process hand-off of local:// deliveries, fewer durable commits per step, separate pools for intake and delivery, and an opt-in data-store read cache. On one laptop with durable PostgreSQL a two-process relay went from 14 to 499 completions per second, p99 250 ms.Tuning for throughput
Answers that beat their wait; less CPU (0.6.6)A local:// answer that reaches a process before it has parked at the wait for it is held and retried instead of dropped. HTTP clients read the CA bundle once. About a fifth less CPU per completed relay.An answer that arrives before its wait, Tuning for throughput
Binary snapshots; restart-proof local deliveries (0.6.7)Instance snapshots are stored in a binary container — an older release cannot read it, so upgrade every replica at once. A local delivery that fails because the database restarted is retried instead of poisoned.The binary container
Replicas no longer send twice (0.6.8)Engines sharing one database each claimed outbox rows another was still delivering, so 0.6.5–0.6.7 sent about half of all outbound messages twice under multi-replica load. The claim now leases its rows (30 s, renewed while in flight).Outbox processing
Autovacuum for the busy tables (0.6.8)On PostgreSQL the outbox queue and the other high-churn tables have their own autovacuum settings, so dead rows no longer stall the outbox claim at high rates. Lower autovacuum_naptime on the server for high-rate deployments.PostgreSQL autovacuum
Correlation keys on audit events (0.6.9)Every audit event in the JSONL and OTLP sinks carries the instance’s <q:alias> keys and values, redacted like any other audit content. SUTRA_AUDIT_OTEL_ENDPOINT no longer panics the engine at start, and the JSONL sink writes exactly one event per line. 0.6.9 is also the first release of the 0.6.8 changes above.Correlation keys on every audit event
Key queries on a data store (0.6.10)A data-task read returns the rows whose key starts with a prefix or lies in a range, in key order and paged, with count and exact-decimal sum aggregates over every matching row, optionally grouped. One index range scan, no locks. expect="absent" claims a key once across replicas, and a data-store conflict can be caught by an error boundary.Querying rows by key
Relays honour a wait’s message type (0.6.10)Several waits on one channel are told apart by messageTypeValue / messageTypePattern, like start events. Also in 0.6.10: FEEL builtins such as context put and list contains parse in the package lint, and the lint checks assignment expressions and multi-instance loop items.Several waits on one channel
A failed send stays with its failed step (0.6.11)A process that sent and then failed on its first pass left that send for the next request on the same engine lane, which then failed with an HTTP 500 or delivered the failed process’s send. Every release up to 0.6.10 is affected.Each step’s sends are its own
Outbox renewal never waits; audit resource attributes (0.6.12)The claim-lease renewal no longer deadlocks with the batch delete of delivered rows (0.6.8–0.6.11) and no longer takes seconds when the engine falls behind. Data stores that share a table no longer race their first use. OTLP audit records carry OTEL_RESOURCE_ATTRIBUTES and the new sutra.telemetry.resource-attributes.Outbox processing, Resource attributes on audit records
Typed-store queries, a migration ledger, deployment manifests (0.6.13)<q:store query=…> and <q:aggregate> run on typed-column stores as SQL aggregates. Module-store migrations run once, recorded in sutra_store_migrations, refuse an edited script, and take ${package.*}/${store.*} placeholders. sutra lint/package/deploy accept a deploy/<name>.yaml manifest that stacks layers, with per-layer .ignore.Querying a typed-column store, Migrations run once, Deployment manifests
JSON from OpenAI audio models; the reason for a failed model call (0.6.14)An OpenAI audio model (gpt-audio…) asked for JSON is asked in words, since it refuses response_format; a fenced JSON answer inside prose is parsed. A failed llm call logs the provider’s HTTP status, error code and message at WARN under SUTRA.LLM.CALL.FAILED, keys redacted, and the failed instance’s stored diagnostic carries it.How JSON is asked for, Finding the reason for a failed call
A failed synchronous channel call can be caught (0.6.15)A permanent failure of a synchronous channel call is a BPMN error on its task, under the failure’s own code: a boundary or an error event sub-process catches it, and the catching path reads callFailure (code, message, node). Uncaught, it fails as before.Catching a failed call
A delivery that can’t be checked is refused; event.body is masked (0.6.17)A delivery with an explicit event id or a declared dedupKey is refused and retried while its duplicate check can’t answer, instead of running unchecked. See Acknowledgement modes. A flow that declares masking now masks event.body and every inbound header too: it used to show them in audit, inspect and snapshots.
One endpoint, one deployment; acceptance receipts; refused deliveries (0.6.18)A second deployment claiming a taken endpoint is refused at deploy, with the first still serving; a new version may take a route from an older one, which then drains whole. An on-persist HTTP channel answers with an acceptance receipt naming the instance, and a duplicate names its original. A declared <q:source dedupKey> always dedupes, and an unrecognised form is refused. Reading GET /sutra/instances takes the admin key when admin auth is on. A broker message with no content type is decoded by the channel’s codec, and a permanent refusal now leaves a WARN and an audit record. A synchronous flow with no <q:reply> answers an empty body.One endpoint, one deployment, Acceptance receipts, Refused deliveries, Upgrading to 0.6.18
Keep a message as text; references in connection properties (0.6.19)A flow can keep a message as raw text in a typed store column and read it back decoded with the new <q:decode> data-task step. event.bodyLossy marks a body whose bytes were not valid UTF-8. The connection properties of kafka, rabbitmq, amqp, gcp-pubsub and file channels now expand references, where 0.6.18 refused them.Keeping a message as text, References in connection properties, Upgrading to 0.6.19
A superseded version keeps its own channels; about half the engine CPU; coverage cycles (0.6.20)A version that a newer one supersedes keeps serving the channels only it declares until it retires, and another owner is refused those endpoints meanwhile. A literal local:// send to another package is refused at lint and deploy. Expressions evaluate without copying the variables and jemalloc is the allocator: about 55% less engine CPU at 200 messages/s. Coverage can be mapped to requirements and recorded in named compliance cycles. RabbitMQ and AMQP credentials accept secret:.Upgrading a deployment in place, Coverage, Upgrading to 0.6.20
TLS and SASL brokers; password hashing in FEEL; the CLI on macOS and Windows (0.6.21)The kafka and amqp (AMQP 1.0) transports connect to TLS brokers natively and authenticate with SASL (Kafka: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512; AMQP 1.0: PLAIN), verified by default. FEEL gains password hash and password verify (argon2id), so a flow can sign a user in against a typed store without storing or logging the password. sutra self-update works on macOS, and Windows paths print without a \\?\ prefix.Connecting to a TLS and SASL broker, Signing a user in against a store, Upgrading to 0.6.21
Waits first; dedup across versions; bounded AMQP sends (0.6.22)A message answers the instance waiting for it before it starts anything, with one version and across the versions of a package while an older one drains; on a broadcast channel other processes’ starts still run. A retry that spans a version flip is a duplicate. AMQP 1.0 sends are bounded (SUTRA_SINK_AMQP_OPERATION_TIMEOUT_MS) and retried; a repeated Kafka TLS or SASL failure is logged once with a count; templates compile once, at deploy.A reply answers the instance waiting for it, Upgrading to 0.6.22
xmlEscape in templates; timers keep up under load (0.6.25){{xmlEscape value}} escapes free text for XML element content and attribute values; a missing value renders empty. Timers fire as fast as they fall due, where up to 0.6.24 the engine fired at most about 64 a second for the whole cluster, so timer-driven steps fell further behind the longer a surge lasted. Run more than one shard (sutra.engine.shards) if you expect timer surges.Escaping free text in an XML template, Template helpers, Upgrading to 0.6.25
0.6.24: the 0.6.23 release, publishedThe same code as 0.6.23. Its release run stopped at the engine image scan on a newly published, unreachable libstdc++ advisory; 0.6.24 carries one triaged entry for it, so the image and GitHub Release publish. 0.6.23’s CLI archives and crates.io crates were already published.Release notes
Build information in --version; self-update picks the newest release (0.6.23)sutra --version and sutra-engine --version name the commit and the commit’s date (sutra 0.6.23 (1a2b3c4, built 2026-11-15)), and sutra version --format json carries commit and built. sutra self-update chooses the newest release from a file-store listing by version, whatever the listing’s order, never offers an older one unless --version pins it, and asks for the running binary’s own libc first.sutra version, Staying current
A draining deployment’s instances stay its owner’s (0.6.17)A message only falls back to draining deployments with the same tenant and module labels as its channel’s deployment. It could previously resume another owner’s draining instance. A package without both labels no longer falls back at all.
Live audio sessions (preview)AudioSession and require: [realtime], opened through an llm channel by a media transport, not by a process.Live audio sessions

Telephony, IVRs, meetings and WebRTC are not part of this engine. They live in the separate open-source repository sutra-voice, which builds on the pieces above.

Before you try a walkthrough

Every walkthrough runs from the root of a checkout of this repository. It needs:

  • Docker with Compose v2, plus curl, python3, sha256sum and GNU base64 (on macOS, use base64 -i <file> wherever a walkthrough says base64 -w0 <file>);
  • an engine image and the sutra CLI, both 0.6.0 or later.

The compose files default to the published ghcr.io/startr-trade/sutra:0.8.3 image, and Installation installs the CLI. To run a walkthrough against your own checkout instead, build both from it:

make image                                        # builds sutra-rust-engine:dev
(cd rust && cargo build --profile release-it -p sutra-cli)
export PATH="$PWD/rust/target/release-it:$PATH"   # the sutra CLI you just built
export SUTRA_ENGINE_IMAGE=sutra-rust-engine:dev   # every example's compose file reads this

Each example’s compose file names its own Compose project (sutra-ai-advanced, sutra-media-recordings, sutra-capacity-pools, sutra-review-loop) and publishes the engine on a dynamic loopback port, so the stacks never collide with each other or with anything you already run. Each walkthrough ends by removing its own stack with docker compose -p <project> down -v.

The output shown in each walkthrough was copied from a real run against the stub. Instance ids, timestamps and audio bytes differ on every run.

Installation

Sutra ships as two things, released together and versioned together:

What it isHow you get it
sutrathe CLI that scaffolds, packages, lints, deploys and inspectsone-line install, below
sutra-enginethe runtime that executes your processesa container image, docker pull

Neither needs a Rust toolchain. You only need one to build from source, which is the last section on this page.

Install the CLI

curl -fsSL https://raw.githubusercontent.com/startr-trade/sutra/main/scripts/install.sh | sh

On Windows (PowerShell):

irm https://raw.githubusercontent.com/startr-trade/sutra/main/scripts/install.ps1 | iex

The installer downloads the release build for your platform, verifies its SHA-256 against the release’s own SHA256SUMS, and installs it — into /usr/local/bin when that is writable, otherwise ~/.local/bin. It edits no shell profile and starts no daemon.

Useful knobs:

# Pin a version instead of taking the newest release
curl -fsSL …/install.sh | sh -s -- --version v0.8.3

# Install somewhere specific
curl -fsSL …/install.sh | SUTRA_INSTALL_DIR=~/bin sh

# Lift the API rate limit when retrying (60 requests/hour anonymous -> 5000 with a token)
GH_TOKEN=$(gh auth token) curl -fsSL …/install.sh | sh

If the install fails

raw.githubusercontent.com rate-limits and answers 429 Too Many Requests under load; so does the GitHub API, at 60 requests an hour per IP without a token. Three ways around it, in the order worth trying:

# 1. the SAME installer, served from the release instead of the raw CDN (different host, and
#    the script is versioned with the binaries it installs)
curl -fsSL https://github.com/startr-trade/sutra/releases/download/v0.8.3/install.sh \
  | sh -s -- --version v0.8.3

# 2. a token, which moves the API to 5000 requests an hour
GH_TOKEN=$(gh auth token) curl -fsSL …/install.sh | sh

# 3. no installer at all — the assets are plain files
#    (swap in your target: aarch64-unknown-linux-musl, aarch64-apple-darwin, x86_64-apple-darwin)
gh release download v0.8.3 -R startr-trade/sutra \
  -p 'sutra-*-x86_64-unknown-linux-musl.tar.gz' -p SHA256SUMS
sha256sum --ignore-missing -c SHA256SUMS
tar xzf sutra-*-x86_64-unknown-linux-musl.tar.gz --strip-components=1 -C ~/.local/bin

The installer retries transient failures and times out rather than hanging, and it resolves the release from three different endpoints (/releases/latest, the release list, then git tags), because each of them has been observed failing while a release was perfectly installable. When all three come up empty it tells you to pin --version, which is the one path that needs no lookup at all.

Prefer to see what you are running before you run it? Download install.sh, read it — it is about a hundred lines of POSIX shell — then execute it. Or skip the script entirely and grab the archive yourself from the releases page.

Verify the install:

sutra --version

It prints the version and the build it came from, for example sutra 0.8.3 (1a2b3c4, built 2026-11-15): the release commit and that commit’s date.

Published targets

linux x86_64, linux aarch64, macOS arm64, macOS x86_64, windows x86_64.

x86_64arm64
Linux✅ static musl✅ static musl
macOS✅ built natively✅ built natively
Windows✅ MSVC—

The Linux builds are static musl, so there is no glibc version to match — they run on an old distribution and a new one alike. The macOS builds are compiled on Apple hardware of each architecture rather than cross-compiled.

This table is the CLI. The engine is a container, and its image is Linux only (linux/amd64 + linux/arm64) — which is not a limitation in practice: on macOS or Windows you author, lint, package and deploy with the native CLI, and the engine runs as a Linux container through Docker Desktop, exactly as it does in production. There is no native macOS or Windows engine build, and none is planned.

On Apple silicon that container runs natively, which is why the absence costs so little. The image is a multi-arch manifest, so docker pull resolves to linux/arm64 on its own, and that arm64 build is compiled on a native arm64 runner rather than cross-built or emulated. Docker on macOS is always a Linux VM, and on Apple silicon that VM is arm64 — so you get native-speed execution of the same artifact production runs. (The reverse would be emulated: an linux/amd64 image on Apple silicon goes through Rosetta 2 or QEMU, which is slower and occasionally fails outright. Nothing here asks you to do that.)

On an Apple-silicon Mac running the installer under Rosetta 2, uname -m answers x86_64, so you get the Intel build — it runs, translated. Pass --version and fetch the aarch64-apple-darwin archive by hand if you want the native one.

The macOS binaries are unsigned and un-notarized. That matters only for how you obtained them: a file downloaded by a browser carries com.apple.quarantine, and Gatekeeper refuses it until you clear the attribute —

xattr -d com.apple.quarantine ./sutra

— while an archive fetched by the installer (or any curl) has no such attribute and simply runs. The installer does not strip it for you: quietly removing a security attribute is not an installer’s business.

Get the engine image

docker pull ghcr.io/startr-trade/sutra:0.8.3

One generic image runs every application: behavior comes entirely from the deployment packages you mount or deploy into it. The quickstart starts it for you with Docker Compose, so you can skip this step if you are heading straight there.

Staying current

sutra self-update --check      # is there a newer release? (changes nothing)
sutra self-update              # replace this binary with the newest release
sutra self-update --runtime    # …and pull the matching engine image
sutra self-update --runtime-only   # only the engine image

Updates are never automatic — nothing runs on a timer or as a side effect of another command. Every download is checksum-verified against the release’s SHA256SUMS before it is installed, and the binary is replaced by an atomic rename, so an interrupted update cannot leave a half-written executable on your PATH. --version <tag> pins or rolls back; without it, self-update only ever moves to a newer release. Release candidates count as older than their release, and compare by number (rc.10 is newer than rc.9).

Keeping the CLI and the engine on the same release is worth doing deliberately: they are built, tested and published from one tag, which is what --runtime is for.

Build from source

You need a stable Rust toolchain. Docker is needed only for the container test tier.

git clone https://github.com/startr-trade/sutra.git
cd sutra

make test                                        # the no-docker suite
cargo install --path rust/crates/sutra-cli       # install the CLI you just built
docker build -t sutra-engine:dev -f rust/Dockerfile rust/   # the engine image

Next

Quickstart — a running engine and your first message flowing through it, in about five minutes.

Quickstart

A running engine and your first message flowing through it. About five minutes, and the only prerequisites are the sutra CLI and Docker — on linux, macOS or windows (x86_64 everywhere, arm64 on linux and macOS). The CLI is native on each; the engine runs as a Linux container.

What you will have at the end: a BPMN process, bound to an HTTP channel, that decodes and schema-validates an XML message, branches on the result, and replies — running on the real engine against a real PostgreSQL, not a simulator.

flowchart LR
    A["sutra create app<br/>scaffold"] --> B["sutra package<br/>seal + lint"]
    B -->|".sutra archive"| C["deploy/deployments/<br/>watched directory"]
    C --> D["docker compose up<br/>engine + PostgreSQL"]
    D --> E["curl a message<br/>→ reply"]
    E -.->|"edit, re-package"| B

Five steps, and the last arrow is the loop you stay in: change something, re-package, and the running engine flips to it.

1. Scaffold an app

sutra create app my-first-app
cd my-first-app

That is a complete, deployable application, not a hello-world stub:

packages/my-first-app-main/     the deployment package (the unit you ship)
  package.yaml                  identity: tenant, module, version
  channels.yaml                 the HTTP channel the process listens on
  bpmn/sample.bpmn              the process
  schemas/sample/sample.xsd     the message contract
  templates/*.hbs               the reply bodies
deploy/compose.yaml             engine + PostgreSQL
deploy/deployments/             what the engine watches
deploy/smoke.sh                 a health-gated end-to-end check

2. Package it

sutra package packages/my-first-app-main --out deploy/deployments

package seals the directory into a single .sutra archive: content-addressed, with a manifest hash over every entry. It also lints on the way in — a template that navigates a field the schema does not declare, or a channel with no process bound to it, fails here rather than at 2 a.m. That archive is now the artifact you would promote through environments.

3. Start the engine

docker compose -f deploy/compose.yaml up -d

Compose starts the engine image alongside its own PostgreSQL and mounts deploy/deployments, which the engine watches. The archive you just wrote is picked up and activated on boot.

The host port is deliberately dynamic — nothing assumes port 8080 is free on your machine:

ENGINE=$(docker compose -f deploy/compose.yaml port engine 8080)
curl -s http://$ENGINE/sutra/health/ready
{"status":"UP","checks":[{"name":"sutra-loader","status":"UP","data":{"deployments":1,"shards":1}}]}

deployments: 1 is your archive, live.

4. Send a message

curl -s -X POST http://$ENGINE/channels/sample-in \
  -H 'Content-Type: application/xml' \
  -H 'X-Api-Key: dev-only-sample-key' \
  --data '<SampleRequest xmlns="urn:sutra:deployment:my-first-app-main"><note>hello</note></SampleRequest>'
<Accepted xmlns="urn:sutra:deployment:my-first-app-main" process="sample"/>

The X-Api-Key header is not optional: channels.yaml declares apikey auth on this channel, because the engine refuses to wire an unauthenticated HTTP intake (SUTRA.CHANNEL.AUTH.MISSING_SCHEME). The scaffold’s dev value is supplied by deploy/compose.yaml; a real deployment resolves ${SAMPLE_API_KEY} from its secret store.

The namespace is not decoration either. The codec validates against schemas/sample/sample.xsd, whose targetNamespace is this deployment’s (urn:sutra:deployment:my-first-app-main), so an unqualified <SampleRequest> is a different element as far as the validator is concerned and comes back rejected with no declaration found. Payload namespaces are how every message standard this engine speaks — ISO 20022, MT, NACHA — identifies its documents.

The reply carries it too, which is the half people miss. Accepted and Rejected are declared in this deployment’s own schema alongside the inbound message, so an unqualified answer would be a different element from the one the schema declares — the same trap, on the way out — and a caller that validates what it receives could not check it at all. The contract runs in both directions. See templates/sample-accepted.hbs, which says so in a comment.

That reply came out the other end of a real process: the channel decoded the XML, validated it against sample.xsd, started an instance, ran a gateway on the validation outcome, rendered a template, and replied on the inbound connection. The reply is literally templates/sample-accepted.hbs; edit it to echo a field ({{payload.note}}) and the change is verified against the schema at package time.

Now watch it reject something. Send a payload the schema forbids:

curl -s -X POST http://$ENGINE/channels/sample-in \
  -H 'Content-Type: application/xml' \
  -H 'X-Api-Key: dev-only-sample-key' \
  --data '<SampleRequest xmlns="urn:sutra:deployment:my-first-app-main"><wrong>hello</wrong></SampleRequest>'
<Rejected xmlns="urn:sutra:deployment:my-first-app-main" process="sample" outcome="FATAL"
          reason="element 'wrong' is not expected at this point of the content model"/>

You get the rejection branch, not a stack trace and not a 500. That is the point of the whole design: a schema violation is a routable outcome your diagram models, so the failure path is as reviewable as the happy path.

The bundled check runs both of those for you:

./deploy/smoke.sh

5. Look inside

# What is deployed, and what state is it in?
curl -s http://$ENGINE/sutra/deployments | jq

# What does the engine think this package binds?
sutra describe packages/my-first-app-main/bpmn/sample.bpmn

# Will a message actually reach a process? (no engine needed)
sutra simulate --channel sample-in --dry-run packages/my-first-app-main/bpmn/sample.bpmn

6. Change something

Edit templates/sample-accepted.hbs, then re-package into the watched directory:

sutra package packages/my-first-app-main --out deploy/deployments

The engine picks up the new archive and flips atomically: in-flight instances finish on the version they started on, new messages land on the new one. No restart, no dropped request. That is the same mechanism you would use in production — there is no separate “dev mode”.

7. Regenerate the documentation

The change you just made is also a documentation change, so refresh the catalog and commit both together:

sutra generate docs --input packages --output catalog

That writes one page per artifact — each BPMN page opening with a diagram in BPMN notation, laid out from the process graph itself (your .bpmn carries no coordinates, and needs none) — plus the SUMMARY.md and book.toml that make the whole catalog a buildable mdBook. mdbook build catalog turns it into a site or a PDF.

This is not optional housekeeping. The catalog is committed, and --check is a gate:

sutra generate docs --input packages --output catalog --check

sutra create ci --provider github (or bitbucket) scaffolds a pipeline that runs exactly that, so a package whose docs have drifted from its BPMN fails the build. CI only ever checks — regenerating is yours, in the same commit as the change that caused it.

Clean up

docker compose -f deploy/compose.yaml down -v

Next

  • Your first deployment — deploying over the API instead of a watched directory, and what “Active” actually means.
  • Concepts — why the message, not a REST call, is the contract.
  • Channels — the other eight transports (Kafka, RabbitMQ, SQS, Pub/Sub, AMQP, …) your process can listen on without changing the process.

Anatomy of an app

The quickstart ran a scaffolded app end to end. This page opens it up: what each file does, how a message travels through them, and how to make your first real change — including watching the engine refuse a change that would have broken in production.

If you skipped the quickstart, scaffold one now:

sutra create app my-first-app && cd my-first-app

The package is the unit

packages/my-first-app-main/       ← the deployment package: what you ship
├── package.yaml                  identity (tenant, module, version) — its slot
├── channels.yaml                 which channels this package binds, and how
├── bpmn/sample.bpmn              the process
├── schemas/sample/
│   ├── sample.xsd                the message contract
│   └── codec-manifest.yaml       which schema serves which message type
├── templates/*.hbs               reply bodies
└── datastores.yaml               business data stores this package declares

Everything outside packages/ is scaffolding around the app — deploy/compose.yaml, deploy/deployments/ (the watched directory), deploy/k8s/, deploy/smoke.sh. You own those and can delete them; the package is the part the engine consumes.

One package directory seals into one .sutra archive. Its identity is tenant--module--version (its slot) plus a content hash of everything inside — which is why a re-deploy of identical bytes is a no-op and a changed package replaces its slot atomically.

How the pieces connect

Follow one message:

flowchart LR
    REQ["POST /channels/sample-in<br/>&lt;SampleRequest&gt;…"] --> CH

    subgraph pkg["packages/my-first-app-main"]
        CH["channels.yaml<br/>sample-in → codec"] --> XSD["schemas/sample.xsd<br/>decode + validate"]
        XSD -->|"valid, typed"| BPMN["bpmn/sample.bpmn<br/>start → gateway"]
        XSD -.->|"invalid"| BPMN
        BPMN -->|"accepted"| TA["templates/<br/>sample-accepted.hbs"]
        BPMN -.->|"rejected"| TR["templates/<br/>sample-rejected.hbs"]
    end

    TA --> RES["HTTP reply"]
    TR -.-> RES
  1. channels.yaml declares sample-in as an HTTP channel bound to POST /channels/sample-in, with a codec. The channel is the doorway: it decides how bytes become a message.
  2. The codec (schemas/sample/) decodes the XML and validates it against sample.xsd before any process runs. A violation here is data, not an exception.
  3. bpmn/sample.bpmn has a start event bound to that channel, so a valid message starts an instance — with the payload already typed. The gateway branches on the validation outcome; the accept and reject paths render different templates.
  4. templates/*.hbs produce the reply body, returned on the same HTTP connection because the process declares <q:reply>.

The important property: steps 2 and 3 are separate on purpose. The process never parses anything. Change the wire format later — XML to JSON, HTTP to Kafka — and the diagram is untouched, because format and transport are channel concerns.

Inspect any of this without running an engine:

sutra describe packages/my-first-app-main/bpmn/sample.bpmn        # processes, events, tasks, gateways
sutra dispatch-graph packages/my-first-app-main/bpmn/sample.bpmn  # which channel reaches which process

Make it yours

Add a field to the contract and use it. First, schemas/sample/sample.xsd — add an amount alongside note:

<xs:element name="amount" type="xs:decimal" minOccurs="0"/>

Then use it in templates/sample-accepted.hbs:

<Accepted><note>{{payload.note}}</note><amount>{{payload.amount}}</amount></Accepted>

Re-package, and the engine’s deploy-time verification runs:

sutra package packages/my-first-app-main --out deploy/deployments

Now break it deliberately — change the template to {{payload.amont}} (typo) and re-package:

SUTRA.LINT.NAVIGATION.UNKNOWN_FIELD
  templates/sample-accepted.hbs:1 — 'amont' is not a field of SampleRequest
  did you mean: amount

Packaging fails. That typo never reaches an engine, never waits for the one request that happens to take that branch at 2 a.m. Every FEEL expression and template path in the package is checked against the schema it claims to read — this is the design’s central bet, and it costs you nothing to see it work.

Fix the typo, re-package, and the running engine picks up the new archive and flips to it with in-flight instances left to finish on the old one.

Where each concern lives

Want to…Go to
Understand the model behind all of thisConcepts
Bind a broker instead of HTTPChannels and transports
Decode a different format, or your ownDeployment packages
Wait for a human decision or a replyWait states
Write decisions as tables or rulesRules: DMN, FEEL, and .srl
Retry, schedule, or time-box workRetries, history, schedules
Test a 30-day timer in millisecondsTesting time

Next

Your first deployment — getting the archive onto an engine you did not start with docker compose, and what “Active” really means.

Your first deployment

The quickstart deployed by dropping an archive into a watched directory. That is one of three ways in, and the right one for local work. This page covers the other two and what they have in common — because “deployed” means something precise here.

What a deploy actually is

The deploy unit is one sealed .sutra archive. sutra package runs the fail-closed validation suite and derives the manifest — including the content-addressed deploymentId = sha256(manifest) — from the package directory; nothing in the manifest is hand-authored.

Two consequences worth internalizing:

  • Deploying is idempotent. Re-deploying identical bytes is a no-op, because the bytes are the identity. There is no “did that apply?” ambiguity.
  • A slot holds one active revision. The slot is the package’s stable tenant--module--version. A changed archive under the same slot replaces the active revision in one transaction — a hot-deploy, not a restart — while instances already running stay pinned to the revision they started on.

Three ways in

PathUse it whenHow
Watched directorylocal dev, air-gapped hostswrite the .sutra into the directory the engine watches
Synchronous APICI/CD, any engine you can reachsutra deploy … --api --engine-url …
ConfigMap patchKubernetessutra deploy … (the default when a cluster context is set)
flowchart LR
    PKG[".sutra archive"] --> W["watched directory"]
    PKG --> A["POST /admin/deployments"]
    PKG --> K["ConfigMap patch"]
    W --> V["re-verify fail-closed<br/>store as the slot's new revision"]
    A --> V
    K --> V
    V --> F["two-phase activation flip"]
    F --> ACT["Active"]
    V -.->|"invalid archive"| REJ["SUTRA.DEPLOY.* refusal<br/>nothing stored"]

All three end at the same place: the archive is stored, validated, and activated by the same two-phase flip. They differ only in how the bytes arrive.

The synchronous API

sutra deploy my-first-app-main.sutra --api --engine-url http://localhost:<port>

This is the deterministic path — the call returns only once the deployment is live, so what you see on the wire is what happened:

  1. The CLI uploads the archive to POST /admin/deployments.
  2. The engine re-verifies it fail-closed — a corrupt or invalid archive is rejected here, before anything is stored.
  3. The archive becomes the new active revision for its slot, in the engine’s own datasource.
  4. The engine runs its two-phase activation flip: drain the old revision if there is one, activate the new one.
  5. The call returns synchronously — 200 {deploymentId, phase: "Active"}, or a 4xx carrying the SUTRA.DEPLOY.* diagnostic that explains the refusal.

There is no propagation window to wait out; the HTTP response is the “it’s live” signal. For a deployment large enough that the flip risks a long-held request (mainly behind an ingress with a short read timeout), the same endpoint takes an async mode: 202 {deploymentId, status: "Pending"}, then poll GET /sutra/deployments/{id} until it reads Active or Failed.

In CI, that exit code is your gate — a failed validation fails the pipeline, not the release.

What you get, and what you don’t

A successful deploy leaves you with:

  • A live deployment. GET /sutra/deployments/{id} reports Active.
  • Live channels. Whatever channels.yaml declared is bound and serving — an HTTP channel accepts requests immediately; a broker channel’s consumer is running (for a singleton: true channel, on whichever replica holds the per-channel lease).
  • Nothing else. Deploying creates no infrastructure — no database, no broker, no ingress. Those are provisioned separately (locally by docker compose, in a cluster by the OpenTofu modules under deploy/). A deploy only registers and activates the package’s processes, channels, and store bindings against infrastructure that already exists.

Hot-deploy and rollback

Re-package under the same slot and deploy again. The archive gets a new content-addressed id, the slot name does not, so the engine flips in-process and lets in-flight instances drain on the old revision. Rollback is the identical operation with the previous archive — there is no separate rollback mode to learn, and no “undo” that behaves differently from a deploy.

For the operator-facing detail — draining, retirement, migrating an instance off a pinned revision — see Deploy, hot-deploy, and rollback.

Next

You have installed the tools, run a message through an engine, understood the package, and deployed it for real. Building BPMN solutions picks up from here with the ideas and file formats behind what you just shipped.

Concepts overview

Sutra is organized around a few load-bearing ideas. This page is the map; deeper chapters will expand each one.

flowchart TD
    MSG["A message arrives<br/>on a channel"] --> CODEC["Codec<br/>format × schema"]
    CODEC -->|"typed payload"| INST["Process instance"]
    CODEC -.->|"soft error"| INST
    INST --> WAIT{"needs to wait?"}
    WAIT -->|"no"| DONE["Reply / emit / complete"]
    WAIT -->|"yes"| PARK["Suspend + persist<br/>keyed by YOUR business key"]
    PARK -->|"correlated message<br/>or timer fires"| INST
    INST <-->|"q:store"| STORE[("Data store<br/>state beyond one instance")]

Everything below is one of these boxes examined closely.

The message is the contract

A start event binds to a channel and a message type. The engine decodes the real wire format and validates it against a schema before the process runs, so malformed input becomes a routable soft error, not an exception deep in the flow.

Codec = format × schema

A codec pairs a format (how bytes are structured) with a schema (what a valid message looks like). Decoding and validation are one step. One schema can declare several message types, and dispatch fans out one process per type. Six formats are built in — JSON, XML, YAML, CSV, raw text, raw bytes; a schema-bound codec comes either from the package itself (XSDs under schemas/<name>/, compiled at deploy time) or from an extension crate implementing the codec SPI, which is how a message standard (an industry wire format with its own envelope grammar and schema editions) is served.

Channels and transports

Processes are triggered by messages arriving on channels — HTTP, five brokers (Kafka, RabbitMQ, AWS SQS, Google Pub/Sub, AMQP 1.0), Dapr pub/sub, Knative Eventing, and an air-gapped file transport. <q:source> binds a start event to a channel + message type. Co-deployed processes hand off over an in-process local:// channel.

Wait-states and correlation

Wait states (userTask, intermediate message catch) suspend → persist → rehydrate → resume. A later message is correlated back to the parked instance by a business key you name (<q:alias> — e.g. an EndToEndId or an order reference), not an engine id, and it’s durable and replica-coherent on PostgreSQL.

Durable state beyond one instance

Wait-state data belongs to one instance. State that outlives an instance, or is shared between them, lives in a data store — declared in the package (datastores.yaml), read and written from the flow through <q:store>, and transactional, with optimistic concurrency (expect="unchanged") and pessimistic locking (forUpdate="true") where a read-modify-write needs them. Each store owns its own connection: the engine’s database is never the module’s. A store is key/value by default and holds a value of any shape; a flat record may additionally declare its structure: and project onto real typed columns in a table you own, so the SQL tooling you already point at that database reads it directly.

Rules — DMN, .srl, and FEEL

A businessRuleTask binds a .dmn decision table or a .srl ruleset (a Drools-inspired rule / when / then DSL). Both compile onto one shared FEEL evaluator — no JVM, no Rete runtime.

The q: vocabulary

Layered BPMN extensions keep diagrams standard while removing boilerplate: <q:dispatch> / <q:case> (a routing table instead of gateway sprawl), <q:validators>, <q:alias>, <q:reply>, <q:variables>, <q:audit>, and <q:coverage>.

Content-addressed deployment

The deploy unit is one sealed .sutra archive; runtime identity is a single opaque deploymentId = sha256(manifest). Deploys are idempotent, hot-reload flips activation without a restart, and secrets never live in the archive — channels reference them by scheme (secret: / env: / vault: / aws-secrets: / …), resolved at runtime through one vendor-neutral SPI.

Multi-tenancy and observability

Per-channel tenant binding, single-database PostgreSQL with per-deployment row-level security across PG / MySQL / MariaDB / MSSQL, OTLP traces / metrics / logs to any collector, plus KEDA autoscaling and leader election.

Deployment packages

A Sutra project is a single deployment package. sutra create app produces exactly one; sutra package <dir> seals it into one immutable, content-addressed .sutra archive; sutra deploy activates that archive on a running engine. There is no build step in between — you author declarative resources, seal them, and hand the sealed archive to a generic engine binary that never needs your source. Secrets are the one thing that never travels inside it: a resource names them by scheme (env:NAME, secret:KEY, vault:…), resolved at runtime, and package-time validation rejects a literal outright (see Channels and transports).

flowchart LR
    DIR["the package directory<br/>bpmn · schemas · templates<br/>channels.yaml · datastores.yaml"]
    DIR -->|"sutra package<br/>validates first, fail-closed"| AR["one immutable .sutra<br/>manifest of per-file digests"]
    AR -->|"deploymentId =<br/>sha256(manifest)"| DEP["sutra deploy"]
    DEP --> SLOT["the slot:<br/>tenant--module--version"]
    AR -.->|"carries env:NAME or<br/>secret:KEY, never a value"| SEC["resolved at runtime,<br/>outside the archive"]

Everything the engine needs to run the module is sealed and hashed, so identical bytes re-deploy as a no-op and the same archive promotes across environments unchanged — while the values that must differ per environment stay references it resolves on the way up.

Package interior

<package>/
├── package.yaml             # manifest: labels {tenant, module, version}, engine.minContract
├── bpmn/**/*.bpmn            # BPMN 2.0 processes
├── rules/                    # DMN decision tables (.dmn) + the .srl rule DSL, by extension
├── scripts/**/*.{hbs,xsl,xslt}     # derive/compute (Handlebars + XSLT)
├── templates/                # render output (Handlebars + XSLT)
├── schemas/                  # each leaf folder under here = one codec
│   └── <codec>/*.xsd + codec-manifest.yaml
├── migrations/<store>/**     # per-datastore SQL migrations
├── channels.yaml             # transport channels (bind codec URNs; hot-reloaded on flip)
└── datastores.yaml           # datastore declarations

Every artifact folder supports nested subfolders — bpmn/orders/checkout/…, templates/eu/invoices/… — and discovery is recursive. sutra create deployment <name> --from packages/my-app scaffolds a sibling package as an explicit copy; packages never inherit from one another.

package.yaml — labels, not scope

labels:
  "module": "money-transfer"
  "tenant": "default"
  "version": "1.0.0"
engine:
  minContract: 1

tenant / module / version are opaque labels — selectors for observability, routing, and row-level-security partitioning (tenant_id) — never a resource-tree dimension. A package archive is fully self-contained: there is no tenants/<id>/ overlay tree, no shared-library modules/ folder, and no inheritance between packages at deploy time. What a tenant “sees” is simply every process in the package deployed under its labels. The authoring tree may still be layered — see Composing a package from layers.

properties: — deployment configuration

properties:
  defaultWindow: "${DEFAULT_WINDOW:PT10S}"   # an env reference with a default
  callTimeout: "env:CALL_TIMEOUT"            # a whole-value env reference
  startAfter: "PT1H"                         # a literal

properties: is a flat map of keys to text. A timer expression reads it as config.<key> (see Timer intervals from FEEL). The archive seals the text as written, so the same archive runs in every environment. The engine resolves env:NAME, ${NAME} and ${NAME:default} from its own environment when it activates the deployment. A reference with no value and no default refuses the deployment (SUTRA.DEPLOY.PROPERTY_UNRESOLVED). Keys are letters, digits and _. Values are strings, so use number(config.x) for arithmetic. Secret references (secret:KEY, ${secret:KEY}, a secret store) are refused, because every timer expression and the audit trail can read the configuration. Editing a property changes the deploymentId, the same as an audit: change. Changing the environment does not, and takes effect when the engine next activates the deployment.

Composing a package from layers

When one process is deployed several ways — the same flow bound to HTTP for one tenant and to a broker for another, or a second edition of the same contract — keep the common content once and let each variant hold only what differs:

order-intake/
├── shared/                          # common to every variant; no package.yaml here
│   ├── bpmn/order-intake.bpmn
│   ├── rules/approval.dmn
│   └── templates/receipt.hbs
└── variants/
    ├── acme--order-intake--1.0.0/
    │   ├── package.yaml             # this variant's labels
    │   └── channels.yaml            # HTTP binding
    └── globex--order-intake--1.0.0/
        ├── package.yaml
        └── channels.yaml            # broker binding

Name the shared folder as an overlay, and the two commands see the same composed tree:

sutra lint    variants/acme--order-intake--1.0.0 --overlay shared
sutra package variants/acme--order-intake--1.0.0 --overlay shared -o dist/

The rules, in the order they bite:

  • Overlays are base layers. --overlay may be repeated; each is applied in the order given, and the package directory is applied last — so a variant’s file always wins over a shared one.
  • An overlay carries no package.yaml. Labels belong to the variant, and a package.yaml found inside an overlay is refused rather than quietly merged.
  • Every overridden path is reported, so a variant shadowing a shared file does it visibly.
  • Nothing is deleted. A variant can add to or replace shared content, never remove it. Content a variant must not carry does not belong in shared/ — or use a deployment manifest, whose layers can drop files with .ignore.
  • The package directory’s name names the archive (acme--order-intake--1.0.0.sutra), which is why variant directories are named after their labels.
  • Composition happens at package time. The sealed archive holds the merged file set and knows nothing about layers, and its deployment id is identical to sealing a hand-merged directory with the same content. Layering is an authoring convenience; it changes nothing about what deploys, or about how the engine loads it.

Deployment manifests — customers, editions, environments

--overlay covers one shared folder plus one variant. When a package is assembled from more dimensions — shared content, a product edition, a customer, an environment — write the stack down once in a deployment manifest and hand that file to sutra lint, sutra package or sutra deploy in place of the package directory:

order-hub/
├── shared/schemas/                  # used by several products; mounted at schemas/
├── app/
│   ├── shared/                      # bpmn/ rules/ templates/ channels.yaml
│   └── editions/
│       └── 2026r1/                  # what this edition changes (bpmn/intake.bpmn)
├── customers/customer-a/
│   ├── common/                      # this customer's templates
│   └── envs/
│       ├── dev/                     # channels.yaml + .ignore
│       └── prod/                    # channels.yaml
└── deploy/
    ├── customer-a.2026r1.dev.yaml
    └── customer-a.2026r1.prod.yaml
# deploy/customer-a.2026r1.dev.yaml
deployment: customer-a-dev            # optional, informational
package:                              # becomes the package's package.yaml
  tenant: customer-a                  # tenant / module / version are label shorthands
  module: order-hub
  version: 1.1.0
  engine: { minContract: 1 }
layers:                               # lowest first; later wins; relative to this file
  - path: ../shared/schemas
    at: schemas                       # mount this folder under schemas/
  - ../app/shared
  - ../app/editions/2026r1
  - ../customers/customer-a/common
  - ../customers/customer-a/envs/dev
target:                               # optional: sutra deploy's flags
  engine-url: http://localhost:8088
  api: true

deploy/customer-a.2026r1.prod.yaml is the same file with envs/prod as its last layer and its own target:. The dev layer overrides channels.yaml and drops a process dev does not run:

# customers/customer-a/envs/dev/.ignore
bpmn/reconcile.bpmn
sutra lint    deploy/customer-a.2026r1.dev.yaml
sutra package deploy/customer-a.2026r1.prod.yaml -o dist/     # → dist/customer-a.2026r1.prod.sutra
sutra deploy  deploy/customer-a.2026r1.dev.yaml

sutra lint prints the resolved stack before its findings, so what will be sealed is never a guess:

manifest deploy/customer-a.2026r1.dev.yaml (deployment customer-a-dev)
  layer 1 ../shared/schemas (at schemas)
  layer 2 ../app/shared
  layer 3 ../app/editions/2026r1
  layer 4 ../customers/customer-a/common
  layer 5 ../customers/customer-a/envs/dev
  override bpmn/intake.bpmn: layer 3 ../app/editions/2026r1 over layer 2 ../app/shared
  override channels.yaml: layer 5 ../customers/customer-a/envs/dev over layer 2 ../app/shared
  removed bpmn/reconcile.bpmn: by layer 5 ../customers/customer-a/envs/dev .ignore 'bpmn/reconcile.bpmn' (was from layer 2 ../app/shared)

The rules:

  • It is only a shorthand. The CLI turns the manifest into a generated package.yaml plus the layer stack, then validates and seals exactly as for a directory. Sealing a manifest produces the same archive bytes and the same deployment id as sealing a hand-merged directory holding the same files and the same package.yaml. The archive is named after the manifest file (customer-a.2026r1.dev.sutra).
  • A closed schema. The top-level keys are deployment, package, layers and target; any other key is refused. package: takes the package.yaml keys (labels, engine, entryProcesses, audit, properties) plus the tenant, module and version shorthands; module and version are required.
  • Layers apply in order, a later layer’s copy of a file replacing an earlier one’s; every override is reported. A layer may not carry package.yaml — the refusal names the layer. A missing layer folder, or an at: that is absolute or climbs out with .., is refused.
  • --overlay still works and stacks above the manifest’s last layer.
  • target: takes sutra deploy’s flag names (context, namespace, configmap, secret-name, engine-url, api, async, wait, wait-timeout, api-key-header). A flag given on the command line wins. Secrets never go in a manifest — pass them with --secret or --secret-from.

.ignore — removing what a lower layer brought

A layer may carry a .ignore file at its root. It uses gitignore syntax — # comments, *, ? and […] within a path segment, ** across segments, a trailing / for a directory, a leading or inner / to anchor, ! to re-include — and:

  • patterns are relative to the package root, not to the layer folder (nor to its at:);
  • when the layer is applied, its patterns first remove matching files contributed by the layers below it, then the layer’s own files are added — a layer never ignores its own files;
  • the last matching pattern decides, a pattern naming a directory covers everything under it, and a ! pattern can bring back a single file under an ignored directory;
  • package.yaml can never be ignored;
  • a pattern that removes nothing is reported as a warning (SUTRA.DEPLOY.PACKAGE.IGNORE_UNMATCHED) — it is usually stale or mistyped;
  • .ignore files are never sealed into the archive.

Referencing resources — convention over configuration

Because the package is self-contained, resources are referenced by their local key, derived from the folder tree — never by tenant/module/version:

ResourceResolved by
BPMN processprocessId — globally unique within the package
Rule (.dmn / .srl)its relative path under rules/
Script / templateits full relative path (folder + filename + extension)
Codeca URN — see below

Codecs are the one exception. A globally-named codec — a format the engine links in (json, xml, yaml, csv, raw-text, raw-bytes) or a codec crate a distribution force-links — is urn:sutra:codec:<name>; a package-defined codec is named by its path under schemas/, / folded to : — schemas/transfer/ becomes urn:transfer, schemas/hr/employee/ becomes urn:hr:employee. channels.yaml binds that URN:

# examples/money-transfer/.../channels.yaml
channels:
  - name: transfer-request
    transport: http
    bind: "POST /channels/transfer-request"
    codec: urn:transfer     # schemas/transfer/*.xsd, declared by schemas/transfer/codec-manifest.yaml

A codec-manifest.yaml sits inside its own schemas/<codec>/ folder and declares the schema kind plus the formats the codec accepts:

# examples/money-transfer/.../schemas/transfer/codec-manifest.yaml
schemaKind: xsd
formats: [xml, json, yaml]

Tabular formats — csv and fixed-width

formats: is not limited to document syntaxes. A schema codec can bind csv or fixed-width, which is what turns a delimited or fixed-column upload from a bag of untyped strings into a typed message:

# examples/call-log-load/.../schemas/cdr/codec-manifest.yaml
schemaKind: xsd
formats: [csv, fixed-width]

csv:
  delimiter: ","          # both default; written here only to show where layout lives
  header: true

fixed-width:
  fields:                 # REQUIRED — a fixed-width line has no structure of its own
    - {name: recordId,    width: 12}
    - {name: msisdn,      width: 16}
    - {name: durationSec, width:  6}

A tabular body is a batch, so it is validated row-wise: each row is checked as one instance of the declared root, in a single decode, before the process starts. Every violation carries a row-indexed path (value[3].durationSec), so a bad cell names its record. An unparseable file is fatal; any row’s violation is a soft error, so the payload still projects and q:onValidation decides the posture. The batch projects under value, so a flow reads payload.value[0].field.

The XSD’s leaf types reach the cells: an xs:int column arrives as a number, not "182". An empty cell reads as absence for an element the type declares minOccurs="0" — without that, an optional column would have to be populated in every row. An empty cell for a required element is still a violation.

Both tabular formats may be declared together. Their content types are disjoint — text/csv / application/csv against text/plain / application/x-fixed-width — so an upload selects its parser unambiguously and one schema serves a CSV feed and a fixed-width feed over the same channel.

Layout is parser config, not schema: it cannot be expressed in an XSD or a JSON Schema (neither has byte offsets), which is why it lives in the manifest. A header-bearing CSV names its own columns and needs no block at all; a fixed-width codec cannot work without one and declaring the format without it is a manifest error.

Because a fixed-width layout is configuration, its columns are checked against the bound type at package time: a column the type does not declare, or a required element with no column, fails sutra lint (SUTRA.CONFIG.CODEC_MANIFEST.REJECTED) rather than every row at runtime. A csv codec gets no equivalent check — its column names arrive in the header, at runtime, so there is nothing to compare against beforehand.

The opaque formats (raw-text, raw-bytes) are not schema-bindable: there is no map under raw bytes for a schema to type, so binding one is refused with that reason.

The reserved token sutra may not be used as a first-level subfolder name under any artifact folder (schemas/sutra/, bpmn/sutra/, …) — that would collide with the engine’s own urn:sutra:* namespace. A deeper sutra (schemas/hr/sutra/) is fine.

Schema bundles — when a codec is a whole profile

schemaKind: xsd and schemaKind: json-schema are the generic case: a folder of schema files the engine validates a decoded document against. Some standards aren’t that shape at all — they’re a whole profile: an envelope grammar, a mapping from a wire-level message name to a schema file, and versioned editions the profile revs on its own release cadence, independent of Sutra releases. For those, schemaKind names a bundle kind a codec crate has registered (see Domain neutrality and the SPI model), and the folder’s job shifts from “here are the schemas” to “here is the configuration that decides which schema backs which message, for this archive version.”

A codec crate registers its own bundle kind this way whenever the standard it serves is a whole profile rather than a bare schema folder — say, a market venue that publishes a wrapper envelope around every message and revs its schema editions on its own release calendar. That codec crate is a proprietary extension, not part of this distribution — the mechanism below is generic; naming a concrete kind is only for illustration. Its manifest maps wrapper element names — not a bare schema namespace — onto schema files archived alongside it:

# schemas/<your-kind>/codec-manifest.yaml
schemaKind: <your-kind>
# schemas/<your-kind>/<your-kind>-manifest.yaml
appHdr: edition-2024/Header_v1.xsd   # optional
incoming:
  OrderConfirmation: edition-2026/OrderConfirmation_v3.xsd
outgoing:
  OrderConfirmation: edition-2026/OrderConfirmation_v3.xsd
  ShipmentNotice: edition-2024/ShipmentNotice_v2.xsd
schemas/<your-kind>/codec-manifest.yaml      # schemaKind: <your-kind>
schemas/<your-kind>/<your-kind>-manifest.yaml   # appHdr? / incoming{wrapper: relpath} / outgoing{wrapper: relpath}
schemas/<your-kind>/<edition-folder>/*.xsd   # free-form folder names — the manifest is the only truth

A few things fall out of that shape:

  • Edition folders sit side by side. Because the venue revs its schemas per release (the same wrapper can move from one schema version to the next between editions), archiving each release’s files under its own folder and repointing the manifest is how a module adopts a new edition — a new version of the archive, no engine or codec change.
  • An unlisted wrapper falls back to the codec crate’s own base schema for its pinned default version, so a bundle only needs to carry the wrappers it actually wants to validate more strictly than that default — the codec stays useful with zero configuration otherwise.
  • An enriched edition can be licensed material. A codec’s base schemas may be freely redistributable while a fuller, usage-guideline edition is a licensed product participants obtain themselves and supply in their own archive — the engine never ships that tier, and neither does any artifact built from this repository.
  • Registration is deployment-scoped, exactly like every other artifact under the registration model: the bundle’s registry key is urn:sutra:codec:<folder-path-with-'/'-folded-to-':'>:<deploymentId>. Naming the folder <your-kind> shadows the globally-registered urn:sutra:codec:<your-kind> for that deployment only — a second version of the same module with a different edition mapping registers under its own deploymentId and runs side by side, no collision.
  • Deploy-time errors are fail-closed: an unknown wrapper name, a wrapper listed under the wrong direction, a missing or uncompilable schema file, or a schema whose namespace doesn’t match the expected one all reject the archive rather than deploy something that silently validates less than the manifest claims.
flowchart LR
    W["a wrapper element name on the wire<br/>OrderConfirmation"] --> M["the bundle manifest<br/>incoming / outgoing maps"]
    CM["codec-manifest.yaml<br/>schemaKind names the bundle kind<br/>a codec crate registered"] --> M
    M -->|"listed"| E["that edition folder's schema<br/>edition-2026/OrderConfirmation_v3.xsd"]
    M -.->|"unlisted"| B["the codec crate's own base schema,<br/>at its pinned default version"]
    NEW["a new edition of the profile"] -.->|"archive its folder alongside,<br/>repoint the manifest,<br/>ship a new archive version"| M

The manifest is the only truth about which schema backs which message — the folder names carry no meaning — so adopting a new edition is a package change and a new deploymentId, never an engine or codec-crate change.

Store migrations, and evolving a projected structure

Every sql data store the package declares brings its own schema, under a folder named for the store:

migrations/<store-id>/V001__<description>.sql
migrations/<store-id>/V002__<description>.sql

<store-id> is the store’s name in datastores.yaml, and the migrations: key of that store points at the folder (migrations: migrations/accounts). The scripts are yours: your dialect, your table names, your indexes and seed rows. The engine generates no DDL for a module store.

One store is the exception: the reserved coverage store, where path-coverage marks are persisted. Its declaration still picks the database, but the engine owns its schema — it ships that DDL per dialect and applies it to the connection on the same first-use path — so the store block carries no migrations: key and a package carries no migrations/coverage/ folder. See Coverage: declared routes as the compliance signal.

Three properties of how the package’s own scripts run are worth designing around:

  • Applied in V<n>__ order, before the store serves its first operation — and serialized across replicas, so two engines booting at once don’t race.
  • Each script runs once per database. Applied scripts are recorded in a history table, sutra_store_migrations, in the store’s own database; a recorded script is skipped on every later boot, and editing one after it was applied refuses the store (SUTRA.RUNTIME.DATASTORE.MIGRATION_CHECKSUM). A change is always a new, higher-numbered script. See Migrations run once for the full rules, including the five ${…} placeholders a script can use.
  • A store’s data carries across a version bump. There is no deployment_id on a business store’s rows: deploying 1.0.1 beside 1.0.0 doesn’t fork the data, and that is the feature, not an oversight.

When a projected structure changes

A store that declares a structure: block adds one obligation: a changed structure ships with the migration that makes the table match, in the same package. sutra lint derives the effective table shape from these very scripts, so it tells you at package time whether the pair is consistent — its job is to detect the mismatch, not to repair it.

Change to the declared typeWhat it costs
Add an optional scalar fieldAdditive. A new nullable column; rows written before it read the field as absent
Add a required scalar fieldLint error until the column exists and is nullable or has a DEFAULT — existing rows cannot satisfy a bare NOT NULL
Remove a fieldThe column becomes unmapped (a warning); existing data is untouched
Widen a facet (maxLength 35 → 70)Lint error until the ALTER ships in a new V… script; clean once it does
Rename a fieldModelled as remove + add. A columns: mapping can keep the physical column name instead
Scalar → nested or repeatedHard stop: STRUCTURE_NOT_FLAT. Either keep the field flat, or drop the structure block and go back to the opaque store — an explicit decision at package time, never a silent shape change

What follows from the table is a packaging rule: a type change and the ALTER that supports it belong in the same package. Lint replays every script in the folder in version order and compares the result against the declared type as it stands, so a package carrying both lands clean, while one carrying only the type change fails the gate rather than the deployment.

Building one by hand vs. scaffolding

sutra create app <name> (see Anatomy of an app) generates a package in this exact shape, verified through the engine’s own loaders before anything is written. Growing it from there:

sutra create bpmn my-process --package packages/my-app --validation fatal
sutra create deployment my-app-eu --from packages/my-app   # explicit variant copy

sutra lint <package-dir> runs the full package-time validation suite (the same checks sutra package runs before sealing) with no output on success — the fast pre-flight to run before every package.

Next

Channels and transports

Processes don’t get called — they’re triggered by messages arriving on channels. channels.yaml (one per package, see Deployment packages) declares each channel: which transport it rides, which codec decodes it, how it acknowledges receipt, and who may send to it.

Transports

One neutral transport SPI, ten implementations, all self-registering behind the same lifecycle trait — the engine binds, activates, and drains every one of them through a single generic path with no if transport == "..." branching anywhere:

transport:Notes
httpThe universal baseline. Also serves /sutra/health/*.
kafkardkafka.
rabbitmqlapin (AMQP 0.9.1).
aws-sqsAWS SDK.
gcp-pubsubGoogle Cloud client.
amqpfe2o3-amqp (AMQP 1.0).
daprDapr pub/sub: the sidecar pushes to the engine’s own HTTP listener. No vendor client — see Dapr and Knative.
knativeKnative Eventing: a Trigger or Subscription pushes to the engine’s own HTTP listener. No vendor client.
llmA language model as an outbound request/reply channel, with ordered fallback candidates — see Calling a language model. Provider adapters are cargo features llm-openai, llm-anthropic, llm-bedrock, all in the default build.
fileAir-gapped: file-spool inbound + file:// outbound sink, no network dependency.

Two further transport: values are engine-internal rather than vendor clients — they have no wire protocol and no listener of their own. local delivers in-process to another channel; pull parks the delivery as a task a worker fetches instead of dialing anything, which is the external-task surface.

Phone calls, IVRs and meetings are not part of this engine’s build. The voip transport, the WebRTC media worker and the voice, IVR, meeting and video examples live in the separate open-source repository sutra-voice. It builds on this engine as a git submodule and ships its own engine image with the voip transport linked in. The model side they rely on stays here: llm channels with audio parts, live audio sessions, q:audit capture="media" and the object store (Recordings and media references).

flowchart LR
    H["http"] --> SPI
    B["kafka · rabbitmq · aws-sqs<br/>gcp-pubsub · amqp"] --> SPI
    PU["dapr · knative<br/>pushed to the HTTP listener"] --> SPI
    F["file<br/>air-gapped spool"] --> SPI
    LP["local · pull<br/>engine-internal, no wire protocol"] -.-> SPI
    SPI["One transport SPI<br/>bind · activate · drain, generically"] --> CD["The channel's codec<br/>decode + schema-validate"]
    CD --> SUB["The processes that subscribed<br/>q:source channel + messageType"]

Which transport a channel rides changes only how the bytes arrive: bind, activate and drain are one generic path, and everything past the doorway — decode, validation, subscription — is identical for all of them.

A hardened or air-gapped build selects a subset of transports at compile time via Cargo features on the distribution crate (cargo build -p sutra-dist --no-default-features --features file), so the unlinked vendor clients (rdkafka, the AWS/GCP SDKs, lapin, fe2o3-amqp) are not compiled in at all. dapr and knative are features too, though neither links a vendor client — leaving them out only removes their routes and outbound sinks. An operator can additionally restrict which transports a running binary accepts via SUTRA_ALLOWED_TRANSPORTS — a channel declaring a disallowed transport fails the deployment with a clear diagnostic, not a silent no-op.

Dapr and Knative

Dapr pub/sub and Knative Eventing each have a transport crate of their own (sutra-transport-dapr, sutra-transport-knative), but neither links a vendor client or opens a connection. Both are push transports: the Dapr sidecar, or a Knative Trigger or Subscription, delivers each event as a CloudEvents HTTP POST to a route on the engine’s own HTTP listener. There is no long-lived consumer to leader-elect — the pusher’s at-least-once delivery is the guarantee, and inbox dedup absorbs its redeliveries. CloudEvents extraction follows the channel’s cloudevents-mode, as on http. Outbound, each rewrites its destination to an HTTP URL and sends it through the same HTTP sink the http transport uses.

daprknative
Inbound routePOST /dapr/{topic}POST /knative/{subscription}
Channel property that binds ittopicsubscription
Pointing the pusher at the engineA declarative Dapr subscription whose route is /dapr/<topic> — the engine does not answer Dapr’s programmatic GET /dapr/subscribeA Trigger or Subscription whose subscriber is /knative/<subscription> on the engine’s HTTP listener
Outbound destinationdapr://<pubsub>/<topic>, published through the local sidecar’s /v1.0/publish/<pubsub>/<topic>knative://<namespace>/<broker>, posted to the Broker ingress
Outbound config (process-wide)SUTRA_SINK_DAPR_SIDECAR_PORT (default 3500)SUTRA_SINK_KNATIVE_BROKER_INGRESS, else K_SINK, else a built-in in-cluster default
ack-mode: on-completeNot supported, by design — the channel boots with SUTRA.ACK.ON_COMPLETE_UNSUPPORTED and runs on-persistSupported — the push response is held until the instance ends, bounded by on-complete.hold-timeout (default 30 s)
Inbound guardA dapr-topic header that disagrees with the path’s topic is rejectedA request carrying some but not all of ce-id, ce-source, ce-type is rejected (400)

The sidecar port and the Broker ingress are process-wide because an engine process has one sidecar and one ingress: a channel’s sidecar.port or broker.url is validated but has no effect.

channels:
  - name: orders-in                   # a Dapr subscription routes topic orders.created here
    transport: dapr
    codec: urn:sutra:codec:json
    topic: orders.created
  - name: shipments-in                # a Knative Trigger delivers to /knative/shipments
    transport: knative
    codec: urn:sutra:codec:json
    subscription: shipments
    ack-mode: on-complete
    on-complete.hold-timeout: PT20S   # ISO-8601 or bare seconds; keep it below the Trigger's timeout

Why the two differ on on-complete. On both, the HTTP response to the push is the acknowledgement; there is no separate ack to defer. Knative bounds that response with a timeout the operator sets on the Trigger or Subscription (delivery.timeout), so the engine can hold it until the instance completes (202), fails (422, which Knative sends to the deadLetterSink rather than retrying) or outlives on-complete.hold-timeout (202 with a warning — that one delivery degrades to on-persist). Keep the hold timeout below the sender’s timeout, or every held delivery turns into a redelivery. Dapr’s equivalent bound belongs to the pub/sub component — Redis Streams’ processingTimeout, Service Bus’s handlerTimeoutInSec — which the engine cannot see, and holding a response past it makes the broker deliver the same message again, concurrently. So Dapr runs on-persist, which answers once the dispatch has run to its first wait state or to completion. The full per-transport matrix is in Acknowledgement modes.

Binding a channel

# examples/money-transfer/.../channels.yaml (abridged)
channels:
  - name: transfer-request
    transport: http
    bind: "POST /channels/transfer-request"
    codec: urn:transfer
    cloudevents-mode: none
    ack-mode: on-complete
    auth:
      scheme: apikey
      apikey:
        value: transfer-demo-key
        header: X-Api-Key

A channel does not name a process. Processes subscribe to channels: a start event’s <q:source channel="transfer-request" messageTypeValue="TransferRequest"/> is what routes a decoded inbound to it (see The q: namespace). This is what lets one channel feed several processes and one process listen on several channels — the money-transfer example runs the same transfer flow off three different intake channels (http, rabbitmq, kafka), each a separate <q:source> on the same underlying <bpmn:transaction>.

Fan-out. By default a channel is point-to-point: for a given (channel, messageType), exactly one subscribing process is allowed (enforced at deploy time). Setting broadcast: true fans a decoded message out to every subscribing process, one instance each — genuine pub/sub.

flowchart LR
    H["intake channel<br/>http"] --> P["one process<br/>one q:source per channel"]
    R["intake channel<br/>rabbitmq"] --> P
    K["intake channel<br/>kafka"] --> P
    N["one channel<br/>+ messageType"] --> S1["exactly one subscribing process<br/>enforced at deploy time"]
    N -.->|"broadcast: true"| S2["every subscribing process<br/>one instance each"]

The channel never names a process; the process names the channel. That is what lets one flow be fed by three transports at once, and why a second subscriber to the same (channel, messageType) is a deploy-time error unless the channel opts into broadcast.

Concurrency admission. A channel may optionally declare maxConcurrentInstances — an admission cap on simultaneously active instances from that channel (a suspended instance still holds its slot). Absent, the channel is unbounded.

Uncorrelated answers. uncorrelated: { start: <processId> } on an inbound channel that a wait subscribes to sends a message that correlates to no live instance to that process, instead of refusing it. The process’s start events on the channel are reserved for such messages. See An answer no instance waits for. A malformed block fails to load, and sutra lint reports a start naming no process of the package, or one with no start event on the channel, both as SUTRA.CONFIG.CHANNEL.UNCORRELATED_INVALID.

Each exclusive endpoint (route, queue, subscription, consumer group, spool directory) belongs to one deployment; see One endpoint, one deployment.

Codecs — format × schema

A codec is a channel-facing named decoder: a parser (json / xml / yaml / csv / a domain wire format) plus an optional schema. The schema is the load-bearing part — it’s what yields a messageType and structural validation. A codec with no schema (a media codec, or a channel with no codec at all) still decodes, but only a catch-all <q:source> (no messageTypeValue/messageTypePattern) can subscribe to it, and sutra lint warns.

Codec names share one urn:sutra:codec:<name> namespace, and codec: in channels.yaml looks identical whichever tier the decoder came from. There are three:

TierExample bindingWhere the decoder comes fromWhat it takes to have one
Built-in formatcodec: urn:sutra:codec:jsonLinked into every engine binaryNothing — it is always there
Package-supplied codeccodec: urn:transferschemas/transfer/*.xsd inside the archive, compiled when the package deploysAuthor the XSDs — no Rust, no engine build
Extension-crate codeccodec: urn:sutra:codec:<name>A crate implementing PayloadCodec, force-linked by a composition rootOne Cargo dependency and one line in that composition root

A built-in format is a pure parser — it decodes, but carries no schema, so it lands in the catch-all-subscription case above. This distribution bundles six of them — json, xml, yaml, csv, raw-text, raw-bytes — and no domain codec at all. Bind one when the payload genuinely has no contract to check, or when that contract is enforced somewhere else.

A package-supplied codec is the zero-install path to a typed contract, and the one both example apps take. schemas/transfer/ holds the XSDs plus a codec-manifest.yaml declaring schemaKind: xsd and the formats the codec accepts; the engine compiles them at deploy time and names the codec after the folder path (urn:transfer — see Deployment packages for the exact folding rule). Nothing about it is second-class: it yields message types, structural validation, and the shape every FEEL path is checked against at load time, exactly as a codec written in Rust does. The engine cannot tell that the schema arrived in an archive rather than a crate.

An extension-crate codec is what a wire format needs when a folder of schemas cannot express it — a grammar that isn’t XML or JSON at all (a fixed-width block structure, or a delimited segment stream), or a whole profile: an envelope grammar, a mapping from wire-level message names to schemas, and versioned editions revved on the standard’s own release cadence. It implements PayloadCodec from sutra-codec-spi, inventory::submit!s a BuiltinCodec next to the impl, and claims its name in the same namespace; a distribution that wants it adds the dependency and force-links it from its own composition root. Every message standard is served this way, by proprietary extension crates built outside this repository. An engine binary that links one resolves it exactly like a built-in; see Domain neutrality and the SPI model for why none of them lives here.

What an extension codec can express — an enveloped profile, generically

The clearest illustration of what the codec SPI has to be able to carry is a market venue’s own profile of a standard rather than the bare standard itself. A generic-format codec is the easy case: one instance document validated against a schema, with the message type read straight off the document’s own root element or namespace. A profile codec decodes a family of messages for a venue that doesn’t fit that shape — every message travels inside a venue-specific envelope, wrapping a header plus a body, and the venue pins its own schema versions on its own release cadence. Worse for identification purposes, the same underlying message can sometimes back several different wrappers, so a bare schema namespace can’t tell you which one you’re looking at.

Such a codec’s message type is therefore the wrapper element’s local name, not a schema namespace slug — which lets declared_message_types() return the venue’s own closed set of wrapper names, rather than the open type set a bare-standard codec declares, so a message-type applicability check (a rules-manifest.yaml entry, a q:source messageTypeValue pin) actually fires for a profile-bound module rather than silently no-op.

Decoding validates the envelope grammar, the header, and the body against the venue-pinned base schemas the codec crate carries, out of the box. The payload view projects the body exactly as a bare-standard document would be — a compatibility guarantee: every alias, DMN input, or fixture written against the underlying message shape reads identically whether the underlying codec is the profile variant or the plain standard. Outbound can be a template-rendered passthrough — encode() returning the rendered envelope bytes verbatim rather than assembling one — so template drift against the venue’s schemas has to be caught by validating rendered output in tests, not by a runtime encode path. A second venue following the identical envelope/wrapper/edition pattern reuses the same machinery rather than re-implementing it.

None of that requires a change to the engine, the channel layer, or this repository — which is the point of the codec SPI. A deployment can go one step further still and override a codec’s own schemas: see Deployment packages for the schemaKind bundle mechanism that lets an archive map its own schema editions per wrapper.

Acknowledgement modes

ack-mode decides when the engine acknowledges an inbound relative to processing:

  • on-persist — ack as soon as the message is durably captured, before the process runs. Broker default. On HTTP this is what makes a channel asynchronous (202 Accepted with an acceptance receipt).
  • on-complete — ack only once the instance reaches a terminal state. HTTP default (classic synchronous request/reply — hold the connection, return the reply body). On a broker, this defers the ack via the engine’s DeferredAckRegistry until the instance completes or fails.

The full per-transport wiring matrix, the bounded-registry knobs, and when to pick which mode live in Acknowledgement modes — that’s the operating-chapter deep dive this summary points at.

AMQP 1.0: deliveries in flight and link credit

An AMQP 1.0 inbound channel processes several deliveries of one binding at once (since 0.8.3):

channels:
  - name: orders-in
    transport: amqp
    host: broker.example
    queue: orders
    address-type: queue
    codec: json
    ack-mode: on-persist
    max-in-flight: 16     # deliveries processed at once (1..1024); 1 = one at a time
    prefetch-count: 10    # link credit floor
  • max-in-flight (default 16) is how many intake calls run at once. Each delivery is still accepted only after its own message is persisted (or, with on-complete, its instance ends), and a duplicate is still absorbed by the inbox. On stop the engine waits for the calls in flight and settles each; on a lost connection the broker redelivers what was not settled.
  • prefetch-count sets the link credit: the receiver grants max(prefetch-count, 2 × max-in-flight) credits, 32 by default. Before 0.8.3 the property was read but not applied.
  • Order. The engine never promised inbound order for a broker source. With max-in-flight above 1, two messages for the same instance can reach it in either order. max-in-flight: 1 processes deliveries one at a time, in arrival order, as before 0.8.3.

Secrets on a channel

A channel’s credentials (broker username/password, an API key) are never literal values in channels.yaml — they’re scheme references (env:NAME, secret:…, vault:…, aws-secrets:…), resolved at channel startup through one vendor-neutral resolver SPI. Package-time validation rejects a literal secret outright.

Next

Calling a language model

A process calls a language model the same way it calls any other counterpart: with a channel-call task. The model sits behind an outbound channel with transport: llm. Whether the call is synchronous or asynchronous is the channel’s ack-mode (see Sync or async). No BPMN element is specific to models, so everything a channel call already has applies unchanged — <q:timeout>, <q:retry>, timer boundaries, error boundaries.

Sync or async

Outbound ack-modePatternWhat happens
on-completesyncThe model is called inline and the answer is merged into the instance’s variables; the flow continues. No outbox row, no park, no database write for the hop, no reply-channel, no <q:alias>.
on-persist (the outbound default)asyncThe request goes through the outbox, the instance parks, and the answer arrives as a message on reply-channel, resuming the instance by <q:alias> correlation.

Only an ack-mode you declare on the outbound channel counts; leave it out and the channel is async, exactly as before.

Sync

- name: summarize-sync
  direction: outbound
  transport: llm
  bind: "llm://summarize-sync"
  ack-mode: on-complete                 # no reply-channel
  llm:
    candidates:
      - { provider: anthropic, model: <model-id>, api-key: "secret:llm/anthropic" }
    timeout-seconds: 30
<serviceTask id="Summarize" implementation="channel:summarize-sync">
  <extensionElements>
    <q:timeout duration="PT45S"/>
  </extensionElements>
  <dataOutputAssociation>
    <assignment><from>payload.text</from><to>summary</to></assignment>
  </dataOutputAssociation>
</serviceTask>

The task sends its requestBody variable (see The request). The answer is the reply message below, and it reaches variables only through the task’s output mapping: payload.text is the answer text, payload.finish, payload.usage.outputTokens, payload.model and so on are the rest; header.x-llm-provider and header.x-llm-model carry the provider and model that answered. Anything you do not map is dropped.

Parallel branches’ sync calls run concurrently; each is bounded by its <q:timeout>.

Async

Add reply-channel and an inbound channel for the answer, and give the task a <q:alias> whose value is unique per call in flight (the next section shows the pair). A channel-call to an async channel with no <q:alias> is rejected at deploy time with SUTRA.DISPATCH.CHANNEL_CALL.ALIAS_REQUIRED.

Replying to the caller

When the process was started by (or is being resumed through) an inbound HTTP channel with ack-mode: on-complete, the caller is waiting and the flow’s <q:reply> answers it:

  • A relayed turn (a message that resumes a parked instance) answers with the <q:reply> its own pass reaches, whether that reply is continue="true" before the instance parks again or sits on the way to the end. Replies the instance passed in earlier turns are not repeated. A turn that reaches no reply answers 200 with an empty body.
  • A plain <q:reply> followed by a park answers nothing — the request is answered when the flow ends, or by a continue="true" reply.
  • An inbound HTTP channel on on-persist answers 202, with the reply as the receipt body.

The two channels (async)

channels:
  # The request side: an outbound channel the process calls.
  - name: summarize
    direction: outbound
    transport: llm
    bind: "llm://summarize"
    llm:
      reply-channel: summarize-reply        # where the answer is delivered
      candidates:                           # tried in order
        - provider: anthropic
          model: <model-id>
          api-key: secret:llm/anthropic
        - provider: openai
          model: <model-id>
          base-url: https://models.example.test/v1   # any compatible server
          api-key: env:LLM_FALLBACK_KEY
      system: "You summarize support tickets in three sentences."
      max-output-tokens: 512
      temperature: 0
      timeout-seconds: 60
      concurrency: 4
      fallback-on: [rate-limited, unavailable, timeout]
      require: [json-output]
    # ack-mode: on-persist is the outbound default

  # The reply side: an inbound channel the answer arrives on.
  - name: summarize-reply
    transport: local
    codec: json
KeyDefaultMeaning
candidatesrequiredOrdered provider + model pairs, each with optional base-url and api-key. api-key is a reference (env:, secret:, vault:, ${…}), never a literal. A candidate may also name a transcription-model, used when a live audio session is opened on it.
reply-channelrequired unless ack-mode: on-completeThe inbound channel, in the same package, the answer is delivered on. A sync channel has none.
modecallcall holds the request open; job submits and completes later (see Jobs and webhooks). Refused on ack-mode: on-complete.
systemnoneSystem prompt. A request may override it.
max-output-tokens, temperatureprovider defaultPassed through. A request may override them.
timeout-seconds60Deadline for one candidate’s call (for a job: the submit call).
concurrency4Calls in flight on this channel, per engine replica.
fallback-on[rate-limited, unavailable, timeout]Which transient failures move on to the next candidate. A rejected request or a filtered answer never does.
requirenonetools, json-output, audio-input, audio-output, audio-understanding, video-understanding or realtime: skip candidates that cannot do it (see Audio, images and video). Applies to every call on the channel.
output-voiceprovider defaultThe voice of a spoken answer. A request’s outputVoice overrides it.
voicenativenative sends audio to the model as audio; pipeline transcribes it, asks a text model, and speaks the answer (see The voice pipeline).
speechnoneThe speech provider of a voice: pipeline channel: provider, api-key, base-url, transcribe-model, speech-model, optional voice and language.
media-storenoneThe object store (sutra.object-stores.<name>, engine configuration) this channel reads media references from. Without it, a request carrying a reference is refused.
media-max-bytes20971520 (20 MiB)The largest object one media reference may name.

Providers

provider:ServiceDefault endpointNotes
openaiOpenAIhttps://api.openai.com/v1Also any compatible server via base-url. With an audio chat model (a model id containing audio), audio in and spoken answers out.
anthropicAnthropichttps://api.anthropic.com
aws-bedrockAmazon Bedrock (Converse API)none — set base-url: https://bedrock-runtime.<region>.amazonaws.comAuthenticates with a Bedrock API key; IAM request signing is not supported.
geminiGoogle Gemini, native APIhttps://generativelanguage.googleapis.comGemini’s own generateContent API: audio, images and video in, JSON answers, tools, streaming. The key travels in the x-goog-api-key header.
google-geminiGoogle Geminihttps://generativelanguage.googleapis.com/v1beta/openaiGemini’s Chat Completions-compatible endpoint: text only. Use gemini for audio or video.
mistralMistral AIhttps://api.mistral.ai/v1
deepseekDeepSeekhttps://api.deepseek.com
xaixAIhttps://api.x.ai/v1
kimiMoonshot AI (Kimi)https://api.moonshot.ai/v1
meta-llamaMeta Llama APIhttps://api.llama.com/compat/v1

Every service except anthropic, aws-bedrock and gemini speaks the Chat Completions wire format and is served by one adapter; the provider: id only picks the default endpoint. Override base-url for a regional host, a proxy, or a self-hosted server.

Which providers exist depends on how the engine binary was built (llm-openai, llm-anthropic, llm-bedrock, llm-gemini cargo features; the standard image includes all four). A channel naming a provider the binary does not include fails at load with SUTRA.LLM.PROVIDER.UNKNOWN.

Jobs and webhooks

Some providers accept a request and finish it later (background responses, batch jobs). With mode: job the channel submits the request, the provider holds the work, and the result is delivered on reply-channel when it completes — found by polling, by a webhook, or both. The flow is the async one above: send, park, resume by <q:alias>. Jobs need ack-mode: on-persist and an engine with a database.

- name: summarize
  direction: outbound
  transport: llm
  bind: "llm://summarize"
  llm:
    mode: job
    reply-channel: summarize-reply
    candidates:
      - provider: openai
        model: <model-id>
        api-key: secret:llm/openai
        completion: callback              # the provider notifies us
        callback-channel: openai-hooks
        poll-interval-seconds: 300        # safety net: a webhook can be lost or late
        job-timeout-seconds: 86400
      - provider: anthropic
        model: <model-id>
        api-key: secret:llm/anthropic
        completion: poll                  # no webhooks: poll
        poll-interval-seconds: 30

# The webhook endpoint: an inbound llm channel.
- name: openai-hooks
  transport: llm
  bind: "POST /llm/openai"                # always POST /llm/<path>, on the shared HTTP listener
  llm:
    callback-for: openai
    callback-secret: secret:llm/openai-webhook
KeyWhereDefaultMeaning
completioncandidatepollpoll or callback.
callback-channelcandidatenoneRequired with completion: callback, and only then. Names an inbound transport: llm channel in the same package whose callback-for is this candidate’s provider.
poll-interval-secondscandidate30 (poll), 300 (callback)Poll cadence; with callback it is a safety-net sweep.
job-timeout-secondscandidate86400A job not finished by then is marked failed.
callback-secretcandidate or callback channelnoneSecret reference used to verify the webhook signature. A callback is accepted if the callback channel’s secret or any referencing candidate’s secret verifies, so rotation is a two-step change. At least one is required.
callback-forcallback channelnoneRequired: the provider whose webhooks this endpoint takes.

The candidate job keys are refused on a mode: call channel. Candidates that cannot do jobs are skipped at submit, like a candidate that lacks a require: capability.

The webhook endpoint answers 404 for an unknown path, 401 for a bad signature, 400 for a body the provider adapter cannot parse, 503 when the job row cannot be claimed or the database is unavailable (so the provider retries the webhook), and 200 otherwise (also when it had nothing left to deliver — the same job is delivered once, whether the poller or the webhook finishes it first). A job that fails, is rejected, or passes job-timeout-seconds delivers nothing and logs SUTRA.LLM.CALL.FAILED; the flow’s own <q:timeout> handles the missing answer. Broker-routed callbacks are not supported.

Misconfiguration fails at load with SUTRA.LLM.CONFIG.INVALID: mode: job on an on-complete channel or without a database, no candidate that supports jobs, completion: callback on a provider without webhooks, a missing or dangling callback-channel, a provider mismatch, no secret, or a callback bind that is not POST /llm/<path> or reuses a path.

What each provider supports

provider:StreamingJobsWebhooks
openaiyesyesyes
anthropicyesyes (polled)no
geminiyesnono
google-gemini, mistral, deepseek, xai, kimi, meta-llama (Chat Completions services)yesnono
aws-bedrocknonono

Streaming to the caller

A token stream means nothing inside a flow — a variable is useful only once complete — but it matters to the caller waiting on the last step. Two switches turn it on:

- name: ask
  transport: http
  bind: "POST /ask"
  ack-mode: on-complete
  reply-stream: sse            # inbound HTTP channel: answer as server-sent events
<serviceTask id="Answer" implementation="channel:summarize-sync" q:stream="true"> ... </serviceTask>

The call must go to a sync (ack-mode: on-complete) transport: llm channel, and the provider must support streaming. The response becomes text/event-stream:

event: chunk
data: Hel

event: chunk
data: lo

event: reply
data: {"summary":"Hello"}
EventDataMeaning
chunka piece of the answer textRepeats. A multi-line piece uses one data: line per line.
resetemptyA candidate failed mid-answer and the next one started over: discard the chunks received so far.
replythe flow’s reply bodyThe flow finished; the stream then closes.
errorthe diagnostic code only (never its message)The flow failed; the stream then closes.

Chunks go to the caller only. The flow continues with the whole accumulated answer in variables and audit, exactly as for a non-streamed call. If the flow streams nothing, or the channel has no reply-stream, the request is answered as an ordinary response and q:stream is ignored.

Deploy-time errors: SUTRA.CONFIG.STREAM.TARGET_NOT_SYNC_LLM for q:stream="true" on anything but a channel-call to a sync llm channel, and SUTRA.CONFIG.STREAM.REPLY_STREAM_UNSUPPORTED for reply-stream: sse on anything but an inbound http channel with ack-mode: on-complete.

The request

The channel-call task’s input is a JSON object:

FieldMeaning
promptA single user message.
messagesA conversation: [{ "role": "user" | "assistant" | "tool", "content": "…", "toolCallId": "…" }]. Use prompt, messages, or both (prompt is appended).
systemOverrides the channel’s system.
jsontrue asks for a JSON object as the answer. The reply carries it parsed under json, or jsonError saying why it did not parse. The answer may be bare JSON or one fenced block (```json), with or without prose around it. See How JSON is asked for.
maxOutputTokens, temperatureOverride the channel’s values.
tools[{ "name", "description", "parameters": <JSON Schema> }] the model may call.
messages[].audioAudio on a user turn: { "data": <base64>, "format": <media type> }. See Audio, images and video.
messages[].videoVideo on a user turn: a clip { "data", "format": "video/…" }, or sampled frames { "frames": [{ "data", "format": "image/…", "atMs" }] }.
messages[].imageOne image on a user turn, { "data", "format": "image/…" }, read as a one-frame video (the candidate needs video-understanding).
ref, sha256Any audio, video, frame or image part may give { "ref": <object URI>, "sha256": <hex> } instead of data: a media reference.
audioOutputAsk for a spoken answer in this format, for example "audio/L16;rate=8000".
outputVoiceThe spoken answer’s voice; overrides the channel’s output-voice.

Build the prompt in the process — a template task, a FEEL string, or a <q:param> — so it is versioned with the flow that uses it.

How JSON is asked for

Every provider declares json-output, but not every model has a JSON mode, so each adapter asks in the way its model accepts. Either way the transport parses the answer into json.

Provider / modelHow json: true is sent
openai and the Chat Completions services, text modelsresponse_format: {type: json_object} (text.format for a job). The JSON modes refuse a request that never says “JSON”, so when neither system nor any message mentions it, Respond with a single JSON object and nothing else. is added to the system text.
openai audio chat models (a model id containing audio: gpt-audio…, gpt-4o-audio-preview)In words only: the instruction above in the system text, no response_format. These models refuse the JSON mode with HTTP 400 (“…response_format of type json_object is not supported with this model”). They often answer with the JSON in a fenced block inside prose; the transport finds and parses it.
gemini (native)responseMimeType: application/json; with tools, in words.
anthropic, aws-bedrockIn words (neither has a JSON mode).

So a channel with require: [audio-understanding, json-output] still selects an OpenAI audio model. A model asked in words can still answer prose with no JSON in it: the reply then carries jsonError, and a flow that needs the object should check for it.

The reply

The answer is this JSON message — delivered on reply-channel for an async call, the response itself for a sync one:

{
  "text": "…",
  "toolCalls": [{ "id": "…", "name": "…", "arguments": { } }],
  "finish": "stop | length | tool-calls | filtered | …",
  "usage": { "inputTokens": 812, "outputTokens": 96 },
  "provider": "anthropic",
  "model": "<model-id>",
  "providerRequestId": "…",
  "request": { }
}

request echoes the request body, so an async task’s <q:alias> can correlate on a field the process put there (for example body.request.ticketId). The headers of the request ride along too, plus x-llm-provider and x-llm-model.

A finish of filtered means the provider withheld the answer. It is a result, not a failure: route on it with a gateway rather than retrying.

Structured answers. A call that asked for JSON (json: true) adds json: the answer parsed into an object or array (a fenced code block is unwrapped), so the flow reads its fields directly, as payload.json.category, instead of parsing text. When the answer does not parse, json is absent and jsonError says why: either the answer was cut off at max-output-tokens (finish: "length"), or it is not a JSON object or array. Reasoning models such as gemini-2.5-flash think before they answer, and the thinking counts against max-output-tokens, so give a JSON channel on such a model room (the examples use 2048).

Audio. A call involving audio adds audio ({ "data": <base64>, "format": <the audioOutput asked for> }), transcript (what the spoken answer says) and audioPreserved (true when the model heard the audio itself, false when it went through the pipeline). In the echoed request, every audio or video part keeps its format and has its data replaced by bytes (the decoded size), so the reply never carries the media back.

Audio, images and video

Preview. Video understanding (clips, frames, images) and live audio sessions are preview in 0.6: they have been run against the real Gemini and OpenAI APIs as well as against in-process stand-ins of them, and their shapes may still change. Audio on a turn-based call (audio, audioOutput, voice: pipeline) is not preview.

Audio stays audio. A user turn’s audio goes to the model as audio, so an audio-native model hears the voice itself (tone, stress, hesitation), not a transcript of it. Speech-to-text in front of a text model happens only when a channel asks for it with voice: pipeline.

- name: voice-model
  direction: outbound
  transport: llm
  bind: "llm://voice-model"
  ack-mode: on-complete
  llm:
    candidates:
      - provider: openai
        model: <audio-model-id>          # an audio chat model: its id contains `audio`
        api-key: env:OPENAI_API_KEY
    require: [audio-understanding]       # skip any candidate that would flatten the voice
    output-voice: alloy

The request names the format it has and the format it wants back:

{
  "messages": [{ "role": "user", "content": "Answer the caller.",
                 "audio": { "data": "<base64>", "format": "audio/PCMU" } }],
  "audioOutput": "audio/L16;rate=8000"
}

Formats are media types. audio/wav; audio/L16;rate=<hz>[;channels=<n>] (also written audio/pcm;rate=<hz>: signed 16-bit little-endian, mono unless stated); audio/PCMU and audio/PCMA (G.711 µ-law and A-law, 8 kHz); and the compressed audio/mpeg, audio/flac, audio/opus, audio/ogg, audio/aac, audio/webm and audio/mp4. The transport converts between the call’s formats and the provider’s in-process: WAV wrap and unwrap, G.711, channel mixing and linear resampling. A compressed format is passed through, never converted. If the provider does not take it as it is, the call fails with SUTRA.LLM.AUDIO.UNSUPPORTED in the diagnostic.

Audio is never dropped. A candidate that cannot take what the call carries (audio input, a spoken answer, video) is skipped, exactly like a candidate without tools. If none can, the call fails permanently. A mode: job channel carries no audio or video and refuses such a call.

require: valueA candidate must…
audio-inputaccept audio on a message
audio-outputanswer in audio
audio-understandinghear the voice itself (emotion, tone), not a transcript
video-understandingread video: a clip, sampled frames or an image
realtimeopen a live audio session
provider:Audio inSpoken answerImages and video
openai, with an audio chat model (id containing audio)WAV, MP3WAV, MP3, FLAC, Opus, AACno
geminiWAV, MP3, FLAC, AAC, Ogg, Opus, M4A, WebMnoclips, frames, images
every other providernonono

Images and video (preview). A user turn may carry video, either a clip ({ "data", "format": "video/mp4" }) or a run of sampled frames ({ "frames": [{ "data", "format": "image/png", "atMs": 0 }, …] }), or one image ({ "data", "format": "image/png" }), which is read as a one-frame video. Only gemini reads them today. Combined with json: true, the flow gets the answer parsed under json:

- name: vision-model
  direction: outbound
  transport: llm
  bind: "llm://vision-model"
  ack-mode: on-complete
  llm:
    candidates:
      - provider: gemini
        model: <gemini-model-id>
        api-key: env:GEMINI_API_KEY
    require: [video-understanding, json-output]

The voice pipeline

For a model without audio, or for cost, a channel can transcribe, answer in text and speak the answer, explicitly:

  llm:
    voice: pipeline
    speech:
      provider: openai
      api-key: env:OPENAI_API_KEY
      transcribe-model: <transcription-model-id>
      speech-model: <speech-model-id>
      voice: alloy                  # optional; a request's outputVoice overrides it
      language: en                  # optional transcription hint
    candidates:
      - provider: anthropic
        model: <model-id>
        api-key: env:ANTHROPIC_API_KEY

Each audio turn is transcribed by speech and becomes text, appended to the turn’s content. The candidates answer as text. With audioOutput, speech speaks the answer, and the reply’s transcript is the answer text. The reply then says audioPreserved: false: the voice never reached the model, so a flow that needs it (an emotion analysis, say) can refuse the answer. A native answer says audioPreserved: true. A text-only call on a pipeline channel is not touched.

voice is native (the default) or pipeline. llm.speech is required with pipeline and refused without it. The speech provider must do both transcription and speech synthesis (today, openai). A pipeline channel cannot require an audio capability and cannot be mode: job. Each of these is a SUTRA.LLM.CONFIG.INVALID at load.

Media references

Any audio, video (the clip, or each frame) or image part may name an object in an object store, { "ref": "s3://…", "sha256": "<hex>" }, instead of inlining data. The channel’s media-store names the store it may read. The transport reads the object, checks its SHA-256 and size, and sends it as if it had been inline; variables, snapshots and audit only ever hold the reference. Recordings and media references covers the object store, the refusals and a walkthrough.

Try it: audio, the pipeline and vision

This runs three flows of the examples/ai-advanced package against the stub model server:

FlowChannelModel channelCandidate
shared/bpmn/voice.bpmnPOST /channels/voicevoice-modelopenai, audio model, require: [audio-understanding]
shared/bpmn/voice-pipeline.bpmnPOST /channels/voice-pipelinevoice-pipeline-modelvoice: pipeline: openai speech, anthropic text model
shared/bpmn/vision.bpmnPOST /channels/visionvision-modelgemini, require: [video-understanding, json-output]

Prerequisites: Before you try a walkthrough. No key is needed. In variants/live/channels.yaml the same channels name real models, with their keys as env:OPENAI_API_KEY, env:ANTHROPIC_API_KEY and env:GEMINI_API_KEY, resolved from the engine’s environment.

The vision flow builds its request in one FEEL data task. It sends either the image or the frames, and asks for JSON:

<bpmn:assignment>
  <bpmn:from>{messages: [if payload.image != null then {role: "user", content: "Describe the image as one JSON object: scene (one sentence), objects (a list of short names) and framesSeen (1).", image: payload.image} else {role: "user", content: "These are the frames of one short clip, in order. Describe the whole clip as ONE JSON object, not one per frame: scene (one sentence), objects (a list of short names) and framesSeen (how many frames you received).", video: payload.video}], json: true}</bpmn:from>
  <bpmn:to>requestBody</bpmn:to>
</bpmn:assignment>

It then reads the parsed answer’s fields directly:

<bpmn:from>{requestId: payload.requestId, scene: answer.json.scene, objects: answer.json.objects, framesSeen: answer.json.framesSeen, provider: answer.provider}</bpmn:from>

1. Package the stub variant and start the stack (engine, PostgreSQL, the stub):

C=examples/ai-advanced/deploy/compose.yaml
sutra package examples/ai-advanced/variants/stub --overlay examples/ai-advanced/shared \
    --out examples/ai-advanced/deploy/deployments
docker compose -f $C up -d --wait
PORT=$(docker compose -f $C port engine 8080 | cut -d: -f2)
H=(-H 'Content-Type: application/json' -H 'X-Api-Key: dev-only-advanced-key')
until curl -sf "http://127.0.0.1:$PORT/sutra/deployments" | grep -q '"phase":"Active"'; do sleep 1; done

sutra package prints one SUTRA.CHANNEL.NO_SCHEMA warning per schema-less JSON channel, and then:

packaged examples/ai-advanced/deploy/deployments/stub.sutra (deploymentId dep-163c73f2de054d4963d7812b)

2. Audio in, a spoken answer out. The sample recording is a quarter-second WAV. The request asks for 16 kHz linear PCM back. The helper shortens the returned audio for display:

AUDIO=$(base64 -w0 examples/ai-advanced/sample/question.wav)
short() { python3 -c 'import json,sys; r=json.load(sys.stdin); r["audio"]["data"]=r["audio"]["data"][:12]+"..."; print(json.dumps(r, indent=2))'; }
curl -s -X POST "http://127.0.0.1:$PORT/channels/voice" "${H[@]}" \
    -d "{\"requestId\":\"Q-1\",\"audio\":{\"data\":\"$AUDIO\",\"format\":\"audio/wav\"},\"audioOutput\":\"audio/L16;rate=16000\"}" | short
{
  "audio": {
    "data": "oA+gD6APoA+g...",
    "format": "audio/L16;rate=16000"
  },
  "audioPreserved": true,
  "provider": "openai",
  "requestId": "Q-1",
  "text": "STUB SPOKEN ANSWER: your appointment is confirmed for Tuesday.",
  "transcript": "STUB SPOKEN ANSWER: your appointment is confirmed for Tuesday."
}

The stub spoke a 24 kHz WAV, and the transport unwrapped and resampled it to the audio/L16;rate=16000 the request asked for.

3. The same question through the pipeline, asking for G.711 µ-law back:

curl -s -X POST "http://127.0.0.1:$PORT/channels/voice-pipeline" "${H[@]}" \
    -d "{\"requestId\":\"Q-2\",\"audio\":{\"data\":\"$AUDIO\",\"format\":\"audio/wav\"},\"audioOutput\":\"audio/PCMU\"}" | short
{
  "audio": {
    "data": "r6+vr6+vr6+v...",
    "format": "audio/PCMU"
  },
  "audioPreserved": false,
  "provider": "anthropic",
  "requestId": "Q-2",
  "text": "STUB SUMMARY: the customer reports a problem and asks for it to be looked at.",
  "transcript": "STUB SUMMARY: the customer reports a problem and asks for it to be looked at."
}

The text model (Anthropic) answered, and audioPreserved: false says it never heard the voice.

4. One image to the native Gemini provider:

IMG=$(base64 -w0 examples/ai-advanced/sample/frame-1.png)
curl -s -X POST "http://127.0.0.1:$PORT/channels/vision" "${H[@]}" \
    -d "{\"requestId\":\"V-1\",\"image\":{\"data\":\"$IMG\",\"format\":\"image/png\"}}"; echo
{"framesSeen":1,"objects":["bicycle","door"],"provider":"gemini","requestId":"V-1","scene":"STUB SCENE: a bicycle leans against a blue door."}

5. Two sampled video frames:

F2=$(base64 -w0 examples/ai-advanced/sample/frame-2.png)
curl -s -X POST "http://127.0.0.1:$PORT/channels/vision" "${H[@]}" \
    -d "{\"requestId\":\"V-2\",\"video\":{\"frames\":[{\"data\":\"$IMG\",\"format\":\"image/png\",\"atMs\":0},{\"data\":\"$F2\",\"format\":\"image/png\",\"atMs\":500}]}}"; echo
{"framesSeen":2,"objects":["bicycle","door"],"provider":"gemini","requestId":"V-2","scene":"STUB SCENE: a bicycle leans against a blue door."}

scene, objects and framesSeen came from answer.json: the flow never parsed text.

6. What the stub was asked, by route:

docker compose -f $C exec model-stub wget -qO- http://127.0.0.1:8000/stats; echo
{"chat": 1, "chat-audio-in": 1, "chat-audio-out": 1, "transcriptions": 1, "messages": 1, "speech": 1, "gemini-generate": 2}

The native call is one audio chat completion. The pipeline is one transcription, one Anthropic message and one speech synthesis. The two vision calls are two Gemini generateContent requests.

7. Clean up:

docker compose -p sutra-ai-advanced down -v

Live audio sessions (preview)

A provider adapter can also open a live audio session (AudioSession): audio frames in, transcript deltas and the model’s audio and text out. The openai adapter does it over the Realtime WebSocket API for a model id containing realtime, and the gemini adapter does it over the Live API for a model id containing live. The gemini adapter never puts the API key in a URL. It mints a single-use ephemeral token with the key (sent in the x-goog-api-key header) and opens the session with that token in the handshake’s Authorization header.

A process never opens a session. A media transport does, such as the telephony transport of sutra-voice, and it opens it through an llm channel of its own deployment, so it gets the same routing as a process’s call (ordered candidates, fallback-on, require:, key references, concurrency, and the sutra.llm.call span, with session = true). The media goes from that transport to the provider directly and never passes through a process. Such a channel answers inline, so it declares ack-mode: on-complete and no reply channel:

  - name: live-transcription
    direction: outbound
    transport: llm
    bind: "llm://live-transcription"
    ack-mode: on-complete
    llm:
      require: [realtime]            # only candidates with live audio sessions
      concurrency: 16                # an open session holds one permit for its life
      candidates:
        - provider: openai
          model: <realtime-model-id>
          transcription-model: <transcription-model-id>
          api-key: env:OPENAI_API_KEY
        - provider: gemini           # the fallback
          model: <live-model-id>
          api-key: env:GEMINI_API_KEY

A session opens on the first candidate that has realtime and what require: names. A failure listed in fallback-on moves it to the next candidate, exactly as for a call. A transport may also ask for a transcription-only session: on openai that is a Realtime transcription session, which reports the start and end of each stretch of speech as offsets into the audio. Gemini Live has no listen-only session, so it is asked to stay silent and anything it says is ignored. A mode: job channel cannot serve a transport.

Turn detection. A session either lets the provider’s voice-activity detection decide where the caller’s turn ends (the default), or the transport marks each turn itself (manual turns). With the gemini adapter, provider detection runs at a low end-of-speech sensitivity with 800 ms of silence, so a pause inside a sentence does not end the turn. Provider detection is tuned for real speech: in our checks, gemini-3.8-live took only the first word of a synthetic (text-to- speech) question for speech and answered that, while it handled a recorded human voice, and gemini-2.5-flash-native-audio-latest handled both. Manual turns handled every case. If your callers’ audio is unusual (synthetic prompts, heavy noise), test with your own recordings or use manual turns.

There is no walkthrough here, because a live session needs a media transport to open it and the engine in this repository ships none. The sutra-voice repository runs it end to end. examples/ai-advanced/live-check.sh checks the adapters themselves against the real APIs.

Failures

What happenedSync (on-complete)Async (on-persist)
Every candidate failed transiently, or the call timed outWith <q:retry>: the call is re-issued inline with the policy’s backoff, all within the <q:timeout> budget; once spent, the failure’s code (SUTRA.LLM.CALL.FAILED, or SUTRA.DISPATCH.CHANNEL_CALL.TIMEOUT) is raised as a BPMN error on the task. Without it: a timeout fires the task’s #timeout boundary, which raises SUTRA.DISPATCH.CHANNEL_CALL.TIMEOUT; any other failure raises SUTRA.LLM.CALL.FAILED. Uncaught, the instance fails.The delivery is retried with the outbox’s backoff. If it never succeeds, the task’s <q:timeout> fires, or the outbox attempt ceiling poisons it — both are retryable failures under <q:retry>.
The request was rejected (bad key, unknown model, malformed request)A permanent failure: no retry. SUTRA.LLM.CALL.FAILED is raised as a BPMN error on the task; a boundary on that code, a boundary with no code, or an error event sub-process catches it. Uncaught, the instance fails.The delivery is poisoned and stays redrivable; the provider’s code does not reach the task. The task’s <q:timeout> fires (SUTRA.DISPATCH.CHANNEL_CALL.TIMEOUT). With sutra.outbox.retry.max-attempts set, the row turns terminal first and the task gets SUTRA.OUTBOUND.DELIVERY_ATTEMPTS_EXHAUSTED: through <q:retry> when it has one, then as a BPMN error when something catches it.

A sync retry never parks a durable backoff; a flow that needs one should use the async channel.

Catching a failed call

A sync call’s permanent failure is a BPMN error on its task, so a flow that can carry on without the answer says so with a boundary:

<bpmn:error id="callFailed" errorCode="SUTRA.LLM.CALL.FAILED"/>
...
<bpmn:boundaryEvent id="ModelRefused" attachedToRef="AskModel">
  <bpmn:outgoing>F5</bpmn:outgoing>
  <bpmn:errorEventDefinition errorRef="callFailed"/>
</bpmn:boundaryEvent>

An empty <bpmn:errorEventDefinition/> catches any failure. Boundaries are tried in document order, so declare one on a specific code before a catch-all. With no boundary on the task, a process-level error event sub-process is next; a channel call cannot sit inside a sub-process, so there is no scope between them. A <q:reply> inside an event sub-process does not answer a waiting HTTP caller, so when a caller waits, use a boundary and reply on its route. With nothing to catch it, the instance fails as it always did.

The catching path reads callFailure, a context the engine sets when the failure is raised:

FieldWhat it holds
callFailure.codeThe code the boundary matched.
callFailure.messageThe reason: for a model call the channel, provider and model, the HTTP status, the provider’s code and its message cut to 300 characters, with credentials redacted. Never the prompt or the answer.
callFailure.nodeThe id of the task whose call failed.

An answer the call’s own contract refuses (an output mapping that cannot be evaluated) is caught the same way, and its message says only that the answer was refused, since the diagnostic may quote it. A <q:timeout> that fires with no modelled route sets callFailure too. The variable stays set after the catch, like any other, until the next failure replaces it. A flow’s own callFailure variable is overwritten.

A failure under <q:retry> is caught only when the budget is spent, under the failure’s code, not SUTRA.RUNTIME.RETRY.EXHAUSTED (which stays the code of the failed instance when nothing catches it). A loop from the boundary back to the call starts on a fresh budget.

The same holds for any synchronous channel, not only llm: the HTTP transport’s code is SUTRA.OUTBOUND.SEND.FAILED.

Try it: a model that refuses the request

This runs four flows of the examples/ai-advanced stub variant against a stub model that answers every call with HTTP 400:

FlowChannelThe callWhat catches it
variants/stub/bpmn/refused-guarded.bpmnPOST /channels/refused-guardedWith the HTTP caller waitingA boundary on SUTRA.LLM.CALL.FAILED
variants/stub/bpmn/refused-unguarded.bpmnPOST /channels/refused-unguardedWith the HTTP caller waitingNothing
variants/stub/bpmn/refused-later-guarded.bpmnPOST /channels/refused-later-guardedOn a stored instance, after a timerA boundary with no code
variants/stub/bpmn/refused-later-unguarded.bpmnPOST /channels/refused-later-unguardedOn a stored instance, after a timerNothing

Prerequisites: Before you try a walkthrough. No key is needed.

1. Package the stub variant and start the stack:

C=examples/ai-advanced/deploy/compose.yaml
sutra package examples/ai-advanced/variants/stub --overlay examples/ai-advanced/shared \
    --out examples/ai-advanced/deploy/deployments
docker compose -f $C up -d --wait
PORT=$(docker compose -f $C port engine 8080 | cut -d: -f2)
H=(-H 'Content-Type: application/json' -H 'X-Api-Key: dev-only-advanced-key')
until curl -sf "http://127.0.0.1:$PORT/sutra/deployments" | grep -q '"phase":"Active"'; do sleep 1; done

2. Caught, with the caller waiting. The boundary routes to a task that answers from callFailure:

<bpmn:from>{ticketId: payload.ticketId, recovered: true, code: callFailure.code, reason: callFailure.message}</bpmn:from>
curl -s -X POST "http://127.0.0.1:$PORT/channels/refused-guarded" "${H[@]}" \
    -d '{"ticketId":"T-1","description":"My parcel never arrived."}' | python3 -m json.tool

The caller gets the recovery path’s reply, with HTTP 200:

{
    "code": "SUTRA.LLM.CALL.FAILED",
    "reason": "llm channel 'refusing-model': every candidate failed; last (openai/refuses-everything): HTTP 400 (model_refuses): This model refuses the request.",
    "recovered": true,
    "ticketId": "T-1"
}

3. Uncaught, with the caller waiting. The same flow without the boundary:

curl -s -w '\nHTTP %{http_code}\n' -X POST "http://127.0.0.1:$PORT/channels/refused-unguarded" "${H[@]}" \
    -d '{"ticketId":"T-2","description":"My parcel never arrived."}'

The caller gets the failure:

{"type":"urn:bpm:diag:SUTRA.LLM.CALL.FAILED","title":"SUTRA.LLM.CALL.FAILED","status":500,"detail":"llm channel 'refusing-model': every candidate failed; last (openai/refuses-everything): HTTP 400 (model_refuses): This model refuses the request.","code":"SUTRA.LLM.CALL.FAILED","attributes":{"incident":"SUTRA.INBOUND.NON_IDEMPOTENT_FAILURE"}}
HTTP 500

4. On a stored instance. No caller waits here, which is the case that used to lose work: a timer stores the instance, then the call runs. The guarded flow’s boundary hands the reason to the stub’s sink and completes; the unguarded flow fails durably:

for flow in refused-later-guarded refused-later-unguarded; do
  curl -s -o /dev/null -w "$flow: HTTP %{http_code}\n" -X POST "http://127.0.0.1:$PORT/channels/$flow" "${H[@]}" \
      -d '{"ticketId":"T-3","description":"My parcel never arrived."}'
done
sleep 12
docker compose -f $C exec model-stub wget -qO- http://127.0.0.1:8000/sink; echo
curl -s "http://127.0.0.1:$PORT/sutra/instances?status=COMPLETED" | grep -o '"status":"[A-Z]*"'
curl -s "http://127.0.0.1:$PORT/sutra/instances?status=FAILED" | grep -o '"status":"[A-Z]*"'
docker compose -f $C logs engine | grep INSTANCE_FAILED | grep -o '"process":"[^"]*","cause_code":"[^"]*"'
refused-later-guarded: HTTP 202
refused-later-unguarded: HTTP 202
["{\"code\":\"SUTRA.LLM.CALL.FAILED\",\"reason\":\"llm channel 'refusing-model': every candidate failed; last (openai/refuses-everything): HTTP 400 (model_refuses): This model refuses the request.\",\"ticketId\":\"T-3\"}"]
"status":"COMPLETED"
"status":"FAILED"
"process":"refused-later-unguarded","cause_code":"SUTRA.LLM.CALL.FAILED"

5. Clean up:

docker compose -p sutra-ai-advanced down -v

Finding the reason for a failed call

The provider’s reason for a failure is logged at WARN, for sync and async calls alike, under the code SUTRA.LLM.CALL.FAILED: the same code the engine’s instance-failed line names as its cause_code, so one search for that code finds both. Two lines are written:

  • llm candidate failed: <reason>, once per candidate tried, with channel, provider, model, candidate (its index), class (rate-limited, unavailable, timeout or permanent), http_status, provider_code and fall_back;
  • llm call failed: <diagnostic>, once for the call, with the same fields plus trace_id (from the call’s traceparent) and instance (when the engine stamped the instance id on the call, as it does for an audited process; - otherwise).

The reason is the provider’s HTTP status, its error code (code, status or type of its error object) and its message, on one line and cut to 300 characters, for example HTTP 400 (invalid_request_error): Invalid parameter: 'response_format' of type 'json_object' is not supported with this model. Every URL in it has its query values and credentials replaced by <redacted>, and the candidate’s key is replaced wherever a provider echoes it. The same reason is in the failure diagnostic, which the engine stores with a failed instance (and with a dead-lettered start), with http_status and provider_code as attributes. No admin API is needed to read it.

Duplicate calls. Async delivery is at-least-once: a crash after the model answered but before the delivery was recorded calls the model again; inbox deduplication absorbs the duplicate reply, but the call is paid for twice (a job channel likewise risks a duplicate provider job if the submit’s record fails). A sync call lives only in memory: a crash mid-call loses it, and the client’s retry re-runs the flow from the start, calling the model again. Sync calls made by a timer or scheduled start run inline too; only a broadcast fan-out degrades to the async path, with a SUTRA.CHANNEL.SYNC_CALL_DEGRADED warning.

Tools and agent loops

The transport never runs tools. When finish is tool-calls, route the token to tasks that run the requested tools (other channels), append their results to messages as tool turns, and loop back to the model task. Bound the loop with a counter and a non-interrupting timer boundary.

What is not logged

Prompts and answers are never written to spans, logs or diagnostics. Each call opens a sutra.llm.call span carrying the channel, provider, model, finish reason and token counts. A failure’s log lines carry the provider’s error message, never the request, and keys are redacted from it (see Finding the reason for a failed call).

Examples

Runnable packages in the source tree. The two ai- examples keep their flows once under shared/ and one channels.yaml per provider under variants/, composed by sutra package --overlay:

ExampleVariantsFlows
examples/ai-summarizestub-openai, stub-anthropic, openai, anthropic, geminisync, streamed (SSE), async
examples/ai-advancedstub, livecross-provider fallback (Anthropic first, OpenAI second), JSON mode, a tool call, an OpenAI background job (polled), an OpenAI background job (webhook), an Anthropic Message Batch, audio in and out, audio in with a parsed JSON answer out (an OpenAI audio model), the voice pipeline, images and video frames to native Gemini
examples/media-recordings—a stored recording read by media reference from an object store, audited by reference, deleted on erasure and retention (walkthrough)
sutra package examples/ai-summarize/variants/anthropic --overlay examples/ai-summarize/shared \
    --out examples/ai-summarize/deploy/deployments
sutra package examples/ai-advanced/variants/stub --overlay examples/ai-advanced/shared \
    --out examples/ai-advanced/deploy/deployments

They share a stub model server (examples/ai-summarize/stub/model-stub.py) that speaks the OpenAI, Anthropic and native Gemini wire formats with fixed answers, including streaming, tools, JSON mode, audio, speech, transcription, background jobs, batches, signed webhooks and a switchable outage for fallback.

  • CI (tier 2, no keys): the stub-openai, stub-anthropic and ai-advanced stub variants, composed through the overlay path, run against the stub on a real engine and PostgreSQL — every flow above except vision (exercised by the walkthrough in this chapter), with the replies, the SSE frames, the sink deliveries and the instance_state and llm_job rows checked. Every variant, the real-provider ones included, is also linted in tier 1.
  • Live (your keys): examples/ai-advanced/live-check.sh runs the openai, anthropic and gemini variants and the ai-advanced live variant against the real APIs, including a fallback where both the failing Anthropic candidate and the answering OpenAI one are real, and — with a tunnel and OPENAI_WEBHOOK_SECRET — a webhook-completed job.

The q: namespace

Standard BPMN 2.0 has no opinion on channels, message types, correlation, or replies. Sutra adds a small, layered set of extension elements — the q: namespace, urn:sutra:q:1.0 — that live entirely inside <bpmn:extensionElements>, so a Sutra process is still a valid, portable BPMN 2.0 diagram; the q: attributes just tell the engine how to wire it to the outside world. The authoritative shape is xsd/q.xsd; the engine’s parser validates every <q:*> element against it, and the same XSD drives the sutra-modeler-plugin property panels.

<bpmn:definitions xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL"
                  xmlns:q="urn:sutra:q:1.0" ...>

At a glance

ElementAttaches toWhat it declares
q:sourcestart events, wait-capable nodesThe inbound trigger: channel, message type, ack mode, dedup, data class.
q:validators / q:redactorsq:sourceThe validation and redaction chains over the decoded payload.
q:aliaswait-capable nodesA correlation key derived by FEEL — how a later inbound finds the parked instance.
q:replytasks, end eventsAn outbound reply on the inbound’s own channel; continue="true" is respond-and-continue (reply now, keep executing detached); status (0.8.0) sets the HTTP status of a synchronous answer.
q:sendthrow events, send tasksAn unsolicited outbound message to a channel destination.
q:headerq:send / q:replyAn author-declared outbound header; on a synchronous q:reply, an HTTP response header (0.8.0).
q:decodedata task (bpmn:serviceTask with no @implementation)Parses text the flow holds, such as a message kept in a typed store column, with a registered codec, and binds the result (0.6.19).
q:parambpmn:serviceTaskA scoped, per-invocation input to a registered task or template.
q:retryregistered-task and channel-call bpmn:serviceTaskPer-task retry policy — attempts, backoff, non-retryable codes.
q:timeoutchannel-call bpmn:serviceTaskSynthesizes a timer boundary on the call.
q:stream (attribute)channel-call bpmn:serviceTaskq:stream="true" streams a synchronous model answer to the original caller. Default false.
q:storebpmn:dataStoreReferenceBinds the reference to a durable key in a declared data store; reads, writes, deletes (0.7.0) and key queries.
q:dispatch / q:casebpmn:processContent-based dispatch to called elements.
q:variables / q:variablebpmn:processDeclared process variables — type or schema, transient / sensitive / source.
q:onValidationbpmn:processThe structural-failure policy (route / reject / error).
q:processbpmn:processThe retry-safety (idempotency) assertion.
q:auditbpmn:processAudit capture level (none / metadata / payload / media) and data-class tagging.
q:coveragebpmn:processAn opt-in tracked compliance path.
flowchart TD
    PROC["on the process itself:<br/>q:variables · q:onValidation · q:process<br/>q:audit · q:coverage · q:dispatch"]

    subgraph flow["…and on the nodes inside it"]
        S(["start event"]) --> T["serviceTask"]
        T --> W["wait node"]
        W --> X(["throw · send"])
        DS[("dataStoreReference")]
    end

    S -.- A["q:source<br/>q:validators"]
    T -.- B["q:param · q:retry<br/>q:timeout · q:reply"]
    W -.- C["q:source · q:alias"]
    X -.- D["q:send · q:header"]
    DS -.- F["q:store"]

Nothing in the vocabulary is a new node type: every element hangs off a standard BPMN element’s extension elements, which is why a process carrying all of it is still a valid, portable BPMN 2.0 diagram.

q:source — the inbound trigger

Every message-consuming node — a start event or a wait-state catch — declares exactly one q:source. It names the channel, the message type it accepts, and the variable the decoded payload lands in:

<bpmn:startEvent id="Start">
  <bpmn:extensionElements>
    <q:source channel="transfer-request" messageTypeValue="TransferRequest"/>
  </bpmn:extensionElements>
</bpmn:startEvent>
  • channel (required) — the channels.yaml channel name.
  • messageTypeValue / messageTypePattern — subscribe to one exact type, or a family via regex, matched against the codec’s decoded message type. Neither set = accept anything the channel’s codec yields.
  • name (default payload) — the process-variable name the decoded body is projected under (payload.fromId, payload.body.CdtTrfTxInf... for a structured codec).
  • ack (default on-persist) — see Acknowledgement modes.
  • dedupKey — an expression identifying a redelivered duplicate (e.g. header.X-Request-Id), distinct from the process-level retry-safety assertion below. dedupKey accepts header.<name>, ce.id, body.<path> and amqp.message-id; any other form is refused at lint, package and deploy with SUTRA.PARSE.Q_SOURCE.DEDUP_KEY_UNRECOGNISED. It dedupes on a start event and on a wait.
  • dataClass (default none) — pii / pci / phi / financial; drives redaction policy.
  • filterExpression — a FEEL predicate over the message attributes the codec reported. It narrows whichever of the message-type selectors above applies rather than replacing one, so “exact beats pattern beats accept-anything” is unchanged.

Several waits on one channel

Since 0.6.10. A message on a channel that a WAIT subscribes to (a relay) chooses among the waits by the same rules a start event uses: an exact messageTypeValue beats a messageTypePattern, which beats a wait that declares no type, and filterExpression narrows within a tier. A wait pinned to another message type never takes the message. Among the waits that accept it, the one the correlated instance is actually parked at is resumed, the most specific first. So two waits on one channel can be told apart by message type alone:

<bpmn:receiveTask id="WaitAnswer">
  <bpmn:extensionElements>
    <q:source channel="answers-in" name="answer" messageTypeValue="Answer"/>
    <q:alias name="answerRef" expression="answer.ref" unique="true"/>
  </bpmn:extensionElements>
</bpmn:receiveTask>
<bpmn:receiveTask id="WaitCancel">
  <bpmn:extensionElements>
    <q:source channel="answers-in" name="cancel" messageTypeValue="Cancel"/>
    <q:alias name="cancelRef" expression="cancel.ref" unique="true"/>
  </bpmn:extensionElements>
</bpmn:receiveTask>

Before 0.6.10 a relay ignored the wait’s message type, and such waits needed a filterExpression on messageType to stay apart. That still works.

Selecting on attributes, not just a message type

One string cannot always say which process should handle a message. A message standard may use the same message type in more than one flavour — the same type arriving one way is a request to act on, and arriving the other way is an acknowledgement of something you sent — and the flavour, not the type, decides the handler.

A codec reports those facts as attributes alongside the message type, and a start event selects on the pair:

<q:source channel="rail" messageTypeValue="CreditTransfer"
          filterExpression="attr.direction = &quot;incoming&quot;"/>

Two start events may now pin the same message type and be told apart by an attribute, where before that was an ambiguous handler. The predicate sees the attributes and messageType — not headers, not the payload: admitting either would make the package-time checks below impossible and would let one business message route differently depending on the transport it arrived over.

A codec declares the attribute keys it can report, and optionally their closed value domains. sutra lint then refuses a predicate reading a key the bound channel’s codec can never emit, or comparing against a value outside its domain — so "Incoming" for "incoming" fails at package time instead of producing a start event that silently never fires. It also refuses two subscribers on one non-broadcast channel whose predicates it cannot prove disjoint; the recognised form is and / or / not over equality, inequality, in, and relational comparisons on numeric attributes. That limit is permanent rather than provisional: the gate is a proof obligation, and arbitrary FEEL cannot be proven disjoint. A codec declaring no attributes is open — the predicate still works and lint warns that it cannot verify it rather than blocking.

The codec itself is not declared on q:source — it comes from the channel (YAML is authoritative for transport/codec binding; BPMN is authoritative for process flow). Declaring a codec on q:source is a parse error.

q:validators and q:redactors (nested under q:source)

<q:source channel="transfer-request" messageTypeValue="TransferRequest">
  <q:validators>
    <q:complexValidator source="transfer-limits.dmn"/>
    <q:complexValidator source="transfer-fields.srl"/>
    <q:simpleValidator ref="iso-4217-currency" path="payload.amount.currency"/>
  </q:validators>
  <q:redactors>
    <q:redactor ref="pci"/>
  </q:redactors>
</q:source>

q:validators is a mixed, ordered container: a q:complexValidator runs a whole-payload ruleset (a .dmn or .srl file, or a built-in like iso-xsd) — a chain can mix .dmn and .srl entries freely, run in declaration order, with every entry’s issues accumulating into one result (see Composing a validator chain); a q:simpleValidator checks one field at a FEEL path against a registered content validator (iso-3166-country, iso-4217-currency, iso-9362-bic). q:redactors names registered ContentRedactors that mask sensitive spans in every observability surface (audit, logs, traces) without touching the value the flow actually sees.

q:reply and q:send — outbound

<bpmn:serviceTask id="OkReply" implementation="transfer-result.hbs">
  <bpmn:extensionElements>
    <q:reply mode="native" contentType="application/xml"/>
  </bpmn:extensionElements>
</bpmn:serviceTask>

q:reply answers the caller that started this instance — mode="native" (default) preserves the symmetric reply behavior, or emit a CloudEvent (cloudevent-binary / cloudevent-structured / match-inbound). q:send is the unsolicited counterpart — an intermediate throw event emitting to its own destination (@destination or @channel), with no inbound caller to answer. Both accept <q:header name="…" value="…"/> children carrying FEEL-derived values onto transport headers/application-properties.

q:reply’s continue="true" is respond-and-continue: flush the reply the moment the task completes, then park the instance and self-resume the remaining nodes asynchronously — the caller gets its answer without waiting on the tail of the flow.

A flow with no <q:reply> answers a synchronous HTTP call with 200 and an empty body; see Upgrading to 0.6.18.

Status and response headers (0.8.0)

On a synchronous HTTP channel (ack-mode: on-complete, the HTTP default), a <q:reply> with no destination can set the status, and its <q:header> children become the response’s headers:

<q:reply contentType="application/json" status="201">
  <q:header name="Location" value="'/items/' + req.id"/>
  <q:header name="X-Request-Id" value="req.id"/>
</q:reply>

status is literal first, else FEEL. Digits alone ("201") are the status. Any other text is a FEEL expression over the variables the reply renders against ("if found then 200 else 404"). It must give a whole number or a string of digits, and null keeps the default. Without the attribute the answer is 200, as before. Headers are evaluated like a <q:send>’s: a null value leaves the header out, and a repeated name adds another field line. A reply that declares a status or headers answers with them even when the flow rendered no body, so a 401 with its challenge needs no template.

RuleAt deploy (sutra lint)At the answer
The status is 200–599. A 1xx is not a final answer.A literal outside is refused.A value outside, a fraction, or a non-number is a 500.
A 3xx needs a Location header.A literal 3xx with no Location header is refused.A 3xx whose Location is null is a 500.
A 204 or 304 carries no body.—Body and Content-Type are not sent.
A header name is an HTTP token.Refused otherwise.—
The flow may not set Connection, Keep-Alive, Proxy-Connection, Transfer-Encoding, TE, Trailer, Upgrade, Content-Length, Content-Type (use contentType) or any Sutra-* name.Refused.—
A header value has no control character (CR, LF, NUL, …).—A 500.

The status refusals use SUTRA.OUTBOUND.REPLY_STATUS_INVALID, the header ones SUTRA.OUTBOUND.REPLY_HEADER_INVALID. The 500 is the usual application/problem+json. It names the header, never its value.

The status applies at the end of the flow, at a <q:reply continue="true">, and on the answer to a relayed message. An ack-mode: on-persist channel always answers 202, and adds the reply’s headers next to its receipt. A broker intake has no status. sutra lint warns about both (SUTRA.CHANNEL.REPLY_STATUS_IGNORED). A <q:reply> with a destination goes out of band, and its headers travel with it as before. Answering with a status and headers walks through it.

sequenceDiagram
    participant C as Caller
    participant E as Engine

    C->>E: inbound message
    E->>E: run as far as the reply task
    E-->>C: reply flushed the moment that task completes
    E->>E: park, then self-resume the remaining nodes

The caller’s answer is bounded by the task that produces it rather than by the tail of the process — the remaining nodes run on a resume nobody is waiting on.

The contract an outgoing message must satisfy

A message a flow produces is held to a codec’s contract before it leaves, the same way an inbound message is held to one on the way in. The codec’s own decode step is the validator, so the check is literally “read back what we are about to send” — there is no second contract to keep in step with the first.

This matters most for output a template, script or decision produces. Those renders reach the wire without passing through an encode step at all, so without this check an engine that validates every byte it accepts would validate none of the bytes it emits — and a flow could answer a strictly-validated request with a structurally invalid message.

Attribute on q:send / q:replyMeans
codec="urn:…"the codec whose contract this output must satisfy, stated rather than inferred
messageType="…"the message type this output emits — checked against what that codec can produce
<bpmn:serviceTask id="Answer" implementation="ack.hbs">
  <bpmn:extensionElements>
    <q:reply codec="urn:acknowledgements" messageType="Acknowledgement"/>
  </bpmn:extensionElements>
</bpmn:serviceTask>

State the codec when the output’s contract is not the one the bytes happen to leave by. A flow is a translator: it may accept one wire format and answer or forward in another, so the codec that decoded the request need not govern the answer.

The two forms differ in what happens when you say nothing, and the asymmetry is deliberate:

  • A <q:send> infers the codec of the @channel it names. That is not a guess — the bytes are addressed to that channel, and its codec is exactly what the consumers there will decode them with. A <q:send destination="…"> names an endpoint rather than a channel and so has nothing to infer from, which makes stating the codec the only way to hold it to a contract.
  • A <q:reply> infers nothing, and is unvalidated unless you state one. The tempting fallback is the channel the reply rides back out on, but a channel’s codec is the contract its requests satisfy, and a reply is by definition a different message — a codec bound to validate a transfer request has no declaration for the acceptance that answers it, so inferring it would reject correct replies.

A failure is refused rather than sent (SUTRA.OUTBOUND.VALIDATION_REJECT), and the diagnostic names the offending field. The default is fail-closed for the same reason q:onValidation’s is: a flow that declared no posture toward a validation failure has said nothing about handling one, so the invalid message must not leave. Egress bound to a bare format, or to no codec, asserts no schema and is unaffected — there is nothing there to fail.

Where the refusal lands depends on where the step runs. On a request the caller is driving, it comes back to that caller as the error, and nothing is persisted. On a resumed step (the tail after a <q:reply continue="true">, a timer, a <q:retry> re-drive, a relayed message), there may be no caller left to tell, so the instance fails instead. It is marked FAILED with this code (the failureCode on GET /sutra/instances/{id}), and its waits and timers are resolved so nothing re-runs.

sutra lint moves what it can to package time. It reads a template’s own unrendered text with the governing codec and compares the answer against template-manifest.yaml — outputMessageType, and outputAttributes for the codec-reported attributes. That works because a template is text with holes and the holes are leaf content: the element structure deciding classification is already there. It catches a template that renders the wrong message while declaring the right one. A template the codec cannot classify unrendered is reported unverifiable rather than wrong; a stated codec that does not resolve is an error, because stating it is how you ask for the check.

q:alias — correlation by your business key

<q:alias name="e2eId" expression="payload.E2EId" unique="true" onConflict="correlate"/>

A friendly key derived from a FEEL expression over the process variables — durable, and re-evaluated on rehydration. unique="true" with onConflict="correlate" is what lets a later message on a different channel find and resume the exact parked instance it belongs to, by a key you named (an EndToEndId, an order reference) rather than an engine-internal id. See Wait states and human tasks. A correlation alias’s value also appears in an acceptance receipt, masked if sensitive; see Acceptance receipts.

q:receipt — what a 202 names

An HTTP channel with ack-mode: on-persist answers 202 Accepted with a receipt the engine writes itself, so a caller learns which instance it started without the flow doing anything:

FieldWhat it is
instanceIdThe instance started or resumed: the handle for GET /sutra/instances/{id}.
deploymentId, labelsThe deployment serving the channel, and its tenant, module, version.
aliasesThe instance’s <q:alias> values at the moment of the answer, masked by the rules below.
idempotencyKeyThe key the caller supplied, or the one a <q:source dedupKey> derived. Absent otherwise: the body hash used in its place recognises no retry.
duplicate, originalInstanceIdThis delivery was already seen; the instance the first one started.
attributesThe values the flow chose to expose, below.

It travels in two headers: Sutra-Instance-Id and Sutra-Receipt (the whole receipt as one line of ASCII JSON, cut down to its identifying fields with "truncated": true if it would pass 4 KiB). A flow that replied nothing also gets it as the application/json body; a body the flow authored with <q:reply continue="true"> is left untouched. Other channels answer as before, except that a duplicate on any HTTP channel gets this 202. Queue transports have no synchronous answer, so no receipt.

<q:receipt> adds values of the flow’s choosing. It hangs off the process’s own extensionElements, and without it the flow exposes nothing:

<bpmn:process id="case-open">
  <bpmn:extensionElements>
    <q:receipt>
      <q:attribute name="caseId" expression="req.caseId"/>
    </q:receipt>
  </bpmn:extensionElements>

When it is evaluated. At the answer, over the variables a <q:reply continue="true"> renders against: everything the intake bound, plus what the flow computed before replying or parking.

What is masked. An expression reading a sensitive variable shows ***REDACTED***, an alias included. One reading a redactor-controlled variable shows that path of its masked projection, or the mask if it is not a plain path. While a redactor is active, anything under event other than channel, contentType, dedupKey, idempotencyKey and receivedAt is masked too, since it repeats the payload. A null is left out, and an expression that fails to evaluate is left out and logged: the work is already accepted and does not fail over its receipt. Values are strings.

sutra lint refuses an expression that does not parse and a repeated name (SUTRA.CONFIG.RECEIPT.ATTRIBUTE_DUPLICATE), and warns about an attribute reading a sensitive variable (SUTRA.CONFIG.RECEIPT.ATTRIBUTE_SENSITIVE), since the caller would only ever see the mask.

A duplicate’s receipt names the original instance, and carries no aliases or attributes: nothing ran to produce them. It names no instance when the first delivery ended without one or was recorded before 0.6.18.

A caller that lost its 202 can ask GET /sutra/instances?alias=<name>:<value>. See Acceptance receipts.

q:decode — read held text as a message

event.body is the message as text, so a flow can keep it in a typed store column with an ordinary assignment. <q:decode> is the other half: it parses text the flow holds with a registered codec and binds the result. It hangs off a data task, and anywhere else is a parse error (SUTRA.PARSE.Q_DECODE.NOT_ON_DATA_TASK). The walkthrough is Keeping a message as text.

<bpmn:serviceTask id="Load">
  <bpmn:extensionElements>
    <q:decode name="message" expression="row.body" codec="json" contentType="row.contentType">
      <q:redactors><q:redactor ref="card"/></q:redactors> <!-- optional -->
    </q:decode>
  </bpmn:extensionElements>
  <bpmn:dataInputAssociation>
    <bpmn:sourceRef>refRead</bpmn:sourceRef><bpmn:targetRef>row</bpmn:targetRef>
  </bpmn:dataInputAssociation>
</bpmn:serviceTask>
AttributeRequiredWhat it is
nameyesThe variable the decoded value is bound to.
expressionyesFEEL giving a string.
codecyesA built-in format name or a package schema codec URN, as on a channel.
contentTypenoFEEL. Picks the parser within a format’s contract (json, xml and yaml are one contract), and is passed to the codec.

Order. A data task runs its store reads, its decodes in document order, its assignments, then its store writes.

The instance’s own codec. It resolves in the instance’s pinned deployment, so a redeploy does not change how an old instance decodes. Validators and <q:onValidation> do not run: this is not intake. A codec’s non-fatal issues do not stop the step.

Failures can be caught with an error boundary or an event sub-process. Uncaught, they fail the instance. Their text names issue codes and paths only, never the input.

CodeWhen
SUTRA.RUNTIME.DECODE.NOT_TEXTThe expression gave null or something other than a string. A timer start has no event, so event.body there gives this.
SUTRA.RUNTIME.DECODE.FAILEDThe codec refused the text or its content type, or the codec or a redactor isn’t registered.
SUTRA.RUNTIME.DECODE.TOO_LARGEThe text is over the engine’s sutra.codec.max-payload-bytes (10 MiB by default; 0 there still leaves that default here). Nothing was decoded.

Text only. A valid UTF-8 body round-trips exactly. Otherwise invalid bytes become U+FFFD in event.body, and event.bodyLossy is true, so a flow can route the message away; a clean message has no such key. There is no binary form. A body whose prolog declares another encoding, such as ISO-8859-1 XML, cannot be kept faithfully. A NUL byte survives the conversion but PostgreSQL TEXT refuses it, and that store write fails the instance.

Masking. A <q:redactors> child writes a masked projection to the <name>.redacted companion, and a crashed redactor masks the whole value. Otherwise declare the target sensitive. A store column has no at-rest encryption, and sensitive does not reach it. In a package where a process masks its source, sutra lint:

CodeSeverityWhen
SUTRA.CONFIG.BPMN.RAW_BODY_UNMASKEDwarning, an error in a later releaseA variable holds raw event.body text and isn’t sensitive. This covers a copy, and a variable read from a store that a masking process wrote it to, even in a process with no masking of its own.
SUTRA.CONFIG.BPMN.RAW_BODY_STOREDwarningRaw event.body text is written to a store.
SUTRA.CONFIG.DECODE.REDACTORS_MISSINGerrorA <q:decode> reads such text, and its target has neither <q:redactors> nor sensitive.

The check follows a store by name within one package; another package reading the same database isn’t seen, and <q:alias> and <q:param> don’t count as copies. It also refuses a codec or redactor that doesn’t resolve (SUTRA.CONFIG.DECODE.CODEC_UNKNOWN, SUTRA.CONFIG.DECODE.REDACTOR_UNKNOWN), and a step missing an attribute fails at parse (SUTRA.PARSE.Q_DECODE.INCOMPLETE).

After a wait. A decode reads variables and store rows, never a delivery, so a resume or a retry decodes the same text. But a resuming message replaces event, and so does a synchronous channel-call reply unless the task declares an output mapping. So capture on the step right after the intake you mean. A call activity sees its caller’s event.

Cost. The decoded target and any capture variable are persisted at a park like any other, so a large message costs two or three times its size per park unless declared transient. A flow that doesn’t use <q:decode> pays nothing.

q:retry — per-task retry policies

<bpmn:serviceTask id="Score" implementation="registered:score">
  <bpmn:extensionElements>
    <q:retry maxAttempts="3" initialDelay="PT1S" backoffCoefficient="2.0"
             maxDelay="PT5M" nonRetryableCodes="SUTRA.TASK.VALIDATION"/>
  </bpmn:extensionElements>
</bpmn:serviceTask>

Valid on both kinds of <bpmn:serviceTask> — a registered task and a channel-call task (implementation="channel:<name>"). maxAttempts is the total invocation budget including the first attempt; attempt n+1 waits min(initialDelay × backoffCoefficient^(n-1), maxDelay); nonRetryableCodes names structured codes that fail immediately regardless of budget.

The load-bearing property: a retry wait is a durable timer park, never a sleep. The instance persists with the failed task still pending and an armed timer at the backoff instant, so the backoff survives restarts and hot-deploys and blocks no execution lane.

What counts as a failed attempt is deliberately narrow. On a registered task: an uncaught error from the task function. On a channel-call task, exactly two things — the route-less <q:timeout> boundary firing, and the request delivery being terminally poisoned by the outbox attempt ceiling. A correlated business response is never a retry trigger (the counterpart answered; re-sending would double-submit), and BPMN errors always route to their boundaries instead. Because a modelled outcome always beats a policy, a channel-call <q:retry> requires the route-less <q:timeout> form — a timer boundary with drawn outgoing flows alongside a retry policy is a load error.

@initialDelay and @maxDelay may be FEEL expressions instead of literals, for example initialDelay="config.retryDelay". They are evaluated each time a backoff is armed.

Full treatment, including the re-drive’s fresh-idempotency-key contract and what exhaustion does: Retries, history, and schedules.

q:timeout — a deadline on a channel call

<bpmn:serviceTask id="Score" implementation="channel:score-request">
  <bpmn:extensionElements>
    <q:timeout duration="PT2M"/>
  </bpmn:extensionElements>
</bpmn:serviceTask>

Synthesizes a timer boundary on the call, so a counterpart that never answers cannot park the instance forever. Without a <q:retry> alongside it, a fired route-less timeout is a catchable BPMN error; with one, it is a retryable task failure first.

@duration may also be a FEEL expression, such as duration="request.budget" or duration="config.callTimeout". It is evaluated when the call parks. See Timer intervals from FEEL.

q:stream — stream a channel-call answer

q:stream is an optional boolean attribute (not an element) on a bpmn:serviceTask. Writing q:stream="true" on a channel-call task (implementation="channel:<name>") marks it as one whose synchronous model answer should be streamed to the original caller, while the complete answer is still accumulated and resumes the flow as an ordinary response. It defaults to false, and omitting it changes nothing. Declaring it true on a task whose implementation is not a channel call fails the load with SUTRA.CONFIG.BPMN.STREAM.NOT_CHANNEL_CALL; a value that is not true/false fails with SUTRA.CONFIG.BPMN.STREAM.INVALID. The attribute is a backwards-compatible addition to the frozen urn:sutra:q:1.0 namespace. See Calling a language model.

q:store — durable, cross-instance data

<bpmn:dataStoreReference id="dsrFrom" name="accounts" dataStoreRef="accountsStore">
  <bpmn:extensionElements>
    <q:store key="payload.fromId" forUpdate="true"/>
  </bpmn:extensionElements>
</bpmn:dataStoreReference>

Binds a <bpmn:dataStoreReference> to a key in a datastores.yaml-declared store. forUpdate takes a pessimistic row lock (serializes concurrent writers on the same key); field replaces one field of a stored map value; expect="unchanged" is an optimistic compare-and-set instead of a lock; expect="absent" (0.6.10) writes only when the key does not exist yet — a claim exactly one replica wins. A lost compare-and-set raises SUTRA.RUNTIME.DATASTORE.CONFLICT, which an error boundary on the data task can catch.

On a write, delete="true" (0.7.0) removes the key’s row instead of writing a value. A missing key is a no-op, and expect="unchanged" makes the delete conditional on the revision the instance read. See Deleting a row.

On a read, query (0.6.10) turns the binding into a key query: every row whose key starts with key (query="prefix") or lies in [key, to) (query="range"), keys compared bytewise, in key order, limit per page (default 1000, maximum 10 000) after the key after; rows="false" returns aggregates only. Each child <q:aggregate fn="count|sum" field="…" groupBy="…"/> is computed over every matching row:

<q:store key="payload.meter + '|' + payload.day + '|'" query="prefix" limit="500" after="payload.after">
  <q:aggregate fn="sum" field="kwh"/>
  <q:aggregate fn="sum" field="kwh" groupBy="kind"/>
</q:store>

The target variable receives {rows: [{key, value}], aggregates: [...], truncated, next}. See Querying rows by key for the result shape, the number rules and the index the store’s table needs.

q:dispatch / q:case — a routing table instead of gateway sprawl

<q:dispatch default="fallback" onNoMatch="error">
  <q:case when="payload.type = 'A'" calledElement="handle-a"/>
  <q:case when="payload.type = 'B'" calledElement="handle-b"/>
</q:dispatch>

A declarative dynamic call-activity dispatch — one FEEL condition per case, instead of an exclusive gateway fanning into N call activities.

q:onValidation — the structural-failure policy

<q:onValidation mode="route" errorCode="T505"/>

mode="route" surfaces the validation summary (outcome, tier, firstReasonCode, firstIssue, issues) as variables so the BPMN’s own gateway decides what to do; reject / error short-circuit. The engine never interprets what a soft error means — that decision stays in your process, which is what keeps the engine domain-neutral.

The three modes are not interchangeable:

ModeWhat happens
routethe payload reaches the flow; you triage payload.validation.issues yourself
rejectthe message is refused at intake; no instance starts; the caller gets the issue list
errora BPMN error is raised into the flow, for an error boundary or event sub-process to catch

Declaring nothing means reject. An intake that carries a validation contract — a schema-backed codec, or a declared <q:validators> chain — and states no policy refuses a failing payload rather than passing it on. A flow that says nothing about handling failure has no handler for it by definition, so the failure must not enter it.

That default is reject rather than error deliberately: error is still the flow taking control, and raising a BPMN error into a process with no matching boundary reports “uncaught BPMN error” instead of naming the offending field — a worse diagnostic for the same refusal.

Schema-less ingress is untouched. A channel binding a bare format (csv, json, …) has no contract to fail, and the format layer is fail-open by construction, so nothing changes there.

If you want the payload through regardless, say so — <q:onValidation mode="route"/> — and the intent is visible in the diagram instead of resting on a default. sutra lint warns (SUTRA.CONFIG.VALIDATION.POSTURE_UNDECLARED) on any contract-bearing intake that declares no policy, naming both choices.

q:process — the retry-safety assertion

<bpmn:extensionElements><q:process idempotent="true"/></bpmn:extensionElements>

Hung off the <bpmn:process> itself. idempotent="true" is your assertion that re-running this process on the same input converges to one end state — so a redelivered message is safe to re-process any number of times. The default is false (fail-closed): an execution failure on a non-idempotent process is consumed (ack, no requeue) and recorded as an incident rather than blind-retried. This is a different concern from q:source’s dedupKey, which only detects that a message is a redelivery — it says nothing about whether re-processing it is safe.

What the assertion gates, and what it does not. It governs the failure path: whether a delivery that failed is requeued or dead-lettered. It does not gate replay after a park. An instance that parks — at a wait node, a <q:retry> backoff, or a <q:reply continue="true"> — is re-driven at-least-once by the durable poller, unconditionally. A multi-instance loop is one durable step, so a crash mid-loop replays it from the first item rather than resuming where it stopped.

Both rest on the same property: every effect the flow repeats must converge. A store write keyed by something derived from the record does (re-writing is an upsert). A <q:send>, an append, or a counter increment does not — and no assertion makes it so. Declaring idempotent="true" over effects that do not converge does not make replay safe; it only changes what happens after a failure.

q:variables, q:audit, q:coverage

  • q:variables — declares process variables up front (name + scalar type or a schema reference), so deploy-time static validation can check every FEEL path against them, and mark a variable transient (never persisted, so it is dropped at any park — reading it after a wait state, a <q:retry> backoff or a <q:reply continue="true"> is a validation error; it is for scratch that is re-derived, never for anything the flow still needs on the other side), sensitive (persisted but redacted downstream), or source-bound (payload-initialized from a named channel).
  • q:audit — per-process audit sink/target/capture-level configuration. capture is none, metadata, payload (the default) or, since 0.6, media: metadata-level events that carry the media references found among the variables, never the media (see Recordings and media references).
  • q:coverage — declares one tracked compliance route (path id + the ordered flows it must traverse) for the path-coverage reporting the CLI’s sutra coverage commands drive. This is the intra-process half of the mechanism; a route spanning several correlated processes is declared differently. See Coverage: declared routes as the compliance signal for both shapes, the curation guidance, and the CLI walkthrough.

Next

Rules: DMN, FEEL, and .srl

A businessRuleTask binds one decision, authored either as a .dmn decision table or as a .srl ruleset. Both compile onto the same FEEL evaluator (sutra-feel) — no JVM, no Rete runtime, and no second expression language to learn.

flowchart LR
    BRT["bpmn:businessRuleTask"] --> R{"routed by<br/>file extension"}
    R -->|".dmn"| D["DMN 1.5 decision table<br/>all seven hit policies"]
    R -->|".srl"| S["ruleset: rule / when / then<br/>one deterministic pass"]
    D --> F["sutra-feel — one evaluator<br/>DECIMAL64 · determinism denylist"]
    S --> F

Choosing a table or a ruleset is a choice of framing, not of runtime — an expression means the same thing in a gateway condition, a q:alias, a DMN cell, and an .srl when.

FEEL — the shared expression language

FEEL (Friendly Enough Expression Language) is what every condition, gateway expression, q:alias key, q:store key, and rule body is written in. Sutra’s implementation is a from-scratch Rust evaluator (lexer, parser, AST, a determinism denylist that forbids non-reproducible constructs on a replay path, and DECIMAL64 numeric semantics — 16 significant digits, HALF_EVEN rounding).

Try an expression standalone, before it’s anywhere near a process:

sutra explain 'payload.amount > 100'
sutra explain --context vars.txt 'fromAccount.balance - payload.amount'

sutra explain is a one-shot evaluator or a REPL (omit the expression) — useful for debugging a gateway condition or a q:alias expression without deploying anything. See Crates: sutra-feel and sutra-dmn if you want FEEL evaluation embedded in your own Rust project, independent of the engine.

Durations with fractional seconds

Since 0.8.0. A days-and-time duration holds its seconds to the nanosecond (digits past the ninth are cut off), through parsing, string(), comparison, arithmetic with dates, times and other durations, multiplication and division by a number, and the seconds accessor. Seconds take XML Schema’s decimal forms (PT0.5S, PT.5S, PT0.S). From sutra explain:

$ sutra explain 'duration("PT0.001S")'
duration("PT0.001S")  =>  PT0.001S
$ sutra explain 'duration("PT4.5S").seconds'
duration("PT4.5S").seconds  =>  4.5
$ sutra explain 'string(duration("P1DT2H3M4.123456789S"))'
string(duration("P1DT2H3M4.123456789S"))  =>  P1DT2H3M4.123456789S
$ sutra explain 'duration("PT1.5S") * 3'
duration("PT1.5S") * 3  =>  PT4.5S
$ sutra explain 'date and time("2026-11-19T10:00:00Z") + duration("PT0.250S")'
date and time("2026-11-19T10:00:00Z") + duration("PT0.250S")  =>  2026-11-19T10:00:00.25Z

Only seconds take a fraction. Until 0.7.0 the parser dropped everything after the decimal point, so duration("PT0.001S") was zero and duration("PT1.5H") was one hour; now the second is refused:

$ sutra explain 'duration("PT1.5H")'
[ERROR] SUTRA.FEEL.COMPILE.TYPE_MISMATCH — duration() cannot parse the string (feel:inline:1:1)

Write PT1H30M instead. A timer whose FEEL expression returns duration("PT0.001S") now waits 1 ms instead of firing at once.

Password hashing

Two builtins let a flow check a password against a stored hash without ever storing the password (new in 0.6.21):

FunctionReturns
password hash(password)An argon2id hash in PHC string form, $argon2id$v=19$m=19456,t=2,p=1$<salt>$<hash>, with a fresh 16-byte salt from the operating system. null if the argument is not a string or is empty.
password verify(password, hash)true if the password matches the hash, false if it does not. null if either argument is not a string, or if the hash is malformed, is not argon2id, or asks for more than m = 65 536 KiB, t = 10 or p = 4. The comparison is constant-time.
  • The parameters are fixed in code: argon2id, m = 19 456 KiB, t = 2, p = 1, the OWASP minimum. A package can’t change them. password verify reads the parameters from the stored hash, so the cap keeps a planted hash from tying up a worker and its memory.
  • password hash is denied at replay-bound sites, like uuid(), because it draws a random salt. Use it in a task, not in a q:alias or a when; sutra lint refuses it there with SUTRA.FEEL.DETERMINISM.UNSAFE_BUILTIN. password verify is deterministic and works anywhere.
  • Cost. One verify takes about 22 ms (criterion median 22.3 ms on an Intel i7-10750H laptop, release build), and holds its executor worker for that time. That suits a sign-in; don’t call these functions for every message.

Signing a user in against a store builds a sign-in flow with them and checks that the password reaches no table, audit record or log line.

DMN decision tables

A .dmn file under rules/ is a standard DMN 1.5 decision table; the engine’s evaluator supports all seven hit policies. A businessRuleTask names the decision, and its named output columns merge back into the process’s instance variables:

<!-- examples/approval-hold/.../rules/approval-decide.dmn -->
<decision id="approvalDecide" name="Review decision from risk score">
  <decisionTable hitPolicy="FIRST">
    <input id="i_risk"><inputExpression typeRef="number"><text>riskScore</text></inputExpression></input>
    <output id="o_decision" name="decision" typeRef="string"/>
    <rule id="r_approve">
      <inputEntry><text>&lt; 50</text></inputEntry>
      <outputEntry><text>"approve"</text></outputEntry>
    </rule>
    <rule id="r_review">
      <inputEntry><text>-</text></inputEntry>
      <outputEntry><text>"review"</text></outputEntry>
    </rule>
  </decisionTable>
</decision>

Sutra’s DMN conformance is measured against the real OMG DMN-TCK, not asserted — see DMN-TCK conformance for the numbers and what they cover.

.srl — a Drools-inspired rule DSL over FEEL

.srl is Sutra’s own rule language: rule / when / then / end framing around FEEL conditions and a small, closed set of side-effecting actions. It targets the same use case DMN’s COLLECT hit policy does — several independent business-validation rules over one payload — in a syntax closer to a Drools ruleset than a decision table.

rule "currency-not-usd"
when
  exists(payload.amount.currency) and payload.amount.currency != "USD"
then
  report(
    "SUTRA.VALIDATE.CURRENCY_NOT_USD",
    "payload.amount.currency",
    "Currency must be USD; got " + payload.amount.currency
  );
end

Grammar, in full:

ruleset := rule*
rule    := "rule" STRING attr* "when" <condition> "then" action* "end"
attr    := "salience" INTEGER | "activation-group" STRING
action  := verb ";"
verb    := "report" "(" <feel_expr> "," <feel_expr> "," <feel_expr> ")"
         | "set"    "(" IDENT "," <feel_expr> ")"
  • Every condition and action argument is an embedded FEEL expression — .srl adds only the rule framing on top.
  • Two action verbs today: set(target, expr) binds a value into the working context (visible to later rules) and the output map; report(code, path, message) appends a structured issue. insert / retract are reserved for a future stateful phase and are a clean parse error, not silently accepted.
  • Evaluation is a single deterministic forward pass — a stable-sorted agenda by (-salience, declaration order), each rule firing at most once, activation-group giving first-match-wins semantics within a group. This is sequential-agenda, not a Rete network.
  • Fail-closed: a parse error, or a condition/action that errors at evaluation, is a hard error — never a silently-skipped rule.
  • A FEEL if / then / else used inside a when condition must be parenthesised (when (if a then b else c) …), because the bare keyword then that ends the condition is matched at paren depth 0.

.srl and .dmn both live under a package’s rules/ folder (see Deployment packages) and are routed by file extension — no separate configuration names one or the other.

Composing a validator chain

A <q:source>’s <q:validators> list (see The q: namespace) can mix <q:complexValidator> entries pointing at both .dmn and .srl files in one chain — nothing requires a chain to stick to one rule engine, and nothing requires a business ruleset to live in a single file.

  • Declaration order is evaluation order. The chain runs top to bottom exactly as written in the BPMN.
  • Every validator sees the same payload projection. Whatever the channel’s codec decoded is what each entry in the chain evaluates against — an earlier validator’s issues don’t change what a later one reads, only what accumulates alongside it.
  • Issues accumulate, they never replace. Each entry’s reported issues append to the same validation.* result; validation.outcome and validation.tier are derived from the whole accumulated list, so a gateway keyed on them is indifferent to which file in the chain produced which code.
  • validation.firstReasonCode follows declaration order. When two entries both report an issue, firstReasonCode is whichever fired first in chain order — this only affects which code a reply template surfaces, never whether a gateway routes to reject.

This is what lets you split one business ruleset across engines by what each rule needs, instead of by an arbitrary file boundary. A module with a mix of clock-dependent and stateless checks (an extension-crate workload, not one of this repository’s bundled examples) does exactly that:

<q:validators>
  <q:complexValidator source="intake-timing.dmn"/>
  <q:complexValidator source="intake-fields.srl"/>
</q:validators>

The DMN table keeps the rules that need a clock — a staleness window and a clock-skew/future-dated window on some received-at timestamp — because the engine-injected evaluation clock is only available as a DMN validator’s reserved now input. The .srl file carries the stateless field checks (a required-currency check, a positive-amount check, a required-identifier check) that have no temporal dependency at all and read more naturally as named rules than as columns on the decision table. Both files reason over the identical payload projection the codec decoded, report through the same SUTRA.VALIDATE.* code family, and their issues land in one accumulated validation.issues list — the split between engines is invisible to the BPMN gateway that routes on the outcome.

flowchart LR
    P["the payload projection<br/>the codec decoded"] --> V1["1 · intake-timing.dmn"]
    P --> V2["2 · intake-fields.srl"]
    V1 -->|"issues append"| ACC["one accumulated result<br/>validation.issues · outcome · tier"]
    V2 -->|"issues append"| ACC
    ACC --> G{"the BPMN gateway<br/>routes on the outcome"}

Both entries read the identical projection and append into one result, so the gateway cannot tell which file — or which rule engine — produced a given code; only firstReasonCode carries the declared order.

Which to reach for

  • DMN if the logic is naturally tabular (a rate card, an approval matrix, one output per row) or you want a diagram a business analyst can review directly.
  • .srl if the logic is a set of independent validation rules over a payload, where a Drools-style when/then reads more naturally than a table.
  • Both, in one chain, when a ruleset is naturally mixed — some rules need something only a DMN validator gets (the injected evaluation clock), others are better named individually than columned into a table. See Composing a validator chain above for exactly how a .dmn and a .srl entry combine, illustrated above with a clock-dependent-vs- stateless split.

Next

Data stores

A data object (<bpmn:dataObject>) is a per-instance variable — private, transient, born and dying with one instance. A data store (<bpmn:dataStore> / <bpmn:dataStoreReference>) is the opposite axis: durable, keyed state that outlives and is shared across instances — an account ledger, a customer index, the covered/uncovered flags behind a compliance route.

Status note. Data stores are exercised end to end by the money-transfer example — a real sql-backed ledger under transactional, pessimistic-lock control. (Path coverage rides a data store of an unusual kind: its marks are persisted in a store the package declares — the reserved coverage name, whose connection picks the database they land in — but the engine owns that store’s schema and applies it itself, so it is neither a key→value business store nor one you write DDL for. See Coverage: declared routes as the compliance signal.) Some of the wider surface described in the design record — a file-backed store, an optimistic-concurrency compare-and-set on every provider — is still in motion; treat anything below not shown against the money-transfer example as directionally accurate rather than a finished contract, and check rust/crates/sutra-datastore for the current provider set before depending on specifics.

Choosing a store shape

A business store has two shapes. The choice between them is narrower than the volume of mechanism the rest of this chapter spends on the second one suggests, so it is worth settling first. (The reserved coverage store is neither — the engine owns its schema and applies it, so a package writes no coverage DDL and makes no shape choice at all; see Coverage.)

Key/value is the general case, and it is always sufficient. By default a store is a key→value store: one row per key, the value serialized whole as JSON text. Any structure fits it — nested, repeated, open-ended, a different shape next release — because nothing about the value’s shape is written down anywhere for it to disagree with. The table behind it is one fixed generic shape (store name, key, value, rev, updated_at) shared by every key/value store on that connection, and it does not change when your data does. From a standards point of view this is the only shape that carries every scenario, which is why it is the default and stays the default. A store with no structure: block is this store, and none of the projection machinery below applies to it.

Projection is an integration affordance, not an upgrade. Declaring a structure: does not make a store more correct, more durable or more capable, and it is not a tier you graduate to. It exists for one reason: so that everything else that already speaks SQL can read the data. A BI tool, a dashboard, a nightly reporting query, an analyst with read access to that database — none of them know what Sutra is, and against a key/value store the most any of them can see is an opaque document to parse. Against a projected table they see balance and opened_at as columns, with types. That is one more way to ease integration, not a better kind of store.

The trade, stated plainly. Projection costs two things and buys one:

  • It costs the flat-only constraint. The declared type must be entirely scalar, and a type that later grows a nested or repeated child stops projecting at all — see Flat only.
  • It costs ownership of the DDL and its evolution. You write the CREATE TABLE, and every change to the declared type is a migration you author and sequence. A key/value store needs a table too, but it is the same generic one every time — written once, never revisited. A projected table is your record’s own shape, so it has to move whenever that shape moves.
  • It buys direct SQL access to the data, for anything that speaks SQL, without going through the engine.

If nothing outside the process ever reads the store, that affordance buys nothing, and key/value is the right answer.

flowchart TD
    W["one write through q:store"]

    subgraph KV["Key/value — the default"]
        KROW[("a shared generic table<br/>name · key · value · rev")]
    end

    subgraph PJ["Projected — a structure: block"]
        PROW[("this store's own table<br/>one typed column per field")]
    end

    W --> KROW
    W --> PROW
    KROW -.->|"an opaque document<br/>to parse"| BI["any SQL tool that has<br/>never heard of Sutra"]
    PROW -->|"balance, opened_at:<br/>columns, with types"| BI

Both shapes hold the same record and answer the same <q:store> access with the same locking and rev semantics; the only thing projection moves is the arrow on the right — an opaque document versus columns a reader can use directly, bought with the flat-only constraint and ownership of the DDL.

The shape is a property of the store, not of the flow — within limits worth knowing. A store’s shape is declared in datastores.yaml and nowhere else. The BPMN is unchanged either way: the same <bpmn:dataStoreReference>, the same <q:store> with the same key / field / forUpdate / expect attributes, resolved by store name through the same registry. Pessimistic locking, <bpmn:transaction> enlistment and rollback, delete, and the rev bookkeeping behind expect="unchanged" — seeded at 1, bumped on every write, a conflict when the revision moved — carry the same semantics under both shapes. Adding or removing a structure: block is a package-level change, not a process-level one.

What does change is what the store accepts and what it hands back — and that is not nothing:

  • A projected store refuses a write carrying a field the structure does not declare (UNDECLARED_FIELD); a key/value store simply stores it. This is the difference most likely to bite, because a process that has quietly been writing a wider record than the declared type works against key/value and fails against projection — and nothing catches it beforehand: neither sutra lint nor the executor checks a <q:store field="…"> name against the declared structure, so the first evidence is the refused write.
  • It refuses a value that isn’t a record at all — a scalar, an array, an explicit null (VALUE_NOT_A_RECORD). A key/value store holds any of those happily.
  • It refuses every operation until the live table satisfies the declaration (PROJECTION_UNSATISFIABLE, checked on first use). A key/value store has no declaration to drift from, so it has no such failure mode.
  • A field-narrowed write is a read-modify-write of the whole value under both shapes, so one that creates a key writes a record containing only that field. On a key/value store that is simply a one-field document; on a projected store every other declared column is bound NULL, so unless they are all nullable or defaulted in your DDL that insert runs straight into your own constraints. Make the first write of a key a full record rather than a narrowed one.
  • Values that pass through a typed column come back in the column’s canonical form rather than the text that was written: an explicit null and an absent optional field become the same thing on read, and a dateTime comes back as your dialect’s rendering of that instant. The exact scope of what is and isn’t preserved is in A worked example.

None of those change how a process addresses the store; several of them change whether a given write succeeds. They are the checklist for a shape change, not a reason to avoid one.

Column names become a published interface. The moment a dashboard reads that table, its column names are a contract with someone who has never seen your schema — a report knows account_balance, not the XSD element it was derived from. Column names are derived by convention from the declared field names, so renaming a field renames a column, and silently breaks that report. That is what the columns: override exists for: pin the physical name, and the schema can be renamed underneath it while the published surface holds still. It is worth setting before you need it rather than after someone’s query starts returning nothing — the naming rules and the override syntax are under Column names.

Declaring a store

One flat datastores.yaml per package (see Deployment packages). The module owns its store — the connection is declared here, resolved from environment references, and its migrations ship inside the package:

# examples/money-transfer/.../datastores.yaml (abridged)
datastores:
  - name: accounts
    type: sql
    sql:
      url-ref: env:ACCOUNTS_DB_URL
      username-ref: env:ACCOUNTS_DB_USER
      password-ref: env:ACCOUNTS_DB_PASSWORD
      migrations: migrations/accounts     # V<n>__*.sql, each run once per database
    dataClass: financial                  # sensitivity tag — redacted in audit/logs/traces

A store’s migrations run the first time the store is used, under a database lock, so replicas never run them at the same moment. The lock is one per database (per server on MySQL and MariaDB), not one per store, because several stores often share one table and the same CREATE TABLE IF NOT EXISTS. Before 0.6.12 the lock was per store, and two stores sharing a table that were first used at the same moment could fail that first request with HTTP 500 (pg_type_typname_nsp_index); the next request succeeded.

Migrations run once

The engine records every script it applies in a history table, sutra_store_migrations, created in the store’s own database the first time a store there migrates. The table has one row per store and version: store_name, version, script (the file name), checksum (the SHA-256 of the script as it ran) and applied_at. On first use, under the lock described above, each script is checked against that table:

  • Recorded with the same checksum: skipped. A restart, a second replica or a redeploy of the same package runs nothing.
  • Not recorded: it runs, then its row is written. On PostgreSQL and SQL Server the script and its row commit in one transaction, so a script that fails leaves neither its changes nor a row behind. MySQL and MariaDB commit DDL implicitly, so there the script runs first and its row is written after it succeeds. Either way, a failed script records nothing and fails the store’s first use; the next use tries again. Because of that transaction, a statement PostgreSQL refuses inside one (CREATE INDEX CONCURRENTLY, for example) cannot be in a migration script; create such an index outside the package’s migrations.
  • Recorded with a different checksum: the store refuses to start with SUTRA.RUNTIME.DATASTORE.MIGRATION_CHECKSUM, naming the store, the file, the version and both checksums. A script that has been applied must not be edited. Restore it, and put the change in a new script with the next version number.

Rows for scripts the package no longer ships are ignored.

Versions and order. V<n>__<description>.sql has version <n> with leading zeros dropped, so V001__accounts.sql and V1__accounts.sql are the same version, and two scripts of one store with the same version are refused (SUTRA.CONFIG.DATASTORE.MIGRATION_VERSION_DUPLICATE). When every script is named this way they run in numeric order, so V2 runs before V10. A script with any other name still works: its version is its file name without .sql, and a folder that mixes the two styles runs in plain file-name order. sutra lint requires the V<n>__ form.

Databases migrated before the history table existed. Releases before 0.6.13 re-ran every script on every boot and kept no record, which is why those scripts had to be idempotent. Nothing special is needed to move such a database over: it has no rows for the store, so each script runs one more time, harmlessly, and is recorded. From then on it runs once.

Placeholders. A script can use five placeholders, replaced before it runs and before its checksum is taken:

PlaceholderReplaced with
${package.tenant}the package’s tenant label
${package.module}the package’s module label
${package.version}the package’s version label
${store.name}the store’s name in datastores.yaml
${store.table}the table the store writes: its sql.table (or its name) when it declares a structure:, otherwise data_store
-- migrations/accounts/V002__seed.sql
INSERT INTO data_store (store_name, store_key, store_value)
VALUES ('${store.name}', 'settings', '{"owner":"${package.tenant}"}');

Any other ${…} is an error, and so is a ${ with no } on the same line. Both are reported by sutra lint and again when the engine loads the package (SUTRA.CONFIG.DATASTORE.MIGRATION_PLACEHOLDER, naming the file, the line and the placeholder). Write $${ to get a literal ${. A lone $ or $$, such as PostgreSQL dollar quoting, is left alone. Because ${package.version} is part of the checksummed text, a script that uses it counts as edited when the version label changes. Use it only in scripts you add for a specific release.

The engine’s own datasource (instances, outbox, lease, audit, inbox) is never a store’s backing connection — a sql store always resolves its own connection, and fails closed if it declares none. This is what makes a generic engine image self-sufficient against any number of modules that each bring their own store: no baked default datasource, no baked migrations.

A store declared this way is the key/value shape: one row per key, the value serialized whole as JSON text, opaque to the database. Adding a structure: block projects a flat record onto real typed columns instead — see Typed columns: declaring the structure a store holds below.

Referencing a store from BPMN

<bpmn:dataStore id="accountsStore" name="accounts"/>   <!-- definitions scope -->

<bpmn:dataStoreReference id="dsrFrom" name="accounts" dataStoreRef="accountsStore">
  <bpmn:extensionElements>
    <q:store key="payload.fromId" forUpdate="true"/>
  </bpmn:extensionElements>
</bpmn:dataStoreReference>

<q:store> (see The q: namespace) is what turns a bare reference into a keyed access: key is a FEEL expression producing the row key, forUpdate="true" takes a pessimistic row lock so concurrent writers on the same key serialize, field narrows a write to one field of a stored map value, and expect="unchanged" is an optimistic compare-and-set alternative to a lock. expect="absent", delete and query are covered in Claiming a key once, Deleting a row and Querying rows by key. A data task — a serviceTask with no implementation — reads and writes through its data associations exactly like it would a data object, except the source/target is a store reference instead of a variable.

Typed columns: declaring the structure a store holds

An opaque store hands the database a string. Nothing inside the value is queryable, indexable or typed, and the only way to read a field is to fetch the whole document by key and pick it apart in the process. For a record whose fields are all scalars that is a blob tax on data that could perfectly well be ordinary columns.

So a store may declare the structure it stores, not just its key:

datastores:
  - name: accounts
    type: sql
    structure:
      schema: urn:accounts        # a schemas/<folder> codec of this package
      type: AccountRecord         # a complexType, a root element, or a JSON Schema definition
    sql:
      url-ref: env:ACCOUNTS_DB_URL
      migrations: migrations/accounts

schema + type resolve through the schemas the package already compiles for its codecs (the path-derived URN rule is in Deployment packages), so there is no second schema source to keep in step and no shape inferred from observed data — the declaration is the contract.

A store with no structure: block behaves exactly as it always did. Same opaque key→JSON row, same providers, same <q:store> semantics, and none of the diagnostics below ever fire for it. Declaring a structure is opt-in per store; a record too nested to project simply doesn’t declare one.

Flat only

The declared type must be entirely scalar. Every child is classified in declared order:

Declared shapeBecomes
Scalar leaf, at most once (a simple-type element or an attribute)a column
Scalar leaf, minOccurs="0"a nullable column
Scalar element inside a choicea nullable column (at most one branch is populated)
A complex child — a nested objectlint error
Anything that may occur more than oncelint error
Open content (xs:any, JSON Schema additionalProperties: true)lint error

The table is the declared fields. There is no residue column for the parts that didn’t fit, no column-versus-JSON merge on read, and no partial projection. That is the whole simplification, and it is what makes a projected row readable by anything that speaks SQL without knowing Sutra exists.

A type that can’t be expressed that way is rejected at package time with SUTRA.CONFIG.DATASTORE.STRUCTURE_NOT_FLAT, naming the offending child:

data store 'accounts': declared structure type 'AccountRecord' is not flat: 'owner' is not a
scalar leaf (nested content). Flatten the type, or remove the 'structure' block and keep the
opaque store.

Those two remedies are the whole menu, deliberately:

  • Flatten the type — pull the nested part up into scalar siblings (owner/name → ownerName), or move it to a store of its own keyed by the same business id.
  • Drop the structure: block — the store goes back to being the opaque key→JSON store it was, which is a perfectly good answer for a record that genuinely isn’t flat.

There is nothing in between, because anything in between would be a silent partial projection — some fields queryable, some hidden in a blob, and a merge rule nobody can see from the declaration.

Because the structure is closed and there is nowhere for an unexpected field to go, a write carrying a field the structure does not declare is a fail-closed runtime error (SUTRA.RUNTIME.DATASTORE.UNDECLARED_FIELD) naming the field. Silently dropping it is the one outcome this design refuses. Writing anything that isn’t a record at all — a scalar, an array, an explicit null for the whole value — is refused the same way, as SUTRA.RUNTIME.DATASTORE.VALUE_NOT_A_RECORD: a projected row is its declared fields, so there is nowhere for a bare value to land either.

Control columns

A projected table is never only the declared fields. Three columns belong to the engine, on every projected table, whether or not your own type ever mentions them:

ColumnRole
store_keythe store key — whatever the store’s <q:store key="…"> expression produces — and the table’s PRIMARY KEY
revthe optimistic-concurrency revision, bumped on every write; what put_if_revision keys on
updated_atthe write timestamp

There is no store_name column — a projected table is one store, unlike the opaque table that multiplexes many stores by name. The runtime binds and maintains all three itself, so a declared field may not claim one of their names: that’s a naming collision (SUTRA.CONFIG.DATASTORE.COLUMN_NAME_INVALID), resolved the same way as any other one — map the field to a different column under columns:. Your own migration has to create all three alongside your declared columns, or sutra lint raises COLUMN_MISSING naming whichever is absent — a projected table missing store_key, rev or updated_at cannot be served no matter how correct its declared columns are. The worked example below shows all three in the DDL.

Column names

A declared field’s column name is derived by convention: lowerCamel → snake_case, ASCII-folded, runs of anything else collapsed to a single _.

Declared fieldColumn
accountIdaccount_id
openedAtopened_at
iso4217Codeiso4217_code
IBANCodeiban_code

The convention is not always enough, and when it isn’t, sutra lint says so (SUTRA.CONFIG.DATASTORE.COLUMN_NAME_INVALID) rather than guessing: two fields folding to the same column, a fold that lands on a SQL reserved word, a name over the 63-character identifier cap (PostgreSQL’s — the narrowest of the three shipped dialects, so a name that clears it is portable), or a fold that isn’t a usable identifier at all. Every one of them is resolved by naming the column yourself:

    structure:
      schema: urn:accounts
      type: AccountRecord
      columns:
        openedAt: opened_on       # the DDL already calls it this
        order:    order_seq       # `order` is reserved

The mapping exists for two reasons. The first is that you write the DDL and may already have column names you can’t change. The second is that it pins the physical name against a later schema rename — which matters as soon as anything outside the engine reads the table, because at that point the column names are a published interface and the convention is no longer yours alone to change (see Choosing a store shape). The mapping is checked by the same rules — an override that is itself reserved or over-length is still an error, and an override naming a field the type doesn’t declare is an error rather than a silent no-op.

You own the DDL; the engine verifies it

The engine generates no DDL for a projected store and owns no part of its table shape. That table is created by the store’s own migrations, in your own dialect, shipped inside the package under migrations/<store>/ (see Deployment packages).

sutra lint derives the effective table shape statically — from the package’s own migrations/<store>/V*.sql, replayed in version order (CREATE TABLE plus ALTER TABLE ADD/ALTER/DROP COLUMN), with no database connection and no credentials — and compares it against the projection. The table it compares against is the one named after the store — or, if the migrations create exactly one table, that one — or the one you name explicitly:

    sql:
      url-ref: env:ACCOUNTS_DB_URL
      migrations: migrations/accounts
      table: account_ledger       # when the table isn't named after the store

What it checks the column types against is this mapping. It is advisory — the shape lint expects to find, not DDL anything emits:

DeclaredPostgreSQLMySQL / MariaDBSQL Server
xs:string + maxLength nVARCHAR(n)VARCHAR(n)NVARCHAR(n)
xs:string, unboundedTEXTLONGTEXTNVARCHAR(MAX)
xs:decimal + totalDigits p / fractionDigits sNUMERIC(p,s)DECIMAL(p,s)DECIMAL(p,s)
xs:int / xs:long / xs:shortINTEGER / BIGINT / SMALLINTsameINT / BIGINT / SMALLINT
xs:booleanBOOLEANTINYINT(1)BIT
xs:date / xs:dateTime / xs:timeDATE / TIMESTAMPTZ / TIMEDATE / DATETIME / TIMEDATE / DATETIME2 / TIME
xs:base64BinaryBYTEABLOBVARBINARY(MAX)
an enumeration facetthe base type (a CHECK or a lookup table is your choice)samesame

Two declared shapes this release refuses outright — not silently degraded, not weakly verified — and one honest limit on verification alone:

  • xs:base64Binary is refused, not projected. The advisory mapping above (BYTEA / BLOB / VARBINARY(MAX)) is what the column would need to be; the runtime doesn’t marshal binary values through a projected column yet. A store declaring one is refused when the engine resolves it — at deploy time, not at lint time — naming the field:

    data store 'accounts' cannot project structure type 'AccountRecord': field 'attachment' is
    declared 'base64Binary', which a projected column does not carry in this release — declare it
    as a string, or remove the 'structure' block and keep the opaque store.
    

    Lint doesn’t catch this on its own: its DDL parser treats BYTEA/BLOB/VARBINARY as an ordinary column type and matches it happily, so a package can lint clean and still refuse to deploy. Don’t read lint’s silence as proof a binary field works.

  • A JSON-Schema-declared structure is refused, not weakly verified. Only a package schemas/<folder> XSD codec carries the enumerable, facet-bearing field list a typed column is checked against — a JSON Schema definition has no such list at this phase. sutra lint reports that honestly as a DDL_UNVERIFIABLE warning (unprovable, not necessarily wrong) — but the engine that actually resolves the store at deploy time draws a harder line and refuses to load the deployment:

    data store 'accounts' declares structure schema 'urn:accounts', which this package provides no
    XSD codec for. A projected store's type must be declared by a schemas/<folder> XSD codec of the
    same package — only XSD carries the declared facets a typed column is checked against.
    

    So a lint-clean package with a JSON-Schema structure: still will not deploy. Declare the structure against an XSD if you want the store to exist at all, not just to have its column types checked.

  • DDL outside the parsed subset degrades to a warning, never to a false error. A migration that creates the table in a PL/pgSQL block, a T-SQL procedural guard, a table created outside the package entirely — lint reports that it could not derive the shape and raises no column diagnostic for that store. A linter that cries wolf on legitimate DDL is worse than no linter, because authors learn to ignore it. See Troubleshooting for how to read that warning.

Lint proves the package-time case. The deployed table is checked separately, but not on every operation: first use of a projected store rides the same once-per-store-instance gate that runs its migrations (the one already serialized across replicas by an advisory lock), and there the provider reads the live table’s actual columns and fails the store closed if the projection isn’t satisfiable, naming every offending column at once. That is the defence against a table that drifted from the package’s own migrations — a hand-applied ALTER — and it fails loudly because a silent partial write is far worse than a refusal. It costs one catalog round-trip the first time the store is used, and nothing on any operation after that.

flowchart TD
    L["<b>lint</b> — package time<br/>replays your V*.sql, no database"]
    L -.->|"COLUMN_MISSING<br/>KEY_MISMATCH"| LE["package rejected"]
    L --> D["<b>deploy</b> — the store resolves"]
    D -.->|"a field no codec<br/>of this package declares"| DE["deployment refused"]
    D --> F["<b>first use</b> — read the<br/>live table's columns, once"]
    F -.->|"the table does not<br/>satisfy the declaration"| FE["PROJECTION_UNSATISFIABLE<br/>fails closed, names every column"]
    F --> R["<b>every operation after</b><br/>no further catalog round-trip"]
    R -.->|"undeclared field,<br/>or not a record"| RE["UNDECLARED_FIELD<br/>VALUE_NOT_A_RECORD"]

Each gate sees something the one before it could not — lint has the migrations but no database, deploy has the resolved store but no table, first use has the live catalog — which is why a package can pass lint (or draw only a warning from it) and still refuse to deploy.

A worked example

A flat record, declared in the package’s own codec schema (schemas/accounts/accounts.xsd, the codec urn:accounts):

<xs:element name="AccountRecord">
  <xs:complexType>
    <xs:sequence>
      <xs:element name="accountId" type="AccountId"/>
      <xs:element name="balance"   type="Money"/>
      <xs:element name="openedAt"  type="xs:date"/>
      <xs:element name="active"    type="xs:boolean"/>
      <xs:element name="note"      type="Note" minOccurs="0"/>
    </xs:sequence>
  </xs:complexType>
</xs:element>

<xs:simpleType name="AccountId">
  <xs:restriction base="xs:string"><xs:maxLength value="35"/></xs:restriction>
</xs:simpleType>
<xs:simpleType name="Money">
  <xs:restriction base="xs:decimal">
    <xs:totalDigits value="18"/><xs:fractionDigits value="2"/>
  </xs:restriction>
</xs:simpleType>
<xs:simpleType name="Note">
  <xs:restriction base="xs:string"><xs:maxLength value="140"/></xs:restriction>
</xs:simpleType>

Its table, written by hand, in the dialect the store actually runs on (migrations/accounts/V001__accounts.sql) — store_key, rev and updated_at are the engine’s control columns, alongside one column per declared field:

CREATE TABLE IF NOT EXISTS accounts (
  store_key   VARCHAR(512)  NOT NULL,
  account_id  VARCHAR(35)   NOT NULL,
  balance     NUMERIC(18,2) NOT NULL DEFAULT 0,
  opened_at   DATE          NOT NULL,
  active      BOOLEAN       NOT NULL DEFAULT TRUE,
  note        VARCHAR(140),              -- the optional field's column admits NULL
  rev         BIGINT        NOT NULL DEFAULT 1,
  updated_at  TIMESTAMPTZ   NOT NULL DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY (store_key)                -- the store key, never a declared field
);

CREATE INDEX IF NOT EXISTS accounts_opened_at ON accounts (opened_at);

And the declaration that ties them together — no columns: block, because every field folds to the name the DDL already uses:

datastores:
  - name: accounts
    type: sql
    structure:
      schema: urn:accounts
      type: AccountRecord
    sql:
      url-ref: env:ACCOUNTS_DB_URL
      migrations: migrations/accounts
    dataClass: financial

sutra lint raises nothing on this package: every declared field has a column, every column’s type holds the declared facet range (VARCHAR(35) for maxLength="35", NUMERIC(18,2) for 18 total / 2 fractional digits), the one optional field maps to a nullable column, the three control columns are all present, and the table’s primary key is store_key itself — never a declared field — which is what lint requires of every projected table (KEY_MISMATCH otherwise: a business key would let two store keys collide on one row).

What lands in the database is a table anything can read:

store_keyaccount_idbalanceopened_atactivenoterevupdated_at
alicealice100.002026-01-01trueNULL12026-01-01T00:00:00Z

Round-tripping is lexical, not merely typed: every value travels as text in both directions — bound with the dialect’s write-side cast, read back through its canonical text rendering — so a put followed by a get returns the same shape it wrote, including a decimal’s written scale, decided by your own NUMERIC(18,2) rather than by anything in between. That holds exactly for xs:string, xs:decimal/the integer family, xs:boolean, xs:date, and an absent optional field — and an explicit JSON null on an optional field stores and reads back exactly like an absent one: both are NULL in the column, and both come back with the key omitted, never an explicit null.

xs:dateTime and xs:time are the one exception to “lexical”: the column stores an instant, not the string you sent, so a get returns the dialect’s own rendering of that instant, normalised toward ISO-8601 (a T separator, an explicit +HH:MM offset) — not necessarily the offset or the sub-second digits you wrote. Write 2026-08-04T10:00:00.000-05:00 into a dateTime column and expect back whatever your dialect renders that same instant as (typically UTC, for a timezone-aware column type), not the literal string. forUpdate, expect="unchanged" and the revision bookkeeping behave exactly as they do for an opaque store.

If AccountRecord later grows an owner sub-record, the package stops linting with STRUCTURE_NOT_FLAT — an explicit, package-time decision (flatten it, or give up projection for this store) rather than a silent shape change. The evolution rules are in Deployment packages.

Querying rows by key

Since 0.6.10 (typed-column stores since 0.6.13). A data-task read can return every row of a store whose key starts with a prefix, or lies in a range, together with aggregates over all of those rows. It suits append-only data — one row per event, keyed so that the rows you report on together share a prefix — where a single summary row would be a write hot spot.

<bpmn:dataStoreReference id="refDay" dataStoreRef="readingsStore">
  <bpmn:extensionElements>
    <q:store key="payload.meter + '|' + payload.day + '|'" query="prefix"
             limit="2" after="payload.after">
      <q:aggregate fn="sum" field="kwh"/>
      <q:aggregate fn="sum" field="kwh" groupBy="kind"/>
    </q:store>
  </bpmn:extensionElements>
</bpmn:dataStoreReference>
Attribute / elementMeaning
query="prefix"keys that start with key (an empty prefix is refused: it would read the whole store)
query="range", tokeys from key (inclusive) to to (exclusive); both are FEEL strings
limitrows per page, default 1000, maximum 10 000
afterFEEL: only keys after this one — the previous page’s next, or null for the first page
rows="false"aggregates only
<q:aggregate fn="count">{count} over every matching row
<q:aggregate fn="sum" field="…">{count, sum}; field is a dotted path into the stored value
groupBy="…"{groups: {"<value>": {count, sum?}}}, one cell per value of that path

The read’s target variable receives:

{"rows": [{"key": "m-7|2026-10-30|0001", "value": {"kind": "import", "kwh": "1.25"}}, ...],
 "aggregates": [{"count": 5, "sum": 7.55},
                {"groups": {"export": {"count": 2, "sum": 1.15}, "import": {"count": 3, "sum": 6.40}}}],
 "truncated": true, "next": "m-7|2026-10-30|0002"}
  • Keys compare bytewise, so m-7|2026-10-01 … m-7|2026-10-31 covers a month and a | separator behaves predictably. Rows come back in ascending key order. truncated says more rows follow, and next is the key to pass as after.
  • Aggregates cover every matching row, whatever limit and after say, so every page reports the same totals.
  • Sums are exact decimals, never floats. A sum adds JSON numbers and strings that hold a decimal literal ("30.05"). A row whose field is missing or null is counted but not summed. Any other value — text, a boolean, an object — fails the task with SUTRA.RUNTIME.DATASTORE.QUERY_NOT_NUMERIC. A row without the groupBy field groups under "null". No matching row gives {count: 0, sum: 0} or {groups: {}}, never null.
  • It is a plain read. It takes no locks, reads committed rows, never uses the read cache, and does not see the uncommitted writes of an enclosing <bpmn:transaction>. It cannot be combined with forUpdate, cannot be a write, and runs on sql stores — key/value or typed-column (see Querying a typed-column store).

The index a key query needs

A key query is one range scan of an index whose first two columns are (store_name, store_key), with store_key in a bytewise collation. Declare both in the store’s migration:

CREATE TABLE IF NOT EXISTS data_store (
  tenant_id       VARCHAR(64)  NOT NULL,   -- filled by the engine on insert
  module_id       VARCHAR(256) NOT NULL,
  module_version  VARCHAR(64)  NOT NULL,
  store_name      VARCHAR(128) NOT NULL,
  store_key       VARCHAR(512) COLLATE "C" NOT NULL,
  store_value     TEXT         NOT NULL,
  rev             BIGINT       NOT NULL DEFAULT 1,
  updated_at      TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY (store_name, store_key)
);

On MySQL/MariaDB use a …_bin collation for store_key (ascii_bin, utf8mb4_bin); on SQL Server an NVARCHAR key with a …_BIN2 collation. Before a store’s first query the engine checks both and refuses rather than scanning: SUTRA.RUNTIME.DATASTORE.QUERY_KEY_ORDER names a key column that does not sort bytewise, SUTRA.RUNTIME.DATASTORE.QUERY_KEY_INDEX a table with no such index — for example one keyed (tenant_id, module_id, module_version, store_name, store_key). Each message carries the DDL that fixes it. Stores that never run a key query are unaffected.

On MySQL a JSON number is held as a double, exact to about 15 significant digits; store amounts with more digits as strings there. PostgreSQL keeps the inputs’ decimal scale (6.40); MySQL and SQL Server report the sum without trailing zeros.

Querying a typed-column store

Since 0.6.13. A store that declares a structure: is queried with the same <q:store query=…>, over its own table. Each row’s value is the typed record, exactly as a plain read of that key returns it, and the aggregates are SQL COUNT and SUM over the declared columns, with GROUP BY the grouping column. For the accounts store from Typed columns:

<q:store key="'branch-7|'" query="prefix" limit="100" after="payload.after">
  <q:aggregate fn="sum" field="balance" groupBy="active"/>
  <q:aggregate fn="count" groupBy="note"/>
</q:store>
{"rows": [{"key": "branch-7|alice",
           "value": {"accountId": "alice", "balance": 100.50, "openedAt": "2026-01-01", "active": true}}],
 "aggregates": [{"groups": {"true": {"count": 2, "sum": 250.50}, "false": {"count": 1, "sum": 0.00}}},
                {"groups": {"null": {"count": 3}}}],
 "truncated": false, "next": null}
  • field and groupBy name one declared field — balance, not balance.value, and not a field the structure does not declare. sutra lint refuses either, naming the structure and its fields; at run time the same mistake fails with SUTRA.RUNTIME.DATASTORE.QUERY_UNSUPPORTED.
  • A summed field must be numeric — declared xs:decimal or an integer type. sutra lint refuses a sum over a string, date or boolean field. (If one reaches the engine anyway, every present value counts as not numeric and the task fails with SUTRA.RUNTIME.DATASTORE.QUERY_NOT_NUMERIC, as on a key/value store.)
  • Sums are exact. They are computed on the column itself, so a NUMERIC(18,2) sums to two decimal places on PostgreSQL. MySQL and SQL Server report the sum without trailing zeros.
  • An absent optional field is a NULL column. A sum skips it (the row still counts), and it groups under "null", just as a missing field does on a key/value store. Group keys read the way the record does: "true"/"false" for a boolean, the decimal text for a number, yyyy-mm-dd for a date.

The index requirement is the same idea with one column: the table’s first index column must be store_key, in a bytewise collation. A typed table normally already has PRIMARY KEY (store_key); add the collation:

CREATE TABLE IF NOT EXISTS accounts (
  store_key   VARCHAR(512)  COLLATE "C" NOT NULL,   -- MySQL: … COLLATE utf8mb4_bin; SQL Server: NVARCHAR … COLLATE Latin1_General_100_BIN2
  account_id  VARCHAR(35)   NOT NULL,
  balance     NUMERIC(18,2) NOT NULL,
  opened_at   DATE          NOT NULL,
  active      BOOLEAN       NOT NULL,
  note        VARCHAR(140),
  rev         BIGINT        NOT NULL DEFAULT 1,
  updated_at  TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY (store_key)
);

The first query checks both and refuses with SUTRA.RUNTIME.DATASTORE.QUERY_KEY_ORDER or SUTRA.RUNTIME.DATASTORE.QUERY_KEY_INDEX, naming the table and the DDL that fixes it — for example a table whose primary key is (account_id, store_key).

Claiming a key once

Since 0.6.10. expect="absent" makes a write an insert that applies only when the key does not exist yet. The database decides, so of any number of instances — on any number of replicas — that write the same key, exactly one wins. (A forUpdate read of a missing key locks nothing, so “read, see nothing, write” cannot do this.) The losing write changes nothing and raises SUTRA.RUNTIME.DATASTORE.CONFLICT. Catch it with an error boundary on the data task to take a “someone else got there first” path; uncaught, it fails the instance:

<bpmn:error id="taken" errorCode="SUTRA.RUNTIME.DATASTORE.CONFLICT"/>

<bpmn:dataStoreReference id="refRun" dataStoreRef="runsStore">
  <bpmn:extensionElements><q:store key="'daily|' + day" expect="absent"/></bpmn:extensionElements>
</bpmn:dataStoreReference>

<bpmn:serviceTask id="Claim">           <!-- writes `claim` to refRun -->
  <bpmn:dataOutputAssociation>
    <bpmn:sourceRef>claim</bpmn:sourceRef><bpmn:targetRef>refRun</bpmn:targetRef>
  </bpmn:dataOutputAssociation>
</bpmn:serviceTask>
<bpmn:boundaryEvent id="AlreadyClaimed" attachedToRef="Claim">
  <bpmn:errorEventDefinition errorRef="taken"/>
</bpmn:boundaryEvent>

The same conflict from expect="unchanged" is catchable the same way.

Walkthrough: readings by meter and day

A package with two processes: record-reading writes one row per reading with expect="absent" (so a resent reading is refused), and day-report runs the key query above and answers with a template. The files:

readings/
├── package.yaml            # labels: default / readings / 1.0.0
├── datastores.yaml         # store `readings`, env-indirected connection, migrations/readings
├── migrations/readings/V001__data_store.sql   # the table above
├── channels.yaml           # reading-in, day-report: HTTP + json + apikey
├── bpmn/readings.bpmn
└── templates/day-report.hbs

record-reading keys the row by meter, day and sequence number:

<q:store key="payload.meter + '|' + payload.day + '|' + payload.seq" expect="absent"/>

day-report reads the day into report, then picks the parts the answer shows:

<bpmn:serviceTask id="Query" name="Read the day">
  <bpmn:dataInputAssociation>
    <bpmn:sourceRef>refDay</bpmn:sourceRef><bpmn:targetRef>report</bpmn:targetRef>
  </bpmn:dataInputAssociation>
  <bpmn:dataInputAssociation>
    <bpmn:assignment><bpmn:from>report.aggregates[1]</bpmn:from><bpmn:to>total</bpmn:to></bpmn:assignment>
  </bpmn:dataInputAssociation>
  <bpmn:dataInputAssociation>
    <bpmn:assignment><bpmn:from>report.aggregates[2].groups</bpmn:from><bpmn:to>byKind</bpmn:to></bpmn:assignment>
  </bpmn:dataInputAssociation>
  <bpmn:dataInputAssociation>
    <bpmn:assignment><bpmn:from>for row in report.rows return row.key</bpmn:from><bpmn:to>keys</bpmn:to></bpmn:assignment>
  </bpmn:dataInputAssociation>
</bpmn:serviceTask>
{"keys": [{{#each keys}}{{#if @index}}, {{/if}}"{{this}}"{{/each}}],
 "readings": {{total.count}}, "kwh": {{total.sum}},
 "importKwh": {{byKind.import.sum}}, "exportKwh": {{byKind.export.sum}},
 "next": {{#if report.next}}"{{report.next}}"{{else}}null{{/if}}}

Package it and start PostgreSQL and the engine (the engine refuses a superuser database role, so it gets its own):

sutra package readings --out deployments
docker network create readings-net
docker run -d --name readings-db --network readings-net -e POSTGRES_PASSWORD=dev-only postgres:16-alpine
docker exec readings-db psql -U postgres \
  -c "CREATE ROLE sutra_engine LOGIN PASSWORD 'dev-only' NOSUPERUSER NOBYPASSRLS" \
  -c "ALTER SCHEMA public OWNER TO sutra_engine"
docker run -d --name readings-engine --network readings-net -p 127.0.0.1::8080 \
  -v "$PWD/deployments:/deployments:ro" -e SUTRA_DEPLOYMENTS_DIR=/deployments \
  -e SUTRA_DATASOURCE_URL=postgres://readings-db:5432/postgres \
  -e SUTRA_DATASOURCE_USERNAME=sutra_engine -e SUTRA_DATASOURCE_PASSWORD=dev-only \
  -e SUTRA_ADMIN_OIDC_DEV_DISABLED=true \
  -e READINGS_DB_URL=postgres://readings-db:5432/postgres \
  -e READINGS_DB_USER=sutra_engine -e READINGS_DB_PASSWORD=dev-only \
  -e READINGS_API_KEY=dev-only-readings-key ghcr.io/startr-trade/sutra:0.6.13
PORT=$(docker port readings-engine 8080 | head -1 | cut -d: -f2)
H=(-H 'Content-Type: application/json' -H 'X-Api-Key: dev-only-readings-key')

Record five readings for meter m-7 on 2026-10-30 (amounts as decimal strings or numbers, both sum exactly) and one on the next day:

for r in '"0001","kwh":"1.25","kind":"import"' '"0002","kwh":0.75,"kind":"export"' \
         '"0003","kwh":"2.10","kind":"import"' '"0004","kwh":"0.40","kind":"export"' \
         '"0005","kwh":"3.05","kind":"import"'; do
  curl -s -o /dev/null -w "%{http_code}\n" -X POST "http://127.0.0.1:$PORT/channels/reading-in" "${H[@]}" \
    -d "{\"meter\":\"m-7\",\"day\":\"2026-10-30\",\"seq\":$r}"
done
curl -s -o /dev/null -w "%{http_code}\n" -X POST "http://127.0.0.1:$PORT/channels/reading-in" "${H[@]}" \
  -d '{"meter":"m-7","day":"2026-10-31","seq":"0001","kwh":"9.99","kind":"import"}'

Each answers 200. Resending reading 0003 is refused, because the key exists:

{"type":"urn:bpm:diag:SUTRA.RUNTIME.DATASTORE.CONFLICT", ... "detail":"Optimistic-concurrency
conflict writing data store 'readings' key 'm-7|2026-10-30|0003' at data task 'Record': the key
already exists (expect=\"absent\"). ..."}

Ask for the day, two rows per page, passing each page’s next as the next after:

curl -s -X POST "http://127.0.0.1:$PORT/channels/day-report" "${H[@]}" \
  -d '{"meter":"m-7","day":"2026-10-30","after":null}'
{"keys": ["m-7|2026-10-30|0001", "m-7|2026-10-30|0002"],
 "readings": 5, "kwh": 7.55,
 "importKwh": 6.40, "exportKwh": 1.15,
 "next": "m-7|2026-10-30|0002"}

With "after":"m-7|2026-10-30|0002" the keys are 0003 and 0004 (next …|0004); with "after":"m-7|2026-10-30|0004" only 0005, and "next": null. The totals are the same on every page, and the 2026-10-31 reading is never counted. Clean up with docker rm -f readings-engine readings-db && docker network rm readings-net.

Deleting a row

Since 0.7.0. delete="true" on a write removes the key’s row instead of writing a value. The reference is the targetRef of the data task’s dataOutputAssociation, as for any write. The association needs no <sourceRef>, and one that is given is ignored.

<bpmn:dataStoreReference id="refRemove" dataStoreRef="itemsStore">
  <bpmn:extensionElements><q:store key="payload.itemId" delete="true"/></bpmn:extensionElements>
</bpmn:dataStoreReference>

<bpmn:serviceTask id="Remove">
  <bpmn:dataOutputAssociation><bpmn:targetRef>refRemove</bpmn:targetRef></bpmn:dataOutputAssociation>
</bpmn:serviceTask>
FormRow presentRow missing
delete="true"deletedno-op
delete="true" expect="unchanged", key read earlier in the instancedeleted while its revision is still the one read, otherwise a conflicta conflict (the row is gone)
delete="true" expect="unchanged", key never read, or read as missinga conflict (the row appeared)no-op

The conflict is SUTRA.RUNTIME.DATASTORE.CONFLICT. An error boundary on the data task catches it, as it catches a lost claim.

Read, check, delete. A data task runs its reads before its writes. So one task can read a key through one reference and delete it through another with expect="unchanged". If anything changes the row between the read and the delete, the delete does not apply.

<bpmn:dataStoreReference id="refItem" dataStoreRef="itemsStore">
  <bpmn:extensionElements><q:store key="payload.itemId"/></bpmn:extensionElements>
</bpmn:dataStoreReference>
<bpmn:dataStoreReference id="refRemove" dataStoreRef="itemsStore">
  <bpmn:extensionElements><q:store key="payload.itemId" delete="true" expect="unchanged"/></bpmn:extensionElements>
</bpmn:dataStoreReference>

<bpmn:serviceTask id="Remove">
  <bpmn:dataInputAssociation><bpmn:sourceRef>refItem</bpmn:sourceRef><bpmn:targetRef>item</bpmn:targetRef></bpmn:dataInputAssociation>
  <bpmn:dataOutputAssociation><bpmn:targetRef>refRemove</bpmn:targetRef></bpmn:dataOutputAssociation>
</bpmn:serviceTask>

A delete follows the same rules as a write of the same key:

  • Transactions. Inside a <bpmn:transaction> the delete runs on the scope’s store transaction, so it commits or rolls back with the scope. Outside one it commits on its own.
  • Locks. The DELETE takes the row lock itself. It waits behind a forUpdate read of the same key in another instance until that instance commits.
  • The read cache. A delete through the engine invalidates the key at once. Other replicas see the row gone on their next read, because a deleted row no longer has the revision they cached.
  • Store shapes and databases. Deletes work on key/value and typed-column stores, on PostgreSQL, MySQL, MariaDB and SQL Server.

delete cannot be combined with field, forUpdate, expect="absent", query, to, rows, limit, after or a <q:aggregate>. A delete reference cannot be the source of a read either. sutra lint, sealing and deploy all refuse these with SUTRA.PARSE.STORE.DELETE_INVALID.

A re-created row never reuses a revision. A row’s revision starts at the database’s own clock (microseconds since the epoch, read in the insert itself, so all replicas share one clock) and goes up by one with each write. Revisions are strictly increasing per row but not dense. A key that is deleted and created again therefore starts above every revision the old row had. A replica that cached the old row reads the new one, and an expect="unchanged" write or delete that read the old row is refused. This relies on the database clock moving forward: if it is stepped backwards between the delete and the re-create, the guarantee can break.

Walkthrough: removing an item seen through two replicas

A package with an items store that keeps a read cache, a results store, and three processes, each started over HTTP: item-write writes a row, item-copy reads one row and copies it into results, and item-delete runs the read-check-delete above. Two engines share one database, so one can delete a row the other has cached. The files:

items/
├── package.yaml            # labels: default / items / 1.0.0
├── datastores.yaml         # stores `items` (cache: {enabled: true}) and `results`, env-indirected
├── migrations/items/V001__data_store.sql     # the key/value table, as in Querying rows by key
├── migrations/results/V001__data_store.sql   # the same table
├── channels.yaml           # item-in, copy-in, delete-in: HTTP + json + apikey
└── bpmn/items.bpmn

item-copy copies the row into results under copy|<key>|<tag>:

<bpmn:dataStoreReference id="refItem" dataStoreRef="itemsStore">
  <bpmn:extensionElements><q:store key="payload.key"/></bpmn:extensionElements>
</bpmn:dataStoreReference>
<bpmn:dataStoreReference id="refCopy" dataStoreRef="resultsStore">
  <bpmn:extensionElements><q:store key="'copy|' + payload.key + '|' + payload.tag"/></bpmn:extensionElements>
</bpmn:dataStoreReference>

<bpmn:serviceTask id="Copy" name="Copy the item">
  <bpmn:dataInputAssociation>
    <bpmn:sourceRef>refItem</bpmn:sourceRef><bpmn:targetRef>copied</bpmn:targetRef>
  </bpmn:dataInputAssociation>
  <bpmn:dataOutputAssociation>
    <bpmn:sourceRef>copied</bpmn:sourceRef><bpmn:targetRef>refCopy</bpmn:targetRef>
  </bpmn:dataOutputAssociation>
</bpmn:serviceTask>

Package it, then start PostgreSQL and two engines, items-a and items-b:

sutra package items --out deployments
docker network create items-net
docker run -d --name items-db --network items-net -e POSTGRES_PASSWORD=dev-only postgres:16-alpine
until docker exec items-db pg_isready -U postgres -h 127.0.0.1; do sleep 1; done; sleep 2
docker exec items-db psql -U postgres \
  -c "CREATE ROLE sutra_engine LOGIN PASSWORD 'dev-only' NOSUPERUSER NOBYPASSRLS" \
  -c "ALTER SCHEMA public OWNER TO sutra_engine"
for r in a b; do
  docker run -d --name items-$r --network items-net -p 127.0.0.1::8080 \
    -v "$PWD/deployments:/deployments:ro" -e SUTRA_DEPLOYMENTS_DIR=/deployments \
    -e SUTRA_DATASOURCE_URL=postgres://items-db:5432/postgres \
    -e SUTRA_DATASOURCE_USERNAME=sutra_engine -e SUTRA_DATASOURCE_PASSWORD=dev-only \
    -e SUTRA_ADMIN_OIDC_DEV_DISABLED=true \
    -e ITEMS_DB_URL=postgres://items-db:5432/postgres \
    -e ITEMS_DB_USER=sutra_engine -e ITEMS_DB_PASSWORD=dev-only \
    -e ITEMS_API_KEY=dev-only-items-key ghcr.io/startr-trade/sutra:0.8.2
  until curl -sf -o /dev/null "http://127.0.0.1:$(docker port items-$r 8080 | head -1 | cut -d: -f2)/sutra/health/ready"; do
    sleep 1
  done
done
A=http://127.0.0.1:$(docker port items-a 8080 | head -1 | cut -d: -f2)
B=http://127.0.0.1:$(docker port items-b 8080 | head -1 | cut -d: -f2)
C=(-s -o /dev/null -w "%{http_code}\n" -H 'Content-Type: application/json' -H 'X-Api-Key: dev-only-items-key')
sql() { docker exec items-db psql -U postgres -tAc "$1"; }
  1. Write an item through items-a. Its revision is the database clock at the insert:

    curl "${C[@]}" -X POST "$A/channels/item-in" -d '{"key":"item-0001","entry":{"amount":"12.50"}}'
    sql "SELECT store_value, rev FROM data_store WHERE store_name='items' AND store_key='item-0001'"
    
    200
    {"amount":"12.50"}|1795060962963065
    
  2. Read it through items-b, which now holds it in its cache:

    curl "${C[@]}" -X POST "$B/channels/copy-in" -d '{"key":"item-0001","tag":"before"}'
    sql "SELECT store_value FROM data_store WHERE store_name='results' AND store_key='copy|item-0001|before'"
    
    200
    {"amount":"12.50"}
    
  3. Delete it through items-a:

    curl "${C[@]}" -X POST "$A/channels/delete-in" -d '{"key":"item-0001"}'
    sql "SELECT count(*) FROM data_store WHERE store_name='items' AND store_key='item-0001'"
    
    200
    0
    
  4. Read it through items-b again. Its cached entry’s revision no longer matches (the row is gone), so it serves nothing stale:

    curl "${C[@]}" -X POST "$B/channels/copy-in" -d '{"key":"item-0001","tag":"after"}'
    sql "SELECT store_value FROM data_store WHERE store_name='results' AND store_key='copy|item-0001|after'"
    
    200
    null
    
  5. Create the item again with another value through items-a, and read it through items-b. The new row starts above every revision the old one had, so items-b reads the new value:

    curl "${C[@]}" -X POST "$A/channels/item-in" -d '{"key":"item-0001","entry":{"amount":"2.00"}}'
    sql "SELECT store_value, rev FROM data_store WHERE store_name='items' AND store_key='item-0001'"
    curl "${C[@]}" -X POST "$B/channels/copy-in" -d '{"key":"item-0001","tag":"again"}'
    sql "SELECT store_value FROM data_store WHERE store_name='results' AND store_key='copy|item-0001|again'"
    
    200
    {"amount":"2.00"}|1795060963446990
    200
    {"amount":"2.00"}
    
  6. Delete it twice, once through each engine. Both answer 200: the second delete is of a missing key, a no-op.

    curl "${C[@]}" -X POST "$A/channels/delete-in" -d '{"key":"item-0001"}'
    curl "${C[@]}" -X POST "$B/channels/delete-in" -d '{"key":"item-0001"}'
    
  7. Change the delete reference to <q:store key="payload.key" delete="true" forUpdate="true"/> and lint the package with sutra lint items. It is refused before anything runs:

    [ERROR] SUTRA.DEPLOY.PACKAGE.CONFIG_INVALID — BPMN bpmn/items.bpmn fails to parse: [SUTRA.PARSE.STORE.DELETE_INVALID] <serviceTask> 'Remove' declares a delete on data store 'items' that cannot run: @forUpdate locks a row for a read; a delete takes the row lock itself (lock it with a forUpdate read through another reference). (items/bpmn/items.bpmn)
    

Revisions differ on every run. Clean up with docker rm -f items-a items-b items-db && docker network rm items-net.

Caching reads

Since 0.6.5. A data task reads a value and its revision in one round trip. A store that is read far more often than it is written — settings, reference data — can also keep a read cache:

datastores:
  - name: refdata
    type: sql
    sql:
      url-ref: env:REF_DB_URL
      username-ref: env:REF_DB_USER
      password-ref: env:REF_DB_PASSWORD
    cache:
      enabled: true      # default false
      maxEntries: 1000   # keys kept; the least recently read is evicted first
      ttl: PT0S          # default: check the revision on every read
ttlA read of a cached keyA write made through any engineA write made by SQL outside the engine
PT0S (default)one revision check (SELECT rev) instead of a full readseen by the next read on every replica: each write bumps the row’s revisionseen by the next read if the SQL bumps rev, otherwise only once the key is evicted
positive, e.g. PT5Sno query at all while the key was checked less than ttl agothrough this engine: seen at once; through another replica: within ttlwithin ttl

The cache is shared by every lane of the engine process and is per deployment. A write or delete made through the engine — a data task’s, or one inside a <bpmn:transaction> — invalidates its key at once. Every other replica sees it on its next read, a key deleted and created again included: the re-created row never reuses a revision (see Deleting a row). Reads inside a transaction always go to the database, because they must see the transaction’s own snapshot and locks. Keep ttl at PT0S unless a few seconds of staleness after an operator’s edit is acceptable. When you change rows by hand, bump rev with them:

UPDATE data_store SET store_value = '{"maxAmount": 500}', rev = rev + 1
 WHERE store_name = 'refdata' AND store_key = 'default';

When the database cannot be reached

A data task whose database cannot be reached (a dropped connection, a pool timeout, a server restarting) fails with SUTRA.RUNTIME.DATASTORE.UNAVAILABLE. That is not a failed instance: the step is not committed and the delivery is refused for a retry (HTTP 503, a broker redelivery, the outbox’s backoff for a hop between processes, the timer poller’s backoff), whatever the process declares about idempotency. It is never dead-lettered. (Since 0.8.2.)

Isolation and atomicity

Two separate guarantees, worth keeping distinct:

  • Isolation (no lost update between concurrent writers) comes from serializing access to a key — either a pessimistic forUpdate="true" lock, or funneling all writers for that channel through a single active consumer (see the singleton channel property in Channels and transports).
  • Atomicity (all writes commit or none do) comes from a <bpmn:transaction> scope: writes inside it commit together on a normal end, or roll back together on a cancel end / error. An external store write cannot auto-rollback on its own — the transaction sub-process is what gives you the boundary.

The worked example combines both: a singleton channel serializes writers, and the debit/credit pair runs inside a <bpmn:transaction> so a rejected transfer touches no row.

Keeping a message as text

A flow can keep a message exactly as it arrived, as text in a typed store column, and work with it decoded again later. This walkthrough keeps notes in a TEXT column, reads one back through <q:decode>, and then masks them.

You need the engine and PostgreSQL; this walkthrough was run on PostgreSQL 16. xsd/q.xsd is the reference for the element.

1. A typed store with a text column

package.yaml labels the module notes, version 1.0.0. datastores.yaml gives the store its row type:

datastores:
  - name: notes
    type: sql
    structure:
      schema: urn:note-row
      type: NoteRow
    sql:
      url-ref: env:NOTES_DB_URL
      username-ref: env:NOTES_DB_USER
      password-ref: env:NOTES_DB_PASSWORD
      migrations: migrations/notes

schemas/note-row/codec-manifest.yaml is schemaKind: xsd and formats: [xml]. schemas/note-row/note-row.xsd declares the row. body has no length limit:

<xs:element name="NoteRow">
  <xs:complexType>
    <xs:sequence>
      <xs:element name="noteId"      type="NoteId"/>    <!-- maxLength 64 -->
      <xs:element name="contentType" type="MediaType" minOccurs="0"/>  <!-- maxLength 128 -->
      <xs:element name="body"        type="xs:string"/>
    </xs:sequence>
  </xs:complexType>
</xs:element>

migrations/notes/V001__notes.sql:

CREATE TABLE IF NOT EXISTS notes (
  store_key    VARCHAR(512) NOT NULL,
  note_id      VARCHAR(64)  NOT NULL,
  content_type VARCHAR(128),
  body         TEXT         NOT NULL,
  rev          BIGINT       NOT NULL DEFAULT 1,
  updated_at   TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY (store_key)
);

On MySQL, TEXT stops at 64 KiB, so use MEDIUMTEXT or LONGTEXT. On SQL Server, use NVARCHAR(MAX).

2. Capture the text

channels.yaml declares two HTTP channels with the usual API-key auth. note-in has no codec, so it takes any format, and its key comes from a header:

channels:
  - name: note-in
    transport: http
    bind: "POST /channels/note-in"
    idempotency-key-header: X-Note-Id
    cloudevents-mode: none
    auth: { scheme: apikey, apikey: { value: env:NOTES_API_KEY, header: X-Api-Key } }
  - name: note-read
    transport: http
    bind: "POST /channels/note-read"
    codec: json
    cloudevents-mode: none
    auth: { scheme: apikey, apikey: { value: env:NOTES_API_KEY, header: X-Api-Key } }

bpmn/note-keep.bpmn builds the row from event on the step right after the start. A later message would replace event, so capture here. The gateway is step 6:

<bpmn:dataStore id="notesStore" name="notes"/>
<bpmn:process id="note-keep" isExecutable="true">
  <bpmn:extensionElements>
    <q:variables><q:variable name="row" type="object"/></q:variables>
  </bpmn:extensionElements>
  <bpmn:dataStoreReference id="refRow" dataStoreRef="notesStore">
    <bpmn:extensionElements><q:store key="event.dedupKey"/></bpmn:extensionElements>
  </bpmn:dataStoreReference>
  <bpmn:startEvent id="Start">
    <bpmn:extensionElements><q:source channel="note-in"/></bpmn:extensionElements>
  </bpmn:startEvent>
  <bpmn:exclusiveGateway id="IsText" default="ToKeep"/>
  <bpmn:serviceTask id="Keep" name="Keep the text">
    <bpmn:dataInputAssociation>
      <bpmn:assignment>
        <bpmn:from>{noteId: event.dedupKey, contentType: event.contentType, body: event.body}</bpmn:from>
        <bpmn:to>row</bpmn:to>
      </bpmn:assignment>
    </bpmn:dataInputAssociation>
    <bpmn:dataOutputAssociation>
      <bpmn:sourceRef>row</bpmn:sourceRef><bpmn:targetRef>refRow</bpmn:targetRef>
    </bpmn:dataOutputAssociation>
  </bpmn:serviceTask>
  <bpmn:endEvent id="Kept"/>
  <bpmn:endEvent id="NotText"/>
  <bpmn:sequenceFlow id="F1" sourceRef="Start" targetRef="IsText"/>
  <bpmn:sequenceFlow id="ToKeep" sourceRef="IsText" targetRef="Keep"/>
  <bpmn:sequenceFlow id="ToRefuse" sourceRef="IsText" targetRef="NotText">
    <bpmn:conditionExpression>event.bodyLossy = true</bpmn:conditionExpression>
  </bpmn:sequenceFlow>
  <bpmn:sequenceFlow id="F2" sourceRef="Keep" targetRef="Kept"/>
</bpmn:process>

3. A retrieval flow

bpmn/note-read.bpmn reads the row, decodes its text by the row’s own content type, and answers from the decoded fields. The store read runs first, then <q:decode>, within one data task. Text that doesn’t parse raises SUTRA.RUNTIME.DECODE.FAILED, which the boundary event catches:

<bpmn:error id="undecodable" errorCode="SUTRA.RUNTIME.DECODE.FAILED"/>
<bpmn:dataStore id="notesStore" name="notes"/>
<bpmn:process id="note-read" isExecutable="true">
  <bpmn:extensionElements>
    <q:variables>
      <q:variable name="row" type="object"/>
      <q:variable name="note" type="object"/>
    </q:variables>
  </bpmn:extensionElements>
  <bpmn:dataStoreReference id="refRead" dataStoreRef="notesStore">
    <bpmn:extensionElements><q:store key="req.id"/></bpmn:extensionElements>
  </bpmn:dataStoreReference>
  <bpmn:startEvent id="Start">
    <bpmn:extensionElements><q:source channel="note-read" name="req"/></bpmn:extensionElements>
  </bpmn:startEvent>
  <bpmn:serviceTask id="Load" name="Read and decode">
    <bpmn:extensionElements>
      <q:decode name="note" expression="row.body" codec="json" contentType="row.contentType"/>
    </bpmn:extensionElements>
    <bpmn:dataInputAssociation>
      <bpmn:sourceRef>refRead</bpmn:sourceRef><bpmn:targetRef>row</bpmn:targetRef>
    </bpmn:dataInputAssociation>
  </bpmn:serviceTask>
  <bpmn:boundaryEvent id="BadText" attachedToRef="Load">
    <bpmn:errorEventDefinition errorRef="undecodable"/>
  </bpmn:boundaryEvent>
  <bpmn:serviceTask id="Answer" implementation="summary.hbs">
    <bpmn:extensionElements><q:reply mode="native" contentType="application/json"/></bpmn:extensionElements>
  </bpmn:serviceTask>
  <bpmn:serviceTask id="AnswerBad" implementation="undecodable.hbs">
    <bpmn:extensionElements><q:reply mode="native" contentType="application/json"/></bpmn:extensionElements>
  </bpmn:serviceTask>
  <bpmn:endEvent id="Done"/>
  <bpmn:endEvent id="DoneBad"/>
  <bpmn:sequenceFlow id="F1" sourceRef="Start" targetRef="Load"/>
  <bpmn:sequenceFlow id="F2" sourceRef="Load" targetRef="Answer"/>
  <bpmn:sequenceFlow id="F3" sourceRef="Answer" targetRef="Done"/>
  <bpmn:sequenceFlow id="F4" sourceRef="BadText" targetRef="AnswerBad"/>
  <bpmn:sequenceFlow id="F5" sourceRef="AnswerBad" targetRef="DoneBad"/>
</bpmn:process>

templates/summary.hbs:

{"decoded":true,"title":"{{note.title}}","pages":{{note.pages}}}

templates/undecodable.hbs:

{"decoded":false}

4. Package it and start the engine

sutra lint notes
sutra package notes -o deployments

Give the engine a role that can’t bypass row-level security, and a database that role owns:

CREATE ROLE sutra_app LOGIN NOSUPERUSER NOBYPASSRLS PASSWORD '<a password>';
CREATE DATABASE sutra OWNER sutra_app;

Start the engine with SUTRA_DATASOURCE_URL pointing at that database as sutra_app, SUTRA_DEPLOYMENTS_DIR pointing at deployments, NOTES_API_KEY, and NOTES_DB_URL, NOTES_DB_USER and NOTES_DB_PASSWORD for the store. Turn admin auth on with SUTRA_ADMIN_AUTH_SCHEME=apikey, SUTRA_ADMIN_AUTH_KEY_REF=env:SUTRA_ADMIN_API_KEY and SUTRA_ADMIN_API_KEY, because step 7 reads an instance. Set B to the engine’s address.

5. Keep a note and read it back

note.json has odd spacing, a non-ASCII title, and a number with a written scale:

{ "title" : "café notes",
  "pages": 3.50 }
curl -s -o /dev/null -w '%{http_code}\n' -X POST "$B/channels/note-in" \
  -H 'X-Api-Key: <your key>' -H 'Content-Type: application/json' -H 'X-Note-Id: n-1' \
  --data-binary @note.json
200

The column holds the exact text:

SELECT store_key, note_id, content_type, body FROM notes;
 store_key | note_id |   content_type   |           body
-----------+---------+------------------+---------------------------
 n-1       | n-1     | application/json | { "title" : "café notes",+
           |         |                  |   "pages": 3.50 }        +
           |         |                  |

Read it back:

curl -s -X POST "$B/channels/note-read" -H 'X-Api-Key: <your key>' \
  -H 'Content-Type: application/json' -d '{"id":"n-1"}'
{"decoded":true,"title":"café notes","pages":3.50}

The row’s content type picks the parser, so the same flow reads XML. Keep <note><title>second</title><pages>2</pages></note> as application/xml under n-2, and read it:

{"decoded":true,"title":"second","pages":2}

Keep { this is not json as application/json under n-3. It is stored as it came, and reading it takes the error boundary:

{"decoded":false}

6. A message that isn’t text

Send bytes that aren’t valid UTF-8:

printf 'ok\377\376' | curl -s -o /dev/null -w '%{http_code}\n' -X POST "$B/channels/note-in" \
  -H 'X-Api-Key: <your key>' -H 'Content-Type: application/octet-stream' -H 'X-Note-Id: n-4' \
  --data-binary @-
200

The invalid bytes became U+FFFD, so event.bodyLossy is true, and the gateway took ToRefuse. There is no n-4 row. A clean message has no bodyLossy key at all, which is why = true is the test. <q:decode> handles text only; there is no binary form.

7. Masking

Now suppose the titles are private. Add redactors/title.hbs, which holds the path to mask:

/title

Name it on the source. This walkthrough also adds a wait, so there is an instance to look at:

<bpmn:startEvent id="Start">
  <bpmn:extensionElements>
    <q:source channel="note-in"><q:redactors><q:redactor ref="title"/></q:redactors></q:source>
    <q:alias name="noteId" expression="event.dedupKey" unique="true"/>
  </bpmn:extensionElements>
</bpmn:startEvent>
<!-- after Keep -->
<bpmn:receiveTask id="Review" name="Wait for a review">
  <bpmn:extensionElements>
    <q:source channel="note-reviewed" name="review"/>
    <q:alias name="noteId" expression="review.id"/>
  </bpmn:extensionElements>
</bpmn:receiveTask>

sutra lint notes now finds the copies the engine can’t see through, including in note-read, which masks nothing itself but reads what a masking flow stored:

[WARN] SUTRA.CONFIG.BPMN.RAW_BODY_UNMASKED — ... process 'note-keep' declares masking, but variable 'row' holds raw event.body text and is not sensitive, so the raw message shows in audit and inspect and is stored unencrypted. Declare it <q:variable name="row" sensitive="true"/>. This becomes an error in a later release.
[WARN] SUTRA.CONFIG.BPMN.RAW_BODY_STORED — ... process 'note-keep' declares masking, but writes raw event.body text to data store 'notes' through variable 'row'. A store column has no at-rest encryption, and sensitive does not reach it.
[WARN] SUTRA.CONFIG.BPMN.RAW_BODY_UNMASKED — ... process 'note-read' reads text a masking process stored, and variable 'row' holds raw event.body text and is not sensitive, ...
[ERROR] SUTRA.CONFIG.DECODE.REDACTORS_MISSING — ... process 'note-read' reads text a masking process stored, and <q:decode name="note"> parses raw event.body text into a variable with no masking, so the message shows in audit and inspect. Declare it <q:variable name="note" sensitive="true"/>, or give the step <q:redactors>.

Declare row sensitive in both flows and note sensitive in note-read. Only RAW_BODY_STORED remains. It stays because storing the text is the point of this flow, and it is true: the column is cleartext.

Package it and restart the engine with SUTRA_CRYPTO_MASTER_KEY set, so that parked state is encrypted. Keep a note with a private title under n-5, and look at the parked instance. This route needs the admin key:

curl -s -H "X-API-Key: $SUTRA_ADMIN_API_KEY" "$B/sutra/instances?alias=noteId:n-5"
curl -s -H "X-API-Key: $SUTRA_ADMIN_API_KEY" "$B/sutra/instances/<instanceId>"
{"instanceId":"cb729345-dde3-46ab-94c6-0beaa6cbf5d3","deploymentId":"dep-a149aa3f6fbb7bc0a118c185","status":"SUSPENDED","completedNodes":["Keep"],"waitingNodes":["Review"],"sensitive":["row"],"variables":{"attr":"{}","event":"{body=[REDACTED], channel=note-in, contentType=application/json, dedupKey=[REDACTED], headers={accept=[REDACTED], content-length=[REDACTED], content-type=[REDACTED], host=[REDACTED], user-agent=[REDACTED], x-api-key=[REDACTED], x-note-id=[REDACTED]}, idempotencyKey=[REDACTED], receivedAt=2026-11-04T19:21:26.23034207Z}","header":"{accept=[REDACTED], content-length=[REDACTED], content-type=[REDACTED], host=[REDACTED], user-agent=[REDACTED], x-api-key=[REDACTED], x-note-id=[REDACTED]}","noteId":"n-5","payload":"{pages=1, title=[REDACTED]}","payload.validation":"[]","row":"***REDACTED***","validation":"{firstIssue=, firstReasonCode=, issues=[], outcome=OK, tier=n/a}"}}

The title appears nowhere in that answer, and the parked row in instance_state doesn’t contain it either. The notes column does, in the clear. To keep it out of the column, don’t store it in text.

The flow itself still reads the real values. Answer the wait on note-reviewed with {"id":"n-5"}, then read the note:

{"decoded":true,"title":"PRIVATE-TITLE-5","pages":1}

Answering with the title is this flow’s choice. A real one masking its titles would answer something else.

Limits

  • Size. <q:decode> refuses text over sutra.codec.max-payload-bytes with SUTRA.RUNTIME.DECODE.TOO_LARGE. Column limits are the database’s.
  • Encoding. A body whose prolog declares another encoding, such as ISO-8859-1 XML, can’t be kept faithfully. A NUL byte survives into event.body, but PostgreSQL TEXT refuses it, and that write fails the instance.
  • After a wait. A resuming message replaces event. Capture before the wait, as here.
  • Other packages. Lint follows a store by name within one package. Another package reading the same database isn’t seen.

Sensitivity

A store’s dataClass (pii / pci / phi / financial) marks its contents sensitive: values read from it are redacted wherever the engine emits observability data — audit events, structured logs, traces — while the flow itself still sees the real value. The tag travels with the variable a data association reads into, so it stays redacted downstream too.

Next

Signing a user in against a store

A flow can keep users in a typed store and check their passwords with two FEEL builtins, password hash and password verify (new in 0.6.21; reference in Rules: DMN, FEEL, and .srl). This walkthrough seeds one user from a migration, enrols a second through a flow, and signs both in. The password itself is never stored.

You need the engine and PostgreSQL. This walkthrough was run on PostgreSQL 16, with the output shown below.

The two builtins

FunctionReturns
password hash(password)An argon2id hash in PHC string form, $argon2id$v=19$m=19456,t=2,p=1$<salt>$<hash>, with a fresh 16-byte salt from the OS. null if the argument is not a string or is empty.
password verify(password, hash)true if the password matches the hash, false if it does not. null if either argument is not a string, or if the hash is malformed, is not argon2id, or asks for more than m = 65 536 KiB, t = 10 or p = 4. The comparison is constant-time.

The parameters are fixed: argon2id, m = 19 456 KiB, t = 2, p = 1, the OWASP minimum. A package can’t change them. verify reads the parameters from the stored hash, so the cap keeps a planted hash from tying up a worker and its memory.

password hash draws a random salt, so it is denied at replay-bound sites, like uuid(). Use it in a task, not in an alias or a when. password verify is pure and works anywhere.

FEEL evaluation blocks its executor worker, so one verify holds a worker for one hash time, about 22 ms on a 2020 laptop CPU (Intel i7-10750H). That is fine for sign-ins. Don’t call these functions per message.

1. A typed store for users

package.yaml labels the module signin, version 1.0.0. datastores.yaml gives the store its row type:

datastores:
  - name: users
    type: sql
    structure:
      schema: urn:user-row
      type: UserRow
    sql:
      url-ref: env:STORE_DB_URL
      username-ref: env:STORE_DB_USER
      password-ref: env:STORE_DB_PASSWORD
      migrations: migrations/users

schemas/user-row/codec-manifest.yaml is schemaKind: xsd and formats: [xml]. schemas/user-row/user-row.xsd declares the row:

<xs:element name="UserRow">
  <xs:complexType>
    <xs:sequence>
      <xs:element name="userId"       type="UserId"/>   <!-- maxLength 64 -->
      <xs:element name="passwordHash" type="PhcHash"/>  <!-- maxLength 256 -->
    </xs:sequence>
  </xs:complexType>
</xs:element>

migrations/users/V001__users.sql creates the table and seeds one user, ada. The hash is a fixed test vector for the password correct horse:

CREATE TABLE IF NOT EXISTS users (
  store_key     VARCHAR(512) NOT NULL,
  user_id       VARCHAR(64)  NOT NULL,
  password_hash VARCHAR(256) NOT NULL,
  rev           BIGINT       NOT NULL DEFAULT 1,
  updated_at    TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY (store_key)
);
INSERT INTO users (store_key, user_id, password_hash)
VALUES ('ada', 'ada',
        '$argon2id$v=19$m=19456,t=2,p=1$AAECAwQFBgcICQoLDA0ODw$0AuQxA+RjPiJeA19Op7DwrFxnfh9xeTU2T6mFj2wThc')
ON CONFLICT (store_key) DO NOTHING;

2. Two channels

channels.yaml declares two JSON channels with the usual API-key auth. Both take {"user": ..., "password": ...}:

channels:
  - name: enrol
    transport: http
    bind: "POST /channels/enrol"
    codec: json
    cloudevents-mode: none
    auth: { scheme: apikey, apikey: { value: env:SIGN_IN_API_KEY, header: X-Api-Key } }
  - name: sign-in
    transport: http
    bind: "POST /channels/sign-in"
    codec: json
    cloudevents-mode: none
    auth: { scheme: apikey, apikey: { value: env:SIGN_IN_API_KEY, header: X-Api-Key } }

3. Keep the password out of storage

Both flows bind the request to a variable named creds and declare it transient and sensitive. transient keeps it out of the instance store. sensitive redacts it, and the raw request body with it, in logs and audit. Neither flow waits, so creds is only needed for one turn.

bpmn/enrol.bpmn hashes the password and writes the row:

<bpmn:dataStore id="usersStore" name="users"/>
<bpmn:process id="enrol" isExecutable="true">
  <bpmn:extensionElements>
    <q:audit sink="jsonl" capture="payload"/>
    <q:variables>
      <q:variable name="creds" type="object" transient="true" sensitive="true"/>
      <q:variable name="row" type="object"/>
    </q:variables>
  </bpmn:extensionElements>
  <bpmn:dataStoreReference id="refUser" dataStoreRef="usersStore">
    <bpmn:extensionElements><q:store key="creds.user"/></bpmn:extensionElements>
  </bpmn:dataStoreReference>
  <bpmn:startEvent id="Start">
    <bpmn:extensionElements><q:source channel="enrol" name="creds"/></bpmn:extensionElements>
  </bpmn:startEvent>
  <bpmn:serviceTask id="Hash" name="Hash and store">
    <bpmn:dataInputAssociation>
      <bpmn:assignment>
        <bpmn:from>{userId: creds.user, passwordHash: password hash(creds.password)}</bpmn:from>
        <bpmn:to>row</bpmn:to>
      </bpmn:assignment>
    </bpmn:dataInputAssociation>
    <bpmn:dataOutputAssociation>
      <bpmn:sourceRef>row</bpmn:sourceRef><bpmn:targetRef>refUser</bpmn:targetRef>
    </bpmn:dataOutputAssociation>
  </bpmn:serviceTask>
  <bpmn:endEvent id="Done"/>
  <bpmn:sequenceFlow id="F1" sourceRef="Start" targetRef="Hash"/>
  <bpmn:sequenceFlow id="F2" sourceRef="Hash" targetRef="Done"/>
</bpmn:process>

bpmn/sign-in.bpmn reads the row, checks the password, and answers:

<bpmn:dataStore id="usersStore" name="users"/>
<bpmn:process id="sign-in" isExecutable="true">
  <bpmn:extensionElements>
    <q:audit sink="jsonl" capture="payload"/>
    <q:variables>
      <q:variable name="creds" type="object" transient="true" sensitive="true"/>
      <q:variable name="user" type="object"/>
      <q:variable name="signedIn" type="boolean"/>
    </q:variables>
  </bpmn:extensionElements>
  <bpmn:dataStoreReference id="refUser" dataStoreRef="usersStore">
    <bpmn:extensionElements><q:store key="creds.user"/></bpmn:extensionElements>
  </bpmn:dataStoreReference>
  <bpmn:startEvent id="Start">
    <bpmn:extensionElements><q:source channel="sign-in" name="creds"/></bpmn:extensionElements>
  </bpmn:startEvent>
  <bpmn:serviceTask id="Load" name="Read the user">
    <bpmn:dataInputAssociation>
      <bpmn:sourceRef>refUser</bpmn:sourceRef><bpmn:targetRef>user</bpmn:targetRef>
    </bpmn:dataInputAssociation>
  </bpmn:serviceTask>
  <bpmn:serviceTask id="Check" name="Check the password">
    <bpmn:dataInputAssociation>
      <bpmn:assignment>
        <bpmn:from>password verify(creds.password, user.passwordHash) = true</bpmn:from>
        <bpmn:to>signedIn</bpmn:to>
      </bpmn:assignment>
    </bpmn:dataInputAssociation>
  </bpmn:serviceTask>
  <bpmn:serviceTask id="Answer" implementation="signed-in.hbs">
    <bpmn:extensionElements><q:reply mode="native" contentType="application/json"/></bpmn:extensionElements>
  </bpmn:serviceTask>
  <bpmn:endEvent id="Done"/>
  <bpmn:sequenceFlow id="F1" sourceRef="Start" targetRef="Load"/>
  <bpmn:sequenceFlow id="F2" sourceRef="Load" targetRef="Check"/>
  <bpmn:sequenceFlow id="F3" sourceRef="Check" targetRef="Answer"/>
  <bpmn:sequenceFlow id="F4" sourceRef="Answer" targetRef="Done"/>
</bpmn:process>

The = true turns a null from password verify (a malformed stored hash) into false, so the answer is always a boolean.

templates/signed-in.hbs:

{"signedIn":{{signedIn}}}

4. Package it and start the engine

sutra lint signin
sutra package signin -o deployments

Give the engine a role that can’t bypass row-level security, and a database that role owns:

CREATE ROLE sutra_app LOGIN NOSUPERUSER NOBYPASSRLS PASSWORD '<a password>';
CREATE DATABASE sutra OWNER sutra_app;

Start the engine with SUTRA_DATASOURCE_URL pointing at that database as sutra_app, SUTRA_DEPLOYMENTS_DIR pointing at deployments, SIGN_IN_API_KEY, STORE_DB_URL, STORE_DB_USER and STORE_DB_PASSWORD for the store, and SUTRA_AUDIT_JSONL for the audit file. Set B to the engine’s address.

5. Sign in the seeded user

curl -s -X POST "$B/channels/sign-in" -H 'X-Api-Key: <your key>' \
  -H 'Content-Type: application/json' -d '{"user":"ada","password":"correct horse"}'
{"signedIn":true}

With "password":"correct horsf" instead:

{"signedIn":false}

6. Enrol a user

curl -s -o /dev/null -w '%{http_code}\n' -X POST "$B/channels/enrol" -H 'X-Api-Key: <your key>' \
  -H 'Content-Type: application/json' -d '{"user":"grace","password":"<a password>"}'
200

The row holds the hash, with its own salt:

SELECT store_key, user_id, password_hash FROM users ORDER BY store_key;
 store_key | user_id |                                           password_hash
-----------+---------+---------------------------------------------------------------------------------------------------
 ada       | ada     | $argon2id$v=19$m=19456,t=2,p=1$AAECAwQFBgcICQoLDA0ODw$0AuQxA+RjPiJeA19Op7DwrFxnfh9xeTU2T6mFj2wThc
 grace     | grace   | $argon2id$v=19$m=19456,t=2,p=1$egxSk+lhFOUDspiuABfQoQ$QzDs4/To+3jKTJclLpfm3cpGF1N41U2Ig4fZuXsTn3M

Signing grace in with that password answers {"signedIn":true}, and with any other password {"signedIn":false}.

7. Where the password went

Nowhere. A data-only pg_dump of the engine’s database has no row containing either password, and neither does the audit file or the engine log. The audit file records each step, with creds and the raw body redacted:

{"eventType":"NODE_ENTERED","nodeId":"Answer","payload":{"creds":"***REDACTED***","event":{"body":"[REDACTED]", ...},"signedIn":true, ...}, ...}

8. Not at a replay-bound site

Put password hash in an alias and lint refuses the package:

<q:alias name="h" expression="password hash(creds.password)"/>
[ERROR] SUTRA.FEEL.DETERMINISM.UNSAFE_BUILTIN — ... The FEEL expression on process 'sign-in' node 'Start' alias 'h' uses `password hash`, which is denied at replay-bound sites.

Limits

  • Waits. creds is transient, so it is gone after a wait, and lint refuses a read of it after one. Check the password before any wait.
  • Cost. Each call costs one argon2id hash on an executor worker. Rate-limit the sign-in channel if it is exposed.
  • Unknown users. This flow assumes the row exists. A real one should answer false for an unknown user too, after the same amount of work, so the answer time doesn’t reveal which users exist.

Recordings and media references

A process should not carry audio through its variables. A recording is large and personal, and once it is inside a snapshot, an audit row or a log it is copied everywhere the engine writes. In 0.6 the media stays in an object store, and the engine passes a media reference around in its place:

{ "kind": "media", "uri": "s3://recordings/calls/q-1.wav", "sha256": "ee6497…", "format": "audio/wav" }

Four pieces work together:

PieceWhat it does
Named object storesEngine configuration: a bucket (and optional key prefix) the engine may read and delete in.
Media references in a model callAn audio, video or image part gives {ref, sha256} instead of data, and the llm channel reads the object from its media-store.
q:audit capture="media"Audit events carry the references found among the instance’s variables, never the media.
Erasure and retentionWhen an instance’s data has to go, the engine deletes the referenced objects and amends the audit events to say so.

A media transport (a telephony transport, for example) typically writes the recording and hands the process its reference. In the walkthrough below you play that part by uploading a file.

Object stores

A store is declared in the engine configuration, never in a package, by any sutra.object-stores.<name>.* key in the file, or by naming it in sutra.object-stores:

KeyDefaultMeaning
sutra.object-stores—Comma-separated store names; the way to declare a store purely by environment variables.
sutra.object-stores.<name>.urirequireds3://<bucket>[/<prefix>] (or memory://<bucket> for tests: in process, nothing survives a restart).
sutra.object-stores.<name>.endpointAWSA self-hosted or non-AWS store’s base URL (http://store:9000), addressed path-style.
sutra.object-stores.<name>.regionus-east-1Signing region.
sutra.object-stores.<name>.access-key, .secret-keyrequired for s3://Secret references (env:, secret:, vault:, …), resolved at boot.
sutra.object-stores.<name>.encryptionenvelopeenvelope: objects are sealed (encrypted at rest with a fresh data key wrapped under kek), as a recording transport writes them; none: stored as they are.
sutra.object-stores.<name>.kekrequired with envelopeSecret reference to the key-encryption secret the objects were sealed under.
sutra.object-stores.<name>.key-idrecordingsThe key id sealed objects name.

Each key has an env var SUTRA_OBJECT_STORES_<NAME>_<FIELD>, with <NAME> upper-cased and - written as _ (for example SUTRA_OBJECT_STORES_RECORDINGS_SECRET_KEY). A store that cannot be built refuses startup. The client speaks the S3 API with Signature V4, so any S3-compatible store works.

sutra.object-stores.recordings.uri=s3://recordings/calls
sutra.object-stores.recordings.endpoint=http://store:9000
sutra.object-stores.recordings.access-key=env:STORE_ACCESS_KEY
sutra.object-stores.recordings.secret-key=env:STORE_SECRET_KEY
sutra.object-stores.recordings.kek=secret:recordings-kek

Media references in a model call

Any audio, video (the clip, or each frame) or image part of an llm request may give ref and sha256 instead of data:

{
  "messages": [{ "role": "user", "content": "Summarise this recording in one sentence.",
                 "audio": { "ref": "s3://recordings/calls/q-1.wav",
                            "sha256": "ee649768a64c…", "format": "audio/wav" } }]
}

The channel names the one store it may read from:

  llm:
    media-store: recordings        # sutra.object-stores.recordings
    media-max-bytes: 1048576       # optional; 20 MiB by default
    candidates: […]

Before the call is routed, the transport reads each referenced object. It checks the stored bytes against sha256 and media-max-bytes, opens the object when it is sealed, and sends the content to the provider as if it had been inline. The reply’s echoed request keeps the reference, never the content.

What went wrongResult
The channel has no media-store; the reference is outside that store (another store, another bucket, or a key outside its prefix); the object is missing or over the limit; the digest does not match; a sealed object does not openpermanent failure, SUTRA.LLM.MEDIA.REFUSED
media-store names a store the engine configuration does not declareSUTRA.LLM.CONFIG.INVALID when the package loads
The store cannot be reacheda retryable failure

The audit trail carries references, never media

<q:audit capture="media"> on a process selects metadata-level audit events: no variable snapshot. Each event instead carries, under media, every media reference found among the instance’s variables (any object with kind: "media", a uri and a sha256, at any depth). A variable marked sensitive is not searched. The capture also marks every request the process sends (the sutra-audit-capture: media header), so a media transport knows to record the session.

<bpmn:process id="recording-summary" isExecutable="true">
  <bpmn:extensionElements>
    <q:audit capture="media"/>
    …

media is an additive value of the capture attribute, alongside none, metadata and payload, and a package’s audit.capture accepts it too.

Erasure and retention delete the recording

When an instance’s data has to go, its recordings go with it:

  • Subject erasure (POST /admin/subjects/erase) deletes every object that the erased instances’ media events reference.
  • Terminal retention (sutra.instance.retention) deletes them for the instances it purges: retained rows past the window, and finished instances whose row is already gone.

Each media event is then amended to say what was recorded and that it is gone:

{"media": [{"kind": "media", "uri": "…", "sha256": "…", "erased": true}],
 "mediaErasure": {"at": "<time>", "reason": "subject-erasure | retention", "objects": 1}}

A recording is never skipped silently. If no declared store holds it, or the store refuses the delete, erasure answers 500 with the references and changes nothing for that instance, so it can be re-run once that is fixed. Retention keeps the instance and retries on its next sweep, with a WARN in the log.

Walkthrough: a recording summarised, erased, and retired

The examples/media-recordings package has one flow, recording-summary.bpmn:

POST /channels/recording-summary   { recordingId, callerId, recording: {kind, uri, sha256, format} }
  BuildRequest   requestBody := { messages: [{ content, audio: { ref, sha256, format } }] }
  AskModel       summary-model (sync): reads the object, checks digest and size, sends it as audio
  BuildReply     { summary, provider, audioPreserved }, answered at once (<q:reply continue="true">)
  AwaitClose     parks until POST /channels/recording-closed { recordingId }
  End

callerId is declared <q:variable name="callerId" type="string" sensitive="true" subjectKey="true"/>, which makes it the data subject that erasure finds the instance by. The model channel:

  - name: summary-model
    direction: outbound
    transport: llm
    bind: "llm://summary-model"
    ack-mode: on-complete
    llm:
      media-store: recordings          # sutra.object-stores.recordings (engine configuration)
      media-max-bytes: 1048576         # refuse any referenced object over 1 MiB (default 20 MiB)
      require: [audio-understanding]   # only a candidate that hears the voice itself
      candidates:
        - provider: openai
          model: gpt-audio-stub
          base-url: http://model-stub:8000/v1
          api-key: env:LLM_API_KEY     # a real deployment: env:OPENAI_API_KEY, no base-url
      timeout-seconds: 30

deploy/compose.yaml runs the engine, PostgreSQL, MinIO (pinned by digest) and the stub model server. It sets on the engine:

SettingWhy
SUTRA_OBJECT_STORES=recordings, …_URI=s3://recordings/calls, …_ENDPOINT=http://store:9000, …_ACCESS_KEY=env:STORE_ACCESS_KEY, …_SECRET_KEY=env:STORE_SECRET_KEY, …_ENCRYPTION=nonethe store; the sample uploads plain objects, where a recording transport would write sealed ones
SUTRA_AUDIT_SQL=trueaudit events go to the engine database, which GET /admin/instances/{id}/history reads
SUTRA_CRYPTO_MASTER_KEY=env:RECORDINGS_MASTER_KEY (a dev-only placeholder)blind-indexes subjectKey variables so erasure can find the instance; it also encrypts sensitive variables at rest. It is a secret reference like every other secret key (env:, secret:, vault:, …); a literal value is still used as the raw key, with a warning at boot
SUTRA_INSTANCE_RETENTION=PT2M, SUTRA_INSTANCE_RETENTION_SWEEP_INTERVAL=PT15Sa finished instance is purged about two minutes after it ends, instead of the default seven days
SUTRA_ADMIN_OIDC_DEV_DISABLED=true/admin/* without authentication. Dev only.

Prerequisites: Before you try a walkthrough. No key is needed.

1. Start the stack and deploy the package:

C=examples/media-recordings/deploy/compose.yaml
docker compose -f $C up -d --wait
sutra package examples/media-recordings/deployments-src/default--recordings--1.0.0 \
    --out examples/media-recordings/deploy/deployments 2>/dev/null
PORT=$(docker compose -f $C port engine 8080 | cut -d: -f2)
H=(-H 'Content-Type: application/json' -H 'X-Api-Key: dev-only-recordings-key')
until curl -sf "http://127.0.0.1:$PORT/sutra/deployments" | grep -q '"phase":"Active"'; do sleep 1; done
packaged examples/media-recordings/deploy/deployments/default--recordings--1.0.0.sutra (deploymentId dep-4314895f6198e60f7bf51a20)

(sutra package also prints two warnings about the schema-less JSON channel, which you can ignore here.)

2. Put two recordings in the store, as a recording transport would. The MinIO image carries the mc client, already pointed at the server:

docker compose -f $C exec store mc mb local/recordings
for id in q-1 q-2; do
  docker compose -f $C exec -T store mc pipe local/recordings/calls/$id.wav < examples/media-recordings/sample/question.wav
done
SHA=$(sha256sum examples/media-recordings/sample/question.wav | cut -d' ' -f1)
Bucket created successfully `local/recordings`.
 0 B / ? 8044 bytes -> `local/recordings/calls/q-1.wav`
 0 B / ? 8044 bytes -> `local/recordings/calls/q-2.wav`

3. Summarise the first recording by reference:

ask() {  # ask <recordingId> <callerId> <sha256>
  curl -s -w " HTTP %{http_code}\n" -X POST "http://127.0.0.1:$PORT/channels/recording-summary" "${H[@]}" \
    -d "{\"recordingId\":\"$1\",\"callerId\":\"$2\",\"recording\":{\"kind\":\"media\",\"uri\":\"s3://recordings/calls/$1.wav\",\"sha256\":\"$3\",\"format\":\"audio/wav\"}}"
}
ask q-1 caller-1 "$SHA"
{"audioPreserved":true,"provider":"openai","summary":"STUB SUMMARY: the customer reports a problem and asks for it to be looked at."} HTTP 200

The engine read calls/q-1.wav, checked its digest, and sent it to the audio model. The instance is now parked at AwaitClose.

4. A reference the store cannot honour is refused. There is no q-9.wav:

ask q-9 caller-9 "$SHA"
{"type":"urn:bpm:diag:SUTRA.LLM.MEDIA.REFUSED","title":"SUTRA.LLM.MEDIA.REFUSED","status":422,"detail":"'s3://recordings/calls/q-9.wav' does not exist","code":"SUTRA.LLM.MEDIA.REFUSED","attributes":{"incident":"SUTRA.INBOUND.NON_IDEMPOTENT_FAILURE"}} HTTP 422

The model call failed permanently with SUTRA.LLM.MEDIA.REFUSED, and the caller is told so, with 422: the request named an object that cannot be read, so sending it again cannot succeed. The detail names the reference, never the store’s credentials or the object’s content. This process does not declare itself idempotent, so the inbound is consumed and recorded as an incident instead of being redelivered; incident carries that code. A wrong sha256 on an existing object is refused the same way, with does not match its SHA-256.

5. Read the audit trail. Every event carries the reference, and none carries the audio:

curl -s "http://127.0.0.1:$PORT/admin/instances"; echo
ID=$(curl -s "http://127.0.0.1:$PORT/admin/instances" | python3 -c 'import json,sys; print([i["instanceId"] for i in json.load(sys.stdin)["instances"] if i["status"] == "SUSPENDED"][0])')
history() {
  curl -s "http://127.0.0.1:$PORT/admin/instances/$1/history" | python3 -c '
import json, sys
for e in json.load(sys.stdin)["events"]:
    print(e["seq"], e["eventType"], e["nodeId"] or "", json.dumps(e["payload"]) if e["payload"] else "")'
}
history "$ID"
{"instances":[{"instanceId":"f67c1b1e-4c95-4317-9f0f-6fb3ede9f5a2","deploymentId":"dep-4314895f6198e60f7bf51a20","status":"SUSPENDED"}]}
1 INSTANCE_STARTED
2 NODE_ENTERED Start {"media": [{"format": "audio/wav", "kind": "media", "sha256": "ee649768a64c8d99984c36213c629f4a5c84d10fac2d872275f2f11bcafa1e4c", "uri": "s3://recordings/calls/q-1.wav"}]}
3 NODE_LEFT Start
4 NODE_ENTERED BuildRequest {"media": [{"format": "audio/wav", "kind": "media", "sha256": "ee649768a64c8d99984c36213c629f4a5c84d10fac2d872275f2f11bcafa1e4c", "uri": "s3://recordings/calls/q-1.wav"}]}
5 NODE_LEFT BuildRequest
6 NODE_ENTERED AskModel {"media": [{"format": "audio/wav", "kind": "media", "sha256": "ee649768a64c8d99984c36213c629f4a5c84d10fac2d872275f2f11bcafa1e4c", "uri": "s3://recordings/calls/q-1.wav"}]}
7 INSTANCE_SUSPENDED
8 INSTANCE_RESUMED
9 NODE_LEFT AskModel
10 NODE_ENTERED BuildReply {"media": [{"format": "audio/wav", "kind": "media", "sha256": "ee649768a64c8d99984c36213c629f4a5c84d10fac2d872275f2f11bcafa1e4c", "uri": "s3://recordings/calls/q-1.wav"}]}
11 NODE_LEFT BuildReply
12 NODE_ENTERED AwaitClose {"media": [{"format": "audio/wav", "kind": "media", "sha256": "ee649768a64c8d99984c36213c629f4a5c84d10fac2d872275f2f11bcafa1e4c", "uri": "s3://recordings/calls/q-1.wav"}]}
13 INSTANCE_SUSPENDED

The refused instance is not listed: /admin/instances lists live instances. Inspecting the parked one shows its variables, with callerId redacted: it is sensitive, encrypted at rest, and the endpoint decrypts it only to apply the same redaction as everywhere else:

curl -s "http://127.0.0.1:$PORT/admin/instances/$ID" | python3 -c 'import json,sys; r=json.load(sys.stdin); print(r["status"], r["sensitive"], r["variables"]["callerId"])'
SUSPENDED ['callerId'] ***REDACTED***

6. Erase the data subject caller-1. keyId is the tenant and subjectName is the subjectKey variable. With no deploymentId, erasure covers every live deployment of the tenant, wherever it was deployed from (a watched directory, as here, or the database):

curl -s -X POST "http://127.0.0.1:$PORT/admin/subjects/erase" -H 'Content-Type: application/json' \
    -d '{"keyId":"default","subjectName":"callerId","value":"caller-1"}'; echo
docker compose -f $C exec store mc ls -r local/recordings
history "$ID" | sed -n 2p
{"subjectName":"callerId","tenantWide":true,"deploymentsScanned":1,"erased":true,"matched":1,"erasedCount":1,"instanceIds":["f67c1b1e-4c95-4317-9f0f-6fb3ede9f5a2"]}
[2026-10-26 06:52:26 UTC] 7.9KiB STANDARD calls/q-2.wav
2 NODE_ENTERED Start {"media": [{"kind": "media", "uri": "s3://recordings/calls/q-1.wav", "erased": true, "sha256": "ee649768a64c8d99984c36213c629f4a5c84d10fac2d872275f2f11bcafa1e4c"}], "mediaErasure": {"at": "2026-10-26T06:52:26.726921625Z", "reason": "subject-erasure", "objects": 1}}

calls/q-1.wav is gone from the store, and the audit events still say what was recorded, now marked erased.

7. Let retention retire the second recording. Summarise q-2 and close it, so the instance completes:

ask q-2 caller-2 "$SHA"
curl -s -w "HTTP %{http_code}\n" -X POST "http://127.0.0.1:$PORT/channels/recording-closed" "${H[@]}" -d '{"recordingId":"q-2"}'
{"audioPreserved":true,"provider":"openai","summary":"STUB SUMMARY: the customer reports a problem and asks for it to be looked at."} HTTP 200
HTTP 200

Then wait for the purge. With a two-minute retention and a 15-second sweep it takes about two and a half minutes:

until [ -z "$(docker compose -f $C exec -T store mc ls local/recordings/calls/q-2.wav)" ]; do sleep 5; done
docker compose -f $C logs engine | grep -o 'terminal retention deleted the recorded media[^"]*'
docker compose -f $C exec -T engine-db psql -U postgres -d sutra -At \
    -c "select seq, event_type, payload_json from audit_event where payload_json like '%q-2.wav%' order by seq limit 1"
terminal retention deleted the recorded media of finished instances
2|NODE_ENTERED|{"media":[{"kind":"media","uri":"s3://recordings/calls/q-2.wav","erased":true,"sha256":"ee649768a64c8d99984c36213c629f4a5c84d10fac2d872275f2f11bcafa1e4c"}],"mediaErasure":{"at":"2026-10-26T06:54:34.352446787Z","reason":"retention","objects":1}}

The loop ends when mc ls no longer lists the object. The purged instance’s audit rows stay, amended with "reason":"retention".

8. Clean up:

docker compose -p sutra-media-recordings down -v

Next

Wait states and human tasks

Sutra’s wait state is an integration point that waits for an external relay decision — not a human-task management system. When a token reaches one, the engine suspends the instance (persists it, frees the thread); the flow continues only when an external actor — a human via a console you build, or another system — relays a decision back through a channel as a typed message. The engine rehydrates and runs to completion.

Sutra (the engine) providesYou own
The wait point + durable suspend / rehydrate / resumeHuman-task management: assignment, queues, claim, escalation, SLA
A relay channel (typed message in)Forms and any UI
The held instance variables, carried verbatimIdentity / authentication / authorization (who may relay)
Compliance-grade audit of the wait and the relayed decisionNotifications, dashboards

That split is deliberate: full human-task management and forms are a much larger scope than a workflow engine’s core job, and Sutra doesn’t try to own them.

The construct: an intermediate message catch event

The honest BPMN construct is the intermediate message catch event — “pause until message M arrives, then continue.” A userTask is the human-facing flavor of the exact same machinery, for authors who think in terms of “a human decides here” — it carries no assignee/group/queue model in the engine; if present, those are just opaque hints in instance variables.

An intake node for a wait declares a channel exactly like a start event’s q:source — the relayed message goes through the same two-tier validation pipeline (structural codec, then business validators) that every inbound message does. The difference is only in what happens after:

OutcomeStart event (new instance)Wait state (relay)
Structurally invalidNo instance minted; error to callerRelay rejected; the instance stays parked, unchanged
Soft errorsMinted and run; surfaced as variablesRehydrated and resumed; the gateway after the wait routes on the soft errors
ValidMinted and runRehydrated and resumed

The load-bearing property: the wait is the safe state. A hard-invalid relay can never advance or corrupt a parked instance — it is rejected and the instance sits exactly where it was. The relayer fixes the message and retries.

Correlation by a business key you name

<q:alias name="e2eId" expression="payload.E2EId" unique="true" onConflict="correlate"/>

The relayed message doesn’t carry an engine-internal instance id — it carries a business value (an EndToEndId, an order reference, whatever your domain already uses), and the process declares a q:alias deriving that key from its own payload. The engine indexes it durably, on PostgreSQL, so correlation survives a restart and works identically across every replica. See The q: namespace for the full q:alias shape.

A relay is matched by the start event’s alias even when the wait binds its message under a name of its own (<q:source channel="revision" name="revision"/>): while the alias is evaluated, the relayed message is also visible under the start event’s payload name, so payload.E2EId reads the relayed E2EId. The name a wait gives its payload is local to that wait. It is what the steps after it read (revision.note), and it does not change which instance a message finds.

sequenceDiagram
    participant A as Requester
    participant E as Engine
    participant DB as PostgreSQL
    participant R as Relayer

    A->>E: ApprovalRequest on approval-request
    E->>DB: park — snapshot plus alias e2eId from payload.E2EId
    Note over E,DB: durable commit — parked, correlatable, thread freed
    E-->>A: accepted, no business reply yet
    R->>E: ApprovalDecision on approval-decision
    E->>E: structural codec, then business validators
    alt structurally invalid
        E-->>R: relay rejected — the instance stays parked, unchanged
    else valid, or soft errors only
        E->>DB: correlate by e2eId, rehydrate typed variables
        E->>E: the gateway after the wait routes on the soft errors
        E->>DB: run to the end — alias retired
    end

The park is a commit, not a pause: the correlation key is written in the same durable step that suspends the instance, so a relay arriving later — on a different channel, against a different replica — finds it by the key you named. And because a hard-invalid relay is refused before anything is rehydrated, the parked instance is never the thing that pays for a bad message.

An answer that arrives before its wait

Since 0.6.6. Between co-deployed processes an answer can be faster than the process that asked. A local:// delivery goes out the moment the sender’s step commits, so when a process sends a request at one park and reaches the wait for the answer only one step later — a q:reply continue="true", a timer, or any other wait in between — the answer can reach the instance before it has parked there. Its wait key is not written yet, so the engine refuses the answer (SUTRA.RUNTIME.RELAY.CORRELATION_NOT_FOUND, or SUTRA.RUNTIME.RESUME.NOT_WAITING when the instance is parked at another node), having run nothing.

The engine marks such a refusal wait-pending (sutra.waitPending=true on the diagnostic), and a local:// delivery that gets one is held, not dropped: it is offered again after a few milliseconds (155 ms in all, about one step), and then retried with the outbox backoff for up to 8 attempts — about four minutes. If the instance still has not parked by then, the delivery takes the outbox’s usual poison posture: kept with its last error and redelivered at the poison horizon, so nothing is lost. Since 0.7.0 an answer for an instance that has finished is not wait-pending: when every instance that carried the answer’s key has ended, the refusal carries no sutra.waitPending and its detail says the instance has ended. The engine can tell while the finished instance is retained (sutra.instance.retention, seven days by default). Since 0.6.7 a local:// delivery whose step failed because the database was unreachable (a reset connection, a restart) is retried the same way, without the bound, instead of being parked at the poison horizon. A message from outside the engine (HTTP, a broker) keeps the transport’s own handling, because the engine cannot tell a fast answer from a stale one there; send it again, or make the wait come first.

To watch it, run the conformance test that builds the race on purpose. Its package (rust/crates/sutra-conformance/fixtures/wait-race/) sends a request to a second process, waits two seconds on a timer, and only then parks at the wait for the answer, which the second process sends at once. It needs Docker and the engine image built from your checkout (make image):

cd rust && cargo test -p sutra-conformance --test all tc_wait_race -- --ignored --nocapture
test tc_wait_race::tc_wait_race_an_answer_that_beats_the_wait_is_held_until_the_wait_parks ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 46 filtered out; finished in 8.85s

All 20 answers arrive while their instances sit at the timer, and all 20 instances complete, each called back once. On a 0.6.5 engine the same test fails after 90 seconds with every answer still in the outbox, refused with SUTRA.RUNTIME.RELAY.CORRELATION_NOT_FOUND and parked at the poison horizon.

An answer no instance waits for

Since 0.7.0. An answer that correlates to no live instance (a reply after its instance ended, a key nothing ever carried) is refused with SUTRA.RUNTIME.RELAY.CORRELATION_NOT_FOUND, and recorded whatever transport carried it:

  • a WARN log line with the code SUTRA.INBOUND.RELAY_UNCORRELATED, naming the channel, the message type, the dedup key and the correlation keys looked for (keys=orderKey=B-9); never the body or the headers;
  • an inbound.RELAY_UNCORRELATED record in the deployment’s audit journal, with no instance id. Its diagnostic_json holds channel, deployment, messageType, dedupKey, keys (each with process, node, alias, value, ended), ended and outcome.

A key value is masked in both when the alias, or a variable its expression reads, is sensitive, or when a redactor masks its input.

A package can take such an answer instead of refusing it, with no BPMN change, by naming a process on the channel in channels.yaml:

channels:
  - name: order-confirm
    transport: http
    bind: "POST /orders/confirm"
    codec: json
    uncorrelated:
      start: late-confirmation   # a process of this package with a start event on order-confirm

That process’s start events on the channel are reserved for answers no wait correlates: ordinary start resolution passes them over, so an answer that correlates still resumes its instance and starts nothing. Its <q:source> still decides which uncorrelated answers it takes. The started instance reads event.uncorrelated: code, keys (alias name to the raw value looked for) and ended. The audit record then says outcome: started, with startedProcess and instanceId.

To watch both, run the conformance suite (Docker, and the engine image built from your checkout with make image):

cd rust && cargo test -p sutra-conformance --test all tc_uncorrelated_relay -- --ignored --nocapture
test tc_uncorrelated_relay::tc_uncorrelated_relay_a_refused_answer_is_recorded_and_wait_pending_only_while_it_may_park ... ok
test tc_uncorrelated_relay::tc_uncorrelated_relay_a_routed_answer_starts_the_named_process_and_correlation_is_unaffected ... ok

Variables survive the wait typed

The instance variables held across a wait keep their real types. A number comes back a number, a boolean a boolean, a list a list, a context a context, a date/time/duration its own temporal type — and a null comes back null, not an empty string.

This matters because FEEL, correctly, does not coerce. A gateway condition after a wait therefore evaluates against the value’s real type:

After the waitEvaluates as
amount > 100 where amount parked as 1250.75a number comparison — true
not(approved) where approved parked as falsea boolean negation — true
cancelledAt = null where nothing was ever settrue

Two things are deliberately unchanged. First, the inspect projection is not affected: every variable still renders through the display form it always had on GET /admin/instances/{id}, so what an operator’s response looks like did not move — typing changed what survives a wait, not what is shown. (A null now renders as null rather than blank, which is the one visible difference, and it is the correct one.) Second, the correlation key input is not affected: a subject blind-index value keeps hashing the exact string it always hashed.

Two value kinds still persist as their canonical string, because neither is really instance state: a FEEL function closes over an evaluation context that ceases to exist the moment the instance parks, and a range is a comparison shape rather than a value. Both behaved this way before and still do.

The design reasoning — the encoding, the compatibility posture, and why encrypted values are typed too — is in the internals chapter.

Waiting at the same node again: loops

Since 0.6, a process instance can wait at the same node any number of times. A loop that brings the token back to a message catch, a receive task, a user task, a channel call (async or synchronous), a timer catch or an event-based gateway parks a new wait on each pass, with its own wait row, its own timer and <q:timeout> rows, its own <q:retry> budget and its own <q:alias> rows. The step that answers a pass resolves that pass’s rows and retires its correlation keys, so a late message for a past pass resumes nothing. Before 0.6 the token found the node already done and ran past it, or the instance could not park there again.

Behind this, a resume continues from the satisfied node itself instead of replaying the graph from the start event. Nothing upstream runs again, other parked waits carry over untouched, and a converging gateway regains its earlier arrivals from a side-effect-free walk over the completed nodes. The snapshot format is unchanged, so instances parked by an earlier engine resume under the new rules.

Related rules that came with it:

  • A message that correlates to an instance but names a wait the instance is not parked at is refused before anything runs, with SUTRA.RUNTIME.RESUME.NOT_WAITING.
  • A wait node’s own <q:alias> (on a message catch, receive task or user task) is indexed when the node parks, so a message correlates by it.
  • An event-based gateway continues along the arm that fired when the instance resumes.
  • A parallel join inside a loop waits for a full set of tokens each time round.
  • A message on a channel that several wait nodes of one process subscribe to resumes the node the instance is parked at.

Walkthrough: a review loop

The examples/review-loop package sends a draft round review, revision and a short cool-off until a reviewer approves it:

POST /channels/drafts            { draftId }                     rounds := 0
  AwaitReview    user task, answered on POST /channels/drafts/review { draftId, decision }
  Count          rounds := rounds + 1
  Decide         review.decision = "approve" -> End
                 otherwise -> AwaitRevision (message catch, POST /channels/drafts/revision)
                           -> CoolOff (timer catch, PT2S) -> back to AwaitReview

The loop is ordinary BPMN: AwaitReview has two incoming flows, one from the start and one back from CoolOff:

<bpmn:userTask id="AwaitReview" name="Review the draft">
  <bpmn:extensionElements>
    <q:source channel="review" name="review"/>
  </bpmn:extensionElements>
  <bpmn:incoming>F2</bpmn:incoming><bpmn:incoming>F8</bpmn:incoming>
  <bpmn:outgoing>F3</bpmn:outgoing>
</bpmn:userTask>
…
<bpmn:intermediateCatchEvent id="CoolOff" name="Cool-off, 2 s">
  <bpmn:incoming>F7</bpmn:incoming><bpmn:outgoing>F8</bpmn:outgoing>
  <bpmn:timerEventDefinition id="CoolOff_Timer">
    <bpmn:timeDuration>PT2S</bpmn:timeDuration>
  </bpmn:timerEventDefinition>
</bpmn:intermediateCatchEvent>
<bpmn:sequenceFlow id="F8" sourceRef="CoolOff" targetRef="AwaitReview"/>

Every message after the first finds the instance through the start event’s <q:alias name="draftId" expression="payload.draftId" unique="true" onConflict="correlate"/>. The waits bind their messages under names of their own (name="review", name="revision"), and Decide reads review.decision; the alias still reads payload.draftId, because a wait’s name is local to it. The process declares <q:audit capture="metadata"/>, so each step is in the instance’s audit history.

Prerequisites: Before you try a walkthrough.

1. Start the stack and deploy the package. The waits helper lists the instance’s wait rows from the engine database:

C=examples/review-loop/deploy/compose.yaml
docker compose -f $C up -d --wait
sutra package examples/review-loop/deployments-src/default--review--1.0.0 \
    --out examples/review-loop/deploy/deployments 2>/dev/null
PORT=$(docker compose -f $C port engine 8080 | cut -d: -f2)
H=(-H 'Content-Type: application/json' -H 'X-Api-Key: dev-only-review-key')
until curl -sf "http://127.0.0.1:$PORT/sutra/deployments" | grep -q '"phase":"Active"'; do sleep 1; done
send() { curl -s -w " HTTP %{http_code}\n" -X POST "http://127.0.0.1:$PORT/channels/drafts$1" "${H[@]}" -d "$2"; }
waits() {
  docker compose -f $C exec -T engine-db psql -U postgres -d sutra -At -c \
    "select node_id, kind, status from waiting_event order by created_at"
}
packaged examples/review-loop/deploy/deployments/default--review--1.0.0.sutra (deploymentId dep-84173f6525975bed01d706ee)

2. Submit a draft. It parks at the review:

send "" '{"draftId":"d-1"}'
ID=$(curl -s "http://127.0.0.1:$PORT/admin/instances/by-alias/draftId/d-1" | python3 -c 'import json,sys; print(json.load(sys.stdin)["instanceId"])')
waits
 HTTP 200
AwaitReview|MESSAGE|WAITING

3. Round one: ask for a revision, submit it, and let the cool-off run:

send /review '{"draftId":"d-1","decision":"revise"}'
waits
send /revision '{"draftId":"d-1"}'
waits
sleep 4
waits
 HTTP 200
AwaitReview|MESSAGE|RESOLVED
AwaitRevision|MESSAGE|WAITING
 HTTP 200
AwaitReview|MESSAGE|RESOLVED
AwaitRevision|MESSAGE|RESOLVED
CoolOff|TIMER|WAITING
AwaitReview|MESSAGE|WAITING
AwaitRevision|MESSAGE|RESOLVED
CoolOff|TIMER|RESOLVED

After the timer fired, the token came back to AwaitReview, and its wait is armed again (WAITING) for round two.

4. Round two, and a message for a wait the instance has left. The second review asks for another revision. A third review arrives while the instance waits for that revision, so it is refused and nothing runs:

send /review '{"draftId":"d-1","decision":"revise"}'
send /review '{"draftId":"d-1","decision":"approve"}'
 HTTP 200
{"type":"urn:bpm:diag:SUTRA.RUNTIME.RESUME.NOT_WAITING","title":"SUTRA.RUNTIME.RESUME.NOT_WAITING","status":500,"detail":"Instance 43a748ee-92ef-4845-bf59-10492c4b45ee is not waiting at 'AwaitReview' (it is parked at [\"AwaitRevision\"]); this relay on channel 'review' arrived for a wait the instance already left and is refused — nothing was executed.","code":"SUTRA.RUNTIME.RESUME.NOT_WAITING","attributes":{"instanceId":"43a748ee-92ef-4845-bf59-10492c4b45ee","nodeId":"AwaitReview"}} HTTP 500

5. Finish round two, then approve in round three:

send /revision '{"draftId":"d-1"}'
sleep 4
curl -s "http://127.0.0.1:$PORT/admin/instances/$ID" | python3 -c 'import json,sys; r=json.load(sys.stdin); print(r["status"], r["variables"]["rounds"])'
send /review '{"draftId":"d-1","decision":"approve"}'
curl -s "http://127.0.0.1:$PORT/admin/instances"; echo
 HTTP 200
SUSPENDED 2
 HTTP 200
{"instances":[]}

Before the last review, the instance was parked at AwaitReview for the third time with rounds = 2. The approval ended it, so no instance is live any more.

6. Read the audit history. The journal is written asynchronously, so the loop waits for the completion event. Each event’s ordinal is its position in the journal, which is what the history pages by:

until curl -s "http://127.0.0.1:$PORT/admin/instances/$ID/history" | grep -q INSTANCE_COMPLETED; do sleep 1; done
curl -s "http://127.0.0.1:$PORT/admin/instances/$ID/history" | python3 -c '
import json, sys
for e in json.load(sys.stdin)["events"]:
    print(e["ordinal"], e["eventType"], e["nodeId"] or "")'
1 INSTANCE_STARTED
2 NODE_ENTERED Start
3 NODE_LEFT Start
4 NODE_ENTERED Init
5 NODE_LEFT Init
6 NODE_ENTERED AwaitReview
7 INSTANCE_SUSPENDED
8 INSTANCE_RESUMED
9 NODE_LEFT AwaitReview
10 NODE_ENTERED Count
11 NODE_LEFT Count
12 NODE_ENTERED Decide
13 NODE_LEFT Decide
14 NODE_ENTERED AwaitRevision
15 INSTANCE_SUSPENDED
16 INSTANCE_RESUMED
17 NODE_LEFT AwaitRevision
18 NODE_ENTERED CoolOff
19 INSTANCE_SUSPENDED
20 INSTANCE_RESUMED
21 NODE_LEFT CoolOff
22 NODE_ENTERED AwaitReview
23 INSTANCE_SUSPENDED
24 INSTANCE_RESUMED
25 NODE_LEFT AwaitReview
26 NODE_ENTERED Count
27 NODE_LEFT Count
28 NODE_ENTERED Decide
29 NODE_LEFT Decide
30 NODE_ENTERED AwaitRevision
31 INSTANCE_SUSPENDED
32 INSTANCE_RESUMED
33 NODE_LEFT AwaitRevision
34 NODE_ENTERED CoolOff
35 INSTANCE_SUSPENDED
36 INSTANCE_RESUMED
37 NODE_LEFT CoolOff
38 NODE_ENTERED AwaitReview
39 INSTANCE_SUSPENDED
40 INSTANCE_RESUMED
41 NODE_LEFT AwaitReview
42 NODE_ENTERED Count
43 NODE_LEFT Count
44 NODE_ENTERED Decide
45 NODE_LEFT Decide
46 NODE_ENTERED End
47 NODE_LEFT End
48 INSTANCE_COMPLETED

Each round reads the same way: the resume leaves the wait it came through (NODE_LEFT AwaitReview, NODE_LEFT AwaitRevision, NODE_LEFT CoolOff), and Count and Decide run once per round. The last round leaves Decide for End.

7. Clean up:

docker compose -p sutra-review-loop down -v

Worked example: approval-hold

examples/approval-hold is the reference wait-state flow: a startEvent on channel approval-request records the correlation alias (e2eId = payload.E2EId, unique, onConflict="correlate"), then a userTask on channel approval-decision — the hold — then an endEvent. Its bundled conformance test drives the full lifecycle against real PostgreSQL:

  1. Park — an ApprovalRequest on approval-request returns 200 (accept, no business reply yet); the instance parks and its alias is recorded.
  2. Duplicate rejected — a second ApprovalRequest with the same E2EId while parked is rejected (the durable unique-alias guard).
  3. Relay resume — an ApprovalDecision on the separate approval-decision channel — a channel no start event subscribes to — is routed by the dispatcher’s relay path: the same two-tier intake, against the wait node’s q:source, then correlated by E2EId, then resumed.
  4. Retire — the same E2EId is accepted again afterward, proving the alias was retired once the instance completed.
  5. Uncorrelated relay is safe — a decision for an unknown E2EId is rejected and every parked instance is left untouched.

Try it by hand once the package is deployed (see Your first deployment):

# park
curl -sS -X POST localhost:8080/channels/approval-request \
  -H 'Content-Type: application/json' -H 'X-Api-Key: approval-demo-key' \
  -d '{"ApprovalRequest":{"E2EId":"E2E-1","Amount":"1500.00"}}'

# resume it with a decision
curl -sS -X POST localhost:8080/channels/approval-decision \
  -H 'Content-Type: application/json' -H 'X-Api-Key: approval-demo-key' \
  -d '{"ApprovalDecision":{"E2EId":"E2E-1","Decision":"APPROVE"}}'

Run the scenario itself with cd rust && cargo test -p sutra-conformance -- --ignored tc_approval_hold (needs Docker — see Test tiers).

Next

  • Worked example: money-transfer — the other flagship example, showing durable data stores instead of a wait state.
  • External tasks — the pull flavour of the same park: a delivery parked for a worker to fetch rather than dialed out.
  • Replica semantics — how correlation and suspend/resume stay correct across a multi-replica engine.
  • Durable execution — what a park actually persists, and why it is a snapshot rather than an event log.

One endpoint, one deployment

Several deployments can share one engine, but most inbound endpoints hand each message to a single receiver. If two deployments claimed one, the second would either never receive anything or silently compete with the first for messages. So since 0.6.18 the engine refuses the second claim at deploy, with the first deployment still serving.

EndpointWho can hold it
HTTP METHOD pathone deployment
Queue (RabbitMQ, SQS, AMQP 1.0)one deployment
Pub/Sub subscriptionone deployment
Kafka consumer group, on any topicone deployment
File spool directoryone deployment
LLM callback routeone deployment
A channel’s name, under a package’s tenant, module and versionone deployment
Topic (Dapr, Knative subscription, AMQP 1.0)any number; each receives its own copy

Two deployments can read one Kafka topic if they use different group.id values, and one Pub/Sub topic through different subscriptions. Each then gets every message. An unset group.id defaults from the channel name, so the same channel in two deployments lands in one group and is refused.

This walkthrough deploys two packages that want the same route, then fixes the second.

What was and was not run. The walkthrough ran an HTTP route end to end on the db deployment source. RabbitMQ was run end to end too. Claim identities for Kafka, SQS, Pub/Sub, AMQP and file have unit tests with no broker behind them. Dapr fan-out was driven by a direct POST with no sidecar, Knative fan-out has unit tests only, and the advisory lock has no concurrent-replica race test.

Before you start

You need a running engine on the db deployment source, with a database role that can’t bypass row-level security (the engine refuses to boot on a superuser or a BYPASSRLS role with SUTRA.STARTUP.RLS_BYPASS_RISK), and an admin gate, or the admin surface answers 503. This page uses a key: SUTRA_ADMIN_AUTH_SCHEME=apikey and SUTRA_ADMIN_AUTH_KEY_REF=env:<a variable holding the key> on the engine, and SUTRA_ADMIN_API_KEY for the CLI. See Configuration reference.

You also need two package directories, orders/ and notes/, each with a one-step flow that answers a request. They differ only in the module name. This is the part that matters, channels.yaml, in both (the channel is notes-in in notes/):

channels:
  - name: orders-in
    transport: http
    bind: "POST /orders"
    auth:
      scheme: apikey
      apikey:
        value: env:ORDERS_KEY

Both also declare a data store on one connection, datastores.yaml:

datastores:
  - name: accounts
    type: sql
    sql:
      url-ref: env:SHARED_DB_URL

1. Deploy the first package

$ sutra package orders --out out-orders
packaged out-orders/orders.sutra (deploymentId dep-7a4cb8e71418102e546ce735)
$ sutra deploy out-orders/orders.sutra --api --engine-url http://localhost:8080
deployed dep-7a4cb8e71418102e546ce735 (slot acme--orders--1.0.0, rev 1) — Active
$ curl -X POST http://localhost:8080/orders -H 'X-API-Key: <your key>' -H 'Content-Type: application/json' -d '{}'
hello from orders

2. Check the second package before deploying it

sutra lint runs the deploy’s check when you give it a target. --engine-url asks the running engine:

$ sutra lint notes --engine-url http://localhost:8080
[ERROR] SUTRA.DEPLOY.ENDPOINT.COLLISION — http endpoint 'POST /orders' is claimed by channel 'orders-in' of deployment acme--orders--1.0.0 and by channel 'notes-in' of deployment acme--notes--1.0.0; it can belong to only one deployment
[WARN] SUTRA.DEPLOY.DATASTORE.SHARED_CONNECTION — deployment acme--notes--1.0.0 and deployment acme--orders--1.0.0 use the same connection for data store(s) accounts, so they share those stores' rows. If that is not intended, give this deployment its own connection in its datastores.yaml
1 error(s), 2 warning(s)

The count includes a schema warning about these minimal packages, left out here. The call needs the admin surface, so pass --api-key if your engine gates it. With no engine to reach, --active-set takes sealed archives, or a directory of them, instead. That form runs inside the CLI, which links only the HTTP transport, so it reports a channel on any other transport as SUTRA.DEPLOY.ENDPOINT.UNCHECKED rather than judging it. See the CLI reference.

3. Try the deploy anyway

$ sutra package notes --out out-notes
$ sutra deploy out-notes/notes.sutra --api --engine-url http://localhost:8080
[ERROR] SUTRA.DEPLOY.ENDPOINT.COLLISION — engine rejected the deployment (HTTP 409 Conflict): ...
$ echo $?
1

Nothing changed. The route still answers from the first deployment, and the second was not stored:

$ curl -X POST http://localhost:8080/orders -H 'X-API-Key: <your key>' -H 'Content-Type: application/json' -d '{}'
hello from orders
$ curl http://localhost:8080/sutra/deployments
{"active":[{"deploymentId":"dep-7a4cb8e71418102e546ce735","slot":"acme--orders--1.0.0","phase":"Active","ready":true}],"pending":[],"draining":[],"failed":[]}

4. Give the second package its own route

Change the bind in notes/channels.yaml to POST /notes, then package and deploy again:

$ sutra deploy out-notes/notes.sutra --api --engine-url http://localhost:8080
[WARN] SUTRA.DEPLOY.DATASTORE.SHARED_CONNECTION — deployment acme--notes--1.0.0 and deployment acme--orders--1.0.0 use the same connection for data store(s) accounts, so they share those stores' rows. If that is not intended, give this deployment its own connection in its datastores.yaml
deployed dep-b7a5226eb282e88b23a075b7 (slot acme--notes--1.0.0, rev 1) — Active

The deploy went through, and the warning stayed. Both deployments declare a store named accounts on one connection, and rows are keyed by store name and key, so they read and write each other’s rows. Sharing a connection is sometimes right, so this never refuses a deploy. If you didn’t mean it, give each deployment its own connection in its datastores.yaml. The warning names the other deployment and the stores, never the connection. sutra lint with no target also raises it when a sql store’s connection reference names neither the tenant nor the module, because any co-hosted copy of the package would share it.

How the other deployment paths behave

A hot deploy into the same slot is not a collision. New content for acme--orders--1.0.0 replaces it.

A new version takes over from an old one. Versions of one tenant and module count as one owner, so acme--orders--2.0.0 on 1.0.0’s route is accepted, and SUTRA.DEPLOY.ENDPOINT.SUPERSEDED warns. New requests reach 2.0.0, and 1.0.0 drains. Since 0.6.20 the old version keeps serving the channels only it declares until it retires, and another owner is refused those endpoints meanwhile: see Upgrading a deployment in place. Before 0.6.20 it drained whole: every channel of 1.0.0 stopped taking new work, including one that 2.0.0 doesn’t have.

A package with no tenant or module label owns only itself. It can’t take a route, or a channel name, from anything. Moving it to a new route is refused until you undeploy the old one. Label the package and new versions replace old ones for you.

A folder source (sutra.deployments.dir). A refused archive is listed under failed in GET /sutra/deployments, and the engine serves the rest. File names and arrival order decide nothing:

  • The holder keeps the endpoint across restarts and on new pods. The engine records it in its database, so every replica reads the same answer.
  • Two archives on one endpoint with no record of a holder both report Failed and neither deploys. Remove one and the other lands. With no database there is no record, so this is also what a restart gives you if a refused file is still there.
  • The highest version of one tenant and module wins, comparing versions rather than text (1.10.0 beats 1.9.0). To roll back, remove the newer file.
  • A pod that can’t verify the holder’s file, because it is damaged or a secret is missing there, doesn’t give the endpoint away. It passes on only when the holder’s file is gone.

sutra deploy to a ConfigMap. The CLI judges the archive against the ConfigMap’s other entries and refuses before patching it. Add --engine-url to have the engine judge it too, which covers every transport.

References compare as written

A claim is computed where no secrets are to hand, including in the CLI, and its text appears in refusal messages and in the deployment_endpoint_holder table. So it holds a reference as you wrote it, never its value. Two deployments on one queue through ${A}/q and ${B}/q with the same value are not caught, and neither are ${A}/q and the literal.

Upgrading

  • Two deployments already on one queue. The one activated first keeps it. The later one stops being active and is listed under failed until the next deployment change. Its running instances keep their definition.

  • Kafka groups are different. Deployments already sharing a consumer group keep running, with SUTRA.DEPLOY.ENDPOINT.ALREADY_SHARED logged as an error. Any new deploy claiming that group is refused, including new content for one of them. Give each channel its own group.id the next time you change it, and set auto.offset.reset for what a new group should read first, because it starts from its own offsets:

    channels:
      - name: orders-in
        transport: kafka
        bootstrap.servers: kafka:9092
        topic: orders
        group.id: orders-consumers
    
  • Topics. Dapr and Knative channels on one topic now all receive each event, where a second claim used to be refused. The engine creates a deployment_endpoint_holder table at boot.

A reply answers the instance waiting for it

From 0.6.22, when a message arrives on a channel, the engine first asks whether it answers an instance that is parked waiting for it, and only then whether it starts a new one. This holds with one version of a package and across versions while an older one drains:

  • A message on a channel with both a start event and a wait resumes the instance it correlates to, and starts nothing. An uncorrelated message starts, as before.
  • While version 1 drains, a reply for one of its parked instances resumes that instance on version 1, even when version 2 has a start event on the channel, and it is processed with version 1’s own codec, validators and reply.
  • A retry of a message that ran on version 1 is a duplicate on version 2.

This walkthrough shows the first with one package; the tier-2 suite covers the other two. It needs the setup of One endpoint, one deployment: an engine on the db deployment source, a database role that can’t bypass row-level security, and an admin key.

The package

orders/bpmn/flow.bpmn has one process. It starts on order-in, keeps the order’s key as its correlation alias, and parks on order-in again until a message with the same key arrives:

<?xml version="1.0" encoding="UTF-8"?>
<bpmn:definitions xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL"
                  xmlns:q="urn:sutra:q:1.0" id="Definitions_orders"
                  targetNamespace="urn:sutra:module:orders:1.0.0">
  <bpmn:process id="order" name="Order" isExecutable="true">
    <bpmn:startEvent id="Received">
      <bpmn:extensionElements>
        <q:source channel="order-in"/>
        <q:alias name="orderKey" expression="payload.key" unique="true" onConflict="correlate"/>
      </bpmn:extensionElements>
      <bpmn:outgoing>f1</bpmn:outgoing>
    </bpmn:startEvent>
    <bpmn:sequenceFlow id="f1" sourceRef="Received" targetRef="AwaitConfirmation"/>
    <bpmn:receiveTask id="AwaitConfirmation" name="Await confirmation">
      <bpmn:extensionElements>
        <q:source channel="order-in" name="confirmation"/>
      </bpmn:extensionElements>
      <bpmn:incoming>f1</bpmn:incoming><bpmn:outgoing>f2</bpmn:outgoing>
    </bpmn:receiveTask>
    <bpmn:sequenceFlow id="f2" sourceRef="AwaitConfirmation" targetRef="Done"/>
    <bpmn:endEvent id="Done"><bpmn:incoming>f2</bpmn:incoming></bpmn:endEvent>
  </bpmn:process>
</bpmn:definitions>

orders/channels.yaml binds the channel once:

channels:
  - name: order-in
    transport: http
    bind: "POST /orders"
    codec: json
    auth:
      scheme: apikey
      apikey:
        value: env:ORDERS_KEY

sutra lint orders warns SUTRA.CONFIG.CHANNEL.START_AND_WAIT:

[WARN] SUTRA.CONFIG.CHANNEL.START_AND_WAIT — deployment acme/orders/1.0.0 (dep-…): process 'order' start event 'Received' and the wait at order/AwaitConfirmation are both on channel 'order-in' with message types that can overlap. A message that correlates to a parked instance answers its wait and starts nothing, so a repeated key never starts a second instance; give them disjoint message types or separate channels if one is meant

That is what this walkthrough relies on; the warning is there for a flow that meant to start a second instance with a repeated key. package.yaml labels the package acme/orders/1.0.0, and the engine has ORDERS_KEY set to the key the channel checks.

1. Deploy and start one order

$ sutra package orders --out out-orders
packaged out-orders/orders.sutra (deploymentId dep-<id>)
$ sutra deploy out-orders/orders.sutra --api --engine-url http://localhost:8080
deployed dep-<id> (slot acme--orders--1.0.0, rev 1) — Active
$ curl -X POST http://localhost:8080/orders -H 'X-API-Key: <your key>' \
    -H 'Content-Type: application/json' -d '{"key":"A-1"}'

The instance parks at AwaitConfirmation, findable by its key:

$ curl -s -H "X-API-Key: $SUTRA_ADMIN_API_KEY" 'http://localhost:8080/sutra/instances?alias=orderKey:A-1'
{"instances":[{"instanceId":"…","deploymentId":"dep-<id>","status":"SUSPENDED"}]}

2. Send the confirmation

$ curl -X POST http://localhost:8080/orders -H 'X-API-Key: <your key>' \
    -H 'Content-Type: application/json' -d '{"key":"A-1"}'

The same channel, the same key: the message resumes the parked instance, which completes. No second instance starts: the same query now answers {"instances":[]}.

3. Start another order

$ curl -X POST http://localhost:8080/orders -H 'X-API-Key: <your key>' \
    -H 'Content-Type: application/json' -d '{"key":"B-2"}'

No instance waits for B-2, so this starts a new one, which parks as in step 1: /sutra/instances?alias=orderKey:B-2 lists it, SUSPENDED.

Broadcast channels

On a channel with broadcast: true, a correlated message resumes the process that waits for it, and that process does not also start. Every other process with a start event on the channel still receives the message, as broadcast promises.

What it costs

On a channel no version waits on, nothing: the engine knows from its index which channels have waits, and answers in a few hash lookups. On a channel with a wait that can take the message’s type, each message costs one correlation lookup per live version that waits. A message whose type no wait accepts (for example a start type and a reply type on one channel) is never looked up.

Upgrading to 0.6.22

  • A flow that re-sent a business key on a start-and-wait channel to spawn a second instance now answers the parked one.
  • A message whose key names a running instance that has not reached the wait yet is refused with a retryable WAIT_PENDING answer, not started.
  • An engine database runs two additive migrations at first boot: V305 and V306. V306 builds an index on inbox_seen concurrently; it adds I/O in proportion to the table and blocks nothing. If the engine connects with a non-owning role, grant it SELECT, INSERT on the new deployment_owner table.
  • Dedup records written before the upgrade dedupe within their own deployment only, so a retry that spans both the engine upgrade and a version flip can run again.

Upgrading a deployment in place

When a new version of a tenant and module takes over, the old version drains: its running instances finish on it. Since 0.6.20, a channel only the old version declares stays served by it, starts included, until it retires at zero running instances. Another owner cannot take such an endpoint meanwhile.

The packages

  • orders 1.0.0: start (POST /orders) parks an order on a receive task waiting on release (POST /orders/release, correlated by id). A second process starts on legacy (POST /orders/legacy) and replies legacy handled by orders 1.0.0.
  • orders 2.0.0: keeps start, waits on confirm (POST /orders/confirm) instead, and has no release and no legacy.
  • audit 1.0.0, another module (another owner), binds POST /orders/release.

Steps

  1. Package and deploy orders 1.0.0; start two orders, A-1 and A-2. Both park on release.
  2. Deploy orders 2.0.0. SUTRA.DEPLOY.ENDPOINT.SUPERSEDED says 1.0.0 drains, its instances finish on it, and names the endpoints it keeps: POST /orders/release, POST /orders/legacy.
  3. GET /sutra/deployments: 1.0.0 is under draining, with 2 running instances under drainingInstances.
  4. POST /orders/legacy still runs, on 1.0.0.
  5. Deploy audit: refused with 409, SUTRA.DEPLOY.ENDPOINT.COLLISION, naming 1.0.0 and its 2 running instances. Nothing is stored.
  6. Release A-1 and A-2 on POST /orders/release: both finish on 1.0.0.
  7. On the next sweep 1.0.0 retires: draining is empty, and POST /orders/legacy is 404.
  8. Deploy audit again: it lands.
  9. sutra lint on a package that sends q:send destination="local://acme/orders/2.0.0/start": SUTRA.CONFIG.SEND.LOCAL_DESTINATION_CROSSES_PACKAGE, exit 1. Send on an outbound channel the other package listens on instead.
$ sutra package orders-1 --out out1
packaged out1/orders-1.sutra (deploymentId dep-9f805df1cff68d20777326e9)
[exit 0]

$ sutra deploy out1/orders-1.sutra --api --engine-url http://localhost:8080
deployed dep-9f805df1cff68d20777326e9 (slot acme--orders--1.0.0, rev 1) — Active
[exit 0]

$ curl -X POST /orders -H 'X-API-Key: walk-key' -d '{"id":"A-1"}'
 [HTTP 200]

$ curl -X POST /orders -H 'X-API-Key: walk-key' -d '{"id":"A-2"}'
 [HTTP 200]

$ sutra package orders-2 --out out2
packaged out2/orders-2.sutra (deploymentId dep-200e1644e8045c4c02c0e3d0)
[exit 0]

$ sutra deploy out2/orders-2.sutra --api --engine-url http://localhost:8080
[WARN] SUTRA.DEPLOY.ENDPOINT.SUPERSEDED — deployment acme--orders--2.0.0 takes over from acme--orders--1.0.0, an earlier deployment of the same tenant and module. acme--orders--1.0.0 drains: it takes no new work, and its running instances finish on it. Channels only it declares stay served by it until it retires: POST /orders/release, POST /orders/legacy.
deployed dep-200e1644e8045c4c02c0e3d0 (slot acme--orders--2.0.0, rev 1) — Active
[exit 0]

$ curl /sutra/deployments
{"active":[{"deploymentId":"dep-200e1644e8045c4c02c0e3d0","slot":"acme--orders--2.0.0","phase":"Active","ready":true}],"pending":[],"draining":["dep-9f805df1cff68d20777326e9"],"drainingInstances":{"dep-9f805df1cff68d20777326e9":2},"failed":[]}

$ curl -X POST /orders/legacy -H 'X-API-Key: walk-key' -d '{}'
legacy handled by orders 1.0.0
 [HTTP 200]

$ sutra package audit --out outa
packaged outa/audit.sutra (deploymentId dep-dbd1c12988671263d273fbe1)
[exit 0]

$ sutra deploy outa/audit.sutra --api --engine-url http://localhost:8080
[ERROR] SUTRA.DEPLOY.ENDPOINT.COLLISION — engine rejected the deployment (HTTP 409 Conflict): {"code":"SUTRA.DEPLOY.ENDPOINT.COLLISION","error":"[SUTRA.DEPLOY.ENDPOINT.COLLISION] http endpoint 'POST /orders/release' stays with draining deployment acme--orders--1.0.0 (2 running instances) until it retires"}
[exit 1]

$ curl -X POST /orders/release -H 'X-API-Key: walk-key' -d '{"id":"A-1"}'
 [HTTP 200]

$ curl -X POST /orders/release -H 'X-API-Key: walk-key' -d '{"id":"A-2"}'
 [HTTP 200]

$ # wait for the sweep

$ curl /sutra/deployments
{"active":[{"deploymentId":"dep-200e1644e8045c4c02c0e3d0","slot":"acme--orders--2.0.0","phase":"Active","ready":true}],"pending":[],"draining":[],"drainingInstances":{},"failed":[]}

$ curl -X POST /orders/legacy -H 'X-API-Key: walk-key' -d '{}'
{"type":"urn:bpm:diag:SUTRA.RESOLVE.CHANNEL.UNKNOWN","title":"SUTRA.RESOLVE.CHANNEL.UNKNOWN","status":404,"detail":"No channel bound at 'POST /orders/legacy'","code":"SUTRA.RESOLVE.CHANNEL.UNKNOWN"} [HTTP 404]

$ sutra deploy outa/audit.sutra --api --engine-url http://localhost:8080
deployed dep-dbd1c12988671263d273fbe1 (slot acme--audit--1.0.0, rev 1) — Active
[exit 0]

$ sutra lint crossing
[ERROR] SUTRA.CONFIG.SEND.LOCAL_DESTINATION_CROSSES_PACKAGE — deployment acme/sender/1.0.0 (dep-ffffffffffffffffffffffff): process 'send' node 'Send' sends to 'local://acme/orders/2.0.0/start', which is not a 'transport: local' inbound channel of this package (local://acme/sender/1.0.0/<channel>). A local address does not cross packages, versions of one package included: send on an outbound channel the other package listens on
1 error(s), 1 warning(s)
[exit 1]

If you undeploy the new version

Undeploying 2.0.0 gives 1.0.0 back every channel it declares, the ones it shared with 2.0.0 included, until 1.0.0 retires. Here a new order C-1 on the shared POST /orders starts on 1.0.0 (the admin instance list for 1.0.0 counts 2), both orders finish on it, and once it retires POST /orders is 404, because nothing serves it any more.

$ sutra package orders-1 --out out1
packaged out1/orders-1.sutra (deploymentId dep-9f805df1cff68d20777326e9)
[exit 0]

$ sutra deploy out1/orders-1.sutra --api --engine-url http://localhost:8080
deployed dep-9f805df1cff68d20777326e9 (slot acme--orders--1.0.0, rev 1) — Active
[exit 0]

$ curl -X POST /orders -H 'X-API-Key: walk-key' -d '{"id":"A-1"}'
 [HTTP 200]

$ sutra package orders-2 --out out2
packaged out2/orders-2.sutra (deploymentId dep-200e1644e8045c4c02c0e3d0)
[exit 0]

$ sutra deploy out2/orders-2.sutra --api --engine-url http://localhost:8080
[WARN] SUTRA.DEPLOY.ENDPOINT.SUPERSEDED — deployment acme--orders--2.0.0 takes over from acme--orders--1.0.0, an earlier deployment of the same tenant and module. acme--orders--1.0.0 drains: it takes no new work, and its running instances finish on it. Channels only it declares stay served by it until it retires: POST /orders/release, POST /orders/legacy.
deployed dep-200e1644e8045c4c02c0e3d0 (slot acme--orders--2.0.0, rev 1) — Active
[exit 0]

$ sutra undeploy acme--orders--2.0.0 --api --engine-url http://localhost:8080
undeployed slot acme--orders--2.0.0 — Draining
[exit 0]

$ curl /sutra/deployments
{"active":[],"pending":[],"draining":["dep-9f805df1cff68d20777326e9"],"drainingInstances":{"dep-9f805df1cff68d20777326e9":1},"failed":[]}

$ curl -X POST /orders -H 'X-API-Key: walk-key' -d '{"id":"C-1"}'
 [HTTP 200]

$ # instances on 1.0.0 (dep-9f805df1cff68d20777326e9)
2

$ curl -X POST /orders/release -H 'X-API-Key: walk-key' -d '{"id":"A-1"}'
 [HTTP 200]

$ curl -X POST /orders/release -H 'X-API-Key: walk-key' -d '{"id":"C-1"}'
 [HTTP 200]

$ # wait for the sweep

$ curl /sutra/deployments
{"active":[],"pending":[],"draining":[],"drainingInstances":{},"failed":[]}

$ curl -X POST /orders -H 'X-API-Key: walk-key' -d '{"id":"D-1"}'
{"type":"urn:bpm:diag:SUTRA.RESOLVE.CHANNEL.UNKNOWN","title":"SUTRA.RESOLVE.CHANNEL.UNKNOWN","status":404,"detail":"No channel bound at 'POST /orders'","code":"SUTRA.RESOLVE.CHANNEL.UNKNOWN"} [HTTP 404]

What to watch for

  • A version that keeps starting work on the channels it kept never retires. Watch drainingInstances.
  • In a deployments folder, removing 1.0.0’s file stops its channels.
  • Versions already draining when you upgrade the engine to 0.6.20 stop serving as before.
  • SUTRA.DEPLOY.ENDPOINT.CHANNEL_CHANGED and START_ON_WAIT warn at deploy when 2.0.0 changes a channel 1.0.0 has instances parked on (not shown in this run; the tier-2 suite covers them).

See also One endpoint, one deployment, which covers how the other deployment paths behave.

Acceptance receipts

An HTTP channel with ack-mode: on-persist answers 202 Accepted before the work finishes. Since 0.6.18 the engine tells the caller what it accepted, so nobody has to poll the instance list and match a variable. This walkthrough starts an instance, reads its receipt, repeats the request, and finds the instance again by its alias.

You need the engine and a PostgreSQL database; this was run on PostgreSQL 16. Only the HTTP path was run. Queue transports have no synchronous answer, so they get no receipt.

1. Write the package

package.yaml:

labels:
  "tenant": "default"
  "module": "cases"
  "version": "1.0.0"
engine:
  minContract: 1

channels.yaml. Idempotency-Key is the header that makes a retry the same request:

channels:
  - name: case-open
    transport: http
    bind: "POST /channels/case-open"
    codec: json
    ack-mode: on-persist
    idempotency-key-header: Idempotency-Key
    cloudevents-mode: none
    auth:
      scheme: apikey
      apikey:
        value: env:CASES_API_KEY
        header: X-Api-Key
  - name: case-release
    transport: http
    bind: "POST /channels/case-release"
    codec: json
    ack-mode: on-persist
    cloudevents-mode: none
    auth:
      scheme: apikey
      apikey:
        value: env:CASES_API_KEY
        header: X-Api-Key

bpmn/case-open.bpmn. <q:receipt> names the values this flow chooses to expose. Without it, the receipt carries no flow values of its own:

<?xml version="1.0" encoding="UTF-8"?>
<bpmn:definitions xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL"
                  xmlns:q="urn:sutra:q:1.0"
                  id="Definitions_case_open"
                  targetNamespace="urn:sutra:module:cases:1.0.0">
  <bpmn:process id="case-open" isExecutable="true">
    <bpmn:extensionElements>
      <q:variables>
        <q:variable name="holderId" type="string" sensitive="true"/>
      </q:variables>
      <q:receipt>
        <q:attribute name="caseId" expression="req.caseId"/>
        <q:attribute name="holder" expression="holderId"/>
      </q:receipt>
    </bpmn:extensionElements>
    <bpmn:startEvent id="Start">
      <bpmn:extensionElements>
        <q:source channel="case-open" name="req"/>
        <q:alias name="orderId" expression="req.orderId" unique="true" onConflict="correlate"/>
        <q:alias name="holderId" expression="req.holderId"/>
      </bpmn:extensionElements>
      <bpmn:outgoing>F1</bpmn:outgoing>
    </bpmn:startEvent>
    <bpmn:sequenceFlow id="F1" sourceRef="Start" targetRef="Release"/>
    <bpmn:receiveTask id="Release">
      <bpmn:extensionElements><q:source channel="case-release" name="release"/></bpmn:extensionElements>
      <bpmn:incoming>F1</bpmn:incoming><bpmn:outgoing>F2</bpmn:outgoing>
    </bpmn:receiveTask>
    <bpmn:sequenceFlow id="F2" sourceRef="Release" targetRef="Done"/>
    <bpmn:endEvent id="Done"><bpmn:incoming>F2</bpmn:incoming></bpmn:endEvent>
  </bpmn:process>
</bpmn:definitions>

2. Lint and package it

sutra lint cases
sutra package cases -o deployments

Lint warns about the holder attribute, because it reads a sensitive variable and the caller would only ever see the mask. It is kept here to show the masking. A real flow would drop it.

[WARN] SUTRA.CONFIG.RECEIPT.ATTRIBUTE_SENSITIVE — ... process 'case-open' <q:receipt> attribute 'holder' reads sensitive variable 'holderId', so every receipt shows it as ***REDACTED***. Expose a value that is safe for the caller to see, or remove the attribute.

3. Start the engine

Give the engine a PostgreSQL role that isn’t a superuser, and a database that role owns:

CREATE ROLE sutra_app LOGIN NOSUPERUSER NOBYPASSRLS PASSWORD '<a password>';
CREATE DATABASE sutra OWNER sutra_app;

Start the engine with SUTRA_DATASOURCE_URL pointing at that database as sutra_app, SUTRA_DEPLOYMENTS_DIR pointing at deployments, and CASES_API_KEY. Turn admin auth on with SUTRA_ADMIN_AUTH_SCHEME=apikey, SUTRA_ADMIN_AUTH_KEY_REF=env:SUTRA_ADMIN_API_KEY and SUTRA_ADMIN_API_KEY, because step 6 reads instances. Because this role owns its tables, the engine logs SUTRA.STARTUP.RLS_INERT_POSTURE at startup: row-level security is on but doesn’t apply to the table owner. Before running several tenants on one database, give the engine a role that doesn’t own its tables. Set B to the engine’s address, then wait for it:

curl -s "$B/sutra/health/ready"
{"status":"UP","checks":[{"name":"sutra-loader","status":"UP","data":{"deployments":1,"shards":1}}]}

4. Start a case and read the receipt

curl -si -X POST "$B/channels/case-open" \
  -H 'X-Api-Key: <your key>' -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: try-1' \
  -d '{"orderId":"O-100","caseId":"C-7","holderId":"H-SECRET"}'
HTTP/1.1 202 Accepted
content-type: application/json
sutra-instance-id: 315001d4-74f0-42bc-8b97-8dabe7511b35
sutra-receipt: {"deploymentId":"dep-dceba0e28896cca13aa92d0a","labels":{"module":"cases","tenant":"default","version":"1.0.0"},"duplicate":false,"instanceId":"315001d4-74f0-42bc-8b97-8dabe7511b35","aliases":{"holderId":"***REDACTED***","orderId":"O-100"},"attributes":{"caseId":"C-7","holder":"***REDACTED***"},"idempotencyKey":"try-1"}

{"deploymentId":"dep-dceba0e28896cca13aa92d0a","labels":{"module":"cases","tenant":"default","version":"1.0.0"},"duplicate":false,"instanceId":"315001d4-74f0-42bc-8b97-8dabe7511b35","aliases":{"holderId":"***REDACTED***","orderId":"O-100"},"attributes":{"caseId":"C-7","holder":"***REDACTED***"},"idempotencyKey":"try-1"}

The flow replied nothing, so the receipt is also the body. A flow that replies with <q:reply continue="true"> keeps its own body byte for byte, and the receipt rides only in the headers. H-SECRET appears nowhere: holderId is declared sensitive, so both the alias and the attribute show the mask. An alias built from the payload does show that value to the caller, so alias what you are willing to return.

5. Send it again

Repeat the same request, with the same Idempotency-Key:

HTTP/1.1 202 Accepted
content-type: application/json
sutra-instance-id: 315001d4-74f0-42bc-8b97-8dabe7511b35

{"deploymentId":"dep-dceba0e28896cca13aa92d0a","labels":{"module":"cases","tenant":"default","version":"1.0.0"},"duplicate":true,"instanceId":"315001d4-74f0-42bc-8b97-8dabe7511b35","originalInstanceId":"315001d4-74f0-42bc-8b97-8dabe7511b35","aliases":{},"attributes":{},"idempotencyKey":"try-1"}

Nothing new started. The answer names the instance the first request started. It has no aliases or attributes because nothing ran to produce them. A duplicate on any HTTP channel, including an on-complete one, gets this 202.

6. Find the instance after losing the 202

With admin auth on, reading instances takes the admin key. Without it, or with a wrong one, the engine refuses:

curl -si "$B/sutra/instances?alias=orderId:O-100"
HTTP/1.1 401 Unauthorized

{"error":"unauthorized","reason":"missing or invalid admin auth key/secret"}

Send the key, in X-API-Key unless you set SUTRA_ADMIN_AUTH_HEADER:

curl -s -H "X-API-Key: $SUTRA_ADMIN_API_KEY" "$B/sutra/instances?alias=orderId:O-100"
{"instances":[{"instanceId":"315001d4-74f0-42bc-8b97-8dabe7511b35","deploymentId":"dep-dceba0e28896cca13aa92d0a","status":"SUSPENDED"}]}

An alias nothing is bound to gives {"instances":[]}. The lookup finds live instances. A finished instance’s aliases have retired, so add includeTerminal=true to find it. That form walks the deployment’s alias rows, so don’t poll it. The same gate covers the plain list and GET /sutra/instances/{id}; see Upgrading to 0.6.18 for the case where admin auth is off.

What to know

  • idempotencyKey appears only when the caller supplied a key or a <q:source dedupKey> derived one. Without one, a repeated request starts a new instance, so nothing is echoed.
  • Attribute values are strings. A null is left out. An expression that fails to evaluate is left out and logged, because the work is already accepted.
  • Sutra-Receipt is one line of ASCII JSON. Past 4 KiB it keeps only the identifying fields and says "truncated": true. Sutra-Instance-Id is always there.
  • A duplicate names no instance if the first delivery ended without one, or was recorded before 0.6.18.
  • Two deliveries racing before either commits behave as before: the loser gets SUTRA.INBOUND.DUPLICATE_CONCURRENT.
  • A <q:attribute> with a repeated name is the lint error SUTRA.CONFIG.RECEIPT.ATTRIBUTE_DUPLICATE; one missing its name or expression fails at load.

Answering with a status and headers

Since 0.8.0 a flow that answers a synchronous HTTP request chooses the status and the response headers, so it can behave like a REST API (201 with Location, 404, 409) or like an auth check for a reverse proxy (200 with identity headers, or 401 with a challenge). The reference is Status and response headers. This walkthrough deploys one package with four endpoints and calls each with curl.

You need Docker, curl, the sutra CLI and an empty working directory. It was run as written with the engine built from this release and postgres:16-alpine; the output shown is from that run.

1. Write the package

replies/package.yaml:

labels:
  "tenant": "default"
  "module": "replies"
  "version": "1.0.0"
engine:
  minContract: 1

replies/channels.yaml: four synchronous HTTP channels with the same key on each:

channels:
  - name: items
    transport: http
    bind: "POST /channels/items"
    codec: json
    auth:
      scheme: apikey
      apikey:
        value: env:REPLIES_API_KEY
        header: X-Api-Key
  - name: lookup
    transport: http
    bind: "POST /channels/lookup"
    codec: json
    auth:
      scheme: apikey
      apikey:
        value: env:REPLIES_API_KEY
        header: X-Api-Key
  - name: check
    transport: http
    bind: "POST /channels/check"
    codec: json
    auth:
      scheme: apikey
      apikey:
        value: env:REPLIES_API_KEY
        header: X-Api-Key
  - name: bad
    transport: http
    bind: "POST /channels/bad"
    codec: json
    auth:
      scheme: apikey
      apikey:
        value: env:REPLIES_API_KEY
        header: X-Api-Key

replies/templates/item.hbs:

{"id":"{{req.id}}"}

replies/bpmn/reply-status.bpmn, four processes:

<?xml version="1.0" encoding="UTF-8"?>
<bpmn:definitions xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL"
                  xmlns:q="urn:sutra:q:1.0"
                  targetNamespace="urn:sutra:module:replies:1.0.0">
  <!-- 201 with a Location -->
  <bpmn:process id="create" isExecutable="true">
    <bpmn:startEvent id="CreateStart">
      <bpmn:extensionElements><q:source channel="items" name="req"/></bpmn:extensionElements>
    </bpmn:startEvent>
    <bpmn:sequenceFlow id="C1" sourceRef="CreateStart" targetRef="Create"/>
    <bpmn:serviceTask id="Create" implementation="item.hbs">
      <bpmn:extensionElements>
        <q:reply contentType="application/json" status="201">
          <q:header name="Location" value="'/items/' + req.id"/>
          <q:header name="X-Request-Id" value="req.id"/>
        </q:reply>
      </bpmn:extensionElements>
    </bpmn:serviceTask>
    <bpmn:sequenceFlow id="C2" sourceRef="Create" targetRef="CreateDone"/>
    <bpmn:endEvent id="CreateDone"/>
  </bpmn:process>

  <!-- a status chosen by FEEL -->
  <bpmn:process id="lookup" isExecutable="true">
    <bpmn:startEvent id="LookupStart">
      <bpmn:extensionElements><q:source channel="lookup" name="req"/></bpmn:extensionElements>
    </bpmn:startEvent>
    <bpmn:sequenceFlow id="L1" sourceRef="LookupStart" targetRef="Lookup"/>
    <bpmn:serviceTask id="Lookup" implementation="item.hbs">
      <bpmn:extensionElements>
        <q:reply contentType="application/json" status="if req.id = '42' then 200 else 404"/>
      </bpmn:extensionElements>
    </bpmn:serviceTask>
    <bpmn:sequenceFlow id="L2" sourceRef="Lookup" targetRef="LookupDone"/>
    <bpmn:endEvent id="LookupDone"/>
  </bpmn:process>

  <!-- an auth check: no body, 200 with identity headers or 401 with a challenge -->
  <bpmn:process id="check" isExecutable="true">
    <bpmn:startEvent id="CheckStart">
      <bpmn:extensionElements><q:source channel="check" name="req"/></bpmn:extensionElements>
    </bpmn:startEvent>
    <bpmn:sequenceFlow id="K1" sourceRef="CheckStart" targetRef="CheckDone"/>
    <bpmn:endEvent id="CheckDone">
      <bpmn:extensionElements>
        <q:reply status="if req.token = 'good' then 200 else 401">
          <q:header name="X-User-Id" value="if req.token = 'good' then req.user else null"/>
          <q:header name="WWW-Authenticate" value="if req.token = 'good' then null else 'Bearer realm=&quot;sim&quot;'"/>
        </q:reply>
      </bpmn:extensionElements>
    </bpmn:endEvent>
  </bpmn:process>

  <!-- whatever status the caller sends: shows the refusal -->
  <bpmn:process id="bad" isExecutable="true">
    <bpmn:startEvent id="BadStart">
      <bpmn:extensionElements><q:source channel="bad" name="req"/></bpmn:extensionElements>
    </bpmn:startEvent>
    <bpmn:sequenceFlow id="B1" sourceRef="BadStart" targetRef="BadDone"/>
    <bpmn:endEvent id="BadDone">
      <bpmn:extensionElements><q:reply status="req.code"/></bpmn:extensionElements>
    </bpmn:endEvent>
  </bpmn:process>
</bpmn:definitions>

create answers 201 with two headers and the template’s body. lookup picks its status with FEEL. check renders no body: its end event answers 200 with X-User-Id for the right token, else 401 with WWW-Authenticate; a header whose value is null is left out. bad sends whatever status the caller asks for, to show what happens when it cannot be sent.

2. Check it

sutra lint replies/

Only the schema-less-channel warnings appear:

[WARN] SUTRA.CHANNEL.NO_SCHEMA — deployment default/replies/1.0.0 (dep-ffffffffffffffffffffffff): channel 'items' has 1 subscriber(s) but its codec (json) declares no message types — ...
...
0 error(s), 4 warning(s)

A literal status is checked here, before anything is deployed. Change lookup’s status to status="302" and lint refuses it, because a redirect needs a Location; status="99" is refused because a 1xx is not a final answer:

[ERROR] SUTRA.OUTBOUND.REPLY_STATUS_INVALID — deployment default/replies/1.0.0 (dep-ffffffffffffffffffffffff): process 'lookup' node 'Lookup' <q:reply>: status 302 is a redirect, which needs a Location header (declare <q:header name="Location">, and make sure it is not null).
1 error(s), 4 warning(s)
[ERROR] SUTRA.OUTBOUND.REPLY_STATUS_INVALID — deployment default/replies/1.0.0 (dep-ffffffffffffffffffffffff): process 'lookup' node 'Lookup' <q:reply>: status 99 is outside 200-599; a reply is a final answer, so a 1xx is refused too.
1 error(s), 4 warning(s)

Change it back.

3. Package it and start the engine

mkdir -p deployments
sutra package replies --out deployments
chmod -R a+rX deployments
docker network create replies-net
docker run -d --name replies-db --network replies-net -e POSTGRES_PASSWORD=dev-only postgres:16-alpine
until docker exec replies-db pg_isready -U postgres -h 127.0.0.1; do sleep 1; done
sleep 2
docker exec replies-db psql -U postgres \
  -c "CREATE ROLE sutra_engine LOGIN PASSWORD 'dev-only' NOSUPERUSER NOBYPASSRLS" \
  -c "ALTER SCHEMA public OWNER TO sutra_engine"
docker run -d --name replies-engine --network replies-net -p 127.0.0.1::8080 \
  -v "$PWD/deployments:/deployments:ro" -e SUTRA_DEPLOYMENTS_DIR=/deployments \
  -e SUTRA_DATASOURCE_URL=postgres://replies-db:5432/postgres \
  -e SUTRA_DATASOURCE_USERNAME=sutra_engine -e SUTRA_DATASOURCE_PASSWORD=dev-only \
  -e REPLIES_API_KEY=demo-key \
  ghcr.io/startr-trade/sutra:0.8.3
PORT=$(docker port replies-engine 8080 | head -1 | cut -d: -f2)
until curl -sf "http://127.0.0.1:$PORT/sutra/health/ready"; do sleep 1; done
{"status":"UP","checks":[{"name":"sutra-loader","status":"UP","data":{"deployments":1,"shards":1}}]}

4. Call it

A create answers 201 with its Location:

curl -si -X POST "localhost:$PORT/channels/items" -H 'X-Api-Key: demo-key' \
  -H 'Content-Type: application/json' -d '{"id":"7"}'
HTTP/1.1 201 Created
content-type: application/json
location: /items/7
x-request-id: 7
content-length: 11
date: Thu, 19 Nov 2026 09:13:19 GMT

{"id":"7"}

A lookup answers 200 for {"id":"42"} and 404 for anything else, with the flow’s body either way:

curl -si -X POST "localhost:$PORT/channels/lookup" -H 'X-Api-Key: demo-key' \
  -H 'Content-Type: application/json' -d '{"id":"41"}'
HTTP/1.1 404 Not Found
content-type: application/json
content-length: 12
date: Thu, 19 Nov 2026 09:13:19 GMT

{"id":"41"}

The auth check, with the right token and then a wrong one:

curl -si -X POST "localhost:$PORT/channels/check" -H 'X-Api-Key: demo-key' \
  -H 'Content-Type: application/json' -d '{"token":"good","user":"u-1"}'
curl -si -X POST "localhost:$PORT/channels/check" -H 'X-Api-Key: demo-key' \
  -H 'Content-Type: application/json' -d '{"token":"bad"}'
HTTP/1.1 200 OK
content-type: application/json
x-user-id: u-1
content-length: 0
date: Thu, 19 Nov 2026 09:13:19 GMT

HTTP/1.1 401 Unauthorized
content-type: application/json
www-authenticate: Bearer realm="sim"
content-length: 0
date: Thu, 19 Nov 2026 09:13:19 GMT

Each answer carries only the header whose value was not null. A reverse proxy that runs a subrequest per call (an auth_request-style check) can pass X-User-Id on to the upstream.

5. A status that cannot be sent

A computed status is checked when the answer is sent. bad with {"code":700}:

curl -si -X POST "localhost:$PORT/channels/bad" -H 'X-Api-Key: demo-key' \
  -H 'Content-Type: application/json' -d '{"code":700}'
HTTP/1.1 500 Internal Server Error
content-type: application/problem+json
content-length: 355
date: Thu, 19 Nov 2026 09:13:19 GMT

{"type":"urn:bpm:diag:SUTRA.OUTBOUND.REPLY_STATUS_INVALID","title":"SUTRA.OUTBOUND.REPLY_STATUS_INVALID","status":500,"detail":"The reply of process 'bad' at node 'BadDone': status 700 is outside 200-599; a reply is a final answer, so a 1xx is refused too.","code":"SUTRA.OUTBOUND.REPLY_STATUS_INVALID","attributes":{"nodeId":"BadDone","processId":"bad"}}

The same call with {"code":409} answers HTTP/1.1 409 Conflict with an empty body. A header value with a line break or another control character is refused the same way, with SUTRA.OUTBOUND.REPLY_HEADER_INVALID; the problem names the header, never its value.

6. Clean up

docker rm -f replies-engine replies-db
docker network rm replies-net

Where it does not apply

  • An ack-mode: on-persist channel always answers 202, the acceptance; the reply’s headers are added to it.
  • A broker intake (Kafka, AMQP, …) has no status.
  • sutra lint warns about both with SUTRA.CHANNEL.REPLY_STATUS_IGNORED.
  • A <q:reply> with a destination goes out of band, and its headers travel with that message.

External tasks: the pull worker surface

The engine is otherwise push-only: an outbound emission lands in the outbox and the relay dials a transport. A channel declaring transport: pull inverts that last hop — instead of dialing anything, the delivery is parked as a fetchable task, and workers come and get it.

That is the whole feature. It exists for work the engine cannot dial out to: a worker behind a NAT, on a laptop, in a language that has no inbound listener, or simply one that would rather poll than be called.

Declaring a pull channel

channels:
  - name: score-request
    transport: pull
    bind: "pull://acme/scoring/1.0.0/score-request"

Nothing in the BPMN changes. A <q:send> or a channel-call <bpmn:serviceTask> targeting this channel behaves exactly as it would over HTTP or a broker; only the last hop is different. And because the worker’s answer comes back in on the same channel, the author’s <q:alias> correlation is unchanged too — a pull task is not a new resume path.

Pull needs a datasource: a parked task is a database row. Without one the surface answers 503.

The worker protocol

Three operations, all under /sutra/external-tasks.

Fetch and lock

POST /sutra/external-tasks/fetch-and-lock
{ "workerId": "scorer-7", "channels": ["score-request"],
  "lockDuration": "PT1M", "maxTasks": 10, "asyncResponseTimeout": "PT20S" }

Claims up to maxTasks fetchable tasks on the named channels and locks them to workerId. A worker names topics (channel names), never deployment ids — the fetch walks the live deployment set for you.

When nothing is available the request is held open as a bounded long poll: it wakes the moment a task is parked on one of the channels you asked for, and answers with an empty list when asyncResponseTimeout elapses. It never hangs, and the wait is always bounded by the operator’s ceiling.

Each returned task carries its id, channel, instance id, headers, body and content type, its attempt count, and its remaining failure budget.

Complete

POST /sutra/external-tasks/{id}/complete
{ "workerId": "scorer-7", "result": { ... } }

Feeds the worker’s result back through the engine’s ordinary inbound path — the same seam every transport delivers through. Correlation, validation, and inbox dedup all behave exactly as they do for a pushed reply; there is no second resume entry point.

Omitting the result re-delivers the original request payload. That is the fire-and-forget shape: the work happened outside, and the flow is waiting only on the fact of it.

Failure

POST /sutra/external-tasks/{id}/failure
{ "workerId": "scorer-7", "errorMessage": "downstream 500", "retries": 2, "retryTimeout": "PT30S" }

Releases the lock and spends one of the task’s retries — or sets the remaining budget explicitly — deferring the next fetch by retryTimeout.

Lock expiry is the only recovery mechanism you need

A locked task is invisible to every other worker until its lock expires, at which point it becomes fetchable again. There is no sweeper, no reaper, and no timeout job: expiry is part of the predicate that decides what a fetch may claim, so a worker that dies mid-task costs exactly one lock duration and never costs the work.

stateDiagram-v2
    [*] --> fetchable: the pull sink parks<br/>the delivery as a task row
    fetchable --> locked: fetch-and-lock<br/>workerId + lockDuration
    locked --> fetchable: lock expires — no sweeper,<br/>it is the claim predicate
    locked --> fetchable: failure with budget left,<br/>after retryTimeout
    locked --> [*]: complete — dispatched,<br/>then the row is deleted
    locked --> terminal: budget spent —<br/>retained with its last error

There is no reaper because an expired lock is not a state anyone has to clean up: it is simply no longer an obstacle to the next fetch, so availability is instant rather than one sweep tick late.

A completion or failure from a worker that no longer owns the lock fails closed, and the refusal names which of the three situations it is:

CodeMeaning
SUTRA.EXTERNAL_TASK.LOCK_LOSTYour lock expired or was released. The task is fetchable again — by you or by anyone.
SUTRA.EXTERNAL_TASK.LOCK_HELDAnother worker holds it.
SUTRA.EXTERNAL_TASK.TERMINALThe task spent its budget and can never be completed or failed again.
SUTRA.EXTERNAL_TASK.NOT_FOUNDNo such task on any live deployment.

A stale worker never receives a 200 it could mistake for success.

At-least-once, and what makes that safe

The task row is deleted only after the engine has accepted the completion. A crash in the window between the two re-offers the task, so the work is never lost — the surface is at-least-once, deliberately, because the inverse ordering (delete, then dispatch) would be at-most-once and would drop work outright on the same crash.

sequenceDiagram
    participant W as Worker
    participant E as Engine
    participant TR as Task row

    W->>E: complete — workerId + result
    E->>TR: ownership-guarded — is this lock still yours?
    E->>E: dispatch the result through the ordinary inbound path
    Note over E,TR: crash here and the task is re-offered — inbox dedup absorbs the duplicate
    E->>TR: delete the row
    E-->>W: 200

The ordering is the whole guarantee: the crash window can only produce a duplicate the idempotency key already absorbs, where the inverse order would lose the work with nothing recording that it ever happened.

What makes the duplicate harmless is that each task carries the originating outbound delivery’s key as an explicit idempotency key, and the completion re-enters under it. Inbox dedup absorbs the second attempt. A worker does not have to do anything to get this — it is a property of the surface, not of the worker.

If the engine refuses a completion on the inbound path (a validation reject, say), the answer is 422 with the engine’s own code carried as attributes.causeCode — which is what tells the worker whether re-fetching later can ever help. The task is retained and becomes fetchable again.

Two reserved headers a worker never sees

For completeness: internally, the dispatcher stamps two reserved headers — sutra-park-deployment and sutra-park-instance — onto a delivery whose destination scheme is pull, and the pull sink strips them as it parks the task. They carry the owning deployment (the isolation key) and instance to the parking side without widening the transport contract every other sink implements, and the scheme gate means they can never leak onto a network transport. They never appear in a fetched task’s headers; nothing a worker builds should look for them, and their presence in a payload you construct yourself has no effect — the sink’s own stamp always wins.

The worker retry budget

A freshly parked task starts with sutra.external-task.retries failures (default 3). Each reported failure spends one and defers the next fetch by sutra.external-task.retry-timeout (default PT10S).

A spent budget makes the task terminal: never fetched again, retained with its last error. That is the pull-side twin of the outbox’s poison horizon, and it exists for the same reason — so “we gave up” can never degrade into “it silently vanished”. A terminal task no longer counts toward its deployment’s retirement quiescence gate.

Bounds are rejections, never clamps

Every knob a worker can send is a ceiling an operator sets, and a request over the ceiling is a 400 — not a silent adjustment down:

KeyDefaultCeiling on
sutra.external-task.default-lock-durationPT30SThe lock granted when a fetch names none; also the completion grace window.
sutra.external-task.max-lock-durationPT1HlockDuration.
sutra.external-task.max-async-response-timeoutPT30SasyncResponseTimeout (and its default).
sutra.external-task.max-tasks100maxTasks (and its default).
sutra.external-task.retries3The starting failure budget.
sutra.external-task.retry-timeoutPT10SBackoff before a failed-with-budget task is fetchable again.

The reason it is a reject is a correctness one: a worker that believes it holds a longer lock than it actually does is a duplicate-execution bug waiting to happen. Silently clamping would create exactly that belief. Durations are ISO-8601 everywhere on this surface, the same grammar the engine’s cadence keys and <q:retry> use.

The engine also boots fail-closed if default-lock-duration exceeds its own ceiling or is zero.

Posture

These are operate-surface routes (/sutra/*), not administrative ones: a completion is an ordinary delivery, not a privileged control operation. They carry the same cluster-internal posture the rest of /sutra/* does. A deployment that needs authenticated workers puts them behind the same ingress policy the rest of the operate surface already needs — see Configuration reference for the separately gated admin surface.

How it composes with <q:retry>

Two budgets, at two layers, that never overlap:

Governed byCountsOn exhaustion
Worker failuressutra.external-task.retries (operator config)A worker reporting a failure on a fetched taskThe task turns terminal; the instance stays parked on its wait
Task failures<q:retry> (the author’s model)A route-less <q:timeout> firing, or a poisoned request deliveryThe instance fails durably

A worker retrying its own work never touches the instance. A <q:retry> policy on a pull-backed channel call still governs the task-level outcome — its timeout is still the thing that eventually fires if no worker ever completes. See Retries, history, and schedules.

Next

Retries, history, and schedules

Three execution-semantics features an author reaches for once a flow has to survive contact with an unreliable world: a per-task retry policy, execution history that outlives the instance, and the full set of BPMN timer schedules.

<q:retry> — a per-task retry policy

A <bpmn:serviceTask> may declare a retry policy inline. It applies to both kinds of service task: a registered task (a function the embedding application registered) and a channel-call task (implementation="channel:<name>" — the wait-state call every pure-BPMN deployment uses through the shipped binary).

<bpmn:serviceTask id="Score" implementation="registered:score">
  <bpmn:extensionElements>
    <q:retry maxAttempts="3" initialDelay="PT1S" backoffCoefficient="2.0"
             maxDelay="PT5M" nonRetryableCodes="SUTRA.TASK.VALIDATION"/>
  </bpmn:extensionElements>
</bpmn:serviceTask>
AttributeDefaultMeaning
maxAttemptsrequiredTotal invocation budget, including the first attempt.
initialDelayPT1SWait before attempt 2.
backoffCoefficient2.0Delay multiplier — attempt n+1 waits min(initialDelay × coefficient^(n-1), maxDelay).
maxDelayPT5MCeiling on any single backoff wait.
nonRetryableCodes—Structured codes that fail immediately, budget notwithstanding.

So maxAttempts="4" initialDelay="PT1S" backoffCoefficient="2.0" waits 1 s, then 2 s, then 4 s between its four attempts — each capped at maxDelay.

A retry wait is a durable park, never a sleep

This is the load-bearing property. A failed attempt with budget remaining persists the instance with the failed task still pending and an armed timer due at the backoff instant; the ordinary timer poller re-drives it when that instant arrives.

Nothing sleeps, and nothing is held in memory. The consequences are the ones you want:

  • the backoff survives a restart, a rolling upgrade, and a hot-deploy (a retry park is pinned to its deployment like every other park);
  • no execution lane is blocked during the wait, so a task backing off costs other instances nothing;
  • a PT5M backoff is a PT5M backoff whether or not the replica that started it is still alive.
sequenceDiagram
    participant L as Execution lane
    participant DB as PostgreSQL
    participant T as Timer poller

    loop while the budget holds
        L->>L: attempt fails
        L->>DB: re-park — task still pending, timer armed at the backoff instant
        Note over L,DB: a durable park, not a sleep — the lane is free for other instances
        T->>DB: backoff instant due
        T->>L: re-drive the task
    end
    L->>DB: budget spent — durable FAILED, retained and inspectable

A backoff is a row in a database, not a held thread — which is why it survives a restart and a hot-deploy, and why a task waiting five minutes costs every other instance nothing.

Declaring <q:retry> on a task makes its process stateful — the park needs persistence, so a process that was otherwise run-to-completion now requires a configured datasource.

What counts as a failed attempt

The failure set is deliberately narrow and different for each task kind.

On a registered task: an uncaught error from the task function.

On a channel-call task, exactly two things:

  • the route-less <q:timeout> boundary firing before the correlated response arrived (classification SUTRA.DISPATCH.CHANNEL_CALL.TIMEOUT). Without a policy a fired timeout is a catchable BPMN error; with one it becomes a retryable task failure first;
  • the request delivery being marked terminally poisoned by the outbox attempt ceiling (classification SUTRA.OUTBOUND.DELIVERY_ATTEMPTS_EXHAUSTED, reachable only when sutra.outbox.retry.max-attempts is configured). The failure reaches the task the moment the delivery gives up, rather than waiting out the whole timeout window.

Both classifications are stable structured codes, so nonRetryableCodes can name them — “retry a poisoned delivery but never a timeout” is expressible.

What is never a failed attempt:

  • A correlated business response. The counterpart answered. Whatever the answer says is the process’s business to route on; re-sending would double-submit. A response is a completion, full stop.
  • A BPMN error. Errors route to their boundary events, unchanged, on either task kind.

Modelled outcomes always beat a policy

A timer boundary event with drawn outgoing flows is a modelled outcome — the author has said what happens on timeout, so that is what happens, exactly as a BPMN error routes to its boundary instead of consuming a retry budget. Which is why the loader refuses the combination: a channel-call <q:retry> requires the route-less <q:timeout> form, and a routed timer boundary alongside a retry policy is a load error (SUTRA.CONFIG.BPMN.RETRY_NOT_APPLICABLE). A policy that could never fire would otherwise load as a silent near-no-op.

What a re-drive actually does on a channel call

Re-driving a channel call is not “wait for the same answer again” — it re-issues the request:

  • a fresh outbound request built from the same persisted variables, with a fresh idempotency key on the wire (so a counterpart doing its own dedup sees a genuinely new request, not a redelivery);
  • a fresh timeout window;
  • the dead attempt’s outstanding request withdrawn, so a late answer to a superseded attempt cannot race the new one.

A response that arrives for the dead attempt is refused with SUTRA.DISPATCH.CHANNEL_CALL.RETRY_PENDING — an honest verdict rather than a “no such instance” miss, because the correlation is deliberately kept alive. Once the re-issued request goes out, the same correlation serves its response normally.

sequenceDiagram
    participant E as Engine
    participant C as Counterpart

    E->>C: attempt n, idempotency key K1
    Note over E: route-less q:timeout fires — a retryable task failure
    E->>E: park the backoff, withdraw attempt n's pending delivery
    C-->>E: late answer to attempt n
    E-->>C: refused — SUTRA.DISPATCH.CHANNEL_CALL.RETRY_PENDING
    E->>C: attempt n+1, fresh key K2, fresh timeout window
    C-->>E: answer — a completion, never a retry trigger

A re-drive is a new request rather than a second wait for the old one, so the counterpart’s own deduplication sees it as new; and the superseded attempt is refused by name instead of being silently absorbed or lost.

Exhaustion

On a channel-call task whose failure the flow catches (a boundary or an error event sub-process), a spent budget raises the failure’s own code as a BPMN error on the task; see Catching a failed call. Otherwise a spent budget — or a nonRetryableCodes hit — fails the instance durably with SUTRA.RUNTIME.RETRY.EXHAUSTED. The instance becomes FAILED: retained, inspectable, blocking its deployment’s retirement, and repairable through instance migration. See Replica semantics.

Two budgets that never overlap

If a channel uses the pull transport, its worker-side failures are governed by the external task’s own budget (sutra.external-task.retries), not by <q:retry>. The two live at different layers and never interact — see External tasks.

The outbox poison ceiling

Outbound deliveries retry with backoff forever by default: an unreachable counterpart is treated as temporarily unreachable, and giving up silently is worse than waiting. Since 0.6.20 each failed attempt that will be retried logs a WARN with the destination, the cause’s code and message, the attempt number and next_attempt_at. The backoff base, sutra.outbox.retry.base-delay, defaults to PT1S.

sutra.outbox.retry.max-attempts is the opt-in ceiling. A delivery that exhausts it becomes terminally poisoned: retained with its last error, never retried again, one incident recorded if the delivery was required, and — importantly — no longer counted by the draining-deployment retirement gate. “We gave up” is a durable, visible state, never a silent disappearance.

Configuring the ceiling is also what makes SUTRA.OUTBOUND.DELIVERY_ATTEMPTS_EXHAUSTED reachable as a <q:retry> failure at all.

Execution history: a finished instance is not a 404

A terminal instance used to vanish. It doesn’t: the terminal step re-stamps the stored snapshot to COMPLETED / TERMINATED in the same transaction that resolves its waits, retires its correlation aliases, and enqueues its final emissions.

SurfaceWhat it answers for a finished instance
GET /sutra/instances/{id} and its admin twinThe retained projection — status, node progress, variables — for as long as retention keeps the row.
GET /admin/instancesExcludes finished instances by default; includeTerminal=true (or a terminal status filter) includes them.
GET /admin/instances/{id}/historyThe audit journal, paged: how it got there, event by event, with whatever payload the process captured. Page with ?afterOrdinal= and the response’s nextAfterOrdinal: each event’s ordinal is its position in the journal, assigned in commit order, so no event written later lands behind a cursor already returned. ?afterSeq= still works, but can miss a transport event, which shares the seq of the engine event it followed. Admin-only — an audit row can carry business data.

Retention. sutra.instance.retention (ISO-8601, default P7D) is how long terminal snapshots stay queryable; a lease-gated sweeper purges past-window rows on sutra.instance.retention-sweep-interval (default PT1H). PT0S restores delete-at-completion for a deployment that wants no history at all.

FAILED is always retained, regardless of the window — see above. And the audit journal itself stays opt-in (sutra.audit.sql): with it off, the history endpoint answers an explanatory empty shape rather than a misleading 404, and the retained terminal snapshot still answers the inspect route.

stateDiagram-v2
    [*] --> RUNNING
    RUNNING --> COMPLETED: terminal step re-stamps the snapshot
    RUNNING --> TERMINATED: terminal step re-stamps the snapshot
    RUNNING --> FAILED: budget spent or non-retryable code
    COMPLETED --> [*]: swept past sutra.instance.retention
    TERMINATED --> [*]: swept past sutra.instance.retention
    note right of FAILED
      Retained regardless of the window.
      Blocks its deployment's retirement; repairable by migration.
    end note

Finishing is a status change, not a disappearance — and FAILED is the one status the retention sweeper never reaches.

Timer completeness: durations, dates, cycles, and timer starts

All three BPMN timer definitions are supported, on all three timer hosts (start event, intermediate catch, boundary event):

DefinitionShapeExample
timeDurationISO-8601 duration — relative to armingPT30M, P2D
timeDateISO-8601 instant — an absolute deadline2026-12-31T23:59:00Z
timeCycleISO-8601 repeating intervalR3/PT12H, R/P1D

Deliberately not supported, and rejected as such rather than silently mis-parsed: cron syntax, and calendar-arithmetic durations (P1M meaning “one calendar month”). A malformed expression and an unsupported-but-well-formed one produce different diagnostics — “you wrote it wrong” and “the engine does not do that” are different problems.

Timer start events are schedules

A start event carrying a timer definition is a schedule, not a wait: it has no instance until it fires. Schedules are armed when a deployment becomes active, re-armed on boot, and resolved when the deployment stops being active — so hot-deploy handoff, undeploy, and retirement all need no extra operator step.

A fire mints an instance through the ordinary dispatch path, with empty variables: tenancy, quotas, audit capture, and coverage marking all apply exactly as they do for a message-triggered start.

Two rules keep cycles honest:

  • A cycle never bursts. If the engine was down across three occurrences of an R/PT1H cycle, it fires once and moves to the next slot — missed occurrences are coalesced, not replayed.
  • A repeat budget is spent by the grid, not by the firing. Each skipped slot of an R3/PT1H consumes one repeat, so the schedule can never outlive its third slot.

A start event carrying both a <q:source> and a timer definition is a load error — a start is triggered by a message or by a clock, never ambiguously by both.

Timer intervals from FEEL

Since 0.6.4. Every interval above can be computed instead of written down. That covers the body of a <timeDuration>, <timeDate> or <timeCycle>, a <q:timeout duration>, and a <q:retry initialDelay> or maxDelay. The rule is literal first, else FEEL:

The textWhat it is
parses as an ISO-8601 literal of its form (PT30M, 2026-12-31T23:59:00Z, R3/PT12H)that literal, exactly as before
looks like a literal but is written wrong (PT5X, P1M, 2026-03-01, R0/PT1H)a load error, with the same code as before
anything else that parses as FEEL (payload.window, config.retryDelay)an expression
anything elsea load error

Since 0.6.5 an expression may read only names something provides: config, a declared variable or data object, the payload, or a variable the process (or a process that calls it) sets before the timer. Any other name would be null when the timer is armed, so sutra lint, sutra package and the engine refuse it with SUTRA.CONFIG.BPMN.TIMER_EXPRESSION_UNKNOWN_NAME. A literal written wrong now says which literal was meant. From sutra lint on a package with one of each:

[ERROR] SUTRA.DEPLOY.PACKAGE.CONFIG_INVALID — BPMN bpmn/wait.bpmn fails to parse: [SUTRA.DISPATCH.TIMER.DURATION_INVALID] <intermediateCatchEvent> Pause timer is not schedulable: 'PT10SEC' has an unparseable number ''; did you mean PT10S?
[ERROR] SUTRA.CONFIG.BPMN.TIMER_EXPRESSION_UNKNOWN_NAME — deployment default/window/1.0.0 (dep-ffffffffffffffffffffffff): process 'wait' node 'Hold' timeDuration expression 'holdFor' reads 'holdFor', which nothing in scope provides (not config, a declared variable, the payload or a variable the process sets), so it would be null when the timer is armed. Declare it (<q:variable>), set it before the timer, or read it from config.<key>.

An expression is evaluated once, when its timer is armed:

TimerArmed whenThe expression can read
intermediate catch (also an event-based gateway arm)the token arrivesthe instance’s variables and config
boundary (timeDuration, timeDate)the host activity parksthe instance’s variables and config
<q:timeout>the call parks (it is a boundary)the instance’s variables and config
<q:retry> delayseach failed attempt arms its backoffthe instance’s variables and config
start event (timeDuration, timeDate, timeCycle)the deployment activatesconfig only

The instance’s variables are whatever the instance holds at that moment: the payload, a DMN decision’s output, a value read from a store. config holds the deployment’s properties:, all as strings. A start event has no instance yet, so its expression may read config and nothing else.

What the expression must return:

FormAccepted
durationan ISO-8601 duration string, or a FEEL days-and-time duration (duration("PT" + string(n) + "S"))
datean instant string with a zone, or a FEEL date and time with an offset or zone
cyclea repeating-interval string (R/PT1H)

A null, a number, a negative or calendar-length duration (P1M), an instant with no zone, or any other value fails the instance once with SUTRA.RUNTIME.TIMER.EXPRESSION_INVALID. It gets no default and no retry: <q:retry> does not count the failure, and an error boundary does not catch it. If the timer is armed by the request that starts the instance, the request is refused with that code and nothing is stored. On a start event, the deployment refuses to activate.

Only the instant the expression resolves to is stored: the wait row’s due time, the retry backoff, the schedule row’s spec. A resume, a replay, a restart or a replica hand-off reads that instant and never evaluates the expression again. A loop that comes back to the timer arms it again, so the expression is evaluated again.

Walkthrough: a delivery window

examples/delivery-window waits at a receive task for a delivery and flags the order late when the window closes. The order can name its own window. Otherwise the window is the deployment’s default:

POST /channels/orders             { orderId, window? }
  AwaitDelivery   receive task, answered on POST /channels/orders/delivered { orderId }
                  boundary WindowClosed: payload.window, else config.defaultWindow
  -> Delivered    a delivery inside the window
  -> Late         the window closed first
<bpmn:boundaryEvent id="WindowClosed" name="No delivery in the window" attachedToRef="AwaitDelivery">
  <bpmn:outgoing>F3</bpmn:outgoing>
  <bpmn:timerEventDefinition id="WindowClosed_Timer">
    <bpmn:timeDuration>if payload.window = null then config.defaultWindow else payload.window</bpmn:timeDuration>
  </bpmn:timerEventDefinition>
</bpmn:boundaryEvent>
# package.yaml
properties:
  defaultWindow: "${DEFAULT_WINDOW:PT10S}"

The example’s compose.yaml gives the engine DEFAULT_WINDOW=PT8S.

Prerequisites: Before you try a walkthrough, with an engine image and a sutra CLI of 0.6.4 or later.

1. Start the stack and deploy the package. The timers helper lists the timer rows, with how many seconds after arming each one is due:

C=examples/delivery-window/deploy/compose.yaml
docker compose -f $C up -d --wait
sutra package examples/delivery-window/deployments-src/default--window--1.0.0 \
    --out examples/delivery-window/deploy/deployments 2>/dev/null
PORT=$(docker compose -f $C port engine 8080 | cut -d: -f2)
H=(-H 'Content-Type: application/json' -H 'X-Api-Key: dev-only-window-key')
until curl -sf "http://127.0.0.1:$PORT/sutra/deployments" | grep -q '"phase":"Active"'; do sleep 1; done
send() { curl -s -w " HTTP %{http_code}\n" -X POST "http://127.0.0.1:$PORT/channels/orders$1" "${H[@]}" -d "$2"; }
timers() {
  docker compose -f $C exec -T engine-db psql -U postgres -d sutra -At -c \
    "select node_id, status, extract(epoch from timer_due_at - created_at)::int as seconds
       from waiting_event where kind = 'TIMER' order by created_at"
}
id_of() { curl -s "http://127.0.0.1:$PORT/admin/instances/by-alias/orderId/$1" | python3 -c 'import json,sys; print(json.load(sys.stdin)["instanceId"])'; }
status_of() { curl -s "http://127.0.0.1:$PORT/sutra/instances/$1" | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["status"], d.get("failureCode") or "")'; }
packaged examples/delivery-window/deploy/deployments/default--window--1.0.0.sutra (deploymentId dep-99c53c0c698e17b631a9fdcf)

2. An order with its own window. The boundary is due 3 seconds after the receive task parked. Nothing is delivered, so after 5 seconds the instance has taken the Late path:

send "" '{"orderId":"o-1","window":"PT3S"}'
O1=$(id_of o-1)
timers
sleep 5
status_of $O1
curl -s "http://127.0.0.1:$PORT/admin/instances/$O1/history" | python3 -c '
import json, sys
for e in json.load(sys.stdin)["events"]:
    if e["eventType"] == "NODE_LEFT": print(e["nodeId"])'
 HTTP 200
WindowClosed|WAITING|3
COMPLETED
Start
AwaitDelivery
Late

3. An order without one. The window is config.defaultWindow: PT8S, from the engine’s environment. A delivery inside it completes the instance on the normal path:

send "" '{"orderId":"o-2"}'
O2=$(id_of o-2)
timers
send /delivered '{"orderId":"o-2"}'
status_of $O2
 HTTP 200
WindowClosed|RESOLVED|3
WindowClosed|WAITING|8
 HTTP 200
COMPLETED

4. A window that is not a duration. The expression returns the string "soon" when the receive task parks. The request is refused and nothing is stored:

send "" '{"orderId":"o-3","window":"soon"}'
{"type":"urn:bpm:diag:SUTRA.RUNTIME.TIMER.EXPRESSION_INVALID","title":"SUTRA.RUNTIME.TIMER.EXPRESSION_INVALID","status":500,"detail":"timer 'WindowClosed' <timeDuration> expression 'if payload.window = null then config.defaultWindow else payload.window' could not be armed: it returned the string \"soon\", which is not a usable duration: 'soon' is not an ISO-8601 duration (must start with 'P')","code":"SUTRA.RUNTIME.TIMER.EXPRESSION_INVALID","attributes":{"incident":"SUTRA.INBOUND.NON_IDEMPOTENT_FAILURE"}} HTTP 500

When the timer is armed later in an instance’s life (after a first wait, say), the same failure leaves the instance FAILED, with this code as its failureCode.

5. Change the default without touching the archive. Restart the engine with a different DEFAULT_WINDOW. The archive and its deploymentId stay the same; new orders get the new window:

docker compose -f $C stop engine
DEFAULT_WINDOW=PT20S docker compose -f $C up -d --wait engine
PORT=$(docker compose -f $C port engine 8080 | cut -d: -f2)
until curl -sf "http://127.0.0.1:$PORT/sutra/deployments" | grep -q '"phase":"Active"'; do sleep 1; done
send "" '{"orderId":"o-4"}'
timers
 HTTP 200
WindowClosed|RESOLVED|3
WindowClosed|RESOLVED|8
WindowClosed|WAITING|20

6. Clean up:

docker compose -p sutra-delivery-window down -v

Next

Escaping free text in an XML template

A template renders {{x}} verbatim. That is right for values you control, and wrong for free text a caller sent you: a customer called O'Brien & Sons or a note with a < in it makes the message malformed XML. {{xmlEscape x}} (new in 0.6.25; reference in Template helpers) escapes the value for XML.

This walkthrough answers a JSON order with an XML receipt twice, once escaped and once verbatim. It was run against the 0.6.25 engine and PostgreSQL 16, with the output shown below.

1. The package

package.yaml labels the module receipt, version 1.0.0. channels.yaml declares two JSON channels, one per template:

channels:
  - name: order
    transport: http
    bind: "POST /channels/order"
    codec: json
    cloudevents-mode: none
    auth: { scheme: apikey, apikey: { value: env:RECEIPT_API_KEY, header: X-Api-Key } }
  - name: order-raw
    transport: http
    bind: "POST /channels/order-raw"
    codec: json
    cloudevents-mode: none
    auth: { scheme: apikey, apikey: { value: env:RECEIPT_API_KEY, header: X-Api-Key } }

bpmn/receipt.bpmn binds the request to order and answers with a template:

<bpmn:process id="receipt" isExecutable="true">
  <bpmn:extensionElements>
    <q:variables>
      <q:variable name="order" type="object"/>
    </q:variables>
  </bpmn:extensionElements>
  <bpmn:startEvent id="Start">
    <bpmn:extensionElements><q:source channel="order" name="order"/></bpmn:extensionElements>
  </bpmn:startEvent>
  <bpmn:serviceTask id="Answer" implementation="receipt.hbs">
    <bpmn:extensionElements><q:reply mode="native" contentType="application/xml"/></bpmn:extensionElements>
  </bpmn:serviceTask>
  <bpmn:endEvent id="Done"/>
  <bpmn:sequenceFlow id="F1" sourceRef="Start" targetRef="Answer"/>
  <bpmn:sequenceFlow id="F2" sourceRef="Answer" targetRef="Done"/>
</bpmn:process>

bpmn/receipt-raw.bpmn is the same process with the id receipt-raw, the channel order-raw and the template receipt-raw.hbs.

templates/receipt.hbs escapes every value taken from the request:

<Receipt customer="{{xmlEscape order.customer}}"><Note>{{xmlEscape order.note}}</Note><Ref>{{xmlEscape order.ref}}</Ref></Receipt>

templates/receipt-raw.hbs does not:

<Receipt customer="{{order.customer}}"><Note>{{order.note}}</Note></Receipt>

2. Package it and start the engine

sutra lint receipt
sutra package receipt -o deployments

Lint reports no errors. It warns that the json channels declare no message types, which is expected for a schema-less channel.

Start the engine with SUTRA_DEPLOYMENTS_DIR pointing at deployments, SUTRA_DATASOURCE_URL and its credentials pointing at PostgreSQL, and RECEIPT_API_KEY set. Set B to the engine’s address.

3. Send free text

cat > order.json <<'JSON'
{"customer":"O'Brien & Sons","note":"Leave at the door if <2 kg. \"Fragile\" — 壊れ物","ref":"A-1042"}
JSON
curl -s -X POST "$B/channels/order" -H 'X-Api-Key: <your key>' \
  -H 'Content-Type: application/json' -d @order.json
<Receipt customer="O&apos;Brien &amp; Sons"><Note>Leave at the door if &lt;2 kg. &quot;Fragile&quot; — 壊れ物</Note><Ref>A-1042</Ref></Receipt>

The five XML special characters are escaped; the dash and the Japanese text pass through unchanged.

The same order on the verbatim channel:

curl -s -X POST "$B/channels/order-raw" -H 'X-Api-Key: <your key>' \
  -H 'Content-Type: application/json' -d @order.json
<Receipt customer="O'Brien & Sons"><Note>Leave at the door if <2 kg. "Fragile" — 壊れ物</Note></Receipt>

Save each answer to a file and give it to an XML parser. Python’s xml.dom.minidom accepts the first and refuses the second:

receipt.xml:     well-formed
receipt-raw.xml: not well-formed (invalid token): line 1, column 28

4. A missing value

Leave ref out:

curl -s -X POST "$B/channels/order" -H 'X-Api-Key: <your key>' \
  -H 'Content-Type: application/json' -d '{"customer":"Ada","note":"none"}'
<Receipt customer="Ada"><Note>none</Note><Ref></Ref></Receipt>

xmlEscape renders a missing or null value empty. A plain {{order.note}} on a missing value fails the render instead, as strict mode requires. Leaving note out on order-raw answers HTTP 500:

unresolved template reference {{order.note}} (strict rendering: every referenced value must be present; use {{coalesce x "default"}} for optional values)

If an element must not be empty, check the value in the flow before the template runs.

Testing time: fast-forwarding durable timers

A flow that waits PT24H for a reminder, or fires an R3/PT12H schedule three times, is exactly the kind of thing that is hard to test and therefore usually untested. Sutra’s answer is a virtual clock: the engine’s notion of “now” becomes something a test drives explicitly, so a day-long timer settles in wall-clock seconds against a real database, with real durable rows and the real timer poller.

sequenceDiagram
    participant T as Test
    participant E as Engine
    participant DB as PostgreSQL

    T->>E: send the message that starts the flow
    E->>DB: park: durable timer row, due in PT24H
    Note over E,DB: real row, real poller — only "now" is virtual
    T->>E: fast-forward the clock past the due time
    E->>DB: timer is due: claim + fire
    E-->>T: the reminder path ran

A day-long timer settles in milliseconds because the clock moved, not because the timer was shortened — the durable rows and the code path are the production ones.

Two ways in: a CLI command for BPMN authors, and an embedded seam for anyone writing tests in Rust.

sutra test simulate — no test code required

sutra test simulate --deployments <DIR> --datasource <URL> \
    (--advance <DURATION> | --until-quiescent) [flags]

Boots a real engine on a dynamic port against a directory of sealed deployment archives with a virtual clock installed, fast-forwards it, reports, and shuts down cleanly.

This is unrelated to sutra simulate, which is a dry-run routing report over a single BPMN file and never boots anything. They are separate commands on purpose.

FlagMeaning
--deployments <DIR>Directory of sealed .sutra archives to serve. Required.
--datasource <URL> (--datasource-username / --datasource-password)Engine datasource. Required — the same canonical SUTRA_DATASOURCE_* env names a real engine container reads.
--advance <DURATION>Fast-forward the virtual clock by this ISO-8601 duration, firing everything due along the way, then stop and report.
--until-quiescentFast-forward until nothing is armed and nothing is live, or --timeout elapses.
--timeout <DURATION>Real wall-clock budget for the fast-forward loop, either mode. Default PT30S.
--start <RFC3339>Virtual start instant. Default: the real current instant.
--allow-existing-dataProceed even though the datasource already holds instances.

Exactly one of --advance / --until-quiescent is required.

It refuses to run against data it did not expect

Fast-forwarding a virtual clock against a database that already holds real, in-flight instances would durably fire their real timers early. A datasource mixup must not silently do that, so before booting anything the command checks that the target database holds no instances, and refuses with exit 2 if it does.

--allow-existing-data is the explicit acknowledgement — and also the right shape for “seed one instance externally, then fast-forward it”.

--datasource being required is part of the same posture: a persistence-less run has no durable timers to advance at all, so the command refuses rather than pretending to have done something.

Quiescence, precisely

--until-quiescent stops when, simultaneously: no waiting row is an armed timer, no schedule is armed, and no instance is still live.

A FAILED instance is deliberately not counted as finished — it needs an operator, not a clock. So a fixture that fails under fast-forward correctly reports timedOut: true rather than a false “quiescent”, and the summary’s instancesFailed says why.

Output

Human-readable progress goes to stderr. The final summary is one JSON object on stdout and nothing else touches stdout, so piping into jq is always safe:

{
  "mode": "until-quiescent",
  "deployments": "/path/to/deployments",
  "allowExistingData": false,
  "timedOut": false,
  "quiescent": true,
  "preExistingInstances": 0,
  "virtualStart": "2026-08-06T00:00:00Z",
  "virtualEnd": "2026-08-07T00:00:03Z",
  "virtualSecondsAdvanced": 86403.2,
  "wallSeconds": 1.84,
  "instancesStarted": 1,
  "instancesCompleted": 1,
  "instancesFailed": 0,
  "instancesTerminated": 0,
  "instancesLive": 0,
  "timersFired": 1,
  "schedulesFired": 0
}

schedulesFired counts single-shot and bounded (R<n>/…) timer-start fires exactly. An unbounded cycle (R/…) has no repeat budget to difference, so its fires show up in instancesStarted but are not separately counted in this release.

Use it to prove, in your application’s own CI, that a durable-timer or cyclic-schedule flow settles correctly — without a wall-clock-length test run.

The embedded seam

sutra_engine::TestClock is a manually-advanced virtual clock, and sutra_engine::fast_forward_until is the paired driver. Install the clock on the engine config before serving, and every temporal read in that boot uses it: timer park due instants, <q:retry> backoff instants, the timer poller’s per-tick claim instant, and schedule arming.

#![allow(unused)]
fn main() {
let clock = sutra_engine::TestClock::starting_now();
let engine = sutra_engine::serve(sutra_engine::EngineConfig {
    now_override: Some(clock.clone()),
    ..config
}).await?;

// park a PT24H timer, then:
let settled = sutra_engine::fast_forward_until(
    &pool, &clock, std::time::Duration::from_secs(10),
    || async { live_instance_count(&pool).await == 0 },
).await;
assert!(settled);
}

Every clone of the clock is a handle onto the same instant, so the code under test and the test driving it always agree. starting_now() starts at the real current instant — only the fast-forward is virtual — which keeps any absolute-time assertion elsewhere in the test (log timestamps, audit rows) sane.

The driver loop is deliberately simple: jump the clock to the earliest armed due instant, give the real timer poller a tick to claim and fire it, re-check your condition; repeat until the condition holds or the real wall-clock timeout elapses. It only ever touches the clock and the database — never the poller’s internals. Set the poller’s tick interval low for the test boot so a tick is cheap.

It is unreachable from a deployed engine — by construction

There is no config key, no environment variable, and no CLI flag on the engine binary that installs a virtual clock. Wiring one in is always an explicit choice in Rust at the call site, so an operator has no way to reach it in a deployed engine.

sutra test simulate is not a hole in that: it is a different binary that constructs an engine configuration directly in code — exactly the sanctioned pattern, just packaged as a reusable tool instead of a bespoke integration test per application. The engine’s own configuration loading is untouched by it.

What it is good for

  • A timeDuration catch timer or <q:timeout> boundary that would otherwise need a real wait.
  • A timeCycle schedule firing its full repeat budget.
  • A <q:retry> backoff curve running to exhaustion — every attempt’s park is a real durable row, and fast-forward walks them all.

See Retries, history, and schedules for what each of those constructs does, and Testing tiers for where a test like this belongs in the suite.

Next

Coverage: declared routes as the compliance signal

Compliance in a Sutra deployment isn’t reconstructed from logs after the fact. A module owner declares the business-event routes that matter — “the happy-path completion,” “the reject branch,” “the end-to-end route across all three collaborating processes” — and the engine ticks each one off as real instances walk it. That declaration is the whole compliance surface: a route nobody declared can never show up as covered or uncovered, so what you choose to declare is exactly what you can ever detect. This chapter covers the two shapes a route can take, how to declare each, the CLI that drives them, and — the part that actually determines whether the feature is useful — how to curate the declared set down to what’s worth watching.

What “covered” means

A route is an ordered path through execution: for an intra-process route, an ordered list of one process’s own sequence-flow ids; for a cross-process route, a per-process segment of flows for each participant in a correlated cascade, tied together by a business key that threads every hop. A route is marked covered the moment an instance (or, cross-process, a correlated group of instances) walks every one of its flows in order; it stays uncovered since the last reset otherwise. There is no partial credit and no percentage for anything you didn’t declare — total is exactly the number of declared routes, covered is how many have flipped, and the uncovered list is the rest.

Both shapes land in one place: the coverage data store the deployment declares in its own datastores.yaml, as a typed coverage_metric row per declared route carrying a covered boolean — entirely separate from the audit trail, so coverage never reads audit tables and audit never reads coverage’s. That’s also what makes coverage cheap to skip: a process with no declared route pays nothing (no flow-trace capture, no writes). See Where coverage is stored for the division of labour that store implies: you pick the database, the engine owns the schema.

ShapeDeclaredSpans
Intra-process<q:coverage path="…" flows="…"/> inline on a <bpmn:process>one process
Cross-processa coverage/<name>.yaml file, URN-identified urn:sutra:coverage:<name>several correlated processes of one deployment
flowchart LR
    D1["q:coverage on a process<br/>an ordered list of flow ids"] --> SEED
    D2["a coverage/name.yaml route<br/>a segment per participant process"] --> SEED
    SEED["on activation — seed one coverage_metric row<br/>per declared route, covered = false"] --> ST[("the coverage store<br/>the deployment declares")]
    RUN["a real instance's fired-flow trace"] -->|"contains the route's flows<br/>in order, as a subsequence"| UP["guarded UPDATE … AND NOT covered —<br/>the affected-row count is the answer to<br/>did this run newly cover it"]
    UP --> ST
    ST --> REP["total, covered, uncovered —<br/>one aggregate and one ordered query,<br/>in a single REPEATABLE READ snapshot"]

Declaration is what creates the measurable surface: total is exactly the seeded set, so a route nobody declared can never appear as covered or uncovered, and the percentage is derived on read rather than kept as a counter that could drift.

Intra-process: <q:coverage> inline

The simplest case declares a route directly on the process it belongs to, as an ordered list of sequence-flow ids — the same ids the diagram already shows on every arrow:

<bpmn:process id="transfer">
  <bpmn:extensionElements>
    <q:coverage path="accept" flows="Flow_TxToOk Flow_OkToEnd"/>
    <q:coverage path="reject" flows="Flow_CancelToReject Flow_RejectToEnd"/>
  </bpmn:extensionElements>
  …
</bpmn:process>

A run covers a path when its fired-flow trace contains flows as an ordered subsequence — so whatever intake channel started the instance (money-transfer’s transfer.bpmn has three) is irrelevant to whether accept or reject gets marked, and a wait-state park in the middle doesn’t break the match. See Worked example: money-transfer for this declaration driving a real ACID transfer flow end to end.

Fail-closed checks at load: a flow id the process doesn’t have, a set of flows that isn’t a contiguous route (each flow’s target must be the next flow’s source), or a reused path id inside one process all evict the module rather than silently declare something meaningless. A process that declares at least one <q:coverage> must also declare a coverage store in datastores.yaml (SUTRA.CONFIG.COVERAGE.STORE_MISSING at lint time) — that store is where the marks are persisted, so declaring routes without it is declaring something with nowhere to record it. The scaffold satisfies the rule for you; Where coverage is stored is what the declaration actually buys.

Cross-process: a coverage file

A route that spans more than one participant process can’t live inside a single <bpmn:process> — there’s no BPMN element for “the flow that starts here and finishes three hops away in a different process.” Instead it’s declared in a coverage file under the package’s coverage/ folder, identified the same way a template is: path-derived, urn:sutra:coverage:<folder…>:<file>. A file holds one or more correlations — the business key that ties instances of the cascade together, and the links (hops) between them — plus one or more routes (coverages:) that each state a complete set of per-process segments.

The shape that motivates this is a relay cascade: three collaborating services — an intake service, a routing hub, and a fulfillment service — each its own <bpmn:process>, handing off over channels rather than calling each other directly. No single process sees the whole exchange, so “did the end-to-end handshake complete, and which way did it end” is exactly the question none of them can answer alone. A coverage/e2e.yaml (urn:sutra:coverage:e2e) declares the routes that span all three:

correlations:
  - id: relay-case
    key: caseId                            # default hop key — every leg correlates on the same case id
    links:                                 # the <q:send> -> <q:source>/imec hops between the 3 processes
      # forward: intake -> hub (spawns the hub)
      - { from: intake-svc:Task_ForwardToHub,        to: hub-svc:Start_Received }
      # forward: hub -> fulfillment (spawns fulfillment)
      - { from: hub-svc:Task_ForwardToFulfillment,   to: fulfillment-svc:Start_Posting }
      # reply: fulfillment -> hub (resumes the hub's imec) - accept and reject both land here
      - { from: fulfillment-svc:Task_ReplyAccept,    to: hub-svc:Imec_AwaitFulfillment }
      - { from: fulfillment-svc:Task_ReplyReject,    to: hub-svc:Imec_AwaitFulfillment }
      # reply: hub -> intake (resumes the intake's imec)
      - { from: hub-svc:Task_ReplyToIntake,          to: intake-svc:Imec_AwaitReply }
    coverages:
      - path: e2e-accepted                 # the accepted branch
        segments:
          intake-svc:      [Flow_p1_start, Flow_p1_await, Flow_p1_done]
          hub-svc:         [Flow_p2_start, Flow_p2_await, Flow_p2_reply, Flow_p2_done]
          fulfillment-svc: [Flow_p3_start, Flow_p3_accept, Flow_p3_accept_end]
      - path: e2e-rejected                 # the rejected branch — same p1/p2, different p3
        segments:
          intake-svc:      [Flow_p1_start, Flow_p1_await, Flow_p1_done]
          hub-svc:         [Flow_p2_start, Flow_p2_await, Flow_p2_reply, Flow_p2_done]
          fulfillment-svc: [Flow_p3_start, Flow_p3_reject, Flow_p3_reject_end]

A few things about this shape:

  • links are structural, not payload access. Each hop names a <q:send> node on one side and either a start-event <q:source> (a spawn — the hop starts a new instance) or an imec relay-wait <q:source> (a relay — the hop resumes a parked instance) on the other, matched against the actual channel wiring. sutra lint validates every link resolves against real channel bindings, that every segments flow id is contiguous within its own process, and that the correlation key resolves at both ends of every hop — a broken link or an unresolvable key evicts the module rather than deploying a route that can never complete.
  • The correlation key is how the runtime reconstructs the cascade, since three separately dispatched instances have no shared instance id. Here it’s a single value — a case id — carried as an author-declared header rather than a payload field: every <q:send> sets it (<q:header name="caseId" value="caseId"/>) and every consuming <q:source> reads it back (<q:alias name="caseId" expression="header.caseId"/>). A hop can override the correlation’s default key when its own leg correlates on a different value — this cascade doesn’t need to, since one case id threads every hop end to end.
  • Each route is fully self-contained. e2e-accepted and e2e-rejected repeat their identical intake-svc and hub-svc segments rather than one inheriting from the other — there is no route-to-route inheritance, by design, so reading any one route never requires cross-referencing another.
  • A route can be declared but structurally rare. e2e-rejected is only reachable when the fulfillment process takes its reject branch — under normal traffic that rarely happens, so e2e-rejected can sit uncovered indefinitely. That’s not a bug in the declaration; it’s the compliance signal doing its job: “the reject path exists and we watch for it,” distinct from “the reject path fired.” It also makes a clean assertion for a test campaign: drive one ordinary case, expect e2e-accepted to reconstruct as covered and e2e-rejected not to, then reset and confirm both go back to uncovered.

At load, each route’s per-process segments are injected as ordinary intra-process coverage paths on their own process (so the existing per-process marking is unchanged); on completion each segment writes a correlation-tagged record, and a route flips covered once every one of its segments has landed in the same correlated group. Reading total/covered/percentage for a cross-process route is the same store query as an intra-process one — the mechanism composes rather than duplicating.

flowchart LR
    subgraph SEG["one route's per-process segments"]
        A["intake-svc"]
        B["hub-svc"]
        C["fulfillment-svc"]
    end
    A -->|"a correlation-<br/>tagged record"| G
    B --> G
    C --> G
    G["one group, keyed by<br/>the caseId every hop carries"] -->|"all segments in"| F["the route flips covered<br/>counted once in total"]

Three separately dispatched instances share no instance id, so the correlation key is the whole reconstruction; the segments are marking cursors, which is why many of them collapse onto the one route flag a report reads.

Composing several definitions in one package

Nothing limits a package to one kind of declaration. A deployment can carry <q:coverage> on several of its own processes and one or more coverage/*.yaml files, each with its own correlation and its own set of routes — money-transfer’s inline accept/reject pair and the file-based e2e-accepted/e2e-rejected pair above are the two shapes, and a single package is free to declare both kinds side by side: an intra-process route for “did this one process’s own retry/compensation branch fire,” alongside a cross-process file for “did the whole multi-participant handshake complete.” Each declaration is independent — an intra-process path only needs its own process’s flow ids; a coverage file only needs the processIds and channel wiring it names — so there’s no coordination cost to adding more of either kind as a module grows.

Where coverage is stored

Coverage marks are persisted in the coverage data store the deployment declares — a reserved store name in its own datastores.yaml, declared like any other store. That declaration is how you choose the database: the connection it names is where the marks land, and the URL scheme picks the dialect (PostgreSQL, MySQL/MariaDB and SQL Server all work). The engine hosts no coverage in its own database.

What you do not write is coverage SQL. The coverage tables are a built-in feature, so the engine owns their schema: it ships the coverage_metric / coverage_fragment DDL per dialect and applies it to that connection the first time the store is used — the same idempotent, lock-serialized first-use path a module’s own migrations/<store>/ scripts take, with engine-shipped scripts instead of package-supplied ones. So the store block carries no migrations: key, and a package carries no migrations/coverage/ folder at all:

datastores:
  - name: coverage                        # the reserved name — this declaration picks the DATABASE
    type: sql
    sql:
      url-ref: env:ACCOUNTS_DB_URL
      username-ref: env:ACCOUNTS_DB_USER
      password-ref: env:ACCOUNTS_DB_PASSWORD
                                          # no `migrations:` — the engine owns this store's schema

Pointing it at a connection some other store already uses is fine and often convenient — money-transfer’s coverage store names the same database as its accounts ledger. The coverage tables are the engine’s own; they sit beside the business ones rather than mixing with them.

On activation, a deployment seeds one row per declared route into that store’s coverage_metric table with covered = false — intra-process path ids and cross-process route ids alike. (A cross-process route’s per-process segments are marking cursors, not routes: many segments collapse onto the one route flag, which is why total counts the route once.) Execution flips flags with a guarded UPDATE … AND NOT covered, so first-covers-wins is settled by the write itself — the affected-row count is the answer to “did this run newly cover it,” never a read-then-write race. Nothing else writes there. That typed table is the whole coverage substrate, and three consequences follow from it.

Counts are SQL, not a fold in the engine. total and covered are one aggregate over the seeded set (COUNT(CASE WHEN … THEN 1 END) — the portable pivot, which returns a count on every shipped dialect), and the uncovered routes are their own ordered query. Both run inside one REPEATABLE READ transaction, so the count and the list can never disagree: they describe one snapshot. A percentage is still derived on read, never stored — there is no counter to drift.

Flags are keyed by deployment and route id. deployment_id is a column on both tables and a bound predicate on every statement the engine issues; that, not row-level security, is the isolation. (RLS is an engine-database convention — it wants table ownership and a per-transaction setting the engine cannot assume on a connection you own, and two of the three shipped dialects have no equivalent anyway.) Two processes in one deployment that both declare a path called accept share one flag; the CLI’s total has always counted them once. Name a route for the business event it represents rather than for the branch it sits on, and the question never comes up. The same keying is what makes a runtime report of a cross-process route read against the route flag rather than the segment that happened to finish — one number, and the same one sutra coverage check --archive reports.

A declared store the engine can’t open fails loudly. Coverage exists wherever you declare a store, so the failure mode is no longer “this deployment has no database of its own to record in” — it is a coverage store whose connection is wrong, unset or unreachable. The engine logs an error at boot naming that deployment, and a coverage:report / coverage:reset op fails with SUTRA.CONFIG.COVERAGE.STORE_MISSING naming the cause. It does not report 0%:

a report of 0% here would be indistinguishable from a real measurement of nothing covered

A deployment that declares <q:coverage> paths but no coverage store at all is the same story one step earlier: sutra lint errors on it at package time, and an engine that loads such a package anyway warns at boot and fails those two ops with the same code. The reasoning extends to a store that opens but whose read fails — the op surfaces the underlying error rather than counting every route as uncovered. A compliance signal that quietly degrades to “nothing is covered” is worse than one that stops and says why. (Runtime marking stays best-effort — a metric side-effect must never fail a business instance. The loud surface is the report, which is what a human or a CI gate actually reads.)

A package built against an older engine may still carry a migrations/coverage/ folder. It is dead weight: nothing in it was ever coverage DDL — the script there created the same generic table a business store already creates — and the engine no longer reads a migrations: key on the reserved coverage store. Delete the folder.

The sutra coverage CLI

init — enumerate and seed

Single-process form: point it at a BPMN file and it walks every start→end route over the engine’s own execution semantics (gateways, sub-processes, the lot), then seeds <q:coverage> declarations plus a matching admin scaffold (report/reset BPMN, reply templates, the two admin channels, and the coverage store declaration — no SQL, because none of it is yours to write):

$ sutra coverage init bpmn/transfer.bpmn --process transfer
coverage init: bpmn/transfer.bpmn — process 'transfer', 2 path(s) declared (0 kept, 2 new)
  path path-1: Flow_TxToOk Flow_OkToEnd
  path path-2: Flow_CancelToReject Flow_RejectToEnd
  updated  bpmn/transfer.bpmn
  created  bpmn/coverage-report.bpmn
  created  bpmn/coverage-reset.bpmn
  created  templates/coverage-report.hbs
  created  templates/coverage-reset.hbs
  updated  channels.yaml
  updated  datastores.yaml

Route enumeration refuses beyond a 256-route cap (--max-paths raises it): a process with enough parallel/inclusive branching to combinatorially explode past that is a sign the process itself is too fine-grained for route-level compliance tracking, not that the cap is wrong — raise --max-paths only when you’ve confirmed the extra routes are genuinely distinct business outcomes, not gateway noise. A second init run keeps any path id whose flows still match (renames survive), refuses to clobber hand-edited declarations without --force, and never replaces a file it can’t safely re-parse.

Cross-process form: name a coverage file and the processIds it should span instead of a single BPMN file. This does not enumerate a path set at all (there’s no combinatorial explosion to cap) — it emits the connectable graph: each process’s own flow adjacency, plus every inter-process hop it can infer from <q:send>/<q:source>/imec wiring, as a commented scaffold with one starter route to trim:

$ sutra coverage init coverage/e2e.yaml intake-svc hub-svc fulfillment-svc
coverage init: coverage/e2e.yaml — urn:sutra:coverage:e2e
  processes: intake-svc, hub-svc, fulfillment-svc
  intra-process adjacency: 9 flow(s) across 3 process(es)
  inter-process hops: 5
    intake-svc:Task_ForwardToHub --to-hub--> hub-svc:Start_Received [spawn fire-and-forget] key=caseId
    hub-svc:Task_ForwardToFulfillment --to-fulfillment--> fulfillment-svc:Start_Posting [spawn fire-and-forget] key=caseId
    fulfillment-svc:Task_ReplyAccept --to-hub-imec--> hub-svc:Imec_AwaitFulfillment [relay request-reply] key=caseId
    fulfillment-svc:Task_ReplyReject --to-hub-imec--> hub-svc:Imec_AwaitFulfillment [relay request-reply] key=caseId
    hub-svc:Task_ReplyToIntake --to-intake-imec--> intake-svc:Imec_AwaitReply [relay request-reply] key=caseId
  scaffold written — draw connected walks from it (sutra lint validates connectivity)

The written file’s coverages: starts with exactly one route (path-1) whose segments list every connectable flow id per process — the raw graph, not a curated path. --single restricts the graph to intra-process adjacency only (no inter-process hops, for a package still being wired up one participant at a time); --force overwrites an existing file.

check — read covered/uncovered, drive assertions

Bare BPMN file: a drift lint, not a store read. It confirms every declared path is still an ordered subsequence of the process’s current flow graph — the check that catches a declaration silently breaking when someone reworks the diagram — and reports (informationally) any enumerated route no declared path covers:

$ sutra coverage check bpmn/transfer.bpmn
coverage check: 0 error(s), 0 note(s)

--archive selects the store-backed, correlation-aware check instead: it reads the deployment’s seeded metric flags, union-finds the cross-process reconstruction records to flip any route that has now fully completed, and reports the same total/covered/percentage a live dashboard would read — the fail-closed CI gate and the runtime signal are the identical query. --database-url points at the database the deployment’s coverage store declares, in whichever of the three shipped dialects it names; the CLI reads and writes exactly what the engine does, and creates nothing the engine would not create itself:

$ sutra coverage check --archive deployed.sutra --database-url "$SUTRA_DB_URL" --threshold 100
coverage check (cross-process) — deployment 3f9a1c…
  total: 2  covered: 1  coveragePercentage: 50.00%
  newly covered this run (1):
    + urn:sutra:coverage:e2e:e2e-accepted
  uncovered (1):
    - urn:sutra:coverage:e2e:e2e-rejected
  threshold: 100.00%  =>  FAIL

--threshold (default 100) is the gate: the process exits non-zero whenever the percentage falls below it, so a CI pipeline can fail a release that hasn’t exercised every route it declared as required — or, set lower, tolerate a known-rare route like e2e-rejected without losing visibility into whether it ever fires.

reset — re-seed a deployment’s declared routes

$ sutra coverage reset --archive deployed.sutra --database-url "$SUTRA_DB_URL"
coverage reset — deployment 3f9a1c…: 2 path(s) re-seeded covered=false, reconstruction fragments cleared

Every declared path (intra- and cross-process alike) goes back to covered = false and every cross-process reconstruction record is cleared — the clean baseline a new test campaign, or a new reporting period, starts from. The rows stay seeded; reset is one scoped update that flips the flags, so total is unchanged and the cleared count is exactly how many routes had been covered.

Requirements and compliance cycles

Since 0.6.20 a package can map its declared paths to requirements, and a deployment can keep named compliance cycles, so you can ask “how much of release 4.2’s requirements has the test campaign covered?”

  • A requirements map. coverage/requirements-map.yaml maps each declared path to requirement ids, a weight (default 1, above zero) and the test cases that walk it. sutra coverage check then adds a compliance object: per-requirement and weighted percentages, uncovered paths with their cases, the cases to run, and paths with no cases or no mapping. --threshold gates the weighted percentage. Every declared path counts; there is no way to exclude one. A package with no map reports as before. Lint: SUTRA.CONFIG.COVERAGE.MAPPING_ROUTE_UNKNOWN (error, a key that matches no declared path) and SUTRA.CONFIG.COVERAGE.MAPPING_ROUTE_UNMAPPED (warning, a declared path the map leaves out, counted at weight 1).
  • Cycles. sutra coverage cycle declare|activate|deactivate <LABEL> and sutra coverage cycle list, plus --cycle on coverage check and coverage reset. While a cycle is active, every mark for that deployment records to it. A mark belongs to the cycle active when its instance completes. One cycle can be active per deployment: a second activation is refused (SUTRA.RUNTIME.COVERAGE.CYCLE_ALREADY_ACTIVE), as is activating an undeclared label (CYCLE_UNDECLARED); reactivating a cycle continues it. With none active, coverage records as before, and check and reset without --cycle read the marks recorded outside any cycle. Cycles belong to a deployment id, so a changed package starts with none. The sutra.coverage.percent gauge and the path counter are not cycle-aware.

Walkthrough

The money-transfer example package with this coverage/requirements-map.yaml:

routes:
  accept:
    requirements: [REQ-HAPPY]
    weight: 3
    cases: [case-accept]
  reject:
    requirements: [REQ-REFUSAL]
    cases: [case-reject, case-reject-frozen]

sutra lint and sutra package accept it (no MAPPING_ROUTE_* finding: both declared paths are mapped). The engine runs from a deployments folder on PostgreSQL; the CLI reaches the same database through SUTRA_DB_URL. Steps: declare a cycle and activate it; check it (nothing covered, every case to run); post an accepted request; check again (weighted 75%: accept weighs 3 of 4); the JSON form with --threshold 75 passes, because the threshold gates the weighted percentage; deactivate, so a rejected request records outside the cycle; a second active cycle is refused; reactivating release-4.2 continues it, and the rejected request now covers it fully.

$ sutra coverage cycle declare release-4.2 --archive archives/money-transfer.sutra
coverage cycle declare — deployment dep-43a59818955d7e27dd83b82d: cycle 'release-4.2' declared, 2 path(s) seeded
[exit 0]

$ sutra coverage cycle activate release-4.2 --archive archives/money-transfer.sutra
coverage cycle activate — deployment dep-43a59818955d7e27dd83b82d: cycle 'release-4.2' is active; coverage now records to it
[exit 0]

$ sutra coverage check --archive archives/money-transfer.sutra --cycle release-4.2
coverage check (cross-process) — deployment dep-43a59818955d7e27dd83b82d
  cycle: release-4.2
  total: 2  covered: 0  coveragePercentage: 0.00%
  uncovered (2):
    - accept
    - reject
  weighted: 0.00%  (0 of 4 by weight)
  requirements (2):
    REQ-HAPPY: 0.00%  (0 of 1 routes)
    REQ-REFUSAL: 0.00%  (0 of 1 routes)
  uncovered routes and the cases that cover them:
    - accept: case-accept
    - reject: case-reject, case-reject-frozen
  cases to run: case-accept, case-reject, case-reject-frozen
  threshold: 100.00%  =>  FAIL
[exit 1]

POST accept: HTTP 200

$ sutra coverage check --archive archives/money-transfer.sutra --cycle release-4.2
coverage check (cross-process) — deployment dep-43a59818955d7e27dd83b82d
  cycle: release-4.2
  total: 2  covered: 1  coveragePercentage: 50.00%
  uncovered (1):
    - reject
  weighted: 75.00%  (3 of 4 by weight)
  requirements (2):
    REQ-HAPPY: 100.00%  (1 of 1 routes)
    REQ-REFUSAL: 0.00%  (0 of 1 routes)
  uncovered routes and the cases that cover them:
    - reject: case-reject, case-reject-frozen
  cases to run: case-reject, case-reject-frozen
  threshold: 100.00%  =>  FAIL
[exit 1]

$ sutra --format json coverage check --archive archives/money-transfer.sutra --cycle release-4.2 --threshold 75
{"command":"coverage check","form":"cross-process","deploymentId":"dep-43a59818955d7e27dd83b82d","total":2,"covered":1,"coveragePercentage":50.0,"threshold":75.0,"uncovered":["reject"],"newlyCovered":[],"passed":true,"cy …
[exit 0]

$ sutra coverage cycle deactivate release-4.2 --archive archives/money-transfer.sutra
coverage cycle deactivate — deployment dep-43a59818955d7e27dd83b82d: cycle 'release-4.2' deactivated; coverage records outside any cycle
[exit 0]

POST reject: HTTP 200

$ sutra coverage cycle list --archive archives/money-transfer.sutra
coverage cycle list — deployment dep-43a59818955d7e27dd83b82d: 1 cycle(s), no-cycle coverage 50.00%  (1 of 2)
    release-4.2: 50.00%  (1 of 2)
[exit 0]

$ sutra coverage cycle declare release-4.3 --archive archives/money-transfer.sutra
coverage cycle declare — deployment dep-43a59818955d7e27dd83b82d: cycle 'release-4.3' declared, 2 path(s) seeded
[exit 0]

$ sutra coverage cycle activate release-4.3 --archive archives/money-transfer.sutra
coverage cycle activate — deployment dep-43a59818955d7e27dd83b82d: cycle 'release-4.3' is active; coverage now records to it
[exit 0]

$ sutra coverage cycle activate release-4.2 --archive archives/money-transfer.sutra
coverage cycle activate — deployment dep-43a59818955d7e27dd83b82d: SUTRA.RUNTIME.COVERAGE.CYCLE_ALREADY_ACTIVE: cycle 'release-4.3' is active, and only one can be. Deactivate it first; 'release-4.2' was not activated
[exit 1]

$ sutra coverage cycle deactivate release-4.3 --archive archives/money-transfer.sutra
coverage cycle deactivate — deployment dep-43a59818955d7e27dd83b82d: cycle 'release-4.3' deactivated; coverage records outside any cycle
[exit 0]

$ sutra coverage cycle activate release-4.2 --archive archives/money-transfer.sutra
coverage cycle activate — deployment dep-43a59818955d7e27dd83b82d: cycle 'release-4.2' is active; coverage now records to it
[exit 0]

POST reject: HTTP 200

$ sutra coverage check --archive archives/money-transfer.sutra --cycle release-4.2
coverage check (cross-process) — deployment dep-43a59818955d7e27dd83b82d
  cycle: release-4.2
  total: 2  covered: 2  coveragePercentage: 100.00%
  uncovered: none
  weighted: 100.00%  (4 of 4 by weight)
  requirements (2):
    REQ-HAPPY: 100.00%  (1 of 1 routes)
    REQ-REFUSAL: 100.00%  (1 of 1 routes)
  threshold: 100.00%  =>  PASS
[exit 0]

$ sutra coverage cycle deactivate release-4.2 --archive archives/money-transfer.sutra
coverage cycle deactivate — deployment dep-43a59818955d7e27dd83b82d: cycle 'release-4.2' deactivated; coverage records outside any cycle
[exit 0]

$ sutra coverage cycle list --archive archives/money-transfer.sutra
coverage cycle list — deployment dep-43a59818955d7e27dd83b82d: 2 cycle(s), no-cycle coverage 50.00%  (1 of 2)
    release-4.2: 100.00%  (2 of 2)
    release-4.3: 0.00%  (0 of 2)
[exit 0]

The compliance object of the JSON check above:

{"weightedPercentage":75.0,"weightedTotal":4.0,"weightedCovered":3.0,"requirements":[{"requirement":"REQ-HAPPY","total":1,"covered":1,"coveragePercentage":100.0,"uncovered":[]},{"requirement":"REQ-REFUSAL","total":1,"covered":0,"coveragePercentage":0.0,"uncovered":["reject"]}],"uncoveredRoutes":[{"path":"reject","requirements":["REQ-REFUSAL"],"weight":1.0,"cases":["case-reject","case-reject-frozen"]}],"casesToRun":["case-reject","case-reject-frozen"],"routesWithoutCases":[],"unmappedRoutes":[]}

To re-run one case, send <cycle>:<case>:<attempt> as the idempotency key on a channel that declares idempotency-key-header: a repeat of an attempt does nothing, the next attempt runs. This is covered by the tier-2 conformance suite and not walked here.

Curation: init enumerates, you decide what matters

init’s job is mechanical completeness — surface every structurally valid route (or, cross-process, the whole connectable graph) so nothing gets missed by hand. It is deliberately not trying to guess which of those routes represents a business event worth watching, and treating its raw output as the final declared set is the single most common way this feature stops being useful.

Notice, in the single-process example above, that init names its two routes path-1 and path-2 — it has no way to know one is the accepted-transfer outcome and the other is the rejected one. Money-transfer’s shipped transfer.bpmn renames them to accept and reject by hand — that rename is the curation step, and it’s why sutra coverage check --archive output reads as a compliance report (“the reject path never fired this quarter”) rather than a cryptic route id. The same discipline applies to the cross-process scaffold: coverage init’s starter route lists every connectable flow per process, which is almost never the walk you actually care about — the e2e.yaml above is that raw scaffold trimmed down to exactly the two branches (accepted vs. rejected) the cascade wants visibility into, with routes named for the outcome they represent rather than left as path-1.

This matters because “uncovered” is the entire signal, and noise defeats it. A moderately branchy process can mechanically enumerate dozens of technically-distinct full paths — different orderings through independent parallel branches that don’t correspond to different business outcomes at all. Declare all of them and every report is dominated by routes nobody ever intended to track individually, burying the one genuinely-uncovered path that matters under a wall of noise. Keep the declared set as large as the compliance surface you actually want visibility into — named after the business events they represent — and no larger.

Next

Worked example: money-transfer

examples/money-transfer is the flagship demonstration of full ACID semantics on a single writable ledger, driven from one BPMN process over three different transports (HTTP, RabbitMQ, Kafka). It exercises most of what the last few chapters covered — deployment packages, channels, the q: namespace, and data stores — in one real flow.

The shape

examples/money-transfer/deployments-src/default--money-transfer--1.0.0/
├── package.yaml                         # labels {tenant: default, module: money-transfer, version: 1.0.0}
├── bpmn/transfer.bpmn                   # the ACID transfer flow
├── bpmn/balance-query.bpmn              # read-only balance lookup
├── bpmn/coverage-report.bpmn            # admin: path-coverage report
├── bpmn/coverage-reset.bpmn             # admin: path-coverage reset
├── channels.yaml                        # 6 channels — 3 intake transports + balance + 2 admin
├── datastores.yaml                      # `accounts` (the ledger) + `coverage` (engine-owned schema)
├── migrations/accounts/V001__accounts.sql
├── schemas/transfer/transfer.xsd        # TransferRequest + BalanceQuery
├── schemas/transfer/codec-manifest.yaml
└── templates/*.hbs                      # reply rendering (accept / reject / balance / coverage)

What it demonstrates

A module-owned SQL data store. datastores.yaml declares accounts — the ledger, one row per account holding {balance, frozen} — with its own connection (env:ACCOUNTS_DB_URL/_USER/ _PASSWORD) and its own idempotent migration under migrations/accounts/. See Data stores.

Per-channel singleton serialization. Three intake channels — transfer-request (HTTP), transfer-queue (RabbitMQ), transfer-topic (Kafka) — all drive the same transfer.bpmn, each declaring singleton: true. On a queue transport that’s leader-gated across replicas (a per-channel PostgreSQL-lease election — see Replica semantics); on HTTP the single-writer guarantee instead comes from the <bpmn:transaction> scope plus FOR UPDATE row locks in the flow itself. The read-only balance channel and the coverage-* admin channels are not singletons — they scale across every replica, since they only read or reset shared state rather than serialize writes to it.

flowchart LR
    H["transfer-request — HTTP"] -->|"singleton"| T
    Q["transfer-queue — RabbitMQ"] -->|"singleton, leader-gated<br/>across replicas"| T
    K["transfer-topic — Kafka"] -->|"singleton, leader-gated<br/>across replicas"| T
    T["transfer.bpmn<br/>the ACID transfer"] --> ACC[("accounts — the ledger")]
    B["balance"] -->|"not a singleton —<br/>every replica serves it"| BQ["balance-query.bpmn<br/>read only"]
    BQ --> ACC
    CV["coverage-report, coverage-reset"] -->|"not singletons"| AD["the two admin flows"]

All six channels bind the same urn:transfer codec, and the three write intakes converge on one process rather than three — so the write path is serialized once, by the channel on a broker and by the transaction plus FOR UPDATE locks on HTTP, while the read-only paths stay free to scale.

A custom module schema. schemas/transfer/transfer.xsd declares TransferRequest and BalanceQuery as sibling root elements; the codec registers as urn:transfer (see Deployment packages for the path-derived URN rule), and every channel in channels.yaml binds codec: urn:transfer.

The flow, node by node

transfer.bpmn has three start events — one per intake channel — that all converge on a single <bpmn:transaction> sub-process named Transfer:

Start (http) ─┐
StartQueue ────┼──▶ Transfer [transaction]:
StartKafka ───┘        SubStart → LoadFrom → LoadTo → Valid?
                          ├─ ok:      Compute → Persist → SubEnd (normal end → COMMIT)
                          └─ invalid: DecideReason → cancel end event (→ ROLLBACK)
                     → (committed) OkReply (render TransferAccepted) → End
                     → (cancelled, via a boundary cancel event on Transfer) RejectReply → End

Full ACID, mapped onto real BPMN + q: elements:

  • Atomicity — LoadFrom/LoadTo/Compute/Persist all run inside the <bpmn:transaction>; a normal end commits both balance writes together, a cancel end (reached from the Valid? gateway’s invalid branch) rolls back — no partial transfer ever lands.
  • Consistency — the Valid? exclusive gateway checks fromAccount.frozen, toAccount.frozen, and fromAccount.balance < payload.amount before any write, in a visible FEEL condition on the sequence flow.
  • Isolation — LoadFrom/LoadTo read their rows with <q:store key="payload.fromId" forUpdate="true"/> — a pessimistic lock serializing concurrent transfers that touch the same account — layered under the channel-level singleton/exclusive-consumer contract.
  • Durability — Persist writes both new balances back to the accounts store in the same transaction; a later instance (on any replica) sees the committed values.

Compute and DecideReason are pure FEEL data-assignment nodes (<bpmn:assignment> pairs of a <from> FEEL expression and a <to> target variable) — no service-task implementation code anywhere in this flow.

sequenceDiagram
    participant IN as Intake channel
    participant TX as Transfer, a transaction sub-process
    participant DB as accounts store

    IN->>TX: TransferRequest, decoded by urn:transfer
    TX->>DB: LoadFrom — q:store forUpdate, a pessimistic row lock
    TX->>DB: LoadTo — q:store forUpdate, a pessimistic row lock
    TX->>TX: Valid? — frozen flags and sufficient balance, in FEEL, before any write
    alt valid
        TX->>TX: Compute — pure FEEL assignment
        TX->>DB: Persist — both new balances
        Note over TX,DB: normal end of the sub-process — COMMIT, both writes together
        TX-->>IN: OkReply renders TransferAccepted
    else invalid
        TX->>TX: DecideReason, then a cancel end event
        Note over TX,DB: ROLLBACK — no partial transfer ever lands
        TX-->>IN: RejectReply, via the boundary cancel event on Transfer
    end

The gateway sits between the locks and the writes, so consistency is checked while the rows are already held — and the only two ways out of the sub-process are the two ends that commit or roll back, which is what makes the reply and the durable outcome agree.

Compliance path coverage

<q:coverage path="accept" flows="Flow_TxToOk Flow_OkToEnd"/>
<q:coverage path="reject" flows="Flow_CancelToReject Flow_RejectToEnd"/>

The two business outcomes are declared as tracked routes on the outer process, named for what they mean (not left as the raw path-1/path-2 sutra coverage init would seed) — the curation step Coverage: declared routes as the compliance signal covers in full. A run is “covered” once its fired-flow trace contains a path’s flows, in order, as a subsequence — so any of the three intake transports covers accept when it commits, and covers reject when it cancels. Both marks land in the coverage store datastores.yaml declares — pointed, for convenience, at the same database as the accounts ledger, and carrying no migration of its own because the engine owns the coverage schema. The coverage-report / coverage-reset admin channels read and clear this compliance metric; see that chapter for the CLI walkthrough (init/check/reset), the cross-process form a multi-participant collaboration needs instead, and how to curate a declared set that stays a signal instead of noise.

Try it

# seal it
cd rust && cargo build -p sutra-cli --release
target/release/sutra package ../examples/money-transfer/deployments-src/default--money-transfer--1.0.0 \
    --out /tmp/pkgs

# deploy it (see Your first deployment for the full API-deploy walkthrough)
target/release/sutra deploy /tmp/pkgs/default--money-transfer--1.0.0.sutra \
    --api --engine-url http://localhost:<port>

# send a transfer
curl -sS -X POST localhost:<port>/channels/transfer-request \
  -H 'Content-Type: application/json' -H 'X-Api-Key: transfer-demo-key' \
  -d '{"TransferRequest":{"FromAccount":"alice","ToAccount":"bob","Amount":"25.00"}}'

Run the full ordered scenario (durability, cross-instance reads, insufficient-funds and frozen-account rejection, atomicity rollback, isolation under concurrency, then the coverage report/reset pair) against real PostgreSQL:

cd rust && cargo test -p sutra-conformance -- --ignored tc_money_transfer_acid_ledger

Next

Worked example: loading a batch file

examples/call-log-load/ is a package that takes a file of records over HTTP — CSV or fixed-width, the same records either way — validates every row against an XSD before the process starts, transforms each record into a second XSD, and writes it as a typed database row.

One channel, one process, two schemas, two wire forms.

The shape

POST /channels/cdr-upload        codec: urn:cdr   (formats: [csv, fixed-width])
   │
   ├── the CODEC validates the WHOLE FILE, row by row, cell by cell
   ▼
 cdr-load.bpmn
   Prep      rowCount := count(payload.value)
   Accept    Handlebars receipt + <q:reply continue="true">   ──►  202 + receipt HERE
   PerRow    multi-instance over payload.value                     (the load runs detached)
     Map       <scriptTask> — one CallDetailRecord ► one CallLogEntry
     Persist   data task     — call_log[entry.entryId]
   End

Answering now, loading later

The channel declares ack-mode: on-persist: the caller is answered as soon as the upload is durably captured, never held for the load. That alone would send a bare 202. What makes it a useful 202 is <q:reply continue="true"> on Accept — the process renders its receipt, that render becomes the response body, and the process then parks and resumes detached to run the rows:

HTTP/1.1 202 Accepted
Content-Type: application/json

{"batchId":"d36d2c59-0fea-4349-b410-a4b4c7715ab9","rowsAccepted":4,"status":"loading"}

The two declarations are independent and both are needed. The ack mode decides when the caller is answered; continue="true" decides that the reply happens at the park instead of at the end of the process. Drop the reply and the caller gets an empty 202 it cannot tell from a lost message; drop continue="true" and the receipt waits for the whole load. See Acknowledgement modes.

Binding a tabular format to a schema

A bare codec: csv on a channel is a parser: it would hand the flow a bag of untyped strings with nothing asserted. Binding a schema codec whose manifest declares the tabular format is what makes the file a typed message:

# schemas/cdr/codec-manifest.yaml
schemaKind: xsd
formats: [csv, fixed-width]

csv:
  delimiter: ","
  header: true

fixed-width:
  fields:
    - {name: recordId,     width: 12}
    - {name: msisdn,       width: 16}
    - {name: peerMsisdn,   width: 16}
    - {name: startTime,    width: 22}
    - {name: durationSec,  width:  6}
    - {name: direction,    width: 11}
    - {name: cellId,       width: 10}
    - {name: chargeAmount, width: 10}
    - {name: rateCode,     width:  8}

The schema those columns are checked against is an ordinary XSD — the facets do the work:

<xs:simpleType name="Msisdn">
  <xs:restriction base="xs:string"><xs:pattern value="\+[0-9]{8,15}"/></xs:restriction>
</xs:simpleType>
<xs:simpleType name="Direction">
  <xs:restriction base="xs:string">
    <xs:enumeration value="originated"/><xs:enumeration value="received"/>
  </xs:restriction>
</xs:simpleType>

<xs:element name="CallDetailRecord">
  <xs:complexType><xs:sequence>
    <xs:element name="recordId"    type="RecordId"/>
    <xs:element name="msisdn"      type="Msisdn"/>
    <xs:element name="startTime"   type="xs:dateTime"/>
    <xs:element name="durationSec" type="DurationSeconds"/>   <!-- xs:int, 0..86400 -->
    <xs:element name="direction"   type="Direction"/>
    <xs:element name="rateCode"    type="RateCode" minOccurs="0"/>
  </xs:sequence></xs:complexType>
</xs:element>

Two wire forms, one schema

Both files carry the same four records. The content-type selects the parser; nothing downstream knows which arrived.

recordId,msisdn,peerMsisdn,startTime,durationSec,direction,cellId,chargeAmount,rateCode
CDR-100001,+14155550101,+14155550187,2026-09-06T09:14:02Z,182,originated,CELL-0042,4.5500,PEAK
CDR-100003,+14155550188,+14155550101,2026-09-06T10:02:19Z,930,received,CELL-0117,0.0000,
CDR-100001  +14155550101    +14155550187    2026-09-06T09:14:02Z  182   originated CELL-0042 4.5500    PEAK
CDR-100003  +14155550188    +14155550101    2026-09-06T10:02:19Z  930   received   CELL-0117 0.0000
curl -X POST .../channels/cdr-upload -H 'Content-Type: text/csv'   --data-binary @sample/call-logs.csv
curl -X POST .../channels/cdr-upload -H 'Content-Type: text/plain' --data-binary @sample/call-logs.fixed-width.txt

Adding a vendor with a different delimiter is a manifest edit. The BPMN never changes.

The nuances

A batch projects under value

An array root projects under value, so the flow reads payload.value and a row is payload.value[0].recordId. The loop iterates it in place:

<bpmn:multiInstanceLoopCharacteristics isSequential="true">
  <bpmn:loopDataInputRef>payload.value</bpmn:loopDataInputRef>
  <bpmn:inputDataItem name="row"/>
</bpmn:multiInstanceLoopCharacteristics>

loopDataInputRef is a FEEL expression, not a variable name. Copying the collection into a variable first would put a second copy of the whole batch in the instance snapshot on every park.

The XSD’s leaf types reach the cells

A CSV cell is text on the wire, but the bound type decides what it is. durationSec is declared xs:int, so it arrives as a number — which is why the transform can render it unquoted into JSON and the store can write it into an INTEGER column without a cast anywhere.

An empty cell is absence — for an optional element

Row CDR-100003 leaves rateCode empty. A tabular row has a cell for every column whether or not it carries a value, so an empty cell for an element the type declares minOccurs="0" reads as absent. Without that rule the empty cell would become <rateCode></rateCode> and fail the element’s own enumeration — and every optional column would have to be populated in every row.

An empty cell for a required element is untouched: that is a genuine data error and is still reported.

A bad cell names its row

One decode validates every row and every cell, and each violation carries a row-indexed path:

value[1]   value '4155550999' does not match pattern '\+[0-9]{8,15}'
value[2]   the value of element 'startTime' is not valid
value[3]   value 'sideways' is not one of the enumerated values [originated, received]

The row index is the part that locates the problem; the message names the element. Rows 0 and 4 are not reported.

The whole batch is refused, in the caller’s own format

<q:onValidation mode="reject"/> makes a partly-bad file an all-or-nothing refusal. Because the caller posted text/csv, the RFC 7807 problem document comes back as a table — one row per issue, diffable against the file that was sent, rather than JSON to re-parse looking for row 4,217.

A fixed-width layout is checked at package time

A fixed-width record’s columns are its only field names, so they must agree with the bound type or every row would fail at runtime for what is really a configuration mistake. sutra lint catches it:

[ERROR] SUTRA.CONFIG.CODEC_MANIFEST.REJECTED — codec 'cdr': fixed-width layout declares
column(s) ["recordIdTYPO"] that type 'CallDetailRecord' does not declare …

A csv codec gets no equivalent check: its column names arrive in the header, at runtime, so there is nothing to compare against beforehand. That asymmetry is the price of a self-describing format.

What to know before scaling it up

The multi-instance loop is one durable step. That has three consequences worth stating plainly:

  • The batch is the unit of recovery. A crash part-way through replays the loop from the first record rather than resuming where it stopped. Here that converges, because every per-record effect is a store write keyed by entry.entryId and the write is an upsert — the same rows are rewritten, not duplicated. Add a per-record <q:send>, an append or a counter and that is no longer true.
  • Say so. <q:process idempotent="true"/> is the assertion that re-running converges. Without it the fail-closed default applies and a mid-load failure is dead-lettered and consumed rather than retried — the wrong posture for a file load, and wrong by silence.
  • The decoded batch lives in the snapshot. That is unavoidable; what is avoidable is duplicating it. Iterate payload.value in place, and mark per-iteration scratch transient so it never reaches the database.

If batches outgrow what one step should own, the move is to deliver them in chunks — a file or broker transport carrying N records per message. The BPMN names channels, never transports, so nothing in the flow changes.

Run it

The example ships its own two-service stack — engine and PostgreSQL, every secret already set — so trying it is up, package, curl. You need the sutra CLI and Docker; this is the same local model as the Quickstart, pointed at a package that already exists.

1. Start the engine

docker compose -f examples/call-log-load/deploy/compose.yaml up -d

2. Seal the package into the watched directory

sutra package examples/call-log-load/deployments-src/default--call-log--1.0.0 \
     --out examples/call-log-load/deploy/deployments
packaged …/default--call-log--1.0.0.sutra (deploymentId dep-397f20f9669c0e2eaabd0e74)

package lints on the way in — the fixed-width layout is checked against the bound XSD, the store’s structure against the migration’s DDL — so a mistake in either fails here rather than on the first upload. The engine watches that directory and activates the archive without a restart.

The host port is dynamic, because nothing assumes 8080 is free on your machine:

ENGINE=$(docker compose -f examples/call-log-load/deploy/compose.yaml port engine 8080)
curl -s http://$ENGINE/sutra/health/ready
{"status":"UP","checks":[{"name":"sutra-loader","status":"UP","data":{"deployments":1,"shards":1}}]}

3. Upload a batch

curl -s -X POST http://$ENGINE/channels/cdr-upload \
  -H 'Content-Type: text/csv' \
  -H 'X-Api-Key: dev-only-cdr-key' \
  --data-binary @examples/call-log-load/sample/call-logs.csv
{"batchId":"45217d89-a520-4d7b-ba2a-453d1d9fa38a","rowsAccepted":4,"status":"loading"}

202 Accepted, in about 30 ms. The load has not finished — it has barely started. That is ack-mode: on-persist plus <q:reply continue="true"> doing exactly what the section above describes.

4. Read the rows back

docker compose -f examples/call-log-load/deploy/compose.yaml exec engine-db \
  psql -U sutra_engine -d sutra -c \
  'select entry_id, subscriber, bearing, duration_seconds, rated_amount, billable from call_log order by entry_id'
  entry_id  |  subscriber  | bearing  | duration_seconds | rated_amount | billable
------------+--------------+----------+------------------+--------------+----------
 CDR-100001 | +14155550101 | outgoing |              182 |       4.5500 | t
 CDR-100002 | +14155550101 | incoming |               45 |       0.0000 | f
 CDR-100003 | +14155550188 | incoming |              930 |       0.0000 | f
 CDR-100004 | +14155550101 | outgoing |             1204 |      21.6720 | t

Typed columns, not a JSON blob: duration_seconds is an integer and rated_amount a NUMERIC because the XSD’s leaf types were applied to the CSV’s untyped cells at decode. bearing and billable do not exist in the uploaded file at all — the transform derived them from direction.

psql needs no password here: sutra_engine is the engine’s own role (owner of sutra), and inside the container the local socket trusts it. The bootstrap superuser is postgres, which you only need for creating roles or databases.

5. Watch it refuse a bad batch

curl -i -X POST http://$ENGINE/channels/cdr-upload \
  -H 'Content-Type: text/csv' \
  -H 'X-Api-Key: dev-only-cdr-key' \
  --data-binary @examples/call-log-load/sample/call-logs-with-a-bad-row.csv
HTTP/1.1 400 Bad Request
content-type: text/csv

field,value
type,urn:bpm:diag:SUTRA.INBOUND.VALIDATION_REJECT
status,400
detail,"Inbound rejected on intake node Start of process cdr-load: 6 validation issue(s); mode=reject."
issueCount,6

Three things at once. It is a 400, not a 500 — a malformed file is the caller’s to fix, and a 5xx would invite a retry of identical bytes. It comes back as CSV, because that is what was posted. And re-running the query from step 4 still shows four rows: validation runs at intake over the whole file, so a batch with one bad cell writes nothing at all.

Drop the X-Api-Key header and you get 401 instead — the channel declares apikey auth, and the engine will not wire an unauthenticated HTTP intake.

6. The same records, as fixed-width

curl -s -X POST http://$ENGINE/channels/cdr-upload \
  -H 'Content-Type: text/plain' \
  -H 'X-Api-Key: dev-only-cdr-key' \
  --data-binary @examples/call-log-load/sample/call-logs.fixed-width.txt

Another 202 with a receipt, and the table still holds four rows — same schema, same keys, so the same records upserted rather than appended. One channel, one XSD, two wire forms, chosen by content-type.

Clean up

docker compose -f examples/call-log-load/deploy/compose.yaml down -v

Into an app you already have

If you are already running a scaffolded app rather than this stack, copy the package into it and give the engine the secrets the package names:

cp -r examples/call-log-load/deployments-src/default--call-log--1.0.0 \
      my-app/packages/call-log-load
# my-app/deploy/.env — read by the engine service (env_file), gitignored by the scaffold
CDR_UPLOAD_API_KEY=dev-only-cdr-key
CALL_LOG_DB_URL=postgres://engine-db:5432/sutra
CALL_LOG_DB_USER=sutra_engine
CALL_LOG_DB_PASSWORD=sutra-dev-only

Then sutra package packages/call-log-load --out deploy/deployments and recreate the engine (docker compose -f deploy/compose.yaml up -d --force-recreate engine) so it picks the new environment up.

This step is not optional and it fails loudly. channels.yaml and datastores.yaml name their secrets by reference — env:CDR_UPLOAD_API_KEY, env:CALL_LOG_DB_URL — and the engine resolves them, in its own environment. Exporting them in your shell does nothing; the engine is not your shell. An unresolvable channel-auth reference is fatal:

startup failed — refusing to serve
  HTTP channel auth value could not be resolved: secret-ref 'env:CDR_UPLOAD_API_KEY'
  resolves to no value (environment variable 'CDR_UPLOAD_API_KEY' is not set).

The engine will not open an unauthenticated port instead. An unresolvable store reference is narrower — that store is not registered and the rest of the deployment still serves — so watch the startup log for store NOT registered too.

Next

Engine layering

Sutra is a single Cargo workspace under rust/. The crates split cleanly along the layers a message passes through, plus one composition root that assembles them into the thing you actually run.

The layers

sutra-dist            composition root — force-links concrete codecs/transports/resolvers,
                       produces the `sutra-engine` binary (see Dockerfile)
   │
sutra-engine           the engine LIBRARY — config, deploy/activation, admin API, OTel,
                       audit sinks, the outbox tick loop. Domain-neutral: it collects
                       codecs/transports/resolvers generically via their SPIs and names none.
   │
sutra-channels         protocol-neutral channel binding + dispatch: decode → two-tier
                       validate → route to a start event or a parked wait → execute
   │
sutra-executor         the token executor: gateways, sub-processes, data associations,
                       compensation, path-coverage tracking — synchronous and stateful paths
   │
sutra-bpmn             the BPMN 2.0 + q: extension model and loader
   │
sutra-feel / sutra-dmn / sutra-srl / sutra-templates
                       the expression + decision + template languages every task type runs on
   │
sutra-persistence      durable state: instances, outbox, inbox dedup, lease, audit — PostgreSQL
                       (MySQL/MariaDB/SQL Server dialects follow the same pattern)

A message arrives on a channel (sutra-channels), gets decoded and validated by a codec (sutra-codec-spi + the concrete codec crates), is routed to a process, and the process runs on the token executor (sutra-executor) over the BPMN model (sutra-bpmn), evaluating FEEL/DMN/.srl expressions (sutra-feel, sutra-dmn, sutra-srl) and rendering templates (sutra-templates) along the way. Anything durable — the instance snapshot, the outbox, the inbox dedup row, a data store write — goes through sutra-persistence (or a module’s own store, for data stores; see Data stores).

flowchart LR
    IN["inbound message"] --> CH["sutra-channels<br/>decode · two-tier validate · route"]
    CO["sutra-codec-spi<br/>+ the concrete codecs"] -.->|"codec SPI"| CH
    CH -->|"typed payload"| EX["sutra-executor<br/>token execution"]
    EX --> BP["sutra-bpmn<br/>model + q: extensions"]
    EX --> LG["sutra-feel · sutra-dmn<br/>sutra-srl · sutra-templates"]
    EX -->|"durable state"| PE["sutra-persistence"]
    EX -.->|"a q:store binding"| DS["a module's own data store"]

Decode-and-validate happens once, at the doorway; everything below the channel layer only ever sees a typed payload, and every durable write leaves through one crate.

The composition root

sutra-engine — the library — never names a concrete codec, transport, or secret-resolver implementation. sutra-dist is the one crate allowed to: it force-links the schema-less formats (json, xml, yaml, csv, raw-text, raw-bytes), the redactors, the vendor secret-resolvers, and the feature-selected transports, and produces the sutra-engine binary the container image ships (docker build -f rust/Dockerfile rust/). It force-links no domain codec: every message standard is a proprietary extension crate built outside this repository, registering through the same SPIs and force-linked by its own composition root, which is precisely what this split is for. This split is what lets a hardened build drop everything it doesn’t need — cargo build -p sutra-engine --no-default-features --features file links no broker client at all — without touching a line of the neutral engine or channel code.

flowchart LR
    F["sutra-formats<br/>json · xml · yaml<br/>csv · raw-text · raw-bytes"] --> D
    R["the redactors"] --> D
    S["vendor secret-resolvers"] --> D
    T["transports"] -->|"feature-selected"| D
    E["sutra-engine (the library)<br/>names no concrete"] --> D
    D["sutra-dist<br/>composition root — force-links"] --> B["the sutra-engine binary<br/>the container image ships"]

Every concrete implementation enters the binary at exactly one crate — which is what lets a hardened build link no broker client at all while the neutral crates it depends on stay untouched.

See Domain neutrality and the SPI model for exactly how that boundary is drawn and mechanically enforced, and what a third party has to write to add a new transport or codec.

Where the tooling sits

sutra-cli (the sutra binary) depends on the model/loader layer read-only for its inspection commands (describe, dispatch-graph, simulate, explain), and on sutra-persistence + sutra-channels for the commands that touch a running engine or a database behind it (deploy and migrate against the engine’s own; coverage check --archive against the one a deployment’s coverage store declares). It is not part of the engine’s own runtime dependency graph — a deployed engine binary has no CLI code linked into it.

Next

Domain neutrality and the SPI model

The engine core never names a business domain, a vendor, or a wire format. Not SWIFT, not FedNow, not ISO 20022, not Kafka, not PostgreSQL-as-a-brand. Payments, health, and EDI live entirely in codecs, transports, validators, and deployment packages — never in the token executor, the BPMN model, or the channel dispatcher. This is the single rule everything in this chapter exists to serve, and it is not a style guideline: it is checked by a build gate on every change.

Why this is load-bearing

A domain-neutral core is what lets one engine binary serve payments today and healthcare tomorrow without a fork, what lets a hardened build strip out everything it doesn’t need down to the byte, and what keeps the surface a security review has to reason about small and stable. The moment a business term leaks into the executor or the model, that guarantee is gone for everyone, silently.

The gate: sutra-archtest

rust/crates/sutra-archtest is the Rust twin of what used to be an ArchUnit test on the retired reference baseline. It walks a fixed set of crates’ src/ trees, strips comments (a domain term cited in a doc comment is fine — only a term baked into an identifier or a string literal is a violation), and fails the build on any hit against a denylist:

#![allow(unused)]
fn main() {
pub const DOMAIN_DENYLIST: &[&str] = &[
    "swift", "fednow", "fedwire", "pacs", "camt", "nacha", "edifact", "hl7", "iso20022", "x12",
];
}

Enforced — must be domain-literal-free:

  • The neutral core: sutra-executor, sutra-bpmn, sutra-feel, sutra-dmn, sutra-srl, sutra-templates, sutra-persistence, sutra-datastore, and the codec/format/schema SPI itself (sutra-codec-spi).
  • The assembly/binding layer: sutra-channels (protocol-neutral channel binding) and sutra-engine (the library) — domain-literal-free since the transport and codec extraction and the sutra-dist composition-root split.

Deliberately excluded — the legitimate domain edge, by design:

  • sutra-dist — the composition root. It force-links the built-in formats, the redactors, the vendor secret-resolvers, and the feature-selected transports, so it necessarily names them. That’s the one place wiring concretes to SPIs is supposed to happen — not a core leak.
  • The concrete implementations themselves: sutra-formats, sutra-codec-schema, every sutra-transport-<vendor> crate, and sutra-redactor-pci.
  • Tooling and test crates (sutra-cli, sutra-conformance, sutra-testkit, the generators).

Read that exclusion list again for what isn’t on it. This repository contains no message-standard codec at all. The built-in codec set is the six schema-less formats in sutra-formats — json, xml, yaml, csv, raw-text, raw-bytes — and sutra-dist force-links nothing else that registers one. Every wire standard — SWIFT MT, SWIFT MX, rail variants such as FedNow, and the EDI/segment families (HL7 v2, X12, NACHA/ACH, EDIFACT) — is a proprietary extension crate maintained outside this repository, consuming exactly the public SPIs a third party consumes.

That is the strongest available evidence that the SPI boundary is real rather than aspirational: adding a message standard to a distribution costs one Cargo dependency and one use <crate> as _; line in that distribution’s own composition root. Nothing in sutra-engine, sutra-channels, or this repository’s composition root changes, and there is no central list anywhere to add a name to. The gate has nothing to exclude for those crates because they are not here; the neutral core cannot leak a term that never enters its dependency graph.

flowchart TD
    subgraph here["This repository"]
        CORE["the neutral core<br/>no domain literal, gated"]
        FMT["sutra-formats<br/>six schema-less formats"]
        DIST["sutra-dist"]
    end

    subgraph out["Outside this repository"]
        STD["a codec crate<br/>against the public SPI"]
        OWN["its composition root"]
    end

    CORE --> DIST
    FMT --> DIST
    DIST --> B1["the stock binary"]
    CORE --> OWN
    STD -->|"one dependency,<br/>one link line"| OWN
    OWN --> B2["that distribution's binary"]

A distribution adds a wire standard by force-linking one more crate into its own composition root — there is no central list in this repository to append a name to, which is what makes the neutrality gate cheap to keep passing.

make lint runs this gate alongside clippy on every change (see Contributing) — a business term landing in a gated crate fails CI, not a code review comment.

The mechanism: self-registration behind a neutral SPI

Every extension point follows the same shape: a small SPI crate defines a trait and a process-wide registry; each concrete implementation is its own crate that self-registers into that registry via inventory at link time. The neutral crate collects what got linked in — it never imports or names a specific implementation. Implementing an extension is registering it; there is no separate central list to forget to update (a gap that bit an earlier iteration of the codec set, closed by moving to this pattern).

flowchart LR
    CR["a composition root"] -->|"force-links the crate"| IMPL["a concrete crate<br/>a transport, a codec, a resolver, a redactor"]
    IMPL -->|"inventory::submit! at link time"| REG[("the SPI crate's<br/>process-wide registry")]
    REG -->|"collected, sorted by name"| NEU["the neutral engine<br/>drives every entry through the same trait"]

The neutral side only ever iterates what got linked in, so adding an implementation is a link-time fact rather than an edit to anything the engine owns.

Transports — sutra-transport-spi

#![allow(unused)]
fn main() {
pub struct TransportFactory {
    pub transport: &'static str,          // the `transport:` value in channels.yaml this wires
    pub spawn: TransportSpawn,             // (defs, engine, pool, envref-resolver, runtime) -> Arc<dyn TransportChannels>
    pub register_sink: fn(&mut SinkRegistry),
    pub handles_on_complete: bool,         // self-declared capability — see below
}
inventory::collect!(TransportFactory);
}

Each sutra-transport-<vendor> crate submits one TransportFactory. The engine assembly iterates transport_factories() (sorted by name for determinism) and drives every one of them through the identical TransportChannels lifecycle trait (rewire on an activation flip, drain on shutdown) — there is no if transport == "kafka" anywhere in the engine.

Capability self-declaration, not a hardcoded transport check. handles_on_complete is how a transport tells the engine whether it can realize ack-mode: on-complete (a broker deferring its settle through the engine’s ack registry, or an on-listener transport like HTTP holding the connection to completion). When a channel declares on-complete on a transport whose factory reports false, the engine assembly emits SUTRA.ACK.ON_COMPLETE_UNSUPPORTED and runs on-persist instead — a loud, generic degrade driven by a flag the transport itself set, not a list of vendor names the engine maintains. See Acknowledgement modes for the full per-transport wiring table this flag produces.

Codecs — sutra-codec-spi

#![allow(unused)]
fn main() {
pub trait PayloadCodec {
    fn name(&self) -> &str;
    fn accepted_content_types(&self) -> Vec<String>;
    fn decode(&self, body: &[u8], content_type: Option<&str>) -> DecodeResult;   // never panics
    fn declared_message_types(&self) -> Vec<String> { Vec::new() }
    fn shape_of(&self, message_type: Option<&str>) -> Option<SchemaShape> { None }
    fn bind_refs(&self, resolve: CodecRefResolver) -> Result<(), String> { Ok(()) }
    fn seal(&self, body: &[u8], content_type: Option<&str>) -> Result<Option<Vec<u8>>, String> { Ok(None) }
    // + encode (the reply direction)
}
}

Sealing and references. A codec that must change the bytes it sends, for example to add a digital signature, implements seal. The engine calls it after the codec’s own read-back of an outbound message passed and before the message is committed to the outbox, for sends and replies alike, template renders included. A seal error refuses the message (SUTRA.OUTBOUND.SEAL_FAILED). A codec whose bundle files name references (secret:NAME, env:NAME) resolves them in bind_refs. The engine calls it once, when the deployment is planned on the runtime host, through the same resolver channel credentials use. An unresolvable reference refuses the deployment (SUTRA.CODEC.BIND_REFS_FAILED). The engine still names no format and no algorithm: what sealing means is the codec’s business.

A zero-config, globally-named codec — each built-in format (json, xml, yaml, csv, raw-text, raw-bytes), and any extension crate that wants a global name — submits a BuiltinCodec { name, make }; builtin_codecs() collects and sorts them, and each is addressable as urn:sutra:codec:<name> with no per-package configuration. A schema-backed user codec (compiled from a package’s own schemas/<name>/*.xsd) does not self-register this way — it’s instantiated per package, per the path-derived URN rule in Deployment packages. Either way, the engine only ever calls through the PayloadCodec trait; it has no branch for “this is the such-and-such standard one,” and no way to acquire one — the type it would have to name is not in its dependency graph.

Bundle codec kinds — a codec crate registers its own schema-folder shape

The newest extension point in this family follows the identical inventory-pull shape, one level into how a package’s own schema folder is interpreted. schemaKind: xsd and schemaKind: json-schema are generic — a folder of schema files the engine validates against — but some standards are a whole profile: an envelope grammar, a mapping from wire-level message names to schema files, versioned editions revved on the standard’s own release cadence. A codec crate that needs that shape submits a BundleCodecKind { kind, build } in sutra-codec-schema::bundle next to its PayloadCodec impl — implementing the kind is registering it, exactly like a BuiltinCodec, and sutra-codec-schema itself stays free of any knowledge of which standards use it. A package’s schemas/<name>/codec-manifest.yaml declaring schemaKind: <kind> is what selects a registered bundle over the generic ones; an unserved kind is a fail-closed deploy error naming the kinds the running build actually serves.

A market-infrastructure rail codec — an extension crate outside this repository — is the first user of this extension point (see Channels and transports and Deployment packages for the manifest shape and deployment-scoped registration it produces). It’s also a useful case study for the SPI pattern more broadly: everything specific to one rail’s profile — its envelope root names and namespaces, its per-direction wrapper tables, its quirks (a wrapper shipped without the profile’s usual name prefix, several wrappers backed by one underlying message) — is a single data value the codec’s internal walker/validator/projection code reads generically. A second rail that follows the same envelope-and-wrapper-editions pattern would supply its own profile value and reuse that machinery rather than re-implement it — the same “data, not a branch” discipline the rest of this chapter describes, one layer further in.

Data stores — sutra-datastore

The DataStore SPI is the same shape one level down: a provider (sql today; see Data stores) resolves its own connection from datastores.yaml — never the engine’s internal datasource — and exposes get/put/get_for_update/ put_if_revision against (store_name, store_key). The engine’s executor calls through this trait for every <q:store>-bound data association; it has no idea whether the value on the other side is an account balance or a customer record.

Secrets — sutra-envref-spi

The vendor-neutral seam behind env:, secret:, ${…} placeholders, and vendor schemes like vault:… / aws-secrets:…. A vendor resolver crate (sutra-envref-vault, sutra-envref-aws, …) submits an EnvRefResolverEntry; the engine resolves a whole reference generically and names no vendor SDK. This is what lets channels.yaml and datastores.yaml carry credentials as references, never literals — see Configuration reference.

Redactors — sutra-redactor-spi

A sutra-redactor-<standard> crate submits a RedactorEntry that locates sensitive spans in a decoded payload (a JSON-Pointer-shaped path + a reason code); the engine masks every located path on every observability surface and marks it for encryption at rest. Fail-closed by construction: a redactor that panics or errors tells the engine to over-mask the whole bound payload rather than risk a leak — the opposite failure mode from a validator crash, which becomes an ordinary business-reject issue instead.

Validators

<q:simpleValidator ref="…"> names a field-content validator (iso-3166-country, iso-4217-currency, iso-9362-bic, …) out of a neutral registry the core knows nothing about (see The q: namespace). The concrete validators are domain content and live in extension crates, outside this repository, for the same reason the domain codecs do — which is why the neutrality gate has nothing to exclude here.

The lifecycle bus — ExecutionListener

One more neutral seam worth knowing about even though it isn’t a “plug in a new vendor” SPI: every cross-cutting concern that needs to react to instance/token/task lifecycle events — audit, OTel metrics, the deferred-ack registry — implements the same ExecutionListener trait and is fanned out from a plain Vec<Rc<dyn ExecutionListener>> the executor calls on on_instance_started / on_instance_completed / on_instance_suspended / etc. There’s no dependency-injection container; listener registration happens explicitly at executor-construction time in the engine assembly. See Acknowledgement modes for DeferredAckRegistry as a concrete example, and Observability for the OTel listener.

Where business content actually lives

Given all of the above, a real deployment’s domain-specific content lives in exactly two places:

  1. Inside a deployment package — the BPMN processes, DMN/.srl rules, channels.yaml, datastores.yaml, and the package’s own XSD-backed codec (see Deployment packages). This is where almost all business logic belongs, and it needs zero Rust code.
  2. In an extension crate, only when the domain needs a new kind of codec, transport, validator, or redactor that doesn’t already exist — implemented against the relevant SPI above and linked in by a composition root (sutra-dist, or your own if you build a custom binary).

Worked example: adding a transport

Say you need a new broker Sutra doesn’t ship a transport for. The shape is fixed by the pattern above:

  1. Create sutra-transport-<vendor>, depending on sutra-transport-spi and sutra-channels.
  2. Implement TransportChannels (transport(), consumer_count(), rewire(active), drain(), stop_all_detached(), and the one optional capability inbound_router() — only HTTP-shaped transports return Some).
  3. Implement the inbound spawn function with the uniform signature (&[ChannelDefinition], EngineHandle, Option<PgPool>, EnvRefResolver, tokio::runtime::Handle) -> Result<Arc<dyn TransportChannels>, Diagnostic>, and an outbound sink registrar fn(&mut SinkRegistry).
  4. Decide whether your transport can realize ack-mode: on-complete (deferred settle, or holding the connection) and set handles_on_complete honestly — get this wrong and channels either silently under-deliver on a promise, or trigger the loud unsupported diagnostic needlessly.
  5. inventory::submit! { TransportFactory { transport: "your-broker", spawn, register_sink, handles_on_complete } } next to your implementation.
  6. Add your crate as an optional dependency behind a same-named Cargo feature in whatever composition root force-links it (sutra-dist, or a custom one) — nothing in sutra-engine or sutra-channels changes.

A new codec follows the identical shape against PayloadCodec instead, submitting a BuiltinCodec for a zero-config global codec, or shipping as a package-local schema-backed codec with no self-registration at all. Every message-standard codec in existence takes that first path — including the ones a downstream distribution ships — so it is a well-travelled route, not a theoretical extension point kept alive by a single in-tree user.

Next

  • Deployment model — how a package’s declared channels/codecs/stores get resolved against whatever is linked into the running binary.
  • Contributing — the repo map, including where each SPI crate and its concrete implementations live.

Deployment model

A .sutra archive’s runtime identity is a single opaque deploymentId = sha256(manifest) — the manifest is derived at seal time (sutra package), never hand-authored. Deploying is therefore naturally idempotent: re-deploying identical bytes is a no-op. This page is the architecture behind Your first deployment and Deploy, hot-deploy, and rollback — how activation actually works underneath the CLI.

The database is the source of truth

Sealed archives are stored in the engine’s own datasource — the same core database that backs instance state, the outbox, and the lease table — as a deployment_archive row keyed by a stable slot (the archive’s tenant--module--version key). Exactly one row per slot is ever active; deploying a new revision to an existing slot replaces the old active row in one transaction. This is what makes hot-deploy a replace, not a restart: the slot name is stable across versions, only the content-addressed deploymentId changes.

Loading is symmetric — on boot, the engine loads its active set from the database (WHERE status='active'), not by scanning a directory. A restarted or newly-scaled replica rehydrates its active set from the shared database, not from whatever a local volume happens to reflect.

Deploy is an API call

POST /admin/deployments is the one control path onto a running engine (auth-gated — see Configuration reference for the admin auth scheme). It:

  1. Accepts the sealed archive’s bytes.
  2. Re-verifies them fail-closed (the same archive-integrity check sutra package already ran client-side).
  3. Stores the new revision as the slot’s active row, in a transaction.
  4. Runs the two-phase activation flip in-process (drain the old revision if one exists, activate the new one).
  5. Returns synchronously: 200 {deploymentId, phase: "Active"}, or a 4xx carrying the SUTRA.DEPLOY.* reject diagnostic.

Because the HTTP response is the activation signal, there is no propagation window to reason about, no separate “did it actually take?” step — this is the property that makes deploys deterministic rather than eventually-consistent from the caller’s point of view.

sequenceDiagram
    participant C as sutra deploy
    participant R1 as The replica that took the call
    participant DB as The engine's database
    participant R2 as Every other replica
    C->>R1: POST /admin/deployments — the sealed archive
    R1->>R1: re-verify the archive, fail-closed
    R1->>DB: store as the slot's active row
    Note over DB: one transaction — exactly one active row per slot
    R1->>R1: two-phase flip — drain the old revision, activate the new
    R1-->>C: 200 {deploymentId, phase "Active"}
    DB--)R2: LISTEN/NOTIFY after commit, version-poll as the fallback
    R2->>R2: converge on the committed active set

The caller’s answer arrives only after the flip, so it is definitive; the rest of the fleet converges off the committed row rather than off anything the deploying replica tells it.

DELETE /admin/deployments/{slot|id} marks a row draining; the engine’s activation flip drains it (no new intake, retire once quiescent).

stateDiagram-v2
    [*] --> Active: POST /admin/deployments
    Active --> Draining: a new revision takes the slot
    Active --> Draining: DELETE by slot or id
    Draining --> Retired: no new intake, quiescent
    Retired --> [*]

The slot outlives its revisions: taking a slot moves the old row sideways into draining rather than deleting it, which is what makes a hot-deploy a replace rather than a restart.

Sync vs. async — the same call, two response shapes

For a small or local deploy, the synchronous form above is the whole story: one request, one definitive answer. For a large deployment where the engine’s own plan-and-flip work risks a long-held request — mainly a concern behind an ingress with a short read timeout — the identical endpoint accepts an async mode:

  • Sync (default): POST /admin/deployments blocks until the flip completes, returning Active or a reject.
  • Async (opt-in): the same POST returns immediately with 202 {deploymentId, status: "Pending"}; activation runs in the background. The caller learns the outcome one of two ways:
    • Poll GET /sutra/deployments/{id} — short, ingress-safe requests — until it flips to Active or Failed.
    • A completion event. The caller opts in with callback=<https-url> (a webhook) and/or notify=<broker-uri> at deploy time; on completion the engine emits a CloudEvent (com.sutra.deployment.activated / com.sutra.deployment.failed) carrying {deploymentId, slot, revision, status[, error]}. This rides the engine’s existing outbound spine — one more durable outbox entry, delivered by the same dispatcher that sends every other outbound message — so a deploy-complete notification is not a separate mechanism, just another emission.
flowchart LR
    P["POST /admin/deployments"] ==>|"sync — the default"| A["blocks through the flip<br/>200 Active, or a SUTRA.DEPLOY.* reject"]
    P -->|"async — opt-in"| B["202 Pending, activation runs behind it"]
    B -->|"poll"| C["GET /sutra/deployments/{id}<br/>until Active or Failed"]
    B -->|"push"| D["a CloudEvent to callback= and/or notify="]
    D -->|"one more outbox row"| E["the same dispatcher that sends<br/>every other outbound message"]

This is a long-running-operation (LRO) shape, not a bespoke deploy protocol: accept fast, do the work in the background, let the caller choose polling or a push notification.

Multi-replica convergence

The replica that handled the deploy call activates locally and returns synchronously; every other replica converges asynchronously on the committed database state — on PostgreSQL via LISTEN/ NOTIFY after the commit, with a version-poll fallback for dialects with no equivalent (MySQL/MariaDB/SQL Server) and as the resilience backstop on Postgres itself. A single-replica run needs no convergence step at all — the one replica that deployed is already the whole fleet.

Next

Multi-tenancy and isolation

Sutra runs many tenants behind one engine and one database, with isolation enforced at the storage layer rather than by giving each tenant its own infrastructure.

flowchart LR
    M["inbound message"] --> CH["channel dispatcher<br/>tenant = the package's own label"]
    CH --> Q{"per-tenant ceilings<br/>concurrent instances · inbound rate"}
    Q -.->|"over the cap"| REJ["HTTP 429 · broker nack"]
    Q -->|"admitted"| EX["the executor"]
    EX --> P["the persistence layer<br/>injects the tenant WHERE clause"]
    P --> DB[("PostgreSQL row-level security<br/>SET LOCAL app.current_tenant")]

Two different gates: the ceilings are an admission concern, enforced before an inbound ever reaches the executor, while the tenant partition itself is enforced twice on the way to disk — the second time by the database, which no query can talk its way around.

Tenant identity is a package label

A deployment package’s tenant is an opaque label in package.yaml (alongside module and version — see Deployment packages):

labels:
  "module": "money-transfer"
  "tenant": "default"
  "version": "1.0.0"

The engine never interprets this string — it’s a selector for observability, routing, and the row-level-security partition key described below. Because a package is fully self-contained (no shared resource tree, no inheritance between packages), a tenant’s isolation boundary is simply “whatever package it was labeled into” — there’s no separate tenant-configuration document a package’s processes have to be matched against.

Storage isolation: row-level filtering + PostgreSQL RLS

Every durable table the engine owns — instance state, the outbox, the inbox dedup table, the alias index, the audit event log, the wait-state table, the dead-letter store — carries a tenant column and a PostgreSQL row-level-security policy keyed on a session variable, applied by the engine’s own SQL migrations. Defense in depth, two layers:

  1. Application-level filtering — every query goes through the persistence layer, which requires a tenant argument and injects the WHERE clause; there’s no ad hoc database access path around it.
  2. PostgreSQL RLS — SET LOCAL app.current_tenant = '…' at the start of each request transaction, with a CREATE POLICY … USING (tenant_id = current_setting('app.current_tenant')) on every table. Even a query that forgot its own WHERE clause cannot return another tenant’s rows — the database itself is the second line of defense, not just the repository layer.

A startup check closes the obvious way to defeat this. RLS policies are silently bypassed for any role with BYPASSRLS, SUPERUSER, or ownership of the target tables. The engine probes its own connecting role at startup and refuses to start if either flag is set — an operator who pointed the engine at postgres or a role minted with BYPASSRLS finds out at boot, not after a cross-tenant leak. Production role setup:

CREATE ROLE sutra_app LOGIN PASSWORD '<from-secret>';
ALTER ROLE sutra_app NOBYPASSRLS;
GRANT SELECT, INSERT, UPDATE, DELETE ON <engine tables> TO sutra_app;

(Migrations themselves run as a separate, table-owning role that the engine runtime never uses.)

Quotas

Two per-tenant ceilings, enforced by the channel dispatcher before an inbound reaches the executor:

  • maxConcurrentInstances — a hard cap on simultaneously in-flight instances for the tenant, checked against a live count in instance_state — globally coherent across every replica, since every replica queries the same table.
  • maxInboundRatePerMinute — a sliding 60-second admission window. This one is per-replica, not fleet-wide: each replica tracks its own window in memory rather than paying a synchronous round-trip to the database on every inbound message. A tenant saturating one replica’s window can still be admitted by a second replica until its window also fills — a deliberate latency/coherence trade-off. A hard global rate guarantee, if you need one, belongs in front of the engine (an API gateway or service-mesh L7 limiter), not in the dispatcher.

Both rejections surface as the same class of diagnostic the payload-size cap uses (see Limits and quotas), translated to the right per-transport signal (HTTP 429, a broker nack, …).

flowchart LR
    IA["inbound → Replica A"] --> DA{"admit?"}
    IB["inbound → Replica B"] --> DB2{"admit?"}
    WA["A's own 60s window<br/>in memory"] --> DA
    WB["B's own 60s window<br/>in memory"] --> DB2
    C[("live count in instance_state<br/>one shared table")] --> DA
    C --> DB2

The concurrency cap is one shared count, so it holds fleet-wide; the rate window is per-replica memory, so a fleet of N replicas admits up to N windows’ worth before anything says no.

Audit isolation

Audit rows carry the tenant column and are covered by the same RLS policy as every other engine table. Where a JSONL audit sink is configured, it writes per-tenant directories, so a log shipper can fan out per-tenant index/retention policy downstream without the engine knowing anything about that policy itself.

Next

  • Replica semantics — the mechanisms (leases, row locks, SKIP LOCKED) that keep all of this correct across a multi-replica engine.
  • Limits and quotas — the operator-facing configuration surface for the ceilings above.

Replica semantics

Sutra runs as an active-active stateless replica set — all durable state lives in PostgreSQL, and no replica owns any particular instance, tenant, or channel by default. Any replica can pick up any token at any time, which is what lets the engine scale horizontally with no sticky routing.

What every replica does

Each replica is one engine process running inbound channel listeners (the HTTP server, broker consumers), the token executor for instances it picks up from persistence, an outbox worker sending replies it claims, and the health/metrics endpoints. Execution inside the process runs on one or more identical actor lanes — see Execution lanes; lanes are an in-process concern and change nothing on this page. Three pieces of work don’t tolerate concurrency and are held by exactly one replica at a time instead: timer firing, stuck-instance scanning, and terminal-history purging.

flowchart TD
    R1["Replica 1"] --> DB
    R2["Replica 2"] --> DB
    R3["Replica 3"] --> DB
    DB[("The engine's own PostgreSQL")] --- LEASE["a lease row per role<br/>timers · sweeps · purges<br/>singleton channels"]
    DB --- CLAIM["the instance row's owner<br/>gates advancing one instance"]
    DB --- OUT["outbox rows<br/>FOR UPDATE SKIP LOCKED"]
    DB --- IBX["the inbox unique index<br/>ON CONFLICT DO NOTHING"]

Every replica is identical and interchangeable; each of the four things that must not happen twice is a different row-level primitive in the one shared database, not a coordination service.

Leader election: a PostgreSQL-backed lease

There is no Kubernetes-native Lease object and no separate coordination service — leader election is a lease row in the engine’s own PostgreSQL database (DbLeaderElection over PgLeaseStore). One poll task per role tries to acquire the lease at a fixed cadence (ttl = 30s, poll = 10s); a successful acquire flips that replica to leader for that role, a contended one flips it to follower. There’s no push notification — polling is the mechanism.

Roles are dynamic, not a fixed pair: the engine registers one lease per singleton channel role (channel_role(tenant, channel)) the first time it’s needed, so declaring a channel singleton: true (see Channels and transports and the money-transfer example) starts contending a lease without any separate configuration. When no engine datasource is configured at all, every replica simply leads unconditionally (AlwaysLeading) — there’s no third posture to reason about.

Instance ownership claims

Leases gate roles. What keeps two replicas from advancing the same instance at the same moment is a different, finer mechanism: a per-instance ownership claim, taken by a compare-and-set on the instance’s own row.

Every resume path claims first. A correlated relay arriving on a channel, a timer firing, an administrative migration — each one takes the claim before it rehydrates anything, and a claim it cannot take is a refusal, never a wait. The refusal is retry-safe by design and carries a structured code (SUTRA.RUNTIME.RESUME.CLAIM_HELD, or the admin surface’s SUTRA.ADMIN.MIGRATE.CLAIM_HELD): a broker relay is requeued for redelivery, a timer fire is deferred to a later tick, an admin call answers 409 having read and written nothing. Nothing is ever half-applied while contended, and nothing blocks waiting for a lock held on another replica’s timescale.

The claim is released inside the same transaction that commits the step’s own writes, so “committed” and “unowned again” are one atomic fact. Every exit that does not commit — an early refusal, an error, a panic — releases the claim explicitly on its way out, so a claim never outlives the work it was protecting.

sequenceDiagram
    participant A as Replica A
    participant I as The instance's own row
    participant B as Replica B
    A->>I: compare-and-set — take ownership if free or stale
    I-->>A: claimed
    B->>I: compare-and-set for the same instance
    I-->>B: no match — SUTRA.RUNTIME.RESUME.CLAIM_HELD
    Note over B: relay requeued, timer fire deferred, admin 409
    A->>I: commit the step's writes and release the claim
    Note over A,I: one transaction — committed and unowned are the same fact

The whole mechanism is two round trips against one row: the loser of the race has read and written nothing, and the winner never holds the claim past the commit it was protecting.

The claim is re-entrant for the same owner, and the invariant that makes that safe is worth stating exactly: an owner id names one execution lane in one process, and a lane advances instances one at a time. Same owner therefore means same lane, which already means serialized — so re-claiming what you already hold is a heartbeat refresh rather than contention. Owner ids carry the lane index for exactly this reason, and the administrative migration path takes a deliberately distinct owner suffix so that a migration can never re-enter past a resume this same replica has in flight.

The stuck-instance sweep is the backstop for the case no protocol can prevent — a replica that dies mid-step and never releases. The instance-sweeper lease role scans on sutra.instance.sweep-interval (default PT1M) and clears any claim whose owner has been silent longer than sutra.instance.claim-timeout (default PT5M). It sweeps by age and is owner-blind, so it needs to know nothing about how owner ids are shaped.

Durable FAILED is a state, not a disappearance

An instance whose execution fails fatally is not deleted and does not silently linger as a mystery park. Its snapshot is re-stamped FAILED in place, carrying the structured failure code and the captured detail, and every waiting row it held is resolved in the same transaction — so no timer refires against it and no relay finds a live wait to satisfy.

FAILED is deliberately not terminal:

  • it is always retained, regardless of the history-retention window, because it needs an operator rather than a clock;
  • it keeps blocking its deployment’s retirement (below), so a dead-but-unhandled instance cannot be quietly swept under a hot-deploy;
  • it is the one state an instance can be migrated and resumed out of — repair the model, move the instance onto it, then explicitly bring it back.

Restoring it is exactly the inverse of the failure commit: the failure keys are dropped, the status goes back to suspended, and the parks the failure tore down are re-armed — after which the instance comes back through the ordinary claim-guarded paths. There is no privileged resume entry point.

Inbox dedup via row locks

An inbound message’s (tenant, channel, event_id) triple is inserted with ON CONFLICT DO NOTHING; whichever replica’s insert actually produced a row owns starting that instance, and any other replica that raced it (or a genuine redelivery) sees nothing came back and treats it as a dedup hit. This is a plain unique-index conflict — no application-level locking, and it works identically whichever replica happens to receive the redelivered copy. A delivery that can’t be checked because the database doesn’t answer is refused and retried, not run (details).

Outbox processing: SKIP LOCKED and a claim lease

Every reply-in-waiting sits in an outbox table. Each replica’s outbox dispatcher claims a batch with FOR UPDATE SKIP LOCKED, so replicas never contend on the same row. Those row locks end when the claim’s transaction commits, which is before the rows are delivered. So the same statement also leases each claimed row: it moves the row’s due time 30 seconds ahead. No claim on any replica returns the row before then. While the row waits for a delivery worker or is being delivered, the claiming replica renews the lease every 10 seconds. A successful delivery deletes the row, and a failed one replaces the lease with its retry backoff. Under normal operation no row is sent twice, however many replicas share the database.

A replica that dies holding rows stops renewing their leases. Within 30 seconds the leases lapse, and the next replica to claim delivers the rows. The rare case where the original send went out just before the crash is absorbed downstream: the outbox row’s key travels as the message’s own idempotency token.

The renewal never waits for a row lock. It skips a row another transaction holds, because the only holders of an in-flight row are the transactions that end its lease on purpose (the delete of a delivered row, a retry’s backoff, a poison) or another replica’s claim of a lease that has already lapsed. It renews at most 1024 rows per transaction, leaves out rows already delivered and waiting for their delete, and has a database connection of its own. A renewal round slower than 5 seconds is logged at warn.

Fixed in 0.6.12. From 0.6.8 to 0.6.11 the renewal was one statement over every in-flight row, and it locked rows that the batch delete of delivered rows was locking too, each in its own order. Under a burst of deliveries PostgreSQL aborted one of them with deadlock detected: a lost delete was retried row by row (outbox delete failed: deadlock detected, nothing lost), a lost renewal renewed nothing that round. When the engine fell behind, the renewal took 1 to 3 seconds.

Fixed in 0.6.8. Releases 0.6.5 to 0.6.7 took the row locks but wrote no lease. A second replica’s claim could pick up a row the first was still delivering, and under multi-replica load about half of all outbound messages were sent twice. Nothing was lost and every instance still completed once. A single engine was not affected.

Retiring a draining deployment: a three-legged gate

A hot-deploy leaves the previous graph draining rather than deleting it, precisely so instances pinned to it keep resuming on the definition they started under. It retires only when it is genuinely quiescent — and “quiescent” is three independent facts, all of which must hold:

  1. No active instances pinned to it. Retained terminal history does not count (it would otherwise pin a deployment open for a whole retention window); FAILED instances do count, because they are live work awaiting an operator.
  2. No pending outbox rows minted by it. An emission belongs to the channel bindings that produced it, so it drains where it was made. A delivery that exhausted an opt-in attempt ceiling is marked terminally poisoned and stops counting — “we gave up” is a durable, visible state, and it must not hold a deployment open forever.
  3. No parked external tasks waiting on it. A task parked for a pull worker is outstanding work with no in-flight anything to observe; retiring underneath it would strand the worker’s completion. Tasks that turned terminal after exhausting their worker budget stop counting, for the same reason poisoned deliveries do.
flowchart LR
    I["active instances pinned to it<br/>FAILED counts, terminal history does not"] --> G{"quiescent?"}
    O["pending outbox rows it minted<br/>terminally poisoned ones stop counting"] --> G
    T["parked external tasks<br/>budget-exhausted ones stop counting"] --> G
    G -->|"all three read zero"| R["retired"]
    G -.->|"any is non-zero"| D["stays draining"]

All three legs are database-scoped counts, so the gate reads the same from every replica and is unaffected by which replica happens to run the sweep.

Recovery on pod death

Because token state lives in the database, a dead replica leaves nothing to recover from memory:

  • Mid-token-execution — the instance’s claim goes stale with its owner; the stuck-instance sweep clears it and the next replica to see ready work picks the instance up.
  • Mid inbox-to-instance handoff — an inbox row without a matching instance past a recovery threshold gets retried; the same unique-index dedup makes the retry safe.
  • Mid outbox send — the dead replica’s claim lease lapses within 30 seconds and the row is delivered by whichever replica claims it next.
  • The leader itself dies — its lease expires within its TTL and another replica’s next poll acquires it; timer scheduling pauses for at most that window, while outbox processing continues uninterrupted on every other replica throughout.

Scaling signal

CPU is not a meaningful autoscaling signal for an event-driven engine — the right one is backlog depth (inbox rows waiting to start an instance, outbox rows waiting to send), which is what a KEDA ScaledObject querying the engine’s own tables drives off. See the OpenTofu module under deploy/modules/sutra/ for the shipped shape.

Non-goals

No active-passive mode (every replica processes real work; the lease only gates the singleton pieces), no sticky session routing, no tenant-to-replica affinity (isolation is a PostgreSQL RLS concern — see Multi-tenancy and isolation — not a placement one), and no engine-managed broker resources (queues/topics are provisioned by your own infrastructure tooling; the engine connects to what already exists).

Next

  • Execution lanes — the in-process half: N actor lanes under one replica, and why claims rather than routing keep them correct.
  • Multi-tenancy and isolation — the other half of “many tenants, one engine, one database.”
  • Deployment model — how the same PostgreSQL-backed convergence pattern (LISTEN/NOTIFY, version polling) applies to deploy activation across a fleet.
  • Ownership and claims: the design reasoning — the compare-and-set, the re-entrancy invariant, and the owner-suffix conventions.

Execution lanes

Inside one replica the engine executes on N identical actor lanes. Every piece of instance-addressed work is routed onto a lane by a stable hash of the instance id, and one lane drains one request at a time — so all work for a single instance runs in arrival order, on one lane, with nothing else interleaved into it. That property is the contract, and it holds identically at every N.

Lanes are an in-process scale-out story. They sit underneath — and are entirely independent of — the horizontal one: a replica has N lanes; a deployment has M replicas; the two multiply. See Replica semantics for the cross-replica half.

Why lanes exist

A single serial execution lane is a convoy. One slow commit does not merely delay its own instance — it delays every instance behind it on the same lane, however unrelated. Splitting the lane into N lanes removes the convoy without changing what happens within a lane: the ordering guarantees a single lane gave are the guarantees each of the N lanes still gives.

Nothing else about execution changes. Lanes do not partition data, do not shard the database, and do not introduce a placement or affinity model an operator has to reason about. They are a parallelism mechanism inside a process, nothing more.

Routing: a stable hash of the instance id

The routing key is the instance id. It is the unit of durable state (one snapshot row), the unit of mutual exclusion (one ownership claim), and the unit the guarantee names. Nothing else works as a key: a correlation alias is not stable per instance (one instance can carry several alias rows with different values), a deployment or a tenant is far too coarse to spread load, and a channel splits one instance across lanes the moment its relays arrive on a channel other than its spawn’s.

Work arrives in three shapes, and the router handles each differently:

ArrivalRouting
The id is already known — a timer fire carries the instance it belongs toStraight to that instance’s lane. No hop.
The id does not exist yet — a spawn from an inbound message, or a due timer start eventAny lane may mint it; arrivals are spread round-robin. No hop, ever.
The id is learned mid-pipeline — a relayResolved on the arrival lane, then handed off if the owner lane is a different one.
flowchart LR
    T["timer fire<br/>carries its instance id"] -->|"hash(instanceId)"| OWN["the instance's owner lane"]
    S["spawn<br/>no instance id yet"] -->|"round-robin"| ANY["any lane mints it — no hop, ever"]
    R["relay<br/>names a business key only"] --> ARR["the arrival lane<br/>decode · validate · correlate"]
    ARR -->|"resolved id, at most one hop"| OWN

Only a relay can hop, and only once — because the id it routes on does not exist until the arrival lane has decoded and correlated the delivery.

The cross-lane handoff

A relay does not name an instance id on the wire. It names a business key, and the engine only learns the instance after channel resolution, decoding, intake validation, and evaluating the <q:alias> correlation expression. All of that runs on the lane the delivery arrived on, exactly as it always has.

When that resolution names an instance owned by a different lane, the arrival lane does not execute it and does not push it into the other lane’s queue. It answers its caller with a handoff: the already-decoded, already-validated resume request. The caller’s own task — the HTTP handler, the broker consumer, the outbox worker — then enqueues it on the owner lane and waits for that lane’s answer. Two rules make this safe by construction:

  • Lane loops never send into another lane’s queue. Only caller-side tasks do. The mutual-block deadlock (lane A stuck sending into a full queue on B while B is stuck sending into A’s) therefore cannot arise at all.
  • At most one hop. The lane that receives the resolved request runs it where it lands. It re-runs only the race-sensitive part — claim, load, terminal/failed/suspended guards, deployment pin resolution, resume — never the decode, validation, or correlation, which are deterministic over the delivery and already done.
sequenceDiagram
    participant K as Caller task — HTTP handler, consumer, outbox worker
    participant A as Lane A, where the delivery arrived
    participant B as Lane B, the instance's owner
    K->>A: the delivery
    A->>A: decode, intake validation, q:alias correlation
    A-->>K: handoff — the already-decoded resume request
    K->>B: enqueue the resolved request
    B->>B: claim, load, guards, pin resolution, resume
    B-->>K: the answer
    Note over A,B: a lane loop never sends into another lane's queue

The hop travels back out through the caller rather than lane-to-lane, which is what makes the mutual-block deadlock impossible and turns a full queue into backpressure on the transport.

Only relays ever hop. Spawns and timer fires never do.

Continuations before new intake

Since 0.6.5 each lane has two queues. Work that continues an instance already admitted — a local:// delivery handed over by the outbox, a resume routed from another lane — goes on the continuation queue, which the lane serves first; new intake waits on the other. Each queue is first-in, first-out, and per-instance serialization is unchanged: a lane still runs one request at a time. Under a burst, the effect is that the instances already started finish first, and intake meets backpressure, instead of a growing backlog of half-finished work.

Claims are the correctness mechanism; routing is only affinity

This is the property worth internalizing: routing is an optimization, not the thing that keeps execution correct. Correctness comes from the per-instance ownership claim described in Replica semantics — and the claim’s owner identity is lane-scoped, not just process-scoped.

The consequence is that a mis-route is harmless. If work for an instance ever lands on the wrong lane — a bug, a hash change, anything — the claim bounces it exactly as it bounces a competing replica today: a broker relay is requeued, a timer fire is deferred and retried. A mis-route degrades to visible, retry-safe contention. It can never degrade to two lanes interleaving inside one instance.

That is also why the claim-bounce meter below doubles as the mis-route alarm.

The activation flip

A deploy activation rebuilds the engine’s live view of processes, codecs, validators, and channels. Under lanes, the controller sends that rebuild to every lane and waits for all of them before it replaces the live deployment set and rewires transports.

sequenceDiagram
    participant C as The activation controller
    participant L0 as Lane 0
    participant LN as Lane N-1
    C->>L0: rebuild processes, codecs, validators, channels
    C->>LN: rebuild
    L0-->>C: applied between two requests
    LN-->>C: applied between two requests
    Note over C,LN: the controller waits for every lane
    C->>C: replace the live deployment set, rewire transports

The await-all barrier is the only thing lanes add to the flip: the later stages cannot begin until every lane has applied its rebuild.

Per-lane atomicity is what matters, and it is preserved: a lane applies its flip between two requests, never inside one, so nothing is ever observed half-flipped. Because every step of an instance runs on that instance’s one lane, its flip is a single point in its own queue — no instance can straddle two definitions. During the fan-out window two different instances can be served either side of the flip, which is indistinguishable from two deliveries ordered around a single flip point.

Lanes never block on store I/O

Each lane is one asynchronous loop awaiting one request to completion before it dequeues the next. Every persistence call on the execution path is awaited rather than blocked on, so a lane waiting on the database parks on the runtime instead of holding a thread hostage — which is what keeps tail behavior sane when lanes outnumber available pool connections.

The ordering properties are unchanged by this, deliberately: because the loop awaits each request to completion before recv-ing the next, the commit still happens-before the reply and happens-before the next request is dequeued. There is no intra-lane pipelining and no completion re-entry, so there are no re-entry rules to get wrong.

Each step’s sends are its own

While a step runs, the <q:send>s and channel-call requests it makes are collected on its lane. They are committed with the step, in the same transaction, or not at all. A step that fails drops them, and every step starts with nothing collected, so one request’s sends never ride along with the next request on the lane. An engine that finds anything left over at the start of a step discards it and logs an error.

Releases up to and including 0.6.10 broke this in one case. A wait-state process that sent and then failed on its first pass left the send behind. The next request on the lane then committed it as its own: a parking request failed with an HTTP 500 (outbox entry … does not belong to step instance), and a run-to-end request delivered the failed process’s send. See the 0.6.11 release notes.

What changes at N > 1 — say it out loud

Incidental cross-instance serialization disappears. With one lane, two concurrent deliveries to two different instances of the same flow never interleave — as a side effect of there being one lane, not as a promise. At N > 1 they genuinely run in parallel.

This was never the contract, and every documented concurrency mechanism is unaffected:

  • per-channel singleton / serial consumption (a transport-side property — one delivery in flight — which lanes do not touch);
  • per-channel and per-tenant admission caps (see Limits and quotas);
  • optimistic expect="unchanged" writes and pessimistic forUpdate locks on data stores (see Data stores).

But a deployment that has been silently leaning on the single-lane side effect will observe new interleavings. That is precisely why the default is one lane and turning it up is an explicit operator action.

Two other narrowings are worth naming, and both only affect a persistence-less engine (no datasource configured — a dev/test posture): in-process alias uniqueness and in-process inbox dedup become per-lane rather than per-process. Under any pooled production posture both are database-backed and cross-lane safe, exactly as they are already cross-replica safe.

What lanes do not change

Every background role stays exactly one per replica, or one per cluster by lease — none of them becomes lane-aware beyond dispatching into the router:

RoleUnder lanes
Timer poller (lease-gated)Unchanged as a role. Timer fires route by their instance id; schedule fires spread round-robin. Its per-tick fire loop gains bounded concurrency up to the lane count, so a timer burst is not capped at one lane.
Stuck-instance sweep, terminal-retention sweepUnchanged. Both sweep by age and are owner-blind.
Outbox workerUnchanged. Its in-process delivery sink re-enters through the router like any transport.
Deployment watcher and quiescence sweepUnchanged as roles; the activation flip is the fan-out above.
Deferred-ack registry and its timeout sweepUnchanged — one per replica, shared across lanes and across activation flips.
Per-channel singleton consumersUnchanged. Their serialization is transport-side (one delivery in flight), so it survives lanes intact.

Configuring and observing lanes

sutra.engine.shards sets the lane count (default 1); sutra.engine.shard-queue-capacity optionally bounds each lane’s mailbox, in which case a full queue makes the caller wait, so backpressure propagates outward to the transport — never sideways into another lane. Full key detail: Configuration reference.

Meters shipped with the feature, all carrying the lane index as a dimension:

MeterWhat it tells you
sutra.engine.shard.queue-depthPer-lane backlog. Sustained skew across lanes means a hot instance or a hot arrival burst, not an undersized fleet.
sutra.engine.shard.dispatches / .parks / .resumesPer-lane work rates.
sutra.engine.shard.handoffsCross-lane relay hops. Expected and healthy — it rises with lane count by construction.
sutra.engine.shard.claim-bounces (split relay / timer)The mis-route alarm. On a correct rollout it should read near zero outside genuine cross-replica contention.

The live lane count is readable without reading configuration: GET /sutra/health/ready reports it in the loader check’s data.shards, read off the running router rather than echoed back from config — which is what lets a smoke test assert that a container actually came up with the lane count you meant.

Lane death is a health condition, not just a log line. A lane can only die outside the per-dispatch panic containment (a failure in the lane’s own build), and after that every piece of work hashed to it would answer SUTRA.RUNTIME.UNEXPECTED forever while the process otherwise looks healthy. Both probes therefore watch for it: GET /sutra/health/live returns 503 with the dead lane indexes in data.deadLanes — the signal an orchestrator should restart on, because a dead lane’s key space has no other home inside the process — and GET /sutra/health/ready goes DOWN at the same moment so no new traffic routes to the replica while the restart is pending.

Next

Observability

Sutra exports all three signals — traces, metrics, and logs — through OpenTelemetry, and is fail-open about it: telemetry can never affect message processing. No endpoint configured means no exporters and zero overhead; a bad config value falls back to a default with a warning; an exporter failure only logs, it never propagates into intake or dispatch.

No telemetry, no phone-home

Sutra collects nothing on its own behalf. The engine binary, the sutra CLI, and the container image sutra-dist produces send no usage statistics, no error or crash reports, and no telemetry of any kind to any Sutra-affiliated destination — there isn’t one, and no code path sends anywhere by default. Nothing phones home, in any build, ever.

The only way data leaves a running engine to a system outside it is the operator-configured OpenTelemetry export this chapter describes, and export is opt-in, not opt-out (see Configuration reference). With no OTLP endpoint configured, the engine emits structured JSON logs to stdout and nothing else — the boot log says so in plain text (telemetry export off (no OTLP endpoint configured) — JSON stdout logs only), and that posture is enforced in the engine’s own telemetry bootstrap (rust/crates/sutra-engine/src/otel.rs), not just asserted here. Point sutra.telemetry.otlp.endpoint (or the standard OTEL_EXPORTER_OTLP_ENDPOINT) at a collector you run and the signals below start exporting there; leave it unset and nothing crosses the process boundary except whatever you separately configured elsewhere (a channel, a datastore, an audit sink) — this is a property you can verify by reading the source or watching an unconfigured engine’s own boot log, not a policy statement to take on faith.

flowchart LR
    subgraph eng["The engine process"]
        SP["traces<br/>sutra.dispatch · decode · validate · execute"]
        ME["metrics<br/>sutra.instance.* · token.* · task.* · shard.*"]
        LO["logs<br/>structured JSON"]
    end
    LO ==> OUT["stdout — always, no configuration"]
    SP -.-> COL["a collector you run"]
    ME -.-> COL
    LO -.->|"only once an OTLP endpoint is configured"| COL

Three signals, one destination, and it is one you configured: with no OTLP endpoint set there are no exporters at all and stdout is the only thing that crosses the process boundary.

Traces

The engine’s existing tracing spans (sutra.dispatch, sutra.resolve, sutra.decode, sutra.validate, sutra.execute, sutra.outbox.send) export through a tracing-opentelemetry layer with no call-site changes — the same spans that back local RUST_LOG debugging (see Troubleshooting BPMN solutions) are what leaves the process as OTLP.

A suspended instance does not leak an open span. Because a wait state can hold an instance for an arbitrary length of time (see Wait states and human tasks), the executor fires a listener event at suspend that force-ends every span open for that instance — otherwise a long park would show up as a pathologically long trace. Each segment of a stateful flow’s lifecycle (the initial run, then each resume) gets its own trace with its own traceId; there is no trace-of-traces joining them, since OpenTelemetry traces are flat. What ties the segments together for a human reading a Gantt view is the instance id, stamped as a plain span attribute (bpm.instance.id) on every span belonging to that instance — filter on it and the segments line up on one timeline, with the waits showing as the gaps between them.

flowchart LR
    S1["the initial run<br/>its own traceId"] -->|"suspend force-ends<br/>every span still open"| W(["the wait — nothing open"])
    W -->|"resume"| S2["the next segment<br/>a new traceId"]
    S1 -.->|"bpm.instance.id"| K["the attribute that lines<br/>the segments up in a Gantt view"]
    S2 -.->|"bpm.instance.id"| K

A month-long park costs no open span; the instance id, not a parent trace, is what stitches the segments back together.

Metrics

An ExecutionListener (the same lifecycle bus described in Domain neutrality and the SPI model) maps instance/token/task events onto a fixed set of meter names (sutra.instance.*, sutra.token.*, sutra.task.*, sutra.coverage.path_covered — see Coverage: declared routes as the compliance signal for what that one actually tracks), tagged with the deployment id and a configurable label allowlist. Alongside those, the sutra.engine.shard.* family reports per-execution-lane queue depth, work rates, cross-lane handoffs, and claim bounces — see Execution lanes. Delta vs. cumulative temporality follows the standard OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE variable — set it to delta for an Elasticsearch-backed collector, which drops cumulative histograms.

Logs

Structured JSON on stdout, always, with no configuration required — this is the log path every deployment gets whether or not OTLP is configured. When an OTLP endpoint is configured, the same log records additionally export over OTLP. The field shape (timestamp/level/loggerName/ message/service.name, plus traceId/spanId inside a sampled span) is stable and is what any log-processing pipeline should key off. See Logging and audit for the operator-facing configuration.

Cardinality discipline

Tenant id is deliberately not a tag on high-cardinality metrics (per-task duration histograms, for instance) — only on instance-counting metrics, where the cardinality stays bounded by tenant count rather than by tenant × task-name × outcome. This is the same discipline that keeps the label allowlist above short by default.

The reference stack

The repo ships a reference EFK-family stack as a set of OpenTofu modules (deploy/modules/efk-stack) — an OTel Collector, Elasticsearch, Kibana, and a Fluent Bit DaemonSet for host-level logs — deployed alongside the engine module in a dev/small-prod cluster, or pointed at an external, already-owned observability stack in a larger one. It’s a reference, not a requirement: the engine’s only actual contract is the three endpoint inputs (OTLP, an optional log-forward target, an optional direct Elasticsearch endpoint for the audit fan-out) — any OTLP-speaking collector on the other end works.

Next

  • Logging and audit — configuring the endpoints above, and the audit trail as a separate, compliance-oriented sink from telemetry.
  • Troubleshooting BPMN solutions — using traces and audit together to retrace what one message actually did.

Durable execution: snapshots and typed values

This chapter is the why behind what a parked instance actually is. If you only need the observable behaviour, Wait states and human tasks has it; this is the reasoning underneath.

A snapshot at a quiescent point, not an event log

Sutra persists an instance as a snapshot taken at a quiescent point — the moment execution has nothing left to do until the outside world answers. It does not persist a log of events and it does not reconstruct state by replaying one.

The two models differ in what they make cheap and what they make hazardous.

An event-sourced engine rebuilds state by re-running history. That gives it a free audit trail and free time-travel, and it buys them with a permanent constraint: the code that produced the history must keep being able to reproduce it. Every task function, every expression, every library the flow ever touched becomes part of the replay contract, forever, for every instance still alive. Determinism stops being a property of one execution and becomes a property of every version of the engine and the application, jointly, across time.

A snapshot engine stores what the instance is rather than how it got there. The replay contract shrinks to a single step: from this state, given this input, do the next thing. Nothing older than the current snapshot is ever re-executed, so an upgrade cannot retroactively change what already happened.

flowchart TD
    subgraph ev["Event replay"]
        E1["event"] --> E2["event"] --> E3["event"] --> EN["state now,<br/>re-derived every time"]
    end

    subgraph sn["Snapshot"]
        SNAP["state at the last<br/>quiescent point"] -->|"+ this input"| NEXT["do the next thing"]
    end

In the replay shape every arrow is a permanent contract — the code must keep being able to reproduce all of them, for every instance still alive. In the snapshot shape there is exactly one arrow, and it only ever points forward.

The audit trail an event-sourced engine gets for free is then an explicit, separately-configured concern here — the audit journal — and that turns out to be the honest arrangement anyway: an audit record has different retention, different access control, and different redaction requirements from execution state. Conflating them means one policy has to serve both.

The properties that fall out of the choice:

  • A step is one transaction. The snapshot write, the waiting rows, the correlation aliases, and the outbound emissions all commit together, or none of them do. There is no window in which an instance has parked but not registered its correlation, or emitted but not recorded that it did.
  • Encoding is deterministic. Identical logical state produces identical bytes: the container is sorted, and every value has exactly one canonical rendering. Re-encoding a decoded snapshot is byte-identical — which is what makes migration and rollback verifiable rather than hopeful.
  • The frontier is explicit. A snapshot names the nodes it is waiting at, the nodes it has completed, and the start node it was routed through. Resume is “replay the completed set as done, satisfy this wait, continue” — a bounded operation over the recorded frontier, not a re-execution of the past.
flowchart LR
    S["the routed start node"] -.->|"replayed as done"| A["completed node"]
    A -.->|"replayed as done"| B["completed node"]
    B --> W["the wait it parked at"]
    W -->|"satisfied"| N["next node, executed for real"]
    N --> R["…continue"]

Resume re-walks the graph without re-running it: the dashed edges are bookkeeping over the recorded frontier, and the first genuine execution after a wait is the node after it.

The typed value encoding

Instance variables ride inside the snapshot container. How they are encoded is the part with the most design in it.

The defect it fixes

A parked instance used to flatten every variable to its display string. A number became "1250.75". A boolean became "false". A list became "[1, 2]". A null became the empty string. Resume restored all of them as strings.

That is not cosmetic. FEEL — correctly — does not coerce a string in arithmetic or comparison, so an exclusive gateway re-evaluating amount > 100 after a wait compared a string to a number and got null, which a gateway condition reads as false. A transaction that plainly exceeded its limit took the under-limit branch. not(approved) was worse: "false" is a non-empty string, so a restored boolean was never false again. The same wait state that made the engine durable made its decisions wrong.

The loss happened in exactly two places — the park ran every value through the display formatter, and resume wrapped every restored value back into a string — and both are gone.

The encoding

A typed value is <tag>|<payload>: one ASCII tag byte, a separator, then the payload. Only the first separator is structural, so a string whose own text is n|42 round-trips unambiguously as s|n|42.

TagTypePayload
znullempty
bbooleantrue / false
nnumbercanonical decimal text, scale-faithful
sstringthe raw text
ddateISO-8601 date
ttimeFEEL time literal body
idate and timeFEEL date-and-time literal body
udurationISO-8601 duration
jlist / contextJSON

The separator is chosen to be a character the container’s own escaping never touches, so a tagged scalar costs exactly two bytes and the row stays legible to an operator reading it:

sutra.var.amount=n|1250.75
sutra.var.approved=b|false
sutra.var.cancelledAt=z|
sutra.var.inboundId=s|INB-7
sutra.var.lines=j|[1,{"sku":"A-1"}]

Lists and contexts ride JSON. The four temporal types have no JSON counterpart, so they ride a single-key object ({"@d":…} and siblings), and a user context key that would collide — any key starting @ — is escaped by doubling its leading @. The two are never confusable in either direction.

Why the generation integer says 4, not 3

The snapshot format carries a generation integer, and typing takes 4.

Generation 3 was already spent. It is the generation at-rest encryption introduced, where the integer means this snapshot carries ciphertext. Redefining 3 to mean typed would misread every encrypted row already persisted — and distinguishing the two by probing values for something that looks like a tag is exactly the kind of heuristic a persisted format must never depend on.

So the conflation stops at 3. Encryption has always been detected structurally, off the ciphertext key prefix — the read path never consults the generation integer for it at all — which means 4 means one thing only: the value encoding. An encrypted snapshot with typed values is simply 4. There is no generation for “typed and encrypted”, and there never needs to be.

Compatibility: lenient decode, lowest-generation encode

Decode is version detection, not migration. An older snapshot is not tag-decoded at all: every value becomes a string, byte for byte what it was — including a legacy value that happens to look like a tag. An old row read and re-written untouched reproduces its original key/value map (in the binary container since 0.6.7). There is no silent upgrade of values on load.

Encode emits the lowest generation that can carry the state: 4 when at least one variable actually needs the typed form, else 3 when something is encrypted, else 2. An instance whose variables are all strings therefore still writes the exact bytes it wrote before typing existed.

That asymmetry did real work for generations 2 to 4: it kept a golden corpus valid across the feature and let a fleet mid-upgrade write rows an older replica could still read. The binary container of 0.6.7 (below) gives that up on purpose.

The decode path is also deliberately lenient in one further respect: a malformed engine-internal counter reads as zero rather than as an error. A snapshot must never become unloadable. An instance that cannot be read is an instance no operator can inspect, migrate, or terminate — a strictness that turns a small corruption into an unrecoverable one is the wrong trade.

The binary container (format 5, since 0.6.7)

Up to 0.6.6 the key/value map was stored as Properties text: every character of every key and value went through escaping on write and unescaping on read, which made encoding and decoding a visible share of each step’s CPU. Since 0.6.7 the same map is stored as a binary container:

\uSNAPSHOT/5\n                      marker, 13 bytes
entry count                          u32, little-endian
per entry, keys ascending:           u32 key length, key bytes (UTF-8), u32 value length, value bytes

The map inside is unchanged — the same keys, the same tagged values, the same sutra.snapshot generation, the same encryption envelope and key id — so everything above still holds, and the byte-level patchers work on either container. Identical state still produces identical bytes.

  • Reading old rows. A row written by any earlier release starts with a sutra. key, never with the marker, so it is read as text exactly as before. It is rewritten in the binary container the next time its instance commits (a re-park, completion, a FAILED or terminal re-stamp, a migration). Nothing is migrated in bulk.
  • Older engines cannot read new rows. A release before 0.6.7 reads the marker as a malformed Properties escape and refuses the row (“failed to decode persisted instance snapshot: malformed \uXXXX encoding”); the instance is left untouched. So:
    • upgrade the whole fleet at once — stop every older replica, then start 0.6.7 — rather than rolling replica by replica; an older replica still running would refuse every instance a 0.6.7 replica has committed;
    • rolling back to 0.6.6 or earlier works only for instances no 0.6.7 engine has committed; finish or cancel the rest first.
  • Fail-closed. A truncated container, a length that runs past the end, trailing bytes or non-UTF-8 text is a decode error, never a partial instance.

The byte-level key patchers

Three operations rewrite a stored snapshot without decoding it: marking an instance failed, marking it terminal, and re-pinning it during migration. Each patches the raw key/value map in place — engine-internal keys only — and never touches a variable value.

This is deliberate, and the reason is a security one rather than a performance one. Decoding and re-encoding a variable would need the tenant’s data-encryption key. That means:

  • an operation as mundane as “mark this instance dead” would acquire a dependency on key availability, and would fail — or worse, partially succeed — when the key backend is down;
  • it would have to re-derive which variables are sensitive, from a snapshot that at resume time no longer carries that set in the shape the encryptor wanted;
  • and if either went wrong, it would persist a previously-encrypted value in the clear.

Marking an instance dead, finished, or re-pinned must never need the tenant key and must never be able to downgrade an at-rest value to plaintext. Treating every variable as an opaque string is how that is guaranteed structurally rather than promised procedurally.

Typing does not weaken the property, by construction: to the patchers a value is still an opaque string. A snapshot carrying a tag this codec does not recognise rides a patch through untouched rather than being “repaired” — a forward-compatibility posture the tests pin explicitly.

Encryption: the tag rides inside the ciphertext

At the typed generation, the plaintext that gets encrypted is the tagged form. A sensitive number is a number again after the resume, exactly like a non-sensitive one.

The alternative — an outer tag sitting beside the ciphertext — was rejected because it would disclose the type of every encrypted value to anyone who could read the row, for no benefit at all. “This encrypted field is a number, and that one is a list” is information, and it is information the row does not need to carry.

Everything else about the envelope is unchanged. The authenticated-data binding still ties a ciphertext to its key, its instance, and its variable name, and still deliberately excludes the deployment id — which is precisely what keeps “a migration changes only the pin” true, and what lets a migrated instance’s encrypted values still decrypt (see Migration internals). Decode still fails closed on a missing cipher or a failed authentication.

What is deliberately not typed

  • A subject blind-index input. A subject value feeds a keyed hash over index rows that are already persisted. It keeps hashing the exact string it hashed before typing existed; deriving it from the typed value would orphan every index entry written to date — and those rows are how erasure and disclosure find an instance at all.
  • The operator inspect projection. A published contract: every variable still renders through its display form. Typing changed what survives a wait, not what a response looks like.
  • Functions and ranges. A FEEL function closes over an evaluation context that ceases to exist the moment the instance parks; a range is a comparison shape rather than instance state. Both persist as their canonical string — which is what they always did, so nothing regressed. They simply did not become typed.

The one behaviour change a consumer can observe

Every restored value now evaluates as its real type. Where a gateway condition’s answer differs from a pre-typing release, the previous answer was wrong — that is the entire point of the change. The only other visible difference is that a null variable now restores as null (and renders as null rather than blank) instead of restoring as an empty string.

The snapshot key registry

Everything a stored snapshot’s key/value map can carry (in either container), in one place — useful when reading a raw row. sutra.-prefixed keys are engine-internal: opaque strings to the byte-level key patchers and never tag-decoded. sutra.var.<name> entries are the user variables the typed encoding exists for.

KeyWritten byMeaning
sutra.snapshotcodecFormat generation: 2 plaintext-untyped, 3 encrypted-untyped, 4 typed (encryption stays orthogonal — detected off the sutra.enc. prefix, never the version).
sutra.statusexecutor / re-stampsInstance status; terminal and FAILED re-stamps are byte-level patches.
sutra.deploymentIdexecutor / migrationThe content-hash pin; rewritten only by the validated migration operation.
sutra.processIdexecutor / migrationThe owning process definition id; rewritten only by a cross-process re-home.
sutra.waitingNodes / sutra.completedNodes / sutra.startNodeexecutorThe replay frontier; node ids, mapped on migration.
sutra.retry.<nodeId>retry machineryDurable attempt counter for a retry policy; the key is renamed on migration, the counter never resets; absent unless the task has a policy. A malformed counter reads as zero — a snapshot must never become unloadable.
sutra.retryWait.<nodeId>retry machineryThe backoff-window marker on a channel-call task: present exactly while a dead attempt’s backoff timer is pending, carrying the failure’s classification code. Renamed with the node on migration.
sutra.auditSeqauditPer-instance audit sequence floor.
sutra.var.<name>executorA user variable — tagged typed value at generation 4; a plain string in older rows.
sutra.enc.<name>crypto envelopeCiphertext for a sensitive variable; the authenticated data binds key id, instance id, and variable name — and deliberately excludes the deployment id, so migration stays decryptable. At generation 4 the encrypted plaintext is the tagged form.
sutra.keyId / sutra.sensitivecrypto envelopeThe tenant key anchor (plaintext, self-describing) and the sensitive-name set.
sutra.coverage.<pathId>coveragePath-coverage cursors — keyed by declared path id, not node id; migration must not touch them.
sutra.failureCode / sutra.failureDetailFAILED re-stampStructured code + captured message of the fatal failure, for inspection and post-repair migration.

Next

Ownership and claims

Two things must never happen to one instance: two workers advancing it concurrently, and a worker holding it forever. The ownership claim is the mechanism for both. The observable behaviour is in Replica semantics; this is why it is shaped the way it is.

The claim is a compare-and-set, and a failure is a refusal

Every resume path — a correlated relay, a timer fire, an administrative migration — takes the claim before it rehydrates anything, by a conditional update on the instance’s own row: take ownership if it is free or stale, otherwise match nothing.

The critical design choice is what happens when it matches nothing. It refuses. It does not wait, retry in place, or queue behind the holder.

Waiting would be wrong on three counts. It would make the runtime of a request depend on another replica’s work, on that work’s timescale, with no bound anybody can reason about. It would hold a transport resource — an in-flight HTTP request, a broker prefetch slot — for the duration. And it would make a contended path a fundamentally different code path from an uncontended one, which is exactly the kind of asymmetry that hides bugs until the day of the incident.

Refusing instead composes with mechanisms that already exist. A broker relay is requeued and redelivered; a timer fire is deferred to a later tick with backoff; an administrative call answers 409 having read and written nothing. The refusal carries a structured code (SUTRA.RUNTIME.RESUME.CLAIM_HELD, or the admin surface’s SUTRA.ADMIN.MIGRATE.CLAIM_HELD) so a caller can tell “you lost a race, try again” apart from “this will never work”.

flowchart LR
    C{"claim: free or stale?"} -->|"yes"| RUN["rehydrate, run the step"]
    C -->|"no"| REF["refuse — nothing rehydrated"]
    REF --> RQ["broker relay:<br/>requeued"]
    REF --> TM["timer fire:<br/>deferred, with backoff"]
    REF --> AD["admin call: 409,<br/>nothing read or written"]

The claim comes before any rehydration, so a contended path ends at the diamond rather than partway through a step — and every branch out of it is a mechanism that already existed, which is why contention needed no new recovery machinery of its own.

Release is part of the commit, and part of every other exit

The claim is released inside the same transaction that commits the step’s writes. “Committed” and “unowned again” are therefore one atomic fact, and there is no window in which an instance is durably advanced but still shows an owner — a window a sweeper would eventually have to clean up, and that a concurrent resume would misread as contention in the meantime.

Every path that does not commit has to release too: an early refusal, a validation error, an unexpected failure, a panic. Originally that was a scope-guard — release on drop, automatically, whatever the exit. When lane loops became asynchronous, the guard had to change shape, because a destructor cannot await and releasing is a database call.

So the guard became an explicit release on every exit, including the panic path, which is re-raised identically afterwards. The mechanism moved; the invariant did not. It is worth being honest that this trades a compiler-enforced guarantee for a reviewed one — the mitigation is that the exits are few, enumerated, and directly tested, and that the sweep below is the backstop for the case where the process is not around to run any exit at all.

Re-entrancy, and the invariant that makes it safe

The compare-and-set deliberately grants a re-claim to the same owner. That looks like a hole until you see the invariant it rests on.

Originally the reasoning was: an owner id names one process, and one process advances instances on a single execution thread — so the same owner cannot possibly be in two places at once. Re-claiming what you already hold is a heartbeat refresh, not contention.

Introducing N execution lanes inside one process invalidated that reasoning exactly. Under a per-process owner id, lane B claiming an instance that lane A is mid-step on would succeed re-entrantly: a silent double-resume, which is precisely the corruption the claim exists to prevent. All-green tests at one lane would have proved nothing, and the cross-replica tests use different owners, so they could never have caught it.

The fix is to make the owner id lane-scoped, which rotates the invariant into its honest form:

same owner ⇒ same lane ⇒ already serialised

The words changed and the guarantee did not. That is the shape to want from an invariant when the system underneath it moves.

Routing is affinity; claims are correctness

This is the property the whole lane design leans on, and it is worth stating as its own claim.

Instance-id hash routing exists to give a lane affinity for the instances it owns — fewer cross-lane hops, better cache behaviour, less queue churn. It is not what makes execution correct.

If work for instance X ever lands on the wrong lane — a routing bug, a hash change, a handoff rule misapplied — the claim bounces it exactly as it bounces a competing replica: CLAIM_HELD, requeue for a relay, defer for a timer fire. Mis-routing degrades to visible, retry-safe contention. It can never degrade to interleaved execution.

That is a much better failure mode than the alternative, where routing is the mutual exclusion and a routing bug is a silent data-corruption bug. It also gives the rollout a direct observable: the claim-bounce meter, split by relay and timer, should read near zero at a healthy multi-lane rollout, so a mis-route shows up as a number going up rather than as a support ticket six weeks later.

Owner-id suffixes, and why they cannot collide

An owner id is an opaque string to the store and to the sweeper — neither parses it — which is what lets the conventions layer without a schema change:

SuffixWho uses itWhy it is distinct
Lane indexEvery execution laneSo same-owner re-entry means same-lane re-entry (above).
A migration markerThe administrative migrate operationSo a migration cannot re-enter past a resume this same replica has in flight.

The migration case is the subtle one, and it is the direct consequence of re-entrancy. Claiming under the bare replica identity would succeed against a resume in flight on that very replica — the one race the claim exists to prevent, reintroduced by an operator action. Under a distinct owner the compare-and-set fails honestly, and a resume that starts after the migration claims bounces off it in turn.

The two conventions cannot collide because they occupy different, non-overlapping parts of the identity: a lane suffix is a lane index on the execution path, and the migration marker is not a lane index at all and is never produced by a lane. There is no string a lane can generate that an administrative claim can also generate.

Heartbeats: deliberately not wired

A claim could refresh itself mid-step. It doesn’t, and that is a decision rather than an omission.

A step runs between quiescent points — milliseconds to low seconds — which is orders of magnitude inside the default claim timeout. Adding more execution lanes shortens the queue wait before a step; it does not lengthen the step itself. So the mechanism would fire constantly and protect against nothing.

The condition that would change the answer is a genuinely long-running step. The pull-worker surface is the shape that could produce one — but it deliberately does not: a task parked for a worker is not a step in progress. The instance parks, its claim is released with the commit, and the worker’s completion re-enters as a fresh delivery later. The long wait lives in a task row’s lock, not in an instance claim, which is exactly why no heartbeat is needed. See The pull surface.

The sweep is the only backstop that can exist

No release protocol survives a process that stops existing. The stuck-instance sweep is the answer: a lease-gated role that clears any claim whose owner has been silent longer than the claim timeout.

stateDiagram-v2
    [*] --> Unowned
    Unowned --> Held: compare-and-set, free or stale
    Held --> Held: same owner, so same lane — re-claim granted
    Held --> Unowned: released inside the committing transaction
    Held --> Unowned: released on every non-committing exit
    Held --> Unowned: swept, owner silent past the claim timeout

Three ways out of Held, in decreasing order of how much the design leans on them: the commit itself, the enumerated non-committing exits, and the sweep — which exists only for the process that is no longer around to run any exit at all.

It sweeps by age and is owner-blind — it does not parse owner ids, does not know how many lanes a replica has, and does not care. That is what let the lane-index suffix land with no change to the sweep at all: owner cardinality grew, and the sweep predicate never mentioned owners in the first place.

The tuning trade is plain in the two keys. A short claim timeout recovers faster from a dead replica and risks stealing an instance from a live-but-slow one — which the claim’s own compare-and-set then makes safe rather than catastrophic, since the original owner’s commit no longer matches. A long one is the conservative direction. The defaults sit far enough above normal step duration that the first case does not arise in practice.

Next

Execution lanes: the design

Execution lanes describes what lanes do. This chapter is the reasoning: what the single-lane design was actually buying, how each of those properties was preserved rather than re-argued, the one genuine cross-lane race that had to be fixed, and the runtime shape that had to be falsified before the right one was found.

What the single lane was buying

Before lanes, the engine was one actor thread draining one queue, with every store call blocked on rather than awaited. That is easy to dismiss as an accident of an early implementation. It wasn’t — four properties rode on it, and each had to be accounted for.

Because a dispatch was synchronous end to end, the step’s commit happened-before the reply to the caller, and happened-before the next request was dequeued. From that:

  1. Reply implies committed. A transport that got an answer knows the park is durable. This is what makes an acknowledgement mode meaningful at all.
  2. At most one store transaction in flight per process. Connection-pool behaviour was trivially bounded.
  3. A park can never race its own completion. The deferred-acknowledgement registration that follows a commit was serialised against every terminal event, because nothing else could run in between.
  4. The activation flip is atomic with respect to dispatches. Nothing is ever observed half-flipped.

The honest observation — the one that set the whole sequencing — is that blocking on a dedicated thread is not itself the scaling defect. The defect is having one serial lane, where a single slow commit convoys every unrelated instance behind it. N lanes fix the convoy while changing nothing else. Removing the blocking is a separate change that buys different things and should be judged on its own.

So the work split: first N lanes with the blocking intact, then the conversion to awaiting, with the ordering properties preserved verbatim at each step. Each half is reviewable on its own terms, and a regression can be attributed to one of them rather than to “the concurrency change”.

The total-serialization audit

The interesting engineering was not writing a router. It was enumerating everything that implicitly relied on there being one thread — because every such reliance was invisible, uncommented, and correct right up until it wasn’t.

The method was to walk each site and classify it into exactly one of four buckets:

VerdictMeaning
SafeIt relied on one step running wholly in one place, not on one thread existing. A step still does.
Safe with a changePer-process scratch becomes per-lane scratch; a generated id gets the lane index mixed in so it stays unique.
Accepted narrowingA per-process property becomes per-lane. Sound only where the production posture does not rely on it.
Genuine raceIt relied on total order across different instances. Must be fixed.

Most sites landed safe, and for one recurring reason: per-instance state has a durable home, and a step runs wholly on one lane. In-memory audit sequence counters, for instance, are re-seeded from the snapshot before every resume, so a stale entry on a lane that last saw an instance is simply overwritten wherever the next resume lands.

The accepted narrowings were the ones worth writing down honestly, because they are real degradations rather than non-issues. In-process correlation-alias uniqueness and in-process inbox deduplication become per-lane instead of per-process. Both are only reachable in a persistence-less posture (a dev or test engine with no datasource). Under any pooled production posture both are database-backed — a unique index and a row lock — and already cross-replica safe, which strictly implies cross-lane safe. Documenting the narrowing is the price of not pretending it doesn’t exist.

One further site is a deliberate removal rather than a narrowing: incidental cross-instance serialization. Two deliveries to two different instances of one flow never interleaved, as a side effect of there being one lane. That was never a contract, but a deployment could have been leaning on it. It is the reason the default lane count is one, and the reason the configuration page says the sentence out loud rather than burying it.

The one genuine race, and its inversion

Exactly one site was a real cross-lane race, and it is worth walking through because the fix is smaller and stranger than the problem.

The deferred-acknowledgement registry lets a transport hold an inbound acknowledgement until the instance actually completes. The original order was: commit the park, then register the settle callback. Safe under one lane, because nothing could run between the two.

Under lanes it breaks. The park runs on the lane the delivery arrived on — call it A. The instance’s first relay routes to the instance’s owner lane — call it B. B can claim, resume, and complete the instance in the window between A’s commit and A’s registration. The terminal event then fires with no registration present, the acknowledgement never fires, and the delivery dangles until the registry’s timeout sweep negatively acknowledges it. Microseconds wide, invisible under load, catastrophic when it hits.

The fix is to invert the order: register before committing, and deregister if the commit fails.

The window closes cleanly, and the argument is the load-bearing part:

Before the park commits, no correlation alias row exists — alias rows ride the step transaction. So no relay anywhere can correlate to this instance, and therefore no terminal event for it can fire from anywhere except this same dispatch, which is the one blocked on the commit.

A failed commit deregisters, and the caller sees the same error it always did. The registry’s documented invariant rotates from “registered ⇒ the transport was told to defer” to “the transport was told to defer ⇒ registered ∧ committed” — which is the direction the transport actually depends on, and arguably what it should have said all along.

At one lane the inversion is unobservable, which is why it landed unconditionally rather than as a behind-a-flag branch. A correctness fix that only applies in one configuration is a correctness fix you have to reason about twice.

The lesson generalizes: the ordering was safe because of a property (total order) that was never written down as a requirement. Auditing for implicit reliance is the only way those surface before production does it for you.

Handoff rules

A relay does not carry an instance id. Resolving one needs the decode, the intake validation, and the correlation expression — all of which need the lane-resident registries. So resolution happens where the delivery arrived, and the resolved request is then handed to the owner lane. Two rules constrain it, and both are structural rather than advisory.

Lane loops never send into another lane’s queue. Only caller-side tasks do. This is what removes the inter-lane deadlock case by construction: if lane loops could enqueue into each other, lane A blocked sending into B’s full queue while B is blocked sending into A’s is a genuine deadlock, and no amount of capacity tuning eliminates it — it just makes it rarer and therefore worse. Routing the hop back out through the caller’s own task means a full queue applies backpressure to a transport, where backpressure belongs.

At most one hop. The lane that receives a resolved request runs it where it lands. It cannot need a second hop — the instance id is fixed, so re-resolution would name the same lane — and more importantly, correctness must not depend on the hop landing correctly. It doesn’t: a wrong landing bounces on the claim. The hop is affinity; the claim is correctness. See Ownership and claims.

The receiving lane re-runs only the race-sensitive part — claim, load, guards, pin resolution, resume. It does not repeat decode, validation, or correlation, which are deterministic over the delivery and already done. Redoing them would be wasted work and, worse, a second place where those semantics could drift.

The runtime shape, including the one that was wrong

Three shapes were weighed for removing the blocking store calls.

(a) One asynchronous loop per lane, awaited to completion per request. Every persistence call becomes awaited; the loop is “receive a request, await it fully, receive the next”. Because the loop awaits each request to completion before receiving the next, every ordering property above is preserved verbatim per lane — commit still happens-before reply and before the next dequeue. No completion re-entry exists, so there are no re-entry rules to get wrong.

(b) Blocking loop plus I/O offload with completion re-entry. Split a step at the commit, hand the write to a worker, return to the loop, and re-enter a completion event later. This is the only option that adds intra-lane pipelining — and it was rejected. It breaks reply-implies-committed unless responders are parked on the completion; it needs per-key in-process mailbox holds, which is a second serialization mechanism layered on top of claims; and its failure modes (a commit landing after a crash-restart re-queued the work, a lost completion, a mailbox stuck held) are precisely the subtle-ordering-bug class the design exists to avoid. Two mechanisms enforcing one invariant is how invariants get broken.

(c) N blocking lanes. Zero semantic change, and N× I/O concurrency arrives from N threads.

The order shipped was (c), then (a); (b) was rejected outright. (c) makes the convoy fix reviewable and bisectable on its own. (a) then removes the blocking as a mechanical conversion whose diff is wide but whose semantics are provably unchanged. What (a) buys concretely: lanes park on the runtime instead of holding threads when the pool is the bottleneck, an entire class of “blocked inside an async runtime” panic disappears, and the door stays open to pipelining later without another trait migration.

The shape that was falsified

The first cut of (a) gave each lane its own runtime with its own I/O reactor. It looked like the clean version: a lane is fully self-contained, owning its execution and its I/O.

Its own acceptance gate — the “one lane must be identical to before” bar — falsified it. When a lane shut down, its reactor died with it, and pooled database connections registered on that reactor were stranded: sockets nobody would ever poll again. The visible symptom was a restart hang. The successor engine’s lease acquisition waited out the full lease TTL before it could proceed, roughly half the time. A restart-timing flake, not a wrong answer — the kind of thing that gets triaged as flaky infrastructure for months.

The fix inverted the ownership: lanes own no reactor at all. Each lane’s loop is driven on its own dedicated thread via the process-wide runtime’s handle, so the reactor topology is identical to what it was before the conversion, and lane lifetime is decoupled from I/O registration lifetime. Restart handover got materially faster as a side effect, because nothing waits out a lease TTL any more.

Two things are worth taking from this. First, the “obviously clean” decomposition was wrong because it decomposed the wrong resource: execution is per-lane, but I/O registration is per-process, and conflating a lifetime with a locality is a recurring shape of bug. Second, the identity bar is what caught it. An acceptance criterion of “one lane behaves exactly as before” is a much stronger instrument than a suite of feature tests, because it fails on things nobody thought to assert.

The activation flip

Activation rebuilds the engine’s live view of processes, codecs, validators, and channels. Under one lane that rebuild was a single request, atomic against every dispatch.

Under lanes the controller sends the rebuild to every lane and awaits all of them before replacing the live deployment set and rewiring transports. Per-lane atomicity is preserved; that is all that is needed, and the argument for why is worth being explicit about.

The flip never promised cross-component simultaneity in the first place. Even under one lane the later stages — replacing the deployment set, reconciling schedules, rewiring transports — were already non-atomic with respect to the actor swap; deliveries kept flowing between them. What was promised, and is still promised, is per-dispatch consistency: nothing is ever observed half-flipped.

Per-lane atomicity delivers exactly that, because an instance’s steps all run on its one lane, whose flip is a single point in its own queue. No instance can straddle two definitions. During the fan-out window two different instances can be served either side of the flip — which is indistinguishable from two deliveries ordered around a single flip point, which is what happened before. The draining-deployment correlation tail is part of the rebuilt view and flips with it, per lane.

Where false confidence would have come from

The risks that mattered were not the ones the code made obvious. They were the ones where a green test suite would have been actively misleading:

RiskWhy a green suite would have liedWhat actually catches it
Silent double-resume via re-entrant claimOne-lane suites cannot exercise it; cross-replica tests use different ownersLane-scoped owner makes the window structurally impossible; a deliberate mis-route test proves the bounce
The park/completion acknowledgement raceThe window is microseconds — stock tests pass almost alwaysThe ordering inversion, plus a race test with an injected pause exactly in the window
Flip skew across lanesSingle-replica flip tests never produce skewAwait-all barrier, plus a flip-under-load test at several lanes
Ordering regressions from extracting the router“It is just plumbing” reviewsA byte-identical outcome-sequence bar at one lane
Hot-instance skew pinning a laneAveraged throughput hides it entirelyPer-lane queue-depth is a shipped meter, not a debugging afterthought
Pool exhaustion at N concurrent commitsTesting against a generous poolA soak against a deliberately small pool
The per-lane narrowings surprising a persistence-less deploymentThe tests exercise the pooled pathDocumented, and unchanged under any pooled posture

The pattern in that table is the transferable part: for each risk, name the specific green result that would have been misleading, then design the test that would not have been. An all-green-at-one-lane suite proves nothing whatsoever about cross-lane windows.

Next

Retry machinery

Retries, history, and schedules covers what <q:retry> does. This chapter is why it is built the way it is — and the channel-call half turned out to have a genuinely hard problem in it.

A backoff is a park, not a sleep

The first decision constrains everything else: a retry wait is a durable timer park.

The alternative — sleep in place and try again — is not merely inelegant here, it is unavailable. A lane executes one request at a time, so an in-path delay does not delay one instance; it freezes every instance queued behind it on that lane. A ten-second backoff would become a ten-second stall for unrelated work. Even with lanes, a sleeping lane is a lane doing nothing while holding a queue.

So a failed attempt with budget remaining re-parks the instance: the failed task stays pending, a timer row is armed at the backoff instant, and the ordinary timer poller re-drives it when the instant arrives. Deliberately the same seam that respond-and-continue already uses — not a parallel mechanism. A second scheduling path would be a second set of races to reason about, and the existing one already survives restarts, hot-deploys, and replica death.

Everything good about the behaviour follows from that one choice. The backoff survives a restart. A PT5M wait is PT5M whether or not the replica that started it is still alive. No execution capacity is consumed while waiting. And a retry park is pinned to its deployment exactly like every other park, so a hot-deploy cannot silently move a backing-off task onto a different definition.

The cost is honest and stated: a <q:retry> task makes its process stateful. The park needs persistence. A process that was otherwise run-to-completion now requires a datasource, and the structural classifier accounts for it rather than discovering it at runtime.

Attempt state lives on the snapshot

Attempt counters are snapshot keys, one per node, not a column on the waiting row. Three reasons, in increasing order of importance.

A column would be erased by the very step that increments it. The retry park resolves its timer row and creates a new one each attempt. Attempt state stored there would be destroyed by the re-park — the mechanism would delete its own bookkeeping.

Attempt state is instance state. It must ride the failure re-stamp untouched when the instance eventually fails, so that an operator inspecting a dead instance can see how many attempts it burned. Putting it on the snapshot gets that for free, because the failure re-stamp is a key patch over the snapshot’s map.

It costs zero migration and stays byte-identical when unused. The snapshot container is an open-keyed map, so a new key family needs no schema change anywhere. And because the key is absent unless a task actually has a policy, a process with no <q:retry> writes byte-identical snapshots to one that predates the feature — which keeps a byte-for-byte golden corpus valid across the whole feature. A feature that changes the persisted bytes of flows that don’t use it is a feature that cannot be verified cheaply. See Durable execution for the encoding and the patchers.

A malformed counter reads as zero rather than failing the load, for the same reason the rest of the decode path is lenient: a snapshot must never become unloadable.

The channel-call problem

Extending <q:retry> to channel-call tasks was initially skipped, on the theory that the outbox’s own retry curve plus the timeout boundary already covered them. A survey of what the code actually did disproved it:

  • the outbox curve retries one delivery, by default forever — it is about reaching a counterpart, not about the task having another go;
  • a timeout without a policy simply kills the instance.

So there was a real gap: no way to say “if the counterpart doesn’t answer, ask again”. Closing it meant answering a question the registered-task case never has to: what, exactly, is a failed attempt of a call?

The honest failure set

Derived from what the dispatcher and executor actually do, and nothing else:

  1. The route-less <q:timeout> boundary firing. With a policy present the timeout is a retryable task failure first. Without one, the pre-existing catchable-error behaviour is preserved byte for byte.
  2. A terminally-poisoned request delivery. The outbox exhausted its configured attempt ceiling while the task waited. Before this existed, the instance simply stayed parked forever — the delivery had given up and nothing told the task.

Both are stable structured codes, which is what makes them expressible in nonRetryableCodes. (“Never retry timeouts” is a reasonable policy, and it has to be sayable.)

Not failures, deliberately:

  • A correlated business response. The counterpart answered. What the answer says — approved, declined, rejected — is the process’s business to route on with a gateway. Retrying because a business answer was unwelcome would double-submit, and re-issuing an instruction because the first answer was “declined” is exactly the class of bug an engine must make structurally impossible.
  • BPMN errors. They route to their boundaries, unchanged, on either task kind.

Modelled outcomes beat policies — enforced at load

A timer boundary with drawn outgoing flows is a modelled outcome: the author has said what happens. So it wins, exactly as a BPMN error routes to its boundary rather than consuming a retry budget.

Which means a channel-call <q:retry> alongside a routed timer boundary is a policy that could never fire. Rather than let that load as a silent near-no-op, the loader refuses the combination. A configuration that cannot do anything should not be accepted quietly; the author believes they have retries and does not.

The hard part: a dead attempt and an in-flight one look identical

Here is the problem that shaped the rest of the design.

Consider a channel-call node in a backoff window — its previous request failed, it is waiting to re-issue — and a channel-call node whose request is in flight right now. Through the durable facts available (the wait frontier plus the attempt counter) they look exactly the same: a node waiting, with attempts already burned.

They demand opposite treatment. A late response to the dead attempt must be refused; a response to the live attempt must resume the flow. A due timer on the first is the re-drive; a due timer on the second is stale residue.

The registered-task shortcut does not work here. For a registered task, “attempts burned and not completed” implies a backoff window. For a call node it does not, because a counterpart’s relay can name the node mid-retry.

The backoff-window marker

The resolution is one more durable key: a marker set while a node is in a backoff window, carrying the classification of the failure that parked it. It is carried exactly like the attempt counters — an open-keyed snapshot entry, no migration, absent by default so unaffected snapshots stay byte-identical, and renamed along with the node mapping when an instance is migrated.

It keys four decisions, each made under the instance claim from durable facts alone:

SituationVerdict
A due timer on a marked call nodeThe backoff re-drive. Explicit, because the registered-task inference is unsound here.
A due timer on an unmarked call nodeStale residue. Resolved, never driven.
A timeout boundary firing on a marked hostStale — a poisoned delivery beat it to the failure. Firing would double-count one attempt.
A response correlating to a marked nodeRefused (SUTRA.DISPATCH.CHANNEL_CALL.RETRY_PENDING) — it belongs to the dead attempt.

That last row has a deliberate sub-decision: the correlation row stays live rather than being retired. A counterpart answering a superseded request gets an honest “that attempt is gone, a retry is pending” verdict instead of a “no such instance” miss — the same posture durable FAILED takes, where the truth is more useful than a 404. And when the re-drive re-emits, the same correlation serves the new attempt’s response normally, with no re-registration.

The re-emission contract

A re-drive is not “wait for the same answer again”. It re-runs the park’s side effects from durable state:

  • a fresh request emission, built over the same persisted variables, with a fresh idempotency key on the wire;
  • a fresh timeout window;
  • the response wait re-incarnated — a node both resolved and re-parked in one step is a new incarnation, and is recorded as such rather than as an update of the dead one.

The fresh idempotency key is the subtle one and it is deliberate. The retry is a genuinely new request, and a counterpart doing its own deduplication must see it as new. Reusing the key would invite the counterpart to answer “already handled” for an attempt it never actually completed — turning the retry into a silent no-op precisely when it matters.

Withdrawing the dead attempt’s deliveries

The backoff park deletes the dead attempt’s outbox rows — pending and poisoned alike — in the same transaction that arms the backoff.

This is a deliberate, narrow exception to an otherwise absolute rule: the outbox never deletes an undelivered row. The rule exists because a deleted undelivered row is lost work with no trace, and it is a good rule. Two concrete races justify the exception:

  • A superseded request delivered late would race the re-drive’s fresh emission into a double-submit — two live requests for one logical call.
  • A superseded request poisoning later would fire a failure against the live attempt, consuming a budget slot that attempt never spent.

Both are worse than the loss the rule protects against — and the loss does not actually occur here, because the durable record of the failure survives elsewhere: in the marker, in the attempt counter, and in the incident the poison already recorded. Nothing is forgotten; only the pending delivery is withdrawn.

sequenceDiagram
    participant I as Instance
    participant D as Durable state
    participant C as Counterpart

    I->>C: request, attempt n, idempotency key A
    I->>I: the timeout boundary fires, or the delivery is terminally poisoned
    I->>D: arm the backoff timer, set the marker, delete attempt n's outbox rows
    Note over I,D: one transaction — nothing of the dead attempt stays pending
    C->>I: a late response to attempt n
    I-->>C: refused, SUTRA.DISPATCH.CHANNEL_CALL.RETRY_PENDING — the correlation row stays live
    D->>I: a due timer on the marked node — the re-drive
    I->>C: fresh request, attempt n+1, fresh idempotency key B
    C-->>I: response, served by the same correlation, no re-registration

Withdrawing the dead attempt’s deliveries is what makes the fresh idempotency key safe: only one request for this call is ever live, so a counterpart doing its own deduplication sees a genuinely new one rather than a replay it can answer “already handled”.

The general shape is worth naming: an exception to a safety rule is defensible when you can point at the specific corruption the rule would cause, and show the information the rule was protecting is preserved by another mechanism.

The poison wake is a prompt, never a fact

When the outbox gives up on a request, the task waiting on it needs to hear about it. The dispatcher fires a best-effort in-process wake — a fresh serialized turn, like a timer fire.

But the wake is treated as a prompt to go and look, never as evidence. The engine acts only on durable evidence re-read under the instance claim: a poisoned outbox row for exactly this instance and this node. A wake that arrives spuriously finds nothing and does nothing.

And a wake that is lost — a crash, a shutdown, a dropped in-process message — is recovered by the timeout boundary, which the loader guarantees every channel call has. That guarantee is what lets the wake be best-effort at all: the fast path is a notification, the correctness path is the boundary the model was required to declare anyway.

This is a shape worth reusing. An in-process notification that races with a crash is fine as long as (a) the receiver re-derives the fact from durable state, and (b) something durable eventually detects the same condition without the notification.

The whole thing is claim-arbitrated

Every decision above — relay versus timer versus poison, marked versus unmarked, re-drive versus resolve — is made under the instance claim, from durable facts. The races between a late relay, a due timer, and a poison notification arbitrate through the claim exactly as cross-replica races do; the marker only tells a holder which situation it is in, never who gets to act. See Ownership and claims.

The determinism discipline is unchanged throughout: no lane-blocking sleep anywhere, attempt state on the snapshot, every re-drive decision from durable facts, and byte-identical snapshots for every process that never backs off a call.

Exhaustion

A spent budget — or a nonRetryableCodes hit — lands on the same fatal path a registered task’s exhaustion does: a durable FAILED snapshot carrying the structured code. Identical for both task kinds, deliberately, because “how many kinds of failure state does this engine have” should have exactly one answer.

From there the instance is retained, inspectable, blocks its deployment’s retirement, and is repairable by migration — which is the loop the whole design is pointed at: fail loudly, keep everything, let an operator fix the model and bring it back.

stateDiagram-v2
    Attempt: Attempt<br/>the task runs, or the request is emitted
    Backoff: Backoff park<br/>timer armed, marker set,<br/>the dead attempt's outbox rows deleted
    Failed: FAILED<br/>a durable snapshot with the code
    [*] --> Attempt
    Attempt --> [*]: success, or a<br/>business response
    Attempt --> [*]: a BPMN error,<br/>routed to its boundary
    Attempt --> Backoff: failed,<br/>budget left
    Backoff --> Attempt: due timer<br/>on the marked node
    Attempt --> Failed: nonRetryableCodes
    Attempt --> Failed: budget spent
    Failed --> [*]: retained, inspectable,<br/>repairable by migration

Only two things are a failed attempt of a channel call — a route-less <q:timeout> firing and a terminally-poisoned delivery. An answer and a BPMN error both leave without touching the budget, because each is an outcome the author already routed; retrying an unwelcome answer would double-submit.

Next

Migration internals

Instance migration is the operator’s guide. This chapter is the reasoning: a row-security problem with a silent failure mode, why compatibility is derived from resume behaviour rather than from BPMN taxonomy, how the re-arm set is worked out, and why batch independence is structural rather than promised.

The two-scope problem

Every other commit in the engine runs inside one deployment scope: open a transaction, set the scope, write, commit. Row-level security policies then confine every statement to that scope, which is how tenant and deployment isolation is enforced at the database rather than in application code.

Migration breaks the pattern by definition. It reads rows pinned to the source and writes rows pinned to the target.

The obvious implementation — a single UPDATE that sets the deployment id — cannot work under an enforcing policy. The shipped policies are visibility predicates with no separate write predicate, and the database then reuses the visibility predicate as the write check for an update. So the statement fails at both possible scopes, and the two failures are not equally bad:

Scope set toWhat happensSeverity
SourceThe old row is visible and passes, then the re-scoped new row is rejected by the implied write check.An outright error. Loud, obvious, safe.
TargetThe old row is not even visible, so the statement succeeds and matches nothing.Silent.

The second is the dangerous one, and it is worth dwelling on why. It does not raise. It does not warn. It returns success having moved zero rows — so an operator running a migration would see it succeed, and the instance would still be pinned to the broken model. Every downstream signal would agree that everything was fine.

That is the failure mode this design exists to make impossible, which is why both halves are pinned by a test running as a genuinely non-bypassing database role. The justification for the shape is executable rather than asserted — a claim about what row security does under an enforcing role is exactly the kind of claim that rots quietly.

The scope flip inside one commit envelope

What makes the operation possible is that the scope setting is transaction-local, not transaction-immutable. One transaction may re-scope itself between statements.

So the commit runs two phases inside one envelope:

  1. Scoped to the source — lock the instance row, re-assert the ownership claim and the non-terminal status, read every row belonging to the instance, delete them.
  2. Scoped to the target — insert the rewritten rows.

Atomicity is unchanged: a phase-2 failure rolls phase 1 back, and no session ever observes the instance under both pins or under neither. Isolation is not weakened either, and the distinction is precise — the transaction is never scoped to two deployments at once. It finishes with one, then scopes to the other. There is no window in which a statement could see both.

The re-assertion in phase 1 is deliberate. The claim was taken before validation; re-verifying it under the row lock, inside the commit, closes the gap between “claimed” and “moved”.

flowchart TD
    CLM["claim the instance"] --> VAL["validate every locus<br/>all violations, not the first"]

    subgraph TX["one transaction"]
        direction TB
        LK["scoped to the SOURCE:<br/>lock the instance row"] --> RA["re-assert the claim and<br/>non-terminal status"]
        RA --> RD["read the instance's rows,<br/>then delete them"]
        RD --> INS["re-scoped to the TARGET:<br/>insert the rewritten rows"]
    end

    VAL --> LK
    INS --> OK["commit"]
    RA -.->|"claim gone,<br/>or terminal"| NO["refused —<br/>nothing moved"]
    INS -.->|"phase 2 fails"| RB["phase 1 rolls back:<br/>never both pins, never neither"]

The scope is re-set between statements rather than held for the transaction, which is what lets one commit read under the source pin and write under the target one — and the re-assertion sits inside that envelope precisely so a claim that died between validation and commit stops the move instead of racing it.

The snapshot moves as a byte-level patch

The snapshot rewrite is a key patch over the raw map — the same shape marking an instance failed or terminal uses, and for the same reasons, plus one specific to migration.

A decode-and-re-encode would need the tenant’s data-encryption key, would have to re-derive an encryption set the resume-time snapshot no longer carries in that shape, and would persist a previously-encrypted value in the clear the moment either went wrong. See Durable execution.

That carried-through ciphertext still decrypts under the new pin is not luck. The authenticated-data binding ties a ciphertext to its key, its instance and its variable name, and deliberately excludes the deployment id — a property the encryption design chose precisely so that “a version migration changes only the pin” would stay true. A design decision made in one subsystem paying off in another, years later, is what a well-chosen invariant looks like.

Rewritten: the deployment pin; the wait frontier, completed set and routed start (each entry mapped through the node mapping); the per-node retry attempt counters — the key is renamed, the counter untouched, because a burned budget follows its task; the audit sequence floor, bumped past the migration event.

Untouched: variables plain and encrypted, the key anchor and sensitive-name set, the status, the failure keys, and coverage cursors — those last are keyed by declared path id, not node id, so applying the node mapping to them would corrupt them. The distinction is invisible unless you look, which is why it is written down.

Compatibility is derived from what resume does

The locus model is the substance of validation, and its one real idea is this: the question is never “is this a userTask”. It is “can a parked message wait resume here”.

Several distinct BPMN constructs answer yes to the second question, and the set is not the same as any single element type. So each locus is classified by the resume behaviour it will actually get, and the target node is checked against capability flags rather than an element enum.

Two classifications carry the weight:

A retry park is not merely a timer park. It is a timer row whose node has a durable attempt count and is absent from the completed set — which is exactly the condition the executor itself uses to route a due timer to re-run the task rather than to fire the timer. So the compatibility rule reuses the executor’s own test rather than reinventing one. Landing a retry park on a merely timer-capable node would silently re-run nothing.

A continue-reply park is read from the source graph. Whether a parked node carries a respond-and-continue reply is a property of the definition the instance is running under, not of the target. Which is why a source deployment whose plan is no longer registered is a hard refusal rather than a fallback: validating a continue-reply park against the timer rule would pass the wrong check and produce a confidently-wrong verdict. Refusing to answer beats answering wrong.

The capability index each plan projects is small and computed at activation, so validation never re-verifies and re-plans two sealed archives on an administrative request path. It covers draining deployments as well as active ones — a migration’s source is by definition a deployment that has been flipped away from. Sub-process bodies are flattened into their parent’s index, because durable state records a node id with no scope path.

Every violation, not the first

The validator reports all violations. Fixing a mapping should take one round trip, not one per node — a validator that stops at the first error turns a ten-node rename into ten deploy-test cycles.

The same instinct drives the refusals that could have been silent: a mapping entry naming a node the instance neither parks at nor has completed is refused, because a typo must never read as “identity mapping, then”.

The resume re-arm set: why the frontier alone is wrong

resume: true brings a FAILED instance back. Working out what to re-arm is the subtlest part of the whole operation.

The failure commit did two things: key-patched the snapshot to FAILED with its failure code and detail, and resolved every waiting row in one statement so no timer refires and no relay finds a live wait. The frontier itself was left untouched.

Resume is the inverse of both halves. The snapshot half is straightforward — status back to suspended, failure keys dropped, output byte-identical to the pre-failure snapshot because the failure keys are emit-only-when-present, and like every other re-stamp it is a raw patch, so reviving an instance never needs the tenant key and can never downgrade its at-rest protection. It fails closed on any status but FAILED.

The rows are the hard half, and the frontier is not enough. A <q:timeout> synthesizes a boundary with a derived id, and a timer boundary event has its own id — neither appears in the frontier. Re-arming by frontier alone would silently drop the timeout the park was armed with, producing an instance that resumes and then waits forever with no deadline.

The rows are recoverable exactly, though, and the derivation is a nice piece of reasoning about what the data already tells you:

They were resolved by one statement, so they share a single resolution timestamp — and it is the instance’s latest, because nothing touches a FAILED instance’s rows afterwards (the poller’s stale and failed branches only update rows still waiting).

So the re-arm set is the frontier’s own rows, plus every row sharing the instance’s latest resolution timestamp. It is gated on a non-empty frontier: an instance that had nothing parked has nothing to restore, and an older satisfied wait is spent history rather than a park.

The parks are re-armed in the migration’s own transaction, so migrate-and-resume is one commit and a crash cannot leave a half-revived instance.

Afterwards the instance comes back through ordinary paths: the row is unowned (the claim died with the source row), the snapshot is suspended, and the parks are armed. A re-armed timer whose instant has passed is claimed by the ordinary poller on its next tick; a message park waits for its next correlated inbound. There is no new resume entry point and no privileged re-drive, which is precisely why the claim-guarded concurrency story still holds afterwards.

flowchart LR
    F["the wait frontier's own rows"] --> SET
    L["every row sharing the instance's<br/>latest resolution timestamp"] --> SET
    T["a synthesized q:timeout boundary,<br/>a timer boundary event"] -.->|"neither appears<br/>in the frontier"| L
    SET["the re-arm set —<br/>gated on a non-empty frontier"] --> ARM["re-armed in the migration's own transaction,<br/>status back to suspended, failure keys dropped"]
    ARM --> ORD["ordinary paths — the poller claims a due timer,<br/>a message park waits for its next correlated inbound"]

The frontier alone would silently drop the deadline the park was armed with; the second input recovers it exactly, because the failure commit resolved every waiting row in one statement and nothing touches a FAILED instance’s rows afterwards.

A defect this closed

The first version derived loci from waiting rows that were still marked waiting. For a FAILED instance there are none — its failure commit resolved them all. So every frontier entry fell through to the “no row behind it” branch and was validated as a message wait. A dead instance parked on a timer was therefore checked against the wrong rule, and warned about a missing row that was sitting right there.

Since a FAILED instance is the operation’s prime use case, that was not a corner case. Reading the park set as described above fixed it — and note the property that makes the fix trustworthy: what validation checks is now exactly the set resume re-arms. One derivation, two consumers, no possibility of drift.

Batch independence is structural, not promised

The batch endpoint is not a second implementation. A request body parses once into a migration plan; a single internal operation applies that plan to exactly one instance and returns a verdict rather than an HTTP response; and both endpoints are thin wrappers over it.

So “each instance validates and migrates independently” is not a promise the batch endpoint makes and must be trusted to keep. It is structural: the batch is a loop over the single-instance operation, and the attempt is the loop’s return value. There is no shared mutable state for one instance’s outcome to leak into another’s, because there is nowhere to put it.

Everything else follows:

  • One claim, one transaction, one report entry per instance. No batch-wide transaction — which is the point, and which makes the crash story trivial: a mid-batch crash leaves every instance either fully migrated or completely untouched, because that is already true of each instance’s own commit and there is nothing larger to be half-applied.
  • The outcome is a single enum, not a combination of booleans, so no caller has to infer a state from flags that might contradict each other.
  • The status describes the batch, not its instances. A run where every instance refused still answers 200: the call was accepted, executed to completion, and reported in full. Scripts key on the totals and the per-instance outcomes.

Contention is reported, never retried

An instance whose claim is held bounces, and the batch moves on. Retrying inside the call is refused on two grounds, and both are about honesty rather than convenience:

  • It makes an administrative request’s runtime unbounded — the claim is held by another replica’s work, on that work’s timescale.
  • It turns the report into a claim about a moment that has already passed. A report that says “everything moved” after internally retrying for thirty seconds describes a world that no longer exists by the time it is read.

The caller re-runs the same request instead. The selector is deterministic and whatever moved is no longer under the source pin, so a re-run converges — which is a much better contract than “we tried hard”.

Selection is ordered by instance id, then limited. Ordering by a timestamp that moves under a live population is exactly how a caller silently skips work, and a paging bug that skips work without saying so is the worst kind.

Two request-level contradictions are caught from the request alone rather than N times over in the per-instance reports — resuming a selection of suspended instances (it selects exactly what resume refuses), and re-homing a mixed population into one process (one mapping could only be right for one of them).

What deliberately does not move

Pending outbound rows stay under the source pin. An emission was minted by the source deployment’s channel bindings and is dispatched against them — and the dispatcher covers draining deployments, so it drains where it was made. Re-targeting a message at bindings that never produced it would be worse than leaving it: the destination, headers, and codec were all decided by a configuration the target may not even have.

The audit journal moves scope but keeps its node ids verbatim. A trail names where something happened, under the graph that was live at the time; rewriting the ids would falsify the record. The migration event itself records source and target, so the provenance is recoverable without corrupting the history.

Both are reported — the outbound rows as a warning — because the operator should know why, not just what.

Next

The pull surface: design

External tasks is the worker-facing guide. This chapter is why the pull surface is a hundred lines of new machinery rather than a subsystem — which is almost entirely down to one decision.

Ownership transfer is the design

A channel declaring transport: pull gets a sink like any other transport. The sink claims its URI scheme, exactly as an HTTP or broker sink claims theirs. What it does with a delivery is different: instead of dialing anything, it parks the delivery as a task row — and then answers Delivered.

That answer is the whole design. Ownership transfers from the outbox row to the task row, and the relay deletes the outbox row exactly as it would for a genuinely delivered push.

Everything else falls out of it for free:

  • The relay’s retry and poison machinery applies to a failed park automatically. If parking fails — the database is briefly unavailable, say — the sink answers a retryable failure, and the outbox’s existing backoff curve retries the park with no new code. If the operator configured an attempt ceiling, an unparkable delivery poisons exactly like an undeliverable one, records its incident, and stops pinning a draining deployment. None of that had to be re-implemented for pull; it was already there, one layer up.
  • The hot claim predicate is untouched. The outbox worker’s SKIP LOCKED batch claim — the query that runs constantly on every replica — does not learn about pull at all. It sees a delivery, hands it to a sink, and gets an answer. Adding a delivery mode without touching the hottest query in the system is the payoff for putting the new behaviour behind an existing polymorphic seam rather than beside it.
  • Parking is off the execution path. The sink runs on the outbox tick loop, not on an execution lane. It touches the engine actor not at all: a park is a store write plus a wake-up.

The alternative shape — a first-class “pull delivery” concept understood by the dispatcher, the outbox worker, and the retirement gate — would have been a genuinely new subsystem with its own retry semantics, its own poison horizon, and its own failure modes to reason about. Modelling pull as a sink that happens to hand ownership to a different table keeps one delivery pipeline.

The completion is not a new resume path

A worker’s result is rebuilt into an ordinary inbound message and re-enters the engine through the same seam every transport delivers through. Not a nested call, not a privileged entry point: one more serialized turn, indistinguishable from a pushed reply.

That is what keeps three things unchanged rather than duplicated:

  • Correlation. The parked headers ride through, so the author’s <q:alias> resolves the waiting instance the way it always has.
  • Validation. The completion goes through the same intake pipeline — a worker cannot smuggle a structurally invalid payload past the checks a pushed message faces.
  • Deduplication. The completion carries the originating delivery’s key as an explicit idempotency key, so inbox dedup covers it with no worker-side cooperation.

A second resume entry point would have needed all three re-implemented, and would have been the place where they drifted.

Lock expiry inside the claim predicate: no sweeper, by construction

A worker that dies mid-task must not hold its task forever. The usual answer is a reaper — a background job that scans for expired locks and releases them — which is a role to schedule, a lease to gate, a cadence to tune, and a lag between expiry and availability.

There is none here. Lock expiry is part of the predicate that decides what a fetch may claim: a task is fetchable when it is not terminal, its backoff has elapsed, and it is either unlocked or holding an expired lock. An expired lock is therefore not a state anyone has to clean up. It is simply not an obstacle to the next fetch.

stateDiagram-v2
    [*] --> Fetchable: the sink parks<br/>the delivery
    Fetchable --> Locked: a fetch claims it
    Locked --> Fetchable: the lock expires —<br/>the predicate stops excluding it
    Locked --> Fetchable: failure, budget left,<br/>after the backoff
    Locked --> Terminal: failure,<br/>budget spent
    Locked --> [*]: completion dispatched,<br/>then the row deleted

No edge out of Locked is driven by a background job: each one is either a worker’s own call or the claim predicate declining to exclude the row. That is what “no sweeper” actually buys.

The consequences are worth naming, because “no component” is easy to undervalue:

  • No lag. A task is available the instant its lock expires, not on the next sweep tick.
  • Nothing to gate. No lease role, no leader, no cadence key, no failure mode where the sweeper is down and locks pile up.
  • Nothing to get wrong under concurrency. The claim and the expiry check are the same statement, so there is no window between “the sweeper released it” and “someone claimed it”.

The cost of a dead worker is therefore exactly one lock duration — which is a knob the operator already sets, rather than a second knob about how often to look.

The completion path extends the same idea: it re-takes the lock across the dispatch, so the lock cannot expire mid-flight and let a second worker pick the task up while the first one’s result is still inside the engine.

Dispatch, then delete — and why the inverse loses work

The completion order is: verify and hold the lock, dispatch the result through the inbound path, then delete the task row.

That makes the surface at-least-once. A crash between the dispatch and the delete re-offers the task, and a duplicate completion is absorbed by inbox deduplication under the delivery’s idempotency key.

sequenceDiagram
    participant W as Worker
    participant T as Task row
    participant E as Engine intake
    W->>T: complete, guarded by the lock it holds
    T->>T: re-take the lock across the dispatch
    T->>E: dispatch the result through the ordinary inbound path
    Note over E: correlation, validation and inbox dedup,<br/>under the delivery's own idempotency key
    E-->>T: accepted
    Note over W,T: a crash before the delete re-offers the task —<br/>a duplicate, never a loss
    T->>T: delete the task row

The delete is last on purpose. The crash window between the dispatch and the delete costs a duplicate that already has a dedup key waiting for it; inverting the two would make that same window cost the work itself.

The inverse order — delete, then dispatch — would make it at-most-once, and the same crash would lose the work outright: the task is gone, the instance is still parked, and nothing anywhere records that the worker ever answered. The instance waits forever on a result that was produced and thrown away.

Between a duplicate the system already knows how to absorb and a silent loss it cannot detect, the duplicate is not a close call. This is the standard shape of the choice, and it is worth stating in those terms: at-least-once plus deduplication beats at-most-once plus hope, whenever deduplication is available — and here it is available by construction, because the idempotency key was minted by the delivery that created the task.

A worker that completes with no result re-delivers the original request payload. That is the fire-and-forget shape: the work happened outside, and the flow waits only on the fact of it.

If the engine refuses the completion on the inbound path, the row stays, still locked until its grace window elapses, and becomes fetchable again. The refusal carries the engine’s own code as an attribute — not just prose — because that code is the only thing that tells a worker whether re-fetching later can ever help (a transiently unavailable engine) or never will (a validation reject that will fail identically forever).

Zero rows affected is never success

Every worker-facing mutation is ownership-guarded: it matches only if this worker still holds this task’s lock. Which means the interesting case is a statement that matches nothing — and the critical rule is that this is never reported as success.

A 200 to a worker whose lock lapsed is a duplicate-execution bug. The worker believes its result landed; it did not; the task is fetchable again and someone else will do the work too.

So a zero-row result triggers a second, unguarded read to find out which situation it is, and the answer is one of four verdicts:

VerdictSituationWhat the worker should do
LOCK_LOSTThe lock expired or was released; the task is fetchable againStop. Do not re-complete — fetch fresh work.
LOCK_HELDAnother worker holds itStop. It is not yours.
TERMINALThe budget is spent; it can never be completed or failed againStop permanently.
NOT_FOUNDNo such task on any live deploymentStop; the task is gone.

Splitting them apart matters because they call for different worker behaviour, and a single generic “conflict” would leave a worker author guessing — usually by retrying, which is wrong for three of the four.

The extra read is deliberate and cheap: it happens only on the failure path, and it reads lock state only. The payload never rides a failure path — there is no reason for a task’s body to travel through an error response.

Separating “no such task” from “you do not hold it” needs one more distinction: the guarded statement is attempted across the live deployment set, and if it matches nowhere, an unguarded existence probe distinguishes a lock problem from a genuinely absent task. A worker names topics, never deployment ids — the same posture the outbox worker and the instance listing take — so the surface does the walking.

Bounds are rejections, never clamps

Every bound a worker can send — lock duration, long-poll wait, batch size — is a ceiling the operator sets, and an over-ceiling request is a 400.

Silently clamping would be the friendlier-looking choice and it is the wrong one, for a reason specific to locks: a worker that believes it holds a longer lock than it does is a duplicate-execution bug. It paces its work against a deadline that already passed, the task goes fetchable, another worker takes it, and the first one’s completion is refused after the work was already done twice. A clamp does not prevent that; it creates the belief that causes it.

The same reasoning applies to the others, less dramatically: a clamped long poll makes a worker’s own timeout arithmetic wrong, and a clamped batch size makes its throughput planning wrong. In each case the reject is information and the clamp is a lie the system tells quietly.

Omitting a bound is different from asking for too much: an omitted value takes the default, not the ceiling. The default is a considered value; the ceiling is a limit. Conflating them would hand every silent request the most expensive setting available.

The engine also refuses to boot if the default lock duration exceeds its own ceiling, or is zero — a configuration that is internally contradictory should fail at startup, where somebody is watching, rather than at the first fetch.

The long poll wakes, and cannot miss

A fetch that finds nothing waits, bounded, for a task to be parked on one of its channels. The wake-up is a broadcast keyed by channel, so a fetch filters to the topics it asked for instead of re-querying on every unrelated park.

Two properties make it safe:

  • Subscribe before the first query. A task parked in the window between the query and the wait still wakes the waiter. The classic lost-wakeup ordering bug is closed by ordering, not by a timeout that hides it.
  • Missing a wake-up is harmless. A woken fetch always re-runs the claim query; the wake-up carries no payload beyond “look again”. So a lagging subscriber is not an error — it costs at most one extra round of a loop it was already in. The ring buffer is deliberately small, because it is a doorbell rather than a queue.

And the wait is always bounded by the operator’s ceiling, so a fetch answers with an empty list rather than hanging. A long poll that can hang is a client-side resource leak with extra steps.

Two budgets that never overlap

The worker-failure budget and a <q:retry> policy sit at different layers and never interact:

GovernsCounted byOn exhaustion
Worker budgetA worker failing at its own workThe task rowThe task turns terminal — never fetched again, retained with its last error
<q:retry>The task-level outcome in the modelThe instance snapshotThe instance fails durably

A worker retrying its own work never touches the instance; the instance stays parked on its wait, exactly as it would while a worker was simply slow. Conversely a <q:retry> timeout on a pull-backed call still fires if no worker ever completes — the pull surface changed the last hop, not the model’s semantics.

A terminal task no longer counts toward its deployment’s retirement quiescence gate, for the same reason a poisoned delivery doesn’t: “we gave up” must be a durable, visible state, and it must not hold a deployment open forever. That symmetry is intentional — the terminal task is the pull-side twin of the outbox’s poison horizon. See Retry machinery.

Posture

These are operate-surface routes, not administrative ones, and the classification is a judgement worth stating: a completion is an ordinary delivery, not a privileged control operation. It goes through the same intake as a pushed reply, and it can do nothing an inbound message on that channel could not do.

So it carries the operate surface’s cluster-internal posture rather than the administrative surface’s gate. A deployment needing authenticated workers puts them behind the same ingress policy the rest of the operate surface already needs.

Next

Configuration reference

The engine reads exactly one family of configuration keys — canonical sutra.* properties, each with a canonical SUTRA_* environment-variable mirror. There is no separate framework prefix and no legacy alias layer to reason about.

Sources and precedence

canonical env  >  config file  >  built-in default

The config file path comes from SUTRA_CONFIG (default sutra.properties, read from the engine’s working directory, and only if present); file values may themselves use ${ENV} / env:NAME indirection. A key that has both a sutra.* file entry and its SUTRA_* environment variable set always resolves from the environment.

rust/crates/sutra-engine/src/config.rs is the single source of truth for the exact key list — this page is a map of what’s there and how the pieces fit together, not a duplicate of it; treat that file (and otel.rs for the telemetry keys) as authoritative if this page and the running engine ever disagree.

Deployment source

sutra.* keyenvmeaning
sutra.deployment.sourceSUTRA_DEPLOYMENT_SOURCEdir (default) — watch a folder of sealed archives — or db — the database-backed store, activated only via POST /admin/deployments. See Deployment model.
sutra.deployments.dirSUTRA_DEPLOYMENTS_DIRRequired for the dir source: the directory of .sutra archives the engine watches.
sutra.deployments.poll-intervalSUTRA_DEPLOYMENTS_POLL_INTERVALHow often the dir source rescans (default a few seconds).
sutra.http.portSUTRA_HTTP_PORTListen port; 0 binds an OS-assigned port (always use 0 locally — see Quickstart).

The engine’s own datasource

sutra.* keyenv
sutra.datasource.urlSUTRA_DATASOURCE_URL
sutra.datasource.usernameSUTRA_DATASOURCE_USERNAME
sutra.datasource.passwordSUTRA_DATASOURCE_PASSWORD
sutra.datasource.max-connectionsSUTRA_DATASOURCE_MAX_CONNECTIONS
sutra.datasource.dispatch-max-connectionsSUTRA_DATASOURCE_DISPATCH_MAX_CONNECTIONS

Since 0.6.5 the engine opens two pools over this database. The main pool (max-connections, default 8, the size it always had) serves the execution lanes: intake, steps and resumes. Size it at least to sutra.engine.shards, because each lane holds a connection while it commits a step. The dispatch pool (dispatch-max-connections, default 4 since 0.8.3, 2 before) serves the outbox dispatcher and the timer poller, so a burst of intake can never leave delivery waiting for a connection. The claimer and the delete batcher hold one connection each; the others serve the defers and requeues of failed sends in a burst.

This is the engine-internal database (instances, outbox, lease, audit, inbox) — never a package’s own business data store, which owns its connection independently in datastores.yaml (see Data stores). Note the CLI’s sutra migrate reads a different-named set (SUTRA_DB_URL / SUTRA_DB_USERNAME / SUTRA_DB_PASSWORD / SUTRA_DB_SCHEMA) even when pointed at the same database — see the CLI reference.

Outbox and acknowledgement

sutra.* keyenv
sutra.outbox.tick-intervalSUTRA_OUTBOX_TICK_INTERVAL
sutra.outbox.workersSUTRA_OUTBOX_WORKERS
sutra.outbox.destination-concurrencySUTRA_OUTBOX_DESTINATION_CONCURRENCY
sutra.outbox.max-bufferedSUTRA_OUTBOX_MAX_BUFFERED
sutra.outbox.lag-warnSUTRA_OUTBOX_LAG_WARN
sutra.outbox.batch-sizeSUTRA_OUTBOX_BATCH_SIZE
sutra.outbox.retry.base-delay / .max-delay / .jitterSUTRA_OUTBOX_RETRY_BASE_DELAY / _MAX_DELAY / _JITTER
sutra.outbox.retry.max-attemptsSUTRA_OUTBOX_RETRY_MAX_ATTEMPTS
sutra.ack.deferred.capacitySUTRA_ACK_DEFERRED_CAPACITY
sutra.ack.deferred.timeoutSUTRA_ACK_DEFERRED_TIMEOUT
sutra.ack.deferred.sweep-intervalSUTRA_ACK_DEFERRED_SWEEP_INTERVAL

The outbox delivers many rows at once, under three bounds (since 0.8.3):

  • sutra.outbox.workers (default 32, 1..1024) is the most sends in flight at once, across every destination. 1 restores one delivery at a time. Before 0.8.3 it was a count of workers rows were spread over by instance, defaulting to the lane count and at least 8.
  • sutra.outbox.destination-concurrency (default 32) is the most sends in flight to one destination authority: scheme, host and port, the path ignored. It is clamped to workers with a WARN. A row waiting on a busy destination holds no global slot, so a slow destination does not delay the others.
  • sutra.outbox.max-buffered (default 4096, from workers to 100 000) is the most rows held in memory, in flight or queued. Past it the claimer waits.

One instance’s deliveries still go out one after another, in the order they were enqueued; different instances never share a queue. sutra.outbox.lag-warn (default PT10S, PT0S = off) logs one WARN a minute while delivered rows exceed it, enqueue to delivery. A step that enqueues wakes the dispatcher at once, and a local:// delivery is handed to it in-process as soon as the step’s commit is durable, so the tick (tick-interval, default PT5S) only paces retries that fall due and rows another replica enqueued. sutra.outbox.batch-size (default 50) is how many due rows one claim takes per deployment. Delivered rows are deleted in batches. See Tuning for throughput.

Outbound deliveries retry with backoff forever by default. sutra.outbox.retry.base-delay defaults to PT1S: retry n waits about base-delay × 2ⁿ, so the first retry comes about 2 s after the failure. Since 0.6.20 each failed attempt that will be retried logs a WARN with the destination, the cause’s code and message, the attempt number and next_attempt_at. sutra.outbox.retry.max-attempts is the opt-in ceiling: a delivery that exhausts it is marked terminally poisoned — retained with its last error, never retried again, one incident recorded for a required delivery, and no longer counted by the draining-deployment retirement gate. “We gave up” is a durable, visible state, never a silent disappearance.

Full explanation of the deferred-ack registry the sutra.ack.deferred.* keys tune: Acknowledgement modes. Configuring the attempt ceiling is also what makes a poisoned delivery reachable as a <q:retry> failure — see Retries, history, and schedules.

Instance lifecycle — ownership and retention

sutra.* keyenv
sutra.instance.sweep-intervalSUTRA_INSTANCE_SWEEP_INTERVAL
sutra.instance.claim-timeoutSUTRA_INSTANCE_CLAIM_TIMEOUT
sutra.instance.retentionSUTRA_INSTANCE_RETENTION
sutra.instance.retention-sweep-intervalSUTRA_INSTANCE_RETENTION_SWEEP_INTERVAL

Every resume claims the instance first, so two replicas can never advance one instance concurrently; the stuck-instance sweep (sweep-interval, default PT1M) clears claims whose owner has been silent longer than claim-timeout (default PT5M).

A finished instance is history, not a 404: terminal (completed/terminated) snapshots are retained for sutra.instance.retention (ISO-8601 duration, default P7D) and served by GET /sutra/instances/{id} and GET /admin/instances/{id}/history; a lease-gated sweeper purges rows past the window on the retention-sweep-interval cadence (default PT1H). PT0S restores delete-at-completion. Failed instances are always retained regardless of the window — they need an operator before their deployment may retire; the operator action is instance migration. See Retries, history, and schedules for what the retention window makes queryable.

With <q:audit capture="media">, retention also deletes the recordings the purged instances’ audit events reference, from whichever declared object store holds them (below).

Capacity pools

sutra.* keyenv
sutra.capacity-poolsSUTRA_CAPACITY_POOLS
sutra.capacity-pools.<name>.slotsSUTRA_CAPACITY_POOLS_<NAME>_SLOTS
sutra.capacity-pools.<name>.reserve.inboundSUTRA_CAPACITY_POOLS_<NAME>_RESERVE_INBOUND
sutra.capacity-pools.<name>.reserve.outboundSUTRA_CAPACITY_POOLS_<NAME>_RESERVE_OUTBOUND
sutra.capacity-pools.<name>.tenantsSUTRA_CAPACITY_POOLS_<NAME>_TENANTS

Named, cluster-wide slot counts that channels of any deployment draw from with pool: <name>. Full explanation, defaults and a walkthrough: Capacity pools and sessions.

Object stores

sutra.* keyenv
sutra.object-storesSUTRA_OBJECT_STORES
sutra.object-stores.<name>.uriSUTRA_OBJECT_STORES_<NAME>_URI
sutra.object-stores.<name>.endpointSUTRA_OBJECT_STORES_<NAME>_ENDPOINT
sutra.object-stores.<name>.regionSUTRA_OBJECT_STORES_<NAME>_REGION
sutra.object-stores.<name>.access-key / .secret-keySUTRA_OBJECT_STORES_<NAME>_ACCESS_KEY / _SECRET_KEY
sutra.object-stores.<name>.encryptionSUTRA_OBJECT_STORES_<NAME>_ENCRYPTION
sutra.object-stores.<name>.kekSUTRA_OBJECT_STORES_<NAME>_KEK
sutra.object-stores.<name>.key-idSUTRA_OBJECT_STORES_<NAME>_KEY_ID

Named S3-compatible buckets the engine reads recorded media from (an llm channel’s media-store) and deletes it in (erasure and retention). Credentials and the key-encryption secret are references, resolved at boot; a store that cannot be built refuses startup. Full explanation and a walkthrough: Recordings and media references.

External tasks (the pull worker surface)

sutra.* keyenv
sutra.external-task.default-lock-durationSUTRA_EXTERNAL_TASK_DEFAULT_LOCK_DURATION
sutra.external-task.max-lock-durationSUTRA_EXTERNAL_TASK_MAX_LOCK_DURATION
sutra.external-task.max-async-response-timeoutSUTRA_EXTERNAL_TASK_MAX_ASYNC_RESPONSE_TIMEOUT
sutra.external-task.max-tasksSUTRA_EXTERNAL_TASK_MAX_TASKS
sutra.external-task.retriesSUTRA_EXTERNAL_TASK_RETRIES
sutra.external-task.retry-timeoutSUTRA_EXTERNAL_TASK_RETRY_TIMEOUT

A channel declaring transport: pull parks its deliveries as fetchable tasks instead of dialing an endpoint; workers drive them over POST /sutra/external-tasks/fetch-and-lock (bounded long poll) and .../{id}/complete / .../{id}/failure. The defaults: a PT30S lock when the fetch names none (ceiling PT1H — a longer request is rejected, never clamped), a PT30S long-poll ceiling, 100 tasks per fetch, a worker-failure budget of 3 with PT10S between attempts. A spent budget turns the task terminal (failed) — retained, never fetchable again. The engine boots fail-closed if default-lock-duration exceeds its own ceiling or is zero. Full treatment: External tasks.

Execution lanes

sutra.* keyenvdefault
sutra.engine.shardsSUTRA_ENGINE_SHARDS1
sutra.engine.shard-queue-capacitySUTRA_ENGINE_SHARD_QUEUE_CAPACITYunbounded

sutra.engine.shards is the number of identical actor lanes the engine executes on inside one replica. At N > 1 all work for one instance — routed by a stable hash of its id — still runs on one lane, in arrival order, one request at a time: per-instance serialization is the contract, and it holds at every N. Values above 1 are accepted; the default stays 1 because turning it up has one advertised consequence.

Say it out loud before you raise it. Incidental cross-instance serialization disappears at N > 1. Two concurrent deliveries to two different instances of the same flow never interleave under a single lane — as a side effect of there being one lane, never as a promise. At N > 1 they genuinely run in parallel. A deployment silently leaning on that side effect will observe new interleavings. Every supported concurrency mechanism is unaffected: per-channel singleton / serial consumption, per-channel and per-tenant admission caps, and optimistic expect="unchanged" / pessimistic forUpdate data-store writes.

shard-queue-capacity bounds each lane’s mailbox (unset = unbounded; zero is rejected — “unset” is how you say unbounded). A bounded send awaits on the caller’s task, so backpressure propagates outward to the transport that offered the work — an in-flight HTTP request, a broker prefetch window, a poller tick — and never sideways into another lane.

Meters ship with the feature, each carrying the lane index as a dimension: sutra.engine.shard.queue-depth (per-lane backlog and skew), …dispatches / …parks / …resumes, …handoffs (cross-lane relay hops — expected, and rising with lane count by construction), and …claim-bounces split relay / timer. That last one is the mis-route alarm: on a healthy rollout it reads near zero outside genuine cross-replica contention.

The live lane count is readable without reading config: GET /sutra/health/ready reports it under the loader check’s data.shards, read off the running router rather than echoed back from configuration.

Full picture: Execution lanes.

Broker connectivity events

KeyDefaultMeaning
sutra.engine.connectivity-events.channel (env SUTRA_CONNECTIVITY_EVENTS_CHANNEL)unset(0.8.0) A channel name. When set, every broker outage a transport sees on one of a deployment’s channels is delivered to that deployment as an inbound JSON message on the channel of this name, if the deployment declares one. Unset: nothing is collected.

Declare the channel in the package as an in-process intake and start a process on it:

channels:
  - name: connectivity-in
    transport: local
    codec: json

Each outage produces two messages, one when it starts and one when it ends:

{"code":"SUTRA.INBOUND.AMQP.OUTAGE_ENDED","channel":"orders-in","direction":"inbound","state":"up",
 "startedAt":"2026-11-19T10:00:00.120Z","endedAt":"2026-11-19T10:01:00.420Z","ms":60300,
 "detail":"amqp broker mq.example:5672"}
  • state is down when the outage starts (no endedAt or ms) and up when it ends.
  • code is the transport’s own log code for the change, so the message and the engine log line match.
  • direction: inbound is an intake losing its connection. direction: outbound is a send channel whose send could not reach its broker; each send channel that sent during the outage reports its own window, ending on its first send that reaches the broker again. A send channel that sent nothing during the outage reports nothing.
  • A deployment only receives events for its own channels. One that declares no channel of the configured name receives nothing.
  • Delivery is best effort: up to 1024 events wait per deployment; past that the oldest is dropped and SUTRA.CONNECTIVITY.EVENTS_DROPPED is logged. An event the deployment refuses is logged as SUTRA.CONNECTIVITY.EVENT_UNDELIVERED and never retried.
  • The AMQP 1.0 transport reports these events. Transports without a long-lived broker connection (HTTP, for example) report none.

Broker outages as messages runs it against a broker that is stopped and started.

Limits

sutra.* keyenv
sutra.codec.max-payload-bytesSUTRA_CODEC_MAX_PAYLOAD_BYTES

Full explanation: Limits and quotas.

Admin API auth

sutra.* keyenv
sutra.admin.auth.schemeSUTRA_ADMIN_AUTH_SCHEME (apikey | bearer)
sutra.admin.auth.key-refSUTRA_ADMIN_AUTH_KEY_REF
sutra.admin.auth.headerSUTRA_ADMIN_AUTH_HEADER (default X-API-Key)
sutra.admin.oidc.issuer / .audience / .jwks / .role-claim / .required-roleSUTRA_ADMIN_OIDC_*
sutra.admin.oidc.dev-disabledSUTRA_ADMIN_OIDC_DEV_DISABLED

The /admin/* surface (deployments, instance inspection, subject erasure) is gated fail-closed: unconfigured returns 503, a missing/invalid credential returns 401, and a valid token missing the required role/claim returns 403 — never silently open. The auth-key scheme is the same static-secret model channels use for inbound HTTP auth, and it takes precedence when both are set. Gating is disabled only via the explicit sutra.admin.oidc.dev-disabled=true escape hatch. From 0.6.18 the gate also covers GET /sutra/instances, its ?alias= search and GET /sutra/instances/{id} when admin auth is configured. An engine with no admin configuration at all still answers them without a key, so keep such an engine where only operators can reach it. See Upgrading to 0.6.18.

Audit sinks

sutra.* keyenv
sutra.audit.jsonl.pathSUTRA_AUDIT_JSONL
sutra.audit.otel.endpointSUTRA_AUDIT_OTEL_ENDPOINT
sutra.audit.otel.service-nameSUTRA_AUDIT_OTEL_SERVICE_NAME (default sutra-audit, 0.6.12)
sutra.audit.sqlSUTRA_AUDIT_SQL

The SQL sink writes the audit journal behind execution and best-effort: an event is queued and written by the audit writer, not inside the transaction of the step that produced it, so a crash can lose events not yet written. Since 0.8.1 the writer takes whatever queued while its previous write ran, up to 256 events, and writes it as one multi-row insert per deployment in one transaction, where it used to commit every event on its own. On a durable PostgreSQL that is what lets the journal keep up with a busy engine. The journal is the same: each instance’s events in order, deduplicated, and a batch that fails is written again event by event. Nothing to configure.

See Logging and audit.

Telemetry (OTel)

sutra.* keycanonical envstandard OTEL_* env also accepted
sutra.telemetry.otlp.endpointSUTRA_TELEMETRY_OTLP_ENDPOINTOTEL_EXPORTER_OTLP_ENDPOINT
sutra.telemetry.service-nameSUTRA_TELEMETRY_SERVICE_NAMEOTEL_SERVICE_NAME
sutra.telemetry.enabledSUTRA_TELEMETRY_ENABLEDOTEL_SDK_DISABLED (inverted)
sutra.telemetry.metrics.export-intervalSUTRA_TELEMETRY_METRICS_EXPORT_INTERVALOTEL_METRIC_EXPORT_INTERVAL
sutra.telemetry.metrics.temporality-preferenceSUTRA_TELEMETRY_METRICS_TEMPORALITY_PREFERENCEOTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE
sutra.telemetry.metric-labelsSUTRA_TELEMETRY_METRIC_LABELS— (default tenant,module,version)
sutra.telemetry.resource-attributesSUTRA_TELEMETRY_RESOURCE_ATTRIBUTESOTEL_RESOURCE_ATTRIBUTES (both apply; the sutra.* value wins per key; 0.6.12)

No OTLP endpoint configured means no exporters at all, and this is the default — telemetry export is opt-in, never opt-out, and Sutra itself collects nothing regardless: see No telemetry, no phone-home for the exact guarantee, and Observability for what each signal actually carries once you do turn export on.

Secrets — never literal values

Every credential-shaped field across channels.yaml, datastores.yaml, and the admin-auth keys above is a reference, resolved at startup or channel-activation time — env:NAME, secret:KEY (a file under the mounted secrets directory, default /etc/sutra/secrets / SUTRA_SECRETS_DIR), ${NAME} / ${NAME:default} placeholders, or a vendor scheme (vault:…, aws-secrets:…, azure-kv:…, gcp-secret:…) resolved by whichever sutra-envref-<vendor> crate the binary was built with. A literal secret value in a resource file is rejected at package-validation time, not just discouraged by convention. See Domain neutrality and the SPI model for how a vendor secret backend plugs in.

Next

Acknowledgement modes

An inbound channel declares an ack-mode that decides when the engine acknowledges receipt relative to processing. How each transport actually realizes that intent differs — a broker gets a native ack/nack, HTTP gets a status code — but the two values mean the same thing everywhere.

The two modes

  • on-persist — acknowledge the moment the engine has durably captured the inbound, before the BPMN process runs. On a broker: ack immediately, releasing the delivery slot early (lower broker-side latency; a redelivery after a mid-process crash is caught by inbox dedup). On HTTP: reply 202 Accepted with an acceptance receipt (see Acceptance receipts) and process asynchronously — the fire-and-forget intake, whose eventual reply (if any) rides an outbound channel instead of the original connection.
  • on-complete — acknowledge only once the instance reaches a terminal state (INSTANCE_COMPLETED or INSTANCE_FAILED). On a broker: the ack is deferred — registered against the engine’s DeferredAckRegistry and held until the instance finishes. On HTTP: hold the connection open until completion and return the reply body (the body of the flow’s <q:reply>, or an empty body if it declares none; the flow’s variables never answer the caller) — the classic synchronous request/reply.
sequenceDiagram
    participant S as Broker / HTTP caller
    participant E as Engine intake
    participant P as Process instance
    S->>E: deliver
    E->>E: durably capture the inbound
    Note over E: on-persist acknowledges here
    E-->>S: broker ack, or HTTP 202 Accepted with the receipt
    E->>P: start
    P-->>S: eventual reply, if any, on an outbound channel
sequenceDiagram
    participant S as Broker / HTTP caller
    participant E as Engine intake
    participant P as Process instance
    S->>E: deliver
    E->>E: durably capture the inbound
    E->>P: start
    Note over S,E: broker — ack deferred in the registry<br/>HTTP — connection held open
    P-->>E: INSTANCE_COMPLETED or INSTANCE_FAILED
    Note over E: on-complete acknowledges here
    E-->>S: broker ack or nack, or the HTTP reply body

The two are identical up to the durable capture; the mode moves only where the ack arrow sits. That one move is the whole trade — release the delivery slot early and let inbox dedup absorb a redelivery, or hold it so the broker’s own redelivery is your crash-recovery path.

The default differs by transport, because the natural mode differs: broker channels default to on-persist (release the delivery slot early); HTTP channels default to on-complete (synchronous request/reply). An HTTP channel opts into asynchronous intake by declaring ack-mode: on-persist explicitly.

channels:
  - name: orders-inbound
    transport: http
    bind: "POST /channels/orders-inbound"
    ack-mode: on-persist     # HTTP: 202 Accepted as soon as intake is durable

An asynchronous channel can still answer with a document

on-persist decides when the caller is answered — now, not when the work finishes. It does not decide what with. A process that renders a reply before it parks, with <q:reply continue="true">, sends that reply as the body of the 202; a process that renders nothing gets the receipt as the body of the 202.

That combination is what a long-running intake usually wants. The caller is not held for the work, and it still gets back something it can act on — a batch id to quote, a count of what was accepted — instead of an empty body it cannot distinguish from a message that went nowhere:

HTTP/1.1 202 Accepted
Content-Type: application/json

{"batchId":"d36d2c59-0fea-4349-b410-a4b4c7715ab9","rowsAccepted":4,"status":"loading"}

The same render under on-complete comes back as a 200 instead. Either way continue="true" is what puts the reply at the park rather than at the end of the process, which is the only reason a load measured in minutes can answer its caller in milliseconds. See Batch loading for a worked example.

When to pick which

Use on-persist when…Use on-complete when…
The process is short-livedThe process takes real time (minutes, not milliseconds)
You want the broker slot released fastYou want the broker to redeliver if the engine crashes mid-process
Downstream tolerates at-least-once with inbox dedup catching the duplicateThe side effect is non-idempotent and you want broker redelivery as the restart-recovery path
You don’t want a bounded in-memory registry on the engineYou can afford the deferred-ack registry’s configured capacity of pending entries

The mode is set per channel — different channels on the same engine can use different modes.

Per-transport wiring — what actually realizes on-complete

on-complete’s deferred-settle mechanism is a capability each transport factory self-declares (see Domain neutrality and the SPI model) — the engine never hardcodes a per-vendor branch. Current wiring:

Transporton-completeMechanism
RabbitMQwiredDeferred settle via the registry: basic.ack on completion, basic.nack(requeue=false) on failure/timeout/overflow.
KafkawiredSettle commands over an internal channel to the consumer task; per-partition low-watermark commits, so an out-of-order settle can never mask an earlier nack.
AWS SQSwiredAck = delete_message. The visibility timeout keeps running while parked — size it (and the registry timeout) against expected instance duration.
Google Pub/SubwiredPer-message ack()/nack() handles held in the callback; same lease-timeout caveat as SQS.
AMQP 1.0wiredDispositions bridged to the session task (accept / reject).
FilewiredThe terminal file move (.done/ / .failed/) happens at the instance’s terminal event.
HTTPnativeConnection-hold is its on-complete — no registry involved.
Knativewired (response-hold)The push response is held to the terminal event, bounded by a hold timeout; expiry degrades to a loud warning rather than losing the signal.
Daprnot supportedDapr’s own pub/sub components own redelivery timers; holding the push response would multiply duplicate deliveries rather than strengthen the guarantee. Declaring on-complete here boots with a loud diagnostic and runs on-persist instead.

A channel declaring ack-mode: on-complete on a transport whose factory reports it can’t realize it never fails silently — the engine emits SUTRA.ACK.ON_COMPLETE_UNSUPPORTED at startup and runs on-persist.

AWS SQS, beyond the settle behaviour in the table:

  • A message with no content-type attribute is decoded with the channel’s codec, or read as raw bytes on a channel with no codec.
  • A refused message is deleted. It leaves a WARN SUTRA.INBOUND.DELIVERY_DROPPED and, if an audit sink is on, an inbound.DELIVERY_DROPPED audit record. See Refused deliveries and connection references.
  • region and queue.url are required, and accept references. See References in connection properties, which covers the other transports too.

The deferred-ack registry

The registry (sutra-channels) is a bounded, insertion-ordered structure with three operator knobs (see Configuration reference):

KeyDefaultBehavior
sutra.ack.deferred.capacity10 000At capacity, register() nacks the oldest entry (SUTRA.ACK.DEFERRED_OVERFLOW) before accepting the new one.
sutra.ack.deferred.timeout1 hourThe sweep nacks any entry older than this (SUTRA.ACK.DEFERRED_TIMEOUT) — the broker slot frees, but the instance itself keeps running; inbox dedup absorbs any resulting redelivery.
sutra.ack.deferred.sweep-interval1 minuteCadence of the background sweep task.

Eviction is an operator-visible failure mode, not a silent memory leak — repeated SUTRA.ACK.DEFERRED_OVERFLOW events mean either raise the capacity or find the runaway processes that aren’t reaching a terminal state. Diagnostic codes at each stage: SUTRA.ACK.DEFERRED_REGISTERED (debug), SUTRA.ACK.DEFERRED_ACKED (debug), SUTRA.ACK.DEFERRED_NACKED (warn — a permanent reject), SUTRA.ACK.DEFERRED_OVERFLOW (warn), SUTRA.ACK.DEFERRED_TIMEOUT (warn), SUTRA.ACK.ON_COMPLETE_UNSUPPORTED (warn, at startup).

stateDiagram-v2
    [*] --> Registered: on-complete inbound, ack withheld
    Registered --> Acked: instance completed
    Registered --> Nacked: instance failed
    Registered --> Nacked: sweep, entry older than the timeout
    Registered --> Nacked: at capacity, oldest entry evicted

The completion and failure edges are the mode working as intended. The sweep and capacity edges are the ones an operator watches: a timeout nack frees the broker slot while the instance keeps running, and repeated overflow means either the capacity is too low or something is never reaching a terminal state.

Migration note

Omitting ack-mode entirely preserves today’s behavior exactly: a broker channel stays on-persist, and an HTTP channel stays synchronous (on-complete). Turning on a broker’s deferred ack, or an HTTP channel’s async intake, is a one-line addition to channels.yaml and takes effect on the next deployment poll.

When a delivery can’t be checked for a duplicate

Before it runs a delivery with an explicit event id or a declared dedupKey="body.<path>", the engine checks whether that key was already processed. If the database doesn’t answer the check, the delivery is refused and nothing runs, with SUTRA.INBOUND.DEDUP_CHECK_UNAVAILABLE. This holds even for a flow that never touches the database, because a duplicate can’t be undone afterwards: a data task’s store write commits before the step that would have caught it.

A delivery also gets refused, with SUTRA.INBOUND.DEDUP_IN_DOUBT, when an earlier delivery of the same event is recorded but a synchronous-call chain still holds that record. That chain is either running, or it failed and couldn’t release the record. The refusal stops the retry being swallowed as a duplicate of work that may never have finished. It also applies to a retry during a healthy, running chain, which gets a refusal until the chain ends. If the chain’s process is gone, or it could not release the record, the sweep resolves it within sutra.instance.inflight-chain-max-age, 15 minutes by default, so the refusal can last that long.

A refused delivery is safe to repeat with the same id. What the sender sees depends on the transport:

TransportWhat happens
HTTP503 with a problem document carrying the code. There’s no Retry-After header.
Dapr, Knative500, which the sender’s own retry policy treats as retryable.
RabbitMQbasic.nack(requeue=true).
AMQP 1.0The delivery is released, so the broker redelivers it.
KafkaThe consumer seeks back to the record, so the next poll reads it again.
SQSThe message stays in flight, and returns after its visibility timeout.
Pub/SubThe message is nacked, and redelivered after its ack deadline.
FileThe file moves back to the spool root for a later poll.
local:// hopsRetried with the outbox’s backoff.

What counts as an explicit event id:

  • On HTTP, Dapr and Knative: the channel’s idempotency-key-header, or a CloudEvent id.
  • On a queue transport: the engine’s own outbox key. A message from another producer doesn’t have one.
  • On the file transport: the file name.
  • local:// hops and external-task completions always have one.

A delivery with neither isn’t refused. It never reaches the check, and a repeat of it is simply a new delivery. That includes a message from another producer on a queue transport, unless its flow declares a dedupKey.

While a database is down, expect more retries, not lost work: nothing was accepted that wasn’t run.

Next

Refused deliveries and connection references

This page covers what the engine does with a message it permanently refuses on a broker or file channel, what changed in 0.6.18, and how references behave in a channel’s connection properties. For the ack modes themselves see Acknowledgement modes.

A message with no content type

Before 0.6.18 (back to 0.2.0-rc.1), a broker message with no content type on a channel that declares a codec was deleted with nothing recorded: no instance, no log line, no audit record. The source filled in application/octet-stream, the codec refused that, and the drop left no trace.

Now a message with no content type is decoded with the channel’s codec. If the codec refuses the body, it follows the normal reject path. A message that states a type the codec does not accept is still refused. A channel with no codec reads a message with no type as raw bytes, as before. The one visible difference there is that event.contentType is empty rather than application/octet-stream, the same as on HTTP and file channels.

What a permanent refusal leaves behind

The same silence covered every permanent refusal on these transports: a body the codec refused, a stated type it does not accept, a message no start event matched, a tenant quota. HTTP, Dapr and Knative answer the sender with a status, and the file transport moves the file to .failed/. Every other transport now leaves two things:

  • A WARN log line with code SUTRA.INBOUND.DELIVERY_DROPPED, the refusal’s own code as cause_code, the channel and the dedup key. It never carries the body.
  • When an audit sink is on, a record inbound.DELIVERY_DROPPED in the deployment’s journal. Its diagnostic_code is the refusal’s, and its diagnostic_json names the channel. No instance owns it, so its instance_id is NULL, not empty: select it with instance_id IS NULL.

Two limits of this release:

  • sutra.audit.sql is off by default. An engine with no audit sink writes no audit record, and the WARN is the only trace. Turn a sink on if you need the record. See Logging and audit.
  • A refused body is not kept. On SQS a refused message is deleted, and nothing in the engine can redrive it. A sender that needs the body back keeps its own copy.

What the refusal did to the message depends on the transport, and a copy may still exist:

TransportWhat happensWhere a copy can still be
aws-sqsdeletednowhere, unless the sender keeps its own
kafkaoffset committed past itin the topic, until retention removes it
gcp-pubsubackedin a snapshot or a seek window, if you keep one
rabbitmq, amqprejected without requeuein the dead-letter route, if the broker has one

A rejected credential gets the same audit record, beside the WARN the transport already wrote. A message settled as a duplicate is not a drop: it logs at INFO with SUTRA.INBOUND.DUPLICATE_DELIVERY. A deferred nack for a failed instance, SUTRA.ACK.DEFERRED_NACKED, is now WARN rather than INFO.

To check whether you lost messages on an earlier version, compare what your senders sent with the instances and rows the engine produced for that channel over the time it ran. On Kafka, compare the topic’s offsets with the same counts, and replay what is missing with content-type set. The engine holds nothing to query, because earlier versions wrote nothing.

References in connection properties

Since 0.6.19 every inbound transport expands them: see References in connection properties. Version 0.6.18 refused ${ on all but aws-sqs.

References in connection properties

A channel’s broker address, queue or spool directory can come from the environment or a mounted secret, so one package runs unchanged in each environment. This walkthrough runs a RabbitMQ channel whose whole connection is references, then shows what happens when one doesn’t resolve.

You need the engine, PostgreSQL and a broker. This was run on PostgreSQL 16 and RabbitMQ 3.13.

Which properties, and which forms

TransportProperties that may be a reference
kafkabootstrap.servers, topic, group.id, tls-ca-file
rabbitmqhost, port, virtual-host, queue, exchange
amqphost, port, queue, topic, tls-ca-file
gcp-pubsubproject-id, subscription, topic, endpoint-override
filespool.dir, directory
aws-sqsregion, queue.url, endpoint-override, wait-time-seconds, max-messages, visibility-timeout-seconds
FormReads
${NAME}the environment variable, and it can sit inside other text: ${BROKER_HOST}:9092
${NAME:default}the same, or the default when it isn’t set
env:NAMEthe environment variable; the whole value only
secret:KEY, ${secret:KEY}the file KEY in SUTRA_SECRETS_DIR, default /etc/sutra/secrets
vault:, aws-secrets:, azure-kv:, gcp-secret:the vendor store, in a build that links it; the whole value only, or inside ${…}

Other properties of these channels, such as prefetch-count, stay literal, and so do the TLS and SASL switches (tls, security.protocol, sasl.mechanism, tls-insecure-skip-verify).

Credentials (username, password on rabbitmq and amqp; since 0.6.21 sasl.username, sasl.password on kafka, and tls-ca-pem on amqp and kafka) take the whole-value forms above, ${secret:KEY} included since 0.6.20, and k8s:secret/<name>#<key>. A literal, or text with a reference inside it such as pre${X}, is refused with SUTRA.STARTUP.REFUSED.HARDCODED_SECRET. See section 5, and Connecting to a TLS and SASL broker for the TLS and SASL settings.

1. The channel

channels.yaml. The host and port come from the environment, the port with a default, the virtual host from env:, and the queue name from a mounted secret file:

channels:
  - name: note-in
    transport: rabbitmq
    host: ${BROKER_HOST}
    port: ${BROKER_PORT:5672}
    virtual-host: env:BROKER_VHOST
    queue: secret:notes-queue
    username: ${BROKER_USER}
    password: ${BROKER_PASSWORD}
    ack-mode: on-persist
    cloudevents-mode: none

The flow is note-keep from Keeping a message as text, which writes the message to a notes table. secrets/notes-queue holds the queue’s name, notes-from-a-secret. Create that queue on the broker in a virtual host called refs.

sutra lint refnotes
sutra package refnotes -o out

2. A reference that doesn’t resolve

Start the engine with SUTRA_DEPLOYMENTS_DIR pointing at out, the secrets folder mounted, and every setting except BROKER_HOST. It stops with exit code 1 and this error:

rabbitmq channel wiring failed: [SUTRA.INBOUND.RABBITMQ.CONFIG_INVALID] channel 'note-in' property 'host' has an unresolved reference: placeholder '${BROKER_HOST}' resolves to no value (environment variable 'BROKER_HOST' is not set and no default is given)

The message names the channel, the property and the variable.

Now set BROKER_HOST, and set BROKER_PORT to text that is no port. It also stops with exit code 1, and the message doesn’t repeat what the variable held:

rabbitmq channel wiring failed: [SUTRA.INBOUND.RABBITMQ.CONFIG_INVALID] channel 'note-in' property 'port' has a reference that does not resolve to an integer in 1..=65535

3. All of them set

Set BROKER_HOST and leave BROKER_PORT unset, so that the default applies. The engine starts, and the queue named by the secret has one consumer:

rabbitmqctl -q list_queues -p refs name consumers --no-table-headers
notes-from-a-secret	1

Publish a message to that queue with a message_id of r-1 and a content type of application/json. The row is there:

SELECT store_key, content_type, body FROM notes;
 store_key |   content_type   |              body
-----------+------------------+---------------------------------
 r-1       | application/json | {"title":"through a reference"}
(1 row)

The engine’s log names the queue it reached:

{"channel":"note-in","queue":"notes-from-a-secret","level":"INFO","message":"rabbitmq consumer up"}

So a value you read from a secret store shows up in connection log lines, as a literal one would. These are connection settings. Keep credentials in the credential properties.

4. An activation in a running engine

An activation is not refused for this. With the engine on the db deployment source, version 1.0.0 deployed and consuming, deploy a 1.1.0 whose queue is ${NOTES_QUEUE_NOT_SET}:

sutra deploy out2/refnotes.sutra --api --engine-url "$B"
deployed dep-f26c5a0b3b788bca8f9833e5 (slot default--refnotes--1.1.0, rev 1) — Active

1.1.0 reports Active and ready, and has no consumer. The engine logs a WARN:

{"code":"SUTRA.INBOUND.RABBITMQ.CONFIG_INVALID","level":"WARN","message":"rabbitmq consumer NOT started during flip: channel 'note-in' property 'queue' has an unresolved reference: placeholder '${NOTES_QUEUE_NOT_SET}' resolves to no value (environment variable 'NOTES_QUEUE_NOT_SET' is not set and no default is given)"}

No queue with that name was created, and 1.0.0 keeps its consumer. So watch for this code after a deploy, because the deploy’s own answer won’t show it.

If the channel with the bad reference already had a running consumer, that consumer stops. The next activation in which the reference resolves starts it again.

5. Broker credentials from a secret, and a blank reference

Since 0.6.20, username and password on rabbitmq and amqp take secret:KEY and ${secret:KEY}, besides ${NAME}, env:NAME and k8s:secret/<name>#<key>. A literal is still refused with SUTRA.STARTUP.REFUSED.HARDCODED_SECRET.

A reference that resolves to blank now warns SUTRA.CONFIG.PROPERTY.BLANK_REFERENCE, naming the channel, the property and the reference as written, never a value. Write ${NAME:} for an optional setting: it does not warn.

A RabbitMQ channel whose credentials come from secret files (/etc/sutra/secrets/broker-user, /etc/sutra/secrets/broker-password), and whose exchange is ${H} with H set but empty. The broker has only the user in that file, so the consumer connects only if the credentials resolved. Run B is the same package with exchange: ${H:}.

using sutra 0.6.20
$ cat refnotes/channels.yaml
channels:
  - name: note-in
    transport: rabbitmq
    host: ${BROKER_HOST}
    virtual-host: env:BROKER_VHOST
    queue: secret:notes-queue
    exchange: ${H}
    username: secret:broker-user
    password: ${secret:broker-password}
    ack-mode: on-persist
    cloudevents-mode: none
$ sutra lint refnotes
[WARN] SUTRA.CHANNEL.NO_SCHEMA — deployment default/refnotes/1.0.0 (dep-ffffffffffffffffffffffff): channel 'note-in' has 1 subscriber(s) but its codec (none) declares no message types — no type can be derived, so only a catch-all <q:source> can match. Schema-less ingress is legitimate; this warns so
[WARN] SUTRA.DEPLOY.DATASTORE.SHARED_CONNECTION — the connection of data store(s) notes names neither this package's tenant nor its module, so a copy co-hosted on one engine resolves the same connection and shares those stores' rows. If that is not intended, give each variant its own reference in it
0 error(s), 2 warning(s)
[exit 0]
$ sutra package refnotes -o out-a
[WARN] SUTRA.CHANNEL.NO_SCHEMA — deployment default/refnotes/1.0.0 (dep-ffffffffffffffffffffffff): channel 'note-in' has 1 subscriber(s) but its codec (none) declares no message types — no type can be derived, so only a catch-all <q:source> can match. Schema-less ingress is legitimate; this warns so it is explicit.
packaged out-a/refnotes.sutra (deploymentId dep-11d6a5f464d563879221824f)
[exit 0]
package refnotes-b exit=0
broker up: 1

== Run A: secret: credentials, exchange: ${H} with H set but empty
ready: {"status":"UP","checks":[{"name":"sutra-loader","status":"UP","data":{"deploymen
queue, consumers: notes-from-a-secret	1
Message published
 store_key |                    body                    
-----------+--------------------------------------------
 s-1       | {"title":"credentials from a secret file"}
(1 row)

BLANK_REFERENCE lines:
{"code":"SUTRA.CONFIG.PROPERTY.BLANK_REFERENCE","channel":"note-in","property":"exchange","timestamp":"2026-11-10T05:26:09.248+00:00","sequence":6,"level":"WARN","loggerName":"sutra_transport_spi","message":"channel 'note-in' property 'exchange' reference '${H}' resolved to blank, so the property is unset: the transport's built-in default applies, or a required property is refused. Write the refer
HARDCODED_SECRET lines: 0
broker password in the log: 0

== Run B: exchange: ${H:} with H set but empty
ready: {"status":"UP","checks":[{"name":"sutra-loader","status":"UP","data":{"deploymen
queue, consumers: notes-from-a-secret	1
Message published
 store_key |              body               
-----------+---------------------------------
 s-2       | {"title":"an optional setting"}
(1 row)

BLANK_REFERENCE lines: 0
broker password in the log: 0
removed, left: 0

The full warning text, one line in the log:

channel 'note-in' property 'exchange' reference '${H}' resolved to blank, so the property is unset: the transport's built-in default applies, or a required property is refused. Write the reference with an empty default, such as ${NAME:}, to leave it blank on purpose

What to watch for

  • A reference that resolves to blank is the property left unset, so ${NAME:} writes an optional setting. A variable that is set but empty counts too. A required property is then refused: Kafka bootstrap.servers and topic, RabbitMQ queue, AMQP host, Pub/Sub project-id, the file spool directory. One with a default takes it: RabbitMQ host becomes localhost, and Kafka group.id becomes sutra-<channel>. Since 0.6.20 the engine logs SUTRA.CONFIG.PROPERTY.BLANK_REFERENCE first (0.6.19 was silent); ${NAME:} is the explicit form and does not warn.
  • A value that starts with a secret scheme and a colon is a reference. A Kafka broker called vault, written bootstrap.servers: vault:9092, fails to resolve. Write it ${BROKER:vault:9092}. kafka:9092 is fine.
  • A changed value restarts the consumer at the next activation. An activation is what reads it again; nothing watches it in between.
  • Endpoint claims compare references as written, so two deployments on one queue under different names aren’t caught. See One endpoint, one deployment.

Connecting to a TLS and SASL broker

Since 0.6.21 the kafka and amqp (AMQP 1.0) transports connect to TLS brokers natively, with no sidecar. Kafka also authenticates with SASL PLAIN, SCRAM-SHA-256 or SCRAM-SHA-512, and AMQP 1.0 with SASL PLAIN.

What this was tested against. Local containers only: Apache Kafka 3.8 and Apache ActiveMQ Artemis. Azure Service Bus, Confluent Cloud, Amazon MSK and other managed services have not been tested and are not claimed to work. Azure Service Bus’s CBS token flow is not implemented. Client certificates (mTLS) and SASL OAUTHBEARER and GSSAPI are not supported.

The settings

For an inbound channel:

PropertyTransportMeaning
security.protocolkafkaPLAINTEXT (default), SSL, SASL_PLAINTEXT or SASL_SSL
sasl.mechanismkafkaPLAIN, SCRAM-SHA-256 or SCRAM-SHA-512
sasl.username, sasl.passwordkafkaSecret references
tls: trueamqpUse TLS. The default port becomes 5671
username, passwordamqpSecret references. SASL PLAIN is used when both are set
tls-ca-filebothPath to a PEM bundle, used in place of the system CAs
tls-ca-pembothSecret reference to PEM text, used in place of the system CAs
tls-insecure-skip-verifybothtrue turns off certificate and host-name checks

A credential, and tls-ca-pem, must be a reference as a whole value: env:NAME, secret:KEY, ${NAME}, ${secret:KEY}, a vendor store scheme the engine is built with, or k8s:secret/<name>#<key> (see References in connection properties). A literal value stops the channel at startup with SUTRA.STARTUP.REFUSED.HARDCODED_SECRET. tls-ca-file takes references like an address does; the switches (tls, security.protocol, sasl.mechanism, tls-insecure-skip-verify) stay literal.

With no CA setting, the broker is checked against the system CAs, including any set with SSL_CERT_FILE. Certificate and host-name checks are on unless you set tls-insecure-skip-verify: true, and the engine logs a warning whenever that setting applies.

A setting that could not take effect stops the channel at startup rather than being ignored: a CA without TLS, sasl.* without a SASL protocol, both CA settings at once, or SASL GSSAPI or OAUTHBEARER. Kafka’s kafka.* passthrough does not accept sasl.*, ssl.* or enable.ssl.certificate.verification.

Outbound sends use one engine-wide configuration per transport, set in the environment:

VariableMeaning
SUTRA_SINK_KAFKA_BOOTSTRAP_SERVERSThe brokers
SUTRA_SINK_KAFKA_SECURITY_PROTOCOLAs security.protocol
SUTRA_SINK_KAFKA_SASL_MECHANISM, _SASL_USERNAME, _SASL_PASSWORDAs the sasl.* properties. These hold values, so set them from your secret store
SUTRA_SINK_KAFKA_TLS_CA_FILE, SUTRA_SINK_KAFKA_TLS_INSECURE_SKIP_VERIFYTrust
SUTRA_SINK_AMQP_USERNAME, SUTRA_SINK_AMQP_PASSWORDDefault credentials
SUTRA_SINK_AMQP_TLS_CA_FILE, SUTRA_SINK_AMQP_TLS_INSECURE_SKIP_VERIFYTrust for amqp10s:// destinations
SUTRA_SINK_AMQP_OPERATION_TIMEOUT_MSThe bound on each broker round trip of a send (opening the connection with TLS and SASL, the session, the attach, the wait for the broker to accept the message, the close), default 30000. A broker that does not answer in time fails the attempt with the retryable SUTRA.OUTBOUND.AMQP.SEND_TIMEOUT and the outbox sends again; a send accepted just before its answer timed out can arrive twice, so receivers dedupe on sutra-outbox-key. Since 0.6.22.
SUTRA_SINK_AMQP_MAX_CONNECTIONSHow many connections the sink keeps open to one broker (scheme, host, port and credentials), default 32 (8 in 0.8.2), matching sutra.outbox.destination-concurrency so a broker destination’s sends do not queue for a connection. Each has one session and a sender link per target address; a send takes an idle one and, with all busy, waits for one for at most the operation timeout. A connection that fails a send is dropped and the send tried once more on a fresh one; one idle for two minutes is closed. Since 0.8.2.

Walkthrough: a message through both brokers

You need Docker, openssl, the sutra CLI and an empty working directory. This creates a private CA, starts both brokers with TLS and a login, and runs a package that answers each message. It was run as written, with sutra 0.6.21, apache/kafka:3.8.0, apache/activemq-artemis:latest-alpine and postgres:16-alpine; the output shown is from that run.

1. Create the certificates. The server certificate must name the host the engine connects to.

openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes -days 7 \
  -subj "/CN=demo broker CA" -keyout ca.key -out ca.pem
for broker in tlsdemo-kafka tlsdemo-artemis; do
  openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
    -subj "/CN=$broker" -keyout "$broker.key" -out "$broker.csr"
  openssl x509 -req -in "$broker.csr" -CA ca.pem -CAkey ca.key -CAcreateserial -days 7 \
    -extfile <(printf "subjectAltName=DNS:%s" "$broker") -out "$broker.crt"
done
chmod 644 ./*.key ./*.crt ca.pem
BROKER_PASSWORD=$(openssl rand -hex 16)
docker network create tlsdemo-net

The chmod lets the containers read these files. The keys are throwaway, so don’t reuse this step for real keys.

2. Start Kafka with a plaintext listener inside its container for the commands below, and a SASL_SSL listener for the engine. Save this as server.properties:

process.roles=broker,controller
node.id=1
controller.quorum.voters=1@localhost:9093
controller.listener.names=CONTROLLER
inter.broker.listener.name=INTERNAL
listeners=INTERNAL://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093,SECURE://0.0.0.0:9094
advertised.listeners=INTERNAL://localhost:9092,SECURE://tlsdemo-kafka:9094
listener.security.protocol.map=INTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT,SECURE:SASL_SSL
sasl.enabled.mechanisms=SCRAM-SHA-256
listener.name.secure.scram-sha-256.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required;
ssl.keystore.type=PEM
ssl.keystore.location=/certs/kafka-keystore.pem
log.dirs=/tmp/kraft-logs
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
cat tlsdemo-kafka.key tlsdemo-kafka.crt > kafka-keystore.pem
chmod 644 kafka-keystore.pem server.properties
docker run -d --name tlsdemo-kafka --network tlsdemo-net \
  -v "$PWD/server.properties:/certs/server.properties:ro" \
  -v "$PWD/kafka-keystore.pem:/certs/kafka-keystore.pem:ro" \
  --entrypoint /bin/sh apache/kafka:3.8.0 -c "
    /opt/kafka/bin/kafka-storage.sh format -t MkU3OEVBNTcwNTJENDM2Qk -c /certs/server.properties \
      --add-scram 'SCRAM-SHA-256=[name=orders-svc,password=$BROKER_PASSWORD]' &&
    exec /opt/kafka/bin/kafka-server-start.sh /certs/server.properties"
until docker logs tlsdemo-kafka 2>&1 | grep -q 'Kafka Server started'; do sleep 1; done
for topic in orders-in orders-out; do
  docker exec tlsdemo-kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 \
    --create --topic "$topic"
done

3. Start Artemis with an AMQPS acceptor.

openssl pkcs12 -export -in tlsdemo-artemis.crt -inkey tlsdemo-artemis.key -out artemis.p12 \
  -passout pass:changeit -name broker
chmod 644 artemis.p12
docker run -d --name tlsdemo-artemis --network tlsdemo-net \
  -v "$PWD/artemis.p12:/certs/artemis.p12:ro" \
  --entrypoint /bin/sh apache/activemq-artemis:latest-alpine -c "
    /opt/activemq-artemis/bin/artemis create --user orders-svc --password $BROKER_PASSWORD \
      --silent --require-login --http-host 0.0.0.0 --relax-jolokia . &&
    sed -i 's#</acceptors>#<acceptor name=\"amqps\">tcp://0.0.0.0:5671?sslEnabled=true;keyStorePath=/certs/artemis.p12;keyStorePassword=changeit;keyStoreType=PKCS12;protocols=AMQP;saslMechanisms=PLAIN</acceptor></acceptors>#' etc/broker.xml &&
    exec ./bin/artemis run"
until docker logs tlsdemo-artemis 2>&1 | grep -q 'Server is now active'; do sleep 1; done
for q in orders-in orders-out; do
  docker exec tlsdemo-artemis ./bin/artemis queue create --user orders-svc --password "$BROKER_PASSWORD" \
    --url tcp://localhost:61616 --name "$q" --address "$q" --anycast --durable --auto-create-address \
    --preserve-on-no-consumers --silent
done

The two queues are created up front, as anycast queues, so a message sent before anyone reads it is kept. Left to Artemis, an address the engine attaches to first is created as multicast, and the producer below would send to a queue no one reads.

4. Write the package. Create tls-relay/package.yaml:

labels:
  "tenant": "default"
  "module": "tls-relay"
  "version": "1.0.0"
engine:
  minContract: 1

Create tls-relay/channels.yaml:

channels:
  - name: kafka-in
    transport: kafka
    codec: json
    cloudevents-mode: none
    bootstrap.servers: tlsdemo-kafka:9094
    topic: orders-in
    group.id: tls-relay
    security.protocol: SASL_SSL
    sasl.mechanism: SCRAM-SHA-256
    sasl.username: env:BROKER_USER
    sasl.password: secret:BROKER_PASSWORD
    tls-ca-pem: secret:BROKER_CA
  - name: kafka-out
    direction: outbound
    transport: kafka
    bind: kafka://orders-out
    cloudevents-mode: none
  - name: amqp-in
    transport: amqp
    codec: json
    cloudevents-mode: none
    host: tlsdemo-artemis
    tls: true
    tls-ca-pem: secret:BROKER_CA
    username: env:BROKER_USER
    password: secret:BROKER_PASSWORD
    queue: orders-in
  - name: amqp-out
    direction: outbound
    transport: amqp
    bind: amqp10s://tlsdemo-artemis/orders-out
    cloudevents-mode: none

Create tls-relay/bpmn/tls-relay.bpmn, with one process per transport. Each builds an answer and sends it out. This is the Kafka one; add a copy for AMQP with its ids prefixed amqp, its process id amqp-relay, and the channels amqp-in and amqp-out:

<?xml version="1.0" encoding="UTF-8"?>
<bpmn:definitions xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL"
                  xmlns:q="urn:sutra:q:1.0" id="Defs"
                  targetNamespace="urn:sutra:module:tls-relay:1.0.0">
  <bpmn:process id="kafka-relay" isExecutable="true">
    <bpmn:extensionElements>
      <q:variables><q:variable name="responseBody" type="object"/></q:variables>
    </bpmn:extensionElements>
    <bpmn:startEvent id="kafkaStart">
      <bpmn:extensionElements><q:source channel="kafka-in"/></bpmn:extensionElements>
      <bpmn:outgoing>kafkaF1</bpmn:outgoing>
    </bpmn:startEvent>
    <bpmn:sequenceFlow id="kafkaF1" sourceRef="kafkaStart" targetRef="kafkaBuild"/>
    <bpmn:serviceTask id="kafkaBuild">
      <bpmn:incoming>kafkaF1</bpmn:incoming><bpmn:outgoing>kafkaF2</bpmn:outgoing>
      <bpmn:dataInputAssociation>
        <bpmn:assignment>
          <bpmn:from>{ref: payload.ref, status: "ANSWERED"}</bpmn:from>
          <bpmn:to>responseBody</bpmn:to>
        </bpmn:assignment>
      </bpmn:dataInputAssociation>
    </bpmn:serviceTask>
    <bpmn:sequenceFlow id="kafkaF2" sourceRef="kafkaBuild" targetRef="kafkaAnswer"/>
    <bpmn:sendTask id="kafkaAnswer">
      <bpmn:extensionElements><q:send channel="kafka-out"/></bpmn:extensionElements>
      <bpmn:incoming>kafkaF2</bpmn:incoming><bpmn:outgoing>kafkaF3</bpmn:outgoing>
    </bpmn:sendTask>
    <bpmn:sequenceFlow id="kafkaF3" sourceRef="kafkaAnswer" targetRef="kafkaEnd"/>
    <bpmn:endEvent id="kafkaEnd"><bpmn:incoming>kafkaF3</bpmn:incoming></bpmn:endEvent>
  </bpmn:process>
</bpmn:definitions>
mkdir -p deployments secrets
sutra lint tls-relay
sutra package tls-relay --out deployments
[WARN] SUTRA.CHANNEL.NO_SCHEMA — deployment default/tls-relay/1.0.0 (...): channel 'kafka-in' has 1 subscriber(s) but its codec (json) declares no message types — ...
[WARN] SUTRA.CHANNEL.NO_SCHEMA — deployment default/tls-relay/1.0.0 (...): channel 'amqp-in' has 1 subscriber(s) but its codec (json) declares no message types — ...
0 error(s), 2 warning(s)
packaged deployments/tls-relay.sutra (deploymentId dep-e890994c8e67a1c5b02e6db2)

The two warnings say the channels are schema-less, which is what this package intends.

5. Start the engine. The inbound channels read the password and the CA from the mounted secrets directory. The engine-wide sinks take theirs from the environment. The database role has no superuser or BYPASSRLS rights, because the engine refuses to start on one that does.

printf '%s' "$BROKER_PASSWORD" > secrets/BROKER_PASSWORD
cp ca.pem secrets/BROKER_CA
chmod -R a+rX deployments secrets
docker run -d --name tlsdemo-db --network tlsdemo-net -e POSTGRES_PASSWORD=dev-only postgres:16-alpine
until docker exec tlsdemo-db pg_isready -U postgres -h 127.0.0.1; do sleep 1; done
sleep 2
docker exec tlsdemo-db psql -U postgres \
  -c "CREATE ROLE sutra_engine LOGIN PASSWORD 'dev-only' NOSUPERUSER NOBYPASSRLS" \
  -c "ALTER SCHEMA public OWNER TO sutra_engine"
docker run -d --name tlsdemo-engine --network tlsdemo-net -p 127.0.0.1::8080 \
  -v "$PWD/deployments:/deployments:ro" -e SUTRA_DEPLOYMENTS_DIR=/deployments \
  -v "$PWD/secrets:/etc/sutra/secrets:ro" \
  -v "$PWD/ca.pem:/etc/sutra/broker-ca.pem:ro" \
  -e SUTRA_DATASOURCE_URL=postgres://tlsdemo-db:5432/postgres \
  -e SUTRA_DATASOURCE_USERNAME=sutra_engine -e SUTRA_DATASOURCE_PASSWORD=dev-only \
  -e BROKER_USER=orders-svc \
  -e SUTRA_SINK_KAFKA_BOOTSTRAP_SERVERS=tlsdemo-kafka:9094 \
  -e SUTRA_SINK_KAFKA_SECURITY_PROTOCOL=SASL_SSL \
  -e SUTRA_SINK_KAFKA_SASL_MECHANISM=SCRAM-SHA-256 \
  -e SUTRA_SINK_KAFKA_SASL_USERNAME=orders-svc \
  -e SUTRA_SINK_KAFKA_SASL_PASSWORD="$BROKER_PASSWORD" \
  -e SUTRA_SINK_KAFKA_TLS_CA_FILE=/etc/sutra/broker-ca.pem \
  -e SUTRA_SINK_AMQP_USERNAME=orders-svc \
  -e SUTRA_SINK_AMQP_PASSWORD="$BROKER_PASSWORD" \
  -e SUTRA_SINK_AMQP_TLS_CA_FILE=/etc/sutra/broker-ca.pem \
  ghcr.io/startr-trade/sutra:0.6.21
PORT=$(docker port tlsdemo-engine 8080 | head -1 | cut -d: -f2)
until curl -sf "http://127.0.0.1:$PORT/sutra/health/ready"; do sleep 1; done
{"status":"UP","checks":[{"name":"sutra-loader","status":"UP","data":{"deployments":1,"shards":1}}]}

6. Send a message through Kafka and read the answer.

echo '{"ref":"kafka-1"}' | docker exec -i tlsdemo-kafka /opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server localhost:9092 --topic orders-in
docker exec tlsdemo-kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic orders-out --from-beginning --max-messages 1 --timeout-ms 60000
{"ref":"kafka-1","status":"ANSWERED"}
Processed a total of 1 messages

The message went in and the answer came out over the SASL_SSL listener, with SCRAM-SHA-256 and the private CA; the commands above use the broker’s internal plaintext listener only to drive it.

7. Send a message through Artemis and read the answer.

docker exec tlsdemo-artemis ./bin/artemis producer --user orders-svc --password "$BROKER_PASSWORD" \
  --url tcp://localhost:61616 --protocol amqp --destination queue://orders-in --message-count 1 \
  --message '{"ref":"amqp-1"}'
docker exec tlsdemo-artemis ./bin/artemis browser --user orders-svc --password "$BROKER_PASSWORD" \
  --url tcp://localhost:61616 --destination queue://orders-out --message-count 1 --verbose
Producer orders-in, thread=0 Produced: 1 messages
...
Consumer ActiveMQQueue[orders-out], thread=0 browsed: 1 messages

The engine read the message over AMQPS (port 5671, SASL PLAIN, the private CA) and sent its answer to amqp10s://tlsdemo-artemis/orders-out. The engine’s log holds the broker password 0 times.

8. See a refusal. Restart the engine with a wrong password in secrets/BROKER_PASSWORD (and in the two SUTRA_SINK_*_PASSWORD variables), and read its log:

docker logs tlsdemo-engine 2>&1 | grep -E 'BROKER_(AUTH|TLS)_FAILED'
{"channel":"amqp-in","code":"SUTRA.INBOUND.AMQP.BROKER_AUTH_FAILED",...,"level":"WARN",...,"message":"amqp consumer unavailable — retrying in 1s: amqp channel 'amqp-in' could not connect to tlsdemo-artemis:5671: the broker refused the SASL credentials (outcome Auth)",...}
{"client":"channel 'kafka-in'","code":"SUTRA.INBOUND.KAFKA.BROKER_AUTH_FAILED",...,"level":"WARN",...,"message":"kafka channel 'kafka-in': sasl_ssl://tlsdemo-kafka:9094/bootstrap: SASL authentication error: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 ...",...}

Put the right password back and a certificate from another CA in secrets/BROKER_CA, restart, and the log reads:

{"client":"channel 'kafka-in'","code":"SUTRA.INBOUND.KAFKA.BROKER_TLS_FAILED",...,"message":"kafka channel 'kafka-in': sasl_ssl://tlsdemo-kafka:9094/bootstrap: SSL handshake failed: error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified ...",...}
{"channel":"amqp-in","code":"SUTRA.INBOUND.AMQP.BROKER_TLS_FAILED",...,"message":"amqp consumer unavailable — retrying in 1s: amqp channel 'amqp-in' could not connect to tlsdemo-artemis:5671: the TLS handshake failed: invalid peer certificate: UnknownIssuer",...}

Neither the wrong password nor the right one appeared anywhere in the log.

CodeMeaning
SUTRA.{INBOUND,OUTBOUND}.{KAFKA,AMQP}.BROKER_AUTH_FAILEDThe broker refused the credentials
SUTRA.{INBOUND,OUTBOUND}.{KAFKA,AMQP}.BROKER_TLS_FAILEDThe CA does not trust the broker’s certificate, the certificate does not name the host, or the trust settings could not be loaded

Neither message includes a password. Both consumers keep retrying and an outbound send stays in the outbox, so both recover once you correct the setting and restart the engine. The AMQP consumer backs off (1 s, 2 s, …). The Kafka consumer logs one line per attempt by its client, which retries quickly: with the wrong CA this run logged about 740 BROKER_TLS_FAILED lines in 20 seconds, so alert on the code rather than on the line count.

9. Clean up.

docker rm -f tlsdemo-engine tlsdemo-db tlsdemo-artemis tlsdemo-kafka
docker network rm tlsdemo-net

Connecting the engine to IBM MQ

The engine reaches IBM MQ through the queue manager’s AMQP 1.0 channel, with the amqp transport. There is no MQ client library in the engine: the queue manager speaks AMQP on a port of its own, and MQ applications and other queue managers see ordinary MQ messages on ordinary MQ queues.

What this was tested against. IBM MQ Advanced for Developers 10.0.1.0 in containers, on one machine: two queue managers joined by a sender/receiver channel pair over mutual TLS, local queues, a remote queue definition, a topic with administrative subscriptions, the AMQP channel in plain text and with TLS, SASL PLAIN, concurrent sends, and the queue manager stopped and started while messages were being sent. IBM MQ on z/OS, MQ appliances, IBM MQ as a service and the native MQI client have not been tested and are not claimed to work.

What MQ needs from a channel

MQ maps AMQP onto its own message model, and two of its rules decide how a channel is written:

  • An address without a capability is a topic. MQ reads an AMQP address as a topic string unless the link says it names a queue. Set address-type: queue on an inbound channel and ?type=queue on a destination. Without it a send goes to the topic of that name (with no subscriber it reaches no one, and a user without publish authority there is refused with AMQXR0102E), and an inbound channel subscribes to that topic instead of reading the queue.
  • The body section decides the MQ format. A message that comes over AMQP is stored as MQSTR when its body is one AMQP string, as MQFMT_NONE when it is one AMQP binary value, and as an embedded AMQP message (MQAMQP) otherwise, which includes the engine’s default data section. Add body=text for XML, JSON and other text, or body=binary for bytes, to every destination an MQ application reads. An engine reading the queue over AMQP gets the same bytes either way.

Two more rules the transport follows by itself since 0.7.0: every message names its destination in its to property, which MQ requires (AMQXR0032E otherwise), and every connection has its own container id, because MQ treats the container id as the client’s identity and closes the older of two connections that share one.

The settings

An inbound channel takes the amqp properties described in Connecting to a TLS and SASL broker, and one more:

PropertyMeaning
address-typequeue or topic, sent to the broker as the link’s capability. Unset sends none

A destination (amqp10:// or amqp10s://host[:port]/<address>) takes these query options. Each is off unless you set it, so an existing destination sends what it always did:

OptionValuesEffect
typequeue, topicThe link’s capability
durabletrue, falseThe message’s durable flag: MQ’s persistent / not persistent
ttl-msmillisecondsThe message’s time to live: MQ’s expiry, in tenths of a second
bodydata (default), binary, textdata is one AMQP data section, binary one AMQP binary value, text one AMQP string. With text a payload that is not UTF-8 is refused with SUTRA.OUTBOUND.AMQP.CONFIG_INVALID
application-propertiesall (default), nonenone sends no application properties: no sutra-outbox-key, no content-type, no headers. The outbox key is still the message-id
property-namesverbatim (default), identifieridentifier rewrites every application-property name to [A-Za-z_$][A-Za-z0-9_$]*: sutra-outbox-key becomes sutra_outbox_key, which the engine reads too. IBM MQ 10.0.1 kept names with - as they were; IBM documents earlier releases as refusing them, and JMS applications can read only identifier names

A value an option cannot take stops the send permanently with SUTRA.OUTBOUND.AMQP.CONFIG_INVALID.

Four headers carry AMQP message properties both ways. A received message has them set from the message, and an outbound message that sets them sends them as those properties rather than as application properties:

HeaderAMQP propertyIn MQ
x-amqp-message-idmessage-idInbound: MQ’s MsgId, 24 bytes as 48 hex digits. Outbound: kept in the AMQPMsgId property; MQ makes its own MsgId. The engine sends the outbox key when the header is unset
x-amqp-correlation-idcorrelation-idCorrelId
x-amqp-reply-toreply-toReplyToQ
x-amqp-subjectsubjectthe AMQPSubject property

How a message looks to MQ

What an MQ application reading the queue saw, from the queue manager’s own amqsbcg:

MQ fieldFrom the engine
MsgTypeMQMT_DATAGRAM (8), always
FormatMQSTR with body=text, blank (MQFMT_NONE) with body=binary, MQAMQP with the default data section
CodedCharSetId1208, always; the text is the engine’s UTF-8 bytes unchanged
Persistence1 with durable=true, 0 otherwise: IBM documents a message without the flag as not persistent, whatever the queue’s DEFPSIST
Expiryttl-ms in tenths of a second, unlimited (-1) without it
MsgIdmade by the queue manager; the engine’s message id is kept in the AMQPMsgId property
CorrelIdthe x-amqp-correlation-id text, padded with zero bytes to 24
ReplyToQ, ReplyToQMgrthe x-amqp-reply-to queue, and the queue manager the engine is connected to
UserIdentifierthe user the AMQP channel authenticated
PutApplNameIBM MQ AMQP Support 10.0.1.0

Message properties and MQRFH2. The AMQP header and properties become MQ message properties in an mq_amqp folder, a reply-to also in a jms folder, and the application properties in usr. An application that reads a queue with the default PROPCTL(COMPAT) and does not ask for properties gets them as an MQRFH2 header in front of the body, with Format MQHRF2, when the message has a usr or jms folder; the mq_amqp folder alone is not handed to it. So for a reader that expects a plain message, either send it with application-properties=none and no x-amqp-reply-to, or set PROPCTL(NONE) on the queue it reads. The sender channel’s own PROPCTL does not help: it applies only to queue managers of version 6 and earlier.

Going the other way, a message an MQ application put arrives with its MsgId in x-amqp-message-id (48 hex digits), its ReplyToQ in x-amqp-reply-to, and an MQSTR body as text. The MsgId is also the idempotency key when the message has no sutra-outbox-key.

When the queue manager goes away

An inbound channel reconnects by itself, backing off from 1 s to 30 s, so it is attached again at most about 30 s after the queue manager is back. An outbound send that cannot reach the queue manager fails as retryable and the outbox sends it again. Both log the outage once when it starts and once when it ends, with its start, end and length:

CodeLogged
SUTRA.INBOUND.AMQP.OUTAGE_STARTEDWARN, when a consumer loses its connection or cannot open one; outage_started
SUTRA.INBOUND.AMQP.OUTAGE_ENDEDINFO, when it is attached again; outage_started, outage_ended, outage_ms
SUTRA.OUTBOUND.AMQP.OUTAGE_STARTEDWARN, per broker (host:port), at the first send that cannot connect
SUTRA.OUTBOUND.AMQP.OUTAGE_ENDEDINFO, at the first send that connects again

The walkthrough below shows both lines from a real restart.

The same outage can also reach the package as messages, so a deployment can record its own outage windows: set SUTRA_CONNECTIVITY_EVENTS_CHANNEL and declare that channel (see Broker connectivity events and Broker outages as messages).

An AMQP channel that was running when the queue manager stopped starts again with it. What MQ keeps across the restart is MQ’s rule, not the engine’s: persistent messages (durable=true) are kept, and so are non-persistent ones on a queue defined with NPMCLASS(HIGH) after a normal shutdown; other non-persistent messages are lost. A message the engine had received but not yet settled is delivered again after the reconnect, and the engine’s inbox drops the duplicate.

What the AMQP channel cannot do

These are limits of MQ’s AMQP support, not settings:

  • MQPER_PERSISTENCE_AS_Q_DEF. A message is persistent or not, by durable; the queue’s default does not apply. Set durable to match the queues a destination writes to.
  • ReplyToQMgr. It is always the queue manager the engine is connected to, and the engine addresses only queues that queue manager can open: its local queues, aliases and remote queue definitions. To answer a ReplyToQ on another queue manager, define a remote queue definition for it.
  • A binary CorrelId or MsgId. An outbound x-amqp-correlation-id is text, at most 24 bytes as MQ keeps it. The engine cannot set a MsgId: the queue manager makes one.
  • Other descriptor fields. Report, Feedback, GroupId, MsgSeqNumber, Offset and MsgFlags keep their defaults, and segmentation and message groups are not available.
  • One AMQP session per connection. The engine opens one connection, with one session, per inbound channel. Outbound it keeps a pool of up to 32 connections per queue manager (8 before 0.8.3) (SUTRA_SINK_AMQP_MAX_CONNECTIONS), each with one session and a sender link per queue, which fits. Those connections stay open (closed after two minutes idle), so each is a running instance of the AMQP channel: allow for the engine’s replicas times the bound in the channel’s instance limits.
  • Message size. An AMQP channel accepts messages up to its MAXMSGL (4 MB unless set); raise it, and the queues’, for larger messages.

Where a peer needs exact control of the message descriptor, the AMQP channel is the wrong road: that takes an MQI client, which the engine does not have.

Walkthrough: a message through IBM MQ

You need Docker, openssl, the sutra CLI and an empty working directory. This builds an IBM MQ developer image with AMQP, gives the queue manager a certificate, defines its AMQP channel, queues and a topic, runs a package that answers each message, and restarts the queue manager under it. It was run as written with IBM MQ Advanced for Developers 10.0.1.0, postgres:16-alpine and the engine built from this release; the output shown is from that run.

1. Build an IBM MQ image with AMQP. The published developer image leaves out MQ’s AMQP component. Copy it in from the developer archive of the same release. The archive is about 550 MB and IBM’s download site can be slow on one connection; only its small amqp/ directory is used:

curl -fLO https://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/messaging/mqadv/10.0.1.0-IBM-MQ-Advanced-for-Developers-Non-Install-LinuxX64.tar.gz
mkdir -p mq-amqp
tar xzf 10.0.1.0-IBM-MQ-Advanced-for-Developers-Non-Install-LinuxX64.tar.gz -C mq-amqp amqp
printf 'FROM icr.io/ibm-messaging/mq:10.0.1.0-r1\nCOPY --chown=1001:0 amqp/ /opt/mqm/amqp/\n' > mq-amqp/Dockerfile
docker build -q -t mq-amqp:10.0.1.0 mq-amqp

Running the image means accepting the IBM MQ Advanced for Developers licence (LICENSE=accept below). It is for development and testing only.

2. Create the certificates. The queue manager’s certificate must name the host the engine connects to. The AMQP channel reads it from a PKCS#12 key repository with a password; it cannot use the CMS repository the image builds from /etc/mqm/pki.

openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes -days 7 \
  -subj "/CN=demo MQ CA" -keyout ca.key -out ca.pem 2>/dev/null
openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
  -subj "/CN=mqdemo-qm" -keyout qm.key -out qm.csr 2>/dev/null
openssl x509 -req -in qm.csr -CA ca.pem -CAkey ca.key -CAcreateserial -days 7 \
  -extfile <(printf "subjectAltName=DNS:mqdemo-qm") -out qm.crt 2>/dev/null
KEYRING_PASSWORD=$(openssl rand -hex 16)
MQ_PASSWORD=$(openssl rand -hex 16)
mkdir -p tls
openssl pkcs12 -export -name qm -in qm.crt -inkey qm.key -certfile ca.pem \
  -out tls/qm.p12 -passout "pass:$KEYRING_PASSWORD"
chmod 644 tls/qm.p12 ca.pem
docker network create mqdemo-net

3. Define the queue manager’s objects. An AMQP channel with TLS that requires a user and password, an inbound and an outbound queue, a topic, and an administrative subscription that copies every publication to a queue. The key repository is set before the AMQP service starts, because the service reads it then.

cat > qm.mqsc <<EOF
ALTER QMGR SSLKEYR('/tls/qm.p12') KEYRPWD('$KEYRING_PASSWORD') CERTLABL('qm')
REFRESH SECURITY TYPE(SSL)
DEFINE CHANNEL('ENGINE.AMQPS') CHLTYPE(AMQP) PORT(5671) MCAUSER('app') +
       SSLCIPH('ANY_TLS12_OR_HIGHER') SSLCAUTH(OPTIONAL) REPLACE
SET CHLAUTH('ENGINE.AMQPS') TYPE(ADDRESSMAP) ADDRESS('*') USERSRC(CHANNEL) CHCKCLNT(REQUIRED) +
    ACTION(REPLACE)
DEFINE QLOCAL('ORDERS.IN') REPLACE
DEFINE QLOCAL('ORDERS.OUT') REPLACE
DEFINE TOPIC('ORDERS.EVENTS') TOPICSTR('orders/events') REPLACE
DEFINE QLOCAL('ORDERS.EVENTS.COPY') REPLACE
DEFINE SUB('ORDERS.EVENTS.COPY') TOPICSTR('orders/events') DEST('ORDERS.EVENTS.COPY') REPLACE
SET AUTHREC PROFILE('ORDERS.**') PRINCIPAL('app') OBJTYPE(QUEUE) AUTHADD(BROWSE,GET,INQ,PUT)
SET AUTHREC PROFILE('ORDERS.**') PRINCIPAL('app') OBJTYPE(TOPIC) AUTHADD(PUB,SUB)
ALTER SERVICE(SYSTEM.AMQP.SERVICE) CONTROL(QMGR)
START SERVICE(SYSTEM.AMQP.SERVICE)
EOF
chmod 644 qm.mqsc

4. Start the queue manager, then the AMQP channel. The channel can start only once the AMQP service is up, so retry until it runs. After a restart it starts again by itself.

docker run -d --name mqdemo-qm --network mqdemo-net \
  -e LICENSE=accept -e MQ_QMGR_NAME=QM1 -e MQ_APP_PASSWORD="$MQ_PASSWORD" \
  -v "$PWD/qm.mqsc:/etc/mqm/30-demo.mqsc:ro" -v "$PWD/tls:/tls:ro" \
  mq-amqp:10.0.1.0
until docker logs mqdemo-qm 2>&1 | grep -q 'Started queue manager'; do sleep 1; done
until docker exec mqdemo-qm bash -c "echo \"START CHANNEL('ENGINE.AMQPS')
DISPLAY CHSTATUS('ENGINE.AMQPS') CHLTYPE(AMQP)\" | runmqsc QM1" | grep -q 'STATUS(RUNNING)'; do
  sleep 1
done

5. Write the package in mq-relay/ (mkdir -p mq-relay/bpmn). mq-relay/package.yaml:

labels:
  "tenant": "default"
  "module": "mq-relay"
  "version": "1.0.0"
engine:
  minContract: 1

mq-relay/channels.yaml. The inbound channel names its queue as a queue; both outbound destinations send the payload as text and no application properties, so an MQ application reading the default queues gets a plain MQSTR message:

channels:
  - name: mq-in
    transport: amqp
    codec: json
    cloudevents-mode: none
    host: mqdemo-qm
    tls: true
    tls-ca-pem: secret:MQ_CA
    username: env:MQ_USER
    password: secret:MQ_PASSWORD
    queue: ORDERS.IN
    address-type: queue
  - name: mq-out
    direction: outbound
    transport: amqp
    bind: amqp10s://mqdemo-qm/ORDERS.OUT?type=queue&body=text&application-properties=none
    cloudevents-mode: none
  - name: mq-events
    direction: outbound
    transport: amqp
    bind: amqp10s://mqdemo-qm/orders/events?type=topic&body=text&application-properties=none
    cloudevents-mode: none

mq-relay/bpmn/mq-relay.bpmn answers each message, then announces it on the topic:

<?xml version="1.0" encoding="UTF-8"?>
<bpmn:definitions xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL"
                  xmlns:q="urn:sutra:q:1.0" id="Defs"
                  targetNamespace="urn:sutra:module:mq-relay:1.0.0">
  <bpmn:process id="mq-relay" isExecutable="true">
    <bpmn:extensionElements>
      <q:variables>
        <q:variable name="responseBody" type="object"/>
      </q:variables>
    </bpmn:extensionElements>
    <bpmn:startEvent id="start">
      <bpmn:extensionElements><q:source channel="mq-in"/></bpmn:extensionElements>
      <bpmn:outgoing>f1</bpmn:outgoing>
    </bpmn:startEvent>
    <bpmn:sequenceFlow id="f1" sourceRef="start" targetRef="build"/>
    <bpmn:serviceTask id="build">
      <bpmn:incoming>f1</bpmn:incoming><bpmn:outgoing>f2</bpmn:outgoing>
      <bpmn:dataInputAssociation>
        <bpmn:assignment>
          <bpmn:from>{ref: payload.ref, status: "ACCEPTED"}</bpmn:from>
          <bpmn:to>responseBody</bpmn:to>
        </bpmn:assignment>
      </bpmn:dataInputAssociation>
    </bpmn:serviceTask>
    <bpmn:sequenceFlow id="f2" sourceRef="build" targetRef="answer"/>
    <bpmn:sendTask id="answer">
      <bpmn:extensionElements><q:send channel="mq-out"/></bpmn:extensionElements>
      <bpmn:incoming>f2</bpmn:incoming><bpmn:outgoing>f3</bpmn:outgoing>
    </bpmn:sendTask>
    <bpmn:sequenceFlow id="f3" sourceRef="answer" targetRef="announce"/>
    <bpmn:sendTask id="announce">
      <bpmn:extensionElements><q:send channel="mq-events"/></bpmn:extensionElements>
      <bpmn:incoming>f3</bpmn:incoming><bpmn:outgoing>f4</bpmn:outgoing>
    </bpmn:sendTask>
    <bpmn:sequenceFlow id="f4" sourceRef="announce" targetRef="end"/>
    <bpmn:endEvent id="end"><bpmn:incoming>f4</bpmn:incoming></bpmn:endEvent>
  </bpmn:process>
</bpmn:definitions>
mkdir -p deployments secrets
sutra lint mq-relay
sutra package mq-relay --out deployments
[WARN] SUTRA.CHANNEL.NO_SCHEMA — deployment default/mq-relay/1.0.0 (...): channel 'mq-in' has 1 subscriber(s) but its codec (json) declares no message types — ...
0 error(s), 1 warning(s)
packaged deployments/mq-relay.sutra (deploymentId dep-b95d811099f0245b07e2d9c1)

6. Start the engine. The inbound channel reads the password and the CA from the mounted secrets directory; the outbound sends take theirs from the environment.

printf '%s' "$MQ_PASSWORD" > secrets/MQ_PASSWORD
cp ca.pem secrets/MQ_CA
chmod -R a+rX deployments secrets
docker run -d --name mqdemo-db --network mqdemo-net -e POSTGRES_PASSWORD=dev-only postgres:16-alpine
until docker exec mqdemo-db pg_isready -U postgres -h 127.0.0.1; do sleep 1; done
sleep 2
docker exec mqdemo-db psql -U postgres \
  -c "CREATE ROLE sutra_engine LOGIN PASSWORD 'dev-only' NOSUPERUSER NOBYPASSRLS" \
  -c "ALTER SCHEMA public OWNER TO sutra_engine"
docker run -d --name mqdemo-engine --network mqdemo-net -p 127.0.0.1::8080 \
  -v "$PWD/deployments:/deployments:ro" -e SUTRA_DEPLOYMENTS_DIR=/deployments \
  -v "$PWD/secrets:/etc/sutra/secrets:ro" \
  -v "$PWD/ca.pem:/etc/sutra/mq-ca.pem:ro" \
  -e SUTRA_DATASOURCE_URL=postgres://mqdemo-db:5432/postgres \
  -e SUTRA_DATASOURCE_USERNAME=sutra_engine -e SUTRA_DATASOURCE_PASSWORD=dev-only \
  -e MQ_USER=app \
  -e SUTRA_SINK_AMQP_USERNAME=app \
  -e SUTRA_SINK_AMQP_PASSWORD="$MQ_PASSWORD" \
  -e SUTRA_SINK_AMQP_TLS_CA_FILE=/etc/sutra/mq-ca.pem \
  ghcr.io/startr-trade/sutra:0.8.2
PORT=$(docker port mqdemo-engine 8080 | head -1 | cut -d: -f2)
until curl -sf "http://127.0.0.1:$PORT/sutra/health/ready"; do sleep 1; done
{"status":"UP","checks":[{"name":"sutra-loader","status":"UP","data":{"deployments":1,"shards":1}}]}

7. Put a message as an MQ application would, and read the answer and the announcement.

docker exec mqdemo-qm bash -c \
  "printf '%s\n\n' '{\"ref\":\"mq-1\"}' | /opt/mqm/samp/bin/amqsput ORDERS.IN QM1"
until [ "$(docker exec mqdemo-qm bash -c "echo \"DISPLAY QLOCAL('ORDERS.OUT') CURDEPTH\" | runmqsc QM1" | grep -o 'CURDEPTH([0-9]*)')" = "CURDEPTH(1)" ]; do sleep 1; done
docker exec mqdemo-qm /opt/mqm/samp/bin/amqsbcg ORDERS.OUT QM1 | grep -E "Format|CodedCharSetId"
docker exec mqdemo-qm /opt/mqm/samp/bin/amqsget ORDERS.OUT QM1
until [ "$(docker exec mqdemo-qm bash -c "echo \"DISPLAY QLOCAL('ORDERS.EVENTS.COPY') CURDEPTH\" | runmqsc QM1" | grep -o 'CURDEPTH([0-9]*)')" = "CURDEPTH(1)" ]; do sleep 1; done
docker exec mqdemo-qm /opt/mqm/samp/bin/amqsget ORDERS.EVENTS.COPY QM1
Sample AMQSPUT0 start
target queue is ORDERS.IN
Sample AMQSPUT0 end
  Encoding : 273  CodedCharSetId : 1208
  Format : 'MQSTR   '
Sample AMQSGET0 start
message <{"ref":"mq-1","status":"ACCEPTED"}>
no more messages
Sample AMQSGET0 end
Sample AMQSGET0 start
message <{"ref":"mq-1","status":"ACCEPTED"}>
no more messages
Sample AMQSGET0 end

amqsput put an MQSTR message; the engine read it over the AMQP channel with TLS and SASL PLAIN and answered on ORDERS.OUT as MQSTR in CCSID 1208, with no MQRFH2 in front of it. Its one publication to orders/events reached ORDERS.EVENTS.COPY through the subscription.

8. Restart the queue manager and send another message once it is back:

docker restart mqdemo-qm
until [ "$(docker logs mqdemo-qm 2>&1 | grep -c 'Started queue manager')" -ge 2 ]; do sleep 1; done
docker exec mqdemo-qm bash -c \
  "printf '%s\n\n' '{\"ref\":\"mq-2\"}' | /opt/mqm/samp/bin/amqsput ORDERS.IN QM1"
until [ "$(docker exec mqdemo-qm bash -c "echo \"DISPLAY QLOCAL('ORDERS.OUT') CURDEPTH\" | runmqsc QM1" | grep -o 'CURDEPTH([0-9]*)')" = "CURDEPTH(1)" ]; do sleep 1; done
docker exec mqdemo-qm /opt/mqm/samp/bin/amqsget ORDERS.OUT QM1
docker logs mqdemo-engine 2>&1 | grep -E 'OUTAGE_(STARTED|ENDED)'
Sample AMQSPUT0 start
target queue is ORDERS.IN
Sample AMQSPUT0 end
Sample AMQSGET0 start
message <{"ref":"mq-2","status":"ACCEPTED"}>
no more messages
Sample AMQSGET0 end
{"channel":"mq-in","code":"SUTRA.INBOUND.AMQP.OUTAGE_STARTED","outage_started":"2026-11-19T04:24:51.086333388Z",...,"level":"WARN",...,"message":"amqp consumer stream failed — reconnecting",...}
{"channel":"mq-in","code":"SUTRA.INBOUND.AMQP.OUTAGE_ENDED","outage_started":"2026-11-19T04:24:51.086333388Z","outage_ended":"2026-11-19T04:25:06.679943301Z","outage_ms":15593,...,"level":"INFO",...,"message":"amqp consumer reconnected after an outage",...}

The consumer lost the queue manager when it stopped and was attached again 15.6 s later, by itself. Part of that is the queue manager stopping and starting; the rest is the consumer’s backoff, which waits up to 30 s between attempts, so the gap differs from run to run (31.7 s in an earlier run). The engine’s log held the queue manager password 0 times.

9. Clean up.

docker rm -f mqdemo-engine mqdemo-db mqdemo-qm
docker network rm mqdemo-net

Broker outages as messages

Since 0.8.0. The engine logs every broker outage it sees (see When the queue manager goes away). With sutra.engine.connectivity-events.channel set, the same outage also reaches the deployment that owns the channel, as a JSON message on a channel of that name, so a process can record its own outage windows, alert, or hold work. The keys and the message are described in Broker connectivity events.

This walkthrough runs an AMQP 1.0 broker (Apache ActiveMQ Artemis), a package that consumes one queue and sends to another, and a small HTTP server that prints what the package records. It stops the broker, sends while it is down, and starts it again. You need Docker, curl, the sutra CLI and an empty working directory. It was run as written with the engine built from this release, apache/activemq-artemis:latest-alpine, python:3.12-alpine and postgres:16-alpine; the output shown is from that run.

1. Start the broker and a capture server

capture.py prints the body of every POST it receives:

from http.server import BaseHTTPRequestHandler, HTTPServer

class Capture(BaseHTTPRequestHandler):
    def do_POST(self):
        body = self.rfile.read(int(self.headers.get("Content-Length", 0)))
        print(body.decode().strip(), flush=True)
        self.send_response(204)
        self.end_headers()

    def log_message(self, *args):
        pass

HTTPServer(("", 8000), Capture).serve_forever()
docker network create outages-net
docker run -d --name outages-broker --network outages-net \
  -e ARTEMIS_USER=demo -e ARTEMIS_PASSWORD=demo-only \
  apache/activemq-artemis:latest-alpine
until docker logs outages-broker 2>&1 | grep -q 'Server is now active'; do sleep 1; done
docker run -d --name outages-capture --network outages-net \
  -v "$PWD/capture.py:/capture.py:ro" python:3.12-alpine python -u /capture.py

2. Write the package

outages/package.yaml:

labels:
  "tenant": "default"
  "module": "outages"
  "version": "1.0.0"
engine:
  minContract: 1

outages/channels.yaml. orders-in and orders-out are the AMQP channels whose outages are reported. connectivity-in is the channel the events arrive on: an in-process intake with the JSON codec. record-out sends each event to the capture server:

channels:
  - name: orders-in
    transport: amqp
    codec: json
    cloudevents-mode: none
    host: outages-broker
    username: env:BROKER_USER
    password: env:BROKER_PASSWORD
    queue: orders.in
  - name: orders-out
    direction: outbound
    transport: amqp
    bind: amqp10://outages-broker/orders.out
    cloudevents-mode: none
  - name: emit
    transport: http
    bind: "POST /channels/emit"
    codec: json
    auth:
      scheme: apikey
      apikey:
        value: env:OUTAGES_API_KEY
        header: X-Api-Key
  - name: connectivity-in
    transport: local
    codec: json
  - name: record-out
    direction: outbound
    transport: http
    bind: "http://outages-capture:8000/connectivity"

outages/templates/out.hbs:

{"n":"{{payload.n}}"}

outages/templates/record.hbs copies the event’s fields:

{"code":"{{payload.code}}","channel":"{{payload.channel}}","direction":"{{payload.direction}}","state":"{{payload.state}}","startedAt":"{{payload.startedAt}}"{{#if payload.endedAt}},"endedAt":"{{payload.endedAt}}","ms":{{payload.ms}}{{/if}}}

outages/bpmn/outages.bpmn:

<?xml version="1.0" encoding="UTF-8"?>
<bpmn:definitions xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL"
                  xmlns:q="urn:sutra:q:1.0"
                  targetNamespace="urn:sutra:module:outages:1.0.0">
  <!-- consumes orders-in, so the intake is attached to the broker -->
  <bpmn:process id="listen" isExecutable="true">
    <bpmn:startEvent id="S1">
      <bpmn:extensionElements><q:source channel="orders-in" name="payload"/></bpmn:extensionElements>
    </bpmn:startEvent>
    <bpmn:sequenceFlow id="l1" sourceRef="S1" targetRef="E1"/>
    <bpmn:endEvent id="E1"/>
  </bpmn:process>

  <!-- POST /channels/emit sends one message on orders-out -->
  <bpmn:process id="emit" isExecutable="true">
    <bpmn:startEvent id="S2">
      <bpmn:extensionElements><q:source channel="emit" name="payload"/></bpmn:extensionElements>
    </bpmn:startEvent>
    <bpmn:sequenceFlow id="e1" sourceRef="S2" targetRef="Send"/>
    <bpmn:serviceTask id="Send" implementation="out.hbs">
      <bpmn:extensionElements>
        <q:send channel="orders-out" mode="native" contentType="application/json"/>
      </bpmn:extensionElements>
    </bpmn:serviceTask>
    <bpmn:sequenceFlow id="e2" sourceRef="Send" targetRef="E2"/>
    <bpmn:endEvent id="E2"/>
  </bpmn:process>

  <!-- each connectivity event starts an instance, which records it -->
  <bpmn:process id="record" isExecutable="true">
    <bpmn:startEvent id="S3">
      <bpmn:extensionElements><q:source channel="connectivity-in" name="payload"/></bpmn:extensionElements>
    </bpmn:startEvent>
    <bpmn:sequenceFlow id="r1" sourceRef="S3" targetRef="Record"/>
    <bpmn:serviceTask id="Record" implementation="record.hbs">
      <bpmn:extensionElements>
        <q:send channel="record-out" mode="native" contentType="application/json"/>
      </bpmn:extensionElements>
    </bpmn:serviceTask>
    <bpmn:sequenceFlow id="r2" sourceRef="Record" targetRef="E3"/>
    <bpmn:endEvent id="E3"/>
  </bpmn:process>
</bpmn:definitions>
sutra lint outages
mkdir -p deployments
sutra package outages --out deployments
chmod -R a+rX deployments

Lint reports 0 errors and 11 warnings: three SUTRA.CHANNEL.NO_SCHEMA and eight SUTRA.CONFIG.TEMPLATE.FIELD_UNVERIFIABLE, because the JSON codec declares no schema.

3. Start the engine with the events channel set

SUTRA_CONNECTIVITY_EVENTS_CHANNEL=connectivity-in names the channel. A deployment that declares no channel of that name receives nothing.

docker run -d --name outages-db --network outages-net -e POSTGRES_PASSWORD=dev-only postgres:16-alpine
until docker exec outages-db pg_isready -U postgres -h 127.0.0.1; do sleep 1; done
sleep 2
docker exec outages-db psql -U postgres \
  -c "CREATE ROLE sutra_engine LOGIN PASSWORD 'dev-only' NOSUPERUSER NOBYPASSRLS" \
  -c "ALTER SCHEMA public OWNER TO sutra_engine"
docker run -d --name outages-engine --network outages-net -p 127.0.0.1::8080 \
  -v "$PWD/deployments:/deployments:ro" -e SUTRA_DEPLOYMENTS_DIR=/deployments \
  -e SUTRA_DATASOURCE_URL=postgres://outages-db:5432/postgres \
  -e SUTRA_DATASOURCE_USERNAME=sutra_engine -e SUTRA_DATASOURCE_PASSWORD=dev-only \
  -e SUTRA_CONNECTIVITY_EVENTS_CHANNEL=connectivity-in \
  -e BROKER_USER=demo -e BROKER_PASSWORD=demo-only \
  -e SUTRA_SINK_AMQP_USERNAME=demo -e SUTRA_SINK_AMQP_PASSWORD=demo-only \
  -e OUTAGES_API_KEY=demo-key \
  ghcr.io/startr-trade/sutra:0.8.3
PORT=$(docker port outages-engine 8080 | head -1 | cut -d: -f2)
until curl -sf "http://127.0.0.1:$PORT/sutra/health/ready"; do sleep 1; done
{"status":"UP","checks":[{"name":"sutra-loader","status":"UP","data":{"deployments":1,"shards":1}}]}

Send one message while the broker is up, so the send channel has connected. Nothing is recorded:

curl -s -o /dev/null -w '%{http_code}\n' -X POST "localhost:$PORT/channels/emit" -H 'X-Api-Key: demo-key' \
  -H 'Content-Type: application/json' -d '{"n":"1"}'
sleep 3
docker logs outages-capture
200

4. Stop the broker, send during the outage, start it again

docker stop outages-broker
until docker logs outages-capture 2>&1 | grep -q '"orders-in".*"down"'; do sleep 1; done
curl -s -o /dev/null -w '%{http_code}\n' -X POST "localhost:$PORT/channels/emit" -H 'X-Api-Key: demo-key' \
  -H 'Content-Type: application/json' -d '{"n":"2"}'
until docker logs outages-capture 2>&1 | grep -q '"orders-out".*"down"'; do sleep 1; done
sleep 5
docker start outages-broker
until [ "$(docker logs outages-capture 2>&1 | grep -c '"up"')" -ge 2 ]; do sleep 1; done
docker logs outages-capture

The send during the outage is accepted (200): it waits in the outbox and goes out once the broker is back. What the package recorded:

{"code":"SUTRA.INBOUND.AMQP.OUTAGE_STARTED","channel":"orders-in","direction":"inbound","state":"down","startedAt":"2026-11-19T09:15:09.146540438Z"}
{"code":"SUTRA.OUTBOUND.AMQP.OUTAGE_STARTED","channel":"orders-out","direction":"outbound","state":"down","startedAt":"2026-11-19T09:15:09.521903086Z"}
{"code":"SUTRA.OUTBOUND.AMQP.OUTAGE_ENDED","channel":"orders-out","direction":"outbound","state":"up","startedAt":"2026-11-19T09:15:09.521903086Z","endedAt":"2026-11-19T09:15:21.185206554Z","ms":11663}
{"code":"SUTRA.INBOUND.AMQP.OUTAGE_ENDED","channel":"orders-in","direction":"inbound","state":"up","startedAt":"2026-11-19T09:15:09.146540438Z","endedAt":"2026-11-19T09:15:24.169169142Z","ms":15022}

Two windows, one per channel. The intake’s started when it lost its connection and ended when it was attached again; the send channel’s started at the send that could not connect and ended at the outbox’s retry that reached the broker. Each up repeats its startedAt, so the two messages of a window pair up. The engine’s own log has the same four changes with the same codes and times:

docker logs outages-engine 2>&1 | grep -E 'OUTAGE_(STARTED|ENDED)'
{"channel":"orders-in","code":"SUTRA.INBOUND.AMQP.OUTAGE_STARTED","outage_started":"2026-11-19T09:15:09.146540438Z",...,"level":"WARN",...}
{"code":"SUTRA.OUTBOUND.AMQP.OUTAGE_STARTED","broker":"outages-broker:5672","destination":"amqp10://outages-broker/orders.out","outage_started":"2026-11-19T09:15:09.521911293Z",...,"level":"WARN",...}
{"code":"SUTRA.OUTBOUND.AMQP.OUTAGE_ENDED","broker":"outages-broker:5672","destination":"amqp10://outages-broker/orders.out","outage_started":"2026-11-19T09:15:09.521911293Z","outage_ended":"2026-11-19T09:15:21.185210334Z","outage_ms":11663,...,"level":"INFO",...}
{"channel":"orders-in","code":"SUTRA.INBOUND.AMQP.OUTAGE_ENDED","outage_started":"2026-11-19T09:15:09.146540438Z","outage_ended":"2026-11-19T09:15:24.169169142Z","outage_ms":15022,...,"level":"INFO",...}

How long each window lasts depends on the broker’s restart and on the intake’s reconnect backoff, which waits up to 30 s between attempts, so it differs from run to run.

5. Clean up

docker rm -f outages-engine outages-db outages-broker outages-capture
docker network rm outages-net

Tuning for throughput

Since 0.6.5. This page shows how to measure what one engine and one PostgreSQL complete per second on your hardware, and which settings move it, without giving up durability: every number here was taken with fsync, synchronous_commit and full_page_writes on.

What limits a durable engine

An instance’s state is safe only once PostgreSQL has flushed its write-ahead log to disk, and a disk flushes a limited number of times per second. A consumer NVMe drive manages about 300; a server SSD with a power-loss-protected cache manages tens of thousands. Measure yours with PostgreSQL’s own tool:

/usr/lib/postgresql/18/bin/pg_test_fsync -s 2 -f /path/on/the/database/disk/testfile

What decides throughput is how many commits share each flush. The engine helps in three ways:

  • it commits only what must be durable before anything is acknowledged: a step’s state, its aliases, its waits and its outbound messages, in one transaction. Bookkeeping that a crash cannot hurt — an ownership claim, the outbox’s row locks, the delete of an already delivered row — does not wait for its own flush, and the inbox dedup record rides the step’s own transaction;
  • it delivers outbound messages on several workers at once (sutra.outbox.workers), deletes delivered rows in batches, and hands local:// deliveries straight from the sender’s commit to a worker;
  • with several execution lanes (sutra.engine.shards), several steps commit at the same time, and PostgreSQL flushes them together.

So on a slow disk, add lanes: each lane is one more commit that can share a flush.

What may skip its own flush, and why

The server keeps synchronous_commit = on. Three kinds of transaction set synchronous_commit = off for themselves alone, and only because a crash that loses them can neither lose an acknowledged state nor invent one:

TransactionWhat it writesWhy it may skip its flush
instance ownership claimwhich engine is resuming an instancea claim held by an engine that crashed is void anyway; the step that follows commits synchronously
outbox claim and its lease renewalrow locks, and a lease: the claimed rows’ due time moved 30 s aheada lost lease only makes the rows claimable again, so they are sent again: at-least-once, deduplicated on the idempotency key
delete of delivered outbox rowsremoves rows already sentif it is lost, the row is sent again: outbound delivery has always been at-least-once, and receivers deduplicate on the idempotency key

The inbox dedup record of a delivery is not on the list: it is written in the same transaction as the step that processes the delivery, so it exists exactly when the step’s effects do.

Everything else — every step’s state, aliases, waits and outbound rows, every data-store write, every retry or poison mark — commits synchronously, as before. The tier-2 suite kills the engine, and separately the database, in the middle of a load and checks that every accepted request still completes exactly once.

The workload

The repository carries a workload that exercises the whole durable path, in tools/sutra-load-test:

  • fixtures/relay/ — two processes. relay-intake takes an order over HTTP, validates it with a small .srl ruleset, reads two rows from a data store, forwards it over a local channel to relay-worker, waits for the worker’s answer on a second local channel, and calls back an HTTP endpoint. Three durable hops per order.
  • profiles/relay.js — k6 posts orders at a constant rate.
  • sink/ — a small HTTP server that receives the callbacks and measures each order’s latency from the moment k6 sent it.
  • relay-bench.sh — starts a throwaway PostgreSQL cluster (durable settings) on the disk you run it from, the sink and the engine, warms up, runs k6, waits until every order has called back, and prints one line of results plus the busiest SQL statements.

It needs k6, jq, PostgreSQL’s server binaries (PG_BIN_DIR, default /usr/lib/postgresql/18/bin) and an optimised build of the engine and the CLI. The runs below used the faster-to-build release-it profile:

(cd rust && cargo build --profile release-it -p sutra-dist -p sutra-cli)
export SUTRA_ENGINE_BIN=$PWD/rust/target/release-it/sutra-engine
export SUTRA_CLI=$PWD/rust/target/release-it/sutra

The script refuses to put the database on tmpfs, where a flush costs nothing and the numbers would mean nothing.

Step 1: the defaults

RATE=100 DURATION=60s tools/sutra-load-test/relay-bench.sh
[relay-bench] building the sink
[relay-bench] warm-up 10/s x 10s
[relay-bench] measured run 100/s x 60s
| offered | sent | settled | dup | drained | settles/s | settle p50 | settle p99 | intake p50 | intake p99 | commits/settle | WAL flushes/settle | WAL KB/settle | WAL fsync busy | engine CPU (cores) | PG CPU (cores) | CPU ms/settle (engine+PG) | engine env |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 100/s x 60s | 5167 | 5167 | 0 | yes | 70.1 | 6729 ms | 13395 ms | 6708 ms | 10001 ms | 16.0 | 3.01 | 10.2 | 63% | 0.24 | 0.25 | 7.09 | `defaults` |
Top statements by calls (calls | total ms | query):
72550 | 30 | BEGIN
72550 | 24 | COMMIT
46805 | 187 | SELECT set_config($1, $2, $3)
25745 | 140 | SELECT set_config($1, $2, $3), set_config($4, $5, $6)
15501 | 402 | INSERT INTO outbox_entry (entry_id, deployment_id, instance_id, body, content_type, destination, headers_json,
15396 | 332 | DELETE FROM outbox_entry WHERE deployment_id = $1 AND entry_id = ANY($2)
10334 | 118 | SELECT instance_id FROM alias_index WHERE deployment_id = $1 AND alias_name = $2 AND alias_value = $3 AND live
10334 | 144 | SELECT store_value, rev FROM data_store WHERE store_name = $1 AND store_key = $2

One lane executes one step at a time, so its commits never share a flush; on this disk that caps the engine near 70 orders per second, and the rest of the offered load queues at intake (intake p99). settled can exceed sent: k6 gives up on a request after 10 seconds and does not count it, but the intake had already accepted it, and it completes like any other.

Step 2: lanes, and a pool to match

Each lane holds a database connection while it commits, so raise the main pool with the lanes:

RATE=100 DURATION=60s ENGINE_ENV="SUTRA_ENGINE_SHARDS=8" tools/sutra-load-test/relay-bench.sh
[relay-bench] warm-up 10/s x 10s
[relay-bench] measured run 100/s x 60s
| offered | sent | settled | dup | drained | settles/s | settle p50 | settle p99 | intake p50 | intake p99 | commits/settle | WAL flushes/settle | WAL KB/settle | WAL fsync busy | engine CPU (cores) | PG CPU (cores) | CPU ms/settle (engine+PG) | engine env |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 100/s x 60s | 6001 | 6001 | 0 | yes | 99.8 | 20 ms | 72 ms | 5 ms | 26 ms | 15.9 | 2.81 | 10.2 | 87% | 0.35 | 0.37 | 7.31 | `SUTRA_ENGINE_SHARDS=8` |

Then the target rate:

RATE=500 DURATION=60s \
  ENGINE_ENV="SUTRA_ENGINE_SHARDS=32 SUTRA_DATASOURCE_MAX_CONNECTIONS=40" \
  tools/sutra-load-test/relay-bench.sh
[relay-bench] warm-up 10/s x 10s
[relay-bench] measured run 500/s x 60s
| offered | sent | settled | dup | drained | settles/s | settle p50 | settle p99 | intake p50 | intake p99 | commits/settle | WAL flushes/settle | WAL KB/settle | WAL fsync busy | engine CPU (cores) | PG CPU (cores) | CPU ms/settle (engine+PG) | engine env |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 500/s x 60s | 30001 | 30001 | 0 | yes | 499.1 | 31 ms | 185 ms | 8 ms | 52 ms | 14.1 | 0.57 | 10.2 | 89% | 1.79 | 2.01 | 7.69 | `SUTRA_ENGINE_SHARDS=32 SUTRA_DATASOURCE_MAX_CONNECTIONS=40` |

Read the columns this way:

ColumnMeaning
settles/sorders that called back, per second, from the start of the run to the last callback
settle p50 / p99time from k6 sending the order to the sink receiving the callback
intake p50 / p99time for the intake to answer 202
commits/settledatabase transactions per order, read-only ones included
WAL flushes/settledisk flushes per order — below 1 means commits share flushes
WAL fsync busyshare of the run the disk spent flushing
engine CPUcores the engine used
PG CPUcores PostgreSQL’s backends used (since 0.6.6)
CPU ms/settleengine plus PostgreSQL CPU time per order, in milliseconds — the number to watch when the host, not the disk, is the limit (since 0.6.6)

When WAL fsync busy is near 100% and adding lanes no longer raises settles/s, the disk is the limit: a faster disk, or PostgreSQL’s commit_delay (which makes a flush wait a few microseconds for more commits to join it, still durable), are what move it next. When the disk is not busy but every core is, the CPU is the limit: CPU ms/settle times the rate you want is the core count it takes. 0.6.6 cut it by about a fifth on this workload: with SUTRA_OUTBOX_WORKERS=32 SUTRA_OUTBOX_TICK_INTERVAL=PT0.01S added to the last run, 8.05 ms on 0.6.5 and 6.33 ms on 0.6.6.

The settings

SettingDefaultRaise it when
sutra.engine.shards1the disk is busy less than the time it takes to flush, and orders queue at intake. Read Execution lanes first: lanes run different instances in parallel.
sutra.datasource.max-connections8you raise the lanes: keep it at least the lane count, plus a few
sutra.datasource.dispatch-max-connections4rarely; the dispatcher’s claim and its batched delete each hold one connection, the rest serve the defers of failed sends
sutra.outbox.workers32many destinations are slow at once: it bounds the sends in flight across all of them
sutra.outbox.destination-concurrency32one destination is slow and can take more parallel requests; see Outbound bursts
sutra.outbox.max-buffered4096sutra.outbox.buffered sits at the bound during a burst and memory allows
sutra.outbox.batch-size50a deployment enqueues far more than 50 rows between claims
sutra.outbox.tick-intervalPT5Syou want retries and rows from other replicas picked up sooner; local deliveries do not wait for it
cache: on a data storeoffa store is read on every request and changed rarely — see Caching reads

Outbound bursts

Each process instance’s outbound rows form one ordered chain: they leave one after another, in the order they were enqueued, while different instances’ chains send in parallel. Two bounds apply to a send, taken in this order: one per destination authority (scheme, host and port; sutra.outbox.destination-concurrency) and one across every destination (sutra.outbox.workers). A row waiting for a busy destination holds no global slot, so a slow counterparty delays only its own rows. The claimer stops claiming only when sutra.outbox.max-buffered rows are held in memory.

Three instruments show whether outbound delivery keeps up:

  • sutra.outbox.delivery.lag (histogram, milliseconds, by scheme): enqueue to delivery, once per delivered row. Its p99 is the number to watch.
  • sutra.outbox.in-flight (gauge): at sutra.outbox.workers for long, the global bound is the limit.
  • sutra.outbox.buffered (gauge): at sutra.outbox.max-buffered, the claimer is waiting.

The defaults come from a burst bench: tools/sutra-load-test/relay-bench.sh BURST=3500 enqueues 3,500 rows in about 6 s across two HTTP destinations, one answering at once and one after 500 ms (30% of instances), on durable PostgreSQL. Lag is measured by the receiving end, against the instant the engine built each request.

Engineworkers / destination-concurrencyDrainFast lag p50 / p99 / maxSlow lag p99Slow rows/s
0.8.28 workers by instance84.5 s35.9 s / 73.9 s / 78.9 s73.6 s12.5
0.8.38 / 865.5 s6.3 s / 18.8 s / 19.1 s59.1 s16.1
0.8.316 / 865.5 s5 ms / 7 ms / 10 ms59.8 s16.0
0.8.316 / 1632.8 s172 ms / 666 ms / 864 ms27.6 s32.1
0.8.332 / 1632.9 s5 ms / 11 ms / 17 ms27.5 s32.1
0.8.332 / 32 (default)16.5 s25 ms / 166 ms / 187 ms12.3 s64.2
0.8.364 / 1632.9 s4 ms / 7 ms / 9 ms27.8 s32.0
0.8.364 / 3216.5 s4 ms / 7 ms / 15 ms12.3 s64.2
0.8.3128 / 3216.5 s5 ms / 8 ms / 11 ms12.2 s64.2

Peak PostgreSQL connections stayed at 8–9 in every run.

The defaults are the smallest pair whose fast-destination p99 stays under 2 s and whose drain time is within 10% of the best. The slow destination drains at about destination-concurrency / its latency; the fast one is only held back once the slow one can take every global slot (8 / 8 above).

Watch outbound lag under a burst

This walkthrough runs the burst bench twice against a local OpenTelemetry Collector, first with a global bound so small that the slow destination takes all of it, then with the defaults, and reads sutra.outbox.delivery.lag from the collector each time. It needs Docker, k6, PostgreSQL 18’s server binaries and a release build of the engine and CLI (cargo build --profile release-it -p sutra-dist -p sutra-cli).

Step 1: start a collector that receives OTLP and exposes the metrics for Prometheus:

cat > /tmp/otelcol.yaml <<'YAML'
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
exporters:
  prometheus:
    endpoint: 0.0.0.0:8889
service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheus]
YAML
docker run -d --name walk-otelcol -p 127.0.0.1::4317 -p 127.0.0.1::8889 \
  -v /tmp/otelcol.yaml:/etc/otelcol-contrib/config.yaml:ro otel/opentelemetry-collector-contrib
OTLP=$(docker port walk-otelcol 4317/tcp | head -1)
PROM=$(docker port walk-otelcol 8889/tcp | head -1)

Step 2: a burst with the slow destination holding the fast one. Eight sends in flight in total and eight per destination: the slow destination can take every slot.

export SUTRA_ENGINE_BIN=rust/target/release-it/sutra-engine SUTRA_CLI=rust/target/release-it/sutra
BURST=3500 ENGINE_ENV="SUTRA_TELEMETRY_OTLP_ENDPOINT=http://$OTLP \
  SUTRA_TELEMETRY_METRICS_EXPORT_INTERVAL=1000 SUTRA_TELEMETRY_SERVICE_NAME=walk-8 \
  SUTRA_OUTBOX_WORKERS=8 SUTRA_OUTBOX_DESTINATION_CONCURRENCY=8" \
  tools/sutra-load-test/relay-bench.sh
curl -s "http://$PROM/metrics" | grep 'sutra_outbox_delivery_lag_milliseconds_bucket{job="walk-8"' \
  | grep -E 'le="(100|1000|10000|\+Inf)"'

The bench’s block shows the fast destination waiting: on our run its lag was 6.5 s / 17.5 s / 17.7 s (p50 / p99 / max), and the histogram has 41 of the 3,500 rows within 100 ms, 231 within a second and 1,746 within ten.

Step 3: the same burst with the defaults (32 / 32):

BURST=3500 ENGINE_ENV="SUTRA_TELEMETRY_OTLP_ENDPOINT=http://$OTLP \
  SUTRA_TELEMETRY_METRICS_EXPORT_INTERVAL=1000 SUTRA_TELEMETRY_SERVICE_NAME=walk-default" \
  tools/sutra-load-test/relay-bench.sh
curl -s "http://$PROM/metrics" | grep 'sutra_outbox_delivery_lag_milliseconds_bucket{job="walk-default"' \
  | grep -E 'le="(100|1000|10000|\+Inf)"'

The fast destination no longer waits: 26 ms / 165 ms / 184 ms, and 2,139 rows land within 100 ms. The rows still over a second are the slow destination’s own (1,048 of them, drained at 64 rows/s), and the whole burst drains in 16.5 s instead of 66.7 s. The histogram carries only the scheme attribute, so both HTTP destinations share one series; the per-destination split comes from the bench’s sink.

Step 4: clean up. docker rm -f walk-otelcol. The bench stops its own engine, sinks and database.

PostgreSQL autovacuum

Several engine tables churn at the request rate:

  • outbox_entry gets one insert and one delete per delivery;
  • instance_state is rewritten on every step;
  • waiting_event, alias_index and inbox_seen gain and lose rows as instances park, finish and age out.

Each delete or rewrite leaves a dead row version, and the engine’s hot queries step over those until a vacuum removes them. The claim of due outbox rows is the one that suffers most.

PostgreSQL’s defaults vacuum a table once 20% of it is dead and look for such tables once a minute. That is too late for a queue: it stays small while it produces thousands of dead rows a second. One measured run at about 1,700 requests a second used the defaults. Every two and a half minutes the outbox claim took 1–1.6 s, throughput fell below 100 a second for some 40 s, and p99 latency reached 38 s.

Since 0.6.8 the engine’s migrations set per-table values. You need do nothing for them:

TableVacuum afterCost throttle
outbox_entry5,000 dead rows, whatever the table’s size (autovacuum_vacuum_scale_factor = 0, autovacuum_vacuum_threshold = 5000)none (autovacuum_vacuum_cost_delay = 0); the table is small
instance_state, waiting_event, alias_index, inbox_seen1% of the table plus 5,000 rows (0.01, 5000)the server’s setting; these tables can be large

Statistics are refreshed on the same kind of trigger. At low rates 5,000 dead rows take a long time to accumulate, so these settings cost nothing there.

One setting is server-wide: autovacuum_naptime. It sets how often autovacuum wakes to look, and a table cannot override it. For a deployment running hundreds of requests a second or more, lower it in postgresql.conf, or in your managed service’s database flags:

autovacuum_naptime = 1s              # default 1min
autovacuum_vacuum_cost_limit = 2000  # default 200 (via vacuum_cost_limit); lets the larger tables keep up

The run above, repeated with naptime 1s and an unthrottled global autovacuum, went on for ten minutes at 2,500 requests a second with p99 329 ms and no slow statements. A busy autovacuum worker uses some CPU and I/O, so watch both after the change.

MySQL/MariaDB and SQL Server have no per-table equivalent. InnoDB purge and SQL Server’s ghost cleanup are server-wide, and the engine ships no setting for them.

Profiling

PERF_DIR=/some/dir records a CPU profile of the engine and of the database backends over the measured run, and writes a flamegraph of each plus a list of the hottest functions. It needs perf usable by your user (kernel.perf_event_paranoid of 1 or less) and inferno (cargo install inferno). Build the engine with frame pointers so the stacks are whole:

(cd rust && RUSTFLAGS="-C force-frame-pointers=yes" cargo build --profile release-it -p sutra-dist)
PERF_DIR=/tmp/relay-profile RATE=100 DURATION=30s tools/sutra-load-test/relay-bench.sh

On the run above the engine used well under one core; most of its time went to the database protocol, and the rest of the wall clock was the disk.

Limits and quotas

Engine-wide ceilings, each with a safe shipped default and a consistent override shape — the same canonical sutra.* / SUTRA_* pair described in Configuration reference.

Inbound payload byte cap

Every inbound channel — broker or HTTP — delivers raw bytes to the engine before a codec ever parses them, and a producer can publish an arbitrarily large payload. The cap closes that gap: a message is byte-length-checked before codec.decode() runs, so nothing is allocated or parsed for an oversized message.

ConfigDefaultDisabled value
sutra.codec.max-payload-bytes (SUTRA_CODEC_MAX_PAYLOAD_BYTES)10 MiB (10485760)0

A per-channel override lives on the channel’s own definition (not as a separate engine-wide property), and is not clamped by the global value — it can raise or lower the effective cap for just that channel:

channels:
  - name: bulk-statements-acme
    payload-cap-bytes: 104857600   # raised for a trusted large-file channel
  - name: heartbeat-acme
    payload-cap-bytes: 4096        # tightened for a strict small-event channel

Negative values are rejected at startup (SUTRA.CONFIG.PROPERTY.INVALID) — this catches a typo like -1 before it silently disables the cap the way some other libraries interpret that value.

On rejection. Exceeding the effective cap rejects the message with SUTRA.INBOUND.PAYLOAD_TOO_LARGE (ERROR), carrying the channel id, the actual payload size, and the effective cap that was applied — enough for an operator to know exactly which config key to raise. This is a permanent reject, not a retryable one:

TransportRejection translates to
HTTP413
RabbitMQ / AMQPbasic_nack with requeue=false → the broker’s DLX (or dropped if none configured)
KafkaOffset committed — the poison record is skipped, not replayed (wire a dead-letter topic in BPMN if you need one)
AWS SQSDeleteMessage — removed, not redelivered
GCP Pub/Submessage.ack() — removed, not redelivered
FileThe source file is moved to the failed/ sub-directory
Dapr500 to the sidecar
Knative500 to the Broker

Dapr and Knative are the exception to “permanent” today. Both answer this rejection — and every other intake rejection except a malformed CloudEvent — with 500, and both pushers treat 500 as retryable: the Dapr sidecar retries under the component’s retry policy, and a Knative Broker redelivers until the Trigger’s delivery retries are spent. The same oversized message therefore arrives again until the pusher gives up. Configure a dead-letter topic on the Dapr subscription, or a deadLetterSink on the Trigger, so that it ends somewhere visible.

Per-tenant quotas

Two more admission checks, enforced before a message reaches the executor — see Multi-tenancy and isolation for the full detail:

  • maxConcurrentInstances — a hard cap on simultaneously in-flight instances for a tenant, coherent across every replica.
  • maxInboundRatePerMinute — a per-replica sliding 60-second admission window.

Neither is applied unless a tenant opts in — an unconfigured tenant is unlimited on both dimensions.

What’s not yet a configurable limit

The threat-model backlog names a few more ceilings that aren’t wired yet — a FEEL evaluation wall-clock/memory budget per expression, and a per-tenant audit-write rate cap. Until they land, the payload cap and the two tenant quotas above are the complete set of per-message and per-tenant limits. Limits on a shared physical resource (lines, seats, connections) are capacity pools.

Next

Capacity pools and sessions

Some limits are not per channel or per tenant but physical: a number of lines, licences, seats or connections to a downstream system, shared by every deployment that uses them. 0.6 adds two things for that:

  • Capacity pools: named, cluster-wide counts of slots, declared in the engine configuration, which inbound and outbound channels of any deployment draw from with pool: <name>.
  • session-end:: an inbound channel can end an instance’s session (give back its slots) before the instance reaches its end, while the instance runs on.

Active is not incomplete

max-concurrent-instances on an inbound channel caps how many of the instances it admitted may be active at once. With use-only-in-flight-for-concurrency-cap: false, a parked instance (waiting on a user task, a model job, a reply) still counts. Normally an instance stops counting when it ends. session-end: lets it stop earlier, for a flow whose session (a connection, a conversation, a line) can finish before its follow-up work does:

channels:
  - name: session-start
    transport: http
    bind: "POST /sessions"
    max-concurrent-instances: 200
    use-only-in-flight-for-concurrency-cap: false
    pool: lines                      # optional: also draw from a capacity pool
    pool-call-key: header.x-call-id  # optional: which call a message belongs to
    session-end:
      on-message: session-closed     # a correlated message on this channel ends the session
      on-reply: true                 # and/or: the flow's reply on session-start ends it
  - name: session-closed
    transport: http
    bind: "POST /sessions/closed"
KeyMeaning
session-end.on-messageAnother inbound channel of the same file. A message on it is correlated to an instance through the <q:alias onConflict="correlate"> of the process that starts on the session channel. When the session channel declares pool-call-key: header.<name> and the message carries that header, only the call it names is ended; otherwise the instance’s whole session is released. If a wait in the flow subscribes to the channel too, the message is also delivered as usual. If none does, the message is accepted for the release alone, and a message that matches no live instance is accepted and does nothing.
session-end.on-replytrue: the flow’s <q:reply> on the session channel (including continue="true") ends the replying instance’s inbound call: its channel slot and the pool slot that call took. Pool slots the instance holds for outbound calls stay.

A release is idempotent, and nothing takes a released slot back: not a later park, not a resume, not an outbound request. A request the instance put in the outbox that is still waiting for a slot when its session ends (or its call, for a call-scoped end), or one it sends afterwards, is still sent, without a slot. End, failure or a cancel still release whatever nothing released earlier. A malformed block (on an outbound channel, an unknown key, an on-message that is not another inbound channel of the file) fails to load with SUTRA.CONFIG.CHANNEL.SESSION_END_INVALID.

Capacity pools

A pool is engine configuration:

KeyDefaultMeaning
sutra.capacity-pools—Comma-separated pool names; the way to declare a pool purely by environment variables.
sutra.capacity-pools.<name>.slotsrequiredTotal slots. At least 1.
sutra.capacity-pools.<name>.reserve.inbound0Slots outbound may never take while inbound holds fewer than this many.
sutra.capacity-pools.<name>.reserve.outbound0The same, kept for outbound. The two reserves together may not exceed slots.
sutra.capacity-pools.<name>.tenantsevery tenantComma-separated allow-list of tenants whose channels may name the pool.

The env vars are SUTRA_CAPACITY_POOLS_<NAME>_SLOTS, …_RESERVE_INBOUND, …_RESERVE_OUTBOUND and …_TENANTS, with <NAME> upper-cased and - written as _. Pool names are lowercase letters, digits and dashes.

sutra.capacity-pools.lines.slots=30
sutra.capacity-pools.lines.reserve.inbound=6
sutra.capacity-pools.lines.tenants=default

The count is kept in the engine database (the capacity_pool and capacity_slot tables), so every replica sees the same pool. Without a datasource it is counted per process. Stranded slots (an instance that finished without releasing, a replica that died mid-call) are returned by the instance sweep (sutra.instance.sweep-interval).

A channel draws from a pool with pool: <name>:

  • Inbound. A new instance takes a slot for its call before its first step runs, or is rejected as busy with SUTRA.INBOUND.POOL.EXHAUSTED (HTTP 503), and no instance is created. A relay or a timer that resumes an existing instance takes no slot.
  • Outbound. A request (a <q:send> or a channel call) on a pooled channel takes a slot when the outbox sends it. While the pool is full the request waits. It is put back without counting a delivery attempt (its last diagnostic reads SUTRA.OUTBOUND.POOL.WAITING) and re-checked on every outbox tick, oldest first. The wait never blocks the flow, and a channel call’s <q:timeout> still fires. A channel-call request still waiting when its instance has finished is abandoned rather than sent (SUTRA.OUTBOUND.POOL.ABANDONED). A pooled outbound channel is never synchronous: ack-mode: on-complete on it takes the outbox path, since only the outbox can hold a request back without holding the caller’s connection.

One slot per active call. An instance holds a slot for one call, identified by its key: on an inbound start, the value of the channel’s pool-call-key header, or else the delivery’s own identity; on an outbound request, the outbound channel’s pool-call-key header, or else the outbox entry’s identity. So an instance that bridges its caller to a second party holds two slots, one per leg, and requests that carry the same call key share one slot. A call’s slot is released when that call ends: a session-end: on-message naming its key (the far leg hanging up frees that leg’s slot alone), or on-reply for the inbound call. When the instance ends, fails or is cancelled, every slot it still holds is released. A park never releases a slot.

pool-call-key takes the form header.<name>, matched case-insensitively, and needs a pool: on the same channel.

Reserves. reserve.inbound: R keeps R slots that outbound may not take while inbound holds fewer than R (and the same the other way round). A direction may take a slot when held + 1 + max(0, reserve(other) − held(other)) ≤ slots. If the pool’s store fails, inbound is admitted (as the channel cap’s count is) and outbound waits.

Load-time errorWhen
SUTRA.CONFIG.CHANNEL.POOL_UNDECLAREDa channel names a pool the engine configuration does not declare
SUTRA.CONFIG.CHANNEL.POOL_TENANT_NOT_ALLOWEDthe pool’s tenants list excludes the channel’s tenant
SUTRA.CONFIG.CHANNEL.POOL_CALL_KEY_INVALIDa malformed pool-call-key, or one on a channel with no pool:

Walkthrough: two lines, one callback slot

The examples/capacity-pools package runs one flow, session.bpmn:

POST /channels/sessions          { sessionId }   takes a `lines` slot (or 503)
  BuildCallback   responseBody := { sessionId, note }
  Notify          send on agent-callback          takes a `callbacks` slot when sent (or waits)
  AwaitWrapUp     parks until POST /channels/sessions/wrap-up { sessionId }
  End             releases whatever is still held

POST /channels/sessions/closed   { sessionId }   session-end: releases the session's slots

Its channels name the pools; nothing in the BPMN does:

  - name: session-start
    transport: http
    bind: "POST /channels/sessions"
    codec: json
    cloudevents-mode: none
    pool: lines
    session-end:
      on-message: session-closed     # a correlated message there releases the session's slots
    auth: …

  - name: agent-callback
    direction: outbound
    transport: http
    bind: "http://sink:8000/sink"
    cloudevents-mode: none
    pool: callbacks

deploy/compose.yaml declares the pools on the engine, deliberately small so that three requests fill them:

      SUTRA_CAPACITY_POOLS: lines,callbacks
      SUTRA_CAPACITY_POOLS_LINES_SLOTS: "2"
      SUTRA_CAPACITY_POOLS_CALLBACKS_SLOTS: "1"
      SUTRA_OUTBOX_TICK_INTERVAL: PT1S     # a waiting request is re-checked every second

The sink service is the stub server from examples/ai-summarize, used only for its /sink route, which records every body it is sent.

Prerequisites: Before you try a walkthrough.

1. Start the stack and deploy the package:

C=examples/capacity-pools/deploy/compose.yaml
docker compose -f $C up -d --wait
sutra package examples/capacity-pools/deployments-src/default--lines--1.0.0 \
    --out examples/capacity-pools/deploy/deployments 2>/dev/null
PORT=$(docker compose -f $C port engine 8080 | cut -d: -f2)
H=(-H 'Content-Type: application/json' -H 'X-Api-Key: dev-only-lines-key')
until curl -sf "http://127.0.0.1:$PORT/sutra/deployments" | grep -q '"phase":"Active"'; do sleep 1; done
post() { curl -s -w " HTTP %{http_code}\n" -X POST "http://127.0.0.1:$PORT/channels/$1" "${H[@]}" -d "{\"sessionId\":\"$2\"}"; }
slots() {
  docker compose -f $C exec -T engine-db psql -U postgres -d sutra -c \
    "select pool, direction, count(*) from capacity_slot group by pool, direction order by pool"
}
packaged examples/capacity-pools/deploy/deployments/default--lines--1.0.0.sutra (deploymentId dep-a833cc0807faa199083bcb7b)

2. Open two sessions, and wait until the first one’s callback has been delivered. Both lines are now taken. The first session’s callback took the only callbacks slot, so the second one’s callback waits:

post sessions s-1
post sessions s-2
until docker compose -f $C exec sink wget -qO- http://127.0.0.1:8000/sink | grep -q s-1; do sleep 1; done
docker compose -f $C exec sink wget -qO- http://127.0.0.1:8000/sink; echo
slots
 HTTP 200
 HTTP 200
["{\"note\":\"call the session back\",\"sessionId\":\"s-1\"}"]
   pool    | direction | count
-----------+-----------+-------
 callbacks | OUTBOUND  |     1
 lines     | INBOUND   |     2
(2 rows)

s-1’s callback holds the callbacks slot until s-1’s instance ends, or its session does.

3. The waiting request sits in the outbox, with no attempt counted:

docker compose -f $C exec -T engine-db psql -U postgres -d sutra -At -c \
    "select convert_from(body, 'UTF8'), attempt_count, last_diagnostic_json::jsonb->>'code' from outbox_entry"
{"note":"call the session back","sessionId":"s-2"}|0|SUTRA.OUTBOUND.POOL.WAITING

4. A third session is turned away, because the pool is full:

post sessions s-3
{"type":"urn:bpm:diag:SUTRA.INBOUND.POOL.EXHAUSTED","title":"SUTRA.INBOUND.POOL.EXHAUSTED","status":503,"detail":"Channel 'session-start' draws from capacity pool 'lines', which has no free slot (2 inbound + 0 outbound of 2); the inbound is rejected (busy).","code":"SUTRA.INBOUND.POOL.EXHAUSTED","attributes":{"channel":"session-start","pool":"lines","slots":"2"}} HTTP 503

5. End s-2’s session while its callback waits. s-2 gives back its line. Its callback is still sent, at once, although s-1 still holds the only callbacks slot: an ended session takes no slot and holds none. A third session now fits, and its callback waits for the slot s-1 holds. The lines are now s-1’s and s-3’s, and the callback slot is still s-1’s:

post sessions/closed s-2
until docker compose -f $C exec sink wget -qO- http://127.0.0.1:8000/sink | grep -q s-2; do sleep 1; done
docker compose -f $C exec sink wget -qO- http://127.0.0.1:8000/sink; echo
post sessions s-3
slots
curl -s "http://127.0.0.1:$PORT/admin/instances" | python3 -c 'import json,sys; print(len(json.load(sys.stdin)["instances"]), "instances parked")'
 HTTP 200
["{\"note\":\"call the session back\",\"sessionId\":\"s-1\"}", "{\"note\":\"call the session back\",\"sessionId\":\"s-2\"}"]
 HTTP 200
   pool    | direction | count
-----------+-----------+-------
 callbacks | OUTBOUND  |     1
 lines     | INBOUND   |     2
(2 rows)

3 instances parked

s-2’s instance is still parked at AwaitWrapUp: a session end releases slots, and the instance runs on.

6. Wrap up s-1. It ends, and gives back its line and the callback slot. s-3’s waiting callback takes the slot, so one line (s-3’s) and the callback slot (also s-3’s) are held:

post sessions/wrap-up s-1
until docker compose -f $C exec sink wget -qO- http://127.0.0.1:8000/sink | grep -q s-3; do sleep 1; done
slots
 HTTP 200
   pool    | direction | count
-----------+-----------+-------
 callbacks | OUTBOUND  |     1
 lines     | INBOUND   |     1
(2 rows)

7. Clean up:

docker compose -p sutra-capacity-pools down -v

Next

Deploy, hot-deploy, and rollback

The mechanics behind this page are covered in full in Deployment model; this is the operator-facing walkthrough of the same three operations.

Deploy

sutra deploy my-app.sutra --api --engine-url http://localhost:<port>

sutra deploy has two paths: the API path (--api, against an engine running the db deployment source) is synchronous — it returns only once the deployment is Active, or fails fast with the SUTRA.DEPLOY.* reject diagnostic. The ConfigMap path (the default, for a Kubernetes dir deployment source) patches the deployments ConfigMap and is asynchronous — the engine’s own watcher picks up the change on its next poll; add --wait to have the CLI poll GET /sutra/deployments/{id} until it reports Active.

For a large deployment where a synchronous call risks a long-held request behind an ingress, --async (API path) submits and returns 202 {deploymentId, Pending} immediately, then the CLI polls to completion itself.

Hot-deploy

A hot-deploy is just a normal deploy call against an existing slot (the archive’s stable tenant--module--version key): re-package the same source directory, and re-deploy. The new archive gets a new content-addressed deploymentId, but the slot name is unchanged, so the engine replaces the slot’s active revision in one transaction and runs its two-phase activation flip — drain the old revision, activate the new one — with no restart. In-flight instances on the old revision keep running to completion in the background; new inbound picks up the new revision immediately.

flowchart LR
    PKG["re-packaged archive<br/>new deploymentId, same slot"] --> FLIP["two-phase activation flip<br/>one transaction, no restart"]
    FLIP -->|"activate"| NEW["new revision<br/>active"]
    FLIP -->|"drain"| OLD["previous revision<br/>draining"]
    IN["new inbound"] --> NEW
    OLD -.->|"in-flight instances<br/>run to completion"| RET["retires once quiescent"]

Nothing in flight is moved by a hot-deploy: new traffic follows the active set from the moment the flip commits, while instances that started on the previous revision finish on the graph they started under.

# edit the package source, then:
sutra package packages/my-app --out /tmp/pkgs
sutra deploy /tmp/pkgs/my-app.sutra --api --engine-url http://localhost:<port>

For the edit-save-redeploy loop while developing, skip the manual re-package step entirely:

sutra deploy --watch packages/my-app --engine-url http://localhost:<port>

Each save is re-packaged and validated first — the same static validation sutra lint runs. Deployment only fires if validation passes; on a finding, the CLI reports it and skips the deploy, so a broken edit never reaches the engine.

Rollback

Rollback is the identical operation in reverse: re-package (or simply keep) the previous archive for the same slot, and re-deploy it. Its still-draining deploymentId resurrects rather than being rebuilt from scratch:

sutra deploy /tmp/pkgs/my-app-previous.sutra --api --engine-url http://localhost:<port>

Removing a deployment

sutra undeploy my-app.sutra --api --engine-url http://localhost:<port>

The engine drains it — refuses new intake, lets in-flight instances finish, retires the slot once it reaches zero instances and zero pending outbox entries. On the ConfigMap path, pair this with sutra deployments list <dir> to find the right archive/deploymentId first.

Deploy, hot-deploy, rollback and undeploy are all edges of one revision lifecycle:

stateDiagram-v2
    [*] --> Active: deploy --api, synchronous
    [*] --> Pending: deploy --async, 202 accepted
    Pending --> Active: activation flip completes
    Pending --> Failed: SUTRA.DEPLOY.* reject
    Active --> Draining: a newer revision<br/>takes the slot, or undeploy
    Draining --> Active: re-deploy<br/>that same archive
    Draining --> [*]: retired — zero instances,<br/>zero pending outbox

Rollback is not a separate operation, it is the Draining --> Active edge — which is why the previous archive’s still-draining deploymentId resurrects rather than being rebuilt.

Two drain behaviors worth knowing before you need them:

  • Inbound routes always follow the active set. A slot whose only revisions are draining serves no inbound routes at all (SUTRA.RESOLVE.CHANNEL.UNKNOWN on its paths); its parked instances still resume through relay correlation the moment you deploy a new active revision into the slot — which is also the recovery move when an undeploy left work parked.
  • Accumulated drains are safe. Several draining revisions of one slot are a legal store state (interrupted drains accumulate across restarts); boot registers each channel key once, newest draining revision first, and instances pinned to any of the revisions remain resumable. No cleanup is required before a restart.

Checking what’s live

sutra deployments list <dir> [--label KEY=VALUE]...   # ConfigMap/dir source: what's on disk

Against a running engine, GET /sutra/deployments/{id} is the authoritative live status — Active, Pending, Draining, or Failed with a reason.

Next

Instance migration

An instance is pinned to a deployment — by content hash — the moment it first parks, and every resume path resolves that pin fail-closed. That is what stops accidental version skew: a hot-deploy leaves the previous graph draining precisely so pinned instances keep resuming on the definition they started under, and a lost pin refuses rather than guesses.

It has one consequence with no escape hatch. An instance parked on a broken model stays parked on the broken model until somebody cancels it. Migration is the sanctioned way off the pin.

It is deliberately not a versioning mechanism. There is no in-process branch to author and no fleet of workers to label. There is one operator action, with a machine-readable report, that can be dry-run before it is done, refused with every violation listed rather than the first, and read back out of the audit journal afterwards.

The two endpoints

POST /admin/instances/{id}/migrateOne instance.
POST /admin/instances/migrateA filtered population off one source pin.

Both are on the gated admin surface (see Configuration reference), and both take the same shape of request: a target deployment, an optional node mapping, and the dryRun / resume / targetProcessId switches.

curl -sS -X POST "$ENGINE/admin/instances/$ID/migrate" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{ "targetDeploymentId": "dep-9f3c…",
        "nodeMapping": { "Approve": "ApproveV2" },
        "dryRun": true }'

Validation is the substance

The engine loads both graphs — the source from the deployment the instance is pinned to, the target from the active set — and derives every live locus from durable state: the snapshot’s wait frontier, its routed start event, its retry attempt counters, and the instance’s waiting rows.

Each locus is mapped through nodeMapping (identity where the map is silent), and the target node must exist and be compatible with what resume actually does there. That last phrase is the whole idea. Compatibility is not a BPMN type check; it is a check against the resume behaviour the locus will get:

LocusThe target node must beBecause resume…
MESSAGE_WAITrelay-resumable — a userTask, an intermediate message catch, or a channel-call serviceTaskcorrelates an inbound, marks the node satisfied, and continues from its outgoing edges
TIMER_WAITtimer-capable — a timer catch event, a timer boundary, or a <q:timeout>-synthesized boundaryroutes a due fire through the timer node’s own semantics
RETRY_PARKa serviceTask carrying <q:retry>re-runs the task rather than treating the node as done — so merely timer-capable is not enough
CONTINUE_REPLY_PARKa node carrying <q:reply continue="true">takes the relay path, not the timer path, and re-drives the parked tail
ROUTED_STARTa start eventreplays multi-start routing from it
RETRY_BUDGETa node carrying <q:retry>a burned attempt budget landing on a policy-less node would silently apply attempts nobody declared

Every violation is reported, not just the first. Fixing a mapping should take one round trip, not one per node.

flowchart TD
    DUR["Durable state<br/>wait frontier · routed start<br/>retry counters · waiting rows"] --> LOCI["Live loci"]
    SRC["Source graph<br/>the deployment the instance is pinned to"] --> LOCI
    LOCI --> MAP["nodeMapping<br/>identity where the map is silent"]
    TGT["Target graph<br/>from the active set"] --> CHK
    MAP --> CHK{"target node exists,<br/>and matches what resume<br/>actually does there?"}
    CHK -->|"no"| REF["Refused — every violation listed"]
    CHK -->|"yes"| GO["Migrate"]

Both graphs are loaded, so the check runs against the target’s real nodes rather than against the mapping’s good intentions — and a refusal comes back complete, not one node at a time.

Some findings are warnings rather than refusals:

  • A completed node the target does not declare — not a live locus, so not fatal. But replay-as-done matches by id, so if the target still reaches that node it will be executed again. Dropping a node is legitimate model evolution, so this is loud rather than blocking.
  • A frontier entry with no waiting row behind it — validated as a message wait and migrated anyway; the frontier is the resume-side truth and is never dropped.
  • Pending outbox rows on the source — see What moves.

Dry run

dryRun: true mutates nothing: it takes no ownership claim and writes no row, and returns the same full report. Because nothing was locked while it was produced, a dry-run report is advisory — a concurrent resume can move the frontier out from under it. The report says so.

Node mapping is checked, not merely applied

A mapping entry naming a node the instance neither parks at nor has completed is refused (MAPPING_INVALID), not ignored. A typo must never read as “identity mapping, then”.

Lifecycle rules

  • The target must be active. A draining target is refused: a draining deployment retires the moment it is quiescent, so migrating onto one strands the instance again and costs a migration to do it. Unknown ids are the same refusal.
  • The source graph must still be registered (active or draining). Quiescence gating means a live instance always keeps its deployment registered, so this only bites after a forced retirement — and it is a hard refusal rather than a fallback, because validating a continue-reply park against the wrong rule would pass the wrong check.
  • Terminal instances are a validation error. Completed and terminated instances are retained history, and re-pinning history rewrites the record of where it ran.
  • FAILED instances are migratable — and that is the prime use case.
  • A no-op is refused (TARGET_SAME_AS_SOURCE) rather than silently rewriting a row for nothing.

Migration never auto-resumes — unless you ask

Migration re-pins and rewrites. Full stop. A FAILED instance stays FAILED under the new pin, and bringing it back is a separate, explicit decision — resumed: false is in every response so no caller has to infer it.

resume: true is that decision, and it closes the repair loop the operation exists for: fix the model, migrate the dead instance onto it, bring it back. On a successfully migrated FAILED instance it clears the failure state, re-stamps the snapshot suspended, and re-arms the parks the failure commit tore down — all inside the migration’s own transaction, so a crash can never leave a half-revived instance.

It then comes back through the ordinary paths: a re-armed timer whose due instant has passed is claimed by the timer poller on its next tick; a message park waits for its next correlated inbound exactly as it would have. There is no privileged re-drive, which is exactly why the claim-guarded concurrency story below still holds afterwards. A burned <q:retry> budget stays burned — resume is not a retry reset.

On an instance that is not FAILED, resume: true is a validation error (RESUME_NOT_FAILED), not a no-op. A suspended instance is not stuck; it is parked, and it resumes on its own correlation or its own timer with no operator action at all. Reporting resumed: false for it would let a caller believe they had woken something.

What moves, and what deliberately does not

Moves?Why
The instance snapshotYes — re-pinned, node ids rewrittenIt is the instance.
Waiting rowsYes — node ids mapped; kind, due instant and timestamps intactA park with an hour left still has an hour left.
Correlation aliasesYes, verbatimMust: relay correlation resolves the instance through them. Leaving them behind makes a migrated instance unreachable.
Subject index rowsYes, verbatimMust: erasure and disclosure find instances through them. Leaving them behind makes a migrated instance un-erasable.
Audit journalYes — scope only; node ids left verbatimA trail names where something happened, under the graph that was live then. Rewriting the ids would falsify it; a SUTRA.INSTANCE_MIGRATED event records from→to, so the provenance is recoverable.
Pending outbox rowsNoAn emission was minted by the source deployment’s channel bindings and is dispatched against them — and the dispatcher covers draining deployments, so it drains where it was made. Re-targeting a message at bindings that never produced it would be worse than leaving it. Reported as a warning.

Coverage cursors travel verbatim, because they are keyed by declared path id rather than by node id — the node mapping must not touch them.

Encrypted variables ride through as ciphertext and still decrypt under the new pin. That is by construction rather than by luck: the authenticated-data binding for an at-rest value deliberately excludes the deployment id, precisely so that “a migration changes only the pin” stays true.

Concurrency: the claim, and why it bounces

A real run claims the instance first, through the same per-instance ownership machinery every resume path uses (see Replica semantics) — and under a distinct owner identity, so a migration can never slip past a resume this same replica has in flight. The claim is re-verified under the row lock inside the commit itself, closing the gap between “claimed” and “moved”.

Contention is a retry-safe 409 (SUTRA.ADMIN.MIGRATE.CLAIM_HELD) with nothing read, rewritten or committed. A resume that starts after the migration claims bounces off it in turn.

The move itself is one transaction: snapshot, waiting rows, aliases, subject index and audit scope all land under the new pin together, or none do. No session ever observes the instance under both pins or under neither.

sequenceDiagram
    participant M as Migrate call
    participant C as The instance row
    participant R as A resume path
    alt the claim is free
        M->>C: claim, under a distinct owner identity
        C-->>M: taken
        R->>C: claim, to resume
        C-->>R: CLAIM_HELD — requeue or defer
        M->>C: commit, claim re-verified under the row lock
        Note over M,C: snapshot, waiting rows, aliases, subject index,<br/>audit scope: all together, or none
    else a resume already holds it
        M->>C: claim, under a distinct owner identity
        C-->>M: 409 CLAIM_HELD, nothing read, rewritten or committed
    end

Whichever side gets there first, the other one bounces retry-safely — the distinct owner identity is what stops a migration re-entering past a resume already in flight on the same replica.

Batch migration

POST /admin/instances/migrate applies the same validation, the same compatibility matrix and the same node mapping to a filtered population.

Selection. filter.sourceDeploymentId is required — one migration names one source graph and one target graph, and a node mapping that is correct for one source is meaningless for another. There is deliberately no “every deployment” mode. Optional narrowing: processId, status (SUSPENDED or FAILED — the only two an instance can migrate in), includeTerminal, and limit (default 100, clamped to 1000).

Selection is ordered by instance id, then limited. Ordering by a timestamp that moves under a live population is exactly how a caller silently skips work.

Retained terminal rows are excluded unless includeTerminal asks for them — a busy deployment holds a whole retention window of finished instances, and a report where they crowd out the live ones is worse than useless. The flag exists so a caller who wants to know why an instance was not moved gets an explicit INSTANCE_TERMINAL refusal instead of a silent omission.

Every instance is its own transaction. One claim, one commit, one report entry. There is deliberately no batch-wide transaction — which is the point, and which makes the crash story trivial: a mid-batch crash leaves every instance either fully migrated or completely untouched, because that is already true of each instance’s own commit and there is nothing larger to be half applied. Nothing about one instance can decide another’s fate.

Per-instance outcome is a single enum, so no caller has to infer one from a combination of booleans:

OutcomeMeaning
MIGRATEDMoved.
VALIDA dry run validated it.
REFUSEDValidation said no; the report says why.
BOUNCEDIts ownership claim was held. Nothing read or written — re-run to pick it up.
NOT_FOUNDSelected, then gone.
ERRORSomething else failed for this instance alone.

The HTTP status describes the batch, not its instances. A run in which every instance refused still answers 200: the call was accepted, executed to completion, and reported in full — which is exactly what was asked. 400 is a malformed request, 422 a batch-level refusal (target not active; source and target the same pin), 503 no persistence. Scripts key on totals and on each entry’s outcome, never on the status line.

Contention is reported, never retried. Retrying inside the call would make an admin request’s runtime unbounded — the claim is held by another replica’s work, on that work’s timescale, not yours — and would turn the report into a claim about a moment that has already passed. Re-run the same request instead: the selector is deterministic and whatever moved is no longer under the source pin, so a re-run converges.

Two contradictions are caught from the request alone rather than N times over in the per-instance reports: resume with status: SUSPENDED (it selects exactly the instances resume refuses), and targetProcessId without processId (re-homing a mixed population into one process is never what anyone means, and one mapping could only be right for one of them).

Cross-process re-homing

targetProcessId re-homes the instance into a different process of the target deployment. Naming the instance’s own process id is not a cross-process migration — being explicit must never change semantics.

Identity is never implicit across a process boundary. Every live locus must carry an explicit nodeMapping entry, or the migration is refused with CROSS_PROCESS_UNMAPPED — a different code from the ordinary unmapped-node refusal, because the danger is the opposite one: the id probably does exist in the target. Two processes that both declare Approve are not thereby the same Approve, and an accidental collision between unrelated graphs must never read as a deliberate mapping.

Everything else is unchanged. The same compatibility matrix is evaluated against the target process’s graph; a target deployment that does not declare the named process is PROCESS_ABSENT, and its message names both the process asked for and the ones actually there. The snapshot’s process id and the waiting rows’ process id are rewritten together, because the timer poller reports it and the admin listing renders it — a row still naming the source process would describe the instance as living somewhere it no longer does.

One honest wrinkle: a cross-process move can carry a coverage cursor for a path the target process does not declare. It is inert, and remapping it would need a path mapping nobody supplied.

Status codes and diagnostics

StatusMeaning
200Migrated, or dry-run validated. The body is the full report.
400The id is not a UUID, or the body is malformed / missing the target.
404No such instance.
409Retry-safe refusal: claim held, a unique-live alias collides under the target, or the validated move did not commit. Nothing moved.
422Validation failed. The report lists every violation.
503No persistence configured.

All codes are under SUTRA.ADMIN.MIGRATE. — TARGET_NOT_ACTIVE, TARGET_SAME_AS_SOURCE, SOURCE_UNRESOLVABLE, PROCESS_ABSENT, NODE_UNMAPPED, NODE_INCOMPATIBLE, MAPPING_INVALID, INSTANCE_TERMINAL, CROSS_PROCESS_UNMAPPED, RESUME_NOT_FAILED, CLAIM_HELD, ALIAS_CONFLICT, COMMIT_FAILED — plus the warnings named above. CLAIM_HELD is the admin twin of the resume paths’ own, and carries the same retry-safe posture.

A worked repair loop

  1. An instance fails. It is FAILED, retained, and blocking its deployment’s retirement.
  2. Fix the model; package and deploy it. The new deployment becomes active; the old one drains.
  3. Dry-run the migration against the new deployment id. Read the report; supply nodeMapping entries for anything renamed until it validates clean.
  4. Run it for real with resume: true.
  5. The instance comes back through the ordinary timer/correlation paths. Once it and its siblings finish, the old deployment goes quiescent and retires on its own.

For a population, do steps 3–4 with the batch endpoint and a filter.sourceDeploymentId of the old pin, then re-run until totals shows no BOUNCED.

Next

Logging and audit

Three separate surfaces, deliberately not conflated: engine logs, CLI logs, and the audit trail. Each has its own destination and its own purpose. None of the three is collected by Sutra itself — see No telemetry, no phone-home for that guarantee; everything below stays on the host (or goes to a destination you configure) unless you point it somewhere else.

Engine logs

Structured JSON on stdout, always, with no configuration required. The field shape (timestamp / level / loggerName / message / service.name, plus traceId/spanId inside a sampled span) is stable, so a log-processing pipeline can key off it directly. RUST_LOG filters verbosity using standard tracing EnvFilter syntax (default info) — e.g. RUST_LOG=sutra_channels=debug,sutra_engine::deploy=trace to raise one module without drowning in the rest. See Troubleshooting BPMN solutions for reading these logs alongside traces and the audit trail when tracking down one message’s path through the engine.

When an OTLP endpoint is configured (see Configuration reference), the same log records additionally export over OTLP — stdout is never replaced, only supplemented.

CLI logs

The sutra CLI writes its own logs to stderr, opt-in via -v / -vv / -vvv (info / debug / trace) — never mixed into stdout, which is reserved for report output (text or --format json). This is why every command in the CLI reference is safe to pipe: sutra describe my-process.bpmn --format json | jq . never has a stray log line corrupt the JSON.

Audit sinks

The audit trail is a separate, compliance-oriented record of what a process instance did — every INSTANCE_STARTED/INSTANCE_COMPLETED/INSTANCE_SUSPENDED/INSTANCE_RESUMED event, validation outcomes, and relay/resume decisions — independent of the telemetry pipeline described in Observability. Three sinks, any combination of which can be active at once:

ConfigenvWhat it does
sutra.audit.jsonl.pathSUTRA_AUDIT_JSONLWrites one JSONL file (or a per-tenant directory tree) an operator can tail, or feed to sutra audit-replay (below) offline.
sutra.audit.otel.endpointSUTRA_AUDIT_OTEL_ENDPOINTShips audit events over OTLP to a collector.
sutra.audit.sqlSUTRA_AUDIT_SQLPersists audit rows durably in the engine’s own datasource, under the same row-level-security policy as every other engine table — see Multi-tenancy and isolation.

sutra.audit.sql is off by default. With no sink on, an engine writes no audit record, including the inbound.DELIVERY_DROPPED record for a refused delivery; see Refused deliveries.

A process’s events go to one sink: the one its <q:audit sink="…"> names, or else the highest-priority sink that is switched on (sql, then jsonl, then otel).

  • JSONL. Each event is one line, written in a single append, so a file shared by every executor lane never holds two events on one line (fixed in 0.6.9).
  • OTLP. The records are OTLP log records on the instrumentation scope sutra.audit, sent through a logs pipeline of their own to SUTRA_AUDIT_OTEL_ENDPOINT, never through the telemetry endpoint. The engine builds that pipeline once at start and flushes it on shutdown. Before 0.6.9, setting this variable made the engine panic at start.

Resource attributes on audit records

The OTLP audit records carry the same resource attributes as the engine’s own telemetry, so a collector can tell which deployment, environment or test run an audit record belongs to:

SourceExampleOn audit records
OTEL_RESOURCE_ATTRIBUTESdeployment.environment=staging,example.run_id=run-42yes
sutra.telemetry.resource-attributes / SUTRA_TELEMETRY_RESOURCE_ATTRIBUTES (0.6.12)example.from_config=yesyes, and on every telemetry signal; wins over OTEL_RESOURCE_ATTRIBUTES for the same key
sutra.audit.otel.service-name / SUTRA_AUDIT_OTEL_SERVICE_NAME (0.6.12)orders-auditthe records’ service.name; default sutra-audit
OTEL_SERVICE_NAMEorders-engineno: it names the engine’s telemetry, never the audit stream

To try it, point the audit pipeline at a collector whose debug exporter prints what it receives (verbosity: detailed), and start the engine with:

SUTRA_AUDIT_OTEL_ENDPOINT=http://collector:4317
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=conformance,example.run_id=run-42
SUTRA_TELEMETRY_RESOURCE_ATTRIBUTES=example.from_config=yes

Run any process with <q:audit sink="otel">. Each batch of audit records the collector prints starts with the resource, which includes (with the telemetry.sdk.* attributes, in any order):

Resource attributes:
     -> deployment.environment: Str(conformance)
     -> example.run_id: Str(run-42)
     -> example.from_config: Str(yes)
     -> service.name: Str(sutra-audit)

The conformance suite tc_audit_sinks runs exactly this against a real collector.

What an audit event captures

A process chooses how much each of its audit events carries, with <q:audit capture="…">:

captureEach event carries
noneno audit event at all
metadatathe event, node and diagnostic, with no variable content
payload (the default)the redacted variable snapshot
media (0.6)metadata, plus the media references found among the variables ({kind: "media", uri, sha256, …}), never the media itself

With media, subject erasure and terminal retention also delete the recordings those events reference, and amend each event to say they are gone. See Recordings and media references for the mechanism and a walkthrough.

Correlation keys on every audit event

From 0.6.9, each audit event carries the instance’s correlation keys: every <q:alias> the process declares, with its current value. That lets you find all of one business transaction’s events by the key your systems already use, without first looking up the instance id.

SinkWhere the keys are
JSONLa correlation object on the line: "correlation":{"draftId":"d-7"}
OTLPone attribute per key, named correlation.<alias name>
SQLnot stored; the audit_event row is unchanged
  • Where the values come from. A start event’s aliases are bound into the instance’s variables under the alias name, and the variables travel with the instance, so every event reads its keys from there, including after a resume on another replica. No database read is added per event.
  • Wait-node aliases. An alias declared on a wait node is evaluated over the current variables. Until it has a value, the key is left out.
  • Transport events and unaudited processes. Transport events carry no keys, and a process that is not audited computes none.

Redaction applies to the keys as it does everywhere else:

  • A key shows ***REDACTED*** when the alias name, or any variable its expression reads, is declared sensitive.
  • When the expression reads a payload that has a <q:redactors> chain, the key comes from the masked projection, never from the raw payload. When the expression is more than a plain path, the key shows ***REDACTED***.

Try it

The review-loop example correlates every message by its start event’s <q:alias name="draftId" expression="payload.draftId">. Its compose file switches on the SQL sink; this override sends the audit trail to a JSONL file instead. You need Docker, the sutra CLI and an engine image, both 0.6.9 or later. From the repository root:

C=examples/review-loop/deploy/compose.yaml
cat > /tmp/audit-jsonl.yaml <<'YAML'
services:
  engine:
    environment:
      SUTRA_AUDIT_SQL: "false"
      SUTRA_AUDIT_JSONL: /tmp/audit/audit.jsonl
YAML
docker compose -f $C -f /tmp/audit-jsonl.yaml up -d --wait
sutra package examples/review-loop/deployments-src/default--review--1.0.0 \
    --out examples/review-loop/deploy/deployments 2>/dev/null
PORT=$(docker compose -f $C port engine 8080 | cut -d: -f2)
H=(-H 'Content-Type: application/json' -H 'X-Api-Key: dev-only-review-key')
until curl -sf "http://127.0.0.1:$PORT/sutra/deployments" | grep -q '"phase":"Active"'; do sleep 1; done
send() { curl -s -o /dev/null -w "HTTP %{http_code}\n" -X POST "http://127.0.0.1:$PORT/channels/drafts$1" "${H[@]}" -d "$2"; }
send "" '{"draftId":"d-7"}'
send /review '{"draftId":"d-7","decision":"approve"}'
docker compose -f $C cp engine:/tmp/audit/audit.jsonl - | tar -xO | head -3
docker compose -p sutra-review-loop down -v

Both requests answer HTTP 200, and every line carries the draft’s key (ids and times differ):

{"instanceId":"66ed7453-…","deploymentId":"dep-84173f…","tenant":"default","seq":1,"at":"…","eventType":"INSTANCE_STARTED","payload":{},"correlation":{"draftId":"d-7"}}
{"instanceId":"66ed7453-…","deploymentId":"dep-84173f…","tenant":"default","seq":2,"at":"…","eventType":"NODE_ENTERED","nodeId":"Start","payload":{},"correlation":{"draftId":"d-7"}}
{"instanceId":"66ed7453-…","deploymentId":"dep-84173f…","tenant":"default","seq":3,"at":"…","eventType":"NODE_LEFT","nodeId":"Start","payload":{},"correlation":{"draftId":"d-7"}}

Replaying an instance’s audit trail offline

sutra audit-replay <instance-id> --from-jsonl <path-to-file-or-dir> [--tenant <id>] [--until <EVENT_TYPE>]

Walks the JSONL audit stream for one instance id and prints its events in order — useful for reconstructing what a specific production instance did without needing direct database access. --until stops replay after a given event type (e.g. INSTANCE_COMPLETED).

How sensitive data is masked

A variable tagged sensitive (via q:variables, see The q: namespace) or read from a dataClass-tagged data store (see Data stores) is redacted on every one of the surfaces above — audit sinks, structured logs, and traces alike — by the same redactor mechanism described in Domain neutrality and the SPI model. The flow itself still sees the real value; only what the engine emits is masked.

A source that declares masking, with q:redactors or a sensitive payload variable, masks the rest of the inbound message as well: event.body and every header value in event.headers and header, whole, because they carry the same data and can carry credentials. The flow still reads the real values. This has limits you should know:

  • With no master key, a snapshot holds values unencrypted.
  • A flow that declares no masking shows all of it, including a credential sent in a header such as an API key, on inspect, in the snapshot, and in audit output with capture="payload". To avoid that, declare a sensitive payload variable, avoid capture="payload", limit access to the inspect API and the database, or end authentication at a gateway that strips the header.
  • A q:alias value is stored in clear, so an alias built from a header holds its value. An explicit event id is stored in clear too, and so is the trace id with an outbound send.
  • A copy of the body in another variable is only masked if that variable is sensitive. sutra lint warns about this with SUTRA.CONFIG.BPMN.RAW_BODY_UNMASKED. Writing the body to a data store warns with SUTRA.CONFIG.BPMN.RAW_BODY_STORED, because store columns have no at-rest encryption. The lint doesn’t cover copies of a header.

Next

  • Observability — the telemetry (trace/metric/log) side this page’s audit trail is deliberately kept separate from.
  • Troubleshooting BPMN solutions — putting logs, traces, and the audit trail together to answer “what happened to this message?”

Troubleshooting BPMN solutions

This page is for the person building and running a Sutra solution — a deployment package that isn’t behaving the way you expect. (If instead you’re debugging the engine itself — a crash, a suspicious executor code path, a failing test in the workspace — see Debugging the engine, which is written for contributors.)

Every command below is read against rust/crates/sutra-cli/src/commands/ — what’s described is what the code actually does, not an aspirational surface.

Start with sutra lint

sutra lint packages/my-app

Runs the exact fail-closed validation suite sutra package runs before sealing an archive, and prints nothing on success. This is always the first thing to run — most “why doesn’t this deploy” questions are answered here, before you ever touch a running engine.

Reading a SUTRA.* diagnostic code

Every rejection, warning, and structured log line the engine emits carries a code shaped like SUTRA.<CATEGORY>.<REASON>. The category tells you which layer to look at:

CategoryLayerExample
SUTRA.PARSE.*A document failed to parse or failed schema validationSUTRA.PARSE.XSD.SCHEMA_VIOLATION
SUTRA.CONFIG.*A deploy-time configuration problem (channels.yaml, datastores.yaml, a codec manifest)SUTRA.CONFIG.CODEC_MANIFEST.MISSING
SUTRA.INBOUND.*An inbound message was rejected before or during dispatchSUTRA.INBOUND.NO_START_EVENT_FOR_MESSAGE_TYPE, SUTRA.INBOUND.PAYLOAD_TOO_LARGE, SUTRA.INBOUND.ALIAS_CONFLICT_REJECT
SUTRA.ACK.*Acknowledgement / deferred-ack registry eventsSUTRA.ACK.DEFERRED_TIMEOUT — see Acknowledgement modes
SUTRA.VALIDATE.*A q:validators entry (complex or simple) couldn’t be resolved or runSUTRA.VALIDATE.VALIDATOR_NOT_FOUND
SUTRA.RUNTIME.*A failure while executing an already-dispatched instanceSUTRA.RUNTIME.RELAY.CORRELATION_NOT_FOUND, SUTRA.RUNTIME.DATASTORE.CONFLICT
SUTRA.OUTBOUND.*A reply or send failed to encode or deliverSUTRA.OUTBOUND.ENCODE_FAILED
SUTRA.DEPLOY.*The admin deploy API rejected an archivereturned as the body of a 4xx from POST /admin/deployments

--format json on any CLI command that reports diagnostics gives you the code, severity, and any attributes as structured data — pipe it to jq rather than screen-scraping text.

Two families get their own sections below, because their codes are read in groups rather than one at a time: codec diagnostics and data-store diagnostics.

Codec diagnostics: parse and schema failures

Whichever codec a channel is bound to, a decode lands on one of three outcomes, and it is that contract — not any codec’s own vocabulary — that the rest of the engine reacts to:

  • OK — a payload, no issues.
  • SOFT_ERRORS — a usable payload plus structural issues. The instance still starts and still runs; the issues are data your process can branch on. This is the outcome people are most often surprised by, and it’s deliberate: a schema violation is a business decision to route, not an exception to catch.
  • FATAL — no payload at all. The engine rejects the inbound per the channel’s configured posture.

Every issue carries the same five slots — code, severity (ERROR / WARNING / INFO), path (JSON-Pointer-shaped into the payload, or the source location for a schema violation), message, and value (the offending value, or the reason code a validator wants surfaced). They reach your process as <source>.validation (the full list) plus a frozen validation summary: outcome, tier (structural / content), firstReasonCode, firstIssue, and issues. <q:onValidation mode="route"/> is what hands that to your own gateway instead of short-circuiting — see The q: namespace.

The codes this engine emits

The middle segment of a SUTRA.PARSE.* code names the layer that produced it, which is the fastest way to separate a malformed-bytes problem from a schema problem:

CodeOutcomeMeans, and what to check
SUTRA.PARSE.JSON.PARSE_ERROR, SUTRA.PARSE.XML.PARSE_ERROR, SUTRA.PARSE.YAML.PARSE_ERRORFATALThe bytes aren’t well-formed in the format the channel declared. Nothing schema-related has run yet — look at what the sender actually put on the wire, and at the channel’s content-type handling.
SUTRA.PARSE.XSD.SCHEMA_VIOLATIONSOFT_ERRORSA well-formed document that violates the package’s own XSD. Collect-all: one issue per violation, never just the first, each carrying line:column in the path slot.
SUTRA.PARSE.JSON_SCHEMA.SCHEMA_VIOLATIONSOFT_ERRORSThe same, for a schemaKind: json-schema codec.
SUTRA.RUNTIME.CODEC.DECODE_FAILEDFATALA package codec couldn’t read the bytes at all — a parse or transcode failure before validation could run. Most often a payload sent in a format the codec-manifest’s formats list doesn’t include.
SUTRA.INBOUND.CODEC_NOT_FOUNDrejected before decodeThe channel names a codec this binary doesn’t serve. Check what the running build actually links — see Channels and transports.
SUTRA.OUTBOUND.ENCODE_FAILED—The reply direction: the payload couldn’t be encoded for the outbound channel.

Deploy time is a separate, earlier, fail-closed gate: SUTRA.CONFIG.SCHEMA.INVALID (a schema that doesn’t compile), SUTRA.CONFIG.CODEC_MANIFEST.MISSING / .INVALID, and SUTRA.CONFIG.CODEC_LAYOUT.INVALID (a loose file, mixed kinds, or an empty codec folder under schemas/) all reject the archive rather than deploy a codec that would validate less than it claims. sutra lint reports every one of them before you deploy anything — which is why it’s the first command on this page.

A code from a codec you added

An extension codec (see Channels and transports) claims its own middle segment — SUTRA.PARSE.<STANDARD>.* and SUTRA.VALIDATE.<STANDARD>.* — and documents its own set; the engine neither interprets nor rewrites them. What the engine guarantees is everything above: the three-way outcome, the five issue slots, and the same surfaces. So an unfamiliar code reads exactly the way the table does — the middle segment tells you which codec owns it, and that codec’s own documentation tells you what it means. Wherever it came from, it reaches you the same three ways: the validation variables in the process, --format json diagnostic output, and the audit trail, so Tracing one message end to end applies unchanged.

Data-store diagnostics: projected stores

A store that declares a structure: block is verified at package time against the store’s own migrations — no database connection, no credentials. A store with no structure: block raises none of these codes at all; it stays the opaque key→JSON store it always was.

The posture is three-state, and reading it correctly saves a lot of time: a definite fault is an error, an unprovable one is a warning worded as unprovable, and a projection that matches raises nothing. Only errors fail the command — a warning here is information, not a gate.

CodeSeverityMeans, and the fix
SUTRA.CONFIG.DATASTORE.STRUCTURE_NOT_FLATERRORThe declared type has a nested, repeated or open child (or no projectable child at all), so it can’t become a flat row. The message names the child. Flatten the type, or remove the structure block and keep the opaque store — those are the only two remedies, deliberately
SUTRA.CONFIG.DATASTORE.COLUMN_MISSINGERRORA declared field projects to a column the effective table doesn’t have, or the table is missing one of the three control columns (store_key, rev, updated_at) every projected table needs — the message names whichever is absent. Add it in a new V-numbered migration, or (a declared field only) map it to a column that already exists under columns: — there’s no columns: remedy for a missing control column
SUTRA.CONFIG.DATASTORE.COLUMN_TYPE_MISMATCHERRORThe column can’t hold the declared value space — VARCHAR(10) for a maxLength="35" field, an integer column for a fractional decimal — or its nullability contradicts the declaration (an optional field against a NOT NULL column with no DEFAULT, or a column an ALTER adds as NOT NULL with no DEFAULT, which pre-existing rows could never satisfy). Widen or relax the column in a new migration, or narrow the declared type
SUTRA.CONFIG.DATASTORE.KEY_MISMATCHERRORThe table’s identity isn’t a key over the projected columns: it declares no PRIMARY KEY (nor a unique constraint) at all, or a key column the projection never writes, or one that maps to an optional field. A projected store reads, upserts and compare-and-sets exactly one row by key, so it needs a key it can always write
SUTRA.CONFIG.DATASTORE.COLUMN_NAME_INVALIDERRORA folded column name collides with another field’s, lands on a reserved word, exceeds the 63-character identifier cap, or isn’t a usable identifier. Name the column yourself under columns: — the mapping is checked by the same rules, so the override has to be usable too
SUTRA.CONFIG.DATASTORE.DDL_UNVERIFIABLEWARNINGNothing is wrong — something simply couldn’t be proven. See below
SUTRA.CONFIG.DATASTORE.COLUMN_UNMAPPEDWARNINGThe table has columns the projection never writes. Usually fine (a legacy or an operator column). One case is sharper and the message says so: an unmapped NOT NULL column with no DEFAULT would make every insert fail — give it a DEFAULT, declare it in the structure type, or drop it

A structure block pointing at a schema or a type the package doesn’t declare is reported as SUTRA.CONFIG.DATASTORE.INVALID — the same code any other malformed store declaration uses — not as one of the projection codes above.

DDL_UNVERIFIABLE is not an error, and shouldn’t be read as one

… could not be fully parsed — the statement CREATE OR REPLACE FUNCTION f( … is outside the DDL subset this lint parses — so the effective table shape was not derived and the declared structure was not verified; no column diagnostic is raised for this store (it may be valid; it is simply not provable here)

Lint replays a package’s migrations/<store>/V*.sql through a deliberately small SQL subset — CREATE TABLE, ALTER TABLE ADD/ALTER/DROP COLUMN, and the key-bearing constraints, across the three shipped dialects’ spellings. Real migrations routinely contain more than that: a PL/pgSQL trigger body, a T-SQL procedural guard, a table created by an operator outside the package. When the parser meets something it doesn’t model, it stops trying to verify that store and says so. It never converts that into a column error, because a linter that cries wolf on legitimate DDL is one authors learn to ignore.

The same wording covers the narrower cases, and they’re worth telling apart:

  • The store’s shape wasn’t derived at all — an out-of-subset statement, no migrations shipped for the store, the table created elsewhere, or several candidate tables and none named after the store (name the projected one with sql: table: <name>).
  • Individual fields couldn’t be compared — the column’s type is outside the set lint compares (JSONB, UUID, a domain type), or the declared type has no ruled column mapping. The column exists; only the type comparison is withheld, aggregated into one line per store.
  • The declared schema isn’t an enumerable XSD — an engine-provided codec (its type set is open), or a JSON-Schema / schema-bundle codec folder. JSON Schema carries no length or precision information, so there is genuinely nothing to compare against here — but don’t read the warning as “unverified for now, deploy anyway”: a JSON-Schema-declared structure: isn’t merely unverified, it is refused outright when the engine resolves the store at deploy time (see Typed columns). Declare the structure against an XSD if you want the store to exist at all, not just to have its column types checked.

Note that degradation is per store, not per statement: one out-of-subset statement anywhere in migrations/<store>/ withholds verification for that store’s whole table. So the way to get the warning to go away is to keep the store’s own DDL inside the subset (plain CREATE TABLE / ALTER TABLE, with procedural setup living in a different store’s schema or outside the package), or to give lint the missing pointer — sql: table: — when the shape is parseable but ambiguous. If neither is worth it, the warning is a fair thing to live with: first-use verification against the live table still fails the store closed if the projection genuinely doesn’t hold.

Runtime codes: UNDECLARED_FIELD, VALUE_NOT_A_RECORD, PROJECTION_UNSATISFIABLE

Three runtime codes ride a projected store’s own operations. Where to look for them matters: unlike SUTRA.RUNTIME.DATASTORE.CONFLICT, which is the failed instance’s own diagnostic code, these three travel inside the message of a SUTRA.RUNTIME.UNEXPECTED diagnostic. So filter on the message text, not the code, when you go looking:

CodeFires when, and the fix
SUTRA.RUNTIME.DATASTORE.UNDECLARED_FIELDA write carried a field the structure doesn’t declare. A projected row is its declared scalars — there is no residue column, so an extra field has nowhere to go, and the write is refused naming the field rather than dropping it. Two causes in practice: the process is writing a value assembled from a different (wider) shape than the declared type, or the declared type has fallen behind a schema change. Declare the field and ship the column, or stop writing it
SUTRA.RUNTIME.DATASTORE.VALUE_NOT_A_RECORDA projected store was handed a value that isn’t a record at all — a scalar, an array, or null. A projected row is its declared fields, so there is nowhere for a bare value to land either. Write a record shaped like the declared type
SUTRA.RUNTIME.DATASTORE.PROJECTION_UNSATISFIABLEThe package-time drift check, caught live: on first use of the store — once per store instance, the same gate that runs its migrations, never on every operation — the provider reads the live table’s actual columns and fails the store closed if the projection can’t be satisfied, naming every offending column at once (one missing, an optional field’s column NOT NULL, or a mandatory unmapped column with no DEFAULT). Usually a hand-applied ALTER that drifted from the package’s own migrations — re-apply them (or ship the missing ALTER) before the store can be used

A silent partial write is the worse outcome in every one of these, so all three are deliberately loud rather than best-effort. SUTRA.CONFIG.DATASTORE.COLUMN_NAME_INVALID, in the table above, is this family’s plan-time member: a declared field folding onto a control column name is refused before the store is ever served — both lint and the engine’s own plan-time resolution run the same rule, sourced from one shared constant, so the two sides cannot drift apart.

The inspection commands

None of these execute anything — they read a BPMN file (or a running engine) and report. Use them in roughly this order when a process isn’t routing, resuming, or replying the way you expect.

sutra describe — what does the engine see in this file?

sutra describe bpmn/transfer.bpmn

A structural summary: processes, start/end events, every service/user task (with its implementation, codec, validator, and redactor refs), gateways, channel sources, and reply refs. Read-only — it never invokes the engine’s real loader, so it’s safe to run against a file you’re mid-edit on. --format json for scripting.

sutra dispatch-graph — visualize the routing

sutra dispatch-graph bpmn/transfer.bpmn --format mermaid

Emits a graphviz dot or mermaid diagram of the file’s dispatch tree — nodes are BPMN elements, edges are sequence flows. Paste the mermaid output straight into a markdown viewer to see the shape of a flow you didn’t author, or to sanity-check q:dispatch/q:case routing before review.

sutra simulate --dry-run — will this channel actually reach my process?

sutra simulate bpmn/transfer.bpmn --channel transfer-request --dry-run

Resolves which process (and start event) a named channel would route to, from the file’s own q:source declarations — no execution, --dry-run is required. If the channel name doesn’t match any declared source, the error lists every channel the file does declare, so a typo is obvious immediately. This is the fastest way to answer “why isn’t my message reaching the process I think it should” without deploying anything.

sutra explain — evaluate a FEEL expression in isolation

sutra explain 'fromAccount.frozen or toAccount.frozen or fromAccount.balance < payload.amount'
sutra explain --context vars.txt 'payload.amount > 100'
sutra explain   # no expression: a REPL on stdin, `:quit` to exit

Built directly on sutra-feel — the same evaluator the engine runs, standalone. --context takes a flat key: value (or key=value) file (values coerce to boolean/number when they parse as such, else stay strings). Use this the moment a gateway takes the branch you didn’t expect: paste the exact condition in, supply the variables you think are in scope, and see what FEEL actually does with them — usually a type coercion or a missing-path surprise, not a logic bug.

sutra coverage check — is my compliance path actually being exercised?

sutra coverage check bpmn/transfer.bpmn                       # drift lint: declared paths still valid?
sutra coverage check --archive deployed.sutra \
    --database-url "$SUTRA_DB_URL" --threshold 100             # correlation-aware, store-backed check

The bare form is a drift lint: every q:coverage declaration must still be an ordered subsequence of a real route through the current flow graph — it catches a path declaration that silently broke when someone reworked the diagram. The --archive form reads the deployment’s seeded coverage flags out of the database its coverage store declares (--database-url names it — any of the three shipped dialects), and fails closed below --threshold — the gate a CI pipeline runs after a test campaign. See The q: namespace for what q:coverage declares and the money-transfer worked example for it end to end.

Tracing one message end to end

Three surfaces, used together:

  1. The audit trail — the definitive record of what an instance did: INSTANCE_STARTED → node-by-node progress → INSTANCE_COMPLETED / INSTANCE_FAILED / INSTANCE_SUSPENDED / INSTANCE_RESUMED. If you have a JSONL audit sink configured (see Logging and audit), replay one instance’s history offline:
    sutra audit-replay <instance-id> --from-jsonl /path/to/audit --until INSTANCE_COMPLETED
    
  2. Traces — if OTel is configured (see Observability), every span carries bpm.instance.id; filter your trace backend on that id to see the sutra.dispatch / sutra.decode / sutra.validate / sutra.execute span chain for exactly that instance, across every resume segment if it went through a wait state.
  3. The admin API — against a live engine, GET /admin/instances/{id} (or GET /admin/instances/by-alias/{key}/{value} if you only have the business correlation key, not the instance id) shows the instance’s current state directly. POST /admin/instances/{id}/cancel is the one instance-level control operation exposed — there is no generic “retry” endpoint; retrying means fixing whatever rejected the input and re-sending it (or, for a parked wait, sending a corrected relay message).

Common failures, worked

“My message got a 200/202 but nothing happened downstream.” Check ack-mode first (see Acknowledgement modes) — on-persist on an HTTP channel means the 202 you got back is only proof of durable receipt, not completion; the actual reply (if any) rides an outbound channel, not the original connection. Then check the audit trail for the instance.

“A message vanished from a broker channel.” A permanent refusal on a broker transport (a body the codec refused, a type it does not accept, no matching start event, a tenant quota) deletes, commits past or rejects the message, and from 0.6.18 logs a WARN with SUTRA.INBOUND.DELIVERY_DROPPED; its cause_code says why. Search the log for that code. See Refused deliveries and connection references.

“The message never became an instance at all.” Run sutra simulate --channel <name> --dry-run against the deployed BPMN first — a messageTypeValue/messageTypePattern mismatch on the q:source is the most common cause, followed by the channel’s codec rejecting the payload outright (SUTRA.INBOUND.CODEC_NOT_FOUND / a FATAL DecodeResult — check sutra describe for what codec the channel is actually bound to).

“A relay message didn’t resume my parked instance.” SUTRA.RUNTIME.RELAY.CORRELATION_NOT_FOUND in the audit trail means the relay’s q:alias expression didn’t match any parked instance’s alias — run sutra explain with the relay payload’s variables against the exact alias expression from the BPMN to see what key it actually produces, and compare it to what the original request set.

“Senders get a 503 with SUTRA.INBOUND.DEDUP_CHECK_UNAVAILABLE.” The delivery carries an event id, and the database didn’t answer the duplicate check, so nothing ran. Fix the database, or wait for it: a retry with the same id is safe, and it runs once. Stopping the database and posting with an event id gives this answer:

{"type":"urn:bpm:diag:SUTRA.INBOUND.DEDUP_CHECK_UNAVAILABLE","title":"SUTRA.INBOUND.DEDUP_CHECK_UNAVAILABLE","status":503,"detail":"event 'outage-1' on channel 'event-in' could not be checked for a duplicate because the dedup store did not answer; it was not processed, and a retry is safe","code":"SUTRA.INBOUND.DEDUP_CHECK_UNAVAILABLE"}

The document also carries an attributes object. Once the database is back, the same POST succeeds, and repeating it succeeds without running the flow again. See Acknowledgement modes for what each transport does.

“Senders get a 503 with SUTRA.INBOUND.DEDUP_IN_DOUBT.” An earlier delivery of the same event is recorded, but a synchronous-call chain still holds that record. It clears when the chain ends. If the chain’s process is gone, or the chain couldn’t release the record, it clears when the sweep runs, within sutra.instance.inflight-chain-max-age (15 minutes by default). Keep retrying.

“A gateway takes the wrong branch.” sutra explain the condition expression directly, with a --context file built from the actual variable values at that point (read them off the audit trail or a trace span if you’re not sure) — this isolates a FEEL semantics surprise from a “my BPMN is wired wrong” problem in one step.

“My coverage report shows a path that should be covered as uncovered.” Confirm the flows listed on the q:coverage declaration are still the exact ids in the current diagram (sutra coverage check bpmn/your-process.bpmn, the drift-lint form) before assuming the flow genuinely isn’t being exercised.

“Every request answers SUTRA.RUNTIME.UNEXPECTED — engine actor is not running.” An execution lane died — something panicked outside the per-dispatch containment, which in practice means the lane’s own build/rebuild failed (the boot log has the panic). The process is a zombie for that lane’s share of the key space and will not heal in place: both health probes report it (GET /sutra/health/live → 503 with data.deadLanes; /ready goes DOWN at the same moment), so an orchestrator with a liveness probe restarts the replica automatically. If you run without probes, restart it yourself and read the boot log for the panic that killed the lane.

“My coverage op now fails with SUTRA.CONFIG.COVERAGE.STORE_MISSING.” Coverage marks are persisted in the coverage data store the deployment declares in datastores.yaml, so the op fails whenever that store isn’t there to write to — rather than answering 0%, because a fabricated 0% is indistinguishable from a real measurement of nothing covered. Two causes, both named in the message: the deployment declares <q:coverage> paths but no coverage store (a package-shape requirement sutra lint errors on with this same code — declare the store; you supply no coverage SQL, since the engine owns that schema and applies it on first use), or it declares one whose connection could not be opened (the boot log carries an error naming that deployment — fix the URL or the environment reference it resolves from). The same code with a different message means the store opened but the read failed; the underlying error is in the message. See Coverage.

Next

Upgrading to 0.6.18

Two changes in 0.6.18 can break a client that worked on 0.6.17: reads of GET /sutra/instances* now need the admin key, and a synchronous HTTP flow with no <q:reply> now answers an empty body. This page covers those two and points to the rest. No archive format, deploymentId or snapshot format changes. Two additive migrations run at boot: V304 (a nullable inbox_seen.instance_id) and V1002 (a deployment_endpoint_holder table). The tier-3 (Kubernetes) suites were not run for this release.

Reading instances takes the admin key

GET /sutra/instances, its ?alias= search and GET /sutra/instances/{id} now sit behind the admin gate when admin auth is on (sutra.admin.auth.* or sutra.admin.oidc.*).

  • No credential, or a wrong one, gets 401. A valid OIDC token without the admin role gets 403.
  • Send what /admin/* takes: the admin key in X-API-Key (or the header you set with SUTRA_ADMIN_AUTH_HEADER), or the bearer token.
  • POST /sutra/instances/{id}/cancel is not part of this change.
$ curl -si "$B/sutra/instances?alias=orderId:O-100"
HTTP/1.1 401 Unauthorized

{"error":"unauthorized","reason":"missing or invalid admin auth key/secret"}

An engine with no admin configuration at all still answers these routes without a key. The gate applies only when admin auth is configured, so an engine that never set it up behaves as before. Anyone who can reach it can list instances and read one by id, including a value found through an alias. Either configure admin auth (see Configuration reference) or keep that engine where only operators can reach it.

A flow with no <q:reply> answers an empty body

On a synchronous HTTP channel (ack-mode: on-complete, the default), a flow with no <q:reply> now answers 200 with an empty body. Every earlier version, from 0.2.0-rc.1 through 0.6.17, returned the instance’s whole variable map as JSON: event.body, the request’s headers including the caller’s X-Api-Key, and every variable the flow set. A declared reply, and status codes, are unchanged.

What to do:

  1. List the HTTP channels whose flow has no <q:reply> and whose callers read the answer.
  2. Add a <q:reply> that returns exactly what the caller needs, and change the client to read only that.
  3. If a log, a proxy or a client stored the old answers, they hold those credentials. Rotate the keys.

Also changed

  • A 202 from an on-persist HTTP channel that used to be empty now carries a JSON body when the flow replied nothing: Acceptance receipts.
  • A second deployment on a taken route, queue or consumer group is refused, and a new version drains the old one whole: One endpoint, one deployment.
  • A declared <q:source dedupKey> now always dedupes. The recognised forms are header.<name>, ce.id, body.<path> and amqp.message-id. A new package with any other form is refused with SUTRA.PARSE.Q_SOURCE.DEDUP_KEY_UNRECOGNISED; one already deployed keeps running, dedupes nothing as before, and logs the code as an error. A repeat that used to run now answers as a duplicate. Don’t key on a secret header: the value is stored in inbox_seen and can appear in refusal text and logs.
  • Messages with no content type, refused deliveries, and references in connection properties: Refused deliveries and connection references.
  • A package with no tenant or module label on the db source gets a slot of its own and no longer replaces another unlabeled package. Undeploy the old one first, or label it.

Upgrading to 0.6.19

Check these against what you run:

  • A connection property that starts with a secret scheme and a colon. env:, secret:, vault:, aws-secrets:, azure-kv: and gcp-secret: now mean a reference in the properties listed below, so a Kafka broker called vault written bootstrap.servers: vault:9092, or a queue called env:orders, now fails to resolve and stops the channel. Write such a value through a default: ${BROKER:vault:9092}. Other text with a colon, such as kafka:9092, is unaffected.
  • A ${NAME} that 0.6.18 refused. It now resolves, so the channel starts. Before 0.6.18 such a value was used as a literal name, so if you move a Kafka group.id from a literal ${NAME} to a resolved one, it is a new consumer group and starts from auto.offset.reset (default earliest), not from the old group’s offsets.
  • RAW_BODY_UNMASKED and RAW_BODY_STORED warn in more places. They now also fire on a flow that reads a store a masking flow wrote raw text to. These are warnings, and RAW_BODY_UNMASKED becomes an error in a later release.

There is no migration, archive format, deploymentId or snapshot format change.

Upgrading to 0.6.20

2026-11-10. A superseded version now keeps serving the channels only it declares, until it retires, where 0.6.18 and 0.6.19 drained it whole. Another owner is refused such an endpoint until then. A literal local:// send to another package, or to another version of the same package, is refused at lint and deploy. The engine uses about half the CPU it did under load: evaluation no longer copies the variables, and jemalloc is the allocator. Coverage can be checked against requirements, in named compliance cycles. RabbitMQ and AMQP broker credentials accept secret: references. Two additive migrations run at boot: V1003 (deployments) and V904 (coverage, at the store’s next first use). There is no archive format, deploymentId or snapshot format change.

Before you upgrade

Check these against what you run:

  • A q:send to a literal local:// address that is not one of the package’s own transport: local channels. Another package, another version of this package, and a bare local://<name> are now refused by sutra lint, sutra package and every deploy path, with SUTRA.CONFIG.SEND.LOCAL_DESTINATION_CROSSES_PACKAGE. A destination containing ${ is not judged. A stored db deployment with such a send keeps running and logs the error at each load. A file in a deployments folder with one goes Failed at the next scan, including the first boot after the upgrade. Replace the send with an outbound channel the other package listens on.
  • A calledElement, or a <q:dispatch> case or default, that names no process of the package is refused the same way, with SUTRA.CONFIG.CALL_ACTIVITY.UNKNOWN_CALLED_ELEMENT, with the same exemption for stored rows and the same effect on a folder. A calledElement that a <q:dispatch> replaces is never used, so it is not checked.
  • An old version can now hold an endpoint for as long as it has running instances. If you move an endpoint from one module to another (another owner) while the old module’s previous version is draining, the deploy is refused until that version retires. A version with instances parked indefinitely holds its channels indefinitely. /sutra/deployments shows the count under drainingInstances.
  • New traffic can start on the old version after the flip, on the channels it keeps. Moving that traffic to the new version is up to your integration.
  • New warnings. SUTRA.CONFIG.PROPERTY.BLANK_REFERENCE, SUTRA.DEPLOY.ENDPOINT.CHANNEL_CHANGED, SUTRA.DEPLOY.ENDPOINT.START_ON_WAIT and the lint warning SUTRA.CONFIG.CHANNEL.START_AND_WAIT are described below. None of them refuses anything.

Upgrading a deployment in place

When a new version of a tenant and module takes over, the old version drains as before: it takes no new work on the channels the new version has, and its running instances finish on it. New in 0.6.20, an inbound channel that no newer version declares, by name or on any of its endpoints, stays served by the old version, starts included, until the old version retires at zero running instances and zero pending outbox. SUTRA.DEPLOY.ENDPOINT.SUPERSEDED names those endpoints, and the timer starts that stop at the drain:

[WARN] SUTRA.DEPLOY.ENDPOINT.SUPERSEDED — deployment acme--orders--2.0.0 takes over from acme--orders--1.0.0, an earlier deployment of the same tenant and module. acme--orders--1.0.0 drains: it takes no new work, and its running instances finish on it. Channels only it declares stay served by it until it retires: POST /orders/release, POST /orders/legacy.
  • What the old version keeps is recomputed at every change. Undeploying the new version gives the old one back every channel it declares, the ones it shared with the new version included, until it retires; then they stop. Undeploying the old version itself, while it is active, stops them.
  • Retirement is rechecked. A version that just retired is checked again until it is quiet twice; one that had work arrive after all goes back to draining.
  • A version that keeps feeding itself over its own channel never retires. The engine does not stop it. GET /sutra/deployments now shows each draining version’s running instances under drainingInstances ({"<deploymentId>": n}), and GET /sutra/deployments/{id} gains instances for a draining version. The draining array is unchanged.
  • Versions already draining when you upgrade the engine stop serving as before. Only a version superseded on 0.6.20 keeps channels.
  • In a deployments folder, removing the old version’s file stops its channels, and the answer survives a restart.

Another owner is refused an endpoint a draining version keeps, until it retires:

[SUTRA.DEPLOY.ENDPOINT.COLLISION] http endpoint 'POST /orders/release' stays with draining deployment acme--orders--1.0.0 (2 running instances) until it retires

0.6.19 let another owner take it at once. POST /admin/deployments/check gives the same answer.

Upgrade warnings. These name what changes and how many instances are parked on it, and never print an auth value:

  • SUTRA.DEPLOY.ENDPOINT.CHANNEL_CHANGED: the deploy changes a channel an older version has instances parked on: its inbound auth, codec, schema, address or transport, or “renamed or replaced” when its endpoint is now served under another channel name.
  • SUTRA.DEPLOY.ENDPOINT.START_ON_WAIT: a new start event listens where the older version waits. Start events are tried first, so a reply meant for a parked instance starts a new one.
  • SUTRA.CONFIG.CHANNEL.START_AND_WAIT: sutra lint warns about the same shape within one package.

Unchanged: a resume on a channel the new version changed is processed with the new version’s binding; an old instance parked on a channel the new version serves under another name is still unreachable from it (now warned); a retry across a version flip still runs again.

Schema: migration V1003 adds deployment_archive.retains_channels on PostgreSQL, MySQL and SQL Server. It is FALSE for rows drained before it existed.

The walkthrough is Upgrading a deployment in place; One endpoint, one deployment describes how the other deployment paths behave.

Less engine CPU under load

  • Expressions evaluate without copying the variables. A FEEL dotted path walks nested values by reference and copies only the value it ends on. Data tasks, exclusive gateways and q:param inputs evaluate against the live variables, and each operation still sees the previous one’s result. A strict template engine gets a render model built without null entries in one pass, with the same JSON and key order.
  • jemalloc is the engine’s allocator. It is a default feature, jemalloc, of sutra-dist, and is never linked on MSVC targets. A build with --no-default-features drops it, along with the other defaults, and uses the system allocator. jemalloc fixes its page size when it is built: an arm64 image build sets JEMALLOC_SYS_WITH_LG_PAGE=16 (64 KiB), so the binary also runs on kernels with 16 or 64 KiB pages. If you build your own arm64 binary, set it too, or the binary aborts at start on such a kernel. amd64 is unaffected.
  • Measured on a laptop at 200 messages/s for 60 s, every message completed, none duplicated: steady engine CPU fell from about 316% to about 139% of one core, roughly 42% from the evaluation change and a further 24% from jemalloc. With database writes off, 267% to 121%. jemalloc used less CPU than mimalloc and had the lower p99. The runs were bound by the laptop’s disk, so their latencies are only indicative; the CPU figures are the reliable ones. A relay benchmark at 500 messages/s showed no change.

Coverage against requirements, in compliance cycles

The walkthrough is in Coverage.

  • A requirements map. A package can ship coverage/requirements-map.yaml, mapping each declared path to requirement ids, a weight (default 1, above zero) and the test cases that walk it. sutra coverage check then adds a compliance object: per-requirement and weighted percentages, uncovered paths with their cases, the cases to run, and paths with no cases or no mapping. --threshold gates the weighted percentage. Every declared path counts; there is no way to exclude one. A package with no map reports as before. Lint: SUTRA.CONFIG.COVERAGE.MAPPING_ROUTE_UNKNOWN (error, a key that matches no declared path) and SUTRA.CONFIG.COVERAGE.MAPPING_ROUTE_UNMAPPED (warning, a declared path the map leaves out, counted at weight 1).
  • Cycles. sutra coverage cycle declare|activate|deactivate <LABEL> and sutra coverage cycle list, plus --cycle on coverage check and coverage reset. While a cycle is active, every mark for that deployment records to it, including the coverage:report and coverage:reset operations. A mark belongs to the cycle active when its instance completes. One cycle can be active per deployment: a second activation is refused (SUTRA.RUNTIME.COVERAGE.CYCLE_ALREADY_ACTIVE), as is activating an undeclared label (CYCLE_UNDECLARED); reactivating a cycle continues it. With none active, coverage records as before, and check and reset without --cycle read the marks recorded outside any cycle. Cycles belong to a deployment id, so a changed package starts with none. The sutra.coverage.percent gauge and the path counter are not cycle-aware.
  • To re-run a case, send <cycle>:<case>:<attempt> as the idempotency key on a channel that declares idempotency-key-header: a repeat of an attempt does nothing, the next attempt runs.
  • Schema: V904 adds four coverage tables (coverage_cycle, coverage_active_cycle, coverage_cycle_metric, coverage_cycle_fragment) on PostgreSQL, MySQL/MariaDB and SQL Server. Existing rows are untouched and read as “no cycle”.

Connection references

See References in connection properties.

  • RabbitMQ and AMQP broker credentials accept secret:KEY and ${secret:KEY}. username and password now take exactly the whole-value forms the reference resolver resolves, plus k8s:secret/<name>#<key>. 0.6.19 refused the two secret: forms there. A literal, or text with a reference inside it such as pre${X}, is still refused with SUTRA.STARTUP.REFUSED.HARDCODED_SECRET.
  • RabbitMQ now accepts a lowercase ${name} in credentials, as AMQP already did. It used to refuse it.
  • A reference that resolves to blank now warns SUTRA.CONFIG.PROPERTY.BLANK_REFERENCE, naming the channel, the property and the reference as written, never a value. 0.6.19 applied the transport’s default silently: RabbitMQ host became localhost, Kafka group.id became sutra-<channel>. That still happens, and a blank required property is still refused, now after the warning. The explicit-empty forms ${NAME:} and ${secret:KEY:} do not warn, so write an optional setting that way. This covers the connection properties of kafka, rabbitmq, amqp, gcp-pubsub, file and aws-sqs, not credentials or data-store URLs.

Delivery and data stores

  • Each failed delivery attempt that will be retried now logs a WARN with the destination, the cause’s code and message, the attempt number and next_attempt_at. Before, a retried attempt logged nothing, so a callback that arrived seconds late gave no cause.
  • A closed in-process hand-off no longer delays delivery by 30 seconds. Outbox rows are inserted due after a short grace for the in-process hand-off. When the hand-off had closed before the commit, its rows waited out that grace. Now exactly those rows are made due at once, if they are still as inserted (never a leased, deferred or poisoned row), and the dispatcher is woken, so the next poll tick delivers them. Delivery stays at-least-once with the same dedup.
  • A coded data-store refusal keeps its code. An edited migration that was already applied refuses the store with SUTRA.RUNTIME.DATASTORE.MIGRATION_CHECKSUM, but the step failed with SUTRA.RUNTIME.UNEXPECTED, so the dead letter’s cause_code and its log line never showed the cause. It now fails with the store’s code and logs at ERROR, naming the store and the script, never the key. Routing is unchanged: a non-idempotent flow is still dead-lettered, an idempotent one requeued.

Fixed in the documentation

  • sutra.outbox.retry.base-delay defaults to PT1S, as the code always had it; the configuration guide said PT5S.
  • The 0.6.18 notes said a dropped delivery’s audit record has an empty instance_id. No instance owns it, so it is NULL: select it with instance_id IS NULL.

Tooling

  • Load steps report resources. tools/sutra-load-test run.sh and relay-bench.sh now report engine CPU and RSS and PostgreSQL CPU, IOPS (pg_stat_io) and WAL volume for each step, from step-metrics.sh, as a table and a .metrics.json. A metric it can’t read prints unavailable, never 0.

Security

  • RUSTSEC-2026-0459 is ignored in cargo audit, time-boxed. It is in libsqlite3-sys 0.30.1, which is in Cargo.lock only because sqlx declares its optional SQLite driver through a weak feature. No crate enables SQLite, so it is compiled into nothing: cargo tree -i libsqlite3-sys -e all --target all is empty, and cargo deny, which resolves the real graph, does not see it. No sqlx release allows the fixed 0.39 yet. The ignore is dropped when one does.

For extension authors

  • sutra_envref_spi::is_whole_reference (whether a value is exactly one whole reference form) and is_explicit_empty_default are new.
  • StoreError::with_code and StoreError::code are new in sutra-executor; a store adapter that keeps a refusal’s code passes it through them.
  • OutboxSignal::hand_off in sutra-channels now returns the ids it could not hand off.
  • sutra-channels gains wait_subscriptions and message_types_can_overlap.

Upgrading to 0.6.21

2026-11-13. The kafka and amqp (AMQP 1.0) transports connect to TLS brokers natively, with no sidecar, and authenticate with SASL: Kafka with PLAIN, SCRAM-SHA-256 or SCRAM-SHA-512, AMQP 1.0 with PLAIN. Brokers are verified by default. FEEL gains password hash and password verify (argon2id), so a flow can sign a user in against a password hash in a typed store without storing or logging the password. The CLI’s self-update works on macOS, and paths printed on Windows no longer carry a \\?\ prefix. There is no migration, and no archive format, deploymentId or snapshot format change.

Before you upgrade

Check these against what you run:

  • Kafka’s kafka.* passthrough now refuses sasl.*, ssl.* and enable.ssl.certificate.verification. A channel that sets one of them stops at startup. Use the typed properties below instead. They could not have secured a connection before, because the engine’s Kafka client was built without TLS.
  • A Kafka sasl.username, sasl.password or tls-ca-pem written as a literal is refused with SUTRA.STARTUP.REFUSED.HARDCODED_SECRET, like the AMQP and RabbitMQ credentials. Write a reference.
  • A plaintext AMQP sink whose credentials the broker refuses now reports SUTRA.OUTBOUND.AMQP.BROKER_AUTH_FAILED, where it reported SEND_FAILED. Update an alert that matches the old code.
  • The engine binary now includes its own OpenSSL 3.6.3, for the Kafka client (librdkafka has no other TLS backend). The engine still loads no system libssl: readelf -d lists only libgcc_s, libm, libc and the loader. An image scanner cannot see this copy; it is tracked through rust/Cargo.lock, so cargo audit reports an advisory filed against the openssl-src crate. The image is about 8 MB larger. Building from source now needs perl, for OpenSSL’s build.

TLS and SASL on Kafka and AMQP 1.0

For an inbound channel:

PropertyTransportMeaning
security.protocolkafkaPLAINTEXT (default), SSL, SASL_PLAINTEXT or SASL_SSL
sasl.mechanismkafkaPLAIN, SCRAM-SHA-256 or SCRAM-SHA-512
sasl.username, sasl.passwordkafkaReferences, never literals
tls: trueamqpUse TLS. The default port becomes 5671
username, passwordamqpReferences. SASL PLAIN is used when both are set
tls-ca-filebothA PEM bundle used in place of the system CAs. Takes references, like an address
tls-ca-pembothA reference to PEM text, used in place of the system CAs
tls-insecure-skip-verifybothtrue turns off the certificate and host-name checks
  • Verification is on by default, against the system CAs (SSL_CERT_FILE and SSL_CERT_DIR are honoured), unless a CA setting replaces them. tls-insecure-skip-verify: true is the only way to turn it off, and the engine logs a warning whenever it applies.
  • A setting that could not take effect stops the channel at startup instead of being ignored: a CA setting without TLS, sasl.* without a SASL protocol, both CA settings at once, or SASL GSSAPI or OAUTHBEARER.
  • The protocol switches stay literal. tls, security.protocol, sasl.mechanism and tls-insecure-skip-verify do not expand references; addresses, credentials and tls-ca-file do.
  • Outbound sends use one engine-wide setting per transport, from the environment: SUTRA_SINK_KAFKA_SECURITY_PROTOCOL, SUTRA_SINK_KAFKA_SASL_MECHANISM, SUTRA_SINK_KAFKA_SASL_USERNAME, SUTRA_SINK_KAFKA_SASL_PASSWORD, SUTRA_SINK_KAFKA_TLS_CA_FILE, SUTRA_SINK_KAFKA_TLS_INSECURE_SKIP_VERIFY; and for amqp10s:// destinations SUTRA_SINK_AMQP_TLS_CA_FILE and SUTRA_SINK_AMQP_TLS_INSECURE_SKIP_VERIFY, with the existing SUTRA_SINK_AMQP_USERNAME and SUTRA_SINK_AMQP_PASSWORD.
  • New diagnostics: SUTRA.INBOUND.KAFKA.BROKER_TLS_FAILED, SUTRA.INBOUND.KAFKA.BROKER_AUTH_FAILED, the same two under OUTBOUND.KAFKA, INBOUND.AMQP and OUTBOUND.AMQP. BROKER_TLS_FAILED means the broker’s certificate is not trusted or does not name the host, or the trust settings could not be read; BROKER_AUTH_FAILED means the broker refused the credentials. Neither prints a password or anything the broker sent back during SASL. A consumer keeps retrying and an outbound message stays in the outbox, so both recover once the setting is corrected. The AMQP consumer backs off between attempts; the Kafka client retries quickly and logs each attempt, so a wrong CA gives dozens of lines a second: alert on the code, not the line count.
  • What it was tested against: Apache Kafka 3.8 and Apache ActiveMQ Artemis, in local containers. Managed Kafka services and Azure Service Bus have not been tested and are not claimed to work; Azure Service Bus’s CBS token flow is not implemented. Client certificates (mTLS) are not supported.

The walkthrough is Connecting to a TLS and SASL broker: a message through both brokers, a refused login and an untrusted CA.

Password hashing in FEEL

  • password hash(password) returns an argon2id hash in PHC string form ($argon2id$v=19$m=19456,t=2,p=1$…), with a fresh 16-byte salt from the operating system. The parameters are fixed in code. An empty or non-string password gives null.
  • password verify(password, hash) returns true or false, comparing in constant time. It returns null for a malformed hash, one that is not argon2id, or one that asks for more than m = 65 536 KiB, t = 10 or p = 4, so a stored hash cannot make a verify arbitrarily expensive.
  • password hash is denied where an expression must give the same answer on replay, like uuid(): sutra lint refuses it there with SUTRA.FEEL.DETERMINISM.UNSAFE_BUILTIN. password verify is deterministic and allowed everywhere.
  • Cost: one verify takes about 22 ms (criterion median 22.3 ms, range 21.1 to 23.8 ms, on an Intel i7-10750H laptop, release build).
  • Keep the plaintext out of every surface by reading the sign-in message into a variable marked transient and sensitive: it is then neither persisted nor written to the audit log or the engine log. Signing a user in against a store builds such a flow, and checks the database dump, the audit file and the engine log for the password.
  • New dependency: argon2 0.5 (with password-hash and blake2), with default features off.

CLI on macOS and Windows

  • sutra self-update works on macOS. It passed GNU tar’s --wildcards, which macOS’s bsdtar refuses. It now unpacks the archive into a scratch folder and takes the binary from there.
  • Windows paths are printed without the \\?\ prefix. Package loading, sutra lint, sutra create and the documentation generators resolve paths without it, so a printed path or a path comparison matches what you typed. The catalog’s JSON repoRoot showed it.
  • sutra compat-baseline against a git reference gives tar a relative archive name, because GNU tar, which Git for Windows puts first on PATH, reads C:\… as a remote host.
  • These were tested on Linux, which exercises the Linux-visible half of each. A new hosted job, cli-portability, runs the CLI and package-loader tests and clippy on macOS 15 and Windows Server 2025 for every change; its first run is this release’s.

Building and CI

  • make ci runs the hosted pipeline’s checks locally, cheapest first: make fmt-check, make lint, make cross, make test, make audit and make scan-deps.
  • make test runs cargo-nextest when it is installed, plus the doctests, and otherwise cargo test; both run every test before reporting. The hosted pipeline installs nextest.
  • make cross runs clippy for the CLI and its workspace crates against Windows (MSVC) and both macOS targets, skipping a target whose C toolchain the host lacks.
  • The workflows pin every action by commit and every runner by version, and are checked with actionlint. A new architecture test fails the build when a workflow’s Rust toolchain differs from rust-toolchain.toml.
  • .gitattributes makes text files LF in every checkout. The BPMN and DMN schemas published by the OMG are kept byte for byte as published.

Security

  • The image scan configuration’s justifications for the base image’s libssl3 now say why: the engine links no libssl, and its Kafka client’s OpenSSL is compiled in (see “Before you upgrade”).
  • Seven new base-image findings are ignored, time-boxed, with their reasons in .trivyignore.yaml: five in libssl3 (CMS, QUIC and AES-GCM-SIV), which the engine never loads and whose code paths its compiled-in OpenSSL does not use for TLS over TCP, and two in glibc (regex and the wide scanf family). Debian ships no fix for any of them.
  • quinn-proto 0.11.19 and quinn-udp 0.5.16 in Cargo.lock, for RUSTSEC-2026-0523, RUSTSEC-2026-0524 and RUSTSEC-2026-0538. Neither crate is compiled into the engine or the CLI; they are in the lockfile only as an optional dependency of reqwest that no crate enables (cargo tree -i quinn-proto -e all --target all is empty).

For extension authors

  • sutra_transport_spi::is_secret_reference is new: whether a value is a whole secret reference (any registered scheme, ${…}, or k8s:secret/<name>#<key>).
  • sutra-transport-amqp gains AmqpTlsProperties, register_amqp_sink_with_tls, sink_tls_settings_from_env and AmqpMessageSink::with_tls_settings, and AmqpChannelProperties a tls_settings field.
  • sutra-transport-kafka gains KafkaBrokerSecurity, KafkaSinkSecurity, register_kafka_sink_with_security and KafkaMessageSink::with_security, and KafkaChannelProperties a security field.
  • BrokerFailureRecorder is new. Code that builds AmqpChannelProperties or KafkaChannelProperties with a struct literal must add the new field.

Upgrading to 0.6.22

2026-11-15. A message now answers the instance waiting for it before it starts anything, with one version of a package and across versions while an older one drains. A retry that spans a version flip is a duplicate. A repeated Kafka TLS or SASL failure is logged once with a count instead of hundreds of times. Large templates compile once per process. The engine image’s CA bundle no longer trusts two roots Debian distrusted (DLA-4845-1). Two additive migrations run at first boot: V305 and V306 (inbox). There is no archive format, deploymentId or snapshot format change.

Before you upgrade

Check these against what you run:

  • A channel with both a start event and a wait of a compatible message type. A message that correlates to a parked instance now resumes it and starts nothing, also with one version. sutra lint already warns these with SUTRA.CONFIG.CHANNEL.START_AND_WAIT. A flow that re-sent a business key on such a channel to start a second instance now answers the first.
  • A message whose key names a running instance that has not reached its wait yet is refused with a retryable WAIT_PENDING answer instead of starting a new instance.
  • A retry across an upgrade is a duplicate (202, naming the original instance). A changed key header, dedupKey path or channel name produces another key, and that message runs again.
  • A non-owning database role needs SELECT, INSERT on the new deployment_owner table, which inbox_seen’s row-security policy reads.
  • inbox_seen gets an index built concurrently at first boot (V306). It adds I/O in proportion to the table and blocks no writes. Two replicas booting together build it once.

Waits first

The walkthrough is A reply answers the instance waiting for it.

Before a start event is resolved, the engine asks whether the message answers a wait an instance of the channel’s owner is parked at, in any of the owner’s live versions. A hit resumes that instance and is the whole delivery. A miss is dispatched exactly as before.

  • A resume is processed as the version it is parked on: its own binding, codec, validators and reply codec, under the address and auth of the version that serves the channel. A message that version 2’s codec refuses can still resume a parked version 1 instance under version 1’s codec; it can never start anything.
  • A second process waiting on one channel is now tried; before, only the first was.
  • On a broadcast: true channel the process holding the correlated wait resumes and does not start; every other process’s start on the channel still runs.
  • Cost: nothing on a channel no live version waits on (an index answers in a few hash lookups). On a channel with a wait of a compatible type, one correlation lookup per live version that waits.

Measured (the neutral relay benchmark, tools/sutra-load-test/relay-bench.sh, 500/s for 60 s, 32 lanes, a pool of 40, 32 outbox workers, durable PostgreSQL; two runs each, interleaved, on one machine; release-it builds):

buildsettles/ssettle p50settle p99WAL flushes/settlecommits/settleCPU ms/settle (engine+PG)
0.6.21497.7 / 498.526 / 27 ms116 / 130 ms0.59 / 0.5713.2 / 13.26.18 / 6.61
0.6.22498.6 / 498.727 / 26 ms200 / 76 ms0.58 / 0.5913.2 / 13.36.68 / 6.62

Every order settled exactly once in every run. The differences are within run-to-run noise (p99 spans 76 to 200 ms across the four runs).

Dedup per owner

A dedup record now carries its owner, so a key version 1 recorded is a duplicate on version 2, and two versions racing one key commit it once. Records written before the upgrade dedupe within their own deployment only (no backfill), so a retry spanning both the engine upgrade and a version flip can run again.

Fixes

  • Kafka: against a wrong CA, librdkafka reported every reconnect, each time with different text, and the engine logged about 740 BROKER_TLS_FAILED lines in 20 s. The first failure under a code is now logged in full, and repeats at most once per 30 s with their count, until a success resets it.
  • Templates: compiling a Handlebars template costs time in proportion to the square of its size (about 0.8 s for 100 KB and 3.4 s for 200 KB). Each engine lane compiled each template again on its first render after every activation. A deployment’s templates are now compiled when it is deployed (logged as deployment templates precompiled, with the time it took), once per process, so the first message pays nothing: a 210 KB template’s first message answered in 16 ms. A template that does not compile is refused by sutra lint as before, and fails at render as before if it reaches one.
  • AMQP 1.0 sends are bounded. Opening the connection (TLS and SASL included), beginning the session, attaching the sender and waiting for the broker to settle the message each get at most SUTRA_SINK_AMQP_OPERATION_TIMEOUT_MS (default 30000). A broker that stops answering fails the attempt with the retryable SUTRA.OUTBOUND.AMQP.SEND_TIMEOUT and the outbox sends again; before, one delivery could wait forever. A send that timed out waiting for the broker to settle it may already have been accepted, so the retry can deliver it twice: receivers dedupe on sutra-outbox-key.
  • CA bundle: the image carries a CA bundle that no longer trusts “AAA Certificate Services” and “emSign ECC Root CA - G3” (DLA-4845-1).
  • rustix 1.1.6 (RUSTSEC-2026-0486).

Known

  • SUTRA.INBOUND.CAPABILITY_MISMATCH answers HTTP 500. A body the channel’s codec or format contract cannot take (a nested body on a flat csv channel, a content type the codec does not accept) is the caller’s fault, and a later release will answer it with 415. Nothing changes in 0.6.22.

Upgrading to 0.6.25

2026-11-19. Templates gain an xmlEscape helper, so free text relayed into an XML message stays well-formed. Timers no longer fall behind under sustained load: up to 0.6.24 the engine fired at most about 64 timers a second for the whole cluster, whatever the shard count, so at higher rates every parked timer fired later than the one before it. A distribution’s version line may carry its build without the build reaching image tags. There is no migration, and no archive format, deploymentId or snapshot format change.

Before you upgrade

Check this against what you run:

  • If you expect timer surges, run more than one shard. Set sutra.engine.shards (SUTRA_ENGINE_SHARDS, default 1; see Configuration). Due timers now fire as fast as they fall due. With one shard, that work shares the single execution lane with new work, so new instances answer more slowly while a timer backlog drains. In the test below, at 200 new instances a second, each parking a 1 s timer, the median new-instance answer went from 2.5 ms to 59 ms with one shard, and from 1.9 ms to 2.6 ms with four.

Timers drain instead of firing one batch per tick

Every 500 ms the leader replica’s timer poller looks for due timers. Up to 0.6.24 it claimed at most 32 per deployment, fired them and waited for the next tick. That is about 64 timer fires a second for the whole cluster. Above that rate the backlog grew for as long as the load lasted: instances started on time, and every timer-driven step after them ran later and later.

Now each tick keeps claiming while a claim comes back full, taking the deployments that are behind in turn so one backlog cannot starve another. It stops when every claim comes back short, after one second of draining (or the tick, if that is longer), or when the replica stops being the leader. Each round’s timers finish firing before the next claim. Up to max(shards, 16) timers fire at once. Timer start events are unchanged. There is no new configuration key, and a timer still fires at least once.

Measured on a debug build against PostgreSQL 16. Each instance starts over HTTP, parks a PT1S timer, then sends a follow-up message; the lag is the follow-up’s arrival against its due time.

ShardsRate, for 15 sFollow-up lag p99, 0.6.240.6.25
1200 a second31.8 s0.78 s
4400 a second78.2 s0.50 s

The lag that remains comes from the 500 ms tick.

xmlEscape

{{xmlEscape value}} escapes & < > " ' in free text for XML element content and attribute values. A null or missing value renders empty. Default {{x}} interpolation is unchanged. Reference: Template helpers. Walkthrough: Escaping free text in an XML template.

For distributions

sutra_cli::product_version() reads a distribution’s version up to the first (. A version block that starts 2.0.0-rc.6 (1a2b3c4, built 2026-11-16) reports 2.0.0-rc.6, so the runtime image tag, the release tag create app scaffolds and codecs no longer carry the build suffix. A version block with the build on its own line works as before.

Upgrading to 0.8.0

2026-11-19. A synchronous HTTP answer sets its own status and headers with <q:reply status="…"> and the reply’s <q:header> children. It adds an attribute to the q: namespace (urn:sutra:q:1.0), which is why this release is 0.8.0. The addition is backward compatible: existing packages load and run unchanged, and an answer without the attribute or headers is unchanged. Broker outages can reach the deployment as messages. FEEL durations keep fractional seconds. No migration, and no archive or snapshot format change.

Coming from the 0.6.25 image: there is no 0.7.0 image

The v0.7.0 tag, its CLI archives and its crates.io crates were published, but its release run stopped at the engine image scan on a base-image advisory published the same day (CVE-2026-95690, below), so there is no ghcr.io/startr-trade/sutra:0.7.0 image and no 0.7.0 GitHub Release. 0.8.0 carries every 0.7.0 change. If you run the 0.6.25 image, move straight to 0.8.0 and read Upgrading to 0.7.0 as well: its migration (V103) and its behaviour changes (data-store revisions seeded from the database clock, uncorrelated messages logged and audited) arrive with this image.

Before you upgrade

Check these against what you run:

  • A FEEL duration with a fraction on a unit other than seconds is now refused. duration("PT1.5H") used to be cut to PT1H; now duration() cannot parse it, as XML Schema and the FEEL specification require. Write PT1H30M or PT5400S. See Durations with fractional seconds.
  • .seconds on a days-and-time duration returns the fraction. duration("PT4.5S").seconds is 4.5, where it was 4. An expression that needs a whole number should apply floor().
  • A fractional seconds value is no longer dropped. duration("PT0.001S") was zero, so a timer whose FEEL expression returned it fired at once; it now waits 1 ms. string() keeps the fraction.

Status and headers on a synchronous answer

On an HTTP channel that answers synchronously (ack-mode: on-complete, the default), a <q:reply> with no destination can set the status (digits, or a FEEL expression), and its <q:header> children become the response’s headers. A flow can answer 201 with a Location, 404 or 409, or act as an auth check that answers 200 with identity headers or 401 with a challenge. Statuses outside 200–599, a 3xx without a Location, and headers a flow may not set are refused at load by sutra lint, sealing and deploy alike; a computed status or header value that cannot be sent is a 500 that names the header, never its value.

Reference: Status and response headers. Walkthrough: Answering with a status and headers.

Broker outages as messages

Set sutra.engine.connectivity-events.channel (env SUTRA_CONNECTIVITY_EVENTS_CHANNEL) to a channel name, declare that channel in a package (transport: local, codec: json), and each broker outage start and end the AMQP 1.0 transport sees on one of the package’s channels arrives as a JSON message there. Unset, nothing changes.

Reference: Broker connectivity events. Walkthrough: Broker outages as messages.

For transport authors

sutra_channels::connectivity (ConnectivityReporter, ConnectivityEvent, current_send(), re-exported by sutra-transport-spi) and the default method InboundIntake::connectivity(), a no-op unless the engine installs a sink. Existing transports compile unchanged and report nothing.

Image scan

CVE-2026-95690 (libssl3, DTLS) in the distroless base has no fixed Debian package. The engine never loads libssl3: on the 0.8.0 binary, readelf -d lists libgcc_s, libm, libc and the loader as its only libraries and the binary names no libssl to load at run time, and nothing in it speaks DTLS. The entry is time-boxed in .trivyignore.yaml to 2026-12-16 with the other base-image entries.

Upgrading to 0.7.0

2026-11-19. A data task can delete a store row with <q:store key="…" delete="true"/>. It adds an attribute to the q: namespace (urn:sutra:q:1.0), which is why this release is 0.7.0. The addition is backward compatible: existing packages load and run unchanged. A data-store row’s revision now starts from the database clock, so a row deleted and created again never reuses a revision. A message that correlates to no live instance is recorded, and a channel can send it to a process instead of refusing it. The AMQP 1.0 transport works with IBM MQ. Codecs can seal what they send and resolve secret references in their own bundle files. One migration, V103, runs at first boot on PostgreSQL. There is no archive or snapshot format change.

0.7.0 has no engine image. Its CLI archives and crates were published; its image was not (see Upgrading to 0.8.0). Engine image users take these changes with 0.8.0.

Before you upgrade

Check these against what you run:

  • Rows created after the upgrade start from a large revision, not 1. An insert into a data store now seeds rev with the database clock in microseconds since the epoch (about 1.8 × 10^15 in 2026); each write still adds 1. Rows written before the upgrade keep their revisions. Anything outside the engine that reads rev and expects a new row at 1, or expects revisions with no gaps, must stop expecting it. The rev column must be a 64-bit integer (BIGINT), as in every example in Data stores; check a typed store whose table you created yourself.
  • V103 builds an index concurrently at first boot (PostgreSQL: alias_index_retired_lookup, a partial index over the correlation keys of ended instances). It blocks no writes and adds I/O in proportion to the table. From now on a completing instance writes one more index entry per correlation key it retires.
  • A message that correlates to no live instance is now a WARN log line and an audit record. A sender that keeps answering ended instances shows up in both. The HTTP answer is unchanged.
  • A local:// delivery for an instance that has ended is given up at once. The refusal no longer carries sutra.waitPending=true when every instance that carried the key has ended, so it is not retried for about four minutes first.
  • If you expect timer surges, run more than one shard (sutra.engine.shards, SUTRA_ENGINE_SHARDS, default 1; see Configuration). Unchanged since 0.6.25: due timers fire as fast as they fall due, and with one shard that work shares the single execution lane with new work.

Deleting a data-store row

delete="true" on a write removes the key’s row instead of writing a value. Deleting a missing key is a no-op. With expect="unchanged" the delete applies only while the row’s revision is the one the instance read; otherwise it fails with SUTRA.RUNTIME.DATASTORE.CONFLICT, which an error boundary catches. A delete runs where a write runs (on a <bpmn:transaction> scope’s store transaction, else on its own), waits behind a forUpdate row lock, and invalidates the read cache like a write. It works on key/value and typed stores on PostgreSQL, MySQL, MariaDB and SQL Server. Combinations that make no sense for a delete are refused at load with SUTRA.PARSE.STORE.DELETE_INVALID.

Reference and walkthrough: Deleting a row.

Revisions start from the database clock

Until 0.6.25 every insert started a row at revision 1. Once a process can delete rows, a key can go from revision 1, to deleted, to created again at revision 1 with another value. Another replica’s read cache would then serve the old row, and an expect="unchanged" write or delete that read the old row would apply. Now the insert reads the database’s own clock inside the insert statement, so every replica shares one clock. Revisions keep going up by one per write and never repeat for a key, but they have gaps. This relies on the database clock moving forward: if it is stepped backwards between a delete and the re-create, the guarantee can break.

A message no instance waits for

A message on a channel a wait subscribes to that finds no live instance carrying its key (SUTRA.RUNTIME.RELAY.CORRELATION_NOT_FOUND) is now recorded whatever transport carried it: a WARN log line with SUTRA.INBOUND.RELAY_UNCORRELATED, and an inbound.RELAY_UNCORRELATED audit record. Both name the channel, the message type, the dedup key and the keys looked for, never the body or the headers, and sensitive key values are masked.

uncorrelated: { start: <processId> } on the channel in channels.yaml sends such a message to a process of the package instead, with no BPMN change. Details and a run of both: An answer no instance waits for.

IBM MQ

The amqp transport now works with an IBM MQ queue manager’s AMQP channel. Two fixes apply to every broker: each connection has its own container id (IBM MQ closes an open connection when another opens with the same id, so concurrent sends and replicas cut each other off), and the sink sets the message’s to property (IBM MQ refuses a message without it). New options, all off by default, pick queue or topic addressing, persistence, expiry, the body form and the message properties; the source and sink log when the broker is lost and when it is back. Also changed: the sink sends the outbox key as message-id, and the source renders a binary message id as hex.

Setup and a walkthrough: Connecting the engine to IBM MQ.

For codec authors

PayloadCodec gains two default methods, inert unless a codec overrides them: seal, the last transform of an outbound message before the outbox commit (an error refuses the message with SUTRA.OUTBOUND.SEAL_FAILED), and bind_refs, which resolves secret: and env: references named in the codec’s bundle files once, when the deployment is planned (an error refuses the deployment with SUTRA.CODEC.BIND_REFS_FAILED). Existing codecs are unchanged. See Domain neutrality and the SPI model.

Fixed

  • A deployment flip could serve an older set of versions, or route a channel to a version whose lanes were not built yet (database deployment source). Concurrent re-activations could interleave, so a request for the new version’s channel answered 404 (RESOLVE.CHANNEL.UNKNOWN), or a broadcast briefly had one subscriber. Re-activations are now serialized, and each reads the store afresh.
  • Building the engine image from source could ship stale code when several checkouts were built on one host: the Dockerfile’s shared cargo cache let changed crates pass as fresh. The build now decides by content. Published images are not affected.

BPMN 2.0 element support

Sutra executes BPMN 2.0. This page explains the rule that decides what it accepts; for the element-by-element answer, read the generated table rather than any prose:

rust/crates/sutra-bpmn/bpmn-support.md

That file is generated from the engine’s own Node enum and is compile-enforced: three exhaustive matches mean a new node kind does not compile until it is described there, and a test loads a minimal process per element to check the page against the loader rather than against a comment. It therefore cannot drift from the implementation — which a hand-maintained list here would.

The rule

Where BPMN 2.0 already spells something the engine does, the engine accepts the standard spelling and maps it onto existing behaviour. The q: namespace carries what BPMN does not model, and stays visibly an extension.

In practice that means four things:

  1. A conformant element the engine can execute is accepted, even where an extension already expressed the same thing. <task> loads as the pass-through <manualTask> already is.
  2. An element the engine genuinely cannot execute is rejected, loudly, with the diagnostic naming what to model instead. Silent inertness is worse than a refusal: a model that loads and quietly ignores half your diagram is harder to debug than one that will not load.
  3. Accepting something silently when you probably meant otherwise is itself a defect. <task> loads and sutra lint warns, because an author who meant <serviceTask> and typed <task> has written a step that does nothing.
  4. Where BPMN and q: both spell something, BPMN’s spelling is an alias onto the same model, not a second mechanism — and declaring one name both ways is a lint warning rather than a silent precedence.

What is rejected

One flow node: <event>.

That is not a gap. tEvent is abstract="true" in the BPMN 2.0 XSD — <event> is the substitution-group head that concrete events substitute for, so no conformant document can use it as a flow node and there is nothing for any engine to implement.

A handful of forms are also refused, each with a reason the diagnostic states:

FormWhy
<timeCycle> on an intermediate catcha token parks there exactly once, so later occurrences have nothing to resume
<timeCycle> on an interrupting boundarythe first firing cancels the host, so later occurrences are unreachable
a bounded R<n> cycle on a boundarythe remaining budget would have to survive the host’s park, which needs durable per-instance state the engine does not keep. Use R/<interval>
@instantiate="true" on <receiveTask> / <eventBasedGateway>process instantiation needs the intake-selection surface a <startEvent> has
eventGatewayType="Parallel"meaningful only when instantiating; treating it as Exclusive would invert “wait for all” into “wait for one”
an <eventBasedGateway> arm that does not parkan arm that resolves instantly makes every other arm unreachable
a durable park inside an inline sub-processthe inline runners cannot park mid-scope — model it at the top level

Things that are accepted but carry no token

<dataObject> / <dataObjectReference>, <laneSet> / <lane>, artifacts and <documentation> are modelling constructs, not flow nodes. A data object does now contribute a declared variable when a data association names it — see the q: namespace for how that interacts with <q:variables>.

The wait-state column is the one to read first

A process containing any node that parks is stateful, and ineligible for synchronous execution. Most “does Sutra support X?” questions are really “does X park, and what does that imply” — which is why the generated table has a column for it, checked against the engine’s own Node::is_wait_state().

sutra CLI reference

The single sutra binary (crate rust/crates/sutra-cli) carries the whole authoring-to-operations toolchain: scaffolding, validation, packaging, deployment, and analysis. Build it from source:

cd rust && cargo build --release -p sutra-cli    # -> rust/target/release/sutra
cargo run -p sutra-cli -- <args>                  # or run ad hoc

Or install a published build — no Rust toolchain needed. The CLI runs on linux x86_64/arm64, macOS x86_64/arm64 and windows x86_64; see installation. The engine image it deploys to is Linux only, so on macOS and Windows the CLI is native and the engine is a container.

Commands are grouped below by workflow: author → validate → package → deploy → operate → analyze, with the generators last.

Global conventions

ConventionBehavior
--format <FORMAT>text (default) or json for report commands; dot or mermaid for dispatch-graph
-v / -vv / -vvvLog verbosity (info / debug / trace); logs always go to stderr, reports to stdout
Exit 0Clean run, no findings
Exit 1Findings — the input has a diagnosable problem (breaking compat change, routing miss, coverage below threshold)
Exit 2Usage or infrastructure — bad flags, missing files, unparseable input, unreachable database

Database-touching commands (migrate, coverage check/reset, crypto) share one set of connection options, each with an env fallback: --url (SUTRA_DB_URL), --user (SUTRA_DB_USERNAME), --password (SUTRA_DB_PASSWORD), and — for migrate — --schema (SUTRA_DB_SCHEMA). These are the CLI’s own database-connection names, distinct from the engine’s SUTRA_DATASOURCE_* — see Configuration reference.


Author

sutra create app

sutra create app <NAME> [--dir <DIR>]

Scaffolds an application workspace: a sample standalone deployment package under packages/<name>-main/ plus deploy assets (a compose file, a deployments drop-directory, a k8s manifest, a health-gated smoke script). Idempotent-safe — existing files are never overwritten. See Quickstart and Anatomy of an app.

sutra create deployment

sutra create deployment <NAME> [--dir <DIR>] [--from <PACKAGE_DIR>]

Scaffolds a fresh standalone deployment-package skeleton, or copies an existing package with --from — the explicit variant model: packages never inherit, a variant is a copy. See Deployment packages.

sutra create bpmn

sutra create bpmn <PROCESS> [--package <PACKAGE_DIR>] [--validation fatal|soft] [flags]

Generates a process with the validation-gateway wiring (plus accepted/rejected reply templates) into a package, verified through the engine’s own BPMN loader before being written.

FlagMeaning
--validation <MODE>fatal (default): only FATAL outcomes take the rejected branch; soft: FATAL and SOFT_ERRORS both reject
--channel <NAME>Inbound channel the start event subscribes to (default <process>-in)
--message-type <TYPE>Inbound message type (default <Process>Request)
--forceOverwrite an existing user-edited file (never implicit)

sutra create ci

sutra create ci --provider github|bitbucket [--dir <DIR>]

Writes the pipeline for an existing workspace — .github/workflows/sutra.yml or bitbucket-pipelines.yml — as four steps in the order that fails cheapest first:

  1. sutra package over every directory under packages/ — fail-closed validation, so a package the engine would refuse to load never reaches a deployments directory;
  2. sutra generate docs … --check — the committed catalog still matches what it documents;
  3. deploy/smoke.sh against the real engine and the archive just sealed;
  4. publish — the catalog book to GitHub Pages, or printed to a PDF release artifact on a v* tag under Bitbucket.

The provider is explicit rather than detected: a scaffolded pipeline lands in a repository that usually already has one, with its own runners and conventions, and guessing would be guessing wrong in the file most likely to be deleted on sight.

The pipeline checks documentation and never writes it. Regenerating in CI would author bot commits and race step 2 against its own output. When step 2 fails the fix is local — run sutra generate docs and commit the refreshed pages alongside the change that caused them, which is what keeps the diff reviewable.

The release it installs is read from the same distribution seam self-update uses, pinned to the version of the CLI that wrote the file. A distribution’s CLI, its engine image and its release channel are one matched set: a pipeline that fetched some other distribution’s binary would resolve a different codec registry than the archive it had just sealed.


Validate

sutra lint

sutra lint <PACKAGE_DIR | MANIFEST.yaml> [--overlay <DIR>]... [--active-set <PATH>]... [--engine-url <URL>]

Runs the full package-time validation suite (the same fail-closed checks sutra package runs) and emits nothing on success. The fast pre-flight before every package.

--overlay composes base layers underneath <PACKAGE_DIR> before validating — the same composition package performs, so you lint exactly what would be sealed. See Composing a package from layers.

Given a target, lint also runs the check a deploy runs. --active-set names sealed archives, or directories of them, standing in for what the target runs. --engine-url asks a running engine (POST /admin/deployments/check; --api-key / --api-key-header as for deploy), which links every transport it can serve and resolves connection references in its own environment. --active-set judges only the transports the CLI links (HTTP) and reports a channel on any other as SUTRA.DEPLOY.ENDPOINT.UNCHECKED. An exclusive endpoint another deployment holds is the error SUTRA.DEPLOY.ENDPOINT.COLLISION; a data store that would share rows is the warning SUTRA.DEPLOY.DATASTORE.SHARED_CONNECTION. See One endpoint, one deployment.

Given a deployment manifest (a .yaml file) instead of a directory, lint first prints the resolved stack — each layer in order, each overridden file with the layer that won and the one it overrode, and each .ignore removal — then the findings; --format json carries the same under a stack key. See Deployment manifests.

sutra describe

sutra describe <BPMN_FILE> [--format json]

Prints a structural summary of a BPMN file: processes, events, tasks, gateways, channels. See Troubleshooting BPMN solutions.

sutra dispatch-graph

sutra dispatch-graph <BPMN_FILE> --format dot|mermaid

Emits a graphviz or mermaid diagram of a BPMN file’s dispatch tree.

sutra simulate

sutra simulate <BPMN_FILE> --channel <CHANNEL> --dry-run

Reports which process an inbound channel message would route to. --dry-run is required — routing report only, no execution.

sutra explain

sutra explain [EXPRESSION] [--context <FILE>]

Evaluates a FEEL expression — one-shot, or a REPL on stdin when the expression is omitted. See Rules: DMN, FEEL, and .srl.

sutra compat-baseline

sutra compat-baseline --baseline <PATH_OR_REF> [--current <PATH>] [flags]

Compares current BPMN signatures against a baseline directory or git ref and reports breaking changes — a CI gate on process-contract compatibility between releases.


Package

sutra package

sutra package <INPUT | MANIFEST.yaml> [--overlay <DIR>]... [-o|--out <DIR>]

Seals a deployment-package directory into one immutable .sutra archive, running the full validation suite fail-closed first. The archive manifest (per-file digests, the content-addressed deployment id) is derived, never authored. See Deployment packages.

--overlay composes one or more base layers underneath <INPUT> — each applied in the order given, with <INPUT> applied last and winning any conflict — so a shared folder plus a small per-variant folder seal into one archive without being copied together by hand. An overlay may not contain package.yaml, overridden paths are reported, and nothing is ever deleted. The resulting deployment id is identical to sealing a hand-merged directory of the same content. See Composing a package from layers.

<INPUT> may instead be a deployment manifest — a .yaml / .yml file naming the package labels and an ordered stack of layer folders, each optionally mounted with at: and able to drop lower-layer files with a .ignore. It seals to <manifest-file-name>.sutra, byte-identical to sealing the same merged tree by hand; --overlay flags stack above its last layer:

sutra package deploy/customer-a.2026r1.prod.yaml -o dist/   # → dist/customer-a.2026r1.prod.sutra

See Deployment manifests for the format and a worked customer / edition / environment example.

sutra deployments list

sutra deployments list <DIR> [--label KEY=VALUE]...

Inspects a directory of sealed .sutra archives — lists each archive’s deployment id and labels.

sutra openapi

sutra openapi <ARCHIVE>

Emits a sealed archive’s generated OpenAPI 3.1 surface (channels → BPMNs, message types, endpoint nature, data stores) — the same document the engine serves live per deployment id.


Deploy

sutra migrate

sutra migrate [status|verify] [--url <URL>] [--dry-run] [flags]

Applies engine schema migrations to the engine-internal database, or inspects them (status — applied vs. pending; verify — ledger integrity, expected head, checksum drift).

sutra crypto provision-dek

sutra crypto provision-dek --key-id <KEY_ID> --kek <REF> [--url <URL>] [flags]

Provisions a KEK-wrapped per-tenant data-encryption key for envelope encryption of sensitive instance variables at rest.

sutra deploy

sutra deploy [ARCHIVE] [flags]

Hot-deploys a sealed .sutra archive onto a running engine. See Deploy, hot-deploy, and rollback and Deployment model for the full mechanics.

[ARCHIVE] may also be a deployment manifest (.yaml): it is packaged first, exactly as sutra package would, and its target: block supplies any of the flags below that the command line leaves out — the command line wins. See Deployment manifests.

FlagMeaning
--apiDeploy via the engine’s synchronous admin API instead of a ConfigMap patch; requires --engine-url
--asyncWith --api: submit async, then poll until Active
--watch <PKG_DIR>Watch a package source directory and re-deploy on change (validate-then-deploy loop); implies --api
--wait / --wait-timeout <SECS>ConfigMap path: poll until Active
--engine-url <URL>Engine base URL for --wait / --api
--api-key <KEY> / --api-key-header <HEADER>Admin auth for --api (default header X-API-Key)
--secret <KEY=VALUE> / --secret-from <FILE>Estate-secret keys ensured/merged before the ConfigMap patch

A deployment claiming an exclusive endpoint another active deployment holds is refused with SUTRA.DEPLOY.ENDPOINT.COLLISION and the old state serving (exit 1). On the ConfigMap path deploy runs the check first, asking the engine when --engine-url is given. A new version of one tenant and module may take an endpoint from an older one; the older drains whole (SUTRA.DEPLOY.ENDPOINT.SUPERSEDED warns). --engine-url times out after 30 seconds and says why it got no verdict. See One endpoint, one deployment.

sutra undeploy

sutra undeploy <DEPLOYMENT> [flags]

Removes a deployment — the engine drains it (no new intake) and retires it at zero instances and zero pending outbox.


Operate and analyze

sutra coverage

sutra coverage init  <FILE> [PROCESS_ID]... [flags]   # seed declarations / scaffold admin set
sutra coverage check [BPMN_FILE] [flags]              # drift lint, or the store-backed check
sutra coverage reset --archive <FILE> [flags]         # re-seed the store covered=false

Path-coverage tooling for q:coverage route declarations (intra-process) and coverage/*.yaml files (cross-process) — see Coverage: declared routes as the compliance signal for the full walkthrough of both shapes and every flag above, and Troubleshooting BPMN solutions for reading a report that doesn’t match what you expected.

sutra test simulate

sutra test simulate --deployments <DIR> --datasource <URL>
                    (--advance <DURATION> | --until-quiescent) [flags]

Boots a real engine on a dynamic port against a directory of sealed deployment archives with a virtual clock installed, fast-forwards it, reports, and shuts down — so a PT24H timer or an R3/PT12H schedule settles in wall-clock seconds. Unrelated to sutra simulate above, which is a dry-run routing report over one BPMN file and boots nothing.

FlagMeaning
--deployments <DIR>Directory of sealed .sutra archives to serve (required)
--datasource <URL> / --datasource-username / --datasource-passwordEngine datasource — the engine’s own SUTRA_DATASOURCE_* env names, not the CLI’s SUTRA_DB_* set (required)
--advance <DURATION>Fast-forward the virtual clock by this ISO-8601 duration, firing everything due along the way, then stop
--until-quiescentFast-forward until nothing is armed and nothing is live, or --timeout elapses
--timeout <DURATION>Real wall-clock budget for the engine boot plus the fast-forward loop, either mode — since 0.6.5 a boot that has not finished within it ends the run (exit 2) instead of waiting (default PT30S)
--start <RFC3339>Virtual start instant (default: the real current instant)
--allow-existing-dataProceed even though the datasource already holds instances

Exactly one of --advance / --until-quiescent is required. Safety: the target database must hold no instances or the run refuses with exit 2 — pointing this at a database with real in-flight instances would durably fire their real timers early; --allow-existing-data is the explicit acknowledgement.

Fixed in 0.6.5: at a log level that logs while the engine boots (-v, or RUST_LOG), a package in --deployments made 0.6.4 hang after “schema up to date” and ignore --timeout. The CLI held the terminal’s stderr lock for the whole command, and the engine’s runtime threads block on that lock to write their first log line. The CLI now writes to stderr without holding it, and --timeout covers the boot.

Progress is text on stderr; the final summary is one JSON object on stdout and nothing else touches stdout, so sutra test simulate … | jq . is always safe. See Testing time for the summary’s fields and the embedded seam behind this command.

sutra audit-replay

sutra audit-replay <INSTANCE_ID> --from-jsonl <PATH> [--tenant <TENANT>] [--until <EVENT_TYPE>]

Walks a process instance’s audit events from a JSONL stream — reconstructing what a specific instance did, offline. See Logging and audit.

sutra version

sutra version            # sutra 0.8.3 (1a2b3c4, built 2026-11-15)
sutra --version          # identical text
sutra version --format json

Prints the tool version and the build it came from. The parenthesis names the commit the binary was built from (-dirty appended when tracked files had uncommitted changes) and that commit’s date in UTC. It is never the wall clock, so two builds of one commit print the same line. An unknown part is left out rather than guessed: sutra 0.8.3 (1a2b3c4) without a date, plain sutra 0.8.3 when git is unavailable and no build variable was set. A release build takes both from SUTRA_BUILD_COMMIT and SOURCE_DATE_EPOCH; a local build reads them from git. The engine image’s sutra-engine --version prints the same, and its sutra-engine starting log line carries version, commit and built fields.

The program name is derived from the running binary, not hardcoded: a distribution that embeds this CLI as a library (sutra_cli::run) under its own binary name prints that name, and one that versions itself independently of the engine (sutra_cli::run_with_version) prints its own version with the embedded engine’s underneath:

<tool> 2.0.0
sutra  0.8.3 (engine)

The tool’s own version is its first line read up to the first (, so a distribution may print its build on the same line: <tool> 2.0.0 (1a2b3c4, built 2026-11-16) still reports 2.0.0, which is also the version self-update compares against and the tag of the runtime image create app scaffolds (0.6.25).

--format json is the structured form and always separates the two — version is the reporting tool’s own, engine the embedded engine’s (equal for this binary). commit and built are present when known:

{"name":"sutra","version":"0.8.3","engine":"0.8.3","commit":"1a2b3c4","built":"2026-11-15"}

Generate

The generators all recompute output that is derived, not authored, and all share one --check drift gate that regenerates without writing — which is the shape a CI or pre-commit gate wants. That shared contract is why they sit under one verb. It is also what separates them from sutra create, which scaffolds files that become yours: a scaffold is headed “edit freely — this file is yours” and needs --force to overwrite your edits, while a generated page is headed “Do not edit above the MANUAL NOTES sentinel” and --check fails the build if you edited it. Two verbs, opposite guarantees — never one.

sutra generate docs

sutra generate docs --input <FOLDER> [--output <DIR>] [--check]

Recurses a folder of authored deployment artifacts — BPMN processes, DMN/.srl rules, Handlebars/XSLT templates and their manifests, channels.yaml, package.yaml, coverage files — and emits a deterministic markdown catalog, one page per artifact. It parses through the engine’s own loaders, so each page describes exactly what the engine loads rather than a second parser’s opinion of it. --check generates into a temporary directory and reports drift instead of writing anything. sutra generate catalog is its sibling for Rust source — same --output / --check contract, one page per source file, rooted at --repo-root. A deployment package wants docs; catalog documents the engine’s own source tree.

Each BPMN page opens with a diagram in BPMN notation, auto-laid-out from the process graph — authored BPMN carries no <bpmndi:BPMNDiagram>, so no coordinates are needed in the source. The SVG is embedded in the page, scaled to the column, and linked to a full-size copy written beside it (<name>.svg) that opens in a new tab.

sutra generate schema-handler

sutra generate schema-handler <SCHEMAS_DIR> <OUT_DIR> [--full] [--check]

Compiles a directory of XSD schemas into Rust sources: the decode tables, the canonical map projection, and the shape metadata a schema-bound codec is built on. The schema files are the only input, and emission is byte-identical run to run after rustfmt — which is what makes --check (regenerate in memory, diff against the tree at <OUT_DIR>, exit 1 on any difference) a usable drift gate. The default emission is the slim, data-driven form; --full additionally emits the typed model. A write run touches only the files the generator itself produces and never the hand-maintained ones alongside them — support.rs and Cargo.toml are yours.

Both paths are arguments, not conventions: it is a neutral tool over whatever corpus you point it at, and the crate it emits lives wherever the caller wants it — including in a repository that composes this one.

Template helpers

A Handlebars template (templates/*.hbs) renders in strict mode: a value that resolves to nothing (a missing key, a misspelt path, a null part-way along a path) fails the render. {{x}} renders x verbatim, with no escaping, because templates produce XML, JSON and plain-text messages, not HTML.

The engine registers the helpers below and no others. Helper arguments are the exception to strict mode: a missing path is a null argument, never a failed render. So (eq maybe "X") can test for absence, and {{coalesce maybe "default"}} can supply a default.

HelperRenders
eq a b, neq a btrue or false. Numbers compare as numbers, so 1 equals 1.0.
gt a b, gte a b, lt a b, lte a btrue or false. Two numbers compare as numbers and two strings in code-point order; anything else is false.
and a b …, or a b …, not atrue or false, with Handlebars truthiness: false, null, "", 0 and [] are false.
{{#let value as |name|}}…{{/let}}Its body, always, with value bound to name. A null binding fails the render only where the body uses it.
substring value from toThe characters of value from index from up to, not including, to (to the end if to is left out). A range outside the value fails the render.
replace value "a" "b"value with every a replaced by b, literally (no patterns).
coalesce value fallback …The first argument that is not null.
xmlEscape valuevalue with & < > " ' replaced by &amp; &lt; &gt; &quot; &apos;. A null or missing value renders empty. (0.6.25)

uuid and now are not helpers: the engine puts them in the render model, so a template reads them like any other value.

xmlEscape

Use xmlEscape for any free text that goes into an XML message: a name, a note, an address line. Without it, a customer called O'Brien & Sons or a note containing < produces a message that is not well-formed XML, and the receiver refuses it.

<Receipt customer="{{xmlEscape order.customer}}"><Note>{{xmlEscape order.note}}</Note></Receipt>
  • It is safe in element content and in attribute values in either kind of quotes.
  • Every other character passes through unchanged, non-Latin scripts included.
  • A number or boolean renders in its usual text form. A null or missing value renders empty, where {{order.note}} on a missing value would fail the render.
  • Escaping is opt-in per value. {{x}} stays verbatim, so existing templates render as before. Don’t put xmlEscape on a value that already holds XML markup you want kept: it escapes that too.

Escaping free text in an XML template runs it end to end.

Crates: sutra-feel, sutra-dmn and bpmn-svg

Every crate in the workspace is unpublished by default (publish = false at the workspace level) except three, which are deliberately carved out as standalone, embeddable libraries: sutra-feel, sutra-dmn and bpmn-svg. All are pure Rust, #![forbid(unsafe_code)], and have no dependency on channels or persistence — you can use any of them in a project that has nothing to do with this engine at all.

[dependencies]
sutra-feel = "0.8.3"
sutra-dmn  = "0.8.3"   # depends on sutra-feel directly; use it alone if you only need DMN
bpmn-svg   = "0.1"      # versioned on its own; depends on neither

sutra-feel — the FEEL expression language

FEEL (Friendly Enough Expression Language) is the expression language the OMG DMN specification defines, reused throughout the BPMN/DMN tooling ecosystem wherever a small, side-effect-free expression needs to be embedded in a larger model — a decision rule, a gateway condition, a data-mapping assignment. sutra-feel is a complete, standalone implementation: lexer, parser, AST, and evaluator, with no dependency on DMN or BPMN machinery.

#![allow(unused)]
fn main() {
use sutra_feel::{expressions, FeelContext, FeelValue};

let mut context = FeelContext::new();
context.insert("age".to_string(), FeelValue::Number("42".parse().unwrap()));

let result = expressions::eval("age >= 18", &context).unwrap();
assert_eq!(result, FeelValue::Boolean(true));
}

expressions::eval is the main entry point; expressions::parse parses without evaluating, and expressions::eval_boolean is a convenience wrapper for expressions that must produce a boolean (gateway-condition shaped callers). expressions::paths extracts which context paths an expression reads without evaluating it — useful for static analysis over a set of expressions.

Highlights:

  • Source-position-aware diagnostics (errors carry line/column, pinned to a caller-supplied source URI) — the errors this crate raises are the same ones you’d see from sutra explain (see the CLI reference) or from a deploy-time validation failure.
  • DECIMAL64 numeric semantics (16 significant digits, HALF_EVEN rounding, via bigdecimal) — arithmetic matches the FEEL specification, not native floating point.
  • Full temporal support: dates, times, date-and-time, and durations, including IANA-timezone-aware @Region/City zone-qualified literals, with the timezone database bundled at compile time (no host tzdata dependency).
  • Ranges/intervals, contexts (maps), lists, and function values, plus a documented determinism denylist for expressions that must stay side-effect-free (the engine uses this to keep replay deterministic across a wait-state resume — see Wait states and human tasks).

sutra-dmn — the DMN 1.5 decision engine

DMN (Decision Model and Notation) is the OMG standard for decision tables, decision requirement graphs (DRGs), and business knowledge models, tied together by FEEL. sutra-dmn loads .dmn XML files, validates them, and evaluates them — single decision tables (all seven OMG hit policies: UNIQUE, ANY, PRIORITY, FIRST, OUTPUT ORDER, RULE ORDER, and COLLECT with SUM/COUNT/MIN/MAX aggregation) as well as full DRGs, where a decision depends on other decisions, invokes business knowledge models, or calls decision services. FEEL parsing and evaluation is delegated directly to sutra-feel.

#![allow(unused)]
fn main() {
use sutra_dmn::DmnDecisionEngine;
use sutra_feel::{FeelContext, FeelValue};

let engine = DmnDecisionEngine::new();
let result = engine.evaluate("tier.dmn", TIER_DMN_XML.as_bytes(), &input).unwrap();
assert_eq!(result.get("tier"), Some(&FeelValue::from("GOLD")));
}

DmnDecisionEngine::evaluate is the single-file entry point, returning a map of output-clause name to FeelValue — this is exactly what backs a businessRuleTask in the engine (see Rules: DMN, FEEL, and .srl). For DRG evaluation across a graph of decisions/BKMs/decision services with imports, see the crate’s drg module; for structural validation without evaluation, see DmnRulesetValidator.

Conformance is measured, not asserted — see DMN-TCK conformance for the current standing and what it does and doesn’t cover.

bpmn-svg — BPMN diagrams as SVG

bpmn-svg draws BPMN 2.0 process diagrams as self-contained SVG, with a layered auto-layout for files that carry no BPMN-DI. parse reads standard BPMN into a small model and ignores BPMN-DI and vendor extensions, and render draws it. The same input always gives byte-identical SVG, and an option writes colours as CSS variables for light and dark themes. sutra docgen draws through it. Its README has a runnable example and its known limits.

Why these and not the rest of the workspace

Everything else in the workspace — the BPMN model and executor, the channel/transport layer, the persistence layer, the CLI — is Sutra-engine-internal: coupled to the engine’s own conventions (diagnostics, deployment packages, the q: namespace) in a way that wouldn’t make sense as a general-purpose crate. FEEL and DMN evaluation, by contrast, are useful on their own to anyone building rules or decision logic in Rust, independent of whether BPMN or Sutra is anywhere in the picture, and drawing a BPMN file is useful to anything that handles one — so those three are published, versioned, and supported as standalone libraries.

The model-call crates

Unpublished, and listed because they are the extension points behind the llm transport (see Calling a language model):

CrateWhat it is
sutra-llm-spiThe provider-neutral SPI: request, response and failure types, the LlmProvider trait, the registration each provider crate submits, and a shared JSON/SSE HTTP client. Names no provider.
sutra-llm-openaiThe Chat Completions adapter, serving openai and the compatible services (google-gemini, mistral, deepseek, xai, kimi, meta-llama) or any compatible server via base-url. Streaming and background jobs with webhooks.
sutra-llm-anthropicThe Anthropic Messages API adapter. Streaming; jobs through Message Batches, polled.
sutra-llm-bedrockThe Amazon Bedrock Converse API adapter (aws-bedrock), authenticated with a Bedrock API key.
sutra-transport-llmThe llm transport: ordered candidates with fallback, capability filtering, sync and async calls, jobs, signed webhooks and SSE streaming. Names no provider.

sutra-dist links the three adapters behind the llm-openai, llm-anthropic and llm-bedrock features, all on by default.

Next

DMN-TCK conformance

Sutra’s DMN/FEEL evaluator (sutra-dmn + sutra-feel, see Crates: sutra-feel and sutra-dmn) is checked against the real OMG DMN Technology Compatibility Kit — the same corpus other DMN implementations are measured against — rather than relying on an internal test suite alone.

Current standing

LevelResult
Compliance level 2126/126 (100%)
Compliance level 33365/3385 absolute (99.4%), and 100% of attempted assertions (0 semantic failures among cases the engine attempts)

Measured against the TCK as of 2026-09-17 (commit 0456e1a), including its newest tests for multiple model imports and DMN 1.5’s implicit type conversions.

The gap between “100% of attempted” and “99.4% absolute” is 20 enumerated assertions the evaluator deliberately doesn’t attempt: 14 are reflective execution of {java|pmml: …} external function bodies (the grammar parses and validates their binding shape; invoking one returns a clear, deliberate semantic rejection rather than a wrong answer), and 6 are dates with nine-digit years, beyond the range the engine’s temporal values represent. This is the honest distinction the numbers are built to preserve: a wrong value on a supported construct is a real conformance defect, and the harness that produces these numbers is built specifically to never launder one into an “unsupported” bucket. Level 2 has no such gap at all.

What this does and doesn’t tell you

What it tells you: decision tables (all seven OMG hit policies, including COLLECT with aggregation), decision requirement graphs, business knowledge models, decision services, and the breadth of FEEL itself (temporal types and arithmetic, ranges/intervals, list/context comprehensions, string and numeric builtins, filters and projections) behave per specification across a large, independently-authored corpus — not just the cases Sutra’s own contributors thought to write.

What it doesn’t tell you: conformance to a language specification says nothing about Sutra’s BPMN coverage, its channel/transport behavior, its persistence and replica semantics, or its security posture — those are covered elsewhere in this book (see Building BPMN solutions and Architecture). It also isn’t a performance claim — conformance and throughput are separate properties.

Where this comes from

The conformance measurement is produced by a development-time harness in the workspace — it is a tool contributors run when working on the FEEL/DMN evaluator, not something a user building a BPMN solution ever needs to invoke. See Debugging the engine if you’re contributing to sutra-feel or sutra-dmn and need to reproduce or extend this measurement yourself.

Next

Contributing

Contributions are welcome! This page covers the repo map, the day-to-day dev workflow, the code style and PR bar, and the license. If you’re an AI agent working on this codebase, also read CLAUDE.md at the repository root — it’s a shorter, more mechanical companion to this page written specifically for that audience.

Repository map

The engine, tooling, and libraries are a single Cargo workspace under rust/:

PathWhat it is
rust/The Cargo workspace — engine, CLI, libraries, tools. Start at rust/README.md; build/test tiers in rust/TESTING.md.
docs/This book (mdBook) — book.toml + src/SUMMARY.md.
examples/End-to-end example apps (money-transfer, approval-hold). See Worked example: money-transfer.
deploy/Reusable OpenTofu deployment modules.
xsd/Authoritative XSD schemas for BPMN, DMN, and the q: extension namespace.
openapi/API specifications.
scripts/Repository-level dev/ops scripts.
tools/User-facing tooling: sutra-vscode, sutra-modeler-plugin, sutra-load-test.

Within rust/crates/, the layering described in Engine layering maps directly onto directories — sutra-engine (the library), sutra-dist (the composition root that builds the actual binary), sutra-channels, sutra-executor, sutra-bpmn, sutra-feel/sutra-dmn/sutra-srl/sutra-templates, sutra-persistence, sutra-formats (the built-in formats) alongside sutra-codec-schema, one sutra-transport-<vendor> crate per broker, and the SPI crates each of those builds against (sutra-codec-spi, sutra-transport-spi, sutra-datastore, sutra-envref-spi, sutra-redactor-spi) — see Domain neutrality and the SPI model for exactly how the pieces fit and how to add a new one.

Development workflow

You need a stable Rust toolchain (rustup); Docker is only needed for the container/integration test tiers, and tofu plus a kind cluster only for the Kubernetes tier. make help (from the repo root) lists every target; full detail lives in rust/TESTING.md.

cd rust && cargo build                 # whole workspace, debug
cargo build --release -p sutra-cli     # the `sutra` CLI binary
cargo build --release -p sutra-dist    # the `sutra-engine` binary (the composition root)

Test tiers

Tiermake targetWhat it needs
1make testRust toolchain only — the default gate while iterating
2make test-docker (P=<crate> narrows to one crate)A Docker daemon
1+2make test-allA Docker daemon
3make test-k8sA running kind cluster (make -C deploy/k8s-it init first)

The CI workflow (.github/workflows/ci.yml) mirrors the local gate chain exactly. make ci runs the same jobs locally, cheapest first: make actionlint, make fmt-check, make lint, make cross, make test, make audit and make scan-deps (Trivy over the lockfiles, secrets and IaC files). Every Trivy scan prints its vulnerability database’s UpdatedAt; make scan-deps RELEASE=1 and make scan-image RELEASE=1 fetch the latest database first, as a release gate should. A contributor who is green locally is green in CI. No CI job builds an image: the release builds the image it ships, once, and scans it before tagging it. Tier-2 runs nightly rather than per-PR (a Docker-heavy suite is slow and resource-flaky on a small hosted runner for signal tier-1 already covers on most changes); tier-3 stays a local/milestone gate, since it needs a provisioned kind cluster CI doesn’t have. CodeQL runs on its own schedule, and a weekly Trivy run scans the tree and the newest published image for vulnerabilities disclosed since they shipped.

Tier-1 covers the whole workspace — no crate is held out of it, so a green make test means the whole workspace is green, not a subset of it.

Lint and supply-chain gates

make lint    # cargo clippy --workspace --all-targets -- -D warnings,
             #   plus the sutra-archtest domain-neutrality suite
make audit   # cargo audit (RustSec advisories) + cargo deny check (rust/deny.toml)

make lint failing on a domain-neutrality violation means a business term landed in a crate the gate enforces — move the concrete logic into its own extension crate instead.

The generated catalog (on demand, not committed)

make catalog         # regenerate the artifact-documentation catalog under catalog/
make catalog-check   # verify it's in sync

The catalog — one page per source file plus its dependency relationships, produced by sutra-catalog-gen — is generated on demand and git-ignored; there’s no committed baseline to diff against, so it isn’t part of the CI gate today. Run it locally when you want the dependency picture for a change you’re making; make install-hooks wires an optional pre-commit regeneration hook if you want it kept fresh automatically.

Code style and the PR bar

  1. Discuss anything large first. A new transport, a new codec, an architectural change — open an issue before you invest the time.
  2. Keep PRs focused. One logical change reviews faster than a bundle of unrelated ones.
  3. Be green locally before pushing:
    cd rust
    cargo fmt --all
    cargo clippy --workspace --all-targets -- -D warnings
    cargo test  --workspace
    
  4. Add a CHANGELOG.md entry under [Unreleased] for a user-visible change.
  5. Commit messages are short, imperative, area-prefixed:
    feat(channels): add AMQP 1.0 outbound reply path
    fix(feel): correct DECIMAL64 rounding on division
    docs(concepts): clarify wait-state correlation keys
    

The zero-warning clippy bar and the domain-neutrality gate are both hard requirements, not style preferences — see Domain neutrality and the SPI model for why the latter exists.

Diagrams in this book

Diagrams are mermaid in fenced ```mermaid blocks, rendered by the mdbook-mermaid preprocessor (put it on your PATH, or your local build shows the source instead of the picture). Four rules keep them consistent:

  1. No hardcoded colors — no style … fill:#…, no classDef. The book has a light and a dark theme and mermaid picks its own palette for each; a hardcoded fill is unreadable in one of them. Shape, label and subgraph carry the meaning.
  2. A diagram earns its place by compressing something the reader would otherwise hold in their head — an ordering, a state machine, a fan-in, a comparison. A bullet list drawn as boxes is not a diagram.
  3. Keep it small (roughly a dozen nodes). If it needs more, it is two diagrams.
  4. Label the edges where the label is the information (park, after PT30S, hash(instanceId)), and put one or two sentences under the diagram naming the takeaway.

The diagram must say what the prose says: no fact appears only in a picture, and nothing in a picture contradicts the text.

Diagrams render at their natural size rather than being scaled down into the content column (that scaling is what makes wide diagrams unreadable), so a wide one scrolls inside its own block. Clicking any diagram opens it full size in a new tab, where the browser’s own zoom, save and print apply — which is also the escape hatch when a diagram is genuinely large.

Where designs live

Deeper design rationale than this book covers — why a mechanism was built the way it was, staged plans for work in flight — lives in docs/design/ in the source (private working) repository this public repo is curated from; a subset is rewritten into the chapters of this book as it stabilizes, rather than published wholesale. If you’re contributing a change whose “why” isn’t already captured in this book, a short design note alongside your PR is the right place for it.

License

Sutra is dual-licensed under MIT OR Apache-2.0 (LICENSE-MIT, LICENSE-APACHE). Unless you state otherwise, any contribution you submit is licensed under the same terms, with no additional conditions — you retain copyright over your own contributions.

Please also read our Code of Conduct and Security Policy.

Next

  • Debugging the engine — tracing targets, reproducing a bug at the right test tier, and the development-only tools (like the DMN-TCK harness) that live alongside the workspace.

Debugging the engine

This page is for contributors working on the engine itself — its executor, model, channel dispatch, or the FEEL/DMN evaluators. If you’re building a solution on top of Sutra and something in your BPMN/rules/config isn’t behaving, see Troubleshooting BPMN solutions instead — that page (and its CLI-based tools) is the one written for that audience.

Tracing targets

Every crate logs through tracing, and its module path is the target RUST_LOG filters on — so you can raise verbosity on exactly the layer you’re chasing a bug through without drowning in noise from the rest of the workspace:

RUST_LOG=sutra_channels=debug,sutra_engine::deploy=trace cargo test -p sutra-channels
RUST_LOG=sutra_executor=trace cargo run -p sutra-dist

The default filter is info. When you’re not sure which crate owns the behavior you’re chasing, start from Engine layering — a message’s path through the system (channel → dispatch → executor → persistence) is also the order to add targets in.

Reproduce at the lowest tier that can show the bug

The workspace’s three test tiers (rust/TESTING.md; summarized in Contributing) aren’t just a CI cost control — they’re the natural escalation order for reproducing a bug:

  1. Tier 1 first. Most bugs in the executor, the BPMN model, FEEL/DMN/.srl, or template rendering reproduce with a plain unit or no-docker integration test — no Postgres, no broker, nothing to spin up. Add it as a new module in the crate’s tests/all.rs (a new file under tests/all/ plus one mod <name>; line — not a new top-level tests/*.rs, which would reintroduce a separate link unit) and run it in isolation:
    cargo test -p sutra-executor gateways_test::your_new_case
    
  2. Escalate to tier 2 only if the bug needs real infrastructure — a specific PostgreSQL locking behavior, a RabbitMQ redelivery edge case, a replica-convergence race. Mark the test #[ignore = "docker"] and call sutra_testkit::reap_on_exit(container.id()) on whatever container handle you spin up, so the shared-fixture pattern doesn’t leak it (see rust/TESTING.md’s “Reaper” section for why that call is required rather than relying on Drop).
  3. Escalate to tier 3 only if the bug is genuinely multi-replica or Kubernetes-specific — leader-election handoff, ConfigMap-driven activation timing, ingress-level deploy behavior. This tier needs a provisioned kind cluster (make -C deploy/k8s-it init) and is the most expensive to iterate on — reach for it last, and only once you’ve ruled out that a tier-1 or tier-2 test could show the same thing.

A bug fixed without a new failing test at the tier that caught it isn’t really pinned — the next regression in the same area won’t be caught until it reaches production again. Write the test first, watch it fail against the unfixed code, then fix it.

Catalog-guided navigation

make catalog          # regenerate catalog/ (git-ignored, generated on demand)

sutra-catalog-gen walks every crate and emits one markdown page per source file, plus a bidirectional relationships table — what a file depends on, and what depends on it. When you’re about to change a shared type (an SPI trait, a diagnostic code, a config key) and need to know every call site that will need updating, this is faster and more complete than a plain grep: it’s generated from the actual syn-parsed dependency graph, not a text search that can miss a re-export or an indirect reference.

The DMN-TCK harness — a development tool, not a user-facing command

The DMN-TCK conformance numbers this book cites come from a harness that ships in the workspace but is gated and external: the OMG TCK corpus itself is ASL-2.0-licensed and isn’t vendored into this (unlicensed) repository, so it’s never something a person building a BPMN solution runs. If you’re working on sutra-feel or sutra-dmn:

git clone https://github.com/dmn-tck/tck.git   # or wherever your checkout lives
SUTRA_DMN_TCK_DIR=/path/to/tck cargo test -p sutra-dmn --test tck -- --ignored --nocapture

The harness classifies every assertion as PASS (matches expected), FAIL (a real conformance gap — the engine produced a different value), or UNSUPPORTED (the engine couldn’t evaluate the construct at all — recorded, not counted as a failure). Two env-gated side outputs help when you’re closing a gap: SUTRA_DMN_TCK_DUMP=<file> writes every non-pass outcome for gap analysis, and SUTRA_DMN_TCK_RESULTS_DIR=<dir> writes the official vendor-submission tck_results.csv/tck_results.properties pair. When you land an increment, re-run the harness and update the numbers in DMN-TCK conformance alongside your change — that page is a measurement, and a stale measurement is worse than none.

Next