Introduction
Sutra is a Rust-native, message-native workflow engine built on BPMN 2.0. It turns a standard BPMN process into a declarative way to consume typed, schema-validated messages off any channel, route and correlate them, pause for human decisions, and reply — as a single statically-compiled binary (no JVM, no GC) that is multi-tenant, content-addressed, and hot-deployable.
Most workflow engines treat the payload as an opaque blob and start a process with a REST call. Sutra makes the message a first-class, typed contract: a start event binds to a channel (HTTP or a broker) and a message type; the engine decodes the real wire format, validates it against its schema on the way in, and drives the process with the typed payload, surfacing violations as routable soft errors rather than exceptions. The engine itself ships the schema-less structural formats (JSON, XML, YAML, CSV, raw text and bytes); a typed contract comes either from the deployment package — drop your XSDs into it and the engine compiles them when the package deploys — or from an extension crate implementing the codec SPI, which is how a message standard (an industry wire format with its own envelope grammar and schema editions) is served.
flowchart LR
W["Wire bytes<br/>HTTP · Kafka · SQS · …"] --> C["Channel<br/>transport + auth"]
C --> D["Codec<br/>decode + schema-validate"]
D -->|"typed payload"| P["Process<br/>BPMN 2.0 + DMN"]
D -.->|"violation"| R["Reject path<br/>a branch you modelled"]
P --> O["Reply · emit · park for a human"]
Validation sits in the doorway, not inside your process — which is why a schema violation is a branch you drew rather than an exception you catch.
The engine core is domain-neutral: no business vertical lives in the engine itself — it lives entirely in codecs, channels, and modules. It also collects no telemetry of its own — no usage statistics, no crash reports, nothing phones home, ever; the only data a running engine ever sends anywhere is what you explicitly configure it to export (see No telemetry, no phone-home) — worth stating plainly for an engine meant to carry sensitive, typed messages.
Why Sutra exists
Nearly every production-grade BPMN 2.0 engine is JVM-based; the production-grade native-language orchestrators (Temporal, Cadence, Argo) deliberately aren’t BPMN. Sutra is built for that gap — a native-compiled, standards-oriented BPMN engine that treats the message as the contract.
What Sutra ships
Sutra is four things, and this book covers all of them:
- The engine — a container image built from the
sutra-distcomposition root (docker build -f rust/Dockerfile rust/), the thing you actually run. It force-links the built-in formats, transports, redactors, and secret resolvers into one deployable binary. - The
sutraCLI — one binary (rust/crates/sutra-cli) that scaffolds, validates, packages, deploys, and inspects deployments. See Getting started and the CLI reference. - Two standalone library crates —
sutra-feel(the FEEL expression language) andsutra-dmn(the DMN 1.5 decision-table core). Both compile and run independently of the engine — embed FEEL or DMN evaluation in your own Rust project without pulling in BPMN, channels, or persistence. - This book — Getting Started, Building BPMN Solutions, Architecture, Operating, Reference, and Contributing.
Where these run. The CLI is published for linux x86_64/arm64, macOS x86_64/arm64 and
windows x86_64; the engine image is Linux only (linux/amd64 + linux/arm64). So a Mac
or Windows developer authors, lints, packages and deploys natively, and the engine runs as a
Linux container — the same way it runs in production. The two library crates are plain Rust and
build wherever cargo does. Details in
installation.
Honest capability summary
- BPMN 2.0 coverage. Start/end/intermediate events, all four gateway kinds, embedded / transaction / ad-hoc / event sub-processes, service/script/business-rule/user tasks, data objects and data stores, link/escalation/error events, compensation, and path-coverage instrumentation for compliance reporting — see Coverage: declared routes as the compliance signal.
- DMN + FEEL conformance — measured, not asserted. Sutra’s DMN/FEEL evaluator is checked against the real OMG DMN Technical Compatibility Kit. Current standing: compliance level 2, 100% (126/126); compliance level 3, 99.4% absolute (3365/3385), with 100% of attempted cases passing (0 semantic failures — the remaining gap is 20 enumerated assertions it deliberately doesn’t attempt: external Java function execution, and dates with nine-digit years). See DMN-TCK conformance for what that means and what is deliberately out of scope.
- Channels and transports. HTTP, five brokers (Kafka, RabbitMQ, AWS SQS, Google Pub/Sub, AMQP 1.0), Dapr pub/sub, Knative Eventing, and an air-gapped file transport — nine in all, behind one neutral transport SPI — see Channels and transports.
- Typed codecs — the machinery, not a catalog of standards. Decode and schema-validation are
one step, and what a codec yields is a
messageType, a walkable payload, and a shape every FEEL path in the process is checked against at load time. Six formats are built in (JSON, XML, YAML, CSV, raw text, raw bytes). A schema-bound codec comes from one of two places: the deployment package itself — XSDs underschemas/<name>/, compiled at deploy time, no Rust and no engine build — or an extension crate against the public codec SPI, which is what a message standard (an industry wire format with its own envelope grammar, wrapper profiles, and schema editions) needs and what a downstream distribution force-links for itself. The engine ships no domain codec of its own; see Domain neutrality and the SPI model. - Stateful execution. Wait states (
userTask, intermediate message catch) suspend, persist, and rehydrate on PostgreSQL, correlated by a business key you name — see Wait states and human tasks. State shared across instances lives in a data store, which is key/value by default — one row per key, holding a value of any shape — and which can additionally project a flat record onto real typed columns, so the SQL tooling you already point at that database reads it directly. - What’s still thin. There is no operations console — the admin HTTP API is complete, but a UI over it is roadmap, not shipped. Worker helper libraries beyond the HTTP pull API are ecosystem work that hasn’t happened yet, and macOS has no published CLI binary (it builds from source). Where a chapter below is a stub, it says so explicitly rather than padding.
Where to go next
- Installation — one line for the CLI, one pull for the engine.
- Quickstart — scaffold an app and watch a typed message flow through: decode → validate → route → reply.
- Concepts — the ideas that make Sutra different: typed message
contracts, channels, wait-states, the
q:vocabulary, and content-addressed deployment.
This book is a work in progress. Chapters are being filled in as the 1.0 release approaches.
What’s new in 0.8
Engine 0.8.0 adds one attribute to the q: namespace, so a flow can choose the status and headers
of a synchronous HTTP answer. The addition is backward compatible: a package written for 0.7 or 0.6
loads and runs unchanged. What changes on upgrade is listed in
Upgrading to 0.8.0. 0.8.0 is also the first engine image since
0.6.25: 0.7.0 published no image, so everything in What’s new in 0.7 arrives
with it.
| Feature | What it gives you | Try it |
|---|---|---|
| Outbound delivery no longer queues behind a slow destination (0.8.3) | Each instance’s deliveries form their own ordered chain, under a global bound (sutra.outbox.workers, now 32) and a per-destination bound (sutra.outbox.destination-concurrency, 32). On a 3,500-row burst against a fast and a slow destination, the fast one’s p99 delivery lag went from 73.9 s to 166 ms. New metrics sutra.outbox.delivery.lag, sutra.outbox.in-flight, sutra.outbox.buffered. | Watch outbound lag under a burst |
| AMQP 1.0 channels process deliveries concurrently (0.8.3) | max-in-flight (default 16) intake calls per binding; a message is still accepted only after it is persisted, and 1 keeps the previous one-at-a-time behaviour. prefetch-count now sets the link credit. | Channels |
| FEEL scopes without deep copies (0.8.3) | Filters, quantifiers, for loops, context literals and function calls no longer copy the whole variable scope per item; results are identical. | Rules |
| Status and headers on a synchronous answer | <q:reply status="201"> (digits or FEEL) and the reply’s <q:header> children set the HTTP status and response headers, so a flow can answer like a REST API or an auth check. | Answering with a status and headers |
| Broker outages as messages | With sutra.engine.connectivity-events.channel set, each outage start and end the AMQP transport sees on a deployment’s channel reaches that deployment as a JSON message. | Broker outages as messages |
| FEEL durations keep fractional seconds | duration("PT0.001S") is one millisecond, not zero, through parsing, string(), arithmetic, comparison and .seconds. | Durations with fractional seconds |
| No accepted message lost across a database restart (0.8.2) | A data task that cannot reach its database (a restart breaks every pooled connection) now refuses the delivery for a retry (HTTP 503, broker redelivery, outbox backoff) instead of dead-lettering it, and a step that has run rides out a short outage before giving up, so it is recorded rather than run again. New code SUTRA.RUNTIME.DATASTORE.UNAVAILABLE. | Data stores |
| The AMQP 1.0 sink keeps its connections (0.8.2) | Up to 8 connections per broker (SUTRA_SINK_AMQP_MAX_CONNECTIONS), each with one session and a link per target, instead of a connection per message: on a durable queue 22.3 → 227 messages/s one at a time. Delivery guarantee unchanged. | IBM MQ |
| The SQL audit sink writes in batches (0.8.1) | With sutra.audit.sql on, the audit writer commits once per batch of up to 256 events instead of once per event, so on a durable PostgreSQL the journal keeps up with a busy engine (on a 50 orders/s relay workload it had been 85,770 rows, about five minutes, behind). Still best-effort and behind execution; nothing to configure. | Audit sinks |
What’s new in 0.7
Engine 0.7.0 adds one attribute to the q: namespace, so a process can delete a data-store row.
The addition is backward compatible: a package written for 0.6 loads and runs unchanged. What
changes on upgrade is listed in Upgrading to 0.7.0.
0.7.0 published its CLI and crates but no engine image; the first image with these features is
0.8.0 (see Upgrading to 0.8.0).
| Feature | What it gives you | Try it |
|---|---|---|
| Delete a data-store row from a process | <q:store key="…" delete="true"/> on a data task’s write removes the row; with expect="unchanged" it applies only if the row is unchanged since the instance read it. A row created again never reuses a revision, so no replica’s read cache serves the old one. | Deleting a row |
| Answers no instance waits for | A message that correlates to no live instance is logged and audited, and uncorrelated: { start: … } on its channel can send it to a process instead of refusing it. | An answer no instance waits for |
| IBM MQ over AMQP 1.0 | The amqp transport works with an IBM MQ queue manager’s AMQP channel: queues and topics, plain MQSTR messages with no MQRFH2 header, TLS and SASL, and logged outage windows. | Connecting the engine to IBM MQ |
| Codecs that seal what they send | PayloadCodec::seal and bind_refs let an extension codec sign outbound messages, template renders included, with keys named as references. | Domain neutrality and the SPI model |
What’s new in 0.6
Engine 0.6.0 lets a process work with sound, images and stored recordings, share a limited number of slots across deployments, and loop back to a wait as many times as it needs. Every feature below has a walkthrough you can run on your own machine with the bundled stub model server, so no provider key is needed.
| Feature | What it gives you | Try it |
|---|---|---|
| Audio in model calls | audio parts on a user turn, formats written as media types, audioOutput for a spoken answer, and audio / transcript / audioPreserved on the reply. require: gains audio-input, audio-output, audio-understanding and video-understanding. | Audio, images and video |
voice: pipeline | Speech-to-text, then a text model, then text-to-speech, through an llm.speech provider, for models that take no audio. | The voice pipeline |
| Video and image understanding (preview) | Video clips, sampled frames and a one-frame image part. | Try it |
Native gemini provider | Gemini’s own API, which carries audio, images and video. | Providers |
| Structured answers | A call with json: true gets the parsed answer under json, so a flow reads its fields directly. | The reply |
| Media references and object stores | Named object stores in the engine configuration. A media part gives {ref, sha256} instead of inline data, and the channel reads it from llm.media-store. | Recordings and media references |
q:audit capture="media" | Audit events carry media references, never the media. | Recordings: the audit trail |
| Erasure and retention delete recordings | Subject erasure and terminal retention delete the referenced objects and amend the audit events. | Recordings: erasure and retention |
Capacity pools and session-end: | Named, cluster-wide slot counts drawn from by inbound and outbound channels. A full pool answers 503 inbound and holds outbound requests in the outbox. A session can end before its instance does. | Capacity pools and sessions |
| Re-enterable waits | A loop back to a message catch, user task, timer or channel call parks a new wait each time round. | Waiting at the same node again |
| Timer intervals from FEEL (0.6.4) | A timeDuration, timeDate or timeCycle, a q:timeout and the q:retry delays can each be a FEEL expression over the instance’s variables and config.<key>. Deployment properties: in package.yaml hold per-environment values. | Timer intervals from FEEL |
| Durable throughput (0.6.5) | A concurrent outbox dispatcher, in-process hand-off of local:// deliveries, fewer durable commits per step, separate pools for intake and delivery, and an opt-in data-store read cache. On one laptop with durable PostgreSQL a two-process relay went from 14 to 499 completions per second, p99 250 ms. | Tuning for throughput |
| Answers that beat their wait; less CPU (0.6.6) | A local:// answer that reaches a process before it has parked at the wait for it is held and retried instead of dropped. HTTP clients read the CA bundle once. About a fifth less CPU per completed relay. | An answer that arrives before its wait, Tuning for throughput |
| Binary snapshots; restart-proof local deliveries (0.6.7) | Instance snapshots are stored in a binary container — an older release cannot read it, so upgrade every replica at once. A local delivery that fails because the database restarted is retried instead of poisoned. | The binary container |
| Replicas no longer send twice (0.6.8) | Engines sharing one database each claimed outbox rows another was still delivering, so 0.6.5–0.6.7 sent about half of all outbound messages twice under multi-replica load. The claim now leases its rows (30 s, renewed while in flight). | Outbox processing |
| Autovacuum for the busy tables (0.6.8) | On PostgreSQL the outbox queue and the other high-churn tables have their own autovacuum settings, so dead rows no longer stall the outbox claim at high rates. Lower autovacuum_naptime on the server for high-rate deployments. | PostgreSQL autovacuum |
| Correlation keys on audit events (0.6.9) | Every audit event in the JSONL and OTLP sinks carries the instance’s <q:alias> keys and values, redacted like any other audit content. SUTRA_AUDIT_OTEL_ENDPOINT no longer panics the engine at start, and the JSONL sink writes exactly one event per line. 0.6.9 is also the first release of the 0.6.8 changes above. | Correlation keys on every audit event |
| Key queries on a data store (0.6.10) | A data-task read returns the rows whose key starts with a prefix or lies in a range, in key order and paged, with count and exact-decimal sum aggregates over every matching row, optionally grouped. One index range scan, no locks. expect="absent" claims a key once across replicas, and a data-store conflict can be caught by an error boundary. | Querying rows by key |
| Relays honour a wait’s message type (0.6.10) | Several waits on one channel are told apart by messageTypeValue / messageTypePattern, like start events. Also in 0.6.10: FEEL builtins such as context put and list contains parse in the package lint, and the lint checks assignment expressions and multi-instance loop items. | Several waits on one channel |
| A failed send stays with its failed step (0.6.11) | A process that sent and then failed on its first pass left that send for the next request on the same engine lane, which then failed with an HTTP 500 or delivered the failed process’s send. Every release up to 0.6.10 is affected. | Each step’s sends are its own |
| Outbox renewal never waits; audit resource attributes (0.6.12) | The claim-lease renewal no longer deadlocks with the batch delete of delivered rows (0.6.8–0.6.11) and no longer takes seconds when the engine falls behind. Data stores that share a table no longer race their first use. OTLP audit records carry OTEL_RESOURCE_ATTRIBUTES and the new sutra.telemetry.resource-attributes. | Outbox processing, Resource attributes on audit records |
| Typed-store queries, a migration ledger, deployment manifests (0.6.13) | <q:store query=…> and <q:aggregate> run on typed-column stores as SQL aggregates. Module-store migrations run once, recorded in sutra_store_migrations, refuse an edited script, and take ${package.*}/${store.*} placeholders. sutra lint/package/deploy accept a deploy/<name>.yaml manifest that stacks layers, with per-layer .ignore. | Querying a typed-column store, Migrations run once, Deployment manifests |
| JSON from OpenAI audio models; the reason for a failed model call (0.6.14) | An OpenAI audio model (gpt-audio…) asked for JSON is asked in words, since it refuses response_format; a fenced JSON answer inside prose is parsed. A failed llm call logs the provider’s HTTP status, error code and message at WARN under SUTRA.LLM.CALL.FAILED, keys redacted, and the failed instance’s stored diagnostic carries it. | How JSON is asked for, Finding the reason for a failed call |
| A failed synchronous channel call can be caught (0.6.15) | A permanent failure of a synchronous channel call is a BPMN error on its task, under the failure’s own code: a boundary or an error event sub-process catches it, and the catching path reads callFailure (code, message, node). Uncaught, it fails as before. | Catching a failed call |
A delivery that can’t be checked is refused; event.body is masked (0.6.17) | A delivery with an explicit event id or a declared dedupKey is refused and retried while its duplicate check can’t answer, instead of running unchecked. See Acknowledgement modes. A flow that declares masking now masks event.body and every inbound header too: it used to show them in audit, inspect and snapshots. | |
| One endpoint, one deployment; acceptance receipts; refused deliveries (0.6.18) | A second deployment claiming a taken endpoint is refused at deploy, with the first still serving; a new version may take a route from an older one, which then drains whole. An on-persist HTTP channel answers with an acceptance receipt naming the instance, and a duplicate names its original. A declared <q:source dedupKey> always dedupes, and an unrecognised form is refused. Reading GET /sutra/instances takes the admin key when admin auth is on. A broker message with no content type is decoded by the channel’s codec, and a permanent refusal now leaves a WARN and an audit record. A synchronous flow with no <q:reply> answers an empty body. | One endpoint, one deployment, Acceptance receipts, Refused deliveries, Upgrading to 0.6.18 |
| Keep a message as text; references in connection properties (0.6.19) | A flow can keep a message as raw text in a typed store column and read it back decoded with the new <q:decode> data-task step. event.bodyLossy marks a body whose bytes were not valid UTF-8. The connection properties of kafka, rabbitmq, amqp, gcp-pubsub and file channels now expand references, where 0.6.18 refused them. | Keeping a message as text, References in connection properties, Upgrading to 0.6.19 |
| A superseded version keeps its own channels; about half the engine CPU; coverage cycles (0.6.20) | A version that a newer one supersedes keeps serving the channels only it declares until it retires, and another owner is refused those endpoints meanwhile. A literal local:// send to another package is refused at lint and deploy. Expressions evaluate without copying the variables and jemalloc is the allocator: about 55% less engine CPU at 200 messages/s. Coverage can be mapped to requirements and recorded in named compliance cycles. RabbitMQ and AMQP credentials accept secret:. | Upgrading a deployment in place, Coverage, Upgrading to 0.6.20 |
| TLS and SASL brokers; password hashing in FEEL; the CLI on macOS and Windows (0.6.21) | The kafka and amqp (AMQP 1.0) transports connect to TLS brokers natively and authenticate with SASL (Kafka: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512; AMQP 1.0: PLAIN), verified by default. FEEL gains password hash and password verify (argon2id), so a flow can sign a user in against a typed store without storing or logging the password. sutra self-update works on macOS, and Windows paths print without a \\?\ prefix. | Connecting to a TLS and SASL broker, Signing a user in against a store, Upgrading to 0.6.21 |
| Waits first; dedup across versions; bounded AMQP sends (0.6.22) | A message answers the instance waiting for it before it starts anything, with one version and across the versions of a package while an older one drains; on a broadcast channel other processes’ starts still run. A retry that spans a version flip is a duplicate. AMQP 1.0 sends are bounded (SUTRA_SINK_AMQP_OPERATION_TIMEOUT_MS) and retried; a repeated Kafka TLS or SASL failure is logged once with a count; templates compile once, at deploy. | A reply answers the instance waiting for it, Upgrading to 0.6.22 |
xmlEscape in templates; timers keep up under load (0.6.25) | {{xmlEscape value}} escapes free text for XML element content and attribute values; a missing value renders empty. Timers fire as fast as they fall due, where up to 0.6.24 the engine fired at most about 64 a second for the whole cluster, so timer-driven steps fell further behind the longer a surge lasted. Run more than one shard (sutra.engine.shards) if you expect timer surges. | Escaping free text in an XML template, Template helpers, Upgrading to 0.6.25 |
| 0.6.24: the 0.6.23 release, published | The same code as 0.6.23. Its release run stopped at the engine image scan on a newly published, unreachable libstdc++ advisory; 0.6.24 carries one triaged entry for it, so the image and GitHub Release publish. 0.6.23’s CLI archives and crates.io crates were already published. | Release notes |
Build information in --version; self-update picks the newest release (0.6.23) | sutra --version and sutra-engine --version name the commit and the commit’s date (sutra 0.6.23 (1a2b3c4, built 2026-11-15)), and sutra version --format json carries commit and built. sutra self-update chooses the newest release from a file-store listing by version, whatever the listing’s order, never offers an older one unless --version pins it, and asks for the running binary’s own libc first. | sutra version, Staying current |
| A draining deployment’s instances stay its owner’s (0.6.17) | A message only falls back to draining deployments with the same tenant and module labels as its channel’s deployment. It could previously resume another owner’s draining instance. A package without both labels no longer falls back at all. | |
| Live audio sessions (preview) | AudioSession and require: [realtime], opened through an llm channel by a media transport, not by a process. | Live audio sessions |
Telephony, IVRs, meetings and WebRTC are not part of this engine. They live in the separate open-source repository sutra-voice, which builds on the pieces above.
Before you try a walkthrough
Every walkthrough runs from the root of a checkout of this repository. It needs:
- Docker with Compose v2, plus
curl,python3,sha256sumand GNUbase64(on macOS, usebase64 -i <file>wherever a walkthrough saysbase64 -w0 <file>); - an engine image and the
sutraCLI, both 0.6.0 or later.
The compose files default to the published ghcr.io/startr-trade/sutra:0.8.3 image, and
Installation installs the CLI. To run a walkthrough against
your own checkout instead, build both from it:
make image # builds sutra-rust-engine:dev
(cd rust && cargo build --profile release-it -p sutra-cli)
export PATH="$PWD/rust/target/release-it:$PATH" # the sutra CLI you just built
export SUTRA_ENGINE_IMAGE=sutra-rust-engine:dev # every example's compose file reads this
Each example’s compose file names its own Compose project (sutra-ai-advanced,
sutra-media-recordings, sutra-capacity-pools, sutra-review-loop) and publishes the engine
on a dynamic loopback port, so the stacks never collide with each other or with anything you
already run. Each walkthrough ends by removing its own stack with docker compose -p <project> down -v.
The output shown in each walkthrough was copied from a real run against the stub. Instance ids, timestamps and audio bytes differ on every run.
Installation
Sutra ships as two things, released together and versioned together:
| What it is | How you get it | |
|---|---|---|
sutra | the CLI that scaffolds, packages, lints, deploys and inspects | one-line install, below |
sutra-engine | the runtime that executes your processes | a container image, docker pull |
Neither needs a Rust toolchain. You only need one to build from source, which is the last section on this page.
Install the CLI
curl -fsSL https://raw.githubusercontent.com/startr-trade/sutra/main/scripts/install.sh | sh
On Windows (PowerShell):
irm https://raw.githubusercontent.com/startr-trade/sutra/main/scripts/install.ps1 | iex
The installer downloads the release build for your platform, verifies its SHA-256 against
the release’s own SHA256SUMS, and installs it — into /usr/local/bin when that is
writable, otherwise ~/.local/bin. It edits no shell profile and starts no daemon.
Useful knobs:
# Pin a version instead of taking the newest release
curl -fsSL …/install.sh | sh -s -- --version v0.8.3
# Install somewhere specific
curl -fsSL …/install.sh | SUTRA_INSTALL_DIR=~/bin sh
# Lift the API rate limit when retrying (60 requests/hour anonymous -> 5000 with a token)
GH_TOKEN=$(gh auth token) curl -fsSL …/install.sh | sh
If the install fails
raw.githubusercontent.com rate-limits and answers 429 Too Many Requests under load; so
does the GitHub API, at 60 requests an hour per IP without a token. Three ways around it, in the
order worth trying:
# 1. the SAME installer, served from the release instead of the raw CDN (different host, and
# the script is versioned with the binaries it installs)
curl -fsSL https://github.com/startr-trade/sutra/releases/download/v0.8.3/install.sh \
| sh -s -- --version v0.8.3
# 2. a token, which moves the API to 5000 requests an hour
GH_TOKEN=$(gh auth token) curl -fsSL …/install.sh | sh
# 3. no installer at all — the assets are plain files
# (swap in your target: aarch64-unknown-linux-musl, aarch64-apple-darwin, x86_64-apple-darwin)
gh release download v0.8.3 -R startr-trade/sutra \
-p 'sutra-*-x86_64-unknown-linux-musl.tar.gz' -p SHA256SUMS
sha256sum --ignore-missing -c SHA256SUMS
tar xzf sutra-*-x86_64-unknown-linux-musl.tar.gz --strip-components=1 -C ~/.local/bin
The installer retries transient failures and times out rather than hanging, and it resolves the
release from three different endpoints (/releases/latest, the release list, then git tags),
because each of them has been observed failing while a release was perfectly installable. When
all three come up empty it tells you to pin --version, which is the one path that needs no
lookup at all.
Prefer to see what you are running before you run it? Download install.sh, read it — it is
about a hundred lines of POSIX shell — then execute it. Or skip the script entirely and grab
the archive yourself from the releases page.
Verify the install:
sutra --version
It prints the version and the build it came from, for example
sutra 0.8.3 (1a2b3c4, built 2026-11-15): the release commit and that commit’s date.
Published targets
linux x86_64, linux aarch64, macOS arm64, macOS x86_64, windows x86_64.
| x86_64 | arm64 | |
|---|---|---|
| Linux | ✅ static musl | ✅ static musl |
| macOS | ✅ built natively | ✅ built natively |
| Windows | ✅ MSVC | — |
The Linux builds are static musl, so there is no glibc version to match — they run on an old distribution and a new one alike. The macOS builds are compiled on Apple hardware of each architecture rather than cross-compiled.
This table is the CLI. The engine is a container, and its image is Linux only
(linux/amd64 + linux/arm64) — which is not a limitation in practice: on macOS or Windows you
author, lint, package and deploy with the native CLI, and the engine runs as a Linux container
through Docker Desktop, exactly as it does in production. There is no native macOS or Windows
engine build, and none is planned.
On Apple silicon that container runs natively, which is why the absence costs so little. The
image is a multi-arch manifest, so docker pull resolves to linux/arm64 on its own, and that
arm64 build is compiled on a native arm64 runner rather than cross-built or emulated. Docker on
macOS is always a Linux VM, and on Apple silicon that VM is arm64 — so you get native-speed
execution of the same artifact production runs. (The reverse would be emulated: an linux/amd64
image on Apple silicon goes through Rosetta 2 or QEMU, which is slower and occasionally fails
outright. Nothing here asks you to do that.)
On an Apple-silicon Mac running the installer under Rosetta 2, uname -m answers x86_64, so
you get the Intel build — it runs, translated. Pass --version and fetch the
aarch64-apple-darwin archive by hand if you want the native one.
The macOS binaries are unsigned and un-notarized. That matters only for how you obtained
them: a file downloaded by a browser carries com.apple.quarantine, and Gatekeeper refuses
it until you clear the attribute —
xattr -d com.apple.quarantine ./sutra
— while an archive fetched by the installer (or any curl) has no such attribute and simply
runs. The installer does not strip it for you: quietly removing a security attribute is not an
installer’s business.
Get the engine image
docker pull ghcr.io/startr-trade/sutra:0.8.3
One generic image runs every application: behavior comes entirely from the deployment packages you mount or deploy into it. The quickstart starts it for you with Docker Compose, so you can skip this step if you are heading straight there.
Staying current
sutra self-update --check # is there a newer release? (changes nothing)
sutra self-update # replace this binary with the newest release
sutra self-update --runtime # …and pull the matching engine image
sutra self-update --runtime-only # only the engine image
Updates are never automatic — nothing runs on a timer or as a side effect of another
command. Every download is checksum-verified against the release’s SHA256SUMS before it is
installed, and the binary is replaced by an atomic rename, so an interrupted update cannot
leave a half-written executable on your PATH. --version <tag> pins or rolls back; without it,
self-update only ever moves to a newer release. Release candidates count as older than their
release, and compare by number (rc.10 is newer than rc.9).
Keeping the CLI and the engine on the same release is worth doing deliberately: they are
built, tested and published from one tag, which is what --runtime is for.
Build from source
You need a stable Rust toolchain. Docker is needed only for the container test tier.
git clone https://github.com/startr-trade/sutra.git
cd sutra
make test # the no-docker suite
cargo install --path rust/crates/sutra-cli # install the CLI you just built
docker build -t sutra-engine:dev -f rust/Dockerfile rust/ # the engine image
Next
Quickstart — a running engine and your first message flowing through it, in about five minutes.
Quickstart
A running engine and your first message flowing through it. About five minutes, and the only
prerequisites are the sutra CLI and Docker — on linux, macOS or windows
(x86_64 everywhere, arm64 on linux and macOS). The CLI is native on each; the engine runs as
a Linux container.
What you will have at the end: a BPMN process, bound to an HTTP channel, that decodes and schema-validates an XML message, branches on the result, and replies — running on the real engine against a real PostgreSQL, not a simulator.
flowchart LR
A["sutra create app<br/>scaffold"] --> B["sutra package<br/>seal + lint"]
B -->|".sutra archive"| C["deploy/deployments/<br/>watched directory"]
C --> D["docker compose up<br/>engine + PostgreSQL"]
D --> E["curl a message<br/>→ reply"]
E -.->|"edit, re-package"| B
Five steps, and the last arrow is the loop you stay in: change something, re-package, and the running engine flips to it.
1. Scaffold an app
sutra create app my-first-app
cd my-first-app
That is a complete, deployable application, not a hello-world stub:
packages/my-first-app-main/ the deployment package (the unit you ship)
package.yaml identity: tenant, module, version
channels.yaml the HTTP channel the process listens on
bpmn/sample.bpmn the process
schemas/sample/sample.xsd the message contract
templates/*.hbs the reply bodies
deploy/compose.yaml engine + PostgreSQL
deploy/deployments/ what the engine watches
deploy/smoke.sh a health-gated end-to-end check
2. Package it
sutra package packages/my-first-app-main --out deploy/deployments
package seals the directory into a single .sutra archive: content-addressed, with a
manifest hash over every entry. It also lints on the way in — a template that navigates a
field the schema does not declare, or a channel with no process bound to it, fails here rather
than at 2 a.m. That archive is now the artifact you would promote through environments.
3. Start the engine
docker compose -f deploy/compose.yaml up -d
Compose starts the engine image alongside its own PostgreSQL and mounts deploy/deployments,
which the engine watches. The archive you just wrote is picked up and activated on boot.
The host port is deliberately dynamic — nothing assumes port 8080 is free on your machine:
ENGINE=$(docker compose -f deploy/compose.yaml port engine 8080)
curl -s http://$ENGINE/sutra/health/ready
{"status":"UP","checks":[{"name":"sutra-loader","status":"UP","data":{"deployments":1,"shards":1}}]}
deployments: 1 is your archive, live.
4. Send a message
curl -s -X POST http://$ENGINE/channels/sample-in \
-H 'Content-Type: application/xml' \
-H 'X-Api-Key: dev-only-sample-key' \
--data '<SampleRequest xmlns="urn:sutra:deployment:my-first-app-main"><note>hello</note></SampleRequest>'
<Accepted xmlns="urn:sutra:deployment:my-first-app-main" process="sample"/>
The X-Api-Key header is not optional: channels.yaml declares apikey auth on this
channel, because the engine refuses to wire an unauthenticated HTTP intake
(SUTRA.CHANNEL.AUTH.MISSING_SCHEME). The scaffold’s dev value is supplied by
deploy/compose.yaml; a real deployment resolves ${SAMPLE_API_KEY} from its secret store.
The namespace is not decoration either. The codec validates against
schemas/sample/sample.xsd, whose targetNamespace is this deployment’s
(urn:sutra:deployment:my-first-app-main), so an unqualified <SampleRequest> is a different
element as far as the validator is concerned and comes back rejected with no declaration
found. Payload namespaces are how every message standard this engine speaks — ISO 20022, MT,
NACHA — identifies its documents.
The reply carries it too, which is the half people miss. Accepted and Rejected are
declared in this deployment’s own schema alongside the inbound message, so an unqualified
answer would be a different element from the one the schema declares — the same trap, on the
way out — and a caller that validates what it receives could not check it at all. The contract
runs in both directions. See templates/sample-accepted.hbs, which says so in a comment.
That reply came out the other end of a real process: the channel decoded the XML, validated it
against sample.xsd, started an instance, ran a gateway on the validation outcome, rendered a
template, and replied on the inbound connection. The reply is literally
templates/sample-accepted.hbs; edit it to echo a field ({{payload.note}}) and the change is
verified against the schema at package time.
Now watch it reject something. Send a payload the schema forbids:
curl -s -X POST http://$ENGINE/channels/sample-in \
-H 'Content-Type: application/xml' \
-H 'X-Api-Key: dev-only-sample-key' \
--data '<SampleRequest xmlns="urn:sutra:deployment:my-first-app-main"><wrong>hello</wrong></SampleRequest>'
<Rejected xmlns="urn:sutra:deployment:my-first-app-main" process="sample" outcome="FATAL"
reason="element 'wrong' is not expected at this point of the content model"/>
You get the rejection branch, not a stack trace and not a 500. That is the point of the whole design: a schema violation is a routable outcome your diagram models, so the failure path is as reviewable as the happy path.
The bundled check runs both of those for you:
./deploy/smoke.sh
5. Look inside
# What is deployed, and what state is it in?
curl -s http://$ENGINE/sutra/deployments | jq
# What does the engine think this package binds?
sutra describe packages/my-first-app-main/bpmn/sample.bpmn
# Will a message actually reach a process? (no engine needed)
sutra simulate --channel sample-in --dry-run packages/my-first-app-main/bpmn/sample.bpmn
6. Change something
Edit templates/sample-accepted.hbs, then re-package into the watched directory:
sutra package packages/my-first-app-main --out deploy/deployments
The engine picks up the new archive and flips atomically: in-flight instances finish on the version they started on, new messages land on the new one. No restart, no dropped request. That is the same mechanism you would use in production — there is no separate “dev mode”.
7. Regenerate the documentation
The change you just made is also a documentation change, so refresh the catalog and commit both together:
sutra generate docs --input packages --output catalog
That writes one page per artifact — each BPMN page opening with a diagram in BPMN notation, laid
out from the process graph itself (your .bpmn carries no coordinates, and needs none) — plus
the SUMMARY.md and book.toml that make the whole catalog a buildable mdBook. mdbook build catalog turns it into a site or a PDF.
This is not optional housekeeping. The catalog is committed, and --check is a gate:
sutra generate docs --input packages --output catalog --check
sutra create ci --provider github (or bitbucket) scaffolds a pipeline that runs exactly that,
so a package whose docs have drifted from its BPMN fails the build. CI only ever checks —
regenerating is yours, in the same commit as the change that caused it.
Clean up
docker compose -f deploy/compose.yaml down -v
Next
- Your first deployment — deploying over the API instead of a watched directory, and what “Active” actually means.
- Concepts — why the message, not a REST call, is the contract.
- Channels — the other eight transports (Kafka, RabbitMQ, SQS, Pub/Sub, AMQP, …) your process can listen on without changing the process.
Anatomy of an app
The quickstart ran a scaffolded app end to end. This page opens it up: what each file does, how a message travels through them, and how to make your first real change — including watching the engine refuse a change that would have broken in production.
If you skipped the quickstart, scaffold one now:
sutra create app my-first-app && cd my-first-app
The package is the unit
packages/my-first-app-main/ ← the deployment package: what you ship
├── package.yaml identity (tenant, module, version) — its slot
├── channels.yaml which channels this package binds, and how
├── bpmn/sample.bpmn the process
├── schemas/sample/
│ ├── sample.xsd the message contract
│ └── codec-manifest.yaml which schema serves which message type
├── templates/*.hbs reply bodies
└── datastores.yaml business data stores this package declares
Everything outside packages/ is scaffolding around the app — deploy/compose.yaml,
deploy/deployments/ (the watched directory), deploy/k8s/, deploy/smoke.sh. You own those
and can delete them; the package is the part the engine consumes.
One package directory seals into one .sutra archive. Its identity is
tenant--module--version (its slot) plus a content hash of everything inside — which is why
a re-deploy of identical bytes is a no-op and a changed package replaces its slot atomically.
How the pieces connect
Follow one message:
flowchart LR
REQ["POST /channels/sample-in<br/><SampleRequest>…"] --> CH
subgraph pkg["packages/my-first-app-main"]
CH["channels.yaml<br/>sample-in → codec"] --> XSD["schemas/sample.xsd<br/>decode + validate"]
XSD -->|"valid, typed"| BPMN["bpmn/sample.bpmn<br/>start → gateway"]
XSD -.->|"invalid"| BPMN
BPMN -->|"accepted"| TA["templates/<br/>sample-accepted.hbs"]
BPMN -.->|"rejected"| TR["templates/<br/>sample-rejected.hbs"]
end
TA --> RES["HTTP reply"]
TR -.-> RES
channels.yamldeclaressample-inas an HTTP channel bound toPOST /channels/sample-in, with a codec. The channel is the doorway: it decides how bytes become a message.- The codec (
schemas/sample/) decodes the XML and validates it againstsample.xsdbefore any process runs. A violation here is data, not an exception. bpmn/sample.bpmnhas a start event bound to that channel, so a valid message starts an instance — with the payload already typed. The gateway branches on the validation outcome; the accept and reject paths render different templates.templates/*.hbsproduce the reply body, returned on the same HTTP connection because the process declares<q:reply>.
The important property: steps 2 and 3 are separate on purpose. The process never parses anything. Change the wire format later — XML to JSON, HTTP to Kafka — and the diagram is untouched, because format and transport are channel concerns.
Inspect any of this without running an engine:
sutra describe packages/my-first-app-main/bpmn/sample.bpmn # processes, events, tasks, gateways
sutra dispatch-graph packages/my-first-app-main/bpmn/sample.bpmn # which channel reaches which process
Make it yours
Add a field to the contract and use it. First, schemas/sample/sample.xsd — add an amount
alongside note:
<xs:element name="amount" type="xs:decimal" minOccurs="0"/>
Then use it in templates/sample-accepted.hbs:
<Accepted><note>{{payload.note}}</note><amount>{{payload.amount}}</amount></Accepted>
Re-package, and the engine’s deploy-time verification runs:
sutra package packages/my-first-app-main --out deploy/deployments
Now break it deliberately — change the template to {{payload.amont}} (typo) and re-package:
SUTRA.LINT.NAVIGATION.UNKNOWN_FIELD
templates/sample-accepted.hbs:1 — 'amont' is not a field of SampleRequest
did you mean: amount
Packaging fails. That typo never reaches an engine, never waits for the one request that happens to take that branch at 2 a.m. Every FEEL expression and template path in the package is checked against the schema it claims to read — this is the design’s central bet, and it costs you nothing to see it work.
Fix the typo, re-package, and the running engine picks up the new archive and flips to it with in-flight instances left to finish on the old one.
Where each concern lives
| Want to… | Go to |
|---|---|
| Understand the model behind all of this | Concepts |
| Bind a broker instead of HTTP | Channels and transports |
| Decode a different format, or your own | Deployment packages |
| Wait for a human decision or a reply | Wait states |
| Write decisions as tables or rules | Rules: DMN, FEEL, and .srl |
| Retry, schedule, or time-box work | Retries, history, schedules |
| Test a 30-day timer in milliseconds | Testing time |
Next
Your first deployment — getting the archive onto an engine you did not
start with docker compose, and what “Active” really means.
Your first deployment
The quickstart deployed by dropping an archive into a watched directory. That is one of three ways in, and the right one for local work. This page covers the other two and what they have in common — because “deployed” means something precise here.
What a deploy actually is
The deploy unit is one sealed .sutra archive. sutra package runs the fail-closed validation
suite and derives the manifest — including the content-addressed deploymentId = sha256(manifest)
— from the package directory; nothing in the manifest is hand-authored.
Two consequences worth internalizing:
- Deploying is idempotent. Re-deploying identical bytes is a no-op, because the bytes are the identity. There is no “did that apply?” ambiguity.
- A slot holds one active revision. The slot is the package’s stable
tenant--module--version. A changed archive under the same slot replaces the active revision in one transaction — a hot-deploy, not a restart — while instances already running stay pinned to the revision they started on.
Three ways in
| Path | Use it when | How |
|---|---|---|
| Watched directory | local dev, air-gapped hosts | write the .sutra into the directory the engine watches |
| Synchronous API | CI/CD, any engine you can reach | sutra deploy … --api --engine-url … |
| ConfigMap patch | Kubernetes | sutra deploy … (the default when a cluster context is set) |
flowchart LR
PKG[".sutra archive"] --> W["watched directory"]
PKG --> A["POST /admin/deployments"]
PKG --> K["ConfigMap patch"]
W --> V["re-verify fail-closed<br/>store as the slot's new revision"]
A --> V
K --> V
V --> F["two-phase activation flip"]
F --> ACT["Active"]
V -.->|"invalid archive"| REJ["SUTRA.DEPLOY.* refusal<br/>nothing stored"]
All three end at the same place: the archive is stored, validated, and activated by the same two-phase flip. They differ only in how the bytes arrive.
The synchronous API
sutra deploy my-first-app-main.sutra --api --engine-url http://localhost:<port>
This is the deterministic path — the call returns only once the deployment is live, so what you see on the wire is what happened:
- The CLI uploads the archive to
POST /admin/deployments. - The engine re-verifies it fail-closed — a corrupt or invalid archive is rejected here, before anything is stored.
- The archive becomes the new active revision for its slot, in the engine’s own datasource.
- The engine runs its two-phase activation flip: drain the old revision if there is one, activate the new one.
- The call returns synchronously —
200 {deploymentId, phase: "Active"}, or a4xxcarrying theSUTRA.DEPLOY.*diagnostic that explains the refusal.
There is no propagation window to wait out; the HTTP response is the “it’s live” signal. For a
deployment large enough that the flip risks a long-held request (mainly behind an ingress with a
short read timeout), the same endpoint takes an async mode: 202 {deploymentId, status: "Pending"}, then poll GET /sutra/deployments/{id} until it reads Active or Failed.
In CI, that exit code is your gate — a failed validation fails the pipeline, not the release.
What you get, and what you don’t
A successful deploy leaves you with:
- A live deployment.
GET /sutra/deployments/{id}reportsActive. - Live channels. Whatever
channels.yamldeclared is bound and serving — an HTTP channel accepts requests immediately; a broker channel’s consumer is running (for asingleton: truechannel, on whichever replica holds the per-channel lease). - Nothing else. Deploying creates no infrastructure — no database, no broker, no ingress.
Those are provisioned separately (locally by
docker compose, in a cluster by the OpenTofu modules underdeploy/). A deploy only registers and activates the package’s processes, channels, and store bindings against infrastructure that already exists.
Hot-deploy and rollback
Re-package under the same slot and deploy again. The archive gets a new content-addressed id, the slot name does not, so the engine flips in-process and lets in-flight instances drain on the old revision. Rollback is the identical operation with the previous archive — there is no separate rollback mode to learn, and no “undo” that behaves differently from a deploy.
For the operator-facing detail — draining, retirement, migrating an instance off a pinned revision — see Deploy, hot-deploy, and rollback.
Next
You have installed the tools, run a message through an engine, understood the package, and deployed it for real. Building BPMN solutions picks up from here with the ideas and file formats behind what you just shipped.
Concepts overview
Sutra is organized around a few load-bearing ideas. This page is the map; deeper chapters will expand each one.
flowchart TD
MSG["A message arrives<br/>on a channel"] --> CODEC["Codec<br/>format × schema"]
CODEC -->|"typed payload"| INST["Process instance"]
CODEC -.->|"soft error"| INST
INST --> WAIT{"needs to wait?"}
WAIT -->|"no"| DONE["Reply / emit / complete"]
WAIT -->|"yes"| PARK["Suspend + persist<br/>keyed by YOUR business key"]
PARK -->|"correlated message<br/>or timer fires"| INST
INST <-->|"q:store"| STORE[("Data store<br/>state beyond one instance")]
Everything below is one of these boxes examined closely.
The message is the contract
A start event binds to a channel and a message type. The engine decodes the real wire format and validates it against a schema before the process runs, so malformed input becomes a routable soft error, not an exception deep in the flow.
Codec = format × schema
A codec pairs a format (how bytes are structured) with a schema (what a valid message
looks like). Decoding and validation are one step. One schema can declare several message types,
and dispatch fans out one process per type. Six formats are built in — JSON, XML, YAML, CSV, raw
text, raw bytes; a schema-bound codec comes either from the package itself (XSDs under
schemas/<name>/, compiled at deploy time) or from an extension crate implementing the codec
SPI, which is how a message standard (an industry wire format with its own envelope grammar and
schema editions) is served.
Channels and transports
Processes are triggered by messages arriving on channels — HTTP, five brokers (Kafka,
RabbitMQ, AWS SQS, Google Pub/Sub, AMQP 1.0), Dapr pub/sub, Knative Eventing, and an air-gapped file
transport. <q:source>
binds a start event to a channel + message type. Co-deployed processes hand off over an
in-process local:// channel.
Wait-states and correlation
Wait states (userTask, intermediate message catch) suspend → persist → rehydrate → resume.
A later message is correlated back to the parked instance by a business key you name
(<q:alias> — e.g. an EndToEndId or an order reference), not an engine id, and it’s durable and
replica-coherent on PostgreSQL.
Durable state beyond one instance
Wait-state data belongs to one instance. State that outlives an instance, or is shared between
them, lives in a data store — declared in the package (datastores.yaml),
read and written from the flow through <q:store>, and transactional, with optimistic
concurrency (expect="unchanged") and pessimistic locking (forUpdate="true") where a
read-modify-write needs them. Each store owns its own connection: the engine’s database is
never the module’s. A store is key/value by default and holds a value of any shape; a flat
record may additionally declare its structure: and project onto real typed columns in a table
you own, so the SQL tooling you already point at that database reads it directly.
Rules — DMN, .srl, and FEEL
A businessRuleTask binds a .dmn decision table or a .srl ruleset (a Drools-inspired
rule / when / then DSL). Both compile onto one shared FEEL evaluator — no JVM, no Rete
runtime.
The q: vocabulary
Layered BPMN extensions keep diagrams standard while removing boilerplate: <q:dispatch> /
<q:case> (a routing table instead of gateway sprawl), <q:validators>, <q:alias>,
<q:reply>, <q:variables>, <q:audit>, and <q:coverage>.
Content-addressed deployment
The deploy unit is one sealed .sutra archive; runtime identity is a single opaque
deploymentId = sha256(manifest). Deploys are idempotent, hot-reload flips activation without a
restart, and secrets never live in the archive — channels reference them by scheme
(secret: / env: / vault: / aws-secrets: / …), resolved at runtime through one
vendor-neutral SPI.
Multi-tenancy and observability
Per-channel tenant binding, single-database PostgreSQL with per-deployment row-level security across PG / MySQL / MariaDB / MSSQL, OTLP traces / metrics / logs to any collector, plus KEDA autoscaling and leader election.
Deployment packages
A Sutra project is a single deployment package. sutra create app produces exactly one;
sutra package <dir> seals it into one immutable, content-addressed .sutra archive; sutra deploy activates that archive on a running engine. There is no build step in between — you
author declarative resources, seal them, and hand the sealed archive to a generic engine binary
that never needs your source. Secrets are the one thing that never travels inside it: a resource
names them by scheme (env:NAME, secret:KEY, vault:…), resolved at runtime, and package-time
validation rejects a literal outright (see Channels and transports).
flowchart LR
DIR["the package directory<br/>bpmn · schemas · templates<br/>channels.yaml · datastores.yaml"]
DIR -->|"sutra package<br/>validates first, fail-closed"| AR["one immutable .sutra<br/>manifest of per-file digests"]
AR -->|"deploymentId =<br/>sha256(manifest)"| DEP["sutra deploy"]
DEP --> SLOT["the slot:<br/>tenant--module--version"]
AR -.->|"carries env:NAME or<br/>secret:KEY, never a value"| SEC["resolved at runtime,<br/>outside the archive"]
Everything the engine needs to run the module is sealed and hashed, so identical bytes re-deploy as a no-op and the same archive promotes across environments unchanged — while the values that must differ per environment stay references it resolves on the way up.
Package interior
<package>/
├── package.yaml # manifest: labels {tenant, module, version}, engine.minContract
├── bpmn/**/*.bpmn # BPMN 2.0 processes
├── rules/ # DMN decision tables (.dmn) + the .srl rule DSL, by extension
├── scripts/**/*.{hbs,xsl,xslt} # derive/compute (Handlebars + XSLT)
├── templates/ # render output (Handlebars + XSLT)
├── schemas/ # each leaf folder under here = one codec
│ └── <codec>/*.xsd + codec-manifest.yaml
├── migrations/<store>/** # per-datastore SQL migrations
├── channels.yaml # transport channels (bind codec URNs; hot-reloaded on flip)
└── datastores.yaml # datastore declarations
Every artifact folder supports nested subfolders — bpmn/orders/checkout/…, templates/eu/invoices/…
— and discovery is recursive. sutra create deployment <name> --from packages/my-app
scaffolds a sibling package as an explicit copy; packages never inherit from one another.
package.yaml — labels, not scope
labels:
"module": "money-transfer"
"tenant": "default"
"version": "1.0.0"
engine:
minContract: 1
tenant / module / version are opaque labels — selectors for observability, routing, and
row-level-security partitioning (tenant_id) — never a resource-tree dimension. A package
archive is fully self-contained: there is no tenants/<id>/ overlay tree, no shared-library
modules/ folder, and no inheritance between packages at deploy time. What a tenant “sees” is
simply every process in the package deployed under its labels. The authoring tree may still be
layered — see Composing a package from layers.
properties: — deployment configuration
properties:
defaultWindow: "${DEFAULT_WINDOW:PT10S}" # an env reference with a default
callTimeout: "env:CALL_TIMEOUT" # a whole-value env reference
startAfter: "PT1H" # a literal
properties: is a flat map of keys to text. A timer expression reads it as config.<key>
(see Timer intervals from FEEL). The
archive seals the text as written, so the same archive runs in every environment. The engine
resolves env:NAME, ${NAME} and ${NAME:default} from its own environment when it activates
the deployment. A reference with no value and no default refuses the deployment
(SUTRA.DEPLOY.PROPERTY_UNRESOLVED). Keys are letters, digits and _. Values are strings, so
use number(config.x) for arithmetic. Secret references (secret:KEY, ${secret:KEY}, a secret
store) are refused, because every timer expression and the audit trail can read the
configuration. Editing a property changes the deploymentId, the same as an audit: change.
Changing the environment does not, and takes effect when the engine next activates the
deployment.
Composing a package from layers
When one process is deployed several ways — the same flow bound to HTTP for one tenant and to a broker for another, or a second edition of the same contract — keep the common content once and let each variant hold only what differs:
order-intake/
├── shared/ # common to every variant; no package.yaml here
│ ├── bpmn/order-intake.bpmn
│ ├── rules/approval.dmn
│ └── templates/receipt.hbs
└── variants/
├── acme--order-intake--1.0.0/
│ ├── package.yaml # this variant's labels
│ └── channels.yaml # HTTP binding
└── globex--order-intake--1.0.0/
├── package.yaml
└── channels.yaml # broker binding
Name the shared folder as an overlay, and the two commands see the same composed tree:
sutra lint variants/acme--order-intake--1.0.0 --overlay shared
sutra package variants/acme--order-intake--1.0.0 --overlay shared -o dist/
The rules, in the order they bite:
- Overlays are base layers.
--overlaymay be repeated; each is applied in the order given, and the package directory is applied last — so a variant’s file always wins over a shared one. - An overlay carries no
package.yaml. Labels belong to the variant, and apackage.yamlfound inside an overlay is refused rather than quietly merged. - Every overridden path is reported, so a variant shadowing a shared file does it visibly.
- Nothing is deleted. A variant can add to or replace shared content, never remove it. Content a
variant must not carry does not belong in
shared/— or use a deployment manifest, whose layers can drop files with.ignore. - The package directory’s name names the archive (
acme--order-intake--1.0.0.sutra), which is why variant directories are named after their labels. - Composition happens at package time. The sealed archive holds the merged file set and knows nothing about layers, and its deployment id is identical to sealing a hand-merged directory with the same content. Layering is an authoring convenience; it changes nothing about what deploys, or about how the engine loads it.
Deployment manifests — customers, editions, environments
--overlay covers one shared folder plus one variant. When a package is assembled from more
dimensions — shared content, a product edition, a customer, an environment — write the stack down
once in a deployment manifest and hand that file to sutra lint, sutra package or
sutra deploy in place of the package directory:
order-hub/
├── shared/schemas/ # used by several products; mounted at schemas/
├── app/
│ ├── shared/ # bpmn/ rules/ templates/ channels.yaml
│ └── editions/
│ └── 2026r1/ # what this edition changes (bpmn/intake.bpmn)
├── customers/customer-a/
│ ├── common/ # this customer's templates
│ └── envs/
│ ├── dev/ # channels.yaml + .ignore
│ └── prod/ # channels.yaml
└── deploy/
├── customer-a.2026r1.dev.yaml
└── customer-a.2026r1.prod.yaml
# deploy/customer-a.2026r1.dev.yaml
deployment: customer-a-dev # optional, informational
package: # becomes the package's package.yaml
tenant: customer-a # tenant / module / version are label shorthands
module: order-hub
version: 1.1.0
engine: { minContract: 1 }
layers: # lowest first; later wins; relative to this file
- path: ../shared/schemas
at: schemas # mount this folder under schemas/
- ../app/shared
- ../app/editions/2026r1
- ../customers/customer-a/common
- ../customers/customer-a/envs/dev
target: # optional: sutra deploy's flags
engine-url: http://localhost:8088
api: true
deploy/customer-a.2026r1.prod.yaml is the same file with envs/prod as its last layer and its
own target:. The dev layer overrides channels.yaml and drops a process dev does not run:
# customers/customer-a/envs/dev/.ignore
bpmn/reconcile.bpmn
sutra lint deploy/customer-a.2026r1.dev.yaml
sutra package deploy/customer-a.2026r1.prod.yaml -o dist/ # → dist/customer-a.2026r1.prod.sutra
sutra deploy deploy/customer-a.2026r1.dev.yaml
sutra lint prints the resolved stack before its findings, so what will be sealed is never a
guess:
manifest deploy/customer-a.2026r1.dev.yaml (deployment customer-a-dev)
layer 1 ../shared/schemas (at schemas)
layer 2 ../app/shared
layer 3 ../app/editions/2026r1
layer 4 ../customers/customer-a/common
layer 5 ../customers/customer-a/envs/dev
override bpmn/intake.bpmn: layer 3 ../app/editions/2026r1 over layer 2 ../app/shared
override channels.yaml: layer 5 ../customers/customer-a/envs/dev over layer 2 ../app/shared
removed bpmn/reconcile.bpmn: by layer 5 ../customers/customer-a/envs/dev .ignore 'bpmn/reconcile.bpmn' (was from layer 2 ../app/shared)
The rules:
- It is only a shorthand. The CLI turns the manifest into a generated
package.yamlplus the layer stack, then validates and seals exactly as for a directory. Sealing a manifest produces the same archive bytes and the same deployment id as sealing a hand-merged directory holding the same files and the samepackage.yaml. The archive is named after the manifest file (customer-a.2026r1.dev.sutra). - A closed schema. The top-level keys are
deployment,package,layersandtarget; any other key is refused.package:takes thepackage.yamlkeys (labels,engine,entryProcesses,audit,properties) plus thetenant,moduleandversionshorthands;moduleandversionare required. - Layers apply in order, a later layer’s copy of a file replacing an earlier one’s; every
override is reported. A layer may not carry
package.yaml— the refusal names the layer. A missing layer folder, or anat:that is absolute or climbs out with.., is refused. --overlaystill works and stacks above the manifest’s last layer.target:takessutra deploy’s flag names (context,namespace,configmap,secret-name,engine-url,api,async,wait,wait-timeout,api-key-header). A flag given on the command line wins. Secrets never go in a manifest — pass them with--secretor--secret-from.
.ignore — removing what a lower layer brought
A layer may carry a .ignore file at its root. It uses gitignore syntax — # comments, *, ?
and […] within a path segment, ** across segments, a trailing / for a directory, a leading
or inner / to anchor, ! to re-include — and:
- patterns are relative to the package root, not to the layer folder (nor to its
at:); - when the layer is applied, its patterns first remove matching files contributed by the layers below it, then the layer’s own files are added — a layer never ignores its own files;
- the last matching pattern decides, a pattern naming a directory covers everything under it, and a
!pattern can bring back a single file under an ignored directory; package.yamlcan never be ignored;- a pattern that removes nothing is reported as a warning (
SUTRA.DEPLOY.PACKAGE.IGNORE_UNMATCHED) — it is usually stale or mistyped; .ignorefiles are never sealed into the archive.
Referencing resources — convention over configuration
Because the package is self-contained, resources are referenced by their local key, derived
from the folder tree — never by tenant/module/version:
| Resource | Resolved by |
|---|---|
| BPMN process | processId — globally unique within the package |
Rule (.dmn / .srl) | its relative path under rules/ |
| Script / template | its full relative path (folder + filename + extension) |
| Codec | a URN — see below |
Codecs are the one exception. A globally-named codec — a format the engine links in (json,
xml, yaml, csv, raw-text, raw-bytes) or a codec crate a distribution force-links — is
urn:sutra:codec:<name>; a package-defined codec is named by its path under schemas/, /
folded to : — schemas/transfer/ becomes urn:transfer, schemas/hr/employee/ becomes
urn:hr:employee. channels.yaml binds that URN:
# examples/money-transfer/.../channels.yaml
channels:
- name: transfer-request
transport: http
bind: "POST /channels/transfer-request"
codec: urn:transfer # schemas/transfer/*.xsd, declared by schemas/transfer/codec-manifest.yaml
A codec-manifest.yaml sits inside its own schemas/<codec>/ folder and declares the schema kind
plus the formats the codec accepts:
# examples/money-transfer/.../schemas/transfer/codec-manifest.yaml
schemaKind: xsd
formats: [xml, json, yaml]
Tabular formats — csv and fixed-width
formats: is not limited to document syntaxes. A schema codec can bind csv or fixed-width,
which is what turns a delimited or fixed-column upload from a bag of untyped strings into a typed
message:
# examples/call-log-load/.../schemas/cdr/codec-manifest.yaml
schemaKind: xsd
formats: [csv, fixed-width]
csv:
delimiter: "," # both default; written here only to show where layout lives
header: true
fixed-width:
fields: # REQUIRED — a fixed-width line has no structure of its own
- {name: recordId, width: 12}
- {name: msisdn, width: 16}
- {name: durationSec, width: 6}
A tabular body is a batch, so it is validated row-wise: each row is checked as one instance
of the declared root, in a single decode, before the process starts. Every violation carries a
row-indexed path (value[3].durationSec), so a bad cell names its record. An unparseable file is
fatal; any row’s violation is a soft error, so the payload still projects and q:onValidation
decides the posture. The batch projects under value, so a flow reads payload.value[0].field.
The XSD’s leaf types reach the cells: an xs:int column arrives as a number, not "182". An empty
cell reads as absence for an element the type declares minOccurs="0" — without that, an
optional column would have to be populated in every row. An empty cell for a required element is
still a violation.
Both tabular formats may be declared together. Their content types are disjoint — text/csv /
application/csv against text/plain / application/x-fixed-width — so an upload selects its
parser unambiguously and one schema serves a CSV feed and a fixed-width feed over the same channel.
Layout is parser config, not schema: it cannot be expressed in an XSD or a JSON Schema (neither has byte offsets), which is why it lives in the manifest. A header-bearing CSV names its own columns and needs no block at all; a fixed-width codec cannot work without one and declaring the format without it is a manifest error.
Because a fixed-width layout is configuration, its columns are checked against the bound type at
package time: a column the type does not declare, or a required element with no column, fails
sutra lint (SUTRA.CONFIG.CODEC_MANIFEST.REJECTED) rather than every row at runtime. A csv codec
gets no equivalent check — its column names arrive in the header, at runtime, so there is nothing
to compare against beforehand.
The opaque formats (raw-text, raw-bytes) are not schema-bindable: there is no map under raw
bytes for a schema to type, so binding one is refused with that reason.
The reserved token sutra may not be used as a first-level subfolder name under any artifact
folder (schemas/sutra/, bpmn/sutra/, …) — that would collide with the engine’s own
urn:sutra:* namespace. A deeper sutra (schemas/hr/sutra/) is fine.
Schema bundles — when a codec is a whole profile
schemaKind: xsd and schemaKind: json-schema are the generic case: a folder of schema files the
engine validates a decoded document against. Some standards aren’t that shape at all — they’re a
whole profile: an envelope grammar, a mapping from a wire-level message name to a schema file,
and versioned editions the profile revs on its own release cadence, independent of Sutra releases.
For those, schemaKind names a bundle kind a codec crate has registered (see Domain
neutrality and the SPI model),
and the folder’s job shifts from “here are the schemas” to “here is the configuration that decides
which schema backs which message, for this archive version.”
A codec crate registers its own bundle kind this way whenever the standard it serves is a whole profile rather than a bare schema folder — say, a market venue that publishes a wrapper envelope around every message and revs its schema editions on its own release calendar. That codec crate is a proprietary extension, not part of this distribution — the mechanism below is generic; naming a concrete kind is only for illustration. Its manifest maps wrapper element names — not a bare schema namespace — onto schema files archived alongside it:
# schemas/<your-kind>/codec-manifest.yaml
schemaKind: <your-kind>
# schemas/<your-kind>/<your-kind>-manifest.yaml
appHdr: edition-2024/Header_v1.xsd # optional
incoming:
OrderConfirmation: edition-2026/OrderConfirmation_v3.xsd
outgoing:
OrderConfirmation: edition-2026/OrderConfirmation_v3.xsd
ShipmentNotice: edition-2024/ShipmentNotice_v2.xsd
schemas/<your-kind>/codec-manifest.yaml # schemaKind: <your-kind>
schemas/<your-kind>/<your-kind>-manifest.yaml # appHdr? / incoming{wrapper: relpath} / outgoing{wrapper: relpath}
schemas/<your-kind>/<edition-folder>/*.xsd # free-form folder names — the manifest is the only truth
A few things fall out of that shape:
- Edition folders sit side by side. Because the venue revs its schemas per release (the same wrapper can move from one schema version to the next between editions), archiving each release’s files under its own folder and repointing the manifest is how a module adopts a new edition — a new version of the archive, no engine or codec change.
- An unlisted wrapper falls back to the codec crate’s own base schema for its pinned default version, so a bundle only needs to carry the wrappers it actually wants to validate more strictly than that default — the codec stays useful with zero configuration otherwise.
- An enriched edition can be licensed material. A codec’s base schemas may be freely redistributable while a fuller, usage-guideline edition is a licensed product participants obtain themselves and supply in their own archive — the engine never ships that tier, and neither does any artifact built from this repository.
- Registration is deployment-scoped, exactly like every other artifact under the
registration model: the bundle’s registry key is
urn:sutra:codec:<folder-path-with-'/'-folded-to-':'>:<deploymentId>. Naming the folder<your-kind>shadows the globally-registeredurn:sutra:codec:<your-kind>for that deployment only — a second version of the same module with a different edition mapping registers under its owndeploymentIdand runs side by side, no collision. - Deploy-time errors are fail-closed: an unknown wrapper name, a wrapper listed under the wrong direction, a missing or uncompilable schema file, or a schema whose namespace doesn’t match the expected one all reject the archive rather than deploy something that silently validates less than the manifest claims.
flowchart LR
W["a wrapper element name on the wire<br/>OrderConfirmation"] --> M["the bundle manifest<br/>incoming / outgoing maps"]
CM["codec-manifest.yaml<br/>schemaKind names the bundle kind<br/>a codec crate registered"] --> M
M -->|"listed"| E["that edition folder's schema<br/>edition-2026/OrderConfirmation_v3.xsd"]
M -.->|"unlisted"| B["the codec crate's own base schema,<br/>at its pinned default version"]
NEW["a new edition of the profile"] -.->|"archive its folder alongside,<br/>repoint the manifest,<br/>ship a new archive version"| M
The manifest is the only truth about which schema backs which message — the folder names carry no
meaning — so adopting a new edition is a package change and a new deploymentId, never an engine
or codec-crate change.
Store migrations, and evolving a projected structure
Every sql data store the package declares brings its own schema, under a folder named for the
store:
migrations/<store-id>/V001__<description>.sql
migrations/<store-id>/V002__<description>.sql
<store-id> is the store’s name in datastores.yaml, and the migrations: key of that store
points at the folder (migrations: migrations/accounts). The scripts are yours: your dialect, your
table names, your indexes and seed rows. The engine generates no DDL for a module store.
One store is the exception: the reserved coverage store, where path-coverage marks are
persisted. Its declaration still picks the database, but the engine owns its schema — it ships that
DDL per dialect and applies it to the connection on the same first-use path — so the store block
carries no migrations: key and a package carries no migrations/coverage/ folder. See
Coverage: declared routes as the compliance signal.
Three properties of how the package’s own scripts run are worth designing around:
- Applied in
V<n>__order, before the store serves its first operation — and serialized across replicas, so two engines booting at once don’t race. - Each script runs once per database. Applied scripts are recorded in a history table,
sutra_store_migrations, in the store’s own database; a recorded script is skipped on every later boot, and editing one after it was applied refuses the store (SUTRA.RUNTIME.DATASTORE.MIGRATION_CHECKSUM). A change is always a new, higher-numbered script. See Migrations run once for the full rules, including the five${…}placeholders a script can use. - A store’s data carries across a version bump. There is no
deployment_idon a business store’s rows: deploying1.0.1beside1.0.0doesn’t fork the data, and that is the feature, not an oversight.
When a projected structure changes
A store that declares a structure: block
adds one obligation: a changed structure ships with the migration that makes the table match, in
the same package. sutra lint derives the effective table shape from these very scripts, so it
tells you at package time whether the pair is consistent — its job is to detect the mismatch, not
to repair it.
| Change to the declared type | What it costs |
|---|---|
| Add an optional scalar field | Additive. A new nullable column; rows written before it read the field as absent |
| Add a required scalar field | Lint error until the column exists and is nullable or has a DEFAULT — existing rows cannot satisfy a bare NOT NULL |
| Remove a field | The column becomes unmapped (a warning); existing data is untouched |
Widen a facet (maxLength 35 → 70) | Lint error until the ALTER ships in a new V… script; clean once it does |
| Rename a field | Modelled as remove + add. A columns: mapping can keep the physical column name instead |
| Scalar → nested or repeated | Hard stop: STRUCTURE_NOT_FLAT. Either keep the field flat, or drop the structure block and go back to the opaque store — an explicit decision at package time, never a silent shape change |
What follows from the table is a packaging rule: a type change and the ALTER that supports it
belong in the same package. Lint replays every script in the folder in version order and compares
the result against the declared type as it stands, so a package carrying both lands clean, while
one carrying only the type change fails the gate rather than the deployment.
Building one by hand vs. scaffolding
sutra create app <name> (see Anatomy of an app) generates a
package in this exact shape, verified through the engine’s own loaders before anything is
written. Growing it from there:
sutra create bpmn my-process --package packages/my-app --validation fatal
sutra create deployment my-app-eu --from packages/my-app # explicit variant copy
sutra lint <package-dir> runs the full package-time validation suite (the same checks sutra package runs before sealing) with no output on success — the fast pre-flight to run before every
package.
Next
- Channels and transports — how
channels.yamlbinds a transport, a codec, and an ack mode. - The q: namespace — the BPMN extension vocabulary that wires a process to its channel.
Channels and transports
Processes don’t get called — they’re triggered by messages arriving on channels.
channels.yaml (one per package, see Deployment packages) declares each
channel: which transport it rides, which codec decodes it, how it acknowledges receipt, and who
may send to it.
Transports
One neutral transport SPI, ten implementations, all self-registering behind the same lifecycle
trait — the engine binds, activates, and drains every one of them through a single generic path
with no if transport == "..." branching anywhere:
transport: | Notes |
|---|---|
http | The universal baseline. Also serves /sutra/health/*. |
kafka | rdkafka. |
rabbitmq | lapin (AMQP 0.9.1). |
aws-sqs | AWS SDK. |
gcp-pubsub | Google Cloud client. |
amqp | fe2o3-amqp (AMQP 1.0). |
dapr | Dapr pub/sub: the sidecar pushes to the engine’s own HTTP listener. No vendor client — see Dapr and Knative. |
knative | Knative Eventing: a Trigger or Subscription pushes to the engine’s own HTTP listener. No vendor client. |
llm | A language model as an outbound request/reply channel, with ordered fallback candidates — see Calling a language model. Provider adapters are cargo features llm-openai, llm-anthropic, llm-bedrock, all in the default build. |
file | Air-gapped: file-spool inbound + file:// outbound sink, no network dependency. |
Two further transport: values are engine-internal rather than vendor clients — they have no wire
protocol and no listener of their own. local delivers in-process to another channel; pull parks
the delivery as a task a worker fetches instead of dialing anything, which is the
external-task surface.
Phone calls, IVRs and meetings are not part of this engine’s build. The voip transport, the
WebRTC media worker and the voice, IVR, meeting and video examples live in the separate
open-source repository sutra-voice. It builds on
this engine as a git submodule and ships its own engine image with the voip transport linked in.
The model side they rely on stays here: llm channels with
audio parts,
live audio sessions, q:audit capture="media" and
the object store (Recordings and media references).
flowchart LR
H["http"] --> SPI
B["kafka · rabbitmq · aws-sqs<br/>gcp-pubsub · amqp"] --> SPI
PU["dapr · knative<br/>pushed to the HTTP listener"] --> SPI
F["file<br/>air-gapped spool"] --> SPI
LP["local · pull<br/>engine-internal, no wire protocol"] -.-> SPI
SPI["One transport SPI<br/>bind · activate · drain, generically"] --> CD["The channel's codec<br/>decode + schema-validate"]
CD --> SUB["The processes that subscribed<br/>q:source channel + messageType"]
Which transport a channel rides changes only how the bytes arrive: bind, activate and drain are one generic path, and everything past the doorway — decode, validation, subscription — is identical for all of them.
A hardened or air-gapped build selects a subset of transports at compile time via Cargo features on
the distribution crate (cargo build -p sutra-dist --no-default-features --features file), so the
unlinked vendor clients (rdkafka, the AWS/GCP SDKs, lapin, fe2o3-amqp) are not compiled in at
all. dapr and knative are features too, though neither links a vendor client — leaving them out
only removes their routes and outbound sinks. An operator can additionally restrict which
transports a running binary accepts via SUTRA_ALLOWED_TRANSPORTS — a channel declaring a
disallowed transport fails the deployment with a clear diagnostic, not a silent no-op.
Dapr and Knative
Dapr pub/sub and Knative Eventing each have a transport crate of their own (sutra-transport-dapr,
sutra-transport-knative), but neither links a vendor client or opens a connection. Both are
push transports: the Dapr sidecar, or a Knative Trigger or Subscription, delivers each event as
a CloudEvents HTTP POST to a route on the engine’s own HTTP listener. There is no long-lived
consumer to leader-elect — the pusher’s at-least-once delivery is the guarantee, and inbox dedup
absorbs its redeliveries. CloudEvents extraction follows the channel’s cloudevents-mode, as on
http. Outbound, each rewrites its destination to an HTTP URL and sends it through the same HTTP
sink the http transport uses.
dapr | knative | |
|---|---|---|
| Inbound route | POST /dapr/{topic} | POST /knative/{subscription} |
| Channel property that binds it | topic | subscription |
| Pointing the pusher at the engine | A declarative Dapr subscription whose route is /dapr/<topic> — the engine does not answer Dapr’s programmatic GET /dapr/subscribe | A Trigger or Subscription whose subscriber is /knative/<subscription> on the engine’s HTTP listener |
| Outbound destination | dapr://<pubsub>/<topic>, published through the local sidecar’s /v1.0/publish/<pubsub>/<topic> | knative://<namespace>/<broker>, posted to the Broker ingress |
| Outbound config (process-wide) | SUTRA_SINK_DAPR_SIDECAR_PORT (default 3500) | SUTRA_SINK_KNATIVE_BROKER_INGRESS, else K_SINK, else a built-in in-cluster default |
ack-mode: on-complete | Not supported, by design — the channel boots with SUTRA.ACK.ON_COMPLETE_UNSUPPORTED and runs on-persist | Supported — the push response is held until the instance ends, bounded by on-complete.hold-timeout (default 30 s) |
| Inbound guard | A dapr-topic header that disagrees with the path’s topic is rejected | A request carrying some but not all of ce-id, ce-source, ce-type is rejected (400) |
The sidecar port and the Broker ingress are process-wide because an engine process has one sidecar
and one ingress: a channel’s sidecar.port or broker.url is validated but has no effect.
channels:
- name: orders-in # a Dapr subscription routes topic orders.created here
transport: dapr
codec: urn:sutra:codec:json
topic: orders.created
- name: shipments-in # a Knative Trigger delivers to /knative/shipments
transport: knative
codec: urn:sutra:codec:json
subscription: shipments
ack-mode: on-complete
on-complete.hold-timeout: PT20S # ISO-8601 or bare seconds; keep it below the Trigger's timeout
Why the two differ on on-complete. On both, the HTTP response to the push is the
acknowledgement; there is no separate ack to defer. Knative bounds that response with a timeout the
operator sets on the Trigger or Subscription (delivery.timeout), so the engine can hold it until
the instance completes (202), fails (422, which Knative sends to the deadLetterSink rather than
retrying) or outlives on-complete.hold-timeout (202 with a warning — that one delivery degrades
to on-persist). Keep the hold timeout below the sender’s timeout, or every held delivery turns
into a redelivery. Dapr’s equivalent bound belongs to the pub/sub component — Redis Streams’
processingTimeout, Service Bus’s handlerTimeoutInSec — which the engine cannot see, and holding a
response past it makes the broker deliver the same message again, concurrently. So Dapr runs
on-persist, which answers once the dispatch has run to its first wait state or to completion. The
full per-transport matrix is in Acknowledgement modes.
Binding a channel
# examples/money-transfer/.../channels.yaml (abridged)
channels:
- name: transfer-request
transport: http
bind: "POST /channels/transfer-request"
codec: urn:transfer
cloudevents-mode: none
ack-mode: on-complete
auth:
scheme: apikey
apikey:
value: transfer-demo-key
header: X-Api-Key
A channel does not name a process. Processes subscribe to channels: a start event’s
<q:source channel="transfer-request" messageTypeValue="TransferRequest"/> is what routes a
decoded inbound to it (see The q: namespace). This is what lets one channel feed
several processes and one process listen on several channels — the money-transfer example runs
the same transfer flow off three different intake channels (http, rabbitmq, kafka), each a
separate <q:source> on the same underlying <bpmn:transaction>.
Fan-out. By default a channel is point-to-point: for a given (channel, messageType), exactly
one subscribing process is allowed (enforced at deploy time). Setting broadcast: true fans a
decoded message out to every subscribing process, one instance each — genuine pub/sub.
flowchart LR
H["intake channel<br/>http"] --> P["one process<br/>one q:source per channel"]
R["intake channel<br/>rabbitmq"] --> P
K["intake channel<br/>kafka"] --> P
N["one channel<br/>+ messageType"] --> S1["exactly one subscribing process<br/>enforced at deploy time"]
N -.->|"broadcast: true"| S2["every subscribing process<br/>one instance each"]
The channel never names a process; the process names the channel. That is what lets one flow be fed
by three transports at once, and why a second subscriber to the same (channel, messageType) is a
deploy-time error unless the channel opts into broadcast.
Concurrency admission. A channel may optionally declare maxConcurrentInstances — an
admission cap on simultaneously active instances from that channel (a suspended instance still
holds its slot). Absent, the channel is unbounded.
Uncorrelated answers. uncorrelated: { start: <processId> } on an inbound channel that a wait
subscribes to sends a message that correlates to no live instance to that process, instead of
refusing it. The process’s start events on the channel are reserved for such messages. See
An answer no instance waits for. A malformed
block fails to load, and sutra lint reports a start naming no process of the package, or one
with no start event on the channel, both as SUTRA.CONFIG.CHANNEL.UNCORRELATED_INVALID.
Each exclusive endpoint (route, queue, subscription, consumer group, spool directory) belongs to one deployment; see One endpoint, one deployment.
Codecs — format × schema
A codec is a channel-facing named decoder: a parser (json / xml / yaml / csv / a
domain wire format) plus an optional schema. The schema is the load-bearing part — it’s what
yields a messageType and structural validation. A codec with no schema (a media codec, or a
channel with no codec at all) still decodes, but only a catch-all <q:source> (no
messageTypeValue/messageTypePattern) can subscribe to it, and sutra lint warns.
Codec names share one urn:sutra:codec:<name> namespace, and codec: in channels.yaml looks
identical whichever tier the decoder came from. There are three:
| Tier | Example binding | Where the decoder comes from | What it takes to have one |
|---|---|---|---|
| Built-in format | codec: urn:sutra:codec:json | Linked into every engine binary | Nothing — it is always there |
| Package-supplied codec | codec: urn:transfer | schemas/transfer/*.xsd inside the archive, compiled when the package deploys | Author the XSDs — no Rust, no engine build |
| Extension-crate codec | codec: urn:sutra:codec:<name> | A crate implementing PayloadCodec, force-linked by a composition root | One Cargo dependency and one line in that composition root |
A built-in format is a pure parser — it decodes, but carries no schema, so it lands in the
catch-all-subscription case above. This distribution bundles six of them — json, xml, yaml,
csv, raw-text, raw-bytes — and no domain codec at all. Bind one when the payload genuinely
has no contract to check, or when that contract is enforced somewhere else.
A package-supplied codec is the zero-install path to a typed contract, and the one both
example apps take. schemas/transfer/ holds the XSDs plus a codec-manifest.yaml declaring
schemaKind: xsd and the formats the codec accepts; the engine compiles them at deploy time and
names the codec after the folder path (urn:transfer — see
Deployment packages for the exact folding rule). Nothing about it is
second-class: it yields message types, structural validation, and the shape every FEEL path is
checked against at load time, exactly as a codec written in Rust does. The engine cannot tell that
the schema arrived in an archive rather than a crate.
An extension-crate codec is what a wire format needs when a folder of schemas cannot express
it — a grammar that isn’t XML or JSON at all (a fixed-width block structure, or a delimited segment
stream), or a whole profile: an envelope grammar, a mapping from wire-level message names to
schemas, and versioned editions revved on the standard’s own release cadence. It implements
PayloadCodec from sutra-codec-spi, inventory::submit!s a BuiltinCodec next to the impl, and
claims its name in the same namespace; a distribution that wants it adds the dependency and
force-links it from its own composition root. Every message standard is served this way, by
proprietary extension crates built outside this repository. An engine binary that links one
resolves it exactly like a built-in; see
Domain neutrality and the SPI model for why none of them
lives here.
What an extension codec can express — an enveloped profile, generically
The clearest illustration of what the codec SPI has to be able to carry is a market venue’s own profile of a standard rather than the bare standard itself. A generic-format codec is the easy case: one instance document validated against a schema, with the message type read straight off the document’s own root element or namespace. A profile codec decodes a family of messages for a venue that doesn’t fit that shape — every message travels inside a venue-specific envelope, wrapping a header plus a body, and the venue pins its own schema versions on its own release cadence. Worse for identification purposes, the same underlying message can sometimes back several different wrappers, so a bare schema namespace can’t tell you which one you’re looking at.
Such a codec’s message type is therefore the wrapper element’s local name, not a schema
namespace slug — which lets declared_message_types() return the venue’s own closed set of
wrapper names, rather than the open type set a bare-standard codec declares, so a message-type
applicability check (a rules-manifest.yaml entry, a q:source messageTypeValue pin) actually
fires for a profile-bound module rather than silently no-op.
Decoding validates the envelope grammar, the header, and the body against the venue-pinned base
schemas the codec crate carries, out of the box. The payload view projects the body exactly as a
bare-standard document would be — a compatibility guarantee: every alias, DMN input, or fixture
written against the underlying message shape reads identically whether the underlying codec is the
profile variant or the plain standard. Outbound can be a template-rendered passthrough — encode()
returning the rendered envelope bytes verbatim rather than assembling one — so template drift
against the venue’s schemas has to be caught by validating rendered output in tests, not by a
runtime encode path. A second venue following the identical envelope/wrapper/edition pattern
reuses the same machinery rather than re-implementing it.
None of that requires a change to the engine, the channel layer, or this repository — which is the
point of the codec SPI. A deployment can go one step further still and override a codec’s own
schemas: see
Deployment packages for
the schemaKind bundle mechanism that lets an archive map its own schema editions per wrapper.
Acknowledgement modes
ack-mode decides when the engine acknowledges an inbound relative to processing:
on-persist— ack as soon as the message is durably captured, before the process runs. Broker default. On HTTP this is what makes a channel asynchronous (202 Acceptedwith an acceptance receipt).on-complete— ack only once the instance reaches a terminal state. HTTP default (classic synchronous request/reply — hold the connection, return the reply body). On a broker, this defers the ack via the engine’sDeferredAckRegistryuntil the instance completes or fails.
The full per-transport wiring matrix, the bounded-registry knobs, and when to pick which mode live in Acknowledgement modes — that’s the operating-chapter deep dive this summary points at.
AMQP 1.0: deliveries in flight and link credit
An AMQP 1.0 inbound channel processes several deliveries of one binding at once (since 0.8.3):
channels:
- name: orders-in
transport: amqp
host: broker.example
queue: orders
address-type: queue
codec: json
ack-mode: on-persist
max-in-flight: 16 # deliveries processed at once (1..1024); 1 = one at a time
prefetch-count: 10 # link credit floor
max-in-flight(default16) is how many intake calls run at once. Each delivery is still accepted only after its own message is persisted (or, withon-complete, its instance ends), and a duplicate is still absorbed by the inbox. On stop the engine waits for the calls in flight and settles each; on a lost connection the broker redelivers what was not settled.prefetch-countsets the link credit: the receiver grantsmax(prefetch-count, 2 × max-in-flight)credits, 32 by default. Before 0.8.3 the property was read but not applied.- Order. The engine never promised inbound order for a broker source. With
max-in-flightabove 1, two messages for the same instance can reach it in either order.max-in-flight: 1processes deliveries one at a time, in arrival order, as before 0.8.3.
Secrets on a channel
A channel’s credentials (broker username/password, an API key) are never literal values in
channels.yaml — they’re scheme references (env:NAME, secret:…, vault:…,
aws-secrets:…), resolved at channel startup through one vendor-neutral resolver SPI. Package-time
validation rejects a literal secret outright.
Next
- The q: namespace — how a BPMN process subscribes to a channel.
- Calling a language model — the
llmtransport, sync and async. - External tasks: the pull worker surface — the
pulltransport in full. - Acknowledgement modes — the operator-facing deep dive.
Calling a language model
A process calls a language model the same way it calls any other counterpart: with a
channel-call task. The model sits behind an outbound channel with transport: llm. Whether
the call is synchronous or asynchronous is the channel’s ack-mode (see
Sync or async). No BPMN element is specific to models, so everything a channel
call already has applies unchanged — <q:timeout>, <q:retry>, timer boundaries, error
boundaries.
Sync or async
Outbound ack-mode | Pattern | What happens |
|---|---|---|
on-complete | sync | The model is called inline and the answer is merged into the instance’s variables; the flow continues. No outbox row, no park, no database write for the hop, no reply-channel, no <q:alias>. |
on-persist (the outbound default) | async | The request goes through the outbox, the instance parks, and the answer arrives as a message on reply-channel, resuming the instance by <q:alias> correlation. |
Only an ack-mode you declare on the outbound channel counts; leave it out and the channel is
async, exactly as before.
Sync
- name: summarize-sync
direction: outbound
transport: llm
bind: "llm://summarize-sync"
ack-mode: on-complete # no reply-channel
llm:
candidates:
- { provider: anthropic, model: <model-id>, api-key: "secret:llm/anthropic" }
timeout-seconds: 30
<serviceTask id="Summarize" implementation="channel:summarize-sync">
<extensionElements>
<q:timeout duration="PT45S"/>
</extensionElements>
<dataOutputAssociation>
<assignment><from>payload.text</from><to>summary</to></assignment>
</dataOutputAssociation>
</serviceTask>
The task sends its requestBody variable (see The request). The answer is the
reply message below, and it reaches variables only through the task’s output
mapping: payload.text is the answer text, payload.finish, payload.usage.outputTokens,
payload.model and so on are the rest; header.x-llm-provider and header.x-llm-model carry
the provider and model that answered. Anything you do not map is dropped.
Parallel branches’ sync calls run concurrently; each is bounded by its <q:timeout>.
Async
Add reply-channel and an inbound channel for the answer, and give the task a <q:alias> whose
value is unique per call in flight (the next section shows the pair). A channel-call to an async
channel with no <q:alias> is rejected at deploy time with
SUTRA.DISPATCH.CHANNEL_CALL.ALIAS_REQUIRED.
Replying to the caller
When the process was started by (or is being resumed through) an inbound HTTP channel with
ack-mode: on-complete, the caller is waiting and the flow’s <q:reply> answers it:
- A relayed turn (a message that resumes a parked instance) answers with the
<q:reply>its own pass reaches, whether that reply iscontinue="true"before the instance parks again or sits on the way to the end. Replies the instance passed in earlier turns are not repeated. A turn that reaches no reply answers200with an empty body. - A plain
<q:reply>followed by a park answers nothing — the request is answered when the flow ends, or by acontinue="true"reply. - An inbound HTTP channel on
on-persistanswers202, with the reply as the receipt body.
The two channels (async)
channels:
# The request side: an outbound channel the process calls.
- name: summarize
direction: outbound
transport: llm
bind: "llm://summarize"
llm:
reply-channel: summarize-reply # where the answer is delivered
candidates: # tried in order
- provider: anthropic
model: <model-id>
api-key: secret:llm/anthropic
- provider: openai
model: <model-id>
base-url: https://models.example.test/v1 # any compatible server
api-key: env:LLM_FALLBACK_KEY
system: "You summarize support tickets in three sentences."
max-output-tokens: 512
temperature: 0
timeout-seconds: 60
concurrency: 4
fallback-on: [rate-limited, unavailable, timeout]
require: [json-output]
# ack-mode: on-persist is the outbound default
# The reply side: an inbound channel the answer arrives on.
- name: summarize-reply
transport: local
codec: json
| Key | Default | Meaning |
|---|---|---|
candidates | required | Ordered provider + model pairs, each with optional base-url and api-key. api-key is a reference (env:, secret:, vault:, ${…}), never a literal. A candidate may also name a transcription-model, used when a live audio session is opened on it. |
reply-channel | required unless ack-mode: on-complete | The inbound channel, in the same package, the answer is delivered on. A sync channel has none. |
mode | call | call holds the request open; job submits and completes later (see Jobs and webhooks). Refused on ack-mode: on-complete. |
system | none | System prompt. A request may override it. |
max-output-tokens, temperature | provider default | Passed through. A request may override them. |
timeout-seconds | 60 | Deadline for one candidate’s call (for a job: the submit call). |
concurrency | 4 | Calls in flight on this channel, per engine replica. |
fallback-on | [rate-limited, unavailable, timeout] | Which transient failures move on to the next candidate. A rejected request or a filtered answer never does. |
require | none | tools, json-output, audio-input, audio-output, audio-understanding, video-understanding or realtime: skip candidates that cannot do it (see Audio, images and video). Applies to every call on the channel. |
output-voice | provider default | The voice of a spoken answer. A request’s outputVoice overrides it. |
voice | native | native sends audio to the model as audio; pipeline transcribes it, asks a text model, and speaks the answer (see The voice pipeline). |
speech | none | The speech provider of a voice: pipeline channel: provider, api-key, base-url, transcribe-model, speech-model, optional voice and language. |
media-store | none | The object store (sutra.object-stores.<name>, engine configuration) this channel reads media references from. Without it, a request carrying a reference is refused. |
media-max-bytes | 20971520 (20 MiB) | The largest object one media reference may name. |
Providers
provider: | Service | Default endpoint | Notes |
|---|---|---|---|
openai | OpenAI | https://api.openai.com/v1 | Also any compatible server via base-url. With an audio chat model (a model id containing audio), audio in and spoken answers out. |
anthropic | Anthropic | https://api.anthropic.com | |
aws-bedrock | Amazon Bedrock (Converse API) | none — set base-url: https://bedrock-runtime.<region>.amazonaws.com | Authenticates with a Bedrock API key; IAM request signing is not supported. |
gemini | Google Gemini, native API | https://generativelanguage.googleapis.com | Gemini’s own generateContent API: audio, images and video in, JSON answers, tools, streaming. The key travels in the x-goog-api-key header. |
google-gemini | Google Gemini | https://generativelanguage.googleapis.com/v1beta/openai | Gemini’s Chat Completions-compatible endpoint: text only. Use gemini for audio or video. |
mistral | Mistral AI | https://api.mistral.ai/v1 | |
deepseek | DeepSeek | https://api.deepseek.com | |
xai | xAI | https://api.x.ai/v1 | |
kimi | Moonshot AI (Kimi) | https://api.moonshot.ai/v1 | |
meta-llama | Meta Llama API | https://api.llama.com/compat/v1 |
Every service except anthropic, aws-bedrock and gemini speaks the Chat Completions wire
format and is served by one adapter; the provider: id only picks the default endpoint. Override base-url for
a regional host, a proxy, or a self-hosted server.
Which providers exist depends on how the engine binary was built (llm-openai, llm-anthropic,
llm-bedrock, llm-gemini cargo features; the standard image includes all four). A channel naming a provider
the binary does not include fails at load with SUTRA.LLM.PROVIDER.UNKNOWN.
Jobs and webhooks
Some providers accept a request and finish it later (background responses, batch jobs). With
mode: job the channel submits the request, the provider holds the work, and the result is
delivered on reply-channel when it completes — found by polling, by a webhook, or both. The
flow is the async one above: send, park, resume by <q:alias>. Jobs need ack-mode: on-persist
and an engine with a database.
- name: summarize
direction: outbound
transport: llm
bind: "llm://summarize"
llm:
mode: job
reply-channel: summarize-reply
candidates:
- provider: openai
model: <model-id>
api-key: secret:llm/openai
completion: callback # the provider notifies us
callback-channel: openai-hooks
poll-interval-seconds: 300 # safety net: a webhook can be lost or late
job-timeout-seconds: 86400
- provider: anthropic
model: <model-id>
api-key: secret:llm/anthropic
completion: poll # no webhooks: poll
poll-interval-seconds: 30
# The webhook endpoint: an inbound llm channel.
- name: openai-hooks
transport: llm
bind: "POST /llm/openai" # always POST /llm/<path>, on the shared HTTP listener
llm:
callback-for: openai
callback-secret: secret:llm/openai-webhook
| Key | Where | Default | Meaning |
|---|---|---|---|
completion | candidate | poll | poll or callback. |
callback-channel | candidate | none | Required with completion: callback, and only then. Names an inbound transport: llm channel in the same package whose callback-for is this candidate’s provider. |
poll-interval-seconds | candidate | 30 (poll), 300 (callback) | Poll cadence; with callback it is a safety-net sweep. |
job-timeout-seconds | candidate | 86400 | A job not finished by then is marked failed. |
callback-secret | candidate or callback channel | none | Secret reference used to verify the webhook signature. A callback is accepted if the callback channel’s secret or any referencing candidate’s secret verifies, so rotation is a two-step change. At least one is required. |
callback-for | callback channel | none | Required: the provider whose webhooks this endpoint takes. |
The candidate job keys are refused on a mode: call channel. Candidates that cannot do jobs are
skipped at submit, like a candidate that lacks a require: capability.
The webhook endpoint answers 404 for an unknown path, 401 for a bad signature, 400 for a
body the provider adapter cannot parse, 503 when the job row cannot be claimed or the database is
unavailable (so the provider retries the webhook), and 200 otherwise (also when it had nothing
left to deliver — the same job is delivered once, whether the poller or the webhook finishes it
first).
A job that fails, is rejected, or passes job-timeout-seconds delivers nothing and logs
SUTRA.LLM.CALL.FAILED; the flow’s own <q:timeout> handles the missing answer. Broker-routed
callbacks are not supported.
Misconfiguration fails at load with SUTRA.LLM.CONFIG.INVALID: mode: job on an on-complete
channel or without a database, no candidate that supports jobs, completion: callback on a
provider without webhooks, a missing or dangling callback-channel, a provider mismatch, no
secret, or a callback bind that is not POST /llm/<path> or reuses a path.
What each provider supports
provider: | Streaming | Jobs | Webhooks |
|---|---|---|---|
openai | yes | yes | yes |
anthropic | yes | yes (polled) | no |
gemini | yes | no | no |
google-gemini, mistral, deepseek, xai, kimi, meta-llama (Chat Completions services) | yes | no | no |
aws-bedrock | no | no | no |
Streaming to the caller
A token stream means nothing inside a flow — a variable is useful only once complete — but it matters to the caller waiting on the last step. Two switches turn it on:
- name: ask
transport: http
bind: "POST /ask"
ack-mode: on-complete
reply-stream: sse # inbound HTTP channel: answer as server-sent events
<serviceTask id="Answer" implementation="channel:summarize-sync" q:stream="true"> ... </serviceTask>
The call must go to a sync (ack-mode: on-complete) transport: llm channel, and the provider
must support streaming. The response becomes text/event-stream:
event: chunk
data: Hel
event: chunk
data: lo
event: reply
data: {"summary":"Hello"}
| Event | Data | Meaning |
|---|---|---|
chunk | a piece of the answer text | Repeats. A multi-line piece uses one data: line per line. |
reset | empty | A candidate failed mid-answer and the next one started over: discard the chunks received so far. |
reply | the flow’s reply body | The flow finished; the stream then closes. |
error | the diagnostic code only (never its message) | The flow failed; the stream then closes. |
Chunks go to the caller only. The flow continues with the whole accumulated answer in variables
and audit, exactly as for a non-streamed call. If the flow streams nothing, or the channel has no
reply-stream, the request is answered as an ordinary response and q:stream is ignored.
Deploy-time errors: SUTRA.CONFIG.STREAM.TARGET_NOT_SYNC_LLM for q:stream="true" on anything
but a channel-call to a sync llm channel, and SUTRA.CONFIG.STREAM.REPLY_STREAM_UNSUPPORTED
for reply-stream: sse on anything but an inbound http channel with ack-mode: on-complete.
The request
The channel-call task’s input is a JSON object:
| Field | Meaning |
|---|---|
prompt | A single user message. |
messages | A conversation: [{ "role": "user" | "assistant" | "tool", "content": "…", "toolCallId": "…" }]. Use prompt, messages, or both (prompt is appended). |
system | Overrides the channel’s system. |
json | true asks for a JSON object as the answer. The reply carries it parsed under json, or jsonError saying why it did not parse. The answer may be bare JSON or one fenced block (```json), with or without prose around it. See How JSON is asked for. |
maxOutputTokens, temperature | Override the channel’s values. |
tools | [{ "name", "description", "parameters": <JSON Schema> }] the model may call. |
messages[].audio | Audio on a user turn: { "data": <base64>, "format": <media type> }. See Audio, images and video. |
messages[].video | Video on a user turn: a clip { "data", "format": "video/…" }, or sampled frames { "frames": [{ "data", "format": "image/…", "atMs" }] }. |
messages[].image | One image on a user turn, { "data", "format": "image/…" }, read as a one-frame video (the candidate needs video-understanding). |
ref, sha256 | Any audio, video, frame or image part may give { "ref": <object URI>, "sha256": <hex> } instead of data: a media reference. |
audioOutput | Ask for a spoken answer in this format, for example "audio/L16;rate=8000". |
outputVoice | The spoken answer’s voice; overrides the channel’s output-voice. |
Build the prompt in the process — a template task, a FEEL string, or a <q:param> — so it is
versioned with the flow that uses it.
How JSON is asked for
Every provider declares json-output, but not every model has a JSON mode, so each adapter asks
in the way its model accepts. Either way the transport parses the answer into json.
| Provider / model | How json: true is sent |
|---|---|
openai and the Chat Completions services, text models | response_format: {type: json_object} (text.format for a job). The JSON modes refuse a request that never says “JSON”, so when neither system nor any message mentions it, Respond with a single JSON object and nothing else. is added to the system text. |
openai audio chat models (a model id containing audio: gpt-audio…, gpt-4o-audio-preview) | In words only: the instruction above in the system text, no response_format. These models refuse the JSON mode with HTTP 400 (“…response_format of type json_object is not supported with this model”). They often answer with the JSON in a fenced block inside prose; the transport finds and parses it. |
gemini (native) | responseMimeType: application/json; with tools, in words. |
anthropic, aws-bedrock | In words (neither has a JSON mode). |
So a channel with require: [audio-understanding, json-output] still selects an OpenAI audio
model. A model asked in words can still answer prose with no JSON in it: the reply then carries
jsonError, and a flow that needs the object should check for it.
The reply
The answer is this JSON message — delivered on reply-channel for an async call, the response
itself for a sync one:
{
"text": "…",
"toolCalls": [{ "id": "…", "name": "…", "arguments": { } }],
"finish": "stop | length | tool-calls | filtered | …",
"usage": { "inputTokens": 812, "outputTokens": 96 },
"provider": "anthropic",
"model": "<model-id>",
"providerRequestId": "…",
"request": { }
}
request echoes the request body, so an async task’s <q:alias> can correlate on a field the
process put there (for example body.request.ticketId). The headers of the request ride along
too, plus x-llm-provider and x-llm-model.
A finish of filtered means the provider withheld the answer. It is a result, not a failure:
route on it with a gateway rather than retrying.
Structured answers. A call that asked for JSON (json: true) adds json: the answer parsed
into an object or array (a fenced code block is unwrapped), so the flow reads its fields
directly, as payload.json.category, instead of parsing text. When the answer does not parse,
json is absent and jsonError says why: either the answer was cut off at max-output-tokens
(finish: "length"), or it is not a JSON object or array. Reasoning models such as
gemini-2.5-flash think before they answer, and the thinking counts against
max-output-tokens, so give a JSON channel on such a model room (the examples use 2048).
Audio. A call involving audio adds audio ({ "data": <base64>, "format": <the audioOutput asked for> }), transcript (what the spoken answer says) and audioPreserved (true when the
model heard the audio itself, false when it went through the pipeline).
In the echoed request, every audio or video part keeps its format and has its data replaced
by bytes (the decoded size), so the reply never carries the media back.
Audio, images and video
Preview. Video understanding (clips, frames, images) and live audio sessions are preview in 0.6: they have been run against the real Gemini and OpenAI APIs as well as against in-process stand-ins of them, and their shapes may still change. Audio on a turn-based call (
audio,audioOutput,voice: pipeline) is not preview.
Audio stays audio. A user turn’s audio goes to the model as audio, so an audio-native model
hears the voice itself (tone, stress, hesitation), not a transcript of it. Speech-to-text in
front of a text model happens only when a channel asks for it with
voice: pipeline.
- name: voice-model
direction: outbound
transport: llm
bind: "llm://voice-model"
ack-mode: on-complete
llm:
candidates:
- provider: openai
model: <audio-model-id> # an audio chat model: its id contains `audio`
api-key: env:OPENAI_API_KEY
require: [audio-understanding] # skip any candidate that would flatten the voice
output-voice: alloy
The request names the format it has and the format it wants back:
{
"messages": [{ "role": "user", "content": "Answer the caller.",
"audio": { "data": "<base64>", "format": "audio/PCMU" } }],
"audioOutput": "audio/L16;rate=8000"
}
Formats are media types. audio/wav; audio/L16;rate=<hz>[;channels=<n>] (also written
audio/pcm;rate=<hz>: signed 16-bit little-endian, mono unless stated); audio/PCMU and
audio/PCMA (G.711 µ-law and A-law, 8 kHz); and the compressed audio/mpeg, audio/flac,
audio/opus, audio/ogg, audio/aac, audio/webm and audio/mp4. The transport converts
between the call’s formats and the provider’s in-process: WAV wrap and unwrap, G.711, channel
mixing and linear resampling. A compressed format is passed through, never converted. If the
provider does not take it as it is, the call fails with SUTRA.LLM.AUDIO.UNSUPPORTED in the
diagnostic.
Audio is never dropped. A candidate that cannot take what the call carries (audio input, a
spoken answer, video) is skipped, exactly like a candidate without tools. If none can, the call
fails permanently. A mode: job channel carries no audio or video and refuses such a call.
require: value | A candidate must… |
|---|---|
audio-input | accept audio on a message |
audio-output | answer in audio |
audio-understanding | hear the voice itself (emotion, tone), not a transcript |
video-understanding | read video: a clip, sampled frames or an image |
realtime | open a live audio session |
provider: | Audio in | Spoken answer | Images and video |
|---|---|---|---|
openai, with an audio chat model (id containing audio) | WAV, MP3 | WAV, MP3, FLAC, Opus, AAC | no |
gemini | WAV, MP3, FLAC, AAC, Ogg, Opus, M4A, WebM | no | clips, frames, images |
| every other provider | no | no | no |
Images and video (preview). A user turn may carry video, either a clip ({ "data", "format": "video/mp4" }) or a run of sampled frames ({ "frames": [{ "data", "format": "image/png", "atMs": 0 }, …] }), or one image ({ "data", "format": "image/png" }), which is
read as a one-frame video. Only gemini reads them today. Combined with json: true, the flow
gets the answer parsed under json:
- name: vision-model
direction: outbound
transport: llm
bind: "llm://vision-model"
ack-mode: on-complete
llm:
candidates:
- provider: gemini
model: <gemini-model-id>
api-key: env:GEMINI_API_KEY
require: [video-understanding, json-output]
The voice pipeline
For a model without audio, or for cost, a channel can transcribe, answer in text and speak the answer, explicitly:
llm:
voice: pipeline
speech:
provider: openai
api-key: env:OPENAI_API_KEY
transcribe-model: <transcription-model-id>
speech-model: <speech-model-id>
voice: alloy # optional; a request's outputVoice overrides it
language: en # optional transcription hint
candidates:
- provider: anthropic
model: <model-id>
api-key: env:ANTHROPIC_API_KEY
Each audio turn is transcribed by speech and becomes text, appended to the turn’s content.
The candidates answer as text. With audioOutput, speech speaks the answer, and the reply’s
transcript is the answer text. The reply then says audioPreserved: false: the voice never
reached the model, so a flow that needs it (an emotion analysis, say) can refuse the answer. A
native answer says audioPreserved: true. A text-only call on a pipeline channel is not touched.
voice is native (the default) or pipeline. llm.speech is required with pipeline and
refused without it. The speech provider must do both transcription and speech synthesis (today,
openai). A pipeline channel cannot require an audio capability and cannot be mode: job.
Each of these is a SUTRA.LLM.CONFIG.INVALID at load.
Media references
Any audio, video (the clip, or each frame) or image part may name an object in an object
store, { "ref": "s3://…", "sha256": "<hex>" }, instead of inlining data. The channel’s
media-store names the store it may read. The transport reads the object, checks its SHA-256
and size, and sends it as if it had been inline; variables, snapshots and audit only ever hold
the reference. Recordings and media references covers the object
store, the refusals and a walkthrough.
Try it: audio, the pipeline and vision
This runs three flows of the examples/ai-advanced package against the stub model server:
| Flow | Channel | Model channel | Candidate |
|---|---|---|---|
shared/bpmn/voice.bpmn | POST /channels/voice | voice-model | openai, audio model, require: [audio-understanding] |
shared/bpmn/voice-pipeline.bpmn | POST /channels/voice-pipeline | voice-pipeline-model | voice: pipeline: openai speech, anthropic text model |
shared/bpmn/vision.bpmn | POST /channels/vision | vision-model | gemini, require: [video-understanding, json-output] |
Prerequisites: Before you try a walkthrough.
No key is needed. In variants/live/channels.yaml the same channels name real models, with
their keys as env:OPENAI_API_KEY, env:ANTHROPIC_API_KEY and env:GEMINI_API_KEY, resolved
from the engine’s environment.
The vision flow builds its request in one FEEL data task. It sends either the image or the frames, and asks for JSON:
<bpmn:assignment>
<bpmn:from>{messages: [if payload.image != null then {role: "user", content: "Describe the image as one JSON object: scene (one sentence), objects (a list of short names) and framesSeen (1).", image: payload.image} else {role: "user", content: "These are the frames of one short clip, in order. Describe the whole clip as ONE JSON object, not one per frame: scene (one sentence), objects (a list of short names) and framesSeen (how many frames you received).", video: payload.video}], json: true}</bpmn:from>
<bpmn:to>requestBody</bpmn:to>
</bpmn:assignment>
It then reads the parsed answer’s fields directly:
<bpmn:from>{requestId: payload.requestId, scene: answer.json.scene, objects: answer.json.objects, framesSeen: answer.json.framesSeen, provider: answer.provider}</bpmn:from>
1. Package the stub variant and start the stack (engine, PostgreSQL, the stub):
C=examples/ai-advanced/deploy/compose.yaml
sutra package examples/ai-advanced/variants/stub --overlay examples/ai-advanced/shared \
--out examples/ai-advanced/deploy/deployments
docker compose -f $C up -d --wait
PORT=$(docker compose -f $C port engine 8080 | cut -d: -f2)
H=(-H 'Content-Type: application/json' -H 'X-Api-Key: dev-only-advanced-key')
until curl -sf "http://127.0.0.1:$PORT/sutra/deployments" | grep -q '"phase":"Active"'; do sleep 1; done
sutra package prints one SUTRA.CHANNEL.NO_SCHEMA warning per schema-less JSON channel, and
then:
packaged examples/ai-advanced/deploy/deployments/stub.sutra (deploymentId dep-163c73f2de054d4963d7812b)
2. Audio in, a spoken answer out. The sample recording is a quarter-second WAV. The request asks for 16 kHz linear PCM back. The helper shortens the returned audio for display:
AUDIO=$(base64 -w0 examples/ai-advanced/sample/question.wav)
short() { python3 -c 'import json,sys; r=json.load(sys.stdin); r["audio"]["data"]=r["audio"]["data"][:12]+"..."; print(json.dumps(r, indent=2))'; }
curl -s -X POST "http://127.0.0.1:$PORT/channels/voice" "${H[@]}" \
-d "{\"requestId\":\"Q-1\",\"audio\":{\"data\":\"$AUDIO\",\"format\":\"audio/wav\"},\"audioOutput\":\"audio/L16;rate=16000\"}" | short
{
"audio": {
"data": "oA+gD6APoA+g...",
"format": "audio/L16;rate=16000"
},
"audioPreserved": true,
"provider": "openai",
"requestId": "Q-1",
"text": "STUB SPOKEN ANSWER: your appointment is confirmed for Tuesday.",
"transcript": "STUB SPOKEN ANSWER: your appointment is confirmed for Tuesday."
}
The stub spoke a 24 kHz WAV, and the transport unwrapped and resampled it to the
audio/L16;rate=16000 the request asked for.
3. The same question through the pipeline, asking for G.711 µ-law back:
curl -s -X POST "http://127.0.0.1:$PORT/channels/voice-pipeline" "${H[@]}" \
-d "{\"requestId\":\"Q-2\",\"audio\":{\"data\":\"$AUDIO\",\"format\":\"audio/wav\"},\"audioOutput\":\"audio/PCMU\"}" | short
{
"audio": {
"data": "r6+vr6+vr6+v...",
"format": "audio/PCMU"
},
"audioPreserved": false,
"provider": "anthropic",
"requestId": "Q-2",
"text": "STUB SUMMARY: the customer reports a problem and asks for it to be looked at.",
"transcript": "STUB SUMMARY: the customer reports a problem and asks for it to be looked at."
}
The text model (Anthropic) answered, and audioPreserved: false says it never heard the voice.
4. One image to the native Gemini provider:
IMG=$(base64 -w0 examples/ai-advanced/sample/frame-1.png)
curl -s -X POST "http://127.0.0.1:$PORT/channels/vision" "${H[@]}" \
-d "{\"requestId\":\"V-1\",\"image\":{\"data\":\"$IMG\",\"format\":\"image/png\"}}"; echo
{"framesSeen":1,"objects":["bicycle","door"],"provider":"gemini","requestId":"V-1","scene":"STUB SCENE: a bicycle leans against a blue door."}
5. Two sampled video frames:
F2=$(base64 -w0 examples/ai-advanced/sample/frame-2.png)
curl -s -X POST "http://127.0.0.1:$PORT/channels/vision" "${H[@]}" \
-d "{\"requestId\":\"V-2\",\"video\":{\"frames\":[{\"data\":\"$IMG\",\"format\":\"image/png\",\"atMs\":0},{\"data\":\"$F2\",\"format\":\"image/png\",\"atMs\":500}]}}"; echo
{"framesSeen":2,"objects":["bicycle","door"],"provider":"gemini","requestId":"V-2","scene":"STUB SCENE: a bicycle leans against a blue door."}
scene, objects and framesSeen came from answer.json: the flow never parsed text.
6. What the stub was asked, by route:
docker compose -f $C exec model-stub wget -qO- http://127.0.0.1:8000/stats; echo
{"chat": 1, "chat-audio-in": 1, "chat-audio-out": 1, "transcriptions": 1, "messages": 1, "speech": 1, "gemini-generate": 2}
The native call is one audio chat completion. The pipeline is one transcription, one Anthropic
message and one speech synthesis. The two vision calls are two Gemini generateContent requests.
7. Clean up:
docker compose -p sutra-ai-advanced down -v
Live audio sessions (preview)
A provider adapter can also open a live audio session (AudioSession): audio frames in,
transcript deltas and the model’s audio and text out. The openai adapter does it over the
Realtime WebSocket API for a model id containing realtime, and the gemini adapter does it over
the Live API for a model id containing live. The gemini adapter never puts the API key in a
URL. It mints a single-use ephemeral token with the key (sent in the x-goog-api-key header) and
opens the session with that token in the handshake’s Authorization header.
A process never opens a session. A media transport does, such as the telephony transport of
sutra-voice, and it opens it through an llm
channel of its own deployment, so it gets the same routing as a process’s call (ordered
candidates, fallback-on, require:, key references, concurrency, and the sutra.llm.call
span, with session = true). The media goes from that transport to the provider directly and
never passes through a process. Such a channel answers inline, so it declares
ack-mode: on-complete and no reply channel:
- name: live-transcription
direction: outbound
transport: llm
bind: "llm://live-transcription"
ack-mode: on-complete
llm:
require: [realtime] # only candidates with live audio sessions
concurrency: 16 # an open session holds one permit for its life
candidates:
- provider: openai
model: <realtime-model-id>
transcription-model: <transcription-model-id>
api-key: env:OPENAI_API_KEY
- provider: gemini # the fallback
model: <live-model-id>
api-key: env:GEMINI_API_KEY
A session opens on the first candidate that has realtime and what require: names. A failure
listed in fallback-on moves it to the next candidate, exactly as for a call. A transport may
also ask for a transcription-only session: on openai that is a Realtime transcription
session, which reports the start and end of each stretch of speech as offsets into the audio.
Gemini Live has no listen-only session, so it is asked to stay silent and anything it says is
ignored. A mode: job channel cannot serve a transport.
Turn detection. A session either lets the provider’s voice-activity detection decide where
the caller’s turn ends (the default), or the transport marks each turn itself (manual turns).
With the gemini adapter, provider detection runs at a low end-of-speech sensitivity with 800 ms
of silence, so a pause inside a sentence does not end the turn. Provider detection is tuned for
real speech: in our checks, gemini-3.8-live took only the first word of a synthetic (text-to-
speech) question for speech and answered that, while it handled a recorded human voice, and
gemini-2.5-flash-native-audio-latest handled both. Manual turns handled every case. If your
callers’ audio is unusual (synthetic prompts, heavy noise), test with your own recordings or use
manual turns.
There is no walkthrough here, because a live session needs a media transport to open it and the
engine in this repository ships none. The sutra-voice repository runs it end to end.
examples/ai-advanced/live-check.sh checks the adapters themselves against the real APIs.
Failures
| What happened | Sync (on-complete) | Async (on-persist) |
|---|---|---|
| Every candidate failed transiently, or the call timed out | With <q:retry>: the call is re-issued inline with the policy’s backoff, all within the <q:timeout> budget; once spent, the failure’s code (SUTRA.LLM.CALL.FAILED, or SUTRA.DISPATCH.CHANNEL_CALL.TIMEOUT) is raised as a BPMN error on the task. Without it: a timeout fires the task’s #timeout boundary, which raises SUTRA.DISPATCH.CHANNEL_CALL.TIMEOUT; any other failure raises SUTRA.LLM.CALL.FAILED. Uncaught, the instance fails. | The delivery is retried with the outbox’s backoff. If it never succeeds, the task’s <q:timeout> fires, or the outbox attempt ceiling poisons it — both are retryable failures under <q:retry>. |
| The request was rejected (bad key, unknown model, malformed request) | A permanent failure: no retry. SUTRA.LLM.CALL.FAILED is raised as a BPMN error on the task; a boundary on that code, a boundary with no code, or an error event sub-process catches it. Uncaught, the instance fails. | The delivery is poisoned and stays redrivable; the provider’s code does not reach the task. The task’s <q:timeout> fires (SUTRA.DISPATCH.CHANNEL_CALL.TIMEOUT). With sutra.outbox.retry.max-attempts set, the row turns terminal first and the task gets SUTRA.OUTBOUND.DELIVERY_ATTEMPTS_EXHAUSTED: through <q:retry> when it has one, then as a BPMN error when something catches it. |
A sync retry never parks a durable backoff; a flow that needs one should use the async channel.
Catching a failed call
A sync call’s permanent failure is a BPMN error on its task, so a flow that can carry on without the answer says so with a boundary:
<bpmn:error id="callFailed" errorCode="SUTRA.LLM.CALL.FAILED"/>
...
<bpmn:boundaryEvent id="ModelRefused" attachedToRef="AskModel">
<bpmn:outgoing>F5</bpmn:outgoing>
<bpmn:errorEventDefinition errorRef="callFailed"/>
</bpmn:boundaryEvent>
An empty <bpmn:errorEventDefinition/> catches any failure. Boundaries are tried in document order,
so declare one on a specific code before a catch-all. With no boundary on the task, a process-level
error event sub-process is next; a channel call cannot sit inside a sub-process, so there is no
scope between them. A <q:reply> inside an event sub-process does not answer a waiting HTTP caller,
so when a caller waits, use a boundary and reply on its route. With nothing to catch it, the instance
fails as it always did.
The catching path reads callFailure, a context the engine sets when the failure is raised:
| Field | What it holds |
|---|---|
callFailure.code | The code the boundary matched. |
callFailure.message | The reason: for a model call the channel, provider and model, the HTTP status, the provider’s code and its message cut to 300 characters, with credentials redacted. Never the prompt or the answer. |
callFailure.node | The id of the task whose call failed. |
An answer the call’s own contract refuses (an output mapping that cannot be evaluated) is caught the
same way, and its message says only that the answer was refused, since the diagnostic may quote it.
A <q:timeout> that fires with no modelled route sets callFailure too. The variable stays set
after the catch, like any other, until the next failure replaces it. A flow’s own callFailure
variable is overwritten.
A failure under <q:retry> is caught only when the budget is spent, under the failure’s code, not
SUTRA.RUNTIME.RETRY.EXHAUSTED (which stays the code of the failed instance when nothing catches
it). A loop from the boundary back to the call starts on a fresh budget.
The same holds for any synchronous channel, not only llm: the HTTP transport’s code is
SUTRA.OUTBOUND.SEND.FAILED.
Try it: a model that refuses the request
This runs four flows of the examples/ai-advanced stub variant against a stub model that answers
every call with HTTP 400:
| Flow | Channel | The call | What catches it |
|---|---|---|---|
variants/stub/bpmn/refused-guarded.bpmn | POST /channels/refused-guarded | With the HTTP caller waiting | A boundary on SUTRA.LLM.CALL.FAILED |
variants/stub/bpmn/refused-unguarded.bpmn | POST /channels/refused-unguarded | With the HTTP caller waiting | Nothing |
variants/stub/bpmn/refused-later-guarded.bpmn | POST /channels/refused-later-guarded | On a stored instance, after a timer | A boundary with no code |
variants/stub/bpmn/refused-later-unguarded.bpmn | POST /channels/refused-later-unguarded | On a stored instance, after a timer | Nothing |
Prerequisites: Before you try a walkthrough. No key is needed.
1. Package the stub variant and start the stack:
C=examples/ai-advanced/deploy/compose.yaml
sutra package examples/ai-advanced/variants/stub --overlay examples/ai-advanced/shared \
--out examples/ai-advanced/deploy/deployments
docker compose -f $C up -d --wait
PORT=$(docker compose -f $C port engine 8080 | cut -d: -f2)
H=(-H 'Content-Type: application/json' -H 'X-Api-Key: dev-only-advanced-key')
until curl -sf "http://127.0.0.1:$PORT/sutra/deployments" | grep -q '"phase":"Active"'; do sleep 1; done
2. Caught, with the caller waiting. The boundary routes to a task that answers from
callFailure:
<bpmn:from>{ticketId: payload.ticketId, recovered: true, code: callFailure.code, reason: callFailure.message}</bpmn:from>
curl -s -X POST "http://127.0.0.1:$PORT/channels/refused-guarded" "${H[@]}" \
-d '{"ticketId":"T-1","description":"My parcel never arrived."}' | python3 -m json.tool
The caller gets the recovery path’s reply, with HTTP 200:
{
"code": "SUTRA.LLM.CALL.FAILED",
"reason": "llm channel 'refusing-model': every candidate failed; last (openai/refuses-everything): HTTP 400 (model_refuses): This model refuses the request.",
"recovered": true,
"ticketId": "T-1"
}
3. Uncaught, with the caller waiting. The same flow without the boundary:
curl -s -w '\nHTTP %{http_code}\n' -X POST "http://127.0.0.1:$PORT/channels/refused-unguarded" "${H[@]}" \
-d '{"ticketId":"T-2","description":"My parcel never arrived."}'
The caller gets the failure:
{"type":"urn:bpm:diag:SUTRA.LLM.CALL.FAILED","title":"SUTRA.LLM.CALL.FAILED","status":500,"detail":"llm channel 'refusing-model': every candidate failed; last (openai/refuses-everything): HTTP 400 (model_refuses): This model refuses the request.","code":"SUTRA.LLM.CALL.FAILED","attributes":{"incident":"SUTRA.INBOUND.NON_IDEMPOTENT_FAILURE"}}
HTTP 500
4. On a stored instance. No caller waits here, which is the case that used to lose work: a timer stores the instance, then the call runs. The guarded flow’s boundary hands the reason to the stub’s sink and completes; the unguarded flow fails durably:
for flow in refused-later-guarded refused-later-unguarded; do
curl -s -o /dev/null -w "$flow: HTTP %{http_code}\n" -X POST "http://127.0.0.1:$PORT/channels/$flow" "${H[@]}" \
-d '{"ticketId":"T-3","description":"My parcel never arrived."}'
done
sleep 12
docker compose -f $C exec model-stub wget -qO- http://127.0.0.1:8000/sink; echo
curl -s "http://127.0.0.1:$PORT/sutra/instances?status=COMPLETED" | grep -o '"status":"[A-Z]*"'
curl -s "http://127.0.0.1:$PORT/sutra/instances?status=FAILED" | grep -o '"status":"[A-Z]*"'
docker compose -f $C logs engine | grep INSTANCE_FAILED | grep -o '"process":"[^"]*","cause_code":"[^"]*"'
refused-later-guarded: HTTP 202
refused-later-unguarded: HTTP 202
["{\"code\":\"SUTRA.LLM.CALL.FAILED\",\"reason\":\"llm channel 'refusing-model': every candidate failed; last (openai/refuses-everything): HTTP 400 (model_refuses): This model refuses the request.\",\"ticketId\":\"T-3\"}"]
"status":"COMPLETED"
"status":"FAILED"
"process":"refused-later-unguarded","cause_code":"SUTRA.LLM.CALL.FAILED"
5. Clean up:
docker compose -p sutra-ai-advanced down -v
Finding the reason for a failed call
The provider’s reason for a failure is logged at WARN, for sync and async calls alike, under the
code SUTRA.LLM.CALL.FAILED: the same code the engine’s instance-failed line names as its
cause_code, so one search for that code finds both. Two lines are written:
llm candidate failed: <reason>, once per candidate tried, withchannel,provider,model,candidate(its index),class(rate-limited,unavailable,timeoutorpermanent),http_status,provider_codeandfall_back;llm call failed: <diagnostic>, once for the call, with the same fields plustrace_id(from the call’straceparent) andinstance(when the engine stamped the instance id on the call, as it does for an audited process;-otherwise).
The reason is the provider’s HTTP status, its error code (code, status or type of its error
object) and its message, on one line and cut to 300 characters, for example
HTTP 400 (invalid_request_error): Invalid parameter: 'response_format' of type 'json_object' is not supported with this model. Every URL in it has its query values and credentials replaced
by <redacted>, and the candidate’s key is replaced wherever a provider echoes it. The same
reason is in the failure diagnostic, which the engine stores with a failed instance (and with a
dead-lettered start), with http_status and provider_code as attributes. No admin API is
needed to read it.
Duplicate calls. Async delivery is at-least-once: a crash after the model answered but before
the delivery was recorded calls the model again; inbox deduplication absorbs the duplicate reply,
but the call is paid for twice (a job channel likewise risks a duplicate provider job if the
submit’s record fails). A sync call lives only in memory: a crash mid-call loses it, and the
client’s retry re-runs the flow from the start, calling the model again. Sync calls made by a
timer or scheduled start run inline too; only a broadcast fan-out degrades to the async path,
with a SUTRA.CHANNEL.SYNC_CALL_DEGRADED warning.
Tools and agent loops
The transport never runs tools. When finish is tool-calls, route the token to tasks that run
the requested tools (other channels), append their results to messages as tool turns, and
loop back to the model task. Bound the loop with a counter and a non-interrupting timer boundary.
What is not logged
Prompts and answers are never written to spans, logs or diagnostics. Each call opens a
sutra.llm.call span carrying the channel, provider, model, finish reason and token counts. A
failure’s log lines carry the provider’s error message, never the request, and keys are redacted
from it (see Finding the reason for a failed call).
Examples
Runnable packages in the source tree. The two ai- examples keep their flows once under
shared/ and one channels.yaml per provider under variants/, composed by
sutra package --overlay:
| Example | Variants | Flows |
|---|---|---|
examples/ai-summarize | stub-openai, stub-anthropic, openai, anthropic, gemini | sync, streamed (SSE), async |
examples/ai-advanced | stub, live | cross-provider fallback (Anthropic first, OpenAI second), JSON mode, a tool call, an OpenAI background job (polled), an OpenAI background job (webhook), an Anthropic Message Batch, audio in and out, audio in with a parsed JSON answer out (an OpenAI audio model), the voice pipeline, images and video frames to native Gemini |
examples/media-recordings | — | a stored recording read by media reference from an object store, audited by reference, deleted on erasure and retention (walkthrough) |
sutra package examples/ai-summarize/variants/anthropic --overlay examples/ai-summarize/shared \
--out examples/ai-summarize/deploy/deployments
sutra package examples/ai-advanced/variants/stub --overlay examples/ai-advanced/shared \
--out examples/ai-advanced/deploy/deployments
They share a stub model server (examples/ai-summarize/stub/model-stub.py) that speaks the
OpenAI, Anthropic and native Gemini wire formats with fixed answers, including streaming, tools,
JSON mode, audio, speech, transcription, background jobs, batches, signed webhooks and a
switchable outage for fallback.
- CI (tier 2, no keys): the
stub-openai,stub-anthropicand ai-advancedstubvariants, composed through the overlay path, run against the stub on a real engine and PostgreSQL — every flow above exceptvision(exercised by the walkthrough in this chapter), with the replies, the SSE frames, the sink deliveries and theinstance_stateandllm_jobrows checked. Every variant, the real-provider ones included, is also linted in tier 1. - Live (your keys):
examples/ai-advanced/live-check.shruns theopenai,anthropicandgeminivariants and the ai-advancedlivevariant against the real APIs, including a fallback where both the failing Anthropic candidate and the answering OpenAI one are real, and — with a tunnel andOPENAI_WEBHOOK_SECRET— a webhook-completed job.
The q: namespace
Standard BPMN 2.0 has no opinion on channels, message types, correlation, or replies. Sutra adds a
small, layered set of extension elements — the q: namespace, urn:sutra:q:1.0 — that live
entirely inside <bpmn:extensionElements>, so a Sutra process is still a valid, portable BPMN 2.0
diagram; the q: attributes just tell the engine how to wire it to the outside world. The
authoritative shape is xsd/q.xsd; the engine’s parser validates every <q:*> element against it,
and the same XSD drives the sutra-modeler-plugin property panels.
<bpmn:definitions xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL"
xmlns:q="urn:sutra:q:1.0" ...>
At a glance
| Element | Attaches to | What it declares |
|---|---|---|
q:source | start events, wait-capable nodes | The inbound trigger: channel, message type, ack mode, dedup, data class. |
q:validators / q:redactors | q:source | The validation and redaction chains over the decoded payload. |
q:alias | wait-capable nodes | A correlation key derived by FEEL — how a later inbound finds the parked instance. |
q:reply | tasks, end events | An outbound reply on the inbound’s own channel; continue="true" is respond-and-continue (reply now, keep executing detached); status (0.8.0) sets the HTTP status of a synchronous answer. |
q:send | throw events, send tasks | An unsolicited outbound message to a channel destination. |
q:header | q:send / q:reply | An author-declared outbound header; on a synchronous q:reply, an HTTP response header (0.8.0). |
q:decode | data task (bpmn:serviceTask with no @implementation) | Parses text the flow holds, such as a message kept in a typed store column, with a registered codec, and binds the result (0.6.19). |
q:param | bpmn:serviceTask | A scoped, per-invocation input to a registered task or template. |
q:retry | registered-task and channel-call bpmn:serviceTask | Per-task retry policy — attempts, backoff, non-retryable codes. |
q:timeout | channel-call bpmn:serviceTask | Synthesizes a timer boundary on the call. |
q:stream (attribute) | channel-call bpmn:serviceTask | q:stream="true" streams a synchronous model answer to the original caller. Default false. |
q:store | bpmn:dataStoreReference | Binds the reference to a durable key in a declared data store; reads, writes, deletes (0.7.0) and key queries. |
q:dispatch / q:case | bpmn:process | Content-based dispatch to called elements. |
q:variables / q:variable | bpmn:process | Declared process variables — type or schema, transient / sensitive / source. |
q:onValidation | bpmn:process | The structural-failure policy (route / reject / error). |
q:process | bpmn:process | The retry-safety (idempotency) assertion. |
q:audit | bpmn:process | Audit capture level (none / metadata / payload / media) and data-class tagging. |
q:coverage | bpmn:process | An opt-in tracked compliance path. |
flowchart TD
PROC["on the process itself:<br/>q:variables · q:onValidation · q:process<br/>q:audit · q:coverage · q:dispatch"]
subgraph flow["…and on the nodes inside it"]
S(["start event"]) --> T["serviceTask"]
T --> W["wait node"]
W --> X(["throw · send"])
DS[("dataStoreReference")]
end
S -.- A["q:source<br/>q:validators"]
T -.- B["q:param · q:retry<br/>q:timeout · q:reply"]
W -.- C["q:source · q:alias"]
X -.- D["q:send · q:header"]
DS -.- F["q:store"]
Nothing in the vocabulary is a new node type: every element hangs off a standard BPMN element’s extension elements, which is why a process carrying all of it is still a valid, portable BPMN 2.0 diagram.
q:source — the inbound trigger
Every message-consuming node — a start event or a wait-state catch — declares exactly one
q:source. It names the channel, the message type it accepts, and the variable the decoded
payload lands in:
<bpmn:startEvent id="Start">
<bpmn:extensionElements>
<q:source channel="transfer-request" messageTypeValue="TransferRequest"/>
</bpmn:extensionElements>
</bpmn:startEvent>
channel(required) — thechannels.yamlchannel name.messageTypeValue/messageTypePattern— subscribe to one exact type, or a family via regex, matched against the codec’s decoded message type. Neither set = accept anything the channel’s codec yields.name(defaultpayload) — the process-variable name the decoded body is projected under (payload.fromId,payload.body.CdtTrfTxInf...for a structured codec).ack(defaulton-persist) — see Acknowledgement modes.dedupKey— an expression identifying a redelivered duplicate (e.g.header.X-Request-Id), distinct from the process-level retry-safety assertion below.dedupKeyacceptsheader.<name>,ce.id,body.<path>andamqp.message-id; any other form is refused at lint, package and deploy withSUTRA.PARSE.Q_SOURCE.DEDUP_KEY_UNRECOGNISED. It dedupes on a start event and on a wait.dataClass(defaultnone) —pii/pci/phi/financial; drives redaction policy.filterExpression— a FEEL predicate over the message attributes the codec reported. It narrows whichever of the message-type selectors above applies rather than replacing one, so “exact beats pattern beats accept-anything” is unchanged.
Several waits on one channel
Since 0.6.10. A message on a channel that a WAIT subscribes to (a relay) chooses among the waits
by the same rules a start event uses: an exact messageTypeValue beats a messageTypePattern,
which beats a wait that declares no type, and filterExpression narrows within a tier. A wait
pinned to another message type never takes the message. Among the waits that accept it, the one
the correlated instance is actually parked at is resumed, the most specific first. So two waits on
one channel can be told apart by message type alone:
<bpmn:receiveTask id="WaitAnswer">
<bpmn:extensionElements>
<q:source channel="answers-in" name="answer" messageTypeValue="Answer"/>
<q:alias name="answerRef" expression="answer.ref" unique="true"/>
</bpmn:extensionElements>
</bpmn:receiveTask>
<bpmn:receiveTask id="WaitCancel">
<bpmn:extensionElements>
<q:source channel="answers-in" name="cancel" messageTypeValue="Cancel"/>
<q:alias name="cancelRef" expression="cancel.ref" unique="true"/>
</bpmn:extensionElements>
</bpmn:receiveTask>
Before 0.6.10 a relay ignored the wait’s message type, and such waits needed a filterExpression
on messageType to stay apart. That still works.
Selecting on attributes, not just a message type
One string cannot always say which process should handle a message. A message standard may use the same message type in more than one flavour — the same type arriving one way is a request to act on, and arriving the other way is an acknowledgement of something you sent — and the flavour, not the type, decides the handler.
A codec reports those facts as attributes alongside the message type, and a start event selects on the pair:
<q:source channel="rail" messageTypeValue="CreditTransfer"
filterExpression="attr.direction = "incoming""/>
Two start events may now pin the same message type and be told apart by an attribute, where before
that was an ambiguous handler. The predicate sees the attributes and messageType — not headers,
not the payload: admitting either would make the package-time checks below impossible and would
let one business message route differently depending on the transport it arrived over.
A codec declares the attribute keys it can report, and optionally their closed value domains.
sutra lint then refuses a predicate reading a key the bound channel’s codec can never emit, or
comparing against a value outside its domain — so "Incoming" for "incoming" fails at package
time instead of producing a start event that silently never fires. It also refuses two subscribers
on one non-broadcast channel whose predicates it cannot prove disjoint; the recognised form is
and / or / not over equality, inequality, in, and relational comparisons on numeric
attributes. That limit is permanent rather than provisional: the gate is a proof obligation, and
arbitrary FEEL cannot be proven disjoint. A codec declaring no attributes is open — the
predicate still works and lint warns that it cannot verify it rather than blocking.
The codec itself is not declared on q:source — it comes from the channel (YAML is
authoritative for transport/codec binding; BPMN is authoritative for process flow). Declaring a
codec on q:source is a parse error.
q:validators and q:redactors (nested under q:source)
<q:source channel="transfer-request" messageTypeValue="TransferRequest">
<q:validators>
<q:complexValidator source="transfer-limits.dmn"/>
<q:complexValidator source="transfer-fields.srl"/>
<q:simpleValidator ref="iso-4217-currency" path="payload.amount.currency"/>
</q:validators>
<q:redactors>
<q:redactor ref="pci"/>
</q:redactors>
</q:source>
q:validators is a mixed, ordered container: a q:complexValidator runs a whole-payload ruleset
(a .dmn or .srl file, or a built-in like iso-xsd) — a chain can mix .dmn and .srl entries
freely, run in declaration order, with every entry’s issues accumulating into one result (see
Composing a validator chain); a q:simpleValidator checks
one field at a FEEL path against a registered content validator (iso-3166-country,
iso-4217-currency, iso-9362-bic). q:redactors names registered ContentRedactors that mask
sensitive spans in every observability surface (audit, logs, traces) without touching the value
the flow actually sees.
q:reply and q:send — outbound
<bpmn:serviceTask id="OkReply" implementation="transfer-result.hbs">
<bpmn:extensionElements>
<q:reply mode="native" contentType="application/xml"/>
</bpmn:extensionElements>
</bpmn:serviceTask>
q:reply answers the caller that started this instance — mode="native" (default) preserves the
symmetric reply behavior, or emit a CloudEvent (cloudevent-binary / cloudevent-structured /
match-inbound). q:send is the unsolicited counterpart — an intermediate throw event emitting to
its own destination (@destination or @channel), with no inbound caller to answer. Both accept
<q:header name="…" value="…"/> children carrying FEEL-derived values onto transport
headers/application-properties.
q:reply’s continue="true" is respond-and-continue: flush the reply the moment the task
completes, then park the instance and self-resume the remaining nodes asynchronously — the caller
gets its answer without waiting on the tail of the flow.
A flow with no <q:reply> answers a synchronous HTTP call with 200 and an empty body; see
Upgrading to 0.6.18.
Status and response headers (0.8.0)
On a synchronous HTTP channel (ack-mode: on-complete, the HTTP default), a <q:reply> with no
destination can set the status, and its <q:header> children become the response’s headers:
<q:reply contentType="application/json" status="201">
<q:header name="Location" value="'/items/' + req.id"/>
<q:header name="X-Request-Id" value="req.id"/>
</q:reply>
status is literal first, else FEEL. Digits alone ("201") are the status. Any other text is a
FEEL expression over the variables the reply renders against ("if found then 200 else 404"). It
must give a whole number or a string of digits, and null keeps the default. Without the attribute
the answer is 200, as before. Headers are evaluated like a <q:send>’s: a null value leaves the
header out, and a repeated name adds another field line. A reply that declares a status or headers
answers with them even when the flow rendered no body, so a 401 with its challenge needs no
template.
| Rule | At deploy (sutra lint) | At the answer |
|---|---|---|
| The status is 200–599. A 1xx is not a final answer. | A literal outside is refused. | A value outside, a fraction, or a non-number is a 500. |
A 3xx needs a Location header. | A literal 3xx with no Location header is refused. | A 3xx whose Location is null is a 500. |
| A 204 or 304 carries no body. | — | Body and Content-Type are not sent. |
| A header name is an HTTP token. | Refused otherwise. | — |
The flow may not set Connection, Keep-Alive, Proxy-Connection, Transfer-Encoding, TE, Trailer, Upgrade, Content-Length, Content-Type (use contentType) or any Sutra-* name. | Refused. | — |
| A header value has no control character (CR, LF, NUL, …). | — | A 500. |
The status refusals use SUTRA.OUTBOUND.REPLY_STATUS_INVALID, the header ones
SUTRA.OUTBOUND.REPLY_HEADER_INVALID. The 500 is the usual application/problem+json. It names
the header, never its value.
The status applies at the end of the flow, at a <q:reply continue="true">, and on the answer to a
relayed message. An ack-mode: on-persist channel always answers 202, and adds the reply’s headers
next to its receipt. A broker intake has no status. sutra lint warns about both
(SUTRA.CHANNEL.REPLY_STATUS_IGNORED). A <q:reply> with a destination goes out of band, and its
headers travel with it as before. Answering with a status and headers
walks through it.
sequenceDiagram
participant C as Caller
participant E as Engine
C->>E: inbound message
E->>E: run as far as the reply task
E-->>C: reply flushed the moment that task completes
E->>E: park, then self-resume the remaining nodes
The caller’s answer is bounded by the task that produces it rather than by the tail of the process — the remaining nodes run on a resume nobody is waiting on.
The contract an outgoing message must satisfy
A message a flow produces is held to a codec’s contract before it leaves, the same way an inbound message is held to one on the way in. The codec’s own decode step is the validator, so the check is literally “read back what we are about to send” — there is no second contract to keep in step with the first.
This matters most for output a template, script or decision produces. Those renders reach the wire without passing through an encode step at all, so without this check an engine that validates every byte it accepts would validate none of the bytes it emits — and a flow could answer a strictly-validated request with a structurally invalid message.
Attribute on q:send / q:reply | Means |
|---|---|
codec="urn:…" | the codec whose contract this output must satisfy, stated rather than inferred |
messageType="…" | the message type this output emits — checked against what that codec can produce |
<bpmn:serviceTask id="Answer" implementation="ack.hbs">
<bpmn:extensionElements>
<q:reply codec="urn:acknowledgements" messageType="Acknowledgement"/>
</bpmn:extensionElements>
</bpmn:serviceTask>
State the codec when the output’s contract is not the one the bytes happen to leave by. A flow is a translator: it may accept one wire format and answer or forward in another, so the codec that decoded the request need not govern the answer.
The two forms differ in what happens when you say nothing, and the asymmetry is deliberate:
- A
<q:send>infers the codec of the@channelit names. That is not a guess — the bytes are addressed to that channel, and its codec is exactly what the consumers there will decode them with. A<q:send destination="…">names an endpoint rather than a channel and so has nothing to infer from, which makes stating the codec the only way to hold it to a contract. - A
<q:reply>infers nothing, and is unvalidated unless you state one. The tempting fallback is the channel the reply rides back out on, but a channel’s codec is the contract its requests satisfy, and a reply is by definition a different message — a codec bound to validate a transfer request has no declaration for the acceptance that answers it, so inferring it would reject correct replies.
A failure is refused rather than sent (SUTRA.OUTBOUND.VALIDATION_REJECT), and the diagnostic
names the offending field. The default is fail-closed for the same reason
q:onValidation’s is: a flow that declared no
posture toward a validation failure has said nothing about handling one, so the invalid message
must not leave. Egress bound to a bare format, or to no codec, asserts no schema and is
unaffected — there is nothing there to fail.
Where the refusal lands depends on where the step runs. On a request the caller is driving, it
comes back to that caller as the error, and nothing is persisted. On a resumed step (the tail
after a <q:reply continue="true">, a timer, a <q:retry> re-drive, a relayed message), there
may be no caller left to tell, so the instance fails instead. It is marked FAILED with this code
(the failureCode on GET /sutra/instances/{id}), and its waits and timers are resolved so
nothing re-runs.
sutra lint moves what it can to package time. It reads a template’s own unrendered text with
the governing codec and compares the answer against template-manifest.yaml — outputMessageType,
and outputAttributes for the codec-reported attributes. That works because a template is text
with holes and the holes are leaf content: the element structure deciding classification is already
there. It catches a template that renders the wrong message while declaring the right one. A
template the codec cannot classify unrendered is reported unverifiable rather than wrong; a
stated codec that does not resolve is an error, because stating it is how you ask for the check.
q:alias — correlation by your business key
<q:alias name="e2eId" expression="payload.E2EId" unique="true" onConflict="correlate"/>
A friendly key derived from a FEEL expression over the process variables — durable, and
re-evaluated on rehydration. unique="true" with onConflict="correlate" is what lets a later
message on a different channel find and resume the exact parked instance it belongs to, by a key
you named (an EndToEndId, an order reference) rather than an engine-internal id. See
Wait states and human tasks. A correlation alias’s value also appears in an acceptance
receipt, masked if sensitive; see Acceptance receipts.
q:receipt — what a 202 names
An HTTP channel with ack-mode: on-persist answers 202 Accepted with a receipt the engine writes
itself, so a caller learns which instance it started without the flow doing anything:
| Field | What it is |
|---|---|
instanceId | The instance started or resumed: the handle for GET /sutra/instances/{id}. |
deploymentId, labels | The deployment serving the channel, and its tenant, module, version. |
aliases | The instance’s <q:alias> values at the moment of the answer, masked by the rules below. |
idempotencyKey | The key the caller supplied, or the one a <q:source dedupKey> derived. Absent otherwise: the body hash used in its place recognises no retry. |
duplicate, originalInstanceId | This delivery was already seen; the instance the first one started. |
attributes | The values the flow chose to expose, below. |
It travels in two headers: Sutra-Instance-Id and Sutra-Receipt (the whole receipt as one line of
ASCII JSON, cut down to its identifying fields with "truncated": true if it would pass 4 KiB). A
flow that replied nothing also gets it as the application/json body; a body the flow authored with
<q:reply continue="true"> is left untouched. Other channels answer as before, except that a duplicate on
any HTTP channel gets this 202. Queue transports have no synchronous answer, so no
receipt.
<q:receipt> adds values of the flow’s choosing. It hangs off the process’s own
extensionElements, and without it the flow exposes nothing:
<bpmn:process id="case-open">
<bpmn:extensionElements>
<q:receipt>
<q:attribute name="caseId" expression="req.caseId"/>
</q:receipt>
</bpmn:extensionElements>
When it is evaluated. At the answer, over the variables a <q:reply continue="true"> renders
against: everything the intake bound, plus what the flow computed before replying or parking.
What is masked. An expression reading a sensitive variable shows ***REDACTED***, an alias
included. One reading a redactor-controlled variable shows that path of its masked projection, or the
mask if it is not a plain path. While a redactor is active, anything under event other than
channel, contentType, dedupKey, idempotencyKey and receivedAt is masked too, since it
repeats the payload. A null is left out, and an expression that fails to evaluate is left out and
logged: the work is already accepted and does not fail over its receipt. Values are strings.
sutra lint refuses an expression that does not parse and a repeated name
(SUTRA.CONFIG.RECEIPT.ATTRIBUTE_DUPLICATE), and warns about an attribute reading a sensitive
variable (SUTRA.CONFIG.RECEIPT.ATTRIBUTE_SENSITIVE), since the caller would only ever see the mask.
A duplicate’s receipt names the original instance, and carries no aliases or attributes: nothing ran to produce them. It names no instance when the first delivery ended without one or was recorded before 0.6.18.
A caller that lost its 202 can ask GET /sutra/instances?alias=<name>:<value>. See
Acceptance receipts.
q:decode — read held text as a message
event.body is the message as text, so a flow can keep it in a typed store column with an ordinary
assignment. <q:decode> is the other half: it parses text the flow holds with a registered codec and
binds the result. It hangs off a data task, and anywhere else is a parse error
(SUTRA.PARSE.Q_DECODE.NOT_ON_DATA_TASK). The walkthrough is
Keeping a message as text.
<bpmn:serviceTask id="Load">
<bpmn:extensionElements>
<q:decode name="message" expression="row.body" codec="json" contentType="row.contentType">
<q:redactors><q:redactor ref="card"/></q:redactors> <!-- optional -->
</q:decode>
</bpmn:extensionElements>
<bpmn:dataInputAssociation>
<bpmn:sourceRef>refRead</bpmn:sourceRef><bpmn:targetRef>row</bpmn:targetRef>
</bpmn:dataInputAssociation>
</bpmn:serviceTask>
| Attribute | Required | What it is |
|---|---|---|
name | yes | The variable the decoded value is bound to. |
expression | yes | FEEL giving a string. |
codec | yes | A built-in format name or a package schema codec URN, as on a channel. |
contentType | no | FEEL. Picks the parser within a format’s contract (json, xml and yaml are one contract), and is passed to the codec. |
Order. A data task runs its store reads, its decodes in document order, its assignments, then its store writes.
The instance’s own codec. It resolves in the instance’s pinned deployment, so a redeploy does not
change how an old instance decodes. Validators and <q:onValidation> do not run: this is not intake.
A codec’s non-fatal issues do not stop the step.
Failures can be caught with an error boundary or an event sub-process. Uncaught, they fail the instance. Their text names issue codes and paths only, never the input.
| Code | When |
|---|---|
SUTRA.RUNTIME.DECODE.NOT_TEXT | The expression gave null or something other than a string. A timer start has no event, so event.body there gives this. |
SUTRA.RUNTIME.DECODE.FAILED | The codec refused the text or its content type, or the codec or a redactor isn’t registered. |
SUTRA.RUNTIME.DECODE.TOO_LARGE | The text is over the engine’s sutra.codec.max-payload-bytes (10 MiB by default; 0 there still leaves that default here). Nothing was decoded. |
Text only. A valid UTF-8 body round-trips exactly. Otherwise invalid bytes become U+FFFD in
event.body, and event.bodyLossy is true, so a flow can route the message away; a clean message
has no such key. There is no binary form. A body whose prolog declares another encoding, such as
ISO-8859-1 XML, cannot be kept faithfully. A NUL byte survives the conversion but PostgreSQL TEXT
refuses it, and that store write fails the instance.
Masking. A <q:redactors> child writes a masked projection to the <name>.redacted companion,
and a crashed redactor masks the whole value. Otherwise declare the target sensitive. A store column
has no at-rest encryption, and sensitive does not reach it. In a package where a process masks its
source, sutra lint:
| Code | Severity | When |
|---|---|---|
SUTRA.CONFIG.BPMN.RAW_BODY_UNMASKED | warning, an error in a later release | A variable holds raw event.body text and isn’t sensitive. This covers a copy, and a variable read from a store that a masking process wrote it to, even in a process with no masking of its own. |
SUTRA.CONFIG.BPMN.RAW_BODY_STORED | warning | Raw event.body text is written to a store. |
SUTRA.CONFIG.DECODE.REDACTORS_MISSING | error | A <q:decode> reads such text, and its target has neither <q:redactors> nor sensitive. |
The check follows a store by name within one package; another package reading the same database isn’t
seen, and <q:alias> and <q:param> don’t count as copies. It also refuses a codec or redactor that
doesn’t resolve (SUTRA.CONFIG.DECODE.CODEC_UNKNOWN, SUTRA.CONFIG.DECODE.REDACTOR_UNKNOWN), and a
step missing an attribute fails at parse (SUTRA.PARSE.Q_DECODE.INCOMPLETE).
After a wait. A decode reads variables and store rows, never a delivery, so a resume or a retry
decodes the same text. But a resuming message replaces event, and so does a synchronous channel-call
reply unless the task declares an output mapping. So capture on the step right after the intake you
mean. A call activity sees its caller’s event.
Cost. The decoded target and any capture variable are persisted at a park like any other, so a
large message costs two or three times its size per park unless declared transient. A flow that
doesn’t use <q:decode> pays nothing.
q:retry — per-task retry policies
<bpmn:serviceTask id="Score" implementation="registered:score">
<bpmn:extensionElements>
<q:retry maxAttempts="3" initialDelay="PT1S" backoffCoefficient="2.0"
maxDelay="PT5M" nonRetryableCodes="SUTRA.TASK.VALIDATION"/>
</bpmn:extensionElements>
</bpmn:serviceTask>
Valid on both kinds of <bpmn:serviceTask> — a registered task and a channel-call task
(implementation="channel:<name>"). maxAttempts is the total invocation budget including the
first attempt; attempt n+1 waits min(initialDelay × backoffCoefficient^(n-1), maxDelay);
nonRetryableCodes names structured codes that fail immediately regardless of budget.
The load-bearing property: a retry wait is a durable timer park, never a sleep. The instance persists with the failed task still pending and an armed timer at the backoff instant, so the backoff survives restarts and hot-deploys and blocks no execution lane.
What counts as a failed attempt is deliberately narrow. On a registered task: an uncaught error
from the task function. On a channel-call task, exactly two things — the route-less
<q:timeout> boundary firing, and the request delivery being terminally poisoned by the outbox
attempt ceiling. A correlated business response is never a retry trigger (the counterpart
answered; re-sending would double-submit), and BPMN errors always route to their boundaries
instead. Because a modelled outcome always beats a policy, a channel-call <q:retry> requires the
route-less <q:timeout> form — a timer boundary with drawn outgoing flows alongside a retry
policy is a load error.
@initialDelay and @maxDelay may be FEEL expressions instead of literals, for example
initialDelay="config.retryDelay". They are evaluated each time a backoff is armed.
Full treatment, including the re-drive’s fresh-idempotency-key contract and what exhaustion does: Retries, history, and schedules.
q:timeout — a deadline on a channel call
<bpmn:serviceTask id="Score" implementation="channel:score-request">
<bpmn:extensionElements>
<q:timeout duration="PT2M"/>
</bpmn:extensionElements>
</bpmn:serviceTask>
Synthesizes a timer boundary on the call, so a counterpart that never answers cannot park the
instance forever. Without a <q:retry> alongside it, a fired route-less timeout is a catchable
BPMN error; with one, it is a retryable task failure first.
@duration may also be a FEEL expression, such as duration="request.budget" or
duration="config.callTimeout". It is evaluated when the call parks. See
Timer intervals from FEEL.
q:stream — stream a channel-call answer
q:stream is an optional boolean attribute (not an element) on a bpmn:serviceTask. Writing
q:stream="true" on a channel-call task (implementation="channel:<name>") marks it as one whose
synchronous model answer should be streamed to the original caller, while the complete answer is
still accumulated and resumes the flow as an ordinary response. It defaults to false, and omitting
it changes nothing. Declaring it true on a task whose implementation is not a channel call fails
the load with SUTRA.CONFIG.BPMN.STREAM.NOT_CHANNEL_CALL; a value that is not true/false fails
with SUTRA.CONFIG.BPMN.STREAM.INVALID. The attribute is a backwards-compatible addition to the
frozen urn:sutra:q:1.0 namespace. See Calling a language model.
q:store — durable, cross-instance data
<bpmn:dataStoreReference id="dsrFrom" name="accounts" dataStoreRef="accountsStore">
<bpmn:extensionElements>
<q:store key="payload.fromId" forUpdate="true"/>
</bpmn:extensionElements>
</bpmn:dataStoreReference>
Binds a <bpmn:dataStoreReference> to a key in a datastores.yaml-declared store. forUpdate
takes a pessimistic row lock (serializes concurrent writers on the same key); field replaces one
field of a stored map value; expect="unchanged" is an optimistic compare-and-set instead of a
lock; expect="absent" (0.6.10) writes only when the key does not exist yet — a claim exactly one
replica wins. A lost compare-and-set raises SUTRA.RUNTIME.DATASTORE.CONFLICT, which an error
boundary on the data task can catch.
On a write, delete="true" (0.7.0) removes the key’s row instead of writing a value. A missing
key is a no-op, and expect="unchanged" makes the delete conditional on the revision the instance
read. See Deleting a row.
On a read, query (0.6.10) turns the binding into a key query: every row whose key starts with
key (query="prefix") or lies in [key, to) (query="range"), keys compared bytewise, in key
order, limit per page (default 1000, maximum 10 000) after the key after; rows="false"
returns aggregates only. Each child <q:aggregate fn="count|sum" field="…" groupBy="…"/> is
computed over every matching row:
<q:store key="payload.meter + '|' + payload.day + '|'" query="prefix" limit="500" after="payload.after">
<q:aggregate fn="sum" field="kwh"/>
<q:aggregate fn="sum" field="kwh" groupBy="kind"/>
</q:store>
The target variable receives {rows: [{key, value}], aggregates: [...], truncated, next}. See
Querying rows by key for the result shape, the number rules
and the index the store’s table needs.
q:dispatch / q:case — a routing table instead of gateway sprawl
<q:dispatch default="fallback" onNoMatch="error">
<q:case when="payload.type = 'A'" calledElement="handle-a"/>
<q:case when="payload.type = 'B'" calledElement="handle-b"/>
</q:dispatch>
A declarative dynamic call-activity dispatch — one FEEL condition per case, instead of an exclusive gateway fanning into N call activities.
q:onValidation — the structural-failure policy
<q:onValidation mode="route" errorCode="T505"/>
mode="route" surfaces the validation summary (outcome, tier, firstReasonCode,
firstIssue, issues) as variables so the BPMN’s own gateway decides what to do; reject /
error short-circuit. The engine never interprets what
a soft error means — that decision stays in your process, which is what keeps the engine
domain-neutral.
The three modes are not interchangeable:
| Mode | What happens |
|---|---|
route | the payload reaches the flow; you triage payload.validation.issues yourself |
reject | the message is refused at intake; no instance starts; the caller gets the issue list |
error | a BPMN error is raised into the flow, for an error boundary or event sub-process to catch |
Declaring nothing means reject. An intake that carries a validation contract — a
schema-backed codec, or a declared <q:validators> chain — and states no policy refuses a failing
payload rather than passing it on. A flow that says nothing about handling failure has no handler
for it by definition, so the failure must not enter it.
That default is reject rather than error deliberately: error is still the flow taking
control, and raising a BPMN error into a process with no matching boundary reports “uncaught BPMN
error” instead of naming the offending field — a worse diagnostic for the same refusal.
Schema-less ingress is untouched. A channel binding a bare format (csv, json, …) has no
contract to fail, and the format layer is fail-open by construction, so nothing changes there.
If you want the payload through regardless, say so — <q:onValidation mode="route"/> — and the
intent is visible in the diagram instead of resting on a default. sutra lint warns
(SUTRA.CONFIG.VALIDATION.POSTURE_UNDECLARED) on any contract-bearing intake that declares no
policy, naming both choices.
q:process — the retry-safety assertion
<bpmn:extensionElements><q:process idempotent="true"/></bpmn:extensionElements>
Hung off the <bpmn:process> itself. idempotent="true" is your assertion that re-running this
process on the same input converges to one end state — so a redelivered message is safe to
re-process any number of times. The default is false (fail-closed): an execution failure on a
non-idempotent process is consumed (ack, no requeue) and recorded as an incident rather than
blind-retried. This is a different concern from q:source’s dedupKey, which only detects that a
message is a redelivery — it says nothing about whether re-processing it is safe.
What the assertion gates, and what it does not. It governs the failure path: whether a
delivery that failed is requeued or dead-lettered. It does not gate replay after a park. An
instance that parks — at a wait node, a <q:retry> backoff, or a <q:reply continue="true"> — is
re-driven at-least-once by the durable poller, unconditionally. A multi-instance loop is one
durable step, so a crash mid-loop replays it from the first item rather than resuming where it
stopped.
Both rest on the same property: every effect the flow repeats must converge. A store write keyed
by something derived from the record does (re-writing is an upsert). A <q:send>, an append, or a
counter increment does not — and no assertion makes it so. Declaring idempotent="true" over
effects that do not converge does not make replay safe; it only changes what happens after a
failure.
q:variables, q:audit, q:coverage
q:variables— declares process variables up front (name + scalar type or a schema reference), so deploy-time static validation can check every FEEL path against them, and mark a variabletransient(never persisted, so it is dropped at any park — reading it after a wait state, a<q:retry>backoff or a<q:reply continue="true">is a validation error; it is for scratch that is re-derived, never for anything the flow still needs on the other side),sensitive(persisted but redacted downstream), orsource-bound (payload-initialized from a named channel).q:audit— per-process audit sink/target/capture-level configuration.captureisnone,metadata,payload(the default) or, since 0.6,media: metadata-level events that carry the media references found among the variables, never the media (see Recordings and media references).q:coverage— declares one tracked compliance route (pathid + the orderedflowsit must traverse) for the path-coverage reporting the CLI’ssutra coveragecommands drive. This is the intra-process half of the mechanism; a route spanning several correlated processes is declared differently. See Coverage: declared routes as the compliance signal for both shapes, the curation guidance, and the CLI walkthrough.
Next
- Rules: DMN, FEEL, and .srl — the decision layer these expressions run on.
- Retries, history, and schedules —
q:retryandq:timeoutend to end, plus timer definitions and execution history. - Worked example: money-transfer — most of the elements on this page, in one real process.
Rules: DMN, FEEL, and .srl
A businessRuleTask binds one decision, authored either as a .dmn decision table or as a
.srl ruleset. Both compile onto the same FEEL evaluator (sutra-feel) — no JVM, no Rete
runtime, and no second expression language to learn.
flowchart LR
BRT["bpmn:businessRuleTask"] --> R{"routed by<br/>file extension"}
R -->|".dmn"| D["DMN 1.5 decision table<br/>all seven hit policies"]
R -->|".srl"| S["ruleset: rule / when / then<br/>one deterministic pass"]
D --> F["sutra-feel — one evaluator<br/>DECIMAL64 · determinism denylist"]
S --> F
Choosing a table or a ruleset is a choice of framing, not of runtime — an expression means the same
thing in a gateway condition, a q:alias, a DMN cell, and an .srl when.
FEEL — the shared expression language
FEEL (Friendly Enough Expression Language) is what every condition, gateway expression, q:alias
key, q:store key, and rule body is written in. Sutra’s implementation is a from-scratch Rust
evaluator (lexer, parser, AST, a determinism denylist that forbids non-reproducible constructs on
a replay path, and DECIMAL64 numeric semantics — 16 significant digits, HALF_EVEN rounding).
Try an expression standalone, before it’s anywhere near a process:
sutra explain 'payload.amount > 100'
sutra explain --context vars.txt 'fromAccount.balance - payload.amount'
sutra explain is a one-shot evaluator or a REPL (omit the expression) — useful for debugging a
gateway condition or a q:alias expression without deploying anything. See
Crates: sutra-feel and sutra-dmn if you want FEEL evaluation embedded in
your own Rust project, independent of the engine.
Durations with fractional seconds
Since 0.8.0. A days-and-time duration holds its seconds to the nanosecond (digits past the ninth
are cut off), through parsing, string(), comparison, arithmetic with dates, times and other
durations, multiplication and division by a number, and the seconds accessor. Seconds take XML
Schema’s decimal forms (PT0.5S, PT.5S, PT0.S). From sutra explain:
$ sutra explain 'duration("PT0.001S")'
duration("PT0.001S") => PT0.001S
$ sutra explain 'duration("PT4.5S").seconds'
duration("PT4.5S").seconds => 4.5
$ sutra explain 'string(duration("P1DT2H3M4.123456789S"))'
string(duration("P1DT2H3M4.123456789S")) => P1DT2H3M4.123456789S
$ sutra explain 'duration("PT1.5S") * 3'
duration("PT1.5S") * 3 => PT4.5S
$ sutra explain 'date and time("2026-11-19T10:00:00Z") + duration("PT0.250S")'
date and time("2026-11-19T10:00:00Z") + duration("PT0.250S") => 2026-11-19T10:00:00.25Z
Only seconds take a fraction. Until 0.7.0 the parser dropped everything after the decimal point, so
duration("PT0.001S") was zero and duration("PT1.5H") was one hour; now the second is refused:
$ sutra explain 'duration("PT1.5H")'
[ERROR] SUTRA.FEEL.COMPILE.TYPE_MISMATCH — duration() cannot parse the string (feel:inline:1:1)
Write PT1H30M instead. A timer whose FEEL expression
returns duration("PT0.001S") now waits 1 ms instead of firing at once.
Password hashing
Two builtins let a flow check a password against a stored hash without ever storing the password (new in 0.6.21):
| Function | Returns |
|---|---|
password hash(password) | An argon2id hash in PHC string form, $argon2id$v=19$m=19456,t=2,p=1$<salt>$<hash>, with a fresh 16-byte salt from the operating system. null if the argument is not a string or is empty. |
password verify(password, hash) | true if the password matches the hash, false if it does not. null if either argument is not a string, or if the hash is malformed, is not argon2id, or asks for more than m = 65 536 KiB, t = 10 or p = 4. The comparison is constant-time. |
- The parameters are fixed in code: argon2id, m = 19 456 KiB, t = 2, p = 1, the OWASP minimum.
A package can’t change them.
password verifyreads the parameters from the stored hash, so the cap keeps a planted hash from tying up a worker and its memory. password hashis denied at replay-bound sites, likeuuid(), because it draws a random salt. Use it in a task, not in aq:aliasor awhen;sutra lintrefuses it there withSUTRA.FEEL.DETERMINISM.UNSAFE_BUILTIN.password verifyis deterministic and works anywhere.- Cost. One verify takes about 22 ms (criterion median 22.3 ms on an Intel i7-10750H laptop, release build), and holds its executor worker for that time. That suits a sign-in; don’t call these functions for every message.
Signing a user in against a store builds a sign-in flow with them and checks that the password reaches no table, audit record or log line.
DMN decision tables
A .dmn file under rules/ is a standard DMN 1.5 decision table; the engine’s evaluator supports
all seven hit policies. A businessRuleTask names the decision, and its named output columns merge
back into the process’s instance variables:
<!-- examples/approval-hold/.../rules/approval-decide.dmn -->
<decision id="approvalDecide" name="Review decision from risk score">
<decisionTable hitPolicy="FIRST">
<input id="i_risk"><inputExpression typeRef="number"><text>riskScore</text></inputExpression></input>
<output id="o_decision" name="decision" typeRef="string"/>
<rule id="r_approve">
<inputEntry><text>< 50</text></inputEntry>
<outputEntry><text>"approve"</text></outputEntry>
</rule>
<rule id="r_review">
<inputEntry><text>-</text></inputEntry>
<outputEntry><text>"review"</text></outputEntry>
</rule>
</decisionTable>
</decision>
Sutra’s DMN conformance is measured against the real OMG DMN-TCK, not asserted — see DMN-TCK conformance for the numbers and what they cover.
.srl — a Drools-inspired rule DSL over FEEL
.srl is Sutra’s own rule language: rule / when / then / end framing around FEEL conditions and
a small, closed set of side-effecting actions. It targets the same use case DMN’s COLLECT hit
policy does — several independent business-validation rules over one payload — in a syntax closer
to a Drools ruleset than a decision table.
rule "currency-not-usd"
when
exists(payload.amount.currency) and payload.amount.currency != "USD"
then
report(
"SUTRA.VALIDATE.CURRENCY_NOT_USD",
"payload.amount.currency",
"Currency must be USD; got " + payload.amount.currency
);
end
Grammar, in full:
ruleset := rule*
rule := "rule" STRING attr* "when" <condition> "then" action* "end"
attr := "salience" INTEGER | "activation-group" STRING
action := verb ";"
verb := "report" "(" <feel_expr> "," <feel_expr> "," <feel_expr> ")"
| "set" "(" IDENT "," <feel_expr> ")"
- Every condition and action argument is an embedded FEEL expression —
.srladds only the rule framing on top. - Two action verbs today:
set(target, expr)binds a value into the working context (visible to later rules) and the output map;report(code, path, message)appends a structured issue.insert/retractare reserved for a future stateful phase and are a clean parse error, not silently accepted. - Evaluation is a single deterministic forward pass — a stable-sorted agenda by
(-salience, declaration order), each rule firing at most once,activation-groupgiving first-match-wins semantics within a group. This is sequential-agenda, not a Rete network. - Fail-closed: a parse error, or a condition/action that errors at evaluation, is a hard error — never a silently-skipped rule.
- A FEEL
if / then / elseused inside awhencondition must be parenthesised (when (if a then b else c) …), because the bare keywordthenthat ends the condition is matched at paren depth 0.
.srl and .dmn both live under a package’s rules/ folder (see
Deployment packages) and are routed by file extension — no separate
configuration names one or the other.
Composing a validator chain
A <q:source>’s <q:validators> list (see
The q: namespace) can mix
<q:complexValidator> entries pointing at both .dmn and .srl files in one chain — nothing
requires a chain to stick to one rule engine, and nothing requires a business ruleset to live in a
single file.
- Declaration order is evaluation order. The chain runs top to bottom exactly as written in the BPMN.
- Every validator sees the same payload projection. Whatever the channel’s codec decoded is what each entry in the chain evaluates against — an earlier validator’s issues don’t change what a later one reads, only what accumulates alongside it.
- Issues accumulate, they never replace. Each entry’s reported issues append to the same
validation.*result;validation.outcomeandvalidation.tierare derived from the whole accumulated list, so a gateway keyed on them is indifferent to which file in the chain produced which code. validation.firstReasonCodefollows declaration order. When two entries both report an issue,firstReasonCodeis whichever fired first in chain order — this only affects which code a reply template surfaces, never whether a gateway routes to reject.
This is what lets you split one business ruleset across engines by what each rule needs, instead of by an arbitrary file boundary. A module with a mix of clock-dependent and stateless checks (an extension-crate workload, not one of this repository’s bundled examples) does exactly that:
<q:validators>
<q:complexValidator source="intake-timing.dmn"/>
<q:complexValidator source="intake-fields.srl"/>
</q:validators>
The DMN table keeps the rules that need a clock — a staleness window and a clock-skew/future-dated
window on some received-at timestamp — because the engine-injected evaluation clock is only
available as a DMN validator’s reserved now input. The .srl file carries the stateless field
checks (a required-currency check, a positive-amount check, a required-identifier check) that have
no temporal dependency at all and read more naturally as named rules than as columns on the
decision table. Both files reason over the identical payload projection the codec decoded, report
through the same SUTRA.VALIDATE.* code family, and their issues land in one accumulated
validation.issues list — the split between engines is invisible to the BPMN gateway that routes
on the outcome.
flowchart LR
P["the payload projection<br/>the codec decoded"] --> V1["1 · intake-timing.dmn"]
P --> V2["2 · intake-fields.srl"]
V1 -->|"issues append"| ACC["one accumulated result<br/>validation.issues · outcome · tier"]
V2 -->|"issues append"| ACC
ACC --> G{"the BPMN gateway<br/>routes on the outcome"}
Both entries read the identical projection and append into one result, so the gateway cannot tell
which file — or which rule engine — produced a given code; only firstReasonCode carries the
declared order.
Which to reach for
- DMN if the logic is naturally tabular (a rate card, an approval matrix, one output per row) or you want a diagram a business analyst can review directly.
.srlif the logic is a set of independent validation rules over a payload, where a Drools-stylewhen/thenreads more naturally than a table.- Both, in one chain, when a ruleset is naturally mixed — some rules need something only a DMN
validator gets (the injected evaluation clock), others are better named individually than
columned into a table. See Composing a validator chain above for
exactly how a
.dmnand a.srlentry combine, illustrated above with a clock-dependent-vs- stateless split.
Next
- Data stores — durable state a rule or task reads/writes.
- Worked example: money-transfer — FEEL data-assignment nodes in a real transaction flow.
Data stores
A data object (<bpmn:dataObject>) is a per-instance variable — private, transient, born and
dying with one instance. A data store (<bpmn:dataStore> / <bpmn:dataStoreReference>) is the
opposite axis: durable, keyed state that outlives and is shared across instances — an account
ledger, a customer index, the covered/uncovered flags behind a compliance route.
Status note. Data stores are exercised end to end by the money-transfer example — a real
sql-backed ledger under transactional, pessimistic-lock control. (Path coverage rides a data store of an unusual kind: its marks are persisted in a store the package declares — the reservedcoveragename, whose connection picks the database they land in — but the engine owns that store’s schema and applies it itself, so it is neither a key→value business store nor one you write DDL for. See Coverage: declared routes as the compliance signal.) Some of the wider surface described in the design record — afile-backed store, an optimistic-concurrency compare-and-set on every provider — is still in motion; treat anything below not shown against the money-transfer example as directionally accurate rather than a finished contract, and checkrust/crates/sutra-datastorefor the current provider set before depending on specifics.
Choosing a store shape
A business store has two shapes. The choice between them is narrower than the volume of mechanism
the rest of this chapter spends on the second one suggests, so it is worth settling first. (The
reserved coverage store is neither — the engine owns its schema and applies it, so a package
writes no coverage DDL and makes no shape choice at all; see
Coverage.)
Key/value is the general case, and it is always sufficient. By default a store is a key→value
store: one row per key, the value serialized whole as JSON text. Any structure fits it — nested,
repeated, open-ended, a different shape next release — because nothing about the value’s shape is
written down anywhere for it to disagree with. The table behind it is one fixed generic shape
(store name, key, value, rev, updated_at) shared by every key/value store on that connection,
and it does not change when your data does. From a standards point of view this is the only shape
that carries every scenario, which is why it is the default and stays the default. A store with
no structure: block is this store, and none of the projection machinery below applies to it.
Projection is an integration affordance, not an upgrade. Declaring a structure: does not make
a store more correct, more durable or more capable, and it is not a tier you graduate to. It exists
for one reason: so that everything else that already speaks SQL can read the data. A BI tool, a
dashboard, a nightly reporting query, an analyst with read access to that database — none of them
know what Sutra is, and against a key/value store the most any of them can see is an opaque
document to parse. Against a projected table they see balance and opened_at as columns, with
types. That is one more way to ease integration, not a better kind of store.
The trade, stated plainly. Projection costs two things and buys one:
- It costs the flat-only constraint. The declared type must be entirely scalar, and a type that later grows a nested or repeated child stops projecting at all — see Flat only.
- It costs ownership of the DDL and its evolution. You write the
CREATE TABLE, and every change to the declared type is a migration you author and sequence. A key/value store needs a table too, but it is the same generic one every time — written once, never revisited. A projected table is your record’s own shape, so it has to move whenever that shape moves. - It buys direct SQL access to the data, for anything that speaks SQL, without going through the engine.
If nothing outside the process ever reads the store, that affordance buys nothing, and key/value is the right answer.
flowchart TD
W["one write through q:store"]
subgraph KV["Key/value — the default"]
KROW[("a shared generic table<br/>name · key · value · rev")]
end
subgraph PJ["Projected — a structure: block"]
PROW[("this store's own table<br/>one typed column per field")]
end
W --> KROW
W --> PROW
KROW -.->|"an opaque document<br/>to parse"| BI["any SQL tool that has<br/>never heard of Sutra"]
PROW -->|"balance, opened_at:<br/>columns, with types"| BI
Both shapes hold the same record and answer the same <q:store> access with the same locking and
rev semantics; the only thing projection moves is the arrow on the right — an opaque document
versus columns a reader can use directly, bought with the flat-only constraint and ownership of the
DDL.
The shape is a property of the store, not of the flow — within limits worth knowing. A store’s
shape is declared in datastores.yaml and nowhere else. The BPMN is unchanged either way: the same
<bpmn:dataStoreReference>, the same <q:store> with the same key / field / forUpdate /
expect attributes, resolved by store name through the same registry. Pessimistic locking,
<bpmn:transaction> enlistment and rollback, delete, and the rev bookkeeping behind
expect="unchanged" — seeded at 1, bumped on every write, a conflict when the revision moved —
carry the same semantics under both shapes. Adding or removing a structure: block is a
package-level change, not a process-level one.
What does change is what the store accepts and what it hands back — and that is not nothing:
- A projected store refuses a write carrying a field the structure does not declare
(
UNDECLARED_FIELD); a key/value store simply stores it. This is the difference most likely to bite, because a process that has quietly been writing a wider record than the declared type works against key/value and fails against projection — and nothing catches it beforehand: neithersutra lintnor the executor checks a<q:store field="…">name against the declared structure, so the first evidence is the refused write. - It refuses a value that isn’t a record at all — a scalar, an array, an explicit
null(VALUE_NOT_A_RECORD). A key/value store holds any of those happily. - It refuses every operation until the live table satisfies the declaration
(
PROJECTION_UNSATISFIABLE, checked on first use). A key/value store has no declaration to drift from, so it has no such failure mode. - A
field-narrowed write is a read-modify-write of the whole value under both shapes, so one that creates a key writes a record containing only that field. On a key/value store that is simply a one-field document; on a projected store every other declared column is boundNULL, so unless they are all nullable or defaulted in your DDL that insert runs straight into your own constraints. Make the first write of a key a full record rather than a narrowed one. - Values that pass through a typed column come back in the column’s canonical form rather than
the text that was written: an explicit
nulland an absent optional field become the same thing on read, and adateTimecomes back as your dialect’s rendering of that instant. The exact scope of what is and isn’t preserved is in A worked example.
None of those change how a process addresses the store; several of them change whether a given write succeeds. They are the checklist for a shape change, not a reason to avoid one.
Column names become a published interface. The moment a dashboard reads that table, its column
names are a contract with someone who has never seen your schema — a report knows account_balance,
not the XSD element it was derived from. Column names are derived by convention from the declared
field names, so renaming a field renames a column, and silently breaks that report. That is what
the columns: override exists for: pin the physical name, and the schema can be renamed underneath
it while the published surface holds still. It is worth setting before you need it rather than
after someone’s query starts returning nothing — the naming rules and the override syntax are under
Column names.
Declaring a store
One flat datastores.yaml per package (see Deployment packages). The
module owns its store — the connection is declared here, resolved from environment references,
and its migrations ship inside the package:
# examples/money-transfer/.../datastores.yaml (abridged)
datastores:
- name: accounts
type: sql
sql:
url-ref: env:ACCOUNTS_DB_URL
username-ref: env:ACCOUNTS_DB_USER
password-ref: env:ACCOUNTS_DB_PASSWORD
migrations: migrations/accounts # V<n>__*.sql, each run once per database
dataClass: financial # sensitivity tag — redacted in audit/logs/traces
A store’s migrations run the first time the store is used, under a database lock, so replicas
never run them at the same moment. The lock is one per database (per server on MySQL and MariaDB),
not one per store, because several stores often share one table and the same CREATE TABLE IF NOT EXISTS. Before 0.6.12 the lock was per store, and two stores sharing a table that were first used
at the same moment could fail that first request with HTTP 500
(pg_type_typname_nsp_index); the next request succeeded.
Migrations run once
The engine records every script it applies in a history table, sutra_store_migrations, created
in the store’s own database the first time a store there migrates. The table has one row per store
and version: store_name, version, script (the file name), checksum (the SHA-256 of the
script as it ran) and applied_at. On first use, under the lock described above, each script is
checked against that table:
- Recorded with the same checksum: skipped. A restart, a second replica or a redeploy of the same package runs nothing.
- Not recorded: it runs, then its row is written. On PostgreSQL and SQL Server the script and
its row commit in one transaction, so a script that fails leaves neither its changes nor a row
behind. MySQL and MariaDB commit DDL implicitly, so there the script runs first and its row is
written after it succeeds. Either way, a failed script records nothing and fails the store’s first
use; the next use tries again.
Because of that transaction, a statement PostgreSQL refuses inside one (
CREATE INDEX CONCURRENTLY, for example) cannot be in a migration script; create such an index outside the package’s migrations. - Recorded with a different checksum: the store refuses to start with
SUTRA.RUNTIME.DATASTORE.MIGRATION_CHECKSUM, naming the store, the file, the version and both checksums. A script that has been applied must not be edited. Restore it, and put the change in a new script with the next version number.
Rows for scripts the package no longer ships are ignored.
Versions and order. V<n>__<description>.sql has version <n> with leading zeros dropped, so
V001__accounts.sql and V1__accounts.sql are the same version, and two scripts of one store
with the same version are refused (SUTRA.CONFIG.DATASTORE.MIGRATION_VERSION_DUPLICATE). When
every script is named this way they run in numeric order, so V2 runs before V10. A script with
any other name still works: its version is its file name without .sql, and a folder that mixes
the two styles runs in plain file-name order. sutra lint requires the V<n>__ form.
Databases migrated before the history table existed. Releases before 0.6.13 re-ran every script on every boot and kept no record, which is why those scripts had to be idempotent. Nothing special is needed to move such a database over: it has no rows for the store, so each script runs one more time, harmlessly, and is recorded. From then on it runs once.
Placeholders. A script can use five placeholders, replaced before it runs and before its checksum is taken:
| Placeholder | Replaced with |
|---|---|
${package.tenant} | the package’s tenant label |
${package.module} | the package’s module label |
${package.version} | the package’s version label |
${store.name} | the store’s name in datastores.yaml |
${store.table} | the table the store writes: its sql.table (or its name) when it declares a structure:, otherwise data_store |
-- migrations/accounts/V002__seed.sql
INSERT INTO data_store (store_name, store_key, store_value)
VALUES ('${store.name}', 'settings', '{"owner":"${package.tenant}"}');
Any other ${…} is an error, and so is a ${ with no } on the same line. Both are reported by
sutra lint and again when the engine loads the package
(SUTRA.CONFIG.DATASTORE.MIGRATION_PLACEHOLDER, naming the file, the line and the placeholder).
Write $${ to get a literal ${. A lone $ or $$, such as PostgreSQL dollar quoting, is left
alone. Because ${package.version} is part of the checksummed text, a script that uses it counts
as edited when the version label changes. Use it only in scripts you add for a specific release.
The engine’s own datasource (instances, outbox, lease, audit, inbox) is never a store’s
backing connection — a sql store always resolves its own connection, and fails closed if it
declares none. This is what makes a generic engine image self-sufficient against any number of
modules that each bring their own store: no baked default datasource, no baked migrations.
A store declared this way is the key/value shape: one row per key, the value serialized whole
as JSON text, opaque to the database. Adding a structure: block projects a flat record onto real
typed columns instead — see
Typed columns: declaring the structure a store holds
below.
Referencing a store from BPMN
<bpmn:dataStore id="accountsStore" name="accounts"/> <!-- definitions scope -->
<bpmn:dataStoreReference id="dsrFrom" name="accounts" dataStoreRef="accountsStore">
<bpmn:extensionElements>
<q:store key="payload.fromId" forUpdate="true"/>
</bpmn:extensionElements>
</bpmn:dataStoreReference>
<q:store> (see The q: namespace) is what turns a bare reference into a keyed
access: key is a FEEL expression producing the row key, forUpdate="true" takes a pessimistic
row lock so concurrent writers on the same key serialize, field narrows a write to one field of a
stored map value, and expect="unchanged" is an optimistic compare-and-set alternative to a lock.
expect="absent", delete and query are covered in Claiming a key once,
Deleting a row and Querying rows by key.
A data task — a serviceTask with no implementation — reads and writes through its data
associations exactly like it would a data object, except the source/target is a store reference
instead of a variable.
Typed columns: declaring the structure a store holds
An opaque store hands the database a string. Nothing inside the value is queryable, indexable or typed, and the only way to read a field is to fetch the whole document by key and pick it apart in the process. For a record whose fields are all scalars that is a blob tax on data that could perfectly well be ordinary columns.
So a store may declare the structure it stores, not just its key:
datastores:
- name: accounts
type: sql
structure:
schema: urn:accounts # a schemas/<folder> codec of this package
type: AccountRecord # a complexType, a root element, or a JSON Schema definition
sql:
url-ref: env:ACCOUNTS_DB_URL
migrations: migrations/accounts
schema + type resolve through the schemas the package already compiles for its codecs (the
path-derived URN rule is in Deployment packages),
so there is no second schema source to keep in step and no shape inferred from observed data — the
declaration is the contract.
A store with no
structure:block behaves exactly as it always did. Same opaque key→JSON row, same providers, same<q:store>semantics, and none of the diagnostics below ever fire for it. Declaring a structure is opt-in per store; a record too nested to project simply doesn’t declare one.
Flat only
The declared type must be entirely scalar. Every child is classified in declared order:
| Declared shape | Becomes |
|---|---|
| Scalar leaf, at most once (a simple-type element or an attribute) | a column |
Scalar leaf, minOccurs="0" | a nullable column |
Scalar element inside a choice | a nullable column (at most one branch is populated) |
| A complex child — a nested object | lint error |
| Anything that may occur more than once | lint error |
Open content (xs:any, JSON Schema additionalProperties: true) | lint error |
The table is the declared fields. There is no residue column for the parts that didn’t fit, no column-versus-JSON merge on read, and no partial projection. That is the whole simplification, and it is what makes a projected row readable by anything that speaks SQL without knowing Sutra exists.
A type that can’t be expressed that way is rejected at package time with
SUTRA.CONFIG.DATASTORE.STRUCTURE_NOT_FLAT, naming the offending child:
data store 'accounts': declared structure type 'AccountRecord' is not flat: 'owner' is not a
scalar leaf (nested content). Flatten the type, or remove the 'structure' block and keep the
opaque store.
Those two remedies are the whole menu, deliberately:
- Flatten the type — pull the nested part up into scalar siblings (
owner/name→ownerName), or move it to a store of its own keyed by the same business id. - Drop the
structure:block — the store goes back to being the opaque key→JSON store it was, which is a perfectly good answer for a record that genuinely isn’t flat.
There is nothing in between, because anything in between would be a silent partial projection — some fields queryable, some hidden in a blob, and a merge rule nobody can see from the declaration.
Because the structure is closed and there is nowhere for an unexpected field to go, a write
carrying a field the structure does not declare is a fail-closed runtime error
(SUTRA.RUNTIME.DATASTORE.UNDECLARED_FIELD) naming the field. Silently dropping it is the one
outcome this design refuses. Writing anything that isn’t a record at all — a scalar, an array, an
explicit null for the whole value — is refused the same way, as
SUTRA.RUNTIME.DATASTORE.VALUE_NOT_A_RECORD: a projected row is its declared fields, so there is
nowhere for a bare value to land either.
Control columns
A projected table is never only the declared fields. Three columns belong to the engine, on every projected table, whether or not your own type ever mentions them:
| Column | Role |
|---|---|
store_key | the store key — whatever the store’s <q:store key="…"> expression produces — and the table’s PRIMARY KEY |
rev | the optimistic-concurrency revision, bumped on every write; what put_if_revision keys on |
updated_at | the write timestamp |
There is no store_name column — a projected table is one store, unlike the opaque table that
multiplexes many stores by name. The runtime binds and maintains all three itself, so a declared
field may not claim one of their names: that’s a naming collision
(SUTRA.CONFIG.DATASTORE.COLUMN_NAME_INVALID), resolved the same way as any other one — map the
field to a different column under columns:. Your own migration has to create all three alongside
your declared columns, or sutra lint raises COLUMN_MISSING naming whichever is absent — a
projected table missing store_key, rev or updated_at cannot be served no matter how correct
its declared columns are. The worked example below shows all three in the DDL.
Column names
A declared field’s column name is derived by convention: lowerCamel → snake_case, ASCII-folded,
runs of anything else collapsed to a single _.
| Declared field | Column |
|---|---|
accountId | account_id |
openedAt | opened_at |
iso4217Code | iso4217_code |
IBANCode | iban_code |
The convention is not always enough, and when it isn’t, sutra lint says so
(SUTRA.CONFIG.DATASTORE.COLUMN_NAME_INVALID) rather than guessing: two fields folding to the
same column, a fold that lands on a SQL reserved word, a name over the 63-character identifier cap
(PostgreSQL’s — the narrowest of the three shipped dialects, so a name that clears it is portable),
or a fold that isn’t a usable identifier at all. Every one of them is resolved by naming the column
yourself:
structure:
schema: urn:accounts
type: AccountRecord
columns:
openedAt: opened_on # the DDL already calls it this
order: order_seq # `order` is reserved
The mapping exists for two reasons. The first is that you write the DDL and may already have column names you can’t change. The second is that it pins the physical name against a later schema rename — which matters as soon as anything outside the engine reads the table, because at that point the column names are a published interface and the convention is no longer yours alone to change (see Choosing a store shape). The mapping is checked by the same rules — an override that is itself reserved or over-length is still an error, and an override naming a field the type doesn’t declare is an error rather than a silent no-op.
You own the DDL; the engine verifies it
The engine generates no DDL for a projected store and owns no part of its table shape. That table
is created by the store’s own migrations, in your own dialect, shipped inside the
package under
migrations/<store>/ (see Deployment packages).
sutra lint derives the effective table shape statically — from the package’s own
migrations/<store>/V*.sql, replayed in version order (CREATE TABLE plus
ALTER TABLE ADD/ALTER/DROP COLUMN), with no database connection and no credentials — and
compares it against the projection. The table it compares against is the one named after the store
— or, if the migrations create exactly one table, that one — or the one you name explicitly:
sql:
url-ref: env:ACCOUNTS_DB_URL
migrations: migrations/accounts
table: account_ledger # when the table isn't named after the store
What it checks the column types against is this mapping. It is advisory — the shape lint expects to find, not DDL anything emits:
| Declared | PostgreSQL | MySQL / MariaDB | SQL Server |
|---|---|---|---|
xs:string + maxLength n | VARCHAR(n) | VARCHAR(n) | NVARCHAR(n) |
xs:string, unbounded | TEXT | LONGTEXT | NVARCHAR(MAX) |
xs:decimal + totalDigits p / fractionDigits s | NUMERIC(p,s) | DECIMAL(p,s) | DECIMAL(p,s) |
xs:int / xs:long / xs:short | INTEGER / BIGINT / SMALLINT | same | INT / BIGINT / SMALLINT |
xs:boolean | BOOLEAN | TINYINT(1) | BIT |
xs:date / xs:dateTime / xs:time | DATE / TIMESTAMPTZ / TIME | DATE / DATETIME / TIME | DATE / DATETIME2 / TIME |
xs:base64Binary | BYTEA | BLOB | VARBINARY(MAX) |
an enumeration facet | the base type (a CHECK or a lookup table is your choice) | same | same |
Two declared shapes this release refuses outright — not silently degraded, not weakly verified — and one honest limit on verification alone:
-
xs:base64Binaryis refused, not projected. The advisory mapping above (BYTEA/BLOB/VARBINARY(MAX)) is what the column would need to be; the runtime doesn’t marshal binary values through a projected column yet. A store declaring one is refused when the engine resolves it — at deploy time, not at lint time — naming the field:data store 'accounts' cannot project structure type 'AccountRecord': field 'attachment' is declared 'base64Binary', which a projected column does not carry in this release — declare it as a string, or remove the 'structure' block and keep the opaque store.Lint doesn’t catch this on its own: its DDL parser treats
BYTEA/BLOB/VARBINARYas an ordinary column type and matches it happily, so a package can lint clean and still refuse to deploy. Don’t read lint’s silence as proof a binary field works. -
A JSON-Schema-declared structure is refused, not weakly verified. Only a package
schemas/<folder>XSD codec carries the enumerable, facet-bearing field list a typed column is checked against — a JSON Schema definition has no such list at this phase.sutra lintreports that honestly as aDDL_UNVERIFIABLEwarning (unprovable, not necessarily wrong) — but the engine that actually resolves the store at deploy time draws a harder line and refuses to load the deployment:data store 'accounts' declares structure schema 'urn:accounts', which this package provides no XSD codec for. A projected store's type must be declared by a schemas/<folder> XSD codec of the same package — only XSD carries the declared facets a typed column is checked against.So a lint-clean package with a JSON-Schema
structure:still will not deploy. Declare the structure against an XSD if you want the store to exist at all, not just to have its column types checked. -
DDL outside the parsed subset degrades to a warning, never to a false error. A migration that creates the table in a PL/pgSQL block, a T-SQL procedural guard, a table created outside the package entirely — lint reports that it could not derive the shape and raises no column diagnostic for that store. A linter that cries wolf on legitimate DDL is worse than no linter, because authors learn to ignore it. See Troubleshooting for how to read that warning.
Lint proves the package-time case. The deployed table is checked separately, but not on every
operation: first use of a projected store rides the same once-per-store-instance gate that runs
its migrations (the one already serialized across replicas by an advisory lock), and there the
provider reads the live table’s actual columns and fails the store closed if the projection
isn’t satisfiable, naming every offending column at once. That is the defence against a table that
drifted from the package’s own migrations — a hand-applied ALTER — and it fails loudly because a
silent partial write is far worse than a refusal. It costs one catalog round-trip the first time
the store is used, and nothing on any operation after that.
flowchart TD
L["<b>lint</b> — package time<br/>replays your V*.sql, no database"]
L -.->|"COLUMN_MISSING<br/>KEY_MISMATCH"| LE["package rejected"]
L --> D["<b>deploy</b> — the store resolves"]
D -.->|"a field no codec<br/>of this package declares"| DE["deployment refused"]
D --> F["<b>first use</b> — read the<br/>live table's columns, once"]
F -.->|"the table does not<br/>satisfy the declaration"| FE["PROJECTION_UNSATISFIABLE<br/>fails closed, names every column"]
F --> R["<b>every operation after</b><br/>no further catalog round-trip"]
R -.->|"undeclared field,<br/>or not a record"| RE["UNDECLARED_FIELD<br/>VALUE_NOT_A_RECORD"]
Each gate sees something the one before it could not — lint has the migrations but no database, deploy has the resolved store but no table, first use has the live catalog — which is why a package can pass lint (or draw only a warning from it) and still refuse to deploy.
A worked example
A flat record, declared in the package’s own codec schema
(schemas/accounts/accounts.xsd, the codec urn:accounts):
<xs:element name="AccountRecord">
<xs:complexType>
<xs:sequence>
<xs:element name="accountId" type="AccountId"/>
<xs:element name="balance" type="Money"/>
<xs:element name="openedAt" type="xs:date"/>
<xs:element name="active" type="xs:boolean"/>
<xs:element name="note" type="Note" minOccurs="0"/>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:simpleType name="AccountId">
<xs:restriction base="xs:string"><xs:maxLength value="35"/></xs:restriction>
</xs:simpleType>
<xs:simpleType name="Money">
<xs:restriction base="xs:decimal">
<xs:totalDigits value="18"/><xs:fractionDigits value="2"/>
</xs:restriction>
</xs:simpleType>
<xs:simpleType name="Note">
<xs:restriction base="xs:string"><xs:maxLength value="140"/></xs:restriction>
</xs:simpleType>
Its table, written by hand, in the dialect the store actually runs on
(migrations/accounts/V001__accounts.sql) — store_key, rev and updated_at are the engine’s
control columns, alongside one column per declared field:
CREATE TABLE IF NOT EXISTS accounts (
store_key VARCHAR(512) NOT NULL,
account_id VARCHAR(35) NOT NULL,
balance NUMERIC(18,2) NOT NULL DEFAULT 0,
opened_at DATE NOT NULL,
active BOOLEAN NOT NULL DEFAULT TRUE,
note VARCHAR(140), -- the optional field's column admits NULL
rev BIGINT NOT NULL DEFAULT 1,
updated_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (store_key) -- the store key, never a declared field
);
CREATE INDEX IF NOT EXISTS accounts_opened_at ON accounts (opened_at);
And the declaration that ties them together — no columns: block, because every field folds to the
name the DDL already uses:
datastores:
- name: accounts
type: sql
structure:
schema: urn:accounts
type: AccountRecord
sql:
url-ref: env:ACCOUNTS_DB_URL
migrations: migrations/accounts
dataClass: financial
sutra lint raises nothing on this package: every declared field has a column, every column’s type
holds the declared facet range (VARCHAR(35) for maxLength="35", NUMERIC(18,2) for
18 total / 2 fractional digits), the one optional field maps to a nullable column, the three
control columns are all present, and the table’s primary key is store_key
itself — never a declared field — which is what lint requires of every projected table
(KEY_MISMATCH otherwise: a business key would let two store keys collide on one row).
What lands in the database is a table anything can read:
store_key | account_id | balance | opened_at | active | note | rev | updated_at |
|---|---|---|---|---|---|---|---|
alice | alice | 100.00 | 2026-01-01 | true | NULL | 1 | 2026-01-01T00:00:00Z |
Round-tripping is lexical, not merely typed: every value travels as text in both directions —
bound with the dialect’s write-side cast, read back through its canonical text rendering — so a
put followed by a get returns the same shape it wrote, including a decimal’s written scale,
decided by your own NUMERIC(18,2) rather than by anything in between. That holds exactly for
xs:string, xs:decimal/the integer family, xs:boolean, xs:date, and an absent optional
field — and an explicit JSON null on an optional field stores and reads back exactly like an
absent one: both are NULL in the column, and both come back with the key omitted, never an
explicit null.
xs:dateTime and xs:time are the one exception to “lexical”: the column stores an instant, not
the string you sent, so a get returns the dialect’s own rendering of that instant, normalised
toward ISO-8601 (a T separator, an explicit +HH:MM offset) — not necessarily the offset or the
sub-second digits you wrote. Write 2026-08-04T10:00:00.000-05:00 into a dateTime column and
expect back whatever your dialect renders that same instant as (typically UTC, for a
timezone-aware column type), not the literal string. forUpdate, expect="unchanged" and the
revision bookkeeping behave exactly as they do for an opaque store.
If AccountRecord later grows an owner sub-record, the package stops linting with
STRUCTURE_NOT_FLAT — an explicit, package-time decision (flatten it, or give up projection for
this store) rather than a silent shape change. The evolution rules are in
Deployment packages.
Querying rows by key
Since 0.6.10 (typed-column stores since 0.6.13). A data-task read can return every row of a store whose key starts with a prefix, or lies in a range, together with aggregates over all of those rows. It suits append-only data — one row per event, keyed so that the rows you report on together share a prefix — where a single summary row would be a write hot spot.
<bpmn:dataStoreReference id="refDay" dataStoreRef="readingsStore">
<bpmn:extensionElements>
<q:store key="payload.meter + '|' + payload.day + '|'" query="prefix"
limit="2" after="payload.after">
<q:aggregate fn="sum" field="kwh"/>
<q:aggregate fn="sum" field="kwh" groupBy="kind"/>
</q:store>
</bpmn:extensionElements>
</bpmn:dataStoreReference>
| Attribute / element | Meaning |
|---|---|
query="prefix" | keys that start with key (an empty prefix is refused: it would read the whole store) |
query="range", to | keys from key (inclusive) to to (exclusive); both are FEEL strings |
limit | rows per page, default 1000, maximum 10 000 |
after | FEEL: only keys after this one — the previous page’s next, or null for the first page |
rows="false" | aggregates only |
<q:aggregate fn="count"> | {count} over every matching row |
<q:aggregate fn="sum" field="…"> | {count, sum}; field is a dotted path into the stored value |
groupBy="…" | {groups: {"<value>": {count, sum?}}}, one cell per value of that path |
The read’s target variable receives:
{"rows": [{"key": "m-7|2026-10-30|0001", "value": {"kind": "import", "kwh": "1.25"}}, ...],
"aggregates": [{"count": 5, "sum": 7.55},
{"groups": {"export": {"count": 2, "sum": 1.15}, "import": {"count": 3, "sum": 6.40}}}],
"truncated": true, "next": "m-7|2026-10-30|0002"}
- Keys compare bytewise, so
m-7|2026-10-01…m-7|2026-10-31covers a month and a|separator behaves predictably. Rows come back in ascending key order.truncatedsays more rows follow, andnextis the key to pass asafter. - Aggregates cover every matching row, whatever
limitandaftersay, so every page reports the same totals. - Sums are exact decimals, never floats. A sum adds JSON numbers and strings that hold a
decimal literal (
"30.05"). A row whose field is missing ornullis counted but not summed. Any other value — text, a boolean, an object — fails the task withSUTRA.RUNTIME.DATASTORE.QUERY_NOT_NUMERIC. A row without thegroupByfield groups under"null". No matching row gives{count: 0, sum: 0}or{groups: {}}, never null. - It is a plain read. It takes no locks, reads committed rows, never uses the
read cache, and does not see the uncommitted writes of an enclosing
<bpmn:transaction>. It cannot be combined withforUpdate, cannot be a write, and runs onsqlstores — key/value or typed-column (see Querying a typed-column store).
The index a key query needs
A key query is one range scan of an index whose first two columns are (store_name, store_key),
with store_key in a bytewise collation. Declare both in the store’s migration:
CREATE TABLE IF NOT EXISTS data_store (
tenant_id VARCHAR(64) NOT NULL, -- filled by the engine on insert
module_id VARCHAR(256) NOT NULL,
module_version VARCHAR(64) NOT NULL,
store_name VARCHAR(128) NOT NULL,
store_key VARCHAR(512) COLLATE "C" NOT NULL,
store_value TEXT NOT NULL,
rev BIGINT NOT NULL DEFAULT 1,
updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (store_name, store_key)
);
On MySQL/MariaDB use a …_bin collation for store_key (ascii_bin, utf8mb4_bin); on SQL Server
an NVARCHAR key with a …_BIN2 collation. Before a store’s first query the engine checks both and
refuses rather than scanning: SUTRA.RUNTIME.DATASTORE.QUERY_KEY_ORDER names a key column that does
not sort bytewise, SUTRA.RUNTIME.DATASTORE.QUERY_KEY_INDEX a table with no such index — for example
one keyed (tenant_id, module_id, module_version, store_name, store_key). Each message carries the
DDL that fixes it. Stores that never run a key query are unaffected.
On MySQL a JSON number is held as a double, exact to about 15 significant digits; store amounts
with more digits as strings there. PostgreSQL keeps the inputs’ decimal scale (6.40); MySQL and
SQL Server report the sum without trailing zeros.
Querying a typed-column store
Since 0.6.13. A store that declares a structure: is queried with the same <q:store query=…>,
over its own table. Each row’s value is the typed record, exactly as a plain read of that key
returns it, and the aggregates are SQL COUNT and SUM over the declared columns, with
GROUP BY the grouping column. For the accounts store from
Typed columns:
<q:store key="'branch-7|'" query="prefix" limit="100" after="payload.after">
<q:aggregate fn="sum" field="balance" groupBy="active"/>
<q:aggregate fn="count" groupBy="note"/>
</q:store>
{"rows": [{"key": "branch-7|alice",
"value": {"accountId": "alice", "balance": 100.50, "openedAt": "2026-01-01", "active": true}}],
"aggregates": [{"groups": {"true": {"count": 2, "sum": 250.50}, "false": {"count": 1, "sum": 0.00}}},
{"groups": {"null": {"count": 3}}}],
"truncated": false, "next": null}
fieldandgroupByname one declared field —balance, notbalance.value, and not a field the structure does not declare.sutra lintrefuses either, naming the structure and its fields; at run time the same mistake fails withSUTRA.RUNTIME.DATASTORE.QUERY_UNSUPPORTED.- A summed field must be numeric — declared
xs:decimalor an integer type.sutra lintrefuses a sum over a string, date or boolean field. (If one reaches the engine anyway, every present value counts as not numeric and the task fails withSUTRA.RUNTIME.DATASTORE.QUERY_NOT_NUMERIC, as on a key/value store.) - Sums are exact. They are computed on the column itself, so a
NUMERIC(18,2)sums to two decimal places on PostgreSQL. MySQL and SQL Server report the sum without trailing zeros. - An absent optional field is a NULL column. A sum skips it (the row still counts), and it
groups under
"null", just as a missing field does on a key/value store. Group keys read the way the record does:"true"/"false"for a boolean, the decimal text for a number,yyyy-mm-ddfor a date.
The index requirement is the same idea with one column: the table’s first index column must be
store_key, in a bytewise collation. A typed table normally already has PRIMARY KEY (store_key);
add the collation:
CREATE TABLE IF NOT EXISTS accounts (
store_key VARCHAR(512) COLLATE "C" NOT NULL, -- MySQL: … COLLATE utf8mb4_bin; SQL Server: NVARCHAR … COLLATE Latin1_General_100_BIN2
account_id VARCHAR(35) NOT NULL,
balance NUMERIC(18,2) NOT NULL,
opened_at DATE NOT NULL,
active BOOLEAN NOT NULL,
note VARCHAR(140),
rev BIGINT NOT NULL DEFAULT 1,
updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (store_key)
);
The first query checks both and refuses with SUTRA.RUNTIME.DATASTORE.QUERY_KEY_ORDER or
SUTRA.RUNTIME.DATASTORE.QUERY_KEY_INDEX, naming the table and the DDL that fixes it — for example
a table whose primary key is (account_id, store_key).
Claiming a key once
Since 0.6.10. expect="absent" makes a write an insert that applies only when the key does not
exist yet. The database decides, so of any number of instances — on any number of replicas — that
write the same key, exactly one wins. (A forUpdate read of a missing key locks nothing, so “read,
see nothing, write” cannot do this.) The losing write changes nothing and raises
SUTRA.RUNTIME.DATASTORE.CONFLICT. Catch it with an error boundary on the data task to take a
“someone else got there first” path; uncaught, it fails the instance:
<bpmn:error id="taken" errorCode="SUTRA.RUNTIME.DATASTORE.CONFLICT"/>
<bpmn:dataStoreReference id="refRun" dataStoreRef="runsStore">
<bpmn:extensionElements><q:store key="'daily|' + day" expect="absent"/></bpmn:extensionElements>
</bpmn:dataStoreReference>
<bpmn:serviceTask id="Claim"> <!-- writes `claim` to refRun -->
<bpmn:dataOutputAssociation>
<bpmn:sourceRef>claim</bpmn:sourceRef><bpmn:targetRef>refRun</bpmn:targetRef>
</bpmn:dataOutputAssociation>
</bpmn:serviceTask>
<bpmn:boundaryEvent id="AlreadyClaimed" attachedToRef="Claim">
<bpmn:errorEventDefinition errorRef="taken"/>
</bpmn:boundaryEvent>
The same conflict from expect="unchanged" is catchable the same way.
Walkthrough: readings by meter and day
A package with two processes: record-reading writes one row per reading with expect="absent"
(so a resent reading is refused), and day-report runs the key query above and answers with a
template. The files:
readings/
├── package.yaml # labels: default / readings / 1.0.0
├── datastores.yaml # store `readings`, env-indirected connection, migrations/readings
├── migrations/readings/V001__data_store.sql # the table above
├── channels.yaml # reading-in, day-report: HTTP + json + apikey
├── bpmn/readings.bpmn
└── templates/day-report.hbs
record-reading keys the row by meter, day and sequence number:
<q:store key="payload.meter + '|' + payload.day + '|' + payload.seq" expect="absent"/>
day-report reads the day into report, then picks the parts the answer shows:
<bpmn:serviceTask id="Query" name="Read the day">
<bpmn:dataInputAssociation>
<bpmn:sourceRef>refDay</bpmn:sourceRef><bpmn:targetRef>report</bpmn:targetRef>
</bpmn:dataInputAssociation>
<bpmn:dataInputAssociation>
<bpmn:assignment><bpmn:from>report.aggregates[1]</bpmn:from><bpmn:to>total</bpmn:to></bpmn:assignment>
</bpmn:dataInputAssociation>
<bpmn:dataInputAssociation>
<bpmn:assignment><bpmn:from>report.aggregates[2].groups</bpmn:from><bpmn:to>byKind</bpmn:to></bpmn:assignment>
</bpmn:dataInputAssociation>
<bpmn:dataInputAssociation>
<bpmn:assignment><bpmn:from>for row in report.rows return row.key</bpmn:from><bpmn:to>keys</bpmn:to></bpmn:assignment>
</bpmn:dataInputAssociation>
</bpmn:serviceTask>
{"keys": [{{#each keys}}{{#if @index}}, {{/if}}"{{this}}"{{/each}}],
"readings": {{total.count}}, "kwh": {{total.sum}},
"importKwh": {{byKind.import.sum}}, "exportKwh": {{byKind.export.sum}},
"next": {{#if report.next}}"{{report.next}}"{{else}}null{{/if}}}
Package it and start PostgreSQL and the engine (the engine refuses a superuser database role, so it gets its own):
sutra package readings --out deployments
docker network create readings-net
docker run -d --name readings-db --network readings-net -e POSTGRES_PASSWORD=dev-only postgres:16-alpine
docker exec readings-db psql -U postgres \
-c "CREATE ROLE sutra_engine LOGIN PASSWORD 'dev-only' NOSUPERUSER NOBYPASSRLS" \
-c "ALTER SCHEMA public OWNER TO sutra_engine"
docker run -d --name readings-engine --network readings-net -p 127.0.0.1::8080 \
-v "$PWD/deployments:/deployments:ro" -e SUTRA_DEPLOYMENTS_DIR=/deployments \
-e SUTRA_DATASOURCE_URL=postgres://readings-db:5432/postgres \
-e SUTRA_DATASOURCE_USERNAME=sutra_engine -e SUTRA_DATASOURCE_PASSWORD=dev-only \
-e SUTRA_ADMIN_OIDC_DEV_DISABLED=true \
-e READINGS_DB_URL=postgres://readings-db:5432/postgres \
-e READINGS_DB_USER=sutra_engine -e READINGS_DB_PASSWORD=dev-only \
-e READINGS_API_KEY=dev-only-readings-key ghcr.io/startr-trade/sutra:0.6.13
PORT=$(docker port readings-engine 8080 | head -1 | cut -d: -f2)
H=(-H 'Content-Type: application/json' -H 'X-Api-Key: dev-only-readings-key')
Record five readings for meter m-7 on 2026-10-30 (amounts as decimal strings or numbers, both
sum exactly) and one on the next day:
for r in '"0001","kwh":"1.25","kind":"import"' '"0002","kwh":0.75,"kind":"export"' \
'"0003","kwh":"2.10","kind":"import"' '"0004","kwh":"0.40","kind":"export"' \
'"0005","kwh":"3.05","kind":"import"'; do
curl -s -o /dev/null -w "%{http_code}\n" -X POST "http://127.0.0.1:$PORT/channels/reading-in" "${H[@]}" \
-d "{\"meter\":\"m-7\",\"day\":\"2026-10-30\",\"seq\":$r}"
done
curl -s -o /dev/null -w "%{http_code}\n" -X POST "http://127.0.0.1:$PORT/channels/reading-in" "${H[@]}" \
-d '{"meter":"m-7","day":"2026-10-31","seq":"0001","kwh":"9.99","kind":"import"}'
Each answers 200. Resending reading 0003 is refused, because the key exists:
{"type":"urn:bpm:diag:SUTRA.RUNTIME.DATASTORE.CONFLICT", ... "detail":"Optimistic-concurrency
conflict writing data store 'readings' key 'm-7|2026-10-30|0003' at data task 'Record': the key
already exists (expect=\"absent\"). ..."}
Ask for the day, two rows per page, passing each page’s next as the next after:
curl -s -X POST "http://127.0.0.1:$PORT/channels/day-report" "${H[@]}" \
-d '{"meter":"m-7","day":"2026-10-30","after":null}'
{"keys": ["m-7|2026-10-30|0001", "m-7|2026-10-30|0002"],
"readings": 5, "kwh": 7.55,
"importKwh": 6.40, "exportKwh": 1.15,
"next": "m-7|2026-10-30|0002"}
With "after":"m-7|2026-10-30|0002" the keys are 0003 and 0004 (next …|0004); with
"after":"m-7|2026-10-30|0004" only 0005, and "next": null. The totals are the same on every
page, and the 2026-10-31 reading is never counted. Clean up with
docker rm -f readings-engine readings-db && docker network rm readings-net.
Deleting a row
Since 0.7.0. delete="true" on a write removes the key’s row instead of writing a value. The
reference is the targetRef of the data task’s dataOutputAssociation, as for any write. The
association needs no <sourceRef>, and one that is given is ignored.
<bpmn:dataStoreReference id="refRemove" dataStoreRef="itemsStore">
<bpmn:extensionElements><q:store key="payload.itemId" delete="true"/></bpmn:extensionElements>
</bpmn:dataStoreReference>
<bpmn:serviceTask id="Remove">
<bpmn:dataOutputAssociation><bpmn:targetRef>refRemove</bpmn:targetRef></bpmn:dataOutputAssociation>
</bpmn:serviceTask>
| Form | Row present | Row missing |
|---|---|---|
delete="true" | deleted | no-op |
delete="true" expect="unchanged", key read earlier in the instance | deleted while its revision is still the one read, otherwise a conflict | a conflict (the row is gone) |
delete="true" expect="unchanged", key never read, or read as missing | a conflict (the row appeared) | no-op |
The conflict is SUTRA.RUNTIME.DATASTORE.CONFLICT. An error boundary on the data task catches it, as
it catches a lost claim.
Read, check, delete. A data task runs its reads before its writes. So one task can read a key
through one reference and delete it through another with expect="unchanged". If anything changes
the row between the read and the delete, the delete does not apply.
<bpmn:dataStoreReference id="refItem" dataStoreRef="itemsStore">
<bpmn:extensionElements><q:store key="payload.itemId"/></bpmn:extensionElements>
</bpmn:dataStoreReference>
<bpmn:dataStoreReference id="refRemove" dataStoreRef="itemsStore">
<bpmn:extensionElements><q:store key="payload.itemId" delete="true" expect="unchanged"/></bpmn:extensionElements>
</bpmn:dataStoreReference>
<bpmn:serviceTask id="Remove">
<bpmn:dataInputAssociation><bpmn:sourceRef>refItem</bpmn:sourceRef><bpmn:targetRef>item</bpmn:targetRef></bpmn:dataInputAssociation>
<bpmn:dataOutputAssociation><bpmn:targetRef>refRemove</bpmn:targetRef></bpmn:dataOutputAssociation>
</bpmn:serviceTask>
A delete follows the same rules as a write of the same key:
- Transactions. Inside a
<bpmn:transaction>the delete runs on the scope’s store transaction, so it commits or rolls back with the scope. Outside one it commits on its own. - Locks. The
DELETEtakes the row lock itself. It waits behind aforUpdateread of the same key in another instance until that instance commits. - The read cache. A delete through the engine invalidates the key at once. Other replicas see the row gone on their next read, because a deleted row no longer has the revision they cached.
- Store shapes and databases. Deletes work on key/value and typed-column stores, on PostgreSQL, MySQL, MariaDB and SQL Server.
delete cannot be combined with field, forUpdate, expect="absent", query, to, rows,
limit, after or a <q:aggregate>. A delete reference cannot be the source of a read either.
sutra lint, sealing and deploy all refuse these with SUTRA.PARSE.STORE.DELETE_INVALID.
A re-created row never reuses a revision. A row’s revision starts at the database’s own clock (microseconds since the epoch, read in the insert itself, so all replicas share one clock) and goes up by one with each write. Revisions are strictly increasing per row but not dense. A key that is deleted and created again therefore starts above every revision the old row had. A replica that cached the old row reads the new one, and an
expect="unchanged"write or delete that read the old row is refused. This relies on the database clock moving forward: if it is stepped backwards between the delete and the re-create, the guarantee can break.
Walkthrough: removing an item seen through two replicas
A package with an items store that keeps a read cache, a results store, and three processes,
each started over HTTP: item-write writes a row, item-copy reads one row and copies it into
results, and item-delete runs the read-check-delete above. Two engines share one database, so
one can delete a row the other has cached. The files:
items/
├── package.yaml # labels: default / items / 1.0.0
├── datastores.yaml # stores `items` (cache: {enabled: true}) and `results`, env-indirected
├── migrations/items/V001__data_store.sql # the key/value table, as in Querying rows by key
├── migrations/results/V001__data_store.sql # the same table
├── channels.yaml # item-in, copy-in, delete-in: HTTP + json + apikey
└── bpmn/items.bpmn
item-copy copies the row into results under copy|<key>|<tag>:
<bpmn:dataStoreReference id="refItem" dataStoreRef="itemsStore">
<bpmn:extensionElements><q:store key="payload.key"/></bpmn:extensionElements>
</bpmn:dataStoreReference>
<bpmn:dataStoreReference id="refCopy" dataStoreRef="resultsStore">
<bpmn:extensionElements><q:store key="'copy|' + payload.key + '|' + payload.tag"/></bpmn:extensionElements>
</bpmn:dataStoreReference>
<bpmn:serviceTask id="Copy" name="Copy the item">
<bpmn:dataInputAssociation>
<bpmn:sourceRef>refItem</bpmn:sourceRef><bpmn:targetRef>copied</bpmn:targetRef>
</bpmn:dataInputAssociation>
<bpmn:dataOutputAssociation>
<bpmn:sourceRef>copied</bpmn:sourceRef><bpmn:targetRef>refCopy</bpmn:targetRef>
</bpmn:dataOutputAssociation>
</bpmn:serviceTask>
Package it, then start PostgreSQL and two engines, items-a and items-b:
sutra package items --out deployments
docker network create items-net
docker run -d --name items-db --network items-net -e POSTGRES_PASSWORD=dev-only postgres:16-alpine
until docker exec items-db pg_isready -U postgres -h 127.0.0.1; do sleep 1; done; sleep 2
docker exec items-db psql -U postgres \
-c "CREATE ROLE sutra_engine LOGIN PASSWORD 'dev-only' NOSUPERUSER NOBYPASSRLS" \
-c "ALTER SCHEMA public OWNER TO sutra_engine"
for r in a b; do
docker run -d --name items-$r --network items-net -p 127.0.0.1::8080 \
-v "$PWD/deployments:/deployments:ro" -e SUTRA_DEPLOYMENTS_DIR=/deployments \
-e SUTRA_DATASOURCE_URL=postgres://items-db:5432/postgres \
-e SUTRA_DATASOURCE_USERNAME=sutra_engine -e SUTRA_DATASOURCE_PASSWORD=dev-only \
-e SUTRA_ADMIN_OIDC_DEV_DISABLED=true \
-e ITEMS_DB_URL=postgres://items-db:5432/postgres \
-e ITEMS_DB_USER=sutra_engine -e ITEMS_DB_PASSWORD=dev-only \
-e ITEMS_API_KEY=dev-only-items-key ghcr.io/startr-trade/sutra:0.8.2
until curl -sf -o /dev/null "http://127.0.0.1:$(docker port items-$r 8080 | head -1 | cut -d: -f2)/sutra/health/ready"; do
sleep 1
done
done
A=http://127.0.0.1:$(docker port items-a 8080 | head -1 | cut -d: -f2)
B=http://127.0.0.1:$(docker port items-b 8080 | head -1 | cut -d: -f2)
C=(-s -o /dev/null -w "%{http_code}\n" -H 'Content-Type: application/json' -H 'X-Api-Key: dev-only-items-key')
sql() { docker exec items-db psql -U postgres -tAc "$1"; }
-
Write an item through
items-a. Its revision is the database clock at the insert:curl "${C[@]}" -X POST "$A/channels/item-in" -d '{"key":"item-0001","entry":{"amount":"12.50"}}' sql "SELECT store_value, rev FROM data_store WHERE store_name='items' AND store_key='item-0001'"200 {"amount":"12.50"}|1795060962963065 -
Read it through
items-b, which now holds it in its cache:curl "${C[@]}" -X POST "$B/channels/copy-in" -d '{"key":"item-0001","tag":"before"}' sql "SELECT store_value FROM data_store WHERE store_name='results' AND store_key='copy|item-0001|before'"200 {"amount":"12.50"} -
Delete it through
items-a:curl "${C[@]}" -X POST "$A/channels/delete-in" -d '{"key":"item-0001"}' sql "SELECT count(*) FROM data_store WHERE store_name='items' AND store_key='item-0001'"200 0 -
Read it through
items-bagain. Its cached entry’s revision no longer matches (the row is gone), so it serves nothing stale:curl "${C[@]}" -X POST "$B/channels/copy-in" -d '{"key":"item-0001","tag":"after"}' sql "SELECT store_value FROM data_store WHERE store_name='results' AND store_key='copy|item-0001|after'"200 null -
Create the item again with another value through
items-a, and read it throughitems-b. The new row starts above every revision the old one had, soitems-breads the new value:curl "${C[@]}" -X POST "$A/channels/item-in" -d '{"key":"item-0001","entry":{"amount":"2.00"}}' sql "SELECT store_value, rev FROM data_store WHERE store_name='items' AND store_key='item-0001'" curl "${C[@]}" -X POST "$B/channels/copy-in" -d '{"key":"item-0001","tag":"again"}' sql "SELECT store_value FROM data_store WHERE store_name='results' AND store_key='copy|item-0001|again'"200 {"amount":"2.00"}|1795060963446990 200 {"amount":"2.00"} -
Delete it twice, once through each engine. Both answer
200: the second delete is of a missing key, a no-op.curl "${C[@]}" -X POST "$A/channels/delete-in" -d '{"key":"item-0001"}' curl "${C[@]}" -X POST "$B/channels/delete-in" -d '{"key":"item-0001"}' -
Change the delete reference to
<q:store key="payload.key" delete="true" forUpdate="true"/>and lint the package withsutra lint items. It is refused before anything runs:[ERROR] SUTRA.DEPLOY.PACKAGE.CONFIG_INVALID — BPMN bpmn/items.bpmn fails to parse: [SUTRA.PARSE.STORE.DELETE_INVALID] <serviceTask> 'Remove' declares a delete on data store 'items' that cannot run: @forUpdate locks a row for a read; a delete takes the row lock itself (lock it with a forUpdate read through another reference). (items/bpmn/items.bpmn)
Revisions differ on every run. Clean up with
docker rm -f items-a items-b items-db && docker network rm items-net.
Caching reads
Since 0.6.5. A data task reads a value and its revision in one round trip. A store that is read far more often than it is written — settings, reference data — can also keep a read cache:
datastores:
- name: refdata
type: sql
sql:
url-ref: env:REF_DB_URL
username-ref: env:REF_DB_USER
password-ref: env:REF_DB_PASSWORD
cache:
enabled: true # default false
maxEntries: 1000 # keys kept; the least recently read is evicted first
ttl: PT0S # default: check the revision on every read
ttl | A read of a cached key | A write made through any engine | A write made by SQL outside the engine |
|---|---|---|---|
PT0S (default) | one revision check (SELECT rev) instead of a full read | seen by the next read on every replica: each write bumps the row’s revision | seen by the next read if the SQL bumps rev, otherwise only once the key is evicted |
positive, e.g. PT5S | no query at all while the key was checked less than ttl ago | through this engine: seen at once; through another replica: within ttl | within ttl |
The cache is shared by every lane of the engine process and is per deployment. A write or delete
made through the engine — a data task’s, or one inside a <bpmn:transaction> — invalidates its
key at once. Every other replica sees it on its next read, a key deleted and created again
included: the re-created row never reuses a revision (see Deleting a row). Reads inside a transaction always go to the database, because they must see the
transaction’s own snapshot and locks. Keep ttl at PT0S unless a few seconds of staleness
after an operator’s edit is acceptable. When you change rows by hand, bump rev with them:
UPDATE data_store SET store_value = '{"maxAmount": 500}', rev = rev + 1
WHERE store_name = 'refdata' AND store_key = 'default';
When the database cannot be reached
A data task whose database cannot be reached (a dropped connection, a pool timeout, a server
restarting) fails with SUTRA.RUNTIME.DATASTORE.UNAVAILABLE. That is not a failed instance: the
step is not committed and the delivery is refused for a retry (HTTP 503, a broker redelivery, the
outbox’s backoff for a hop between processes, the timer poller’s backoff), whatever the process
declares about idempotency. It is never dead-lettered. (Since 0.8.2.)
Isolation and atomicity
Two separate guarantees, worth keeping distinct:
- Isolation (no lost update between concurrent writers) comes from serializing access to a
key — either a pessimistic
forUpdate="true"lock, or funneling all writers for that channel through a single active consumer (see thesingletonchannel property in Channels and transports). - Atomicity (all writes commit or none do) comes from a
<bpmn:transaction>scope: writes inside it commit together on a normal end, or roll back together on a cancel end / error. An external store write cannot auto-rollback on its own — the transaction sub-process is what gives you the boundary.
The worked example combines both: a singleton channel serializes writers,
and the debit/credit pair runs inside a <bpmn:transaction> so a rejected transfer touches no row.
Keeping a message as text
A flow can keep a message exactly as it arrived, as text in a typed store column, and work with it
decoded again later. This walkthrough keeps notes in a TEXT column, reads one back through
<q:decode>, and then masks them.
You need the engine and PostgreSQL; this walkthrough was run on PostgreSQL 16. xsd/q.xsd is the
reference for the element.
1. A typed store with a text column
package.yaml labels the module notes, version 1.0.0. datastores.yaml gives the store its row
type:
datastores:
- name: notes
type: sql
structure:
schema: urn:note-row
type: NoteRow
sql:
url-ref: env:NOTES_DB_URL
username-ref: env:NOTES_DB_USER
password-ref: env:NOTES_DB_PASSWORD
migrations: migrations/notes
schemas/note-row/codec-manifest.yaml is schemaKind: xsd and formats: [xml].
schemas/note-row/note-row.xsd declares the row. body has no length limit:
<xs:element name="NoteRow">
<xs:complexType>
<xs:sequence>
<xs:element name="noteId" type="NoteId"/> <!-- maxLength 64 -->
<xs:element name="contentType" type="MediaType" minOccurs="0"/> <!-- maxLength 128 -->
<xs:element name="body" type="xs:string"/>
</xs:sequence>
</xs:complexType>
</xs:element>
migrations/notes/V001__notes.sql:
CREATE TABLE IF NOT EXISTS notes (
store_key VARCHAR(512) NOT NULL,
note_id VARCHAR(64) NOT NULL,
content_type VARCHAR(128),
body TEXT NOT NULL,
rev BIGINT NOT NULL DEFAULT 1,
updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (store_key)
);
On MySQL, TEXT stops at 64 KiB, so use MEDIUMTEXT or LONGTEXT. On SQL Server, use
NVARCHAR(MAX).
2. Capture the text
channels.yaml declares two HTTP channels with the usual API-key auth. note-in has no codec, so
it takes any format, and its key comes from a header:
channels:
- name: note-in
transport: http
bind: "POST /channels/note-in"
idempotency-key-header: X-Note-Id
cloudevents-mode: none
auth: { scheme: apikey, apikey: { value: env:NOTES_API_KEY, header: X-Api-Key } }
- name: note-read
transport: http
bind: "POST /channels/note-read"
codec: json
cloudevents-mode: none
auth: { scheme: apikey, apikey: { value: env:NOTES_API_KEY, header: X-Api-Key } }
bpmn/note-keep.bpmn builds the row from event on the step right after the start. A later message
would replace event, so capture here. The gateway is step 6:
<bpmn:dataStore id="notesStore" name="notes"/>
<bpmn:process id="note-keep" isExecutable="true">
<bpmn:extensionElements>
<q:variables><q:variable name="row" type="object"/></q:variables>
</bpmn:extensionElements>
<bpmn:dataStoreReference id="refRow" dataStoreRef="notesStore">
<bpmn:extensionElements><q:store key="event.dedupKey"/></bpmn:extensionElements>
</bpmn:dataStoreReference>
<bpmn:startEvent id="Start">
<bpmn:extensionElements><q:source channel="note-in"/></bpmn:extensionElements>
</bpmn:startEvent>
<bpmn:exclusiveGateway id="IsText" default="ToKeep"/>
<bpmn:serviceTask id="Keep" name="Keep the text">
<bpmn:dataInputAssociation>
<bpmn:assignment>
<bpmn:from>{noteId: event.dedupKey, contentType: event.contentType, body: event.body}</bpmn:from>
<bpmn:to>row</bpmn:to>
</bpmn:assignment>
</bpmn:dataInputAssociation>
<bpmn:dataOutputAssociation>
<bpmn:sourceRef>row</bpmn:sourceRef><bpmn:targetRef>refRow</bpmn:targetRef>
</bpmn:dataOutputAssociation>
</bpmn:serviceTask>
<bpmn:endEvent id="Kept"/>
<bpmn:endEvent id="NotText"/>
<bpmn:sequenceFlow id="F1" sourceRef="Start" targetRef="IsText"/>
<bpmn:sequenceFlow id="ToKeep" sourceRef="IsText" targetRef="Keep"/>
<bpmn:sequenceFlow id="ToRefuse" sourceRef="IsText" targetRef="NotText">
<bpmn:conditionExpression>event.bodyLossy = true</bpmn:conditionExpression>
</bpmn:sequenceFlow>
<bpmn:sequenceFlow id="F2" sourceRef="Keep" targetRef="Kept"/>
</bpmn:process>
3. A retrieval flow
bpmn/note-read.bpmn reads the row, decodes its text by the row’s own content type, and answers from
the decoded fields. The store read runs first, then <q:decode>, within one data task. Text that
doesn’t parse raises SUTRA.RUNTIME.DECODE.FAILED, which the boundary event catches:
<bpmn:error id="undecodable" errorCode="SUTRA.RUNTIME.DECODE.FAILED"/>
<bpmn:dataStore id="notesStore" name="notes"/>
<bpmn:process id="note-read" isExecutable="true">
<bpmn:extensionElements>
<q:variables>
<q:variable name="row" type="object"/>
<q:variable name="note" type="object"/>
</q:variables>
</bpmn:extensionElements>
<bpmn:dataStoreReference id="refRead" dataStoreRef="notesStore">
<bpmn:extensionElements><q:store key="req.id"/></bpmn:extensionElements>
</bpmn:dataStoreReference>
<bpmn:startEvent id="Start">
<bpmn:extensionElements><q:source channel="note-read" name="req"/></bpmn:extensionElements>
</bpmn:startEvent>
<bpmn:serviceTask id="Load" name="Read and decode">
<bpmn:extensionElements>
<q:decode name="note" expression="row.body" codec="json" contentType="row.contentType"/>
</bpmn:extensionElements>
<bpmn:dataInputAssociation>
<bpmn:sourceRef>refRead</bpmn:sourceRef><bpmn:targetRef>row</bpmn:targetRef>
</bpmn:dataInputAssociation>
</bpmn:serviceTask>
<bpmn:boundaryEvent id="BadText" attachedToRef="Load">
<bpmn:errorEventDefinition errorRef="undecodable"/>
</bpmn:boundaryEvent>
<bpmn:serviceTask id="Answer" implementation="summary.hbs">
<bpmn:extensionElements><q:reply mode="native" contentType="application/json"/></bpmn:extensionElements>
</bpmn:serviceTask>
<bpmn:serviceTask id="AnswerBad" implementation="undecodable.hbs">
<bpmn:extensionElements><q:reply mode="native" contentType="application/json"/></bpmn:extensionElements>
</bpmn:serviceTask>
<bpmn:endEvent id="Done"/>
<bpmn:endEvent id="DoneBad"/>
<bpmn:sequenceFlow id="F1" sourceRef="Start" targetRef="Load"/>
<bpmn:sequenceFlow id="F2" sourceRef="Load" targetRef="Answer"/>
<bpmn:sequenceFlow id="F3" sourceRef="Answer" targetRef="Done"/>
<bpmn:sequenceFlow id="F4" sourceRef="BadText" targetRef="AnswerBad"/>
<bpmn:sequenceFlow id="F5" sourceRef="AnswerBad" targetRef="DoneBad"/>
</bpmn:process>
templates/summary.hbs:
{"decoded":true,"title":"{{note.title}}","pages":{{note.pages}}}
templates/undecodable.hbs:
{"decoded":false}
4. Package it and start the engine
sutra lint notes
sutra package notes -o deployments
Give the engine a role that can’t bypass row-level security, and a database that role owns:
CREATE ROLE sutra_app LOGIN NOSUPERUSER NOBYPASSRLS PASSWORD '<a password>';
CREATE DATABASE sutra OWNER sutra_app;
Start the engine with SUTRA_DATASOURCE_URL pointing at that database as sutra_app,
SUTRA_DEPLOYMENTS_DIR pointing at deployments, NOTES_API_KEY, and NOTES_DB_URL,
NOTES_DB_USER and NOTES_DB_PASSWORD for the store. Turn admin auth on with
SUTRA_ADMIN_AUTH_SCHEME=apikey, SUTRA_ADMIN_AUTH_KEY_REF=env:SUTRA_ADMIN_API_KEY and
SUTRA_ADMIN_API_KEY, because step 7 reads an instance. Set B to the engine’s address.
5. Keep a note and read it back
note.json has odd spacing, a non-ASCII title, and a number with a written scale:
{ "title" : "café notes",
"pages": 3.50 }
curl -s -o /dev/null -w '%{http_code}\n' -X POST "$B/channels/note-in" \
-H 'X-Api-Key: <your key>' -H 'Content-Type: application/json' -H 'X-Note-Id: n-1' \
--data-binary @note.json
200
The column holds the exact text:
SELECT store_key, note_id, content_type, body FROM notes;
store_key | note_id | content_type | body
-----------+---------+------------------+---------------------------
n-1 | n-1 | application/json | { "title" : "café notes",+
| | | "pages": 3.50 } +
| | |
Read it back:
curl -s -X POST "$B/channels/note-read" -H 'X-Api-Key: <your key>' \
-H 'Content-Type: application/json' -d '{"id":"n-1"}'
{"decoded":true,"title":"café notes","pages":3.50}
The row’s content type picks the parser, so the same flow reads XML. Keep
<note><title>second</title><pages>2</pages></note> as application/xml under n-2, and read it:
{"decoded":true,"title":"second","pages":2}
Keep { this is not json as application/json under n-3. It is stored as it came, and reading it
takes the error boundary:
{"decoded":false}
6. A message that isn’t text
Send bytes that aren’t valid UTF-8:
printf 'ok\377\376' | curl -s -o /dev/null -w '%{http_code}\n' -X POST "$B/channels/note-in" \
-H 'X-Api-Key: <your key>' -H 'Content-Type: application/octet-stream' -H 'X-Note-Id: n-4' \
--data-binary @-
200
The invalid bytes became U+FFFD, so event.bodyLossy is true, and the gateway took ToRefuse.
There is no n-4 row. A clean message has no bodyLossy key at all, which is why = true is the
test. <q:decode> handles text only; there is no binary form.
7. Masking
Now suppose the titles are private. Add redactors/title.hbs, which holds the path to mask:
/title
Name it on the source. This walkthrough also adds a wait, so there is an instance to look at:
<bpmn:startEvent id="Start">
<bpmn:extensionElements>
<q:source channel="note-in"><q:redactors><q:redactor ref="title"/></q:redactors></q:source>
<q:alias name="noteId" expression="event.dedupKey" unique="true"/>
</bpmn:extensionElements>
</bpmn:startEvent>
<!-- after Keep -->
<bpmn:receiveTask id="Review" name="Wait for a review">
<bpmn:extensionElements>
<q:source channel="note-reviewed" name="review"/>
<q:alias name="noteId" expression="review.id"/>
</bpmn:extensionElements>
</bpmn:receiveTask>
sutra lint notes now finds the copies the engine can’t see through, including in note-read, which
masks nothing itself but reads what a masking flow stored:
[WARN] SUTRA.CONFIG.BPMN.RAW_BODY_UNMASKED — ... process 'note-keep' declares masking, but variable 'row' holds raw event.body text and is not sensitive, so the raw message shows in audit and inspect and is stored unencrypted. Declare it <q:variable name="row" sensitive="true"/>. This becomes an error in a later release.
[WARN] SUTRA.CONFIG.BPMN.RAW_BODY_STORED — ... process 'note-keep' declares masking, but writes raw event.body text to data store 'notes' through variable 'row'. A store column has no at-rest encryption, and sensitive does not reach it.
[WARN] SUTRA.CONFIG.BPMN.RAW_BODY_UNMASKED — ... process 'note-read' reads text a masking process stored, and variable 'row' holds raw event.body text and is not sensitive, ...
[ERROR] SUTRA.CONFIG.DECODE.REDACTORS_MISSING — ... process 'note-read' reads text a masking process stored, and <q:decode name="note"> parses raw event.body text into a variable with no masking, so the message shows in audit and inspect. Declare it <q:variable name="note" sensitive="true"/>, or give the step <q:redactors>.
Declare row sensitive in both flows and note sensitive in note-read. Only
RAW_BODY_STORED remains. It stays because storing the text is the point of this flow, and it is true:
the column is cleartext.
Package it and restart the engine with SUTRA_CRYPTO_MASTER_KEY set, so that parked state is
encrypted. Keep a note with a private title under n-5, and look at the parked instance. This route
needs the admin key:
curl -s -H "X-API-Key: $SUTRA_ADMIN_API_KEY" "$B/sutra/instances?alias=noteId:n-5"
curl -s -H "X-API-Key: $SUTRA_ADMIN_API_KEY" "$B/sutra/instances/<instanceId>"
{"instanceId":"cb729345-dde3-46ab-94c6-0beaa6cbf5d3","deploymentId":"dep-a149aa3f6fbb7bc0a118c185","status":"SUSPENDED","completedNodes":["Keep"],"waitingNodes":["Review"],"sensitive":["row"],"variables":{"attr":"{}","event":"{body=[REDACTED], channel=note-in, contentType=application/json, dedupKey=[REDACTED], headers={accept=[REDACTED], content-length=[REDACTED], content-type=[REDACTED], host=[REDACTED], user-agent=[REDACTED], x-api-key=[REDACTED], x-note-id=[REDACTED]}, idempotencyKey=[REDACTED], receivedAt=2026-11-04T19:21:26.23034207Z}","header":"{accept=[REDACTED], content-length=[REDACTED], content-type=[REDACTED], host=[REDACTED], user-agent=[REDACTED], x-api-key=[REDACTED], x-note-id=[REDACTED]}","noteId":"n-5","payload":"{pages=1, title=[REDACTED]}","payload.validation":"[]","row":"***REDACTED***","validation":"{firstIssue=, firstReasonCode=, issues=[], outcome=OK, tier=n/a}"}}
The title appears nowhere in that answer, and the parked row in instance_state doesn’t contain it
either. The notes column does, in the clear. To keep it out of the column, don’t store it in text.
The flow itself still reads the real values. Answer the wait on note-reviewed with {"id":"n-5"},
then read the note:
{"decoded":true,"title":"PRIVATE-TITLE-5","pages":1}
Answering with the title is this flow’s choice. A real one masking its titles would answer something else.
Limits
- Size.
<q:decode>refuses text oversutra.codec.max-payload-byteswithSUTRA.RUNTIME.DECODE.TOO_LARGE. Column limits are the database’s. - Encoding. A body whose prolog declares another encoding, such as ISO-8859-1 XML, can’t be kept
faithfully. A NUL byte survives into
event.body, but PostgreSQLTEXTrefuses it, and that write fails the instance. - After a wait. A resuming message replaces
event. Capture before the wait, as here. - Other packages. Lint follows a store by name within one package. Another package reading the same database isn’t seen.
Sensitivity
A store’s dataClass (pii / pci / phi / financial) marks its contents sensitive: values
read from it are redacted wherever the engine emits observability data — audit events, structured
logs, traces — while the flow itself still sees the real value. The tag travels with the variable
a data association reads into, so it stays redacted downstream too.
Next
- Wait states and human tasks — the other durable, cross-request mechanism: a suspended instance rather than a shared store.
- Worked example: money-transfer — a data store under full ACID control.
- Deployment packages — where a store’s migrations live, and what a change to a projected structure costs.
- Troubleshooting — every projection diagnostic, what causes it, and how to fix it.
Signing a user in against a store
A flow can keep users in a typed store and check their passwords with two FEEL builtins,
password hash and password verify (new in 0.6.21; reference in
Rules: DMN, FEEL, and .srl). This walkthrough seeds one user from a migration, enrols a
second through a flow, and signs both in. The password itself is never stored.
You need the engine and PostgreSQL. This walkthrough was run on PostgreSQL 16, with the output shown below.
The two builtins
| Function | Returns |
|---|---|
password hash(password) | An argon2id hash in PHC string form, $argon2id$v=19$m=19456,t=2,p=1$<salt>$<hash>, with a fresh 16-byte salt from the OS. null if the argument is not a string or is empty. |
password verify(password, hash) | true if the password matches the hash, false if it does not. null if either argument is not a string, or if the hash is malformed, is not argon2id, or asks for more than m = 65 536 KiB, t = 10 or p = 4. The comparison is constant-time. |
The parameters are fixed: argon2id, m = 19 456 KiB, t = 2, p = 1, the OWASP minimum. A package
can’t change them. verify reads the parameters from the stored hash, so the cap keeps a planted
hash from tying up a worker and its memory.
password hash draws a random salt, so it is denied at replay-bound sites, like uuid(). Use it
in a task, not in an alias or a when. password verify is pure and works anywhere.
FEEL evaluation blocks its executor worker, so one verify holds a worker for one hash time, about 22 ms on a 2020 laptop CPU (Intel i7-10750H). That is fine for sign-ins. Don’t call these functions per message.
1. A typed store for users
package.yaml labels the module signin, version 1.0.0. datastores.yaml gives the store its
row type:
datastores:
- name: users
type: sql
structure:
schema: urn:user-row
type: UserRow
sql:
url-ref: env:STORE_DB_URL
username-ref: env:STORE_DB_USER
password-ref: env:STORE_DB_PASSWORD
migrations: migrations/users
schemas/user-row/codec-manifest.yaml is schemaKind: xsd and formats: [xml].
schemas/user-row/user-row.xsd declares the row:
<xs:element name="UserRow">
<xs:complexType>
<xs:sequence>
<xs:element name="userId" type="UserId"/> <!-- maxLength 64 -->
<xs:element name="passwordHash" type="PhcHash"/> <!-- maxLength 256 -->
</xs:sequence>
</xs:complexType>
</xs:element>
migrations/users/V001__users.sql creates the table and seeds one user, ada. The hash is a fixed
test vector for the password correct horse:
CREATE TABLE IF NOT EXISTS users (
store_key VARCHAR(512) NOT NULL,
user_id VARCHAR(64) NOT NULL,
password_hash VARCHAR(256) NOT NULL,
rev BIGINT NOT NULL DEFAULT 1,
updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (store_key)
);
INSERT INTO users (store_key, user_id, password_hash)
VALUES ('ada', 'ada',
'$argon2id$v=19$m=19456,t=2,p=1$AAECAwQFBgcICQoLDA0ODw$0AuQxA+RjPiJeA19Op7DwrFxnfh9xeTU2T6mFj2wThc')
ON CONFLICT (store_key) DO NOTHING;
2. Two channels
channels.yaml declares two JSON channels with the usual API-key auth. Both take
{"user": ..., "password": ...}:
channels:
- name: enrol
transport: http
bind: "POST /channels/enrol"
codec: json
cloudevents-mode: none
auth: { scheme: apikey, apikey: { value: env:SIGN_IN_API_KEY, header: X-Api-Key } }
- name: sign-in
transport: http
bind: "POST /channels/sign-in"
codec: json
cloudevents-mode: none
auth: { scheme: apikey, apikey: { value: env:SIGN_IN_API_KEY, header: X-Api-Key } }
3. Keep the password out of storage
Both flows bind the request to a variable named creds and declare it transient and sensitive.
transient keeps it out of the instance store. sensitive redacts it, and the raw request body
with it, in logs and audit. Neither flow waits, so creds is only needed for one turn.
bpmn/enrol.bpmn hashes the password and writes the row:
<bpmn:dataStore id="usersStore" name="users"/>
<bpmn:process id="enrol" isExecutable="true">
<bpmn:extensionElements>
<q:audit sink="jsonl" capture="payload"/>
<q:variables>
<q:variable name="creds" type="object" transient="true" sensitive="true"/>
<q:variable name="row" type="object"/>
</q:variables>
</bpmn:extensionElements>
<bpmn:dataStoreReference id="refUser" dataStoreRef="usersStore">
<bpmn:extensionElements><q:store key="creds.user"/></bpmn:extensionElements>
</bpmn:dataStoreReference>
<bpmn:startEvent id="Start">
<bpmn:extensionElements><q:source channel="enrol" name="creds"/></bpmn:extensionElements>
</bpmn:startEvent>
<bpmn:serviceTask id="Hash" name="Hash and store">
<bpmn:dataInputAssociation>
<bpmn:assignment>
<bpmn:from>{userId: creds.user, passwordHash: password hash(creds.password)}</bpmn:from>
<bpmn:to>row</bpmn:to>
</bpmn:assignment>
</bpmn:dataInputAssociation>
<bpmn:dataOutputAssociation>
<bpmn:sourceRef>row</bpmn:sourceRef><bpmn:targetRef>refUser</bpmn:targetRef>
</bpmn:dataOutputAssociation>
</bpmn:serviceTask>
<bpmn:endEvent id="Done"/>
<bpmn:sequenceFlow id="F1" sourceRef="Start" targetRef="Hash"/>
<bpmn:sequenceFlow id="F2" sourceRef="Hash" targetRef="Done"/>
</bpmn:process>
bpmn/sign-in.bpmn reads the row, checks the password, and answers:
<bpmn:dataStore id="usersStore" name="users"/>
<bpmn:process id="sign-in" isExecutable="true">
<bpmn:extensionElements>
<q:audit sink="jsonl" capture="payload"/>
<q:variables>
<q:variable name="creds" type="object" transient="true" sensitive="true"/>
<q:variable name="user" type="object"/>
<q:variable name="signedIn" type="boolean"/>
</q:variables>
</bpmn:extensionElements>
<bpmn:dataStoreReference id="refUser" dataStoreRef="usersStore">
<bpmn:extensionElements><q:store key="creds.user"/></bpmn:extensionElements>
</bpmn:dataStoreReference>
<bpmn:startEvent id="Start">
<bpmn:extensionElements><q:source channel="sign-in" name="creds"/></bpmn:extensionElements>
</bpmn:startEvent>
<bpmn:serviceTask id="Load" name="Read the user">
<bpmn:dataInputAssociation>
<bpmn:sourceRef>refUser</bpmn:sourceRef><bpmn:targetRef>user</bpmn:targetRef>
</bpmn:dataInputAssociation>
</bpmn:serviceTask>
<bpmn:serviceTask id="Check" name="Check the password">
<bpmn:dataInputAssociation>
<bpmn:assignment>
<bpmn:from>password verify(creds.password, user.passwordHash) = true</bpmn:from>
<bpmn:to>signedIn</bpmn:to>
</bpmn:assignment>
</bpmn:dataInputAssociation>
</bpmn:serviceTask>
<bpmn:serviceTask id="Answer" implementation="signed-in.hbs">
<bpmn:extensionElements><q:reply mode="native" contentType="application/json"/></bpmn:extensionElements>
</bpmn:serviceTask>
<bpmn:endEvent id="Done"/>
<bpmn:sequenceFlow id="F1" sourceRef="Start" targetRef="Load"/>
<bpmn:sequenceFlow id="F2" sourceRef="Load" targetRef="Check"/>
<bpmn:sequenceFlow id="F3" sourceRef="Check" targetRef="Answer"/>
<bpmn:sequenceFlow id="F4" sourceRef="Answer" targetRef="Done"/>
</bpmn:process>
The = true turns a null from password verify (a malformed stored hash) into false, so the
answer is always a boolean.
templates/signed-in.hbs:
{"signedIn":{{signedIn}}}
4. Package it and start the engine
sutra lint signin
sutra package signin -o deployments
Give the engine a role that can’t bypass row-level security, and a database that role owns:
CREATE ROLE sutra_app LOGIN NOSUPERUSER NOBYPASSRLS PASSWORD '<a password>';
CREATE DATABASE sutra OWNER sutra_app;
Start the engine with SUTRA_DATASOURCE_URL pointing at that database as sutra_app,
SUTRA_DEPLOYMENTS_DIR pointing at deployments, SIGN_IN_API_KEY, STORE_DB_URL, STORE_DB_USER
and STORE_DB_PASSWORD for the store, and SUTRA_AUDIT_JSONL for the audit file. Set B to the
engine’s address.
5. Sign in the seeded user
curl -s -X POST "$B/channels/sign-in" -H 'X-Api-Key: <your key>' \
-H 'Content-Type: application/json' -d '{"user":"ada","password":"correct horse"}'
{"signedIn":true}
With "password":"correct horsf" instead:
{"signedIn":false}
6. Enrol a user
curl -s -o /dev/null -w '%{http_code}\n' -X POST "$B/channels/enrol" -H 'X-Api-Key: <your key>' \
-H 'Content-Type: application/json' -d '{"user":"grace","password":"<a password>"}'
200
The row holds the hash, with its own salt:
SELECT store_key, user_id, password_hash FROM users ORDER BY store_key;
store_key | user_id | password_hash
-----------+---------+---------------------------------------------------------------------------------------------------
ada | ada | $argon2id$v=19$m=19456,t=2,p=1$AAECAwQFBgcICQoLDA0ODw$0AuQxA+RjPiJeA19Op7DwrFxnfh9xeTU2T6mFj2wThc
grace | grace | $argon2id$v=19$m=19456,t=2,p=1$egxSk+lhFOUDspiuABfQoQ$QzDs4/To+3jKTJclLpfm3cpGF1N41U2Ig4fZuXsTn3M
Signing grace in with that password answers {"signedIn":true}, and with any other password
{"signedIn":false}.
7. Where the password went
Nowhere. A data-only pg_dump of the engine’s database has no row containing either password, and
neither does the audit file or the engine log. The audit file records each step, with creds and
the raw body redacted:
{"eventType":"NODE_ENTERED","nodeId":"Answer","payload":{"creds":"***REDACTED***","event":{"body":"[REDACTED]", ...},"signedIn":true, ...}, ...}
8. Not at a replay-bound site
Put password hash in an alias and lint refuses the package:
<q:alias name="h" expression="password hash(creds.password)"/>
[ERROR] SUTRA.FEEL.DETERMINISM.UNSAFE_BUILTIN — ... The FEEL expression on process 'sign-in' node 'Start' alias 'h' uses `password hash`, which is denied at replay-bound sites.
Limits
- Waits.
credsistransient, so it is gone after a wait, and lint refuses a read of it after one. Check the password before any wait. - Cost. Each call costs one argon2id hash on an executor worker. Rate-limit the sign-in channel if it is exposed.
- Unknown users. This flow assumes the row exists. A real one should answer
falsefor an unknown user too, after the same amount of work, so the answer time doesn’t reveal which users exist.
Recordings and media references
A process should not carry audio through its variables. A recording is large and personal, and once it is inside a snapshot, an audit row or a log it is copied everywhere the engine writes. In 0.6 the media stays in an object store, and the engine passes a media reference around in its place:
{ "kind": "media", "uri": "s3://recordings/calls/q-1.wav", "sha256": "ee6497…", "format": "audio/wav" }
Four pieces work together:
| Piece | What it does |
|---|---|
| Named object stores | Engine configuration: a bucket (and optional key prefix) the engine may read and delete in. |
| Media references in a model call | An audio, video or image part gives {ref, sha256} instead of data, and the llm channel reads the object from its media-store. |
q:audit capture="media" | Audit events carry the references found among the instance’s variables, never the media. |
| Erasure and retention | When an instance’s data has to go, the engine deletes the referenced objects and amends the audit events to say so. |
A media transport (a telephony transport, for example) typically writes the recording and hands the process its reference. In the walkthrough below you play that part by uploading a file.
Object stores
A store is declared in the engine configuration, never in a package, by any
sutra.object-stores.<name>.* key in the file, or by naming it in sutra.object-stores:
| Key | Default | Meaning |
|---|---|---|
sutra.object-stores | — | Comma-separated store names; the way to declare a store purely by environment variables. |
sutra.object-stores.<name>.uri | required | s3://<bucket>[/<prefix>] (or memory://<bucket> for tests: in process, nothing survives a restart). |
sutra.object-stores.<name>.endpoint | AWS | A self-hosted or non-AWS store’s base URL (http://store:9000), addressed path-style. |
sutra.object-stores.<name>.region | us-east-1 | Signing region. |
sutra.object-stores.<name>.access-key, .secret-key | required for s3:// | Secret references (env:, secret:, vault:, …), resolved at boot. |
sutra.object-stores.<name>.encryption | envelope | envelope: objects are sealed (encrypted at rest with a fresh data key wrapped under kek), as a recording transport writes them; none: stored as they are. |
sutra.object-stores.<name>.kek | required with envelope | Secret reference to the key-encryption secret the objects were sealed under. |
sutra.object-stores.<name>.key-id | recordings | The key id sealed objects name. |
Each key has an env var SUTRA_OBJECT_STORES_<NAME>_<FIELD>, with <NAME> upper-cased and -
written as _ (for example SUTRA_OBJECT_STORES_RECORDINGS_SECRET_KEY). A store that cannot be
built refuses startup. The client speaks the S3 API with Signature V4, so any S3-compatible store
works.
sutra.object-stores.recordings.uri=s3://recordings/calls
sutra.object-stores.recordings.endpoint=http://store:9000
sutra.object-stores.recordings.access-key=env:STORE_ACCESS_KEY
sutra.object-stores.recordings.secret-key=env:STORE_SECRET_KEY
sutra.object-stores.recordings.kek=secret:recordings-kek
Media references in a model call
Any audio, video (the clip, or each frame) or image part of an llm request may give
ref and sha256 instead of data:
{
"messages": [{ "role": "user", "content": "Summarise this recording in one sentence.",
"audio": { "ref": "s3://recordings/calls/q-1.wav",
"sha256": "ee649768a64c…", "format": "audio/wav" } }]
}
The channel names the one store it may read from:
llm:
media-store: recordings # sutra.object-stores.recordings
media-max-bytes: 1048576 # optional; 20 MiB by default
candidates: […]
Before the call is routed, the transport reads each referenced object. It checks the stored
bytes against sha256 and media-max-bytes, opens the object when it is sealed, and sends the
content to the provider as if it had been inline. The reply’s echoed request keeps the
reference, never the content.
| What went wrong | Result |
|---|---|
The channel has no media-store; the reference is outside that store (another store, another bucket, or a key outside its prefix); the object is missing or over the limit; the digest does not match; a sealed object does not open | permanent failure, SUTRA.LLM.MEDIA.REFUSED |
media-store names a store the engine configuration does not declare | SUTRA.LLM.CONFIG.INVALID when the package loads |
| The store cannot be reached | a retryable failure |
The audit trail carries references, never media
<q:audit capture="media"> on a process selects metadata-level audit events: no variable
snapshot. Each event instead carries, under media, every media reference found among the
instance’s variables (any object with kind: "media", a uri and a sha256, at any depth). A
variable marked sensitive is not searched. The capture also marks every request the process
sends (the sutra-audit-capture: media header), so a media transport knows to record the
session.
<bpmn:process id="recording-summary" isExecutable="true">
<bpmn:extensionElements>
<q:audit capture="media"/>
…
media is an additive value of the capture attribute, alongside none, metadata and
payload, and a package’s audit.capture accepts it too.
Erasure and retention delete the recording
When an instance’s data has to go, its recordings go with it:
- Subject erasure (
POST /admin/subjects/erase) deletes every object that the erased instances’ media events reference. - Terminal retention (
sutra.instance.retention) deletes them for the instances it purges: retained rows past the window, and finished instances whose row is already gone.
Each media event is then amended to say what was recorded and that it is gone:
{"media": [{"kind": "media", "uri": "…", "sha256": "…", "erased": true}],
"mediaErasure": {"at": "<time>", "reason": "subject-erasure | retention", "objects": 1}}
A recording is never skipped silently. If no declared store holds it, or the store refuses the
delete, erasure answers 500 with the references and changes nothing for that instance, so it
can be re-run once that is fixed. Retention keeps the instance and retries on its next sweep,
with a WARN in the log.
Walkthrough: a recording summarised, erased, and retired
The examples/media-recordings package has one flow, recording-summary.bpmn:
POST /channels/recording-summary { recordingId, callerId, recording: {kind, uri, sha256, format} }
BuildRequest requestBody := { messages: [{ content, audio: { ref, sha256, format } }] }
AskModel summary-model (sync): reads the object, checks digest and size, sends it as audio
BuildReply { summary, provider, audioPreserved }, answered at once (<q:reply continue="true">)
AwaitClose parks until POST /channels/recording-closed { recordingId }
End
callerId is declared <q:variable name="callerId" type="string" sensitive="true" subjectKey="true"/>, which makes it the data subject that erasure finds the instance by. The
model channel:
- name: summary-model
direction: outbound
transport: llm
bind: "llm://summary-model"
ack-mode: on-complete
llm:
media-store: recordings # sutra.object-stores.recordings (engine configuration)
media-max-bytes: 1048576 # refuse any referenced object over 1 MiB (default 20 MiB)
require: [audio-understanding] # only a candidate that hears the voice itself
candidates:
- provider: openai
model: gpt-audio-stub
base-url: http://model-stub:8000/v1
api-key: env:LLM_API_KEY # a real deployment: env:OPENAI_API_KEY, no base-url
timeout-seconds: 30
deploy/compose.yaml runs the engine, PostgreSQL, MinIO (pinned by digest) and the stub model
server. It sets on the engine:
| Setting | Why |
|---|---|
SUTRA_OBJECT_STORES=recordings, …_URI=s3://recordings/calls, …_ENDPOINT=http://store:9000, …_ACCESS_KEY=env:STORE_ACCESS_KEY, …_SECRET_KEY=env:STORE_SECRET_KEY, …_ENCRYPTION=none | the store; the sample uploads plain objects, where a recording transport would write sealed ones |
SUTRA_AUDIT_SQL=true | audit events go to the engine database, which GET /admin/instances/{id}/history reads |
SUTRA_CRYPTO_MASTER_KEY=env:RECORDINGS_MASTER_KEY (a dev-only placeholder) | blind-indexes subjectKey variables so erasure can find the instance; it also encrypts sensitive variables at rest. It is a secret reference like every other secret key (env:, secret:, vault:, …); a literal value is still used as the raw key, with a warning at boot |
SUTRA_INSTANCE_RETENTION=PT2M, SUTRA_INSTANCE_RETENTION_SWEEP_INTERVAL=PT15S | a finished instance is purged about two minutes after it ends, instead of the default seven days |
SUTRA_ADMIN_OIDC_DEV_DISABLED=true | /admin/* without authentication. Dev only. |
Prerequisites: Before you try a walkthrough. No key is needed.
1. Start the stack and deploy the package:
C=examples/media-recordings/deploy/compose.yaml
docker compose -f $C up -d --wait
sutra package examples/media-recordings/deployments-src/default--recordings--1.0.0 \
--out examples/media-recordings/deploy/deployments 2>/dev/null
PORT=$(docker compose -f $C port engine 8080 | cut -d: -f2)
H=(-H 'Content-Type: application/json' -H 'X-Api-Key: dev-only-recordings-key')
until curl -sf "http://127.0.0.1:$PORT/sutra/deployments" | grep -q '"phase":"Active"'; do sleep 1; done
packaged examples/media-recordings/deploy/deployments/default--recordings--1.0.0.sutra (deploymentId dep-4314895f6198e60f7bf51a20)
(sutra package also prints two warnings about the schema-less JSON channel, which you can
ignore here.)
2. Put two recordings in the store, as a recording transport would. The MinIO image carries
the mc client, already pointed at the server:
docker compose -f $C exec store mc mb local/recordings
for id in q-1 q-2; do
docker compose -f $C exec -T store mc pipe local/recordings/calls/$id.wav < examples/media-recordings/sample/question.wav
done
SHA=$(sha256sum examples/media-recordings/sample/question.wav | cut -d' ' -f1)
Bucket created successfully `local/recordings`.
0 B / ? 8044 bytes -> `local/recordings/calls/q-1.wav`
0 B / ? 8044 bytes -> `local/recordings/calls/q-2.wav`
3. Summarise the first recording by reference:
ask() { # ask <recordingId> <callerId> <sha256>
curl -s -w " HTTP %{http_code}\n" -X POST "http://127.0.0.1:$PORT/channels/recording-summary" "${H[@]}" \
-d "{\"recordingId\":\"$1\",\"callerId\":\"$2\",\"recording\":{\"kind\":\"media\",\"uri\":\"s3://recordings/calls/$1.wav\",\"sha256\":\"$3\",\"format\":\"audio/wav\"}}"
}
ask q-1 caller-1 "$SHA"
{"audioPreserved":true,"provider":"openai","summary":"STUB SUMMARY: the customer reports a problem and asks for it to be looked at."} HTTP 200
The engine read calls/q-1.wav, checked its digest, and sent it to the audio model. The instance
is now parked at AwaitClose.
4. A reference the store cannot honour is refused. There is no q-9.wav:
ask q-9 caller-9 "$SHA"
{"type":"urn:bpm:diag:SUTRA.LLM.MEDIA.REFUSED","title":"SUTRA.LLM.MEDIA.REFUSED","status":422,"detail":"'s3://recordings/calls/q-9.wav' does not exist","code":"SUTRA.LLM.MEDIA.REFUSED","attributes":{"incident":"SUTRA.INBOUND.NON_IDEMPOTENT_FAILURE"}} HTTP 422
The model call failed permanently with SUTRA.LLM.MEDIA.REFUSED, and the caller is told so, with
422: the request named an object that cannot be read, so sending it again cannot succeed. The
detail names the reference, never the store’s credentials or the object’s content. This process
does not declare itself idempotent, so the inbound is consumed and recorded as an incident instead
of being redelivered; incident carries that code. A wrong sha256 on an existing object is
refused the same way, with does not match its SHA-256.
5. Read the audit trail. Every event carries the reference, and none carries the audio:
curl -s "http://127.0.0.1:$PORT/admin/instances"; echo
ID=$(curl -s "http://127.0.0.1:$PORT/admin/instances" | python3 -c 'import json,sys; print([i["instanceId"] for i in json.load(sys.stdin)["instances"] if i["status"] == "SUSPENDED"][0])')
history() {
curl -s "http://127.0.0.1:$PORT/admin/instances/$1/history" | python3 -c '
import json, sys
for e in json.load(sys.stdin)["events"]:
print(e["seq"], e["eventType"], e["nodeId"] or "", json.dumps(e["payload"]) if e["payload"] else "")'
}
history "$ID"
{"instances":[{"instanceId":"f67c1b1e-4c95-4317-9f0f-6fb3ede9f5a2","deploymentId":"dep-4314895f6198e60f7bf51a20","status":"SUSPENDED"}]}
1 INSTANCE_STARTED
2 NODE_ENTERED Start {"media": [{"format": "audio/wav", "kind": "media", "sha256": "ee649768a64c8d99984c36213c629f4a5c84d10fac2d872275f2f11bcafa1e4c", "uri": "s3://recordings/calls/q-1.wav"}]}
3 NODE_LEFT Start
4 NODE_ENTERED BuildRequest {"media": [{"format": "audio/wav", "kind": "media", "sha256": "ee649768a64c8d99984c36213c629f4a5c84d10fac2d872275f2f11bcafa1e4c", "uri": "s3://recordings/calls/q-1.wav"}]}
5 NODE_LEFT BuildRequest
6 NODE_ENTERED AskModel {"media": [{"format": "audio/wav", "kind": "media", "sha256": "ee649768a64c8d99984c36213c629f4a5c84d10fac2d872275f2f11bcafa1e4c", "uri": "s3://recordings/calls/q-1.wav"}]}
7 INSTANCE_SUSPENDED
8 INSTANCE_RESUMED
9 NODE_LEFT AskModel
10 NODE_ENTERED BuildReply {"media": [{"format": "audio/wav", "kind": "media", "sha256": "ee649768a64c8d99984c36213c629f4a5c84d10fac2d872275f2f11bcafa1e4c", "uri": "s3://recordings/calls/q-1.wav"}]}
11 NODE_LEFT BuildReply
12 NODE_ENTERED AwaitClose {"media": [{"format": "audio/wav", "kind": "media", "sha256": "ee649768a64c8d99984c36213c629f4a5c84d10fac2d872275f2f11bcafa1e4c", "uri": "s3://recordings/calls/q-1.wav"}]}
13 INSTANCE_SUSPENDED
The refused instance is not listed: /admin/instances lists live instances. Inspecting the
parked one shows its variables, with callerId redacted: it is sensitive, encrypted at rest,
and the endpoint decrypts it only to apply the same redaction as everywhere else:
curl -s "http://127.0.0.1:$PORT/admin/instances/$ID" | python3 -c 'import json,sys; r=json.load(sys.stdin); print(r["status"], r["sensitive"], r["variables"]["callerId"])'
SUSPENDED ['callerId'] ***REDACTED***
6. Erase the data subject caller-1. keyId is the tenant and subjectName is the
subjectKey variable. With no deploymentId, erasure covers every live deployment of the
tenant, wherever it was deployed from (a watched directory, as here, or the database):
curl -s -X POST "http://127.0.0.1:$PORT/admin/subjects/erase" -H 'Content-Type: application/json' \
-d '{"keyId":"default","subjectName":"callerId","value":"caller-1"}'; echo
docker compose -f $C exec store mc ls -r local/recordings
history "$ID" | sed -n 2p
{"subjectName":"callerId","tenantWide":true,"deploymentsScanned":1,"erased":true,"matched":1,"erasedCount":1,"instanceIds":["f67c1b1e-4c95-4317-9f0f-6fb3ede9f5a2"]}
[2026-10-26 06:52:26 UTC] 7.9KiB STANDARD calls/q-2.wav
2 NODE_ENTERED Start {"media": [{"kind": "media", "uri": "s3://recordings/calls/q-1.wav", "erased": true, "sha256": "ee649768a64c8d99984c36213c629f4a5c84d10fac2d872275f2f11bcafa1e4c"}], "mediaErasure": {"at": "2026-10-26T06:52:26.726921625Z", "reason": "subject-erasure", "objects": 1}}
calls/q-1.wav is gone from the store, and the audit events still say what was recorded, now
marked erased.
7. Let retention retire the second recording. Summarise q-2 and close it, so the instance
completes:
ask q-2 caller-2 "$SHA"
curl -s -w "HTTP %{http_code}\n" -X POST "http://127.0.0.1:$PORT/channels/recording-closed" "${H[@]}" -d '{"recordingId":"q-2"}'
{"audioPreserved":true,"provider":"openai","summary":"STUB SUMMARY: the customer reports a problem and asks for it to be looked at."} HTTP 200
HTTP 200
Then wait for the purge. With a two-minute retention and a 15-second sweep it takes about two and a half minutes:
until [ -z "$(docker compose -f $C exec -T store mc ls local/recordings/calls/q-2.wav)" ]; do sleep 5; done
docker compose -f $C logs engine | grep -o 'terminal retention deleted the recorded media[^"]*'
docker compose -f $C exec -T engine-db psql -U postgres -d sutra -At \
-c "select seq, event_type, payload_json from audit_event where payload_json like '%q-2.wav%' order by seq limit 1"
terminal retention deleted the recorded media of finished instances
2|NODE_ENTERED|{"media":[{"kind":"media","uri":"s3://recordings/calls/q-2.wav","erased":true,"sha256":"ee649768a64c8d99984c36213c629f4a5c84d10fac2d872275f2f11bcafa1e4c"}],"mediaErasure":{"at":"2026-10-26T06:54:34.352446787Z","reason":"retention","objects":1}}
The loop ends when mc ls no longer lists the object. The purged instance’s audit rows stay,
amended with "reason":"retention".
8. Clean up:
docker compose -p sutra-media-recordings down -v
Next
- Calling a language model: audio, images and video in model calls.
- Logging and audit: the audit sinks and capture levels.
- Configuration reference: every
sutra.*key.
Wait states and human tasks
Sutra’s wait state is an integration point that waits for an external relay decision — not a human-task management system. When a token reaches one, the engine suspends the instance (persists it, frees the thread); the flow continues only when an external actor — a human via a console you build, or another system — relays a decision back through a channel as a typed message. The engine rehydrates and runs to completion.
| Sutra (the engine) provides | You own |
|---|---|
| The wait point + durable suspend / rehydrate / resume | Human-task management: assignment, queues, claim, escalation, SLA |
| A relay channel (typed message in) | Forms and any UI |
| The held instance variables, carried verbatim | Identity / authentication / authorization (who may relay) |
| Compliance-grade audit of the wait and the relayed decision | Notifications, dashboards |
That split is deliberate: full human-task management and forms are a much larger scope than a workflow engine’s core job, and Sutra doesn’t try to own them.
The construct: an intermediate message catch event
The honest BPMN construct is the intermediate message catch event — “pause until message M
arrives, then continue.” A userTask is the human-facing flavor of the exact same machinery, for
authors who think in terms of “a human decides here” — it carries no assignee/group/queue model in
the engine; if present, those are just opaque hints in instance variables.
An intake node for a wait declares a channel exactly like a start event’s q:source — the relayed
message goes through the same two-tier validation pipeline (structural codec, then business
validators) that every inbound message does. The difference is only in what happens after:
| Outcome | Start event (new instance) | Wait state (relay) |
|---|---|---|
| Structurally invalid | No instance minted; error to caller | Relay rejected; the instance stays parked, unchanged |
| Soft errors | Minted and run; surfaced as variables | Rehydrated and resumed; the gateway after the wait routes on the soft errors |
| Valid | Minted and run | Rehydrated and resumed |
The load-bearing property: the wait is the safe state. A hard-invalid relay can never advance or corrupt a parked instance — it is rejected and the instance sits exactly where it was. The relayer fixes the message and retries.
Correlation by a business key you name
<q:alias name="e2eId" expression="payload.E2EId" unique="true" onConflict="correlate"/>
The relayed message doesn’t carry an engine-internal instance id — it carries a business value
(an EndToEndId, an order reference, whatever your domain already uses), and the process declares a
q:alias deriving that key from its own payload. The engine indexes it durably, on PostgreSQL, so
correlation survives a restart and works identically across every replica. See
The q: namespace for the full q:alias shape.
A relay is matched by the start event’s alias even when the wait binds its message under a name of
its own (<q:source channel="revision" name="revision"/>): while the alias is evaluated, the
relayed message is also visible under the start event’s payload name, so payload.E2EId reads the
relayed E2EId. The name a wait gives its payload is local to that wait. It is what the steps
after it read (revision.note), and it does not change which instance a message finds.
sequenceDiagram
participant A as Requester
participant E as Engine
participant DB as PostgreSQL
participant R as Relayer
A->>E: ApprovalRequest on approval-request
E->>DB: park — snapshot plus alias e2eId from payload.E2EId
Note over E,DB: durable commit — parked, correlatable, thread freed
E-->>A: accepted, no business reply yet
R->>E: ApprovalDecision on approval-decision
E->>E: structural codec, then business validators
alt structurally invalid
E-->>R: relay rejected — the instance stays parked, unchanged
else valid, or soft errors only
E->>DB: correlate by e2eId, rehydrate typed variables
E->>E: the gateway after the wait routes on the soft errors
E->>DB: run to the end — alias retired
end
The park is a commit, not a pause: the correlation key is written in the same durable step that suspends the instance, so a relay arriving later — on a different channel, against a different replica — finds it by the key you named. And because a hard-invalid relay is refused before anything is rehydrated, the parked instance is never the thing that pays for a bad message.
An answer that arrives before its wait
Since 0.6.6. Between co-deployed processes an answer can be faster than the process that asked.
A local:// delivery goes out the moment the sender’s step commits, so when a process sends a
request at one park and reaches the wait for the answer only one step later — a
q:reply continue="true", a timer, or any other wait in between — the answer can reach the
instance before it has parked there. Its wait key is not written yet, so the engine refuses the
answer (SUTRA.RUNTIME.RELAY.CORRELATION_NOT_FOUND, or SUTRA.RUNTIME.RESUME.NOT_WAITING when the
instance is parked at another node), having run nothing.
The engine marks such a refusal wait-pending (sutra.waitPending=true on the diagnostic), and a
local:// delivery that gets one is held, not dropped: it is offered again after a few
milliseconds (155 ms in all, about one step), and then retried with the outbox backoff for up to
8 attempts — about four minutes. If the instance still has not parked by then, the delivery takes
the outbox’s usual poison posture: kept with its last error and redelivered at the poison horizon,
so nothing is lost. Since 0.7.0 an answer for an instance that has finished is not wait-pending: when every
instance that carried the answer’s key has ended, the refusal carries no sutra.waitPending
and its detail says the instance has ended. The engine can tell while the finished instance is
retained (sutra.instance.retention, seven days by default). Since 0.6.7 a
local:// delivery whose step failed because the database was unreachable (a reset connection, a
restart) is retried the same way, without the bound, instead of being parked at the poison horizon. A message
from outside the engine (HTTP, a broker) keeps the transport’s own handling, because the engine
cannot tell a fast answer from a stale one there; send it again, or make the wait come first.
To watch it, run the conformance test that builds the race on purpose. Its package
(rust/crates/sutra-conformance/fixtures/wait-race/) sends a request to a second process, waits two
seconds on a timer, and only then parks at the wait for the answer, which the second process sends
at once. It needs Docker and the engine image built from your checkout (make image):
cd rust && cargo test -p sutra-conformance --test all tc_wait_race -- --ignored --nocapture
test tc_wait_race::tc_wait_race_an_answer_that_beats_the_wait_is_held_until_the_wait_parks ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 46 filtered out; finished in 8.85s
All 20 answers arrive while their instances sit at the timer, and all 20 instances complete, each
called back once. On a 0.6.5 engine the same test fails after 90 seconds with every answer still in
the outbox, refused with SUTRA.RUNTIME.RELAY.CORRELATION_NOT_FOUND and parked at the poison horizon.
An answer no instance waits for
Since 0.7.0. An answer that correlates to no live instance (a reply after its instance ended,
a key nothing ever carried) is refused with SUTRA.RUNTIME.RELAY.CORRELATION_NOT_FOUND, and
recorded whatever transport carried it:
- a WARN log line with the code
SUTRA.INBOUND.RELAY_UNCORRELATED, naming the channel, the message type, the dedup key and the correlation keys looked for (keys=orderKey=B-9); never the body or the headers; - an
inbound.RELAY_UNCORRELATEDrecord in the deployment’s audit journal, with no instance id. Itsdiagnostic_jsonholdschannel,deployment,messageType,dedupKey,keys(each withprocess,node,alias,value,ended),endedandoutcome.
A key value is masked in both when the alias, or a variable its expression reads, is sensitive,
or when a redactor masks its input.
A package can take such an answer instead of refusing it, with no BPMN change, by naming a
process on the channel in channels.yaml:
channels:
- name: order-confirm
transport: http
bind: "POST /orders/confirm"
codec: json
uncorrelated:
start: late-confirmation # a process of this package with a start event on order-confirm
That process’s start events on the channel are reserved for answers no wait correlates:
ordinary start resolution passes them over, so an answer that correlates still resumes its
instance and starts nothing. Its <q:source> still decides which uncorrelated answers it takes.
The started instance reads event.uncorrelated: code, keys (alias name to the raw value
looked for) and ended. The audit record then says outcome: started, with startedProcess
and instanceId.
To watch both, run the conformance suite (Docker, and the engine image built from your checkout
with make image):
cd rust && cargo test -p sutra-conformance --test all tc_uncorrelated_relay -- --ignored --nocapture
test tc_uncorrelated_relay::tc_uncorrelated_relay_a_refused_answer_is_recorded_and_wait_pending_only_while_it_may_park ... ok
test tc_uncorrelated_relay::tc_uncorrelated_relay_a_routed_answer_starts_the_named_process_and_correlation_is_unaffected ... ok
Variables survive the wait typed
The instance variables held across a wait keep their real types. A number comes back a number, a boolean a boolean, a list a list, a context a context, a date/time/duration its own temporal type — and a null comes back null, not an empty string.
This matters because FEEL, correctly, does not coerce. A gateway condition after a wait therefore evaluates against the value’s real type:
| After the wait | Evaluates as |
|---|---|
amount > 100 where amount parked as 1250.75 | a number comparison — true |
not(approved) where approved parked as false | a boolean negation — true |
cancelledAt = null where nothing was ever set | true |
Two things are deliberately unchanged. First, the inspect projection is not affected: every
variable still renders through the display form it always had on GET /admin/instances/{id}, so
what an operator’s response looks like did not move — typing changed what survives a wait, not
what is shown. (A null now renders as null rather than blank, which is the one visible
difference, and it is the correct one.) Second, the correlation key input is not affected:
a subject blind-index value keeps hashing the exact string it always hashed.
Two value kinds still persist as their canonical string, because neither is really instance state: a FEEL function closes over an evaluation context that ceases to exist the moment the instance parks, and a range is a comparison shape rather than a value. Both behaved this way before and still do.
The design reasoning — the encoding, the compatibility posture, and why encrypted values are typed too — is in the internals chapter.
Waiting at the same node again: loops
Since 0.6, a process instance can wait at the same node any number of times. A loop that brings
the token back to a message catch, a receive task, a user task, a channel call (async or
synchronous), a timer catch or an event-based gateway parks a new wait on each pass, with its
own wait row, its own timer and <q:timeout> rows, its own <q:retry> budget and its own
<q:alias> rows. The step that answers a pass resolves that pass’s rows and retires its
correlation keys, so a late message for a past pass resumes nothing. Before 0.6 the token found
the node already done and ran past it, or the instance could not park there again.
Behind this, a resume continues from the satisfied node itself instead of replaying the graph from the start event. Nothing upstream runs again, other parked waits carry over untouched, and a converging gateway regains its earlier arrivals from a side-effect-free walk over the completed nodes. The snapshot format is unchanged, so instances parked by an earlier engine resume under the new rules.
Related rules that came with it:
- A message that correlates to an instance but names a wait the instance is not parked at is
refused before anything runs, with
SUTRA.RUNTIME.RESUME.NOT_WAITING. - A wait node’s own
<q:alias>(on a message catch, receive task or user task) is indexed when the node parks, so a message correlates by it. - An event-based gateway continues along the arm that fired when the instance resumes.
- A parallel join inside a loop waits for a full set of tokens each time round.
- A message on a channel that several wait nodes of one process subscribe to resumes the node the instance is parked at.
Walkthrough: a review loop
The examples/review-loop package sends a draft round review, revision and a short cool-off
until a reviewer approves it:
POST /channels/drafts { draftId } rounds := 0
AwaitReview user task, answered on POST /channels/drafts/review { draftId, decision }
Count rounds := rounds + 1
Decide review.decision = "approve" -> End
otherwise -> AwaitRevision (message catch, POST /channels/drafts/revision)
-> CoolOff (timer catch, PT2S) -> back to AwaitReview
The loop is ordinary BPMN: AwaitReview has two incoming flows, one from the start and one back
from CoolOff:
<bpmn:userTask id="AwaitReview" name="Review the draft">
<bpmn:extensionElements>
<q:source channel="review" name="review"/>
</bpmn:extensionElements>
<bpmn:incoming>F2</bpmn:incoming><bpmn:incoming>F8</bpmn:incoming>
<bpmn:outgoing>F3</bpmn:outgoing>
</bpmn:userTask>
…
<bpmn:intermediateCatchEvent id="CoolOff" name="Cool-off, 2 s">
<bpmn:incoming>F7</bpmn:incoming><bpmn:outgoing>F8</bpmn:outgoing>
<bpmn:timerEventDefinition id="CoolOff_Timer">
<bpmn:timeDuration>PT2S</bpmn:timeDuration>
</bpmn:timerEventDefinition>
</bpmn:intermediateCatchEvent>
<bpmn:sequenceFlow id="F8" sourceRef="CoolOff" targetRef="AwaitReview"/>
Every message after the first finds the instance through the start event’s
<q:alias name="draftId" expression="payload.draftId" unique="true" onConflict="correlate"/>.
The waits bind their messages under names of their own (name="review", name="revision"), and
Decide reads review.decision; the alias still reads payload.draftId, because a wait’s name is
local to it. The process declares <q:audit capture="metadata"/>, so each step is in the
instance’s audit history.
Prerequisites: Before you try a walkthrough.
1. Start the stack and deploy the package. The waits helper lists the instance’s wait rows
from the engine database:
C=examples/review-loop/deploy/compose.yaml
docker compose -f $C up -d --wait
sutra package examples/review-loop/deployments-src/default--review--1.0.0 \
--out examples/review-loop/deploy/deployments 2>/dev/null
PORT=$(docker compose -f $C port engine 8080 | cut -d: -f2)
H=(-H 'Content-Type: application/json' -H 'X-Api-Key: dev-only-review-key')
until curl -sf "http://127.0.0.1:$PORT/sutra/deployments" | grep -q '"phase":"Active"'; do sleep 1; done
send() { curl -s -w " HTTP %{http_code}\n" -X POST "http://127.0.0.1:$PORT/channels/drafts$1" "${H[@]}" -d "$2"; }
waits() {
docker compose -f $C exec -T engine-db psql -U postgres -d sutra -At -c \
"select node_id, kind, status from waiting_event order by created_at"
}
packaged examples/review-loop/deploy/deployments/default--review--1.0.0.sutra (deploymentId dep-84173f6525975bed01d706ee)
2. Submit a draft. It parks at the review:
send "" '{"draftId":"d-1"}'
ID=$(curl -s "http://127.0.0.1:$PORT/admin/instances/by-alias/draftId/d-1" | python3 -c 'import json,sys; print(json.load(sys.stdin)["instanceId"])')
waits
HTTP 200
AwaitReview|MESSAGE|WAITING
3. Round one: ask for a revision, submit it, and let the cool-off run:
send /review '{"draftId":"d-1","decision":"revise"}'
waits
send /revision '{"draftId":"d-1"}'
waits
sleep 4
waits
HTTP 200
AwaitReview|MESSAGE|RESOLVED
AwaitRevision|MESSAGE|WAITING
HTTP 200
AwaitReview|MESSAGE|RESOLVED
AwaitRevision|MESSAGE|RESOLVED
CoolOff|TIMER|WAITING
AwaitReview|MESSAGE|WAITING
AwaitRevision|MESSAGE|RESOLVED
CoolOff|TIMER|RESOLVED
After the timer fired, the token came back to AwaitReview, and its wait is armed again
(WAITING) for round two.
4. Round two, and a message for a wait the instance has left. The second review asks for another revision. A third review arrives while the instance waits for that revision, so it is refused and nothing runs:
send /review '{"draftId":"d-1","decision":"revise"}'
send /review '{"draftId":"d-1","decision":"approve"}'
HTTP 200
{"type":"urn:bpm:diag:SUTRA.RUNTIME.RESUME.NOT_WAITING","title":"SUTRA.RUNTIME.RESUME.NOT_WAITING","status":500,"detail":"Instance 43a748ee-92ef-4845-bf59-10492c4b45ee is not waiting at 'AwaitReview' (it is parked at [\"AwaitRevision\"]); this relay on channel 'review' arrived for a wait the instance already left and is refused — nothing was executed.","code":"SUTRA.RUNTIME.RESUME.NOT_WAITING","attributes":{"instanceId":"43a748ee-92ef-4845-bf59-10492c4b45ee","nodeId":"AwaitReview"}} HTTP 500
5. Finish round two, then approve in round three:
send /revision '{"draftId":"d-1"}'
sleep 4
curl -s "http://127.0.0.1:$PORT/admin/instances/$ID" | python3 -c 'import json,sys; r=json.load(sys.stdin); print(r["status"], r["variables"]["rounds"])'
send /review '{"draftId":"d-1","decision":"approve"}'
curl -s "http://127.0.0.1:$PORT/admin/instances"; echo
HTTP 200
SUSPENDED 2
HTTP 200
{"instances":[]}
Before the last review, the instance was parked at AwaitReview for the third time with
rounds = 2. The approval ended it, so no instance is live any more.
6. Read the audit history. The journal is written asynchronously, so the loop waits for the
completion event. Each event’s ordinal is its position in the journal, which is what the history
pages by:
until curl -s "http://127.0.0.1:$PORT/admin/instances/$ID/history" | grep -q INSTANCE_COMPLETED; do sleep 1; done
curl -s "http://127.0.0.1:$PORT/admin/instances/$ID/history" | python3 -c '
import json, sys
for e in json.load(sys.stdin)["events"]:
print(e["ordinal"], e["eventType"], e["nodeId"] or "")'
1 INSTANCE_STARTED
2 NODE_ENTERED Start
3 NODE_LEFT Start
4 NODE_ENTERED Init
5 NODE_LEFT Init
6 NODE_ENTERED AwaitReview
7 INSTANCE_SUSPENDED
8 INSTANCE_RESUMED
9 NODE_LEFT AwaitReview
10 NODE_ENTERED Count
11 NODE_LEFT Count
12 NODE_ENTERED Decide
13 NODE_LEFT Decide
14 NODE_ENTERED AwaitRevision
15 INSTANCE_SUSPENDED
16 INSTANCE_RESUMED
17 NODE_LEFT AwaitRevision
18 NODE_ENTERED CoolOff
19 INSTANCE_SUSPENDED
20 INSTANCE_RESUMED
21 NODE_LEFT CoolOff
22 NODE_ENTERED AwaitReview
23 INSTANCE_SUSPENDED
24 INSTANCE_RESUMED
25 NODE_LEFT AwaitReview
26 NODE_ENTERED Count
27 NODE_LEFT Count
28 NODE_ENTERED Decide
29 NODE_LEFT Decide
30 NODE_ENTERED AwaitRevision
31 INSTANCE_SUSPENDED
32 INSTANCE_RESUMED
33 NODE_LEFT AwaitRevision
34 NODE_ENTERED CoolOff
35 INSTANCE_SUSPENDED
36 INSTANCE_RESUMED
37 NODE_LEFT CoolOff
38 NODE_ENTERED AwaitReview
39 INSTANCE_SUSPENDED
40 INSTANCE_RESUMED
41 NODE_LEFT AwaitReview
42 NODE_ENTERED Count
43 NODE_LEFT Count
44 NODE_ENTERED Decide
45 NODE_LEFT Decide
46 NODE_ENTERED End
47 NODE_LEFT End
48 INSTANCE_COMPLETED
Each round reads the same way: the resume leaves the wait it came through (NODE_LEFT AwaitReview, NODE_LEFT AwaitRevision, NODE_LEFT CoolOff), and Count and Decide run once
per round. The last round leaves Decide for End.
7. Clean up:
docker compose -p sutra-review-loop down -v
Worked example: approval-hold
examples/approval-hold is the reference wait-state flow: a startEvent on channel
approval-request records the correlation alias (e2eId = payload.E2EId, unique,
onConflict="correlate"), then a userTask on channel approval-decision — the hold — then an
endEvent. Its bundled conformance test drives the full lifecycle against real PostgreSQL:
- Park — an
ApprovalRequestonapproval-requestreturns200(accept, no business reply yet); the instance parks and its alias is recorded. - Duplicate rejected — a second
ApprovalRequestwith the sameE2EIdwhile parked is rejected (the durable unique-alias guard). - Relay resume — an
ApprovalDecisionon the separateapproval-decisionchannel — a channel no start event subscribes to — is routed by the dispatcher’s relay path: the same two-tier intake, against the wait node’sq:source, then correlated byE2EId, then resumed. - Retire — the same
E2EIdis accepted again afterward, proving the alias was retired once the instance completed. - Uncorrelated relay is safe — a decision for an unknown
E2EIdis rejected and every parked instance is left untouched.
Try it by hand once the package is deployed (see Your first deployment):
# park
curl -sS -X POST localhost:8080/channels/approval-request \
-H 'Content-Type: application/json' -H 'X-Api-Key: approval-demo-key' \
-d '{"ApprovalRequest":{"E2EId":"E2E-1","Amount":"1500.00"}}'
# resume it with a decision
curl -sS -X POST localhost:8080/channels/approval-decision \
-H 'Content-Type: application/json' -H 'X-Api-Key: approval-demo-key' \
-d '{"ApprovalDecision":{"E2EId":"E2E-1","Decision":"APPROVE"}}'
Run the scenario itself with cd rust && cargo test -p sutra-conformance -- --ignored tc_approval_hold
(needs Docker — see Test tiers).
Next
- Worked example: money-transfer — the other flagship example, showing durable data stores instead of a wait state.
- External tasks — the pull flavour of the same park: a delivery parked for a worker to fetch rather than dialed out.
- Replica semantics — how correlation and suspend/resume stay correct across a multi-replica engine.
- Durable execution — what a park actually persists, and why it is a snapshot rather than an event log.
One endpoint, one deployment
Several deployments can share one engine, but most inbound endpoints hand each message to a single receiver. If two deployments claimed one, the second would either never receive anything or silently compete with the first for messages. So since 0.6.18 the engine refuses the second claim at deploy, with the first deployment still serving.
| Endpoint | Who can hold it |
|---|---|
HTTP METHOD path | one deployment |
| Queue (RabbitMQ, SQS, AMQP 1.0) | one deployment |
| Pub/Sub subscription | one deployment |
| Kafka consumer group, on any topic | one deployment |
| File spool directory | one deployment |
| LLM callback route | one deployment |
| A channel’s name, under a package’s tenant, module and version | one deployment |
| Topic (Dapr, Knative subscription, AMQP 1.0) | any number; each receives its own copy |
Two deployments can read one Kafka topic if they use different group.id values, and one Pub/Sub
topic through different subscriptions. Each then gets every message. An unset group.id defaults from
the channel name, so the same channel in two deployments lands in one group and is refused.
This walkthrough deploys two packages that want the same route, then fixes the second.
What was and was not run. The walkthrough ran an HTTP route end to end on the db deployment source.
RabbitMQ was run end to end too. Claim identities for Kafka, SQS, Pub/Sub, AMQP and file have unit tests with
no broker behind them. Dapr fan-out was driven by a direct POST with no sidecar, Knative fan-out has unit tests
only, and the advisory lock has no concurrent-replica race test.
Before you start
You need a running engine on the db deployment source, with a database role that can’t bypass
row-level security (the engine refuses to boot on a superuser or a BYPASSRLS role with
SUTRA.STARTUP.RLS_BYPASS_RISK), and an admin gate, or the admin surface answers 503. This page uses a key:
SUTRA_ADMIN_AUTH_SCHEME=apikey and SUTRA_ADMIN_AUTH_KEY_REF=env:<a variable holding the key> on the engine,
and SUTRA_ADMIN_API_KEY for the CLI. See Configuration reference.
You also need two package directories, orders/ and notes/, each with a one-step flow that answers a
request. They differ only in the module name. This is the part that matters, channels.yaml, in both (the
channel is notes-in in notes/):
channels:
- name: orders-in
transport: http
bind: "POST /orders"
auth:
scheme: apikey
apikey:
value: env:ORDERS_KEY
Both also declare a data store on one connection, datastores.yaml:
datastores:
- name: accounts
type: sql
sql:
url-ref: env:SHARED_DB_URL
1. Deploy the first package
$ sutra package orders --out out-orders
packaged out-orders/orders.sutra (deploymentId dep-7a4cb8e71418102e546ce735)
$ sutra deploy out-orders/orders.sutra --api --engine-url http://localhost:8080
deployed dep-7a4cb8e71418102e546ce735 (slot acme--orders--1.0.0, rev 1) — Active
$ curl -X POST http://localhost:8080/orders -H 'X-API-Key: <your key>' -H 'Content-Type: application/json' -d '{}'
hello from orders
2. Check the second package before deploying it
sutra lint runs the deploy’s check when you give it a target. --engine-url asks the running engine:
$ sutra lint notes --engine-url http://localhost:8080
[ERROR] SUTRA.DEPLOY.ENDPOINT.COLLISION — http endpoint 'POST /orders' is claimed by channel 'orders-in' of deployment acme--orders--1.0.0 and by channel 'notes-in' of deployment acme--notes--1.0.0; it can belong to only one deployment
[WARN] SUTRA.DEPLOY.DATASTORE.SHARED_CONNECTION — deployment acme--notes--1.0.0 and deployment acme--orders--1.0.0 use the same connection for data store(s) accounts, so they share those stores' rows. If that is not intended, give this deployment its own connection in its datastores.yaml
1 error(s), 2 warning(s)
The count includes a schema warning about these minimal packages, left out here. The call needs the admin
surface, so pass --api-key if your engine gates it. With no engine to reach, --active-set takes sealed
archives, or a directory of them, instead. That form runs inside the CLI, which links only the HTTP transport,
so it reports a channel on any other transport as SUTRA.DEPLOY.ENDPOINT.UNCHECKED rather than judging it.
See the CLI reference.
3. Try the deploy anyway
$ sutra package notes --out out-notes
$ sutra deploy out-notes/notes.sutra --api --engine-url http://localhost:8080
[ERROR] SUTRA.DEPLOY.ENDPOINT.COLLISION — engine rejected the deployment (HTTP 409 Conflict): ...
$ echo $?
1
Nothing changed. The route still answers from the first deployment, and the second was not stored:
$ curl -X POST http://localhost:8080/orders -H 'X-API-Key: <your key>' -H 'Content-Type: application/json' -d '{}'
hello from orders
$ curl http://localhost:8080/sutra/deployments
{"active":[{"deploymentId":"dep-7a4cb8e71418102e546ce735","slot":"acme--orders--1.0.0","phase":"Active","ready":true}],"pending":[],"draining":[],"failed":[]}
4. Give the second package its own route
Change the bind in notes/channels.yaml to POST /notes, then package and deploy again:
$ sutra deploy out-notes/notes.sutra --api --engine-url http://localhost:8080
[WARN] SUTRA.DEPLOY.DATASTORE.SHARED_CONNECTION — deployment acme--notes--1.0.0 and deployment acme--orders--1.0.0 use the same connection for data store(s) accounts, so they share those stores' rows. If that is not intended, give this deployment its own connection in its datastores.yaml
deployed dep-b7a5226eb282e88b23a075b7 (slot acme--notes--1.0.0, rev 1) — Active
The deploy went through, and the warning stayed. Both deployments declare a store named accounts on one
connection, and rows are keyed by store name and key, so they read and write each other’s rows. Sharing a
connection is sometimes right, so this never refuses a deploy. If you didn’t mean it, give each deployment
its own connection in its datastores.yaml. The warning names the other deployment and the stores, never the
connection. sutra lint with no target also raises it when a sql store’s connection reference names
neither the tenant nor the module, because any co-hosted copy of the package would share it.
How the other deployment paths behave
A hot deploy into the same slot is not a collision. New content for acme--orders--1.0.0 replaces it.
A new version takes over from an old one. Versions of one tenant and module count as one owner, so
acme--orders--2.0.0 on 1.0.0’s route is accepted, and SUTRA.DEPLOY.ENDPOINT.SUPERSEDED warns. New requests
reach 2.0.0, and 1.0.0 drains. Since 0.6.20 the old version keeps serving the channels only it declares until it retires, and another owner
is refused those endpoints meanwhile: see Upgrading a deployment in place. Before 0.6.20 it
drained whole: every channel of 1.0.0 stopped taking new work, including one that 2.0.0 doesn’t have.
A package with no tenant or module label owns only itself. It can’t take a route, or a channel
name, from anything. Moving it to a new route is refused until you undeploy the old one. Label the package and
new versions replace old ones for you.
A folder source (sutra.deployments.dir). A refused archive is listed under failed in
GET /sutra/deployments, and the engine serves the rest. File names and arrival order decide nothing:
- The holder keeps the endpoint across restarts and on new pods. The engine records it in its database, so every replica reads the same answer.
- Two archives on one endpoint with no record of a holder both report
Failedand neither deploys. Remove one and the other lands. With no database there is no record, so this is also what a restart gives you if a refused file is still there. - The highest version of one tenant and module wins, comparing versions rather than text (
1.10.0beats1.9.0). To roll back, remove the newer file. - A pod that can’t verify the holder’s file, because it is damaged or a secret is missing there, doesn’t give the endpoint away. It passes on only when the holder’s file is gone.
sutra deploy to a ConfigMap. The CLI judges the archive against the ConfigMap’s other entries and
refuses before patching it. Add --engine-url to have the engine judge it too, which covers every transport.
References compare as written
A claim is computed where no secrets are to hand, including in the CLI, and its text appears in refusal messages and in the
deployment_endpoint_holder table. So it holds a reference as you wrote it, never its value. Two deployments on one queue
through ${A}/q and ${B}/q with the same value are not caught, and neither are ${A}/q and the literal.
Upgrading
-
Two deployments already on one queue. The one activated first keeps it. The later one stops being active and is listed under
faileduntil the next deployment change. Its running instances keep their definition. -
Kafka groups are different. Deployments already sharing a consumer group keep running, with
SUTRA.DEPLOY.ENDPOINT.ALREADY_SHAREDlogged as an error. Any new deploy claiming that group is refused, including new content for one of them. Give each channel its owngroup.idthe next time you change it, and setauto.offset.resetfor what a new group should read first, because it starts from its own offsets:channels: - name: orders-in transport: kafka bootstrap.servers: kafka:9092 topic: orders group.id: orders-consumers -
Topics. Dapr and Knative channels on one topic now all receive each event, where a second claim used to be refused. The engine creates a
deployment_endpoint_holdertable at boot.
A reply answers the instance waiting for it
From 0.6.22, when a message arrives on a channel, the engine first asks whether it answers an instance that is parked waiting for it, and only then whether it starts a new one. This holds with one version of a package and across versions while an older one drains:
- A message on a channel with both a start event and a wait resumes the instance it correlates to, and starts nothing. An uncorrelated message starts, as before.
- While version 1 drains, a reply for one of its parked instances resumes that instance on version 1, even when version 2 has a start event on the channel, and it is processed with version 1’s own codec, validators and reply.
- A retry of a message that ran on version 1 is a duplicate on version 2.
This walkthrough shows the first with one package; the tier-2 suite covers the other two. It needs the setup of
One endpoint, one deployment: an engine on the db
deployment source, a database role that can’t bypass row-level security, and an admin key.
The package
orders/bpmn/flow.bpmn has one process. It starts on order-in, keeps the order’s key as its
correlation alias, and parks on order-in again until a message with the same key arrives:
<?xml version="1.0" encoding="UTF-8"?>
<bpmn:definitions xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL"
xmlns:q="urn:sutra:q:1.0" id="Definitions_orders"
targetNamespace="urn:sutra:module:orders:1.0.0">
<bpmn:process id="order" name="Order" isExecutable="true">
<bpmn:startEvent id="Received">
<bpmn:extensionElements>
<q:source channel="order-in"/>
<q:alias name="orderKey" expression="payload.key" unique="true" onConflict="correlate"/>
</bpmn:extensionElements>
<bpmn:outgoing>f1</bpmn:outgoing>
</bpmn:startEvent>
<bpmn:sequenceFlow id="f1" sourceRef="Received" targetRef="AwaitConfirmation"/>
<bpmn:receiveTask id="AwaitConfirmation" name="Await confirmation">
<bpmn:extensionElements>
<q:source channel="order-in" name="confirmation"/>
</bpmn:extensionElements>
<bpmn:incoming>f1</bpmn:incoming><bpmn:outgoing>f2</bpmn:outgoing>
</bpmn:receiveTask>
<bpmn:sequenceFlow id="f2" sourceRef="AwaitConfirmation" targetRef="Done"/>
<bpmn:endEvent id="Done"><bpmn:incoming>f2</bpmn:incoming></bpmn:endEvent>
</bpmn:process>
</bpmn:definitions>
orders/channels.yaml binds the channel once:
channels:
- name: order-in
transport: http
bind: "POST /orders"
codec: json
auth:
scheme: apikey
apikey:
value: env:ORDERS_KEY
sutra lint orders warns SUTRA.CONFIG.CHANNEL.START_AND_WAIT:
[WARN] SUTRA.CONFIG.CHANNEL.START_AND_WAIT — deployment acme/orders/1.0.0 (dep-…): process 'order' start event 'Received' and the wait at order/AwaitConfirmation are both on channel 'order-in' with message types that can overlap. A message that correlates to a parked instance answers its wait and starts nothing, so a repeated key never starts a second instance; give them disjoint message types or separate channels if one is meant
That is what this walkthrough relies on; the warning is there for a flow that meant to start a
second instance with a repeated key. package.yaml labels the package acme/orders/1.0.0, and
the engine has ORDERS_KEY set to the key the channel checks.
1. Deploy and start one order
$ sutra package orders --out out-orders
packaged out-orders/orders.sutra (deploymentId dep-<id>)
$ sutra deploy out-orders/orders.sutra --api --engine-url http://localhost:8080
deployed dep-<id> (slot acme--orders--1.0.0, rev 1) — Active
$ curl -X POST http://localhost:8080/orders -H 'X-API-Key: <your key>' \
-H 'Content-Type: application/json' -d '{"key":"A-1"}'
The instance parks at AwaitConfirmation, findable by its key:
$ curl -s -H "X-API-Key: $SUTRA_ADMIN_API_KEY" 'http://localhost:8080/sutra/instances?alias=orderKey:A-1'
{"instances":[{"instanceId":"…","deploymentId":"dep-<id>","status":"SUSPENDED"}]}
2. Send the confirmation
$ curl -X POST http://localhost:8080/orders -H 'X-API-Key: <your key>' \
-H 'Content-Type: application/json' -d '{"key":"A-1"}'
The same channel, the same key: the message resumes the parked instance, which completes. No second
instance starts: the same query now answers {"instances":[]}.
3. Start another order
$ curl -X POST http://localhost:8080/orders -H 'X-API-Key: <your key>' \
-H 'Content-Type: application/json' -d '{"key":"B-2"}'
No instance waits for B-2, so this starts a new one, which parks as in step 1:
/sutra/instances?alias=orderKey:B-2 lists it, SUSPENDED.
Broadcast channels
On a channel with broadcast: true, a correlated message resumes the process that waits for it,
and that process does not also start. Every other process with a start event on the channel still
receives the message, as broadcast promises.
What it costs
On a channel no version waits on, nothing: the engine knows from its index which channels have waits, and answers in a few hash lookups. On a channel with a wait that can take the message’s type, each message costs one correlation lookup per live version that waits. A message whose type no wait accepts (for example a start type and a reply type on one channel) is never looked up.
Upgrading to 0.6.22
- A flow that re-sent a business key on a start-and-wait channel to spawn a second instance now answers the parked one.
- A message whose key names a running instance that has not reached the wait yet is refused with a
retryable
WAIT_PENDINGanswer, not started. - An engine database runs two additive migrations at first boot: V305 and V306. V306 builds an
index on
inbox_seenconcurrently; it adds I/O in proportion to the table and blocks nothing. If the engine connects with a non-owning role, grant itSELECT, INSERTon the newdeployment_ownertable. - Dedup records written before the upgrade dedupe within their own deployment only, so a retry that spans both the engine upgrade and a version flip can run again.
Upgrading a deployment in place
When a new version of a tenant and module takes over, the old version drains: its running instances finish on it. Since 0.6.20, a channel only the old version declares stays served by it, starts included, until it retires at zero running instances. Another owner cannot take such an endpoint meanwhile.
The packages
orders1.0.0:start(POST /orders) parks an order on a receive task waiting onrelease(POST /orders/release, correlated byid). A second process starts onlegacy(POST /orders/legacy) and replieslegacy handled by orders 1.0.0.orders2.0.0: keepsstart, waits onconfirm(POST /orders/confirm) instead, and has noreleaseand nolegacy.audit1.0.0, another module (another owner), bindsPOST /orders/release.
Steps
- Package and deploy
orders1.0.0; start two orders,A-1andA-2. Both park onrelease. - Deploy
orders2.0.0.SUTRA.DEPLOY.ENDPOINT.SUPERSEDEDsays 1.0.0 drains, its instances finish on it, and names the endpoints it keeps:POST /orders/release, POST /orders/legacy. GET /sutra/deployments: 1.0.0 is underdraining, with 2 running instances underdrainingInstances.POST /orders/legacystill runs, on 1.0.0.- Deploy
audit: refused with409,SUTRA.DEPLOY.ENDPOINT.COLLISION, naming 1.0.0 and its 2 running instances. Nothing is stored. - Release
A-1andA-2onPOST /orders/release: both finish on 1.0.0. - On the next sweep 1.0.0 retires:
drainingis empty, andPOST /orders/legacyis404. - Deploy
auditagain: it lands. sutra linton a package that sendsq:send destination="local://acme/orders/2.0.0/start":SUTRA.CONFIG.SEND.LOCAL_DESTINATION_CROSSES_PACKAGE, exit 1. Send on an outbound channel the other package listens on instead.
$ sutra package orders-1 --out out1
packaged out1/orders-1.sutra (deploymentId dep-9f805df1cff68d20777326e9)
[exit 0]
$ sutra deploy out1/orders-1.sutra --api --engine-url http://localhost:8080
deployed dep-9f805df1cff68d20777326e9 (slot acme--orders--1.0.0, rev 1) — Active
[exit 0]
$ curl -X POST /orders -H 'X-API-Key: walk-key' -d '{"id":"A-1"}'
[HTTP 200]
$ curl -X POST /orders -H 'X-API-Key: walk-key' -d '{"id":"A-2"}'
[HTTP 200]
$ sutra package orders-2 --out out2
packaged out2/orders-2.sutra (deploymentId dep-200e1644e8045c4c02c0e3d0)
[exit 0]
$ sutra deploy out2/orders-2.sutra --api --engine-url http://localhost:8080
[WARN] SUTRA.DEPLOY.ENDPOINT.SUPERSEDED — deployment acme--orders--2.0.0 takes over from acme--orders--1.0.0, an earlier deployment of the same tenant and module. acme--orders--1.0.0 drains: it takes no new work, and its running instances finish on it. Channels only it declares stay served by it until it retires: POST /orders/release, POST /orders/legacy.
deployed dep-200e1644e8045c4c02c0e3d0 (slot acme--orders--2.0.0, rev 1) — Active
[exit 0]
$ curl /sutra/deployments
{"active":[{"deploymentId":"dep-200e1644e8045c4c02c0e3d0","slot":"acme--orders--2.0.0","phase":"Active","ready":true}],"pending":[],"draining":["dep-9f805df1cff68d20777326e9"],"drainingInstances":{"dep-9f805df1cff68d20777326e9":2},"failed":[]}
$ curl -X POST /orders/legacy -H 'X-API-Key: walk-key' -d '{}'
legacy handled by orders 1.0.0
[HTTP 200]
$ sutra package audit --out outa
packaged outa/audit.sutra (deploymentId dep-dbd1c12988671263d273fbe1)
[exit 0]
$ sutra deploy outa/audit.sutra --api --engine-url http://localhost:8080
[ERROR] SUTRA.DEPLOY.ENDPOINT.COLLISION — engine rejected the deployment (HTTP 409 Conflict): {"code":"SUTRA.DEPLOY.ENDPOINT.COLLISION","error":"[SUTRA.DEPLOY.ENDPOINT.COLLISION] http endpoint 'POST /orders/release' stays with draining deployment acme--orders--1.0.0 (2 running instances) until it retires"}
[exit 1]
$ curl -X POST /orders/release -H 'X-API-Key: walk-key' -d '{"id":"A-1"}'
[HTTP 200]
$ curl -X POST /orders/release -H 'X-API-Key: walk-key' -d '{"id":"A-2"}'
[HTTP 200]
$ # wait for the sweep
$ curl /sutra/deployments
{"active":[{"deploymentId":"dep-200e1644e8045c4c02c0e3d0","slot":"acme--orders--2.0.0","phase":"Active","ready":true}],"pending":[],"draining":[],"drainingInstances":{},"failed":[]}
$ curl -X POST /orders/legacy -H 'X-API-Key: walk-key' -d '{}'
{"type":"urn:bpm:diag:SUTRA.RESOLVE.CHANNEL.UNKNOWN","title":"SUTRA.RESOLVE.CHANNEL.UNKNOWN","status":404,"detail":"No channel bound at 'POST /orders/legacy'","code":"SUTRA.RESOLVE.CHANNEL.UNKNOWN"} [HTTP 404]
$ sutra deploy outa/audit.sutra --api --engine-url http://localhost:8080
deployed dep-dbd1c12988671263d273fbe1 (slot acme--audit--1.0.0, rev 1) — Active
[exit 0]
$ sutra lint crossing
[ERROR] SUTRA.CONFIG.SEND.LOCAL_DESTINATION_CROSSES_PACKAGE — deployment acme/sender/1.0.0 (dep-ffffffffffffffffffffffff): process 'send' node 'Send' sends to 'local://acme/orders/2.0.0/start', which is not a 'transport: local' inbound channel of this package (local://acme/sender/1.0.0/<channel>). A local address does not cross packages, versions of one package included: send on an outbound channel the other package listens on
1 error(s), 1 warning(s)
[exit 1]
If you undeploy the new version
Undeploying 2.0.0 gives 1.0.0 back every channel it declares, the ones it shared with 2.0.0
included, until 1.0.0 retires. Here a new order C-1 on the shared POST /orders starts on 1.0.0
(the admin instance list for 1.0.0 counts 2), both orders finish on it, and once it retires
POST /orders is 404, because nothing serves it any more.
$ sutra package orders-1 --out out1
packaged out1/orders-1.sutra (deploymentId dep-9f805df1cff68d20777326e9)
[exit 0]
$ sutra deploy out1/orders-1.sutra --api --engine-url http://localhost:8080
deployed dep-9f805df1cff68d20777326e9 (slot acme--orders--1.0.0, rev 1) — Active
[exit 0]
$ curl -X POST /orders -H 'X-API-Key: walk-key' -d '{"id":"A-1"}'
[HTTP 200]
$ sutra package orders-2 --out out2
packaged out2/orders-2.sutra (deploymentId dep-200e1644e8045c4c02c0e3d0)
[exit 0]
$ sutra deploy out2/orders-2.sutra --api --engine-url http://localhost:8080
[WARN] SUTRA.DEPLOY.ENDPOINT.SUPERSEDED — deployment acme--orders--2.0.0 takes over from acme--orders--1.0.0, an earlier deployment of the same tenant and module. acme--orders--1.0.0 drains: it takes no new work, and its running instances finish on it. Channels only it declares stay served by it until it retires: POST /orders/release, POST /orders/legacy.
deployed dep-200e1644e8045c4c02c0e3d0 (slot acme--orders--2.0.0, rev 1) — Active
[exit 0]
$ sutra undeploy acme--orders--2.0.0 --api --engine-url http://localhost:8080
undeployed slot acme--orders--2.0.0 — Draining
[exit 0]
$ curl /sutra/deployments
{"active":[],"pending":[],"draining":["dep-9f805df1cff68d20777326e9"],"drainingInstances":{"dep-9f805df1cff68d20777326e9":1},"failed":[]}
$ curl -X POST /orders -H 'X-API-Key: walk-key' -d '{"id":"C-1"}'
[HTTP 200]
$ # instances on 1.0.0 (dep-9f805df1cff68d20777326e9)
2
$ curl -X POST /orders/release -H 'X-API-Key: walk-key' -d '{"id":"A-1"}'
[HTTP 200]
$ curl -X POST /orders/release -H 'X-API-Key: walk-key' -d '{"id":"C-1"}'
[HTTP 200]
$ # wait for the sweep
$ curl /sutra/deployments
{"active":[],"pending":[],"draining":[],"drainingInstances":{},"failed":[]}
$ curl -X POST /orders -H 'X-API-Key: walk-key' -d '{"id":"D-1"}'
{"type":"urn:bpm:diag:SUTRA.RESOLVE.CHANNEL.UNKNOWN","title":"SUTRA.RESOLVE.CHANNEL.UNKNOWN","status":404,"detail":"No channel bound at 'POST /orders'","code":"SUTRA.RESOLVE.CHANNEL.UNKNOWN"} [HTTP 404]
What to watch for
- A version that keeps starting work on the channels it kept never retires. Watch
drainingInstances. - In a deployments folder, removing 1.0.0’s file stops its channels.
- Versions already draining when you upgrade the engine to 0.6.20 stop serving as before.
SUTRA.DEPLOY.ENDPOINT.CHANNEL_CHANGEDandSTART_ON_WAITwarn at deploy when 2.0.0 changes a channel 1.0.0 has instances parked on (not shown in this run; the tier-2 suite covers them).
See also One endpoint, one deployment, which covers how the other deployment paths behave.
Acceptance receipts
An HTTP channel with ack-mode: on-persist answers 202 Accepted before the work finishes. Since 0.6.18
the engine tells the caller what it accepted, so nobody has to poll the instance list and match a variable.
This walkthrough starts an instance, reads its receipt, repeats the request, and finds the instance again by
its alias.
You need the engine and a PostgreSQL database; this was run on PostgreSQL 16. Only the HTTP path was run. Queue transports have no synchronous answer, so they get no receipt.
1. Write the package
package.yaml:
labels:
"tenant": "default"
"module": "cases"
"version": "1.0.0"
engine:
minContract: 1
channels.yaml. Idempotency-Key is the header that makes a retry the same request:
channels:
- name: case-open
transport: http
bind: "POST /channels/case-open"
codec: json
ack-mode: on-persist
idempotency-key-header: Idempotency-Key
cloudevents-mode: none
auth:
scheme: apikey
apikey:
value: env:CASES_API_KEY
header: X-Api-Key
- name: case-release
transport: http
bind: "POST /channels/case-release"
codec: json
ack-mode: on-persist
cloudevents-mode: none
auth:
scheme: apikey
apikey:
value: env:CASES_API_KEY
header: X-Api-Key
bpmn/case-open.bpmn. <q:receipt> names the values this flow chooses to expose. Without it, the receipt
carries no flow values of its own:
<?xml version="1.0" encoding="UTF-8"?>
<bpmn:definitions xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL"
xmlns:q="urn:sutra:q:1.0"
id="Definitions_case_open"
targetNamespace="urn:sutra:module:cases:1.0.0">
<bpmn:process id="case-open" isExecutable="true">
<bpmn:extensionElements>
<q:variables>
<q:variable name="holderId" type="string" sensitive="true"/>
</q:variables>
<q:receipt>
<q:attribute name="caseId" expression="req.caseId"/>
<q:attribute name="holder" expression="holderId"/>
</q:receipt>
</bpmn:extensionElements>
<bpmn:startEvent id="Start">
<bpmn:extensionElements>
<q:source channel="case-open" name="req"/>
<q:alias name="orderId" expression="req.orderId" unique="true" onConflict="correlate"/>
<q:alias name="holderId" expression="req.holderId"/>
</bpmn:extensionElements>
<bpmn:outgoing>F1</bpmn:outgoing>
</bpmn:startEvent>
<bpmn:sequenceFlow id="F1" sourceRef="Start" targetRef="Release"/>
<bpmn:receiveTask id="Release">
<bpmn:extensionElements><q:source channel="case-release" name="release"/></bpmn:extensionElements>
<bpmn:incoming>F1</bpmn:incoming><bpmn:outgoing>F2</bpmn:outgoing>
</bpmn:receiveTask>
<bpmn:sequenceFlow id="F2" sourceRef="Release" targetRef="Done"/>
<bpmn:endEvent id="Done"><bpmn:incoming>F2</bpmn:incoming></bpmn:endEvent>
</bpmn:process>
</bpmn:definitions>
2. Lint and package it
sutra lint cases
sutra package cases -o deployments
Lint warns about the holder attribute, because it reads a sensitive variable and the caller would only
ever see the mask. It is kept here to show the masking. A real flow would drop it.
[WARN] SUTRA.CONFIG.RECEIPT.ATTRIBUTE_SENSITIVE — ... process 'case-open' <q:receipt> attribute 'holder' reads sensitive variable 'holderId', so every receipt shows it as ***REDACTED***. Expose a value that is safe for the caller to see, or remove the attribute.
3. Start the engine
Give the engine a PostgreSQL role that isn’t a superuser, and a database that role owns:
CREATE ROLE sutra_app LOGIN NOSUPERUSER NOBYPASSRLS PASSWORD '<a password>';
CREATE DATABASE sutra OWNER sutra_app;
Start the engine with SUTRA_DATASOURCE_URL pointing at that database as sutra_app,
SUTRA_DEPLOYMENTS_DIR pointing at deployments, and CASES_API_KEY. Turn admin auth on with
SUTRA_ADMIN_AUTH_SCHEME=apikey, SUTRA_ADMIN_AUTH_KEY_REF=env:SUTRA_ADMIN_API_KEY and
SUTRA_ADMIN_API_KEY, because step 6 reads instances. Because this role owns its tables, the engine logs
SUTRA.STARTUP.RLS_INERT_POSTURE at startup: row-level security is on but doesn’t apply to the table owner.
Before running several tenants on one database, give the engine a role that doesn’t own its tables. Set B to
the engine’s address, then wait for it:
curl -s "$B/sutra/health/ready"
{"status":"UP","checks":[{"name":"sutra-loader","status":"UP","data":{"deployments":1,"shards":1}}]}
4. Start a case and read the receipt
curl -si -X POST "$B/channels/case-open" \
-H 'X-Api-Key: <your key>' -H 'Content-Type: application/json' \
-H 'Idempotency-Key: try-1' \
-d '{"orderId":"O-100","caseId":"C-7","holderId":"H-SECRET"}'
HTTP/1.1 202 Accepted
content-type: application/json
sutra-instance-id: 315001d4-74f0-42bc-8b97-8dabe7511b35
sutra-receipt: {"deploymentId":"dep-dceba0e28896cca13aa92d0a","labels":{"module":"cases","tenant":"default","version":"1.0.0"},"duplicate":false,"instanceId":"315001d4-74f0-42bc-8b97-8dabe7511b35","aliases":{"holderId":"***REDACTED***","orderId":"O-100"},"attributes":{"caseId":"C-7","holder":"***REDACTED***"},"idempotencyKey":"try-1"}
{"deploymentId":"dep-dceba0e28896cca13aa92d0a","labels":{"module":"cases","tenant":"default","version":"1.0.0"},"duplicate":false,"instanceId":"315001d4-74f0-42bc-8b97-8dabe7511b35","aliases":{"holderId":"***REDACTED***","orderId":"O-100"},"attributes":{"caseId":"C-7","holder":"***REDACTED***"},"idempotencyKey":"try-1"}
The flow replied nothing, so the receipt is also the body. A flow that replies with
<q:reply continue="true"> keeps its own body byte for byte, and the receipt rides only in the headers.
H-SECRET appears nowhere: holderId is declared sensitive, so both the alias and the attribute show the
mask. An alias built from the payload does show that value to the caller, so alias what you are willing to
return.
5. Send it again
Repeat the same request, with the same Idempotency-Key:
HTTP/1.1 202 Accepted
content-type: application/json
sutra-instance-id: 315001d4-74f0-42bc-8b97-8dabe7511b35
{"deploymentId":"dep-dceba0e28896cca13aa92d0a","labels":{"module":"cases","tenant":"default","version":"1.0.0"},"duplicate":true,"instanceId":"315001d4-74f0-42bc-8b97-8dabe7511b35","originalInstanceId":"315001d4-74f0-42bc-8b97-8dabe7511b35","aliases":{},"attributes":{},"idempotencyKey":"try-1"}
Nothing new started. The answer names the instance the first request started. It has no aliases or
attributes because nothing ran to produce them. A duplicate on any HTTP channel, including an on-complete
one, gets this 202.
6. Find the instance after losing the 202
With admin auth on, reading instances takes the admin key. Without it, or with a wrong one, the engine refuses:
curl -si "$B/sutra/instances?alias=orderId:O-100"
HTTP/1.1 401 Unauthorized
{"error":"unauthorized","reason":"missing or invalid admin auth key/secret"}
Send the key, in X-API-Key unless you set SUTRA_ADMIN_AUTH_HEADER:
curl -s -H "X-API-Key: $SUTRA_ADMIN_API_KEY" "$B/sutra/instances?alias=orderId:O-100"
{"instances":[{"instanceId":"315001d4-74f0-42bc-8b97-8dabe7511b35","deploymentId":"dep-dceba0e28896cca13aa92d0a","status":"SUSPENDED"}]}
An alias nothing is bound to gives {"instances":[]}. The lookup finds live instances. A finished instance’s
aliases have retired, so add includeTerminal=true to find it. That form walks the deployment’s alias rows, so
don’t poll it. The same gate covers the plain list and GET /sutra/instances/{id}; see
Upgrading to 0.6.18 for the case where
admin auth is off.
What to know
idempotencyKeyappears only when the caller supplied a key or a<q:source dedupKey>derived one. Without one, a repeated request starts a new instance, so nothing is echoed.- Attribute values are strings. A null is left out. An expression that fails to evaluate is left out and logged, because the work is already accepted.
Sutra-Receiptis one line of ASCII JSON. Past 4 KiB it keeps only the identifying fields and says"truncated": true.Sutra-Instance-Idis always there.- A duplicate names no instance if the first delivery ended without one, or was recorded before 0.6.18.
- Two deliveries racing before either commits behave as before: the loser gets
SUTRA.INBOUND.DUPLICATE_CONCURRENT. - A
<q:attribute>with a repeated name is the lint errorSUTRA.CONFIG.RECEIPT.ATTRIBUTE_DUPLICATE; one missing its name or expression fails at load.
Answering with a status and headers
Since 0.8.0 a flow that answers a synchronous HTTP request chooses the status and the response
headers, so it can behave like a REST API (201 with Location, 404, 409) or like an auth check for
a reverse proxy (200 with identity headers, or 401 with a challenge). The reference is
Status and response headers. This walkthrough
deploys one package with four endpoints and calls each with curl.
You need Docker, curl, the sutra CLI and an empty working directory. It was run as written with
the engine built from this release and postgres:16-alpine; the output shown is from that run.
1. Write the package
replies/package.yaml:
labels:
"tenant": "default"
"module": "replies"
"version": "1.0.0"
engine:
minContract: 1
replies/channels.yaml: four synchronous HTTP channels with the same key on each:
channels:
- name: items
transport: http
bind: "POST /channels/items"
codec: json
auth:
scheme: apikey
apikey:
value: env:REPLIES_API_KEY
header: X-Api-Key
- name: lookup
transport: http
bind: "POST /channels/lookup"
codec: json
auth:
scheme: apikey
apikey:
value: env:REPLIES_API_KEY
header: X-Api-Key
- name: check
transport: http
bind: "POST /channels/check"
codec: json
auth:
scheme: apikey
apikey:
value: env:REPLIES_API_KEY
header: X-Api-Key
- name: bad
transport: http
bind: "POST /channels/bad"
codec: json
auth:
scheme: apikey
apikey:
value: env:REPLIES_API_KEY
header: X-Api-Key
replies/templates/item.hbs:
{"id":"{{req.id}}"}
replies/bpmn/reply-status.bpmn, four processes:
<?xml version="1.0" encoding="UTF-8"?>
<bpmn:definitions xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL"
xmlns:q="urn:sutra:q:1.0"
targetNamespace="urn:sutra:module:replies:1.0.0">
<!-- 201 with a Location -->
<bpmn:process id="create" isExecutable="true">
<bpmn:startEvent id="CreateStart">
<bpmn:extensionElements><q:source channel="items" name="req"/></bpmn:extensionElements>
</bpmn:startEvent>
<bpmn:sequenceFlow id="C1" sourceRef="CreateStart" targetRef="Create"/>
<bpmn:serviceTask id="Create" implementation="item.hbs">
<bpmn:extensionElements>
<q:reply contentType="application/json" status="201">
<q:header name="Location" value="'/items/' + req.id"/>
<q:header name="X-Request-Id" value="req.id"/>
</q:reply>
</bpmn:extensionElements>
</bpmn:serviceTask>
<bpmn:sequenceFlow id="C2" sourceRef="Create" targetRef="CreateDone"/>
<bpmn:endEvent id="CreateDone"/>
</bpmn:process>
<!-- a status chosen by FEEL -->
<bpmn:process id="lookup" isExecutable="true">
<bpmn:startEvent id="LookupStart">
<bpmn:extensionElements><q:source channel="lookup" name="req"/></bpmn:extensionElements>
</bpmn:startEvent>
<bpmn:sequenceFlow id="L1" sourceRef="LookupStart" targetRef="Lookup"/>
<bpmn:serviceTask id="Lookup" implementation="item.hbs">
<bpmn:extensionElements>
<q:reply contentType="application/json" status="if req.id = '42' then 200 else 404"/>
</bpmn:extensionElements>
</bpmn:serviceTask>
<bpmn:sequenceFlow id="L2" sourceRef="Lookup" targetRef="LookupDone"/>
<bpmn:endEvent id="LookupDone"/>
</bpmn:process>
<!-- an auth check: no body, 200 with identity headers or 401 with a challenge -->
<bpmn:process id="check" isExecutable="true">
<bpmn:startEvent id="CheckStart">
<bpmn:extensionElements><q:source channel="check" name="req"/></bpmn:extensionElements>
</bpmn:startEvent>
<bpmn:sequenceFlow id="K1" sourceRef="CheckStart" targetRef="CheckDone"/>
<bpmn:endEvent id="CheckDone">
<bpmn:extensionElements>
<q:reply status="if req.token = 'good' then 200 else 401">
<q:header name="X-User-Id" value="if req.token = 'good' then req.user else null"/>
<q:header name="WWW-Authenticate" value="if req.token = 'good' then null else 'Bearer realm="sim"'"/>
</q:reply>
</bpmn:extensionElements>
</bpmn:endEvent>
</bpmn:process>
<!-- whatever status the caller sends: shows the refusal -->
<bpmn:process id="bad" isExecutable="true">
<bpmn:startEvent id="BadStart">
<bpmn:extensionElements><q:source channel="bad" name="req"/></bpmn:extensionElements>
</bpmn:startEvent>
<bpmn:sequenceFlow id="B1" sourceRef="BadStart" targetRef="BadDone"/>
<bpmn:endEvent id="BadDone">
<bpmn:extensionElements><q:reply status="req.code"/></bpmn:extensionElements>
</bpmn:endEvent>
</bpmn:process>
</bpmn:definitions>
create answers 201 with two headers and the template’s body. lookup picks its status with FEEL.
check renders no body: its end event answers 200 with X-User-Id for the right token, else 401
with WWW-Authenticate; a header whose value is null is left out. bad sends whatever status the
caller asks for, to show what happens when it cannot be sent.
2. Check it
sutra lint replies/
Only the schema-less-channel warnings appear:
[WARN] SUTRA.CHANNEL.NO_SCHEMA — deployment default/replies/1.0.0 (dep-ffffffffffffffffffffffff): channel 'items' has 1 subscriber(s) but its codec (json) declares no message types — ...
...
0 error(s), 4 warning(s)
A literal status is checked here, before anything is deployed. Change lookup’s status to
status="302" and lint refuses it, because a redirect needs a Location; status="99" is refused
because a 1xx is not a final answer:
[ERROR] SUTRA.OUTBOUND.REPLY_STATUS_INVALID — deployment default/replies/1.0.0 (dep-ffffffffffffffffffffffff): process 'lookup' node 'Lookup' <q:reply>: status 302 is a redirect, which needs a Location header (declare <q:header name="Location">, and make sure it is not null).
1 error(s), 4 warning(s)
[ERROR] SUTRA.OUTBOUND.REPLY_STATUS_INVALID — deployment default/replies/1.0.0 (dep-ffffffffffffffffffffffff): process 'lookup' node 'Lookup' <q:reply>: status 99 is outside 200-599; a reply is a final answer, so a 1xx is refused too.
1 error(s), 4 warning(s)
Change it back.
3. Package it and start the engine
mkdir -p deployments
sutra package replies --out deployments
chmod -R a+rX deployments
docker network create replies-net
docker run -d --name replies-db --network replies-net -e POSTGRES_PASSWORD=dev-only postgres:16-alpine
until docker exec replies-db pg_isready -U postgres -h 127.0.0.1; do sleep 1; done
sleep 2
docker exec replies-db psql -U postgres \
-c "CREATE ROLE sutra_engine LOGIN PASSWORD 'dev-only' NOSUPERUSER NOBYPASSRLS" \
-c "ALTER SCHEMA public OWNER TO sutra_engine"
docker run -d --name replies-engine --network replies-net -p 127.0.0.1::8080 \
-v "$PWD/deployments:/deployments:ro" -e SUTRA_DEPLOYMENTS_DIR=/deployments \
-e SUTRA_DATASOURCE_URL=postgres://replies-db:5432/postgres \
-e SUTRA_DATASOURCE_USERNAME=sutra_engine -e SUTRA_DATASOURCE_PASSWORD=dev-only \
-e REPLIES_API_KEY=demo-key \
ghcr.io/startr-trade/sutra:0.8.3
PORT=$(docker port replies-engine 8080 | head -1 | cut -d: -f2)
until curl -sf "http://127.0.0.1:$PORT/sutra/health/ready"; do sleep 1; done
{"status":"UP","checks":[{"name":"sutra-loader","status":"UP","data":{"deployments":1,"shards":1}}]}
4. Call it
A create answers 201 with its Location:
curl -si -X POST "localhost:$PORT/channels/items" -H 'X-Api-Key: demo-key' \
-H 'Content-Type: application/json' -d '{"id":"7"}'
HTTP/1.1 201 Created
content-type: application/json
location: /items/7
x-request-id: 7
content-length: 11
date: Thu, 19 Nov 2026 09:13:19 GMT
{"id":"7"}
A lookup answers 200 for {"id":"42"} and 404 for anything else, with the flow’s body either way:
curl -si -X POST "localhost:$PORT/channels/lookup" -H 'X-Api-Key: demo-key' \
-H 'Content-Type: application/json' -d '{"id":"41"}'
HTTP/1.1 404 Not Found
content-type: application/json
content-length: 12
date: Thu, 19 Nov 2026 09:13:19 GMT
{"id":"41"}
The auth check, with the right token and then a wrong one:
curl -si -X POST "localhost:$PORT/channels/check" -H 'X-Api-Key: demo-key' \
-H 'Content-Type: application/json' -d '{"token":"good","user":"u-1"}'
curl -si -X POST "localhost:$PORT/channels/check" -H 'X-Api-Key: demo-key' \
-H 'Content-Type: application/json' -d '{"token":"bad"}'
HTTP/1.1 200 OK
content-type: application/json
x-user-id: u-1
content-length: 0
date: Thu, 19 Nov 2026 09:13:19 GMT
HTTP/1.1 401 Unauthorized
content-type: application/json
www-authenticate: Bearer realm="sim"
content-length: 0
date: Thu, 19 Nov 2026 09:13:19 GMT
Each answer carries only the header whose value was not null. A reverse proxy that runs a
subrequest per call (an auth_request-style check) can pass X-User-Id on to the upstream.
5. A status that cannot be sent
A computed status is checked when the answer is sent. bad with {"code":700}:
curl -si -X POST "localhost:$PORT/channels/bad" -H 'X-Api-Key: demo-key' \
-H 'Content-Type: application/json' -d '{"code":700}'
HTTP/1.1 500 Internal Server Error
content-type: application/problem+json
content-length: 355
date: Thu, 19 Nov 2026 09:13:19 GMT
{"type":"urn:bpm:diag:SUTRA.OUTBOUND.REPLY_STATUS_INVALID","title":"SUTRA.OUTBOUND.REPLY_STATUS_INVALID","status":500,"detail":"The reply of process 'bad' at node 'BadDone': status 700 is outside 200-599; a reply is a final answer, so a 1xx is refused too.","code":"SUTRA.OUTBOUND.REPLY_STATUS_INVALID","attributes":{"nodeId":"BadDone","processId":"bad"}}
The same call with {"code":409} answers HTTP/1.1 409 Conflict with an empty body. A header value
with a line break or another control character is refused the same way, with
SUTRA.OUTBOUND.REPLY_HEADER_INVALID; the problem names the header, never its value.
6. Clean up
docker rm -f replies-engine replies-db
docker network rm replies-net
Where it does not apply
- An
ack-mode: on-persistchannel always answers 202, the acceptance; the reply’s headers are added to it. - A broker intake (Kafka, AMQP, …) has no status.
sutra lintwarns about both withSUTRA.CHANNEL.REPLY_STATUS_IGNORED.- A
<q:reply>with adestinationgoes out of band, and its headers travel with that message.
External tasks: the pull worker surface
The engine is otherwise push-only: an outbound emission lands in the outbox and the relay dials a
transport. A channel declaring transport: pull inverts that last hop — instead of dialing
anything, the delivery is parked as a fetchable task, and workers come and get it.
That is the whole feature. It exists for work the engine cannot dial out to: a worker behind a NAT, on a laptop, in a language that has no inbound listener, or simply one that would rather poll than be called.
Declaring a pull channel
channels:
- name: score-request
transport: pull
bind: "pull://acme/scoring/1.0.0/score-request"
Nothing in the BPMN changes. A <q:send> or a channel-call <bpmn:serviceTask> targeting this
channel behaves exactly as it would over HTTP or a broker; only the last hop is different. And
because the worker’s answer comes back in on the same channel, the author’s <q:alias>
correlation is unchanged too — a pull task is not a new resume path.
Pull needs a datasource: a parked task is a database row. Without one the surface answers 503.
The worker protocol
Three operations, all under /sutra/external-tasks.
Fetch and lock
POST /sutra/external-tasks/fetch-and-lock
{ "workerId": "scorer-7", "channels": ["score-request"],
"lockDuration": "PT1M", "maxTasks": 10, "asyncResponseTimeout": "PT20S" }
Claims up to maxTasks fetchable tasks on the named channels and locks them to workerId. A
worker names topics (channel names), never deployment ids — the fetch walks the live
deployment set for you.
When nothing is available the request is held open as a bounded long poll: it wakes the moment
a task is parked on one of the channels you asked for, and answers with an empty list when
asyncResponseTimeout elapses. It never hangs, and the wait is always bounded by the operator’s
ceiling.
Each returned task carries its id, channel, instance id, headers, body and content type, its attempt count, and its remaining failure budget.
Complete
POST /sutra/external-tasks/{id}/complete
{ "workerId": "scorer-7", "result": { ... } }
Feeds the worker’s result back through the engine’s ordinary inbound path — the same seam every transport delivers through. Correlation, validation, and inbox dedup all behave exactly as they do for a pushed reply; there is no second resume entry point.
Omitting the result re-delivers the original request payload. That is the fire-and-forget shape: the work happened outside, and the flow is waiting only on the fact of it.
Failure
POST /sutra/external-tasks/{id}/failure
{ "workerId": "scorer-7", "errorMessage": "downstream 500", "retries": 2, "retryTimeout": "PT30S" }
Releases the lock and spends one of the task’s retries — or sets the remaining budget explicitly —
deferring the next fetch by retryTimeout.
Lock expiry is the only recovery mechanism you need
A locked task is invisible to every other worker until its lock expires, at which point it becomes fetchable again. There is no sweeper, no reaper, and no timeout job: expiry is part of the predicate that decides what a fetch may claim, so a worker that dies mid-task costs exactly one lock duration and never costs the work.
stateDiagram-v2
[*] --> fetchable: the pull sink parks<br/>the delivery as a task row
fetchable --> locked: fetch-and-lock<br/>workerId + lockDuration
locked --> fetchable: lock expires — no sweeper,<br/>it is the claim predicate
locked --> fetchable: failure with budget left,<br/>after retryTimeout
locked --> [*]: complete — dispatched,<br/>then the row is deleted
locked --> terminal: budget spent —<br/>retained with its last error
There is no reaper because an expired lock is not a state anyone has to clean up: it is simply no longer an obstacle to the next fetch, so availability is instant rather than one sweep tick late.
A completion or failure from a worker that no longer owns the lock fails closed, and the refusal names which of the three situations it is:
| Code | Meaning |
|---|---|
SUTRA.EXTERNAL_TASK.LOCK_LOST | Your lock expired or was released. The task is fetchable again — by you or by anyone. |
SUTRA.EXTERNAL_TASK.LOCK_HELD | Another worker holds it. |
SUTRA.EXTERNAL_TASK.TERMINAL | The task spent its budget and can never be completed or failed again. |
SUTRA.EXTERNAL_TASK.NOT_FOUND | No such task on any live deployment. |
A stale worker never receives a 200 it could mistake for success.
At-least-once, and what makes that safe
The task row is deleted only after the engine has accepted the completion. A crash in the window between the two re-offers the task, so the work is never lost — the surface is at-least-once, deliberately, because the inverse ordering (delete, then dispatch) would be at-most-once and would drop work outright on the same crash.
sequenceDiagram
participant W as Worker
participant E as Engine
participant TR as Task row
W->>E: complete — workerId + result
E->>TR: ownership-guarded — is this lock still yours?
E->>E: dispatch the result through the ordinary inbound path
Note over E,TR: crash here and the task is re-offered — inbox dedup absorbs the duplicate
E->>TR: delete the row
E-->>W: 200
The ordering is the whole guarantee: the crash window can only produce a duplicate the idempotency key already absorbs, where the inverse order would lose the work with nothing recording that it ever happened.
What makes the duplicate harmless is that each task carries the originating outbound delivery’s key as an explicit idempotency key, and the completion re-enters under it. Inbox dedup absorbs the second attempt. A worker does not have to do anything to get this — it is a property of the surface, not of the worker.
If the engine refuses a completion on the inbound path (a validation reject, say), the answer is
422 with the engine’s own code carried as attributes.causeCode — which is what tells the
worker whether re-fetching later can ever help. The task is retained and becomes fetchable again.
Two reserved headers a worker never sees
For completeness: internally, the dispatcher stamps two reserved headers —
sutra-park-deployment and sutra-park-instance — onto a delivery whose destination scheme is
pull, and the pull sink strips them as it parks the task. They carry the owning deployment (the
isolation key) and instance to the parking side without widening the transport contract every
other sink implements, and the scheme gate means they can never leak onto a network transport.
They never appear in a fetched task’s headers; nothing a worker builds should look for them, and
their presence in a payload you construct yourself has no effect — the sink’s own stamp always
wins.
The worker retry budget
A freshly parked task starts with sutra.external-task.retries failures (default 3). Each
reported failure spends one and defers the next fetch by sutra.external-task.retry-timeout
(default PT10S).
A spent budget makes the task terminal: never fetched again, retained with its last error. That is the pull-side twin of the outbox’s poison horizon, and it exists for the same reason — so “we gave up” can never degrade into “it silently vanished”. A terminal task no longer counts toward its deployment’s retirement quiescence gate.
Bounds are rejections, never clamps
Every knob a worker can send is a ceiling an operator sets, and a request over the ceiling is
a 400 — not a silent adjustment down:
| Key | Default | Ceiling on |
|---|---|---|
sutra.external-task.default-lock-duration | PT30S | The lock granted when a fetch names none; also the completion grace window. |
sutra.external-task.max-lock-duration | PT1H | lockDuration. |
sutra.external-task.max-async-response-timeout | PT30S | asyncResponseTimeout (and its default). |
sutra.external-task.max-tasks | 100 | maxTasks (and its default). |
sutra.external-task.retries | 3 | The starting failure budget. |
sutra.external-task.retry-timeout | PT10S | Backoff before a failed-with-budget task is fetchable again. |
The reason it is a reject is a correctness one: a worker that believes it holds a longer lock
than it actually does is a duplicate-execution bug waiting to happen. Silently clamping would
create exactly that belief. Durations are ISO-8601 everywhere on this surface, the same grammar
the engine’s cadence keys and <q:retry> use.
The engine also boots fail-closed if default-lock-duration exceeds its own ceiling or is zero.
Posture
These are operate-surface routes (/sutra/*), not administrative ones: a completion is an
ordinary delivery, not a privileged control operation. They carry the same cluster-internal
posture the rest of /sutra/* does. A deployment that needs authenticated workers puts them
behind the same ingress policy the rest of the operate surface already needs — see
Configuration reference for the separately gated
admin surface.
How it composes with <q:retry>
Two budgets, at two layers, that never overlap:
| Governed by | Counts | On exhaustion | |
|---|---|---|---|
| Worker failures | sutra.external-task.retries (operator config) | A worker reporting a failure on a fetched task | The task turns terminal; the instance stays parked on its wait |
| Task failures | <q:retry> (the author’s model) | A route-less <q:timeout> firing, or a poisoned request delivery | The instance fails durably |
A worker retrying its own work never touches the instance. A <q:retry> policy on a pull-backed
channel call still governs the task-level outcome — its timeout is still the thing that eventually
fires if no worker ever completes. See
Retries, history, and schedules.
Next
- Channels and transports — where
transport:is declared and what the other transports do. - Wait states and human tasks — the park/resume model a pull task rides on.
- Configuration reference — the key family in full.
- The pull surface: the design reasoning — ownership transfer, why zero rows affected is never success, and why there is no sweeper.
Retries, history, and schedules
Three execution-semantics features an author reaches for once a flow has to survive contact with an unreliable world: a per-task retry policy, execution history that outlives the instance, and the full set of BPMN timer schedules.
<q:retry> — a per-task retry policy
A <bpmn:serviceTask> may declare a retry policy inline. It applies to both kinds of service
task: a registered task (a function the embedding application registered) and a
channel-call task (implementation="channel:<name>" — the wait-state call every pure-BPMN
deployment uses through the shipped binary).
<bpmn:serviceTask id="Score" implementation="registered:score">
<bpmn:extensionElements>
<q:retry maxAttempts="3" initialDelay="PT1S" backoffCoefficient="2.0"
maxDelay="PT5M" nonRetryableCodes="SUTRA.TASK.VALIDATION"/>
</bpmn:extensionElements>
</bpmn:serviceTask>
| Attribute | Default | Meaning |
|---|---|---|
maxAttempts | required | Total invocation budget, including the first attempt. |
initialDelay | PT1S | Wait before attempt 2. |
backoffCoefficient | 2.0 | Delay multiplier — attempt n+1 waits min(initialDelay × coefficient^(n-1), maxDelay). |
maxDelay | PT5M | Ceiling on any single backoff wait. |
nonRetryableCodes | — | Structured codes that fail immediately, budget notwithstanding. |
So maxAttempts="4" initialDelay="PT1S" backoffCoefficient="2.0" waits 1 s, then 2 s, then 4 s
between its four attempts — each capped at maxDelay.
A retry wait is a durable park, never a sleep
This is the load-bearing property. A failed attempt with budget remaining persists the instance with the failed task still pending and an armed timer due at the backoff instant; the ordinary timer poller re-drives it when that instant arrives.
Nothing sleeps, and nothing is held in memory. The consequences are the ones you want:
- the backoff survives a restart, a rolling upgrade, and a hot-deploy (a retry park is pinned to its deployment like every other park);
- no execution lane is blocked during the wait, so a task backing off costs other instances nothing;
- a
PT5Mbackoff is aPT5Mbackoff whether or not the replica that started it is still alive.
sequenceDiagram
participant L as Execution lane
participant DB as PostgreSQL
participant T as Timer poller
loop while the budget holds
L->>L: attempt fails
L->>DB: re-park — task still pending, timer armed at the backoff instant
Note over L,DB: a durable park, not a sleep — the lane is free for other instances
T->>DB: backoff instant due
T->>L: re-drive the task
end
L->>DB: budget spent — durable FAILED, retained and inspectable
A backoff is a row in a database, not a held thread — which is why it survives a restart and a hot-deploy, and why a task waiting five minutes costs every other instance nothing.
Declaring <q:retry> on a task makes its process stateful — the park needs persistence, so a
process that was otherwise run-to-completion now requires a configured datasource.
What counts as a failed attempt
The failure set is deliberately narrow and different for each task kind.
On a registered task: an uncaught error from the task function.
On a channel-call task, exactly two things:
- the route-less
<q:timeout>boundary firing before the correlated response arrived (classificationSUTRA.DISPATCH.CHANNEL_CALL.TIMEOUT). Without a policy a fired timeout is a catchable BPMN error; with one it becomes a retryable task failure first; - the request delivery being marked terminally poisoned by the outbox attempt ceiling
(classification
SUTRA.OUTBOUND.DELIVERY_ATTEMPTS_EXHAUSTED, reachable only whensutra.outbox.retry.max-attemptsis configured). The failure reaches the task the moment the delivery gives up, rather than waiting out the whole timeout window.
Both classifications are stable structured codes, so nonRetryableCodes can name them — “retry a
poisoned delivery but never a timeout” is expressible.
What is never a failed attempt:
- A correlated business response. The counterpart answered. Whatever the answer says is the process’s business to route on; re-sending would double-submit. A response is a completion, full stop.
- A BPMN error. Errors route to their boundary events, unchanged, on either task kind.
Modelled outcomes always beat a policy
A timer boundary event with drawn outgoing flows is a modelled outcome — the author has said
what happens on timeout, so that is what happens, exactly as a BPMN error routes to its boundary
instead of consuming a retry budget. Which is why the loader refuses the combination: a
channel-call <q:retry> requires the route-less <q:timeout> form, and a routed timer boundary
alongside a retry policy is a load error (SUTRA.CONFIG.BPMN.RETRY_NOT_APPLICABLE). A policy that
could never fire would otherwise load as a silent near-no-op.
What a re-drive actually does on a channel call
Re-driving a channel call is not “wait for the same answer again” — it re-issues the request:
- a fresh outbound request built from the same persisted variables, with a fresh idempotency key on the wire (so a counterpart doing its own dedup sees a genuinely new request, not a redelivery);
- a fresh timeout window;
- the dead attempt’s outstanding request withdrawn, so a late answer to a superseded attempt cannot race the new one.
A response that arrives for the dead attempt is refused with
SUTRA.DISPATCH.CHANNEL_CALL.RETRY_PENDING — an honest verdict rather than a “no such instance”
miss, because the correlation is deliberately kept alive. Once the re-issued request goes out, the
same correlation serves its response normally.
sequenceDiagram
participant E as Engine
participant C as Counterpart
E->>C: attempt n, idempotency key K1
Note over E: route-less q:timeout fires — a retryable task failure
E->>E: park the backoff, withdraw attempt n's pending delivery
C-->>E: late answer to attempt n
E-->>C: refused — SUTRA.DISPATCH.CHANNEL_CALL.RETRY_PENDING
E->>C: attempt n+1, fresh key K2, fresh timeout window
C-->>E: answer — a completion, never a retry trigger
A re-drive is a new request rather than a second wait for the old one, so the counterpart’s own deduplication sees it as new; and the superseded attempt is refused by name instead of being silently absorbed or lost.
Exhaustion
On a channel-call task whose failure the flow catches (a boundary or an error event sub-process), a
spent budget raises the failure’s own code as a BPMN error on the task; see
Catching a failed call. Otherwise a spent budget — or a
nonRetryableCodes hit — fails the instance durably with SUTRA.RUNTIME.RETRY.EXHAUSTED. The instance becomes FAILED: retained, inspectable, blocking
its deployment’s retirement, and repairable through
instance migration. See
Replica semantics.
Two budgets that never overlap
If a channel uses the pull transport, its worker-side failures are governed by the external
task’s own budget (sutra.external-task.retries), not by <q:retry>. The two live at different
layers and never interact — see External tasks.
The outbox poison ceiling
Outbound deliveries retry with backoff forever by default: an unreachable counterpart is
treated as temporarily unreachable, and giving up silently is worse than waiting. Since 0.6.20 each failed attempt
that will be retried logs a WARN with the destination, the cause’s code and message, the attempt number and
next_attempt_at. The backoff base, sutra.outbox.retry.base-delay, defaults to PT1S.
sutra.outbox.retry.max-attempts is the opt-in ceiling. A delivery that exhausts it becomes
terminally poisoned: retained with its last error, never retried again, one incident recorded
if the delivery was required, and — importantly — no longer counted by the draining-deployment
retirement gate. “We gave up” is a durable, visible state, never a silent disappearance.
Configuring the ceiling is also what makes SUTRA.OUTBOUND.DELIVERY_ATTEMPTS_EXHAUSTED reachable
as a <q:retry> failure at all.
Execution history: a finished instance is not a 404
A terminal instance used to vanish. It doesn’t: the terminal step re-stamps the stored snapshot
to COMPLETED / TERMINATED in the same transaction that resolves its waits, retires its
correlation aliases, and enqueues its final emissions.
| Surface | What it answers for a finished instance |
|---|---|
GET /sutra/instances/{id} and its admin twin | The retained projection — status, node progress, variables — for as long as retention keeps the row. |
GET /admin/instances | Excludes finished instances by default; includeTerminal=true (or a terminal status filter) includes them. |
GET /admin/instances/{id}/history | The audit journal, paged: how it got there, event by event, with whatever payload the process captured. Page with ?afterOrdinal= and the response’s nextAfterOrdinal: each event’s ordinal is its position in the journal, assigned in commit order, so no event written later lands behind a cursor already returned. ?afterSeq= still works, but can miss a transport event, which shares the seq of the engine event it followed. Admin-only — an audit row can carry business data. |
Retention. sutra.instance.retention (ISO-8601, default P7D) is how long terminal snapshots
stay queryable; a lease-gated sweeper purges past-window rows on
sutra.instance.retention-sweep-interval (default PT1H). PT0S restores delete-at-completion
for a deployment that wants no history at all.
FAILED is always retained, regardless of the window — see above. And the audit journal
itself stays opt-in (sutra.audit.sql): with it off, the history endpoint answers an explanatory
empty shape rather than a misleading 404, and the retained terminal snapshot still answers the
inspect route.
stateDiagram-v2
[*] --> RUNNING
RUNNING --> COMPLETED: terminal step re-stamps the snapshot
RUNNING --> TERMINATED: terminal step re-stamps the snapshot
RUNNING --> FAILED: budget spent or non-retryable code
COMPLETED --> [*]: swept past sutra.instance.retention
TERMINATED --> [*]: swept past sutra.instance.retention
note right of FAILED
Retained regardless of the window.
Blocks its deployment's retirement; repairable by migration.
end note
Finishing is a status change, not a disappearance — and FAILED is the one status the retention
sweeper never reaches.
Timer completeness: durations, dates, cycles, and timer starts
All three BPMN timer definitions are supported, on all three timer hosts (start event, intermediate catch, boundary event):
| Definition | Shape | Example |
|---|---|---|
timeDuration | ISO-8601 duration — relative to arming | PT30M, P2D |
timeDate | ISO-8601 instant — an absolute deadline | 2026-12-31T23:59:00Z |
timeCycle | ISO-8601 repeating interval | R3/PT12H, R/P1D |
Deliberately not supported, and rejected as such rather than silently mis-parsed: cron
syntax, and calendar-arithmetic durations (P1M meaning “one calendar month”). A malformed
expression and an unsupported-but-well-formed one produce different diagnostics — “you wrote it
wrong” and “the engine does not do that” are different problems.
Timer start events are schedules
A start event carrying a timer definition is a schedule, not a wait: it has no instance until it fires. Schedules are armed when a deployment becomes active, re-armed on boot, and resolved when the deployment stops being active — so hot-deploy handoff, undeploy, and retirement all need no extra operator step.
A fire mints an instance through the ordinary dispatch path, with empty variables: tenancy, quotas, audit capture, and coverage marking all apply exactly as they do for a message-triggered start.
Two rules keep cycles honest:
- A cycle never bursts. If the engine was down across three occurrences of an
R/PT1Hcycle, it fires once and moves to the next slot — missed occurrences are coalesced, not replayed. - A repeat budget is spent by the grid, not by the firing. Each skipped slot of an
R3/PT1Hconsumes one repeat, so the schedule can never outlive its third slot.
A start event carrying both a <q:source> and a timer definition is a load error — a start is
triggered by a message or by a clock, never ambiguously by both.
Timer intervals from FEEL
Since 0.6.4. Every interval above can be computed instead of written down. That covers the body
of a <timeDuration>, <timeDate> or <timeCycle>, a <q:timeout duration>, and a
<q:retry initialDelay> or maxDelay. The rule is literal first, else FEEL:
| The text | What it is |
|---|---|
parses as an ISO-8601 literal of its form (PT30M, 2026-12-31T23:59:00Z, R3/PT12H) | that literal, exactly as before |
looks like a literal but is written wrong (PT5X, P1M, 2026-03-01, R0/PT1H) | a load error, with the same code as before |
anything else that parses as FEEL (payload.window, config.retryDelay) | an expression |
| anything else | a load error |
Since 0.6.5 an expression may read only names something provides: config, a declared
variable or data object, the payload, or a variable the process (or a process that calls it) sets
before the timer. Any other name would be null when the timer is armed, so sutra lint,
sutra package and the engine refuse it with SUTRA.CONFIG.BPMN.TIMER_EXPRESSION_UNKNOWN_NAME.
A literal written wrong now says which literal was meant. From sutra lint on a package with
one of each:
[ERROR] SUTRA.DEPLOY.PACKAGE.CONFIG_INVALID — BPMN bpmn/wait.bpmn fails to parse: [SUTRA.DISPATCH.TIMER.DURATION_INVALID] <intermediateCatchEvent> Pause timer is not schedulable: 'PT10SEC' has an unparseable number ''; did you mean PT10S?
[ERROR] SUTRA.CONFIG.BPMN.TIMER_EXPRESSION_UNKNOWN_NAME — deployment default/window/1.0.0 (dep-ffffffffffffffffffffffff): process 'wait' node 'Hold' timeDuration expression 'holdFor' reads 'holdFor', which nothing in scope provides (not config, a declared variable, the payload or a variable the process sets), so it would be null when the timer is armed. Declare it (<q:variable>), set it before the timer, or read it from config.<key>.
An expression is evaluated once, when its timer is armed:
| Timer | Armed when | The expression can read |
|---|---|---|
| intermediate catch (also an event-based gateway arm) | the token arrives | the instance’s variables and config |
boundary (timeDuration, timeDate) | the host activity parks | the instance’s variables and config |
<q:timeout> | the call parks (it is a boundary) | the instance’s variables and config |
<q:retry> delays | each failed attempt arms its backoff | the instance’s variables and config |
start event (timeDuration, timeDate, timeCycle) | the deployment activates | config only |
The instance’s variables are whatever the instance holds at that moment: the payload, a DMN
decision’s output, a value read from a store. config holds the deployment’s
properties:, all as strings. A
start event has no instance yet, so its expression may read config and nothing else.
What the expression must return:
| Form | Accepted |
|---|---|
| duration | an ISO-8601 duration string, or a FEEL days-and-time duration (duration("PT" + string(n) + "S")) |
| date | an instant string with a zone, or a FEEL date and time with an offset or zone |
| cycle | a repeating-interval string (R/PT1H) |
A null, a number, a negative or calendar-length duration (P1M), an instant with no zone, or
any other value fails the instance once with SUTRA.RUNTIME.TIMER.EXPRESSION_INVALID. It
gets no default and no retry: <q:retry> does not count the failure, and an error boundary does
not catch it. If the timer is armed by the request that starts the instance, the request is
refused with that code and nothing is stored. On a start event, the deployment refuses to
activate.
Only the instant the expression resolves to is stored: the wait row’s due time, the retry backoff, the schedule row’s spec. A resume, a replay, a restart or a replica hand-off reads that instant and never evaluates the expression again. A loop that comes back to the timer arms it again, so the expression is evaluated again.
Walkthrough: a delivery window
examples/delivery-window waits at a receive task for a delivery and flags the order late when
the window closes. The order can name its own window. Otherwise the window is the deployment’s
default:
POST /channels/orders { orderId, window? }
AwaitDelivery receive task, answered on POST /channels/orders/delivered { orderId }
boundary WindowClosed: payload.window, else config.defaultWindow
-> Delivered a delivery inside the window
-> Late the window closed first
<bpmn:boundaryEvent id="WindowClosed" name="No delivery in the window" attachedToRef="AwaitDelivery">
<bpmn:outgoing>F3</bpmn:outgoing>
<bpmn:timerEventDefinition id="WindowClosed_Timer">
<bpmn:timeDuration>if payload.window = null then config.defaultWindow else payload.window</bpmn:timeDuration>
</bpmn:timerEventDefinition>
</bpmn:boundaryEvent>
# package.yaml
properties:
defaultWindow: "${DEFAULT_WINDOW:PT10S}"
The example’s compose.yaml gives the engine DEFAULT_WINDOW=PT8S.
Prerequisites: Before you try a walkthrough,
with an engine image and a sutra CLI of 0.6.4 or later.
1. Start the stack and deploy the package. The timers helper lists the timer rows, with
how many seconds after arming each one is due:
C=examples/delivery-window/deploy/compose.yaml
docker compose -f $C up -d --wait
sutra package examples/delivery-window/deployments-src/default--window--1.0.0 \
--out examples/delivery-window/deploy/deployments 2>/dev/null
PORT=$(docker compose -f $C port engine 8080 | cut -d: -f2)
H=(-H 'Content-Type: application/json' -H 'X-Api-Key: dev-only-window-key')
until curl -sf "http://127.0.0.1:$PORT/sutra/deployments" | grep -q '"phase":"Active"'; do sleep 1; done
send() { curl -s -w " HTTP %{http_code}\n" -X POST "http://127.0.0.1:$PORT/channels/orders$1" "${H[@]}" -d "$2"; }
timers() {
docker compose -f $C exec -T engine-db psql -U postgres -d sutra -At -c \
"select node_id, status, extract(epoch from timer_due_at - created_at)::int as seconds
from waiting_event where kind = 'TIMER' order by created_at"
}
id_of() { curl -s "http://127.0.0.1:$PORT/admin/instances/by-alias/orderId/$1" | python3 -c 'import json,sys; print(json.load(sys.stdin)["instanceId"])'; }
status_of() { curl -s "http://127.0.0.1:$PORT/sutra/instances/$1" | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["status"], d.get("failureCode") or "")'; }
packaged examples/delivery-window/deploy/deployments/default--window--1.0.0.sutra (deploymentId dep-99c53c0c698e17b631a9fdcf)
2. An order with its own window. The boundary is due 3 seconds after the receive task
parked. Nothing is delivered, so after 5 seconds the instance has taken the Late path:
send "" '{"orderId":"o-1","window":"PT3S"}'
O1=$(id_of o-1)
timers
sleep 5
status_of $O1
curl -s "http://127.0.0.1:$PORT/admin/instances/$O1/history" | python3 -c '
import json, sys
for e in json.load(sys.stdin)["events"]:
if e["eventType"] == "NODE_LEFT": print(e["nodeId"])'
HTTP 200
WindowClosed|WAITING|3
COMPLETED
Start
AwaitDelivery
Late
3. An order without one. The window is config.defaultWindow: PT8S, from the engine’s
environment. A delivery inside it completes the instance on the normal path:
send "" '{"orderId":"o-2"}'
O2=$(id_of o-2)
timers
send /delivered '{"orderId":"o-2"}'
status_of $O2
HTTP 200
WindowClosed|RESOLVED|3
WindowClosed|WAITING|8
HTTP 200
COMPLETED
4. A window that is not a duration. The expression returns the string "soon" when the
receive task parks. The request is refused and nothing is stored:
send "" '{"orderId":"o-3","window":"soon"}'
{"type":"urn:bpm:diag:SUTRA.RUNTIME.TIMER.EXPRESSION_INVALID","title":"SUTRA.RUNTIME.TIMER.EXPRESSION_INVALID","status":500,"detail":"timer 'WindowClosed' <timeDuration> expression 'if payload.window = null then config.defaultWindow else payload.window' could not be armed: it returned the string \"soon\", which is not a usable duration: 'soon' is not an ISO-8601 duration (must start with 'P')","code":"SUTRA.RUNTIME.TIMER.EXPRESSION_INVALID","attributes":{"incident":"SUTRA.INBOUND.NON_IDEMPOTENT_FAILURE"}} HTTP 500
When the timer is armed later in an instance’s life (after a first wait, say), the same failure
leaves the instance FAILED, with this code as its failureCode.
5. Change the default without touching the archive. Restart the engine with a different
DEFAULT_WINDOW. The archive and its deploymentId stay the same; new orders get the new window:
docker compose -f $C stop engine
DEFAULT_WINDOW=PT20S docker compose -f $C up -d --wait engine
PORT=$(docker compose -f $C port engine 8080 | cut -d: -f2)
until curl -sf "http://127.0.0.1:$PORT/sutra/deployments" | grep -q '"phase":"Active"'; do sleep 1; done
send "" '{"orderId":"o-4"}'
timers
HTTP 200
WindowClosed|RESOLVED|3
WindowClosed|RESOLVED|8
WindowClosed|WAITING|20
6. Clean up:
docker compose -p sutra-delivery-window down -v
Next
- The q: namespace —
<q:retry>in the vocabulary table alongside<q:timeout>. - Deployment packages — the
properties:a timer expression reads asconfig.<key>. - Testing time — proving a
PT24Htimer or anR3/PT12Hschedule in wall-clock seconds. - Instance migration — what to do with an instance that exhausted its budget.
- Configuration reference — the retention, sweep, and outbox keys named on this page.
- Retry machinery: the design reasoning — why a backoff is a park, and how a dead attempt is told apart from an in-flight one.
Escaping free text in an XML template
A template renders {{x}} verbatim. That is right for values you control, and wrong for free text
a caller sent you: a customer called O'Brien & Sons or a note with a < in it makes the message
malformed XML. {{xmlEscape x}} (new in 0.6.25; reference in
Template helpers) escapes the value for XML.
This walkthrough answers a JSON order with an XML receipt twice, once escaped and once verbatim. It was run against the 0.6.25 engine and PostgreSQL 16, with the output shown below.
1. The package
package.yaml labels the module receipt, version 1.0.0. channels.yaml declares two JSON
channels, one per template:
channels:
- name: order
transport: http
bind: "POST /channels/order"
codec: json
cloudevents-mode: none
auth: { scheme: apikey, apikey: { value: env:RECEIPT_API_KEY, header: X-Api-Key } }
- name: order-raw
transport: http
bind: "POST /channels/order-raw"
codec: json
cloudevents-mode: none
auth: { scheme: apikey, apikey: { value: env:RECEIPT_API_KEY, header: X-Api-Key } }
bpmn/receipt.bpmn binds the request to order and answers with a template:
<bpmn:process id="receipt" isExecutable="true">
<bpmn:extensionElements>
<q:variables>
<q:variable name="order" type="object"/>
</q:variables>
</bpmn:extensionElements>
<bpmn:startEvent id="Start">
<bpmn:extensionElements><q:source channel="order" name="order"/></bpmn:extensionElements>
</bpmn:startEvent>
<bpmn:serviceTask id="Answer" implementation="receipt.hbs">
<bpmn:extensionElements><q:reply mode="native" contentType="application/xml"/></bpmn:extensionElements>
</bpmn:serviceTask>
<bpmn:endEvent id="Done"/>
<bpmn:sequenceFlow id="F1" sourceRef="Start" targetRef="Answer"/>
<bpmn:sequenceFlow id="F2" sourceRef="Answer" targetRef="Done"/>
</bpmn:process>
bpmn/receipt-raw.bpmn is the same process with the id receipt-raw, the channel order-raw
and the template receipt-raw.hbs.
templates/receipt.hbs escapes every value taken from the request:
<Receipt customer="{{xmlEscape order.customer}}"><Note>{{xmlEscape order.note}}</Note><Ref>{{xmlEscape order.ref}}</Ref></Receipt>
templates/receipt-raw.hbs does not:
<Receipt customer="{{order.customer}}"><Note>{{order.note}}</Note></Receipt>
2. Package it and start the engine
sutra lint receipt
sutra package receipt -o deployments
Lint reports no errors. It warns that the json channels declare no message types, which is
expected for a schema-less channel.
Start the engine with SUTRA_DEPLOYMENTS_DIR pointing at deployments, SUTRA_DATASOURCE_URL
and its credentials pointing at PostgreSQL, and RECEIPT_API_KEY set. Set B to the engine’s
address.
3. Send free text
cat > order.json <<'JSON'
{"customer":"O'Brien & Sons","note":"Leave at the door if <2 kg. \"Fragile\" — 壊れ物","ref":"A-1042"}
JSON
curl -s -X POST "$B/channels/order" -H 'X-Api-Key: <your key>' \
-H 'Content-Type: application/json' -d @order.json
<Receipt customer="O'Brien & Sons"><Note>Leave at the door if <2 kg. "Fragile" — 壊れ物</Note><Ref>A-1042</Ref></Receipt>
The five XML special characters are escaped; the dash and the Japanese text pass through unchanged.
The same order on the verbatim channel:
curl -s -X POST "$B/channels/order-raw" -H 'X-Api-Key: <your key>' \
-H 'Content-Type: application/json' -d @order.json
<Receipt customer="O'Brien & Sons"><Note>Leave at the door if <2 kg. "Fragile" — 壊れ物</Note></Receipt>
Save each answer to a file and give it to an XML parser. Python’s xml.dom.minidom accepts the
first and refuses the second:
receipt.xml: well-formed
receipt-raw.xml: not well-formed (invalid token): line 1, column 28
4. A missing value
Leave ref out:
curl -s -X POST "$B/channels/order" -H 'X-Api-Key: <your key>' \
-H 'Content-Type: application/json' -d '{"customer":"Ada","note":"none"}'
<Receipt customer="Ada"><Note>none</Note><Ref></Ref></Receipt>
xmlEscape renders a missing or null value empty. A plain {{order.note}} on a missing value
fails the render instead, as strict mode requires. Leaving note out on order-raw answers HTTP
500:
unresolved template reference {{order.note}} (strict rendering: every referenced value must be present; use {{coalesce x "default"}} for optional values)
If an element must not be empty, check the value in the flow before the template runs.
Testing time: fast-forwarding durable timers
A flow that waits PT24H for a reminder, or fires an R3/PT12H schedule three times, is exactly
the kind of thing that is hard to test and therefore usually untested. Sutra’s answer is a
virtual clock: the engine’s notion of “now” becomes something a test drives explicitly, so a
day-long timer settles in wall-clock seconds against a real database, with real durable rows and
the real timer poller.
sequenceDiagram
participant T as Test
participant E as Engine
participant DB as PostgreSQL
T->>E: send the message that starts the flow
E->>DB: park: durable timer row, due in PT24H
Note over E,DB: real row, real poller — only "now" is virtual
T->>E: fast-forward the clock past the due time
E->>DB: timer is due: claim + fire
E-->>T: the reminder path ran
A day-long timer settles in milliseconds because the clock moved, not because the timer was shortened — the durable rows and the code path are the production ones.
Two ways in: a CLI command for BPMN authors, and an embedded seam for anyone writing tests in Rust.
sutra test simulate — no test code required
sutra test simulate --deployments <DIR> --datasource <URL> \
(--advance <DURATION> | --until-quiescent) [flags]
Boots a real engine on a dynamic port against a directory of sealed deployment archives with a virtual clock installed, fast-forwards it, reports, and shuts down cleanly.
This is unrelated to sutra simulate, which is a dry-run routing report over a single BPMN
file and never boots anything. They are separate commands on purpose.
| Flag | Meaning |
|---|---|
--deployments <DIR> | Directory of sealed .sutra archives to serve. Required. |
--datasource <URL> (--datasource-username / --datasource-password) | Engine datasource. Required — the same canonical SUTRA_DATASOURCE_* env names a real engine container reads. |
--advance <DURATION> | Fast-forward the virtual clock by this ISO-8601 duration, firing everything due along the way, then stop and report. |
--until-quiescent | Fast-forward until nothing is armed and nothing is live, or --timeout elapses. |
--timeout <DURATION> | Real wall-clock budget for the fast-forward loop, either mode. Default PT30S. |
--start <RFC3339> | Virtual start instant. Default: the real current instant. |
--allow-existing-data | Proceed even though the datasource already holds instances. |
Exactly one of --advance / --until-quiescent is required.
It refuses to run against data it did not expect
Fast-forwarding a virtual clock against a database that already holds real, in-flight instances
would durably fire their real timers early. A datasource mixup must not silently do that, so
before booting anything the command checks that the target database holds no instances, and
refuses with exit 2 if it does.
--allow-existing-data is the explicit acknowledgement — and also the right shape for “seed one
instance externally, then fast-forward it”.
--datasource being required is part of the same posture: a persistence-less run has no durable
timers to advance at all, so the command refuses rather than pretending to have done something.
Quiescence, precisely
--until-quiescent stops when, simultaneously: no waiting row is an armed timer, no schedule is
armed, and no instance is still live.
A FAILED instance is deliberately not counted as finished — it needs an operator, not a clock.
So a fixture that fails under fast-forward correctly reports timedOut: true rather than a false
“quiescent”, and the summary’s instancesFailed says why.
Output
Human-readable progress goes to stderr. The final summary is one JSON object on stdout and
nothing else touches stdout, so piping into jq is always safe:
{
"mode": "until-quiescent",
"deployments": "/path/to/deployments",
"allowExistingData": false,
"timedOut": false,
"quiescent": true,
"preExistingInstances": 0,
"virtualStart": "2026-08-06T00:00:00Z",
"virtualEnd": "2026-08-07T00:00:03Z",
"virtualSecondsAdvanced": 86403.2,
"wallSeconds": 1.84,
"instancesStarted": 1,
"instancesCompleted": 1,
"instancesFailed": 0,
"instancesTerminated": 0,
"instancesLive": 0,
"timersFired": 1,
"schedulesFired": 0
}
schedulesFired counts single-shot and bounded (R<n>/…) timer-start fires exactly. An unbounded
cycle (R/…) has no repeat budget to difference, so its fires show up in instancesStarted but
are not separately counted in this release.
Use it to prove, in your application’s own CI, that a durable-timer or cyclic-schedule flow settles correctly — without a wall-clock-length test run.
The embedded seam
sutra_engine::TestClock is a manually-advanced virtual clock, and
sutra_engine::fast_forward_until is the paired driver. Install the clock on the engine config
before serving, and every temporal read in that boot uses it: timer park due instants, <q:retry>
backoff instants, the timer poller’s per-tick claim instant, and schedule arming.
#![allow(unused)]
fn main() {
let clock = sutra_engine::TestClock::starting_now();
let engine = sutra_engine::serve(sutra_engine::EngineConfig {
now_override: Some(clock.clone()),
..config
}).await?;
// park a PT24H timer, then:
let settled = sutra_engine::fast_forward_until(
&pool, &clock, std::time::Duration::from_secs(10),
|| async { live_instance_count(&pool).await == 0 },
).await;
assert!(settled);
}
Every clone of the clock is a handle onto the same instant, so the code under test and the test
driving it always agree. starting_now() starts at the real current instant — only the
fast-forward is virtual — which keeps any absolute-time assertion elsewhere in the test (log
timestamps, audit rows) sane.
The driver loop is deliberately simple: jump the clock to the earliest armed due instant, give the real timer poller a tick to claim and fire it, re-check your condition; repeat until the condition holds or the real wall-clock timeout elapses. It only ever touches the clock and the database — never the poller’s internals. Set the poller’s tick interval low for the test boot so a tick is cheap.
It is unreachable from a deployed engine — by construction
There is no config key, no environment variable, and no CLI flag on the engine binary that installs a virtual clock. Wiring one in is always an explicit choice in Rust at the call site, so an operator has no way to reach it in a deployed engine.
sutra test simulate is not a hole in that: it is a different binary that constructs an engine
configuration directly in code — exactly the sanctioned pattern, just packaged as a reusable tool
instead of a bespoke integration test per application. The engine’s own configuration loading is
untouched by it.
What it is good for
- A
timeDurationcatch timer or<q:timeout>boundary that would otherwise need a real wait. - A
timeCycleschedule firing its full repeat budget. - A
<q:retry>backoff curve running to exhaustion — every attempt’s park is a real durable row, and fast-forward walks them all.
See Retries, history, and schedules for what each of those constructs does, and Testing tiers for where a test like this belongs in the suite.
Next
- Retries, history, and schedules — the timers being fast-forwarded.
sutraCLI reference — the command in the full CLI map.
Coverage: declared routes as the compliance signal
Compliance in a Sutra deployment isn’t reconstructed from logs after the fact. A module owner declares the business-event routes that matter — “the happy-path completion,” “the reject branch,” “the end-to-end route across all three collaborating processes” — and the engine ticks each one off as real instances walk it. That declaration is the whole compliance surface: a route nobody declared can never show up as covered or uncovered, so what you choose to declare is exactly what you can ever detect. This chapter covers the two shapes a route can take, how to declare each, the CLI that drives them, and — the part that actually determines whether the feature is useful — how to curate the declared set down to what’s worth watching.
What “covered” means
A route is an ordered path through execution: for an intra-process route, an ordered list of
one process’s own sequence-flow ids; for a cross-process route, a per-process segment of flows
for each participant in a correlated cascade, tied together by a business key that threads every
hop. A route is marked covered the moment an instance (or, cross-process, a correlated group of
instances) walks every one of its flows in order; it stays uncovered since the last reset
otherwise. There is no partial credit and no percentage for anything you didn’t declare — total
is exactly the number of declared routes, covered is how many have flipped, and the uncovered
list is the rest.
Both shapes land in one place: the coverage data store the deployment declares in its own
datastores.yaml, as a typed coverage_metric row per declared route carrying a covered boolean
— entirely separate from the audit trail, so coverage never reads audit tables and audit never reads
coverage’s. That’s also what makes coverage cheap to skip: a process with no declared route pays
nothing (no flow-trace capture, no writes). See
Where coverage is stored for the division of labour that store implies:
you pick the database, the engine owns the schema.
| Shape | Declared | Spans |
|---|---|---|
| Intra-process | <q:coverage path="…" flows="…"/> inline on a <bpmn:process> | one process |
| Cross-process | a coverage/<name>.yaml file, URN-identified urn:sutra:coverage:<name> | several correlated processes of one deployment |
flowchart LR
D1["q:coverage on a process<br/>an ordered list of flow ids"] --> SEED
D2["a coverage/name.yaml route<br/>a segment per participant process"] --> SEED
SEED["on activation — seed one coverage_metric row<br/>per declared route, covered = false"] --> ST[("the coverage store<br/>the deployment declares")]
RUN["a real instance's fired-flow trace"] -->|"contains the route's flows<br/>in order, as a subsequence"| UP["guarded UPDATE … AND NOT covered —<br/>the affected-row count is the answer to<br/>did this run newly cover it"]
UP --> ST
ST --> REP["total, covered, uncovered —<br/>one aggregate and one ordered query,<br/>in a single REPEATABLE READ snapshot"]
Declaration is what creates the measurable surface: total is exactly the seeded set, so a route
nobody declared can never appear as covered or uncovered, and the percentage is derived on read
rather than kept as a counter that could drift.
Intra-process: <q:coverage> inline
The simplest case declares a route directly on the process it belongs to, as an ordered list of sequence-flow ids — the same ids the diagram already shows on every arrow:
<bpmn:process id="transfer">
<bpmn:extensionElements>
<q:coverage path="accept" flows="Flow_TxToOk Flow_OkToEnd"/>
<q:coverage path="reject" flows="Flow_CancelToReject Flow_RejectToEnd"/>
</bpmn:extensionElements>
…
</bpmn:process>
A run covers a path when its fired-flow trace contains flows as an ordered subsequence — so
whatever intake channel started the instance (money-transfer’s transfer.bpmn has three) is
irrelevant to whether accept or reject gets marked, and a wait-state park in the middle doesn’t
break the match. See Worked example: money-transfer
for this declaration driving a real ACID transfer flow end to end.
Fail-closed checks at load: a flow id the process doesn’t have, a set of flows that isn’t a
contiguous route (each flow’s target must be the next flow’s source), or a reused path id inside
one process all evict the module rather than silently declare something meaningless. A process that
declares at least one <q:coverage> must also declare a coverage store in datastores.yaml
(SUTRA.CONFIG.COVERAGE.STORE_MISSING at lint time) — that store is where the marks are
persisted, so declaring routes without it is declaring something with nowhere to record it. The
scaffold satisfies the rule for you; Where coverage is stored is what
the declaration actually buys.
Cross-process: a coverage file
A route that spans more than one participant process can’t live inside a single <bpmn:process> —
there’s no BPMN element for “the flow that starts here and finishes three hops away in a different
process.” Instead it’s declared in a coverage file under the package’s coverage/ folder,
identified the same way a template is: path-derived, urn:sutra:coverage:<folder…>:<file>. A file
holds one or more correlations — the business key that ties instances of the cascade together,
and the links (hops) between them — plus one or more routes (coverages:) that each state
a complete set of per-process segments.
The shape that motivates this is a relay cascade: three collaborating services — an intake
service, a routing hub, and a fulfillment service — each its own <bpmn:process>, handing off over
channels rather than calling each other directly. No single process sees the whole exchange, so
“did the end-to-end handshake complete, and which way did it end” is exactly the question none of
them can answer alone. A coverage/e2e.yaml (urn:sutra:coverage:e2e) declares the routes that
span all three:
correlations:
- id: relay-case
key: caseId # default hop key — every leg correlates on the same case id
links: # the <q:send> -> <q:source>/imec hops between the 3 processes
# forward: intake -> hub (spawns the hub)
- { from: intake-svc:Task_ForwardToHub, to: hub-svc:Start_Received }
# forward: hub -> fulfillment (spawns fulfillment)
- { from: hub-svc:Task_ForwardToFulfillment, to: fulfillment-svc:Start_Posting }
# reply: fulfillment -> hub (resumes the hub's imec) - accept and reject both land here
- { from: fulfillment-svc:Task_ReplyAccept, to: hub-svc:Imec_AwaitFulfillment }
- { from: fulfillment-svc:Task_ReplyReject, to: hub-svc:Imec_AwaitFulfillment }
# reply: hub -> intake (resumes the intake's imec)
- { from: hub-svc:Task_ReplyToIntake, to: intake-svc:Imec_AwaitReply }
coverages:
- path: e2e-accepted # the accepted branch
segments:
intake-svc: [Flow_p1_start, Flow_p1_await, Flow_p1_done]
hub-svc: [Flow_p2_start, Flow_p2_await, Flow_p2_reply, Flow_p2_done]
fulfillment-svc: [Flow_p3_start, Flow_p3_accept, Flow_p3_accept_end]
- path: e2e-rejected # the rejected branch — same p1/p2, different p3
segments:
intake-svc: [Flow_p1_start, Flow_p1_await, Flow_p1_done]
hub-svc: [Flow_p2_start, Flow_p2_await, Flow_p2_reply, Flow_p2_done]
fulfillment-svc: [Flow_p3_start, Flow_p3_reject, Flow_p3_reject_end]
A few things about this shape:
linksare structural, not payload access. Each hop names a<q:send>node on one side and either a start-event<q:source>(a spawn — the hop starts a new instance) or animecrelay-wait<q:source>(a relay — the hop resumes a parked instance) on the other, matched against the actual channel wiring.sutra lintvalidates every link resolves against real channel bindings, that everysegmentsflow id is contiguous within its own process, and that the correlationkeyresolves at both ends of every hop — a broken link or an unresolvable key evicts the module rather than deploying a route that can never complete.- The correlation key is how the runtime reconstructs the cascade, since three separately
dispatched instances have no shared instance id. Here it’s a single value — a case id — carried
as an author-declared header rather than a payload field: every
<q:send>sets it (<q:header name="caseId" value="caseId"/>) and every consuming<q:source>reads it back (<q:alias name="caseId" expression="header.caseId"/>). A hop can override the correlation’s defaultkeywhen its own leg correlates on a different value — this cascade doesn’t need to, since one case id threads every hop end to end. - Each route is fully self-contained.
e2e-acceptedande2e-rejectedrepeat their identicalintake-svcandhub-svcsegments rather than one inheriting from the other — there is no route-to-route inheritance, by design, so reading any one route never requires cross-referencing another. - A route can be declared but structurally rare.
e2e-rejectedis only reachable when the fulfillment process takes its reject branch — under normal traffic that rarely happens, soe2e-rejectedcan sit uncovered indefinitely. That’s not a bug in the declaration; it’s the compliance signal doing its job: “the reject path exists and we watch for it,” distinct from “the reject path fired.” It also makes a clean assertion for a test campaign: drive one ordinary case, expecte2e-acceptedto reconstruct as covered ande2e-rejectednot to, thenresetand confirm both go back to uncovered.
At load, each route’s per-process segments are injected as ordinary intra-process coverage paths
on their own process (so the existing per-process marking is unchanged); on completion each
segment writes a correlation-tagged record, and a route flips covered once every one of its
segments has landed in the same correlated group. Reading total/covered/percentage for a
cross-process route is the same store query as an intra-process one — the mechanism composes
rather than duplicating.
flowchart LR
subgraph SEG["one route's per-process segments"]
A["intake-svc"]
B["hub-svc"]
C["fulfillment-svc"]
end
A -->|"a correlation-<br/>tagged record"| G
B --> G
C --> G
G["one group, keyed by<br/>the caseId every hop carries"] -->|"all segments in"| F["the route flips covered<br/>counted once in total"]
Three separately dispatched instances share no instance id, so the correlation key is the whole reconstruction; the segments are marking cursors, which is why many of them collapse onto the one route flag a report reads.
Composing several definitions in one package
Nothing limits a package to one kind of declaration. A deployment can carry <q:coverage> on
several of its own processes and one or more coverage/*.yaml files, each with its own
correlation and its own set of routes — money-transfer’s inline accept/reject pair and the
file-based e2e-accepted/e2e-rejected pair above are the two shapes, and a single package is
free to declare both kinds side by side: an intra-process route for “did this
one process’s own retry/compensation branch fire,” alongside a cross-process file for “did the
whole multi-participant handshake complete.” Each declaration is independent — an intra-process
path only needs its own process’s flow ids; a coverage file only needs the processIds and channel
wiring it names — so there’s no coordination cost to adding more of either kind as a module grows.
Where coverage is stored
Coverage marks are persisted in the coverage data store the deployment declares — a reserved
store name in its own datastores.yaml, declared like any other store. That declaration is how you
choose the database: the connection it names is where the marks land, and the URL scheme picks the
dialect (PostgreSQL, MySQL/MariaDB and SQL Server all work). The engine hosts no coverage in its own
database.
What you do not write is coverage SQL. The coverage tables are a built-in feature, so the engine
owns their schema: it ships the coverage_metric / coverage_fragment DDL per dialect and
applies it to that connection the first time the store is used — the same idempotent,
lock-serialized first-use path a module’s own migrations/<store>/ scripts take, with
engine-shipped scripts instead of package-supplied ones. So the store block carries no
migrations: key, and a package carries no migrations/coverage/ folder at all:
datastores:
- name: coverage # the reserved name — this declaration picks the DATABASE
type: sql
sql:
url-ref: env:ACCOUNTS_DB_URL
username-ref: env:ACCOUNTS_DB_USER
password-ref: env:ACCOUNTS_DB_PASSWORD
# no `migrations:` — the engine owns this store's schema
Pointing it at a connection some other store already uses is fine and often convenient —
money-transfer’s coverage store names the same database as its accounts ledger. The coverage
tables are the engine’s own; they sit beside the business ones rather than mixing with them.
On activation, a deployment seeds one row per declared route into that store’s coverage_metric
table with covered = false — intra-process path ids and cross-process route ids alike. (A
cross-process route’s per-process segments are marking cursors, not routes: many segments collapse
onto the one route flag, which is why total counts the route once.) Execution flips flags with a
guarded UPDATE … AND NOT covered, so first-covers-wins is settled by the write itself — the
affected-row count is the answer to “did this run newly cover it,” never a read-then-write race.
Nothing else writes there. That typed table is the whole coverage substrate, and three consequences
follow from it.
Counts are SQL, not a fold in the engine. total and covered are one aggregate over the
seeded set (COUNT(CASE WHEN … THEN 1 END) — the portable pivot, which returns a count on every
shipped dialect), and the uncovered routes are their own ordered query. Both run inside one
REPEATABLE READ transaction, so the count and the list can never disagree: they describe one
snapshot. A percentage is still derived on read, never stored — there is no counter to drift.
Flags are keyed by deployment and route id. deployment_id is a column on both tables and a
bound predicate on every statement the engine issues; that, not row-level security, is the
isolation. (RLS is an engine-database convention — it wants table ownership and a per-transaction
setting the engine cannot assume on a connection you own, and two of the three shipped dialects have
no equivalent anyway.) Two processes in one deployment that both declare a path called accept
share one flag; the CLI’s total has always counted them once. Name a route for the business event
it represents rather than for the branch it sits on, and the question never comes up. The same
keying is what makes a runtime report of a cross-process route read against the route flag rather
than the segment that happened to finish — one number, and the same one
sutra coverage check --archive reports.
A declared store the engine can’t open fails loudly. Coverage exists wherever you declare a
store, so the failure mode is no longer “this deployment has no database of its own to record in” —
it is a coverage store whose connection is wrong, unset or unreachable. The engine logs an
error at boot naming that deployment, and a coverage:report / coverage:reset op fails
with SUTRA.CONFIG.COVERAGE.STORE_MISSING naming the cause. It does not report 0%:
a report of 0% here would be indistinguishable from a real measurement of nothing covered
A deployment that declares <q:coverage> paths but no coverage store at all is the same story one
step earlier: sutra lint errors on it at package time, and an engine that loads such a package
anyway warns at boot and fails those two ops with the same code. The reasoning extends to a store
that opens but whose read fails — the op surfaces the underlying error rather than counting every
route as uncovered. A compliance signal that quietly degrades to “nothing is covered” is worse than
one that stops and says why. (Runtime marking stays best-effort — a metric side-effect must never
fail a business instance. The loud surface is the report, which is what a human or a CI gate
actually reads.)
A package built against an older engine may still carry a migrations/coverage/ folder. It is dead
weight: nothing in it was ever coverage DDL — the script there created the same generic table a
business store already creates — and the engine no longer reads a migrations: key on the reserved
coverage store. Delete the folder.
The sutra coverage CLI
init — enumerate and seed
Single-process form: point it at a BPMN file and it walks every start→end route over the engine’s
own execution semantics (gateways, sub-processes, the lot), then seeds <q:coverage> declarations
plus a matching admin scaffold (report/reset BPMN, reply templates, the two admin channels, and the
coverage store declaration — no SQL, because none of it is yours to write):
$ sutra coverage init bpmn/transfer.bpmn --process transfer
coverage init: bpmn/transfer.bpmn — process 'transfer', 2 path(s) declared (0 kept, 2 new)
path path-1: Flow_TxToOk Flow_OkToEnd
path path-2: Flow_CancelToReject Flow_RejectToEnd
updated bpmn/transfer.bpmn
created bpmn/coverage-report.bpmn
created bpmn/coverage-reset.bpmn
created templates/coverage-report.hbs
created templates/coverage-reset.hbs
updated channels.yaml
updated datastores.yaml
Route enumeration refuses beyond a 256-route cap (--max-paths raises it): a process with
enough parallel/inclusive branching to combinatorially explode past that is a sign the process
itself is too fine-grained for route-level compliance tracking, not that the cap is wrong — raise
--max-paths only when you’ve confirmed the extra routes are genuinely distinct business
outcomes, not gateway noise. A second init run keeps any path id whose flows still match
(renames survive), refuses to clobber hand-edited declarations without --force, and never
replaces a file it can’t safely re-parse.
Cross-process form: name a coverage file and the processIds it should span instead of a single
BPMN file. This does not enumerate a path set at all (there’s no combinatorial explosion to
cap) — it emits the connectable graph: each process’s own flow adjacency, plus every
inter-process hop it can infer from <q:send>/<q:source>/imec wiring, as a commented scaffold
with one starter route to trim:
$ sutra coverage init coverage/e2e.yaml intake-svc hub-svc fulfillment-svc
coverage init: coverage/e2e.yaml — urn:sutra:coverage:e2e
processes: intake-svc, hub-svc, fulfillment-svc
intra-process adjacency: 9 flow(s) across 3 process(es)
inter-process hops: 5
intake-svc:Task_ForwardToHub --to-hub--> hub-svc:Start_Received [spawn fire-and-forget] key=caseId
hub-svc:Task_ForwardToFulfillment --to-fulfillment--> fulfillment-svc:Start_Posting [spawn fire-and-forget] key=caseId
fulfillment-svc:Task_ReplyAccept --to-hub-imec--> hub-svc:Imec_AwaitFulfillment [relay request-reply] key=caseId
fulfillment-svc:Task_ReplyReject --to-hub-imec--> hub-svc:Imec_AwaitFulfillment [relay request-reply] key=caseId
hub-svc:Task_ReplyToIntake --to-intake-imec--> intake-svc:Imec_AwaitReply [relay request-reply] key=caseId
scaffold written — draw connected walks from it (sutra lint validates connectivity)
The written file’s coverages: starts with exactly one route (path-1) whose segments list
every connectable flow id per process — the raw graph, not a curated path. --single restricts
the graph to intra-process adjacency only (no inter-process hops, for a package still being
wired up one participant at a time); --force overwrites an existing file.
check — read covered/uncovered, drive assertions
Bare BPMN file: a drift lint, not a store read. It confirms every declared path is still an ordered subsequence of the process’s current flow graph — the check that catches a declaration silently breaking when someone reworks the diagram — and reports (informationally) any enumerated route no declared path covers:
$ sutra coverage check bpmn/transfer.bpmn
coverage check: 0 error(s), 0 note(s)
--archive selects the store-backed, correlation-aware check instead: it reads the deployment’s
seeded metric flags, union-finds the cross-process reconstruction records to flip any route that
has now fully completed, and reports the same total/covered/percentage a live dashboard would
read — the fail-closed CI gate and the runtime signal are the identical query. --database-url
points at the database the deployment’s coverage store declares, in whichever of the three shipped
dialects it names; the CLI reads and writes exactly what the engine does, and creates nothing the
engine would not create itself:
$ sutra coverage check --archive deployed.sutra --database-url "$SUTRA_DB_URL" --threshold 100
coverage check (cross-process) — deployment 3f9a1c…
total: 2 covered: 1 coveragePercentage: 50.00%
newly covered this run (1):
+ urn:sutra:coverage:e2e:e2e-accepted
uncovered (1):
- urn:sutra:coverage:e2e:e2e-rejected
threshold: 100.00% => FAIL
--threshold (default 100) is the gate: the process exits non-zero whenever the percentage falls
below it, so a CI pipeline can fail a release that hasn’t exercised every route it declared as
required — or, set lower, tolerate a known-rare route like e2e-rejected without losing visibility
into whether it ever fires.
reset — re-seed a deployment’s declared routes
$ sutra coverage reset --archive deployed.sutra --database-url "$SUTRA_DB_URL"
coverage reset — deployment 3f9a1c…: 2 path(s) re-seeded covered=false, reconstruction fragments cleared
Every declared path (intra- and cross-process alike) goes back to covered = false and every
cross-process reconstruction record is cleared — the clean baseline a new test campaign, or a new
reporting period, starts from. The rows stay seeded; reset is one scoped update that flips the
flags, so total is unchanged and the cleared count is exactly how many routes had been covered.
Requirements and compliance cycles
Since 0.6.20 a package can map its declared paths to requirements, and a deployment can keep named compliance cycles, so you can ask “how much of release 4.2’s requirements has the test campaign covered?”
- A requirements map.
coverage/requirements-map.yamlmaps each declared path to requirement ids, a weight (default 1, above zero) and the test cases that walk it.sutra coverage checkthen adds acomplianceobject: per-requirement and weighted percentages, uncovered paths with their cases, the cases to run, and paths with no cases or no mapping.--thresholdgates the weighted percentage. Every declared path counts; there is no way to exclude one. A package with no map reports as before. Lint:SUTRA.CONFIG.COVERAGE.MAPPING_ROUTE_UNKNOWN(error, a key that matches no declared path) andSUTRA.CONFIG.COVERAGE.MAPPING_ROUTE_UNMAPPED(warning, a declared path the map leaves out, counted at weight 1). - Cycles.
sutra coverage cycle declare|activate|deactivate <LABEL>andsutra coverage cycle list, plus--cycleoncoverage checkandcoverage reset. While a cycle is active, every mark for that deployment records to it. A mark belongs to the cycle active when its instance completes. One cycle can be active per deployment: a second activation is refused (SUTRA.RUNTIME.COVERAGE.CYCLE_ALREADY_ACTIVE), as is activating an undeclared label (CYCLE_UNDECLARED); reactivating a cycle continues it. With none active, coverage records as before, andcheckandresetwithout--cycleread the marks recorded outside any cycle. Cycles belong to a deployment id, so a changed package starts with none. Thesutra.coverage.percentgauge and the path counter are not cycle-aware.
Walkthrough
The money-transfer example package with this coverage/requirements-map.yaml:
routes:
accept:
requirements: [REQ-HAPPY]
weight: 3
cases: [case-accept]
reject:
requirements: [REQ-REFUSAL]
cases: [case-reject, case-reject-frozen]
sutra lint and sutra package accept it (no MAPPING_ROUTE_* finding: both declared paths are
mapped). The engine runs from a deployments folder on PostgreSQL; the CLI reaches the same database
through SUTRA_DB_URL. Steps: declare a cycle and activate it; check it (nothing covered, every case
to run); post an accepted request; check again (weighted 75%: accept weighs 3 of 4); the JSON form
with --threshold 75 passes, because the threshold gates the weighted percentage; deactivate, so a
rejected request records outside the cycle; a second active cycle is refused; reactivating
release-4.2 continues it, and the rejected request now covers it fully.
$ sutra coverage cycle declare release-4.2 --archive archives/money-transfer.sutra
coverage cycle declare — deployment dep-43a59818955d7e27dd83b82d: cycle 'release-4.2' declared, 2 path(s) seeded
[exit 0]
$ sutra coverage cycle activate release-4.2 --archive archives/money-transfer.sutra
coverage cycle activate — deployment dep-43a59818955d7e27dd83b82d: cycle 'release-4.2' is active; coverage now records to it
[exit 0]
$ sutra coverage check --archive archives/money-transfer.sutra --cycle release-4.2
coverage check (cross-process) — deployment dep-43a59818955d7e27dd83b82d
cycle: release-4.2
total: 2 covered: 0 coveragePercentage: 0.00%
uncovered (2):
- accept
- reject
weighted: 0.00% (0 of 4 by weight)
requirements (2):
REQ-HAPPY: 0.00% (0 of 1 routes)
REQ-REFUSAL: 0.00% (0 of 1 routes)
uncovered routes and the cases that cover them:
- accept: case-accept
- reject: case-reject, case-reject-frozen
cases to run: case-accept, case-reject, case-reject-frozen
threshold: 100.00% => FAIL
[exit 1]
POST accept: HTTP 200
$ sutra coverage check --archive archives/money-transfer.sutra --cycle release-4.2
coverage check (cross-process) — deployment dep-43a59818955d7e27dd83b82d
cycle: release-4.2
total: 2 covered: 1 coveragePercentage: 50.00%
uncovered (1):
- reject
weighted: 75.00% (3 of 4 by weight)
requirements (2):
REQ-HAPPY: 100.00% (1 of 1 routes)
REQ-REFUSAL: 0.00% (0 of 1 routes)
uncovered routes and the cases that cover them:
- reject: case-reject, case-reject-frozen
cases to run: case-reject, case-reject-frozen
threshold: 100.00% => FAIL
[exit 1]
$ sutra --format json coverage check --archive archives/money-transfer.sutra --cycle release-4.2 --threshold 75
{"command":"coverage check","form":"cross-process","deploymentId":"dep-43a59818955d7e27dd83b82d","total":2,"covered":1,"coveragePercentage":50.0,"threshold":75.0,"uncovered":["reject"],"newlyCovered":[],"passed":true,"cy …
[exit 0]
$ sutra coverage cycle deactivate release-4.2 --archive archives/money-transfer.sutra
coverage cycle deactivate — deployment dep-43a59818955d7e27dd83b82d: cycle 'release-4.2' deactivated; coverage records outside any cycle
[exit 0]
POST reject: HTTP 200
$ sutra coverage cycle list --archive archives/money-transfer.sutra
coverage cycle list — deployment dep-43a59818955d7e27dd83b82d: 1 cycle(s), no-cycle coverage 50.00% (1 of 2)
release-4.2: 50.00% (1 of 2)
[exit 0]
$ sutra coverage cycle declare release-4.3 --archive archives/money-transfer.sutra
coverage cycle declare — deployment dep-43a59818955d7e27dd83b82d: cycle 'release-4.3' declared, 2 path(s) seeded
[exit 0]
$ sutra coverage cycle activate release-4.3 --archive archives/money-transfer.sutra
coverage cycle activate — deployment dep-43a59818955d7e27dd83b82d: cycle 'release-4.3' is active; coverage now records to it
[exit 0]
$ sutra coverage cycle activate release-4.2 --archive archives/money-transfer.sutra
coverage cycle activate — deployment dep-43a59818955d7e27dd83b82d: SUTRA.RUNTIME.COVERAGE.CYCLE_ALREADY_ACTIVE: cycle 'release-4.3' is active, and only one can be. Deactivate it first; 'release-4.2' was not activated
[exit 1]
$ sutra coverage cycle deactivate release-4.3 --archive archives/money-transfer.sutra
coverage cycle deactivate — deployment dep-43a59818955d7e27dd83b82d: cycle 'release-4.3' deactivated; coverage records outside any cycle
[exit 0]
$ sutra coverage cycle activate release-4.2 --archive archives/money-transfer.sutra
coverage cycle activate — deployment dep-43a59818955d7e27dd83b82d: cycle 'release-4.2' is active; coverage now records to it
[exit 0]
POST reject: HTTP 200
$ sutra coverage check --archive archives/money-transfer.sutra --cycle release-4.2
coverage check (cross-process) — deployment dep-43a59818955d7e27dd83b82d
cycle: release-4.2
total: 2 covered: 2 coveragePercentage: 100.00%
uncovered: none
weighted: 100.00% (4 of 4 by weight)
requirements (2):
REQ-HAPPY: 100.00% (1 of 1 routes)
REQ-REFUSAL: 100.00% (1 of 1 routes)
threshold: 100.00% => PASS
[exit 0]
$ sutra coverage cycle deactivate release-4.2 --archive archives/money-transfer.sutra
coverage cycle deactivate — deployment dep-43a59818955d7e27dd83b82d: cycle 'release-4.2' deactivated; coverage records outside any cycle
[exit 0]
$ sutra coverage cycle list --archive archives/money-transfer.sutra
coverage cycle list — deployment dep-43a59818955d7e27dd83b82d: 2 cycle(s), no-cycle coverage 50.00% (1 of 2)
release-4.2: 100.00% (2 of 2)
release-4.3: 0.00% (0 of 2)
[exit 0]
The compliance object of the JSON check above:
{"weightedPercentage":75.0,"weightedTotal":4.0,"weightedCovered":3.0,"requirements":[{"requirement":"REQ-HAPPY","total":1,"covered":1,"coveragePercentage":100.0,"uncovered":[]},{"requirement":"REQ-REFUSAL","total":1,"covered":0,"coveragePercentage":0.0,"uncovered":["reject"]}],"uncoveredRoutes":[{"path":"reject","requirements":["REQ-REFUSAL"],"weight":1.0,"cases":["case-reject","case-reject-frozen"]}],"casesToRun":["case-reject","case-reject-frozen"],"routesWithoutCases":[],"unmappedRoutes":[]}
To re-run one case, send <cycle>:<case>:<attempt> as the idempotency key on a channel that declares idempotency-key-header: a repeat of an attempt does nothing, the next attempt runs. This is covered by the tier-2 conformance suite and not walked here.
Curation: init enumerates, you decide what matters
init’s job is mechanical completeness — surface every structurally valid route (or, cross-process,
the whole connectable graph) so nothing gets missed by hand. It is deliberately not trying to
guess which of those routes represents a business event worth watching, and treating its raw output
as the final declared set is the single most common way this feature stops being useful.
Notice, in the single-process example above, that init names its two routes path-1 and
path-2 — it has no way to know one is the accepted-transfer outcome and the other is the rejected
one. Money-transfer’s shipped transfer.bpmn renames them to accept and reject by hand — that
rename is the curation step, and it’s why sutra coverage check --archive output reads as a
compliance report (“the reject path never fired this quarter”) rather than a cryptic route id. The
same discipline applies to the cross-process scaffold: coverage init’s starter route lists every
connectable flow per process, which is almost never the walk you actually care about — the
e2e.yaml above is that raw scaffold trimmed down to exactly the two branches (accepted vs.
rejected) the cascade wants visibility into, with routes named for the outcome they represent
rather than left as path-1.
This matters because “uncovered” is the entire signal, and noise defeats it. A moderately branchy process can mechanically enumerate dozens of technically-distinct full paths — different orderings through independent parallel branches that don’t correspond to different business outcomes at all. Declare all of them and every report is dominated by routes nobody ever intended to track individually, burying the one genuinely-uncovered path that matters under a wall of noise. Keep the declared set as large as the compliance surface you actually want visibility into — named after the business events they represent — and no larger.
Next
- Worked example: money-transfer — the single-process form driven by a real ACID transfer flow.
sutraCLI — every flag oninit/check/reset.- Troubleshooting BPMN solutions — reading a coverage report that doesn’t match what you expected.
Worked example: money-transfer
examples/money-transfer is the flagship demonstration of full ACID semantics on a single
writable ledger, driven from one BPMN process over three different transports (HTTP, RabbitMQ,
Kafka). It exercises most of what the last few chapters covered — deployment packages, channels,
the q: namespace, and data stores — in one real flow.
The shape
examples/money-transfer/deployments-src/default--money-transfer--1.0.0/
├── package.yaml # labels {tenant: default, module: money-transfer, version: 1.0.0}
├── bpmn/transfer.bpmn # the ACID transfer flow
├── bpmn/balance-query.bpmn # read-only balance lookup
├── bpmn/coverage-report.bpmn # admin: path-coverage report
├── bpmn/coverage-reset.bpmn # admin: path-coverage reset
├── channels.yaml # 6 channels — 3 intake transports + balance + 2 admin
├── datastores.yaml # `accounts` (the ledger) + `coverage` (engine-owned schema)
├── migrations/accounts/V001__accounts.sql
├── schemas/transfer/transfer.xsd # TransferRequest + BalanceQuery
├── schemas/transfer/codec-manifest.yaml
└── templates/*.hbs # reply rendering (accept / reject / balance / coverage)
What it demonstrates
A module-owned SQL data store. datastores.yaml declares accounts — the ledger, one row per
account holding {balance, frozen} — with its own connection (env:ACCOUNTS_DB_URL/_USER/
_PASSWORD) and its own idempotent migration under migrations/accounts/. See
Data stores.
Per-channel singleton serialization. Three intake channels — transfer-request (HTTP),
transfer-queue (RabbitMQ), transfer-topic (Kafka) — all drive the same transfer.bpmn, each
declaring singleton: true. On a queue transport that’s leader-gated across replicas (a
per-channel PostgreSQL-lease election — see Replica semantics); on
HTTP the single-writer guarantee instead comes from the <bpmn:transaction> scope plus FOR UPDATE row locks in the flow itself. The read-only balance channel and the coverage-* admin
channels are not singletons — they scale across every replica, since they only read or reset
shared state rather than serialize writes to it.
flowchart LR
H["transfer-request — HTTP"] -->|"singleton"| T
Q["transfer-queue — RabbitMQ"] -->|"singleton, leader-gated<br/>across replicas"| T
K["transfer-topic — Kafka"] -->|"singleton, leader-gated<br/>across replicas"| T
T["transfer.bpmn<br/>the ACID transfer"] --> ACC[("accounts — the ledger")]
B["balance"] -->|"not a singleton —<br/>every replica serves it"| BQ["balance-query.bpmn<br/>read only"]
BQ --> ACC
CV["coverage-report, coverage-reset"] -->|"not singletons"| AD["the two admin flows"]
All six channels bind the same urn:transfer codec, and the three write intakes converge on one
process rather than three — so the write path is serialized once, by the channel on a broker and by
the transaction plus FOR UPDATE locks on HTTP, while the read-only paths stay free to scale.
A custom module schema. schemas/transfer/transfer.xsd declares TransferRequest and
BalanceQuery as sibling root elements; the codec registers as urn:transfer (see
Deployment packages for the path-derived URN rule), and every channel in
channels.yaml binds codec: urn:transfer.
The flow, node by node
transfer.bpmn has three start events — one per intake channel — that all converge on a single
<bpmn:transaction> sub-process named Transfer:
Start (http) ─┐
StartQueue ────┼──▶ Transfer [transaction]:
StartKafka ───┘ SubStart → LoadFrom → LoadTo → Valid?
├─ ok: Compute → Persist → SubEnd (normal end → COMMIT)
└─ invalid: DecideReason → cancel end event (→ ROLLBACK)
→ (committed) OkReply (render TransferAccepted) → End
→ (cancelled, via a boundary cancel event on Transfer) RejectReply → End
Full ACID, mapped onto real BPMN + q: elements:
- Atomicity —
LoadFrom/LoadTo/Compute/Persistall run inside the<bpmn:transaction>; a normal end commits both balance writes together, a cancel end (reached from theValid?gateway’s invalid branch) rolls back — no partial transfer ever lands. - Consistency — the
Valid?exclusive gateway checksfromAccount.frozen,toAccount.frozen, andfromAccount.balance < payload.amountbefore any write, in a visible FEEL condition on the sequence flow. - Isolation —
LoadFrom/LoadToread their rows with<q:store key="payload.fromId" forUpdate="true"/>— a pessimistic lock serializing concurrent transfers that touch the same account — layered under the channel-level singleton/exclusive-consumer contract. - Durability —
Persistwrites both new balances back to theaccountsstore in the same transaction; a later instance (on any replica) sees the committed values.
Compute and DecideReason are pure FEEL data-assignment nodes (<bpmn:assignment> pairs of a
<from> FEEL expression and a <to> target variable) — no service-task implementation code
anywhere in this flow.
sequenceDiagram
participant IN as Intake channel
participant TX as Transfer, a transaction sub-process
participant DB as accounts store
IN->>TX: TransferRequest, decoded by urn:transfer
TX->>DB: LoadFrom — q:store forUpdate, a pessimistic row lock
TX->>DB: LoadTo — q:store forUpdate, a pessimistic row lock
TX->>TX: Valid? — frozen flags and sufficient balance, in FEEL, before any write
alt valid
TX->>TX: Compute — pure FEEL assignment
TX->>DB: Persist — both new balances
Note over TX,DB: normal end of the sub-process — COMMIT, both writes together
TX-->>IN: OkReply renders TransferAccepted
else invalid
TX->>TX: DecideReason, then a cancel end event
Note over TX,DB: ROLLBACK — no partial transfer ever lands
TX-->>IN: RejectReply, via the boundary cancel event on Transfer
end
The gateway sits between the locks and the writes, so consistency is checked while the rows are already held — and the only two ways out of the sub-process are the two ends that commit or roll back, which is what makes the reply and the durable outcome agree.
Compliance path coverage
<q:coverage path="accept" flows="Flow_TxToOk Flow_OkToEnd"/>
<q:coverage path="reject" flows="Flow_CancelToReject Flow_RejectToEnd"/>
The two business outcomes are declared as tracked routes on the outer process, named for what they
mean (not left as the raw path-1/path-2 sutra coverage init would seed) — the curation step
Coverage: declared routes as the compliance signal covers in full. A run is
“covered” once its fired-flow trace contains a path’s flows, in order, as a subsequence — so any
of the three intake transports covers accept when it commits, and covers reject when it
cancels. Both marks land in the coverage store datastores.yaml declares — pointed, for
convenience, at the same database as the accounts ledger, and carrying no migration of its own
because the engine owns the coverage schema. The coverage-report / coverage-reset admin
channels read and clear this compliance metric; see that chapter for the CLI walkthrough
(init/check/reset), the cross-process form a multi-participant collaboration needs instead,
and how to curate a declared set that stays a signal instead of noise.
Try it
# seal it
cd rust && cargo build -p sutra-cli --release
target/release/sutra package ../examples/money-transfer/deployments-src/default--money-transfer--1.0.0 \
--out /tmp/pkgs
# deploy it (see Your first deployment for the full API-deploy walkthrough)
target/release/sutra deploy /tmp/pkgs/default--money-transfer--1.0.0.sutra \
--api --engine-url http://localhost:<port>
# send a transfer
curl -sS -X POST localhost:<port>/channels/transfer-request \
-H 'Content-Type: application/json' -H 'X-Api-Key: transfer-demo-key' \
-d '{"TransferRequest":{"FromAccount":"alice","ToAccount":"bob","Amount":"25.00"}}'
Run the full ordered scenario (durability, cross-instance reads, insufficient-funds and frozen-account rejection, atomicity rollback, isolation under concurrency, then the coverage report/reset pair) against real PostgreSQL:
cd rust && cargo test -p sutra-conformance -- --ignored tc_money_transfer_acid_ledger
Next
- Architecture — the layering underneath everything this example exercises.
- Reference: the
sutraCLI — every command used above, in full.
Worked example: loading a batch file
examples/call-log-load/ is a package that takes a file of records over HTTP — CSV or
fixed-width, the same records either way — validates every row against an XSD before the process
starts, transforms each record into a second XSD, and writes it as a typed database row.
One channel, one process, two schemas, two wire forms.
The shape
POST /channels/cdr-upload codec: urn:cdr (formats: [csv, fixed-width])
│
├── the CODEC validates the WHOLE FILE, row by row, cell by cell
▼
cdr-load.bpmn
Prep rowCount := count(payload.value)
Accept Handlebars receipt + <q:reply continue="true"> ──► 202 + receipt HERE
PerRow multi-instance over payload.value (the load runs detached)
Map <scriptTask> — one CallDetailRecord ► one CallLogEntry
Persist data task — call_log[entry.entryId]
End
Answering now, loading later
The channel declares ack-mode: on-persist: the caller is answered as soon as the upload is
durably captured, never held for the load. That alone would send a bare 202. What makes it a
useful 202 is <q:reply continue="true"> on Accept — the process renders its receipt, that
render becomes the response body, and the process then parks and resumes detached to run the rows:
HTTP/1.1 202 Accepted
Content-Type: application/json
{"batchId":"d36d2c59-0fea-4349-b410-a4b4c7715ab9","rowsAccepted":4,"status":"loading"}
The two declarations are independent and both are needed. The ack mode decides when the caller is
answered; continue="true" decides that the reply happens at the park instead of at the end of the
process. Drop the reply and the caller gets an empty 202 it cannot tell from a lost message; drop
continue="true" and the receipt waits for the whole load. See
Acknowledgement modes.
Binding a tabular format to a schema
A bare codec: csv on a channel is a parser: it would hand the flow a bag of untyped strings
with nothing asserted. Binding a schema codec whose manifest declares the tabular format is what
makes the file a typed message:
# schemas/cdr/codec-manifest.yaml
schemaKind: xsd
formats: [csv, fixed-width]
csv:
delimiter: ","
header: true
fixed-width:
fields:
- {name: recordId, width: 12}
- {name: msisdn, width: 16}
- {name: peerMsisdn, width: 16}
- {name: startTime, width: 22}
- {name: durationSec, width: 6}
- {name: direction, width: 11}
- {name: cellId, width: 10}
- {name: chargeAmount, width: 10}
- {name: rateCode, width: 8}
The schema those columns are checked against is an ordinary XSD — the facets do the work:
<xs:simpleType name="Msisdn">
<xs:restriction base="xs:string"><xs:pattern value="\+[0-9]{8,15}"/></xs:restriction>
</xs:simpleType>
<xs:simpleType name="Direction">
<xs:restriction base="xs:string">
<xs:enumeration value="originated"/><xs:enumeration value="received"/>
</xs:restriction>
</xs:simpleType>
<xs:element name="CallDetailRecord">
<xs:complexType><xs:sequence>
<xs:element name="recordId" type="RecordId"/>
<xs:element name="msisdn" type="Msisdn"/>
<xs:element name="startTime" type="xs:dateTime"/>
<xs:element name="durationSec" type="DurationSeconds"/> <!-- xs:int, 0..86400 -->
<xs:element name="direction" type="Direction"/>
<xs:element name="rateCode" type="RateCode" minOccurs="0"/>
</xs:sequence></xs:complexType>
</xs:element>
Two wire forms, one schema
Both files carry the same four records. The content-type selects the parser; nothing downstream knows which arrived.
recordId,msisdn,peerMsisdn,startTime,durationSec,direction,cellId,chargeAmount,rateCode
CDR-100001,+14155550101,+14155550187,2026-09-06T09:14:02Z,182,originated,CELL-0042,4.5500,PEAK
CDR-100003,+14155550188,+14155550101,2026-09-06T10:02:19Z,930,received,CELL-0117,0.0000,
CDR-100001 +14155550101 +14155550187 2026-09-06T09:14:02Z 182 originated CELL-0042 4.5500 PEAK
CDR-100003 +14155550188 +14155550101 2026-09-06T10:02:19Z 930 received CELL-0117 0.0000
curl -X POST .../channels/cdr-upload -H 'Content-Type: text/csv' --data-binary @sample/call-logs.csv
curl -X POST .../channels/cdr-upload -H 'Content-Type: text/plain' --data-binary @sample/call-logs.fixed-width.txt
Adding a vendor with a different delimiter is a manifest edit. The BPMN never changes.
The nuances
A batch projects under value
An array root projects under value, so the flow reads payload.value and a row is
payload.value[0].recordId. The loop iterates it in place:
<bpmn:multiInstanceLoopCharacteristics isSequential="true">
<bpmn:loopDataInputRef>payload.value</bpmn:loopDataInputRef>
<bpmn:inputDataItem name="row"/>
</bpmn:multiInstanceLoopCharacteristics>
loopDataInputRef is a FEEL expression, not a variable name. Copying the collection into a
variable first would put a second copy of the whole batch in the instance snapshot on every park.
The XSD’s leaf types reach the cells
A CSV cell is text on the wire, but the bound type decides what it is. durationSec is declared
xs:int, so it arrives as a number — which is why the transform can render it unquoted into
JSON and the store can write it into an INTEGER column without a cast anywhere.
An empty cell is absence — for an optional element
Row CDR-100003 leaves rateCode empty. A tabular row has a cell for every column whether or not
it carries a value, so an empty cell for an element the type declares minOccurs="0" reads as
absent. Without that rule the empty cell would become <rateCode></rateCode> and fail the
element’s own enumeration — and every optional column would have to be populated in every row.
An empty cell for a required element is untouched: that is a genuine data error and is still reported.
A bad cell names its row
One decode validates every row and every cell, and each violation carries a row-indexed path:
value[1] value '4155550999' does not match pattern '\+[0-9]{8,15}'
value[2] the value of element 'startTime' is not valid
value[3] value 'sideways' is not one of the enumerated values [originated, received]
The row index is the part that locates the problem; the message names the element. Rows 0 and 4 are not reported.
The whole batch is refused, in the caller’s own format
<q:onValidation mode="reject"/> makes a partly-bad file an all-or-nothing refusal. Because the
caller posted text/csv, the RFC 7807 problem document comes back as a table — one row per
issue, diffable against the file that was sent, rather than JSON to re-parse looking for row 4,217.
A fixed-width layout is checked at package time
A fixed-width record’s columns are its only field names, so they must agree with the bound type
or every row would fail at runtime for what is really a configuration mistake. sutra lint catches
it:
[ERROR] SUTRA.CONFIG.CODEC_MANIFEST.REJECTED — codec 'cdr': fixed-width layout declares
column(s) ["recordIdTYPO"] that type 'CallDetailRecord' does not declare …
A csv codec gets no equivalent check: its column names arrive in the header, at runtime, so there is nothing to compare against beforehand. That asymmetry is the price of a self-describing format.
What to know before scaling it up
The multi-instance loop is one durable step. That has three consequences worth stating plainly:
- The batch is the unit of recovery. A crash part-way through replays the loop from the first
record rather than resuming where it stopped. Here that converges, because every per-record
effect is a store write keyed by
entry.entryIdand the write is an upsert — the same rows are rewritten, not duplicated. Add a per-record<q:send>, an append or a counter and that is no longer true. - Say so.
<q:process idempotent="true"/>is the assertion that re-running converges. Without it the fail-closed default applies and a mid-load failure is dead-lettered and consumed rather than retried — the wrong posture for a file load, and wrong by silence. - The decoded batch lives in the snapshot. That is unavoidable; what is avoidable is
duplicating it. Iterate
payload.valuein place, and mark per-iteration scratchtransientso it never reaches the database.
If batches outgrow what one step should own, the move is to deliver them in chunks — a file or broker transport carrying N records per message. The BPMN names channels, never transports, so nothing in the flow changes.
Run it
The example ships its own two-service stack — engine and PostgreSQL, every secret already set —
so trying it is up, package, curl. You need the sutra CLI
and Docker; this is the same local model as the Quickstart,
pointed at a package that already exists.
1. Start the engine
docker compose -f examples/call-log-load/deploy/compose.yaml up -d
2. Seal the package into the watched directory
sutra package examples/call-log-load/deployments-src/default--call-log--1.0.0 \
--out examples/call-log-load/deploy/deployments
packaged …/default--call-log--1.0.0.sutra (deploymentId dep-397f20f9669c0e2eaabd0e74)
package lints on the way in — the fixed-width layout is checked against the bound XSD, the
store’s structure against the migration’s DDL — so a mistake in either fails here rather than on
the first upload. The engine watches that directory and activates the archive without a restart.
The host port is dynamic, because nothing assumes 8080 is free on your machine:
ENGINE=$(docker compose -f examples/call-log-load/deploy/compose.yaml port engine 8080)
curl -s http://$ENGINE/sutra/health/ready
{"status":"UP","checks":[{"name":"sutra-loader","status":"UP","data":{"deployments":1,"shards":1}}]}
3. Upload a batch
curl -s -X POST http://$ENGINE/channels/cdr-upload \
-H 'Content-Type: text/csv' \
-H 'X-Api-Key: dev-only-cdr-key' \
--data-binary @examples/call-log-load/sample/call-logs.csv
{"batchId":"45217d89-a520-4d7b-ba2a-453d1d9fa38a","rowsAccepted":4,"status":"loading"}
202 Accepted, in about 30 ms. The load has not finished — it has barely started. That is
ack-mode: on-persist plus <q:reply continue="true"> doing
exactly what the section above describes.
4. Read the rows back
docker compose -f examples/call-log-load/deploy/compose.yaml exec engine-db \
psql -U sutra_engine -d sutra -c \
'select entry_id, subscriber, bearing, duration_seconds, rated_amount, billable from call_log order by entry_id'
entry_id | subscriber | bearing | duration_seconds | rated_amount | billable
------------+--------------+----------+------------------+--------------+----------
CDR-100001 | +14155550101 | outgoing | 182 | 4.5500 | t
CDR-100002 | +14155550101 | incoming | 45 | 0.0000 | f
CDR-100003 | +14155550188 | incoming | 930 | 0.0000 | f
CDR-100004 | +14155550101 | outgoing | 1204 | 21.6720 | t
Typed columns, not a JSON blob: duration_seconds is an integer and rated_amount a NUMERIC
because the XSD’s leaf types were applied to the CSV’s untyped cells at decode. bearing and
billable do not exist in the uploaded file at all — the transform derived them from direction.
psql needs no password here: sutra_engine is the engine’s own role (owner of sutra), and
inside the container the local socket trusts it. The bootstrap superuser is postgres, which you
only need for creating roles or databases.
5. Watch it refuse a bad batch
curl -i -X POST http://$ENGINE/channels/cdr-upload \
-H 'Content-Type: text/csv' \
-H 'X-Api-Key: dev-only-cdr-key' \
--data-binary @examples/call-log-load/sample/call-logs-with-a-bad-row.csv
HTTP/1.1 400 Bad Request
content-type: text/csv
field,value
type,urn:bpm:diag:SUTRA.INBOUND.VALIDATION_REJECT
status,400
detail,"Inbound rejected on intake node Start of process cdr-load: 6 validation issue(s); mode=reject."
issueCount,6
Three things at once. It is a 400, not a 500 — a malformed file is the caller’s to fix, and a 5xx would invite a retry of identical bytes. It comes back as CSV, because that is what was posted. And re-running the query from step 4 still shows four rows: validation runs at intake over the whole file, so a batch with one bad cell writes nothing at all.
Drop the X-Api-Key header and you get 401 instead — the channel declares apikey auth, and the
engine will not wire an unauthenticated HTTP intake.
6. The same records, as fixed-width
curl -s -X POST http://$ENGINE/channels/cdr-upload \
-H 'Content-Type: text/plain' \
-H 'X-Api-Key: dev-only-cdr-key' \
--data-binary @examples/call-log-load/sample/call-logs.fixed-width.txt
Another 202 with a receipt, and the table still holds four rows — same schema, same keys, so the
same records upserted rather than appended. One channel, one XSD, two wire
forms, chosen by content-type.
Clean up
docker compose -f examples/call-log-load/deploy/compose.yaml down -v
Into an app you already have
If you are already running a scaffolded app rather than this stack, copy the package into it and give the engine the secrets the package names:
cp -r examples/call-log-load/deployments-src/default--call-log--1.0.0 \
my-app/packages/call-log-load
# my-app/deploy/.env — read by the engine service (env_file), gitignored by the scaffold
CDR_UPLOAD_API_KEY=dev-only-cdr-key
CALL_LOG_DB_URL=postgres://engine-db:5432/sutra
CALL_LOG_DB_USER=sutra_engine
CALL_LOG_DB_PASSWORD=sutra-dev-only
Then sutra package packages/call-log-load --out deploy/deployments and recreate the engine
(docker compose -f deploy/compose.yaml up -d --force-recreate engine) so it picks the new
environment up.
This step is not optional and it fails loudly. channels.yaml and datastores.yaml name their
secrets by reference — env:CDR_UPLOAD_API_KEY, env:CALL_LOG_DB_URL — and the engine
resolves them, in its own environment. Exporting them in your shell does nothing; the engine is not
your shell. An unresolvable channel-auth reference is fatal:
startup failed — refusing to serve
HTTP channel auth value could not be resolved: secret-ref 'env:CDR_UPLOAD_API_KEY'
resolves to no value (environment variable 'CDR_UPLOAD_API_KEY' is not set).
The engine will not open an unauthenticated port instead. An unresolvable store reference is
narrower — that store is not registered and the rest of the deployment still serves — so watch the
startup log for store NOT registered too.
Next
- The q: namespace —
q:onValidation,q:process idempotent,q:variables - Deployment packages — the manifest reference
- Data stores — the projected store the rows land in
Engine layering
Sutra is a single Cargo workspace under rust/. The crates split cleanly along the layers a
message passes through, plus one composition root that assembles them into the thing you actually
run.
The layers
sutra-dist composition root — force-links concrete codecs/transports/resolvers,
produces the `sutra-engine` binary (see Dockerfile)
│
sutra-engine the engine LIBRARY — config, deploy/activation, admin API, OTel,
audit sinks, the outbox tick loop. Domain-neutral: it collects
codecs/transports/resolvers generically via their SPIs and names none.
│
sutra-channels protocol-neutral channel binding + dispatch: decode → two-tier
validate → route to a start event or a parked wait → execute
│
sutra-executor the token executor: gateways, sub-processes, data associations,
compensation, path-coverage tracking — synchronous and stateful paths
│
sutra-bpmn the BPMN 2.0 + q: extension model and loader
│
sutra-feel / sutra-dmn / sutra-srl / sutra-templates
the expression + decision + template languages every task type runs on
│
sutra-persistence durable state: instances, outbox, inbox dedup, lease, audit — PostgreSQL
(MySQL/MariaDB/SQL Server dialects follow the same pattern)
A message arrives on a channel (sutra-channels), gets decoded and validated by a codec
(sutra-codec-spi + the concrete codec crates), is routed to a process, and the process runs on
the token executor (sutra-executor) over the BPMN model (sutra-bpmn), evaluating FEEL/DMN/.srl
expressions (sutra-feel, sutra-dmn, sutra-srl) and rendering templates (sutra-templates)
along the way. Anything durable — the instance snapshot, the outbox, the inbox dedup row, a data
store write — goes through sutra-persistence (or a module’s own store, for data stores; see
Data stores).
flowchart LR
IN["inbound message"] --> CH["sutra-channels<br/>decode · two-tier validate · route"]
CO["sutra-codec-spi<br/>+ the concrete codecs"] -.->|"codec SPI"| CH
CH -->|"typed payload"| EX["sutra-executor<br/>token execution"]
EX --> BP["sutra-bpmn<br/>model + q: extensions"]
EX --> LG["sutra-feel · sutra-dmn<br/>sutra-srl · sutra-templates"]
EX -->|"durable state"| PE["sutra-persistence"]
EX -.->|"a q:store binding"| DS["a module's own data store"]
Decode-and-validate happens once, at the doorway; everything below the channel layer only ever sees a typed payload, and every durable write leaves through one crate.
The composition root
sutra-engine — the library — never names a concrete codec, transport, or secret-resolver
implementation. sutra-dist is the one crate allowed to: it force-links the schema-less formats
(json, xml, yaml, csv, raw-text, raw-bytes), the redactors, the vendor
secret-resolvers, and the feature-selected transports, and produces the sutra-engine binary the
container image ships (docker build -f rust/Dockerfile rust/). It force-links no domain codec:
every message standard is a proprietary extension crate built outside this repository,
registering through the same SPIs and force-linked by its own composition root, which is
precisely what this split is for. This split is what lets a hardened build drop everything it
doesn’t need — cargo build -p sutra-engine --no-default-features --features file links no broker
client at all — without touching a line of the neutral engine or channel code.
flowchart LR
F["sutra-formats<br/>json · xml · yaml<br/>csv · raw-text · raw-bytes"] --> D
R["the redactors"] --> D
S["vendor secret-resolvers"] --> D
T["transports"] -->|"feature-selected"| D
E["sutra-engine (the library)<br/>names no concrete"] --> D
D["sutra-dist<br/>composition root — force-links"] --> B["the sutra-engine binary<br/>the container image ships"]
Every concrete implementation enters the binary at exactly one crate — which is what lets a hardened build link no broker client at all while the neutral crates it depends on stay untouched.
See Domain neutrality and the SPI model for exactly how that boundary is drawn and mechanically enforced, and what a third party has to write to add a new transport or codec.
Where the tooling sits
sutra-cli (the sutra binary) depends on the model/loader layer read-only for its inspection
commands (describe, dispatch-graph, simulate, explain), and on sutra-persistence +
sutra-channels for the commands that touch a running engine or a database behind it (deploy and
migrate against the engine’s own; coverage check --archive against the one a deployment’s
coverage store declares). It is not part of the engine’s own runtime dependency
graph — a deployed engine binary has no CLI code linked into it.
Next
- Domain neutrality and the SPI model
- Deployment model — how a package activates against this layering.
Domain neutrality and the SPI model
The engine core never names a business domain, a vendor, or a wire format. Not SWIFT, not FedNow, not ISO 20022, not Kafka, not PostgreSQL-as-a-brand. Payments, health, and EDI live entirely in codecs, transports, validators, and deployment packages — never in the token executor, the BPMN model, or the channel dispatcher. This is the single rule everything in this chapter exists to serve, and it is not a style guideline: it is checked by a build gate on every change.
Why this is load-bearing
A domain-neutral core is what lets one engine binary serve payments today and healthcare tomorrow without a fork, what lets a hardened build strip out everything it doesn’t need down to the byte, and what keeps the surface a security review has to reason about small and stable. The moment a business term leaks into the executor or the model, that guarantee is gone for everyone, silently.
The gate: sutra-archtest
rust/crates/sutra-archtest is the Rust twin of what used to be an ArchUnit test on the retired
reference baseline. It walks a fixed set of crates’ src/ trees, strips comments (a domain term
cited in a doc comment is fine — only a term baked into an identifier or a string literal is a
violation), and fails the build on any hit against a denylist:
#![allow(unused)]
fn main() {
pub const DOMAIN_DENYLIST: &[&str] = &[
"swift", "fednow", "fedwire", "pacs", "camt", "nacha", "edifact", "hl7", "iso20022", "x12",
];
}
Enforced — must be domain-literal-free:
- The neutral core:
sutra-executor,sutra-bpmn,sutra-feel,sutra-dmn,sutra-srl,sutra-templates,sutra-persistence,sutra-datastore, and the codec/format/schema SPI itself (sutra-codec-spi). - The assembly/binding layer:
sutra-channels(protocol-neutral channel binding) andsutra-engine(the library) — domain-literal-free since the transport and codec extraction and thesutra-distcomposition-root split.
Deliberately excluded — the legitimate domain edge, by design:
sutra-dist— the composition root. It force-links the built-in formats, the redactors, the vendor secret-resolvers, and the feature-selected transports, so it necessarily names them. That’s the one place wiring concretes to SPIs is supposed to happen — not a core leak.- The concrete implementations themselves:
sutra-formats,sutra-codec-schema, everysutra-transport-<vendor>crate, andsutra-redactor-pci. - Tooling and test crates (
sutra-cli,sutra-conformance,sutra-testkit, the generators).
Read that exclusion list again for what isn’t on it. This repository contains no
message-standard codec at all. The built-in codec set is the six schema-less formats in
sutra-formats — json, xml, yaml, csv, raw-text, raw-bytes — and sutra-dist
force-links nothing else that registers one. Every wire standard — SWIFT MT, SWIFT MX, rail
variants such as FedNow, and the EDI/segment families (HL7 v2, X12, NACHA/ACH, EDIFACT) — is a
proprietary extension crate maintained outside this repository, consuming exactly the public
SPIs a third party consumes.
That is the strongest available evidence that the SPI boundary is real rather than aspirational:
adding a message standard to a distribution costs one Cargo dependency and one use <crate> as _;
line in that distribution’s own composition root. Nothing in sutra-engine, sutra-channels, or
this repository’s composition root changes, and there is no central list anywhere to add a name
to. The gate has nothing to exclude for those crates because they are not here; the neutral core
cannot leak a term that never enters its dependency graph.
flowchart TD
subgraph here["This repository"]
CORE["the neutral core<br/>no domain literal, gated"]
FMT["sutra-formats<br/>six schema-less formats"]
DIST["sutra-dist"]
end
subgraph out["Outside this repository"]
STD["a codec crate<br/>against the public SPI"]
OWN["its composition root"]
end
CORE --> DIST
FMT --> DIST
DIST --> B1["the stock binary"]
CORE --> OWN
STD -->|"one dependency,<br/>one link line"| OWN
OWN --> B2["that distribution's binary"]
A distribution adds a wire standard by force-linking one more crate into its own composition root — there is no central list in this repository to append a name to, which is what makes the neutrality gate cheap to keep passing.
make lint runs this gate alongside clippy on every change (see
Contributing) — a business term landing in a gated crate fails CI, not a
code review comment.
The mechanism: self-registration behind a neutral SPI
Every extension point follows the same shape: a small SPI crate defines a trait and a
process-wide registry; each concrete implementation is its own crate that self-registers into
that registry via inventory at link time. The neutral crate
collects what got linked in — it never imports or names a specific implementation. Implementing
an extension is registering it; there is no separate central list to forget to update (a gap
that bit an earlier iteration of the codec set, closed by moving to this pattern).
flowchart LR
CR["a composition root"] -->|"force-links the crate"| IMPL["a concrete crate<br/>a transport, a codec, a resolver, a redactor"]
IMPL -->|"inventory::submit! at link time"| REG[("the SPI crate's<br/>process-wide registry")]
REG -->|"collected, sorted by name"| NEU["the neutral engine<br/>drives every entry through the same trait"]
The neutral side only ever iterates what got linked in, so adding an implementation is a link-time fact rather than an edit to anything the engine owns.
Transports — sutra-transport-spi
#![allow(unused)]
fn main() {
pub struct TransportFactory {
pub transport: &'static str, // the `transport:` value in channels.yaml this wires
pub spawn: TransportSpawn, // (defs, engine, pool, envref-resolver, runtime) -> Arc<dyn TransportChannels>
pub register_sink: fn(&mut SinkRegistry),
pub handles_on_complete: bool, // self-declared capability — see below
}
inventory::collect!(TransportFactory);
}
Each sutra-transport-<vendor> crate submits one TransportFactory. The engine assembly iterates
transport_factories() (sorted by name for determinism) and drives every one of them through the
identical TransportChannels lifecycle trait (rewire on an activation flip, drain on
shutdown) — there is no if transport == "kafka" anywhere in the engine.
Capability self-declaration, not a hardcoded transport check. handles_on_complete is how a
transport tells the engine whether it can realize ack-mode: on-complete (a broker deferring its
settle through the engine’s ack registry, or an on-listener transport like HTTP holding the
connection to completion). When a channel declares on-complete on a transport whose factory
reports false, the engine assembly emits SUTRA.ACK.ON_COMPLETE_UNSUPPORTED and runs
on-persist instead — a loud, generic degrade driven by a flag the transport itself set, not a
list of vendor names the engine maintains. See
Acknowledgement modes for the full per-transport wiring table this
flag produces.
Codecs — sutra-codec-spi
#![allow(unused)]
fn main() {
pub trait PayloadCodec {
fn name(&self) -> &str;
fn accepted_content_types(&self) -> Vec<String>;
fn decode(&self, body: &[u8], content_type: Option<&str>) -> DecodeResult; // never panics
fn declared_message_types(&self) -> Vec<String> { Vec::new() }
fn shape_of(&self, message_type: Option<&str>) -> Option<SchemaShape> { None }
fn bind_refs(&self, resolve: CodecRefResolver) -> Result<(), String> { Ok(()) }
fn seal(&self, body: &[u8], content_type: Option<&str>) -> Result<Option<Vec<u8>>, String> { Ok(None) }
// + encode (the reply direction)
}
}
Sealing and references. A codec that must change the bytes it sends, for example to add a
digital signature, implements seal. The engine calls it after the codec’s own read-back of an
outbound message passed and before the message is committed to the outbox, for sends and replies
alike, template renders included. A seal error refuses the message (SUTRA.OUTBOUND.SEAL_FAILED).
A codec whose bundle files name references (secret:NAME, env:NAME) resolves them in
bind_refs. The engine calls it once, when the deployment is planned on the runtime host, through
the same resolver channel credentials use. An unresolvable reference refuses the deployment
(SUTRA.CODEC.BIND_REFS_FAILED). The engine still names no format and no algorithm: what sealing
means is the codec’s business.
A zero-config, globally-named codec — each built-in format (json, xml, yaml, csv,
raw-text, raw-bytes), and any extension crate that wants a global name — submits a
BuiltinCodec { name, make }; builtin_codecs() collects and sorts them, and each is addressable
as urn:sutra:codec:<name> with no per-package configuration. A schema-backed user codec
(compiled from a package’s own schemas/<name>/*.xsd) does not self-register this way — it’s
instantiated per package, per the path-derived URN rule in
Deployment packages. Either way, the engine only ever calls
through the PayloadCodec trait; it has no branch for “this is the such-and-such standard
one,” and no way to acquire one — the type it would have to name is not in its dependency
graph.
Bundle codec kinds — a codec crate registers its own schema-folder shape
The newest extension point in this family follows the identical inventory-pull shape, one level
into how a package’s own schema folder is interpreted. schemaKind: xsd and schemaKind: json-schema are generic — a folder of schema files the engine validates against — but some
standards are a whole profile: an envelope grammar, a mapping from wire-level message names to
schema files, versioned editions revved on the standard’s own release cadence. A codec crate that
needs that shape submits a BundleCodecKind { kind, build } in sutra-codec-schema::bundle next
to its PayloadCodec impl — implementing the kind is registering it, exactly like a
BuiltinCodec, and sutra-codec-schema itself stays free of any knowledge of which standards use
it. A package’s schemas/<name>/codec-manifest.yaml declaring schemaKind: <kind> is what selects
a registered bundle over the generic ones; an unserved kind is a fail-closed deploy error naming
the kinds the running build actually serves.
A market-infrastructure rail codec — an extension crate outside this repository — is the first user of this extension point (see Channels and transports and Deployment packages for the manifest shape and deployment-scoped registration it produces). It’s also a useful case study for the SPI pattern more broadly: everything specific to one rail’s profile — its envelope root names and namespaces, its per-direction wrapper tables, its quirks (a wrapper shipped without the profile’s usual name prefix, several wrappers backed by one underlying message) — is a single data value the codec’s internal walker/validator/projection code reads generically. A second rail that follows the same envelope-and-wrapper-editions pattern would supply its own profile value and reuse that machinery rather than re-implement it — the same “data, not a branch” discipline the rest of this chapter describes, one layer further in.
Data stores — sutra-datastore
The DataStore SPI is the same shape one level down: a provider (sql today; see
Data stores) resolves its own connection from datastores.yaml
— never the engine’s internal datasource — and exposes get/put/get_for_update/
put_if_revision against (store_name, store_key). The engine’s executor calls through this
trait for every <q:store>-bound data association; it has no idea whether the value on the other
side is an account balance or a customer record.
Secrets — sutra-envref-spi
The vendor-neutral seam behind env:, secret:, ${…} placeholders, and vendor schemes like
vault:… / aws-secrets:…. A vendor resolver crate (sutra-envref-vault, sutra-envref-aws, …)
submits an EnvRefResolverEntry; the engine resolves a whole reference generically and names no
vendor SDK. This is what lets channels.yaml and datastores.yaml carry credentials as
references, never literals — see Configuration reference.
Redactors — sutra-redactor-spi
A sutra-redactor-<standard> crate submits a RedactorEntry that locates sensitive spans in a
decoded payload (a JSON-Pointer-shaped path + a reason code); the engine masks every located path
on every observability surface and marks it for encryption at rest. Fail-closed by construction: a
redactor that panics or errors tells the engine to over-mask the whole bound payload rather
than risk a leak — the opposite failure mode from a validator crash, which becomes an ordinary
business-reject issue instead.
Validators
<q:simpleValidator ref="…"> names a field-content validator (iso-3166-country,
iso-4217-currency, iso-9362-bic, …) out of a neutral registry the core knows nothing about
(see The q: namespace). The concrete validators are domain content
and live in extension crates, outside this repository, for the same reason the domain codecs do —
which is why the neutrality gate has nothing to exclude here.
The lifecycle bus — ExecutionListener
One more neutral seam worth knowing about even though it isn’t a “plug in a new vendor” SPI: every
cross-cutting concern that needs to react to instance/token/task lifecycle events — audit, OTel
metrics, the deferred-ack registry — implements the same ExecutionListener trait and is fanned
out from a plain Vec<Rc<dyn ExecutionListener>> the executor calls on on_instance_started /
on_instance_completed / on_instance_suspended / etc. There’s no dependency-injection
container; listener registration happens explicitly at executor-construction time in the engine
assembly. See Acknowledgement modes for DeferredAckRegistry as a
concrete example, and Observability for the OTel listener.
Where business content actually lives
Given all of the above, a real deployment’s domain-specific content lives in exactly two places:
- Inside a deployment package — the BPMN processes, DMN/
.srlrules,channels.yaml,datastores.yaml, and the package’s own XSD-backed codec (see Deployment packages). This is where almost all business logic belongs, and it needs zero Rust code. - In an extension crate, only when the domain needs a new kind of codec, transport,
validator, or redactor that doesn’t already exist — implemented against the relevant SPI above
and linked in by a composition root (
sutra-dist, or your own if you build a custom binary).
Worked example: adding a transport
Say you need a new broker Sutra doesn’t ship a transport for. The shape is fixed by the pattern above:
- Create
sutra-transport-<vendor>, depending onsutra-transport-spiandsutra-channels. - Implement
TransportChannels(transport(),consumer_count(),rewire(active),drain(),stop_all_detached(), and the one optional capabilityinbound_router()— only HTTP-shaped transports returnSome). - Implement the inbound spawn function with the uniform signature
(&[ChannelDefinition], EngineHandle, Option<PgPool>, EnvRefResolver, tokio::runtime::Handle) -> Result<Arc<dyn TransportChannels>, Diagnostic>, and an outbound sink registrarfn(&mut SinkRegistry). - Decide whether your transport can realize
ack-mode: on-complete(deferred settle, or holding the connection) and sethandles_on_completehonestly — get this wrong and channels either silently under-deliver on a promise, or trigger the loud unsupported diagnostic needlessly. inventory::submit! { TransportFactory { transport: "your-broker", spawn, register_sink, handles_on_complete } }next to your implementation.- Add your crate as an optional dependency behind a same-named Cargo feature in whatever
composition root force-links it (
sutra-dist, or a custom one) — nothing insutra-engineorsutra-channelschanges.
A new codec follows the identical shape against PayloadCodec instead, submitting a
BuiltinCodec for a zero-config global codec, or shipping as a package-local schema-backed codec
with no self-registration at all. Every message-standard codec in existence takes that first
path — including the ones a downstream distribution ships — so it is a well-travelled route, not
a theoretical extension point kept alive by a single in-tree user.
Next
- Deployment model — how a package’s declared channels/codecs/stores get resolved against whatever is linked into the running binary.
- Contributing — the repo map, including where each SPI crate and its concrete implementations live.
Deployment model
A .sutra archive’s runtime identity is a single opaque deploymentId = sha256(manifest) — the
manifest is derived at seal time (sutra package), never hand-authored. Deploying is therefore
naturally idempotent: re-deploying identical bytes is a no-op. This page is the architecture
behind Your first deployment and
Deploy, hot-deploy, and rollback — how activation actually
works underneath the CLI.
The database is the source of truth
Sealed archives are stored in the engine’s own datasource — the same core database that backs
instance state, the outbox, and the lease table — as a deployment_archive row keyed by a stable
slot (the archive’s tenant--module--version key). Exactly one row per slot is ever active;
deploying a new revision to an existing slot replaces the old active row in one transaction. This
is what makes hot-deploy a replace, not a restart: the slot name is stable across versions,
only the content-addressed deploymentId changes.
Loading is symmetric — on boot, the engine loads its active set from the database (WHERE status='active'), not by scanning a directory. A restarted or newly-scaled replica rehydrates its
active set from the shared database, not from whatever a local volume happens to reflect.
Deploy is an API call
POST /admin/deployments is the one control path onto a running engine (auth-gated — see
Configuration reference for the admin auth scheme). It:
- Accepts the sealed archive’s bytes.
- Re-verifies them fail-closed (the same archive-integrity check
sutra packagealready ran client-side). - Stores the new revision as the slot’s active row, in a transaction.
- Runs the two-phase activation flip in-process (drain the old revision if one exists, activate the new one).
- Returns synchronously:
200 {deploymentId, phase: "Active"}, or a4xxcarrying theSUTRA.DEPLOY.*reject diagnostic.
Because the HTTP response is the activation signal, there is no propagation window to reason about, no separate “did it actually take?” step — this is the property that makes deploys deterministic rather than eventually-consistent from the caller’s point of view.
sequenceDiagram
participant C as sutra deploy
participant R1 as The replica that took the call
participant DB as The engine's database
participant R2 as Every other replica
C->>R1: POST /admin/deployments — the sealed archive
R1->>R1: re-verify the archive, fail-closed
R1->>DB: store as the slot's active row
Note over DB: one transaction — exactly one active row per slot
R1->>R1: two-phase flip — drain the old revision, activate the new
R1-->>C: 200 {deploymentId, phase "Active"}
DB--)R2: LISTEN/NOTIFY after commit, version-poll as the fallback
R2->>R2: converge on the committed active set
The caller’s answer arrives only after the flip, so it is definitive; the rest of the fleet converges off the committed row rather than off anything the deploying replica tells it.
DELETE /admin/deployments/{slot|id} marks a row draining; the engine’s activation flip drains
it (no new intake, retire once quiescent).
stateDiagram-v2
[*] --> Active: POST /admin/deployments
Active --> Draining: a new revision takes the slot
Active --> Draining: DELETE by slot or id
Draining --> Retired: no new intake, quiescent
Retired --> [*]
The slot outlives its revisions: taking a slot moves the old row sideways into draining rather
than deleting it, which is what makes a hot-deploy a replace rather than a restart.
Sync vs. async — the same call, two response shapes
For a small or local deploy, the synchronous form above is the whole story: one request, one definitive answer. For a large deployment where the engine’s own plan-and-flip work risks a long-held request — mainly a concern behind an ingress with a short read timeout — the identical endpoint accepts an async mode:
- Sync (default):
POST /admin/deploymentsblocks until the flip completes, returningActiveor a reject. - Async (opt-in): the same POST returns immediately with
202 {deploymentId, status: "Pending"}; activation runs in the background. The caller learns the outcome one of two ways:- Poll
GET /sutra/deployments/{id}— short, ingress-safe requests — until it flips toActiveorFailed. - A completion event. The caller opts in with
callback=<https-url>(a webhook) and/ornotify=<broker-uri>at deploy time; on completion the engine emits a CloudEvent (com.sutra.deployment.activated/com.sutra.deployment.failed) carrying{deploymentId, slot, revision, status[, error]}. This rides the engine’s existing outbound spine — one more durable outbox entry, delivered by the same dispatcher that sends every other outbound message — so a deploy-complete notification is not a separate mechanism, just another emission.
- Poll
flowchart LR
P["POST /admin/deployments"] ==>|"sync — the default"| A["blocks through the flip<br/>200 Active, or a SUTRA.DEPLOY.* reject"]
P -->|"async — opt-in"| B["202 Pending, activation runs behind it"]
B -->|"poll"| C["GET /sutra/deployments/{id}<br/>until Active or Failed"]
B -->|"push"| D["a CloudEvent to callback= and/or notify="]
D -->|"one more outbox row"| E["the same dispatcher that sends<br/>every other outbound message"]
This is a long-running-operation (LRO) shape, not a bespoke deploy protocol: accept fast, do the work in the background, let the caller choose polling or a push notification.
Multi-replica convergence
The replica that handled the deploy call activates locally and returns synchronously; every other
replica converges asynchronously on the committed database state — on PostgreSQL via LISTEN/
NOTIFY after the commit, with a version-poll fallback for dialects with no equivalent
(MySQL/MariaDB/SQL Server) and as the resilience backstop on Postgres itself. A single-replica run
needs no convergence step at all — the one replica that deployed is already the whole fleet.
Next
- Deploy, hot-deploy, and rollback — the operator-facing walkthrough of the same mechanism.
- Replica semantics — how the rest of the engine’s durable state stays correct across a replica set, using the same PostgreSQL-backed primitives.
Multi-tenancy and isolation
Sutra runs many tenants behind one engine and one database, with isolation enforced at the storage layer rather than by giving each tenant its own infrastructure.
flowchart LR
M["inbound message"] --> CH["channel dispatcher<br/>tenant = the package's own label"]
CH --> Q{"per-tenant ceilings<br/>concurrent instances · inbound rate"}
Q -.->|"over the cap"| REJ["HTTP 429 · broker nack"]
Q -->|"admitted"| EX["the executor"]
EX --> P["the persistence layer<br/>injects the tenant WHERE clause"]
P --> DB[("PostgreSQL row-level security<br/>SET LOCAL app.current_tenant")]
Two different gates: the ceilings are an admission concern, enforced before an inbound ever reaches the executor, while the tenant partition itself is enforced twice on the way to disk — the second time by the database, which no query can talk its way around.
Tenant identity is a package label
A deployment package’s tenant is an opaque label in package.yaml (alongside module and
version — see Deployment packages):
labels:
"module": "money-transfer"
"tenant": "default"
"version": "1.0.0"
The engine never interprets this string — it’s a selector for observability, routing, and the row-level-security partition key described below. Because a package is fully self-contained (no shared resource tree, no inheritance between packages), a tenant’s isolation boundary is simply “whatever package it was labeled into” — there’s no separate tenant-configuration document a package’s processes have to be matched against.
Storage isolation: row-level filtering + PostgreSQL RLS
Every durable table the engine owns — instance state, the outbox, the inbox dedup table, the alias index, the audit event log, the wait-state table, the dead-letter store — carries a tenant column and a PostgreSQL row-level-security policy keyed on a session variable, applied by the engine’s own SQL migrations. Defense in depth, two layers:
- Application-level filtering — every query goes through the persistence layer, which
requires a tenant argument and injects the
WHEREclause; there’s no ad hoc database access path around it. - PostgreSQL RLS —
SET LOCAL app.current_tenant = '…'at the start of each request transaction, with aCREATE POLICY … USING (tenant_id = current_setting('app.current_tenant'))on every table. Even a query that forgot its ownWHEREclause cannot return another tenant’s rows — the database itself is the second line of defense, not just the repository layer.
A startup check closes the obvious way to defeat this. RLS policies are silently bypassed for
any role with BYPASSRLS, SUPERUSER, or ownership of the target tables. The engine probes its
own connecting role at startup and refuses to start if either flag is set — an operator who
pointed the engine at postgres or a role minted with BYPASSRLS finds out at boot, not after a
cross-tenant leak. Production role setup:
CREATE ROLE sutra_app LOGIN PASSWORD '<from-secret>';
ALTER ROLE sutra_app NOBYPASSRLS;
GRANT SELECT, INSERT, UPDATE, DELETE ON <engine tables> TO sutra_app;
(Migrations themselves run as a separate, table-owning role that the engine runtime never uses.)
Quotas
Two per-tenant ceilings, enforced by the channel dispatcher before an inbound reaches the executor:
maxConcurrentInstances— a hard cap on simultaneously in-flight instances for the tenant, checked against a live count ininstance_state— globally coherent across every replica, since every replica queries the same table.maxInboundRatePerMinute— a sliding 60-second admission window. This one is per-replica, not fleet-wide: each replica tracks its own window in memory rather than paying a synchronous round-trip to the database on every inbound message. A tenant saturating one replica’s window can still be admitted by a second replica until its window also fills — a deliberate latency/coherence trade-off. A hard global rate guarantee, if you need one, belongs in front of the engine (an API gateway or service-mesh L7 limiter), not in the dispatcher.
Both rejections surface as the same class of diagnostic the payload-size cap uses (see Limits and quotas), translated to the right per-transport signal (HTTP 429, a broker nack, …).
flowchart LR
IA["inbound → Replica A"] --> DA{"admit?"}
IB["inbound → Replica B"] --> DB2{"admit?"}
WA["A's own 60s window<br/>in memory"] --> DA
WB["B's own 60s window<br/>in memory"] --> DB2
C[("live count in instance_state<br/>one shared table")] --> DA
C --> DB2
The concurrency cap is one shared count, so it holds fleet-wide; the rate window is per-replica memory, so a fleet of N replicas admits up to N windows’ worth before anything says no.
Audit isolation
Audit rows carry the tenant column and are covered by the same RLS policy as every other engine table. Where a JSONL audit sink is configured, it writes per-tenant directories, so a log shipper can fan out per-tenant index/retention policy downstream without the engine knowing anything about that policy itself.
Next
- Replica semantics — the mechanisms (leases, row locks,
SKIP LOCKED) that keep all of this correct across a multi-replica engine. - Limits and quotas — the operator-facing configuration surface for the ceilings above.
Replica semantics
Sutra runs as an active-active stateless replica set — all durable state lives in PostgreSQL, and no replica owns any particular instance, tenant, or channel by default. Any replica can pick up any token at any time, which is what lets the engine scale horizontally with no sticky routing.
What every replica does
Each replica is one engine process running inbound channel listeners (the HTTP server, broker consumers), the token executor for instances it picks up from persistence, an outbox worker sending replies it claims, and the health/metrics endpoints. Execution inside the process runs on one or more identical actor lanes — see Execution lanes; lanes are an in-process concern and change nothing on this page. Three pieces of work don’t tolerate concurrency and are held by exactly one replica at a time instead: timer firing, stuck-instance scanning, and terminal-history purging.
flowchart TD
R1["Replica 1"] --> DB
R2["Replica 2"] --> DB
R3["Replica 3"] --> DB
DB[("The engine's own PostgreSQL")] --- LEASE["a lease row per role<br/>timers · sweeps · purges<br/>singleton channels"]
DB --- CLAIM["the instance row's owner<br/>gates advancing one instance"]
DB --- OUT["outbox rows<br/>FOR UPDATE SKIP LOCKED"]
DB --- IBX["the inbox unique index<br/>ON CONFLICT DO NOTHING"]
Every replica is identical and interchangeable; each of the four things that must not happen twice is a different row-level primitive in the one shared database, not a coordination service.
Leader election: a PostgreSQL-backed lease
There is no Kubernetes-native Lease object and no separate coordination service — leader
election is a lease row in the engine’s own PostgreSQL database (DbLeaderElection over
PgLeaseStore). One poll task per role tries to acquire the lease at a fixed cadence
(ttl = 30s, poll = 10s); a successful acquire flips that replica to leader for that role, a
contended one flips it to follower. There’s no push notification — polling is the mechanism.
Roles are dynamic, not a fixed pair: the engine registers one lease per singleton channel
role (channel_role(tenant, channel)) the first time it’s needed, so declaring a channel
singleton: true (see Channels and transports and the
money-transfer example) starts contending a lease without any
separate configuration. When no engine datasource is configured at all, every replica simply
leads unconditionally (AlwaysLeading) — there’s no third posture to reason about.
Instance ownership claims
Leases gate roles. What keeps two replicas from advancing the same instance at the same moment is a different, finer mechanism: a per-instance ownership claim, taken by a compare-and-set on the instance’s own row.
Every resume path claims first. A correlated relay arriving on a channel, a timer firing, an
administrative migration — each one takes the claim before it rehydrates anything, and a claim it
cannot take is a refusal, never a wait. The refusal is retry-safe by design and carries a
structured code (SUTRA.RUNTIME.RESUME.CLAIM_HELD, or the admin surface’s
SUTRA.ADMIN.MIGRATE.CLAIM_HELD): a broker relay is requeued for redelivery, a timer fire is
deferred to a later tick, an admin call answers 409 having read and written nothing. Nothing is
ever half-applied while contended, and nothing blocks waiting for a lock held on another
replica’s timescale.
The claim is released inside the same transaction that commits the step’s own writes, so “committed” and “unowned again” are one atomic fact. Every exit that does not commit — an early refusal, an error, a panic — releases the claim explicitly on its way out, so a claim never outlives the work it was protecting.
sequenceDiagram
participant A as Replica A
participant I as The instance's own row
participant B as Replica B
A->>I: compare-and-set — take ownership if free or stale
I-->>A: claimed
B->>I: compare-and-set for the same instance
I-->>B: no match — SUTRA.RUNTIME.RESUME.CLAIM_HELD
Note over B: relay requeued, timer fire deferred, admin 409
A->>I: commit the step's writes and release the claim
Note over A,I: one transaction — committed and unowned are the same fact
The whole mechanism is two round trips against one row: the loser of the race has read and written nothing, and the winner never holds the claim past the commit it was protecting.
The claim is re-entrant for the same owner, and the invariant that makes that safe is worth stating exactly: an owner id names one execution lane in one process, and a lane advances instances one at a time. Same owner therefore means same lane, which already means serialized — so re-claiming what you already hold is a heartbeat refresh rather than contention. Owner ids carry the lane index for exactly this reason, and the administrative migration path takes a deliberately distinct owner suffix so that a migration can never re-enter past a resume this same replica has in flight.
The stuck-instance sweep is the backstop for the case no protocol can prevent — a replica
that dies mid-step and never releases. The instance-sweeper lease role scans on
sutra.instance.sweep-interval (default PT1M) and clears any claim whose owner has been silent
longer than sutra.instance.claim-timeout (default PT5M). It sweeps by age and is
owner-blind, so it needs to know nothing about how owner ids are shaped.
Durable FAILED is a state, not a disappearance
An instance whose execution fails fatally is not deleted and does not silently linger as a
mystery park. Its snapshot is re-stamped FAILED in place, carrying the structured failure
code and the captured detail, and every waiting row it held is resolved in the same transaction —
so no timer refires against it and no relay finds a live wait to satisfy.
FAILED is deliberately not terminal:
- it is always retained, regardless of the history-retention window, because it needs an operator rather than a clock;
- it keeps blocking its deployment’s retirement (below), so a dead-but-unhandled instance cannot be quietly swept under a hot-deploy;
- it is the one state an instance can be migrated and resumed out of — repair the model, move the instance onto it, then explicitly bring it back.
Restoring it is exactly the inverse of the failure commit: the failure keys are dropped, the status goes back to suspended, and the parks the failure tore down are re-armed — after which the instance comes back through the ordinary claim-guarded paths. There is no privileged resume entry point.
Inbox dedup via row locks
An inbound message’s (tenant, channel, event_id) triple is inserted with ON CONFLICT DO NOTHING; whichever replica’s insert actually produced a row owns starting that instance, and any
other replica that raced it (or a genuine redelivery) sees nothing came back and treats it as a
dedup hit. This is a plain unique-index conflict — no application-level locking, and it works
identically whichever replica happens to receive the redelivered copy. A delivery that can’t be
checked because the database doesn’t answer is refused and retried, not run
(details).
Outbox processing: SKIP LOCKED and a claim lease
Every reply-in-waiting sits in an outbox table. Each replica’s outbox dispatcher claims a batch with
FOR UPDATE SKIP LOCKED, so replicas never contend on the same row. Those row locks end when the
claim’s transaction commits, which is before the rows are delivered. So the same statement also
leases each claimed row: it moves the row’s due time 30 seconds ahead. No claim on any replica
returns the row before then. While the row waits for a delivery worker or is being delivered, the
claiming replica renews the lease every 10 seconds. A successful delivery deletes the row, and a
failed one replaces the lease with its retry backoff. Under normal operation no row is sent twice,
however many replicas share the database.
A replica that dies holding rows stops renewing their leases. Within 30 seconds the leases lapse, and the next replica to claim delivers the rows. The rare case where the original send went out just before the crash is absorbed downstream: the outbox row’s key travels as the message’s own idempotency token.
The renewal never waits for a row lock. It skips a row another transaction holds, because the
only holders of an in-flight row are the transactions that end its lease on purpose (the delete of
a delivered row, a retry’s backoff, a poison) or another replica’s claim of a lease that has
already lapsed. It renews at most 1024 rows per transaction, leaves out rows already delivered and
waiting for their delete, and has a database connection of its own. A renewal round slower than
5 seconds is logged at warn.
Fixed in 0.6.12. From 0.6.8 to 0.6.11 the renewal was one statement over every in-flight row, and it locked rows that the batch delete of delivered rows was locking too, each in its own order. Under a burst of deliveries PostgreSQL aborted one of them with
deadlock detected: a lost delete was retried row by row (outbox delete failed: deadlock detected, nothing lost), a lost renewal renewed nothing that round. When the engine fell behind, the renewal took 1 to 3 seconds.
Fixed in 0.6.8. Releases 0.6.5 to 0.6.7 took the row locks but wrote no lease. A second replica’s claim could pick up a row the first was still delivering, and under multi-replica load about half of all outbound messages were sent twice. Nothing was lost and every instance still completed once. A single engine was not affected.
Retiring a draining deployment: a three-legged gate
A hot-deploy leaves the previous graph draining rather than deleting it, precisely so instances pinned to it keep resuming on the definition they started under. It retires only when it is genuinely quiescent — and “quiescent” is three independent facts, all of which must hold:
- No active instances pinned to it. Retained terminal history does not count (it would
otherwise pin a deployment open for a whole retention window);
FAILEDinstances do count, because they are live work awaiting an operator. - No pending outbox rows minted by it. An emission belongs to the channel bindings that produced it, so it drains where it was made. A delivery that exhausted an opt-in attempt ceiling is marked terminally poisoned and stops counting — “we gave up” is a durable, visible state, and it must not hold a deployment open forever.
- No parked external tasks waiting on it. A task parked for a pull worker is outstanding work with no in-flight anything to observe; retiring underneath it would strand the worker’s completion. Tasks that turned terminal after exhausting their worker budget stop counting, for the same reason poisoned deliveries do.
flowchart LR
I["active instances pinned to it<br/>FAILED counts, terminal history does not"] --> G{"quiescent?"}
O["pending outbox rows it minted<br/>terminally poisoned ones stop counting"] --> G
T["parked external tasks<br/>budget-exhausted ones stop counting"] --> G
G -->|"all three read zero"| R["retired"]
G -.->|"any is non-zero"| D["stays draining"]
All three legs are database-scoped counts, so the gate reads the same from every replica and is unaffected by which replica happens to run the sweep.
Recovery on pod death
Because token state lives in the database, a dead replica leaves nothing to recover from memory:
- Mid-token-execution — the instance’s claim goes stale with its owner; the stuck-instance sweep clears it and the next replica to see ready work picks the instance up.
- Mid inbox-to-instance handoff — an inbox row without a matching instance past a recovery threshold gets retried; the same unique-index dedup makes the retry safe.
- Mid outbox send — the dead replica’s claim lease lapses within 30 seconds and the row is delivered by whichever replica claims it next.
- The leader itself dies — its lease expires within its TTL and another replica’s next poll acquires it; timer scheduling pauses for at most that window, while outbox processing continues uninterrupted on every other replica throughout.
Scaling signal
CPU is not a meaningful autoscaling signal for an event-driven engine — the right one is backlog
depth (inbox rows waiting to start an instance, outbox rows waiting to send), which is what a
KEDA ScaledObject querying the engine’s own tables drives off. See the OpenTofu module under
deploy/modules/sutra/ for the shipped shape.
Non-goals
No active-passive mode (every replica processes real work; the lease only gates the singleton pieces), no sticky session routing, no tenant-to-replica affinity (isolation is a PostgreSQL RLS concern — see Multi-tenancy and isolation — not a placement one), and no engine-managed broker resources (queues/topics are provisioned by your own infrastructure tooling; the engine connects to what already exists).
Next
- Execution lanes — the in-process half: N actor lanes under one replica, and why claims rather than routing keep them correct.
- Multi-tenancy and isolation — the other half of “many tenants, one engine, one database.”
- Deployment model — how the same PostgreSQL-backed convergence pattern
(
LISTEN/NOTIFY, version polling) applies to deploy activation across a fleet. - Ownership and claims: the design reasoning — the compare-and-set, the re-entrancy invariant, and the owner-suffix conventions.
Execution lanes
Inside one replica the engine executes on N identical actor lanes. Every piece of
instance-addressed work is routed onto a lane by a stable hash of the instance id, and one lane
drains one request at a time — so all work for a single instance runs in arrival order, on one
lane, with nothing else interleaved into it. That property is the contract, and it holds
identically at every N.
Lanes are an in-process scale-out story. They sit underneath — and are entirely independent of — the horizontal one: a replica has N lanes; a deployment has M replicas; the two multiply. See Replica semantics for the cross-replica half.
Why lanes exist
A single serial execution lane is a convoy. One slow commit does not merely delay its own instance — it delays every instance behind it on the same lane, however unrelated. Splitting the lane into N lanes removes the convoy without changing what happens within a lane: the ordering guarantees a single lane gave are the guarantees each of the N lanes still gives.
Nothing else about execution changes. Lanes do not partition data, do not shard the database, and do not introduce a placement or affinity model an operator has to reason about. They are a parallelism mechanism inside a process, nothing more.
Routing: a stable hash of the instance id
The routing key is the instance id. It is the unit of durable state (one snapshot row), the unit of mutual exclusion (one ownership claim), and the unit the guarantee names. Nothing else works as a key: a correlation alias is not stable per instance (one instance can carry several alias rows with different values), a deployment or a tenant is far too coarse to spread load, and a channel splits one instance across lanes the moment its relays arrive on a channel other than its spawn’s.
Work arrives in three shapes, and the router handles each differently:
| Arrival | Routing |
|---|---|
| The id is already known — a timer fire carries the instance it belongs to | Straight to that instance’s lane. No hop. |
| The id does not exist yet — a spawn from an inbound message, or a due timer start event | Any lane may mint it; arrivals are spread round-robin. No hop, ever. |
| The id is learned mid-pipeline — a relay | Resolved on the arrival lane, then handed off if the owner lane is a different one. |
flowchart LR
T["timer fire<br/>carries its instance id"] -->|"hash(instanceId)"| OWN["the instance's owner lane"]
S["spawn<br/>no instance id yet"] -->|"round-robin"| ANY["any lane mints it — no hop, ever"]
R["relay<br/>names a business key only"] --> ARR["the arrival lane<br/>decode · validate · correlate"]
ARR -->|"resolved id, at most one hop"| OWN
Only a relay can hop, and only once — because the id it routes on does not exist until the arrival lane has decoded and correlated the delivery.
The cross-lane handoff
A relay does not name an instance id on the wire. It names a business key, and the engine only
learns the instance after channel resolution, decoding, intake validation, and evaluating the
<q:alias> correlation expression. All of that runs on the lane the delivery arrived on, exactly
as it always has.
When that resolution names an instance owned by a different lane, the arrival lane does not execute it and does not push it into the other lane’s queue. It answers its caller with a handoff: the already-decoded, already-validated resume request. The caller’s own task — the HTTP handler, the broker consumer, the outbox worker — then enqueues it on the owner lane and waits for that lane’s answer. Two rules make this safe by construction:
- Lane loops never send into another lane’s queue. Only caller-side tasks do. The mutual-block deadlock (lane A stuck sending into a full queue on B while B is stuck sending into A’s) therefore cannot arise at all.
- At most one hop. The lane that receives the resolved request runs it where it lands. It re-runs only the race-sensitive part — claim, load, terminal/failed/suspended guards, deployment pin resolution, resume — never the decode, validation, or correlation, which are deterministic over the delivery and already done.
sequenceDiagram
participant K as Caller task — HTTP handler, consumer, outbox worker
participant A as Lane A, where the delivery arrived
participant B as Lane B, the instance's owner
K->>A: the delivery
A->>A: decode, intake validation, q:alias correlation
A-->>K: handoff — the already-decoded resume request
K->>B: enqueue the resolved request
B->>B: claim, load, guards, pin resolution, resume
B-->>K: the answer
Note over A,B: a lane loop never sends into another lane's queue
The hop travels back out through the caller rather than lane-to-lane, which is what makes the mutual-block deadlock impossible and turns a full queue into backpressure on the transport.
Only relays ever hop. Spawns and timer fires never do.
Continuations before new intake
Since 0.6.5 each lane has two queues. Work that continues an instance already admitted — a
local:// delivery handed over by the outbox, a resume routed from another lane — goes on the
continuation queue, which the lane serves first; new intake waits on the other. Each queue is
first-in, first-out, and per-instance serialization is unchanged: a lane still runs one request at
a time. Under a burst, the effect is that the instances already started finish first, and intake
meets backpressure, instead of a growing backlog of half-finished work.
Claims are the correctness mechanism; routing is only affinity
This is the property worth internalizing: routing is an optimization, not the thing that keeps execution correct. Correctness comes from the per-instance ownership claim described in Replica semantics — and the claim’s owner identity is lane-scoped, not just process-scoped.
The consequence is that a mis-route is harmless. If work for an instance ever lands on the wrong lane — a bug, a hash change, anything — the claim bounces it exactly as it bounces a competing replica today: a broker relay is requeued, a timer fire is deferred and retried. A mis-route degrades to visible, retry-safe contention. It can never degrade to two lanes interleaving inside one instance.
That is also why the claim-bounce meter below doubles as the mis-route alarm.
The activation flip
A deploy activation rebuilds the engine’s live view of processes, codecs, validators, and channels. Under lanes, the controller sends that rebuild to every lane and waits for all of them before it replaces the live deployment set and rewires transports.
sequenceDiagram
participant C as The activation controller
participant L0 as Lane 0
participant LN as Lane N-1
C->>L0: rebuild processes, codecs, validators, channels
C->>LN: rebuild
L0-->>C: applied between two requests
LN-->>C: applied between two requests
Note over C,LN: the controller waits for every lane
C->>C: replace the live deployment set, rewire transports
The await-all barrier is the only thing lanes add to the flip: the later stages cannot begin until every lane has applied its rebuild.
Per-lane atomicity is what matters, and it is preserved: a lane applies its flip between two requests, never inside one, so nothing is ever observed half-flipped. Because every step of an instance runs on that instance’s one lane, its flip is a single point in its own queue — no instance can straddle two definitions. During the fan-out window two different instances can be served either side of the flip, which is indistinguishable from two deliveries ordered around a single flip point.
Lanes never block on store I/O
Each lane is one asynchronous loop awaiting one request to completion before it dequeues the next. Every persistence call on the execution path is awaited rather than blocked on, so a lane waiting on the database parks on the runtime instead of holding a thread hostage — which is what keeps tail behavior sane when lanes outnumber available pool connections.
The ordering properties are unchanged by this, deliberately: because the loop awaits each request to completion before recv-ing the next, the commit still happens-before the reply and happens-before the next request is dequeued. There is no intra-lane pipelining and no completion re-entry, so there are no re-entry rules to get wrong.
Each step’s sends are its own
While a step runs, the <q:send>s and channel-call requests it makes are collected on its lane.
They are committed with the step, in the same transaction, or not at all. A step that fails drops
them, and every step starts with nothing collected, so one request’s sends never ride along with
the next request on the lane. An engine that finds anything left over at the start of a step
discards it and logs an error.
Releases up to and including 0.6.10 broke this in one case. A wait-state process that sent and
then failed on its first pass left the send behind. The next request on the lane then committed it
as its own: a parking request failed with an HTTP 500 (outbox entry … does not belong to step instance), and a run-to-end request delivered the failed process’s send. See the 0.6.11 release
notes.
What changes at N > 1 — say it out loud
Incidental cross-instance serialization disappears. With one lane, two concurrent deliveries to two different instances of the same flow never interleave — as a side effect of there being one lane, not as a promise. At N > 1 they genuinely run in parallel.
This was never the contract, and every documented concurrency mechanism is unaffected:
- per-channel
singleton/ serial consumption (a transport-side property — one delivery in flight — which lanes do not touch); - per-channel and per-tenant admission caps (see Limits and quotas);
- optimistic
expect="unchanged"writes and pessimisticforUpdatelocks on data stores (see Data stores).
But a deployment that has been silently leaning on the single-lane side effect will observe new interleavings. That is precisely why the default is one lane and turning it up is an explicit operator action.
Two other narrowings are worth naming, and both only affect a persistence-less engine (no datasource configured — a dev/test posture): in-process alias uniqueness and in-process inbox dedup become per-lane rather than per-process. Under any pooled production posture both are database-backed and cross-lane safe, exactly as they are already cross-replica safe.
What lanes do not change
Every background role stays exactly one per replica, or one per cluster by lease — none of them becomes lane-aware beyond dispatching into the router:
| Role | Under lanes |
|---|---|
| Timer poller (lease-gated) | Unchanged as a role. Timer fires route by their instance id; schedule fires spread round-robin. Its per-tick fire loop gains bounded concurrency up to the lane count, so a timer burst is not capped at one lane. |
| Stuck-instance sweep, terminal-retention sweep | Unchanged. Both sweep by age and are owner-blind. |
| Outbox worker | Unchanged. Its in-process delivery sink re-enters through the router like any transport. |
| Deployment watcher and quiescence sweep | Unchanged as roles; the activation flip is the fan-out above. |
| Deferred-ack registry and its timeout sweep | Unchanged — one per replica, shared across lanes and across activation flips. |
Per-channel singleton consumers | Unchanged. Their serialization is transport-side (one delivery in flight), so it survives lanes intact. |
Configuring and observing lanes
sutra.engine.shards sets the lane count (default 1); sutra.engine.shard-queue-capacity
optionally bounds each lane’s mailbox, in which case a full queue makes the caller wait, so
backpressure propagates outward to the transport — never sideways into another lane. Full key
detail: Configuration reference.
Meters shipped with the feature, all carrying the lane index as a dimension:
| Meter | What it tells you |
|---|---|
sutra.engine.shard.queue-depth | Per-lane backlog. Sustained skew across lanes means a hot instance or a hot arrival burst, not an undersized fleet. |
sutra.engine.shard.dispatches / .parks / .resumes | Per-lane work rates. |
sutra.engine.shard.handoffs | Cross-lane relay hops. Expected and healthy — it rises with lane count by construction. |
sutra.engine.shard.claim-bounces (split relay / timer) | The mis-route alarm. On a correct rollout it should read near zero outside genuine cross-replica contention. |
The live lane count is readable without reading configuration: GET /sutra/health/ready
reports it in the loader check’s data.shards, read off the running router rather than echoed
back from config — which is what lets a smoke test assert that a container actually came up with
the lane count you meant.
Lane death is a health condition, not just a log line. A lane can only die outside the
per-dispatch panic containment (a failure in the lane’s own build), and after that every piece
of work hashed to it would answer SUTRA.RUNTIME.UNEXPECTED forever while the process
otherwise looks healthy. Both probes therefore watch for it: GET /sutra/health/live returns
503 with the dead lane indexes in data.deadLanes — the signal an orchestrator should
restart on, because a dead lane’s key space has no other home inside the process — and
GET /sutra/health/ready goes DOWN at the same moment so no new traffic routes to the
replica while the restart is pending.
Next
- Replica semantics — ownership claims, leader-gated singletons, and the cross-replica half of the same picture.
- Configuration reference — the two keys.
- Lanes: the design reasoning — why the ordering properties survive, what the serialization audit found, and the runtime shape that had to be falsified first.
Observability
Sutra exports all three signals — traces, metrics, and logs — through OpenTelemetry, and is fail-open about it: telemetry can never affect message processing. No endpoint configured means no exporters and zero overhead; a bad config value falls back to a default with a warning; an exporter failure only logs, it never propagates into intake or dispatch.
No telemetry, no phone-home
Sutra collects nothing on its own behalf. The engine binary, the sutra CLI, and the container
image sutra-dist produces send no usage statistics, no error or crash reports, and no telemetry
of any kind to any Sutra-affiliated destination — there isn’t one, and no code path sends
anywhere by default. Nothing phones home, in any build, ever.
The only way data leaves a running engine to a system outside it is the operator-configured
OpenTelemetry export this chapter describes, and export is opt-in, not opt-out (see
Configuration reference). With no OTLP endpoint
configured, the engine emits structured JSON logs to stdout and nothing else — the boot log says so
in plain text (telemetry export off (no OTLP endpoint configured) — JSON stdout logs only), and
that posture is enforced in the engine’s own telemetry bootstrap
(rust/crates/sutra-engine/src/otel.rs), not just asserted here. Point
sutra.telemetry.otlp.endpoint (or the standard OTEL_EXPORTER_OTLP_ENDPOINT) at a collector you
run and the signals below start exporting there; leave it unset and nothing crosses the process
boundary except whatever you separately configured elsewhere (a channel, a datastore, an audit
sink) — this is a property you can verify by reading the source or watching an unconfigured
engine’s own boot log, not a policy statement to take on faith.
flowchart LR
subgraph eng["The engine process"]
SP["traces<br/>sutra.dispatch · decode · validate · execute"]
ME["metrics<br/>sutra.instance.* · token.* · task.* · shard.*"]
LO["logs<br/>structured JSON"]
end
LO ==> OUT["stdout — always, no configuration"]
SP -.-> COL["a collector you run"]
ME -.-> COL
LO -.->|"only once an OTLP endpoint is configured"| COL
Three signals, one destination, and it is one you configured: with no OTLP endpoint set there are no exporters at all and stdout is the only thing that crosses the process boundary.
Traces
The engine’s existing tracing spans (sutra.dispatch, sutra.resolve, sutra.decode,
sutra.validate, sutra.execute, sutra.outbox.send) export through a
tracing-opentelemetry layer with no call-site changes — the same spans that back local
RUST_LOG debugging (see Troubleshooting BPMN solutions) are
what leaves the process as OTLP.
A suspended instance does not leak an open span. Because a wait state can hold an instance for
an arbitrary length of time (see Wait states and human tasks), the
executor fires a listener event at suspend that force-ends every span open for that instance —
otherwise a long park would show up as a pathologically long trace. Each segment of a
stateful flow’s lifecycle (the initial run, then each resume) gets its own trace with its own
traceId; there is no trace-of-traces joining them, since OpenTelemetry traces are flat. What
ties the segments together for a human reading a Gantt view is the instance id, stamped as a
plain span attribute (bpm.instance.id) on every span belonging to that instance — filter on it
and the segments line up on one timeline, with the waits showing as the gaps between them.
flowchart LR
S1["the initial run<br/>its own traceId"] -->|"suspend force-ends<br/>every span still open"| W(["the wait — nothing open"])
W -->|"resume"| S2["the next segment<br/>a new traceId"]
S1 -.->|"bpm.instance.id"| K["the attribute that lines<br/>the segments up in a Gantt view"]
S2 -.->|"bpm.instance.id"| K
A month-long park costs no open span; the instance id, not a parent trace, is what stitches the segments back together.
Metrics
An ExecutionListener (the same lifecycle bus described in
Domain neutrality and the SPI model) maps instance/token/task events onto
a fixed set of meter names (sutra.instance.*, sutra.token.*, sutra.task.*,
sutra.coverage.path_covered — see Coverage: declared routes as the compliance
signal for what that one actually tracks), tagged with the deployment id
and a configurable label allowlist. Alongside those, the sutra.engine.shard.* family reports
per-execution-lane queue depth, work rates, cross-lane handoffs, and claim bounces — see
Execution lanes.
Delta vs. cumulative temporality follows the standard
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE variable — set it to delta for an
Elasticsearch-backed collector, which drops cumulative histograms.
Logs
Structured JSON on stdout, always, with no configuration required — this is the log path every
deployment gets whether or not OTLP is configured. When an OTLP endpoint is configured, the same
log records additionally export over OTLP. The field shape (timestamp/level/loggerName/
message/service.name, plus traceId/spanId inside a sampled span) is stable and is what any
log-processing pipeline should key off. See Logging and audit for the
operator-facing configuration.
Cardinality discipline
Tenant id is deliberately not a tag on high-cardinality metrics (per-task duration histograms, for instance) — only on instance-counting metrics, where the cardinality stays bounded by tenant count rather than by tenant × task-name × outcome. This is the same discipline that keeps the label allowlist above short by default.
The reference stack
The repo ships a reference EFK-family stack as a set of OpenTofu modules
(deploy/modules/efk-stack) — an OTel Collector, Elasticsearch, Kibana, and a Fluent Bit
DaemonSet for host-level logs — deployed alongside the engine module in a dev/small-prod cluster,
or pointed at an external, already-owned observability stack in a larger one. It’s a reference,
not a requirement: the engine’s only actual contract is the three endpoint inputs (OTLP, an
optional log-forward target, an optional direct Elasticsearch endpoint for the audit fan-out) — any
OTLP-speaking collector on the other end works.
Next
- Logging and audit — configuring the endpoints above, and the audit trail as a separate, compliance-oriented sink from telemetry.
- Troubleshooting BPMN solutions — using traces and audit together to retrace what one message actually did.
Durable execution: snapshots and typed values
This chapter is the why behind what a parked instance actually is. If you only need the observable behaviour, Wait states and human tasks has it; this is the reasoning underneath.
A snapshot at a quiescent point, not an event log
Sutra persists an instance as a snapshot taken at a quiescent point — the moment execution has nothing left to do until the outside world answers. It does not persist a log of events and it does not reconstruct state by replaying one.
The two models differ in what they make cheap and what they make hazardous.
An event-sourced engine rebuilds state by re-running history. That gives it a free audit trail and free time-travel, and it buys them with a permanent constraint: the code that produced the history must keep being able to reproduce it. Every task function, every expression, every library the flow ever touched becomes part of the replay contract, forever, for every instance still alive. Determinism stops being a property of one execution and becomes a property of every version of the engine and the application, jointly, across time.
A snapshot engine stores what the instance is rather than how it got there. The replay contract shrinks to a single step: from this state, given this input, do the next thing. Nothing older than the current snapshot is ever re-executed, so an upgrade cannot retroactively change what already happened.
flowchart TD
subgraph ev["Event replay"]
E1["event"] --> E2["event"] --> E3["event"] --> EN["state now,<br/>re-derived every time"]
end
subgraph sn["Snapshot"]
SNAP["state at the last<br/>quiescent point"] -->|"+ this input"| NEXT["do the next thing"]
end
In the replay shape every arrow is a permanent contract — the code must keep being able to reproduce all of them, for every instance still alive. In the snapshot shape there is exactly one arrow, and it only ever points forward.
The audit trail an event-sourced engine gets for free is then an explicit, separately-configured concern here — the audit journal — and that turns out to be the honest arrangement anyway: an audit record has different retention, different access control, and different redaction requirements from execution state. Conflating them means one policy has to serve both.
The properties that fall out of the choice:
- A step is one transaction. The snapshot write, the waiting rows, the correlation aliases, and the outbound emissions all commit together, or none of them do. There is no window in which an instance has parked but not registered its correlation, or emitted but not recorded that it did.
- Encoding is deterministic. Identical logical state produces identical bytes: the container is sorted, and every value has exactly one canonical rendering. Re-encoding a decoded snapshot is byte-identical — which is what makes migration and rollback verifiable rather than hopeful.
- The frontier is explicit. A snapshot names the nodes it is waiting at, the nodes it has completed, and the start node it was routed through. Resume is “replay the completed set as done, satisfy this wait, continue” — a bounded operation over the recorded frontier, not a re-execution of the past.
flowchart LR
S["the routed start node"] -.->|"replayed as done"| A["completed node"]
A -.->|"replayed as done"| B["completed node"]
B --> W["the wait it parked at"]
W -->|"satisfied"| N["next node, executed for real"]
N --> R["…continue"]
Resume re-walks the graph without re-running it: the dashed edges are bookkeeping over the recorded frontier, and the first genuine execution after a wait is the node after it.
The typed value encoding
Instance variables ride inside the snapshot container. How they are encoded is the part with the most design in it.
The defect it fixes
A parked instance used to flatten every variable to its display string. A number became
"1250.75". A boolean became "false". A list became "[1, 2]". A null became the empty string.
Resume restored all of them as strings.
That is not cosmetic. FEEL — correctly — does not coerce a string in arithmetic or comparison, so
an exclusive gateway re-evaluating amount > 100 after a wait compared a string to a number and
got null, which a gateway condition reads as false. A transaction that plainly exceeded its
limit took the under-limit branch. not(approved) was worse: "false" is a non-empty string, so a
restored boolean was never false again. The same wait state that made the engine durable made its
decisions wrong.
The loss happened in exactly two places — the park ran every value through the display formatter, and resume wrapped every restored value back into a string — and both are gone.
The encoding
A typed value is <tag>|<payload>: one ASCII tag byte, a separator, then the payload. Only the
first separator is structural, so a string whose own text is n|42 round-trips unambiguously
as s|n|42.
| Tag | Type | Payload |
|---|---|---|
z | null | empty |
b | boolean | true / false |
n | number | canonical decimal text, scale-faithful |
s | string | the raw text |
d | date | ISO-8601 date |
t | time | FEEL time literal body |
i | date and time | FEEL date-and-time literal body |
u | duration | ISO-8601 duration |
j | list / context | JSON |
The separator is chosen to be a character the container’s own escaping never touches, so a tagged scalar costs exactly two bytes and the row stays legible to an operator reading it:
sutra.var.amount=n|1250.75
sutra.var.approved=b|false
sutra.var.cancelledAt=z|
sutra.var.inboundId=s|INB-7
sutra.var.lines=j|[1,{"sku":"A-1"}]
Lists and contexts ride JSON. The four temporal types have no JSON counterpart, so they ride a
single-key object ({"@d":…} and siblings), and a user context key that would collide — any key
starting @ — is escaped by doubling its leading @. The two are never confusable in either
direction.
Why the generation integer says 4, not 3
The snapshot format carries a generation integer, and typing takes 4.
Generation 3 was already spent. It is the generation at-rest encryption introduced, where the
integer means this snapshot carries ciphertext. Redefining 3 to mean typed would misread
every encrypted row already persisted — and distinguishing the two by probing values for something
that looks like a tag is exactly the kind of heuristic a persisted format must never depend on.
So the conflation stops at 3. Encryption has always been detected structurally, off the
ciphertext key prefix — the read path never consults the generation integer for it at all — which
means 4 means one thing only: the value encoding. An encrypted snapshot with typed values is
simply 4. There is no generation for “typed and encrypted”, and there never needs to be.
Compatibility: lenient decode, lowest-generation encode
Decode is version detection, not migration. An older snapshot is not tag-decoded at all: every value becomes a string, byte for byte what it was — including a legacy value that happens to look like a tag. An old row read and re-written untouched reproduces its original key/value map (in the binary container since 0.6.7). There is no silent upgrade of values on load.
Encode emits the lowest generation that can carry the state: 4 when at least one variable
actually needs the typed form, else 3 when something is encrypted, else 2. An instance whose
variables are all strings therefore still writes the exact bytes it wrote before typing existed.
That asymmetry did real work for generations 2 to 4: it kept a golden corpus valid across the feature and let a fleet mid-upgrade write rows an older replica could still read. The binary container of 0.6.7 (below) gives that up on purpose.
The decode path is also deliberately lenient in one further respect: a malformed engine-internal counter reads as zero rather than as an error. A snapshot must never become unloadable. An instance that cannot be read is an instance no operator can inspect, migrate, or terminate — a strictness that turns a small corruption into an unrecoverable one is the wrong trade.
The binary container (format 5, since 0.6.7)
Up to 0.6.6 the key/value map was stored as Properties text: every character of every key and value went through escaping on write and unescaping on read, which made encoding and decoding a visible share of each step’s CPU. Since 0.6.7 the same map is stored as a binary container:
\uSNAPSHOT/5\n marker, 13 bytes
entry count u32, little-endian
per entry, keys ascending: u32 key length, key bytes (UTF-8), u32 value length, value bytes
The map inside is unchanged — the same keys, the same tagged values, the same sutra.snapshot
generation, the same encryption envelope and key id — so everything above still holds, and the
byte-level patchers work on either container. Identical state still produces identical bytes.
- Reading old rows. A row written by any earlier release starts with a
sutra.key, never with the marker, so it is read as text exactly as before. It is rewritten in the binary container the next time its instance commits (a re-park, completion, a FAILED or terminal re-stamp, a migration). Nothing is migrated in bulk. - Older engines cannot read new rows. A release before 0.6.7 reads the marker as a malformed
Properties escape and refuses the row (“failed to decode persisted instance snapshot: malformed
\uXXXX encoding”); the instance is left untouched. So:
- upgrade the whole fleet at once — stop every older replica, then start 0.6.7 — rather than rolling replica by replica; an older replica still running would refuse every instance a 0.6.7 replica has committed;
- rolling back to 0.6.6 or earlier works only for instances no 0.6.7 engine has committed; finish or cancel the rest first.
- Fail-closed. A truncated container, a length that runs past the end, trailing bytes or non-UTF-8 text is a decode error, never a partial instance.
The byte-level key patchers
Three operations rewrite a stored snapshot without decoding it: marking an instance failed, marking it terminal, and re-pinning it during migration. Each patches the raw key/value map in place — engine-internal keys only — and never touches a variable value.
This is deliberate, and the reason is a security one rather than a performance one. Decoding and re-encoding a variable would need the tenant’s data-encryption key. That means:
- an operation as mundane as “mark this instance dead” would acquire a dependency on key availability, and would fail — or worse, partially succeed — when the key backend is down;
- it would have to re-derive which variables are sensitive, from a snapshot that at resume time no longer carries that set in the shape the encryptor wanted;
- and if either went wrong, it would persist a previously-encrypted value in the clear.
Marking an instance dead, finished, or re-pinned must never need the tenant key and must never be able to downgrade an at-rest value to plaintext. Treating every variable as an opaque string is how that is guaranteed structurally rather than promised procedurally.
Typing does not weaken the property, by construction: to the patchers a value is still an opaque string. A snapshot carrying a tag this codec does not recognise rides a patch through untouched rather than being “repaired” — a forward-compatibility posture the tests pin explicitly.
Encryption: the tag rides inside the ciphertext
At the typed generation, the plaintext that gets encrypted is the tagged form. A sensitive number is a number again after the resume, exactly like a non-sensitive one.
The alternative — an outer tag sitting beside the ciphertext — was rejected because it would disclose the type of every encrypted value to anyone who could read the row, for no benefit at all. “This encrypted field is a number, and that one is a list” is information, and it is information the row does not need to carry.
Everything else about the envelope is unchanged. The authenticated-data binding still ties a ciphertext to its key, its instance, and its variable name, and still deliberately excludes the deployment id — which is precisely what keeps “a migration changes only the pin” true, and what lets a migrated instance’s encrypted values still decrypt (see Migration internals). Decode still fails closed on a missing cipher or a failed authentication.
What is deliberately not typed
- A subject blind-index input. A subject value feeds a keyed hash over index rows that are already persisted. It keeps hashing the exact string it hashed before typing existed; deriving it from the typed value would orphan every index entry written to date — and those rows are how erasure and disclosure find an instance at all.
- The operator inspect projection. A published contract: every variable still renders through its display form. Typing changed what survives a wait, not what a response looks like.
- Functions and ranges. A FEEL function closes over an evaluation context that ceases to exist the moment the instance parks; a range is a comparison shape rather than instance state. Both persist as their canonical string — which is what they always did, so nothing regressed. They simply did not become typed.
The one behaviour change a consumer can observe
Every restored value now evaluates as its real type. Where a gateway condition’s answer differs
from a pre-typing release, the previous answer was wrong — that is the entire point of the
change. The only other visible difference is that a null variable now restores as null (and renders
as null rather than blank) instead of restoring as an empty string.
The snapshot key registry
Everything a stored snapshot’s key/value map can carry (in either container), in one place — useful when reading a raw
row. sutra.-prefixed keys are engine-internal: opaque strings to the byte-level key patchers and
never tag-decoded. sutra.var.<name> entries are the user variables the typed encoding exists for.
| Key | Written by | Meaning |
|---|---|---|
sutra.snapshot | codec | Format generation: 2 plaintext-untyped, 3 encrypted-untyped, 4 typed (encryption stays orthogonal — detected off the sutra.enc. prefix, never the version). |
sutra.status | executor / re-stamps | Instance status; terminal and FAILED re-stamps are byte-level patches. |
sutra.deploymentId | executor / migration | The content-hash pin; rewritten only by the validated migration operation. |
sutra.processId | executor / migration | The owning process definition id; rewritten only by a cross-process re-home. |
sutra.waitingNodes / sutra.completedNodes / sutra.startNode | executor | The replay frontier; node ids, mapped on migration. |
sutra.retry.<nodeId> | retry machinery | Durable attempt counter for a retry policy; the key is renamed on migration, the counter never resets; absent unless the task has a policy. A malformed counter reads as zero — a snapshot must never become unloadable. |
sutra.retryWait.<nodeId> | retry machinery | The backoff-window marker on a channel-call task: present exactly while a dead attempt’s backoff timer is pending, carrying the failure’s classification code. Renamed with the node on migration. |
sutra.auditSeq | audit | Per-instance audit sequence floor. |
sutra.var.<name> | executor | A user variable — tagged typed value at generation 4; a plain string in older rows. |
sutra.enc.<name> | crypto envelope | Ciphertext for a sensitive variable; the authenticated data binds key id, instance id, and variable name — and deliberately excludes the deployment id, so migration stays decryptable. At generation 4 the encrypted plaintext is the tagged form. |
sutra.keyId / sutra.sensitive | crypto envelope | The tenant key anchor (plaintext, self-describing) and the sensitive-name set. |
sutra.coverage.<pathId> | coverage | Path-coverage cursors — keyed by declared path id, not node id; migration must not touch them. |
sutra.failureCode / sutra.failureDetail | FAILED re-stamp | Structured code + captured message of the fatal failure, for inspection and post-repair migration. |
Next
- Ownership and claims — what guarantees only one worker is ever writing one instance’s snapshot.
- Retry machinery — attempt counters as snapshot keys, and why that choice costs zero migration.
- Migration internals — re-pinning a snapshot without decoding it.
Ownership and claims
Two things must never happen to one instance: two workers advancing it concurrently, and a worker holding it forever. The ownership claim is the mechanism for both. The observable behaviour is in Replica semantics; this is why it is shaped the way it is.
The claim is a compare-and-set, and a failure is a refusal
Every resume path — a correlated relay, a timer fire, an administrative migration — takes the claim before it rehydrates anything, by a conditional update on the instance’s own row: take ownership if it is free or stale, otherwise match nothing.
The critical design choice is what happens when it matches nothing. It refuses. It does not wait, retry in place, or queue behind the holder.
Waiting would be wrong on three counts. It would make the runtime of a request depend on another replica’s work, on that work’s timescale, with no bound anybody can reason about. It would hold a transport resource — an in-flight HTTP request, a broker prefetch slot — for the duration. And it would make a contended path a fundamentally different code path from an uncontended one, which is exactly the kind of asymmetry that hides bugs until the day of the incident.
Refusing instead composes with mechanisms that already exist. A broker relay is requeued and
redelivered; a timer fire is deferred to a later tick with backoff; an administrative call answers
409 having read and written nothing. The refusal carries a structured code
(SUTRA.RUNTIME.RESUME.CLAIM_HELD, or the admin surface’s SUTRA.ADMIN.MIGRATE.CLAIM_HELD) so a
caller can tell “you lost a race, try again” apart from “this will never work”.
flowchart LR
C{"claim: free or stale?"} -->|"yes"| RUN["rehydrate, run the step"]
C -->|"no"| REF["refuse — nothing rehydrated"]
REF --> RQ["broker relay:<br/>requeued"]
REF --> TM["timer fire:<br/>deferred, with backoff"]
REF --> AD["admin call: 409,<br/>nothing read or written"]
The claim comes before any rehydration, so a contended path ends at the diamond rather than partway through a step — and every branch out of it is a mechanism that already existed, which is why contention needed no new recovery machinery of its own.
Release is part of the commit, and part of every other exit
The claim is released inside the same transaction that commits the step’s writes. “Committed” and “unowned again” are therefore one atomic fact, and there is no window in which an instance is durably advanced but still shows an owner — a window a sweeper would eventually have to clean up, and that a concurrent resume would misread as contention in the meantime.
Every path that does not commit has to release too: an early refusal, a validation error, an unexpected failure, a panic. Originally that was a scope-guard — release on drop, automatically, whatever the exit. When lane loops became asynchronous, the guard had to change shape, because a destructor cannot await and releasing is a database call.
So the guard became an explicit release on every exit, including the panic path, which is re-raised identically afterwards. The mechanism moved; the invariant did not. It is worth being honest that this trades a compiler-enforced guarantee for a reviewed one — the mitigation is that the exits are few, enumerated, and directly tested, and that the sweep below is the backstop for the case where the process is not around to run any exit at all.
Re-entrancy, and the invariant that makes it safe
The compare-and-set deliberately grants a re-claim to the same owner. That looks like a hole until you see the invariant it rests on.
Originally the reasoning was: an owner id names one process, and one process advances instances on a single execution thread — so the same owner cannot possibly be in two places at once. Re-claiming what you already hold is a heartbeat refresh, not contention.
Introducing N execution lanes inside one process invalidated that reasoning exactly. Under a per-process owner id, lane B claiming an instance that lane A is mid-step on would succeed re-entrantly: a silent double-resume, which is precisely the corruption the claim exists to prevent. All-green tests at one lane would have proved nothing, and the cross-replica tests use different owners, so they could never have caught it.
The fix is to make the owner id lane-scoped, which rotates the invariant into its honest form:
same owner ⇒ same lane ⇒ already serialised
The words changed and the guarantee did not. That is the shape to want from an invariant when the system underneath it moves.
Routing is affinity; claims are correctness
This is the property the whole lane design leans on, and it is worth stating as its own claim.
Instance-id hash routing exists to give a lane affinity for the instances it owns — fewer cross-lane hops, better cache behaviour, less queue churn. It is not what makes execution correct.
If work for instance X ever lands on the wrong lane — a routing bug, a hash change, a handoff rule
misapplied — the claim bounces it exactly as it bounces a competing replica: CLAIM_HELD, requeue
for a relay, defer for a timer fire. Mis-routing degrades to visible, retry-safe contention. It
can never degrade to interleaved execution.
That is a much better failure mode than the alternative, where routing is the mutual exclusion and a routing bug is a silent data-corruption bug. It also gives the rollout a direct observable: the claim-bounce meter, split by relay and timer, should read near zero at a healthy multi-lane rollout, so a mis-route shows up as a number going up rather than as a support ticket six weeks later.
Owner-id suffixes, and why they cannot collide
An owner id is an opaque string to the store and to the sweeper — neither parses it — which is what lets the conventions layer without a schema change:
| Suffix | Who uses it | Why it is distinct |
|---|---|---|
| Lane index | Every execution lane | So same-owner re-entry means same-lane re-entry (above). |
| A migration marker | The administrative migrate operation | So a migration cannot re-enter past a resume this same replica has in flight. |
The migration case is the subtle one, and it is the direct consequence of re-entrancy. Claiming under the bare replica identity would succeed against a resume in flight on that very replica — the one race the claim exists to prevent, reintroduced by an operator action. Under a distinct owner the compare-and-set fails honestly, and a resume that starts after the migration claims bounces off it in turn.
The two conventions cannot collide because they occupy different, non-overlapping parts of the identity: a lane suffix is a lane index on the execution path, and the migration marker is not a lane index at all and is never produced by a lane. There is no string a lane can generate that an administrative claim can also generate.
Heartbeats: deliberately not wired
A claim could refresh itself mid-step. It doesn’t, and that is a decision rather than an omission.
A step runs between quiescent points — milliseconds to low seconds — which is orders of magnitude inside the default claim timeout. Adding more execution lanes shortens the queue wait before a step; it does not lengthen the step itself. So the mechanism would fire constantly and protect against nothing.
The condition that would change the answer is a genuinely long-running step. The pull-worker surface is the shape that could produce one — but it deliberately does not: a task parked for a worker is not a step in progress. The instance parks, its claim is released with the commit, and the worker’s completion re-enters as a fresh delivery later. The long wait lives in a task row’s lock, not in an instance claim, which is exactly why no heartbeat is needed. See The pull surface.
The sweep is the only backstop that can exist
No release protocol survives a process that stops existing. The stuck-instance sweep is the answer: a lease-gated role that clears any claim whose owner has been silent longer than the claim timeout.
stateDiagram-v2
[*] --> Unowned
Unowned --> Held: compare-and-set, free or stale
Held --> Held: same owner, so same lane — re-claim granted
Held --> Unowned: released inside the committing transaction
Held --> Unowned: released on every non-committing exit
Held --> Unowned: swept, owner silent past the claim timeout
Three ways out of Held, in decreasing order of how much the design leans on them: the commit
itself, the enumerated non-committing exits, and the sweep — which exists only for the process that
is no longer around to run any exit at all.
It sweeps by age and is owner-blind — it does not parse owner ids, does not know how many lanes a replica has, and does not care. That is what let the lane-index suffix land with no change to the sweep at all: owner cardinality grew, and the sweep predicate never mentioned owners in the first place.
The tuning trade is plain in the two keys. A short claim timeout recovers faster from a dead replica and risks stealing an instance from a live-but-slow one — which the claim’s own compare-and-set then makes safe rather than catastrophic, since the original owner’s commit no longer matches. A long one is the conservative direction. The defaults sit far enough above normal step duration that the first case does not arise in practice.
Next
- Execution lanes: the design — what the lane-scoped owner is protecting.
- Migration internals — the distinct-owner claim in the operation that needs it most.
- Replica semantics — the operator-facing view.
Execution lanes: the design
Execution lanes describes what lanes do. This chapter is the reasoning: what the single-lane design was actually buying, how each of those properties was preserved rather than re-argued, the one genuine cross-lane race that had to be fixed, and the runtime shape that had to be falsified before the right one was found.
What the single lane was buying
Before lanes, the engine was one actor thread draining one queue, with every store call blocked on rather than awaited. That is easy to dismiss as an accident of an early implementation. It wasn’t — four properties rode on it, and each had to be accounted for.
Because a dispatch was synchronous end to end, the step’s commit happened-before the reply to the caller, and happened-before the next request was dequeued. From that:
- Reply implies committed. A transport that got an answer knows the park is durable. This is what makes an acknowledgement mode meaningful at all.
- At most one store transaction in flight per process. Connection-pool behaviour was trivially bounded.
- A park can never race its own completion. The deferred-acknowledgement registration that follows a commit was serialised against every terminal event, because nothing else could run in between.
- The activation flip is atomic with respect to dispatches. Nothing is ever observed half-flipped.
The honest observation — the one that set the whole sequencing — is that blocking on a dedicated thread is not itself the scaling defect. The defect is having one serial lane, where a single slow commit convoys every unrelated instance behind it. N lanes fix the convoy while changing nothing else. Removing the blocking is a separate change that buys different things and should be judged on its own.
So the work split: first N lanes with the blocking intact, then the conversion to awaiting, with the ordering properties preserved verbatim at each step. Each half is reviewable on its own terms, and a regression can be attributed to one of them rather than to “the concurrency change”.
The total-serialization audit
The interesting engineering was not writing a router. It was enumerating everything that implicitly relied on there being one thread — because every such reliance was invisible, uncommented, and correct right up until it wasn’t.
The method was to walk each site and classify it into exactly one of four buckets:
| Verdict | Meaning |
|---|---|
| Safe | It relied on one step running wholly in one place, not on one thread existing. A step still does. |
| Safe with a change | Per-process scratch becomes per-lane scratch; a generated id gets the lane index mixed in so it stays unique. |
| Accepted narrowing | A per-process property becomes per-lane. Sound only where the production posture does not rely on it. |
| Genuine race | It relied on total order across different instances. Must be fixed. |
Most sites landed safe, and for one recurring reason: per-instance state has a durable home, and a step runs wholly on one lane. In-memory audit sequence counters, for instance, are re-seeded from the snapshot before every resume, so a stale entry on a lane that last saw an instance is simply overwritten wherever the next resume lands.
The accepted narrowings were the ones worth writing down honestly, because they are real degradations rather than non-issues. In-process correlation-alias uniqueness and in-process inbox deduplication become per-lane instead of per-process. Both are only reachable in a persistence-less posture (a dev or test engine with no datasource). Under any pooled production posture both are database-backed — a unique index and a row lock — and already cross-replica safe, which strictly implies cross-lane safe. Documenting the narrowing is the price of not pretending it doesn’t exist.
One further site is a deliberate removal rather than a narrowing: incidental cross-instance serialization. Two deliveries to two different instances of one flow never interleaved, as a side effect of there being one lane. That was never a contract, but a deployment could have been leaning on it. It is the reason the default lane count is one, and the reason the configuration page says the sentence out loud rather than burying it.
The one genuine race, and its inversion
Exactly one site was a real cross-lane race, and it is worth walking through because the fix is smaller and stranger than the problem.
The deferred-acknowledgement registry lets a transport hold an inbound acknowledgement until the instance actually completes. The original order was: commit the park, then register the settle callback. Safe under one lane, because nothing could run between the two.
Under lanes it breaks. The park runs on the lane the delivery arrived on — call it A. The instance’s first relay routes to the instance’s owner lane — call it B. B can claim, resume, and complete the instance in the window between A’s commit and A’s registration. The terminal event then fires with no registration present, the acknowledgement never fires, and the delivery dangles until the registry’s timeout sweep negatively acknowledges it. Microseconds wide, invisible under load, catastrophic when it hits.
The fix is to invert the order: register before committing, and deregister if the commit fails.
The window closes cleanly, and the argument is the load-bearing part:
Before the park commits, no correlation alias row exists — alias rows ride the step transaction. So no relay anywhere can correlate to this instance, and therefore no terminal event for it can fire from anywhere except this same dispatch, which is the one blocked on the commit.
A failed commit deregisters, and the caller sees the same error it always did. The registry’s documented invariant rotates from “registered ⇒ the transport was told to defer” to “the transport was told to defer ⇒ registered ∧ committed” — which is the direction the transport actually depends on, and arguably what it should have said all along.
At one lane the inversion is unobservable, which is why it landed unconditionally rather than as a behind-a-flag branch. A correctness fix that only applies in one configuration is a correctness fix you have to reason about twice.
The lesson generalizes: the ordering was safe because of a property (total order) that was never written down as a requirement. Auditing for implicit reliance is the only way those surface before production does it for you.
Handoff rules
A relay does not carry an instance id. Resolving one needs the decode, the intake validation, and the correlation expression — all of which need the lane-resident registries. So resolution happens where the delivery arrived, and the resolved request is then handed to the owner lane. Two rules constrain it, and both are structural rather than advisory.
Lane loops never send into another lane’s queue. Only caller-side tasks do. This is what removes the inter-lane deadlock case by construction: if lane loops could enqueue into each other, lane A blocked sending into B’s full queue while B is blocked sending into A’s is a genuine deadlock, and no amount of capacity tuning eliminates it — it just makes it rarer and therefore worse. Routing the hop back out through the caller’s own task means a full queue applies backpressure to a transport, where backpressure belongs.
At most one hop. The lane that receives a resolved request runs it where it lands. It cannot need a second hop — the instance id is fixed, so re-resolution would name the same lane — and more importantly, correctness must not depend on the hop landing correctly. It doesn’t: a wrong landing bounces on the claim. The hop is affinity; the claim is correctness. See Ownership and claims.
The receiving lane re-runs only the race-sensitive part — claim, load, guards, pin resolution, resume. It does not repeat decode, validation, or correlation, which are deterministic over the delivery and already done. Redoing them would be wasted work and, worse, a second place where those semantics could drift.
The runtime shape, including the one that was wrong
Three shapes were weighed for removing the blocking store calls.
(a) One asynchronous loop per lane, awaited to completion per request. Every persistence call becomes awaited; the loop is “receive a request, await it fully, receive the next”. Because the loop awaits each request to completion before receiving the next, every ordering property above is preserved verbatim per lane — commit still happens-before reply and before the next dequeue. No completion re-entry exists, so there are no re-entry rules to get wrong.
(b) Blocking loop plus I/O offload with completion re-entry. Split a step at the commit, hand the write to a worker, return to the loop, and re-enter a completion event later. This is the only option that adds intra-lane pipelining — and it was rejected. It breaks reply-implies-committed unless responders are parked on the completion; it needs per-key in-process mailbox holds, which is a second serialization mechanism layered on top of claims; and its failure modes (a commit landing after a crash-restart re-queued the work, a lost completion, a mailbox stuck held) are precisely the subtle-ordering-bug class the design exists to avoid. Two mechanisms enforcing one invariant is how invariants get broken.
(c) N blocking lanes. Zero semantic change, and N× I/O concurrency arrives from N threads.
The order shipped was (c), then (a); (b) was rejected outright. (c) makes the convoy fix reviewable and bisectable on its own. (a) then removes the blocking as a mechanical conversion whose diff is wide but whose semantics are provably unchanged. What (a) buys concretely: lanes park on the runtime instead of holding threads when the pool is the bottleneck, an entire class of “blocked inside an async runtime” panic disappears, and the door stays open to pipelining later without another trait migration.
The shape that was falsified
The first cut of (a) gave each lane its own runtime with its own I/O reactor. It looked like the clean version: a lane is fully self-contained, owning its execution and its I/O.
Its own acceptance gate — the “one lane must be identical to before” bar — falsified it. When a lane shut down, its reactor died with it, and pooled database connections registered on that reactor were stranded: sockets nobody would ever poll again. The visible symptom was a restart hang. The successor engine’s lease acquisition waited out the full lease TTL before it could proceed, roughly half the time. A restart-timing flake, not a wrong answer — the kind of thing that gets triaged as flaky infrastructure for months.
The fix inverted the ownership: lanes own no reactor at all. Each lane’s loop is driven on its own dedicated thread via the process-wide runtime’s handle, so the reactor topology is identical to what it was before the conversion, and lane lifetime is decoupled from I/O registration lifetime. Restart handover got materially faster as a side effect, because nothing waits out a lease TTL any more.
Two things are worth taking from this. First, the “obviously clean” decomposition was wrong because it decomposed the wrong resource: execution is per-lane, but I/O registration is per-process, and conflating a lifetime with a locality is a recurring shape of bug. Second, the identity bar is what caught it. An acceptance criterion of “one lane behaves exactly as before” is a much stronger instrument than a suite of feature tests, because it fails on things nobody thought to assert.
The activation flip
Activation rebuilds the engine’s live view of processes, codecs, validators, and channels. Under one lane that rebuild was a single request, atomic against every dispatch.
Under lanes the controller sends the rebuild to every lane and awaits all of them before replacing the live deployment set and rewiring transports. Per-lane atomicity is preserved; that is all that is needed, and the argument for why is worth being explicit about.
The flip never promised cross-component simultaneity in the first place. Even under one lane the later stages — replacing the deployment set, reconciling schedules, rewiring transports — were already non-atomic with respect to the actor swap; deliveries kept flowing between them. What was promised, and is still promised, is per-dispatch consistency: nothing is ever observed half-flipped.
Per-lane atomicity delivers exactly that, because an instance’s steps all run on its one lane, whose flip is a single point in its own queue. No instance can straddle two definitions. During the fan-out window two different instances can be served either side of the flip — which is indistinguishable from two deliveries ordered around a single flip point, which is what happened before. The draining-deployment correlation tail is part of the rebuilt view and flips with it, per lane.
Where false confidence would have come from
The risks that mattered were not the ones the code made obvious. They were the ones where a green test suite would have been actively misleading:
| Risk | Why a green suite would have lied | What actually catches it |
|---|---|---|
| Silent double-resume via re-entrant claim | One-lane suites cannot exercise it; cross-replica tests use different owners | Lane-scoped owner makes the window structurally impossible; a deliberate mis-route test proves the bounce |
| The park/completion acknowledgement race | The window is microseconds — stock tests pass almost always | The ordering inversion, plus a race test with an injected pause exactly in the window |
| Flip skew across lanes | Single-replica flip tests never produce skew | Await-all barrier, plus a flip-under-load test at several lanes |
| Ordering regressions from extracting the router | “It is just plumbing” reviews | A byte-identical outcome-sequence bar at one lane |
| Hot-instance skew pinning a lane | Averaged throughput hides it entirely | Per-lane queue-depth is a shipped meter, not a debugging afterthought |
| Pool exhaustion at N concurrent commits | Testing against a generous pool | A soak against a deliberately small pool |
| The per-lane narrowings surprising a persistence-less deployment | The tests exercise the pooled path | Documented, and unchanged under any pooled posture |
The pattern in that table is the transferable part: for each risk, name the specific green result that would have been misleading, then design the test that would not have been. An all-green-at-one-lane suite proves nothing whatsoever about cross-lane windows.
Next
- Execution lanes — the operator-facing chapter.
- Ownership and claims — the mechanism lanes lean on for correctness.
- Durable execution — what a lane commits at a quiescent point.
Retry machinery
Retries, history, and schedules covers what <q:retry>
does. This chapter is why it is built the way it is — and the channel-call half turned out to have
a genuinely hard problem in it.
A backoff is a park, not a sleep
The first decision constrains everything else: a retry wait is a durable timer park.
The alternative — sleep in place and try again — is not merely inelegant here, it is unavailable. A lane executes one request at a time, so an in-path delay does not delay one instance; it freezes every instance queued behind it on that lane. A ten-second backoff would become a ten-second stall for unrelated work. Even with lanes, a sleeping lane is a lane doing nothing while holding a queue.
So a failed attempt with budget remaining re-parks the instance: the failed task stays pending, a timer row is armed at the backoff instant, and the ordinary timer poller re-drives it when the instant arrives. Deliberately the same seam that respond-and-continue already uses — not a parallel mechanism. A second scheduling path would be a second set of races to reason about, and the existing one already survives restarts, hot-deploys, and replica death.
Everything good about the behaviour follows from that one choice. The backoff survives a restart. A
PT5M wait is PT5M whether or not the replica that started it is still alive. No execution
capacity is consumed while waiting. And a retry park is pinned to its deployment exactly like every
other park, so a hot-deploy cannot silently move a backing-off task onto a different definition.
The cost is honest and stated: a <q:retry> task makes its process stateful. The park needs
persistence. A process that was otherwise run-to-completion now requires a datasource, and the
structural classifier accounts for it rather than discovering it at runtime.
Attempt state lives on the snapshot
Attempt counters are snapshot keys, one per node, not a column on the waiting row. Three reasons, in increasing order of importance.
A column would be erased by the very step that increments it. The retry park resolves its timer row and creates a new one each attempt. Attempt state stored there would be destroyed by the re-park — the mechanism would delete its own bookkeeping.
Attempt state is instance state. It must ride the failure re-stamp untouched when the instance eventually fails, so that an operator inspecting a dead instance can see how many attempts it burned. Putting it on the snapshot gets that for free, because the failure re-stamp is a key patch over the snapshot’s map.
It costs zero migration and stays byte-identical when unused. The snapshot container is an
open-keyed map, so a new key family needs no schema change anywhere. And because the key is absent
unless a task actually has a policy, a process with no <q:retry> writes byte-identical
snapshots to one that predates the feature — which keeps a byte-for-byte golden corpus valid
across the whole feature. A feature that changes the persisted bytes of flows that don’t use it is
a feature that cannot be verified cheaply. See Durable
execution for the encoding and the patchers.
A malformed counter reads as zero rather than failing the load, for the same reason the rest of the decode path is lenient: a snapshot must never become unloadable.
The channel-call problem
Extending <q:retry> to channel-call tasks was initially skipped, on the theory that the outbox’s
own retry curve plus the timeout boundary already covered them. A survey of what the code actually
did disproved it:
- the outbox curve retries one delivery, by default forever — it is about reaching a counterpart, not about the task having another go;
- a timeout without a policy simply kills the instance.
So there was a real gap: no way to say “if the counterpart doesn’t answer, ask again”. Closing it meant answering a question the registered-task case never has to: what, exactly, is a failed attempt of a call?
The honest failure set
Derived from what the dispatcher and executor actually do, and nothing else:
- The route-less
<q:timeout>boundary firing. With a policy present the timeout is a retryable task failure first. Without one, the pre-existing catchable-error behaviour is preserved byte for byte. - A terminally-poisoned request delivery. The outbox exhausted its configured attempt ceiling while the task waited. Before this existed, the instance simply stayed parked forever — the delivery had given up and nothing told the task.
Both are stable structured codes, which is what makes them expressible in nonRetryableCodes.
(“Never retry timeouts” is a reasonable policy, and it has to be sayable.)
Not failures, deliberately:
- A correlated business response. The counterpart answered. What the answer says — approved, declined, rejected — is the process’s business to route on with a gateway. Retrying because a business answer was unwelcome would double-submit, and re-issuing an instruction because the first answer was “declined” is exactly the class of bug an engine must make structurally impossible.
- BPMN errors. They route to their boundaries, unchanged, on either task kind.
Modelled outcomes beat policies — enforced at load
A timer boundary with drawn outgoing flows is a modelled outcome: the author has said what happens. So it wins, exactly as a BPMN error routes to its boundary rather than consuming a retry budget.
Which means a channel-call <q:retry> alongside a routed timer boundary is a policy that could
never fire. Rather than let that load as a silent near-no-op, the loader refuses the
combination. A configuration that cannot do anything should not be accepted quietly; the author
believes they have retries and does not.
The hard part: a dead attempt and an in-flight one look identical
Here is the problem that shaped the rest of the design.
Consider a channel-call node in a backoff window — its previous request failed, it is waiting to re-issue — and a channel-call node whose request is in flight right now. Through the durable facts available (the wait frontier plus the attempt counter) they look exactly the same: a node waiting, with attempts already burned.
They demand opposite treatment. A late response to the dead attempt must be refused; a response to the live attempt must resume the flow. A due timer on the first is the re-drive; a due timer on the second is stale residue.
The registered-task shortcut does not work here. For a registered task, “attempts burned and not completed” implies a backoff window. For a call node it does not, because a counterpart’s relay can name the node mid-retry.
The backoff-window marker
The resolution is one more durable key: a marker set while a node is in a backoff window, carrying the classification of the failure that parked it. It is carried exactly like the attempt counters — an open-keyed snapshot entry, no migration, absent by default so unaffected snapshots stay byte-identical, and renamed along with the node mapping when an instance is migrated.
It keys four decisions, each made under the instance claim from durable facts alone:
| Situation | Verdict |
|---|---|
| A due timer on a marked call node | The backoff re-drive. Explicit, because the registered-task inference is unsound here. |
| A due timer on an unmarked call node | Stale residue. Resolved, never driven. |
| A timeout boundary firing on a marked host | Stale — a poisoned delivery beat it to the failure. Firing would double-count one attempt. |
| A response correlating to a marked node | Refused (SUTRA.DISPATCH.CHANNEL_CALL.RETRY_PENDING) — it belongs to the dead attempt. |
That last row has a deliberate sub-decision: the correlation row stays live rather than being
retired. A counterpart answering a superseded request gets an honest “that attempt is gone, a retry
is pending” verdict instead of a “no such instance” miss — the same posture durable FAILED takes,
where the truth is more useful than a 404. And when the re-drive re-emits, the same correlation
serves the new attempt’s response normally, with no re-registration.
The re-emission contract
A re-drive is not “wait for the same answer again”. It re-runs the park’s side effects from durable state:
- a fresh request emission, built over the same persisted variables, with a fresh idempotency key on the wire;
- a fresh timeout window;
- the response wait re-incarnated — a node both resolved and re-parked in one step is a new incarnation, and is recorded as such rather than as an update of the dead one.
The fresh idempotency key is the subtle one and it is deliberate. The retry is a genuinely new request, and a counterpart doing its own deduplication must see it as new. Reusing the key would invite the counterpart to answer “already handled” for an attempt it never actually completed — turning the retry into a silent no-op precisely when it matters.
Withdrawing the dead attempt’s deliveries
The backoff park deletes the dead attempt’s outbox rows — pending and poisoned alike — in the same transaction that arms the backoff.
This is a deliberate, narrow exception to an otherwise absolute rule: the outbox never deletes an undelivered row. The rule exists because a deleted undelivered row is lost work with no trace, and it is a good rule. Two concrete races justify the exception:
- A superseded request delivered late would race the re-drive’s fresh emission into a double-submit — two live requests for one logical call.
- A superseded request poisoning later would fire a failure against the live attempt, consuming a budget slot that attempt never spent.
Both are worse than the loss the rule protects against — and the loss does not actually occur here, because the durable record of the failure survives elsewhere: in the marker, in the attempt counter, and in the incident the poison already recorded. Nothing is forgotten; only the pending delivery is withdrawn.
sequenceDiagram
participant I as Instance
participant D as Durable state
participant C as Counterpart
I->>C: request, attempt n, idempotency key A
I->>I: the timeout boundary fires, or the delivery is terminally poisoned
I->>D: arm the backoff timer, set the marker, delete attempt n's outbox rows
Note over I,D: one transaction — nothing of the dead attempt stays pending
C->>I: a late response to attempt n
I-->>C: refused, SUTRA.DISPATCH.CHANNEL_CALL.RETRY_PENDING — the correlation row stays live
D->>I: a due timer on the marked node — the re-drive
I->>C: fresh request, attempt n+1, fresh idempotency key B
C-->>I: response, served by the same correlation, no re-registration
Withdrawing the dead attempt’s deliveries is what makes the fresh idempotency key safe: only one request for this call is ever live, so a counterpart doing its own deduplication sees a genuinely new one rather than a replay it can answer “already handled”.
The general shape is worth naming: an exception to a safety rule is defensible when you can point at the specific corruption the rule would cause, and show the information the rule was protecting is preserved by another mechanism.
The poison wake is a prompt, never a fact
When the outbox gives up on a request, the task waiting on it needs to hear about it. The dispatcher fires a best-effort in-process wake — a fresh serialized turn, like a timer fire.
But the wake is treated as a prompt to go and look, never as evidence. The engine acts only on durable evidence re-read under the instance claim: a poisoned outbox row for exactly this instance and this node. A wake that arrives spuriously finds nothing and does nothing.
And a wake that is lost — a crash, a shutdown, a dropped in-process message — is recovered by the timeout boundary, which the loader guarantees every channel call has. That guarantee is what lets the wake be best-effort at all: the fast path is a notification, the correctness path is the boundary the model was required to declare anyway.
This is a shape worth reusing. An in-process notification that races with a crash is fine as long as (a) the receiver re-derives the fact from durable state, and (b) something durable eventually detects the same condition without the notification.
The whole thing is claim-arbitrated
Every decision above — relay versus timer versus poison, marked versus unmarked, re-drive versus resolve — is made under the instance claim, from durable facts. The races between a late relay, a due timer, and a poison notification arbitrate through the claim exactly as cross-replica races do; the marker only tells a holder which situation it is in, never who gets to act. See Ownership and claims.
The determinism discipline is unchanged throughout: no lane-blocking sleep anywhere, attempt state on the snapshot, every re-drive decision from durable facts, and byte-identical snapshots for every process that never backs off a call.
Exhaustion
A spent budget — or a nonRetryableCodes hit — lands on the same fatal path a registered task’s
exhaustion does: a durable FAILED snapshot carrying the structured code. Identical for both task
kinds, deliberately, because “how many kinds of failure state does this engine have” should have
exactly one answer.
From there the instance is retained, inspectable, blocks its deployment’s retirement, and is repairable by migration — which is the loop the whole design is pointed at: fail loudly, keep everything, let an operator fix the model and bring it back.
stateDiagram-v2
Attempt: Attempt<br/>the task runs, or the request is emitted
Backoff: Backoff park<br/>timer armed, marker set,<br/>the dead attempt's outbox rows deleted
Failed: FAILED<br/>a durable snapshot with the code
[*] --> Attempt
Attempt --> [*]: success, or a<br/>business response
Attempt --> [*]: a BPMN error,<br/>routed to its boundary
Attempt --> Backoff: failed,<br/>budget left
Backoff --> Attempt: due timer<br/>on the marked node
Attempt --> Failed: nonRetryableCodes
Attempt --> Failed: budget spent
Failed --> [*]: retained, inspectable,<br/>repairable by migration
Only two things are a failed attempt of a channel call — a route-less <q:timeout> firing and a
terminally-poisoned delivery. An answer and a BPMN error both leave without touching the budget,
because each is an outcome the author already routed; retrying an unwelcome answer would
double-submit.
Next
- Retries, history, and schedules — the author-facing chapter.
- Migration internals — how a burned budget travels with its task.
- The pull surface — the other retry budget, and why they never overlap.
Migration internals
Instance migration is the operator’s guide. This chapter is the reasoning: a row-security problem with a silent failure mode, why compatibility is derived from resume behaviour rather than from BPMN taxonomy, how the re-arm set is worked out, and why batch independence is structural rather than promised.
The two-scope problem
Every other commit in the engine runs inside one deployment scope: open a transaction, set the scope, write, commit. Row-level security policies then confine every statement to that scope, which is how tenant and deployment isolation is enforced at the database rather than in application code.
Migration breaks the pattern by definition. It reads rows pinned to the source and writes rows pinned to the target.
The obvious implementation — a single UPDATE that sets the deployment id — cannot work under
an enforcing policy. The shipped policies are visibility predicates with no separate write
predicate, and the database then reuses the visibility predicate as the write check for an update.
So the statement fails at both possible scopes, and the two failures are not equally bad:
| Scope set to | What happens | Severity |
|---|---|---|
| Source | The old row is visible and passes, then the re-scoped new row is rejected by the implied write check. | An outright error. Loud, obvious, safe. |
| Target | The old row is not even visible, so the statement succeeds and matches nothing. | Silent. |
The second is the dangerous one, and it is worth dwelling on why. It does not raise. It does not warn. It returns success having moved zero rows — so an operator running a migration would see it succeed, and the instance would still be pinned to the broken model. Every downstream signal would agree that everything was fine.
That is the failure mode this design exists to make impossible, which is why both halves are pinned by a test running as a genuinely non-bypassing database role. The justification for the shape is executable rather than asserted — a claim about what row security does under an enforcing role is exactly the kind of claim that rots quietly.
The scope flip inside one commit envelope
What makes the operation possible is that the scope setting is transaction-local, not transaction-immutable. One transaction may re-scope itself between statements.
So the commit runs two phases inside one envelope:
- Scoped to the source — lock the instance row, re-assert the ownership claim and the non-terminal status, read every row belonging to the instance, delete them.
- Scoped to the target — insert the rewritten rows.
Atomicity is unchanged: a phase-2 failure rolls phase 1 back, and no session ever observes the instance under both pins or under neither. Isolation is not weakened either, and the distinction is precise — the transaction is never scoped to two deployments at once. It finishes with one, then scopes to the other. There is no window in which a statement could see both.
The re-assertion in phase 1 is deliberate. The claim was taken before validation; re-verifying it under the row lock, inside the commit, closes the gap between “claimed” and “moved”.
flowchart TD
CLM["claim the instance"] --> VAL["validate every locus<br/>all violations, not the first"]
subgraph TX["one transaction"]
direction TB
LK["scoped to the SOURCE:<br/>lock the instance row"] --> RA["re-assert the claim and<br/>non-terminal status"]
RA --> RD["read the instance's rows,<br/>then delete them"]
RD --> INS["re-scoped to the TARGET:<br/>insert the rewritten rows"]
end
VAL --> LK
INS --> OK["commit"]
RA -.->|"claim gone,<br/>or terminal"| NO["refused —<br/>nothing moved"]
INS -.->|"phase 2 fails"| RB["phase 1 rolls back:<br/>never both pins, never neither"]
The scope is re-set between statements rather than held for the transaction, which is what lets one commit read under the source pin and write under the target one — and the re-assertion sits inside that envelope precisely so a claim that died between validation and commit stops the move instead of racing it.
The snapshot moves as a byte-level patch
The snapshot rewrite is a key patch over the raw map — the same shape marking an instance failed or terminal uses, and for the same reasons, plus one specific to migration.
A decode-and-re-encode would need the tenant’s data-encryption key, would have to re-derive an encryption set the resume-time snapshot no longer carries in that shape, and would persist a previously-encrypted value in the clear the moment either went wrong. See Durable execution.
That carried-through ciphertext still decrypts under the new pin is not luck. The authenticated-data binding ties a ciphertext to its key, its instance and its variable name, and deliberately excludes the deployment id — a property the encryption design chose precisely so that “a version migration changes only the pin” would stay true. A design decision made in one subsystem paying off in another, years later, is what a well-chosen invariant looks like.
Rewritten: the deployment pin; the wait frontier, completed set and routed start (each entry mapped through the node mapping); the per-node retry attempt counters — the key is renamed, the counter untouched, because a burned budget follows its task; the audit sequence floor, bumped past the migration event.
Untouched: variables plain and encrypted, the key anchor and sensitive-name set, the status, the failure keys, and coverage cursors — those last are keyed by declared path id, not node id, so applying the node mapping to them would corrupt them. The distinction is invisible unless you look, which is why it is written down.
Compatibility is derived from what resume does
The locus model is the substance of validation, and its one real idea is this: the question is
never “is this a userTask”. It is “can a parked message wait resume here”.
Several distinct BPMN constructs answer yes to the second question, and the set is not the same as any single element type. So each locus is classified by the resume behaviour it will actually get, and the target node is checked against capability flags rather than an element enum.
Two classifications carry the weight:
A retry park is not merely a timer park. It is a timer row whose node has a durable attempt count and is absent from the completed set — which is exactly the condition the executor itself uses to route a due timer to re-run the task rather than to fire the timer. So the compatibility rule reuses the executor’s own test rather than reinventing one. Landing a retry park on a merely timer-capable node would silently re-run nothing.
A continue-reply park is read from the source graph. Whether a parked node carries a respond-and-continue reply is a property of the definition the instance is running under, not of the target. Which is why a source deployment whose plan is no longer registered is a hard refusal rather than a fallback: validating a continue-reply park against the timer rule would pass the wrong check and produce a confidently-wrong verdict. Refusing to answer beats answering wrong.
The capability index each plan projects is small and computed at activation, so validation never re-verifies and re-plans two sealed archives on an administrative request path. It covers draining deployments as well as active ones — a migration’s source is by definition a deployment that has been flipped away from. Sub-process bodies are flattened into their parent’s index, because durable state records a node id with no scope path.
Every violation, not the first
The validator reports all violations. Fixing a mapping should take one round trip, not one per node — a validator that stops at the first error turns a ten-node rename into ten deploy-test cycles.
The same instinct drives the refusals that could have been silent: a mapping entry naming a node the instance neither parks at nor has completed is refused, because a typo must never read as “identity mapping, then”.
The resume re-arm set: why the frontier alone is wrong
resume: true brings a FAILED instance back. Working out what to re-arm is the subtlest part of
the whole operation.
The failure commit did two things: key-patched the snapshot to FAILED with its failure code and
detail, and resolved every waiting row in one statement so no timer refires and no relay finds a
live wait. The frontier itself was left untouched.
Resume is the inverse of both halves. The snapshot half is straightforward — status back to
suspended, failure keys dropped, output byte-identical to the pre-failure snapshot because the
failure keys are emit-only-when-present, and like every other re-stamp it is a raw patch, so
reviving an instance never needs the tenant key and can never downgrade its at-rest protection. It
fails closed on any status but FAILED.
The rows are the hard half, and the frontier is not enough. A <q:timeout> synthesizes a
boundary with a derived id, and a timer boundary event has its own id — neither appears in the
frontier. Re-arming by frontier alone would silently drop the timeout the park was armed with,
producing an instance that resumes and then waits forever with no deadline.
The rows are recoverable exactly, though, and the derivation is a nice piece of reasoning about what the data already tells you:
They were resolved by one statement, so they share a single resolution timestamp — and it is the instance’s latest, because nothing touches a
FAILEDinstance’s rows afterwards (the poller’s stale and failed branches only update rows still waiting).
So the re-arm set is the frontier’s own rows, plus every row sharing the instance’s latest resolution timestamp. It is gated on a non-empty frontier: an instance that had nothing parked has nothing to restore, and an older satisfied wait is spent history rather than a park.
The parks are re-armed in the migration’s own transaction, so migrate-and-resume is one commit and a crash cannot leave a half-revived instance.
Afterwards the instance comes back through ordinary paths: the row is unowned (the claim died with the source row), the snapshot is suspended, and the parks are armed. A re-armed timer whose instant has passed is claimed by the ordinary poller on its next tick; a message park waits for its next correlated inbound. There is no new resume entry point and no privileged re-drive, which is precisely why the claim-guarded concurrency story still holds afterwards.
flowchart LR
F["the wait frontier's own rows"] --> SET
L["every row sharing the instance's<br/>latest resolution timestamp"] --> SET
T["a synthesized q:timeout boundary,<br/>a timer boundary event"] -.->|"neither appears<br/>in the frontier"| L
SET["the re-arm set —<br/>gated on a non-empty frontier"] --> ARM["re-armed in the migration's own transaction,<br/>status back to suspended, failure keys dropped"]
ARM --> ORD["ordinary paths — the poller claims a due timer,<br/>a message park waits for its next correlated inbound"]
The frontier alone would silently drop the deadline the park was armed with; the second input
recovers it exactly, because the failure commit resolved every waiting row in one statement and
nothing touches a FAILED instance’s rows afterwards.
A defect this closed
The first version derived loci from waiting rows that were still marked waiting. For a FAILED
instance there are none — its failure commit resolved them all. So every frontier entry fell through
to the “no row behind it” branch and was validated as a message wait. A dead instance parked on
a timer was therefore checked against the wrong rule, and warned about a missing row that was
sitting right there.
Since a FAILED instance is the operation’s prime use case, that was not a corner case. Reading
the park set as described above fixed it — and note the property that makes the fix trustworthy:
what validation checks is now exactly the set resume re-arms. One derivation, two consumers, no
possibility of drift.
Batch independence is structural, not promised
The batch endpoint is not a second implementation. A request body parses once into a migration plan; a single internal operation applies that plan to exactly one instance and returns a verdict rather than an HTTP response; and both endpoints are thin wrappers over it.
So “each instance validates and migrates independently” is not a promise the batch endpoint makes and must be trusted to keep. It is structural: the batch is a loop over the single-instance operation, and the attempt is the loop’s return value. There is no shared mutable state for one instance’s outcome to leak into another’s, because there is nowhere to put it.
Everything else follows:
- One claim, one transaction, one report entry per instance. No batch-wide transaction — which is the point, and which makes the crash story trivial: a mid-batch crash leaves every instance either fully migrated or completely untouched, because that is already true of each instance’s own commit and there is nothing larger to be half-applied.
- The outcome is a single enum, not a combination of booleans, so no caller has to infer a state from flags that might contradict each other.
- The status describes the batch, not its instances. A run where every instance refused still
answers
200: the call was accepted, executed to completion, and reported in full. Scripts key on the totals and the per-instance outcomes.
Contention is reported, never retried
An instance whose claim is held bounces, and the batch moves on. Retrying inside the call is refused on two grounds, and both are about honesty rather than convenience:
- It makes an administrative request’s runtime unbounded — the claim is held by another replica’s work, on that work’s timescale.
- It turns the report into a claim about a moment that has already passed. A report that says “everything moved” after internally retrying for thirty seconds describes a world that no longer exists by the time it is read.
The caller re-runs the same request instead. The selector is deterministic and whatever moved is no longer under the source pin, so a re-run converges — which is a much better contract than “we tried hard”.
Selection is ordered by instance id, then limited. Ordering by a timestamp that moves under a live population is exactly how a caller silently skips work, and a paging bug that skips work without saying so is the worst kind.
Two request-level contradictions are caught from the request alone rather than N times over in the per-instance reports — resuming a selection of suspended instances (it selects exactly what resume refuses), and re-homing a mixed population into one process (one mapping could only be right for one of them).
What deliberately does not move
Pending outbound rows stay under the source pin. An emission was minted by the source deployment’s channel bindings and is dispatched against them — and the dispatcher covers draining deployments, so it drains where it was made. Re-targeting a message at bindings that never produced it would be worse than leaving it: the destination, headers, and codec were all decided by a configuration the target may not even have.
The audit journal moves scope but keeps its node ids verbatim. A trail names where something happened, under the graph that was live at the time; rewriting the ids would falsify the record. The migration event itself records source and target, so the provenance is recoverable without corrupting the history.
Both are reported — the outbound rows as a warning — because the operator should know why, not just what.
Next
- Instance migration — the operator’s guide.
- Durable execution — the patchers and the encryption binding this relies on.
- Ownership and claims — why the migration claim needs its own owner identity.
The pull surface: design
External tasks is the worker-facing guide. This chapter is why the pull surface is a hundred lines of new machinery rather than a subsystem — which is almost entirely down to one decision.
Ownership transfer is the design
A channel declaring transport: pull gets a sink like any other transport. The sink claims its URI
scheme, exactly as an HTTP or broker sink claims theirs. What it does with a delivery is different:
instead of dialing anything, it parks the delivery as a task row — and then answers
Delivered.
That answer is the whole design. Ownership transfers from the outbox row to the task row, and the relay deletes the outbox row exactly as it would for a genuinely delivered push.
Everything else falls out of it for free:
- The relay’s retry and poison machinery applies to a failed park automatically. If parking fails — the database is briefly unavailable, say — the sink answers a retryable failure, and the outbox’s existing backoff curve retries the park with no new code. If the operator configured an attempt ceiling, an unparkable delivery poisons exactly like an undeliverable one, records its incident, and stops pinning a draining deployment. None of that had to be re-implemented for pull; it was already there, one layer up.
- The hot claim predicate is untouched. The outbox worker’s
SKIP LOCKEDbatch claim — the query that runs constantly on every replica — does not learn about pull at all. It sees a delivery, hands it to a sink, and gets an answer. Adding a delivery mode without touching the hottest query in the system is the payoff for putting the new behaviour behind an existing polymorphic seam rather than beside it. - Parking is off the execution path. The sink runs on the outbox tick loop, not on an execution lane. It touches the engine actor not at all: a park is a store write plus a wake-up.
The alternative shape — a first-class “pull delivery” concept understood by the dispatcher, the outbox worker, and the retirement gate — would have been a genuinely new subsystem with its own retry semantics, its own poison horizon, and its own failure modes to reason about. Modelling pull as a sink that happens to hand ownership to a different table keeps one delivery pipeline.
The completion is not a new resume path
A worker’s result is rebuilt into an ordinary inbound message and re-enters the engine through the same seam every transport delivers through. Not a nested call, not a privileged entry point: one more serialized turn, indistinguishable from a pushed reply.
That is what keeps three things unchanged rather than duplicated:
- Correlation. The parked headers ride through, so the author’s
<q:alias>resolves the waiting instance the way it always has. - Validation. The completion goes through the same intake pipeline — a worker cannot smuggle a structurally invalid payload past the checks a pushed message faces.
- Deduplication. The completion carries the originating delivery’s key as an explicit idempotency key, so inbox dedup covers it with no worker-side cooperation.
A second resume entry point would have needed all three re-implemented, and would have been the place where they drifted.
Lock expiry inside the claim predicate: no sweeper, by construction
A worker that dies mid-task must not hold its task forever. The usual answer is a reaper — a background job that scans for expired locks and releases them — which is a role to schedule, a lease to gate, a cadence to tune, and a lag between expiry and availability.
There is none here. Lock expiry is part of the predicate that decides what a fetch may claim: a task is fetchable when it is not terminal, its backoff has elapsed, and it is either unlocked or holding an expired lock. An expired lock is therefore not a state anyone has to clean up. It is simply not an obstacle to the next fetch.
stateDiagram-v2
[*] --> Fetchable: the sink parks<br/>the delivery
Fetchable --> Locked: a fetch claims it
Locked --> Fetchable: the lock expires —<br/>the predicate stops excluding it
Locked --> Fetchable: failure, budget left,<br/>after the backoff
Locked --> Terminal: failure,<br/>budget spent
Locked --> [*]: completion dispatched,<br/>then the row deleted
No edge out of Locked is driven by a background job: each one is either a worker’s own call or the
claim predicate declining to exclude the row. That is what “no sweeper” actually buys.
The consequences are worth naming, because “no component” is easy to undervalue:
- No lag. A task is available the instant its lock expires, not on the next sweep tick.
- Nothing to gate. No lease role, no leader, no cadence key, no failure mode where the sweeper is down and locks pile up.
- Nothing to get wrong under concurrency. The claim and the expiry check are the same statement, so there is no window between “the sweeper released it” and “someone claimed it”.
The cost of a dead worker is therefore exactly one lock duration — which is a knob the operator already sets, rather than a second knob about how often to look.
The completion path extends the same idea: it re-takes the lock across the dispatch, so the lock cannot expire mid-flight and let a second worker pick the task up while the first one’s result is still inside the engine.
Dispatch, then delete — and why the inverse loses work
The completion order is: verify and hold the lock, dispatch the result through the inbound path, then delete the task row.
That makes the surface at-least-once. A crash between the dispatch and the delete re-offers the task, and a duplicate completion is absorbed by inbox deduplication under the delivery’s idempotency key.
sequenceDiagram
participant W as Worker
participant T as Task row
participant E as Engine intake
W->>T: complete, guarded by the lock it holds
T->>T: re-take the lock across the dispatch
T->>E: dispatch the result through the ordinary inbound path
Note over E: correlation, validation and inbox dedup,<br/>under the delivery's own idempotency key
E-->>T: accepted
Note over W,T: a crash before the delete re-offers the task —<br/>a duplicate, never a loss
T->>T: delete the task row
The delete is last on purpose. The crash window between the dispatch and the delete costs a duplicate that already has a dedup key waiting for it; inverting the two would make that same window cost the work itself.
The inverse order — delete, then dispatch — would make it at-most-once, and the same crash would lose the work outright: the task is gone, the instance is still parked, and nothing anywhere records that the worker ever answered. The instance waits forever on a result that was produced and thrown away.
Between a duplicate the system already knows how to absorb and a silent loss it cannot detect, the duplicate is not a close call. This is the standard shape of the choice, and it is worth stating in those terms: at-least-once plus deduplication beats at-most-once plus hope, whenever deduplication is available — and here it is available by construction, because the idempotency key was minted by the delivery that created the task.
A worker that completes with no result re-delivers the original request payload. That is the fire-and-forget shape: the work happened outside, and the flow waits only on the fact of it.
If the engine refuses the completion on the inbound path, the row stays, still locked until its grace window elapses, and becomes fetchable again. The refusal carries the engine’s own code as an attribute — not just prose — because that code is the only thing that tells a worker whether re-fetching later can ever help (a transiently unavailable engine) or never will (a validation reject that will fail identically forever).
Zero rows affected is never success
Every worker-facing mutation is ownership-guarded: it matches only if this worker still holds this task’s lock. Which means the interesting case is a statement that matches nothing — and the critical rule is that this is never reported as success.
A 200 to a worker whose lock lapsed is a duplicate-execution bug. The worker believes its result
landed; it did not; the task is fetchable again and someone else will do the work too.
So a zero-row result triggers a second, unguarded read to find out which situation it is, and the answer is one of four verdicts:
| Verdict | Situation | What the worker should do |
|---|---|---|
LOCK_LOST | The lock expired or was released; the task is fetchable again | Stop. Do not re-complete — fetch fresh work. |
LOCK_HELD | Another worker holds it | Stop. It is not yours. |
TERMINAL | The budget is spent; it can never be completed or failed again | Stop permanently. |
NOT_FOUND | No such task on any live deployment | Stop; the task is gone. |
Splitting them apart matters because they call for different worker behaviour, and a single generic “conflict” would leave a worker author guessing — usually by retrying, which is wrong for three of the four.
The extra read is deliberate and cheap: it happens only on the failure path, and it reads lock state only. The payload never rides a failure path — there is no reason for a task’s body to travel through an error response.
Separating “no such task” from “you do not hold it” needs one more distinction: the guarded statement is attempted across the live deployment set, and if it matches nowhere, an unguarded existence probe distinguishes a lock problem from a genuinely absent task. A worker names topics, never deployment ids — the same posture the outbox worker and the instance listing take — so the surface does the walking.
Bounds are rejections, never clamps
Every bound a worker can send — lock duration, long-poll wait, batch size — is a ceiling the
operator sets, and an over-ceiling request is a 400.
Silently clamping would be the friendlier-looking choice and it is the wrong one, for a reason specific to locks: a worker that believes it holds a longer lock than it does is a duplicate-execution bug. It paces its work against a deadline that already passed, the task goes fetchable, another worker takes it, and the first one’s completion is refused after the work was already done twice. A clamp does not prevent that; it creates the belief that causes it.
The same reasoning applies to the others, less dramatically: a clamped long poll makes a worker’s own timeout arithmetic wrong, and a clamped batch size makes its throughput planning wrong. In each case the reject is information and the clamp is a lie the system tells quietly.
Omitting a bound is different from asking for too much: an omitted value takes the default, not the ceiling. The default is a considered value; the ceiling is a limit. Conflating them would hand every silent request the most expensive setting available.
The engine also refuses to boot if the default lock duration exceeds its own ceiling, or is zero — a configuration that is internally contradictory should fail at startup, where somebody is watching, rather than at the first fetch.
The long poll wakes, and cannot miss
A fetch that finds nothing waits, bounded, for a task to be parked on one of its channels. The wake-up is a broadcast keyed by channel, so a fetch filters to the topics it asked for instead of re-querying on every unrelated park.
Two properties make it safe:
- Subscribe before the first query. A task parked in the window between the query and the wait still wakes the waiter. The classic lost-wakeup ordering bug is closed by ordering, not by a timeout that hides it.
- Missing a wake-up is harmless. A woken fetch always re-runs the claim query; the wake-up carries no payload beyond “look again”. So a lagging subscriber is not an error — it costs at most one extra round of a loop it was already in. The ring buffer is deliberately small, because it is a doorbell rather than a queue.
And the wait is always bounded by the operator’s ceiling, so a fetch answers with an empty list rather than hanging. A long poll that can hang is a client-side resource leak with extra steps.
Two budgets that never overlap
The worker-failure budget and a <q:retry> policy sit at different layers and never interact:
| Governs | Counted by | On exhaustion | |
|---|---|---|---|
| Worker budget | A worker failing at its own work | The task row | The task turns terminal — never fetched again, retained with its last error |
<q:retry> | The task-level outcome in the model | The instance snapshot | The instance fails durably |
A worker retrying its own work never touches the instance; the instance stays parked on its wait,
exactly as it would while a worker was simply slow. Conversely a <q:retry> timeout on a
pull-backed call still fires if no worker ever completes — the pull surface changed the last hop,
not the model’s semantics.
A terminal task no longer counts toward its deployment’s retirement quiescence gate, for the same reason a poisoned delivery doesn’t: “we gave up” must be a durable, visible state, and it must not hold a deployment open forever. That symmetry is intentional — the terminal task is the pull-side twin of the outbox’s poison horizon. See Retry machinery.
Posture
These are operate-surface routes, not administrative ones, and the classification is a judgement worth stating: a completion is an ordinary delivery, not a privileged control operation. It goes through the same intake as a pushed reply, and it can do nothing an inbound message on that channel could not do.
So it carries the operate surface’s cluster-internal posture rather than the administrative surface’s gate. A deployment needing authenticated workers puts them behind the same ingress policy the rest of the operate surface already needs.
Next
- External tasks — the worker-facing guide.
- Retry machinery — the other budget, and the outbox poison horizon this mirrors.
- Ownership and claims — why a task lock is not an instance claim.
Configuration reference
The engine reads exactly one family of configuration keys — canonical sutra.* properties, each
with a canonical SUTRA_* environment-variable mirror. There is no separate framework prefix and
no legacy alias layer to reason about.
Sources and precedence
canonical env > config file > built-in default
The config file path comes from SUTRA_CONFIG (default sutra.properties, read from the
engine’s working directory, and only if present); file values may themselves use ${ENV} /
env:NAME indirection. A key that has both a sutra.* file entry and its SUTRA_* environment
variable set always resolves from the environment.
rust/crates/sutra-engine/src/config.rs is the single source of truth for the exact key list —
this page is a map of what’s there and how the pieces fit together, not a duplicate of it; treat
that file (and otel.rs for the telemetry keys) as authoritative if this page and the running
engine ever disagree.
Deployment source
sutra.* key | env | meaning |
|---|---|---|
sutra.deployment.source | SUTRA_DEPLOYMENT_SOURCE | dir (default) — watch a folder of sealed archives — or db — the database-backed store, activated only via POST /admin/deployments. See Deployment model. |
sutra.deployments.dir | SUTRA_DEPLOYMENTS_DIR | Required for the dir source: the directory of .sutra archives the engine watches. |
sutra.deployments.poll-interval | SUTRA_DEPLOYMENTS_POLL_INTERVAL | How often the dir source rescans (default a few seconds). |
sutra.http.port | SUTRA_HTTP_PORT | Listen port; 0 binds an OS-assigned port (always use 0 locally — see Quickstart). |
The engine’s own datasource
sutra.* key | env |
|---|---|
sutra.datasource.url | SUTRA_DATASOURCE_URL |
sutra.datasource.username | SUTRA_DATASOURCE_USERNAME |
sutra.datasource.password | SUTRA_DATASOURCE_PASSWORD |
sutra.datasource.max-connections | SUTRA_DATASOURCE_MAX_CONNECTIONS |
sutra.datasource.dispatch-max-connections | SUTRA_DATASOURCE_DISPATCH_MAX_CONNECTIONS |
Since 0.6.5 the engine opens two pools over this database. The main pool
(max-connections, default 8, the size it always had) serves the execution lanes: intake,
steps and resumes. Size it at least to sutra.engine.shards, because each lane holds a
connection while it commits a step. The dispatch pool (dispatch-max-connections, default
4 since 0.8.3, 2 before) serves the outbox dispatcher and the timer poller, so a burst of
intake can never leave delivery waiting for a connection. The claimer and the delete batcher hold
one connection each; the others serve the defers and requeues of failed sends in a burst.
This is the engine-internal database (instances, outbox, lease, audit, inbox) — never a
package’s own business data store, which owns its connection independently in datastores.yaml
(see Data stores). Note the CLI’s sutra migrate reads a
different-named set (SUTRA_DB_URL / SUTRA_DB_USERNAME / SUTRA_DB_PASSWORD /
SUTRA_DB_SCHEMA) even when pointed at the same database — see the
CLI reference.
Outbox and acknowledgement
sutra.* key | env |
|---|---|
sutra.outbox.tick-interval | SUTRA_OUTBOX_TICK_INTERVAL |
sutra.outbox.workers | SUTRA_OUTBOX_WORKERS |
sutra.outbox.destination-concurrency | SUTRA_OUTBOX_DESTINATION_CONCURRENCY |
sutra.outbox.max-buffered | SUTRA_OUTBOX_MAX_BUFFERED |
sutra.outbox.lag-warn | SUTRA_OUTBOX_LAG_WARN |
sutra.outbox.batch-size | SUTRA_OUTBOX_BATCH_SIZE |
sutra.outbox.retry.base-delay / .max-delay / .jitter | SUTRA_OUTBOX_RETRY_BASE_DELAY / _MAX_DELAY / _JITTER |
sutra.outbox.retry.max-attempts | SUTRA_OUTBOX_RETRY_MAX_ATTEMPTS |
sutra.ack.deferred.capacity | SUTRA_ACK_DEFERRED_CAPACITY |
sutra.ack.deferred.timeout | SUTRA_ACK_DEFERRED_TIMEOUT |
sutra.ack.deferred.sweep-interval | SUTRA_ACK_DEFERRED_SWEEP_INTERVAL |
The outbox delivers many rows at once, under three bounds (since 0.8.3):
sutra.outbox.workers(default32, 1..1024) is the most sends in flight at once, across every destination.1restores one delivery at a time. Before 0.8.3 it was a count of workers rows were spread over by instance, defaulting to the lane count and at least 8.sutra.outbox.destination-concurrency(default32) is the most sends in flight to one destination authority: scheme, host and port, the path ignored. It is clamped toworkerswith a WARN. A row waiting on a busy destination holds no global slot, so a slow destination does not delay the others.sutra.outbox.max-buffered(default4096, fromworkersto 100 000) is the most rows held in memory, in flight or queued. Past it the claimer waits.
One instance’s deliveries still go out one after another, in the order they were enqueued;
different instances never share a queue. sutra.outbox.lag-warn (default PT10S, PT0S = off)
logs one WARN a minute while delivered rows exceed it, enqueue to delivery. A step that enqueues wakes the dispatcher at once, and a local:// delivery is handed
to it in-process as soon as the step’s commit is durable, so the tick (tick-interval, default
PT5S) only paces retries that fall due and rows another replica enqueued.
sutra.outbox.batch-size (default 50) is how many due rows one claim takes per deployment.
Delivered rows are deleted in batches. See Tuning for throughput.
Outbound deliveries retry with backoff forever by default. sutra.outbox.retry.base-delay defaults to PT1S:
retry n waits about base-delay × 2ⁿ, so the first retry comes about 2 s after the failure. Since 0.6.20 each
failed attempt that will be retried logs a WARN with the destination, the cause’s code and message, the attempt number
and next_attempt_at. sutra.outbox.retry.max-attempts
is the opt-in ceiling: a delivery that exhausts it is marked terminally poisoned — retained
with its last error, never retried again, one incident recorded for a required delivery,
and no longer counted by the draining-deployment retirement gate. “We gave up” is a durable,
visible state, never a silent disappearance.
Full explanation of the deferred-ack registry the sutra.ack.deferred.* keys tune:
Acknowledgement modes. Configuring the attempt ceiling is also what makes a
poisoned delivery reachable as a <q:retry> failure — see
Retries, history, and schedules.
Instance lifecycle — ownership and retention
sutra.* key | env |
|---|---|
sutra.instance.sweep-interval | SUTRA_INSTANCE_SWEEP_INTERVAL |
sutra.instance.claim-timeout | SUTRA_INSTANCE_CLAIM_TIMEOUT |
sutra.instance.retention | SUTRA_INSTANCE_RETENTION |
sutra.instance.retention-sweep-interval | SUTRA_INSTANCE_RETENTION_SWEEP_INTERVAL |
Every resume claims the instance first, so two replicas can never advance one instance
concurrently; the stuck-instance sweep (sweep-interval, default PT1M) clears claims whose
owner has been silent longer than claim-timeout (default PT5M).
A finished instance is history, not a 404: terminal (completed/terminated) snapshots are
retained for sutra.instance.retention (ISO-8601 duration, default P7D) and served by
GET /sutra/instances/{id} and GET /admin/instances/{id}/history; a lease-gated sweeper
purges rows past the window on the retention-sweep-interval cadence (default PT1H).
PT0S restores delete-at-completion. Failed instances are always retained regardless of the
window — they need an operator before their deployment may retire; the operator action is
instance migration. See
Retries, history, and schedules for what the
retention window makes queryable.
With <q:audit capture="media">, retention also deletes the recordings the purged instances’
audit events reference, from whichever declared object store holds them (below).
Capacity pools
sutra.* key | env |
|---|---|
sutra.capacity-pools | SUTRA_CAPACITY_POOLS |
sutra.capacity-pools.<name>.slots | SUTRA_CAPACITY_POOLS_<NAME>_SLOTS |
sutra.capacity-pools.<name>.reserve.inbound | SUTRA_CAPACITY_POOLS_<NAME>_RESERVE_INBOUND |
sutra.capacity-pools.<name>.reserve.outbound | SUTRA_CAPACITY_POOLS_<NAME>_RESERVE_OUTBOUND |
sutra.capacity-pools.<name>.tenants | SUTRA_CAPACITY_POOLS_<NAME>_TENANTS |
Named, cluster-wide slot counts that channels of any deployment draw from with pool: <name>.
Full explanation, defaults and a walkthrough: Capacity pools and sessions.
Object stores
sutra.* key | env |
|---|---|
sutra.object-stores | SUTRA_OBJECT_STORES |
sutra.object-stores.<name>.uri | SUTRA_OBJECT_STORES_<NAME>_URI |
sutra.object-stores.<name>.endpoint | SUTRA_OBJECT_STORES_<NAME>_ENDPOINT |
sutra.object-stores.<name>.region | SUTRA_OBJECT_STORES_<NAME>_REGION |
sutra.object-stores.<name>.access-key / .secret-key | SUTRA_OBJECT_STORES_<NAME>_ACCESS_KEY / _SECRET_KEY |
sutra.object-stores.<name>.encryption | SUTRA_OBJECT_STORES_<NAME>_ENCRYPTION |
sutra.object-stores.<name>.kek | SUTRA_OBJECT_STORES_<NAME>_KEK |
sutra.object-stores.<name>.key-id | SUTRA_OBJECT_STORES_<NAME>_KEY_ID |
Named S3-compatible buckets the engine reads recorded media from (an llm channel’s
media-store) and deletes it in (erasure and retention). Credentials and the key-encryption
secret are references, resolved at boot; a store that cannot be built refuses startup. Full
explanation and a walkthrough: Recordings and media references.
External tasks (the pull worker surface)
sutra.* key | env |
|---|---|
sutra.external-task.default-lock-duration | SUTRA_EXTERNAL_TASK_DEFAULT_LOCK_DURATION |
sutra.external-task.max-lock-duration | SUTRA_EXTERNAL_TASK_MAX_LOCK_DURATION |
sutra.external-task.max-async-response-timeout | SUTRA_EXTERNAL_TASK_MAX_ASYNC_RESPONSE_TIMEOUT |
sutra.external-task.max-tasks | SUTRA_EXTERNAL_TASK_MAX_TASKS |
sutra.external-task.retries | SUTRA_EXTERNAL_TASK_RETRIES |
sutra.external-task.retry-timeout | SUTRA_EXTERNAL_TASK_RETRY_TIMEOUT |
A channel declaring transport: pull parks its deliveries as fetchable tasks instead of
dialing an endpoint; workers drive them over POST /sutra/external-tasks/fetch-and-lock
(bounded long poll) and .../{id}/complete / .../{id}/failure. The defaults: a PT30S
lock when the fetch names none (ceiling PT1H — a longer request is rejected, never
clamped), a PT30S long-poll ceiling, 100 tasks per fetch, a worker-failure budget of 3
with PT10S between attempts. A spent budget turns the task terminal (failed) — retained,
never fetchable again. The engine boots fail-closed if default-lock-duration exceeds its
own ceiling or is zero. Full treatment:
External tasks.
Execution lanes
sutra.* key | env | default |
|---|---|---|
sutra.engine.shards | SUTRA_ENGINE_SHARDS | 1 |
sutra.engine.shard-queue-capacity | SUTRA_ENGINE_SHARD_QUEUE_CAPACITY | unbounded |
sutra.engine.shards is the number of identical actor lanes the engine executes on inside one
replica. At N > 1 all work for one instance — routed by a stable hash of its id — still runs on
one lane, in arrival order, one request at a time: per-instance serialization is the
contract, and it holds at every N. Values above 1 are accepted; the default stays 1 because
turning it up has one advertised consequence.
Say it out loud before you raise it. Incidental cross-instance serialization disappears at
N > 1. Two concurrent deliveries to two different instances of the same flow never interleave
under a single lane — as a side effect of there being one lane, never as a promise. At N > 1
they genuinely run in parallel. A deployment silently leaning on that side effect will observe
new interleavings. Every supported concurrency mechanism is unaffected: per-channel singleton
/ serial consumption, per-channel and per-tenant admission caps, and optimistic
expect="unchanged" / pessimistic forUpdate data-store writes.
shard-queue-capacity bounds each lane’s mailbox (unset = unbounded; zero is rejected — “unset”
is how you say unbounded). A bounded send awaits on the caller’s task, so backpressure
propagates outward to the transport that offered the work — an in-flight HTTP request, a broker
prefetch window, a poller tick — and never sideways into another lane.
Meters ship with the feature, each carrying the lane index as a dimension:
sutra.engine.shard.queue-depth (per-lane backlog and skew), …dispatches / …parks /
…resumes, …handoffs (cross-lane relay hops — expected, and rising with lane count by
construction), and …claim-bounces split relay / timer. That last one is the mis-route
alarm: on a healthy rollout it reads near zero outside genuine cross-replica contention.
The live lane count is readable without reading config: GET /sutra/health/ready reports it
under the loader check’s data.shards, read off the running router rather than echoed back from
configuration.
Full picture: Execution lanes.
Broker connectivity events
| Key | Default | Meaning |
|---|---|---|
sutra.engine.connectivity-events.channel (env SUTRA_CONNECTIVITY_EVENTS_CHANNEL) | unset | (0.8.0) A channel name. When set, every broker outage a transport sees on one of a deployment’s channels is delivered to that deployment as an inbound JSON message on the channel of this name, if the deployment declares one. Unset: nothing is collected. |
Declare the channel in the package as an in-process intake and start a process on it:
channels:
- name: connectivity-in
transport: local
codec: json
Each outage produces two messages, one when it starts and one when it ends:
{"code":"SUTRA.INBOUND.AMQP.OUTAGE_ENDED","channel":"orders-in","direction":"inbound","state":"up",
"startedAt":"2026-11-19T10:00:00.120Z","endedAt":"2026-11-19T10:01:00.420Z","ms":60300,
"detail":"amqp broker mq.example:5672"}
stateisdownwhen the outage starts (noendedAtorms) andupwhen it ends.codeis the transport’s own log code for the change, so the message and the engine log line match.direction: inboundis an intake losing its connection.direction: outboundis a send channel whose send could not reach its broker; each send channel that sent during the outage reports its own window, ending on its first send that reaches the broker again. A send channel that sent nothing during the outage reports nothing.- A deployment only receives events for its own channels. One that declares no channel of the configured name receives nothing.
- Delivery is best effort: up to 1024 events wait per deployment; past that the oldest is
dropped and
SUTRA.CONNECTIVITY.EVENTS_DROPPEDis logged. An event the deployment refuses is logged asSUTRA.CONNECTIVITY.EVENT_UNDELIVEREDand never retried. - The AMQP 1.0 transport reports these events. Transports without a long-lived broker connection (HTTP, for example) report none.
Broker outages as messages runs it against a broker that is stopped and started.
Limits
sutra.* key | env |
|---|---|
sutra.codec.max-payload-bytes | SUTRA_CODEC_MAX_PAYLOAD_BYTES |
Full explanation: Limits and quotas.
Admin API auth
sutra.* key | env |
|---|---|
sutra.admin.auth.scheme | SUTRA_ADMIN_AUTH_SCHEME (apikey | bearer) |
sutra.admin.auth.key-ref | SUTRA_ADMIN_AUTH_KEY_REF |
sutra.admin.auth.header | SUTRA_ADMIN_AUTH_HEADER (default X-API-Key) |
sutra.admin.oidc.issuer / .audience / .jwks / .role-claim / .required-role | SUTRA_ADMIN_OIDC_* |
sutra.admin.oidc.dev-disabled | SUTRA_ADMIN_OIDC_DEV_DISABLED |
The /admin/* surface (deployments, instance inspection, subject erasure) is gated fail-closed:
unconfigured returns 503, a missing/invalid credential returns 401, and a valid token missing
the required role/claim returns 403 — never silently open. The auth-key scheme is the same
static-secret model channels use for inbound HTTP auth, and it takes precedence when both are set.
Gating is disabled only via the explicit sutra.admin.oidc.dev-disabled=true escape hatch.
From 0.6.18 the gate also covers GET /sutra/instances, its ?alias= search and GET /sutra/instances/{id}
when admin auth is configured. An engine with no admin configuration at all still answers them without a key,
so keep such an engine where only operators can reach it. See Upgrading to
0.6.18.
Audit sinks
sutra.* key | env |
|---|---|
sutra.audit.jsonl.path | SUTRA_AUDIT_JSONL |
sutra.audit.otel.endpoint | SUTRA_AUDIT_OTEL_ENDPOINT |
sutra.audit.otel.service-name | SUTRA_AUDIT_OTEL_SERVICE_NAME (default sutra-audit, 0.6.12) |
sutra.audit.sql | SUTRA_AUDIT_SQL |
The SQL sink writes the audit journal behind execution and best-effort: an event is queued and written by the audit writer, not inside the transaction of the step that produced it, so a crash can lose events not yet written. Since 0.8.1 the writer takes whatever queued while its previous write ran, up to 256 events, and writes it as one multi-row insert per deployment in one transaction, where it used to commit every event on its own. On a durable PostgreSQL that is what lets the journal keep up with a busy engine. The journal is the same: each instance’s events in order, deduplicated, and a batch that fails is written again event by event. Nothing to configure.
See Logging and audit.
Telemetry (OTel)
sutra.* key | canonical env | standard OTEL_* env also accepted |
|---|---|---|
sutra.telemetry.otlp.endpoint | SUTRA_TELEMETRY_OTLP_ENDPOINT | OTEL_EXPORTER_OTLP_ENDPOINT |
sutra.telemetry.service-name | SUTRA_TELEMETRY_SERVICE_NAME | OTEL_SERVICE_NAME |
sutra.telemetry.enabled | SUTRA_TELEMETRY_ENABLED | OTEL_SDK_DISABLED (inverted) |
sutra.telemetry.metrics.export-interval | SUTRA_TELEMETRY_METRICS_EXPORT_INTERVAL | OTEL_METRIC_EXPORT_INTERVAL |
sutra.telemetry.metrics.temporality-preference | SUTRA_TELEMETRY_METRICS_TEMPORALITY_PREFERENCE | OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE |
sutra.telemetry.metric-labels | SUTRA_TELEMETRY_METRIC_LABELS | — (default tenant,module,version) |
sutra.telemetry.resource-attributes | SUTRA_TELEMETRY_RESOURCE_ATTRIBUTES | OTEL_RESOURCE_ATTRIBUTES (both apply; the sutra.* value wins per key; 0.6.12) |
No OTLP endpoint configured means no exporters at all, and this is the default — telemetry export is opt-in, never opt-out, and Sutra itself collects nothing regardless: see No telemetry, no phone-home for the exact guarantee, and Observability for what each signal actually carries once you do turn export on.
Secrets — never literal values
Every credential-shaped field across channels.yaml, datastores.yaml, and the admin-auth keys
above is a reference, resolved at startup or channel-activation time — env:NAME,
secret:KEY (a file under the mounted secrets directory, default /etc/sutra/secrets /
SUTRA_SECRETS_DIR), ${NAME} / ${NAME:default} placeholders, or a vendor scheme
(vault:…, aws-secrets:…, azure-kv:…, gcp-secret:…) resolved by whichever
sutra-envref-<vendor> crate the binary was built with. A literal secret value in a resource
file is rejected at package-validation time, not just discouraged by convention. See
Domain neutrality and the SPI model
for how a vendor secret backend plugs in.
Next
Acknowledgement modes
An inbound channel declares an ack-mode that decides when the engine acknowledges receipt
relative to processing. How each transport actually realizes that intent differs — a broker gets a
native ack/nack, HTTP gets a status code — but the two values mean the same thing everywhere.
The two modes
on-persist— acknowledge the moment the engine has durably captured the inbound, before the BPMN process runs. On a broker: ack immediately, releasing the delivery slot early (lower broker-side latency; a redelivery after a mid-process crash is caught by inbox dedup). On HTTP: reply202 Acceptedwith an acceptance receipt (see Acceptance receipts) and process asynchronously — the fire-and-forget intake, whose eventual reply (if any) rides an outbound channel instead of the original connection.on-complete— acknowledge only once the instance reaches a terminal state (INSTANCE_COMPLETEDorINSTANCE_FAILED). On a broker: the ack is deferred — registered against the engine’sDeferredAckRegistryand held until the instance finishes. On HTTP: hold the connection open until completion and return the reply body (the body of the flow’s<q:reply>, or an empty body if it declares none; the flow’s variables never answer the caller) — the classic synchronous request/reply.
sequenceDiagram
participant S as Broker / HTTP caller
participant E as Engine intake
participant P as Process instance
S->>E: deliver
E->>E: durably capture the inbound
Note over E: on-persist acknowledges here
E-->>S: broker ack, or HTTP 202 Accepted with the receipt
E->>P: start
P-->>S: eventual reply, if any, on an outbound channel
sequenceDiagram
participant S as Broker / HTTP caller
participant E as Engine intake
participant P as Process instance
S->>E: deliver
E->>E: durably capture the inbound
E->>P: start
Note over S,E: broker — ack deferred in the registry<br/>HTTP — connection held open
P-->>E: INSTANCE_COMPLETED or INSTANCE_FAILED
Note over E: on-complete acknowledges here
E-->>S: broker ack or nack, or the HTTP reply body
The two are identical up to the durable capture; the mode moves only where the ack arrow sits. That one move is the whole trade — release the delivery slot early and let inbox dedup absorb a redelivery, or hold it so the broker’s own redelivery is your crash-recovery path.
The default differs by transport, because the natural mode differs: broker channels default
to on-persist (release the delivery slot early); HTTP channels default to on-complete
(synchronous request/reply). An HTTP channel opts into asynchronous intake by declaring
ack-mode: on-persist explicitly.
channels:
- name: orders-inbound
transport: http
bind: "POST /channels/orders-inbound"
ack-mode: on-persist # HTTP: 202 Accepted as soon as intake is durable
An asynchronous channel can still answer with a document
on-persist decides when the caller is answered — now, not when the work finishes. It does not
decide what with. A process that renders a reply before it parks, with
<q:reply continue="true">, sends that reply as the body of the
202; a process that renders nothing gets the receipt as the body of the 202.
That combination is what a long-running intake usually wants. The caller is not held for the work, and it still gets back something it can act on — a batch id to quote, a count of what was accepted — instead of an empty body it cannot distinguish from a message that went nowhere:
HTTP/1.1 202 Accepted
Content-Type: application/json
{"batchId":"d36d2c59-0fea-4349-b410-a4b4c7715ab9","rowsAccepted":4,"status":"loading"}
The same render under on-complete comes back as a 200 instead. Either way continue="true" is
what puts the reply at the park rather than at the end of the process, which is the only reason a
load measured in minutes can answer its caller in milliseconds. See
Batch loading for a worked example.
When to pick which
Use on-persist when… | Use on-complete when… |
|---|---|
| The process is short-lived | The process takes real time (minutes, not milliseconds) |
| You want the broker slot released fast | You want the broker to redeliver if the engine crashes mid-process |
| Downstream tolerates at-least-once with inbox dedup catching the duplicate | The side effect is non-idempotent and you want broker redelivery as the restart-recovery path |
| You don’t want a bounded in-memory registry on the engine | You can afford the deferred-ack registry’s configured capacity of pending entries |
The mode is set per channel — different channels on the same engine can use different modes.
Per-transport wiring — what actually realizes on-complete
on-complete’s deferred-settle mechanism is a capability each transport factory self-declares
(see Domain neutrality and the SPI model) — the engine
never hardcodes a per-vendor branch. Current wiring:
| Transport | on-complete | Mechanism |
|---|---|---|
| RabbitMQ | wired | Deferred settle via the registry: basic.ack on completion, basic.nack(requeue=false) on failure/timeout/overflow. |
| Kafka | wired | Settle commands over an internal channel to the consumer task; per-partition low-watermark commits, so an out-of-order settle can never mask an earlier nack. |
| AWS SQS | wired | Ack = delete_message. The visibility timeout keeps running while parked — size it (and the registry timeout) against expected instance duration. |
| Google Pub/Sub | wired | Per-message ack()/nack() handles held in the callback; same lease-timeout caveat as SQS. |
| AMQP 1.0 | wired | Dispositions bridged to the session task (accept / reject). |
| File | wired | The terminal file move (.done/ / .failed/) happens at the instance’s terminal event. |
| HTTP | native | Connection-hold is its on-complete — no registry involved. |
| Knative | wired (response-hold) | The push response is held to the terminal event, bounded by a hold timeout; expiry degrades to a loud warning rather than losing the signal. |
| Dapr | not supported | Dapr’s own pub/sub components own redelivery timers; holding the push response would multiply duplicate deliveries rather than strengthen the guarantee. Declaring on-complete here boots with a loud diagnostic and runs on-persist instead. |
A channel declaring ack-mode: on-complete on a transport whose factory reports it can’t realize
it never fails silently — the engine emits SUTRA.ACK.ON_COMPLETE_UNSUPPORTED at startup and
runs on-persist.
AWS SQS, beyond the settle behaviour in the table:
- A message with no
content-typeattribute is decoded with the channel’s codec, or read as raw bytes on a channel with no codec. - A refused message is deleted. It leaves a WARN
SUTRA.INBOUND.DELIVERY_DROPPEDand, if an audit sink is on, aninbound.DELIVERY_DROPPEDaudit record. See Refused deliveries and connection references. regionandqueue.urlare required, and accept references. See References in connection properties, which covers the other transports too.
The deferred-ack registry
The registry (sutra-channels) is a bounded, insertion-ordered structure with three operator
knobs (see Configuration reference):
| Key | Default | Behavior |
|---|---|---|
sutra.ack.deferred.capacity | 10 000 | At capacity, register() nacks the oldest entry (SUTRA.ACK.DEFERRED_OVERFLOW) before accepting the new one. |
sutra.ack.deferred.timeout | 1 hour | The sweep nacks any entry older than this (SUTRA.ACK.DEFERRED_TIMEOUT) — the broker slot frees, but the instance itself keeps running; inbox dedup absorbs any resulting redelivery. |
sutra.ack.deferred.sweep-interval | 1 minute | Cadence of the background sweep task. |
Eviction is an operator-visible failure mode, not a silent memory leak — repeated
SUTRA.ACK.DEFERRED_OVERFLOW events mean either raise the capacity or find the runaway processes
that aren’t reaching a terminal state. Diagnostic codes at each stage:
SUTRA.ACK.DEFERRED_REGISTERED (debug), SUTRA.ACK.DEFERRED_ACKED (debug),
SUTRA.ACK.DEFERRED_NACKED (warn — a permanent reject), SUTRA.ACK.DEFERRED_OVERFLOW (warn),
SUTRA.ACK.DEFERRED_TIMEOUT (warn), SUTRA.ACK.ON_COMPLETE_UNSUPPORTED (warn, at startup).
stateDiagram-v2
[*] --> Registered: on-complete inbound, ack withheld
Registered --> Acked: instance completed
Registered --> Nacked: instance failed
Registered --> Nacked: sweep, entry older than the timeout
Registered --> Nacked: at capacity, oldest entry evicted
The completion and failure edges are the mode working as intended. The sweep and capacity edges are the ones an operator watches: a timeout nack frees the broker slot while the instance keeps running, and repeated overflow means either the capacity is too low or something is never reaching a terminal state.
Migration note
Omitting ack-mode entirely preserves today’s behavior exactly: a broker channel stays
on-persist, and an HTTP channel stays synchronous (on-complete). Turning on a broker’s
deferred ack, or an HTTP channel’s async intake, is a one-line addition to channels.yaml and
takes effect on the next deployment poll.
When a delivery can’t be checked for a duplicate
Before it runs a delivery with an explicit event id or a declared dedupKey="body.<path>", the engine
checks whether that key was already processed. If the database doesn’t answer the check, the delivery
is refused and nothing runs,
with SUTRA.INBOUND.DEDUP_CHECK_UNAVAILABLE. This holds even for a flow that never touches the
database, because a duplicate can’t be undone afterwards: a data task’s store write commits before
the step that would have caught it.
A delivery also gets refused, with SUTRA.INBOUND.DEDUP_IN_DOUBT, when an earlier delivery of the
same event is recorded but a synchronous-call chain still holds that record. That chain is either
running, or it failed and couldn’t release the record. The refusal stops the retry being swallowed
as a duplicate of work that may never have finished. It also applies to a retry during a healthy,
running chain, which gets a refusal until the chain ends. If the chain’s process is gone, or it could
not release the record, the sweep resolves it within sutra.instance.inflight-chain-max-age, 15 minutes
by default, so the refusal can last that long.
A refused delivery is safe to repeat with the same id. What the sender sees depends on the transport:
| Transport | What happens |
|---|---|
| HTTP | 503 with a problem document carrying the code. There’s no Retry-After header. |
| Dapr, Knative | 500, which the sender’s own retry policy treats as retryable. |
| RabbitMQ | basic.nack(requeue=true). |
| AMQP 1.0 | The delivery is released, so the broker redelivers it. |
| Kafka | The consumer seeks back to the record, so the next poll reads it again. |
| SQS | The message stays in flight, and returns after its visibility timeout. |
| Pub/Sub | The message is nacked, and redelivered after its ack deadline. |
| File | The file moves back to the spool root for a later poll. |
local:// hops | Retried with the outbox’s backoff. |
What counts as an explicit event id:
- On HTTP, Dapr and Knative: the channel’s
idempotency-key-header, or a CloudEventid. - On a queue transport: the engine’s own outbox key. A message from another producer doesn’t have one.
- On the file transport: the file name.
local://hops and external-task completions always have one.
A delivery with neither isn’t refused. It never reaches the check, and a repeat of it is simply a
new delivery. That includes a message from another producer on a queue transport, unless its flow
declares a dedupKey.
While a database is down, expect more retries, not lost work: nothing was accepted that wasn’t run.
Next
- The money-transfer worked example uses
singleton+ack-modetogether across three transports feeding one process. - Troubleshooting BPMN solutions — reading the
SUTRA.ACK.*codes above out of the audit trail when a message seems to have vanished.
Refused deliveries and connection references
This page covers what the engine does with a message it permanently refuses on a broker or file channel, what changed in 0.6.18, and how references behave in a channel’s connection properties. For the ack modes themselves see Acknowledgement modes.
A message with no content type
Before 0.6.18 (back to 0.2.0-rc.1), a broker message with no content type on a channel that declares a codec
was deleted with nothing recorded: no instance, no log line, no audit record. The source filled in
application/octet-stream, the codec refused that, and the drop left no trace.
Now a message with no content type is decoded with the channel’s codec. If the codec refuses the body, it
follows the normal reject path. A message that states a type the codec does not accept is still refused.
A channel with no codec reads a message with no type as raw bytes, as before. The one visible difference there
is that event.contentType is empty rather than application/octet-stream, the same as on HTTP and file
channels.
What a permanent refusal leaves behind
The same silence covered every permanent refusal on these transports: a body the codec refused, a stated type it
does not accept, a message no start event matched, a tenant quota. HTTP, Dapr and Knative answer the sender with a
status, and the file transport moves the file to .failed/. Every other transport now leaves two things:
- A WARN log line with
codeSUTRA.INBOUND.DELIVERY_DROPPED, the refusal’s own code ascause_code, the channel and the dedup key. It never carries the body. - When an audit sink is on, a record
inbound.DELIVERY_DROPPEDin the deployment’s journal. Itsdiagnostic_codeis the refusal’s, and itsdiagnostic_jsonnames the channel. No instance owns it, so itsinstance_idis NULL, not empty: select it withinstance_id IS NULL.
Two limits of this release:
sutra.audit.sqlis off by default. An engine with no audit sink writes no audit record, and the WARN is the only trace. Turn a sink on if you need the record. See Logging and audit.- A refused body is not kept. On SQS a refused message is deleted, and nothing in the engine can redrive it. A sender that needs the body back keeps its own copy.
What the refusal did to the message depends on the transport, and a copy may still exist:
| Transport | What happens | Where a copy can still be |
|---|---|---|
aws-sqs | deleted | nowhere, unless the sender keeps its own |
kafka | offset committed past it | in the topic, until retention removes it |
gcp-pubsub | acked | in a snapshot or a seek window, if you keep one |
rabbitmq, amqp | rejected without requeue | in the dead-letter route, if the broker has one |
A rejected credential gets the same audit record, beside the WARN the transport already wrote. A message settled
as a duplicate is not a drop: it logs at INFO with SUTRA.INBOUND.DUPLICATE_DELIVERY. A deferred nack for a
failed instance, SUTRA.ACK.DEFERRED_NACKED, is now WARN rather than INFO.
To check whether you lost messages on an earlier version, compare what your senders sent with the instances
and rows the engine produced for that channel over the time it ran. On Kafka, compare the topic’s offsets with the
same counts, and replay what is missing with content-type set. The engine holds nothing to query, because
earlier versions wrote nothing.
References in connection properties
Since 0.6.19 every inbound transport expands them: see References in connection properties. Version 0.6.18 refused ${ on all but aws-sqs.
References in connection properties
A channel’s broker address, queue or spool directory can come from the environment or a mounted secret, so one package runs unchanged in each environment. This walkthrough runs a RabbitMQ channel whose whole connection is references, then shows what happens when one doesn’t resolve.
You need the engine, PostgreSQL and a broker. This was run on PostgreSQL 16 and RabbitMQ 3.13.
Which properties, and which forms
| Transport | Properties that may be a reference |
|---|---|
kafka | bootstrap.servers, topic, group.id, tls-ca-file |
rabbitmq | host, port, virtual-host, queue, exchange |
amqp | host, port, queue, topic, tls-ca-file |
gcp-pubsub | project-id, subscription, topic, endpoint-override |
file | spool.dir, directory |
aws-sqs | region, queue.url, endpoint-override, wait-time-seconds, max-messages, visibility-timeout-seconds |
| Form | Reads |
|---|---|
${NAME} | the environment variable, and it can sit inside other text: ${BROKER_HOST}:9092 |
${NAME:default} | the same, or the default when it isn’t set |
env:NAME | the environment variable; the whole value only |
secret:KEY, ${secret:KEY} | the file KEY in SUTRA_SECRETS_DIR, default /etc/sutra/secrets |
vault:, aws-secrets:, azure-kv:, gcp-secret: | the vendor store, in a build that links it; the whole value only, or inside ${…} |
Other properties of these channels, such as prefetch-count, stay literal, and so do the TLS and SASL switches
(tls, security.protocol, sasl.mechanism, tls-insecure-skip-verify).
Credentials (username, password on rabbitmq and amqp; since 0.6.21 sasl.username, sasl.password on
kafka, and tls-ca-pem on amqp and kafka) take the whole-value forms above, ${secret:KEY} included since 0.6.20, and k8s:secret/<name>#<key>. A literal, or text with a reference inside it such as pre${X},
is refused with SUTRA.STARTUP.REFUSED.HARDCODED_SECRET. See section 5, and
Connecting to a TLS and SASL broker for the TLS and SASL settings.
1. The channel
channels.yaml. The host and port come from the environment, the port with a default, the virtual host from
env:, and the queue name from a mounted secret file:
channels:
- name: note-in
transport: rabbitmq
host: ${BROKER_HOST}
port: ${BROKER_PORT:5672}
virtual-host: env:BROKER_VHOST
queue: secret:notes-queue
username: ${BROKER_USER}
password: ${BROKER_PASSWORD}
ack-mode: on-persist
cloudevents-mode: none
The flow is note-keep from Keeping a message as text, which writes the
message to a notes table. secrets/notes-queue holds the queue’s name, notes-from-a-secret. Create that
queue on the broker in a virtual host called refs.
sutra lint refnotes
sutra package refnotes -o out
2. A reference that doesn’t resolve
Start the engine with SUTRA_DEPLOYMENTS_DIR pointing at out, the secrets folder mounted, and every setting
except BROKER_HOST. It stops with exit code 1 and this error:
rabbitmq channel wiring failed: [SUTRA.INBOUND.RABBITMQ.CONFIG_INVALID] channel 'note-in' property 'host' has an unresolved reference: placeholder '${BROKER_HOST}' resolves to no value (environment variable 'BROKER_HOST' is not set and no default is given)
The message names the channel, the property and the variable.
Now set BROKER_HOST, and set BROKER_PORT to text that is no port. It also stops with exit code 1, and the
message doesn’t repeat what the variable held:
rabbitmq channel wiring failed: [SUTRA.INBOUND.RABBITMQ.CONFIG_INVALID] channel 'note-in' property 'port' has a reference that does not resolve to an integer in 1..=65535
3. All of them set
Set BROKER_HOST and leave BROKER_PORT unset, so that the default applies. The engine starts, and the queue
named by the secret has one consumer:
rabbitmqctl -q list_queues -p refs name consumers --no-table-headers
notes-from-a-secret 1
Publish a message to that queue with a message_id of r-1 and a content type of application/json. The row
is there:
SELECT store_key, content_type, body FROM notes;
store_key | content_type | body
-----------+------------------+---------------------------------
r-1 | application/json | {"title":"through a reference"}
(1 row)
The engine’s log names the queue it reached:
{"channel":"note-in","queue":"notes-from-a-secret","level":"INFO","message":"rabbitmq consumer up"}
So a value you read from a secret store shows up in connection log lines, as a literal one would. These are connection settings. Keep credentials in the credential properties.
4. An activation in a running engine
An activation is not refused for this. With the engine on the db deployment source, version 1.0.0 deployed
and consuming, deploy a 1.1.0 whose queue is ${NOTES_QUEUE_NOT_SET}:
sutra deploy out2/refnotes.sutra --api --engine-url "$B"
deployed dep-f26c5a0b3b788bca8f9833e5 (slot default--refnotes--1.1.0, rev 1) — Active
1.1.0 reports Active and ready, and has no consumer. The engine logs a WARN:
{"code":"SUTRA.INBOUND.RABBITMQ.CONFIG_INVALID","level":"WARN","message":"rabbitmq consumer NOT started during flip: channel 'note-in' property 'queue' has an unresolved reference: placeholder '${NOTES_QUEUE_NOT_SET}' resolves to no value (environment variable 'NOTES_QUEUE_NOT_SET' is not set and no default is given)"}
No queue with that name was created, and 1.0.0 keeps its consumer. So watch for this code after a deploy,
because the deploy’s own answer won’t show it.
If the channel with the bad reference already had a running consumer, that consumer stops. The next activation in which the reference resolves starts it again.
5. Broker credentials from a secret, and a blank reference
Since 0.6.20, username and password on rabbitmq and amqp take secret:KEY and ${secret:KEY}, besides
${NAME}, env:NAME and k8s:secret/<name>#<key>. A literal is still refused with
SUTRA.STARTUP.REFUSED.HARDCODED_SECRET.
A reference that resolves to blank now warns SUTRA.CONFIG.PROPERTY.BLANK_REFERENCE, naming the channel, the property
and the reference as written, never a value. Write ${NAME:} for an optional setting: it does not warn.
A RabbitMQ channel whose credentials come from secret files (/etc/sutra/secrets/broker-user,
/etc/sutra/secrets/broker-password), and whose exchange is ${H} with H set but empty. The
broker has only the user in that file, so the consumer connects only if the credentials resolved.
Run B is the same package with exchange: ${H:}.
using sutra 0.6.20
$ cat refnotes/channels.yaml
channels:
- name: note-in
transport: rabbitmq
host: ${BROKER_HOST}
virtual-host: env:BROKER_VHOST
queue: secret:notes-queue
exchange: ${H}
username: secret:broker-user
password: ${secret:broker-password}
ack-mode: on-persist
cloudevents-mode: none
$ sutra lint refnotes
[WARN] SUTRA.CHANNEL.NO_SCHEMA — deployment default/refnotes/1.0.0 (dep-ffffffffffffffffffffffff): channel 'note-in' has 1 subscriber(s) but its codec (none) declares no message types — no type can be derived, so only a catch-all <q:source> can match. Schema-less ingress is legitimate; this warns so
[WARN] SUTRA.DEPLOY.DATASTORE.SHARED_CONNECTION — the connection of data store(s) notes names neither this package's tenant nor its module, so a copy co-hosted on one engine resolves the same connection and shares those stores' rows. If that is not intended, give each variant its own reference in it
0 error(s), 2 warning(s)
[exit 0]
$ sutra package refnotes -o out-a
[WARN] SUTRA.CHANNEL.NO_SCHEMA — deployment default/refnotes/1.0.0 (dep-ffffffffffffffffffffffff): channel 'note-in' has 1 subscriber(s) but its codec (none) declares no message types — no type can be derived, so only a catch-all <q:source> can match. Schema-less ingress is legitimate; this warns so it is explicit.
packaged out-a/refnotes.sutra (deploymentId dep-11d6a5f464d563879221824f)
[exit 0]
package refnotes-b exit=0
broker up: 1
== Run A: secret: credentials, exchange: ${H} with H set but empty
ready: {"status":"UP","checks":[{"name":"sutra-loader","status":"UP","data":{"deploymen
queue, consumers: notes-from-a-secret 1
Message published
store_key | body
-----------+--------------------------------------------
s-1 | {"title":"credentials from a secret file"}
(1 row)
BLANK_REFERENCE lines:
{"code":"SUTRA.CONFIG.PROPERTY.BLANK_REFERENCE","channel":"note-in","property":"exchange","timestamp":"2026-11-10T05:26:09.248+00:00","sequence":6,"level":"WARN","loggerName":"sutra_transport_spi","message":"channel 'note-in' property 'exchange' reference '${H}' resolved to blank, so the property is unset: the transport's built-in default applies, or a required property is refused. Write the refer
HARDCODED_SECRET lines: 0
broker password in the log: 0
== Run B: exchange: ${H:} with H set but empty
ready: {"status":"UP","checks":[{"name":"sutra-loader","status":"UP","data":{"deploymen
queue, consumers: notes-from-a-secret 1
Message published
store_key | body
-----------+---------------------------------
s-2 | {"title":"an optional setting"}
(1 row)
BLANK_REFERENCE lines: 0
broker password in the log: 0
removed, left: 0
The full warning text, one line in the log:
channel 'note-in' property 'exchange' reference '${H}' resolved to blank, so the property is unset: the transport's built-in default applies, or a required property is refused. Write the reference with an empty default, such as ${NAME:}, to leave it blank on purpose
What to watch for
- A reference that resolves to blank is the property left unset, so
${NAME:}writes an optional setting. A variable that is set but empty counts too. A required property is then refused: Kafkabootstrap.serversandtopic, RabbitMQqueue, AMQPhost, Pub/Subproject-id, the file spool directory. One with a default takes it: RabbitMQhostbecomeslocalhost, and Kafkagroup.idbecomessutra-<channel>. Since 0.6.20 the engine logsSUTRA.CONFIG.PROPERTY.BLANK_REFERENCEfirst (0.6.19 was silent);${NAME:}is the explicit form and does not warn. - A value that starts with a secret scheme and a colon is a reference. A Kafka broker called
vault, writtenbootstrap.servers: vault:9092, fails to resolve. Write it${BROKER:vault:9092}.kafka:9092is fine. - A changed value restarts the consumer at the next activation. An activation is what reads it again; nothing watches it in between.
- Endpoint claims compare references as written, so two deployments on one queue under different names aren’t caught. See One endpoint, one deployment.
Connecting to a TLS and SASL broker
Since 0.6.21 the kafka and amqp (AMQP 1.0) transports connect to TLS brokers natively, with no sidecar. Kafka
also authenticates with SASL PLAIN, SCRAM-SHA-256 or SCRAM-SHA-512, and AMQP 1.0 with SASL PLAIN.
What this was tested against. Local containers only: Apache Kafka 3.8 and Apache ActiveMQ Artemis. Azure Service Bus, Confluent Cloud, Amazon MSK and other managed services have not been tested and are not claimed to work. Azure Service Bus’s CBS token flow is not implemented. Client certificates (mTLS) and SASL OAUTHBEARER and GSSAPI are not supported.
The settings
For an inbound channel:
| Property | Transport | Meaning |
|---|---|---|
security.protocol | kafka | PLAINTEXT (default), SSL, SASL_PLAINTEXT or SASL_SSL |
sasl.mechanism | kafka | PLAIN, SCRAM-SHA-256 or SCRAM-SHA-512 |
sasl.username, sasl.password | kafka | Secret references |
tls: true | amqp | Use TLS. The default port becomes 5671 |
username, password | amqp | Secret references. SASL PLAIN is used when both are set |
tls-ca-file | both | Path to a PEM bundle, used in place of the system CAs |
tls-ca-pem | both | Secret reference to PEM text, used in place of the system CAs |
tls-insecure-skip-verify | both | true turns off certificate and host-name checks |
A credential, and tls-ca-pem, must be a reference as a whole value: env:NAME, secret:KEY, ${NAME},
${secret:KEY}, a vendor store scheme the engine is built with, or k8s:secret/<name>#<key> (see
References in connection properties). A literal value stops the channel at
startup with SUTRA.STARTUP.REFUSED.HARDCODED_SECRET. tls-ca-file takes references like an address does;
the switches (tls, security.protocol, sasl.mechanism, tls-insecure-skip-verify) stay literal.
With no CA setting, the broker is checked against the system CAs, including any set with
SSL_CERT_FILE. Certificate and host-name checks are on unless you set tls-insecure-skip-verify: true, and the engine logs a warning whenever that setting applies.
A setting that could not take effect stops the channel at startup rather than being ignored: a CA
without TLS, sasl.* without a SASL protocol, both CA settings at once, or SASL GSSAPI or OAUTHBEARER. Kafka’s kafka.*
passthrough does not accept sasl.*, ssl.* or enable.ssl.certificate.verification.
Outbound sends use one engine-wide configuration per transport, set in the environment:
| Variable | Meaning |
|---|---|
SUTRA_SINK_KAFKA_BOOTSTRAP_SERVERS | The brokers |
SUTRA_SINK_KAFKA_SECURITY_PROTOCOL | As security.protocol |
SUTRA_SINK_KAFKA_SASL_MECHANISM, _SASL_USERNAME, _SASL_PASSWORD | As the sasl.* properties. These hold values, so set them from your secret store |
SUTRA_SINK_KAFKA_TLS_CA_FILE, SUTRA_SINK_KAFKA_TLS_INSECURE_SKIP_VERIFY | Trust |
SUTRA_SINK_AMQP_USERNAME, SUTRA_SINK_AMQP_PASSWORD | Default credentials |
SUTRA_SINK_AMQP_TLS_CA_FILE, SUTRA_SINK_AMQP_TLS_INSECURE_SKIP_VERIFY | Trust for amqp10s:// destinations |
SUTRA_SINK_AMQP_OPERATION_TIMEOUT_MS | The bound on each broker round trip of a send (opening the connection with TLS and SASL, the session, the attach, the wait for the broker to accept the message, the close), default 30000. A broker that does not answer in time fails the attempt with the retryable SUTRA.OUTBOUND.AMQP.SEND_TIMEOUT and the outbox sends again; a send accepted just before its answer timed out can arrive twice, so receivers dedupe on sutra-outbox-key. Since 0.6.22. |
SUTRA_SINK_AMQP_MAX_CONNECTIONS | How many connections the sink keeps open to one broker (scheme, host, port and credentials), default 32 (8 in 0.8.2), matching sutra.outbox.destination-concurrency so a broker destination’s sends do not queue for a connection. Each has one session and a sender link per target address; a send takes an idle one and, with all busy, waits for one for at most the operation timeout. A connection that fails a send is dropped and the send tried once more on a fresh one; one idle for two minutes is closed. Since 0.8.2. |
Walkthrough: a message through both brokers
You need Docker, openssl, the sutra CLI and an empty working directory. This creates a private
CA, starts both brokers with TLS and a login, and runs a package that answers each message. It was run
as written, with sutra 0.6.21, apache/kafka:3.8.0, apache/activemq-artemis:latest-alpine and
postgres:16-alpine; the output shown is from that run.
1. Create the certificates. The server certificate must name the host the engine connects to.
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes -days 7 \
-subj "/CN=demo broker CA" -keyout ca.key -out ca.pem
for broker in tlsdemo-kafka tlsdemo-artemis; do
openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
-subj "/CN=$broker" -keyout "$broker.key" -out "$broker.csr"
openssl x509 -req -in "$broker.csr" -CA ca.pem -CAkey ca.key -CAcreateserial -days 7 \
-extfile <(printf "subjectAltName=DNS:%s" "$broker") -out "$broker.crt"
done
chmod 644 ./*.key ./*.crt ca.pem
BROKER_PASSWORD=$(openssl rand -hex 16)
docker network create tlsdemo-net
The chmod lets the containers read these files. The keys are throwaway, so don’t reuse this step
for real keys.
2. Start Kafka with a plaintext listener inside its container for the commands below, and a
SASL_SSL listener for the engine. Save this as server.properties:
process.roles=broker,controller
node.id=1
controller.quorum.voters=1@localhost:9093
controller.listener.names=CONTROLLER
inter.broker.listener.name=INTERNAL
listeners=INTERNAL://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093,SECURE://0.0.0.0:9094
advertised.listeners=INTERNAL://localhost:9092,SECURE://tlsdemo-kafka:9094
listener.security.protocol.map=INTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT,SECURE:SASL_SSL
sasl.enabled.mechanisms=SCRAM-SHA-256
listener.name.secure.scram-sha-256.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required;
ssl.keystore.type=PEM
ssl.keystore.location=/certs/kafka-keystore.pem
log.dirs=/tmp/kraft-logs
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
cat tlsdemo-kafka.key tlsdemo-kafka.crt > kafka-keystore.pem
chmod 644 kafka-keystore.pem server.properties
docker run -d --name tlsdemo-kafka --network tlsdemo-net \
-v "$PWD/server.properties:/certs/server.properties:ro" \
-v "$PWD/kafka-keystore.pem:/certs/kafka-keystore.pem:ro" \
--entrypoint /bin/sh apache/kafka:3.8.0 -c "
/opt/kafka/bin/kafka-storage.sh format -t MkU3OEVBNTcwNTJENDM2Qk -c /certs/server.properties \
--add-scram 'SCRAM-SHA-256=[name=orders-svc,password=$BROKER_PASSWORD]' &&
exec /opt/kafka/bin/kafka-server-start.sh /certs/server.properties"
until docker logs tlsdemo-kafka 2>&1 | grep -q 'Kafka Server started'; do sleep 1; done
for topic in orders-in orders-out; do
docker exec tlsdemo-kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 \
--create --topic "$topic"
done
3. Start Artemis with an AMQPS acceptor.
openssl pkcs12 -export -in tlsdemo-artemis.crt -inkey tlsdemo-artemis.key -out artemis.p12 \
-passout pass:changeit -name broker
chmod 644 artemis.p12
docker run -d --name tlsdemo-artemis --network tlsdemo-net \
-v "$PWD/artemis.p12:/certs/artemis.p12:ro" \
--entrypoint /bin/sh apache/activemq-artemis:latest-alpine -c "
/opt/activemq-artemis/bin/artemis create --user orders-svc --password $BROKER_PASSWORD \
--silent --require-login --http-host 0.0.0.0 --relax-jolokia . &&
sed -i 's#</acceptors>#<acceptor name=\"amqps\">tcp://0.0.0.0:5671?sslEnabled=true;keyStorePath=/certs/artemis.p12;keyStorePassword=changeit;keyStoreType=PKCS12;protocols=AMQP;saslMechanisms=PLAIN</acceptor></acceptors>#' etc/broker.xml &&
exec ./bin/artemis run"
until docker logs tlsdemo-artemis 2>&1 | grep -q 'Server is now active'; do sleep 1; done
for q in orders-in orders-out; do
docker exec tlsdemo-artemis ./bin/artemis queue create --user orders-svc --password "$BROKER_PASSWORD" \
--url tcp://localhost:61616 --name "$q" --address "$q" --anycast --durable --auto-create-address \
--preserve-on-no-consumers --silent
done
The two queues are created up front, as anycast queues, so a message sent before anyone reads it is kept. Left to Artemis, an address the engine attaches to first is created as multicast, and the producer below would send to a queue no one reads.
4. Write the package. Create tls-relay/package.yaml:
labels:
"tenant": "default"
"module": "tls-relay"
"version": "1.0.0"
engine:
minContract: 1
Create tls-relay/channels.yaml:
channels:
- name: kafka-in
transport: kafka
codec: json
cloudevents-mode: none
bootstrap.servers: tlsdemo-kafka:9094
topic: orders-in
group.id: tls-relay
security.protocol: SASL_SSL
sasl.mechanism: SCRAM-SHA-256
sasl.username: env:BROKER_USER
sasl.password: secret:BROKER_PASSWORD
tls-ca-pem: secret:BROKER_CA
- name: kafka-out
direction: outbound
transport: kafka
bind: kafka://orders-out
cloudevents-mode: none
- name: amqp-in
transport: amqp
codec: json
cloudevents-mode: none
host: tlsdemo-artemis
tls: true
tls-ca-pem: secret:BROKER_CA
username: env:BROKER_USER
password: secret:BROKER_PASSWORD
queue: orders-in
- name: amqp-out
direction: outbound
transport: amqp
bind: amqp10s://tlsdemo-artemis/orders-out
cloudevents-mode: none
Create tls-relay/bpmn/tls-relay.bpmn, with one process per transport. Each builds an answer and
sends it out. This is the Kafka one; add a copy for AMQP with its ids prefixed amqp, its process id
amqp-relay, and the channels amqp-in and amqp-out:
<?xml version="1.0" encoding="UTF-8"?>
<bpmn:definitions xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL"
xmlns:q="urn:sutra:q:1.0" id="Defs"
targetNamespace="urn:sutra:module:tls-relay:1.0.0">
<bpmn:process id="kafka-relay" isExecutable="true">
<bpmn:extensionElements>
<q:variables><q:variable name="responseBody" type="object"/></q:variables>
</bpmn:extensionElements>
<bpmn:startEvent id="kafkaStart">
<bpmn:extensionElements><q:source channel="kafka-in"/></bpmn:extensionElements>
<bpmn:outgoing>kafkaF1</bpmn:outgoing>
</bpmn:startEvent>
<bpmn:sequenceFlow id="kafkaF1" sourceRef="kafkaStart" targetRef="kafkaBuild"/>
<bpmn:serviceTask id="kafkaBuild">
<bpmn:incoming>kafkaF1</bpmn:incoming><bpmn:outgoing>kafkaF2</bpmn:outgoing>
<bpmn:dataInputAssociation>
<bpmn:assignment>
<bpmn:from>{ref: payload.ref, status: "ANSWERED"}</bpmn:from>
<bpmn:to>responseBody</bpmn:to>
</bpmn:assignment>
</bpmn:dataInputAssociation>
</bpmn:serviceTask>
<bpmn:sequenceFlow id="kafkaF2" sourceRef="kafkaBuild" targetRef="kafkaAnswer"/>
<bpmn:sendTask id="kafkaAnswer">
<bpmn:extensionElements><q:send channel="kafka-out"/></bpmn:extensionElements>
<bpmn:incoming>kafkaF2</bpmn:incoming><bpmn:outgoing>kafkaF3</bpmn:outgoing>
</bpmn:sendTask>
<bpmn:sequenceFlow id="kafkaF3" sourceRef="kafkaAnswer" targetRef="kafkaEnd"/>
<bpmn:endEvent id="kafkaEnd"><bpmn:incoming>kafkaF3</bpmn:incoming></bpmn:endEvent>
</bpmn:process>
</bpmn:definitions>
mkdir -p deployments secrets
sutra lint tls-relay
sutra package tls-relay --out deployments
[WARN] SUTRA.CHANNEL.NO_SCHEMA — deployment default/tls-relay/1.0.0 (...): channel 'kafka-in' has 1 subscriber(s) but its codec (json) declares no message types — ...
[WARN] SUTRA.CHANNEL.NO_SCHEMA — deployment default/tls-relay/1.0.0 (...): channel 'amqp-in' has 1 subscriber(s) but its codec (json) declares no message types — ...
0 error(s), 2 warning(s)
packaged deployments/tls-relay.sutra (deploymentId dep-e890994c8e67a1c5b02e6db2)
The two warnings say the channels are schema-less, which is what this package intends.
5. Start the engine. The inbound channels read the password and the CA from the mounted secrets
directory. The engine-wide sinks take theirs from the environment. The database role has no
superuser or BYPASSRLS rights, because the engine refuses to start on one that does.
printf '%s' "$BROKER_PASSWORD" > secrets/BROKER_PASSWORD
cp ca.pem secrets/BROKER_CA
chmod -R a+rX deployments secrets
docker run -d --name tlsdemo-db --network tlsdemo-net -e POSTGRES_PASSWORD=dev-only postgres:16-alpine
until docker exec tlsdemo-db pg_isready -U postgres -h 127.0.0.1; do sleep 1; done
sleep 2
docker exec tlsdemo-db psql -U postgres \
-c "CREATE ROLE sutra_engine LOGIN PASSWORD 'dev-only' NOSUPERUSER NOBYPASSRLS" \
-c "ALTER SCHEMA public OWNER TO sutra_engine"
docker run -d --name tlsdemo-engine --network tlsdemo-net -p 127.0.0.1::8080 \
-v "$PWD/deployments:/deployments:ro" -e SUTRA_DEPLOYMENTS_DIR=/deployments \
-v "$PWD/secrets:/etc/sutra/secrets:ro" \
-v "$PWD/ca.pem:/etc/sutra/broker-ca.pem:ro" \
-e SUTRA_DATASOURCE_URL=postgres://tlsdemo-db:5432/postgres \
-e SUTRA_DATASOURCE_USERNAME=sutra_engine -e SUTRA_DATASOURCE_PASSWORD=dev-only \
-e BROKER_USER=orders-svc \
-e SUTRA_SINK_KAFKA_BOOTSTRAP_SERVERS=tlsdemo-kafka:9094 \
-e SUTRA_SINK_KAFKA_SECURITY_PROTOCOL=SASL_SSL \
-e SUTRA_SINK_KAFKA_SASL_MECHANISM=SCRAM-SHA-256 \
-e SUTRA_SINK_KAFKA_SASL_USERNAME=orders-svc \
-e SUTRA_SINK_KAFKA_SASL_PASSWORD="$BROKER_PASSWORD" \
-e SUTRA_SINK_KAFKA_TLS_CA_FILE=/etc/sutra/broker-ca.pem \
-e SUTRA_SINK_AMQP_USERNAME=orders-svc \
-e SUTRA_SINK_AMQP_PASSWORD="$BROKER_PASSWORD" \
-e SUTRA_SINK_AMQP_TLS_CA_FILE=/etc/sutra/broker-ca.pem \
ghcr.io/startr-trade/sutra:0.6.21
PORT=$(docker port tlsdemo-engine 8080 | head -1 | cut -d: -f2)
until curl -sf "http://127.0.0.1:$PORT/sutra/health/ready"; do sleep 1; done
{"status":"UP","checks":[{"name":"sutra-loader","status":"UP","data":{"deployments":1,"shards":1}}]}
6. Send a message through Kafka and read the answer.
echo '{"ref":"kafka-1"}' | docker exec -i tlsdemo-kafka /opt/kafka/bin/kafka-console-producer.sh \
--bootstrap-server localhost:9092 --topic orders-in
docker exec tlsdemo-kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 \
--topic orders-out --from-beginning --max-messages 1 --timeout-ms 60000
{"ref":"kafka-1","status":"ANSWERED"}
Processed a total of 1 messages
The message went in and the answer came out over the SASL_SSL listener, with SCRAM-SHA-256 and the private CA; the commands above use the broker’s internal plaintext listener only to drive it.
7. Send a message through Artemis and read the answer.
docker exec tlsdemo-artemis ./bin/artemis producer --user orders-svc --password "$BROKER_PASSWORD" \
--url tcp://localhost:61616 --protocol amqp --destination queue://orders-in --message-count 1 \
--message '{"ref":"amqp-1"}'
docker exec tlsdemo-artemis ./bin/artemis browser --user orders-svc --password "$BROKER_PASSWORD" \
--url tcp://localhost:61616 --destination queue://orders-out --message-count 1 --verbose
Producer orders-in, thread=0 Produced: 1 messages
...
Consumer ActiveMQQueue[orders-out], thread=0 browsed: 1 messages
The engine read the message over AMQPS (port 5671, SASL PLAIN, the private CA) and sent its answer to
amqp10s://tlsdemo-artemis/orders-out. The engine’s log holds the broker password 0 times.
8. See a refusal. Restart the engine with a wrong password in secrets/BROKER_PASSWORD (and in the
two SUTRA_SINK_*_PASSWORD variables), and read its log:
docker logs tlsdemo-engine 2>&1 | grep -E 'BROKER_(AUTH|TLS)_FAILED'
{"channel":"amqp-in","code":"SUTRA.INBOUND.AMQP.BROKER_AUTH_FAILED",...,"level":"WARN",...,"message":"amqp consumer unavailable — retrying in 1s: amqp channel 'amqp-in' could not connect to tlsdemo-artemis:5671: the broker refused the SASL credentials (outcome Auth)",...}
{"client":"channel 'kafka-in'","code":"SUTRA.INBOUND.KAFKA.BROKER_AUTH_FAILED",...,"level":"WARN",...,"message":"kafka channel 'kafka-in': sasl_ssl://tlsdemo-kafka:9094/bootstrap: SASL authentication error: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 ...",...}
Put the right password back and a certificate from another CA in secrets/BROKER_CA, restart, and the
log reads:
{"client":"channel 'kafka-in'","code":"SUTRA.INBOUND.KAFKA.BROKER_TLS_FAILED",...,"message":"kafka channel 'kafka-in': sasl_ssl://tlsdemo-kafka:9094/bootstrap: SSL handshake failed: error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified ...",...}
{"channel":"amqp-in","code":"SUTRA.INBOUND.AMQP.BROKER_TLS_FAILED",...,"message":"amqp consumer unavailable — retrying in 1s: amqp channel 'amqp-in' could not connect to tlsdemo-artemis:5671: the TLS handshake failed: invalid peer certificate: UnknownIssuer",...}
Neither the wrong password nor the right one appeared anywhere in the log.
| Code | Meaning |
|---|---|
SUTRA.{INBOUND,OUTBOUND}.{KAFKA,AMQP}.BROKER_AUTH_FAILED | The broker refused the credentials |
SUTRA.{INBOUND,OUTBOUND}.{KAFKA,AMQP}.BROKER_TLS_FAILED | The CA does not trust the broker’s certificate, the certificate does not name the host, or the trust settings could not be loaded |
Neither message includes a password. Both consumers keep retrying and an outbound send stays in the
outbox, so both recover once you correct the setting and restart the engine. The AMQP consumer backs
off (1 s, 2 s, …). The Kafka consumer logs one line per attempt by its client, which retries quickly:
with the wrong CA this run logged about 740 BROKER_TLS_FAILED lines in 20 seconds, so alert on the code
rather than on the line count.
9. Clean up.
docker rm -f tlsdemo-engine tlsdemo-db tlsdemo-artemis tlsdemo-kafka
docker network rm tlsdemo-net
Connecting the engine to IBM MQ
The engine reaches IBM MQ through the queue manager’s AMQP 1.0 channel, with the amqp transport.
There is no MQ client library in the engine: the queue manager speaks AMQP on a port of its own, and
MQ applications and other queue managers see ordinary MQ messages on ordinary MQ queues.
What this was tested against. IBM MQ Advanced for Developers 10.0.1.0 in containers, on one machine: two queue managers joined by a sender/receiver channel pair over mutual TLS, local queues, a remote queue definition, a topic with administrative subscriptions, the AMQP channel in plain text and with TLS, SASL PLAIN, concurrent sends, and the queue manager stopped and started while messages were being sent. IBM MQ on z/OS, MQ appliances, IBM MQ as a service and the native MQI client have not been tested and are not claimed to work.
What MQ needs from a channel
MQ maps AMQP onto its own message model, and two of its rules decide how a channel is written:
- An address without a capability is a topic. MQ reads an AMQP address as a topic string unless the
link says it names a queue. Set
address-type: queueon an inbound channel and?type=queueon a destination. Without it a send goes to the topic of that name (with no subscriber it reaches no one, and a user without publish authority there is refused withAMQXR0102E), and an inbound channel subscribes to that topic instead of reading the queue. - The body section decides the MQ format. A message that comes over AMQP is stored as
MQSTRwhen its body is one AMQP string, asMQFMT_NONEwhen it is one AMQP binary value, and as an embedded AMQP message (MQAMQP) otherwise, which includes the engine’s defaultdatasection. Addbody=textfor XML, JSON and other text, orbody=binaryfor bytes, to every destination an MQ application reads. An engine reading the queue over AMQP gets the same bytes either way.
Two more rules the transport follows by itself since 0.7.0: every message names its
destination in its to property, which MQ requires (AMQXR0032E otherwise), and every connection has
its own container id, because MQ treats the container id as the client’s identity and closes the
older of two connections that share one.
The settings
An inbound channel takes the amqp properties described in
Connecting to a TLS and SASL broker, and one more:
| Property | Meaning |
|---|---|
address-type | queue or topic, sent to the broker as the link’s capability. Unset sends none |
A destination (amqp10:// or amqp10s://host[:port]/<address>) takes these query options. Each is off
unless you set it, so an existing destination sends what it always did:
| Option | Values | Effect |
|---|---|---|
type | queue, topic | The link’s capability |
durable | true, false | The message’s durable flag: MQ’s persistent / not persistent |
ttl-ms | milliseconds | The message’s time to live: MQ’s expiry, in tenths of a second |
body | data (default), binary, text | data is one AMQP data section, binary one AMQP binary value, text one AMQP string. With text a payload that is not UTF-8 is refused with SUTRA.OUTBOUND.AMQP.CONFIG_INVALID |
application-properties | all (default), none | none sends no application properties: no sutra-outbox-key, no content-type, no headers. The outbox key is still the message-id |
property-names | verbatim (default), identifier | identifier rewrites every application-property name to [A-Za-z_$][A-Za-z0-9_$]*: sutra-outbox-key becomes sutra_outbox_key, which the engine reads too. IBM MQ 10.0.1 kept names with - as they were; IBM documents earlier releases as refusing them, and JMS applications can read only identifier names |
A value an option cannot take stops the send permanently with SUTRA.OUTBOUND.AMQP.CONFIG_INVALID.
Four headers carry AMQP message properties both ways. A received message has them set from the message, and an outbound message that sets them sends them as those properties rather than as application properties:
| Header | AMQP property | In MQ |
|---|---|---|
x-amqp-message-id | message-id | Inbound: MQ’s MsgId, 24 bytes as 48 hex digits. Outbound: kept in the AMQPMsgId property; MQ makes its own MsgId. The engine sends the outbox key when the header is unset |
x-amqp-correlation-id | correlation-id | CorrelId |
x-amqp-reply-to | reply-to | ReplyToQ |
x-amqp-subject | subject | the AMQPSubject property |
How a message looks to MQ
What an MQ application reading the queue saw, from the queue manager’s own amqsbcg:
| MQ field | From the engine |
|---|---|
MsgType | MQMT_DATAGRAM (8), always |
Format | MQSTR with body=text, blank (MQFMT_NONE) with body=binary, MQAMQP with the default data section |
CodedCharSetId | 1208, always; the text is the engine’s UTF-8 bytes unchanged |
Persistence | 1 with durable=true, 0 otherwise: IBM documents a message without the flag as not persistent, whatever the queue’s DEFPSIST |
Expiry | ttl-ms in tenths of a second, unlimited (-1) without it |
MsgId | made by the queue manager; the engine’s message id is kept in the AMQPMsgId property |
CorrelId | the x-amqp-correlation-id text, padded with zero bytes to 24 |
ReplyToQ, ReplyToQMgr | the x-amqp-reply-to queue, and the queue manager the engine is connected to |
UserIdentifier | the user the AMQP channel authenticated |
PutApplName | IBM MQ AMQP Support 10.0.1.0 |
Message properties and MQRFH2. The AMQP header and properties become MQ message properties in an
mq_amqp folder, a reply-to also in a jms folder, and the application properties in usr. An
application that reads a queue with the default PROPCTL(COMPAT) and does not ask for properties gets
them as an MQRFH2 header in front of the body, with Format MQHRF2, when the message has a usr or
jms folder; the mq_amqp folder alone is not handed to it. So for a reader that expects a plain
message, either send it with application-properties=none and no x-amqp-reply-to, or set
PROPCTL(NONE) on the queue it reads. The sender channel’s own PROPCTL does not help: it applies only
to queue managers of version 6 and earlier.
Going the other way, a message an MQ application put arrives with its MsgId in
x-amqp-message-id (48 hex digits), its ReplyToQ in x-amqp-reply-to, and an MQSTR body as text.
The MsgId is also the idempotency key when the message has no sutra-outbox-key.
When the queue manager goes away
An inbound channel reconnects by itself, backing off from 1 s to 30 s, so it is attached again at most about 30 s after the queue manager is back. An outbound send that cannot reach the queue manager fails as retryable and the outbox sends it again. Both log the outage once when it starts and once when it ends, with its start, end and length:
| Code | Logged |
|---|---|
SUTRA.INBOUND.AMQP.OUTAGE_STARTED | WARN, when a consumer loses its connection or cannot open one; outage_started |
SUTRA.INBOUND.AMQP.OUTAGE_ENDED | INFO, when it is attached again; outage_started, outage_ended, outage_ms |
SUTRA.OUTBOUND.AMQP.OUTAGE_STARTED | WARN, per broker (host:port), at the first send that cannot connect |
SUTRA.OUTBOUND.AMQP.OUTAGE_ENDED | INFO, at the first send that connects again |
The walkthrough below shows both lines from a real restart.
The same outage can also reach the package as messages, so a deployment can record its own
outage windows: set SUTRA_CONNECTIVITY_EVENTS_CHANNEL and declare that channel
(see Broker connectivity events and
Broker outages as messages).
An AMQP channel that was running when the queue manager stopped starts again with it. What MQ keeps
across the restart is MQ’s rule, not the engine’s: persistent messages (durable=true) are kept, and
so are non-persistent ones on a queue defined with NPMCLASS(HIGH) after a normal shutdown; other
non-persistent messages are lost. A message the engine had received but not yet settled is delivered
again after the reconnect, and the engine’s inbox drops the duplicate.
What the AMQP channel cannot do
These are limits of MQ’s AMQP support, not settings:
MQPER_PERSISTENCE_AS_Q_DEF. A message is persistent or not, bydurable; the queue’s default does not apply. Setdurableto match the queues a destination writes to.ReplyToQMgr. It is always the queue manager the engine is connected to, and the engine addresses only queues that queue manager can open: its local queues, aliases and remote queue definitions. To answer aReplyToQon another queue manager, define a remote queue definition for it.- A binary
CorrelIdorMsgId. An outboundx-amqp-correlation-idis text, at most 24 bytes as MQ keeps it. The engine cannot set aMsgId: the queue manager makes one. - Other descriptor fields.
Report,Feedback,GroupId,MsgSeqNumber,OffsetandMsgFlagskeep their defaults, and segmentation and message groups are not available. - One AMQP session per connection. The engine opens one connection, with one session, per inbound
channel. Outbound it keeps a pool of up to 32 connections per queue manager (8 before 0.8.3)
(
SUTRA_SINK_AMQP_MAX_CONNECTIONS), each with one session and a sender link per queue, which fits. Those connections stay open (closed after two minutes idle), so each is a running instance of the AMQP channel: allow for the engine’s replicas times the bound in the channel’s instance limits. - Message size. An AMQP channel accepts messages up to its
MAXMSGL(4 MB unless set); raise it, and the queues’, for larger messages.
Where a peer needs exact control of the message descriptor, the AMQP channel is the wrong road: that takes an MQI client, which the engine does not have.
Walkthrough: a message through IBM MQ
You need Docker, openssl, the sutra CLI and an empty working directory. This builds an IBM MQ
developer image with AMQP, gives the queue manager a certificate, defines its AMQP channel, queues and
a topic, runs a package that answers each message, and restarts the queue manager under it. It was
run as written with IBM MQ Advanced for Developers 10.0.1.0, postgres:16-alpine and the engine built
from this release; the output shown is from that run.
1. Build an IBM MQ image with AMQP. The published developer image leaves out MQ’s AMQP component.
Copy it in from the developer archive of the same release. The archive is about 550 MB and IBM’s
download site can be slow on one connection; only its small amqp/ directory is used:
curl -fLO https://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/messaging/mqadv/10.0.1.0-IBM-MQ-Advanced-for-Developers-Non-Install-LinuxX64.tar.gz
mkdir -p mq-amqp
tar xzf 10.0.1.0-IBM-MQ-Advanced-for-Developers-Non-Install-LinuxX64.tar.gz -C mq-amqp amqp
printf 'FROM icr.io/ibm-messaging/mq:10.0.1.0-r1\nCOPY --chown=1001:0 amqp/ /opt/mqm/amqp/\n' > mq-amqp/Dockerfile
docker build -q -t mq-amqp:10.0.1.0 mq-amqp
Running the image means accepting the IBM MQ Advanced for Developers licence (LICENSE=accept
below). It is for development and testing only.
2. Create the certificates. The queue manager’s certificate must name the host the engine connects
to. The AMQP channel reads it from a PKCS#12 key repository with a password; it cannot use the CMS
repository the image builds from /etc/mqm/pki.
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes -days 7 \
-subj "/CN=demo MQ CA" -keyout ca.key -out ca.pem 2>/dev/null
openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
-subj "/CN=mqdemo-qm" -keyout qm.key -out qm.csr 2>/dev/null
openssl x509 -req -in qm.csr -CA ca.pem -CAkey ca.key -CAcreateserial -days 7 \
-extfile <(printf "subjectAltName=DNS:mqdemo-qm") -out qm.crt 2>/dev/null
KEYRING_PASSWORD=$(openssl rand -hex 16)
MQ_PASSWORD=$(openssl rand -hex 16)
mkdir -p tls
openssl pkcs12 -export -name qm -in qm.crt -inkey qm.key -certfile ca.pem \
-out tls/qm.p12 -passout "pass:$KEYRING_PASSWORD"
chmod 644 tls/qm.p12 ca.pem
docker network create mqdemo-net
3. Define the queue manager’s objects. An AMQP channel with TLS that requires a user and password, an inbound and an outbound queue, a topic, and an administrative subscription that copies every publication to a queue. The key repository is set before the AMQP service starts, because the service reads it then.
cat > qm.mqsc <<EOF
ALTER QMGR SSLKEYR('/tls/qm.p12') KEYRPWD('$KEYRING_PASSWORD') CERTLABL('qm')
REFRESH SECURITY TYPE(SSL)
DEFINE CHANNEL('ENGINE.AMQPS') CHLTYPE(AMQP) PORT(5671) MCAUSER('app') +
SSLCIPH('ANY_TLS12_OR_HIGHER') SSLCAUTH(OPTIONAL) REPLACE
SET CHLAUTH('ENGINE.AMQPS') TYPE(ADDRESSMAP) ADDRESS('*') USERSRC(CHANNEL) CHCKCLNT(REQUIRED) +
ACTION(REPLACE)
DEFINE QLOCAL('ORDERS.IN') REPLACE
DEFINE QLOCAL('ORDERS.OUT') REPLACE
DEFINE TOPIC('ORDERS.EVENTS') TOPICSTR('orders/events') REPLACE
DEFINE QLOCAL('ORDERS.EVENTS.COPY') REPLACE
DEFINE SUB('ORDERS.EVENTS.COPY') TOPICSTR('orders/events') DEST('ORDERS.EVENTS.COPY') REPLACE
SET AUTHREC PROFILE('ORDERS.**') PRINCIPAL('app') OBJTYPE(QUEUE) AUTHADD(BROWSE,GET,INQ,PUT)
SET AUTHREC PROFILE('ORDERS.**') PRINCIPAL('app') OBJTYPE(TOPIC) AUTHADD(PUB,SUB)
ALTER SERVICE(SYSTEM.AMQP.SERVICE) CONTROL(QMGR)
START SERVICE(SYSTEM.AMQP.SERVICE)
EOF
chmod 644 qm.mqsc
4. Start the queue manager, then the AMQP channel. The channel can start only once the AMQP service is up, so retry until it runs. After a restart it starts again by itself.
docker run -d --name mqdemo-qm --network mqdemo-net \
-e LICENSE=accept -e MQ_QMGR_NAME=QM1 -e MQ_APP_PASSWORD="$MQ_PASSWORD" \
-v "$PWD/qm.mqsc:/etc/mqm/30-demo.mqsc:ro" -v "$PWD/tls:/tls:ro" \
mq-amqp:10.0.1.0
until docker logs mqdemo-qm 2>&1 | grep -q 'Started queue manager'; do sleep 1; done
until docker exec mqdemo-qm bash -c "echo \"START CHANNEL('ENGINE.AMQPS')
DISPLAY CHSTATUS('ENGINE.AMQPS') CHLTYPE(AMQP)\" | runmqsc QM1" | grep -q 'STATUS(RUNNING)'; do
sleep 1
done
5. Write the package in mq-relay/ (mkdir -p mq-relay/bpmn). mq-relay/package.yaml:
labels:
"tenant": "default"
"module": "mq-relay"
"version": "1.0.0"
engine:
minContract: 1
mq-relay/channels.yaml. The inbound channel names its queue as a queue; both outbound destinations
send the payload as text and no application properties, so an MQ application reading the default
queues gets a plain MQSTR message:
channels:
- name: mq-in
transport: amqp
codec: json
cloudevents-mode: none
host: mqdemo-qm
tls: true
tls-ca-pem: secret:MQ_CA
username: env:MQ_USER
password: secret:MQ_PASSWORD
queue: ORDERS.IN
address-type: queue
- name: mq-out
direction: outbound
transport: amqp
bind: amqp10s://mqdemo-qm/ORDERS.OUT?type=queue&body=text&application-properties=none
cloudevents-mode: none
- name: mq-events
direction: outbound
transport: amqp
bind: amqp10s://mqdemo-qm/orders/events?type=topic&body=text&application-properties=none
cloudevents-mode: none
mq-relay/bpmn/mq-relay.bpmn answers each message, then announces it on the topic:
<?xml version="1.0" encoding="UTF-8"?>
<bpmn:definitions xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL"
xmlns:q="urn:sutra:q:1.0" id="Defs"
targetNamespace="urn:sutra:module:mq-relay:1.0.0">
<bpmn:process id="mq-relay" isExecutable="true">
<bpmn:extensionElements>
<q:variables>
<q:variable name="responseBody" type="object"/>
</q:variables>
</bpmn:extensionElements>
<bpmn:startEvent id="start">
<bpmn:extensionElements><q:source channel="mq-in"/></bpmn:extensionElements>
<bpmn:outgoing>f1</bpmn:outgoing>
</bpmn:startEvent>
<bpmn:sequenceFlow id="f1" sourceRef="start" targetRef="build"/>
<bpmn:serviceTask id="build">
<bpmn:incoming>f1</bpmn:incoming><bpmn:outgoing>f2</bpmn:outgoing>
<bpmn:dataInputAssociation>
<bpmn:assignment>
<bpmn:from>{ref: payload.ref, status: "ACCEPTED"}</bpmn:from>
<bpmn:to>responseBody</bpmn:to>
</bpmn:assignment>
</bpmn:dataInputAssociation>
</bpmn:serviceTask>
<bpmn:sequenceFlow id="f2" sourceRef="build" targetRef="answer"/>
<bpmn:sendTask id="answer">
<bpmn:extensionElements><q:send channel="mq-out"/></bpmn:extensionElements>
<bpmn:incoming>f2</bpmn:incoming><bpmn:outgoing>f3</bpmn:outgoing>
</bpmn:sendTask>
<bpmn:sequenceFlow id="f3" sourceRef="answer" targetRef="announce"/>
<bpmn:sendTask id="announce">
<bpmn:extensionElements><q:send channel="mq-events"/></bpmn:extensionElements>
<bpmn:incoming>f3</bpmn:incoming><bpmn:outgoing>f4</bpmn:outgoing>
</bpmn:sendTask>
<bpmn:sequenceFlow id="f4" sourceRef="announce" targetRef="end"/>
<bpmn:endEvent id="end"><bpmn:incoming>f4</bpmn:incoming></bpmn:endEvent>
</bpmn:process>
</bpmn:definitions>
mkdir -p deployments secrets
sutra lint mq-relay
sutra package mq-relay --out deployments
[WARN] SUTRA.CHANNEL.NO_SCHEMA — deployment default/mq-relay/1.0.0 (...): channel 'mq-in' has 1 subscriber(s) but its codec (json) declares no message types — ...
0 error(s), 1 warning(s)
packaged deployments/mq-relay.sutra (deploymentId dep-b95d811099f0245b07e2d9c1)
6. Start the engine. The inbound channel reads the password and the CA from the mounted secrets directory; the outbound sends take theirs from the environment.
printf '%s' "$MQ_PASSWORD" > secrets/MQ_PASSWORD
cp ca.pem secrets/MQ_CA
chmod -R a+rX deployments secrets
docker run -d --name mqdemo-db --network mqdemo-net -e POSTGRES_PASSWORD=dev-only postgres:16-alpine
until docker exec mqdemo-db pg_isready -U postgres -h 127.0.0.1; do sleep 1; done
sleep 2
docker exec mqdemo-db psql -U postgres \
-c "CREATE ROLE sutra_engine LOGIN PASSWORD 'dev-only' NOSUPERUSER NOBYPASSRLS" \
-c "ALTER SCHEMA public OWNER TO sutra_engine"
docker run -d --name mqdemo-engine --network mqdemo-net -p 127.0.0.1::8080 \
-v "$PWD/deployments:/deployments:ro" -e SUTRA_DEPLOYMENTS_DIR=/deployments \
-v "$PWD/secrets:/etc/sutra/secrets:ro" \
-v "$PWD/ca.pem:/etc/sutra/mq-ca.pem:ro" \
-e SUTRA_DATASOURCE_URL=postgres://mqdemo-db:5432/postgres \
-e SUTRA_DATASOURCE_USERNAME=sutra_engine -e SUTRA_DATASOURCE_PASSWORD=dev-only \
-e MQ_USER=app \
-e SUTRA_SINK_AMQP_USERNAME=app \
-e SUTRA_SINK_AMQP_PASSWORD="$MQ_PASSWORD" \
-e SUTRA_SINK_AMQP_TLS_CA_FILE=/etc/sutra/mq-ca.pem \
ghcr.io/startr-trade/sutra:0.8.2
PORT=$(docker port mqdemo-engine 8080 | head -1 | cut -d: -f2)
until curl -sf "http://127.0.0.1:$PORT/sutra/health/ready"; do sleep 1; done
{"status":"UP","checks":[{"name":"sutra-loader","status":"UP","data":{"deployments":1,"shards":1}}]}
7. Put a message as an MQ application would, and read the answer and the announcement.
docker exec mqdemo-qm bash -c \
"printf '%s\n\n' '{\"ref\":\"mq-1\"}' | /opt/mqm/samp/bin/amqsput ORDERS.IN QM1"
until [ "$(docker exec mqdemo-qm bash -c "echo \"DISPLAY QLOCAL('ORDERS.OUT') CURDEPTH\" | runmqsc QM1" | grep -o 'CURDEPTH([0-9]*)')" = "CURDEPTH(1)" ]; do sleep 1; done
docker exec mqdemo-qm /opt/mqm/samp/bin/amqsbcg ORDERS.OUT QM1 | grep -E "Format|CodedCharSetId"
docker exec mqdemo-qm /opt/mqm/samp/bin/amqsget ORDERS.OUT QM1
until [ "$(docker exec mqdemo-qm bash -c "echo \"DISPLAY QLOCAL('ORDERS.EVENTS.COPY') CURDEPTH\" | runmqsc QM1" | grep -o 'CURDEPTH([0-9]*)')" = "CURDEPTH(1)" ]; do sleep 1; done
docker exec mqdemo-qm /opt/mqm/samp/bin/amqsget ORDERS.EVENTS.COPY QM1
Sample AMQSPUT0 start
target queue is ORDERS.IN
Sample AMQSPUT0 end
Encoding : 273 CodedCharSetId : 1208
Format : 'MQSTR '
Sample AMQSGET0 start
message <{"ref":"mq-1","status":"ACCEPTED"}>
no more messages
Sample AMQSGET0 end
Sample AMQSGET0 start
message <{"ref":"mq-1","status":"ACCEPTED"}>
no more messages
Sample AMQSGET0 end
amqsput put an MQSTR message; the engine read it over the AMQP channel with TLS and SASL PLAIN and
answered on ORDERS.OUT as MQSTR in CCSID 1208, with no MQRFH2 in front of it. Its one publication
to orders/events reached ORDERS.EVENTS.COPY through the subscription.
8. Restart the queue manager and send another message once it is back:
docker restart mqdemo-qm
until [ "$(docker logs mqdemo-qm 2>&1 | grep -c 'Started queue manager')" -ge 2 ]; do sleep 1; done
docker exec mqdemo-qm bash -c \
"printf '%s\n\n' '{\"ref\":\"mq-2\"}' | /opt/mqm/samp/bin/amqsput ORDERS.IN QM1"
until [ "$(docker exec mqdemo-qm bash -c "echo \"DISPLAY QLOCAL('ORDERS.OUT') CURDEPTH\" | runmqsc QM1" | grep -o 'CURDEPTH([0-9]*)')" = "CURDEPTH(1)" ]; do sleep 1; done
docker exec mqdemo-qm /opt/mqm/samp/bin/amqsget ORDERS.OUT QM1
docker logs mqdemo-engine 2>&1 | grep -E 'OUTAGE_(STARTED|ENDED)'
Sample AMQSPUT0 start
target queue is ORDERS.IN
Sample AMQSPUT0 end
Sample AMQSGET0 start
message <{"ref":"mq-2","status":"ACCEPTED"}>
no more messages
Sample AMQSGET0 end
{"channel":"mq-in","code":"SUTRA.INBOUND.AMQP.OUTAGE_STARTED","outage_started":"2026-11-19T04:24:51.086333388Z",...,"level":"WARN",...,"message":"amqp consumer stream failed — reconnecting",...}
{"channel":"mq-in","code":"SUTRA.INBOUND.AMQP.OUTAGE_ENDED","outage_started":"2026-11-19T04:24:51.086333388Z","outage_ended":"2026-11-19T04:25:06.679943301Z","outage_ms":15593,...,"level":"INFO",...,"message":"amqp consumer reconnected after an outage",...}
The consumer lost the queue manager when it stopped and was attached again 15.6 s later, by itself. Part of that is the queue manager stopping and starting; the rest is the consumer’s backoff, which waits up to 30 s between attempts, so the gap differs from run to run (31.7 s in an earlier run). The engine’s log held the queue manager password 0 times.
9. Clean up.
docker rm -f mqdemo-engine mqdemo-db mqdemo-qm
docker network rm mqdemo-net
Broker outages as messages
Since 0.8.0. The engine logs every broker outage it sees (see
When the queue manager goes away). With
sutra.engine.connectivity-events.channel set, the same outage also reaches the deployment that
owns the channel, as a JSON message on a channel of that name, so a process can record its own
outage windows, alert, or hold work. The keys and the message are described in
Broker connectivity events.
This walkthrough runs an AMQP 1.0 broker (Apache ActiveMQ Artemis), a package that consumes one
queue and sends to another, and a small HTTP server that prints what the package records. It stops
the broker, sends while it is down, and starts it again. You need Docker, curl, the sutra CLI and
an empty working directory. It was run as written with the engine built from this release,
apache/activemq-artemis:latest-alpine, python:3.12-alpine and postgres:16-alpine; the output
shown is from that run.
1. Start the broker and a capture server
capture.py prints the body of every POST it receives:
from http.server import BaseHTTPRequestHandler, HTTPServer
class Capture(BaseHTTPRequestHandler):
def do_POST(self):
body = self.rfile.read(int(self.headers.get("Content-Length", 0)))
print(body.decode().strip(), flush=True)
self.send_response(204)
self.end_headers()
def log_message(self, *args):
pass
HTTPServer(("", 8000), Capture).serve_forever()
docker network create outages-net
docker run -d --name outages-broker --network outages-net \
-e ARTEMIS_USER=demo -e ARTEMIS_PASSWORD=demo-only \
apache/activemq-artemis:latest-alpine
until docker logs outages-broker 2>&1 | grep -q 'Server is now active'; do sleep 1; done
docker run -d --name outages-capture --network outages-net \
-v "$PWD/capture.py:/capture.py:ro" python:3.12-alpine python -u /capture.py
2. Write the package
outages/package.yaml:
labels:
"tenant": "default"
"module": "outages"
"version": "1.0.0"
engine:
minContract: 1
outages/channels.yaml. orders-in and orders-out are the AMQP channels whose outages are
reported. connectivity-in is the channel the events arrive on: an in-process intake with the JSON
codec. record-out sends each event to the capture server:
channels:
- name: orders-in
transport: amqp
codec: json
cloudevents-mode: none
host: outages-broker
username: env:BROKER_USER
password: env:BROKER_PASSWORD
queue: orders.in
- name: orders-out
direction: outbound
transport: amqp
bind: amqp10://outages-broker/orders.out
cloudevents-mode: none
- name: emit
transport: http
bind: "POST /channels/emit"
codec: json
auth:
scheme: apikey
apikey:
value: env:OUTAGES_API_KEY
header: X-Api-Key
- name: connectivity-in
transport: local
codec: json
- name: record-out
direction: outbound
transport: http
bind: "http://outages-capture:8000/connectivity"
outages/templates/out.hbs:
{"n":"{{payload.n}}"}
outages/templates/record.hbs copies the event’s fields:
{"code":"{{payload.code}}","channel":"{{payload.channel}}","direction":"{{payload.direction}}","state":"{{payload.state}}","startedAt":"{{payload.startedAt}}"{{#if payload.endedAt}},"endedAt":"{{payload.endedAt}}","ms":{{payload.ms}}{{/if}}}
outages/bpmn/outages.bpmn:
<?xml version="1.0" encoding="UTF-8"?>
<bpmn:definitions xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL"
xmlns:q="urn:sutra:q:1.0"
targetNamespace="urn:sutra:module:outages:1.0.0">
<!-- consumes orders-in, so the intake is attached to the broker -->
<bpmn:process id="listen" isExecutable="true">
<bpmn:startEvent id="S1">
<bpmn:extensionElements><q:source channel="orders-in" name="payload"/></bpmn:extensionElements>
</bpmn:startEvent>
<bpmn:sequenceFlow id="l1" sourceRef="S1" targetRef="E1"/>
<bpmn:endEvent id="E1"/>
</bpmn:process>
<!-- POST /channels/emit sends one message on orders-out -->
<bpmn:process id="emit" isExecutable="true">
<bpmn:startEvent id="S2">
<bpmn:extensionElements><q:source channel="emit" name="payload"/></bpmn:extensionElements>
</bpmn:startEvent>
<bpmn:sequenceFlow id="e1" sourceRef="S2" targetRef="Send"/>
<bpmn:serviceTask id="Send" implementation="out.hbs">
<bpmn:extensionElements>
<q:send channel="orders-out" mode="native" contentType="application/json"/>
</bpmn:extensionElements>
</bpmn:serviceTask>
<bpmn:sequenceFlow id="e2" sourceRef="Send" targetRef="E2"/>
<bpmn:endEvent id="E2"/>
</bpmn:process>
<!-- each connectivity event starts an instance, which records it -->
<bpmn:process id="record" isExecutable="true">
<bpmn:startEvent id="S3">
<bpmn:extensionElements><q:source channel="connectivity-in" name="payload"/></bpmn:extensionElements>
</bpmn:startEvent>
<bpmn:sequenceFlow id="r1" sourceRef="S3" targetRef="Record"/>
<bpmn:serviceTask id="Record" implementation="record.hbs">
<bpmn:extensionElements>
<q:send channel="record-out" mode="native" contentType="application/json"/>
</bpmn:extensionElements>
</bpmn:serviceTask>
<bpmn:sequenceFlow id="r2" sourceRef="Record" targetRef="E3"/>
<bpmn:endEvent id="E3"/>
</bpmn:process>
</bpmn:definitions>
sutra lint outages
mkdir -p deployments
sutra package outages --out deployments
chmod -R a+rX deployments
Lint reports 0 errors and 11 warnings: three SUTRA.CHANNEL.NO_SCHEMA and eight
SUTRA.CONFIG.TEMPLATE.FIELD_UNVERIFIABLE, because the JSON codec declares no schema.
3. Start the engine with the events channel set
SUTRA_CONNECTIVITY_EVENTS_CHANNEL=connectivity-in names the channel. A deployment that declares
no channel of that name receives nothing.
docker run -d --name outages-db --network outages-net -e POSTGRES_PASSWORD=dev-only postgres:16-alpine
until docker exec outages-db pg_isready -U postgres -h 127.0.0.1; do sleep 1; done
sleep 2
docker exec outages-db psql -U postgres \
-c "CREATE ROLE sutra_engine LOGIN PASSWORD 'dev-only' NOSUPERUSER NOBYPASSRLS" \
-c "ALTER SCHEMA public OWNER TO sutra_engine"
docker run -d --name outages-engine --network outages-net -p 127.0.0.1::8080 \
-v "$PWD/deployments:/deployments:ro" -e SUTRA_DEPLOYMENTS_DIR=/deployments \
-e SUTRA_DATASOURCE_URL=postgres://outages-db:5432/postgres \
-e SUTRA_DATASOURCE_USERNAME=sutra_engine -e SUTRA_DATASOURCE_PASSWORD=dev-only \
-e SUTRA_CONNECTIVITY_EVENTS_CHANNEL=connectivity-in \
-e BROKER_USER=demo -e BROKER_PASSWORD=demo-only \
-e SUTRA_SINK_AMQP_USERNAME=demo -e SUTRA_SINK_AMQP_PASSWORD=demo-only \
-e OUTAGES_API_KEY=demo-key \
ghcr.io/startr-trade/sutra:0.8.3
PORT=$(docker port outages-engine 8080 | head -1 | cut -d: -f2)
until curl -sf "http://127.0.0.1:$PORT/sutra/health/ready"; do sleep 1; done
{"status":"UP","checks":[{"name":"sutra-loader","status":"UP","data":{"deployments":1,"shards":1}}]}
Send one message while the broker is up, so the send channel has connected. Nothing is recorded:
curl -s -o /dev/null -w '%{http_code}\n' -X POST "localhost:$PORT/channels/emit" -H 'X-Api-Key: demo-key' \
-H 'Content-Type: application/json' -d '{"n":"1"}'
sleep 3
docker logs outages-capture
200
4. Stop the broker, send during the outage, start it again
docker stop outages-broker
until docker logs outages-capture 2>&1 | grep -q '"orders-in".*"down"'; do sleep 1; done
curl -s -o /dev/null -w '%{http_code}\n' -X POST "localhost:$PORT/channels/emit" -H 'X-Api-Key: demo-key' \
-H 'Content-Type: application/json' -d '{"n":"2"}'
until docker logs outages-capture 2>&1 | grep -q '"orders-out".*"down"'; do sleep 1; done
sleep 5
docker start outages-broker
until [ "$(docker logs outages-capture 2>&1 | grep -c '"up"')" -ge 2 ]; do sleep 1; done
docker logs outages-capture
The send during the outage is accepted (200): it waits in the outbox and goes out once the broker
is back. What the package recorded:
{"code":"SUTRA.INBOUND.AMQP.OUTAGE_STARTED","channel":"orders-in","direction":"inbound","state":"down","startedAt":"2026-11-19T09:15:09.146540438Z"}
{"code":"SUTRA.OUTBOUND.AMQP.OUTAGE_STARTED","channel":"orders-out","direction":"outbound","state":"down","startedAt":"2026-11-19T09:15:09.521903086Z"}
{"code":"SUTRA.OUTBOUND.AMQP.OUTAGE_ENDED","channel":"orders-out","direction":"outbound","state":"up","startedAt":"2026-11-19T09:15:09.521903086Z","endedAt":"2026-11-19T09:15:21.185206554Z","ms":11663}
{"code":"SUTRA.INBOUND.AMQP.OUTAGE_ENDED","channel":"orders-in","direction":"inbound","state":"up","startedAt":"2026-11-19T09:15:09.146540438Z","endedAt":"2026-11-19T09:15:24.169169142Z","ms":15022}
Two windows, one per channel. The intake’s started when it lost its connection and ended when it
was attached again; the send channel’s started at the send that could not connect and ended at the
outbox’s retry that reached the broker. Each up repeats its startedAt, so the two messages of a
window pair up. The engine’s own log has the same four changes with the same codes and times:
docker logs outages-engine 2>&1 | grep -E 'OUTAGE_(STARTED|ENDED)'
{"channel":"orders-in","code":"SUTRA.INBOUND.AMQP.OUTAGE_STARTED","outage_started":"2026-11-19T09:15:09.146540438Z",...,"level":"WARN",...}
{"code":"SUTRA.OUTBOUND.AMQP.OUTAGE_STARTED","broker":"outages-broker:5672","destination":"amqp10://outages-broker/orders.out","outage_started":"2026-11-19T09:15:09.521911293Z",...,"level":"WARN",...}
{"code":"SUTRA.OUTBOUND.AMQP.OUTAGE_ENDED","broker":"outages-broker:5672","destination":"amqp10://outages-broker/orders.out","outage_started":"2026-11-19T09:15:09.521911293Z","outage_ended":"2026-11-19T09:15:21.185210334Z","outage_ms":11663,...,"level":"INFO",...}
{"channel":"orders-in","code":"SUTRA.INBOUND.AMQP.OUTAGE_ENDED","outage_started":"2026-11-19T09:15:09.146540438Z","outage_ended":"2026-11-19T09:15:24.169169142Z","outage_ms":15022,...,"level":"INFO",...}
How long each window lasts depends on the broker’s restart and on the intake’s reconnect backoff, which waits up to 30 s between attempts, so it differs from run to run.
5. Clean up
docker rm -f outages-engine outages-db outages-broker outages-capture
docker network rm outages-net
Tuning for throughput
Since 0.6.5. This page shows how to measure what one engine and one PostgreSQL complete per
second on your hardware, and which settings move it, without giving up durability: every number
here was taken with fsync, synchronous_commit and full_page_writes on.
What limits a durable engine
An instance’s state is safe only once PostgreSQL has flushed its write-ahead log to disk, and a disk flushes a limited number of times per second. A consumer NVMe drive manages about 300; a server SSD with a power-loss-protected cache manages tens of thousands. Measure yours with PostgreSQL’s own tool:
/usr/lib/postgresql/18/bin/pg_test_fsync -s 2 -f /path/on/the/database/disk/testfile
What decides throughput is how many commits share each flush. The engine helps in three ways:
- it commits only what must be durable before anything is acknowledged: a step’s state, its aliases, its waits and its outbound messages, in one transaction. Bookkeeping that a crash cannot hurt — an ownership claim, the outbox’s row locks, the delete of an already delivered row — does not wait for its own flush, and the inbox dedup record rides the step’s own transaction;
- it delivers outbound messages on several workers at once (
sutra.outbox.workers), deletes delivered rows in batches, and handslocal://deliveries straight from the sender’s commit to a worker; - with several execution lanes (
sutra.engine.shards), several steps commit at the same time, and PostgreSQL flushes them together.
So on a slow disk, add lanes: each lane is one more commit that can share a flush.
What may skip its own flush, and why
The server keeps synchronous_commit = on. Three kinds of transaction set
synchronous_commit = off for themselves alone, and only because a crash that loses them can
neither lose an acknowledged state nor invent one:
| Transaction | What it writes | Why it may skip its flush |
|---|---|---|
| instance ownership claim | which engine is resuming an instance | a claim held by an engine that crashed is void anyway; the step that follows commits synchronously |
| outbox claim and its lease renewal | row locks, and a lease: the claimed rows’ due time moved 30 s ahead | a lost lease only makes the rows claimable again, so they are sent again: at-least-once, deduplicated on the idempotency key |
| delete of delivered outbox rows | removes rows already sent | if it is lost, the row is sent again: outbound delivery has always been at-least-once, and receivers deduplicate on the idempotency key |
The inbox dedup record of a delivery is not on the list: it is written in the same transaction as the step that processes the delivery, so it exists exactly when the step’s effects do.
Everything else — every step’s state, aliases, waits and outbound rows, every data-store write, every retry or poison mark — commits synchronously, as before. The tier-2 suite kills the engine, and separately the database, in the middle of a load and checks that every accepted request still completes exactly once.
The workload
The repository carries a workload that exercises the whole durable path, in
tools/sutra-load-test:
fixtures/relay/— two processes.relay-intaketakes an order over HTTP, validates it with a small.srlruleset, reads two rows from a data store, forwards it over a local channel torelay-worker, waits for the worker’s answer on a second local channel, and calls back an HTTP endpoint. Three durable hops per order.profiles/relay.js— k6 posts orders at a constant rate.sink/— a small HTTP server that receives the callbacks and measures each order’s latency from the moment k6 sent it.relay-bench.sh— starts a throwaway PostgreSQL cluster (durable settings) on the disk you run it from, the sink and the engine, warms up, runs k6, waits until every order has called back, and prints one line of results plus the busiest SQL statements.
It needs k6, jq, PostgreSQL’s server binaries (PG_BIN_DIR, default
/usr/lib/postgresql/18/bin) and an optimised build of the engine and the CLI. The runs below
used the faster-to-build release-it profile:
(cd rust && cargo build --profile release-it -p sutra-dist -p sutra-cli)
export SUTRA_ENGINE_BIN=$PWD/rust/target/release-it/sutra-engine
export SUTRA_CLI=$PWD/rust/target/release-it/sutra
The script refuses to put the database on tmpfs, where a flush costs nothing and the numbers
would mean nothing.
Step 1: the defaults
RATE=100 DURATION=60s tools/sutra-load-test/relay-bench.sh
[relay-bench] building the sink
[relay-bench] warm-up 10/s x 10s
[relay-bench] measured run 100/s x 60s
| offered | sent | settled | dup | drained | settles/s | settle p50 | settle p99 | intake p50 | intake p99 | commits/settle | WAL flushes/settle | WAL KB/settle | WAL fsync busy | engine CPU (cores) | PG CPU (cores) | CPU ms/settle (engine+PG) | engine env |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 100/s x 60s | 5167 | 5167 | 0 | yes | 70.1 | 6729 ms | 13395 ms | 6708 ms | 10001 ms | 16.0 | 3.01 | 10.2 | 63% | 0.24 | 0.25 | 7.09 | `defaults` |
Top statements by calls (calls | total ms | query):
72550 | 30 | BEGIN
72550 | 24 | COMMIT
46805 | 187 | SELECT set_config($1, $2, $3)
25745 | 140 | SELECT set_config($1, $2, $3), set_config($4, $5, $6)
15501 | 402 | INSERT INTO outbox_entry (entry_id, deployment_id, instance_id, body, content_type, destination, headers_json,
15396 | 332 | DELETE FROM outbox_entry WHERE deployment_id = $1 AND entry_id = ANY($2)
10334 | 118 | SELECT instance_id FROM alias_index WHERE deployment_id = $1 AND alias_name = $2 AND alias_value = $3 AND live
10334 | 144 | SELECT store_value, rev FROM data_store WHERE store_name = $1 AND store_key = $2
One lane executes one step at a time, so its commits never share a flush; on this disk that caps
the engine near 70 orders per second, and the rest of the offered load queues at
intake (intake p99). settled can exceed sent: k6 gives up on a request after 10 seconds
and does not count it, but the intake had already accepted it, and it completes like any other.
Step 2: lanes, and a pool to match
Each lane holds a database connection while it commits, so raise the main pool with the lanes:
RATE=100 DURATION=60s ENGINE_ENV="SUTRA_ENGINE_SHARDS=8" tools/sutra-load-test/relay-bench.sh
[relay-bench] warm-up 10/s x 10s
[relay-bench] measured run 100/s x 60s
| offered | sent | settled | dup | drained | settles/s | settle p50 | settle p99 | intake p50 | intake p99 | commits/settle | WAL flushes/settle | WAL KB/settle | WAL fsync busy | engine CPU (cores) | PG CPU (cores) | CPU ms/settle (engine+PG) | engine env |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 100/s x 60s | 6001 | 6001 | 0 | yes | 99.8 | 20 ms | 72 ms | 5 ms | 26 ms | 15.9 | 2.81 | 10.2 | 87% | 0.35 | 0.37 | 7.31 | `SUTRA_ENGINE_SHARDS=8` |
Then the target rate:
RATE=500 DURATION=60s \
ENGINE_ENV="SUTRA_ENGINE_SHARDS=32 SUTRA_DATASOURCE_MAX_CONNECTIONS=40" \
tools/sutra-load-test/relay-bench.sh
[relay-bench] warm-up 10/s x 10s
[relay-bench] measured run 500/s x 60s
| offered | sent | settled | dup | drained | settles/s | settle p50 | settle p99 | intake p50 | intake p99 | commits/settle | WAL flushes/settle | WAL KB/settle | WAL fsync busy | engine CPU (cores) | PG CPU (cores) | CPU ms/settle (engine+PG) | engine env |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 500/s x 60s | 30001 | 30001 | 0 | yes | 499.1 | 31 ms | 185 ms | 8 ms | 52 ms | 14.1 | 0.57 | 10.2 | 89% | 1.79 | 2.01 | 7.69 | `SUTRA_ENGINE_SHARDS=32 SUTRA_DATASOURCE_MAX_CONNECTIONS=40` |
Read the columns this way:
| Column | Meaning |
|---|---|
settles/s | orders that called back, per second, from the start of the run to the last callback |
settle p50 / p99 | time from k6 sending the order to the sink receiving the callback |
intake p50 / p99 | time for the intake to answer 202 |
commits/settle | database transactions per order, read-only ones included |
WAL flushes/settle | disk flushes per order — below 1 means commits share flushes |
WAL fsync busy | share of the run the disk spent flushing |
engine CPU | cores the engine used |
PG CPU | cores PostgreSQL’s backends used (since 0.6.6) |
CPU ms/settle | engine plus PostgreSQL CPU time per order, in milliseconds — the number to watch when the host, not the disk, is the limit (since 0.6.6) |
When WAL fsync busy is near 100% and adding lanes no longer raises settles/s, the disk is the
limit: a faster disk, or PostgreSQL’s commit_delay (which makes a flush wait a few microseconds
for more commits to join it, still durable), are what move it next. When the disk is not busy but
every core is, the CPU is the limit: CPU ms/settle times the rate you want is the core count it
takes. 0.6.6 cut it by about a fifth on this workload: with SUTRA_OUTBOX_WORKERS=32 SUTRA_OUTBOX_TICK_INTERVAL=PT0.01S added to the last run, 8.05 ms on 0.6.5 and 6.33 ms on 0.6.6.
The settings
| Setting | Default | Raise it when |
|---|---|---|
sutra.engine.shards | 1 | the disk is busy less than the time it takes to flush, and orders queue at intake. Read Execution lanes first: lanes run different instances in parallel. |
sutra.datasource.max-connections | 8 | you raise the lanes: keep it at least the lane count, plus a few |
sutra.datasource.dispatch-max-connections | 4 | rarely; the dispatcher’s claim and its batched delete each hold one connection, the rest serve the defers of failed sends |
sutra.outbox.workers | 32 | many destinations are slow at once: it bounds the sends in flight across all of them |
sutra.outbox.destination-concurrency | 32 | one destination is slow and can take more parallel requests; see Outbound bursts |
sutra.outbox.max-buffered | 4096 | sutra.outbox.buffered sits at the bound during a burst and memory allows |
sutra.outbox.batch-size | 50 | a deployment enqueues far more than 50 rows between claims |
sutra.outbox.tick-interval | PT5S | you want retries and rows from other replicas picked up sooner; local deliveries do not wait for it |
cache: on a data store | off | a store is read on every request and changed rarely — see Caching reads |
Outbound bursts
Each process instance’s outbound rows form one ordered chain: they leave one after another, in
the order they were enqueued, while different instances’ chains send in parallel. Two bounds
apply to a send, taken in this order: one per destination authority (scheme, host and port;
sutra.outbox.destination-concurrency) and one across every destination
(sutra.outbox.workers). A row waiting for a busy destination holds no global slot, so a slow
counterparty delays only its own rows. The claimer stops claiming only when
sutra.outbox.max-buffered rows are held in memory.
Three instruments show whether outbound delivery keeps up:
sutra.outbox.delivery.lag(histogram, milliseconds, byscheme): enqueue to delivery, once per delivered row. Its p99 is the number to watch.sutra.outbox.in-flight(gauge): atsutra.outbox.workersfor long, the global bound is the limit.sutra.outbox.buffered(gauge): atsutra.outbox.max-buffered, the claimer is waiting.
The defaults come from a burst bench: tools/sutra-load-test/relay-bench.sh BURST=3500 enqueues
3,500 rows in about 6 s across two HTTP destinations, one answering at once and one after
500 ms (30% of instances), on durable PostgreSQL. Lag is measured by the receiving end, against
the instant the engine built each request.
| Engine | workers / destination-concurrency | Drain | Fast lag p50 / p99 / max | Slow lag p99 | Slow rows/s |
|---|---|---|---|---|---|
| 0.8.2 | 8 workers by instance | 84.5 s | 35.9 s / 73.9 s / 78.9 s | 73.6 s | 12.5 |
| 0.8.3 | 8 / 8 | 65.5 s | 6.3 s / 18.8 s / 19.1 s | 59.1 s | 16.1 |
| 0.8.3 | 16 / 8 | 65.5 s | 5 ms / 7 ms / 10 ms | 59.8 s | 16.0 |
| 0.8.3 | 16 / 16 | 32.8 s | 172 ms / 666 ms / 864 ms | 27.6 s | 32.1 |
| 0.8.3 | 32 / 16 | 32.9 s | 5 ms / 11 ms / 17 ms | 27.5 s | 32.1 |
| 0.8.3 | 32 / 32 (default) | 16.5 s | 25 ms / 166 ms / 187 ms | 12.3 s | 64.2 |
| 0.8.3 | 64 / 16 | 32.9 s | 4 ms / 7 ms / 9 ms | 27.8 s | 32.0 |
| 0.8.3 | 64 / 32 | 16.5 s | 4 ms / 7 ms / 15 ms | 12.3 s | 64.2 |
| 0.8.3 | 128 / 32 | 16.5 s | 5 ms / 8 ms / 11 ms | 12.2 s | 64.2 |
Peak PostgreSQL connections stayed at 8–9 in every run.
The defaults are the smallest pair whose fast-destination p99 stays under 2 s and whose drain
time is within 10% of the best. The slow destination drains at about
destination-concurrency / its latency; the fast one is only held back once the slow one can take
every global slot (8 / 8 above).
Watch outbound lag under a burst
This walkthrough runs the burst bench twice against a local OpenTelemetry Collector, first with a
global bound so small that the slow destination takes all of it, then with the defaults, and
reads sutra.outbox.delivery.lag from the collector each time. It needs Docker, k6, PostgreSQL
18’s server binaries and a release build of the engine and CLI (cargo build --profile release-it -p sutra-dist -p sutra-cli).
Step 1: start a collector that receives OTLP and exposes the metrics for Prometheus:
cat > /tmp/otelcol.yaml <<'YAML'
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
exporters:
prometheus:
endpoint: 0.0.0.0:8889
service:
pipelines:
metrics:
receivers: [otlp]
exporters: [prometheus]
YAML
docker run -d --name walk-otelcol -p 127.0.0.1::4317 -p 127.0.0.1::8889 \
-v /tmp/otelcol.yaml:/etc/otelcol-contrib/config.yaml:ro otel/opentelemetry-collector-contrib
OTLP=$(docker port walk-otelcol 4317/tcp | head -1)
PROM=$(docker port walk-otelcol 8889/tcp | head -1)
Step 2: a burst with the slow destination holding the fast one. Eight sends in flight in total and eight per destination: the slow destination can take every slot.
export SUTRA_ENGINE_BIN=rust/target/release-it/sutra-engine SUTRA_CLI=rust/target/release-it/sutra
BURST=3500 ENGINE_ENV="SUTRA_TELEMETRY_OTLP_ENDPOINT=http://$OTLP \
SUTRA_TELEMETRY_METRICS_EXPORT_INTERVAL=1000 SUTRA_TELEMETRY_SERVICE_NAME=walk-8 \
SUTRA_OUTBOX_WORKERS=8 SUTRA_OUTBOX_DESTINATION_CONCURRENCY=8" \
tools/sutra-load-test/relay-bench.sh
curl -s "http://$PROM/metrics" | grep 'sutra_outbox_delivery_lag_milliseconds_bucket{job="walk-8"' \
| grep -E 'le="(100|1000|10000|\+Inf)"'
The bench’s block shows the fast destination waiting: on our run its lag was 6.5 s / 17.5 s / 17.7 s (p50 / p99 / max), and the histogram has 41 of the 3,500 rows within 100 ms, 231 within a second and 1,746 within ten.
Step 3: the same burst with the defaults (32 / 32):
BURST=3500 ENGINE_ENV="SUTRA_TELEMETRY_OTLP_ENDPOINT=http://$OTLP \
SUTRA_TELEMETRY_METRICS_EXPORT_INTERVAL=1000 SUTRA_TELEMETRY_SERVICE_NAME=walk-default" \
tools/sutra-load-test/relay-bench.sh
curl -s "http://$PROM/metrics" | grep 'sutra_outbox_delivery_lag_milliseconds_bucket{job="walk-default"' \
| grep -E 'le="(100|1000|10000|\+Inf)"'
The fast destination no longer waits: 26 ms / 165 ms / 184 ms, and 2,139 rows land within
100 ms. The rows still over a second are the slow destination’s own (1,048 of them, drained at
64 rows/s), and the whole burst drains in 16.5 s instead of 66.7 s. The histogram carries only
the scheme attribute, so both HTTP destinations share one series; the per-destination split
comes from the bench’s sink.
Step 4: clean up. docker rm -f walk-otelcol. The bench stops its own engine, sinks and
database.
PostgreSQL autovacuum
Several engine tables churn at the request rate:
outbox_entrygets one insert and one delete per delivery;instance_stateis rewritten on every step;waiting_event,alias_indexandinbox_seengain and lose rows as instances park, finish and age out.
Each delete or rewrite leaves a dead row version, and the engine’s hot queries step over those until a vacuum removes them. The claim of due outbox rows is the one that suffers most.
PostgreSQL’s defaults vacuum a table once 20% of it is dead and look for such tables once a minute. That is too late for a queue: it stays small while it produces thousands of dead rows a second. One measured run at about 1,700 requests a second used the defaults. Every two and a half minutes the outbox claim took 1–1.6 s, throughput fell below 100 a second for some 40 s, and p99 latency reached 38 s.
Since 0.6.8 the engine’s migrations set per-table values. You need do nothing for them:
| Table | Vacuum after | Cost throttle |
|---|---|---|
outbox_entry | 5,000 dead rows, whatever the table’s size (autovacuum_vacuum_scale_factor = 0, autovacuum_vacuum_threshold = 5000) | none (autovacuum_vacuum_cost_delay = 0); the table is small |
instance_state, waiting_event, alias_index, inbox_seen | 1% of the table plus 5,000 rows (0.01, 5000) | the server’s setting; these tables can be large |
Statistics are refreshed on the same kind of trigger. At low rates 5,000 dead rows take a long time to accumulate, so these settings cost nothing there.
One setting is server-wide: autovacuum_naptime. It sets how often autovacuum wakes to look,
and a table cannot override it. For a deployment running hundreds of requests a second or more,
lower it in postgresql.conf, or in your managed service’s database flags:
autovacuum_naptime = 1s # default 1min
autovacuum_vacuum_cost_limit = 2000 # default 200 (via vacuum_cost_limit); lets the larger tables keep up
The run above, repeated with naptime 1s and an unthrottled global autovacuum, went on for ten
minutes at 2,500 requests a second with p99 329 ms and no slow statements. A busy autovacuum worker
uses some CPU and I/O, so watch both after the change.
MySQL/MariaDB and SQL Server have no per-table equivalent. InnoDB purge and SQL Server’s ghost cleanup are server-wide, and the engine ships no setting for them.
Profiling
PERF_DIR=/some/dir records a CPU profile of the engine and of the database backends over the
measured run, and writes a flamegraph of each plus a list of the hottest functions. It needs
perf usable by your user (kernel.perf_event_paranoid of 1 or less) and
inferno (cargo install inferno). Build the engine with frame
pointers so the stacks are whole:
(cd rust && RUSTFLAGS="-C force-frame-pointers=yes" cargo build --profile release-it -p sutra-dist)
PERF_DIR=/tmp/relay-profile RATE=100 DURATION=30s tools/sutra-load-test/relay-bench.sh
On the run above the engine used well under one core; most of its time went to the database protocol, and the rest of the wall clock was the disk.
Limits and quotas
Engine-wide ceilings, each with a safe shipped default and a consistent override shape — the same
canonical sutra.* / SUTRA_* pair described in
Configuration reference.
Inbound payload byte cap
Every inbound channel — broker or HTTP — delivers raw bytes to the engine before a codec ever
parses them, and a producer can publish an arbitrarily large payload. The cap closes that gap: a
message is byte-length-checked before codec.decode() runs, so nothing is allocated or parsed
for an oversized message.
| Config | Default | Disabled value |
|---|---|---|
sutra.codec.max-payload-bytes (SUTRA_CODEC_MAX_PAYLOAD_BYTES) | 10 MiB (10485760) | 0 |
A per-channel override lives on the channel’s own definition (not as a separate engine-wide property), and is not clamped by the global value — it can raise or lower the effective cap for just that channel:
channels:
- name: bulk-statements-acme
payload-cap-bytes: 104857600 # raised for a trusted large-file channel
- name: heartbeat-acme
payload-cap-bytes: 4096 # tightened for a strict small-event channel
Negative values are rejected at startup (SUTRA.CONFIG.PROPERTY.INVALID) — this catches a typo
like -1 before it silently disables the cap the way some other libraries interpret that value.
On rejection. Exceeding the effective cap rejects the message with
SUTRA.INBOUND.PAYLOAD_TOO_LARGE (ERROR), carrying the channel id, the actual payload size, and
the effective cap that was applied — enough for an operator to know exactly which config key to
raise. This is a permanent reject, not a retryable one:
| Transport | Rejection translates to |
|---|---|
| HTTP | 413 |
| RabbitMQ / AMQP | basic_nack with requeue=false → the broker’s DLX (or dropped if none configured) |
| Kafka | Offset committed — the poison record is skipped, not replayed (wire a dead-letter topic in BPMN if you need one) |
| AWS SQS | DeleteMessage — removed, not redelivered |
| GCP Pub/Sub | message.ack() — removed, not redelivered |
| File | The source file is moved to the failed/ sub-directory |
| Dapr | 500 to the sidecar |
| Knative | 500 to the Broker |
Dapr and Knative are the exception to “permanent” today. Both answer this rejection — and every
other intake rejection except a malformed CloudEvent — with 500, and both pushers treat 500 as
retryable: the Dapr sidecar retries under the component’s retry policy, and a Knative Broker
redelivers until the Trigger’s delivery retries are spent. The same oversized message therefore
arrives again until the pusher gives up. Configure a dead-letter topic on the Dapr subscription, or a
deadLetterSink on the Trigger, so that it ends somewhere visible.
Per-tenant quotas
Two more admission checks, enforced before a message reaches the executor — see Multi-tenancy and isolation for the full detail:
maxConcurrentInstances— a hard cap on simultaneously in-flight instances for a tenant, coherent across every replica.maxInboundRatePerMinute— a per-replica sliding 60-second admission window.
Neither is applied unless a tenant opts in — an unconfigured tenant is unlimited on both dimensions.
What’s not yet a configurable limit
The threat-model backlog names a few more ceilings that aren’t wired yet — a FEEL evaluation wall-clock/memory budget per expression, and a per-tenant audit-write rate cap. Until they land, the payload cap and the two tenant quotas above are the complete set of per-message and per-tenant limits. Limits on a shared physical resource (lines, seats, connections) are capacity pools.
Next
- Configuration reference — the full
sutra.*/SUTRA_*key map these limits live in. - Troubleshooting BPMN solutions — what a rejected-message diagnostic looks like in practice, and how to trace it back to the message that triggered it.
Capacity pools and sessions
Some limits are not per channel or per tenant but physical: a number of lines, licences, seats or connections to a downstream system, shared by every deployment that uses them. 0.6 adds two things for that:
- Capacity pools: named, cluster-wide counts of slots, declared in the engine configuration,
which inbound and outbound channels of any deployment draw from with
pool: <name>. session-end:: an inbound channel can end an instance’s session (give back its slots) before the instance reaches its end, while the instance runs on.
Active is not incomplete
max-concurrent-instances on an inbound channel caps how many of the instances it admitted may
be active at once. With use-only-in-flight-for-concurrency-cap: false, a parked instance
(waiting on a user task, a model job, a reply) still counts. Normally an instance stops counting
when it ends. session-end: lets it stop earlier, for a flow whose session (a connection, a
conversation, a line) can finish before its follow-up work does:
channels:
- name: session-start
transport: http
bind: "POST /sessions"
max-concurrent-instances: 200
use-only-in-flight-for-concurrency-cap: false
pool: lines # optional: also draw from a capacity pool
pool-call-key: header.x-call-id # optional: which call a message belongs to
session-end:
on-message: session-closed # a correlated message on this channel ends the session
on-reply: true # and/or: the flow's reply on session-start ends it
- name: session-closed
transport: http
bind: "POST /sessions/closed"
| Key | Meaning |
|---|---|
session-end.on-message | Another inbound channel of the same file. A message on it is correlated to an instance through the <q:alias onConflict="correlate"> of the process that starts on the session channel. When the session channel declares pool-call-key: header.<name> and the message carries that header, only the call it names is ended; otherwise the instance’s whole session is released. If a wait in the flow subscribes to the channel too, the message is also delivered as usual. If none does, the message is accepted for the release alone, and a message that matches no live instance is accepted and does nothing. |
session-end.on-reply | true: the flow’s <q:reply> on the session channel (including continue="true") ends the replying instance’s inbound call: its channel slot and the pool slot that call took. Pool slots the instance holds for outbound calls stay. |
A release is idempotent, and nothing takes a released slot back: not a later park, not a resume,
not an outbound request. A request the instance put in the outbox that is still waiting for a
slot when its session ends (or its call, for a call-scoped end), or one it sends afterwards, is
still sent, without a slot.
End, failure or a cancel still release whatever nothing released earlier. A malformed block (on
an outbound channel, an unknown key, an on-message that is not another inbound channel of the
file) fails to load with SUTRA.CONFIG.CHANNEL.SESSION_END_INVALID.
Capacity pools
A pool is engine configuration:
| Key | Default | Meaning |
|---|---|---|
sutra.capacity-pools | — | Comma-separated pool names; the way to declare a pool purely by environment variables. |
sutra.capacity-pools.<name>.slots | required | Total slots. At least 1. |
sutra.capacity-pools.<name>.reserve.inbound | 0 | Slots outbound may never take while inbound holds fewer than this many. |
sutra.capacity-pools.<name>.reserve.outbound | 0 | The same, kept for outbound. The two reserves together may not exceed slots. |
sutra.capacity-pools.<name>.tenants | every tenant | Comma-separated allow-list of tenants whose channels may name the pool. |
The env vars are SUTRA_CAPACITY_POOLS_<NAME>_SLOTS, …_RESERVE_INBOUND, …_RESERVE_OUTBOUND and
…_TENANTS, with <NAME> upper-cased and - written as _. Pool names are lowercase letters,
digits and dashes.
sutra.capacity-pools.lines.slots=30
sutra.capacity-pools.lines.reserve.inbound=6
sutra.capacity-pools.lines.tenants=default
The count is kept in the engine database (the capacity_pool and capacity_slot tables), so
every replica sees the same pool. Without a datasource it is counted per process. Stranded slots
(an instance that finished without releasing, a replica that died mid-call) are returned by the
instance sweep (sutra.instance.sweep-interval).
A channel draws from a pool with pool: <name>:
- Inbound. A new instance takes a slot for its call before its first step runs, or is
rejected as busy with
SUTRA.INBOUND.POOL.EXHAUSTED(HTTP 503), and no instance is created. A relay or a timer that resumes an existing instance takes no slot. - Outbound. A request (a
<q:send>or a channel call) on a pooled channel takes a slot when the outbox sends it. While the pool is full the request waits. It is put back without counting a delivery attempt (its last diagnostic readsSUTRA.OUTBOUND.POOL.WAITING) and re-checked on every outbox tick, oldest first. The wait never blocks the flow, and a channel call’s<q:timeout>still fires. A channel-call request still waiting when its instance has finished is abandoned rather than sent (SUTRA.OUTBOUND.POOL.ABANDONED). A pooled outbound channel is never synchronous:ack-mode: on-completeon it takes the outbox path, since only the outbox can hold a request back without holding the caller’s connection.
One slot per active call. An instance holds a slot for one call, identified by its key: on an
inbound start, the value of the channel’s pool-call-key header, or else the delivery’s own
identity; on an outbound request, the outbound channel’s pool-call-key header, or else the
outbox entry’s identity. So an instance that bridges its caller to a second party holds two slots,
one per leg, and requests that carry the same call key share one slot. A call’s slot is released
when that call ends: a session-end: on-message naming its key (the far leg hanging up frees
that leg’s slot alone), or on-reply for the inbound call. When the instance ends, fails or is
cancelled, every slot it still holds is released. A park never releases a slot.
pool-call-key takes the form header.<name>, matched case-insensitively, and needs a pool:
on the same channel.
Reserves. reserve.inbound: R keeps R slots that outbound may not take while inbound holds
fewer than R (and the same the other way round). A direction may take a slot when
held + 1 + max(0, reserve(other) − held(other)) ≤ slots. If the pool’s store fails, inbound is
admitted (as the channel cap’s count is) and outbound waits.
| Load-time error | When |
|---|---|
SUTRA.CONFIG.CHANNEL.POOL_UNDECLARED | a channel names a pool the engine configuration does not declare |
SUTRA.CONFIG.CHANNEL.POOL_TENANT_NOT_ALLOWED | the pool’s tenants list excludes the channel’s tenant |
SUTRA.CONFIG.CHANNEL.POOL_CALL_KEY_INVALID | a malformed pool-call-key, or one on a channel with no pool: |
Walkthrough: two lines, one callback slot
The examples/capacity-pools package runs one flow, session.bpmn:
POST /channels/sessions { sessionId } takes a `lines` slot (or 503)
BuildCallback responseBody := { sessionId, note }
Notify send on agent-callback takes a `callbacks` slot when sent (or waits)
AwaitWrapUp parks until POST /channels/sessions/wrap-up { sessionId }
End releases whatever is still held
POST /channels/sessions/closed { sessionId } session-end: releases the session's slots
Its channels name the pools; nothing in the BPMN does:
- name: session-start
transport: http
bind: "POST /channels/sessions"
codec: json
cloudevents-mode: none
pool: lines
session-end:
on-message: session-closed # a correlated message there releases the session's slots
auth: …
- name: agent-callback
direction: outbound
transport: http
bind: "http://sink:8000/sink"
cloudevents-mode: none
pool: callbacks
deploy/compose.yaml declares the pools on the engine, deliberately small so that three requests
fill them:
SUTRA_CAPACITY_POOLS: lines,callbacks
SUTRA_CAPACITY_POOLS_LINES_SLOTS: "2"
SUTRA_CAPACITY_POOLS_CALLBACKS_SLOTS: "1"
SUTRA_OUTBOX_TICK_INTERVAL: PT1S # a waiting request is re-checked every second
The sink service is the stub server from examples/ai-summarize, used only for its /sink
route, which records every body it is sent.
Prerequisites: Before you try a walkthrough.
1. Start the stack and deploy the package:
C=examples/capacity-pools/deploy/compose.yaml
docker compose -f $C up -d --wait
sutra package examples/capacity-pools/deployments-src/default--lines--1.0.0 \
--out examples/capacity-pools/deploy/deployments 2>/dev/null
PORT=$(docker compose -f $C port engine 8080 | cut -d: -f2)
H=(-H 'Content-Type: application/json' -H 'X-Api-Key: dev-only-lines-key')
until curl -sf "http://127.0.0.1:$PORT/sutra/deployments" | grep -q '"phase":"Active"'; do sleep 1; done
post() { curl -s -w " HTTP %{http_code}\n" -X POST "http://127.0.0.1:$PORT/channels/$1" "${H[@]}" -d "{\"sessionId\":\"$2\"}"; }
slots() {
docker compose -f $C exec -T engine-db psql -U postgres -d sutra -c \
"select pool, direction, count(*) from capacity_slot group by pool, direction order by pool"
}
packaged examples/capacity-pools/deploy/deployments/default--lines--1.0.0.sutra (deploymentId dep-a833cc0807faa199083bcb7b)
2. Open two sessions, and wait until the first one’s callback has been delivered. Both
lines are now taken. The first session’s callback took the only callbacks slot, so the second
one’s callback waits:
post sessions s-1
post sessions s-2
until docker compose -f $C exec sink wget -qO- http://127.0.0.1:8000/sink | grep -q s-1; do sleep 1; done
docker compose -f $C exec sink wget -qO- http://127.0.0.1:8000/sink; echo
slots
HTTP 200
HTTP 200
["{\"note\":\"call the session back\",\"sessionId\":\"s-1\"}"]
pool | direction | count
-----------+-----------+-------
callbacks | OUTBOUND | 1
lines | INBOUND | 2
(2 rows)
s-1’s callback holds the callbacks slot until s-1’s instance ends, or its session does.
3. The waiting request sits in the outbox, with no attempt counted:
docker compose -f $C exec -T engine-db psql -U postgres -d sutra -At -c \
"select convert_from(body, 'UTF8'), attempt_count, last_diagnostic_json::jsonb->>'code' from outbox_entry"
{"note":"call the session back","sessionId":"s-2"}|0|SUTRA.OUTBOUND.POOL.WAITING
4. A third session is turned away, because the pool is full:
post sessions s-3
{"type":"urn:bpm:diag:SUTRA.INBOUND.POOL.EXHAUSTED","title":"SUTRA.INBOUND.POOL.EXHAUSTED","status":503,"detail":"Channel 'session-start' draws from capacity pool 'lines', which has no free slot (2 inbound + 0 outbound of 2); the inbound is rejected (busy).","code":"SUTRA.INBOUND.POOL.EXHAUSTED","attributes":{"channel":"session-start","pool":"lines","slots":"2"}} HTTP 503
5. End s-2’s session while its callback waits. s-2 gives back its line. Its callback is
still sent, at once, although s-1 still holds the only callbacks slot: an ended session takes
no slot and holds none. A third session now fits, and its callback waits for the slot s-1
holds. The lines are now s-1’s and s-3’s, and the callback slot is still s-1’s:
post sessions/closed s-2
until docker compose -f $C exec sink wget -qO- http://127.0.0.1:8000/sink | grep -q s-2; do sleep 1; done
docker compose -f $C exec sink wget -qO- http://127.0.0.1:8000/sink; echo
post sessions s-3
slots
curl -s "http://127.0.0.1:$PORT/admin/instances" | python3 -c 'import json,sys; print(len(json.load(sys.stdin)["instances"]), "instances parked")'
HTTP 200
["{\"note\":\"call the session back\",\"sessionId\":\"s-1\"}", "{\"note\":\"call the session back\",\"sessionId\":\"s-2\"}"]
HTTP 200
pool | direction | count
-----------+-----------+-------
callbacks | OUTBOUND | 1
lines | INBOUND | 2
(2 rows)
3 instances parked
s-2’s instance is still parked at AwaitWrapUp: a session end releases slots, and the instance
runs on.
6. Wrap up s-1. It ends, and gives back its line and the callback slot. s-3’s waiting
callback takes the slot, so one line (s-3’s) and the callback slot (also s-3’s) are held:
post sessions/wrap-up s-1
until docker compose -f $C exec sink wget -qO- http://127.0.0.1:8000/sink | grep -q s-3; do sleep 1; done
slots
HTTP 200
pool | direction | count
-----------+-----------+-------
callbacks | OUTBOUND | 1
lines | INBOUND | 1
(2 rows)
7. Clean up:
docker compose -p sutra-capacity-pools down -v
Next
- Limits and quotas: the payload cap and the per-tenant quotas.
- Configuration reference: every
sutra.*key.
Deploy, hot-deploy, and rollback
The mechanics behind this page are covered in full in Deployment model; this is the operator-facing walkthrough of the same three operations.
Deploy
sutra deploy my-app.sutra --api --engine-url http://localhost:<port>
sutra deploy has two paths: the API path (--api, against an engine running the db
deployment source) is synchronous — it returns only once the deployment is Active, or fails fast
with the SUTRA.DEPLOY.* reject diagnostic. The ConfigMap path (the default, for a Kubernetes
dir deployment source) patches the deployments ConfigMap and is asynchronous — the engine’s own
watcher picks up the change on its next poll; add --wait to have the CLI poll
GET /sutra/deployments/{id} until it reports Active.
For a large deployment where a synchronous call risks a long-held request behind an ingress,
--async (API path) submits and returns 202 {deploymentId, Pending} immediately, then the CLI
polls to completion itself.
Hot-deploy
A hot-deploy is just a normal deploy call against an existing slot (the archive’s stable
tenant--module--version key): re-package the same source directory, and re-deploy. The new
archive gets a new content-addressed deploymentId, but the slot name is unchanged, so the engine
replaces the slot’s active revision in one transaction and runs its two-phase activation flip —
drain the old revision, activate the new one — with no restart. In-flight instances on the old
revision keep running to completion in the background; new inbound picks up the new revision
immediately.
flowchart LR
PKG["re-packaged archive<br/>new deploymentId, same slot"] --> FLIP["two-phase activation flip<br/>one transaction, no restart"]
FLIP -->|"activate"| NEW["new revision<br/>active"]
FLIP -->|"drain"| OLD["previous revision<br/>draining"]
IN["new inbound"] --> NEW
OLD -.->|"in-flight instances<br/>run to completion"| RET["retires once quiescent"]
Nothing in flight is moved by a hot-deploy: new traffic follows the active set from the moment the flip commits, while instances that started on the previous revision finish on the graph they started under.
# edit the package source, then:
sutra package packages/my-app --out /tmp/pkgs
sutra deploy /tmp/pkgs/my-app.sutra --api --engine-url http://localhost:<port>
For the edit-save-redeploy loop while developing, skip the manual re-package step entirely:
sutra deploy --watch packages/my-app --engine-url http://localhost:<port>
Each save is re-packaged and validated first — the same static validation sutra lint runs.
Deployment only fires if validation passes; on a finding, the CLI reports it and skips the deploy,
so a broken edit never reaches the engine.
Rollback
Rollback is the identical operation in reverse: re-package (or simply keep) the previous
archive for the same slot, and re-deploy it. Its still-draining deploymentId resurrects rather
than being rebuilt from scratch:
sutra deploy /tmp/pkgs/my-app-previous.sutra --api --engine-url http://localhost:<port>
Removing a deployment
sutra undeploy my-app.sutra --api --engine-url http://localhost:<port>
The engine drains it — refuses new intake, lets in-flight instances finish, retires the slot once
it reaches zero instances and zero pending outbox entries. On the ConfigMap path, pair this with
sutra deployments list <dir> to find the right archive/deploymentId first.
Deploy, hot-deploy, rollback and undeploy are all edges of one revision lifecycle:
stateDiagram-v2
[*] --> Active: deploy --api, synchronous
[*] --> Pending: deploy --async, 202 accepted
Pending --> Active: activation flip completes
Pending --> Failed: SUTRA.DEPLOY.* reject
Active --> Draining: a newer revision<br/>takes the slot, or undeploy
Draining --> Active: re-deploy<br/>that same archive
Draining --> [*]: retired — zero instances,<br/>zero pending outbox
Rollback is not a separate operation, it is the Draining --> Active edge — which is why the
previous archive’s still-draining deploymentId resurrects rather than being rebuilt.
Two drain behaviors worth knowing before you need them:
- Inbound routes always follow the active set. A slot whose only revisions are draining
serves no inbound routes at all (
SUTRA.RESOLVE.CHANNEL.UNKNOWNon its paths); its parked instances still resume through relay correlation the moment you deploy a new active revision into the slot — which is also the recovery move when an undeploy left work parked. - Accumulated drains are safe. Several draining revisions of one slot are a legal store state (interrupted drains accumulate across restarts); boot registers each channel key once, newest draining revision first, and instances pinned to any of the revisions remain resumable. No cleanup is required before a restart.
Checking what’s live
sutra deployments list <dir> [--label KEY=VALUE]... # ConfigMap/dir source: what's on disk
Against a running engine, GET /sutra/deployments/{id} is the authoritative live status —
Active, Pending, Draining, or Failed with a reason.
Next
- Deployment model — why this is deterministic rather than eventually consistent.
- Instance migration — the sanctioned way to move an instance off the pin it is stuck on, so a draining deployment can finally retire.
- Reference: the
sutraCLI — every flag ondeploy/undeployin full.
Instance migration
An instance is pinned to a deployment — by content hash — the moment it first parks, and every resume path resolves that pin fail-closed. That is what stops accidental version skew: a hot-deploy leaves the previous graph draining precisely so pinned instances keep resuming on the definition they started under, and a lost pin refuses rather than guesses.
It has one consequence with no escape hatch. An instance parked on a broken model stays parked on the broken model until somebody cancels it. Migration is the sanctioned way off the pin.
It is deliberately not a versioning mechanism. There is no in-process branch to author and no fleet of workers to label. There is one operator action, with a machine-readable report, that can be dry-run before it is done, refused with every violation listed rather than the first, and read back out of the audit journal afterwards.
The two endpoints
POST /admin/instances/{id}/migrate | One instance. |
POST /admin/instances/migrate | A filtered population off one source pin. |
Both are on the gated admin surface (see Configuration
reference), and both take the same shape of request: a target
deployment, an optional node mapping, and the dryRun / resume / targetProcessId switches.
curl -sS -X POST "$ENGINE/admin/instances/$ID/migrate" \
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
-d '{ "targetDeploymentId": "dep-9f3c…",
"nodeMapping": { "Approve": "ApproveV2" },
"dryRun": true }'
Validation is the substance
The engine loads both graphs — the source from the deployment the instance is pinned to, the target from the active set — and derives every live locus from durable state: the snapshot’s wait frontier, its routed start event, its retry attempt counters, and the instance’s waiting rows.
Each locus is mapped through nodeMapping (identity where the map is silent), and the target node
must exist and be compatible with what resume actually does there. That last phrase is the
whole idea. Compatibility is not a BPMN type check; it is a check against the resume behaviour the
locus will get:
| Locus | The target node must be | Because resume… |
|---|---|---|
MESSAGE_WAIT | relay-resumable — a userTask, an intermediate message catch, or a channel-call serviceTask | correlates an inbound, marks the node satisfied, and continues from its outgoing edges |
TIMER_WAIT | timer-capable — a timer catch event, a timer boundary, or a <q:timeout>-synthesized boundary | routes a due fire through the timer node’s own semantics |
RETRY_PARK | a serviceTask carrying <q:retry> | re-runs the task rather than treating the node as done — so merely timer-capable is not enough |
CONTINUE_REPLY_PARK | a node carrying <q:reply continue="true"> | takes the relay path, not the timer path, and re-drives the parked tail |
ROUTED_START | a start event | replays multi-start routing from it |
RETRY_BUDGET | a node carrying <q:retry> | a burned attempt budget landing on a policy-less node would silently apply attempts nobody declared |
Every violation is reported, not just the first. Fixing a mapping should take one round trip, not one per node.
flowchart TD
DUR["Durable state<br/>wait frontier · routed start<br/>retry counters · waiting rows"] --> LOCI["Live loci"]
SRC["Source graph<br/>the deployment the instance is pinned to"] --> LOCI
LOCI --> MAP["nodeMapping<br/>identity where the map is silent"]
TGT["Target graph<br/>from the active set"] --> CHK
MAP --> CHK{"target node exists,<br/>and matches what resume<br/>actually does there?"}
CHK -->|"no"| REF["Refused — every violation listed"]
CHK -->|"yes"| GO["Migrate"]
Both graphs are loaded, so the check runs against the target’s real nodes rather than against the mapping’s good intentions — and a refusal comes back complete, not one node at a time.
Some findings are warnings rather than refusals:
- A completed node the target does not declare — not a live locus, so not fatal. But replay-as-done matches by id, so if the target still reaches that node it will be executed again. Dropping a node is legitimate model evolution, so this is loud rather than blocking.
- A frontier entry with no waiting row behind it — validated as a message wait and migrated anyway; the frontier is the resume-side truth and is never dropped.
- Pending outbox rows on the source — see What moves.
Dry run
dryRun: true mutates nothing: it takes no ownership claim and writes no row, and returns the
same full report. Because nothing was locked while it was produced, a dry-run report is
advisory — a concurrent resume can move the frontier out from under it. The report says so.
Node mapping is checked, not merely applied
A mapping entry naming a node the instance neither parks at nor has completed is refused
(MAPPING_INVALID), not ignored. A typo must never read as “identity mapping, then”.
Lifecycle rules
- The target must be active. A draining target is refused: a draining deployment retires the moment it is quiescent, so migrating onto one strands the instance again and costs a migration to do it. Unknown ids are the same refusal.
- The source graph must still be registered (active or draining). Quiescence gating means a live instance always keeps its deployment registered, so this only bites after a forced retirement — and it is a hard refusal rather than a fallback, because validating a continue-reply park against the wrong rule would pass the wrong check.
- Terminal instances are a validation error. Completed and terminated instances are retained history, and re-pinning history rewrites the record of where it ran.
FAILEDinstances are migratable — and that is the prime use case.- A no-op is refused (
TARGET_SAME_AS_SOURCE) rather than silently rewriting a row for nothing.
Migration never auto-resumes — unless you ask
Migration re-pins and rewrites. Full stop. A FAILED instance stays FAILED under the new pin,
and bringing it back is a separate, explicit decision — resumed: false is in every response so
no caller has to infer it.
resume: true is that decision, and it closes the repair loop the operation exists for: fix the
model, migrate the dead instance onto it, bring it back. On a successfully migrated FAILED
instance it clears the failure state, re-stamps the snapshot suspended, and re-arms the parks the
failure commit tore down — all inside the migration’s own transaction, so a crash can never
leave a half-revived instance.
It then comes back through the ordinary paths: a re-armed timer whose due instant has passed
is claimed by the timer poller on its next tick; a message park waits for its next correlated
inbound exactly as it would have. There is no privileged re-drive, which is exactly why the
claim-guarded concurrency story below still holds afterwards. A burned <q:retry> budget stays
burned — resume is not a retry reset.
On an instance that is not FAILED, resume: true is a validation error
(RESUME_NOT_FAILED), not a no-op. A suspended instance is not stuck; it is parked, and it
resumes on its own correlation or its own timer with no operator action at all. Reporting
resumed: false for it would let a caller believe they had woken something.
What moves, and what deliberately does not
| Moves? | Why | |
|---|---|---|
| The instance snapshot | Yes — re-pinned, node ids rewritten | It is the instance. |
| Waiting rows | Yes — node ids mapped; kind, due instant and timestamps intact | A park with an hour left still has an hour left. |
| Correlation aliases | Yes, verbatim | Must: relay correlation resolves the instance through them. Leaving them behind makes a migrated instance unreachable. |
| Subject index rows | Yes, verbatim | Must: erasure and disclosure find instances through them. Leaving them behind makes a migrated instance un-erasable. |
| Audit journal | Yes — scope only; node ids left verbatim | A trail names where something happened, under the graph that was live then. Rewriting the ids would falsify it; a SUTRA.INSTANCE_MIGRATED event records from→to, so the provenance is recoverable. |
| Pending outbox rows | No | An emission was minted by the source deployment’s channel bindings and is dispatched against them — and the dispatcher covers draining deployments, so it drains where it was made. Re-targeting a message at bindings that never produced it would be worse than leaving it. Reported as a warning. |
Coverage cursors travel verbatim, because they are keyed by declared path id rather than by node id — the node mapping must not touch them.
Encrypted variables ride through as ciphertext and still decrypt under the new pin. That is by construction rather than by luck: the authenticated-data binding for an at-rest value deliberately excludes the deployment id, precisely so that “a migration changes only the pin” stays true.
Concurrency: the claim, and why it bounces
A real run claims the instance first, through the same per-instance ownership machinery every resume path uses (see Replica semantics) — and under a distinct owner identity, so a migration can never slip past a resume this same replica has in flight. The claim is re-verified under the row lock inside the commit itself, closing the gap between “claimed” and “moved”.
Contention is a retry-safe 409 (SUTRA.ADMIN.MIGRATE.CLAIM_HELD) with nothing read, rewritten
or committed. A resume that starts after the migration claims bounces off it in turn.
The move itself is one transaction: snapshot, waiting rows, aliases, subject index and audit scope all land under the new pin together, or none do. No session ever observes the instance under both pins or under neither.
sequenceDiagram
participant M as Migrate call
participant C as The instance row
participant R as A resume path
alt the claim is free
M->>C: claim, under a distinct owner identity
C-->>M: taken
R->>C: claim, to resume
C-->>R: CLAIM_HELD — requeue or defer
M->>C: commit, claim re-verified under the row lock
Note over M,C: snapshot, waiting rows, aliases, subject index,<br/>audit scope: all together, or none
else a resume already holds it
M->>C: claim, under a distinct owner identity
C-->>M: 409 CLAIM_HELD, nothing read, rewritten or committed
end
Whichever side gets there first, the other one bounces retry-safely — the distinct owner identity is what stops a migration re-entering past a resume already in flight on the same replica.
Batch migration
POST /admin/instances/migrate applies the same validation, the same compatibility matrix and the
same node mapping to a filtered population.
Selection. filter.sourceDeploymentId is required — one migration names one source graph
and one target graph, and a node mapping that is correct for one source is meaningless for
another. There is deliberately no “every deployment” mode. Optional narrowing: processId,
status (SUSPENDED or FAILED — the only two an instance can migrate in), includeTerminal,
and limit (default 100, clamped to 1000).
Selection is ordered by instance id, then limited. Ordering by a timestamp that moves under a live population is exactly how a caller silently skips work.
Retained terminal rows are excluded unless includeTerminal asks for them — a busy deployment
holds a whole retention window of finished instances, and a report where they crowd out the live
ones is worse than useless. The flag exists so a caller who wants to know why an instance was not
moved gets an explicit INSTANCE_TERMINAL refusal instead of a silent omission.
Every instance is its own transaction. One claim, one commit, one report entry. There is deliberately no batch-wide transaction — which is the point, and which makes the crash story trivial: a mid-batch crash leaves every instance either fully migrated or completely untouched, because that is already true of each instance’s own commit and there is nothing larger to be half applied. Nothing about one instance can decide another’s fate.
Per-instance outcome is a single enum, so no caller has to infer one from a combination of booleans:
| Outcome | Meaning |
|---|---|
MIGRATED | Moved. |
VALID | A dry run validated it. |
REFUSED | Validation said no; the report says why. |
BOUNCED | Its ownership claim was held. Nothing read or written — re-run to pick it up. |
NOT_FOUND | Selected, then gone. |
ERROR | Something else failed for this instance alone. |
The HTTP status describes the batch, not its instances. A run in which every instance refused
still answers 200: the call was accepted, executed to completion, and reported in full — which
is exactly what was asked. 400 is a malformed request, 422 a batch-level refusal (target not
active; source and target the same pin), 503 no persistence. Scripts key on totals and on
each entry’s outcome, never on the status line.
Contention is reported, never retried. Retrying inside the call would make an admin request’s runtime unbounded — the claim is held by another replica’s work, on that work’s timescale, not yours — and would turn the report into a claim about a moment that has already passed. Re-run the same request instead: the selector is deterministic and whatever moved is no longer under the source pin, so a re-run converges.
Two contradictions are caught from the request alone rather than N times over in the per-instance
reports: resume with status: SUSPENDED (it selects exactly the instances resume refuses), and
targetProcessId without processId (re-homing a mixed population into one process is never what
anyone means, and one mapping could only be right for one of them).
Cross-process re-homing
targetProcessId re-homes the instance into a different process of the target deployment.
Naming the instance’s own process id is not a cross-process migration — being explicit must never
change semantics.
Identity is never implicit across a process boundary. Every live locus must carry an explicit
nodeMapping entry, or the migration is refused with CROSS_PROCESS_UNMAPPED — a different code
from the ordinary unmapped-node refusal, because the danger is the opposite one: the id probably
does exist in the target. Two processes that both declare Approve are not thereby the same
Approve, and an accidental collision between unrelated graphs must never read as a deliberate
mapping.
Everything else is unchanged. The same compatibility matrix is evaluated against the target
process’s graph; a target deployment that does not declare the named process is PROCESS_ABSENT,
and its message names both the process asked for and the ones actually there. The snapshot’s
process id and the waiting rows’ process id are rewritten together, because the timer poller
reports it and the admin listing renders it — a row still naming the source process would describe
the instance as living somewhere it no longer does.
One honest wrinkle: a cross-process move can carry a coverage cursor for a path the target process does not declare. It is inert, and remapping it would need a path mapping nobody supplied.
Status codes and diagnostics
| Status | Meaning |
|---|---|
200 | Migrated, or dry-run validated. The body is the full report. |
400 | The id is not a UUID, or the body is malformed / missing the target. |
404 | No such instance. |
409 | Retry-safe refusal: claim held, a unique-live alias collides under the target, or the validated move did not commit. Nothing moved. |
422 | Validation failed. The report lists every violation. |
503 | No persistence configured. |
All codes are under SUTRA.ADMIN.MIGRATE. — TARGET_NOT_ACTIVE, TARGET_SAME_AS_SOURCE,
SOURCE_UNRESOLVABLE, PROCESS_ABSENT, NODE_UNMAPPED, NODE_INCOMPATIBLE, MAPPING_INVALID,
INSTANCE_TERMINAL, CROSS_PROCESS_UNMAPPED, RESUME_NOT_FAILED, CLAIM_HELD,
ALIAS_CONFLICT, COMMIT_FAILED — plus the warnings named above. CLAIM_HELD is the admin twin
of the resume paths’ own, and carries the same retry-safe posture.
A worked repair loop
- An instance fails. It is
FAILED, retained, and blocking its deployment’s retirement. - Fix the model; package and deploy it. The new deployment becomes active; the old one drains.
- Dry-run the migration against the new deployment id. Read the report; supply
nodeMappingentries for anything renamed until it validates clean. - Run it for real with
resume: true. - The instance comes back through the ordinary timer/correlation paths. Once it and its siblings finish, the old deployment goes quiescent and retires on its own.
For a population, do steps 3–4 with the batch endpoint and a filter.sourceDeploymentId of the
old pin, then re-run until totals shows no BOUNCED.
Next
- Deploy, hot-deploy, and rollback — how the pin and the draining tail arise in the first place.
- Replica semantics — ownership claims and durable
FAILED. - Retries, history, and schedules — the most
common way an instance gets to
FAILEDin the first place. - Migration internals: the design reasoning — the two-scope commit, locus derivation, and the re-arm set.
Logging and audit
Three separate surfaces, deliberately not conflated: engine logs, CLI logs, and the audit trail. Each has its own destination and its own purpose. None of the three is collected by Sutra itself — see No telemetry, no phone-home for that guarantee; everything below stays on the host (or goes to a destination you configure) unless you point it somewhere else.
Engine logs
Structured JSON on stdout, always, with no configuration required. The field shape
(timestamp / level / loggerName / message / service.name, plus traceId/spanId
inside a sampled span) is stable, so a log-processing pipeline can key off it directly. RUST_LOG
filters verbosity using standard tracing EnvFilter syntax (default info) — e.g.
RUST_LOG=sutra_channels=debug,sutra_engine::deploy=trace to raise one module without drowning
in the rest. See Troubleshooting BPMN solutions for reading these logs
alongside traces and the audit trail when tracking down one message’s path through the engine.
When an OTLP endpoint is configured (see Configuration reference), the same log records additionally export over OTLP — stdout is never replaced, only supplemented.
CLI logs
The sutra CLI writes its own logs to stderr, opt-in via -v / -vv / -vvv (info / debug /
trace) — never mixed into stdout, which is reserved for report output (text or --format json).
This is why every command in the CLI reference is safe to pipe: sutra describe my-process.bpmn --format json | jq . never has a stray log line corrupt the JSON.
Audit sinks
The audit trail is a separate, compliance-oriented record of what a process instance did — every
INSTANCE_STARTED/INSTANCE_COMPLETED/INSTANCE_SUSPENDED/INSTANCE_RESUMED event, validation
outcomes, and relay/resume decisions — independent of the telemetry pipeline described in
Observability. Three sinks, any combination of which can be
active at once:
| Config | env | What it does |
|---|---|---|
sutra.audit.jsonl.path | SUTRA_AUDIT_JSONL | Writes one JSONL file (or a per-tenant directory tree) an operator can tail, or feed to sutra audit-replay (below) offline. |
sutra.audit.otel.endpoint | SUTRA_AUDIT_OTEL_ENDPOINT | Ships audit events over OTLP to a collector. |
sutra.audit.sql | SUTRA_AUDIT_SQL | Persists audit rows durably in the engine’s own datasource, under the same row-level-security policy as every other engine table — see Multi-tenancy and isolation. |
sutra.audit.sql is off by default. With no sink on, an engine writes no audit record, including the
inbound.DELIVERY_DROPPED record for a refused delivery; see Refused deliveries.
A process’s events go to one sink: the one its <q:audit sink="…"> names, or else the
highest-priority sink that is switched on (sql, then jsonl, then otel).
- JSONL. Each event is one line, written in a single append, so a file shared by every executor lane never holds two events on one line (fixed in 0.6.9).
- OTLP. The records are OTLP log records on the instrumentation scope
sutra.audit, sent through a logs pipeline of their own toSUTRA_AUDIT_OTEL_ENDPOINT, never through the telemetry endpoint. The engine builds that pipeline once at start and flushes it on shutdown. Before 0.6.9, setting this variable made the engine panic at start.
Resource attributes on audit records
The OTLP audit records carry the same resource attributes as the engine’s own telemetry, so a collector can tell which deployment, environment or test run an audit record belongs to:
| Source | Example | On audit records |
|---|---|---|
OTEL_RESOURCE_ATTRIBUTES | deployment.environment=staging,example.run_id=run-42 | yes |
sutra.telemetry.resource-attributes / SUTRA_TELEMETRY_RESOURCE_ATTRIBUTES (0.6.12) | example.from_config=yes | yes, and on every telemetry signal; wins over OTEL_RESOURCE_ATTRIBUTES for the same key |
sutra.audit.otel.service-name / SUTRA_AUDIT_OTEL_SERVICE_NAME (0.6.12) | orders-audit | the records’ service.name; default sutra-audit |
OTEL_SERVICE_NAME | orders-engine | no: it names the engine’s telemetry, never the audit stream |
To try it, point the audit pipeline at a collector whose debug exporter prints what it receives
(verbosity: detailed), and start the engine with:
SUTRA_AUDIT_OTEL_ENDPOINT=http://collector:4317
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=conformance,example.run_id=run-42
SUTRA_TELEMETRY_RESOURCE_ATTRIBUTES=example.from_config=yes
Run any process with <q:audit sink="otel">. Each batch of audit records the collector prints
starts with the resource, which includes (with the telemetry.sdk.* attributes, in any order):
Resource attributes:
-> deployment.environment: Str(conformance)
-> example.run_id: Str(run-42)
-> example.from_config: Str(yes)
-> service.name: Str(sutra-audit)
The conformance suite tc_audit_sinks runs exactly this against a real collector.
What an audit event captures
A process chooses how much each of its audit events carries, with <q:audit capture="…">:
capture | Each event carries |
|---|---|
none | no audit event at all |
metadata | the event, node and diagnostic, with no variable content |
payload (the default) | the redacted variable snapshot |
media (0.6) | metadata, plus the media references found among the variables ({kind: "media", uri, sha256, …}), never the media itself |
With media, subject erasure and terminal retention also delete the recordings those events
reference, and amend each event to say they are gone. See
Recordings and media references for the mechanism and a
walkthrough.
Correlation keys on every audit event
From 0.6.9, each audit event carries the instance’s correlation keys: every <q:alias> the
process declares, with its current value. That lets you find all of one business transaction’s
events by the key your systems already use, without first looking up the instance id.
| Sink | Where the keys are |
|---|---|
| JSONL | a correlation object on the line: "correlation":{"draftId":"d-7"} |
| OTLP | one attribute per key, named correlation.<alias name> |
| SQL | not stored; the audit_event row is unchanged |
- Where the values come from. A start event’s aliases are bound into the instance’s variables under the alias name, and the variables travel with the instance, so every event reads its keys from there, including after a resume on another replica. No database read is added per event.
- Wait-node aliases. An alias declared on a wait node is evaluated over the current variables. Until it has a value, the key is left out.
- Transport events and unaudited processes. Transport events carry no keys, and a process that is not audited computes none.
Redaction applies to the keys as it does everywhere else:
- A key shows
***REDACTED***when the alias name, or any variable its expression reads, is declaredsensitive. - When the expression reads a payload that has a
<q:redactors>chain, the key comes from the masked projection, never from the raw payload. When the expression is more than a plain path, the key shows***REDACTED***.
Try it
The review-loop example correlates every message by its start event’s
<q:alias name="draftId" expression="payload.draftId">. Its compose file switches on the SQL
sink; this override sends the audit trail to a JSONL file instead. You need Docker, the sutra CLI
and an engine image, both 0.6.9 or later. From the repository root:
C=examples/review-loop/deploy/compose.yaml
cat > /tmp/audit-jsonl.yaml <<'YAML'
services:
engine:
environment:
SUTRA_AUDIT_SQL: "false"
SUTRA_AUDIT_JSONL: /tmp/audit/audit.jsonl
YAML
docker compose -f $C -f /tmp/audit-jsonl.yaml up -d --wait
sutra package examples/review-loop/deployments-src/default--review--1.0.0 \
--out examples/review-loop/deploy/deployments 2>/dev/null
PORT=$(docker compose -f $C port engine 8080 | cut -d: -f2)
H=(-H 'Content-Type: application/json' -H 'X-Api-Key: dev-only-review-key')
until curl -sf "http://127.0.0.1:$PORT/sutra/deployments" | grep -q '"phase":"Active"'; do sleep 1; done
send() { curl -s -o /dev/null -w "HTTP %{http_code}\n" -X POST "http://127.0.0.1:$PORT/channels/drafts$1" "${H[@]}" -d "$2"; }
send "" '{"draftId":"d-7"}'
send /review '{"draftId":"d-7","decision":"approve"}'
docker compose -f $C cp engine:/tmp/audit/audit.jsonl - | tar -xO | head -3
docker compose -p sutra-review-loop down -v
Both requests answer HTTP 200, and every line carries the draft’s key (ids and times differ):
{"instanceId":"66ed7453-…","deploymentId":"dep-84173f…","tenant":"default","seq":1,"at":"…","eventType":"INSTANCE_STARTED","payload":{},"correlation":{"draftId":"d-7"}}
{"instanceId":"66ed7453-…","deploymentId":"dep-84173f…","tenant":"default","seq":2,"at":"…","eventType":"NODE_ENTERED","nodeId":"Start","payload":{},"correlation":{"draftId":"d-7"}}
{"instanceId":"66ed7453-…","deploymentId":"dep-84173f…","tenant":"default","seq":3,"at":"…","eventType":"NODE_LEFT","nodeId":"Start","payload":{},"correlation":{"draftId":"d-7"}}
Replaying an instance’s audit trail offline
sutra audit-replay <instance-id> --from-jsonl <path-to-file-or-dir> [--tenant <id>] [--until <EVENT_TYPE>]
Walks the JSONL audit stream for one instance id and prints its events in order — useful for
reconstructing what a specific production instance did without needing direct database access.
--until stops replay after a given event type (e.g. INSTANCE_COMPLETED).
How sensitive data is masked
A variable tagged sensitive (via q:variables, see The q: namespace)
or read from a dataClass-tagged data store (see Data stores) is
redacted on every one of the surfaces above — audit sinks, structured logs, and traces alike — by
the same redactor mechanism described in
Domain neutrality and the SPI model. The flow itself
still sees the real value; only what the engine emits is masked.
A source that declares masking, with q:redactors or a sensitive payload variable, masks the rest
of the inbound message as well: event.body and every header value in event.headers and header,
whole, because they carry the same data and can carry credentials. The flow still reads the real values.
This has limits you should know:
- With no master key, a snapshot holds values unencrypted.
- A flow that declares no masking shows all of it, including a credential sent in a header such as an
API key, on inspect, in the snapshot, and in audit output with
capture="payload". To avoid that, declare asensitivepayload variable, avoidcapture="payload", limit access to the inspect API and the database, or end authentication at a gateway that strips the header. - A
q:aliasvalue is stored in clear, so an alias built from a header holds its value. An explicit event id is stored in clear too, and so is the trace id with an outbound send. - A copy of the body in another variable is only masked if that variable is
sensitive.sutra lintwarns about this withSUTRA.CONFIG.BPMN.RAW_BODY_UNMASKED. Writing the body to a data store warns withSUTRA.CONFIG.BPMN.RAW_BODY_STORED, because store columns have no at-rest encryption. The lint doesn’t cover copies of a header.
Next
- Observability — the telemetry (trace/metric/log) side this page’s audit trail is deliberately kept separate from.
- Troubleshooting BPMN solutions — putting logs, traces, and the audit trail together to answer “what happened to this message?”
Troubleshooting BPMN solutions
This page is for the person building and running a Sutra solution — a deployment package that isn’t behaving the way you expect. (If instead you’re debugging the engine itself — a crash, a suspicious executor code path, a failing test in the workspace — see Debugging the engine, which is written for contributors.)
Every command below is read against rust/crates/sutra-cli/src/commands/ — what’s described is
what the code actually does, not an aspirational surface.
Start with sutra lint
sutra lint packages/my-app
Runs the exact fail-closed validation suite sutra package runs before sealing an archive, and
prints nothing on success. This is always the first thing to run — most “why doesn’t this deploy”
questions are answered here, before you ever touch a running engine.
Reading a SUTRA.* diagnostic code
Every rejection, warning, and structured log line the engine emits carries a code shaped like
SUTRA.<CATEGORY>.<REASON>. The category tells you which layer to look at:
| Category | Layer | Example |
|---|---|---|
SUTRA.PARSE.* | A document failed to parse or failed schema validation | SUTRA.PARSE.XSD.SCHEMA_VIOLATION |
SUTRA.CONFIG.* | A deploy-time configuration problem (channels.yaml, datastores.yaml, a codec manifest) | SUTRA.CONFIG.CODEC_MANIFEST.MISSING |
SUTRA.INBOUND.* | An inbound message was rejected before or during dispatch | SUTRA.INBOUND.NO_START_EVENT_FOR_MESSAGE_TYPE, SUTRA.INBOUND.PAYLOAD_TOO_LARGE, SUTRA.INBOUND.ALIAS_CONFLICT_REJECT |
SUTRA.ACK.* | Acknowledgement / deferred-ack registry events | SUTRA.ACK.DEFERRED_TIMEOUT — see Acknowledgement modes |
SUTRA.VALIDATE.* | A q:validators entry (complex or simple) couldn’t be resolved or run | SUTRA.VALIDATE.VALIDATOR_NOT_FOUND |
SUTRA.RUNTIME.* | A failure while executing an already-dispatched instance | SUTRA.RUNTIME.RELAY.CORRELATION_NOT_FOUND, SUTRA.RUNTIME.DATASTORE.CONFLICT |
SUTRA.OUTBOUND.* | A reply or send failed to encode or deliver | SUTRA.OUTBOUND.ENCODE_FAILED |
SUTRA.DEPLOY.* | The admin deploy API rejected an archive | returned as the body of a 4xx from POST /admin/deployments |
--format json on any CLI command that reports diagnostics gives you the code, severity, and any
attributes as structured data — pipe it to jq rather than screen-scraping text.
Two families get their own sections below, because their codes are read in groups rather than one at a time: codec diagnostics and data-store diagnostics.
Codec diagnostics: parse and schema failures
Whichever codec a channel is bound to, a decode lands on one of three outcomes, and it is that contract — not any codec’s own vocabulary — that the rest of the engine reacts to:
OK— a payload, no issues.SOFT_ERRORS— a usable payload plus structural issues. The instance still starts and still runs; the issues are data your process can branch on. This is the outcome people are most often surprised by, and it’s deliberate: a schema violation is a business decision to route, not an exception to catch.FATAL— no payload at all. The engine rejects the inbound per the channel’s configured posture.
Every issue carries the same five slots — code, severity (ERROR / WARNING / INFO),
path (JSON-Pointer-shaped into the payload, or the source location for a schema violation),
message, and value (the offending value, or the reason code a validator wants surfaced). They
reach your process as <source>.validation (the full list) plus a frozen validation summary:
outcome, tier (structural / content), firstReasonCode, firstIssue, and issues.
<q:onValidation mode="route"/> is what hands that to your own gateway instead of short-circuiting
— see The q: namespace.
The codes this engine emits
The middle segment of a SUTRA.PARSE.* code names the layer that produced it, which is the fastest
way to separate a malformed-bytes problem from a schema problem:
| Code | Outcome | Means, and what to check |
|---|---|---|
SUTRA.PARSE.JSON.PARSE_ERROR, SUTRA.PARSE.XML.PARSE_ERROR, SUTRA.PARSE.YAML.PARSE_ERROR | FATAL | The bytes aren’t well-formed in the format the channel declared. Nothing schema-related has run yet — look at what the sender actually put on the wire, and at the channel’s content-type handling. |
SUTRA.PARSE.XSD.SCHEMA_VIOLATION | SOFT_ERRORS | A well-formed document that violates the package’s own XSD. Collect-all: one issue per violation, never just the first, each carrying line:column in the path slot. |
SUTRA.PARSE.JSON_SCHEMA.SCHEMA_VIOLATION | SOFT_ERRORS | The same, for a schemaKind: json-schema codec. |
SUTRA.RUNTIME.CODEC.DECODE_FAILED | FATAL | A package codec couldn’t read the bytes at all — a parse or transcode failure before validation could run. Most often a payload sent in a format the codec-manifest’s formats list doesn’t include. |
SUTRA.INBOUND.CODEC_NOT_FOUND | rejected before decode | The channel names a codec this binary doesn’t serve. Check what the running build actually links — see Channels and transports. |
SUTRA.OUTBOUND.ENCODE_FAILED | — | The reply direction: the payload couldn’t be encoded for the outbound channel. |
Deploy time is a separate, earlier, fail-closed gate: SUTRA.CONFIG.SCHEMA.INVALID (a schema that
doesn’t compile), SUTRA.CONFIG.CODEC_MANIFEST.MISSING / .INVALID, and
SUTRA.CONFIG.CODEC_LAYOUT.INVALID (a loose file, mixed kinds, or an empty codec folder under
schemas/) all reject the archive rather than deploy a codec that would validate less than it
claims. sutra lint reports every one of them before you deploy anything — which is why it’s the
first command on this page.
A code from a codec you added
An extension codec (see Channels and transports) claims its own middle
segment — SUTRA.PARSE.<STANDARD>.* and SUTRA.VALIDATE.<STANDARD>.* — and documents its own set;
the engine neither interprets nor rewrites them. What the engine guarantees is everything above:
the three-way outcome, the five issue slots, and the same surfaces. So an unfamiliar code reads
exactly the way the table does — the middle segment tells you which codec owns it, and that codec’s
own documentation tells you what it means. Wherever it came from, it reaches you the same three
ways: the validation variables in the process, --format json diagnostic output, and the audit
trail, so Tracing one message end to end applies unchanged.
Data-store diagnostics: projected stores
A store that declares a
structure: block
is verified at package time against the store’s own migrations — no database connection, no
credentials. A store with no structure: block raises none of these codes at all; it stays the
opaque key→JSON store it always was.
The posture is three-state, and reading it correctly saves a lot of time: a definite fault is an error, an unprovable one is a warning worded as unprovable, and a projection that matches raises nothing. Only errors fail the command — a warning here is information, not a gate.
| Code | Severity | Means, and the fix |
|---|---|---|
SUTRA.CONFIG.DATASTORE.STRUCTURE_NOT_FLAT | ERROR | The declared type has a nested, repeated or open child (or no projectable child at all), so it can’t become a flat row. The message names the child. Flatten the type, or remove the structure block and keep the opaque store — those are the only two remedies, deliberately |
SUTRA.CONFIG.DATASTORE.COLUMN_MISSING | ERROR | A declared field projects to a column the effective table doesn’t have, or the table is missing one of the three control columns (store_key, rev, updated_at) every projected table needs — the message names whichever is absent. Add it in a new V-numbered migration, or (a declared field only) map it to a column that already exists under columns: — there’s no columns: remedy for a missing control column |
SUTRA.CONFIG.DATASTORE.COLUMN_TYPE_MISMATCH | ERROR | The column can’t hold the declared value space — VARCHAR(10) for a maxLength="35" field, an integer column for a fractional decimal — or its nullability contradicts the declaration (an optional field against a NOT NULL column with no DEFAULT, or a column an ALTER adds as NOT NULL with no DEFAULT, which pre-existing rows could never satisfy). Widen or relax the column in a new migration, or narrow the declared type |
SUTRA.CONFIG.DATASTORE.KEY_MISMATCH | ERROR | The table’s identity isn’t a key over the projected columns: it declares no PRIMARY KEY (nor a unique constraint) at all, or a key column the projection never writes, or one that maps to an optional field. A projected store reads, upserts and compare-and-sets exactly one row by key, so it needs a key it can always write |
SUTRA.CONFIG.DATASTORE.COLUMN_NAME_INVALID | ERROR | A folded column name collides with another field’s, lands on a reserved word, exceeds the 63-character identifier cap, or isn’t a usable identifier. Name the column yourself under columns: — the mapping is checked by the same rules, so the override has to be usable too |
SUTRA.CONFIG.DATASTORE.DDL_UNVERIFIABLE | WARNING | Nothing is wrong — something simply couldn’t be proven. See below |
SUTRA.CONFIG.DATASTORE.COLUMN_UNMAPPED | WARNING | The table has columns the projection never writes. Usually fine (a legacy or an operator column). One case is sharper and the message says so: an unmapped NOT NULL column with no DEFAULT would make every insert fail — give it a DEFAULT, declare it in the structure type, or drop it |
A structure block pointing at a schema or a type the package doesn’t declare is reported as
SUTRA.CONFIG.DATASTORE.INVALID — the same code any other malformed store declaration uses — not
as one of the projection codes above.
DDL_UNVERIFIABLE is not an error, and shouldn’t be read as one
… could not be fully parsed — the statement
CREATE OR REPLACE FUNCTION f( …is outside the DDL subset this lint parses — so the effective table shape was not derived and the declared structure was not verified; no column diagnostic is raised for this store (it may be valid; it is simply not provable here)
Lint replays a package’s migrations/<store>/V*.sql through a deliberately small SQL subset —
CREATE TABLE, ALTER TABLE ADD/ALTER/DROP COLUMN, and the key-bearing constraints, across the
three shipped dialects’ spellings. Real migrations routinely contain more than that: a PL/pgSQL
trigger body, a T-SQL procedural guard, a table created by an operator outside the package. When
the parser meets something it doesn’t model, it stops trying to verify that store and says so.
It never converts that into a column error, because a linter that cries wolf on legitimate DDL
is one authors learn to ignore.
The same wording covers the narrower cases, and they’re worth telling apart:
- The store’s shape wasn’t derived at all — an out-of-subset statement, no migrations shipped
for the store, the table created elsewhere, or several candidate tables and none named after the
store (name the projected one with
sql: table: <name>). - Individual fields couldn’t be compared — the column’s type is outside the set lint compares
(
JSONB,UUID, a domain type), or the declared type has no ruled column mapping. The column exists; only the type comparison is withheld, aggregated into one line per store. - The declared schema isn’t an enumerable XSD — an engine-provided codec (its type set is
open), or a JSON-Schema / schema-bundle codec folder. JSON Schema carries no length or precision
information, so there is genuinely nothing to compare against here — but don’t read the warning
as “unverified for now, deploy anyway”: a JSON-Schema-declared
structure:isn’t merely unverified, it is refused outright when the engine resolves the store at deploy time (see Typed columns). Declare the structure against an XSD if you want the store to exist at all, not just to have its column types checked.
Note that degradation is per store, not per statement: one out-of-subset statement anywhere in
migrations/<store>/ withholds verification for that store’s whole table. So the way to get the
warning to go away is to keep the store’s own DDL inside the subset (plain CREATE TABLE /
ALTER TABLE, with procedural setup living in a different store’s schema or outside the package),
or to give lint the missing pointer — sql: table: — when the shape is parseable but ambiguous.
If neither is worth it, the warning is a fair thing to live with: first-use verification against
the live table still fails the store closed if the projection genuinely doesn’t hold.
Runtime codes: UNDECLARED_FIELD, VALUE_NOT_A_RECORD, PROJECTION_UNSATISFIABLE
Three runtime codes ride a projected store’s own operations. Where to look for them matters:
unlike SUTRA.RUNTIME.DATASTORE.CONFLICT, which is the failed instance’s own diagnostic code,
these three travel inside the message of a SUTRA.RUNTIME.UNEXPECTED diagnostic. So filter on
the message text, not the code, when you go looking:
| Code | Fires when, and the fix |
|---|---|
SUTRA.RUNTIME.DATASTORE.UNDECLARED_FIELD | A write carried a field the structure doesn’t declare. A projected row is its declared scalars — there is no residue column, so an extra field has nowhere to go, and the write is refused naming the field rather than dropping it. Two causes in practice: the process is writing a value assembled from a different (wider) shape than the declared type, or the declared type has fallen behind a schema change. Declare the field and ship the column, or stop writing it |
SUTRA.RUNTIME.DATASTORE.VALUE_NOT_A_RECORD | A projected store was handed a value that isn’t a record at all — a scalar, an array, or null. A projected row is its declared fields, so there is nowhere for a bare value to land either. Write a record shaped like the declared type |
SUTRA.RUNTIME.DATASTORE.PROJECTION_UNSATISFIABLE | The package-time drift check, caught live: on first use of the store — once per store instance, the same gate that runs its migrations, never on every operation — the provider reads the live table’s actual columns and fails the store closed if the projection can’t be satisfied, naming every offending column at once (one missing, an optional field’s column NOT NULL, or a mandatory unmapped column with no DEFAULT). Usually a hand-applied ALTER that drifted from the package’s own migrations — re-apply them (or ship the missing ALTER) before the store can be used |
A silent partial write is the worse outcome in every one of these, so all three are deliberately
loud rather than best-effort. SUTRA.CONFIG.DATASTORE.COLUMN_NAME_INVALID, in the table above, is
this family’s plan-time member: a declared field folding onto a
control column name is refused before the store is
ever served — both lint and the engine’s own plan-time resolution run the same rule, sourced from
one shared constant, so the two sides cannot drift apart.
The inspection commands
None of these execute anything — they read a BPMN file (or a running engine) and report. Use them in roughly this order when a process isn’t routing, resuming, or replying the way you expect.
sutra describe — what does the engine see in this file?
sutra describe bpmn/transfer.bpmn
A structural summary: processes, start/end events, every service/user task (with its
implementation, codec, validator, and redactor refs), gateways, channel sources, and reply refs.
Read-only — it never invokes the engine’s real loader, so it’s safe to run against a file you’re
mid-edit on. --format json for scripting.
sutra dispatch-graph — visualize the routing
sutra dispatch-graph bpmn/transfer.bpmn --format mermaid
Emits a graphviz dot or mermaid diagram of the file’s dispatch tree — nodes are BPMN elements,
edges are sequence flows. Paste the mermaid output straight into a markdown viewer to see the
shape of a flow you didn’t author, or to sanity-check q:dispatch/q:case routing before review.
sutra simulate --dry-run — will this channel actually reach my process?
sutra simulate bpmn/transfer.bpmn --channel transfer-request --dry-run
Resolves which process (and start event) a named channel would route to, from the file’s own
q:source declarations — no execution, --dry-run is required. If the channel name doesn’t
match any declared source, the error lists every channel the file does declare, so a typo is
obvious immediately. This is the fastest way to answer “why isn’t my message reaching the process
I think it should” without deploying anything.
sutra explain — evaluate a FEEL expression in isolation
sutra explain 'fromAccount.frozen or toAccount.frozen or fromAccount.balance < payload.amount'
sutra explain --context vars.txt 'payload.amount > 100'
sutra explain # no expression: a REPL on stdin, `:quit` to exit
Built directly on sutra-feel — the same evaluator the engine runs, standalone. --context takes
a flat key: value (or key=value) file (values coerce to boolean/number when they parse as
such, else stay strings). Use this the moment a gateway takes the branch you didn’t expect: paste
the exact condition in, supply the variables you think are in scope, and see what FEEL actually
does with them — usually a type coercion or a missing-path surprise, not a logic bug.
sutra coverage check — is my compliance path actually being exercised?
sutra coverage check bpmn/transfer.bpmn # drift lint: declared paths still valid?
sutra coverage check --archive deployed.sutra \
--database-url "$SUTRA_DB_URL" --threshold 100 # correlation-aware, store-backed check
The bare form is a drift lint: every q:coverage declaration must still be an ordered subsequence
of a real route through the current flow graph — it catches a path declaration that silently broke
when someone reworked the diagram. The --archive form reads the deployment’s seeded coverage
flags out of the database its coverage store declares (--database-url names it — any of the
three shipped dialects), and fails closed below --threshold — the gate a CI pipeline runs after a
test campaign. See The q: namespace
for what q:coverage declares and the
money-transfer worked example for it end to end.
Tracing one message end to end
Three surfaces, used together:
- The audit trail — the definitive record of what an instance did:
INSTANCE_STARTED→ node-by-node progress →INSTANCE_COMPLETED/INSTANCE_FAILED/INSTANCE_SUSPENDED/INSTANCE_RESUMED. If you have a JSONL audit sink configured (see Logging and audit), replay one instance’s history offline:sutra audit-replay <instance-id> --from-jsonl /path/to/audit --until INSTANCE_COMPLETED - Traces — if OTel is configured (see Observability),
every span carries
bpm.instance.id; filter your trace backend on that id to see thesutra.dispatch/sutra.decode/sutra.validate/sutra.executespan chain for exactly that instance, across every resume segment if it went through a wait state. - The admin API — against a live engine,
GET /admin/instances/{id}(orGET /admin/instances/by-alias/{key}/{value}if you only have the business correlation key, not the instance id) shows the instance’s current state directly.POST /admin/instances/{id}/cancelis the one instance-level control operation exposed — there is no generic “retry” endpoint; retrying means fixing whatever rejected the input and re-sending it (or, for a parked wait, sending a corrected relay message).
Common failures, worked
“My message got a 200/202 but nothing happened downstream.” Check ack-mode first (see
Acknowledgement modes) — on-persist on an HTTP channel means the 202 you got
back is only proof of durable receipt, not completion; the actual reply (if any) rides an outbound
channel, not the original connection. Then check the audit trail for the instance.
“A message vanished from a broker channel.” A permanent refusal on a broker transport (a body the codec
refused, a type it does not accept, no matching start event, a tenant quota) deletes, commits past or rejects
the message, and from 0.6.18 logs a WARN with SUTRA.INBOUND.DELIVERY_DROPPED; its cause_code says why.
Search the log for that code. See Refused deliveries and connection
references.
“The message never became an instance at all.” Run sutra simulate --channel <name> --dry-run against the deployed BPMN first — a messageTypeValue/messageTypePattern mismatch on
the q:source is the most common cause, followed by the channel’s codec rejecting the payload
outright (SUTRA.INBOUND.CODEC_NOT_FOUND / a FATAL DecodeResult — check sutra describe for
what codec the channel is actually bound to).
“A relay message didn’t resume my parked instance.” SUTRA.RUNTIME.RELAY.CORRELATION_NOT_FOUND
in the audit trail means the relay’s q:alias expression didn’t match any parked instance’s alias
— run sutra explain with the relay payload’s variables against the exact alias expression from
the BPMN to see what key it actually produces, and compare it to what the original request set.
“Senders get a 503 with SUTRA.INBOUND.DEDUP_CHECK_UNAVAILABLE.” The delivery carries an event
id, and the database didn’t answer the duplicate check, so nothing ran. Fix the database, or wait for
it: a retry with the same id is safe, and it runs once. Stopping the database and posting with an
event id gives this answer:
{"type":"urn:bpm:diag:SUTRA.INBOUND.DEDUP_CHECK_UNAVAILABLE","title":"SUTRA.INBOUND.DEDUP_CHECK_UNAVAILABLE","status":503,"detail":"event 'outage-1' on channel 'event-in' could not be checked for a duplicate because the dedup store did not answer; it was not processed, and a retry is safe","code":"SUTRA.INBOUND.DEDUP_CHECK_UNAVAILABLE"}
The document also carries an attributes object. Once the database is back, the same POST succeeds,
and repeating it succeeds without running the flow again. See
Acknowledgement modes for what each
transport does.
“Senders get a 503 with SUTRA.INBOUND.DEDUP_IN_DOUBT.” An earlier delivery of the same event is
recorded, but a synchronous-call chain still holds that record. It clears when the chain ends. If the
chain’s process is gone, or the chain couldn’t release the record, it clears when the sweep runs, within
sutra.instance.inflight-chain-max-age (15 minutes by default). Keep retrying.
“A gateway takes the wrong branch.” sutra explain the condition expression directly, with a
--context file built from the actual variable values at that point (read them off the audit
trail or a trace span if you’re not sure) — this isolates a FEEL semantics surprise from a
“my BPMN is wired wrong” problem in one step.
“My coverage report shows a path that should be covered as uncovered.” Confirm the flows
listed on the q:coverage declaration are still the exact ids in the current diagram
(sutra coverage check bpmn/your-process.bpmn, the drift-lint form) before assuming the flow
genuinely isn’t being exercised.
“Every request answers SUTRA.RUNTIME.UNEXPECTED — engine actor is not running.” An
execution lane died — something panicked outside the per-dispatch containment, which in practice
means the lane’s own build/rebuild failed (the boot log has the panic). The process is a zombie
for that lane’s share of the key space and will not heal in place: both health probes report it
(GET /sutra/health/live → 503 with data.deadLanes; /ready goes DOWN at the same
moment), so an orchestrator with a liveness probe restarts the replica automatically. If you run
without probes, restart it yourself and read the boot log for the panic that killed the lane.
“My coverage op now fails with SUTRA.CONFIG.COVERAGE.STORE_MISSING.” Coverage marks are
persisted in the coverage data store the deployment declares in datastores.yaml, so the op fails
whenever that store isn’t there to write to — rather than answering 0%, because a fabricated 0% is
indistinguishable from a real measurement of nothing covered. Two causes, both named in the message:
the deployment declares <q:coverage> paths but no coverage store (a package-shape
requirement sutra lint errors on with this same code — declare the store; you supply no coverage
SQL, since the engine owns that schema and applies it on first use), or it declares one whose
connection could not be opened (the boot log carries an error naming that deployment — fix the
URL or the environment reference it resolves from). The same code with a different message means the
store opened but the read failed; the underlying error is in the message. See
Coverage.
Next
- Reference: the
sutraCLI — every flag on every command above. - Debugging the engine — the contributor-facing counterpart, for when the problem turns out to be in the engine rather than in your BPMN/rules/config.
Upgrading to 0.6.18
Two changes in 0.6.18 can break a client that worked on 0.6.17: reads of GET /sutra/instances* now need the
admin key, and a synchronous HTTP flow with no <q:reply> now answers an empty body. This page covers those
two and points to the rest. No archive format, deploymentId or snapshot format changes. Two additive migrations
run at boot: V304 (a nullable inbox_seen.instance_id) and V1002 (a deployment_endpoint_holder table).
The tier-3 (Kubernetes) suites were not run for this release.
Reading instances takes the admin key
GET /sutra/instances, its ?alias= search and GET /sutra/instances/{id} now sit behind the admin gate
when admin auth is on (sutra.admin.auth.* or sutra.admin.oidc.*).
- No credential, or a wrong one, gets
401. A valid OIDC token without the admin role gets403. - Send what
/admin/*takes: the admin key inX-API-Key(or the header you set withSUTRA_ADMIN_AUTH_HEADER), or the bearer token. POST /sutra/instances/{id}/cancelis not part of this change.
$ curl -si "$B/sutra/instances?alias=orderId:O-100"
HTTP/1.1 401 Unauthorized
{"error":"unauthorized","reason":"missing or invalid admin auth key/secret"}
An engine with no admin configuration at all still answers these routes without a key. The gate applies only when admin auth is configured, so an engine that never set it up behaves as before. Anyone who can reach it can list instances and read one by id, including a value found through an alias. Either configure admin auth (see Configuration reference) or keep that engine where only operators can reach it.
A flow with no <q:reply> answers an empty body
On a synchronous HTTP channel (ack-mode: on-complete, the default), a flow with no <q:reply> now answers
200 with an empty body. Every earlier version, from 0.2.0-rc.1 through 0.6.17, returned the instance’s whole
variable map as JSON: event.body, the request’s headers including the caller’s X-Api-Key, and every variable
the flow set. A declared reply, and status codes, are unchanged.
What to do:
- List the HTTP channels whose flow has no
<q:reply>and whose callers read the answer. - Add a
<q:reply>that returns exactly what the caller needs, and change the client to read only that. - If a log, a proxy or a client stored the old answers, they hold those credentials. Rotate the keys.
Also changed
- A
202from anon-persistHTTP channel that used to be empty now carries a JSON body when the flow replied nothing: Acceptance receipts. - A second deployment on a taken route, queue or consumer group is refused, and a new version drains the old one whole: One endpoint, one deployment.
- A declared
<q:source dedupKey>now always dedupes. The recognised forms areheader.<name>,ce.id,body.<path>andamqp.message-id. A new package with any other form is refused withSUTRA.PARSE.Q_SOURCE.DEDUP_KEY_UNRECOGNISED; one already deployed keeps running, dedupes nothing as before, and logs the code as an error. A repeat that used to run now answers as a duplicate. Don’t key on a secret header: the value is stored ininbox_seenand can appear in refusal text and logs. - Messages with no content type, refused deliveries, and references in connection properties: Refused deliveries and connection references.
- A package with no
tenantormodulelabel on thedbsource gets a slot of its own and no longer replaces another unlabeled package. Undeploy the old one first, or label it.
Upgrading to 0.6.19
Check these against what you run:
- A connection property that starts with a secret scheme and a colon.
env:,secret:,vault:,aws-secrets:,azure-kv:andgcp-secret:now mean a reference in the properties listed below, so a Kafka broker calledvaultwrittenbootstrap.servers: vault:9092, or a queue calledenv:orders, now fails to resolve and stops the channel. Write such a value through a default:${BROKER:vault:9092}. Other text with a colon, such askafka:9092, is unaffected. - A
${NAME}that 0.6.18 refused. It now resolves, so the channel starts. Before 0.6.18 such a value was used as a literal name, so if you move a Kafkagroup.idfrom a literal${NAME}to a resolved one, it is a new consumer group and starts fromauto.offset.reset(defaultearliest), not from the old group’s offsets. RAW_BODY_UNMASKEDandRAW_BODY_STOREDwarn in more places. They now also fire on a flow that reads a store a masking flow wrote raw text to. These are warnings, andRAW_BODY_UNMASKEDbecomes an error in a later release.
There is no migration, archive format, deploymentId or snapshot format change.
Upgrading to 0.6.20
2026-11-10. A superseded version now keeps serving the channels only it declares, until it retires, where 0.6.18 and 0.6.19 drained it whole. Another owner is refused such an endpoint until then. A literal local:// send to another package, or to another version of the same package, is refused at lint and deploy. The engine uses about half the CPU it did under load: evaluation no longer copies the variables, and jemalloc is the allocator. Coverage can be checked against requirements, in named compliance cycles. RabbitMQ and AMQP broker credentials accept secret: references. Two additive migrations run at boot: V1003 (deployments) and V904 (coverage, at the store’s next first use). There is no archive format, deploymentId or snapshot format change.
Before you upgrade
Check these against what you run:
- A
q:sendto a literallocal://address that is not one of the package’s owntransport: localchannels. Another package, another version of this package, and a barelocal://<name>are now refused bysutra lint,sutra packageand every deploy path, withSUTRA.CONFIG.SEND.LOCAL_DESTINATION_CROSSES_PACKAGE. A destination containing${is not judged. A storeddbdeployment with such a send keeps running and logs the error at each load. A file in a deployments folder with one goesFailedat the next scan, including the first boot after the upgrade. Replace the send with an outbound channel the other package listens on. - A
calledElement, or a<q:dispatch>case or default, that names no process of the package is refused the same way, withSUTRA.CONFIG.CALL_ACTIVITY.UNKNOWN_CALLED_ELEMENT, with the same exemption for stored rows and the same effect on a folder. AcalledElementthat a<q:dispatch>replaces is never used, so it is not checked. - An old version can now hold an endpoint for as long as it has running instances. If you move an endpoint from one module to another (another owner) while the old module’s previous version is draining, the deploy is refused until that version retires. A version with instances parked indefinitely holds its channels indefinitely.
/sutra/deploymentsshows the count underdrainingInstances. - New traffic can start on the old version after the flip, on the channels it keeps. Moving that traffic to the new version is up to your integration.
- New warnings.
SUTRA.CONFIG.PROPERTY.BLANK_REFERENCE,SUTRA.DEPLOY.ENDPOINT.CHANNEL_CHANGED,SUTRA.DEPLOY.ENDPOINT.START_ON_WAITand the lint warningSUTRA.CONFIG.CHANNEL.START_AND_WAITare described below. None of them refuses anything.
Upgrading a deployment in place
When a new version of a tenant and module takes over, the old version drains as before: it takes no new work on the channels the new version has, and its running instances finish on it. New in 0.6.20, an inbound channel that no newer version declares, by name or on any of its endpoints, stays served by the old version, starts included, until the old version retires at zero running instances and zero pending outbox. SUTRA.DEPLOY.ENDPOINT.SUPERSEDED names those endpoints, and the timer starts that stop at the drain:
[WARN] SUTRA.DEPLOY.ENDPOINT.SUPERSEDED — deployment acme--orders--2.0.0 takes over from acme--orders--1.0.0, an earlier deployment of the same tenant and module. acme--orders--1.0.0 drains: it takes no new work, and its running instances finish on it. Channels only it declares stay served by it until it retires: POST /orders/release, POST /orders/legacy.
- What the old version keeps is recomputed at every change. Undeploying the new version gives the old one back every channel it declares, the ones it shared with the new version included, until it retires; then they stop. Undeploying the old version itself, while it is active, stops them.
- Retirement is rechecked. A version that just retired is checked again until it is quiet twice; one that had work arrive after all goes back to draining.
- A version that keeps feeding itself over its own channel never retires. The engine does not stop it.
GET /sutra/deploymentsnow shows each draining version’s running instances underdrainingInstances({"<deploymentId>": n}), andGET /sutra/deployments/{id}gainsinstancesfor a draining version. Thedrainingarray is unchanged. - Versions already draining when you upgrade the engine stop serving as before. Only a version superseded on 0.6.20 keeps channels.
- In a deployments folder, removing the old version’s file stops its channels, and the answer survives a restart.
Another owner is refused an endpoint a draining version keeps, until it retires:
[SUTRA.DEPLOY.ENDPOINT.COLLISION] http endpoint 'POST /orders/release' stays with draining deployment acme--orders--1.0.0 (2 running instances) until it retires
0.6.19 let another owner take it at once. POST /admin/deployments/check gives the same answer.
Upgrade warnings. These name what changes and how many instances are parked on it, and never print an auth value:
SUTRA.DEPLOY.ENDPOINT.CHANNEL_CHANGED: the deploy changes a channel an older version has instances parked on: its inbound auth, codec, schema, address or transport, or “renamed or replaced” when its endpoint is now served under another channel name.SUTRA.DEPLOY.ENDPOINT.START_ON_WAIT: a new start event listens where the older version waits. Start events are tried first, so a reply meant for a parked instance starts a new one.SUTRA.CONFIG.CHANNEL.START_AND_WAIT:sutra lintwarns about the same shape within one package.
Unchanged: a resume on a channel the new version changed is processed with the new version’s binding; an old instance parked on a channel the new version serves under another name is still unreachable from it (now warned); a retry across a version flip still runs again.
Schema: migration V1003 adds deployment_archive.retains_channels on PostgreSQL, MySQL and SQL Server. It is FALSE for rows drained before it existed.
The walkthrough is Upgrading a deployment in place; One endpoint, one deployment describes how the other deployment paths behave.
Less engine CPU under load
- Expressions evaluate without copying the variables. A FEEL dotted path walks nested values by reference and copies only the value it ends on. Data tasks, exclusive gateways and
q:paraminputs evaluate against the live variables, and each operation still sees the previous one’s result. A strict template engine gets a render model built without null entries in one pass, with the same JSON and key order. - jemalloc is the engine’s allocator. It is a default feature,
jemalloc, ofsutra-dist, and is never linked on MSVC targets. A build with--no-default-featuresdrops it, along with the other defaults, and uses the system allocator. jemalloc fixes its page size when it is built: an arm64 image build setsJEMALLOC_SYS_WITH_LG_PAGE=16(64 KiB), so the binary also runs on kernels with 16 or 64 KiB pages. If you build your own arm64 binary, set it too, or the binary aborts at start on such a kernel. amd64 is unaffected. - Measured on a laptop at 200 messages/s for 60 s, every message completed, none duplicated: steady engine CPU fell from about 316% to about 139% of one core, roughly 42% from the evaluation change and a further 24% from jemalloc. With database writes off, 267% to 121%. jemalloc used less CPU than mimalloc and had the lower p99. The runs were bound by the laptop’s disk, so their latencies are only indicative; the CPU figures are the reliable ones. A relay benchmark at 500 messages/s showed no change.
Coverage against requirements, in compliance cycles
The walkthrough is in Coverage.
- A requirements map. A package can ship
coverage/requirements-map.yaml, mapping each declared path to requirement ids, a weight (default 1, above zero) and the test cases that walk it.sutra coverage checkthen adds acomplianceobject: per-requirement and weighted percentages, uncovered paths with their cases, the cases to run, and paths with no cases or no mapping.--thresholdgates the weighted percentage. Every declared path counts; there is no way to exclude one. A package with no map reports as before. Lint:SUTRA.CONFIG.COVERAGE.MAPPING_ROUTE_UNKNOWN(error, a key that matches no declared path) andSUTRA.CONFIG.COVERAGE.MAPPING_ROUTE_UNMAPPED(warning, a declared path the map leaves out, counted at weight 1). - Cycles.
sutra coverage cycle declare|activate|deactivate <LABEL>andsutra coverage cycle list, plus--cycleoncoverage checkandcoverage reset. While a cycle is active, every mark for that deployment records to it, including thecoverage:reportandcoverage:resetoperations. A mark belongs to the cycle active when its instance completes. One cycle can be active per deployment: a second activation is refused (SUTRA.RUNTIME.COVERAGE.CYCLE_ALREADY_ACTIVE), as is activating an undeclared label (CYCLE_UNDECLARED); reactivating a cycle continues it. With none active, coverage records as before, andcheckandresetwithout--cycleread the marks recorded outside any cycle. Cycles belong to a deployment id, so a changed package starts with none. Thesutra.coverage.percentgauge and the path counter are not cycle-aware. - To re-run a case, send
<cycle>:<case>:<attempt>as the idempotency key on a channel that declaresidempotency-key-header: a repeat of an attempt does nothing, the next attempt runs. - Schema: V904 adds four coverage tables (
coverage_cycle,coverage_active_cycle,coverage_cycle_metric,coverage_cycle_fragment) on PostgreSQL, MySQL/MariaDB and SQL Server. Existing rows are untouched and read as “no cycle”.
Connection references
See References in connection properties.
- RabbitMQ and AMQP broker credentials accept
secret:KEYand${secret:KEY}.usernameandpasswordnow take exactly the whole-value forms the reference resolver resolves, plusk8s:secret/<name>#<key>. 0.6.19 refused the twosecret:forms there. A literal, or text with a reference inside it such aspre${X}, is still refused withSUTRA.STARTUP.REFUSED.HARDCODED_SECRET. - RabbitMQ now accepts a lowercase
${name}in credentials, as AMQP already did. It used to refuse it. - A reference that resolves to blank now warns
SUTRA.CONFIG.PROPERTY.BLANK_REFERENCE, naming the channel, the property and the reference as written, never a value. 0.6.19 applied the transport’s default silently: RabbitMQhostbecamelocalhost, Kafkagroup.idbecamesutra-<channel>. That still happens, and a blank required property is still refused, now after the warning. The explicit-empty forms${NAME:}and${secret:KEY:}do not warn, so write an optional setting that way. This covers the connection properties ofkafka,rabbitmq,amqp,gcp-pubsub,fileandaws-sqs, not credentials or data-store URLs.
Delivery and data stores
- Each failed delivery attempt that will be retried now logs a WARN with the destination, the cause’s code and message, the attempt number and
next_attempt_at. Before, a retried attempt logged nothing, so a callback that arrived seconds late gave no cause. - A closed in-process hand-off no longer delays delivery by 30 seconds. Outbox rows are inserted due after a short grace for the in-process hand-off. When the hand-off had closed before the commit, its rows waited out that grace. Now exactly those rows are made due at once, if they are still as inserted (never a leased, deferred or poisoned row), and the dispatcher is woken, so the next poll tick delivers them. Delivery stays at-least-once with the same dedup.
- A coded data-store refusal keeps its code. An edited migration that was already applied refuses the store with
SUTRA.RUNTIME.DATASTORE.MIGRATION_CHECKSUM, but the step failed withSUTRA.RUNTIME.UNEXPECTED, so the dead letter’scause_codeand its log line never showed the cause. It now fails with the store’s code and logs at ERROR, naming the store and the script, never the key. Routing is unchanged: a non-idempotent flow is still dead-lettered, an idempotent one requeued.
Fixed in the documentation
sutra.outbox.retry.base-delaydefaults toPT1S, as the code always had it; the configuration guide saidPT5S.- The 0.6.18 notes said a dropped delivery’s audit record has an empty
instance_id. No instance owns it, so it is NULL: select it withinstance_id IS NULL.
Tooling
- Load steps report resources.
tools/sutra-load-testrun.shandrelay-bench.shnow report engine CPU and RSS and PostgreSQL CPU, IOPS (pg_stat_io) and WAL volume for each step, fromstep-metrics.sh, as a table and a.metrics.json. A metric it can’t read printsunavailable, never 0.
Security
- RUSTSEC-2026-0459 is ignored in
cargo audit, time-boxed. It is inlibsqlite3-sys0.30.1, which is inCargo.lockonly becausesqlxdeclares its optional SQLite driver through a weak feature. No crate enables SQLite, so it is compiled into nothing:cargo tree -i libsqlite3-sys -e all --target allis empty, andcargo deny, which resolves the real graph, does not see it. Nosqlxrelease allows the fixed 0.39 yet. The ignore is dropped when one does.
For extension authors
sutra_envref_spi::is_whole_reference(whether a value is exactly one whole reference form) andis_explicit_empty_defaultare new.StoreError::with_codeandStoreError::codeare new insutra-executor; a store adapter that keeps a refusal’s code passes it through them.OutboxSignal::hand_offinsutra-channelsnow returns the ids it could not hand off.sutra-channelsgainswait_subscriptionsandmessage_types_can_overlap.
Upgrading to 0.6.21
2026-11-13. The kafka and amqp (AMQP 1.0) transports connect to TLS brokers natively, with no sidecar, and authenticate with SASL: Kafka with PLAIN, SCRAM-SHA-256 or SCRAM-SHA-512, AMQP 1.0 with PLAIN. Brokers are verified by default. FEEL gains password hash and password verify (argon2id), so a flow can sign a user in against a password hash in a typed store without storing or logging the password. The CLI’s self-update works on macOS, and paths printed on Windows no longer carry a \\?\ prefix. There is no migration, and no archive format, deploymentId or snapshot format change.
Before you upgrade
Check these against what you run:
- Kafka’s
kafka.*passthrough now refusessasl.*,ssl.*andenable.ssl.certificate.verification. A channel that sets one of them stops at startup. Use the typed properties below instead. They could not have secured a connection before, because the engine’s Kafka client was built without TLS. - A Kafka
sasl.username,sasl.passwordortls-ca-pemwritten as a literal is refused withSUTRA.STARTUP.REFUSED.HARDCODED_SECRET, like the AMQP and RabbitMQ credentials. Write a reference. - A plaintext AMQP sink whose credentials the broker refuses now reports
SUTRA.OUTBOUND.AMQP.BROKER_AUTH_FAILED, where it reportedSEND_FAILED. Update an alert that matches the old code. - The engine binary now includes its own OpenSSL 3.6.3, for the Kafka client (librdkafka has no other TLS backend). The engine still loads no system libssl:
readelf -dlists only libgcc_s, libm, libc and the loader. An image scanner cannot see this copy; it is tracked throughrust/Cargo.lock, socargo auditreports an advisory filed against theopenssl-srccrate. The image is about 8 MB larger. Building from source now needsperl, for OpenSSL’s build.
TLS and SASL on Kafka and AMQP 1.0
For an inbound channel:
| Property | Transport | Meaning |
|---|---|---|
security.protocol | kafka | PLAINTEXT (default), SSL, SASL_PLAINTEXT or SASL_SSL |
sasl.mechanism | kafka | PLAIN, SCRAM-SHA-256 or SCRAM-SHA-512 |
sasl.username, sasl.password | kafka | References, never literals |
tls: true | amqp | Use TLS. The default port becomes 5671 |
username, password | amqp | References. SASL PLAIN is used when both are set |
tls-ca-file | both | A PEM bundle used in place of the system CAs. Takes references, like an address |
tls-ca-pem | both | A reference to PEM text, used in place of the system CAs |
tls-insecure-skip-verify | both | true turns off the certificate and host-name checks |
- Verification is on by default, against the system CAs (
SSL_CERT_FILEandSSL_CERT_DIRare honoured), unless a CA setting replaces them.tls-insecure-skip-verify: trueis the only way to turn it off, and the engine logs a warning whenever it applies. - A setting that could not take effect stops the channel at startup instead of being ignored: a CA setting without TLS,
sasl.*without a SASL protocol, both CA settings at once, or SASL GSSAPI or OAUTHBEARER. - The protocol switches stay literal.
tls,security.protocol,sasl.mechanismandtls-insecure-skip-verifydo not expand references; addresses, credentials andtls-ca-filedo. - Outbound sends use one engine-wide setting per transport, from the environment:
SUTRA_SINK_KAFKA_SECURITY_PROTOCOL,SUTRA_SINK_KAFKA_SASL_MECHANISM,SUTRA_SINK_KAFKA_SASL_USERNAME,SUTRA_SINK_KAFKA_SASL_PASSWORD,SUTRA_SINK_KAFKA_TLS_CA_FILE,SUTRA_SINK_KAFKA_TLS_INSECURE_SKIP_VERIFY; and foramqp10s://destinationsSUTRA_SINK_AMQP_TLS_CA_FILEandSUTRA_SINK_AMQP_TLS_INSECURE_SKIP_VERIFY, with the existingSUTRA_SINK_AMQP_USERNAMEandSUTRA_SINK_AMQP_PASSWORD. - New diagnostics:
SUTRA.INBOUND.KAFKA.BROKER_TLS_FAILED,SUTRA.INBOUND.KAFKA.BROKER_AUTH_FAILED, the same two underOUTBOUND.KAFKA,INBOUND.AMQPandOUTBOUND.AMQP.BROKER_TLS_FAILEDmeans the broker’s certificate is not trusted or does not name the host, or the trust settings could not be read;BROKER_AUTH_FAILEDmeans the broker refused the credentials. Neither prints a password or anything the broker sent back during SASL. A consumer keeps retrying and an outbound message stays in the outbox, so both recover once the setting is corrected. The AMQP consumer backs off between attempts; the Kafka client retries quickly and logs each attempt, so a wrong CA gives dozens of lines a second: alert on the code, not the line count. - What it was tested against: Apache Kafka 3.8 and Apache ActiveMQ Artemis, in local containers. Managed Kafka services and Azure Service Bus have not been tested and are not claimed to work; Azure Service Bus’s CBS token flow is not implemented. Client certificates (mTLS) are not supported.
The walkthrough is Connecting to a TLS and SASL broker: a message through both brokers, a refused login and an untrusted CA.
Password hashing in FEEL
password hash(password)returns an argon2id hash in PHC string form ($argon2id$v=19$m=19456,t=2,p=1$…), with a fresh 16-byte salt from the operating system. The parameters are fixed in code. An empty or non-string password givesnull.password verify(password, hash)returnstrueorfalse, comparing in constant time. It returnsnullfor a malformed hash, one that is not argon2id, or one that asks for more than m = 65 536 KiB, t = 10 or p = 4, so a stored hash cannot make a verify arbitrarily expensive.password hashis denied where an expression must give the same answer on replay, likeuuid():sutra lintrefuses it there withSUTRA.FEEL.DETERMINISM.UNSAFE_BUILTIN.password verifyis deterministic and allowed everywhere.- Cost: one verify takes about 22 ms (criterion median 22.3 ms, range 21.1 to 23.8 ms, on an Intel i7-10750H laptop, release build).
- Keep the plaintext out of every surface by reading the sign-in message into a variable marked
transientandsensitive: it is then neither persisted nor written to the audit log or the engine log. Signing a user in against a store builds such a flow, and checks the database dump, the audit file and the engine log for the password. - New dependency:
argon20.5 (withpassword-hashandblake2), with default features off.
CLI on macOS and Windows
sutra self-updateworks on macOS. It passed GNU tar’s--wildcards, which macOS’s bsdtar refuses. It now unpacks the archive into a scratch folder and takes the binary from there.- Windows paths are printed without the
\\?\prefix. Package loading,sutra lint,sutra createand the documentation generators resolve paths without it, so a printed path or a path comparison matches what you typed. The catalog’s JSONrepoRootshowed it. sutra compat-baselineagainst a git reference givestara relative archive name, because GNU tar, which Git for Windows puts first onPATH, readsC:\…as a remote host.- These were tested on Linux, which exercises the Linux-visible half of each. A new hosted job,
cli-portability, runs the CLI and package-loader tests and clippy on macOS 15 and Windows Server 2025 for every change; its first run is this release’s.
Building and CI
make ciruns the hosted pipeline’s checks locally, cheapest first:make fmt-check,make lint,make cross,make test,make auditandmake scan-deps.make testruns cargo-nextest when it is installed, plus the doctests, and otherwisecargo test; both run every test before reporting. The hosted pipeline installs nextest.make crossruns clippy for the CLI and its workspace crates against Windows (MSVC) and both macOS targets, skipping a target whose C toolchain the host lacks.- The workflows pin every action by commit and every runner by version, and are checked with actionlint. A new architecture test fails the build when a workflow’s Rust toolchain differs from
rust-toolchain.toml. .gitattributesmakes text files LF in every checkout. The BPMN and DMN schemas published by the OMG are kept byte for byte as published.
Security
- The image scan configuration’s justifications for the base image’s libssl3 now say why: the engine links no libssl, and its Kafka client’s OpenSSL is compiled in (see “Before you upgrade”).
- Seven new base-image findings are ignored, time-boxed, with their reasons in
.trivyignore.yaml: five in libssl3 (CMS, QUIC and AES-GCM-SIV), which the engine never loads and whose code paths its compiled-in OpenSSL does not use for TLS over TCP, and two in glibc (regex and the widescanffamily). Debian ships no fix for any of them. quinn-proto0.11.19 andquinn-udp0.5.16 inCargo.lock, for RUSTSEC-2026-0523, RUSTSEC-2026-0524 and RUSTSEC-2026-0538. Neither crate is compiled into the engine or the CLI; they are in the lockfile only as an optional dependency ofreqwestthat no crate enables (cargo tree -i quinn-proto -e all --target allis empty).
For extension authors
sutra_transport_spi::is_secret_referenceis new: whether a value is a whole secret reference (any registered scheme,${…}, ork8s:secret/<name>#<key>).sutra-transport-amqpgainsAmqpTlsProperties,register_amqp_sink_with_tls,sink_tls_settings_from_envandAmqpMessageSink::with_tls_settings, andAmqpChannelPropertiesatls_settingsfield.sutra-transport-kafkagainsKafkaBrokerSecurity,KafkaSinkSecurity,register_kafka_sink_with_securityandKafkaMessageSink::with_security, andKafkaChannelPropertiesasecurityfield.BrokerFailureRecorderis new. Code that buildsAmqpChannelPropertiesorKafkaChannelPropertieswith a struct literal must add the new field.
Upgrading to 0.6.22
2026-11-15. A message now answers the instance waiting for it before it starts anything, with one version of a package and across versions while an older one drains. A retry that spans a version flip is a duplicate. A repeated Kafka TLS or SASL failure is logged once with a count instead of hundreds of times. Large templates compile once per process. The engine image’s CA bundle no longer trusts two roots Debian distrusted (DLA-4845-1). Two additive migrations run at first boot: V305 and V306 (inbox). There is no archive format, deploymentId or snapshot format change.
Before you upgrade
Check these against what you run:
- A channel with both a start event and a wait of a compatible message type. A message that
correlates to a parked instance now resumes it and starts nothing, also with one version.
sutra lintalready warns these withSUTRA.CONFIG.CHANNEL.START_AND_WAIT. A flow that re-sent a business key on such a channel to start a second instance now answers the first. - A message whose key names a running instance that has not reached its wait yet is refused with
a retryable
WAIT_PENDINGanswer instead of starting a new instance. - A retry across an upgrade is a duplicate (202, naming the original instance). A changed key
header,
dedupKeypath or channel name produces another key, and that message runs again. - A non-owning database role needs
SELECT, INSERTon the newdeployment_ownertable, whichinbox_seen’s row-security policy reads. inbox_seengets an index built concurrently at first boot (V306). It adds I/O in proportion to the table and blocks no writes. Two replicas booting together build it once.
Waits first
The walkthrough is A reply answers the instance waiting for it.
Before a start event is resolved, the engine asks whether the message answers a wait an instance of the channel’s owner is parked at, in any of the owner’s live versions. A hit resumes that instance and is the whole delivery. A miss is dispatched exactly as before.
- A resume is processed as the version it is parked on: its own binding, codec, validators and reply codec, under the address and auth of the version that serves the channel. A message that version 2’s codec refuses can still resume a parked version 1 instance under version 1’s codec; it can never start anything.
- A second process waiting on one channel is now tried; before, only the first was.
- On a
broadcast: truechannel the process holding the correlated wait resumes and does not start; every other process’s start on the channel still runs. - Cost: nothing on a channel no live version waits on (an index answers in a few hash lookups). On a channel with a wait of a compatible type, one correlation lookup per live version that waits.
Measured (the neutral relay benchmark, tools/sutra-load-test/relay-bench.sh, 500/s for 60 s,
32 lanes, a pool of 40, 32 outbox workers, durable PostgreSQL; two runs each, interleaved, on one
machine; release-it builds):
| build | settles/s | settle p50 | settle p99 | WAL flushes/settle | commits/settle | CPU ms/settle (engine+PG) |
|---|---|---|---|---|---|---|
| 0.6.21 | 497.7 / 498.5 | 26 / 27 ms | 116 / 130 ms | 0.59 / 0.57 | 13.2 / 13.2 | 6.18 / 6.61 |
| 0.6.22 | 498.6 / 498.7 | 27 / 26 ms | 200 / 76 ms | 0.58 / 0.59 | 13.2 / 13.3 | 6.68 / 6.62 |
Every order settled exactly once in every run. The differences are within run-to-run noise (p99 spans 76 to 200 ms across the four runs).
Dedup per owner
A dedup record now carries its owner, so a key version 1 recorded is a duplicate on version 2, and two versions racing one key commit it once. Records written before the upgrade dedupe within their own deployment only (no backfill), so a retry spanning both the engine upgrade and a version flip can run again.
Fixes
- Kafka: against a wrong CA, librdkafka reported every reconnect, each time with different text,
and the engine logged about 740
BROKER_TLS_FAILEDlines in 20 s. The first failure under a code is now logged in full, and repeats at most once per 30 s with their count, until a success resets it. - Templates: compiling a Handlebars template costs time in proportion to the square of its size
(about 0.8 s for 100 KB and 3.4 s for 200 KB). Each engine lane compiled each template again on its
first render after every activation. A deployment’s templates are now compiled when it is
deployed (logged as
deployment templates precompiled, with the time it took), once per process, so the first message pays nothing: a 210 KB template’s first message answered in 16 ms. A template that does not compile is refused bysutra lintas before, and fails at render as before if it reaches one. - AMQP 1.0 sends are bounded. Opening the connection (TLS and SASL included), beginning the
session, attaching the sender and waiting for the broker to settle the message each get at most
SUTRA_SINK_AMQP_OPERATION_TIMEOUT_MS(default 30000). A broker that stops answering fails the attempt with the retryableSUTRA.OUTBOUND.AMQP.SEND_TIMEOUTand the outbox sends again; before, one delivery could wait forever. A send that timed out waiting for the broker to settle it may already have been accepted, so the retry can deliver it twice: receivers dedupe onsutra-outbox-key. - CA bundle: the image carries a CA bundle that no longer trusts “AAA Certificate Services” and “emSign ECC Root CA - G3” (DLA-4845-1).
rustix1.1.6 (RUSTSEC-2026-0486).
Known
SUTRA.INBOUND.CAPABILITY_MISMATCHanswers HTTP 500. A body the channel’s codec or format contract cannot take (a nested body on a flatcsvchannel, a content type the codec does not accept) is the caller’s fault, and a later release will answer it with 415. Nothing changes in 0.6.22.
Upgrading to 0.6.25
2026-11-19. Templates gain an xmlEscape helper, so free text relayed into an XML
message stays well-formed. Timers no longer fall behind under sustained load: up to 0.6.24 the
engine fired at most about 64 timers a second for the whole cluster, whatever the shard count, so
at higher rates every parked timer fired later than the one before it. A distribution’s version
line may carry its build without the build reaching image tags. There is no migration, and no
archive format, deploymentId or snapshot format change.
Before you upgrade
Check this against what you run:
- If you expect timer surges, run more than one shard. Set
sutra.engine.shards(SUTRA_ENGINE_SHARDS, default1; see Configuration). Due timers now fire as fast as they fall due. With one shard, that work shares the single execution lane with new work, so new instances answer more slowly while a timer backlog drains. In the test below, at 200 new instances a second, each parking a 1 s timer, the median new-instance answer went from 2.5 ms to 59 ms with one shard, and from 1.9 ms to 2.6 ms with four.
Timers drain instead of firing one batch per tick
Every 500 ms the leader replica’s timer poller looks for due timers. Up to 0.6.24 it claimed at most 32 per deployment, fired them and waited for the next tick. That is about 64 timer fires a second for the whole cluster. Above that rate the backlog grew for as long as the load lasted: instances started on time, and every timer-driven step after them ran later and later.
Now each tick keeps claiming while a claim comes back full, taking the deployments that are behind
in turn so one backlog cannot starve another. It stops when every claim comes back short, after one
second of draining (or the tick, if that is longer), or when the replica stops being the leader.
Each round’s timers finish firing before the next claim. Up to max(shards, 16) timers fire at
once. Timer start events are unchanged. There is no new configuration key, and a timer still fires
at least once.
Measured on a debug build against PostgreSQL 16. Each instance starts over HTTP, parks a PT1S
timer, then sends a follow-up message; the lag is the follow-up’s arrival against its due time.
| Shards | Rate, for 15 s | Follow-up lag p99, 0.6.24 | 0.6.25 |
|---|---|---|---|
| 1 | 200 a second | 31.8 s | 0.78 s |
| 4 | 400 a second | 78.2 s | 0.50 s |
The lag that remains comes from the 500 ms tick.
xmlEscape
{{xmlEscape value}} escapes & < > " ' in free text for XML element content and
attribute values. A null or missing value renders empty. Default {{x}} interpolation is
unchanged. Reference: Template helpers.
Walkthrough: Escaping free text in an XML template.
For distributions
sutra_cli::product_version() reads a distribution’s version up to the first (. A version
block that starts 2.0.0-rc.6 (1a2b3c4, built 2026-11-16) reports 2.0.0-rc.6, so the runtime
image tag, the release tag create app scaffolds and codecs no longer carry the build suffix. A
version block with the build on its own line works as before.
Upgrading to 0.8.0
2026-11-19. A synchronous HTTP answer sets its own status and headers with
<q:reply status="…"> and the reply’s <q:header> children. It adds an attribute to the q:
namespace (urn:sutra:q:1.0), which is why this release is 0.8.0. The addition is backward
compatible: existing packages load and run unchanged, and an answer without the attribute or
headers is unchanged. Broker outages can reach the deployment as messages. FEEL durations
keep fractional seconds. No migration, and no archive or snapshot format change.
Coming from the 0.6.25 image: there is no 0.7.0 image
The v0.7.0 tag, its CLI archives and its crates.io crates were published, but its release run
stopped at the engine image scan on a base-image advisory published the same day
(CVE-2026-95690, below), so there is no ghcr.io/startr-trade/sutra:0.7.0 image and no 0.7.0
GitHub Release. 0.8.0 carries every 0.7.0 change. If you run the 0.6.25 image, move straight to
0.8.0 and read Upgrading to 0.7.0 as well: its migration (V103) and its
behaviour changes (data-store revisions seeded from the database clock, uncorrelated messages
logged and audited) arrive with this image.
Before you upgrade
Check these against what you run:
- A FEEL duration with a fraction on a unit other than seconds is now refused.
duration("PT1.5H")used to be cut toPT1H; nowduration()cannot parse it, as XML Schema and the FEEL specification require. WritePT1H30MorPT5400S. See Durations with fractional seconds. .secondson a days-and-time duration returns the fraction.duration("PT4.5S").secondsis4.5, where it was4. An expression that needs a whole number should applyfloor().- A fractional seconds value is no longer dropped.
duration("PT0.001S")was zero, so a timer whose FEEL expression returned it fired at once; it now waits 1 ms.string()keeps the fraction.
Status and headers on a synchronous answer
On an HTTP channel that answers synchronously (ack-mode: on-complete, the default), a
<q:reply> with no destination can set the status (digits, or a FEEL expression), and its
<q:header> children become the response’s headers. A flow can answer 201 with a Location, 404
or 409, or act as an auth check that answers 200 with identity headers or 401 with a challenge.
Statuses outside 200–599, a 3xx without a Location, and headers a flow may not set are refused
at load by sutra lint, sealing and deploy alike; a computed status or header value that cannot be
sent is a 500 that names the header, never its value.
Reference: Status and response headers. Walkthrough: Answering with a status and headers.
Broker outages as messages
Set sutra.engine.connectivity-events.channel (env SUTRA_CONNECTIVITY_EVENTS_CHANNEL) to a
channel name, declare that channel in a package (transport: local, codec: json), and each
broker outage start and end the AMQP 1.0 transport sees on one of the package’s channels arrives
as a JSON message there. Unset, nothing changes.
Reference: Broker connectivity events. Walkthrough: Broker outages as messages.
For transport authors
sutra_channels::connectivity (ConnectivityReporter, ConnectivityEvent, current_send(),
re-exported by sutra-transport-spi) and the default method InboundIntake::connectivity(), a
no-op unless the engine installs a sink. Existing transports compile unchanged and report nothing.
Image scan
CVE-2026-95690 (libssl3, DTLS) in the distroless base has no fixed Debian package. The engine
never loads libssl3: on the 0.8.0 binary, readelf -d lists libgcc_s, libm, libc and the loader as
its only libraries and the binary names no libssl to load at run time, and nothing in it speaks
DTLS. The entry is time-boxed in .trivyignore.yaml to 2026-12-16 with the other base-image
entries.
Upgrading to 0.7.0
2026-11-19. A data task can delete a store row with <q:store key="…" delete="true"/>. It
adds an attribute to the q: namespace (urn:sutra:q:1.0), which is why this release is 0.7.0.
The addition is backward compatible: existing packages load and run unchanged. A data-store row’s
revision now starts from the database clock, so a row deleted and created again never reuses a
revision. A message that correlates to no live instance is recorded, and a channel can send it
to a process instead of refusing it. The AMQP 1.0 transport works with IBM MQ. Codecs can seal
what they send and resolve secret references in their own bundle files. One migration, V103, runs
at first boot on PostgreSQL. There is no archive or snapshot format change.
0.7.0 has no engine image. Its CLI archives and crates were published; its image was not (see Upgrading to 0.8.0). Engine image users take these changes with 0.8.0.
Before you upgrade
Check these against what you run:
- Rows created after the upgrade start from a large revision, not 1. An insert into a data store
now seeds
revwith the database clock in microseconds since the epoch (about 1.8 × 10^15 in 2026); each write still adds 1. Rows written before the upgrade keep their revisions. Anything outside the engine that readsrevand expects a new row at 1, or expects revisions with no gaps, must stop expecting it. Therevcolumn must be a 64-bit integer (BIGINT), as in every example in Data stores; check a typed store whose table you created yourself. - V103 builds an index concurrently at first boot (PostgreSQL:
alias_index_retired_lookup, a partial index over the correlation keys of ended instances). It blocks no writes and adds I/O in proportion to the table. From now on a completing instance writes one more index entry per correlation key it retires. - A message that correlates to no live instance is now a WARN log line and an audit record. A sender that keeps answering ended instances shows up in both. The HTTP answer is unchanged.
- A
local://delivery for an instance that has ended is given up at once. The refusal no longer carriessutra.waitPending=truewhen every instance that carried the key has ended, so it is not retried for about four minutes first. - If you expect timer surges, run more than one shard (
sutra.engine.shards,SUTRA_ENGINE_SHARDS, default1; see Configuration). Unchanged since 0.6.25: due timers fire as fast as they fall due, and with one shard that work shares the single execution lane with new work.
Deleting a data-store row
delete="true" on a write removes the key’s row instead of writing a value. Deleting a missing key
is a no-op. With expect="unchanged" the delete applies only while the row’s revision is the one
the instance read; otherwise it fails with SUTRA.RUNTIME.DATASTORE.CONFLICT, which an error
boundary catches. A delete runs where a write runs (on a <bpmn:transaction> scope’s store
transaction, else on its own), waits behind a forUpdate row lock, and invalidates the read cache
like a write. It works on key/value and typed stores on PostgreSQL, MySQL, MariaDB and SQL Server.
Combinations that make no sense for a delete are refused at load with
SUTRA.PARSE.STORE.DELETE_INVALID.
Reference and walkthrough: Deleting a row.
Revisions start from the database clock
Until 0.6.25 every insert started a row at revision 1. Once a process can delete rows, a key can
go from revision 1, to deleted, to created again at revision 1 with another value. Another
replica’s read cache would then serve the old row, and an expect="unchanged" write or delete that
read the old row would apply. Now the insert reads the database’s own clock inside the insert
statement, so every replica shares one clock. Revisions keep going up by one per write and never
repeat for a key, but they have gaps. This relies on the database clock moving forward: if it is
stepped backwards between a delete and the re-create, the guarantee can break.
A message no instance waits for
A message on a channel a wait subscribes to that finds no live instance carrying its key
(SUTRA.RUNTIME.RELAY.CORRELATION_NOT_FOUND) is now recorded whatever transport carried it: a WARN
log line with SUTRA.INBOUND.RELAY_UNCORRELATED, and an inbound.RELAY_UNCORRELATED audit record.
Both name the channel, the message type, the dedup key and the keys looked for, never the body or
the headers, and sensitive key values are masked.
uncorrelated: { start: <processId> } on the channel in channels.yaml sends such a message to a
process of the package instead, with no BPMN change. Details and a run of both:
An answer no instance waits for.
IBM MQ
The amqp transport now works with an IBM MQ queue manager’s AMQP channel. Two fixes apply to every
broker: each connection has its own container id (IBM MQ closes an open connection when another
opens with the same id, so concurrent sends and replicas cut each other off), and the sink sets the
message’s to property (IBM MQ refuses a message without it). New options, all off by default,
pick queue or topic addressing, persistence, expiry, the body form and the message properties; the
source and sink log when the broker is lost and when it is back. Also changed: the sink sends the
outbox key as message-id, and the source renders a binary message id as hex.
Setup and a walkthrough: Connecting the engine to IBM MQ.
For codec authors
PayloadCodec gains two default methods, inert unless a codec overrides them: seal, the last
transform of an outbound message before the outbox commit (an error refuses the message with
SUTRA.OUTBOUND.SEAL_FAILED), and bind_refs, which resolves secret: and env: references named
in the codec’s bundle files once, when the deployment is planned (an error refuses the deployment
with SUTRA.CODEC.BIND_REFS_FAILED). Existing codecs are unchanged. See
Domain neutrality and the SPI model.
Fixed
- A deployment flip could serve an older set of versions, or route a channel to a version whose
lanes were not built yet (database deployment source). Concurrent re-activations could
interleave, so a request for the new version’s channel answered 404
(
RESOLVE.CHANNEL.UNKNOWN), or a broadcast briefly had one subscriber. Re-activations are now serialized, and each reads the store afresh. - Building the engine image from source could ship stale code when several checkouts were built on one host: the Dockerfile’s shared cargo cache let changed crates pass as fresh. The build now decides by content. Published images are not affected.
BPMN 2.0 element support
Sutra executes BPMN 2.0. This page explains the rule that decides what it accepts; for the element-by-element answer, read the generated table rather than any prose:
rust/crates/sutra-bpmn/bpmn-support.md
That file is generated from the engine’s own Node enum and is compile-enforced: three exhaustive
matches mean a new node kind does not compile until it is described there, and a test loads a
minimal process per element to check the page against the loader rather than against a comment.
It therefore cannot drift from the implementation — which a hand-maintained list here would.
The rule
Where BPMN 2.0 already spells something the engine does, the engine accepts the standard spelling
and maps it onto existing behaviour. The q: namespace carries what BPMN does not model, and
stays visibly an extension.
In practice that means four things:
- A conformant element the engine can execute is accepted, even where an extension already
expressed the same thing.
<task>loads as the pass-through<manualTask>already is. - An element the engine genuinely cannot execute is rejected, loudly, with the diagnostic naming what to model instead. Silent inertness is worse than a refusal: a model that loads and quietly ignores half your diagram is harder to debug than one that will not load.
- Accepting something silently when you probably meant otherwise is itself a defect.
<task>loads andsutra lintwarns, because an author who meant<serviceTask>and typed<task>has written a step that does nothing. - Where BPMN and
q:both spell something, BPMN’s spelling is an alias onto the same model, not a second mechanism — and declaring one name both ways is a lint warning rather than a silent precedence.
What is rejected
One flow node: <event>.
That is not a gap. tEvent is abstract="true" in the BPMN 2.0 XSD — <event> is the
substitution-group head that concrete events substitute for, so no conformant document can use it
as a flow node and there is nothing for any engine to implement.
A handful of forms are also refused, each with a reason the diagnostic states:
| Form | Why |
|---|---|
<timeCycle> on an intermediate catch | a token parks there exactly once, so later occurrences have nothing to resume |
<timeCycle> on an interrupting boundary | the first firing cancels the host, so later occurrences are unreachable |
a bounded R<n> cycle on a boundary | the remaining budget would have to survive the host’s park, which needs durable per-instance state the engine does not keep. Use R/<interval> |
@instantiate="true" on <receiveTask> / <eventBasedGateway> | process instantiation needs the intake-selection surface a <startEvent> has |
eventGatewayType="Parallel" | meaningful only when instantiating; treating it as Exclusive would invert “wait for all” into “wait for one” |
an <eventBasedGateway> arm that does not park | an arm that resolves instantly makes every other arm unreachable |
| a durable park inside an inline sub-process | the inline runners cannot park mid-scope — model it at the top level |
Things that are accepted but carry no token
<dataObject> / <dataObjectReference>, <laneSet> / <lane>, artifacts and <documentation>
are modelling constructs, not flow nodes. A data object does now contribute a declared variable
when a data association names it — see
the q: namespace for how that interacts with <q:variables>.
The wait-state column is the one to read first
A process containing any node that parks is stateful, and ineligible for synchronous execution.
Most “does Sutra support X?” questions are really “does X park, and what does that imply” — which
is why the generated table has a column for it, checked against the engine’s own
Node::is_wait_state().
sutra CLI reference
The single sutra binary (crate rust/crates/sutra-cli) carries the whole authoring-to-operations
toolchain: scaffolding, validation, packaging, deployment, and analysis. Build it from source:
cd rust && cargo build --release -p sutra-cli # -> rust/target/release/sutra
cargo run -p sutra-cli -- <args> # or run ad hoc
Or install a published build — no Rust toolchain needed. The CLI runs on linux x86_64/arm64, macOS x86_64/arm64 and windows x86_64; see installation. The engine image it deploys to is Linux only, so on macOS and Windows the CLI is native and the engine is a container.
Commands are grouped below by workflow: author → validate → package → deploy → operate → analyze, with the generators last.
Global conventions
| Convention | Behavior |
|---|---|
--format <FORMAT> | text (default) or json for report commands; dot or mermaid for dispatch-graph |
-v / -vv / -vvv | Log verbosity (info / debug / trace); logs always go to stderr, reports to stdout |
Exit 0 | Clean run, no findings |
Exit 1 | Findings — the input has a diagnosable problem (breaking compat change, routing miss, coverage below threshold) |
Exit 2 | Usage or infrastructure — bad flags, missing files, unparseable input, unreachable database |
Database-touching commands (migrate, coverage check/reset, crypto) share one set of
connection options, each with an env fallback: --url (SUTRA_DB_URL), --user
(SUTRA_DB_USERNAME), --password (SUTRA_DB_PASSWORD), and — for migrate — --schema
(SUTRA_DB_SCHEMA). These are the CLI’s own database-connection names, distinct from the engine’s
SUTRA_DATASOURCE_* — see Configuration reference.
Author
sutra create app
sutra create app <NAME> [--dir <DIR>]
Scaffolds an application workspace: a sample standalone deployment package under
packages/<name>-main/ plus deploy assets (a compose file, a deployments drop-directory, a k8s
manifest, a health-gated smoke script). Idempotent-safe — existing files are never overwritten.
See Quickstart and Anatomy of an app.
sutra create deployment
sutra create deployment <NAME> [--dir <DIR>] [--from <PACKAGE_DIR>]
Scaffolds a fresh standalone deployment-package skeleton, or copies an existing package with
--from — the explicit variant model: packages never inherit, a variant is a copy. See
Deployment packages.
sutra create bpmn
sutra create bpmn <PROCESS> [--package <PACKAGE_DIR>] [--validation fatal|soft] [flags]
Generates a process with the validation-gateway wiring (plus accepted/rejected reply templates) into a package, verified through the engine’s own BPMN loader before being written.
| Flag | Meaning |
|---|---|
--validation <MODE> | fatal (default): only FATAL outcomes take the rejected branch; soft: FATAL and SOFT_ERRORS both reject |
--channel <NAME> | Inbound channel the start event subscribes to (default <process>-in) |
--message-type <TYPE> | Inbound message type (default <Process>Request) |
--force | Overwrite an existing user-edited file (never implicit) |
sutra create ci
sutra create ci --provider github|bitbucket [--dir <DIR>]
Writes the pipeline for an existing workspace — .github/workflows/sutra.yml or
bitbucket-pipelines.yml — as four steps in the order that fails cheapest first:
sutra packageover every directory underpackages/— fail-closed validation, so a package the engine would refuse to load never reaches a deployments directory;sutra generate docs … --check— the committed catalog still matches what it documents;deploy/smoke.shagainst the real engine and the archive just sealed;- publish — the catalog book to GitHub Pages, or printed to a PDF release artifact on a
v*tag under Bitbucket.
The provider is explicit rather than detected: a scaffolded pipeline lands in a repository that usually already has one, with its own runners and conventions, and guessing would be guessing wrong in the file most likely to be deleted on sight.
The pipeline checks documentation and never writes it. Regenerating in CI would author bot
commits and race step 2 against its own output. When step 2 fails the fix is local — run
sutra generate docs and commit the refreshed pages alongside the change that caused them, which
is what keeps the diff reviewable.
The release it installs is read from the same distribution seam self-update uses, pinned to the
version of the CLI that wrote the file. A distribution’s CLI, its engine image and its release
channel are one matched set: a pipeline that fetched some other distribution’s binary would
resolve a different codec registry than the archive it had just sealed.
Validate
sutra lint
sutra lint <PACKAGE_DIR | MANIFEST.yaml> [--overlay <DIR>]... [--active-set <PATH>]... [--engine-url <URL>]
Runs the full package-time validation suite (the same fail-closed checks sutra package runs)
and emits nothing on success. The fast pre-flight before every package.
--overlay composes base layers underneath <PACKAGE_DIR> before validating — the same
composition package performs, so you lint exactly what would be sealed. See
Composing a package from layers.
Given a target, lint also runs the check a deploy runs. --active-set names sealed archives, or
directories of them, standing in for what the target runs. --engine-url asks a running engine
(POST /admin/deployments/check; --api-key / --api-key-header as for deploy), which links every
transport it can serve and resolves connection references in its own environment. --active-set judges only
the transports the CLI links (HTTP) and reports a channel on any other as
SUTRA.DEPLOY.ENDPOINT.UNCHECKED. An exclusive endpoint another deployment holds is the error
SUTRA.DEPLOY.ENDPOINT.COLLISION; a data store that would share rows is the warning
SUTRA.DEPLOY.DATASTORE.SHARED_CONNECTION. See One endpoint, one
deployment.
Given a deployment manifest (a .yaml file) instead of a directory, lint first prints the
resolved stack — each layer in order, each overridden file with the layer that won and the one it
overrode, and each .ignore removal — then the findings; --format json carries the same under a
stack key. See
Deployment manifests.
sutra describe
sutra describe <BPMN_FILE> [--format json]
Prints a structural summary of a BPMN file: processes, events, tasks, gateways, channels. See Troubleshooting BPMN solutions.
sutra dispatch-graph
sutra dispatch-graph <BPMN_FILE> --format dot|mermaid
Emits a graphviz or mermaid diagram of a BPMN file’s dispatch tree.
sutra simulate
sutra simulate <BPMN_FILE> --channel <CHANNEL> --dry-run
Reports which process an inbound channel message would route to. --dry-run is required —
routing report only, no execution.
sutra explain
sutra explain [EXPRESSION] [--context <FILE>]
Evaluates a FEEL expression — one-shot, or a REPL on stdin when the expression is omitted. See Rules: DMN, FEEL, and .srl.
sutra compat-baseline
sutra compat-baseline --baseline <PATH_OR_REF> [--current <PATH>] [flags]
Compares current BPMN signatures against a baseline directory or git ref and reports breaking changes — a CI gate on process-contract compatibility between releases.
Package
sutra package
sutra package <INPUT | MANIFEST.yaml> [--overlay <DIR>]... [-o|--out <DIR>]
Seals a deployment-package directory into one immutable .sutra archive, running the full
validation suite fail-closed first. The archive manifest (per-file digests, the content-addressed
deployment id) is derived, never authored. See Deployment packages.
--overlay composes one or more base layers underneath <INPUT> — each applied in the order
given, with <INPUT> applied last and winning any conflict — so a shared folder plus a small
per-variant folder seal into one archive without being copied together by hand. An overlay may not
contain package.yaml, overridden paths are reported, and nothing is ever deleted. The resulting
deployment id is identical to sealing a hand-merged directory of the same content. See
Composing a package from layers.
<INPUT> may instead be a deployment manifest — a .yaml / .yml file naming the package
labels and an ordered stack of layer folders, each optionally mounted with at: and able to drop
lower-layer files with a .ignore. It seals to <manifest-file-name>.sutra, byte-identical to
sealing the same merged tree by hand; --overlay flags stack above its last layer:
sutra package deploy/customer-a.2026r1.prod.yaml -o dist/ # → dist/customer-a.2026r1.prod.sutra
See Deployment manifests for the format and a worked customer / edition / environment example.
sutra deployments list
sutra deployments list <DIR> [--label KEY=VALUE]...
Inspects a directory of sealed .sutra archives — lists each archive’s deployment id and labels.
sutra openapi
sutra openapi <ARCHIVE>
Emits a sealed archive’s generated OpenAPI 3.1 surface (channels → BPMNs, message types, endpoint nature, data stores) — the same document the engine serves live per deployment id.
Deploy
sutra migrate
sutra migrate [status|verify] [--url <URL>] [--dry-run] [flags]
Applies engine schema migrations to the engine-internal database, or inspects them (status —
applied vs. pending; verify — ledger integrity, expected head, checksum drift).
sutra crypto provision-dek
sutra crypto provision-dek --key-id <KEY_ID> --kek <REF> [--url <URL>] [flags]
Provisions a KEK-wrapped per-tenant data-encryption key for envelope encryption of sensitive instance variables at rest.
sutra deploy
sutra deploy [ARCHIVE] [flags]
Hot-deploys a sealed .sutra archive onto a running engine. See
Deploy, hot-deploy, and rollback and
Deployment model for the full mechanics.
[ARCHIVE] may also be a deployment manifest (.yaml): it is packaged first, exactly as
sutra package would, and its target: block supplies any of the flags below that the command
line leaves out — the command line wins. See
Deployment manifests.
| Flag | Meaning |
|---|---|
--api | Deploy via the engine’s synchronous admin API instead of a ConfigMap patch; requires --engine-url |
--async | With --api: submit async, then poll until Active |
--watch <PKG_DIR> | Watch a package source directory and re-deploy on change (validate-then-deploy loop); implies --api |
--wait / --wait-timeout <SECS> | ConfigMap path: poll until Active |
--engine-url <URL> | Engine base URL for --wait / --api |
--api-key <KEY> / --api-key-header <HEADER> | Admin auth for --api (default header X-API-Key) |
--secret <KEY=VALUE> / --secret-from <FILE> | Estate-secret keys ensured/merged before the ConfigMap patch |
A deployment claiming an exclusive endpoint another active deployment holds is refused with
SUTRA.DEPLOY.ENDPOINT.COLLISION and the old state serving (exit 1). On the ConfigMap path deploy runs the
check first, asking the engine when --engine-url is given. A new version of one tenant and module may take an
endpoint from an older one; the older drains whole (SUTRA.DEPLOY.ENDPOINT.SUPERSEDED warns).
--engine-url times out after 30 seconds and says why it got no verdict. See One endpoint, one
deployment.
sutra undeploy
sutra undeploy <DEPLOYMENT> [flags]
Removes a deployment — the engine drains it (no new intake) and retires it at zero instances and zero pending outbox.
Operate and analyze
sutra coverage
sutra coverage init <FILE> [PROCESS_ID]... [flags] # seed declarations / scaffold admin set
sutra coverage check [BPMN_FILE] [flags] # drift lint, or the store-backed check
sutra coverage reset --archive <FILE> [flags] # re-seed the store covered=false
Path-coverage tooling for q:coverage route declarations (intra-process) and coverage/*.yaml
files (cross-process) — see
Coverage: declared routes as the compliance signal for the full
walkthrough of both shapes and every flag above, and
Troubleshooting BPMN solutions
for reading a report that doesn’t match what you expected.
sutra test simulate
sutra test simulate --deployments <DIR> --datasource <URL>
(--advance <DURATION> | --until-quiescent) [flags]
Boots a real engine on a dynamic port against a directory of sealed deployment archives with a
virtual clock installed, fast-forwards it, reports, and shuts down — so a PT24H timer or an
R3/PT12H schedule settles in wall-clock seconds. Unrelated to sutra simulate above, which is a
dry-run routing report over one BPMN file and boots nothing.
| Flag | Meaning |
|---|---|
--deployments <DIR> | Directory of sealed .sutra archives to serve (required) |
--datasource <URL> / --datasource-username / --datasource-password | Engine datasource — the engine’s own SUTRA_DATASOURCE_* env names, not the CLI’s SUTRA_DB_* set (required) |
--advance <DURATION> | Fast-forward the virtual clock by this ISO-8601 duration, firing everything due along the way, then stop |
--until-quiescent | Fast-forward until nothing is armed and nothing is live, or --timeout elapses |
--timeout <DURATION> | Real wall-clock budget for the engine boot plus the fast-forward loop, either mode — since 0.6.5 a boot that has not finished within it ends the run (exit 2) instead of waiting (default PT30S) |
--start <RFC3339> | Virtual start instant (default: the real current instant) |
--allow-existing-data | Proceed even though the datasource already holds instances |
Exactly one of --advance / --until-quiescent is required. Safety: the target database must
hold no instances or the run refuses with exit 2 — pointing this at a database with real
in-flight instances would durably fire their real timers early; --allow-existing-data is the
explicit acknowledgement.
Fixed in 0.6.5: at a log level that logs while the engine boots (-v, or RUST_LOG), a
package in --deployments made 0.6.4 hang after “schema up to date” and ignore --timeout. The
CLI held the terminal’s stderr lock for the whole command, and the engine’s runtime threads
block on that lock to write their first log line. The CLI now writes to stderr without holding
it, and --timeout covers the boot.
Progress is text on stderr; the final summary is one JSON object on stdout and nothing
else touches stdout, so sutra test simulate … | jq . is always safe. See
Testing time for the summary’s fields and the embedded seam behind
this command.
sutra audit-replay
sutra audit-replay <INSTANCE_ID> --from-jsonl <PATH> [--tenant <TENANT>] [--until <EVENT_TYPE>]
Walks a process instance’s audit events from a JSONL stream — reconstructing what a specific instance did, offline. See Logging and audit.
sutra version
sutra version # sutra 0.8.3 (1a2b3c4, built 2026-11-15)
sutra --version # identical text
sutra version --format json
Prints the tool version and the build it came from. The parenthesis names the commit the binary
was built from (-dirty appended when tracked files had uncommitted changes) and that commit’s
date in UTC. It is never the wall clock, so two builds of one commit print the same line. An
unknown part is left out rather than guessed: sutra 0.8.3 (1a2b3c4) without a date, plain
sutra 0.8.3 when git is unavailable and no build variable was set. A release build takes both
from SUTRA_BUILD_COMMIT and SOURCE_DATE_EPOCH; a local build reads them from git. The engine
image’s sutra-engine --version prints the same, and its sutra-engine starting log line carries
version, commit and built fields.
The program name is derived from the running binary, not
hardcoded: a distribution that embeds this CLI as a library (sutra_cli::run) under its own
binary name prints that name, and one that versions itself independently of the engine
(sutra_cli::run_with_version) prints its own version with the embedded engine’s underneath:
<tool> 2.0.0
sutra 0.8.3 (engine)
The tool’s own version is its first line read up to the first (, so a distribution may print
its build on the same line: <tool> 2.0.0 (1a2b3c4, built 2026-11-16) still reports 2.0.0, which
is also the version self-update compares against and the tag of the runtime image create app
scaffolds (0.6.25).
--format json is the structured form and always separates the two — version is the
reporting tool’s own, engine the embedded engine’s (equal for this binary). commit and
built are present when known:
{"name":"sutra","version":"0.8.3","engine":"0.8.3","commit":"1a2b3c4","built":"2026-11-15"}
Generate
The generators all recompute output that is derived, not authored, and all share one
--check drift gate that regenerates without writing — which is the shape a CI or pre-commit
gate wants. That shared contract is why they sit under one verb. It is also what separates them
from sutra create, which scaffolds files that become yours: a scaffold is
headed “edit freely — this file is yours” and needs --force to overwrite your edits, while a
generated page is headed “Do not edit above the MANUAL NOTES sentinel” and --check fails the
build if you edited it. Two verbs, opposite guarantees — never one.
sutra generate docs
sutra generate docs --input <FOLDER> [--output <DIR>] [--check]
Recurses a folder of authored deployment artifacts — BPMN processes, DMN/.srl rules,
Handlebars/XSLT templates and their manifests, channels.yaml, package.yaml, coverage files —
and emits a deterministic markdown catalog, one page per artifact. It parses through the engine’s
own loaders, so each page describes exactly what the engine loads rather than a second parser’s
opinion of it. --check generates into a temporary directory and reports drift instead of writing
anything. sutra generate catalog is its sibling for Rust source — same --output / --check
contract, one page per source file, rooted at --repo-root. A deployment package wants docs;
catalog documents the engine’s own source tree.
Each BPMN page opens with a diagram in BPMN notation, auto-laid-out from the process graph —
authored BPMN carries no <bpmndi:BPMNDiagram>, so no coordinates are needed in the source. The
SVG is embedded in the page, scaled to the column, and linked to a full-size copy written beside
it (<name>.svg) that opens in a new tab.
sutra generate schema-handler
sutra generate schema-handler <SCHEMAS_DIR> <OUT_DIR> [--full] [--check]
Compiles a directory of XSD schemas into Rust sources: the decode tables, the canonical map
projection, and the shape metadata a schema-bound codec is built on. The schema files are the only
input, and emission is byte-identical run to run after rustfmt — which is what makes --check
(regenerate in memory, diff against the tree at <OUT_DIR>, exit 1 on any difference) a usable
drift gate. The default emission is the slim, data-driven form; --full additionally emits the
typed model. A write run touches only the files the generator itself produces and never the
hand-maintained ones alongside them — support.rs and Cargo.toml are yours.
Both paths are arguments, not conventions: it is a neutral tool over whatever corpus you point it at, and the crate it emits lives wherever the caller wants it — including in a repository that composes this one.
Template helpers
A Handlebars template (templates/*.hbs) renders in strict mode: a value that resolves to nothing
(a missing key, a misspelt path, a null part-way along a path) fails the render. {{x}} renders
x verbatim, with no escaping, because templates produce XML, JSON and plain-text messages, not
HTML.
The engine registers the helpers below and no others. Helper arguments are the exception to strict
mode: a missing path is a null argument, never a failed render. So (eq maybe "X") can test for
absence, and {{coalesce maybe "default"}} can supply a default.
| Helper | Renders |
|---|---|
eq a b, neq a b | true or false. Numbers compare as numbers, so 1 equals 1.0. |
gt a b, gte a b, lt a b, lte a b | true or false. Two numbers compare as numbers and two strings in code-point order; anything else is false. |
and a b …, or a b …, not a | true or false, with Handlebars truthiness: false, null, "", 0 and [] are false. |
{{#let value as |name|}}…{{/let}} | Its body, always, with value bound to name. A null binding fails the render only where the body uses it. |
substring value from to | The characters of value from index from up to, not including, to (to the end if to is left out). A range outside the value fails the render. |
replace value "a" "b" | value with every a replaced by b, literally (no patterns). |
coalesce value fallback … | The first argument that is not null. |
xmlEscape value | value with & < > " ' replaced by & < > " '. A null or missing value renders empty. (0.6.25) |
uuid and now are not helpers: the engine puts them in the render model, so a template reads
them like any other value.
xmlEscape
Use xmlEscape for any free text that goes into an XML message: a name, a note, an address
line. Without it, a customer called O'Brien & Sons or a note containing < produces a
message that is not well-formed XML, and the receiver refuses it.
<Receipt customer="{{xmlEscape order.customer}}"><Note>{{xmlEscape order.note}}</Note></Receipt>
- It is safe in element content and in attribute values in either kind of quotes.
- Every other character passes through unchanged, non-Latin scripts included.
- A number or boolean renders in its usual text form. A null or missing value renders empty,
where
{{order.note}}on a missing value would fail the render. - Escaping is opt-in per value.
{{x}}stays verbatim, so existing templates render as before. Don’t putxmlEscapeon a value that already holds XML markup you want kept: it escapes that too.
Escaping free text in an XML template runs it end to end.
Crates: sutra-feel, sutra-dmn and bpmn-svg
Every crate in the workspace is unpublished by default (publish = false at the workspace level)
except three, which are deliberately carved out as standalone, embeddable libraries:
sutra-feel,
sutra-dmn and
bpmn-svg. All are pure Rust, #![forbid(unsafe_code)],
and have no dependency on channels or persistence — you can use any of them in a project that has
nothing to do with this engine at all.
[dependencies]
sutra-feel = "0.8.3"
sutra-dmn = "0.8.3" # depends on sutra-feel directly; use it alone if you only need DMN
bpmn-svg = "0.1" # versioned on its own; depends on neither
sutra-feel — the FEEL expression language
FEEL (Friendly Enough Expression Language) is the expression language the OMG DMN specification
defines, reused throughout the BPMN/DMN tooling ecosystem wherever a small, side-effect-free
expression needs to be embedded in a larger model — a decision rule, a gateway condition, a
data-mapping assignment. sutra-feel is a complete, standalone implementation: lexer, parser,
AST, and evaluator, with no dependency on DMN or BPMN machinery.
#![allow(unused)]
fn main() {
use sutra_feel::{expressions, FeelContext, FeelValue};
let mut context = FeelContext::new();
context.insert("age".to_string(), FeelValue::Number("42".parse().unwrap()));
let result = expressions::eval("age >= 18", &context).unwrap();
assert_eq!(result, FeelValue::Boolean(true));
}
expressions::eval is the main entry point; expressions::parse parses without evaluating, and
expressions::eval_boolean is a convenience wrapper for expressions that must produce a boolean
(gateway-condition shaped callers). expressions::paths extracts which context paths an
expression reads without evaluating it — useful for static analysis over a set of expressions.
Highlights:
- Source-position-aware diagnostics (errors carry line/column, pinned to a caller-supplied source
URI) — the errors this crate raises are the same ones you’d see from
sutra explain(see the CLI reference) or from a deploy-time validation failure. - DECIMAL64 numeric semantics (16 significant digits,
HALF_EVENrounding, viabigdecimal) — arithmetic matches the FEEL specification, not native floating point. - Full temporal support: dates, times, date-and-time, and durations, including IANA-timezone-aware
@Region/Cityzone-qualified literals, with the timezone database bundled at compile time (no hosttzdatadependency). - Ranges/intervals, contexts (maps), lists, and function values, plus a documented determinism denylist for expressions that must stay side-effect-free (the engine uses this to keep replay deterministic across a wait-state resume — see Wait states and human tasks).
sutra-dmn — the DMN 1.5 decision engine
DMN (Decision Model and Notation) is the OMG standard for decision tables, decision requirement
graphs (DRGs), and business knowledge models, tied together by FEEL. sutra-dmn loads .dmn XML
files, validates them, and evaluates them — single decision tables (all seven OMG hit policies:
UNIQUE, ANY, PRIORITY, FIRST, OUTPUT ORDER, RULE ORDER, and COLLECT with SUM/COUNT/MIN/MAX
aggregation) as well as full DRGs, where a decision depends on other decisions, invokes business
knowledge models, or calls decision services. FEEL parsing and evaluation is delegated directly to
sutra-feel.
#![allow(unused)]
fn main() {
use sutra_dmn::DmnDecisionEngine;
use sutra_feel::{FeelContext, FeelValue};
let engine = DmnDecisionEngine::new();
let result = engine.evaluate("tier.dmn", TIER_DMN_XML.as_bytes(), &input).unwrap();
assert_eq!(result.get("tier"), Some(&FeelValue::from("GOLD")));
}
DmnDecisionEngine::evaluate is the single-file entry point, returning a map of output-clause
name to FeelValue — this is exactly what backs a businessRuleTask in the engine (see
Rules: DMN, FEEL, and .srl). For DRG evaluation across a graph of
decisions/BKMs/decision services with imports, see the crate’s drg module; for structural
validation without evaluation, see DmnRulesetValidator.
Conformance is measured, not asserted — see DMN-TCK conformance for the current standing and what it does and doesn’t cover.
bpmn-svg — BPMN diagrams as SVG
bpmn-svg draws BPMN 2.0 process diagrams as self-contained SVG, with a layered auto-layout for
files that carry no BPMN-DI. parse reads standard BPMN into a small model and ignores BPMN-DI and
vendor extensions, and render draws it. The same input always gives byte-identical SVG, and an
option writes colours as CSS variables for light and dark themes. sutra docgen draws through it.
Its README has a
runnable example and its known limits.
Why these and not the rest of the workspace
Everything else in the workspace — the BPMN model and executor, the channel/transport layer, the
persistence layer, the CLI — is Sutra-engine-internal: coupled to the engine’s own conventions
(diagnostics, deployment packages, the q: namespace) in a way that wouldn’t make sense as a
general-purpose crate. FEEL and DMN evaluation, by contrast, are useful on their own to anyone
building rules or decision logic in Rust, independent of whether BPMN or Sutra is anywhere in the
picture, and drawing a BPMN file is useful to anything that handles one — so those three are
published, versioned, and supported as standalone libraries.
The model-call crates
Unpublished, and listed because they are the extension points behind the llm transport (see
Calling a language model):
| Crate | What it is |
|---|---|
sutra-llm-spi | The provider-neutral SPI: request, response and failure types, the LlmProvider trait, the registration each provider crate submits, and a shared JSON/SSE HTTP client. Names no provider. |
sutra-llm-openai | The Chat Completions adapter, serving openai and the compatible services (google-gemini, mistral, deepseek, xai, kimi, meta-llama) or any compatible server via base-url. Streaming and background jobs with webhooks. |
sutra-llm-anthropic | The Anthropic Messages API adapter. Streaming; jobs through Message Batches, polled. |
sutra-llm-bedrock | The Amazon Bedrock Converse API adapter (aws-bedrock), authenticated with a Bedrock API key. |
sutra-transport-llm | The llm transport: ordered candidates with fallback, capability filtering, sync and async calls, jobs, signed webhooks and SSE streaming. Names no provider. |
sutra-dist links the three adapters behind the llm-openai, llm-anthropic and llm-bedrock
features, all on by default.
Next
- DMN-TCK conformance — the numbers behind “measured, not asserted.”
- Rules: DMN, FEEL, and .srl — how these two crates show up inside a Sutra process.
DMN-TCK conformance
Sutra’s DMN/FEEL evaluator (sutra-dmn + sutra-feel, see
Crates: sutra-feel and sutra-dmn) is checked against the real OMG
DMN Technology Compatibility Kit — the same corpus other DMN
implementations are measured against — rather than relying on an internal test suite alone.
Current standing
| Level | Result |
|---|---|
| Compliance level 2 | 126/126 (100%) |
| Compliance level 3 | 3365/3385 absolute (99.4%), and 100% of attempted assertions (0 semantic failures among cases the engine attempts) |
Measured against the TCK as of 2026-09-17 (commit 0456e1a), including its newest tests for
multiple model imports and DMN 1.5’s implicit type conversions.
The gap between “100% of attempted” and “99.4% absolute” is 20 enumerated assertions the
evaluator deliberately doesn’t attempt: 14 are reflective execution of {java|pmml: …} external
function bodies (the grammar parses and validates their binding shape; invoking one returns a
clear, deliberate semantic rejection rather than a wrong answer), and 6 are dates with nine-digit
years, beyond the range the engine’s temporal values represent. This is
the honest distinction the numbers are built to preserve: a wrong value on a supported construct
is a real conformance defect, and the harness that produces these numbers is built specifically to
never launder one into an “unsupported” bucket. Level 2 has no such gap at all.
What this does and doesn’t tell you
What it tells you: decision tables (all seven OMG hit policies, including COLLECT with aggregation), decision requirement graphs, business knowledge models, decision services, and the breadth of FEEL itself (temporal types and arithmetic, ranges/intervals, list/context comprehensions, string and numeric builtins, filters and projections) behave per specification across a large, independently-authored corpus — not just the cases Sutra’s own contributors thought to write.
What it doesn’t tell you: conformance to a language specification says nothing about Sutra’s BPMN coverage, its channel/transport behavior, its persistence and replica semantics, or its security posture — those are covered elsewhere in this book (see Building BPMN solutions and Architecture). It also isn’t a performance claim — conformance and throughput are separate properties.
Where this comes from
The conformance measurement is produced by a development-time harness in the workspace — it is a
tool contributors run when working on the FEEL/DMN evaluator, not something a user building a
BPMN solution ever needs to invoke. See
Debugging the engine if you’re contributing to
sutra-feel or sutra-dmn and need to reproduce or extend this measurement yourself.
Next
- Rules: DMN, FEEL, and .srl — using DMN inside a Sutra process.
- Crates: sutra-feel and sutra-dmn — using the same evaluator standalone.
Contributing
Contributions are welcome! This page covers the repo map, the day-to-day dev workflow, the code
style and PR bar, and the license. If you’re an AI agent working on this codebase, also read
CLAUDE.md at the repository root —
it’s a shorter, more mechanical companion to this page written specifically for that audience.
Repository map
The engine, tooling, and libraries are a single Cargo workspace under rust/:
| Path | What it is |
|---|---|
rust/ | The Cargo workspace — engine, CLI, libraries, tools. Start at rust/README.md; build/test tiers in rust/TESTING.md. |
docs/ | This book (mdBook) — book.toml + src/SUMMARY.md. |
examples/ | End-to-end example apps (money-transfer, approval-hold). See Worked example: money-transfer. |
deploy/ | Reusable OpenTofu deployment modules. |
xsd/ | Authoritative XSD schemas for BPMN, DMN, and the q: extension namespace. |
openapi/ | API specifications. |
scripts/ | Repository-level dev/ops scripts. |
tools/ | User-facing tooling: sutra-vscode, sutra-modeler-plugin, sutra-load-test. |
Within rust/crates/, the layering described in
Engine layering maps directly onto directories — sutra-engine (the
library), sutra-dist (the composition root that builds the actual binary), sutra-channels,
sutra-executor, sutra-bpmn, sutra-feel/sutra-dmn/sutra-srl/sutra-templates,
sutra-persistence, sutra-formats (the built-in formats) alongside sutra-codec-schema, one
sutra-transport-<vendor> crate per broker, and the SPI crates each of those builds against
(sutra-codec-spi, sutra-transport-spi, sutra-datastore, sutra-envref-spi,
sutra-redactor-spi) — see
Domain neutrality and the SPI model for exactly how the
pieces fit and how to add a new one.
Development workflow
You need a stable Rust toolchain (rustup); Docker is only needed for the
container/integration test tiers, and tofu plus a kind cluster only for the Kubernetes tier.
make help (from the repo root) lists every target; full detail lives in rust/TESTING.md.
cd rust && cargo build # whole workspace, debug
cargo build --release -p sutra-cli # the `sutra` CLI binary
cargo build --release -p sutra-dist # the `sutra-engine` binary (the composition root)
Test tiers
| Tier | make target | What it needs |
|---|---|---|
| 1 | make test | Rust toolchain only — the default gate while iterating |
| 2 | make test-docker (P=<crate> narrows to one crate) | A Docker daemon |
| 1+2 | make test-all | A Docker daemon |
| 3 | make test-k8s | A running kind cluster (make -C deploy/k8s-it init first) |
The CI workflow (.github/workflows/ci.yml) mirrors the local gate chain exactly. make ci runs
the same jobs locally, cheapest first: make actionlint, make fmt-check, make lint,
make cross, make test, make audit and make scan-deps (Trivy over the lockfiles, secrets and
IaC files). Every Trivy scan prints its vulnerability database’s UpdatedAt; make scan-deps RELEASE=1 and make scan-image RELEASE=1 fetch the latest database first, as a release gate should. A contributor who is green locally is green in CI. No CI job builds an image: the
release builds the image it ships, once, and scans it before tagging it. Tier-2 runs nightly rather than per-PR (a Docker-heavy suite is slow and
resource-flaky on a small hosted runner for signal tier-1 already covers on most changes); tier-3
stays a local/milestone gate, since it needs a provisioned kind cluster CI doesn’t have.
CodeQL runs on its own schedule, and a weekly Trivy run scans the tree and the newest published
image for vulnerabilities disclosed since they shipped.
Tier-1 covers the whole workspace — no crate is held out of it, so a green make test means the
whole workspace is green, not a subset of it.
Lint and supply-chain gates
make lint # cargo clippy --workspace --all-targets -- -D warnings,
# plus the sutra-archtest domain-neutrality suite
make audit # cargo audit (RustSec advisories) + cargo deny check (rust/deny.toml)
make lint failing on a domain-neutrality violation means a business term landed in a crate
the gate enforces — move the
concrete logic into its own extension crate instead.
The generated catalog (on demand, not committed)
make catalog # regenerate the artifact-documentation catalog under catalog/
make catalog-check # verify it's in sync
The catalog — one page per source file plus its dependency relationships, produced by
sutra-catalog-gen — is generated on demand and git-ignored; there’s no committed baseline
to diff against, so it isn’t part of the CI gate today. Run it locally when you want the
dependency picture for a change you’re making; make install-hooks wires an optional pre-commit
regeneration hook if you want it kept fresh automatically.
Code style and the PR bar
- Discuss anything large first. A new transport, a new codec, an architectural change — open an issue before you invest the time.
- Keep PRs focused. One logical change reviews faster than a bundle of unrelated ones.
- Be green locally before pushing:
cd rust cargo fmt --all cargo clippy --workspace --all-targets -- -D warnings cargo test --workspace - Add a
CHANGELOG.mdentry under[Unreleased]for a user-visible change. - Commit messages are short, imperative, area-prefixed:
feat(channels): add AMQP 1.0 outbound reply path fix(feel): correct DECIMAL64 rounding on division docs(concepts): clarify wait-state correlation keys
The zero-warning clippy bar and the domain-neutrality gate are both hard requirements, not style preferences — see Domain neutrality and the SPI model for why the latter exists.
Diagrams in this book
Diagrams are mermaid in fenced ```mermaid blocks, rendered by
the mdbook-mermaid preprocessor (put it on your PATH, or your local build shows the source
instead of the picture). Four rules keep them consistent:
- No hardcoded colors — no
style … fill:#…, noclassDef. The book has a light and a dark theme and mermaid picks its own palette for each; a hardcoded fill is unreadable in one of them. Shape, label and subgraph carry the meaning. - A diagram earns its place by compressing something the reader would otherwise hold in their head — an ordering, a state machine, a fan-in, a comparison. A bullet list drawn as boxes is not a diagram.
- Keep it small (roughly a dozen nodes). If it needs more, it is two diagrams.
- Label the edges where the label is the information (
park,after PT30S,hash(instanceId)), and put one or two sentences under the diagram naming the takeaway.
The diagram must say what the prose says: no fact appears only in a picture, and nothing in a picture contradicts the text.
Diagrams render at their natural size rather than being scaled down into the content column (that scaling is what makes wide diagrams unreadable), so a wide one scrolls inside its own block. Clicking any diagram opens it full size in a new tab, where the browser’s own zoom, save and print apply — which is also the escape hatch when a diagram is genuinely large.
Where designs live
Deeper design rationale than this book covers — why a mechanism was built the way it was, staged
plans for work in flight — lives in docs/design/ in the source (private working) repository
this public repo is curated from; a subset is rewritten into the chapters of this book as it
stabilizes, rather than published wholesale. If you’re contributing a change whose “why” isn’t
already captured in this book, a short design note alongside your PR is the right place for it.
License
Sutra is dual-licensed under MIT OR Apache-2.0 (LICENSE-MIT, LICENSE-APACHE). Unless you state otherwise, any contribution you submit is licensed under the same terms, with no additional conditions — you retain copyright over your own contributions.
Please also read our Code of Conduct and Security Policy.
Next
- Debugging the engine — tracing targets, reproducing a bug at the right test tier, and the development-only tools (like the DMN-TCK harness) that live alongside the workspace.
Debugging the engine
This page is for contributors working on the engine itself — its executor, model, channel dispatch, or the FEEL/DMN evaluators. If you’re building a solution on top of Sutra and something in your BPMN/rules/config isn’t behaving, see Troubleshooting BPMN solutions instead — that page (and its CLI-based tools) is the one written for that audience.
Tracing targets
Every crate logs through tracing, and its module path is the target RUST_LOG filters on — so
you can raise verbosity on exactly the layer you’re chasing a bug through without drowning in
noise from the rest of the workspace:
RUST_LOG=sutra_channels=debug,sutra_engine::deploy=trace cargo test -p sutra-channels
RUST_LOG=sutra_executor=trace cargo run -p sutra-dist
The default filter is info. When you’re not sure which crate owns the behavior you’re chasing,
start from Engine layering — a message’s path through the system
(channel → dispatch → executor → persistence) is also the order to add targets in.
Reproduce at the lowest tier that can show the bug
The workspace’s three test tiers (rust/TESTING.md; summarized in
Contributing) aren’t just a CI cost control — they’re the natural
escalation order for reproducing a bug:
- Tier 1 first. Most bugs in the executor, the BPMN model, FEEL/DMN/
.srl, or template rendering reproduce with a plain unit or no-docker integration test — no Postgres, no broker, nothing to spin up. Add it as a new module in the crate’stests/all.rs(a new file undertests/all/plus onemod <name>;line — not a new top-leveltests/*.rs, which would reintroduce a separate link unit) and run it in isolation:cargo test -p sutra-executor gateways_test::your_new_case - Escalate to tier 2 only if the bug needs real infrastructure — a specific PostgreSQL
locking behavior, a RabbitMQ redelivery edge case, a replica-convergence race. Mark the test
#[ignore = "docker"]and callsutra_testkit::reap_on_exit(container.id())on whatever container handle you spin up, so the shared-fixture pattern doesn’t leak it (seerust/TESTING.md’s “Reaper” section for why that call is required rather than relying onDrop). - Escalate to tier 3 only if the bug is genuinely multi-replica or Kubernetes-specific —
leader-election handoff, ConfigMap-driven activation timing, ingress-level deploy behavior. This
tier needs a provisioned
kindcluster (make -C deploy/k8s-it init) and is the most expensive to iterate on — reach for it last, and only once you’ve ruled out that a tier-1 or tier-2 test could show the same thing.
A bug fixed without a new failing test at the tier that caught it isn’t really pinned — the next regression in the same area won’t be caught until it reaches production again. Write the test first, watch it fail against the unfixed code, then fix it.
Catalog-guided navigation
make catalog # regenerate catalog/ (git-ignored, generated on demand)
sutra-catalog-gen walks every crate and emits one markdown page per source file, plus a
bidirectional relationships table — what a file depends on, and what depends on it. When you’re
about to change a shared type (an SPI trait, a diagnostic code, a config key) and need to know
every call site that will need updating, this is faster and more complete than a plain grep: it’s
generated from the actual syn-parsed dependency graph, not a text search that can miss a
re-export or an indirect reference.
The DMN-TCK harness — a development tool, not a user-facing command
The DMN-TCK conformance numbers this book cites come from a harness that
ships in the workspace but is gated and external: the OMG TCK corpus itself is
ASL-2.0-licensed and isn’t vendored into this (unlicensed) repository, so it’s never something a
person building a BPMN solution runs. If you’re working on sutra-feel or sutra-dmn:
git clone https://github.com/dmn-tck/tck.git # or wherever your checkout lives
SUTRA_DMN_TCK_DIR=/path/to/tck cargo test -p sutra-dmn --test tck -- --ignored --nocapture
The harness classifies every assertion as PASS (matches expected), FAIL (a real conformance gap —
the engine produced a different value), or UNSUPPORTED (the engine couldn’t evaluate the
construct at all — recorded, not counted as a failure). Two env-gated side outputs help when
you’re closing a gap: SUTRA_DMN_TCK_DUMP=<file> writes every non-pass outcome for gap analysis,
and SUTRA_DMN_TCK_RESULTS_DIR=<dir> writes the official vendor-submission
tck_results.csv/tck_results.properties pair. When you land an increment, re-run the harness and
update the numbers in DMN-TCK conformance alongside your change — that
page is a measurement, and a stale measurement is worse than none.
Next
- Domain neutrality and the SPI model — the structural rule most contributions have to respect, and the gate that enforces it.
- Troubleshooting BPMN solutions — the solution-developer counterpart to this page.