Introduction

Sutra is a Rust-native, message-native workflow engine built on BPMN 2.0. It turns a standard BPMN process into a declarative way to consume typed, schema-validated messages off any channel, route and correlate them, pause for human decisions, and reply — as a single statically-compiled binary (no JVM, no GC) that is multi-tenant, content-addressed, and hot-deployable.

Most workflow engines treat the payload as an opaque blob and start a process with a REST call. Sutra makes the message a first-class, typed contract: a start event binds to a channel (HTTP or a broker) and a message type; the engine decodes the real wire format, validates it against its schema on the way in, and drives the process with the typed payload, surfacing violations as routable soft errors rather than exceptions. The engine itself ships the schema-less structural formats (JSON, XML, YAML, CSV, raw text and bytes); a typed contract comes either from the deployment package — drop your XSDs into it and the engine compiles them when the package deploys — or from an extension crate implementing the codec SPI, which is how a message standard (an industry wire format with its own envelope grammar and schema editions) is served.

The engine core is domain-neutral: no business vertical lives in the engine itself — it lives entirely in codecs, channels, and modules. It also collects no telemetry of its own — no usage statistics, no crash reports, nothing phones home, ever; the only data a running engine ever sends anywhere is what you explicitly configure it to export (see No telemetry, no phone-home) — worth stating plainly for an engine meant to carry sensitive, typed messages.

Why Sutra exists

Nearly every production-grade BPMN 2.0 engine is JVM-based; the production-grade native-language orchestrators (Temporal, Cadence, Argo) deliberately aren't BPMN. Sutra is built for that gap — a native-compiled, standards-oriented BPMN engine that treats the message as the contract.

What Sutra ships

Sutra is four things, and this book covers all of them:

1. The engine — a container image built from the sutra-dist composition root (docker build -f rust/Dockerfile rust/), the thing you actually run. It force-links the built-in formats, transports, redactors, and secret resolvers into one deployable binary.
2. The sutra CLI — one binary (rust/crates/sutra-cli) that scaffolds, validates, packages, deploys, and inspects deployments. See Getting started and the CLI reference.
3. Two standalone library crates — sutra-feel (the FEEL expression language) and sutra-dmn (the DMN 1.5 decision-table core). Both compile and run independently of the engine — embed FEEL or DMN evaluation in your own Rust project without pulling in BPMN, channels, or persistence.
4. This book — Getting Started, Building BPMN Solutions, Architecture, Operating, Reference, and Contributing.

Honest capability summary

• BPMN 2.0 coverage. Start/end/intermediate events, all four gateway kinds, embedded / transaction / ad-hoc / event sub-processes, service/script/business-rule/user tasks, data objects and data stores, link/escalation/error events, compensation, and path-coverage instrumentation for compliance reporting — see Coverage: declared routes as the compliance signal.
• DMN + FEEL conformance — measured, not asserted. Sutra's DMN/FEEL evaluator is checked against the real OMG DMN Technical Compatibility Kit. Current standing: compliance level 2, 100% (126/126); compliance level 3, 99.4% absolute (3349/3369), with 100% of attempted cases passing (0 semantic failures — the remaining gap is a small, enumerated set of out-of-scope external-function-execution cases). See DMN-TCK conformance for what that means and what is deliberately out of scope.
• Channels and transports. HTTP, five brokers (Kafka, RabbitMQ, AWS SQS, Google Pub/Sub, AMQP 1.0), and an air-gapped file transport, all behind one neutral transport SPI — see Channels and transports.
• Typed codecs — the machinery, not a catalog of standards. Decode and schema-validation are one step, and what a codec yields is a messageType, a walkable payload, and a shape every FEEL path in the process is checked against at load time. Six formats are built in (JSON, XML, YAML, CSV, raw text, raw bytes). A schema-bound codec comes from one of two places: the deployment package itself — XSDs under schemas/<name>/, compiled at deploy time, no Rust and no engine build — or an extension crate against the public codec SPI, which is what a message standard (an industry wire format with its own envelope grammar, wrapper profiles, and schema editions) needs and what a downstream distribution force-links for itself. The engine ships no domain codec of its own; see Domain neutrality and the SPI model.
• Stateful execution. Wait states (userTask, intermediate message catch) suspend, persist, and rehydrate on PostgreSQL, correlated by a business key you name — see Wait states and human tasks. State shared across instances lives in a data store, which is key/value by default — one row per key, holding a value of any shape — and which can additionally project a flat record onto real typed columns, so the SQL tooling you already point at that database reads it directly.
• What's still thin. A one-line CLI installer and a published container registry image ship with the first tagged release (see Installation); until then the CLI and the engine image are built from source. Where a chapter below is a stub, it says so explicitly rather than padding.

Where to go next

• Installation — get a toolchain and build the engine.
• Your first app — scaffold an app and watch a typed message flow through: decode → validate → route → reply.
• Concepts — the ideas that make Sutra different: typed message contracts, channels, wait-states, the q: vocabulary, and content-addressed deployment.

This book is a work in progress. Chapters are being filled in as the 1.0 release approaches.

Installation

Sutra ships as two things: the sutra-engine runtime (a container image) and the sutra CLI (a single binary that packages, deploys, and inspects deployments).

Coming with the first tagged release: a one-line CLI installer (curl … | sh, like kubectl / kind / tofu) and a published engine image on a container registry. Until then, build from source as below.

Prerequisites

• A stable Rust toolchain (rustup).
• Docker — only for the container / integration test tiers.
• OpenTofu (tofu) + a kind cluster — only for the Kubernetes tiers.

Build from source

git clone https://github.com/startr-trade/sutra.git
cd sutra

# Build + test the workspace (no Docker needed)
make test
make lint

# Build the engine container image
docker build -t sutra-engine:dev -f rust/Dockerfile rust/

# Run the CLI
cd rust && cargo run -p sutra-cli -- --help

Next

Head to Your first app to scaffold and run a Sutra application.

Your first app

The sutra CLI scaffolds a complete, deployable application — BPMN process, channel bindings, a sample message type, and an OpenTofu deploy — so you can watch a typed message flow through the engine end to end.

# Scaffold a new app
cd rust && cargo run -p sutra-cli -- create app my-first-app

This produces an app with:

• a BPMN process wired to a channel and a message type,
• a codec (format × schema) that decodes and validates the inbound payload, and
• a reply on the inbound channel.

From there you package the app into a sealed .sutra archive and deploy it:

sutra package ./my-first-app
sutra deploy  ./my-first-app.sutra

Then send a message on the bound channel and watch it decode, validate, route, and reply.

The runnable, end-to-end walkthroughs live under examples/ — each has its own README with the exact package / deploy / curl lines. Start with money-transfer or approval-hold.

Next

Continue to Your first deployment to deploy that archive over the synchronous API and see exactly what "Active" means. Or skip ahead to Concepts to understand why the message, not a REST call, is the contract.

Your first deployment

sutra deploy has two paths onto a running engine: a ConfigMap patch (the default, for a Kubernetes deployment source) and the synchronous API (--api, against an engine running the db deployment source, or any engine you can reach directly). This page walks the API path, because it is the deterministic one — the call returns only once the deployment is Active, so what you see on the wire matches what actually happened.

What "deploy" means

The deploy unit is one sealed .sutra archive. sutra package runs the full fail-closed validation suite and derives the archive's manifest — including its content-addressed deploymentId = sha256(manifest) — from the package directory; nothing in the manifest is hand-authored. Deploying that archive is idempotent: re-deploying identical bytes is a no-op, and a changed archive under the same slot (its stable tenant--module--version key) replaces the slot's active revision in one transaction — a hot-deploy, not a restart.

Deploy over the API

Point the CLI at a reachable engine and hand it a sealed archive:

sutra deploy my-first-app-main.sutra --api --engine-url http://localhost:<port>

What happens, in order:

1. The CLI uploads the archive bytes to POST /admin/deployments.
2. The engine re-verifies the archive fail-closed (sutra_loader::read_archive) — a corrupt or invalid archive is rejected here, before anything is stored.
3. The engine stores the archive as the new active revision for its slot, in its own datasource.
4. The engine runs its in-process two-phase activation flip (drain the old revision if one exists, activate the new one).
5. The call returns synchronously: 200 {deploymentId, phase: "Active"} on success, or a 4xx carrying the SUTRA.DEPLOY.* reject diagnostic on failure.

There is no propagation window to wait out — the HTTP response is the "it's live" signal. For a deployment large enough that the engine's own plan-and-flip work risks a long-held request (mainly a concern behind a Kubernetes ingress with a short proxy-read-timeout), the same endpoint accepts an async mode: it returns 202 {deploymentId, status: "Pending"} immediately and you poll GET /sutra/deployments/{id} (or request a completion webhook/broker notification) until it flips to Active or Failed. Small local deploys — everything in this book's examples — use the synchronous form.

What you see

A successful deploy leaves you with:

• A running deployment. GET /sutra/deployments/{id} reports Active, the same status sutra deploy --wait polls on the ConfigMap path.
• A live channel. Whatever channels the package declared in channels.yaml are now bound and serving — an HTTP channel accepts requests immediately; a broker channel's consumer is running (or, for a singleton: true channel, running on whichever replica currently holds the per-channel lease).
• Nothing extra. Deploying does not create infrastructure — no database, no broker, no ingress. Those are provisioned separately (locally via docker compose, in a cluster via the OpenTofu modules under deploy/); the deploy call only ever registers and activates the package's processes, channels, and data-store bindings against infrastructure that already exists. See Deployment model for the full model, including how a fleet of replicas converges on the same active set.

Hot-deploy and rollback

Edit a package's source, re-package under the same slot, and re-deploy — the archive gets a new content-addressed deploymentId, but the slot name doesn't change, so the flip happens in-process with in-flight instances on the old revision left to drain. Rolling back is the identical operation in reverse: re-deploy the previous archive under the same slot. See Deploy, hot-deploy, and rollback for the operator-facing detail.

Next

You've scaffolded an app, packaged it, and deployed it. Building BPMN solutions picks up from here — the ideas and file formats behind what you just deployed.

Concepts overview

Sutra is organized around a few load-bearing ideas. This page is the map; deeper chapters will expand each one.

The message is the contract

A start event binds to a channel and a message type. The engine decodes the real wire format and validates it against a schema before the process runs, so malformed input becomes a routable soft error, not an exception deep in the flow.

Codec = format × schema

A codec pairs a format (how bytes are structured) with a schema (what a valid message looks like). Decoding and validation are one step. One schema can declare several message types, and dispatch fans out one process per type. Six formats are built in — JSON, XML, YAML, CSV, raw text, raw bytes; a schema-bound codec comes either from the package itself (XSDs under schemas/<name>/, compiled at deploy time) or from an extension crate implementing the codec SPI, which is how a message standard (an industry wire format with its own envelope grammar and schema editions) is served.

Channels and transports

Processes are triggered by messages arriving on channels — HTTP, five brokers (Kafka, RabbitMQ, AWS SQS, Google Pub/Sub, AMQP 1.0), and an air-gapped file transport. <q:source> binds a start event to a channel + message type. Co-deployed processes hand off over an in-process local:// channel.

Wait-states and correlation

Wait states (userTask, intermediate message catch) suspend → persist → rehydrate → resume. A later message is correlated back to the parked instance by a business key you name (<q:alias> — e.g. an EndToEndId or an order reference), not an engine id, and it's durable and replica-coherent on PostgreSQL.

Durable state beyond one instance

Wait-state data belongs to one instance. State that outlives an instance, or is shared between them, lives in a data store — declared in the package (datastores.yaml), read and written from the flow through <q:store>, and transactional, with optimistic concurrency (expect="unchanged") and pessimistic locking (forUpdate="true") where a read-modify-write needs them. Each store owns its own connection: the engine's database is never the module's. A store is key/value by default and holds a value of any shape; a flat record may additionally declare its structure: and project onto real typed columns in a table you own, so the SQL tooling you already point at that database reads it directly.

Rules — DMN, .srl, and FEEL

A businessRuleTask binds a .dmn decision table or a .srl ruleset (a Drools-inspired rule / when / then DSL). Both compile onto one shared FEEL evaluator — no JVM, no Rete runtime.

The q: vocabulary

Layered BPMN extensions keep diagrams standard while removing boilerplate: <q:dispatch> / <q:case> (a routing table instead of gateway sprawl), <q:validators>, <q:alias>, <q:reply>, <q:variables>, <q:audit>, and <q:coverage>.

Content-addressed deployment

The deploy unit is one sealed .sutra archive; runtime identity is a single opaque deploymentId = sha256(manifest). Deploys are idempotent, hot-reload flips activation without a restart, and secrets never live in the archive — channels reference them by scheme (secret: / env: / vault: / aws-secrets: / …), resolved at runtime through one vendor-neutral SPI.

Multi-tenancy and observability

Per-channel tenant binding, single-database PostgreSQL with per-deployment row-level security across PG / MySQL / MariaDB / MSSQL, OTLP traces / metrics / logs to any collector, plus KEDA autoscaling and leader election.

Deployment packages

A Sutra project is a single deployment package. sutra create app produces exactly one; sutra package <dir> seals it into one immutable, content-addressed .sutra archive; sutra deploy activates that archive on a running engine. There is no build step in between — you author declarative resources, seal them, and hand the sealed archive to a generic engine binary that never needs your source.

Package interior

<package>/
├── package.yaml             # manifest: labels {tenant, module, version}, engine.minContract
├── bpmn/**/*.bpmn            # BPMN 2.0 processes
├── rules/                    # DMN decision tables (.dmn) + the .srl rule DSL, by extension
├── scripts/**/*.{hbs,xsl,xslt}     # derive/compute (Handlebars + XSLT)
├── templates/                # render output (Handlebars + XSLT)
├── schemas/                  # each leaf folder under here = one codec
│   └── <codec>/*.xsd + codec-manifest.yaml
├── migrations/<store>/**     # per-datastore SQL migrations
├── channels.yaml             # transport channels (bind codec URNs; hot-reloaded on flip)
└── datastores.yaml           # datastore declarations

Every artifact folder supports nested subfolders — bpmn/orders/checkout/…, templates/eu/invoices/… — and discovery is recursive. sutra create deployment <name> --from packages/my-app scaffolds a sibling package as an explicit copy; packages never inherit from one another.

package.yaml — labels, not scope

labels:
  "module": "money-transfer"
  "tenant": "default"
  "version": "1.0.0"
engine:
  minContract: 1

tenant / module / version are opaque labels — selectors for observability, routing, and row-level-security partitioning (tenant_id) — never a resource-tree dimension. A package is fully self-contained: there is no tenants/<id>/ overlay tree, no shared-library modules/ folder, and no inheritance between packages. What a tenant "sees" is simply every process in the package deployed under its labels.

Referencing resources — convention over configuration

Because the package is self-contained, resources are referenced by their local key, derived from the folder tree — never by tenant/module/version:

Codecs are the one exception. A globally-named codec — a format the engine links in (json, xml, yaml, csv, raw-text, raw-bytes) or a codec crate a distribution force-links — is urn:sutra:codec:<name>; a package-defined codec is named by its path under schemas/, / folded to : — schemas/transfer/ becomes urn:transfer, schemas/hr/employee/ becomes urn:hr:employee. channels.yaml binds that URN:

# examples/money-transfer/.../channels.yaml
channels:
  - name: transfer-request
    transport: http
    bind: "POST /channels/transfer-request"
    codec: urn:transfer     # schemas/transfer/*.xsd, declared by schemas/transfer/codec-manifest.yaml

A codec-manifest.yaml sits inside its own schemas/<codec>/ folder and declares the schema kind plus the formats the codec accepts:

# examples/money-transfer/.../schemas/transfer/codec-manifest.yaml
schemaKind: xsd
formats: [xml, json, yaml]

The reserved token sutra may not be used as a first-level subfolder name under any artifact folder (schemas/sutra/, bpmn/sutra/, …) — that would collide with the engine's own urn:sutra:* namespace. A deeper sutra (schemas/hr/sutra/) is fine.

Schema bundles — when a codec is a whole profile

schemaKind: xsd and schemaKind: json-schema are the generic case: a folder of schema files the engine validates a decoded document against. Some standards aren't that shape at all — they're a whole profile: an envelope grammar, a mapping from a wire-level message name to a schema file, and versioned editions the profile revs on its own release cadence, independent of Sutra releases. For those, schemaKind names a bundle kind a codec crate has registered (see Domain neutrality and the SPI model), and the folder's job shifts from "here are the schemas" to "here is the configuration that decides which schema backs which message, for this archive version."

A codec crate registers its own bundle kind this way whenever the standard it serves is a whole profile rather than a bare schema folder — say, a market venue that publishes a wrapper envelope around every message and revs its schema editions on its own release calendar. That codec crate is a proprietary extension, not part of this distribution — the mechanism below is generic; naming a concrete kind is only for illustration. Its manifest maps wrapper element names — not a bare schema namespace — onto schema files archived alongside it:

# schemas/<your-kind>/codec-manifest.yaml
schemaKind: <your-kind>

# schemas/<your-kind>/<your-kind>-manifest.yaml
appHdr: edition-2024/Header_v1.xsd   # optional
incoming:
  OrderConfirmation: edition-2026/OrderConfirmation_v3.xsd
outgoing:
  OrderConfirmation: edition-2026/OrderConfirmation_v3.xsd
  ShipmentNotice: edition-2024/ShipmentNotice_v2.xsd

schemas/<your-kind>/codec-manifest.yaml      # schemaKind: <your-kind>
schemas/<your-kind>/<your-kind>-manifest.yaml   # appHdr? / incoming{wrapper: relpath} / outgoing{wrapper: relpath}
schemas/<your-kind>/<edition-folder>/*.xsd   # free-form folder names — the manifest is the only truth

A few things fall out of that shape:

• Edition folders sit side by side. Because the venue revs its schemas per release (the same wrapper can move from one schema version to the next between editions), archiving each release's files under its own folder and repointing the manifest is how a module adopts a new edition — a new version of the archive, no engine or codec change.
• An unlisted wrapper falls back to the codec crate's own base schema for its pinned default version, so a bundle only needs to carry the wrappers it actually wants to validate more strictly than that default — the codec stays useful with zero configuration otherwise.
• An enriched edition can be licensed material. A codec's base schemas may be freely redistributable while a fuller, usage-guideline edition is a licensed product participants obtain themselves and supply in their own archive — the engine never ships that tier, and neither does any artifact built from this repository.
• Registration is deployment-scoped, exactly like every other artifact under the registration model: the bundle's registry key is urn:sutra:codec:<folder-path-with-'/'-folded-to-':'>:<deploymentId>. Naming the folder <your-kind> shadows the globally-registered urn:sutra:codec:<your-kind> for that deployment only — a second version of the same module with a different edition mapping registers under its own deploymentId and runs side by side, no collision.
• Deploy-time errors are fail-closed: an unknown wrapper name, a wrapper listed under the wrong direction, a missing or uncompilable schema file, or a schema whose namespace doesn't match the expected one all reject the archive rather than deploy something that silently validates less than the manifest claims.

Store migrations, and evolving a projected structure

Every sql data store the package declares brings its own schema, under a folder named for the store:

migrations/<store-id>/V001__<description>.sql
migrations/<store-id>/V002__<description>.sql

<store-id> is the store's name in datastores.yaml, and the migrations: key of that store points at the folder (migrations: migrations/accounts). The scripts are yours: your dialect, your table names, your indexes and seed rows. The engine generates no DDL for a module store.

One store is the exception: the reserved coverage store, where path-coverage marks are persisted. Its declaration still picks the database, but the engine owns its schema — it ships that DDL per dialect and applies it to the connection on the same first-use path — so the store block carries no migrations: key and a package carries no migrations/coverage/ folder. See Coverage: declared routes as the compliance signal.

Three properties of how the package's own scripts run are worth designing around:

• Applied in V<n>__ order, once per store instance, before the store serves its first operation — and serialized across replicas, so two engines booting at once don't race.
• There is no migration ledger. The engine's own Flyway-style history table covers engine tables only; a module store's scripts are simply re-run on the next boot. Write them idempotently — CREATE TABLE IF NOT EXISTS, INSERT … ON CONFLICT DO NOTHING, CREATE INDEX IF NOT EXISTS — because that is what makes the re-run a no-op rather than a failure.
• A store's data carries across a version bump. There is no deployment_id on a business store's rows: deploying 1.0.1 beside 1.0.0 doesn't fork the data, and that is the feature, not an oversight.

When a projected structure changes

A store that declares a structure: block adds one obligation: a changed structure ships with the migration that makes the table match, in the same package. sutra lint derives the effective table shape from these very scripts, so it tells you at package time whether the pair is consistent — its job is to detect the mismatch, not to repair it.

What follows from the table is a packaging rule: a type change and the ALTER that supports it belong in the same package. Lint replays every script in the folder in version order and compares the result against the declared type as it stands, so a package carrying both lands clean, while one carrying only the type change fails the gate rather than the deployment.

Building one by hand vs. scaffolding

sutra create app <name> (see Your first app) generates a package in this exact shape, verified through the engine's own loaders before anything is written. Growing it from there:

sutra create bpmn my-process --package packages/my-app --validation fatal
sutra create deployment my-app-eu --from packages/my-app   # explicit variant copy

sutra lint <package-dir> runs the full package-time validation suite (the same checks sutra package runs before sealing) with no output on success — the fast pre-flight to run before every package.

Next

• Channels and transports — how channels.yaml binds a transport, a codec, and an ack mode.
• The q: namespace — the BPMN extension vocabulary that wires a process to its channel.

Channels and transports

Processes don't get called — they're triggered by messages arriving on channels. channels.yaml (one per package, see Deployment packages) declares each channel: which transport it rides, which codec decodes it, how it acknowledges receipt, and who may send to it.

Transports

One neutral transport SPI, seven implementations, all self-registering behind the same lifecycle trait — the engine binds, activates, and drains every one of them through a single generic path with no if transport == "..." branching anywhere:

Two further transport: values are engine-internal rather than vendor clients — they have no wire protocol and no listener of their own. local delivers in-process to another channel; pull parks the delivery as a task a worker fetches instead of dialing anything, which is the external-task surface.

Dapr and Knative Eventing ride the HTTP transport as integration patterns rather than dedicated crates — the engine speaks plain HTTP (+ CloudEvents) to a Dapr sidecar or a Knative broker, so no broker vendor client ever links into the engine for either.

A hardened or air-gapped build selects a subset of transports at compile time via Cargo features (cargo build -p sutra-engine --no-default-features --features file), so the unlinked vendor clients (rdkafka, the AWS/GCP SDKs, lapin, fe2o3-amqp) are not compiled in at all. An operator can additionally restrict which transports a running binary accepts via SUTRA_ALLOWED_TRANSPORTS — a channel declaring a disallowed transport fails the deployment with a clear diagnostic, not a silent no-op.

Binding a channel

# examples/money-transfer/.../channels.yaml (abridged)
channels:
  - name: transfer-request
    transport: http
    bind: "POST /channels/transfer-request"
    codec: urn:transfer
    cloudevents-mode: none
    ack-mode: on-complete
    auth:
      scheme: apikey
      apikey:
        value: transfer-demo-key
        header: X-Api-Key

A channel does not name a process. Processes subscribe to channels: a start event's <q:source channel="transfer-request" messageTypeValue="TransferRequest"/> is what routes a decoded inbound to it (see The q: namespace). This is what lets one channel feed several processes and one process listen on several channels — the money-transfer example runs the same transfer flow off three different intake channels (http, rabbitmq, kafka), each a separate <q:source> on the same underlying <bpmn:transaction>.

Fan-out. By default a channel is point-to-point: for a given (channel, messageType), exactly one subscribing process is allowed (enforced at deploy time). Setting broadcast: true fans a decoded message out to every subscribing process, one instance each — genuine pub/sub.

Concurrency admission. A channel may optionally declare maxConcurrentInstances — an admission cap on simultaneously active instances from that channel (a suspended instance still holds its slot). Absent, the channel is unbounded.

Codecs — format × schema

A codec is a channel-facing named decoder: a parser (json / xml / yaml / csv / a domain wire format) plus an optional schema. The schema is the load-bearing part — it's what yields a messageType and structural validation. A codec with no schema (a media codec, or a channel with no codec at all) still decodes, but only a catch-all <q:source> (no messageTypeValue/messageTypePattern) can subscribe to it, and sutra lint warns.

Codec names share one urn:sutra:codec:<name> namespace, and codec: in channels.yaml looks identical whichever tier the decoder came from. There are three:

A built-in format is a pure parser — it decodes, but carries no schema, so it lands in the catch-all-subscription case above. This distribution bundles six of them — json, xml, yaml, csv, raw-text, raw-bytes — and no domain codec at all. Bind one when the payload genuinely has no contract to check, or when that contract is enforced somewhere else.

A package-supplied codec is the zero-install path to a typed contract, and the one both example apps take. schemas/transfer/ holds the XSDs plus a codec-manifest.yaml declaring schemaKind: xsd and the formats the codec accepts; the engine compiles them at deploy time and names the codec after the folder path (urn:transfer — see Deployment packages for the exact folding rule). Nothing about it is second-class: it yields message types, structural validation, and the shape every FEEL path is checked against at load time, exactly as a codec written in Rust does. The engine cannot tell that the schema arrived in an archive rather than a crate.

An extension-crate codec is what a wire format needs when a folder of schemas cannot express it — a grammar that isn't XML or JSON at all (a fixed-width block structure, or a delimited segment stream), or a whole profile: an envelope grammar, a mapping from wire-level message names to schemas, and versioned editions revved on the standard's own release cadence. It implements PayloadCodec from sutra-codec-spi, inventory::submit!s a BuiltinCodec next to the impl, and claims its name in the same namespace; a distribution that wants it adds the dependency and force-links it from its own composition root. Every message standard is served this way, by proprietary extension crates built outside this repository. An engine binary that links one resolves it exactly like a built-in; see Domain neutrality and the SPI model for why none of them lives here.

What an extension codec can express — an enveloped profile, generically

The clearest illustration of what the codec SPI has to be able to carry is a market venue's own profile of a standard rather than the bare standard itself. A generic-format codec is the easy case: one instance document validated against a schema, with the message type read straight off the document's own root element or namespace. A profile codec decodes a family of messages for a venue that doesn't fit that shape — every message travels inside a venue-specific envelope, wrapping a header plus a body, and the venue pins its own schema versions on its own release cadence. Worse for identification purposes, the same underlying message can sometimes back several different wrappers, so a bare schema namespace can't tell you which one you're looking at.

Such a codec's message type is therefore the wrapper element's local name, not a schema namespace slug — which lets declared_message_types() return the venue's own closed set of wrapper names, rather than the open type set a bare-standard codec declares, so a message-type applicability check (a rules-manifest.yaml entry, a q:source messageTypeValue pin) actually fires for a profile-bound module rather than silently no-op.

Decoding validates the envelope grammar, the header, and the body against the venue-pinned base schemas the codec crate carries, out of the box. The payload view projects the body exactly as a bare-standard document would be — a compatibility guarantee: every alias, DMN input, or fixture written against the underlying message shape reads identically whether the underlying codec is the profile variant or the plain standard. Outbound can be a template-rendered passthrough — encode() returning the rendered envelope bytes verbatim rather than assembling one — so template drift against the venue's schemas has to be caught by validating rendered output in tests, not by a runtime encode path. A second venue following the identical envelope/wrapper/edition pattern reuses the same machinery rather than re-implementing it.

None of that requires a change to the engine, the channel layer, or this repository — which is the point of the codec SPI. A deployment can go one step further still and override a codec's own schemas: see Deployment packages for the schemaKind bundle mechanism that lets an archive map its own schema editions per wrapper.

Acknowledgement modes

ack-mode decides when the engine acknowledges an inbound relative to processing:

• on-persist — ack as soon as the message is durably captured, before the process runs. Broker default. On HTTP this is what makes a channel asynchronous (202 Accepted, no business body).
• on-complete — ack only once the instance reaches a terminal state. HTTP default (classic synchronous request/reply — hold the connection, return the reply body). On a broker, this defers the ack via the engine's DeferredAckRegistry until the instance completes or fails.

The full per-transport wiring matrix, the bounded-registry knobs, and when to pick which mode live in Acknowledgement modes — that's the operating-chapter deep dive this summary points at.

Secrets on a channel

A channel's credentials (broker username/password, an API key) are never literal values in channels.yaml — they're scheme references (env:NAME, secret:…, vault:…, aws-secrets:…), resolved at channel startup through one vendor-neutral resolver SPI. Package-time validation rejects a literal secret outright.

Next

• The q: namespace — how a BPMN process subscribes to a channel.
• External tasks: the pull worker surface — the pull transport in full.
• Acknowledgement modes — the operator-facing deep dive.

The q: namespace

Standard BPMN 2.0 has no opinion on channels, message types, correlation, or replies. Sutra adds a small, layered set of extension elements — the q: namespace, urn:sutra:q:1.0 — that live entirely inside <bpmn:extensionElements>, so a Sutra process is still a valid, portable BPMN 2.0 diagram; the q: attributes just tell the engine how to wire it to the outside world. The authoritative shape is xsd/q.xsd; the engine's parser validates every <q:*> element against it, and the same XSD drives the sutra-modeler-plugin property panels.

<bpmn:definitions xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL"
                  xmlns:q="urn:sutra:q:1.0" ...>

At a glance

q:source — the inbound trigger

Every message-consuming node — a start event or a wait-state catch — declares exactly one q:source. It names the channel, the message type it accepts, and the variable the decoded payload lands in:

<bpmn:startEvent id="Start">
  <bpmn:extensionElements>
    <q:source channel="transfer-request" messageTypeValue="TransferRequest"/>
  </bpmn:extensionElements>
</bpmn:startEvent>

• channel (required) — the channels.yaml channel name.
• messageTypeValue / messageTypePattern — subscribe to one exact type, or a family via regex, matched against the codec's decoded message type. Neither set = accept anything the channel's codec yields.
• name (default payload) — the process-variable name the decoded body is projected under (payload.fromId, payload.body.CdtTrfTxInf... for a structured codec).
• ack (default on-persist) — see Acknowledgement modes.
• dedupKey — an expression identifying a redelivered duplicate (e.g. header.X-Request-Id), distinct from the process-level retry-safety assertion below.
• dataClass (default none) — pii / pci / phi / financial; drives redaction policy.

The codec itself is not declared on q:source — it comes from the channel (YAML is authoritative for transport/codec binding; BPMN is authoritative for process flow). Declaring a codec on q:source is a parse error.

q:validators and q:redactors (nested under q:source)

<q:source channel="transfer-request" messageTypeValue="TransferRequest">
  <q:validators>
    <q:complexValidator source="transfer-limits.dmn"/>
    <q:complexValidator source="transfer-fields.srl"/>
    <q:simpleValidator ref="iso-4217-currency" path="payload.amount.currency"/>
  </q:validators>
  <q:redactors>
    <q:redactor ref="pci"/>
  </q:redactors>
</q:source>

q:validators is a mixed, ordered container: a q:complexValidator runs a whole-payload ruleset (a .dmn or .srl file, or a built-in like iso-xsd) — a chain can mix .dmn and .srl entries freely, run in declaration order, with every entry's issues accumulating into one result (see Composing a validator chain); a q:simpleValidator checks one field at a FEEL path against a registered content validator (iso-3166-country, iso-4217-currency, iso-9362-bic). q:redactors names registered ContentRedactors that mask sensitive spans in every observability surface (audit, logs, traces) without touching the value the flow actually sees.

q:reply and q:send — outbound

<bpmn:serviceTask id="OkReply" implementation="transfer-result.hbs">
  <bpmn:extensionElements>
    <q:reply mode="native" contentType="application/xml"/>
  </bpmn:extensionElements>
</bpmn:serviceTask>

q:reply answers the caller that started this instance — mode="native" (default) preserves the symmetric reply behavior, or emit a CloudEvent (cloudevent-binary / cloudevent-structured / match-inbound). q:send is the unsolicited counterpart — an intermediate throw event emitting to its own destination (@destination or @channel), with no inbound caller to answer. Both accept <q:header name="…" value="…"/> children carrying FEEL-derived values onto transport headers/application-properties.

q:reply's continue="true" is respond-and-continue: flush the reply the moment the task completes, then park the instance and self-resume the remaining nodes asynchronously — the caller gets its answer without waiting on the tail of the flow.

q:alias — correlation by your business key

<q:alias name="e2eId" expression="payload.E2EId" unique="true" onConflict="correlate"/>

A friendly key derived from a FEEL expression over the process variables — durable, and re-evaluated on rehydration. unique="true" with onConflict="correlate" is what lets a later message on a different channel find and resume the exact parked instance it belongs to, by a key you named (an EndToEndId, an order reference) rather than an engine-internal id. See Wait states and human tasks.

q:retry — per-task retry policies

<bpmn:serviceTask id="Score" implementation="registered:score">
  <bpmn:extensionElements>
    <q:retry maxAttempts="3" initialDelay="PT1S" backoffCoefficient="2.0"
             maxDelay="PT5M" nonRetryableCodes="SUTRA.TASK.VALIDATION"/>
  </bpmn:extensionElements>
</bpmn:serviceTask>

Valid on both kinds of <bpmn:serviceTask> — a registered task and a channel-call task (implementation="channel:<name>"). maxAttempts is the total invocation budget including the first attempt; attempt n+1 waits min(initialDelay × backoffCoefficient^(n-1), maxDelay); nonRetryableCodes names structured codes that fail immediately regardless of budget.

The load-bearing property: a retry wait is a durable timer park, never a sleep. The instance persists with the failed task still pending and an armed timer at the backoff instant, so the backoff survives restarts and hot-deploys and blocks no execution lane.

What counts as a failed attempt is deliberately narrow. On a registered task: an uncaught error from the task function. On a channel-call task, exactly two things — the route-less <q:timeout> boundary firing, and the request delivery being terminally poisoned by the outbox attempt ceiling. A correlated business response is never a retry trigger (the counterpart answered; re-sending would double-submit), and BPMN errors always route to their boundaries instead. Because a modelled outcome always beats a policy, a channel-call <q:retry> requires the route-less <q:timeout> form — a timer boundary with drawn outgoing flows alongside a retry policy is a load error.

Full treatment, including the re-drive's fresh-idempotency-key contract and what exhaustion does: Retries, history, and schedules.

q:timeout — a deadline on a channel call

<bpmn:serviceTask id="Score" implementation="channel:score-request">
  <bpmn:extensionElements>
    <q:timeout duration="PT2M"/>
  </bpmn:extensionElements>
</bpmn:serviceTask>

Synthesizes a timer boundary on the call, so a counterpart that never answers cannot park the instance forever. Without a <q:retry> alongside it, a fired route-less timeout is a catchable BPMN error; with one, it is a retryable task failure first.

q:store — durable, cross-instance data

<bpmn:dataStoreReference id="dsrFrom" name="accounts" dataStoreRef="accountsStore">
  <bpmn:extensionElements>
    <q:store key="payload.fromId" forUpdate="true"/>
  </bpmn:extensionElements>
</bpmn:dataStoreReference>

Binds a <bpmn:dataStoreReference> to a key in a datastores.yaml-declared store. forUpdate takes a pessimistic row lock (serializes concurrent writers on the same key); field replaces one field of a stored map value; expect="unchanged" is an optimistic compare-and-set instead of a lock. See Data stores.

q:dispatch / q:case — a routing table instead of gateway sprawl

<q:dispatch default="fallback" onNoMatch="error">
  <q:case when="payload.type = 'A'" calledElement="handle-a"/>
  <q:case when="payload.type = 'B'" calledElement="handle-b"/>
</q:dispatch>

A declarative dynamic call-activity dispatch — one FEEL condition per case, instead of an exclusive gateway fanning into N call activities.

q:onValidation — the structural-failure policy

<q:onValidation mode="route" errorCode="T505"/>

mode="route" surfaces the validation summary (outcome, tier, firstReasonCode, firstIssue, issues) as variables so the BPMN's own gateway decides what to do; reject / error short-circuit. The engine never interprets what a soft error means — that decision stays in your process, which is what keeps the engine domain-neutral.

q:process — the retry-safety assertion

<bpmn:extensionElements><q:process idempotent="true"/></bpmn:extensionElements>

Hung off the <bpmn:process> itself. idempotent="true" is your assertion that re-running this process on the same input converges to one end state — so a redelivered message is safe to re-process any number of times. The default is false (fail-closed): an execution failure on a non-idempotent process is consumed (ack, no requeue) and recorded as an incident rather than blind-retried. This is a different concern from q:source's dedupKey, which only detects that a message is a redelivery — it says nothing about whether re-processing it is safe.

q:variables, q:audit, q:coverage

• q:variables — declares process variables up front (name + scalar type or a schema reference), so deploy-time static validation can check every FEEL path against them, and mark a variable transient (never persisted — reading it after a wait state is a validation error), sensitive (persisted but redacted downstream), or source-bound (payload-initialized from a named channel).
• q:audit — per-process audit sink/target/capture-level configuration.
• q:coverage — declares one tracked compliance route (path id + the ordered flows it must traverse) for the path-coverage reporting the CLI's sutra coverage commands drive. This is the intra-process half of the mechanism; a route spanning several correlated processes is declared differently. See Coverage: declared routes as the compliance signal for both shapes, the curation guidance, and the CLI walkthrough.

Next

• Rules: DMN, FEEL, and .srl — the decision layer these expressions run on.
• Retries, history, and schedules — q:retry and q:timeout end to end, plus timer definitions and execution history.
• Worked example: money-transfer — most of the elements on this page, in one real process.

Rules: DMN, FEEL, and .srl

A businessRuleTask binds one decision, authored either as a .dmn decision table or as a .srl ruleset. Both compile onto the same FEEL evaluator (sutra-feel) — no JVM, no Rete runtime, and no second expression language to learn.

FEEL — the shared expression language

FEEL (Friendly Enough Expression Language) is what every condition, gateway expression, q:alias key, q:store key, and rule body is written in. Sutra's implementation is a from-scratch Rust evaluator (lexer, parser, AST, a determinism denylist that forbids non-reproducible constructs on a replay path, and DECIMAL64 numeric semantics — 16 significant digits, HALF_EVEN rounding).

Try an expression standalone, before it's anywhere near a process:

sutra explain 'payload.amount > 100'
sutra explain --context vars.txt 'fromAccount.balance - payload.amount'

sutra explain is a one-shot evaluator or a REPL (omit the expression) — useful for debugging a gateway condition or a q:alias expression without deploying anything. See Crates: sutra-feel and sutra-dmn if you want FEEL evaluation embedded in your own Rust project, independent of the engine.

DMN decision tables

A .dmn file under rules/ is a standard DMN 1.5 decision table; the engine's evaluator supports all seven hit policies. A businessRuleTask names the decision, and its named output columns merge back into the process's instance variables:

<!-- examples/approval-hold/.../rules/approval-decide.dmn -->
<decision id="approvalDecide" name="Review decision from risk score">
  <decisionTable hitPolicy="FIRST">
    <input id="i_risk"><inputExpression typeRef="number"><text>riskScore</text></inputExpression></input>
    <output id="o_decision" name="decision" typeRef="string"/>
    <rule id="r_approve">
      <inputEntry><text>&lt; 50</text></inputEntry>
      <outputEntry><text>"approve"</text></outputEntry>
    </rule>
    <rule id="r_review">
      <inputEntry><text>-</text></inputEntry>
      <outputEntry><text>"review"</text></outputEntry>
    </rule>
  </decisionTable>
</decision>

Sutra's DMN conformance is measured against the real OMG DMN-TCK, not asserted — see DMN-TCK conformance for the numbers and what they cover.

.srl — a Drools-inspired rule DSL over FEEL

.srl is Sutra's own rule language: rule / when / then / end framing around FEEL conditions and a small, closed set of side-effecting actions. It targets the same use case DMN's COLLECT hit policy does — several independent business-validation rules over one payload — in a syntax closer to a Drools ruleset than a decision table.

rule "currency-not-usd"
when
  exists(payload.amount.currency) and payload.amount.currency != "USD"
then
  report(
    "SUTRA.VALIDATE.CURRENCY_NOT_USD",
    "payload.amount.currency",
    "Currency must be USD; got " + payload.amount.currency
  );
end

Grammar, in full:

ruleset := rule*
rule    := "rule" STRING attr* "when" <condition> "then" action* "end"
attr    := "salience" INTEGER | "activation-group" STRING
action  := verb ";"
verb    := "report" "(" <feel_expr> "," <feel_expr> "," <feel_expr> ")"
         | "set"    "(" IDENT "," <feel_expr> ")"

• Every condition and action argument is an embedded FEEL expression — .srl adds only the rule framing on top.
• Two action verbs today: set(target, expr) binds a value into the working context (visible to later rules) and the output map; report(code, path, message) appends a structured issue. insert / retract are reserved for a future stateful phase and are a clean parse error, not silently accepted.
• Evaluation is a single deterministic forward pass — a stable-sorted agenda by (-salience, declaration order), each rule firing at most once, activation-group giving first-match-wins semantics within a group. This is sequential-agenda, not a Rete network.
• Fail-closed: a parse error, or a condition/action that errors at evaluation, is a hard error — never a silently-skipped rule.
• A FEEL if / then / else used inside a when condition must be parenthesised (when (if a then b else c) …), because the bare keyword then that ends the condition is matched at paren depth 0.

.srl and .dmn both live under a package's rules/ folder (see Deployment packages) and are routed by file extension — no separate configuration names one or the other.

Composing a validator chain

A <q:source>'s <q:validators> list (see The q: namespace) can mix <q:complexValidator> entries pointing at both .dmn and .srl files in one chain — nothing requires a chain to stick to one rule engine, and nothing requires a business ruleset to live in a single file.

• Declaration order is evaluation order. The chain runs top to bottom exactly as written in the BPMN.
• Every validator sees the same payload projection. Whatever the channel's codec decoded is what each entry in the chain evaluates against — an earlier validator's issues don't change what a later one reads, only what accumulates alongside it.
• Issues accumulate, they never replace. Each entry's reported issues append to the same validation.* result; validation.outcome and validation.tier are derived from the whole accumulated list, so a gateway keyed on them is indifferent to which file in the chain produced which code.
• validation.firstReasonCode follows declaration order. When two entries both report an issue, firstReasonCode is whichever fired first in chain order — this only affects which code a reply template surfaces, never whether a gateway routes to reject.

This is what lets you split one business ruleset across engines by what each rule needs, instead of by an arbitrary file boundary. A module with a mix of clock-dependent and stateless checks (an extension-crate workload, not one of this repository's bundled examples) does exactly that:

<q:validators>
  <q:complexValidator source="intake-timing.dmn"/>
  <q:complexValidator source="intake-fields.srl"/>
</q:validators>

The DMN table keeps the rules that need a clock — a staleness window and a clock-skew/future-dated window on some received-at timestamp — because the engine-injected evaluation clock is only available as a DMN validator's reserved now input. The .srl file carries the stateless field checks (a required-currency check, a positive-amount check, a required-identifier check) that have no temporal dependency at all and read more naturally as named rules than as columns on the decision table. Both files reason over the identical payload projection the codec decoded, report through the same SUTRA.VALIDATE.* code family, and their issues land in one accumulated validation.issues list — the split between engines is invisible to the BPMN gateway that routes on the outcome.

Which to reach for

• DMN if the logic is naturally tabular (a rate card, an approval matrix, one output per row) or you want a diagram a business analyst can review directly.
• .srl if the logic is a set of independent validation rules over a payload, where a Drools-style when/then reads more naturally than a table.
• Both, in one chain, when a ruleset is naturally mixed — some rules need something only a DMN validator gets (the injected evaluation clock), others are better named individually than columned into a table. See Composing a validator chain above for exactly how a .dmn and a .srl entry combine, illustrated above with a clock-dependent-vs- stateless split.

Next

• Data stores — durable state a rule or task reads/writes.
• Worked example: money-transfer — FEEL data-assignment nodes in a real transaction flow.

Data stores

A data object (<bpmn:dataObject>) is a per-instance variable — private, transient, born and dying with one instance. A data store (<bpmn:dataStore> / <bpmn:dataStoreReference>) is the opposite axis: durable, keyed state that outlives and is shared across instances — an account ledger, a customer index, the covered/uncovered flags behind a compliance route.

Status note. Data stores are exercised end to end by the money-transfer example — a real sql-backed ledger under transactional, pessimistic-lock control. (Path coverage rides a data store of an unusual kind: its marks are persisted in a store the package declares — the reserved coverage name, whose connection picks the database they land in — but the engine owns that store's schema and applies it itself, so it is neither a key→value business store nor one you write DDL for. See Coverage: declared routes as the compliance signal.) Some of the wider surface described in the design record — a file-backed store, an optimistic-concurrency compare-and-set on every provider — is still in motion; treat anything below not shown against the money-transfer example as directionally accurate rather than a finished contract, and check rust/crates/sutra-datastore for the current provider set before depending on specifics.

Choosing a store shape

A business store has two shapes. The choice between them is narrower than the volume of mechanism the rest of this chapter spends on the second one suggests, so it is worth settling first. (The reserved coverage store is neither — the engine owns its schema and applies it, so a package writes no coverage DDL and makes no shape choice at all; see Coverage.)

Key/value is the general case, and it is always sufficient. By default a store is a key→value store: one row per key, the value serialized whole as JSON text. Any structure fits it — nested, repeated, open-ended, a different shape next release — because nothing about the value's shape is written down anywhere for it to disagree with. The table behind it is one fixed generic shape (store name, key, value, rev, updated_at) shared by every key/value store on that connection, and it does not change when your data does. From a standards point of view this is the only shape that carries every scenario, which is why it is the default and stays the default. A store with no structure: block is this store, and none of the projection machinery below applies to it.

Projection is an integration affordance, not an upgrade. Declaring a structure: does not make a store more correct, more durable or more capable, and it is not a tier you graduate to. It exists for one reason: so that everything else that already speaks SQL can read the data. A BI tool, a dashboard, a nightly reporting query, an analyst with read access to that database — none of them know what Sutra is, and against a key/value store the most any of them can see is an opaque document to parse. Against a projected table they see balance and opened_at as columns, with types. That is one more way to ease integration, not a better kind of store.

The trade, stated plainly. Projection costs two things and buys one:

• It costs the flat-only constraint. The declared type must be entirely scalar, and a type that later grows a nested or repeated child stops projecting at all — see Flat only.
• It costs ownership of the DDL and its evolution. You write the CREATE TABLE, and every change to the declared type is a migration you author and sequence. A key/value store needs a table too, but it is the same generic one every time — written once, never revisited. A projected table is your record's own shape, so it has to move whenever that shape moves.
• It buys direct SQL access to the data, for anything that speaks SQL, without going through the engine.

If nothing outside the process ever reads the store, that affordance buys nothing, and key/value is the right answer.

The shape is a property of the store, not of the flow — within limits worth knowing. A store's shape is declared in datastores.yaml and nowhere else. The BPMN is unchanged either way: the same <bpmn:dataStoreReference>, the same <q:store> with the same key / field / forUpdate / expect attributes, resolved by store name through the same registry. Pessimistic locking, <bpmn:transaction> enlistment and rollback, delete, and the rev bookkeeping behind expect="unchanged" — seeded at 1, bumped on every write, a conflict when the revision moved — carry the same semantics under both shapes. Adding or removing a structure: block is a package-level change, not a process-level one.

What does change is what the store accepts and what it hands back — and that is not nothing:

• A projected store refuses a write carrying a field the structure does not declare (UNDECLARED_FIELD); a key/value store simply stores it. This is the difference most likely to bite, because a process that has quietly been writing a wider record than the declared type works against key/value and fails against projection — and nothing catches it beforehand: neither sutra lint nor the executor checks a <q:store field="…"> name against the declared structure, so the first evidence is the refused write.
• It refuses a value that isn't a record at all — a scalar, an array, an explicit null (VALUE_NOT_A_RECORD). A key/value store holds any of those happily.
• It refuses every operation until the live table satisfies the declaration (PROJECTION_UNSATISFIABLE, checked on first use). A key/value store has no declaration to drift from, so it has no such failure mode.
• A field-narrowed write is a read-modify-write of the whole value under both shapes, so one that creates a key writes a record containing only that field. On a key/value store that is simply a one-field document; on a projected store every other declared column is bound NULL, so unless they are all nullable or defaulted in your DDL that insert runs straight into your own constraints. Make the first write of a key a full record rather than a narrowed one.
• Values that pass through a typed column come back in the column's canonical form rather than the text that was written: an explicit null and an absent optional field become the same thing on read, and a dateTime comes back as your dialect's rendering of that instant. The exact scope of what is and isn't preserved is in A worked example.

None of those change how a process addresses the store; several of them change whether a given write succeeds. They are the checklist for a shape change, not a reason to avoid one.

Column names become a published interface. The moment a dashboard reads that table, its column names are a contract with someone who has never seen your schema — a report knows account_balance, not the XSD element it was derived from. Column names are derived by convention from the declared field names, so renaming a field renames a column, and silently breaks that report. That is what the columns: override exists for: pin the physical name, and the schema can be renamed underneath it while the published surface holds still. It is worth setting before you need it rather than after someone's query starts returning nothing — the naming rules and the override syntax are under Column names.

Declaring a store

One flat datastores.yaml per package (see Deployment packages). The module owns its store — the connection is declared here, resolved from environment references, and its migrations ship inside the package:

# examples/money-transfer/.../datastores.yaml (abridged)
datastores:
  - name: accounts
    type: sql
    sql:
      url-ref: env:ACCOUNTS_DB_URL
      username-ref: env:ACCOUNTS_DB_USER
      password-ref: env:ACCOUNTS_DB_PASSWORD
      migrations: migrations/accounts     # idempotent SQL, run once on first use
    dataClass: financial                  # sensitivity tag — redacted in audit/logs/traces

The engine's own datasource (instances, outbox, lease, audit, inbox) is never a store's backing connection — a sql store always resolves its own connection, and fails closed if it declares none. This is what makes a generic engine image self-sufficient against any number of modules that each bring their own store: no baked default datasource, no baked migrations.

A store declared this way is the key/value shape: one row per key, the value serialized whole as JSON text, opaque to the database. Adding a structure: block projects a flat record onto real typed columns instead — see Typed columns: declaring the structure a store holds below.

Referencing a store from BPMN

<bpmn:dataStore id="accountsStore" name="accounts"/>   <!-- definitions scope -->

<bpmn:dataStoreReference id="dsrFrom" name="accounts" dataStoreRef="accountsStore">
  <bpmn:extensionElements>
    <q:store key="payload.fromId" forUpdate="true"/>
  </bpmn:extensionElements>
</bpmn:dataStoreReference>

<q:store> (see The q: namespace) is what turns a bare reference into a keyed access: key is a FEEL expression producing the row key, forUpdate="true" takes a pessimistic row lock so concurrent writers on the same key serialize, field narrows a write to one field of a stored map value, and expect="unchanged" is an optimistic compare-and-set alternative to a lock. A data task — a serviceTask with no implementation — reads and writes through its data associations exactly like it would a data object, except the source/target is a store reference instead of a variable.

Typed columns: declaring the structure a store holds

An opaque store hands the database a string. Nothing inside the value is queryable, indexable or typed, and the only way to read a field is to fetch the whole document by key and pick it apart in the process. For a record whose fields are all scalars that is a blob tax on data that could perfectly well be ordinary columns.

So a store may declare the structure it stores, not just its key:

datastores:
  - name: accounts
    type: sql
    structure:
      schema: urn:accounts        # a schemas/<folder> codec of this package
      type: AccountRecord         # a complexType, a root element, or a JSON Schema definition
    sql:
      url-ref: env:ACCOUNTS_DB_URL
      migrations: migrations/accounts

schema + type resolve through the schemas the package already compiles for its codecs (the path-derived URN rule is in Deployment packages), so there is no second schema source to keep in step and no shape inferred from observed data — the declaration is the contract.

A store with no structure: block behaves exactly as it always did. Same opaque key→JSON row, same providers, same <q:store> semantics, and none of the diagnostics below ever fire for it. Declaring a structure is opt-in per store; a record too nested to project simply doesn't declare one.

Flat only

The declared type must be entirely scalar. Every child is classified in declared order:

The table is the declared fields. There is no residue column for the parts that didn't fit, no column-versus-JSON merge on read, and no partial projection. That is the whole simplification, and it is what makes a projected row readable by anything that speaks SQL without knowing Sutra exists.

A type that can't be expressed that way is rejected at package time with SUTRA.CONFIG.DATASTORE.STRUCTURE_NOT_FLAT, naming the offending child:

data store 'accounts': declared structure type 'AccountRecord' is not flat: 'owner' is not a
scalar leaf (nested content). Flatten the type, or remove the 'structure' block and keep the
opaque store.

Those two remedies are the whole menu, deliberately:

• Flatten the type — pull the nested part up into scalar siblings (owner/name → ownerName), or move it to a store of its own keyed by the same business id.
• Drop the structure: block — the store goes back to being the opaque key→JSON store it was, which is a perfectly good answer for a record that genuinely isn't flat.

There is nothing in between, because anything in between would be a silent partial projection — some fields queryable, some hidden in a blob, and a merge rule nobody can see from the declaration.

Because the structure is closed and there is nowhere for an unexpected field to go, a write carrying a field the structure does not declare is a fail-closed runtime error (SUTRA.RUNTIME.DATASTORE.UNDECLARED_FIELD) naming the field. Silently dropping it is the one outcome this design refuses. Writing anything that isn't a record at all — a scalar, an array, an explicit null for the whole value — is refused the same way, as SUTRA.RUNTIME.DATASTORE.VALUE_NOT_A_RECORD: a projected row is its declared fields, so there is nowhere for a bare value to land either.

Control columns

A projected table is never only the declared fields. Three columns belong to the engine, on every projected table, whether or not your own type ever mentions them:

There is no store_name column — a projected table is one store, unlike the opaque table that multiplexes many stores by name. The runtime binds and maintains all three itself, so a declared field may not claim one of their names: that's a naming collision (SUTRA.CONFIG.DATASTORE.COLUMN_NAME_INVALID), resolved the same way as any other one — map the field to a different column under columns:. Your own migration has to create all three alongside your declared columns, or sutra lint raises COLUMN_MISSING naming whichever is absent — a projected table missing store_key, rev or updated_at cannot be served no matter how correct its declared columns are. The worked example below shows all three in the DDL.

Column names

A declared field's column name is derived by convention: lowerCamel → snake_case, ASCII-folded, runs of anything else collapsed to a single _.

The convention is not always enough, and when it isn't, sutra lint says so (SUTRA.CONFIG.DATASTORE.COLUMN_NAME_INVALID) rather than guessing: two fields folding to the same column, a fold that lands on a SQL reserved word, a name over the 63-character identifier cap (PostgreSQL's — the narrowest of the three shipped dialects, so a name that clears it is portable), or a fold that isn't a usable identifier at all. Every one of them is resolved by naming the column yourself:

    structure:
      schema: urn:accounts
      type: AccountRecord
      columns:
        openedAt: opened_on       # the DDL already calls it this
        order:    order_seq       # `order` is reserved

The mapping exists for two reasons. The first is that you write the DDL and may already have column names you can't change. The second is that it pins the physical name against a later schema rename — which matters as soon as anything outside the engine reads the table, because at that point the column names are a published interface and the convention is no longer yours alone to change (see Choosing a store shape). The mapping is checked by the same rules — an override that is itself reserved or over-length is still an error, and an override naming a field the type doesn't declare is an error rather than a silent no-op.

You own the DDL; the engine verifies it

The engine generates no DDL for a projected store and owns no part of its table shape. That table is created by the store's own migrations, in your own dialect, shipped inside the package under migrations/<store>/ (see Deployment packages).

sutra lint derives the effective table shape statically — from the package's own migrations/<store>/V*.sql, replayed in version order (CREATE TABLE plus ALTER TABLE ADD/ALTER/DROP COLUMN), with no database connection and no credentials — and compares it against the projection. The table it compares against is the one named after the store — or, if the migrations create exactly one table, that one — or the one you name explicitly:

    sql:
      url-ref: env:ACCOUNTS_DB_URL
      migrations: migrations/accounts
      table: account_ledger       # when the table isn't named after the store

What it checks the column types against is this mapping. It is advisory — the shape lint expects to find, not DDL anything emits:

Two declared shapes this release refuses outright — not silently degraded, not weakly verified — and one honest limit on verification alone:

• xs:base64Binary is refused, not projected. The advisory mapping above (BYTEA / BLOB / VARBINARY(MAX)) is what the column would need to be; the runtime doesn't marshal binary values through a projected column yet. A store declaring one is refused when the engine resolves it — at deploy time, not at lint time — naming the field:

  data store 'accounts' cannot project structure type 'AccountRecord': field 'attachment' is
  declared 'base64Binary', which a projected column does not carry in this release — declare it
  as a string, or remove the 'structure' block and keep the opaque store.
  

  Lint doesn't catch this on its own: its DDL parser treats BYTEA/BLOB/VARBINARY as an ordinary column type and matches it happily, so a package can lint clean and still refuse to deploy. Don't read lint's silence as proof a binary field works.

• A JSON-Schema-declared structure is refused, not weakly verified. Only a package schemas/<folder> XSD codec carries the enumerable, facet-bearing field list a typed column is checked against — a JSON Schema definition has no such list at this phase. sutra lint reports that honestly as a DDL_UNVERIFIABLE warning (unprovable, not necessarily wrong) — but the engine that actually resolves the store at deploy time draws a harder line and refuses to load the deployment:

  data store 'accounts' declares structure schema 'urn:accounts', which this package provides no
  XSD codec for. A projected store's type must be declared by a schemas/<folder> XSD codec of the
  same package — only XSD carries the declared facets a typed column is checked against.
  

  So a lint-clean package with a JSON-Schema structure: still will not deploy. Declare the structure against an XSD if you want the store to exist at all, not just to have its column types checked.

• DDL outside the parsed subset degrades to a warning, never to a false error. A migration that creates the table in a PL/pgSQL block, a T-SQL procedural guard, a table created outside the package entirely — lint reports that it could not derive the shape and raises no column diagnostic for that store. A linter that cries wolf on legitimate DDL is worse than no linter, because authors learn to ignore it. See Troubleshooting for how to read that warning.

Lint proves the package-time case. The deployed table is checked separately, but not on every operation: first use of a projected store rides the same once-per-store-instance gate that runs its migrations (the one already serialized across replicas by an advisory lock), and there the provider reads the live table's actual columns and fails the store closed if the projection isn't satisfiable, naming every offending column at once. That is the defence against a table that drifted from the package's own migrations — a hand-applied ALTER — and it fails loudly because a silent partial write is far worse than a refusal. It costs one catalog round-trip the first time the store is used, and nothing on any operation after that.

A worked example

A flat record, declared in the package's own codec schema (schemas/accounts/accounts.xsd, the codec urn:accounts):

<xs:element name="AccountRecord">
  <xs:complexType>
    <xs:sequence>
      <xs:element name="accountId" type="AccountId"/>
      <xs:element name="balance"   type="Money"/>
      <xs:element name="openedAt"  type="xs:date"/>
      <xs:element name="active"    type="xs:boolean"/>
      <xs:element name="note"      type="Note" minOccurs="0"/>
    </xs:sequence>
  </xs:complexType>
</xs:element>

<xs:simpleType name="AccountId">
  <xs:restriction base="xs:string"><xs:maxLength value="35"/></xs:restriction>
</xs:simpleType>
<xs:simpleType name="Money">
  <xs:restriction base="xs:decimal">
    <xs:totalDigits value="18"/><xs:fractionDigits value="2"/>
  </xs:restriction>
</xs:simpleType>
<xs:simpleType name="Note">
  <xs:restriction base="xs:string"><xs:maxLength value="140"/></xs:restriction>
</xs:simpleType>

Its table, written by hand, in the dialect the store actually runs on (migrations/accounts/V001__accounts.sql) — store_key, rev and updated_at are the engine's control columns, alongside one column per declared field:

CREATE TABLE IF NOT EXISTS accounts (
  store_key   VARCHAR(512)  NOT NULL,
  account_id  VARCHAR(35)   NOT NULL,
  balance     NUMERIC(18,2) NOT NULL DEFAULT 0,
  opened_at   DATE          NOT NULL,
  active      BOOLEAN       NOT NULL DEFAULT TRUE,
  note        VARCHAR(140),              -- the optional field's column admits NULL
  rev         BIGINT        NOT NULL DEFAULT 1,
  updated_at  TIMESTAMPTZ   NOT NULL DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY (store_key)                -- the store key, never a declared field
);

CREATE INDEX IF NOT EXISTS accounts_opened_at ON accounts (opened_at);

And the declaration that ties them together — no columns: block, because every field folds to the name the DDL already uses:

datastores:
  - name: accounts
    type: sql
    structure:
      schema: urn:accounts
      type: AccountRecord
    sql:
      url-ref: env:ACCOUNTS_DB_URL
      migrations: migrations/accounts
    dataClass: financial

sutra lint raises nothing on this package: every declared field has a column, every column's type holds the declared facet range (VARCHAR(35) for maxLength="35", NUMERIC(18,2) for 18 total / 2 fractional digits), the one optional field maps to a nullable column, the three control columns are all present, and the table's primary key is store_key itself — never a declared field — which is what lint requires of every projected table (KEY_MISMATCH otherwise: a business key would let two store keys collide on one row).

What lands in the database is a table anything can read:

Round-tripping is lexical, not merely typed: every value travels as text in both directions — bound with the dialect's write-side cast, read back through its canonical text rendering — so a put followed by a get returns the same shape it wrote, including a decimal's written scale, decided by your own NUMERIC(18,2) rather than by anything in between. That holds exactly for xs:string, xs:decimal/the integer family, xs:boolean, xs:date, and an absent optional field — and an explicit JSON null on an optional field stores and reads back exactly like an absent one: both are NULL in the column, and both come back with the key omitted, never an explicit null.

xs:dateTime and xs:time are the one exception to "lexical": the column stores an instant, not the string you sent, so a get returns the dialect's own rendering of that instant, normalised toward ISO-8601 (a T separator, an explicit +HH:MM offset) — not necessarily the offset or the sub-second digits you wrote. Write 2026-08-04T10:00:00.000-05:00 into a dateTime column and expect back whatever your dialect renders that same instant as (typically UTC, for a timezone-aware column type), not the literal string. forUpdate, expect="unchanged" and the revision bookkeeping behave exactly as they do for an opaque store.

If AccountRecord later grows an owner sub-record, the package stops linting with STRUCTURE_NOT_FLAT — an explicit, package-time decision (flatten it, or give up projection for this store) rather than a silent shape change. The evolution rules are in Deployment packages.

Isolation and atomicity

Two separate guarantees, worth keeping distinct:

• Isolation (no lost update between concurrent writers) comes from serializing access to a key — either a pessimistic forUpdate="true" lock, or funneling all writers for that channel through a single active consumer (see the singleton channel property in Channels and transports).
• Atomicity (all writes commit or none do) comes from a <bpmn:transaction> scope: writes inside it commit together on a normal end, or roll back together on a cancel end / error. An external store write cannot auto-rollback on its own — the transaction sub-process is what gives you the boundary.

The worked example combines both: a singleton channel serializes writers, and the debit/credit pair runs inside a <bpmn:transaction> so a rejected transfer touches no row.

Sensitivity

A store's dataClass (pii / pci / phi / financial) marks its contents sensitive: values read from it are redacted wherever the engine emits observability data — audit events, structured logs, traces — while the flow itself still sees the real value. The tag travels with the variable a data association reads into, so it stays redacted downstream too.

Next

• Wait states and human tasks — the other durable, cross-request mechanism: a suspended instance rather than a shared store.
• Worked example: money-transfer — a data store under full ACID control.
• Deployment packages — where a store's migrations live, and what a change to a projected structure costs.
• Troubleshooting — every projection diagnostic, what causes it, and how to fix it.

Wait states and human tasks

Sutra's wait state is an integration point that waits for an external relay decision — not a human-task management system. When a token reaches one, the engine suspends the instance (persists it, frees the thread); the flow continues only when an external actor — a human via a console you build, or another system — relays a decision back through a channel as a typed message. The engine rehydrates and runs to completion.

That split is deliberate: full human-task management and forms are a much larger scope than a workflow engine's core job, and Sutra doesn't try to own them.

The construct: an intermediate message catch event

The honest BPMN construct is the intermediate message catch event — "pause until message M arrives, then continue." A userTask is the human-facing flavor of the exact same machinery, for authors who think in terms of "a human decides here" — it carries no assignee/group/queue model in the engine; if present, those are just opaque hints in instance variables.

An intake node for a wait declares a channel exactly like a start event's q:source — the relayed message goes through the same two-tier validation pipeline (structural codec, then business validators) that every inbound message does. The difference is only in what happens after:

The load-bearing property: the wait is the safe state. A hard-invalid relay can never advance or corrupt a parked instance — it is rejected and the instance sits exactly where it was. The relayer fixes the message and retries.

Correlation by a business key you name

<q:alias name="e2eId" expression="payload.E2EId" unique="true" onConflict="correlate"/>

The relayed message doesn't carry an engine-internal instance id — it carries a business value (an EndToEndId, an order reference, whatever your domain already uses), and the process declares a q:alias deriving that key from its own payload. The engine indexes it durably, on PostgreSQL, so correlation survives a restart and works identically across every replica. See The q: namespace for the full q:alias shape.

Variables survive the wait typed

The instance variables held across a wait keep their real types. A number comes back a number, a boolean a boolean, a list a list, a context a context, a date/time/duration its own temporal type — and a null comes back null, not an empty string.

This matters because FEEL, correctly, does not coerce. A gateway condition after a wait therefore evaluates against the value's real type:

Two things are deliberately unchanged. First, the inspect projection is not affected: every variable still renders through the display form it always had on GET /admin/instances/{id}, so what an operator's response looks like did not move — typing changed what survives a wait, not what is shown. (A null now renders as null rather than blank, which is the one visible difference, and it is the correct one.) Second, the correlation key input is not affected: a subject blind-index value keeps hashing the exact string it always hashed.

Two value kinds still persist as their canonical string, because neither is really instance state: a FEEL function closes over an evaluation context that ceases to exist the moment the instance parks, and a range is a comparison shape rather than a value. Both behaved this way before and still do.

The design reasoning — the encoding, the compatibility posture, and why encrypted values are typed too — is in the internals chapter.

Worked example: approval-hold

examples/approval-hold is the reference wait-state flow: a startEvent on channel approval-request records the correlation alias (e2eId = payload.E2EId, unique, onConflict="correlate"), then a userTask on channel approval-decision — the hold — then an endEvent. Its bundled conformance test drives the full lifecycle against real PostgreSQL:

1. Park — an ApprovalRequest on approval-request returns 200 (accept, no business reply yet); the instance parks and its alias is recorded.
2. Duplicate rejected — a second ApprovalRequest with the same E2EId while parked is rejected (the durable unique-alias guard).
3. Relay resume — an ApprovalDecision on the separate approval-decision channel — a channel no start event subscribes to — is routed by the dispatcher's relay path: the same two-tier intake, against the wait node's q:source, then correlated by E2EId, then resumed.
4. Retire — the same E2EId is accepted again afterward, proving the alias was retired once the instance completed.
5. Uncorrelated relay is safe — a decision for an unknown E2EId is rejected and every parked instance is left untouched.

Try it by hand once the package is deployed (see Your first deployment):

# park
curl -sS -X POST localhost:8080/channels/approval-request \
  -H 'Content-Type: application/json' -H 'X-Api-Key: approval-demo-key' \
  -d '{"ApprovalRequest":{"E2EId":"E2E-1","Amount":"1500.00"}}'

# resume it with a decision
curl -sS -X POST localhost:8080/channels/approval-decision \
  -H 'Content-Type: application/json' -H 'X-Api-Key: approval-demo-key' \
  -d '{"ApprovalDecision":{"E2EId":"E2E-1","Decision":"APPROVE"}}'

Run the scenario itself with cd rust && cargo test -p sutra-conformance -- --ignored tc_approval_hold (needs Docker — see Test tiers).

Next

• Worked example: money-transfer — the other flagship example, showing durable data stores instead of a wait state.
• External tasks — the pull flavour of the same park: a delivery parked for a worker to fetch rather than dialed out.
• Replica semantics — how correlation and suspend/resume stay correct across a multi-replica engine.
• Durable execution — what a park actually persists, and why it is a snapshot rather than an event log.

External tasks: the pull worker surface

The engine is otherwise push-only: an outbound emission lands in the outbox and the relay dials a transport. A channel declaring transport: pull inverts that last hop — instead of dialing anything, the delivery is parked as a fetchable task, and workers come and get it.

That is the whole feature. It exists for work the engine cannot dial out to: a worker behind a NAT, on a laptop, in a language that has no inbound listener, or simply one that would rather poll than be called.

Declaring a pull channel

channels:
  - name: score-request
    transport: pull
    bind: "pull://acme/scoring/1.0.0/score-request"

Nothing in the BPMN changes. A <q:send> or a channel-call <bpmn:serviceTask> targeting this channel behaves exactly as it would over HTTP or a broker; only the last hop is different. And because the worker's answer comes back in on the same channel, the author's <q:alias> correlation is unchanged too — a pull task is not a new resume path.

Pull needs a datasource: a parked task is a database row. Without one the surface answers 503.

The worker protocol

Three operations, all under /sutra/external-tasks.

Fetch and lock

POST /sutra/external-tasks/fetch-and-lock
{ "workerId": "scorer-7", "channels": ["score-request"],
  "lockDuration": "PT1M", "maxTasks": 10, "asyncResponseTimeout": "PT20S" }

Claims up to maxTasks fetchable tasks on the named channels and locks them to workerId. A worker names topics (channel names), never deployment ids — the fetch walks the live deployment set for you.

When nothing is available the request is held open as a bounded long poll: it wakes the moment a task is parked on one of the channels you asked for, and answers with an empty list when asyncResponseTimeout elapses. It never hangs, and the wait is always bounded by the operator's ceiling.

Each returned task carries its id, channel, instance id, headers, body and content type, its attempt count, and its remaining failure budget.

Complete

POST /sutra/external-tasks/{id}/complete
{ "workerId": "scorer-7", "result": { ... } }

Feeds the worker's result back through the engine's ordinary inbound path — the same seam every transport delivers through. Correlation, validation, and inbox dedup all behave exactly as they do for a pushed reply; there is no second resume entry point.

Omitting the result re-delivers the original request payload. That is the fire-and-forget shape: the work happened outside, and the flow is waiting only on the fact of it.

Failure

POST /sutra/external-tasks/{id}/failure
{ "workerId": "scorer-7", "errorMessage": "downstream 500", "retries": 2, "retryTimeout": "PT30S" }

Releases the lock and spends one of the task's retries — or sets the remaining budget explicitly — deferring the next fetch by retryTimeout.

Lock expiry is the only recovery mechanism you need

A locked task is invisible to every other worker until its lock expires, at which point it becomes fetchable again. There is no sweeper, no reaper, and no timeout job: expiry is part of the predicate that decides what a fetch may claim, so a worker that dies mid-task costs exactly one lock duration and never costs the work.

A completion or failure from a worker that no longer owns the lock fails closed, and the refusal names which of the three situations it is:

A stale worker never receives a 200 it could mistake for success.

At-least-once, and what makes that safe

The task row is deleted only after the engine has accepted the completion. A crash in the window between the two re-offers the task, so the work is never lost — the surface is at-least-once, deliberately, because the inverse ordering (delete, then dispatch) would be at-most-once and would drop work outright on the same crash.

What makes the duplicate harmless is that each task carries the originating outbound delivery's key as an explicit idempotency key, and the completion re-enters under it. Inbox dedup absorbs the second attempt. A worker does not have to do anything to get this — it is a property of the surface, not of the worker.

If the engine refuses a completion on the inbound path (a validation reject, say), the answer is 422 with the engine's own code carried as attributes.causeCode — which is what tells the worker whether re-fetching later can ever help. The task is retained and becomes fetchable again.

Two reserved headers a worker never sees

For completeness: internally, the dispatcher stamps two reserved headers — sutra-park-deployment and sutra-park-instance — onto a delivery whose destination scheme is pull, and the pull sink strips them as it parks the task. They carry the owning deployment (the isolation key) and instance to the parking side without widening the transport contract every other sink implements, and the scheme gate means they can never leak onto a network transport. They never appear in a fetched task's headers; nothing a worker builds should look for them, and their presence in a payload you construct yourself has no effect — the sink's own stamp always wins.

The worker retry budget

A freshly parked task starts with sutra.external-task.retries failures (default 3). Each reported failure spends one and defers the next fetch by sutra.external-task.retry-timeout (default PT10S).

A spent budget makes the task terminal: never fetched again, retained with its last error. That is the pull-side twin of the outbox's poison horizon, and it exists for the same reason — so "we gave up" can never degrade into "it silently vanished". A terminal task no longer counts toward its deployment's retirement quiescence gate.

Bounds are rejections, never clamps

Every knob a worker can send is a ceiling an operator sets, and a request over the ceiling is a 400 — not a silent adjustment down:

The reason it is a reject is a correctness one: a worker that believes it holds a longer lock than it actually does is a duplicate-execution bug waiting to happen. Silently clamping would create exactly that belief. Durations are ISO-8601 everywhere on this surface, the same grammar the engine's cadence keys and <q:retry> use.

The engine also boots fail-closed if default-lock-duration exceeds its own ceiling or is zero.

Posture

These are operate-surface routes (/sutra/*), not administrative ones: a completion is an ordinary delivery, not a privileged control operation. They carry the same cluster-internal posture the rest of /sutra/* does. A deployment that needs authenticated workers puts them behind the same ingress policy the rest of the operate surface already needs — see Configuration reference for the separately gated admin surface.

How it composes with <q:retry>

Two budgets, at two layers, that never overlap:

A worker retrying its own work never touches the instance. A <q:retry> policy on a pull-backed channel call still governs the task-level outcome — its timeout is still the thing that eventually fires if no worker ever completes. See Retries, history, and schedules.

Next

• Channels and transports — where transport: is declared and what the other transports do.
• Wait states and human tasks — the park/resume model a pull task rides on.
• Configuration reference — the key family in full.
• The pull surface: the design reasoning — ownership transfer, why zero rows affected is never success, and why there is no sweeper.

Retries, history, and schedules

Three execution-semantics features an author reaches for once a flow has to survive contact with an unreliable world: a per-task retry policy, execution history that outlives the instance, and the full set of BPMN timer schedules.

<q:retry> — a per-task retry policy

A <bpmn:serviceTask> may declare a retry policy inline. It applies to both kinds of service task: a registered task (a function the embedding application registered) and a channel-call task (implementation="channel:<name>" — the wait-state call every pure-BPMN deployment uses through the shipped binary).

<bpmn:serviceTask id="Score" implementation="registered:score">
  <bpmn:extensionElements>
    <q:retry maxAttempts="3" initialDelay="PT1S" backoffCoefficient="2.0"
             maxDelay="PT5M" nonRetryableCodes="SUTRA.TASK.VALIDATION"/>
  </bpmn:extensionElements>
</bpmn:serviceTask>

So maxAttempts="4" initialDelay="PT1S" backoffCoefficient="2.0" waits 1 s, then 2 s, then 4 s between its four attempts — each capped at maxDelay.

A retry wait is a durable park, never a sleep

This is the load-bearing property. A failed attempt with budget remaining persists the instance with the failed task still pending and an armed timer due at the backoff instant; the ordinary timer poller re-drives it when that instant arrives.

Nothing sleeps, and nothing is held in memory. The consequences are the ones you want:

• the backoff survives a restart, a rolling upgrade, and a hot-deploy (a retry park is pinned to its deployment like every other park);
• no execution lane is blocked during the wait, so a task backing off costs other instances nothing;
• a PT5M backoff is a PT5M backoff whether or not the replica that started it is still alive.

Declaring <q:retry> on a task makes its process stateful — the park needs persistence, so a process that was otherwise run-to-completion now requires a configured datasource.

What counts as a failed attempt

The failure set is deliberately narrow and different for each task kind.

On a registered task: an uncaught error from the task function.

On a channel-call task, exactly two things:

• the route-less <q:timeout> boundary firing before the correlated response arrived (classification SUTRA.DISPATCH.CHANNEL_CALL.TIMEOUT). Without a policy a fired timeout is a catchable BPMN error; with one it becomes a retryable task failure first;
• the request delivery being marked terminally poisoned by the outbox attempt ceiling (classification SUTRA.OUTBOUND.DELIVERY_ATTEMPTS_EXHAUSTED, reachable only when sutra.outbox.retry.max-attempts is configured). The failure reaches the task the moment the delivery gives up, rather than waiting out the whole timeout window.

Both classifications are stable structured codes, so nonRetryableCodes can name them — "retry a poisoned delivery but never a timeout" is expressible.

What is never a failed attempt:

• A correlated business response. The counterpart answered. Whatever the answer says is the process's business to route on; re-sending would double-submit. A response is a completion, full stop.
• A BPMN error. Errors route to their boundary events, unchanged, on either task kind.

Modelled outcomes always beat a policy

A timer boundary event with drawn outgoing flows is a modelled outcome — the author has said what happens on timeout, so that is what happens, exactly as a BPMN error routes to its boundary instead of consuming a retry budget. Which is why the loader refuses the combination: a channel-call <q:retry> requires the route-less <q:timeout> form, and a routed timer boundary alongside a retry policy is a load error (SUTRA.CONFIG.BPMN.RETRY_NOT_APPLICABLE). A policy that could never fire would otherwise load as a silent near-no-op.

What a re-drive actually does on a channel call

Re-driving a channel call is not "wait for the same answer again" — it re-issues the request:

• a fresh outbound request built from the same persisted variables, with a fresh idempotency key on the wire (so a counterpart doing its own dedup sees a genuinely new request, not a redelivery);
• a fresh timeout window;
• the dead attempt's outstanding request withdrawn, so a late answer to a superseded attempt cannot race the new one.

A response that arrives for the dead attempt is refused with SUTRA.DISPATCH.CHANNEL_CALL.RETRY_PENDING — an honest verdict rather than a "no such instance" miss, because the correlation is deliberately kept alive. Once the re-issued request goes out, the same correlation serves its response normally.

Exhaustion

A spent budget — or a nonRetryableCodes hit — fails the instance durably with SUTRA.RUNTIME.RETRY.EXHAUSTED. The instance becomes FAILED: retained, inspectable, blocking its deployment's retirement, and repairable through instance migration. See Replica semantics.

Two budgets that never overlap

If a channel uses the pull transport, its worker-side failures are governed by the external task's own budget (sutra.external-task.retries), not by <q:retry>. The two live at different layers and never interact — see External tasks.

The outbox poison ceiling

Outbound deliveries retry with backoff forever by default: an unreachable counterpart is treated as temporarily unreachable, and giving up silently is worse than waiting.

sutra.outbox.retry.max-attempts is the opt-in ceiling. A delivery that exhausts it becomes terminally poisoned: retained with its last error, never retried again, one incident recorded if the delivery was required, and — importantly — no longer counted by the draining-deployment retirement gate. "We gave up" is a durable, visible state, never a silent disappearance.

Configuring the ceiling is also what makes SUTRA.OUTBOUND.DELIVERY_ATTEMPTS_EXHAUSTED reachable as a <q:retry> failure at all.

Execution history: a finished instance is not a 404

A terminal instance used to vanish. It doesn't: the terminal step re-stamps the stored snapshot to COMPLETED / TERMINATED in the same transaction that resolves its waits, retires its correlation aliases, and enqueues its final emissions.

Retention. sutra.instance.retention (ISO-8601, default P7D) is how long terminal snapshots stay queryable; a lease-gated sweeper purges past-window rows on sutra.instance.retention-sweep-interval (default PT1H). PT0S restores delete-at-completion for a deployment that wants no history at all.

FAILED is always retained, regardless of the window — see above. And the audit journal itself stays opt-in (sutra.audit.sql): with it off, the history endpoint answers an explanatory empty shape rather than a misleading 404, and the retained terminal snapshot still answers the inspect route.

Timer completeness: durations, dates, cycles, and timer starts

All three BPMN timer definitions are supported, on all three timer hosts (start event, intermediate catch, boundary event):

Deliberately not supported, and rejected as such rather than silently mis-parsed: cron syntax, and calendar-arithmetic durations (P1M meaning "one calendar month"). A malformed expression and an unsupported-but-well-formed one produce different diagnostics — "you wrote it wrong" and "the engine does not do that" are different problems.

Timer start events are schedules

A start event carrying a timer definition is a schedule, not a wait: it has no instance until it fires. Schedules are armed when a deployment becomes active, re-armed on boot, and resolved when the deployment stops being active — so hot-deploy handoff, undeploy, and retirement all need no extra operator step.

A fire mints an instance through the ordinary dispatch path, with empty variables: tenancy, quotas, audit capture, and coverage marking all apply exactly as they do for a message-triggered start.

Two rules keep cycles honest:

• A cycle never bursts. If the engine was down across three occurrences of an R/PT1H cycle, it fires once and moves to the next slot — missed occurrences are coalesced, not replayed.
• A repeat budget is spent by the grid, not by the firing. Each skipped slot of an R3/PT1H consumes one repeat, so the schedule can never outlive its third slot.

A start event carrying both a <q:source> and a timer definition is a load error — a start is triggered by a message or by a clock, never ambiguously by both.

Next

• The q: namespace — <q:retry> in the vocabulary table alongside <q:timeout>.
• Testing time — proving a PT24H timer or an R3/PT12H schedule in wall-clock seconds.
• Instance migration — what to do with an instance that exhausted its budget.
• Configuration reference — the retention, sweep, and outbox keys named on this page.
• Retry machinery: the design reasoning — why a backoff is a park, and how a dead attempt is told apart from an in-flight one.

Testing time: fast-forwarding durable timers

A flow that waits PT24H for a reminder, or fires an R3/PT12H schedule three times, is exactly the kind of thing that is hard to test and therefore usually untested. Sutra's answer is a virtual clock: the engine's notion of "now" becomes something a test drives explicitly, so a day-long timer settles in wall-clock seconds against a real database, with real durable rows and the real timer poller.

Two ways in: a CLI command for BPMN authors, and an embedded seam for anyone writing tests in Rust.

sutra test simulate — no test code required

sutra test simulate --deployments <DIR> --datasource <URL> \
    (--advance <DURATION> | --until-quiescent) [flags]

Boots a real engine on a dynamic port against a directory of sealed deployment archives with a virtual clock installed, fast-forwards it, reports, and shuts down cleanly.

This is unrelated to sutra simulate, which is a dry-run routing report over a single BPMN file and never boots anything. They are separate commands on purpose.

Exactly one of --advance / --until-quiescent is required.

It refuses to run against data it did not expect

Fast-forwarding a virtual clock against a database that already holds real, in-flight instances would durably fire their real timers early. A datasource mixup must not silently do that, so before booting anything the command checks that the target database holds no instances, and refuses with exit 2 if it does.

--allow-existing-data is the explicit acknowledgement — and also the right shape for "seed one instance externally, then fast-forward it".

--datasource being required is part of the same posture: a persistence-less run has no durable timers to advance at all, so the command refuses rather than pretending to have done something.

Quiescence, precisely

--until-quiescent stops when, simultaneously: no waiting row is an armed timer, no schedule is armed, and no instance is still live.

A FAILED instance is deliberately not counted as finished — it needs an operator, not a clock. So a fixture that fails under fast-forward correctly reports timedOut: true rather than a false "quiescent", and the summary's instancesFailed says why.

Output

Human-readable progress goes to stderr. The final summary is one JSON object on stdout and nothing else touches stdout, so piping into jq is always safe:

{
  "mode": "until-quiescent",
  "deployments": "/path/to/deployments",
  "allowExistingData": false,
  "timedOut": false,
  "quiescent": true,
  "preExistingInstances": 0,
  "virtualStart": "2026-08-06T00:00:00Z",
  "virtualEnd": "2026-08-07T00:00:03Z",
  "virtualSecondsAdvanced": 86403.2,
  "wallSeconds": 1.84,
  "instancesStarted": 1,
  "instancesCompleted": 1,
  "instancesFailed": 0,
  "instancesTerminated": 0,
  "instancesLive": 0,
  "timersFired": 1,
  "schedulesFired": 0
}

schedulesFired counts single-shot and bounded (R<n>/…) timer-start fires exactly. An unbounded cycle (R/…) has no repeat budget to difference, so its fires show up in instancesStarted but are not separately counted in this release.

Use it to prove, in your application's own CI, that a durable-timer or cyclic-schedule flow settles correctly — without a wall-clock-length test run.

The embedded seam

sutra_engine::TestClock is a manually-advanced virtual clock, and sutra_engine::fast_forward_until is the paired driver. Install the clock on the engine config before serving, and every temporal read in that boot uses it: timer park due instants, <q:retry> backoff instants, the timer poller's per-tick claim instant, and schedule arming.

#![allow(unused)]
fn main() {
let clock = sutra_engine::TestClock::starting_now();
let engine = sutra_engine::serve(sutra_engine::EngineConfig {
    now_override: Some(clock.clone()),
    ..config
}).await?;

// park a PT24H timer, then:
let settled = sutra_engine::fast_forward_until(
    &pool, &clock, std::time::Duration::from_secs(10),
    || async { live_instance_count(&pool).await == 0 },
).await;
assert!(settled);
}

Every clone of the clock is a handle onto the same instant, so the code under test and the test driving it always agree. starting_now() starts at the real current instant — only the fast-forward is virtual — which keeps any absolute-time assertion elsewhere in the test (log timestamps, audit rows) sane.

The driver loop is deliberately simple: jump the clock to the earliest armed due instant, give the real timer poller a tick to claim and fire it, re-check your condition; repeat until the condition holds or the real wall-clock timeout elapses. It only ever touches the clock and the database — never the poller's internals. Set the poller's tick interval low for the test boot so a tick is cheap.

It is unreachable from a deployed engine — by construction

There is no config key, no environment variable, and no CLI flag on the engine binary that installs a virtual clock. Wiring one in is always an explicit choice in Rust at the call site, so an operator has no way to reach it in a deployed engine.

sutra test simulate is not a hole in that: it is a different binary that constructs an engine configuration directly in code — exactly the sanctioned pattern, just packaged as a reusable tool instead of a bespoke integration test per application. The engine's own configuration loading is untouched by it.

What it is good for

• A timeDuration catch timer or <q:timeout> boundary that would otherwise need a real wait.
• A timeCycle schedule firing its full repeat budget.
• A <q:retry> backoff curve running to exhaustion — every attempt's park is a real durable row, and fast-forward walks them all.

See Retries, history, and schedules for what each of those constructs does, and Testing tiers for where a test like this belongs in the suite.

Next

• Retries, history, and schedules — the timers being fast-forwarded.
• sutra CLI reference — the command in the full CLI map.

Coverage: declared routes as the compliance signal

Compliance in a Sutra deployment isn't reconstructed from logs after the fact. A module owner declares the business-event routes that matter — "the happy-path completion," "the reject branch," "the end-to-end route across all three collaborating processes" — and the engine ticks each one off as real instances walk it. That declaration is the whole compliance surface: a route nobody declared can never show up as covered or uncovered, so what you choose to declare is exactly what you can ever detect. This chapter covers the two shapes a route can take, how to declare each, the CLI that drives them, and — the part that actually determines whether the feature is useful — how to curate the declared set down to what's worth watching.

What "covered" means

A route is an ordered path through execution: for an intra-process route, an ordered list of one process's own sequence-flow ids; for a cross-process route, a per-process segment of flows for each participant in a correlated cascade, tied together by a business key that threads every hop. A route is marked covered the moment an instance (or, cross-process, a correlated group of instances) walks every one of its flows in order; it stays uncovered since the last reset otherwise. There is no partial credit and no percentage for anything you didn't declare — total is exactly the number of declared routes, covered is how many have flipped, and the uncovered list is the rest.

Both shapes land in one place: the coverage data store the deployment declares in its own datastores.yaml, as a typed coverage_metric row per declared route carrying a covered boolean — entirely separate from the audit trail, so coverage never reads audit tables and audit never reads coverage's. That's also what makes coverage cheap to skip: a process with no declared route pays nothing (no flow-trace capture, no writes). See Where coverage is stored for the division of labour that store implies: you pick the database, the engine owns the schema.

Intra-process: <q:coverage> inline

The simplest case declares a route directly on the process it belongs to, as an ordered list of sequence-flow ids — the same ids the diagram already shows on every arrow:

<bpmn:process id="transfer">
  <bpmn:extensionElements>
    <q:coverage path="accept" flows="Flow_TxToOk Flow_OkToEnd"/>
    <q:coverage path="reject" flows="Flow_CancelToReject Flow_RejectToEnd"/>
  </bpmn:extensionElements>
  …
</bpmn:process>

A run covers a path when its fired-flow trace contains flows as an ordered subsequence — so whatever intake channel started the instance (money-transfer's transfer.bpmn has three) is irrelevant to whether accept or reject gets marked, and a wait-state park in the middle doesn't break the match. See Worked example: money-transfer for this declaration driving a real ACID transfer flow end to end.

Fail-closed checks at load: a flow id the process doesn't have, a set of flows that isn't a contiguous route (each flow's target must be the next flow's source), or a reused path id inside one process all evict the module rather than silently declare something meaningless. A process that declares at least one <q:coverage> must also declare a coverage store in datastores.yaml (SUTRA.CONFIG.COVERAGE.STORE_MISSING at lint time) — that store is where the marks are persisted, so declaring routes without it is declaring something with nowhere to record it. The scaffold satisfies the rule for you; Where coverage is stored is what the declaration actually buys.

Cross-process: a coverage file

A route that spans more than one participant process can't live inside a single <bpmn:process> — there's no BPMN element for "the flow that starts here and finishes three hops away in a different process." Instead it's declared in a coverage file under the package's coverage/ folder, identified the same way a template is: path-derived, urn:sutra:coverage:<folder…>:<file>. A file holds one or more correlations — the business key that ties instances of the cascade together, and the links (hops) between them — plus one or more routes (coverages:) that each state a complete set of per-process segments.

The shape that motivates this is a relay cascade: three collaborating services — an intake service, a routing hub, and a fulfillment service — each its own <bpmn:process>, handing off over channels rather than calling each other directly. No single process sees the whole exchange, so "did the end-to-end handshake complete, and which way did it end" is exactly the question none of them can answer alone. A coverage/e2e.yaml (urn:sutra:coverage:e2e) declares the routes that span all three:

correlations:
  - id: relay-case
    key: caseId                            # default hop key — every leg correlates on the same case id
    links:                                 # the <q:send> -> <q:source>/imec hops between the 3 processes
      # forward: intake -> hub (spawns the hub)
      - { from: intake-svc:Task_ForwardToHub,        to: hub-svc:Start_Received }
      # forward: hub -> fulfillment (spawns fulfillment)
      - { from: hub-svc:Task_ForwardToFulfillment,   to: fulfillment-svc:Start_Posting }
      # reply: fulfillment -> hub (resumes the hub's imec) - accept and reject both land here
      - { from: fulfillment-svc:Task_ReplyAccept,    to: hub-svc:Imec_AwaitFulfillment }
      - { from: fulfillment-svc:Task_ReplyReject,    to: hub-svc:Imec_AwaitFulfillment }
      # reply: hub -> intake (resumes the intake's imec)
      - { from: hub-svc:Task_ReplyToIntake,          to: intake-svc:Imec_AwaitReply }
    coverages:
      - path: e2e-accepted                 # the accepted branch
        segments:
          intake-svc:      [Flow_p1_start, Flow_p1_await, Flow_p1_done]
          hub-svc:         [Flow_p2_start, Flow_p2_await, Flow_p2_reply, Flow_p2_done]
          fulfillment-svc: [Flow_p3_start, Flow_p3_accept, Flow_p3_accept_end]
      - path: e2e-rejected                 # the rejected branch — same p1/p2, different p3
        segments:
          intake-svc:      [Flow_p1_start, Flow_p1_await, Flow_p1_done]
          hub-svc:         [Flow_p2_start, Flow_p2_await, Flow_p2_reply, Flow_p2_done]
          fulfillment-svc: [Flow_p3_start, Flow_p3_reject, Flow_p3_reject_end]

A few things about this shape:

• links are structural, not payload access. Each hop names a <q:send> node on one side and either a start-event <q:source> (a spawn — the hop starts a new instance) or an imec relay-wait <q:source> (a relay — the hop resumes a parked instance) on the other, matched against the actual channel wiring. sutra lint validates every link resolves against real channel bindings, that every segments flow id is contiguous within its own process, and that the correlation key resolves at both ends of every hop — a broken link or an unresolvable key evicts the module rather than deploying a route that can never complete.
• The correlation key is how the runtime reconstructs the cascade, since three separately dispatched instances have no shared instance id. Here it's a single value — a case id — carried as an author-declared header rather than a payload field: every <q:send> sets it (<q:header name="caseId" value="caseId"/>) and every consuming <q:source> reads it back (<q:alias name="caseId" expression="header.caseId"/>). A hop can override the correlation's default key when its own leg correlates on a different value — this cascade doesn't need to, since one case id threads every hop end to end.
• Each route is fully self-contained. e2e-accepted and e2e-rejected repeat their identical intake-svc and hub-svc segments rather than one inheriting from the other — there is no route-to-route inheritance, by design, so reading any one route never requires cross-referencing another.
• A route can be declared but structurally rare. e2e-rejected is only reachable when the fulfillment process takes its reject branch — under normal traffic that rarely happens, so e2e-rejected can sit uncovered indefinitely. That's not a bug in the declaration; it's the compliance signal doing its job: "the reject path exists and we watch for it," distinct from "the reject path fired." It also makes a clean assertion for a test campaign: drive one ordinary case, expect e2e-accepted to reconstruct as covered and e2e-rejected not to, then reset and confirm both go back to uncovered.

At load, each route's per-process segments are injected as ordinary intra-process coverage paths on their own process (so the existing per-process marking is unchanged); on completion each segment writes a correlation-tagged record, and a route flips covered once every one of its segments has landed in the same correlated group. Reading total/covered/percentage for a cross-process route is the same store query as an intra-process one — the mechanism composes rather than duplicating.

Composing several definitions in one package

Nothing limits a package to one kind of declaration. A deployment can carry <q:coverage> on several of its own processes and one or more coverage/*.yaml files, each with its own correlation and its own set of routes — money-transfer's inline accept/reject pair and the file-based e2e-accepted/e2e-rejected pair above are the two shapes, and a single package is free to declare both kinds side by side: an intra-process route for "did this one process's own retry/compensation branch fire," alongside a cross-process file for "did the whole multi-participant handshake complete." Each declaration is independent — an intra-process path only needs its own process's flow ids; a coverage file only needs the processIds and channel wiring it names — so there's no coordination cost to adding more of either kind as a module grows.

Where coverage is stored

Coverage marks are persisted in the coverage data store the deployment declares — a reserved store name in its own datastores.yaml, declared like any other store. That declaration is how you choose the database: the connection it names is where the marks land, and the URL scheme picks the dialect (PostgreSQL, MySQL/MariaDB and SQL Server all work). The engine hosts no coverage in its own database.

What you do not write is coverage SQL. The coverage tables are a built-in feature, so the engine owns their schema: it ships the coverage_metric / coverage_fragment DDL per dialect and applies it to that connection the first time the store is used — the same idempotent, lock-serialized first-use path a module's own migrations/<store>/ scripts take, with engine-shipped scripts instead of package-supplied ones. So the store block carries no migrations: key, and a package carries no migrations/coverage/ folder at all:

datastores:
  - name: coverage                        # the reserved name — this declaration picks the DATABASE
    type: sql
    sql:
      url-ref: env:ACCOUNTS_DB_URL
      username-ref: env:ACCOUNTS_DB_USER
      password-ref: env:ACCOUNTS_DB_PASSWORD
                                          # no `migrations:` — the engine owns this store's schema

Pointing it at a connection some other store already uses is fine and often convenient — money-transfer's coverage store names the same database as its accounts ledger. The coverage tables are the engine's own; they sit beside the business ones rather than mixing with them.

On activation, a deployment seeds one row per declared route into that store's coverage_metric table with covered = false — intra-process path ids and cross-process route ids alike. (A cross-process route's per-process segments are marking cursors, not routes: many segments collapse onto the one route flag, which is why total counts the route once.) Execution flips flags with a guarded UPDATE … AND NOT covered, so first-covers-wins is settled by the write itself — the affected-row count is the answer to "did this run newly cover it," never a read-then-write race. Nothing else writes there. That typed table is the whole coverage substrate, and three consequences follow from it.

Counts are SQL, not a fold in the engine. total and covered are one aggregate over the seeded set (COUNT(CASE WHEN … THEN 1 END) — the portable pivot, which returns a count on every shipped dialect), and the uncovered routes are their own ordered query. Both run inside one REPEATABLE READ transaction, so the count and the list can never disagree: they describe one snapshot. A percentage is still derived on read, never stored — there is no counter to drift.

Flags are keyed by deployment and route id. deployment_id is a column on both tables and a bound predicate on every statement the engine issues; that, not row-level security, is the isolation. (RLS is an engine-database convention — it wants table ownership and a per-transaction setting the engine cannot assume on a connection you own, and two of the three shipped dialects have no equivalent anyway.) Two processes in one deployment that both declare a path called accept share one flag; the CLI's total has always counted them once. Name a route for the business event it represents rather than for the branch it sits on, and the question never comes up. The same keying is what makes a runtime report of a cross-process route read against the route flag rather than the segment that happened to finish — one number, and the same one sutra coverage check --archive reports.

A declared store the engine can't open fails loudly. Coverage exists wherever you declare a store, so the failure mode is no longer "this deployment has no database of its own to record in" — it is a coverage store whose connection is wrong, unset or unreachable. The engine logs an error at boot naming that deployment, and a coverage:report / coverage:reset op fails with SUTRA.CONFIG.COVERAGE.STORE_MISSING naming the cause. It does not report 0%:

a report of 0% here would be indistinguishable from a real measurement of nothing covered

A deployment that declares <q:coverage> paths but no coverage store at all is the same story one step earlier: sutra lint errors on it at package time, and an engine that loads such a package anyway warns at boot and fails those two ops with the same code. The reasoning extends to a store that opens but whose read fails — the op surfaces the underlying error rather than counting every route as uncovered. A compliance signal that quietly degrades to "nothing is covered" is worse than one that stops and says why. (Runtime marking stays best-effort — a metric side-effect must never fail a business instance. The loud surface is the report, which is what a human or a CI gate actually reads.)

A package built against an older engine may still carry a migrations/coverage/ folder. It is dead weight: nothing in it was ever coverage DDL — the script there created the same generic table a business store already creates — and the engine no longer reads a migrations: key on the reserved coverage store. Delete the folder.

The sutra coverage CLI

init — enumerate and seed

Single-process form: point it at a BPMN file and it walks every start→end route over the engine's own execution semantics (gateways, sub-processes, the lot), then seeds <q:coverage> declarations plus a matching admin scaffold (report/reset BPMN, reply templates, the two admin channels, and the coverage store declaration — no SQL, because none of it is yours to write):

$ sutra coverage init bpmn/transfer.bpmn --process transfer
coverage init: bpmn/transfer.bpmn — process 'transfer', 2 path(s) declared (0 kept, 2 new)
  path path-1: Flow_TxToOk Flow_OkToEnd
  path path-2: Flow_CancelToReject Flow_RejectToEnd
  updated  bpmn/transfer.bpmn
  created  bpmn/coverage-report.bpmn
  created  bpmn/coverage-reset.bpmn
  created  templates/coverage-report.hbs
  created  templates/coverage-reset.hbs
  updated  channels.yaml
  updated  datastores.yaml

Route enumeration refuses beyond a 256-route cap (--max-paths raises it): a process with enough parallel/inclusive branching to combinatorially explode past that is a sign the process itself is too fine-grained for route-level compliance tracking, not that the cap is wrong — raise --max-paths only when you've confirmed the extra routes are genuinely distinct business outcomes, not gateway noise. A second init run keeps any path id whose flows still match (renames survive), refuses to clobber hand-edited declarations without --force, and never replaces a file it can't safely re-parse.

Cross-process form: name a coverage file and the processIds it should span instead of a single BPMN file. This does not enumerate a path set at all (there's no combinatorial explosion to cap) — it emits the connectable graph: each process's own flow adjacency, plus every inter-process hop it can infer from <q:send>/<q:source>/imec wiring, as a commented scaffold with one starter route to trim:

$ sutra coverage init coverage/e2e.yaml intake-svc hub-svc fulfillment-svc
coverage init: coverage/e2e.yaml — urn:sutra:coverage:e2e
  processes: intake-svc, hub-svc, fulfillment-svc
  intra-process adjacency: 9 flow(s) across 3 process(es)
  inter-process hops: 5
    intake-svc:Task_ForwardToHub --to-hub--> hub-svc:Start_Received [spawn fire-and-forget] key=caseId
    hub-svc:Task_ForwardToFulfillment --to-fulfillment--> fulfillment-svc:Start_Posting [spawn fire-and-forget] key=caseId
    fulfillment-svc:Task_ReplyAccept --to-hub-imec--> hub-svc:Imec_AwaitFulfillment [relay request-reply] key=caseId
    fulfillment-svc:Task_ReplyReject --to-hub-imec--> hub-svc:Imec_AwaitFulfillment [relay request-reply] key=caseId
    hub-svc:Task_ReplyToIntake --to-intake-imec--> intake-svc:Imec_AwaitReply [relay request-reply] key=caseId
  scaffold written — draw connected walks from it (sutra lint validates connectivity)

The written file's coverages: starts with exactly one route (path-1) whose segments list every connectable flow id per process — the raw graph, not a curated path. --single restricts the graph to intra-process adjacency only (no inter-process hops, for a package still being wired up one participant at a time); --force overwrites an existing file.

check — read covered/uncovered, drive assertions

Bare BPMN file: a drift lint, not a store read. It confirms every declared path is still an ordered subsequence of the process's current flow graph — the check that catches a declaration silently breaking when someone reworks the diagram — and reports (informationally) any enumerated route no declared path covers:

$ sutra coverage check bpmn/transfer.bpmn
coverage check: 0 error(s), 0 note(s)

--archive selects the store-backed, correlation-aware check instead: it reads the deployment's seeded metric flags, union-finds the cross-process reconstruction records to flip any route that has now fully completed, and reports the same total/covered/percentage a live dashboard would read — the fail-closed CI gate and the runtime signal are the identical query. --database-url points at the database the deployment's coverage store declares, in whichever of the three shipped dialects it names; the CLI reads and writes exactly what the engine does, and creates nothing the engine would not create itself:

$ sutra coverage check --archive deployed.sutra --database-url "$SUTRA_DB_URL" --threshold 100
coverage check (cross-process) — deployment 3f9a1c…
  total: 2  covered: 1  coveragePercentage: 50.00%
  newly covered this run (1):
    + urn:sutra:coverage:e2e:e2e-accepted
  uncovered (1):
    - urn:sutra:coverage:e2e:e2e-rejected
  threshold: 100.00%  =>  FAIL

--threshold (default 100) is the gate: the process exits non-zero whenever the percentage falls below it, so a CI pipeline can fail a release that hasn't exercised every route it declared as required — or, set lower, tolerate a known-rare route like e2e-rejected without losing visibility into whether it ever fires.

reset — re-seed a deployment's declared routes

$ sutra coverage reset --archive deployed.sutra --database-url "$SUTRA_DB_URL"
coverage reset — deployment 3f9a1c…: 2 path(s) re-seeded covered=false, reconstruction fragments cleared

Every declared path (intra- and cross-process alike) goes back to covered = false and every cross-process reconstruction record is cleared — the clean baseline a new test campaign, or a new reporting period, starts from. The rows stay seeded; reset is one scoped update that flips the flags, so total is unchanged and the cleared count is exactly how many routes had been covered.

Curation: init enumerates, you decide what matters

init's job is mechanical completeness — surface every structurally valid route (or, cross-process, the whole connectable graph) so nothing gets missed by hand. It is deliberately not trying to guess which of those routes represents a business event worth watching, and treating its raw output as the final declared set is the single most common way this feature stops being useful.

Notice, in the single-process example above, that init names its two routes path-1 and path-2 — it has no way to know one is the accepted-transfer outcome and the other is the rejected one. Money-transfer's shipped transfer.bpmn renames them to accept and reject by hand — that rename is the curation step, and it's why sutra coverage check --archive output reads as a compliance report ("the reject path never fired this quarter") rather than a cryptic route id. The same discipline applies to the cross-process scaffold: coverage init's starter route lists every connectable flow per process, which is almost never the walk you actually care about — the e2e.yaml above is that raw scaffold trimmed down to exactly the two branches (accepted vs. rejected) the cascade wants visibility into, with routes named for the outcome they represent rather than left as path-1.

This matters because "uncovered" is the entire signal, and noise defeats it. A moderately branchy process can mechanically enumerate dozens of technically-distinct full paths — different orderings through independent parallel branches that don't correspond to different business outcomes at all. Declare all of them and every report is dominated by routes nobody ever intended to track individually, burying the one genuinely-uncovered path that matters under a wall of noise. Keep the declared set as large as the compliance surface you actually want visibility into — named after the business events they represent — and no larger.

Next

• Worked example: money-transfer — the single-process form driven by a real ACID transfer flow.
• sutra CLI — every flag on init/check/reset.
• Troubleshooting BPMN solutions — reading a coverage report that doesn't match what you expected.

Worked example: money-transfer

examples/money-transfer is the flagship demonstration of full ACID semantics on a single writable ledger, driven from one BPMN process over three different transports (HTTP, RabbitMQ, Kafka). It exercises most of what the last few chapters covered — deployment packages, channels, the q: namespace, and data stores — in one real flow.

The shape

examples/money-transfer/deployments-src/default--money-transfer--1.0.0/
├── package.yaml                         # labels {tenant: default, module: money-transfer, version: 1.0.0}
├── bpmn/transfer.bpmn                   # the ACID transfer flow
├── bpmn/balance-query.bpmn              # read-only balance lookup
├── bpmn/coverage-report.bpmn            # admin: path-coverage report
├── bpmn/coverage-reset.bpmn             # admin: path-coverage reset
├── channels.yaml                        # 6 channels — 3 intake transports + balance + 2 admin
├── datastores.yaml                      # `accounts` (the ledger) + `coverage` (engine-owned schema)
├── migrations/accounts/V001__accounts.sql
├── schemas/transfer/transfer.xsd        # TransferRequest + BalanceQuery
├── schemas/transfer/codec-manifest.yaml
└── templates/*.hbs                      # reply rendering (accept / reject / balance / coverage)

What it demonstrates

A module-owned SQL data store. datastores.yaml declares accounts — the ledger, one row per account holding {balance, frozen} — with its own connection (env:ACCOUNTS_DB_URL/_USER/ _PASSWORD) and its own idempotent migration under migrations/accounts/. See Data stores.

Per-channel singleton serialization. Three intake channels — transfer-request (HTTP), transfer-queue (RabbitMQ), transfer-topic (Kafka) — all drive the same transfer.bpmn, each declaring singleton: true. On a queue transport that's leader-gated across replicas (a per-channel PostgreSQL-lease election — see Replica semantics); on HTTP the single-writer guarantee instead comes from the <bpmn:transaction> scope plus FOR UPDATE row locks in the flow itself. The read-only balance channel and the coverage-* admin channels are not singletons — they scale across every replica, since they only read or reset shared state rather than serialize writes to it.

A custom module schema. schemas/transfer/transfer.xsd declares TransferRequest and BalanceQuery as sibling root elements; the codec registers as urn:transfer (see Deployment packages for the path-derived URN rule), and every channel in channels.yaml binds codec: urn:transfer.

The flow, node by node

transfer.bpmn has three start events — one per intake channel — that all converge on a single <bpmn:transaction> sub-process named Transfer:

Start (http) ─┐
StartQueue ────┼──▶ Transfer [transaction]:
StartKafka ───┘        SubStart → LoadFrom → LoadTo → Valid?
                          ├─ ok:      Compute → Persist → SubEnd (normal end → COMMIT)
                          └─ invalid: DecideReason → cancel end event (→ ROLLBACK)
                     → (committed) OkReply (render TransferAccepted) → End
                     → (cancelled, via a boundary cancel event on Transfer) RejectReply → End

Full ACID, mapped onto real BPMN + q: elements:

• Atomicity — LoadFrom/LoadTo/Compute/Persist all run inside the <bpmn:transaction>; a normal end commits both balance writes together, a cancel end (reached from the Valid? gateway's invalid branch) rolls back — no partial transfer ever lands.
• Consistency — the Valid? exclusive gateway checks fromAccount.frozen, toAccount.frozen, and fromAccount.balance < payload.amount before any write, in a visible FEEL condition on the sequence flow.
• Isolation — LoadFrom/LoadTo read their rows with <q:store key="payload.fromId" forUpdate="true"/> — a pessimistic lock serializing concurrent transfers that touch the same account — layered under the channel-level singleton/exclusive-consumer contract.
• Durability — Persist writes both new balances back to the accounts store in the same transaction; a later instance (on any replica) sees the committed values.

Compute and DecideReason are pure FEEL data-assignment nodes (<bpmn:assignment> pairs of a <from> FEEL expression and a <to> target variable) — no service-task implementation code anywhere in this flow.

Compliance path coverage

<q:coverage path="accept" flows="Flow_TxToOk Flow_OkToEnd"/>
<q:coverage path="reject" flows="Flow_CancelToReject Flow_RejectToEnd"/>

The two business outcomes are declared as tracked routes on the outer process, named for what they mean (not left as the raw path-1/path-2 sutra coverage init would seed) — the curation step Coverage: declared routes as the compliance signal covers in full. A run is "covered" once its fired-flow trace contains a path's flows, in order, as a subsequence — so any of the three intake transports covers accept when it commits, and covers reject when it cancels. Both marks land in the coverage store datastores.yaml declares — pointed, for convenience, at the same database as the accounts ledger, and carrying no migration of its own because the engine owns the coverage schema. The coverage-report / coverage-reset admin channels read and clear this compliance metric; see that chapter for the CLI walkthrough (init/check/reset), the cross-process form a multi-participant collaboration needs instead, and how to curate a declared set that stays a signal instead of noise.

Try it

# seal it
cd rust && cargo build -p sutra-cli --release
target/release/sutra package ../examples/money-transfer/deployments-src/default--money-transfer--1.0.0 \
    --out /tmp/pkgs

# deploy it (see Your first deployment for the full API-deploy walkthrough)
target/release/sutra deploy /tmp/pkgs/default--money-transfer--1.0.0.sutra \
    --api --engine-url http://localhost:<port>

# send a transfer
curl -sS -X POST localhost:<port>/channels/transfer-request \
  -H 'Content-Type: application/json' -H 'X-Api-Key: transfer-demo-key' \
  -d '{"TransferRequest":{"FromAccount":"alice","ToAccount":"bob","Amount":"25.00"}}'

Run the full ordered scenario (durability, cross-instance reads, insufficient-funds and frozen-account rejection, atomicity rollback, isolation under concurrency, then the coverage report/reset pair) against real PostgreSQL:

cd rust && cargo test -p sutra-conformance -- --ignored tc_money_transfer_acid_ledger

Next

• Architecture — the layering underneath everything this example exercises.
• Reference: the sutra CLI — every command used above, in full.

Engine layering

Sutra is a single Cargo workspace under rust/. The crates split cleanly along the layers a message passes through, plus one composition root that assembles them into the thing you actually run.

The layers

sutra-dist            composition root — force-links concrete codecs/transports/resolvers,
                       produces the `sutra-engine` binary (see Dockerfile)
   │
sutra-engine           the engine LIBRARY — config, deploy/activation, admin API, OTel,
                       audit sinks, the outbox tick loop. Domain-neutral: it collects
                       codecs/transports/resolvers generically via their SPIs and names none.
   │
sutra-channels         protocol-neutral channel binding + dispatch: decode → two-tier
                       validate → route to a start event or a parked wait → execute
   │
sutra-executor         the token executor: gateways, sub-processes, data associations,
                       compensation, path-coverage tracking — synchronous and stateful paths
   │
sutra-bpmn             the BPMN 2.0 + q: extension model and loader
   │
sutra-feel / sutra-dmn / sutra-srl / sutra-templates
                       the expression + decision + template languages every task type runs on
   │
sutra-persistence      durable state: instances, outbox, inbox dedup, lease, audit — PostgreSQL
                       (MySQL/MariaDB/SQL Server dialects follow the same pattern)

A message arrives on a channel (sutra-channels), gets decoded and validated by a codec (sutra-codec-spi + the concrete codec crates), is routed to a process, and the process runs on the token executor (sutra-executor) over the BPMN model (sutra-bpmn), evaluating FEEL/DMN/.srl expressions (sutra-feel, sutra-dmn, sutra-srl) and rendering templates (sutra-templates) along the way. Anything durable — the instance snapshot, the outbox, the inbox dedup row, a data store write — goes through sutra-persistence (or a module's own store, for data stores; see Data stores).

The composition root

sutra-engine — the library — never names a concrete codec, transport, or secret-resolver implementation. sutra-dist is the one crate allowed to: it force-links the schema-less formats (json, xml, yaml, csv, raw-text, raw-bytes), the redactors, the vendor secret-resolvers, and the feature-selected transports, and produces the sutra-engine binary the container image ships (docker build -f rust/Dockerfile rust/). It force-links no domain codec: every message standard is a proprietary extension crate built outside this repository, registering through the same SPIs and force-linked by its own composition root, which is precisely what this split is for. This split is what lets a hardened build drop everything it doesn't need — cargo build -p sutra-engine --no-default-features --features file links no broker client at all — without touching a line of the neutral engine or channel code.

See Domain neutrality and the SPI model for exactly how that boundary is drawn and mechanically enforced, and what a third party has to write to add a new transport or codec.

Where the tooling sits

sutra-cli (the sutra binary) depends on the model/loader layer read-only for its inspection commands (describe, dispatch-graph, simulate, explain), and on sutra-persistence + sutra-channels for the commands that touch a running engine or a database behind it (deploy and migrate against the engine's own; coverage check --archive against the one a deployment's coverage store declares). It is not part of the engine's own runtime dependency graph — a deployed engine binary has no CLI code linked into it.

Next

• Domain neutrality and the SPI model
• Deployment model — how a package activates against this layering.

Domain neutrality and the SPI model

The engine core never names a business domain, a vendor, or a wire format. Not SWIFT, not FedNow, not ISO 20022, not Kafka, not PostgreSQL-as-a-brand. Payments, health, and EDI live entirely in codecs, transports, validators, and deployment packages — never in the token executor, the BPMN model, or the channel dispatcher. This is the single rule everything in this chapter exists to serve, and it is not a style guideline: it is checked by a build gate on every change.

Why this is load-bearing

A domain-neutral core is what lets one engine binary serve payments today and healthcare tomorrow without a fork, what lets a hardened build strip out everything it doesn't need down to the byte, and what keeps the surface a security review has to reason about small and stable. The moment a business term leaks into the executor or the model, that guarantee is gone for everyone, silently.

The gate: sutra-archtest

rust/crates/sutra-archtest is the Rust twin of what used to be an ArchUnit test on the retired reference baseline. It walks a fixed set of crates' src/ trees, strips comments (a domain term cited in a doc comment is fine — only a term baked into an identifier or a string literal is a violation), and fails the build on any hit against a denylist:

#![allow(unused)]
fn main() {
pub const DOMAIN_DENYLIST: &[&str] = &[
    "swift", "fednow", "fedwire", "pacs", "camt", "nacha", "edifact", "hl7", "iso20022", "x12",
];
}

Enforced — must be domain-literal-free:

• The neutral core: sutra-executor, sutra-bpmn, sutra-feel, sutra-dmn, sutra-srl, sutra-templates, sutra-persistence, sutra-datastore, and the codec/format/schema SPI itself (sutra-codec-spi).
• The assembly/binding layer: sutra-channels (protocol-neutral channel binding) and sutra-engine (the library) — domain-literal-free since the transport and codec extraction and the sutra-dist composition-root split.

Deliberately excluded — the legitimate domain edge, by design:

• sutra-dist — the composition root. It force-links the built-in formats, the redactors, the vendor secret-resolvers, and the feature-selected transports, so it necessarily names them. That's the one place wiring concretes to SPIs is supposed to happen — not a core leak.
• The concrete implementations themselves: sutra-formats, sutra-codec-schema, every sutra-transport-<vendor> crate, and sutra-redactor-pci.
• Tooling and test crates (sutra-cli, sutra-conformance, sutra-testkit, the generators).

Read that exclusion list again for what isn't on it. This repository contains no message-standard codec at all. The built-in codec set is the six schema-less formats in sutra-formats — json, xml, yaml, csv, raw-text, raw-bytes — and sutra-dist force-links nothing else that registers one. Every wire standard — SWIFT MT, SWIFT MX, rail variants such as FedNow, and the EDI/segment families (HL7 v2, X12, NACHA/ACH, EDIFACT) — is a proprietary extension crate maintained outside this repository, consuming exactly the public SPIs a third party consumes.

That is the strongest available evidence that the SPI boundary is real rather than aspirational: adding a message standard to a distribution costs one Cargo dependency and one use <crate> as _; line in that distribution's own composition root. Nothing in sutra-engine, sutra-channels, or this repository's composition root changes, and there is no central list anywhere to add a name to. The gate has nothing to exclude for those crates because they are not here; the neutral core cannot leak a term that never enters its dependency graph.

make lint runs this gate alongside clippy on every change (see Contributing) — a business term landing in a gated crate fails CI, not a code review comment.

The mechanism: self-registration behind a neutral SPI

Every extension point follows the same shape: a small SPI crate defines a trait and a process-wide registry; each concrete implementation is its own crate that self-registers into that registry via inventory at link time. The neutral crate collects what got linked in — it never imports or names a specific implementation. Implementing an extension is registering it; there is no separate central list to forget to update (a gap that bit an earlier iteration of the codec set, closed by moving to this pattern).

Transports — sutra-transport-spi

#![allow(unused)]
fn main() {
pub struct TransportFactory {
    pub transport: &'static str,          // the `transport:` value in channels.yaml this wires
    pub spawn: TransportSpawn,             // (defs, engine, pool, envref-resolver, runtime) -> Arc<dyn TransportChannels>
    pub register_sink: fn(&mut SinkRegistry),
    pub handles_on_complete: bool,         // self-declared capability — see below
}
inventory::collect!(TransportFactory);
}

Each sutra-transport-<vendor> crate submits one TransportFactory. The engine assembly iterates transport_factories() (sorted by name for determinism) and drives every one of them through the identical TransportChannels lifecycle trait (rewire on an activation flip, drain on shutdown) — there is no if transport == "kafka" anywhere in the engine.

Capability self-declaration, not a hardcoded transport check. handles_on_complete is how a transport tells the engine whether it can realize ack-mode: on-complete (a broker deferring its settle through the engine's ack registry, or an on-listener transport like HTTP holding the connection to completion). When a channel declares on-complete on a transport whose factory reports false, the engine assembly emits SUTRA.ACK.ON_COMPLETE_UNSUPPORTED and runs on-persist instead — a loud, generic degrade driven by a flag the transport itself set, not a list of vendor names the engine maintains. See Acknowledgement modes for the full per-transport wiring table this flag produces.

Codecs — sutra-codec-spi

#![allow(unused)]
fn main() {
pub trait PayloadCodec {
    fn name(&self) -> &str;
    fn accepted_content_types(&self) -> Vec<String>;
    fn decode(&self, body: &[u8], content_type: Option<&str>) -> DecodeResult;   // never panics
    fn declared_message_types(&self) -> Vec<String> { Vec::new() }
    fn shape_of(&self, message_type: Option<&str>) -> Option<SchemaShape> { None }
    // + encode (the reply direction)
}
}

A zero-config, globally-named codec — each built-in format (json, xml, yaml, csv, raw-text, raw-bytes), and any extension crate that wants a global name — submits a BuiltinCodec { name, make }; builtin_codecs() collects and sorts them, and each is addressable as urn:sutra:codec:<name> with no per-package configuration. A schema-backed user codec (compiled from a package's own schemas/<name>/*.xsd) does not self-register this way — it's instantiated per package, per the path-derived URN rule in Deployment packages. Either way, the engine only ever calls through the PayloadCodec trait; it has no branch for "this is the such-and-such standard one," and no way to acquire one — the type it would have to name is not in its dependency graph.

Bundle codec kinds — a codec crate registers its own schema-folder shape

The newest extension point in this family follows the identical inventory-pull shape, one level into how a package's own schema folder is interpreted. schemaKind: xsd and schemaKind: json-schema are generic — a folder of schema files the engine validates against — but some standards are a whole profile: an envelope grammar, a mapping from wire-level message names to schema files, versioned editions revved on the standard's own release cadence. A codec crate that needs that shape submits a BundleCodecKind { kind, build } in sutra-codec-schema::bundle next to its PayloadCodec impl — implementing the kind is registering it, exactly like a BuiltinCodec, and sutra-codec-schema itself stays free of any knowledge of which standards use it. A package's schemas/<name>/codec-manifest.yaml declaring schemaKind: <kind> is what selects a registered bundle over the generic ones; an unserved kind is a fail-closed deploy error naming the kinds the running build actually serves.

A market-infrastructure rail codec — an extension crate outside this repository — is the first user of this extension point (see Channels and transports and Deployment packages for the manifest shape and deployment-scoped registration it produces). It's also a useful case study for the SPI pattern more broadly: everything specific to one rail's profile — its envelope root names and namespaces, its per-direction wrapper tables, its quirks (a wrapper shipped without the profile's usual name prefix, several wrappers backed by one underlying message) — is a single data value the codec's internal walker/validator/projection code reads generically. A second rail that follows the same envelope-and-wrapper-editions pattern would supply its own profile value and reuse that machinery rather than re-implement it — the same "data, not a branch" discipline the rest of this chapter describes, one layer further in.

Data stores — sutra-datastore

The DataStore SPI is the same shape one level down: a provider (sql today; see Data stores) resolves its own connection from datastores.yaml — never the engine's internal datasource — and exposes get/put/get_for_update/ put_if_revision against (store_name, store_key). The engine's executor calls through this trait for every <q:store>-bound data association; it has no idea whether the value on the other side is an account balance or a customer record.

Secrets — sutra-envref-spi

The vendor-neutral seam behind env:, secret:, ${…} placeholders, and vendor schemes like vault:… / aws-secrets:…. A vendor resolver crate (sutra-envref-vault, sutra-envref-aws, …) submits an EnvRefResolverEntry; the engine resolves a whole reference generically and names no vendor SDK. This is what lets channels.yaml and datastores.yaml carry credentials as references, never literals — see Configuration reference.

Redactors — sutra-redactor-spi

A sutra-redactor-<standard> crate submits a RedactorEntry that locates sensitive spans in a decoded payload (a JSON-Pointer-shaped path + a reason code); the engine masks every located path on every observability surface and marks it for encryption at rest. Fail-closed by construction: a redactor that panics or errors tells the engine to over-mask the whole bound payload rather than risk a leak — the opposite failure mode from a validator crash, which becomes an ordinary business-reject issue instead.

Validators

<q:simpleValidator ref="…"> names a field-content validator (iso-3166-country, iso-4217-currency, iso-9362-bic, …) out of a neutral registry the core knows nothing about (see The q: namespace). The concrete validators are domain content and live in extension crates, outside this repository, for the same reason the domain codecs do — which is why the neutrality gate has nothing to exclude here.

The lifecycle bus — ExecutionListener

One more neutral seam worth knowing about even though it isn't a "plug in a new vendor" SPI: every cross-cutting concern that needs to react to instance/token/task lifecycle events — audit, OTel metrics, the deferred-ack registry — implements the same ExecutionListener trait and is fanned out from a plain Vec<Rc<dyn ExecutionListener>> the executor calls on on_instance_started / on_instance_completed / on_instance_suspended / etc. There's no dependency-injection container; listener registration happens explicitly at executor-construction time in the engine assembly. See Acknowledgement modes for DeferredAckRegistry as a concrete example, and Observability for the OTel listener.

Where business content actually lives

Given all of the above, a real deployment's domain-specific content lives in exactly two places:

1. Inside a deployment package — the BPMN processes, DMN/.srl rules, channels.yaml, datastores.yaml, and the package's own XSD-backed codec (see Deployment packages). This is where almost all business logic belongs, and it needs zero Rust code.
2. In an extension crate, only when the domain needs a new kind of codec, transport, validator, or redactor that doesn't already exist — implemented against the relevant SPI above and linked in by a composition root (sutra-dist, or your own if you build a custom binary).

Worked example: adding a transport

Say you need a new broker Sutra doesn't ship a transport for. The shape is fixed by the pattern above:

1. Create sutra-transport-<vendor>, depending on sutra-transport-spi and sutra-channels.
2. Implement TransportChannels (transport(), consumer_count(), rewire(active), drain(), stop_all_detached(), and the one optional capability inbound_router() — only HTTP-shaped transports return Some).
3. Implement the inbound spawn function with the uniform signature (&[ChannelDefinition], EngineHandle, Option<PgPool>, EnvRefResolver, tokio::runtime::Handle) -> Result<Arc<dyn TransportChannels>, Diagnostic>, and an outbound sink registrar fn(&mut SinkRegistry).
4. Decide whether your transport can realize ack-mode: on-complete (deferred settle, or holding the connection) and set handles_on_complete honestly — get this wrong and channels either silently under-deliver on a promise, or trigger the loud unsupported diagnostic needlessly.
5. inventory::submit! { TransportFactory { transport: "your-broker", spawn, register_sink, handles_on_complete } } next to your implementation.
6. Add your crate as an optional dependency behind a same-named Cargo feature in whatever composition root force-links it (sutra-dist, or a custom one) — nothing in sutra-engine or sutra-channels changes.

A new codec follows the identical shape against PayloadCodec instead, submitting a BuiltinCodec for a zero-config global codec, or shipping as a package-local schema-backed codec with no self-registration at all. Every message-standard codec in existence takes that first path — including the ones a downstream distribution ships — so it is a well-travelled route, not a theoretical extension point kept alive by a single in-tree user.

Next

• Deployment model — how a package's declared channels/codecs/stores get resolved against whatever is linked into the running binary.
• Contributing — the repo map, including where each SPI crate and its concrete implementations live.

Deployment model

A .sutra archive's runtime identity is a single opaque deploymentId = sha256(manifest) — the manifest is derived at seal time (sutra package), never hand-authored. Deploying is therefore naturally idempotent: re-deploying identical bytes is a no-op. This page is the architecture behind Your first deployment and Deploy, hot-deploy, and rollback — how activation actually works underneath the CLI.

The database is the source of truth

Sealed archives are stored in the engine's own datasource — the same core database that backs instance state, the outbox, and the lease table — as a deployment_archive row keyed by a stable slot (the archive's tenant--module--version key). Exactly one row per slot is ever active; deploying a new revision to an existing slot replaces the old active row in one transaction. This is what makes hot-deploy a replace, not a restart: the slot name is stable across versions, only the content-addressed deploymentId changes.

Loading is symmetric — on boot, the engine loads its active set from the database (WHERE status='active'), not by scanning a directory. A restarted or newly-scaled replica rehydrates its active set from the shared database, not from whatever a local volume happens to reflect.

Deploy is an API call

POST /admin/deployments is the one control path onto a running engine (auth-gated — see Configuration reference for the admin auth scheme). It:

1. Accepts the sealed archive's bytes.
2. Re-verifies them fail-closed (the same archive-integrity check sutra package already ran client-side).
3. Stores the new revision as the slot's active row, in a transaction.
4. Runs the two-phase activation flip in-process (drain the old revision if one exists, activate the new one).
5. Returns synchronously: 200 {deploymentId, phase: "Active"}, or a 4xx carrying the SUTRA.DEPLOY.* reject diagnostic.

Because the HTTP response is the activation signal, there is no propagation window to reason about, no separate "did it actually take?" step — this is the property that makes deploys deterministic rather than eventually-consistent from the caller's point of view.

DELETE /admin/deployments/{slot|id} marks a row draining; the engine's activation flip drains it (no new intake, retire once quiescent).

Sync vs. async — the same call, two response shapes

For a small or local deploy, the synchronous form above is the whole story: one request, one definitive answer. For a large deployment where the engine's own plan-and-flip work risks a long-held request — mainly a concern behind an ingress with a short read timeout — the identical endpoint accepts an async mode:

• Sync (default): POST /admin/deployments blocks until the flip completes, returning Active or a reject.
• Async (opt-in): the same POST returns immediately with 202 {deploymentId, status: "Pending"}; activation runs in the background. The caller learns the outcome one of two ways:
  • Poll GET /sutra/deployments/{id} — short, ingress-safe requests — until it flips to Active or Failed.
  • A completion event. The caller opts in with callback=<https-url> (a webhook) and/or notify=<broker-uri> at deploy time; on completion the engine emits a CloudEvent (com.sutra.deployment.activated / com.sutra.deployment.failed) carrying {deploymentId, slot, revision, status[, error]}. This rides the engine's existing outbound spine — one more durable outbox entry, delivered by the same dispatcher that sends every other outbound message — so a deploy-complete notification is not a separate mechanism, just another emission.

This is a long-running-operation (LRO) shape, not a bespoke deploy protocol: accept fast, do the work in the background, let the caller choose polling or a push notification.

Multi-replica convergence

The replica that handled the deploy call activates locally and returns synchronously; every other replica converges asynchronously on the committed database state — on PostgreSQL via LISTEN/ NOTIFY after the commit, with a version-poll fallback for dialects with no equivalent (MySQL/MariaDB/SQL Server) and as the resilience backstop on Postgres itself. A single-replica run needs no convergence step at all — the one replica that deployed is already the whole fleet.

Next

• Deploy, hot-deploy, and rollback — the operator-facing walkthrough of the same mechanism.
• Replica semantics — how the rest of the engine's durable state stays correct across a replica set, using the same PostgreSQL-backed primitives.

Multi-tenancy and isolation

Sutra runs many tenants behind one engine and one database, with isolation enforced at the storage layer rather than by giving each tenant its own infrastructure.

Tenant identity is a package label

A deployment package's tenant is an opaque label in package.yaml (alongside module and version — see Deployment packages):

labels:
  "module": "money-transfer"
  "tenant": "default"
  "version": "1.0.0"

The engine never interprets this string — it's a selector for observability, routing, and the row-level-security partition key described below. Because a package is fully self-contained (no shared resource tree, no inheritance between packages), a tenant's isolation boundary is simply "whatever package it was labeled into" — there's no separate tenant-configuration document a package's processes have to be matched against.

Storage isolation: row-level filtering + PostgreSQL RLS

Every durable table the engine owns — instance state, the outbox, the inbox dedup table, the alias index, the audit event log, the wait-state table, the dead-letter store — carries a tenant column and a PostgreSQL row-level-security policy keyed on a session variable, applied by the engine's own SQL migrations. Defense in depth, two layers:

1. Application-level filtering — every query goes through the persistence layer, which requires a tenant argument and injects the WHERE clause; there's no ad hoc database access path around it.
2. PostgreSQL RLS — SET LOCAL app.current_tenant = '…' at the start of each request transaction, with a CREATE POLICY … USING (tenant_id = current_setting('app.current_tenant')) on every table. Even a query that forgot its own WHERE clause cannot return another tenant's rows — the database itself is the second line of defense, not just the repository layer.

A startup check closes the obvious way to defeat this. RLS policies are silently bypassed for any role with BYPASSRLS, SUPERUSER, or ownership of the target tables. The engine probes its own connecting role at startup and refuses to start if either flag is set — an operator who pointed the engine at postgres or a role minted with BYPASSRLS finds out at boot, not after a cross-tenant leak. Production role setup:

CREATE ROLE sutra_app LOGIN PASSWORD '<from-secret>';
ALTER ROLE sutra_app NOBYPASSRLS;
GRANT SELECT, INSERT, UPDATE, DELETE ON <engine tables> TO sutra_app;

(Migrations themselves run as a separate, table-owning role that the engine runtime never uses.)

Quotas

Two per-tenant ceilings, enforced by the channel dispatcher before an inbound reaches the executor:

• maxConcurrentInstances — a hard cap on simultaneously in-flight instances for the tenant, checked against a live count in instance_state — globally coherent across every replica, since every replica queries the same table.
• maxInboundRatePerMinute — a sliding 60-second admission window. This one is per-replica, not fleet-wide: each replica tracks its own window in memory rather than paying a synchronous round-trip to the database on every inbound message. A tenant saturating one replica's window can still be admitted by a second replica until its window also fills — a deliberate latency/coherence trade-off. A hard global rate guarantee, if you need one, belongs in front of the engine (an API gateway or service-mesh L7 limiter), not in the dispatcher.

Both rejections surface as the same class of diagnostic the payload-size cap uses (see Limits and quotas), translated to the right per-transport signal (HTTP 429, a broker nack, …).

Audit isolation

Audit rows carry the tenant column and are covered by the same RLS policy as every other engine table. Where a JSONL audit sink is configured, it writes per-tenant directories, so a log shipper can fan out per-tenant index/retention policy downstream without the engine knowing anything about that policy itself.

Next

• Replica semantics — the mechanisms (leases, row locks, SKIP LOCKED) that keep all of this correct across a multi-replica engine.
• Limits and quotas — the operator-facing configuration surface for the ceilings above.

Replica semantics

Sutra runs as an active-active stateless replica set — all durable state lives in PostgreSQL, and no replica owns any particular instance, tenant, or channel by default. Any replica can pick up any token at any time, which is what lets the engine scale horizontally with no sticky routing.

What every replica does

Each replica is one engine process running inbound channel listeners (the HTTP server, broker consumers), the token executor for instances it picks up from persistence, an outbox worker sending replies it claims, and the health/metrics endpoints. Execution inside the process runs on one or more identical actor lanes — see Execution lanes; lanes are an in-process concern and change nothing on this page. Three pieces of work don't tolerate concurrency and are held by exactly one replica at a time instead: timer firing, stuck-instance scanning, and terminal-history purging.

Leader election: a PostgreSQL-backed lease

There is no Kubernetes-native Lease object and no separate coordination service — leader election is a lease row in the engine's own PostgreSQL database (DbLeaderElection over PgLeaseStore). One poll task per role tries to acquire the lease at a fixed cadence (ttl = 30s, poll = 10s); a successful acquire flips that replica to leader for that role, a contended one flips it to follower. There's no push notification — polling is the mechanism.

Roles are dynamic, not a fixed pair: the engine registers one lease per singleton channel role (channel_role(tenant, channel)) the first time it's needed, so declaring a channel singleton: true (see Channels and transports and the money-transfer example) starts contending a lease without any separate configuration. When no engine datasource is configured at all, every replica simply leads unconditionally (AlwaysLeading) — there's no third posture to reason about.

Instance ownership claims

Leases gate roles. What keeps two replicas from advancing the same instance at the same moment is a different, finer mechanism: a per-instance ownership claim, taken by a compare-and-set on the instance's own row.

Every resume path claims first. A correlated relay arriving on a channel, a timer firing, an administrative migration — each one takes the claim before it rehydrates anything, and a claim it cannot take is a refusal, never a wait. The refusal is retry-safe by design and carries a structured code (SUTRA.RUNTIME.RESUME.CLAIM_HELD, or the admin surface's SUTRA.ADMIN.MIGRATE.CLAIM_HELD): a broker relay is requeued for redelivery, a timer fire is deferred to a later tick, an admin call answers 409 having read and written nothing. Nothing is ever half-applied while contended, and nothing blocks waiting for a lock held on another replica's timescale.

The claim is released inside the same transaction that commits the step's own writes, so "committed" and "unowned again" are one atomic fact. Every exit that does not commit — an early refusal, an error, a panic — releases the claim explicitly on its way out, so a claim never outlives the work it was protecting.

The claim is re-entrant for the same owner, and the invariant that makes that safe is worth stating exactly: an owner id names one execution lane in one process, and a lane advances instances one at a time. Same owner therefore means same lane, which already means serialized — so re-claiming what you already hold is a heartbeat refresh rather than contention. Owner ids carry the lane index for exactly this reason, and the administrative migration path takes a deliberately distinct owner suffix so that a migration can never re-enter past a resume this same replica has in flight.

The stuck-instance sweep is the backstop for the case no protocol can prevent — a replica that dies mid-step and never releases. The instance-sweeper lease role scans on sutra.instance.sweep-interval (default PT1M) and clears any claim whose owner has been silent longer than sutra.instance.claim-timeout (default PT5M). It sweeps by age and is owner-blind, so it needs to know nothing about how owner ids are shaped.

Durable FAILED is a state, not a disappearance

An instance whose execution fails fatally is not deleted and does not silently linger as a mystery park. Its snapshot is re-stamped FAILED in place, carrying the structured failure code and the captured detail, and every waiting row it held is resolved in the same transaction — so no timer refires against it and no relay finds a live wait to satisfy.

FAILED is deliberately not terminal:

• it is always retained, regardless of the history-retention window, because it needs an operator rather than a clock;
• it keeps blocking its deployment's retirement (below), so a dead-but-unhandled instance cannot be quietly swept under a hot-deploy;
• it is the one state an instance can be migrated and resumed out of — repair the model, move the instance onto it, then explicitly bring it back.

Restoring it is exactly the inverse of the failure commit: the failure keys are dropped, the status goes back to suspended, and the parks the failure tore down are re-armed — after which the instance comes back through the ordinary claim-guarded paths. There is no privileged resume entry point.

Inbox dedup via row locks

An inbound message's (tenant, channel, event_id) triple is inserted with ON CONFLICT DO NOTHING; whichever replica's insert actually produced a row owns starting that instance, and any other replica that raced it (or a genuine redelivery) sees nothing came back and treats it as a dedup hit. This is a plain unique-index conflict — no application-level locking, and it works identically whichever replica happens to receive the redelivered copy.

Outbox processing: SKIP LOCKED fan-out

Every reply-in-waiting sits in an outbox table; each replica's outbox worker claims a batch with FOR UPDATE SKIP LOCKED, so replicas never contend on the same row and never double-send under normal operation. A replica that dies mid-send leaves its claim stale; another replica's next scan clears the stale claim and retries — broker-side dedup (the outbox row's key riding as the message's own idempotency token) absorbs the rare case where the original send actually went out before the crash.

Retiring a draining deployment: a three-legged gate

A hot-deploy leaves the previous graph draining rather than deleting it, precisely so instances pinned to it keep resuming on the definition they started under. It retires only when it is genuinely quiescent — and "quiescent" is three independent facts, all of which must hold:

1. No active instances pinned to it. Retained terminal history does not count (it would otherwise pin a deployment open for a whole retention window); FAILED instances do count, because they are live work awaiting an operator.
2. No pending outbox rows minted by it. An emission belongs to the channel bindings that produced it, so it drains where it was made. A delivery that exhausted an opt-in attempt ceiling is marked terminally poisoned and stops counting — "we gave up" is a durable, visible state, and it must not hold a deployment open forever.
3. No parked external tasks waiting on it. A task parked for a pull worker is outstanding work with no in-flight anything to observe; retiring underneath it would strand the worker's completion. Tasks that turned terminal after exhausting their worker budget stop counting, for the same reason poisoned deliveries do.

All three legs are database-scoped counts, so the gate reads the same from every replica and is unaffected by which replica happens to run the sweep.

Recovery on pod death

Because token state lives in the database, a dead replica leaves nothing to recover from memory:

• Mid-token-execution — the instance's claim goes stale with its owner; the stuck-instance sweep clears it and the next replica to see ready work picks the instance up.
• Mid inbox-to-instance handoff — an inbox row without a matching instance past a recovery threshold gets retried; the same unique-index dedup makes the retry safe.
• Mid outbox send — the stale claimed_at is cleared and the row is retried by whichever replica's scan finds it next.
• The leader itself dies — its lease expires within its TTL and another replica's next poll acquires it; timer scheduling pauses for at most that window, while outbox processing continues uninterrupted on every other replica throughout.

Scaling signal

CPU is not a meaningful autoscaling signal for an event-driven engine — the right one is backlog depth (inbox rows waiting to start an instance, outbox rows waiting to send), which is what a KEDA ScaledObject querying the engine's own tables drives off. See the OpenTofu module under deploy/modules/sutra/ for the shipped shape.

Non-goals

No active-passive mode (every replica processes real work; the lease only gates the singleton pieces), no sticky session routing, no tenant-to-replica affinity (isolation is a PostgreSQL RLS concern — see Multi-tenancy and isolation — not a placement one), and no engine-managed broker resources (queues/topics are provisioned by your own infrastructure tooling; the engine connects to what already exists).

Next

• Execution lanes — the in-process half: N actor lanes under one replica, and why claims rather than routing keep them correct.
• Multi-tenancy and isolation — the other half of "many tenants, one engine, one database."
• Deployment model — how the same PostgreSQL-backed convergence pattern (LISTEN/NOTIFY, version polling) applies to deploy activation across a fleet.
• Ownership and claims: the design reasoning — the compare-and-set, the re-entrancy invariant, and the owner-suffix conventions.

Execution lanes

Inside one replica the engine executes on N identical actor lanes. Every piece of instance-addressed work is routed onto a lane by a stable hash of the instance id, and one lane drains one request at a time — so all work for a single instance runs in arrival order, on one lane, with nothing else interleaved into it. That property is the contract, and it holds identically at every N.

Lanes are an in-process scale-out story. They sit underneath — and are entirely independent of — the horizontal one: a replica has N lanes; a deployment has M replicas; the two multiply. See Replica semantics for the cross-replica half.

Why lanes exist

A single serial execution lane is a convoy. One slow commit does not merely delay its own instance — it delays every instance behind it on the same lane, however unrelated. Splitting the lane into N lanes removes the convoy without changing what happens within a lane: the ordering guarantees a single lane gave are the guarantees each of the N lanes still gives.

Nothing else about execution changes. Lanes do not partition data, do not shard the database, and do not introduce a placement or affinity model an operator has to reason about. They are a parallelism mechanism inside a process, nothing more.

Routing: a stable hash of the instance id

The routing key is the instance id. It is the unit of durable state (one snapshot row), the unit of mutual exclusion (one ownership claim), and the unit the guarantee names. Nothing else works as a key: a correlation alias is not stable per instance (one instance can carry several alias rows with different values), a deployment or a tenant is far too coarse to spread load, and a channel splits one instance across lanes the moment its relays arrive on a channel other than its spawn's.

Work arrives in three shapes, and the router handles each differently:

The cross-lane handoff

A relay does not name an instance id on the wire. It names a business key, and the engine only learns the instance after channel resolution, decoding, intake validation, and evaluating the <q:alias> correlation expression. All of that runs on the lane the delivery arrived on, exactly as it always has.

When that resolution names an instance owned by a different lane, the arrival lane does not execute it and does not push it into the other lane's queue. It answers its caller with a handoff: the already-decoded, already-validated resume request. The caller's own task — the HTTP handler, the broker consumer, the outbox worker — then enqueues it on the owner lane and waits for that lane's answer. Two rules make this safe by construction:

• Lane loops never send into another lane's queue. Only caller-side tasks do. The mutual-block deadlock (lane A stuck sending into a full queue on B while B is stuck sending into A's) therefore cannot arise at all.
• At most one hop. The lane that receives the resolved request runs it where it lands. It re-runs only the race-sensitive part — claim, load, terminal/failed/suspended guards, deployment pin resolution, resume — never the decode, validation, or correlation, which are deterministic over the delivery and already done.

Only relays ever hop. Spawns and timer fires never do.

Claims are the correctness mechanism; routing is only affinity

This is the property worth internalizing: routing is an optimization, not the thing that keeps execution correct. Correctness comes from the per-instance ownership claim described in Replica semantics — and the claim's owner identity is lane-scoped, not just process-scoped.

The consequence is that a mis-route is harmless. If work for an instance ever lands on the wrong lane — a bug, a hash change, anything — the claim bounces it exactly as it bounces a competing replica today: a broker relay is requeued, a timer fire is deferred and retried. A mis-route degrades to visible, retry-safe contention. It can never degrade to two lanes interleaving inside one instance.

That is also why the claim-bounce meter below doubles as the mis-route alarm.

The activation flip

A deploy activation rebuilds the engine's live view of processes, codecs, validators, and channels. Under lanes, the controller sends that rebuild to every lane and waits for all of them before it replaces the live deployment set and rewires transports.

Per-lane atomicity is what matters, and it is preserved: a lane applies its flip between two requests, never inside one, so nothing is ever observed half-flipped. Because every step of an instance runs on that instance's one lane, its flip is a single point in its own queue — no instance can straddle two definitions. During the fan-out window two different instances can be served either side of the flip, which is indistinguishable from two deliveries ordered around a single flip point.

Lanes never block on store I/O

Each lane is one asynchronous loop awaiting one request to completion before it dequeues the next. Every persistence call on the execution path is awaited rather than blocked on, so a lane waiting on the database parks on the runtime instead of holding a thread hostage — which is what keeps tail behavior sane when lanes outnumber available pool connections.

The ordering properties are unchanged by this, deliberately: because the loop awaits each request to completion before recv-ing the next, the commit still happens-before the reply and happens-before the next request is dequeued. There is no intra-lane pipelining and no completion re-entry, so there are no re-entry rules to get wrong.

What changes at N > 1 — say it out loud

Incidental cross-instance serialization disappears. With one lane, two concurrent deliveries to two different instances of the same flow never interleave — as a side effect of there being one lane, not as a promise. At N > 1 they genuinely run in parallel.

This was never the contract, and every documented concurrency mechanism is unaffected:

• per-channel singleton / serial consumption (a transport-side property — one delivery in flight — which lanes do not touch);
• per-channel and per-tenant admission caps (see Limits and quotas);
• optimistic expect="unchanged" writes and pessimistic forUpdate locks on data stores (see Data stores).

But a deployment that has been silently leaning on the single-lane side effect will observe new interleavings. That is precisely why the default is one lane and turning it up is an explicit operator action.

Two other narrowings are worth naming, and both only affect a persistence-less engine (no datasource configured — a dev/test posture): in-process alias uniqueness and in-process inbox dedup become per-lane rather than per-process. Under any pooled production posture both are database-backed and cross-lane safe, exactly as they are already cross-replica safe.

What lanes do not change

Every background role stays exactly one per replica, or one per cluster by lease — none of them becomes lane-aware beyond dispatching into the router:

Configuring and observing lanes

sutra.engine.shards sets the lane count (default 1); sutra.engine.shard-queue-capacity optionally bounds each lane's mailbox, in which case a full queue makes the caller wait, so backpressure propagates outward to the transport — never sideways into another lane. Full key detail: Configuration reference.

Meters shipped with the feature, all carrying the lane index as a dimension:

The live lane count is readable without reading configuration: GET /sutra/health/ready reports it in the loader check's data.shards, read off the running router rather than echoed back from config — which is what lets a smoke test assert that a container actually came up with the lane count you meant.

Lane death is a health condition, not just a log line. A lane can only die outside the per-dispatch panic containment (a failure in the lane's own build), and after that every piece of work hashed to it would answer SUTRA.RUNTIME.UNEXPECTED forever while the process otherwise looks healthy. Both probes therefore watch for it: GET /sutra/health/live returns 503 with the dead lane indexes in data.deadLanes — the signal an orchestrator should restart on, because a dead lane's key space has no other home inside the process — and GET /sutra/health/ready goes DOWN at the same moment so no new traffic routes to the replica while the restart is pending.

Next

• Replica semantics — ownership claims, leader-gated singletons, and the cross-replica half of the same picture.
• Configuration reference — the two keys.
• Lanes: the design reasoning — why the ordering properties survive, what the serialization audit found, and the runtime shape that had to be falsified first.

Observability

Sutra exports all three signals — traces, metrics, and logs — through OpenTelemetry, and is fail-open about it: telemetry can never affect message processing. No endpoint configured means no exporters and zero overhead; a bad config value falls back to a default with a warning; an exporter failure only logs, it never propagates into intake or dispatch.

No telemetry, no phone-home

Sutra collects nothing on its own behalf. The engine binary, the sutra CLI, and the container image sutra-dist produces send no usage statistics, no error or crash reports, and no telemetry of any kind to any Sutra-affiliated destination — there isn't one, and no code path sends anywhere by default. Nothing phones home, in any build, ever.

The only way data leaves a running engine to a system outside it is the operator-configured OpenTelemetry export this chapter describes, and export is opt-in, not opt-out (see Configuration reference). With no OTLP endpoint configured, the engine emits structured JSON logs to stdout and nothing else — the boot log says so in plain text (telemetry export off (no OTLP endpoint configured) — JSON stdout logs only), and that posture is enforced in the engine's own telemetry bootstrap (rust/crates/sutra-engine/src/otel.rs), not just asserted here. Point sutra.telemetry.otlp.endpoint (or the standard OTEL_EXPORTER_OTLP_ENDPOINT) at a collector you run and the signals below start exporting there; leave it unset and nothing crosses the process boundary except whatever you separately configured elsewhere (a channel, a datastore, an audit sink) — this is a property you can verify by reading the source or watching an unconfigured engine's own boot log, not a policy statement to take on faith.

Traces

The engine's existing tracing spans (sutra.dispatch, sutra.resolve, sutra.decode, sutra.validate, sutra.execute, sutra.outbox.send) export through a tracing-opentelemetry layer with no call-site changes — the same spans that back local RUST_LOG debugging (see Troubleshooting BPMN solutions) are what leaves the process as OTLP.

A suspended instance does not leak an open span. Because a wait state can hold an instance for an arbitrary length of time (see Wait states and human tasks), the executor fires a listener event at suspend that force-ends every span open for that instance — otherwise a long park would show up as a pathologically long trace. Each segment of a stateful flow's lifecycle (the initial run, then each resume) gets its own trace with its own traceId; there is no trace-of-traces joining them, since OpenTelemetry traces are flat. What ties the segments together for a human reading a Gantt view is the instance id, stamped as a plain span attribute (bpm.instance.id) on every span belonging to that instance — filter on it and the segments line up on one timeline, with the waits showing as the gaps between them.

Metrics

An ExecutionListener (the same lifecycle bus described in Domain neutrality and the SPI model) maps instance/token/task events onto a fixed set of meter names (sutra.instance.*, sutra.token.*, sutra.task.*, sutra.coverage.path_covered — see Coverage: declared routes as the compliance signal for what that one actually tracks), tagged with the deployment id and a configurable label allowlist. Alongside those, the sutra.engine.shard.* family reports per-execution-lane queue depth, work rates, cross-lane handoffs, and claim bounces — see Execution lanes. Delta vs. cumulative temporality follows the standard OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE variable — set it to delta for an Elasticsearch-backed collector, which drops cumulative histograms.

Logs

Structured JSON on stdout, always, with no configuration required — this is the log path every deployment gets whether or not OTLP is configured. When an OTLP endpoint is configured, the same log records additionally export over OTLP. The field shape (timestamp/level/loggerName/ message/service.name, plus traceId/spanId inside a sampled span) is stable and is what any log-processing pipeline should key off. See Logging and audit for the operator-facing configuration.

Cardinality discipline

Tenant id is deliberately not a tag on high-cardinality metrics (per-task duration histograms, for instance) — only on instance-counting metrics, where the cardinality stays bounded by tenant count rather than by tenant × task-name × outcome. This is the same discipline that keeps the label allowlist above short by default.

The reference stack

The repo ships a reference EFK-family stack as a set of OpenTofu modules (deploy/modules/efk-stack) — an OTel Collector, Elasticsearch, Kibana, and a Fluent Bit DaemonSet for host-level logs — deployed alongside the engine module in a dev/small-prod cluster, or pointed at an external, already-owned observability stack in a larger one. It's a reference, not a requirement: the engine's only actual contract is the three endpoint inputs (OTLP, an optional log-forward target, an optional direct Elasticsearch endpoint for the audit fan-out) — any OTLP-speaking collector on the other end works.

Next

• Logging and audit — configuring the endpoints above, and the audit trail as a separate, compliance-oriented sink from telemetry.
• Troubleshooting BPMN solutions — using traces and audit together to retrace what one message actually did.

Durable execution: snapshots and typed values

This chapter is the why behind what a parked instance actually is. If you only need the observable behaviour, Wait states and human tasks has it; this is the reasoning underneath.

A snapshot at a quiescent point, not an event log

Sutra persists an instance as a snapshot taken at a quiescent point — the moment execution has nothing left to do until the outside world answers. It does not persist a log of events and it does not reconstruct state by replaying one.

The two models differ in what they make cheap and what they make hazardous.

An event-sourced engine rebuilds state by re-running history. That gives it a free audit trail and free time-travel, and it buys them with a permanent constraint: the code that produced the history must keep being able to reproduce it. Every task function, every expression, every library the flow ever touched becomes part of the replay contract, forever, for every instance still alive. Determinism stops being a property of one execution and becomes a property of every version of the engine and the application, jointly, across time.

A snapshot engine stores what the instance is rather than how it got there. The replay contract shrinks to a single step: from this state, given this input, do the next thing. Nothing older than the current snapshot is ever re-executed, so an upgrade cannot retroactively change what already happened.

The audit trail an event-sourced engine gets for free is then an explicit, separately-configured concern here — the audit journal — and that turns out to be the honest arrangement anyway: an audit record has different retention, different access control, and different redaction requirements from execution state. Conflating them means one policy has to serve both.

The properties that fall out of the choice:

• A step is one transaction. The snapshot write, the waiting rows, the correlation aliases, and the outbound emissions all commit together, or none of them do. There is no window in which an instance has parked but not registered its correlation, or emitted but not recorded that it did.
• Encoding is deterministic. Identical logical state produces identical bytes: the container is sorted, and every value has exactly one canonical rendering. Re-encoding a decoded snapshot is byte-identical — which is what makes migration and rollback verifiable rather than hopeful.
• The frontier is explicit. A snapshot names the nodes it is waiting at, the nodes it has completed, and the start node it was routed through. Resume is "replay the completed set as done, satisfy this wait, continue" — a bounded operation over the recorded frontier, not a re-execution of the past.

The typed value encoding

Instance variables ride inside the snapshot container. How they are encoded is the part with the most design in it.

The defect it fixes

A parked instance used to flatten every variable to its display string. A number became "1250.75". A boolean became "false". A list became "[1, 2]". A null became the empty string. Resume restored all of them as strings.

That is not cosmetic. FEEL — correctly — does not coerce a string in arithmetic or comparison, so an exclusive gateway re-evaluating amount > 100 after a wait compared a string to a number and got null, which a gateway condition reads as false. A transaction that plainly exceeded its limit took the under-limit branch. not(approved) was worse: "false" is a non-empty string, so a restored boolean was never false again. The same wait state that made the engine durable made its decisions wrong.

The loss happened in exactly two places — the park ran every value through the display formatter, and resume wrapped every restored value back into a string — and both are gone.

The encoding

A typed value is <tag>|<payload>: one ASCII tag byte, a separator, then the payload. Only the first separator is structural, so a string whose own text is n|42 round-trips unambiguously as s|n|42.

The separator is chosen to be a character the container's own escaping never touches, so a tagged scalar costs exactly two bytes and the row stays legible to an operator reading it:

sutra.var.amount=n|1250.75
sutra.var.approved=b|false
sutra.var.cancelledAt=z|
sutra.var.inboundId=s|INB-7
sutra.var.lines=j|[1,{"sku":"A-1"}]

Lists and contexts ride JSON. The four temporal types have no JSON counterpart, so they ride a single-key object ({"@d":…} and siblings), and a user context key that would collide — any key starting @ — is escaped by doubling its leading @. The two are never confusable in either direction.

Why the generation integer says 4, not 3

The snapshot format carries a generation integer, and typing takes 4.

Generation 3 was already spent. It is the generation at-rest encryption introduced, where the integer means this snapshot carries ciphertext. Redefining 3 to mean typed would misread every encrypted row already persisted — and distinguishing the two by probing values for something that looks like a tag is exactly the kind of heuristic a persisted format must never depend on.

So the conflation stops at 3. Encryption has always been detected structurally, off the ciphertext key prefix — the read path never consults the generation integer for it at all — which means 4 means one thing only: the value encoding. An encrypted snapshot with typed values is simply 4. There is no generation for "typed and encrypted", and there never needs to be.

Compatibility: lenient decode, lowest-generation encode

Decode is version detection, not migration. An older snapshot is not tag-decoded at all: every value becomes a string, byte for byte what it was — including a legacy value that happens to look like a tag. An old row read and re-written untouched reproduces its original bytes. There is no silent upgrade on load.

Encode emits the lowest generation that can carry the state: 4 when at least one variable actually needs the typed form, else 3 when something is encrypted, else 2. An instance whose variables are all strings therefore still writes the exact bytes it wrote before typing existed.

That asymmetry is doing real work. It keeps a byte-for-byte golden corpus valid across the feature, and — more importantly — it lets a fleet mid-upgrade write rows an older replica can still read. A rolling upgrade does not need every replica to reach the new version before anything is safe.

The decode path is also deliberately lenient in one further respect: a malformed engine-internal counter reads as zero rather than as an error. A snapshot must never become unloadable. An instance that cannot be read is an instance no operator can inspect, migrate, or terminate — a strictness that turns a small corruption into an unrecoverable one is the wrong trade.

The byte-level key patchers

Three operations rewrite a stored snapshot without decoding it: marking an instance failed, marking it terminal, and re-pinning it during migration. Each patches the raw key/value map in place — engine-internal keys only — and never touches a variable value.

This is deliberate, and the reason is a security one rather than a performance one. Decoding and re-encoding a variable would need the tenant's data-encryption key. That means:

• an operation as mundane as "mark this instance dead" would acquire a dependency on key availability, and would fail — or worse, partially succeed — when the key backend is down;
• it would have to re-derive which variables are sensitive, from a snapshot that at resume time no longer carries that set in the shape the encryptor wanted;
• and if either went wrong, it would persist a previously-encrypted value in the clear.

Marking an instance dead, finished, or re-pinned must never need the tenant key and must never be able to downgrade an at-rest value to plaintext. Treating every variable as an opaque string is how that is guaranteed structurally rather than promised procedurally.

Typing does not weaken the property, by construction: to the patchers a value is still an opaque string. A snapshot carrying a tag this codec does not recognise rides a patch through untouched rather than being "repaired" — a forward-compatibility posture the tests pin explicitly.

Encryption: the tag rides inside the ciphertext

At the typed generation, the plaintext that gets encrypted is the tagged form. A sensitive number is a number again after the resume, exactly like a non-sensitive one.

The alternative — an outer tag sitting beside the ciphertext — was rejected because it would disclose the type of every encrypted value to anyone who could read the row, for no benefit at all. "This encrypted field is a number, and that one is a list" is information, and it is information the row does not need to carry.

Everything else about the envelope is unchanged. The authenticated-data binding still ties a ciphertext to its key, its instance, and its variable name, and still deliberately excludes the deployment id — which is precisely what keeps "a migration changes only the pin" true, and what lets a migrated instance's encrypted values still decrypt (see Migration internals). Decode still fails closed on a missing cipher or a failed authentication.

What is deliberately not typed

• A subject blind-index input. A subject value feeds a keyed hash over index rows that are already persisted. It keeps hashing the exact string it hashed before typing existed; deriving it from the typed value would orphan every index entry written to date — and those rows are how erasure and disclosure find an instance at all.
• The operator inspect projection. A published contract: every variable still renders through its display form. Typing changed what survives a wait, not what a response looks like.
• Functions and ranges. A FEEL function closes over an evaluation context that ceases to exist the moment the instance parks; a range is a comparison shape rather than instance state. Both persist as their canonical string — which is what they always did, so nothing regressed. They simply did not become typed.

The one behaviour change a consumer can observe

Every restored value now evaluates as its real type. Where a gateway condition's answer differs from a pre-typing release, the previous answer was wrong — that is the entire point of the change. The only other visible difference is that a null variable now restores as null (and renders as null rather than blank) instead of restoring as an empty string.

The snapshot key registry

Everything a stored snapshot's properties map can carry, in one place — useful when reading a raw row. sutra.-prefixed keys are engine-internal: opaque strings to the byte-level key patchers and never tag-decoded. sutra.var.<name> entries are the user variables the typed encoding exists for.

Next

• Ownership and claims — what guarantees only one worker is ever writing one instance's snapshot.
• Retry machinery — attempt counters as snapshot keys, and why that choice costs zero migration.
• Migration internals — re-pinning a snapshot without decoding it.

Ownership and claims

Two things must never happen to one instance: two workers advancing it concurrently, and a worker holding it forever. The ownership claim is the mechanism for both. The observable behaviour is in Replica semantics; this is why it is shaped the way it is.

The claim is a compare-and-set, and a failure is a refusal

Every resume path — a correlated relay, a timer fire, an administrative migration — takes the claim before it rehydrates anything, by a conditional update on the instance's own row: take ownership if it is free or stale, otherwise match nothing.

The critical design choice is what happens when it matches nothing. It refuses. It does not wait, retry in place, or queue behind the holder.

Waiting would be wrong on three counts. It would make the runtime of a request depend on another replica's work, on that work's timescale, with no bound anybody can reason about. It would hold a transport resource — an in-flight HTTP request, a broker prefetch slot — for the duration. And it would make a contended path a fundamentally different code path from an uncontended one, which is exactly the kind of asymmetry that hides bugs until the day of the incident.

Refusing instead composes with mechanisms that already exist. A broker relay is requeued and redelivered; a timer fire is deferred to a later tick with backoff; an administrative call answers 409 having read and written nothing. The refusal carries a structured code (SUTRA.RUNTIME.RESUME.CLAIM_HELD, or the admin surface's SUTRA.ADMIN.MIGRATE.CLAIM_HELD) so a caller can tell "you lost a race, try again" apart from "this will never work".

Release is part of the commit, and part of every other exit

The claim is released inside the same transaction that commits the step's writes. "Committed" and "unowned again" are therefore one atomic fact, and there is no window in which an instance is durably advanced but still shows an owner — a window a sweeper would eventually have to clean up, and that a concurrent resume would misread as contention in the meantime.

Every path that does not commit has to release too: an early refusal, a validation error, an unexpected failure, a panic. Originally that was a scope-guard — release on drop, automatically, whatever the exit. When lane loops became asynchronous, the guard had to change shape, because a destructor cannot await and releasing is a database call.

So the guard became an explicit release on every exit, including the panic path, which is re-raised identically afterwards. The mechanism moved; the invariant did not. It is worth being honest that this trades a compiler-enforced guarantee for a reviewed one — the mitigation is that the exits are few, enumerated, and directly tested, and that the sweep below is the backstop for the case where the process is not around to run any exit at all.

Re-entrancy, and the invariant that makes it safe

The compare-and-set deliberately grants a re-claim to the same owner. That looks like a hole until you see the invariant it rests on.

Originally the reasoning was: an owner id names one process, and one process advances instances on a single execution thread — so the same owner cannot possibly be in two places at once. Re-claiming what you already hold is a heartbeat refresh, not contention.

Introducing N execution lanes inside one process invalidated that reasoning exactly. Under a per-process owner id, lane B claiming an instance that lane A is mid-step on would succeed re-entrantly: a silent double-resume, which is precisely the corruption the claim exists to prevent. All-green tests at one lane would have proved nothing, and the cross-replica tests use different owners, so they could never have caught it.

The fix is to make the owner id lane-scoped, which rotates the invariant into its honest form:

same owner ⇒ same lane ⇒ already serialised

The words changed and the guarantee did not. That is the shape to want from an invariant when the system underneath it moves.

Routing is affinity; claims are correctness

This is the property the whole lane design leans on, and it is worth stating as its own claim.

Instance-id hash routing exists to give a lane affinity for the instances it owns — fewer cross-lane hops, better cache behaviour, less queue churn. It is not what makes execution correct.

If work for instance X ever lands on the wrong lane — a routing bug, a hash change, a handoff rule misapplied — the claim bounces it exactly as it bounces a competing replica: CLAIM_HELD, requeue for a relay, defer for a timer fire. Mis-routing degrades to visible, retry-safe contention. It can never degrade to interleaved execution.

That is a much better failure mode than the alternative, where routing is the mutual exclusion and a routing bug is a silent data-corruption bug. It also gives the rollout a direct observable: the claim-bounce meter, split by relay and timer, should read near zero at a healthy multi-lane rollout, so a mis-route shows up as a number going up rather than as a support ticket six weeks later.

Owner-id suffixes, and why they cannot collide

An owner id is an opaque string to the store and to the sweeper — neither parses it — which is what lets the conventions layer without a schema change:

The migration case is the subtle one, and it is the direct consequence of re-entrancy. Claiming under the bare replica identity would succeed against a resume in flight on that very replica — the one race the claim exists to prevent, reintroduced by an operator action. Under a distinct owner the compare-and-set fails honestly, and a resume that starts after the migration claims bounces off it in turn.

The two conventions cannot collide because they occupy different, non-overlapping parts of the identity: a lane suffix is a lane index on the execution path, and the migration marker is not a lane index at all and is never produced by a lane. There is no string a lane can generate that an administrative claim can also generate.

Heartbeats: deliberately not wired

A claim could refresh itself mid-step. It doesn't, and that is a decision rather than an omission.

A step runs between quiescent points — milliseconds to low seconds — which is orders of magnitude inside the default claim timeout. Adding more execution lanes shortens the queue wait before a step; it does not lengthen the step itself. So the mechanism would fire constantly and protect against nothing.

The condition that would change the answer is a genuinely long-running step. The pull-worker surface is the shape that could produce one — but it deliberately does not: a task parked for a worker is not a step in progress. The instance parks, its claim is released with the commit, and the worker's completion re-enters as a fresh delivery later. The long wait lives in a task row's lock, not in an instance claim, which is exactly why no heartbeat is needed. See The pull surface.

The sweep is the only backstop that can exist

No release protocol survives a process that stops existing. The stuck-instance sweep is the answer: a lease-gated role that clears any claim whose owner has been silent longer than the claim timeout.

It sweeps by age and is owner-blind — it does not parse owner ids, does not know how many lanes a replica has, and does not care. That is what let the lane-index suffix land with no change to the sweep at all: owner cardinality grew, and the sweep predicate never mentioned owners in the first place.

The tuning trade is plain in the two keys. A short claim timeout recovers faster from a dead replica and risks stealing an instance from a live-but-slow one — which the claim's own compare-and-set then makes safe rather than catastrophic, since the original owner's commit no longer matches. A long one is the conservative direction. The defaults sit far enough above normal step duration that the first case does not arise in practice.

Next

• Execution lanes: the design — what the lane-scoped owner is protecting.
• Migration internals — the distinct-owner claim in the operation that needs it most.
• Replica semantics — the operator-facing view.

Execution lanes: the design

Execution lanes describes what lanes do. This chapter is the reasoning: what the single-lane design was actually buying, how each of those properties was preserved rather than re-argued, the one genuine cross-lane race that had to be fixed, and the runtime shape that had to be falsified before the right one was found.

What the single lane was buying

Before lanes, the engine was one actor thread draining one queue, with every store call blocked on rather than awaited. That is easy to dismiss as an accident of an early implementation. It wasn't — four properties rode on it, and each had to be accounted for.

Because a dispatch was synchronous end to end, the step's commit happened-before the reply to the caller, and happened-before the next request was dequeued. From that:

1. Reply implies committed. A transport that got an answer knows the park is durable. This is what makes an acknowledgement mode meaningful at all.
2. At most one store transaction in flight per process. Connection-pool behaviour was trivially bounded.
3. A park can never race its own completion. The deferred-acknowledgement registration that follows a commit was serialised against every terminal event, because nothing else could run in between.
4. The activation flip is atomic with respect to dispatches. Nothing is ever observed half-flipped.

The honest observation — the one that set the whole sequencing — is that blocking on a dedicated thread is not itself the scaling defect. The defect is having one serial lane, where a single slow commit convoys every unrelated instance behind it. N lanes fix the convoy while changing nothing else. Removing the blocking is a separate change that buys different things and should be judged on its own.

So the work split: first N lanes with the blocking intact, then the conversion to awaiting, with the ordering properties preserved verbatim at each step. Each half is reviewable on its own terms, and a regression can be attributed to one of them rather than to "the concurrency change".

The total-serialization audit

The interesting engineering was not writing a router. It was enumerating everything that implicitly relied on there being one thread — because every such reliance was invisible, uncommented, and correct right up until it wasn't.

The method was to walk each site and classify it into exactly one of four buckets:

Most sites landed safe, and for one recurring reason: per-instance state has a durable home, and a step runs wholly on one lane. In-memory audit sequence counters, for instance, are re-seeded from the snapshot before every resume, so a stale entry on a lane that last saw an instance is simply overwritten wherever the next resume lands.

The accepted narrowings were the ones worth writing down honestly, because they are real degradations rather than non-issues. In-process correlation-alias uniqueness and in-process inbox deduplication become per-lane instead of per-process. Both are only reachable in a persistence-less posture (a dev or test engine with no datasource). Under any pooled production posture both are database-backed — a unique index and a row lock — and already cross-replica safe, which strictly implies cross-lane safe. Documenting the narrowing is the price of not pretending it doesn't exist.

One further site is a deliberate removal rather than a narrowing: incidental cross-instance serialization. Two deliveries to two different instances of one flow never interleaved, as a side effect of there being one lane. That was never a contract, but a deployment could have been leaning on it. It is the reason the default lane count is one, and the reason the configuration page says the sentence out loud rather than burying it.

The one genuine race, and its inversion

Exactly one site was a real cross-lane race, and it is worth walking through because the fix is smaller and stranger than the problem.

The deferred-acknowledgement registry lets a transport hold an inbound acknowledgement until the instance actually completes. The original order was: commit the park, then register the settle callback. Safe under one lane, because nothing could run between the two.

Under lanes it breaks. The park runs on the lane the delivery arrived on — call it A. The instance's first relay routes to the instance's owner lane — call it B. B can claim, resume, and complete the instance in the window between A's commit and A's registration. The terminal event then fires with no registration present, the acknowledgement never fires, and the delivery dangles until the registry's timeout sweep negatively acknowledges it. Microseconds wide, invisible under load, catastrophic when it hits.

The fix is to invert the order: register before committing, and deregister if the commit fails.

The window closes cleanly, and the argument is the load-bearing part:

Before the park commits, no correlation alias row exists — alias rows ride the step transaction. So no relay anywhere can correlate to this instance, and therefore no terminal event for it can fire from anywhere except this same dispatch, which is the one blocked on the commit.

A failed commit deregisters, and the caller sees the same error it always did. The registry's documented invariant rotates from "registered ⇒ the transport was told to defer" to "the transport was told to defer ⇒ registered ∧ committed" — which is the direction the transport actually depends on, and arguably what it should have said all along.

At one lane the inversion is unobservable, which is why it landed unconditionally rather than as a behind-a-flag branch. A correctness fix that only applies in one configuration is a correctness fix you have to reason about twice.

The lesson generalizes: the ordering was safe because of a property (total order) that was never written down as a requirement. Auditing for implicit reliance is the only way those surface before production does it for you.

Handoff rules

A relay does not carry an instance id. Resolving one needs the decode, the intake validation, and the correlation expression — all of which need the lane-resident registries. So resolution happens where the delivery arrived, and the resolved request is then handed to the owner lane. Two rules constrain it, and both are structural rather than advisory.

Lane loops never send into another lane's queue. Only caller-side tasks do. This is what removes the inter-lane deadlock case by construction: if lane loops could enqueue into each other, lane A blocked sending into B's full queue while B is blocked sending into A's is a genuine deadlock, and no amount of capacity tuning eliminates it — it just makes it rarer and therefore worse. Routing the hop back out through the caller's own task means a full queue applies backpressure to a transport, where backpressure belongs.

At most one hop. The lane that receives a resolved request runs it where it lands. It cannot need a second hop — the instance id is fixed, so re-resolution would name the same lane — and more importantly, correctness must not depend on the hop landing correctly. It doesn't: a wrong landing bounces on the claim. The hop is affinity; the claim is correctness. See Ownership and claims.

The receiving lane re-runs only the race-sensitive part — claim, load, guards, pin resolution, resume. It does not repeat decode, validation, or correlation, which are deterministic over the delivery and already done. Redoing them would be wasted work and, worse, a second place where those semantics could drift.

The runtime shape, including the one that was wrong

Three shapes were weighed for removing the blocking store calls.

(a) One asynchronous loop per lane, awaited to completion per request. Every persistence call becomes awaited; the loop is "receive a request, await it fully, receive the next". Because the loop awaits each request to completion before receiving the next, every ordering property above is preserved verbatim per lane — commit still happens-before reply and before the next dequeue. No completion re-entry exists, so there are no re-entry rules to get wrong.

(b) Blocking loop plus I/O offload with completion re-entry. Split a step at the commit, hand the write to a worker, return to the loop, and re-enter a completion event later. This is the only option that adds intra-lane pipelining — and it was rejected. It breaks reply-implies-committed unless responders are parked on the completion; it needs per-key in-process mailbox holds, which is a second serialization mechanism layered on top of claims; and its failure modes (a commit landing after a crash-restart re-queued the work, a lost completion, a mailbox stuck held) are precisely the subtle-ordering-bug class the design exists to avoid. Two mechanisms enforcing one invariant is how invariants get broken.

(c) N blocking lanes. Zero semantic change, and N× I/O concurrency arrives from N threads.

The order shipped was (c), then (a); (b) was rejected outright. (c) makes the convoy fix reviewable and bisectable on its own. (a) then removes the blocking as a mechanical conversion whose diff is wide but whose semantics are provably unchanged. What (a) buys concretely: lanes park on the runtime instead of holding threads when the pool is the bottleneck, an entire class of "blocked inside an async runtime" panic disappears, and the door stays open to pipelining later without another trait migration.

The shape that was falsified

The first cut of (a) gave each lane its own runtime with its own I/O reactor. It looked like the clean version: a lane is fully self-contained, owning its execution and its I/O.

Its own acceptance gate — the "one lane must be identical to before" bar — falsified it. When a lane shut down, its reactor died with it, and pooled database connections registered on that reactor were stranded: sockets nobody would ever poll again. The visible symptom was a restart hang. The successor engine's lease acquisition waited out the full lease TTL before it could proceed, roughly half the time. A restart-timing flake, not a wrong answer — the kind of thing that gets triaged as flaky infrastructure for months.

The fix inverted the ownership: lanes own no reactor at all. Each lane's loop is driven on its own dedicated thread via the process-wide runtime's handle, so the reactor topology is identical to what it was before the conversion, and lane lifetime is decoupled from I/O registration lifetime. Restart handover got materially faster as a side effect, because nothing waits out a lease TTL any more.

Two things are worth taking from this. First, the "obviously clean" decomposition was wrong because it decomposed the wrong resource: execution is per-lane, but I/O registration is per-process, and conflating a lifetime with a locality is a recurring shape of bug. Second, the identity bar is what caught it. An acceptance criterion of "one lane behaves exactly as before" is a much stronger instrument than a suite of feature tests, because it fails on things nobody thought to assert.

The activation flip

Activation rebuilds the engine's live view of processes, codecs, validators, and channels. Under one lane that rebuild was a single request, atomic against every dispatch.

Under lanes the controller sends the rebuild to every lane and awaits all of them before replacing the live deployment set and rewiring transports. Per-lane atomicity is preserved; that is all that is needed, and the argument for why is worth being explicit about.

The flip never promised cross-component simultaneity in the first place. Even under one lane the later stages — replacing the deployment set, reconciling schedules, rewiring transports — were already non-atomic with respect to the actor swap; deliveries kept flowing between them. What was promised, and is still promised, is per-dispatch consistency: nothing is ever observed half-flipped.

Per-lane atomicity delivers exactly that, because an instance's steps all run on its one lane, whose flip is a single point in its own queue. No instance can straddle two definitions. During the fan-out window two different instances can be served either side of the flip — which is indistinguishable from two deliveries ordered around a single flip point, which is what happened before. The draining-deployment correlation tail is part of the rebuilt view and flips with it, per lane.

Where false confidence would have come from

The risks that mattered were not the ones the code made obvious. They were the ones where a green test suite would have been actively misleading:

The pattern in that table is the transferable part: for each risk, name the specific green result that would have been misleading, then design the test that would not have been. An all-green-at-one-lane suite proves nothing whatsoever about cross-lane windows.

Next

• Execution lanes — the operator-facing chapter.
• Ownership and claims — the mechanism lanes lean on for correctness.
• Durable execution — what a lane commits at a quiescent point.

Retry machinery

Retries, history, and schedules covers what <q:retry> does. This chapter is why it is built the way it is — and the channel-call half turned out to have a genuinely hard problem in it.

A backoff is a park, not a sleep

The first decision constrains everything else: a retry wait is a durable timer park.

The alternative — sleep in place and try again — is not merely inelegant here, it is unavailable. A lane executes one request at a time, so an in-path delay does not delay one instance; it freezes every instance queued behind it on that lane. A ten-second backoff would become a ten-second stall for unrelated work. Even with lanes, a sleeping lane is a lane doing nothing while holding a queue.

So a failed attempt with budget remaining re-parks the instance: the failed task stays pending, a timer row is armed at the backoff instant, and the ordinary timer poller re-drives it when the instant arrives. Deliberately the same seam that respond-and-continue already uses — not a parallel mechanism. A second scheduling path would be a second set of races to reason about, and the existing one already survives restarts, hot-deploys, and replica death.

Everything good about the behaviour follows from that one choice. The backoff survives a restart. A PT5M wait is PT5M whether or not the replica that started it is still alive. No execution capacity is consumed while waiting. And a retry park is pinned to its deployment exactly like every other park, so a hot-deploy cannot silently move a backing-off task onto a different definition.

The cost is honest and stated: a <q:retry> task makes its process stateful. The park needs persistence. A process that was otherwise run-to-completion now requires a datasource, and the structural classifier accounts for it rather than discovering it at runtime.

Attempt state lives on the snapshot

Attempt counters are snapshot keys, one per node, not a column on the waiting row. Three reasons, in increasing order of importance.

A column would be erased by the very step that increments it. The retry park resolves its timer row and creates a new one each attempt. Attempt state stored there would be destroyed by the re-park — the mechanism would delete its own bookkeeping.

Attempt state is instance state. It must ride the failure re-stamp untouched when the instance eventually fails, so that an operator inspecting a dead instance can see how many attempts it burned. Putting it on the snapshot gets that for free, because the failure re-stamp is a key patch over the snapshot's map.

It costs zero migration and stays byte-identical when unused. The snapshot container is an open-keyed map, so a new key family needs no schema change anywhere. And because the key is absent unless a task actually has a policy, a process with no <q:retry> writes byte-identical snapshots to one that predates the feature — which keeps a byte-for-byte golden corpus valid across the whole feature. A feature that changes the persisted bytes of flows that don't use it is a feature that cannot be verified cheaply. See Durable execution for the encoding and the patchers.

A malformed counter reads as zero rather than failing the load, for the same reason the rest of the decode path is lenient: a snapshot must never become unloadable.

The channel-call problem

Extending <q:retry> to channel-call tasks was initially skipped, on the theory that the outbox's own retry curve plus the timeout boundary already covered them. A survey of what the code actually did disproved it:

• the outbox curve retries one delivery, by default forever — it is about reaching a counterpart, not about the task having another go;
• a timeout without a policy simply kills the instance.

So there was a real gap: no way to say "if the counterpart doesn't answer, ask again". Closing it meant answering a question the registered-task case never has to: what, exactly, is a failed attempt of a call?

The honest failure set

Derived from what the dispatcher and executor actually do, and nothing else:

1. The route-less <q:timeout> boundary firing. With a policy present the timeout is a retryable task failure first. Without one, the pre-existing catchable-error behaviour is preserved byte for byte.
2. A terminally-poisoned request delivery. The outbox exhausted its configured attempt ceiling while the task waited. Before this existed, the instance simply stayed parked forever — the delivery had given up and nothing told the task.

Both are stable structured codes, which is what makes them expressible in nonRetryableCodes. ("Never retry timeouts" is a reasonable policy, and it has to be sayable.)

Not failures, deliberately:

• A correlated business response. The counterpart answered. What the answer says — approved, declined, rejected — is the process's business to route on with a gateway. Retrying because a business answer was unwelcome would double-submit, and re-issuing an instruction because the first answer was "declined" is exactly the class of bug an engine must make structurally impossible.
• BPMN errors. They route to their boundaries, unchanged, on either task kind.

Modelled outcomes beat policies — enforced at load

A timer boundary with drawn outgoing flows is a modelled outcome: the author has said what happens. So it wins, exactly as a BPMN error routes to its boundary rather than consuming a retry budget.

Which means a channel-call <q:retry> alongside a routed timer boundary is a policy that could never fire. Rather than let that load as a silent near-no-op, the loader refuses the combination. A configuration that cannot do anything should not be accepted quietly; the author believes they have retries and does not.

The hard part: a dead attempt and an in-flight one look identical

Here is the problem that shaped the rest of the design.

Consider a channel-call node in a backoff window — its previous request failed, it is waiting to re-issue — and a channel-call node whose request is in flight right now. Through the durable facts available (the wait frontier plus the attempt counter) they look exactly the same: a node waiting, with attempts already burned.

They demand opposite treatment. A late response to the dead attempt must be refused; a response to the live attempt must resume the flow. A due timer on the first is the re-drive; a due timer on the second is stale residue.

The registered-task shortcut does not work here. For a registered task, "attempts burned and not completed" implies a backoff window. For a call node it does not, because a counterpart's relay can name the node mid-retry.

The backoff-window marker

The resolution is one more durable key: a marker set while a node is in a backoff window, carrying the classification of the failure that parked it. It is carried exactly like the attempt counters — an open-keyed snapshot entry, no migration, absent by default so unaffected snapshots stay byte-identical, and renamed along with the node mapping when an instance is migrated.

It keys four decisions, each made under the instance claim from durable facts alone:

That last row has a deliberate sub-decision: the correlation row stays live rather than being retired. A counterpart answering a superseded request gets an honest "that attempt is gone, a retry is pending" verdict instead of a "no such instance" miss — the same posture durable FAILED takes, where the truth is more useful than a 404. And when the re-drive re-emits, the same correlation serves the new attempt's response normally, with no re-registration.

The re-emission contract

A re-drive is not "wait for the same answer again". It re-runs the park's side effects from durable state:

• a fresh request emission, built over the same persisted variables, with a fresh idempotency key on the wire;
• a fresh timeout window;
• the response wait re-incarnated — a node both resolved and re-parked in one step is a new incarnation, and is recorded as such rather than as an update of the dead one.

The fresh idempotency key is the subtle one and it is deliberate. The retry is a genuinely new request, and a counterpart doing its own deduplication must see it as new. Reusing the key would invite the counterpart to answer "already handled" for an attempt it never actually completed — turning the retry into a silent no-op precisely when it matters.

Withdrawing the dead attempt's deliveries

The backoff park deletes the dead attempt's outbox rows — pending and poisoned alike — in the same transaction that arms the backoff.

This is a deliberate, narrow exception to an otherwise absolute rule: the outbox never deletes an undelivered row. The rule exists because a deleted undelivered row is lost work with no trace, and it is a good rule. Two concrete races justify the exception:

• A superseded request delivered late would race the re-drive's fresh emission into a double-submit — two live requests for one logical call.
• A superseded request poisoning later would fire a failure against the live attempt, consuming a budget slot that attempt never spent.

Both are worse than the loss the rule protects against — and the loss does not actually occur here, because the durable record of the failure survives elsewhere: in the marker, in the attempt counter, and in the incident the poison already recorded. Nothing is forgotten; only the pending delivery is withdrawn.

The general shape is worth naming: an exception to a safety rule is defensible when you can point at the specific corruption the rule would cause, and show the information the rule was protecting is preserved by another mechanism.

The poison wake is a prompt, never a fact

When the outbox gives up on a request, the task waiting on it needs to hear about it. The dispatcher fires a best-effort in-process wake — a fresh serialized turn, like a timer fire.

But the wake is treated as a prompt to go and look, never as evidence. The engine acts only on durable evidence re-read under the instance claim: a poisoned outbox row for exactly this instance and this node. A wake that arrives spuriously finds nothing and does nothing.

And a wake that is lost — a crash, a shutdown, a dropped in-process message — is recovered by the timeout boundary, which the loader guarantees every channel call has. That guarantee is what lets the wake be best-effort at all: the fast path is a notification, the correctness path is the boundary the model was required to declare anyway.

This is a shape worth reusing. An in-process notification that races with a crash is fine as long as (a) the receiver re-derives the fact from durable state, and (b) something durable eventually detects the same condition without the notification.

The whole thing is claim-arbitrated

Every decision above — relay versus timer versus poison, marked versus unmarked, re-drive versus resolve — is made under the instance claim, from durable facts. The races between a late relay, a due timer, and a poison notification arbitrate through the claim exactly as cross-replica races do; the marker only tells a holder which situation it is in, never who gets to act. See Ownership and claims.

The determinism discipline is unchanged throughout: no lane-blocking sleep anywhere, attempt state on the snapshot, every re-drive decision from durable facts, and byte-identical snapshots for every process that never backs off a call.

Exhaustion

A spent budget — or a nonRetryableCodes hit — lands on the same fatal path a registered task's exhaustion does: a durable FAILED snapshot carrying the structured code. Identical for both task kinds, deliberately, because "how many kinds of failure state does this engine have" should have exactly one answer.

From there the instance is retained, inspectable, blocks its deployment's retirement, and is repairable by migration — which is the loop the whole design is pointed at: fail loudly, keep everything, let an operator fix the model and bring it back.

Next

• Retries, history, and schedules — the author-facing chapter.
• Migration internals — how a burned budget travels with its task.
• The pull surface — the other retry budget, and why they never overlap.

Migration internals

Instance migration is the operator's guide. This chapter is the reasoning: a row-security problem with a silent failure mode, why compatibility is derived from resume behaviour rather than from BPMN taxonomy, how the re-arm set is worked out, and why batch independence is structural rather than promised.

The two-scope problem

Every other commit in the engine runs inside one deployment scope: open a transaction, set the scope, write, commit. Row-level security policies then confine every statement to that scope, which is how tenant and deployment isolation is enforced at the database rather than in application code.

Migration breaks the pattern by definition. It reads rows pinned to the source and writes rows pinned to the target.

The obvious implementation — a single UPDATE that sets the deployment id — cannot work under an enforcing policy. The shipped policies are visibility predicates with no separate write predicate, and the database then reuses the visibility predicate as the write check for an update. So the statement fails at both possible scopes, and the two failures are not equally bad:

The second is the dangerous one, and it is worth dwelling on why. It does not raise. It does not warn. It returns success having moved zero rows — so an operator running a migration would see it succeed, and the instance would still be pinned to the broken model. Every downstream signal would agree that everything was fine.

That is the failure mode this design exists to make impossible, which is why both halves are pinned by a test running as a genuinely non-bypassing database role. The justification for the shape is executable rather than asserted — a claim about what row security does under an enforcing role is exactly the kind of claim that rots quietly.

The scope flip inside one commit envelope

What makes the operation possible is that the scope setting is transaction-local, not transaction-immutable. One transaction may re-scope itself between statements.

So the commit runs two phases inside one envelope:

1. Scoped to the source — lock the instance row, re-assert the ownership claim and the non-terminal status, read every row belonging to the instance, delete them.
2. Scoped to the target — insert the rewritten rows.

Atomicity is unchanged: a phase-2 failure rolls phase 1 back, and no session ever observes the instance under both pins or under neither. Isolation is not weakened either, and the distinction is precise — the transaction is never scoped to two deployments at once. It finishes with one, then scopes to the other. There is no window in which a statement could see both.

The re-assertion in phase 1 is deliberate. The claim was taken before validation; re-verifying it under the row lock, inside the commit, closes the gap between "claimed" and "moved".

The snapshot moves as a byte-level patch

The snapshot rewrite is a key patch over the raw map — the same shape marking an instance failed or terminal uses, and for the same reasons, plus one specific to migration.

A decode-and-re-encode would need the tenant's data-encryption key, would have to re-derive an encryption set the resume-time snapshot no longer carries in that shape, and would persist a previously-encrypted value in the clear the moment either went wrong. See Durable execution.

That carried-through ciphertext still decrypts under the new pin is not luck. The authenticated-data binding ties a ciphertext to its key, its instance and its variable name, and deliberately excludes the deployment id — a property the encryption design chose precisely so that "a version migration changes only the pin" would stay true. A design decision made in one subsystem paying off in another, years later, is what a well-chosen invariant looks like.

Rewritten: the deployment pin; the wait frontier, completed set and routed start (each entry mapped through the node mapping); the per-node retry attempt counters — the key is renamed, the counter untouched, because a burned budget follows its task; the audit sequence floor, bumped past the migration event.

Untouched: variables plain and encrypted, the key anchor and sensitive-name set, the status, the failure keys, and coverage cursors — those last are keyed by declared path id, not node id, so applying the node mapping to them would corrupt them. The distinction is invisible unless you look, which is why it is written down.

Compatibility is derived from what resume does

The locus model is the substance of validation, and its one real idea is this: the question is never "is this a userTask". It is "can a parked message wait resume here".

Several distinct BPMN constructs answer yes to the second question, and the set is not the same as any single element type. So each locus is classified by the resume behaviour it will actually get, and the target node is checked against capability flags rather than an element enum.

Two classifications carry the weight:

A retry park is not merely a timer park. It is a timer row whose node has a durable attempt count and is absent from the completed set — which is exactly the condition the executor itself uses to route a due timer to re-run the task rather than to fire the timer. So the compatibility rule reuses the executor's own test rather than reinventing one. Landing a retry park on a merely timer-capable node would silently re-run nothing.

A continue-reply park is read from the source graph. Whether a parked node carries a respond-and-continue reply is a property of the definition the instance is running under, not of the target. Which is why a source deployment whose plan is no longer registered is a hard refusal rather than a fallback: validating a continue-reply park against the timer rule would pass the wrong check and produce a confidently-wrong verdict. Refusing to answer beats answering wrong.

The capability index each plan projects is small and computed at activation, so validation never re-verifies and re-plans two sealed archives on an administrative request path. It covers draining deployments as well as active ones — a migration's source is by definition a deployment that has been flipped away from. Sub-process bodies are flattened into their parent's index, because durable state records a node id with no scope path.

Every violation, not the first

The validator reports all violations. Fixing a mapping should take one round trip, not one per node — a validator that stops at the first error turns a ten-node rename into ten deploy-test cycles.

The same instinct drives the refusals that could have been silent: a mapping entry naming a node the instance neither parks at nor has completed is refused, because a typo must never read as "identity mapping, then".

The resume re-arm set: why the frontier alone is wrong

resume: true brings a FAILED instance back. Working out what to re-arm is the subtlest part of the whole operation.

The failure commit did two things: key-patched the snapshot to FAILED with its failure code and detail, and resolved every waiting row in one statement so no timer refires and no relay finds a live wait. The frontier itself was left untouched.

Resume is the inverse of both halves. The snapshot half is straightforward — status back to suspended, failure keys dropped, output byte-identical to the pre-failure snapshot because the failure keys are emit-only-when-present, and like every other re-stamp it is a raw patch, so reviving an instance never needs the tenant key and can never downgrade its at-rest protection. It fails closed on any status but FAILED.

The rows are the hard half, and the frontier is not enough. A <q:timeout> synthesizes a boundary with a derived id, and a timer boundary event has its own id — neither appears in the frontier. Re-arming by frontier alone would silently drop the timeout the park was armed with, producing an instance that resumes and then waits forever with no deadline.

The rows are recoverable exactly, though, and the derivation is a nice piece of reasoning about what the data already tells you:

They were resolved by one statement, so they share a single resolution timestamp — and it is the instance's latest, because nothing touches a FAILED instance's rows afterwards (the poller's stale and failed branches only update rows still waiting).

So the re-arm set is the frontier's own rows, plus every row sharing the instance's latest resolution timestamp. It is gated on a non-empty frontier: an instance that had nothing parked has nothing to restore, and an older satisfied wait is spent history rather than a park.

The parks are re-armed in the migration's own transaction, so migrate-and-resume is one commit and a crash cannot leave a half-revived instance.

Afterwards the instance comes back through ordinary paths: the row is unowned (the claim died with the source row), the snapshot is suspended, and the parks are armed. A re-armed timer whose instant has passed is claimed by the ordinary poller on its next tick; a message park waits for its next correlated inbound. There is no new resume entry point and no privileged re-drive, which is precisely why the claim-guarded concurrency story still holds afterwards.

A defect this closed

The first version derived loci from waiting rows that were still marked waiting. For a FAILED instance there are none — its failure commit resolved them all. So every frontier entry fell through to the "no row behind it" branch and was validated as a message wait. A dead instance parked on a timer was therefore checked against the wrong rule, and warned about a missing row that was sitting right there.

Since a FAILED instance is the operation's prime use case, that was not a corner case. Reading the park set as described above fixed it — and note the property that makes the fix trustworthy: what validation checks is now exactly the set resume re-arms. One derivation, two consumers, no possibility of drift.

Batch independence is structural, not promised

The batch endpoint is not a second implementation. A request body parses once into a migration plan; a single internal operation applies that plan to exactly one instance and returns a verdict rather than an HTTP response; and both endpoints are thin wrappers over it.

So "each instance validates and migrates independently" is not a promise the batch endpoint makes and must be trusted to keep. It is structural: the batch is a loop over the single-instance operation, and the attempt is the loop's return value. There is no shared mutable state for one instance's outcome to leak into another's, because there is nowhere to put it.

Everything else follows:

• One claim, one transaction, one report entry per instance. No batch-wide transaction — which is the point, and which makes the crash story trivial: a mid-batch crash leaves every instance either fully migrated or completely untouched, because that is already true of each instance's own commit and there is nothing larger to be half-applied.
• The outcome is a single enum, not a combination of booleans, so no caller has to infer a state from flags that might contradict each other.
• The status describes the batch, not its instances. A run where every instance refused still answers 200: the call was accepted, executed to completion, and reported in full. Scripts key on the totals and the per-instance outcomes.

Contention is reported, never retried

An instance whose claim is held bounces, and the batch moves on. Retrying inside the call is refused on two grounds, and both are about honesty rather than convenience:

• It makes an administrative request's runtime unbounded — the claim is held by another replica's work, on that work's timescale.
• It turns the report into a claim about a moment that has already passed. A report that says "everything moved" after internally retrying for thirty seconds describes a world that no longer exists by the time it is read.

The caller re-runs the same request instead. The selector is deterministic and whatever moved is no longer under the source pin, so a re-run converges — which is a much better contract than "we tried hard".

Selection is ordered by instance id, then limited. Ordering by a timestamp that moves under a live population is exactly how a caller silently skips work, and a paging bug that skips work without saying so is the worst kind.

Two request-level contradictions are caught from the request alone rather than N times over in the per-instance reports — resuming a selection of suspended instances (it selects exactly what resume refuses), and re-homing a mixed population into one process (one mapping could only be right for one of them).

What deliberately does not move

Pending outbound rows stay under the source pin. An emission was minted by the source deployment's channel bindings and is dispatched against them — and the dispatcher covers draining deployments, so it drains where it was made. Re-targeting a message at bindings that never produced it would be worse than leaving it: the destination, headers, and codec were all decided by a configuration the target may not even have.

The audit journal moves scope but keeps its node ids verbatim. A trail names where something happened, under the graph that was live at the time; rewriting the ids would falsify the record. The migration event itself records source and target, so the provenance is recoverable without corrupting the history.

Both are reported — the outbound rows as a warning — because the operator should know why, not just what.

Next

• Instance migration — the operator's guide.
• Durable execution — the patchers and the encryption binding this relies on.
• Ownership and claims — why the migration claim needs its own owner identity.

The pull surface: design

External tasks is the worker-facing guide. This chapter is why the pull surface is a hundred lines of new machinery rather than a subsystem — which is almost entirely down to one decision.

Ownership transfer is the design

A channel declaring transport: pull gets a sink like any other transport. The sink claims its URI scheme, exactly as an HTTP or broker sink claims theirs. What it does with a delivery is different: instead of dialing anything, it parks the delivery as a task row — and then answers Delivered.

That answer is the whole design. Ownership transfers from the outbox row to the task row, and the relay deletes the outbox row exactly as it would for a genuinely delivered push.

Everything else falls out of it for free:

• The relay's retry and poison machinery applies to a failed park automatically. If parking fails — the database is briefly unavailable, say — the sink answers a retryable failure, and the outbox's existing backoff curve retries the park with no new code. If the operator configured an attempt ceiling, an unparkable delivery poisons exactly like an undeliverable one, records its incident, and stops pinning a draining deployment. None of that had to be re-implemented for pull; it was already there, one layer up.
• The hot claim predicate is untouched. The outbox worker's SKIP LOCKED batch claim — the query that runs constantly on every replica — does not learn about pull at all. It sees a delivery, hands it to a sink, and gets an answer. Adding a delivery mode without touching the hottest query in the system is the payoff for putting the new behaviour behind an existing polymorphic seam rather than beside it.
• Parking is off the execution path. The sink runs on the outbox tick loop, not on an execution lane. It touches the engine actor not at all: a park is a store write plus a wake-up.

The alternative shape — a first-class "pull delivery" concept understood by the dispatcher, the outbox worker, and the retirement gate — would have been a genuinely new subsystem with its own retry semantics, its own poison horizon, and its own failure modes to reason about. Modelling pull as a sink that happens to hand ownership to a different table keeps one delivery pipeline.

The completion is not a new resume path

A worker's result is rebuilt into an ordinary inbound message and re-enters the engine through the same seam every transport delivers through. Not a nested call, not a privileged entry point: one more serialized turn, indistinguishable from a pushed reply.

That is what keeps three things unchanged rather than duplicated:

• Correlation. The parked headers ride through, so the author's <q:alias> resolves the waiting instance the way it always has.
• Validation. The completion goes through the same intake pipeline — a worker cannot smuggle a structurally invalid payload past the checks a pushed message faces.
• Deduplication. The completion carries the originating delivery's key as an explicit idempotency key, so inbox dedup covers it with no worker-side cooperation.

A second resume entry point would have needed all three re-implemented, and would have been the place where they drifted.

Lock expiry inside the claim predicate: no sweeper, by construction

A worker that dies mid-task must not hold its task forever. The usual answer is a reaper — a background job that scans for expired locks and releases them — which is a role to schedule, a lease to gate, a cadence to tune, and a lag between expiry and availability.

There is none here. Lock expiry is part of the predicate that decides what a fetch may claim: a task is fetchable when it is not terminal, its backoff has elapsed, and it is either unlocked or holding an expired lock. An expired lock is therefore not a state anyone has to clean up. It is simply not an obstacle to the next fetch.

The consequences are worth naming, because "no component" is easy to undervalue:

• No lag. A task is available the instant its lock expires, not on the next sweep tick.
• Nothing to gate. No lease role, no leader, no cadence key, no failure mode where the sweeper is down and locks pile up.
• Nothing to get wrong under concurrency. The claim and the expiry check are the same statement, so there is no window between "the sweeper released it" and "someone claimed it".

The cost of a dead worker is therefore exactly one lock duration — which is a knob the operator already sets, rather than a second knob about how often to look.

The completion path extends the same idea: it re-takes the lock across the dispatch, so the lock cannot expire mid-flight and let a second worker pick the task up while the first one's result is still inside the engine.

Dispatch, then delete — and why the inverse loses work

The completion order is: verify and hold the lock, dispatch the result through the inbound path, then delete the task row.

That makes the surface at-least-once. A crash between the dispatch and the delete re-offers the task, and a duplicate completion is absorbed by inbox deduplication under the delivery's idempotency key.

The inverse order — delete, then dispatch — would make it at-most-once, and the same crash would lose the work outright: the task is gone, the instance is still parked, and nothing anywhere records that the worker ever answered. The instance waits forever on a result that was produced and thrown away.

Between a duplicate the system already knows how to absorb and a silent loss it cannot detect, the duplicate is not a close call. This is the standard shape of the choice, and it is worth stating in those terms: at-least-once plus deduplication beats at-most-once plus hope, whenever deduplication is available — and here it is available by construction, because the idempotency key was minted by the delivery that created the task.

A worker that completes with no result re-delivers the original request payload. That is the fire-and-forget shape: the work happened outside, and the flow waits only on the fact of it.

If the engine refuses the completion on the inbound path, the row stays, still locked until its grace window elapses, and becomes fetchable again. The refusal carries the engine's own code as an attribute — not just prose — because that code is the only thing that tells a worker whether re-fetching later can ever help (a transiently unavailable engine) or never will (a validation reject that will fail identically forever).

Zero rows affected is never success

Every worker-facing mutation is ownership-guarded: it matches only if this worker still holds this task's lock. Which means the interesting case is a statement that matches nothing — and the critical rule is that this is never reported as success.

A 200 to a worker whose lock lapsed is a duplicate-execution bug. The worker believes its result landed; it did not; the task is fetchable again and someone else will do the work too.

So a zero-row result triggers a second, unguarded read to find out which situation it is, and the answer is one of four verdicts:

Splitting them apart matters because they call for different worker behaviour, and a single generic "conflict" would leave a worker author guessing — usually by retrying, which is wrong for three of the four.

The extra read is deliberate and cheap: it happens only on the failure path, and it reads lock state only. The payload never rides a failure path — there is no reason for a task's body to travel through an error response.

Separating "no such task" from "you do not hold it" needs one more distinction: the guarded statement is attempted across the live deployment set, and if it matches nowhere, an unguarded existence probe distinguishes a lock problem from a genuinely absent task. A worker names topics, never deployment ids — the same posture the outbox worker and the instance listing take — so the surface does the walking.

Bounds are rejections, never clamps

Every bound a worker can send — lock duration, long-poll wait, batch size — is a ceiling the operator sets, and an over-ceiling request is a 400.

Silently clamping would be the friendlier-looking choice and it is the wrong one, for a reason specific to locks: a worker that believes it holds a longer lock than it does is a duplicate-execution bug. It paces its work against a deadline that already passed, the task goes fetchable, another worker takes it, and the first one's completion is refused after the work was already done twice. A clamp does not prevent that; it creates the belief that causes it.

The same reasoning applies to the others, less dramatically: a clamped long poll makes a worker's own timeout arithmetic wrong, and a clamped batch size makes its throughput planning wrong. In each case the reject is information and the clamp is a lie the system tells quietly.

Omitting a bound is different from asking for too much: an omitted value takes the default, not the ceiling. The default is a considered value; the ceiling is a limit. Conflating them would hand every silent request the most expensive setting available.

The engine also refuses to boot if the default lock duration exceeds its own ceiling, or is zero — a configuration that is internally contradictory should fail at startup, where somebody is watching, rather than at the first fetch.

The long poll wakes, and cannot miss

A fetch that finds nothing waits, bounded, for a task to be parked on one of its channels. The wake-up is a broadcast keyed by channel, so a fetch filters to the topics it asked for instead of re-querying on every unrelated park.

Two properties make it safe:

• Subscribe before the first query. A task parked in the window between the query and the wait still wakes the waiter. The classic lost-wakeup ordering bug is closed by ordering, not by a timeout that hides it.
• Missing a wake-up is harmless. A woken fetch always re-runs the claim query; the wake-up carries no payload beyond "look again". So a lagging subscriber is not an error — it costs at most one extra round of a loop it was already in. The ring buffer is deliberately small, because it is a doorbell rather than a queue.

And the wait is always bounded by the operator's ceiling, so a fetch answers with an empty list rather than hanging. A long poll that can hang is a client-side resource leak with extra steps.

Two budgets that never overlap

The worker-failure budget and a <q:retry> policy sit at different layers and never interact:

A worker retrying its own work never touches the instance; the instance stays parked on its wait, exactly as it would while a worker was simply slow. Conversely a <q:retry> timeout on a pull-backed call still fires if no worker ever completes — the pull surface changed the last hop, not the model's semantics.

A terminal task no longer counts toward its deployment's retirement quiescence gate, for the same reason a poisoned delivery doesn't: "we gave up" must be a durable, visible state, and it must not hold a deployment open forever. That symmetry is intentional — the terminal task is the pull-side twin of the outbox's poison horizon. See Retry machinery.

Posture

These are operate-surface routes, not administrative ones, and the classification is a judgement worth stating: a completion is an ordinary delivery, not a privileged control operation. It goes through the same intake as a pushed reply, and it can do nothing an inbound message on that channel could not do.

So it carries the operate surface's cluster-internal posture rather than the administrative surface's gate. A deployment needing authenticated workers puts them behind the same ingress policy the rest of the operate surface already needs.

Next

• External tasks — the worker-facing guide.
• Retry machinery — the other budget, and the outbox poison horizon this mirrors.
• Ownership and claims — why a task lock is not an instance claim.

Configuration reference

The engine reads exactly one family of configuration keys — canonical sutra.* properties, each with a canonical SUTRA_* environment-variable mirror. There is no separate framework prefix and no legacy alias layer to reason about.

Sources and precedence

canonical env  >  config file  >  built-in default

The config file path comes from SUTRA_CONFIG (default sutra.properties, read from the engine's working directory, and only if present); file values may themselves use ${ENV} / env:NAME indirection. A key that has both a sutra.* file entry and its SUTRA_* environment variable set always resolves from the environment.

rust/crates/sutra-engine/src/config.rs is the single source of truth for the exact key list — this page is a map of what's there and how the pieces fit together, not a duplicate of it; treat that file (and otel.rs for the telemetry keys) as authoritative if this page and the running engine ever disagree.

Deployment source

The engine's own datasource

This is the engine-internal database (instances, outbox, lease, audit, inbox) — never a package's own business data store, which owns its connection independently in datastores.yaml (see Data stores). Note the CLI's sutra migrate reads a different-named set (SUTRA_DB_URL / SUTRA_DB_USERNAME / SUTRA_DB_PASSWORD / SUTRA_DB_SCHEMA) even when pointed at the same database — see the CLI reference.

Outbox and acknowledgement

Outbound deliveries retry with backoff forever by default. sutra.outbox.retry.max-attempts is the opt-in ceiling: a delivery that exhausts it is marked terminally poisoned — retained with its last error, never retried again, one incident recorded for a required delivery, and no longer counted by the draining-deployment retirement gate. "We gave up" is a durable, visible state, never a silent disappearance.

Full explanation of the deferred-ack registry the sutra.ack.deferred.* keys tune: Acknowledgement modes. Configuring the attempt ceiling is also what makes a poisoned delivery reachable as a <q:retry> failure — see Retries, history, and schedules.

Instance lifecycle — ownership and retention

Every resume claims the instance first, so two replicas can never advance one instance concurrently; the stuck-instance sweep (sweep-interval, default PT1M) clears claims whose owner has been silent longer than claim-timeout (default PT5M).

A finished instance is history, not a 404: terminal (completed/terminated) snapshots are retained for sutra.instance.retention (ISO-8601 duration, default P7D) and served by GET /sutra/instances/{id} and GET /admin/instances/{id}/history; a lease-gated sweeper purges rows past the window on the retention-sweep-interval cadence (default PT1H). PT0S restores delete-at-completion. Failed instances are always retained regardless of the window — they need an operator before their deployment may retire; the operator action is instance migration. See Retries, history, and schedules for what the retention window makes queryable.

External tasks (the pull worker surface)

A channel declaring transport: pull parks its deliveries as fetchable tasks instead of dialing an endpoint; workers drive them over POST /sutra/external-tasks/fetch-and-lock (bounded long poll) and .../{id}/complete / .../{id}/failure. The defaults: a PT30S lock when the fetch names none (ceiling PT1H — a longer request is rejected, never clamped), a PT30S long-poll ceiling, 100 tasks per fetch, a worker-failure budget of 3 with PT10S between attempts. A spent budget turns the task terminal (failed) — retained, never fetchable again. The engine boots fail-closed if default-lock-duration exceeds its own ceiling or is zero. Full treatment: External tasks.

Execution lanes

sutra.engine.shards is the number of identical actor lanes the engine executes on inside one replica. At N > 1 all work for one instance — routed by a stable hash of its id — still runs on one lane, in arrival order, one request at a time: per-instance serialization is the contract, and it holds at every N. Values above 1 are accepted; the default stays 1 because turning it up has one advertised consequence.

Say it out loud before you raise it. Incidental cross-instance serialization disappears at N > 1. Two concurrent deliveries to two different instances of the same flow never interleave under a single lane — as a side effect of there being one lane, never as a promise. At N > 1 they genuinely run in parallel. A deployment silently leaning on that side effect will observe new interleavings. Every supported concurrency mechanism is unaffected: per-channel singleton / serial consumption, per-channel and per-tenant admission caps, and optimistic expect="unchanged" / pessimistic forUpdate data-store writes.

shard-queue-capacity bounds each lane's mailbox (unset = unbounded; zero is rejected — "unset" is how you say unbounded). A bounded send awaits on the caller's task, so backpressure propagates outward to the transport that offered the work — an in-flight HTTP request, a broker prefetch window, a poller tick — and never sideways into another lane.

Meters ship with the feature, each carrying the lane index as a dimension: sutra.engine.shard.queue-depth (per-lane backlog and skew), …dispatches / …parks / …resumes, …handoffs (cross-lane relay hops — expected, and rising with lane count by construction), and …claim-bounces split relay / timer. That last one is the mis-route alarm: on a healthy rollout it reads near zero outside genuine cross-replica contention.

The live lane count is readable without reading config: GET /sutra/health/ready reports it under the loader check's data.shards, read off the running router rather than echoed back from configuration.

Full picture: Execution lanes.

Limits

Full explanation: Limits and quotas.

Admin API auth

The /admin/* surface (deployments, instance inspection, subject erasure) is gated fail-closed: unconfigured returns 503, a missing/invalid credential returns 401, and a valid token missing the required role/claim returns 403 — never silently open. The auth-key scheme is the same static-secret model channels use for inbound HTTP auth, and it takes precedence when both are set. Gating is disabled only via the explicit sutra.admin.oidc.dev-disabled=true escape hatch.

Audit sinks

See Logging and audit.

Telemetry (OTel)

No OTLP endpoint configured means no exporters at all, and this is the default — telemetry export is opt-in, never opt-out, and Sutra itself collects nothing regardless: see No telemetry, no phone-home for the exact guarantee, and Observability for what each signal actually carries once you do turn export on.

Secrets — never literal values

Every credential-shaped field across channels.yaml, datastores.yaml, and the admin-auth keys above is a reference, resolved at startup or channel-activation time — env:NAME, secret:KEY (a file under the mounted secrets directory, default /etc/sutra/secrets / SUTRA_SECRETS_DIR), ${NAME} / ${NAME:default} placeholders, or a vendor scheme (vault:…, aws-secrets:…, azure-kv:…, gcp-secret:…) resolved by whichever sutra-envref-<vendor> crate the binary was built with. A literal secret value in a resource file is rejected at package-validation time, not just discouraged by convention. See Domain neutrality and the SPI model for how a vendor secret backend plugs in.

Next

• Acknowledgement modes
• Limits and quotas
• Logging and audit
• Instance migration
• Execution lanes

Acknowledgement modes

An inbound channel declares an ack-mode that decides when the engine acknowledges receipt relative to processing. How each transport actually realizes that intent differs — a broker gets a native ack/nack, HTTP gets a status code — but the two values mean the same thing everywhere.

The two modes

• on-persist — acknowledge the moment the engine has durably captured the inbound, before the BPMN process runs. On a broker: ack immediately, releasing the delivery slot early (lower broker-side latency; a redelivery after a mid-process crash is caught by inbox dedup). On HTTP: reply 202 Accepted with no business body and process asynchronously — the fire-and-forget intake, whose eventual reply (if any) rides an outbound channel instead of the original connection.
• on-complete — acknowledge only once the instance reaches a terminal state (INSTANCE_COMPLETED or INSTANCE_FAILED). On a broker: the ack is deferred — registered against the engine's DeferredAckRegistry and held until the instance finishes. On HTTP: hold the connection open until completion and return the reply body — the classic synchronous request/reply.

The default differs by transport, because the natural mode differs: broker channels default to on-persist (release the delivery slot early); HTTP channels default to on-complete (synchronous request/reply). An HTTP channel opts into asynchronous intake by declaring ack-mode: on-persist explicitly.

channels:
  - name: orders-inbound
    transport: http
    bind: "POST /channels/orders-inbound"
    ack-mode: on-persist     # HTTP: 202 Accepted, no synchronous reply body

When to pick which

The mode is set per channel — different channels on the same engine can use different modes.

Per-transport wiring — what actually realizes on-complete

on-complete's deferred-settle mechanism is a capability each transport factory self-declares (see Domain neutrality and the SPI model) — the engine never hardcodes a per-vendor branch. Current wiring:

A channel declaring ack-mode: on-complete on a transport whose factory reports it can't realize it never fails silently — the engine emits SUTRA.ACK.ON_COMPLETE_UNSUPPORTED at startup and runs on-persist.

The deferred-ack registry

The registry (sutra-channels) is a bounded, insertion-ordered structure with three operator knobs (see Configuration reference):

Eviction is an operator-visible failure mode, not a silent memory leak — repeated SUTRA.ACK.DEFERRED_OVERFLOW events mean either raise the capacity or find the runaway processes that aren't reaching a terminal state. Diagnostic codes at each stage: SUTRA.ACK.DEFERRED_REGISTERED (debug), SUTRA.ACK.DEFERRED_ACKED (debug), SUTRA.ACK.DEFERRED_NACKED (info — a permanent reject), SUTRA.ACK.DEFERRED_OVERFLOW (warn), SUTRA.ACK.DEFERRED_TIMEOUT (warn), SUTRA.ACK.ON_COMPLETE_UNSUPPORTED (warn, at startup).

Migration note

Omitting ack-mode entirely preserves today's behavior exactly: a broker channel stays on-persist, and an HTTP channel stays synchronous (on-complete). Turning on a broker's deferred ack, or an HTTP channel's async intake, is a one-line addition to channels.yaml and takes effect on the next deployment poll.

Next

• The money-transfer worked example uses singleton + ack-mode together across three transports feeding one process.
• Troubleshooting BPMN solutions — reading the SUTRA.ACK.* codes above out of the audit trail when a message seems to have vanished.

Limits and quotas

Engine-wide ceilings, each with a safe shipped default and a consistent override shape — the same canonical sutra.* / SUTRA_* pair described in Configuration reference.

Inbound payload byte cap

Every inbound channel — broker or HTTP — delivers raw bytes to the engine before a codec ever parses them, and a producer can publish an arbitrarily large payload. The cap closes that gap: a message is byte-length-checked before codec.decode() runs, so nothing is allocated or parsed for an oversized message.

A per-channel override lives on the channel's own definition (not as a separate engine-wide property), and is not clamped by the global value — it can raise or lower the effective cap for just that channel:

channels:
  - name: bulk-statements-acme
    payload-cap-bytes: 104857600   # raised for a trusted large-file channel
  - name: heartbeat-acme
    payload-cap-bytes: 4096        # tightened for a strict small-event channel

Negative values are rejected at startup (SUTRA.CONFIG.PROPERTY.INVALID) — this catches a typo like -1 before it silently disables the cap the way some other libraries interpret that value.

On rejection. Exceeding the effective cap rejects the message with SUTRA.INBOUND.PAYLOAD_TOO_LARGE (ERROR), carrying the channel id, the actual payload size, and the effective cap that was applied — enough for an operator to know exactly which config key to raise. This is a permanent reject, not a retryable one:

Per-tenant quotas

Two more admission checks, enforced before a message reaches the executor — see Multi-tenancy and isolation for the full detail:

• maxConcurrentInstances — a hard cap on simultaneously in-flight instances for a tenant, coherent across every replica.
• maxInboundRatePerMinute — a per-replica sliding 60-second admission window.

Neither is applied unless a tenant opts in — an unconfigured tenant is unlimited on both dimensions.

What's not yet a configurable limit

The threat-model backlog names a few more ceilings that aren't wired yet — a FEEL evaluation wall-clock/memory budget per expression, and a per-tenant audit-write rate cap. Until they land, the payload cap and the two tenant quotas above are the complete set.

Next

• Configuration reference — the full sutra.*/SUTRA_* key map these limits live in.
• Troubleshooting BPMN solutions — what a rejected-message diagnostic looks like in practice, and how to trace it back to the message that triggered it.

Deploy, hot-deploy, and rollback

The mechanics behind this page are covered in full in Deployment model; this is the operator-facing walkthrough of the same three operations.

Deploy

sutra deploy my-app.sutra --api --engine-url http://localhost:<port>

sutra deploy has two paths: the API path (--api, against an engine running the db deployment source) is synchronous — it returns only once the deployment is Active, or fails fast with the SUTRA.DEPLOY.* reject diagnostic. The ConfigMap path (the default, for a Kubernetes dir deployment source) patches the deployments ConfigMap and is asynchronous — the engine's own watcher picks up the change on its next poll; add --wait to have the CLI poll GET /sutra/deployments/{id} until it reports Active.

For a large deployment where a synchronous call risks a long-held request behind an ingress, --async (API path) submits and returns 202 {deploymentId, Pending} immediately, then the CLI polls to completion itself.

Hot-deploy

A hot-deploy is just a normal deploy call against an existing slot (the archive's stable tenant--module--version key): re-package the same source directory, and re-deploy. The new archive gets a new content-addressed deploymentId, but the slot name is unchanged, so the engine replaces the slot's active revision in one transaction and runs its two-phase activation flip — drain the old revision, activate the new one — with no restart. In-flight instances on the old revision keep running to completion in the background; new inbound picks up the new revision immediately.

# edit the package source, then:
sutra package packages/my-app --out /tmp/pkgs
sutra deploy /tmp/pkgs/my-app.sutra --api --engine-url http://localhost:<port>

For the edit-save-redeploy loop while developing, skip the manual re-package step entirely:

sutra deploy --watch packages/my-app --engine-url http://localhost:<port>

Each save is re-packaged and validated first — the same static validation sutra lint runs. Deployment only fires if validation passes; on a finding, the CLI reports it and skips the deploy, so a broken edit never reaches the engine.

Rollback

Rollback is the identical operation in reverse: re-package (or simply keep) the previous archive for the same slot, and re-deploy it. Its still-draining deploymentId resurrects rather than being rebuilt from scratch:

sutra deploy /tmp/pkgs/my-app-previous.sutra --api --engine-url http://localhost:<port>

Removing a deployment

sutra undeploy my-app.sutra --api --engine-url http://localhost:<port>

The engine drains it — refuses new intake, lets in-flight instances finish, retires the slot once it reaches zero instances and zero pending outbox entries. On the ConfigMap path, pair this with sutra deployments list <dir> to find the right archive/deploymentId first.

Two drain behaviors worth knowing before you need them:

• Inbound routes always follow the active set. A slot whose only revisions are draining serves no inbound routes at all (SUTRA.RESOLVE.CHANNEL.UNKNOWN on its paths); its parked instances still resume through relay correlation the moment you deploy a new active revision into the slot — which is also the recovery move when an undeploy left work parked.
• Accumulated drains are safe. Several draining revisions of one slot are a legal store state (interrupted drains accumulate across restarts); boot registers each channel key once, newest draining revision first, and instances pinned to any of the revisions remain resumable. No cleanup is required before a restart.

Checking what's live

sutra deployments list <dir> [--label KEY=VALUE]...   # ConfigMap/dir source: what's on disk

Against a running engine, GET /sutra/deployments/{id} is the authoritative live status — Active, Pending, Draining, or Failed with a reason.

Next

• Deployment model — why this is deterministic rather than eventually consistent.
• Instance migration — the sanctioned way to move an instance off the pin it is stuck on, so a draining deployment can finally retire.
• Reference: the sutra CLI — every flag on deploy/undeploy in full.

Instance migration

An instance is pinned to a deployment — by content hash — the moment it first parks, and every resume path resolves that pin fail-closed. That is what stops accidental version skew: a hot-deploy leaves the previous graph draining precisely so pinned instances keep resuming on the definition they started under, and a lost pin refuses rather than guesses.

It has one consequence with no escape hatch. An instance parked on a broken model stays parked on the broken model until somebody cancels it. Migration is the sanctioned way off the pin.

It is deliberately not a versioning mechanism. There is no in-process branch to author and no fleet of workers to label. There is one operator action, with a machine-readable report, that can be dry-run before it is done, refused with every violation listed rather than the first, and read back out of the audit journal afterwards.

The two endpoints

Both are on the gated admin surface (see Configuration reference), and both take the same shape of request: a target deployment, an optional node mapping, and the dryRun / resume / targetProcessId switches.

curl -sS -X POST "$ENGINE/admin/instances/$ID/migrate" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{ "targetDeploymentId": "dep-9f3c…",
        "nodeMapping": { "Approve": "ApproveV2" },
        "dryRun": true }'

Validation is the substance

The engine loads both graphs — the source from the deployment the instance is pinned to, the target from the active set — and derives every live locus from durable state: the snapshot's wait frontier, its routed start event, its retry attempt counters, and the instance's waiting rows.

Each locus is mapped through nodeMapping (identity where the map is silent), and the target node must exist and be compatible with what resume actually does there. That last phrase is the whole idea. Compatibility is not a BPMN type check; it is a check against the resume behaviour the locus will get:

Every violation is reported, not just the first. Fixing a mapping should take one round trip, not one per node.

Some findings are warnings rather than refusals:

• A completed node the target does not declare — not a live locus, so not fatal. But replay-as-done matches by id, so if the target still reaches that node it will be executed again. Dropping a node is legitimate model evolution, so this is loud rather than blocking.
• A frontier entry with no waiting row behind it — validated as a message wait and migrated anyway; the frontier is the resume-side truth and is never dropped.
• Pending outbox rows on the source — see What moves.

Dry run

dryRun: true mutates nothing: it takes no ownership claim and writes no row, and returns the same full report. Because nothing was locked while it was produced, a dry-run report is advisory — a concurrent resume can move the frontier out from under it. The report says so.

Node mapping is checked, not merely applied

A mapping entry naming a node the instance neither parks at nor has completed is refused (MAPPING_INVALID), not ignored. A typo must never read as "identity mapping, then".

Lifecycle rules

• The target must be active. A draining target is refused: a draining deployment retires the moment it is quiescent, so migrating onto one strands the instance again and costs a migration to do it. Unknown ids are the same refusal.
• The source graph must still be registered (active or draining). Quiescence gating means a live instance always keeps its deployment registered, so this only bites after a forced retirement — and it is a hard refusal rather than a fallback, because validating a continue-reply park against the wrong rule would pass the wrong check.
• Terminal instances are a validation error. Completed and terminated instances are retained history, and re-pinning history rewrites the record of where it ran.
• FAILED instances are migratable — and that is the prime use case.
• A no-op is refused (TARGET_SAME_AS_SOURCE) rather than silently rewriting a row for nothing.

Migration never auto-resumes — unless you ask

Migration re-pins and rewrites. Full stop. A FAILED instance stays FAILED under the new pin, and bringing it back is a separate, explicit decision — resumed: false is in every response so no caller has to infer it.

resume: true is that decision, and it closes the repair loop the operation exists for: fix the model, migrate the dead instance onto it, bring it back. On a successfully migrated FAILED instance it clears the failure state, re-stamps the snapshot suspended, and re-arms the parks the failure commit tore down — all inside the migration's own transaction, so a crash can never leave a half-revived instance.

It then comes back through the ordinary paths: a re-armed timer whose due instant has passed is claimed by the timer poller on its next tick; a message park waits for its next correlated inbound exactly as it would have. There is no privileged re-drive, which is exactly why the claim-guarded concurrency story below still holds afterwards. A burned <q:retry> budget stays burned — resume is not a retry reset.

On an instance that is not FAILED, resume: true is a validation error (RESUME_NOT_FAILED), not a no-op. A suspended instance is not stuck; it is parked, and it resumes on its own correlation or its own timer with no operator action at all. Reporting resumed: false for it would let a caller believe they had woken something.

What moves, and what deliberately does not

Coverage cursors travel verbatim, because they are keyed by declared path id rather than by node id — the node mapping must not touch them.

Encrypted variables ride through as ciphertext and still decrypt under the new pin. That is by construction rather than by luck: the authenticated-data binding for an at-rest value deliberately excludes the deployment id, precisely so that "a migration changes only the pin" stays true.

Concurrency: the claim, and why it bounces

A real run claims the instance first, through the same per-instance ownership machinery every resume path uses (see Replica semantics) — and under a distinct owner identity, so a migration can never slip past a resume this same replica has in flight. The claim is re-verified under the row lock inside the commit itself, closing the gap between "claimed" and "moved".

Contention is a retry-safe 409 (SUTRA.ADMIN.MIGRATE.CLAIM_HELD) with nothing read, rewritten or committed. A resume that starts after the migration claims bounces off it in turn.

The move itself is one transaction: snapshot, waiting rows, aliases, subject index and audit scope all land under the new pin together, or none do. No session ever observes the instance under both pins or under neither.

Batch migration

POST /admin/instances/migrate applies the same validation, the same compatibility matrix and the same node mapping to a filtered population.

Selection. filter.sourceDeploymentId is required — one migration names one source graph and one target graph, and a node mapping that is correct for one source is meaningless for another. There is deliberately no "every deployment" mode. Optional narrowing: processId, status (SUSPENDED or FAILED — the only two an instance can migrate in), includeTerminal, and limit (default 100, clamped to 1000).

Selection is ordered by instance id, then limited. Ordering by a timestamp that moves under a live population is exactly how a caller silently skips work.

Retained terminal rows are excluded unless includeTerminal asks for them — a busy deployment holds a whole retention window of finished instances, and a report where they crowd out the live ones is worse than useless. The flag exists so a caller who wants to know why an instance was not moved gets an explicit INSTANCE_TERMINAL refusal instead of a silent omission.

Every instance is its own transaction. One claim, one commit, one report entry. There is deliberately no batch-wide transaction — which is the point, and which makes the crash story trivial: a mid-batch crash leaves every instance either fully migrated or completely untouched, because that is already true of each instance's own commit and there is nothing larger to be half applied. Nothing about one instance can decide another's fate.

Per-instance outcome is a single enum, so no caller has to infer one from a combination of booleans:

The HTTP status describes the batch, not its instances. A run in which every instance refused still answers 200: the call was accepted, executed to completion, and reported in full — which is exactly what was asked. 400 is a malformed request, 422 a batch-level refusal (target not active; source and target the same pin), 503 no persistence. Scripts key on totals and on each entry's outcome, never on the status line.

Contention is reported, never retried. Retrying inside the call would make an admin request's runtime unbounded — the claim is held by another replica's work, on that work's timescale, not yours — and would turn the report into a claim about a moment that has already passed. Re-run the same request instead: the selector is deterministic and whatever moved is no longer under the source pin, so a re-run converges.

Two contradictions are caught from the request alone rather than N times over in the per-instance reports: resume with status: SUSPENDED (it selects exactly the instances resume refuses), and targetProcessId without processId (re-homing a mixed population into one process is never what anyone means, and one mapping could only be right for one of them).

Cross-process re-homing

targetProcessId re-homes the instance into a different process of the target deployment. Naming the instance's own process id is not a cross-process migration — being explicit must never change semantics.

Identity is never implicit across a process boundary. Every live locus must carry an explicit nodeMapping entry, or the migration is refused with CROSS_PROCESS_UNMAPPED — a different code from the ordinary unmapped-node refusal, because the danger is the opposite one: the id probably does exist in the target. Two processes that both declare Approve are not thereby the same Approve, and an accidental collision between unrelated graphs must never read as a deliberate mapping.

Everything else is unchanged. The same compatibility matrix is evaluated against the target process's graph; a target deployment that does not declare the named process is PROCESS_ABSENT, and its message names both the process asked for and the ones actually there. The snapshot's process id and the waiting rows' process id are rewritten together, because the timer poller reports it and the admin listing renders it — a row still naming the source process would describe the instance as living somewhere it no longer does.

One honest wrinkle: a cross-process move can carry a coverage cursor for a path the target process does not declare. It is inert, and remapping it would need a path mapping nobody supplied.

Status codes and diagnostics

All codes are under SUTRA.ADMIN.MIGRATE. — TARGET_NOT_ACTIVE, TARGET_SAME_AS_SOURCE, SOURCE_UNRESOLVABLE, PROCESS_ABSENT, NODE_UNMAPPED, NODE_INCOMPATIBLE, MAPPING_INVALID, INSTANCE_TERMINAL, CROSS_PROCESS_UNMAPPED, RESUME_NOT_FAILED, CLAIM_HELD, ALIAS_CONFLICT, COMMIT_FAILED — plus the warnings named above. CLAIM_HELD is the admin twin of the resume paths' own, and carries the same retry-safe posture.

A worked repair loop

1. An instance fails. It is FAILED, retained, and blocking its deployment's retirement.
2. Fix the model; package and deploy it. The new deployment becomes active; the old one drains.
3. Dry-run the migration against the new deployment id. Read the report; supply nodeMapping entries for anything renamed until it validates clean.
4. Run it for real with resume: true.
5. The instance comes back through the ordinary timer/correlation paths. Once it and its siblings finish, the old deployment goes quiescent and retires on its own.

For a population, do steps 3–4 with the batch endpoint and a filter.sourceDeploymentId of the old pin, then re-run until totals shows no BOUNCED.

Next

• Deploy, hot-deploy, and rollback — how the pin and the draining tail arise in the first place.
• Replica semantics — ownership claims and durable FAILED.
• Retries, history, and schedules — the most common way an instance gets to FAILED in the first place.
• Migration internals: the design reasoning — the two-scope commit, locus derivation, and the re-arm set.

Logging and audit

Three separate surfaces, deliberately not conflated: engine logs, CLI logs, and the audit trail. Each has its own destination and its own purpose. None of the three is collected by Sutra itself — see No telemetry, no phone-home for that guarantee; everything below stays on the host (or goes to a destination you configure) unless you point it somewhere else.

Engine logs

Structured JSON on stdout, always, with no configuration required. The field shape (timestamp / level / loggerName / message / service.name, plus traceId/spanId inside a sampled span) is stable, so a log-processing pipeline can key off it directly. RUST_LOG filters verbosity using standard tracing EnvFilter syntax (default info) — e.g. RUST_LOG=sutra_channels=debug,sutra_engine::deploy=trace to raise one module without drowning in the rest. See Troubleshooting BPMN solutions for reading these logs alongside traces and the audit trail when tracking down one message's path through the engine.

When an OTLP endpoint is configured (see Configuration reference), the same log records additionally export over OTLP — stdout is never replaced, only supplemented.

CLI logs

The sutra CLI writes its own logs to stderr, opt-in via -v / -vv / -vvv (info / debug / trace) — never mixed into stdout, which is reserved for report output (text or --format json). This is why every command in the CLI reference is safe to pipe: sutra describe my-process.bpmn --format json | jq . never has a stray log line corrupt the JSON.

Audit sinks

The audit trail is a separate, compliance-oriented record of what a process instance did — every INSTANCE_STARTED/INSTANCE_COMPLETED/INSTANCE_SUSPENDED/INSTANCE_RESUMED event, validation outcomes, and relay/resume decisions — independent of the telemetry pipeline described in Observability. Three sinks, any combination of which can be active at once:

Replaying an instance's audit trail offline

sutra audit-replay <instance-id> --from-jsonl <path-to-file-or-dir> [--tenant <id>] [--until <EVENT_TYPE>]

Walks the JSONL audit stream for one instance id and prints its events in order — useful for reconstructing what a specific production instance did without needing direct database access. --until stops replay after a given event type (e.g. INSTANCE_COMPLETED).

Sensitive data never appears in the clear

A variable tagged sensitive (via q:variables, see The q: namespace) or read from a dataClass-tagged data store (see Data stores) is redacted on every one of the surfaces above — audit sinks, structured logs, and traces alike — by the same redactor mechanism described in Domain neutrality and the SPI model. The flow itself still sees the real value; only what the engine emits is masked.

Next

• Observability — the telemetry (trace/metric/log) side this page's audit trail is deliberately kept separate from.
• Troubleshooting BPMN solutions — putting logs, traces, and the audit trail together to answer "what happened to this message?"

Troubleshooting BPMN solutions

This page is for the person building and running a Sutra solution — a deployment package that isn't behaving the way you expect. (If instead you're debugging the engine itself — a crash, a suspicious executor code path, a failing test in the workspace — see Debugging the engine, which is written for contributors.)

Every command below is read against rust/crates/sutra-cli/src/commands/ — what's described is what the code actually does, not an aspirational surface.

Start with sutra lint

sutra lint packages/my-app

Runs the exact fail-closed validation suite sutra package runs before sealing an archive, and prints nothing on success. This is always the first thing to run — most "why doesn't this deploy" questions are answered here, before you ever touch a running engine.

Reading a SUTRA.* diagnostic code

Every rejection, warning, and structured log line the engine emits carries a code shaped like SUTRA.<CATEGORY>.<REASON>. The category tells you which layer to look at:

--format json on any CLI command that reports diagnostics gives you the code, severity, and any attributes as structured data — pipe it to jq rather than screen-scraping text.

Two families get their own sections below, because their codes are read in groups rather than one at a time: codec diagnostics and data-store diagnostics.

Codec diagnostics: parse and schema failures

Whichever codec a channel is bound to, a decode lands on one of three outcomes, and it is that contract — not any codec's own vocabulary — that the rest of the engine reacts to:

• OK — a payload, no issues.
• SOFT_ERRORS — a usable payload plus structural issues. The instance still starts and still runs; the issues are data your process can branch on. This is the outcome people are most often surprised by, and it's deliberate: a schema violation is a business decision to route, not an exception to catch.
• FATAL — no payload at all. The engine rejects the inbound per the channel's configured posture.

Every issue carries the same five slots — code, severity (ERROR / WARNING / INFO), path (JSON-Pointer-shaped into the payload, or the source location for a schema violation), message, and value (the offending value, or the reason code a validator wants surfaced). They reach your process as <source>.validation (the full list) plus a frozen validation summary: outcome, tier (structural / content), firstReasonCode, firstIssue, and issues. <q:onValidation mode="route"/> is what hands that to your own gateway instead of short-circuiting — see The q: namespace.

The codes this engine emits

The middle segment of a SUTRA.PARSE.* code names the layer that produced it, which is the fastest way to separate a malformed-bytes problem from a schema problem:

Deploy time is a separate, earlier, fail-closed gate: SUTRA.CONFIG.SCHEMA.INVALID (a schema that doesn't compile), SUTRA.CONFIG.CODEC_MANIFEST.MISSING / .INVALID, and SUTRA.CONFIG.CODEC_LAYOUT.INVALID (a loose file, mixed kinds, or an empty codec folder under schemas/) all reject the archive rather than deploy a codec that would validate less than it claims. sutra lint reports every one of them before you deploy anything — which is why it's the first command on this page.

A code from a codec you added

An extension codec (see Channels and transports) claims its own middle segment — SUTRA.PARSE.<STANDARD>.* and SUTRA.VALIDATE.<STANDARD>.* — and documents its own set; the engine neither interprets nor rewrites them. What the engine guarantees is everything above: the three-way outcome, the five issue slots, and the same surfaces. So an unfamiliar code reads exactly the way the table does — the middle segment tells you which codec owns it, and that codec's own documentation tells you what it means. Wherever it came from, it reaches you the same three ways: the validation variables in the process, --format json diagnostic output, and the audit trail, so Tracing one message end to end applies unchanged.

Data-store diagnostics: projected stores

A store that declares a structure: block is verified at package time against the store's own migrations — no database connection, no credentials. A store with no structure: block raises none of these codes at all; it stays the opaque key→JSON store it always was.

The posture is three-state, and reading it correctly saves a lot of time: a definite fault is an error, an unprovable one is a warning worded as unprovable, and a projection that matches raises nothing. Only errors fail the command — a warning here is information, not a gate.

A structure block pointing at a schema or a type the package doesn't declare is reported as SUTRA.CONFIG.DATASTORE.INVALID — the same code any other malformed store declaration uses — not as one of the projection codes above.

DDL_UNVERIFIABLE is not an error, and shouldn't be read as one

… could not be fully parsed — the statement CREATE OR REPLACE FUNCTION f( … is outside the DDL subset this lint parses — so the effective table shape was not derived and the declared structure was not verified; no column diagnostic is raised for this store (it may be valid; it is simply not provable here)

Lint replays a package's migrations/<store>/V*.sql through a deliberately small SQL subset — CREATE TABLE, ALTER TABLE ADD/ALTER/DROP COLUMN, and the key-bearing constraints, across the three shipped dialects' spellings. Real migrations routinely contain more than that: a PL/pgSQL trigger body, a T-SQL procedural guard, a table created by an operator outside the package. When the parser meets something it doesn't model, it stops trying to verify that store and says so. It never converts that into a column error, because a linter that cries wolf on legitimate DDL is one authors learn to ignore.

The same wording covers the narrower cases, and they're worth telling apart:

• The store's shape wasn't derived at all — an out-of-subset statement, no migrations shipped for the store, the table created elsewhere, or several candidate tables and none named after the store (name the projected one with sql: table: <name>).
• Individual fields couldn't be compared — the column's type is outside the set lint compares (JSONB, UUID, a domain type), or the declared type has no ruled column mapping. The column exists; only the type comparison is withheld, aggregated into one line per store.
• The declared schema isn't an enumerable XSD — an engine-provided codec (its type set is open), or a JSON-Schema / schema-bundle codec folder. JSON Schema carries no length or precision information, so there is genuinely nothing to compare against here — but don't read the warning as "unverified for now, deploy anyway": a JSON-Schema-declared structure: isn't merely unverified, it is refused outright when the engine resolves the store at deploy time (see Typed columns). Declare the structure against an XSD if you want the store to exist at all, not just to have its column types checked.

Note that degradation is per store, not per statement: one out-of-subset statement anywhere in migrations/<store>/ withholds verification for that store's whole table. So the way to get the warning to go away is to keep the store's own DDL inside the subset (plain CREATE TABLE / ALTER TABLE, with procedural setup living in a different store's schema or outside the package), or to give lint the missing pointer — sql: table: — when the shape is parseable but ambiguous. If neither is worth it, the warning is a fair thing to live with: first-use verification against the live table still fails the store closed if the projection genuinely doesn't hold.

Runtime codes: UNDECLARED_FIELD, VALUE_NOT_A_RECORD, PROJECTION_UNSATISFIABLE

Three runtime codes ride a projected store's own operations. Where to look for them matters: unlike SUTRA.RUNTIME.DATASTORE.CONFLICT, which is the failed instance's own diagnostic code, these three travel inside the message of a SUTRA.RUNTIME.UNEXPECTED diagnostic. So filter on the message text, not the code, when you go looking:

A silent partial write is the worse outcome in every one of these, so all three are deliberately loud rather than best-effort. SUTRA.CONFIG.DATASTORE.COLUMN_NAME_INVALID, in the table above, is this family's plan-time member: a declared field folding onto a control column name is refused before the store is ever served — both lint and the engine's own plan-time resolution run the same rule, sourced from one shared constant, so the two sides cannot drift apart.

The inspection commands

None of these execute anything — they read a BPMN file (or a running engine) and report. Use them in roughly this order when a process isn't routing, resuming, or replying the way you expect.

sutra describe — what does the engine see in this file?

sutra describe bpmn/transfer.bpmn

A structural summary: processes, start/end events, every service/user task (with its implementation, codec, validator, and redactor refs), gateways, channel sources, and reply refs. Read-only — it never invokes the engine's real loader, so it's safe to run against a file you're mid-edit on. --format json for scripting.

sutra dispatch-graph — visualize the routing

sutra dispatch-graph bpmn/transfer.bpmn --format mermaid

Emits a graphviz dot or mermaid diagram of the file's dispatch tree — nodes are BPMN elements, edges are sequence flows. Paste the mermaid output straight into a markdown viewer to see the shape of a flow you didn't author, or to sanity-check q:dispatch/q:case routing before review.

sutra simulate --dry-run — will this channel actually reach my process?

sutra simulate bpmn/transfer.bpmn --channel transfer-request --dry-run

Resolves which process (and start event) a named channel would route to, from the file's own q:source declarations — no execution, --dry-run is required. If the channel name doesn't match any declared source, the error lists every channel the file does declare, so a typo is obvious immediately. This is the fastest way to answer "why isn't my message reaching the process I think it should" without deploying anything.

sutra explain — evaluate a FEEL expression in isolation

sutra explain 'fromAccount.frozen or toAccount.frozen or fromAccount.balance < payload.amount'
sutra explain --context vars.txt 'payload.amount > 100'
sutra explain   # no expression: a REPL on stdin, `:quit` to exit

Built directly on sutra-feel — the same evaluator the engine runs, standalone. --context takes a flat key: value (or key=value) file (values coerce to boolean/number when they parse as such, else stay strings). Use this the moment a gateway takes the branch you didn't expect: paste the exact condition in, supply the variables you think are in scope, and see what FEEL actually does with them — usually a type coercion or a missing-path surprise, not a logic bug.

sutra coverage check — is my compliance path actually being exercised?

sutra coverage check bpmn/transfer.bpmn                       # drift lint: declared paths still valid?
sutra coverage check --archive deployed.sutra \
    --database-url "$SUTRA_DB_URL" --threshold 100             # correlation-aware, store-backed check

The bare form is a drift lint: every q:coverage declaration must still be an ordered subsequence of a real route through the current flow graph — it catches a path declaration that silently broke when someone reworked the diagram. The --archive form reads the deployment's seeded coverage flags out of the database its coverage store declares (--database-url names it — any of the three shipped dialects), and fails closed below --threshold — the gate a CI pipeline runs after a test campaign. See The q: namespace for what q:coverage declares and the money-transfer worked example for it end to end.

Tracing one message end to end

Three surfaces, used together:

1. The audit trail — the definitive record of what an instance did: INSTANCE_STARTED → node-by-node progress → INSTANCE_COMPLETED / INSTANCE_FAILED / INSTANCE_SUSPENDED / INSTANCE_RESUMED. If you have a JSONL audit sink configured (see Logging and audit), replay one instance's history offline:

   sutra audit-replay <instance-id> --from-jsonl /path/to/audit --until INSTANCE_COMPLETED
   

2. Traces — if OTel is configured (see Observability), every span carries bpm.instance.id; filter your trace backend on that id to see the sutra.dispatch / sutra.decode / sutra.validate / sutra.execute span chain for exactly that instance, across every resume segment if it went through a wait state.
3. The admin API — against a live engine, GET /admin/instances/{id} (or GET /admin/instances/by-alias/{key}/{value} if you only have the business correlation key, not the instance id) shows the instance's current state directly. POST /admin/instances/{id}/cancel is the one instance-level control operation exposed — there is no generic "retry" endpoint; retrying means fixing whatever rejected the input and re-sending it (or, for a parked wait, sending a corrected relay message).

Common failures, worked

"My message got a 200/202 but nothing happened downstream." Check ack-mode first (see Acknowledgement modes) — on-persist on an HTTP channel means the 202 you got back is only proof of durable receipt, not completion; the actual reply (if any) rides an outbound channel, not the original connection. Then check the audit trail for the instance.

"The message never became an instance at all." Run sutra simulate --channel <name> --dry-run against the deployed BPMN first — a messageTypeValue/messageTypePattern mismatch on the q:source is the most common cause, followed by the channel's codec rejecting the payload outright (SUTRA.INBOUND.CODEC_NOT_FOUND / a FATAL DecodeResult — check sutra describe for what codec the channel is actually bound to).

"A relay message didn't resume my parked instance." SUTRA.RUNTIME.RELAY.CORRELATION_NOT_FOUND in the audit trail means the relay's q:alias expression didn't match any parked instance's alias — run sutra explain with the relay payload's variables against the exact alias expression from the BPMN to see what key it actually produces, and compare it to what the original request set.

"A gateway takes the wrong branch." sutra explain the condition expression directly, with a --context file built from the actual variable values at that point (read them off the audit trail or a trace span if you're not sure) — this isolates a FEEL semantics surprise from a "my BPMN is wired wrong" problem in one step.

"My coverage report shows a path that should be covered as uncovered." Confirm the flows listed on the q:coverage declaration are still the exact ids in the current diagram (sutra coverage check bpmn/your-process.bpmn, the drift-lint form) before assuming the flow genuinely isn't being exercised.

"Every request answers SUTRA.RUNTIME.UNEXPECTED — engine actor is not running." An execution lane died — something panicked outside the per-dispatch containment, which in practice means the lane's own build/rebuild failed (the boot log has the panic). The process is a zombie for that lane's share of the key space and will not heal in place: both health probes report it (GET /sutra/health/live → 503 with data.deadLanes; /ready goes DOWN at the same moment), so an orchestrator with a liveness probe restarts the replica automatically. If you run without probes, restart it yourself and read the boot log for the panic that killed the lane.

"My coverage op now fails with SUTRA.CONFIG.COVERAGE.STORE_MISSING." Coverage marks are persisted in the coverage data store the deployment declares in datastores.yaml, so the op fails whenever that store isn't there to write to — rather than answering 0%, because a fabricated 0% is indistinguishable from a real measurement of nothing covered. Two causes, both named in the message: the deployment declares <q:coverage> paths but no coverage store (a package-shape requirement sutra lint errors on with this same code — declare the store; you supply no coverage SQL, since the engine owns that schema and applies it on first use), or it declares one whose connection could not be opened (the boot log carries an error naming that deployment — fix the URL or the environment reference it resolves from). The same code with a different message means the store opened but the read failed; the underlying error is in the message. See Coverage.

Next

• Reference: the sutra CLI — every flag on every command above.
• Debugging the engine — the contributor-facing counterpart, for when the problem turns out to be in the engine rather than in your BPMN/rules/config.

sutra CLI reference

The single sutra binary (crate rust/crates/sutra-cli) carries the whole authoring-to-operations toolchain: scaffolding, validation, packaging, deployment, and analysis. Build it from source:

cd rust && cargo build --release -p sutra-cli    # -> rust/target/release/sutra
cargo run -p sutra-cli -- <args>                  # or run ad hoc

Commands are grouped below by workflow: author → validate → package → deploy → operate → analyze, with the generators last.

Global conventions

Database-touching commands (migrate, coverage check/reset, crypto) share one set of connection options, each with an env fallback: --url (SUTRA_DB_URL), --user (SUTRA_DB_USERNAME), --password (SUTRA_DB_PASSWORD), and — for migrate — --schema (SUTRA_DB_SCHEMA). These are the CLI's own database-connection names, distinct from the engine's SUTRA_DATASOURCE_* — see Configuration reference.

Author

sutra create app

sutra create app <NAME> [--dir <DIR>]

Scaffolds an application workspace: a sample standalone deployment package under packages/<name>-main/ plus deploy assets (a compose file, a deployments drop-directory, a k8s manifest, a health-gated smoke script). Idempotent-safe — existing files are never overwritten. See Your first app.

sutra create deployment

sutra create deployment <NAME> [--dir <DIR>] [--from <PACKAGE_DIR>]

Scaffolds a fresh standalone deployment-package skeleton, or copies an existing package with --from — the explicit variant model: packages never inherit, a variant is a copy. See Deployment packages.

sutra create bpmn

sutra create bpmn <PROCESS> [--package <PACKAGE_DIR>] [--validation fatal|soft] [flags]

Generates a process with the validation-gateway wiring (plus accepted/rejected reply templates) into a package, verified through the engine's own BPMN loader before being written.

Validate

sutra lint

sutra lint <PACKAGE_DIR>

Runs the full package-time validation suite (the same fail-closed checks sutra package runs) and emits nothing on success. The fast pre-flight before every package.

sutra describe

sutra describe <BPMN_FILE> [--format json]

Prints a structural summary of a BPMN file: processes, events, tasks, gateways, channels. See Troubleshooting BPMN solutions.

sutra dispatch-graph

sutra dispatch-graph <BPMN_FILE> --format dot|mermaid

Emits a graphviz or mermaid diagram of a BPMN file's dispatch tree.

sutra simulate

sutra simulate <BPMN_FILE> --channel <CHANNEL> --dry-run

Reports which process an inbound channel message would route to. --dry-run is required — routing report only, no execution.

sutra explain

sutra explain [EXPRESSION] [--context <FILE>]

Evaluates a FEEL expression — one-shot, or a REPL on stdin when the expression is omitted. See Rules: DMN, FEEL, and .srl.

sutra compat-baseline

sutra compat-baseline --baseline <PATH_OR_REF> [--current <PATH>] [flags]

Compares current BPMN signatures against a baseline directory or git ref and reports breaking changes — a CI gate on process-contract compatibility between releases.

Package

sutra package

sutra package <INPUT> [-o|--out <DIR>]

Seals a deployment-package directory into one immutable .sutra archive, running the full validation suite fail-closed first. The archive manifest (per-file digests, the content-addressed deployment id) is derived, never authored. See Deployment packages.

sutra deployments list

sutra deployments list <DIR> [--label KEY=VALUE]...

Inspects a directory of sealed .sutra archives — lists each archive's deployment id and labels.

sutra openapi

sutra openapi <ARCHIVE>

Emits a sealed archive's generated OpenAPI 3.1 surface (channels → BPMNs, message types, endpoint nature, data stores) — the same document the engine serves live per deployment id.

Deploy

sutra migrate

sutra migrate [status|verify] [--url <URL>] [--dry-run] [flags]

Applies engine schema migrations to the engine-internal database, or inspects them (status — applied vs. pending; verify — ledger integrity, expected head, checksum drift).

sutra crypto provision-dek

sutra crypto provision-dek --key-id <KEY_ID> --kek <REF> [--url <URL>] [flags]

Provisions a KEK-wrapped per-tenant data-encryption key for envelope encryption of sensitive instance variables at rest.

sutra deploy

sutra deploy [ARCHIVE] [flags]

Hot-deploys a sealed .sutra archive onto a running engine. See Deploy, hot-deploy, and rollback and Deployment model for the full mechanics.

sutra undeploy

sutra undeploy <DEPLOYMENT> [flags]

Removes a deployment — the engine drains it (no new intake) and retires it at zero instances and zero pending outbox.

Operate and analyze

sutra coverage

sutra coverage init  <FILE> [PROCESS_ID]... [flags]   # seed declarations / scaffold admin set
sutra coverage check [BPMN_FILE] [flags]              # drift lint, or the store-backed check
sutra coverage reset --archive <FILE> [flags]         # re-seed the store covered=false

Path-coverage tooling for q:coverage route declarations (intra-process) and coverage/*.yaml files (cross-process) — see Coverage: declared routes as the compliance signal for the full walkthrough of both shapes and every flag above, and Troubleshooting BPMN solutions for reading a report that doesn't match what you expected.

sutra test simulate

sutra test simulate --deployments <DIR> --datasource <URL>
                    (--advance <DURATION> | --until-quiescent) [flags]

Boots a real engine on a dynamic port against a directory of sealed deployment archives with a virtual clock installed, fast-forwards it, reports, and shuts down — so a PT24H timer or an R3/PT12H schedule settles in wall-clock seconds. Unrelated to sutra simulate above, which is a dry-run routing report over one BPMN file and boots nothing.

Exactly one of --advance / --until-quiescent is required. Safety: the target database must hold no instances or the run refuses with exit 2 — pointing this at a database with real in-flight instances would durably fire their real timers early; --allow-existing-data is the explicit acknowledgement.

Progress is text on stderr; the final summary is one JSON object on stdout and nothing else touches stdout, so sutra test simulate … | jq . is always safe. See Testing time for the summary's fields and the embedded seam behind this command.

sutra audit-replay

sutra audit-replay <INSTANCE_ID> --from-jsonl <PATH> [--tenant <TENANT>] [--until <EVENT_TYPE>]

Walks a process instance's audit events from a JSONL stream — reconstructing what a specific instance did, offline. See Logging and audit.

sutra version

sutra version            # sutra 0.2.0-rc.1
sutra --version          # identical text
sutra version --format json

Prints the tool version. The program name is derived from the running binary, not hardcoded: a distribution that embeds this CLI as a library (sutra_cli::run) under its own binary name prints that name, and one that versions itself independently of the engine (sutra_cli::run_with_version) prints its own version with the embedded engine's underneath:

<tool> 2.0.0
sutra  0.2.0-rc.1 (engine)

--format json is the structured form and always separates the two — version is the reporting tool's own, engine the embedded engine's (equal for this binary):

{"name":"sutra","version":"0.2.0-rc.1","engine":"0.2.0-rc.1"}

Generate

sutra docgen

sutra docgen --input <FOLDER> [--output <DIR>] [--check]

Recurses a folder of authored deployment artifacts — BPMN processes, DMN/.srl rules, Handlebars/XSLT templates and their manifests, channels.yaml, package.yaml, coverage files — and emits a deterministic markdown catalog, one page per artifact. It parses through the engine's own loaders, so each page describes exactly what the engine loads rather than a second parser's opinion of it. --check generates into a temporary directory and reports drift instead of writing anything, which is the shape a CI or pre-commit gate wants. sutra catalog is its sibling for Rust source — same --output / --check contract, one page per source file, rooted at --repo-root.

sutra schemagen

sutra schemagen generate <SCHEMAS_DIR> <OUT_DIR> [--full]
sutra schemagen check    <SCHEMAS_DIR> <TREE_DIR> [--full]

Compiles a directory of XSD schemas into Rust sources: the decode tables, the canonical map projection, and the shape metadata a schema-bound codec is built on. The schema files are the only input, and emission is byte-identical run to run after rustfmt — which is what makes check (regenerate in memory, diff against a committed tree, exit 1 on any difference) a usable drift gate. The default emission is the slim, data-driven form; --full additionally emits the typed model. generate writes only the files the generator itself produces and never touches hand-maintained ones alongside them.

Both paths are arguments, not conventions: schemagen is a neutral tool over whatever corpus you point it at, and the crate it emits lives wherever the caller wants it — including in a repository that composes this one.

Crates: sutra-feel and sutra-dmn

Every crate in the workspace is unpublished by default (publish = false at the workspace level) except two, which are deliberately carved out as standalone, embeddable libraries: sutra-feel and sutra-dmn. Both are pure Rust, #![forbid(unsafe_code)], and have no dependency on BPMN, channels, or persistence — you can use either in a project that has nothing to do with workflow engines at all.

[dependencies]
sutra-feel = "0.2.0-rc.1"
sutra-dmn  = "0.2.0-rc.1"   # depends on sutra-feel directly; use it alone if you only need DMN

sutra-feel — the FEEL expression language

FEEL (Friendly Enough Expression Language) is the expression language the OMG DMN specification defines, reused throughout the BPMN/DMN tooling ecosystem wherever a small, side-effect-free expression needs to be embedded in a larger model — a decision rule, a gateway condition, a data-mapping assignment. sutra-feel is a complete, standalone implementation: lexer, parser, AST, and evaluator, with no dependency on DMN or BPMN machinery.

#![allow(unused)]
fn main() {
use sutra_feel::{expressions, FeelContext, FeelValue};

let mut context = FeelContext::new();
context.insert("age".to_string(), FeelValue::Number("42".parse().unwrap()));

let result = expressions::eval("age >= 18", &context).unwrap();
assert_eq!(result, FeelValue::Boolean(true));
}

expressions::eval is the main entry point; expressions::parse parses without evaluating, and expressions::eval_boolean is a convenience wrapper for expressions that must produce a boolean (gateway-condition shaped callers). expressions::paths extracts which context paths an expression reads without evaluating it — useful for static analysis over a set of expressions.

Highlights:

• Source-position-aware diagnostics (errors carry line/column, pinned to a caller-supplied source URI) — the errors this crate raises are the same ones you'd see from sutra explain (see the CLI reference) or from a deploy-time validation failure.
• DECIMAL64 numeric semantics (16 significant digits, HALF_EVEN rounding, via bigdecimal) — arithmetic matches the FEEL specification, not native floating point.
• Full temporal support: dates, times, date-and-time, and durations, including IANA-timezone-aware @Region/City zone-qualified literals, with the timezone database bundled at compile time (no host tzdata dependency).
• Ranges/intervals, contexts (maps), lists, and function values, plus a documented determinism denylist for expressions that must stay side-effect-free (the engine uses this to keep replay deterministic across a wait-state resume — see Wait states and human tasks).

sutra-dmn — the DMN 1.5 decision engine

DMN (Decision Model and Notation) is the OMG standard for decision tables, decision requirement graphs (DRGs), and business knowledge models, tied together by FEEL. sutra-dmn loads .dmn XML files, validates them, and evaluates them — single decision tables (all seven OMG hit policies: UNIQUE, ANY, PRIORITY, FIRST, OUTPUT ORDER, RULE ORDER, and COLLECT with SUM/COUNT/MIN/MAX aggregation) as well as full DRGs, where a decision depends on other decisions, invokes business knowledge models, or calls decision services. FEEL parsing and evaluation is delegated directly to sutra-feel.

#![allow(unused)]
fn main() {
use sutra_dmn::DmnDecisionEngine;
use sutra_feel::{FeelContext, FeelValue};

let engine = DmnDecisionEngine::new();
let result = engine.evaluate("tier.dmn", TIER_DMN_XML.as_bytes(), &input).unwrap();
assert_eq!(result.get("tier"), Some(&FeelValue::from("GOLD")));
}

DmnDecisionEngine::evaluate is the single-file entry point, returning a map of output-clause name to FeelValue — this is exactly what backs a businessRuleTask in the engine (see Rules: DMN, FEEL, and .srl). For DRG evaluation across a graph of decisions/BKMs/decision services with imports, see the crate's drg module; for structural validation without evaluation, see DmnRulesetValidator.

Conformance is measured, not asserted — see DMN-TCK conformance for the current standing and what it does and doesn't cover.

Why these two and not the rest of the workspace

Everything else in the workspace — the BPMN model and executor, the channel/transport layer, the persistence layer, the CLI — is Sutra-engine-internal: coupled to the engine's own conventions (diagnostics, deployment packages, the q: namespace) in a way that wouldn't make sense as a general-purpose crate. FEEL and DMN evaluation, by contrast, are useful on their own to anyone building rules or decision logic in Rust, independent of whether BPMN or Sutra is anywhere in the picture — so those two are published, versioned, and supported as standalone libraries.

Next

• DMN-TCK conformance — the numbers behind "measured, not asserted."
• Rules: DMN, FEEL, and .srl — how these two crates show up inside a Sutra process.

DMN-TCK conformance

Sutra's DMN/FEEL evaluator (sutra-dmn + sutra-feel, see Crates: sutra-feel and sutra-dmn) is checked against the real OMG DMN Technology Compatibility Kit — the same corpus other DMN implementations are measured against — rather than relying on an internal test suite alone.

Current standing

The gap between "100% of attempted" and "99.4% absolute" is a small, enumerated set of constructs the evaluator deliberately doesn't attempt yet — chiefly reflective execution of {java|pmml: …} external function bodies (the grammar parses and validates their binding shape; invoking one returns a clear, deliberate semantic rejection rather than a wrong answer). This is the honest distinction the numbers are built to preserve: a wrong value on a supported construct is a real conformance defect, and the harness that produces these numbers is built specifically to never launder one into an "unsupported" bucket. Level 2 has no such gap at all.

What this does and doesn't tell you

What it tells you: decision tables (all seven OMG hit policies, including COLLECT with aggregation), decision requirement graphs, business knowledge models, decision services, and the breadth of FEEL itself (temporal types and arithmetic, ranges/intervals, list/context comprehensions, string and numeric builtins, filters and projections) behave per specification across a large, independently-authored corpus — not just the cases Sutra's own contributors thought to write.

What it doesn't tell you: conformance to a language specification says nothing about Sutra's BPMN coverage, its channel/transport behavior, its persistence and replica semantics, or its security posture — those are covered elsewhere in this book (see Building BPMN solutions and Architecture). It also isn't a performance claim — conformance and throughput are separate properties.

Where this comes from

The conformance measurement is produced by a development-time harness in the workspace — it is a tool contributors run when working on the FEEL/DMN evaluator, not something a user building a BPMN solution ever needs to invoke. See Debugging the engine if you're contributing to sutra-feel or sutra-dmn and need to reproduce or extend this measurement yourself.

Next

• Rules: DMN, FEEL, and .srl — using DMN inside a Sutra process.
• Crates: sutra-feel and sutra-dmn — using the same evaluator standalone.

Contributing

Contributions are welcome! This page covers the repo map, the day-to-day dev workflow, the code style and PR bar, and the license. If you're an AI agent working on this codebase, also read CLAUDE.md at the repository root — it's a shorter, more mechanical companion to this page written specifically for that audience.

Repository map

The engine, tooling, and libraries are a single Cargo workspace under rust/:

Within rust/crates/, the layering described in Engine layering maps directly onto directories — sutra-engine (the library), sutra-dist (the composition root that builds the actual binary), sutra-channels, sutra-executor, sutra-bpmn, sutra-feel/sutra-dmn/sutra-srl/sutra-templates, sutra-persistence, sutra-formats (the built-in formats) alongside sutra-codec-schema, one sutra-transport-<vendor> crate per broker, and the SPI crates each of those builds against (sutra-codec-spi, sutra-transport-spi, sutra-datastore, sutra-envref-spi, sutra-redactor-spi) — see Domain neutrality and the SPI model for exactly how the pieces fit and how to add a new one.

Development workflow

You need a stable Rust toolchain (rustup); Docker is only needed for the container/integration test tiers, and tofu plus a kind cluster only for the Kubernetes tier. make help (from the repo root) lists every target; full detail lives in rust/TESTING.md.

cd rust && cargo build                 # whole workspace, debug
cargo build --release -p sutra-cli     # the `sutra` CLI binary
cargo build --release -p sutra-dist    # the `sutra-engine` binary (the composition root)

Test tiers

The CI workflow (.github/workflows/ci.yml) mirrors the local gate chain exactly — cargo fmt --all --check, make lint, make test, make audit — so a contributor who is green locally is green in CI. Tier-2 runs nightly rather than per-PR (a Docker-heavy suite is slow and resource-flaky on a small hosted runner for signal tier-1 already covers on most changes); tier-3 stays a local/milestone gate, since it needs a provisioned kind cluster CI doesn't have. CodeQL and Trivy scans run on their own schedules alongside CI.

Tier-1 covers the whole workspace — no crate is held out of it, so a green make test means the whole workspace is green, not a subset of it.

Lint and supply-chain gates

make lint    # cargo clippy --workspace --all-targets -- -D warnings,
             #   plus the sutra-archtest domain-neutrality suite
make audit   # cargo audit (RustSec advisories) + cargo deny check (rust/deny.toml)

make lint failing on a domain-neutrality violation means a business term landed in a crate the gate enforces — move the concrete logic into its own extension crate instead.

The generated catalog (on demand, not committed)

make catalog         # regenerate the artifact-documentation catalog under catalog/
make catalog-check   # verify it's in sync

The catalog — one page per source file plus its dependency relationships, produced by sutra-catalog-gen — is generated on demand and git-ignored; there's no committed baseline to diff against, so it isn't part of the CI gate today. Run it locally when you want the dependency picture for a change you're making; make install-hooks wires an optional pre-commit regeneration hook if you want it kept fresh automatically.

Code style and the PR bar

1. Discuss anything large first. A new transport, a new codec, an architectural change — open an issue before you invest the time.
2. Keep PRs focused. One logical change reviews faster than a bundle of unrelated ones.
3. Be green locally before pushing:

   cd rust
   cargo fmt --all
   cargo clippy --workspace --all-targets -- -D warnings
   cargo test  --workspace
   

4. Add a CHANGELOG.md entry under [Unreleased] for a user-visible change.
5. Commit messages are short, imperative, area-prefixed:

   feat(channels): add AMQP 1.0 outbound reply path
   fix(feel): correct DECIMAL64 rounding on division
   docs(concepts): clarify wait-state correlation keys
   

The zero-warning clippy bar and the domain-neutrality gate are both hard requirements, not style preferences — see Domain neutrality and the SPI model for why the latter exists.

Where designs live

Deeper design rationale than this book covers — why a mechanism was built the way it was, staged plans for work in flight — lives in docs/design/ in the source (private working) repository this public repo is curated from; a subset is rewritten into the chapters of this book as it stabilizes, rather than published wholesale. If you're contributing a change whose "why" isn't already captured in this book, a short design note alongside your PR is the right place for it.

License

Sutra is dual-licensed under MIT OR Apache-2.0 (LICENSE-MIT, LICENSE-APACHE). Unless you state otherwise, any contribution you submit is licensed under the same terms, with no additional conditions — you retain copyright over your own contributions.

Please also read our Code of Conduct and Security Policy.

Next

• Debugging the engine — tracing targets, reproducing a bug at the right test tier, and the development-only tools (like the DMN-TCK harness) that live alongside the workspace.

Debugging the engine

This page is for contributors working on the engine itself — its executor, model, channel dispatch, or the FEEL/DMN evaluators. If you're building a solution on top of Sutra and something in your BPMN/rules/config isn't behaving, see Troubleshooting BPMN solutions instead — that page (and its CLI-based tools) is the one written for that audience.

Tracing targets

Every crate logs through tracing, and its module path is the target RUST_LOG filters on — so you can raise verbosity on exactly the layer you're chasing a bug through without drowning in noise from the rest of the workspace:

RUST_LOG=sutra_channels=debug,sutra_engine::deploy=trace cargo test -p sutra-channels
RUST_LOG=sutra_executor=trace cargo run -p sutra-dist

The default filter is info. When you're not sure which crate owns the behavior you're chasing, start from Engine layering — a message's path through the system (channel → dispatch → executor → persistence) is also the order to add targets in.

Reproduce at the lowest tier that can show the bug

The workspace's three test tiers (rust/TESTING.md; summarized in Contributing) aren't just a CI cost control — they're the natural escalation order for reproducing a bug:

1. Tier 1 first. Most bugs in the executor, the BPMN model, FEEL/DMN/.srl, or template rendering reproduce with a plain unit or no-docker integration test — no Postgres, no broker, nothing to spin up. Add it as a new module in the crate's tests/all.rs (a new file under tests/all/ plus one mod <name>; line — not a new top-level tests/*.rs, which would reintroduce a separate link unit) and run it in isolation:

   cargo test -p sutra-executor gateways_test::your_new_case
   

2. Escalate to tier 2 only if the bug needs real infrastructure — a specific PostgreSQL locking behavior, a RabbitMQ redelivery edge case, a replica-convergence race. Mark the test #[ignore = "docker"] and call sutra_testkit::reap_on_exit(container.id()) on whatever container handle you spin up, so the shared-fixture pattern doesn't leak it (see rust/TESTING.md's "Reaper" section for why that call is required rather than relying on Drop).
3. Escalate to tier 3 only if the bug is genuinely multi-replica or Kubernetes-specific — leader-election handoff, ConfigMap-driven activation timing, ingress-level deploy behavior. This tier needs a provisioned kind cluster (make -C deploy/k8s-it init) and is the most expensive to iterate on — reach for it last, and only once you've ruled out that a tier-1 or tier-2 test could show the same thing.

A bug fixed without a new failing test at the tier that caught it isn't really pinned — the next regression in the same area won't be caught until it reaches production again. Write the test first, watch it fail against the unfixed code, then fix it.

Catalog-guided navigation

make catalog          # regenerate catalog/ (git-ignored, generated on demand)

sutra-catalog-gen walks every crate and emits one markdown page per source file, plus a bidirectional relationships table — what a file depends on, and what depends on it. When you're about to change a shared type (an SPI trait, a diagnostic code, a config key) and need to know every call site that will need updating, this is faster and more complete than a plain grep: it's generated from the actual syn-parsed dependency graph, not a text search that can miss a re-export or an indirect reference.

The DMN-TCK harness — a development tool, not a user-facing command

The DMN-TCK conformance numbers this book cites come from a harness that ships in the workspace but is gated and external: the OMG TCK corpus itself is ASL-2.0-licensed and isn't vendored into this (unlicensed) repository, so it's never something a person building a BPMN solution runs. If you're working on sutra-feel or sutra-dmn:

git clone https://github.com/dmn-tck/tck.git   # or wherever your checkout lives
SUTRA_DMN_TCK_DIR=/path/to/tck cargo test -p sutra-dmn --test tck -- --ignored --nocapture

The harness classifies every assertion as PASS (matches expected), FAIL (a real conformance gap — the engine produced a different value), or UNSUPPORTED (the engine couldn't evaluate the construct at all — recorded, not counted as a failure). Two env-gated side outputs help when you're closing a gap: SUTRA_DMN_TCK_DUMP=<file> writes every non-pass outcome for gap analysis, and SUTRA_DMN_TCK_RESULTS_DIR=<dir> writes the official vendor-submission tck_results.csv/tck_results.properties pair. When you land an increment, re-run the harness and update the numbers in DMN-TCK conformance alongside your change — that page is a measurement, and a stale measurement is worse than none.

Next

• Domain neutrality and the SPI model — the structural rule most contributions have to respect, and the gate that enforces it.
• Troubleshooting BPMN solutions — the solution-developer counterpart to this page.