Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Retries, history, and schedules

Three execution-semantics features an author reaches for once a flow has to survive contact with an unreliable world: a per-task retry policy, execution history that outlives the instance, and the full set of BPMN timer schedules.

<q:retry> — a per-task retry policy

A <bpmn:serviceTask> may declare a retry policy inline. It applies to both kinds of service task: a registered task (a function the embedding application registered) and a channel-call task (implementation="channel:<name>" — the wait-state call every pure-BPMN deployment uses through the shipped binary).

<bpmn:serviceTask id="Score" implementation="registered:score">
  <bpmn:extensionElements>
    <q:retry maxAttempts="3" initialDelay="PT1S" backoffCoefficient="2.0"
             maxDelay="PT5M" nonRetryableCodes="SUTRA.TASK.VALIDATION"/>
  </bpmn:extensionElements>
</bpmn:serviceTask>
AttributeDefaultMeaning
maxAttemptsrequiredTotal invocation budget, including the first attempt.
initialDelayPT1SWait before attempt 2.
backoffCoefficient2.0Delay multiplier — attempt n+1 waits min(initialDelay × coefficient^(n-1), maxDelay).
maxDelayPT5MCeiling on any single backoff wait.
nonRetryableCodes—Structured codes that fail immediately, budget notwithstanding.

So maxAttempts="4" initialDelay="PT1S" backoffCoefficient="2.0" waits 1 s, then 2 s, then 4 s between its four attempts — each capped at maxDelay.

A retry wait is a durable park, never a sleep

This is the load-bearing property. A failed attempt with budget remaining persists the instance with the failed task still pending and an armed timer due at the backoff instant; the ordinary timer poller re-drives it when that instant arrives.

Nothing sleeps, and nothing is held in memory. The consequences are the ones you want:

  • the backoff survives a restart, a rolling upgrade, and a hot-deploy (a retry park is pinned to its deployment like every other park);
  • no execution lane is blocked during the wait, so a task backing off costs other instances nothing;
  • a PT5M backoff is a PT5M backoff whether or not the replica that started it is still alive.
sequenceDiagram
    participant L as Execution lane
    participant DB as PostgreSQL
    participant T as Timer poller

    loop while the budget holds
        L->>L: attempt fails
        L->>DB: re-park — task still pending, timer armed at the backoff instant
        Note over L,DB: a durable park, not a sleep — the lane is free for other instances
        T->>DB: backoff instant due
        T->>L: re-drive the task
    end
    L->>DB: budget spent — durable FAILED, retained and inspectable

A backoff is a row in a database, not a held thread — which is why it survives a restart and a hot-deploy, and why a task waiting five minutes costs every other instance nothing.

Declaring <q:retry> on a task makes its process stateful — the park needs persistence, so a process that was otherwise run-to-completion now requires a configured datasource.

What counts as a failed attempt

The failure set is deliberately narrow and different for each task kind.

On a registered task: an uncaught error from the task function.

On a channel-call task, exactly two things:

  • the route-less <q:timeout> boundary firing before the correlated response arrived (classification SUTRA.DISPATCH.CHANNEL_CALL.TIMEOUT). Without a policy a fired timeout is a catchable BPMN error; with one it becomes a retryable task failure first;
  • the request delivery being marked terminally poisoned by the outbox attempt ceiling (classification SUTRA.OUTBOUND.DELIVERY_ATTEMPTS_EXHAUSTED, reachable only when sutra.outbox.retry.max-attempts is configured). The failure reaches the task the moment the delivery gives up, rather than waiting out the whole timeout window.

Both classifications are stable structured codes, so nonRetryableCodes can name them — “retry a poisoned delivery but never a timeout” is expressible.

What is never a failed attempt:

  • A correlated business response. The counterpart answered. Whatever the answer says is the process’s business to route on; re-sending would double-submit. A response is a completion, full stop.
  • A BPMN error. Errors route to their boundary events, unchanged, on either task kind.

Modelled outcomes always beat a policy

A timer boundary event with drawn outgoing flows is a modelled outcome — the author has said what happens on timeout, so that is what happens, exactly as a BPMN error routes to its boundary instead of consuming a retry budget. Which is why the loader refuses the combination: a channel-call <q:retry> requires the route-less <q:timeout> form, and a routed timer boundary alongside a retry policy is a load error (SUTRA.CONFIG.BPMN.RETRY_NOT_APPLICABLE). A policy that could never fire would otherwise load as a silent near-no-op.

What a re-drive actually does on a channel call

Re-driving a channel call is not “wait for the same answer again” — it re-issues the request:

  • a fresh outbound request built from the same persisted variables, with a fresh idempotency key on the wire (so a counterpart doing its own dedup sees a genuinely new request, not a redelivery);
  • a fresh timeout window;
  • the dead attempt’s outstanding request withdrawn, so a late answer to a superseded attempt cannot race the new one.

A response that arrives for the dead attempt is refused with SUTRA.DISPATCH.CHANNEL_CALL.RETRY_PENDING — an honest verdict rather than a “no such instance” miss, because the correlation is deliberately kept alive. Once the re-issued request goes out, the same correlation serves its response normally.

sequenceDiagram
    participant E as Engine
    participant C as Counterpart

    E->>C: attempt n, idempotency key K1
    Note over E: route-less q:timeout fires — a retryable task failure
    E->>E: park the backoff, withdraw attempt n's pending delivery
    C-->>E: late answer to attempt n
    E-->>C: refused — SUTRA.DISPATCH.CHANNEL_CALL.RETRY_PENDING
    E->>C: attempt n+1, fresh key K2, fresh timeout window
    C-->>E: answer — a completion, never a retry trigger

A re-drive is a new request rather than a second wait for the old one, so the counterpart’s own deduplication sees it as new; and the superseded attempt is refused by name instead of being silently absorbed or lost.

Exhaustion

On a channel-call task whose failure the flow catches (a boundary or an error event sub-process), a spent budget raises the failure’s own code as a BPMN error on the task; see Catching a failed call. Otherwise a spent budget — or a nonRetryableCodes hit — fails the instance durably with SUTRA.RUNTIME.RETRY.EXHAUSTED. The instance becomes FAILED: retained, inspectable, blocking its deployment’s retirement, and repairable through instance migration. See Replica semantics.

Two budgets that never overlap

If a channel uses the pull transport, its worker-side failures are governed by the external task’s own budget (sutra.external-task.retries), not by <q:retry>. The two live at different layers and never interact — see External tasks.

The outbox poison ceiling

Outbound deliveries retry with backoff forever by default: an unreachable counterpart is treated as temporarily unreachable, and giving up silently is worse than waiting. Since 0.6.20 each failed attempt that will be retried logs a WARN with the destination, the cause’s code and message, the attempt number and next_attempt_at. The backoff base, sutra.outbox.retry.base-delay, defaults to PT1S.

sutra.outbox.retry.max-attempts is the opt-in ceiling. A delivery that exhausts it becomes terminally poisoned: retained with its last error, never retried again, one incident recorded if the delivery was required, and — importantly — no longer counted by the draining-deployment retirement gate. “We gave up” is a durable, visible state, never a silent disappearance.

Configuring the ceiling is also what makes SUTRA.OUTBOUND.DELIVERY_ATTEMPTS_EXHAUSTED reachable as a <q:retry> failure at all.

Execution history: a finished instance is not a 404

A terminal instance used to vanish. It doesn’t: the terminal step re-stamps the stored snapshot to COMPLETED / TERMINATED in the same transaction that resolves its waits, retires its correlation aliases, and enqueues its final emissions.

SurfaceWhat it answers for a finished instance
GET /sutra/instances/{id} and its admin twinThe retained projection — status, node progress, variables — for as long as retention keeps the row.
GET /admin/instancesExcludes finished instances by default; includeTerminal=true (or a terminal status filter) includes them.
GET /admin/instances/{id}/historyThe audit journal, paged: how it got there, event by event, with whatever payload the process captured. Page with ?afterOrdinal= and the response’s nextAfterOrdinal: each event’s ordinal is its position in the journal, assigned in commit order, so no event written later lands behind a cursor already returned. ?afterSeq= still works, but can miss a transport event, which shares the seq of the engine event it followed. Admin-only — an audit row can carry business data.

Retention. sutra.instance.retention (ISO-8601, default P7D) is how long terminal snapshots stay queryable; a lease-gated sweeper purges past-window rows on sutra.instance.retention-sweep-interval (default PT1H). PT0S restores delete-at-completion for a deployment that wants no history at all.

FAILED is always retained, regardless of the window — see above. And the audit journal itself stays opt-in (sutra.audit.sql): with it off, the history endpoint answers an explanatory empty shape rather than a misleading 404, and the retained terminal snapshot still answers the inspect route.

stateDiagram-v2
    [*] --> RUNNING
    RUNNING --> COMPLETED: terminal step re-stamps the snapshot
    RUNNING --> TERMINATED: terminal step re-stamps the snapshot
    RUNNING --> FAILED: budget spent or non-retryable code
    COMPLETED --> [*]: swept past sutra.instance.retention
    TERMINATED --> [*]: swept past sutra.instance.retention
    note right of FAILED
      Retained regardless of the window.
      Blocks its deployment's retirement; repairable by migration.
    end note

Finishing is a status change, not a disappearance — and FAILED is the one status the retention sweeper never reaches.

Timer completeness: durations, dates, cycles, and timer starts

All three BPMN timer definitions are supported, on all three timer hosts (start event, intermediate catch, boundary event):

DefinitionShapeExample
timeDurationISO-8601 duration — relative to armingPT30M, P2D
timeDateISO-8601 instant — an absolute deadline2026-12-31T23:59:00Z
timeCycleISO-8601 repeating intervalR3/PT12H, R/P1D

Deliberately not supported, and rejected as such rather than silently mis-parsed: cron syntax, and calendar-arithmetic durations (P1M meaning “one calendar month”). A malformed expression and an unsupported-but-well-formed one produce different diagnostics — “you wrote it wrong” and “the engine does not do that” are different problems.

Timer start events are schedules

A start event carrying a timer definition is a schedule, not a wait: it has no instance until it fires. Schedules are armed when a deployment becomes active, re-armed on boot, and resolved when the deployment stops being active — so hot-deploy handoff, undeploy, and retirement all need no extra operator step.

A fire mints an instance through the ordinary dispatch path, with empty variables: tenancy, quotas, audit capture, and coverage marking all apply exactly as they do for a message-triggered start.

Two rules keep cycles honest:

  • A cycle never bursts. If the engine was down across three occurrences of an R/PT1H cycle, it fires once and moves to the next slot — missed occurrences are coalesced, not replayed.
  • A repeat budget is spent by the grid, not by the firing. Each skipped slot of an R3/PT1H consumes one repeat, so the schedule can never outlive its third slot.

A start event carrying both a <q:source> and a timer definition is a load error — a start is triggered by a message or by a clock, never ambiguously by both.

Timer intervals from FEEL

Since 0.6.4. Every interval above can be computed instead of written down. That covers the body of a <timeDuration>, <timeDate> or <timeCycle>, a <q:timeout duration>, and a <q:retry initialDelay> or maxDelay. The rule is literal first, else FEEL:

The textWhat it is
parses as an ISO-8601 literal of its form (PT30M, 2026-12-31T23:59:00Z, R3/PT12H)that literal, exactly as before
looks like a literal but is written wrong (PT5X, P1M, 2026-03-01, R0/PT1H)a load error, with the same code as before
anything else that parses as FEEL (payload.window, config.retryDelay)an expression
anything elsea load error

Since 0.6.5 an expression may read only names something provides: config, a declared variable or data object, the payload, or a variable the process (or a process that calls it) sets before the timer. Any other name would be null when the timer is armed, so sutra lint, sutra package and the engine refuse it with SUTRA.CONFIG.BPMN.TIMER_EXPRESSION_UNKNOWN_NAME. A literal written wrong now says which literal was meant. From sutra lint on a package with one of each:

[ERROR] SUTRA.DEPLOY.PACKAGE.CONFIG_INVALID — BPMN bpmn/wait.bpmn fails to parse: [SUTRA.DISPATCH.TIMER.DURATION_INVALID] <intermediateCatchEvent> Pause timer is not schedulable: 'PT10SEC' has an unparseable number ''; did you mean PT10S?
[ERROR] SUTRA.CONFIG.BPMN.TIMER_EXPRESSION_UNKNOWN_NAME — deployment default/window/1.0.0 (dep-ffffffffffffffffffffffff): process 'wait' node 'Hold' timeDuration expression 'holdFor' reads 'holdFor', which nothing in scope provides (not config, a declared variable, the payload or a variable the process sets), so it would be null when the timer is armed. Declare it (<q:variable>), set it before the timer, or read it from config.<key>.

An expression is evaluated once, when its timer is armed:

TimerArmed whenThe expression can read
intermediate catch (also an event-based gateway arm)the token arrivesthe instance’s variables and config
boundary (timeDuration, timeDate)the host activity parksthe instance’s variables and config
<q:timeout>the call parks (it is a boundary)the instance’s variables and config
<q:retry> delayseach failed attempt arms its backoffthe instance’s variables and config
start event (timeDuration, timeDate, timeCycle)the deployment activatesconfig only

The instance’s variables are whatever the instance holds at that moment: the payload, a DMN decision’s output, a value read from a store. config holds the deployment’s properties:, all as strings. A start event has no instance yet, so its expression may read config and nothing else.

What the expression must return:

FormAccepted
durationan ISO-8601 duration string, or a FEEL days-and-time duration (duration("PT" + string(n) + "S"))
datean instant string with a zone, or a FEEL date and time with an offset or zone
cyclea repeating-interval string (R/PT1H)

A null, a number, a negative or calendar-length duration (P1M), an instant with no zone, or any other value fails the instance once with SUTRA.RUNTIME.TIMER.EXPRESSION_INVALID. It gets no default and no retry: <q:retry> does not count the failure, and an error boundary does not catch it. If the timer is armed by the request that starts the instance, the request is refused with that code and nothing is stored. On a start event, the deployment refuses to activate.

Only the instant the expression resolves to is stored: the wait row’s due time, the retry backoff, the schedule row’s spec. A resume, a replay, a restart or a replica hand-off reads that instant and never evaluates the expression again. A loop that comes back to the timer arms it again, so the expression is evaluated again.

Walkthrough: a delivery window

examples/delivery-window waits at a receive task for a delivery and flags the order late when the window closes. The order can name its own window. Otherwise the window is the deployment’s default:

POST /channels/orders             { orderId, window? }
  AwaitDelivery   receive task, answered on POST /channels/orders/delivered { orderId }
                  boundary WindowClosed: payload.window, else config.defaultWindow
  -> Delivered    a delivery inside the window
  -> Late         the window closed first
<bpmn:boundaryEvent id="WindowClosed" name="No delivery in the window" attachedToRef="AwaitDelivery">
  <bpmn:outgoing>F3</bpmn:outgoing>
  <bpmn:timerEventDefinition id="WindowClosed_Timer">
    <bpmn:timeDuration>if payload.window = null then config.defaultWindow else payload.window</bpmn:timeDuration>
  </bpmn:timerEventDefinition>
</bpmn:boundaryEvent>
# package.yaml
properties:
  defaultWindow: "${DEFAULT_WINDOW:PT10S}"

The example’s compose.yaml gives the engine DEFAULT_WINDOW=PT8S.

Prerequisites: Before you try a walkthrough, with an engine image and a sutra CLI of 0.6.4 or later.

1. Start the stack and deploy the package. The timers helper lists the timer rows, with how many seconds after arming each one is due:

C=examples/delivery-window/deploy/compose.yaml
docker compose -f $C up -d --wait
sutra package examples/delivery-window/deployments-src/default--window--1.0.0 \
    --out examples/delivery-window/deploy/deployments 2>/dev/null
PORT=$(docker compose -f $C port engine 8080 | cut -d: -f2)
H=(-H 'Content-Type: application/json' -H 'X-Api-Key: dev-only-window-key')
until curl -sf "http://127.0.0.1:$PORT/sutra/deployments" | grep -q '"phase":"Active"'; do sleep 1; done
send() { curl -s -w " HTTP %{http_code}\n" -X POST "http://127.0.0.1:$PORT/channels/orders$1" "${H[@]}" -d "$2"; }
timers() {
  docker compose -f $C exec -T engine-db psql -U postgres -d sutra -At -c \
    "select node_id, status, extract(epoch from timer_due_at - created_at)::int as seconds
       from waiting_event where kind = 'TIMER' order by created_at"
}
id_of() { curl -s "http://127.0.0.1:$PORT/admin/instances/by-alias/orderId/$1" | python3 -c 'import json,sys; print(json.load(sys.stdin)["instanceId"])'; }
status_of() { curl -s "http://127.0.0.1:$PORT/sutra/instances/$1" | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["status"], d.get("failureCode") or "")'; }
packaged examples/delivery-window/deploy/deployments/default--window--1.0.0.sutra (deploymentId dep-99c53c0c698e17b631a9fdcf)

2. An order with its own window. The boundary is due 3 seconds after the receive task parked. Nothing is delivered, so after 5 seconds the instance has taken the Late path:

send "" '{"orderId":"o-1","window":"PT3S"}'
O1=$(id_of o-1)
timers
sleep 5
status_of $O1
curl -s "http://127.0.0.1:$PORT/admin/instances/$O1/history" | python3 -c '
import json, sys
for e in json.load(sys.stdin)["events"]:
    if e["eventType"] == "NODE_LEFT": print(e["nodeId"])'
 HTTP 200
WindowClosed|WAITING|3
COMPLETED
Start
AwaitDelivery
Late

3. An order without one. The window is config.defaultWindow: PT8S, from the engine’s environment. A delivery inside it completes the instance on the normal path:

send "" '{"orderId":"o-2"}'
O2=$(id_of o-2)
timers
send /delivered '{"orderId":"o-2"}'
status_of $O2
 HTTP 200
WindowClosed|RESOLVED|3
WindowClosed|WAITING|8
 HTTP 200
COMPLETED

4. A window that is not a duration. The expression returns the string "soon" when the receive task parks. The request is refused and nothing is stored:

send "" '{"orderId":"o-3","window":"soon"}'
{"type":"urn:bpm:diag:SUTRA.RUNTIME.TIMER.EXPRESSION_INVALID","title":"SUTRA.RUNTIME.TIMER.EXPRESSION_INVALID","status":500,"detail":"timer 'WindowClosed' <timeDuration> expression 'if payload.window = null then config.defaultWindow else payload.window' could not be armed: it returned the string \"soon\", which is not a usable duration: 'soon' is not an ISO-8601 duration (must start with 'P')","code":"SUTRA.RUNTIME.TIMER.EXPRESSION_INVALID","attributes":{"incident":"SUTRA.INBOUND.NON_IDEMPOTENT_FAILURE"}} HTTP 500

When the timer is armed later in an instance’s life (after a first wait, say), the same failure leaves the instance FAILED, with this code as its failureCode.

5. Change the default without touching the archive. Restart the engine with a different DEFAULT_WINDOW. The archive and its deploymentId stay the same; new orders get the new window:

docker compose -f $C stop engine
DEFAULT_WINDOW=PT20S docker compose -f $C up -d --wait engine
PORT=$(docker compose -f $C port engine 8080 | cut -d: -f2)
until curl -sf "http://127.0.0.1:$PORT/sutra/deployments" | grep -q '"phase":"Active"'; do sleep 1; done
send "" '{"orderId":"o-4"}'
timers
 HTTP 200
WindowClosed|RESOLVED|3
WindowClosed|RESOLVED|8
WindowClosed|WAITING|20

6. Clean up:

docker compose -p sutra-delivery-window down -v

Next