Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Logging and audit

Three separate surfaces, deliberately not conflated: engine logs, CLI logs, and the audit trail. Each has its own destination and its own purpose. None of the three is collected by Sutra itself — see No telemetry, no phone-home for that guarantee; everything below stays on the host (or goes to a destination you configure) unless you point it somewhere else.

Engine logs

Structured JSON on stdout, always, with no configuration required. The field shape (timestamp / level / loggerName / message / service.name, plus traceId/spanId inside a sampled span) is stable, so a log-processing pipeline can key off it directly. RUST_LOG filters verbosity using standard tracing EnvFilter syntax (default info) — e.g. RUST_LOG=sutra_channels=debug,sutra_engine::deploy=trace to raise one module without drowning in the rest. See Troubleshooting BPMN solutions for reading these logs alongside traces and the audit trail when tracking down one message’s path through the engine.

When an OTLP endpoint is configured (see Configuration reference), the same log records additionally export over OTLP — stdout is never replaced, only supplemented.

CLI logs

The sutra CLI writes its own logs to stderr, opt-in via -v / -vv / -vvv (info / debug / trace) — never mixed into stdout, which is reserved for report output (text or --format json). This is why every command in the CLI reference is safe to pipe: sutra describe my-process.bpmn --format json | jq . never has a stray log line corrupt the JSON.

Audit sinks

The audit trail is a separate, compliance-oriented record of what a process instance did — every INSTANCE_STARTED/INSTANCE_COMPLETED/INSTANCE_SUSPENDED/INSTANCE_RESUMED event, validation outcomes, and relay/resume decisions — independent of the telemetry pipeline described in Observability. Three sinks, any combination of which can be active at once:

ConfigenvWhat it does
sutra.audit.jsonl.pathSUTRA_AUDIT_JSONLWrites one JSONL file (or a per-tenant directory tree) an operator can tail, or feed to sutra audit-replay (below) offline.
sutra.audit.otel.endpointSUTRA_AUDIT_OTEL_ENDPOINTShips audit events over OTLP to a collector.
sutra.audit.sqlSUTRA_AUDIT_SQLPersists audit rows durably in the engine’s own datasource, under the same row-level-security policy as every other engine table — see Multi-tenancy and isolation.

sutra.audit.sql is off by default. With no sink on, an engine writes no audit record, including the inbound.DELIVERY_DROPPED record for a refused delivery; see Refused deliveries.

A process’s events go to one sink: the one its <q:audit sink="…"> names, or else the highest-priority sink that is switched on (sql, then jsonl, then otel).

  • JSONL. Each event is one line, written in a single append, so a file shared by every executor lane never holds two events on one line (fixed in 0.6.9).
  • OTLP. The records are OTLP log records on the instrumentation scope sutra.audit, sent through a logs pipeline of their own to SUTRA_AUDIT_OTEL_ENDPOINT, never through the telemetry endpoint. The engine builds that pipeline once at start and flushes it on shutdown. Before 0.6.9, setting this variable made the engine panic at start.

Resource attributes on audit records

The OTLP audit records carry the same resource attributes as the engine’s own telemetry, so a collector can tell which deployment, environment or test run an audit record belongs to:

SourceExampleOn audit records
OTEL_RESOURCE_ATTRIBUTESdeployment.environment=staging,example.run_id=run-42yes
sutra.telemetry.resource-attributes / SUTRA_TELEMETRY_RESOURCE_ATTRIBUTES (0.6.12)example.from_config=yesyes, and on every telemetry signal; wins over OTEL_RESOURCE_ATTRIBUTES for the same key
sutra.audit.otel.service-name / SUTRA_AUDIT_OTEL_SERVICE_NAME (0.6.12)orders-auditthe records’ service.name; default sutra-audit
OTEL_SERVICE_NAMEorders-engineno: it names the engine’s telemetry, never the audit stream

To try it, point the audit pipeline at a collector whose debug exporter prints what it receives (verbosity: detailed), and start the engine with:

SUTRA_AUDIT_OTEL_ENDPOINT=http://collector:4317
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=conformance,example.run_id=run-42
SUTRA_TELEMETRY_RESOURCE_ATTRIBUTES=example.from_config=yes

Run any process with <q:audit sink="otel">. Each batch of audit records the collector prints starts with the resource, which includes (with the telemetry.sdk.* attributes, in any order):

Resource attributes:
     -> deployment.environment: Str(conformance)
     -> example.run_id: Str(run-42)
     -> example.from_config: Str(yes)
     -> service.name: Str(sutra-audit)

The conformance suite tc_audit_sinks runs exactly this against a real collector.

What an audit event captures

A process chooses how much each of its audit events carries, with <q:audit capture="…">:

captureEach event carries
noneno audit event at all
metadatathe event, node and diagnostic, with no variable content
payload (the default)the redacted variable snapshot
media (0.6)metadata, plus the media references found among the variables ({kind: "media", uri, sha256, …}), never the media itself

With media, subject erasure and terminal retention also delete the recordings those events reference, and amend each event to say they are gone. See Recordings and media references for the mechanism and a walkthrough.

Correlation keys on every audit event

From 0.6.9, each audit event carries the instance’s correlation keys: every <q:alias> the process declares, with its current value. That lets you find all of one business transaction’s events by the key your systems already use, without first looking up the instance id.

SinkWhere the keys are
JSONLa correlation object on the line: "correlation":{"draftId":"d-7"}
OTLPone attribute per key, named correlation.<alias name>
SQLnot stored; the audit_event row is unchanged
  • Where the values come from. A start event’s aliases are bound into the instance’s variables under the alias name, and the variables travel with the instance, so every event reads its keys from there, including after a resume on another replica. No database read is added per event.
  • Wait-node aliases. An alias declared on a wait node is evaluated over the current variables. Until it has a value, the key is left out.
  • Transport events and unaudited processes. Transport events carry no keys, and a process that is not audited computes none.

Redaction applies to the keys as it does everywhere else:

  • A key shows ***REDACTED*** when the alias name, or any variable its expression reads, is declared sensitive.
  • When the expression reads a payload that has a <q:redactors> chain, the key comes from the masked projection, never from the raw payload. When the expression is more than a plain path, the key shows ***REDACTED***.

Try it

The review-loop example correlates every message by its start event’s <q:alias name="draftId" expression="payload.draftId">. Its compose file switches on the SQL sink; this override sends the audit trail to a JSONL file instead. You need Docker, the sutra CLI and an engine image, both 0.6.9 or later. From the repository root:

C=examples/review-loop/deploy/compose.yaml
cat > /tmp/audit-jsonl.yaml <<'YAML'
services:
  engine:
    environment:
      SUTRA_AUDIT_SQL: "false"
      SUTRA_AUDIT_JSONL: /tmp/audit/audit.jsonl
YAML
docker compose -f $C -f /tmp/audit-jsonl.yaml up -d --wait
sutra package examples/review-loop/deployments-src/default--review--1.0.0 \
    --out examples/review-loop/deploy/deployments 2>/dev/null
PORT=$(docker compose -f $C port engine 8080 | cut -d: -f2)
H=(-H 'Content-Type: application/json' -H 'X-Api-Key: dev-only-review-key')
until curl -sf "http://127.0.0.1:$PORT/sutra/deployments" | grep -q '"phase":"Active"'; do sleep 1; done
send() { curl -s -o /dev/null -w "HTTP %{http_code}\n" -X POST "http://127.0.0.1:$PORT/channels/drafts$1" "${H[@]}" -d "$2"; }
send "" '{"draftId":"d-7"}'
send /review '{"draftId":"d-7","decision":"approve"}'
docker compose -f $C cp engine:/tmp/audit/audit.jsonl - | tar -xO | head -3
docker compose -p sutra-review-loop down -v

Both requests answer HTTP 200, and every line carries the draft’s key (ids and times differ):

{"instanceId":"66ed7453-…","deploymentId":"dep-84173f…","tenant":"default","seq":1,"at":"…","eventType":"INSTANCE_STARTED","payload":{},"correlation":{"draftId":"d-7"}}
{"instanceId":"66ed7453-…","deploymentId":"dep-84173f…","tenant":"default","seq":2,"at":"…","eventType":"NODE_ENTERED","nodeId":"Start","payload":{},"correlation":{"draftId":"d-7"}}
{"instanceId":"66ed7453-…","deploymentId":"dep-84173f…","tenant":"default","seq":3,"at":"…","eventType":"NODE_LEFT","nodeId":"Start","payload":{},"correlation":{"draftId":"d-7"}}

Replaying an instance’s audit trail offline

sutra audit-replay <instance-id> --from-jsonl <path-to-file-or-dir> [--tenant <id>] [--until <EVENT_TYPE>]

Walks the JSONL audit stream for one instance id and prints its events in order — useful for reconstructing what a specific production instance did without needing direct database access. --until stops replay after a given event type (e.g. INSTANCE_COMPLETED).

How sensitive data is masked

A variable tagged sensitive (via q:variables, see The q: namespace) or read from a dataClass-tagged data store (see Data stores) is redacted on every one of the surfaces above — audit sinks, structured logs, and traces alike — by the same redactor mechanism described in Domain neutrality and the SPI model. The flow itself still sees the real value; only what the engine emits is masked.

A source that declares masking, with q:redactors or a sensitive payload variable, masks the rest of the inbound message as well: event.body and every header value in event.headers and header, whole, because they carry the same data and can carry credentials. The flow still reads the real values. This has limits you should know:

  • With no master key, a snapshot holds values unencrypted.
  • A flow that declares no masking shows all of it, including a credential sent in a header such as an API key, on inspect, in the snapshot, and in audit output with capture="payload". To avoid that, declare a sensitive payload variable, avoid capture="payload", limit access to the inspect API and the database, or end authentication at a gateway that strips the header.
  • A q:alias value is stored in clear, so an alias built from a header holds its value. An explicit event id is stored in clear too, and so is the trace id with an outbound send.
  • A copy of the body in another variable is only masked if that variable is sensitive. sutra lint warns about this with SUTRA.CONFIG.BPMN.RAW_BODY_UNMASKED. Writing the body to a data store warns with SUTRA.CONFIG.BPMN.RAW_BODY_STORED, because store columns have no at-rest encryption. The lint doesn’t cover copies of a header.

Next

  • Observability — the telemetry (trace/metric/log) side this page’s audit trail is deliberately kept separate from.
  • Troubleshooting BPMN solutions — putting logs, traces, and the audit trail together to answer “what happened to this message?”