Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Connecting to a TLS and SASL broker

Since 0.6.21 the kafka and amqp (AMQP 1.0) transports connect to TLS brokers natively, with no sidecar. Kafka also authenticates with SASL PLAIN, SCRAM-SHA-256 or SCRAM-SHA-512, and AMQP 1.0 with SASL PLAIN.

What this was tested against. Local containers only: Apache Kafka 3.8 and Apache ActiveMQ Artemis. Azure Service Bus, Confluent Cloud, Amazon MSK and other managed services have not been tested and are not claimed to work. Azure Service Bus’s CBS token flow is not implemented. Client certificates (mTLS) and SASL OAUTHBEARER and GSSAPI are not supported.

The settings

For an inbound channel:

PropertyTransportMeaning
security.protocolkafkaPLAINTEXT (default), SSL, SASL_PLAINTEXT or SASL_SSL
sasl.mechanismkafkaPLAIN, SCRAM-SHA-256 or SCRAM-SHA-512
sasl.username, sasl.passwordkafkaSecret references
tls: trueamqpUse TLS. The default port becomes 5671
username, passwordamqpSecret references. SASL PLAIN is used when both are set
tls-ca-filebothPath to a PEM bundle, used in place of the system CAs
tls-ca-pembothSecret reference to PEM text, used in place of the system CAs
tls-insecure-skip-verifybothtrue turns off certificate and host-name checks

A credential, and tls-ca-pem, must be a reference as a whole value: env:NAME, secret:KEY, ${NAME}, ${secret:KEY}, a vendor store scheme the engine is built with, or k8s:secret/<name>#<key> (see References in connection properties). A literal value stops the channel at startup with SUTRA.STARTUP.REFUSED.HARDCODED_SECRET. tls-ca-file takes references like an address does; the switches (tls, security.protocol, sasl.mechanism, tls-insecure-skip-verify) stay literal.

With no CA setting, the broker is checked against the system CAs, including any set with SSL_CERT_FILE. Certificate and host-name checks are on unless you set tls-insecure-skip-verify: true, and the engine logs a warning whenever that setting applies.

A setting that could not take effect stops the channel at startup rather than being ignored: a CA without TLS, sasl.* without a SASL protocol, both CA settings at once, or SASL GSSAPI or OAUTHBEARER. Kafka’s kafka.* passthrough does not accept sasl.*, ssl.* or enable.ssl.certificate.verification.

Outbound sends use one engine-wide configuration per transport, set in the environment:

VariableMeaning
SUTRA_SINK_KAFKA_BOOTSTRAP_SERVERSThe brokers
SUTRA_SINK_KAFKA_SECURITY_PROTOCOLAs security.protocol
SUTRA_SINK_KAFKA_SASL_MECHANISM, _SASL_USERNAME, _SASL_PASSWORDAs the sasl.* properties. These hold values, so set them from your secret store
SUTRA_SINK_KAFKA_TLS_CA_FILE, SUTRA_SINK_KAFKA_TLS_INSECURE_SKIP_VERIFYTrust
SUTRA_SINK_AMQP_USERNAME, SUTRA_SINK_AMQP_PASSWORDDefault credentials
SUTRA_SINK_AMQP_TLS_CA_FILE, SUTRA_SINK_AMQP_TLS_INSECURE_SKIP_VERIFYTrust for amqp10s:// destinations
SUTRA_SINK_AMQP_OPERATION_TIMEOUT_MSThe bound on each broker round trip of a send (opening the connection with TLS and SASL, the session, the attach, the wait for the broker to accept the message, the close), default 30000. A broker that does not answer in time fails the attempt with the retryable SUTRA.OUTBOUND.AMQP.SEND_TIMEOUT and the outbox sends again; a send accepted just before its answer timed out can arrive twice, so receivers dedupe on sutra-outbox-key. Since 0.6.22.
SUTRA_SINK_AMQP_MAX_CONNECTIONSHow many connections the sink keeps open to one broker (scheme, host, port and credentials), default 32 (8 in 0.8.2), matching sutra.outbox.destination-concurrency so a broker destination’s sends do not queue for a connection. Each has one session and a sender link per target address; a send takes an idle one and, with all busy, waits for one for at most the operation timeout. A connection that fails a send is dropped and the send tried once more on a fresh one; one idle for two minutes is closed. Since 0.8.2.

Walkthrough: a message through both brokers

You need Docker, openssl, the sutra CLI and an empty working directory. This creates a private CA, starts both brokers with TLS and a login, and runs a package that answers each message. It was run as written, with sutra 0.6.21, apache/kafka:3.8.0, apache/activemq-artemis:latest-alpine and postgres:16-alpine; the output shown is from that run.

1. Create the certificates. The server certificate must name the host the engine connects to.

openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes -days 7 \
  -subj "/CN=demo broker CA" -keyout ca.key -out ca.pem
for broker in tlsdemo-kafka tlsdemo-artemis; do
  openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
    -subj "/CN=$broker" -keyout "$broker.key" -out "$broker.csr"
  openssl x509 -req -in "$broker.csr" -CA ca.pem -CAkey ca.key -CAcreateserial -days 7 \
    -extfile <(printf "subjectAltName=DNS:%s" "$broker") -out "$broker.crt"
done
chmod 644 ./*.key ./*.crt ca.pem
BROKER_PASSWORD=$(openssl rand -hex 16)
docker network create tlsdemo-net

The chmod lets the containers read these files. The keys are throwaway, so don’t reuse this step for real keys.

2. Start Kafka with a plaintext listener inside its container for the commands below, and a SASL_SSL listener for the engine. Save this as server.properties:

process.roles=broker,controller
node.id=1
controller.quorum.voters=1@localhost:9093
controller.listener.names=CONTROLLER
inter.broker.listener.name=INTERNAL
listeners=INTERNAL://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093,SECURE://0.0.0.0:9094
advertised.listeners=INTERNAL://localhost:9092,SECURE://tlsdemo-kafka:9094
listener.security.protocol.map=INTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT,SECURE:SASL_SSL
sasl.enabled.mechanisms=SCRAM-SHA-256
listener.name.secure.scram-sha-256.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required;
ssl.keystore.type=PEM
ssl.keystore.location=/certs/kafka-keystore.pem
log.dirs=/tmp/kraft-logs
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
cat tlsdemo-kafka.key tlsdemo-kafka.crt > kafka-keystore.pem
chmod 644 kafka-keystore.pem server.properties
docker run -d --name tlsdemo-kafka --network tlsdemo-net \
  -v "$PWD/server.properties:/certs/server.properties:ro" \
  -v "$PWD/kafka-keystore.pem:/certs/kafka-keystore.pem:ro" \
  --entrypoint /bin/sh apache/kafka:3.8.0 -c "
    /opt/kafka/bin/kafka-storage.sh format -t MkU3OEVBNTcwNTJENDM2Qk -c /certs/server.properties \
      --add-scram 'SCRAM-SHA-256=[name=orders-svc,password=$BROKER_PASSWORD]' &&
    exec /opt/kafka/bin/kafka-server-start.sh /certs/server.properties"
until docker logs tlsdemo-kafka 2>&1 | grep -q 'Kafka Server started'; do sleep 1; done
for topic in orders-in orders-out; do
  docker exec tlsdemo-kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 \
    --create --topic "$topic"
done

3. Start Artemis with an AMQPS acceptor.

openssl pkcs12 -export -in tlsdemo-artemis.crt -inkey tlsdemo-artemis.key -out artemis.p12 \
  -passout pass:changeit -name broker
chmod 644 artemis.p12
docker run -d --name tlsdemo-artemis --network tlsdemo-net \
  -v "$PWD/artemis.p12:/certs/artemis.p12:ro" \
  --entrypoint /bin/sh apache/activemq-artemis:latest-alpine -c "
    /opt/activemq-artemis/bin/artemis create --user orders-svc --password $BROKER_PASSWORD \
      --silent --require-login --http-host 0.0.0.0 --relax-jolokia . &&
    sed -i 's#</acceptors>#<acceptor name=\"amqps\">tcp://0.0.0.0:5671?sslEnabled=true;keyStorePath=/certs/artemis.p12;keyStorePassword=changeit;keyStoreType=PKCS12;protocols=AMQP;saslMechanisms=PLAIN</acceptor></acceptors>#' etc/broker.xml &&
    exec ./bin/artemis run"
until docker logs tlsdemo-artemis 2>&1 | grep -q 'Server is now active'; do sleep 1; done
for q in orders-in orders-out; do
  docker exec tlsdemo-artemis ./bin/artemis queue create --user orders-svc --password "$BROKER_PASSWORD" \
    --url tcp://localhost:61616 --name "$q" --address "$q" --anycast --durable --auto-create-address \
    --preserve-on-no-consumers --silent
done

The two queues are created up front, as anycast queues, so a message sent before anyone reads it is kept. Left to Artemis, an address the engine attaches to first is created as multicast, and the producer below would send to a queue no one reads.

4. Write the package. Create tls-relay/package.yaml:

labels:
  "tenant": "default"
  "module": "tls-relay"
  "version": "1.0.0"
engine:
  minContract: 1

Create tls-relay/channels.yaml:

channels:
  - name: kafka-in
    transport: kafka
    codec: json
    cloudevents-mode: none
    bootstrap.servers: tlsdemo-kafka:9094
    topic: orders-in
    group.id: tls-relay
    security.protocol: SASL_SSL
    sasl.mechanism: SCRAM-SHA-256
    sasl.username: env:BROKER_USER
    sasl.password: secret:BROKER_PASSWORD
    tls-ca-pem: secret:BROKER_CA
  - name: kafka-out
    direction: outbound
    transport: kafka
    bind: kafka://orders-out
    cloudevents-mode: none
  - name: amqp-in
    transport: amqp
    codec: json
    cloudevents-mode: none
    host: tlsdemo-artemis
    tls: true
    tls-ca-pem: secret:BROKER_CA
    username: env:BROKER_USER
    password: secret:BROKER_PASSWORD
    queue: orders-in
  - name: amqp-out
    direction: outbound
    transport: amqp
    bind: amqp10s://tlsdemo-artemis/orders-out
    cloudevents-mode: none

Create tls-relay/bpmn/tls-relay.bpmn, with one process per transport. Each builds an answer and sends it out. This is the Kafka one; add a copy for AMQP with its ids prefixed amqp, its process id amqp-relay, and the channels amqp-in and amqp-out:

<?xml version="1.0" encoding="UTF-8"?>
<bpmn:definitions xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL"
                  xmlns:q="urn:sutra:q:1.0" id="Defs"
                  targetNamespace="urn:sutra:module:tls-relay:1.0.0">
  <bpmn:process id="kafka-relay" isExecutable="true">
    <bpmn:extensionElements>
      <q:variables><q:variable name="responseBody" type="object"/></q:variables>
    </bpmn:extensionElements>
    <bpmn:startEvent id="kafkaStart">
      <bpmn:extensionElements><q:source channel="kafka-in"/></bpmn:extensionElements>
      <bpmn:outgoing>kafkaF1</bpmn:outgoing>
    </bpmn:startEvent>
    <bpmn:sequenceFlow id="kafkaF1" sourceRef="kafkaStart" targetRef="kafkaBuild"/>
    <bpmn:serviceTask id="kafkaBuild">
      <bpmn:incoming>kafkaF1</bpmn:incoming><bpmn:outgoing>kafkaF2</bpmn:outgoing>
      <bpmn:dataInputAssociation>
        <bpmn:assignment>
          <bpmn:from>{ref: payload.ref, status: "ANSWERED"}</bpmn:from>
          <bpmn:to>responseBody</bpmn:to>
        </bpmn:assignment>
      </bpmn:dataInputAssociation>
    </bpmn:serviceTask>
    <bpmn:sequenceFlow id="kafkaF2" sourceRef="kafkaBuild" targetRef="kafkaAnswer"/>
    <bpmn:sendTask id="kafkaAnswer">
      <bpmn:extensionElements><q:send channel="kafka-out"/></bpmn:extensionElements>
      <bpmn:incoming>kafkaF2</bpmn:incoming><bpmn:outgoing>kafkaF3</bpmn:outgoing>
    </bpmn:sendTask>
    <bpmn:sequenceFlow id="kafkaF3" sourceRef="kafkaAnswer" targetRef="kafkaEnd"/>
    <bpmn:endEvent id="kafkaEnd"><bpmn:incoming>kafkaF3</bpmn:incoming></bpmn:endEvent>
  </bpmn:process>
</bpmn:definitions>
mkdir -p deployments secrets
sutra lint tls-relay
sutra package tls-relay --out deployments
[WARN] SUTRA.CHANNEL.NO_SCHEMA — deployment default/tls-relay/1.0.0 (...): channel 'kafka-in' has 1 subscriber(s) but its codec (json) declares no message types — ...
[WARN] SUTRA.CHANNEL.NO_SCHEMA — deployment default/tls-relay/1.0.0 (...): channel 'amqp-in' has 1 subscriber(s) but its codec (json) declares no message types — ...
0 error(s), 2 warning(s)
packaged deployments/tls-relay.sutra (deploymentId dep-e890994c8e67a1c5b02e6db2)

The two warnings say the channels are schema-less, which is what this package intends.

5. Start the engine. The inbound channels read the password and the CA from the mounted secrets directory. The engine-wide sinks take theirs from the environment. The database role has no superuser or BYPASSRLS rights, because the engine refuses to start on one that does.

printf '%s' "$BROKER_PASSWORD" > secrets/BROKER_PASSWORD
cp ca.pem secrets/BROKER_CA
chmod -R a+rX deployments secrets
docker run -d --name tlsdemo-db --network tlsdemo-net -e POSTGRES_PASSWORD=dev-only postgres:16-alpine
until docker exec tlsdemo-db pg_isready -U postgres -h 127.0.0.1; do sleep 1; done
sleep 2
docker exec tlsdemo-db psql -U postgres \
  -c "CREATE ROLE sutra_engine LOGIN PASSWORD 'dev-only' NOSUPERUSER NOBYPASSRLS" \
  -c "ALTER SCHEMA public OWNER TO sutra_engine"
docker run -d --name tlsdemo-engine --network tlsdemo-net -p 127.0.0.1::8080 \
  -v "$PWD/deployments:/deployments:ro" -e SUTRA_DEPLOYMENTS_DIR=/deployments \
  -v "$PWD/secrets:/etc/sutra/secrets:ro" \
  -v "$PWD/ca.pem:/etc/sutra/broker-ca.pem:ro" \
  -e SUTRA_DATASOURCE_URL=postgres://tlsdemo-db:5432/postgres \
  -e SUTRA_DATASOURCE_USERNAME=sutra_engine -e SUTRA_DATASOURCE_PASSWORD=dev-only \
  -e BROKER_USER=orders-svc \
  -e SUTRA_SINK_KAFKA_BOOTSTRAP_SERVERS=tlsdemo-kafka:9094 \
  -e SUTRA_SINK_KAFKA_SECURITY_PROTOCOL=SASL_SSL \
  -e SUTRA_SINK_KAFKA_SASL_MECHANISM=SCRAM-SHA-256 \
  -e SUTRA_SINK_KAFKA_SASL_USERNAME=orders-svc \
  -e SUTRA_SINK_KAFKA_SASL_PASSWORD="$BROKER_PASSWORD" \
  -e SUTRA_SINK_KAFKA_TLS_CA_FILE=/etc/sutra/broker-ca.pem \
  -e SUTRA_SINK_AMQP_USERNAME=orders-svc \
  -e SUTRA_SINK_AMQP_PASSWORD="$BROKER_PASSWORD" \
  -e SUTRA_SINK_AMQP_TLS_CA_FILE=/etc/sutra/broker-ca.pem \
  ghcr.io/startr-trade/sutra:0.6.21
PORT=$(docker port tlsdemo-engine 8080 | head -1 | cut -d: -f2)
until curl -sf "http://127.0.0.1:$PORT/sutra/health/ready"; do sleep 1; done
{"status":"UP","checks":[{"name":"sutra-loader","status":"UP","data":{"deployments":1,"shards":1}}]}

6. Send a message through Kafka and read the answer.

echo '{"ref":"kafka-1"}' | docker exec -i tlsdemo-kafka /opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server localhost:9092 --topic orders-in
docker exec tlsdemo-kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic orders-out --from-beginning --max-messages 1 --timeout-ms 60000
{"ref":"kafka-1","status":"ANSWERED"}
Processed a total of 1 messages

The message went in and the answer came out over the SASL_SSL listener, with SCRAM-SHA-256 and the private CA; the commands above use the broker’s internal plaintext listener only to drive it.

7. Send a message through Artemis and read the answer.

docker exec tlsdemo-artemis ./bin/artemis producer --user orders-svc --password "$BROKER_PASSWORD" \
  --url tcp://localhost:61616 --protocol amqp --destination queue://orders-in --message-count 1 \
  --message '{"ref":"amqp-1"}'
docker exec tlsdemo-artemis ./bin/artemis browser --user orders-svc --password "$BROKER_PASSWORD" \
  --url tcp://localhost:61616 --destination queue://orders-out --message-count 1 --verbose
Producer orders-in, thread=0 Produced: 1 messages
...
Consumer ActiveMQQueue[orders-out], thread=0 browsed: 1 messages

The engine read the message over AMQPS (port 5671, SASL PLAIN, the private CA) and sent its answer to amqp10s://tlsdemo-artemis/orders-out. The engine’s log holds the broker password 0 times.

8. See a refusal. Restart the engine with a wrong password in secrets/BROKER_PASSWORD (and in the two SUTRA_SINK_*_PASSWORD variables), and read its log:

docker logs tlsdemo-engine 2>&1 | grep -E 'BROKER_(AUTH|TLS)_FAILED'
{"channel":"amqp-in","code":"SUTRA.INBOUND.AMQP.BROKER_AUTH_FAILED",...,"level":"WARN",...,"message":"amqp consumer unavailable — retrying in 1s: amqp channel 'amqp-in' could not connect to tlsdemo-artemis:5671: the broker refused the SASL credentials (outcome Auth)",...}
{"client":"channel 'kafka-in'","code":"SUTRA.INBOUND.KAFKA.BROKER_AUTH_FAILED",...,"level":"WARN",...,"message":"kafka channel 'kafka-in': sasl_ssl://tlsdemo-kafka:9094/bootstrap: SASL authentication error: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 ...",...}

Put the right password back and a certificate from another CA in secrets/BROKER_CA, restart, and the log reads:

{"client":"channel 'kafka-in'","code":"SUTRA.INBOUND.KAFKA.BROKER_TLS_FAILED",...,"message":"kafka channel 'kafka-in': sasl_ssl://tlsdemo-kafka:9094/bootstrap: SSL handshake failed: error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified ...",...}
{"channel":"amqp-in","code":"SUTRA.INBOUND.AMQP.BROKER_TLS_FAILED",...,"message":"amqp consumer unavailable — retrying in 1s: amqp channel 'amqp-in' could not connect to tlsdemo-artemis:5671: the TLS handshake failed: invalid peer certificate: UnknownIssuer",...}

Neither the wrong password nor the right one appeared anywhere in the log.

CodeMeaning
SUTRA.{INBOUND,OUTBOUND}.{KAFKA,AMQP}.BROKER_AUTH_FAILEDThe broker refused the credentials
SUTRA.{INBOUND,OUTBOUND}.{KAFKA,AMQP}.BROKER_TLS_FAILEDThe CA does not trust the broker’s certificate, the certificate does not name the host, or the trust settings could not be loaded

Neither message includes a password. Both consumers keep retrying and an outbound send stays in the outbox, so both recover once you correct the setting and restart the engine. The AMQP consumer backs off (1 s, 2 s, …). The Kafka consumer logs one line per attempt by its client, which retries quickly: with the wrong CA this run logged about 740 BROKER_TLS_FAILED lines in 20 seconds, so alert on the code rather than on the line count.

9. Clean up.

docker rm -f tlsdemo-engine tlsdemo-db tlsdemo-artemis tlsdemo-kafka
docker network rm tlsdemo-net