A Kafka container can report healthy and still refuse every client that isn’t on the same host. On the first connection the broker hands back the address in advertised.listeners, and your client then dials that address instead of the one you gave it. Most Compose files you will find advertise localhost:9092, which works from the host and fails from every other container, and we reproduced exactly that failure before writing the configs below.
This Kafka Docker Compose guide deploys the broker in KRaft mode, first as a single node and then as a three-broker cluster with replication factor 3. You’ll stop brokers to watch the ISR shrink and recover, measure throughput with kafka-producer-perf-test, connect a Python client, add Schema Registry and roll an upgrade under load. Everything ran on Ubuntu 26.04 in October 2026 with Kafka 4.3.1 from the official apache/kafka image.
How the cluster fits together
Each broker in the cluster also acts as a KRaft controller, so three containers carry both the data and the metadata quorum. Every broker has two client listeners, and which one you use depends on where your client runs. INTERNAL on port 9092 serves anything on the Compose network, EXTERNAL on a per-broker published port serves clients outside Docker, and the controller port 9093 never leaves the network.
Each external port is unique because a client that bootstraps from one broker gets told to reach partition leaders on the others. If all three advertised the same host port, your client would keep landing on one broker, and the partitions led by the other two would time out even though every container looked healthy.
Prerequisites
Kafka’s resource use is driven by what you keep and how fast you write, not by the broker count. Retention bytes decide the disk, write throughput times the replication factor decides how fast that disk has to be, and RAM beyond the heap goes to the page cache that serves your consumers. So before you pick a VM size, work out those three numbers for your own topics, then check the host against this list:
- Ubuntu 26.04 or 24.04 with Docker Engine and the Compose plugin; the Docker CE install for Ubuntu 26.04 covers both
- 2 GB RAM for the single node, 4 GB or more for three brokers on one host
- Disk sized as retained bytes times replication factor, plus about 30% headroom
- Free host ports 9092 (single node) and 19092, 29092, 39092 (cluster)
Our test VM had 4 vCPU (Intel Core i7-6700), 6 GB of RAM and a virtio disk on a ZFS zvol. That’s a floor for following along, not a production spec. Production brokers belong on separate hosts with identical CPU, RAM and disks, because a replicated write is only acknowledged once the slowest in-sync replica has it. Keep them in one data centre with sub-millisecond latency between them for the same reason. If you are still building the mental model of partitions, offsets and consumer groups, the Kafka and Spark books worth reading explain the design better than any Compose file can.
Set the shell variables
Two values repeat through the guide, so you set them once. HOST_IP is the LAN address clients outside Docker will use, and it goes straight into the advertised listeners, so it has to be an address your clients can actually route to. The export takes the first address hostname -I prints, and the tag is detected rather than typed:
export HOST_IP="$(hostname -I | awk '{print $1}')"
export KAFKA_VERSION=$(curl -s "https://hub.docker.com/v2/repositories/apache/kafka/tags?page_size=100" | grep -oE '"name":"[0-9]+\.[0-9]+\.[0-9]+"' | cut -d '"' -f 4 | sort -V | tail -1)
echo "Kafka image: apache/kafka:${KAFKA_VERSION} Host IP: ${HOST_IP}"We read the tag from Docker Hub on purpose. On the day we tested, downloads.apache.org/kafka/ already listed a 4.4.0 directory while Docker Hub returned HTTP 404 for apache/kafka:4.4.0, so a detection that scraped the download mirror would have handed you a tag that does not exist. The regex also skips the -rc tags Docker Hub carries for release candidates. Check the echo line before going on: if the host has several interfaces and the printed address is not the one your clients reach, set HOST_IP by hand. Ours printed this:
Kafka image: apache/kafka:4.3.1 Host IP: 10.0.1.50Exported variables only live in the current shell. If you reconnect, run them again before any later step, otherwise the .env files below get written with empty values and Compose starts brokers that advertise nothing useful.
Run a single Kafka broker
A single node runs the broker and the controller in one process, which is all a laptop or a CI job needs. Create the project directory and write its .env. The cluster ID comes from Kafka’s own kafka-storage.sh random-uuid, and you’ll generate a fresh one for the cluster later, because every broker in one cluster has to share an ID and two clusters must not.
mkdir -p ~/kafka-single && cd ~/kafka-single
printf 'KAFKA_VERSION=%s\nHOST_IP=%s\nCLUSTER_ID=%s\n' "${KAFKA_VERSION}" "${HOST_IP}" "$(docker run --rm "apache/kafka:${KAFKA_VERSION}" /opt/kafka/bin/kafka-storage.sh random-uuid)" > .env
grep -E '^(HOST_IP|CLUSTER_ID)=' .envYou should see the host address and the cluster ID that Compose substitutes into the file below. If HOST_IP is blank, the export was lost when you changed shells, so set it again and rewrite the file before going on.
HOST_IP=10.0.1.50
CLUSTER_ID=_8ZfgjkOTf2jaU2GvD_rKwNow open the Compose file. Compose reads compose.yaml and .env from the current directory, which is why both live in ~/kafka-single.
vim ~/kafka-single/compose.yamlPaste this configuration. It is the exact file we ran.
services:
kafka:
image: apache/kafka:${KAFKA_VERSION}
hostname: kafka
restart: unless-stopped
ports:
- "9092:9092"
environment:
KAFKA_NODE_ID: 1
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_LISTENERS: INTERNAL://:19092,EXTERNAL://:9092,CONTROLLER://:9093
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:19092,EXTERNAL://${HOST_IP}:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1
KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_LOG_DIRS: /var/lib/kafka/data
CLUSTER_ID: ${CLUSTER_ID}
volumes:
- kafka-data:/var/lib/kafka/data
healthcheck:
test: ["CMD-SHELL", "/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server kafka:19092 > /dev/null 2>&1"]
interval: 10s
timeout: 10s
retries: 10
start_period: 20s
volumes:
kafka-data:The listener lines do the real work here. INTERNAL://kafka:19092 is what containers on the Compose network get told to use, EXTERNAL://${HOST_IP}:9092 is what everything outside Docker gets, and KAFKA_INTER_BROKER_LISTENER_NAME keeps broker traffic on the internal side. The health check probes the internal listener, so healthy means containers can really reach the broker, not just that the JVM started. Start it and wait for that state.
The replication-factor and min-ISR lines matter on a single broker, because the defaults ask for three replicas of the internal offsets, transaction and share-group state topics, which one broker cannot provide. The two share-coordinator lines follow the official single-node example; this guide never uses share groups, but without them the first share group on this node would fail to create its state topic. kafka-configs.sh on our node confirmed both values at 1.
docker compose up -d --wait
docker compose psOn our VM the broker went healthy 8.5 seconds after up, with the image already pulled. Your STATUS column should say (healthy) before you continue.
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
kafka-single-kafka-1 apache/kafka:4.3.1 "/__cacert_entrypoin…" kafka 9 seconds ago Up 8 seconds (healthy) 0.0.0.0:9092->9092/tcp, [::]:9092->9092/tcpCreate a topic and write three messages through the internal listener. You run both from inside the broker container, so no client tooling is needed on the host yet. The -T and < /dev/null on the first line are there for pasting: docker compose exec keeps reading the terminal, and when we pasted multi-line blocks into a shell, an exec without them swallowed the lines that followed it.
docker compose exec -T kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:19092 --create --if-not-exists --topic orders --partitions 3 --replication-factor 1 < /dev/null
printf 'order-1\norder-2\norder-3\n' | docker compose exec -T kafka /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server kafka:19092 --topic ordersNow read the messages back twice, once the way a host or LAN client would and once from a separate container on the Compose network. Both paths have to work. If you only test one of them, the localhost mistake from the intro survives until the first application container tries to connect.
docker run --rm --network host "apache/kafka:${KAFKA_VERSION}" /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server "${HOST_IP}:9092" --topic orders --from-beginning --max-messages 3
docker run --rm --network kafka-single_default "apache/kafka:${KAFKA_VERSION}" /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server kafka:19092 --topic orders --from-beginning --max-messages 3Each consumer prints the three orders and Processed a total of 3 messages. The order of the messages can differ between your runs because they sit in three partitions, and the KIP-848 line is only a hint about the new consumer protocol.
order-1
order-2
order-3
Processed a total of 3 messages
The consumer rebalance protocol (KIP-848) is production-ready! Set group.protocol=consumer to try it out. See https://kafka.apache.org/documentation/#consumer_rebalance_protocolAt that point the single broker used 402 MiB of RAM. Stop it before you build the cluster, because the cluster needs the memory and doesn’t share anything with this project.
docker compose downdown without -v keeps the kafka-single_kafka-data volume. When we brought the broker back up, the three orders were still there and the data directory held 756 KB, so you can stop and start this node without losing your test data.
Deploy a three-broker Kafka Docker Compose cluster
Three brokers is the smallest cluster that survives losing one of them. The KRaft quorum needs a majority of controllers, two of three, and a topic with min.insync.replicas=2 needs two copies before it acknowledges a write. Create a separate project with its own .env and a new cluster ID.
mkdir -p ~/kafka-cluster && cd ~/kafka-cluster
printf 'KAFKA_VERSION=%s\nHOST_IP=%s\nCLUSTER_ID=%s\n' "${KAFKA_VERSION}" "${HOST_IP}" "$(docker run --rm "apache/kafka:${KAFKA_VERSION}" /opt/kafka/bin/kafka-storage.sh random-uuid)" > .envOpen the cluster Compose file in the new directory.
vim ~/kafka-cluster/compose.yamlShared settings sit in YAML anchors so the three services cannot drift apart, which is the most common way hand-copied broker blocks break. Paste the file.
x-kafka-env: &kafka-env
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka1:9093,2@kafka2:9093,3@kafka3:9093
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2
KAFKA_DEFAULT_REPLICATION_FACTOR: 3
KAFKA_MIN_INSYNC_REPLICAS: 2
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_HEAP_OPTS: "-Xms1g -Xmx1g"
KAFKA_LOG_DIRS: /var/lib/kafka/data
CLUSTER_ID: ${CLUSTER_ID}
x-kafka-health: &kafka-health
test: ["CMD-SHELL", "/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 > /dev/null 2>&1"]
interval: 10s
timeout: 10s
retries: 10
start_period: 30s
services:
kafka1:
image: apache/kafka:${KAFKA_VERSION}
hostname: kafka1
restart: unless-stopped
stop_grace_period: 2m
ports:
- "19092:19092"
environment:
<<: *kafka-env
KAFKA_NODE_ID: 1
KAFKA_LISTENERS: INTERNAL://:9092,EXTERNAL://:19092,CONTROLLER://:9093
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:9092,EXTERNAL://${HOST_IP}:19092
volumes:
- kafka1-data:/var/lib/kafka/data
healthcheck: *kafka-health
kafka2:
image: apache/kafka:${KAFKA_VERSION}
hostname: kafka2
restart: unless-stopped
stop_grace_period: 2m
ports:
- "29092:29092"
environment:
<<: *kafka-env
KAFKA_NODE_ID: 2
KAFKA_LISTENERS: INTERNAL://:9092,EXTERNAL://:29092,CONTROLLER://:9093
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka2:9092,EXTERNAL://${HOST_IP}:29092
volumes:
- kafka2-data:/var/lib/kafka/data
healthcheck: *kafka-health
kafka3:
image: apache/kafka:${KAFKA_VERSION}
hostname: kafka3
restart: unless-stopped
stop_grace_period: 2m
ports:
- "39092:39092"
environment:
<<: *kafka-env
KAFKA_NODE_ID: 3
KAFKA_LISTENERS: INTERNAL://:9092,EXTERNAL://:39092,CONTROLLER://:9093
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka3:9092,EXTERNAL://${HOST_IP}:39092
volumes:
- kafka3-data:/var/lib/kafka/data
healthcheck: *kafka-health
volumes:
kafka1-data:
kafka2-data:
kafka3-data:Four values in it are choices you should know about, because the defaults would behave differently:
KAFKA_MIN_INSYNC_REPLICAS: 2so anacks=allwrite needs two copiesKAFKA_HEAP_OPTSat 1 GB per broker, which is what made three fit in 6 GBstop_grace_period: 2mso Docker waits for a clean shutdown- one published port per broker, matching its
EXTERNALadvertised port
We added the grace period after one stop under load. Straight after the throughput benchmark and the Schema Registry test, docker compose stop with Docker’s default 10 second timeout left two brokers in Exited (137), which means Docker had to send SIGKILL. An idle cluster stopped cleanly within the default timeout in a later test (all three exited 143 in 7.9 seconds), so the risk is a busy broker that needs longer than 10 seconds, and 2 minutes gives it room. Bring the cluster up and ask the quorum how it looks.
docker compose up -d --wait
docker compose exec kafka1 /opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server kafka1:9092 describe --statusAll three brokers were healthy 26.7 seconds after up. The quorum output is your real proof that the cluster formed, with one leader, three CurrentVoters and no observers. If CurrentVoters lists only one node, the brokers did not find each other, and KAFKA_CONTROLLER_QUORUM_VOTERS is the first line to check.
ClusterId: mRWV6cAaTd6MaoiusbQcCA
LeaderId: 2
LeaderEpoch: 1
HighWatermark: 45
MaxFollowerLag: 8
MaxFollowerLagTimeMs: 4357
CurrentVoters: [{"id": 1, "endpoints": ["CONTROLLER://kafka1:9093"]}, {"id": 2, "endpoints": ["CONTROLLER://kafka2:9093"]}, {"id": 3, "endpoints": ["CONTROLLER://kafka3:9093"]}]
CurrentObservers: []Idle, each broker sat at about 300 MiB. After the benchmark further down they had grown to 1.1 GiB each, which is the 1 GB heap filling up, so budget for the full heap and not for the idle number you see right after start.
Create a replicated topic
Create a topic with six partitions and three replicas, then describe it so you can see where each copy lives.
docker compose exec -T kafka1 /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka1:9092 --create --if-not-exists --topic payments --partitions 6 --replication-factor 3 < /dev/null
docker compose exec kafka1 /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka1:9092 --describe --topic paymentsLeadership is spread across all three brokers, each partition lists three replicas, and every Isr matches its Replicas. The topic picked up min.insync.replicas=2 from the broker default, so you don’t have to set it per topic.
Topic: payments TopicId: h3qsTr5RSXuBifKL0-BPbQ PartitionCount: 6 ReplicationFactor: 3 Configs: min.insync.replicas=2
Topic: payments Partition: 0 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1 Elr: LastKnownElr:
Topic: payments Partition: 1 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2 Elr: LastKnownElr:
Topic: payments Partition: 2 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3 Elr: LastKnownElr:
Topic: payments Partition: 3 Leader: 2 Replicas: 2,1,3 Isr: 2,1,3 Elr: LastKnownElr:
Topic: payments Partition: 4 Leader: 1 Replicas: 1,3,2 Isr: 1,3,2 Elr: LastKnownElr:
Topic: payments Partition: 5 Leader: 3 Replicas: 3,2,1 Isr: 3,2,1 Elr: LastKnownElr: Write 1,000 messages with acks=all. Kafka 4.2 deprecated --producer-property in the console tools in favour of --command-property, and the old flag is due to go in Kafka 5.0, so the command uses the new one.
seq 1 1000 | sed 's/^/payment-/' | docker compose exec -T kafka1 /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server kafka1:9092 --topic payments --command-property acks=allA successful console producer prints nothing. The offsets in the next section are how you check that the messages landed.
Kill a broker and watch the ISR
Stopping a broker is the test that tells you whether the replication settings are right, and it is far cheaper to run it now than to discover the answer during an outage. Stop kafka3 and describe the topic again.
docker compose stop kafka3
docker compose exec kafka1 /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka1:9092 --describe --topic paymentsWithin five seconds every partition dropped broker 3 from its ISR, and the two partitions it led moved their leadership to brokers 1 and 2. Your output should show Isr lists of two entries everywhere.
With two in-sync replicas left, acks=all writes should still succeed. Write 500 more messages while kafka3 is down to prove it:
seq 1001 1500 | sed 's/^/payment-/' | docker compose exec -T kafka1 /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server kafka1:9092 --topic payments --command-property acks=allAll 500 landed in both of our runs, although one run logged a REQUEST_TIMED_OUT ... retrying warning while the client noticed that kafka3 had gone, which you can ignore when the offsets add up. Now stop kafka2 as well and try one more write. One broker is below min.insync.replicas, so the cluster should refuse the write instead of accepting data it cannot replicate:
docker compose stop kafka2
echo "payment-lost" | docker compose exec -T kafka1 /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server kafka1:9092 --topic payments --command-property acks=allThe console producer’s default of three retries ran out in under a second, and it then logged the rejection. It still exits with status 0 here, so check the log, not $?, when you script this test:
[2026-10-18 14:16:57,578] WARN [Producer clientId=console-producer] Got error produce response with correlation id 5 on topic-partition payments-1, retrying (2 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender)
[2026-10-18 14:16:57,650] WARN [Producer clientId=console-producer] Got error produce response with correlation id 6 on topic-partition payments-1, retrying (1 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender)
[2026-10-18 14:16:57,850] WARN [Producer clientId=console-producer] Got error produce response with correlation id 7 on topic-partition payments-1, retrying (0 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender)
[2026-10-18 14:16:58,323] ERROR Error when sending message to topic payments with key: null, value: 12 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required.That refusal is the behaviour you want. With combined broker and controller nodes, losing two of three also loses the metadata quorum, so a three-node cluster on one host gives you no protection against the host itself. Start both brokers again and wait until no partition is under-replicated. The loop only exits when the describe command succeeds and prints nothing, so a broker that is still refusing connections keeps it waiting instead of ending it early.
docker compose start kafka2 kafka3
until out=$(docker compose exec -T kafka1 /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka1:9092 --describe --under-replicated-partitions < /dev/null) && [ -z "$out" ]; do sleep 2; done
docker compose exec kafka1 /opt/kafka/bin/kafka-get-offsets.sh --bootstrap-server kafka1:9092 --topic paymentsReplication caught up 31 seconds after the start command in our first run and after 196 seconds in a re-run, so give it a few minutes before you call it stuck. The offsets add up to 1,500, so the 500 messages written during the outage are there and the refused one is not. They sit in one or two partitions (two in our first run, one in the re-run) because the console producer sends unkeyed records in batches to one partition at a time.
payments:0:0
payments:1:0
payments:2:1000
payments:3:500
payments:4:0
payments:5:0After the recovery all six partitions were led by broker 1, the only one that never stopped. Kafka moves leadership back on its own, because auto.leader.rebalance.enable is true by default and leader.imbalance.check.interval.seconds is 300 on these brokers. You don’t have to wait five minutes, though, since you can trigger the election straight away.
docker compose exec kafka1 /opt/kafka/bin/kafka-leader-election.sh --bootstrap-server kafka1:9092 --election-type PREFERRED --all-topic-partitionsFour partitions moved back to their preferred leaders, which spreads your write load over all three brokers again.
Successfully completed leader election (PREFERRED) for partitions payments-3, payments-5, payments-0, payments-1Run the same election after any planned broker restart, otherwise one broker carries every partition leader until the five-minute check catches up.
Measure throughput with kafka-producer-perf-test
A benchmark on one VM measures that VM, so read these numbers as a description of our test box, not of Kafka. Create a dedicated topic and run the producer test three times with 1 million 1 KiB records and acks=all.
docker compose exec -T kafka1 /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka1:9092 --create --if-not-exists --topic perf --partitions 6 --replication-factor 3 < /dev/null
docker compose exec -T kafka1 /opt/kafka/bin/kafka-producer-perf-test.sh --topic perf --num-records 1000000 --record-size 1024 --throughput -1 --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --command-property acks=all linger.ms=5 batch.size=65536The consumer test reads the same records back. Use a new --group name for every run, otherwise the group starts from its committed offset and reads nothing. The 3 minute timeout is deliberate. The first consumer group on a fresh cluster spent 42 seconds in rebalance in our first run and 100 seconds in a re-run, and an earlier attempt with --timeout 60000 exited with 0 records consumed. The second group on the same cluster joined in 1.2 seconds:
docker compose exec -T kafka1 /opt/kafka/bin/kafka-consumer-perf-test.sh --bootstrap-server kafka1:9092 --topic perf --num-records 1000000 --group perf-1 --timeout 180000Our four producer runs and two consumer runs gave the results below. Those runs used the older --producer-props and --messages flags, which Kafka 4.2 deprecated in favour of --command-property and --num-records (removal is planned for 5.0). Three later validation runs with the flags shown above produced 34,477, 31,394 and 23,732 records/s. Each was a single first run on its own, so they compare with run 1 rather than with runs 2 and 3.
| Run | Records/s | MB/s | p50 latency | Max latency |
|---|---|---|---|---|
| Producer, acks=all, run 1 | 35,252 | 34.43 | 607 ms | 3,689 ms |
| Producer, acks=all, run 2 | 14,039 | 13.71 | 361 ms | 55,455 ms |
| Producer, acks=all, run 3 | 11,004 | 10.75 | 292 ms | 74,761 ms |
| Producer, acks=1 | 17,619 | 17.21 | 107 ms | 39,641 ms |
| Consumer, first pass | 18,866 | 18.42 | 42 s spent in group rebalance | |
| Consumer, second pass | 211,764 | 206.80 | 1.1 s rebalance |
The falling producer numbers are the disk, not Kafka. All three brokers write to the same virtual disk, so 34 MB/s of records becomes about 100 MB/s of log writes with replication factor 3, and a dd test with conv=fdatasync on the same VM sustained only 41 MB/s. The first run fit in the page cache, and the later ones waited on flushes, which is where the 55 to 75 second worst-case latencies come from. The second consumer pass read 977 MB at 207 MB/s because the data was still cached. On real hardware you would put each broker on its own disk before reading anything into these figures. The capture below is a validation run with the current flags on a fresh cluster: 23,732 records/s from the producer, then 100 seconds of rebalance for the first consumer group and 1.2 seconds for the second.
Our perf topic left about 5.46 GB in each broker’s volume. Delete it once you’ve read the numbers.
docker compose exec kafka1 /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka1:9092 --delete --topic perfDeleting it now also keeps the benchmark data out of the memory and disk numbers of the later steps.
Connect a Python client
Application code outside Docker connects through the external listeners. The current kafka-python release on PyPI worked against Kafka 4.3 in our run, so the kafka-python-ng fork older guides used is not needed. Create the test topic first, then install the client into a virtual environment so it stays out of the system Python:
cd ~/kafka-cluster
docker compose exec -T kafka1 /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka1:9092 --create --if-not-exists --topic events --partitions 6 --replication-factor 3 < /dev/null
sudo apt install -y python3-venv
mkdir -p ~/kafka-client && cd ~/kafka-client
python3 -m venv .venv
.venv/bin/pip install kafka-pythonPip installed kafka-python 3.0.11 on Python 3.14. Now create the producer script.
vim ~/kafka-client/producer.pyThis script sends 100,000 JSON events with acks=all and prints the rate when the last one is acknowledged.
import json
import os
import time
from kafka import KafkaProducer
bootstrap = os.environ.get("KAFKA_BOOTSTRAP", "localhost:19092")
producer = KafkaProducer(
bootstrap_servers=bootstrap,
acks="all",
linger_ms=5,
value_serializer=lambda v: json.dumps(v).encode("utf-8"),
)
start = time.time()
for i in range(100000):
producer.send("events", {"event": f"evt-{i}", "ts": time.time()})
producer.flush()
elapsed = time.time() - start
print(f"Sent 100000 messages in {elapsed:.2f}s ({100000 / elapsed:.0f} msg/s)")The consumer reads them back with a group ID, so a second run continues from the committed offset instead of starting over; with no new messages it sits silent for the full 3 minute timeout and then prints Consumed 0 messages. Its consumer_timeout_ms is 3 minutes on purpose. With 10 seconds, our first run on a cluster where no consumer group had been used yet consumed 0 messages, because joining the group took longer than the timeout. The perf test above shows that first join can take well over a minute. Create it next to the producer.
vim ~/kafka-client/consumer.pyPaste the consumer.
import json
import os
import time
from kafka import KafkaConsumer
bootstrap = os.environ.get("KAFKA_BOOTSTRAP", "localhost:19092")
consumer = KafkaConsumer(
"events",
bootstrap_servers=bootstrap,
group_id="events-reader",
auto_offset_reset="earliest",
consumer_timeout_ms=180000,
value_deserializer=lambda m: json.loads(m.decode("utf-8")),
)
count = 0
start = time.time()
for message in consumer:
count += 1
if count == 100000:
break
elapsed = time.time() - start
print(f"Consumed {count} messages in {elapsed:.2f}s ({count / elapsed:.0f} msg/s)")
consumer.close()Point both scripts at all three external listeners and run them. Listing every broker means your client can still bootstrap when one of them is down.
export KAFKA_BOOTSTRAP="${HOST_IP}:19092,${HOST_IP}:29092,${HOST_IP}:39092"
.venv/bin/python producer.py
.venv/bin/python consumer.pyOn a fresh cluster our producer managed 15,189 messages a second, and the consumer read all 100,000 in 9.14 seconds, which includes joining the first consumer group. kafka-python 3.x warns when you pass a plain lambda as value_deserializer, but the lambda still works.
Sent 100000 messages in 6.58s (15189 msg/s)
/home/josphat/kafka-client/consumer.py:8: DeprecationWarning: value_deserializer does not implement kafka.serializer.Deserializer
consumer = KafkaConsumer(
Consumed 100000 messages in 9.14s (10938 msg/s)Both scripts used the external listeners only, so the same code works unchanged from another machine on the LAN once KAFKA_BOOTSTRAP points at the host address.
Add Schema Registry
Schema Registry is optional. You only need it once several teams share topics and you want a producer to fail when it sends a payload that breaks the agreed format. Keep it in its own override file so the cluster file stays unchanged.
vim ~/kafka-cluster/compose.schema-registry.yamlPaste the service definition. It binds 8081 to the loopback address, because the registry has no authentication in this form and you don’t want it on the LAN.
services:
schema-registry:
image: confluentinc/cp-schema-registry:8.2.4
hostname: schema-registry
restart: unless-stopped
ports:
- "127.0.0.1:8081:8081"
environment:
SCHEMA_REGISTRY_HOST_NAME: schema-registry
SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: kafka1:9092,kafka2:9092,kafka3:9092
SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081
depends_on:
kafka1:
condition: service_healthy
kafka2:
condition: service_healthy
kafka3:
condition: service_healthyStart it together with the cluster file, wait until it answers and register a test schema. The image has no health check, and our registry took 18 to 32 seconds to answer after up -d, so without the until line the first curl prints nothing and the POST fails silently.
cd ~/kafka-cluster
docker compose -f compose.yaml -f compose.schema-registry.yaml up -d
until curl -sf http://127.0.0.1:8081/subjects >/dev/null; do sleep 2; done
curl -s http://127.0.0.1:8081/subjects; echo
curl -s -X POST http://127.0.0.1:8081/subjects/events-value/versions -H "Content-Type: application/vnd.schemaregistry.v1+json" -d '{"schema": "{\"type\":\"record\",\"name\":\"Event\",\"fields\":[{\"name\":\"event\",\"type\":\"string\"},{\"name\":\"ts\",\"type\":\"double\"}]}"}'; echoThe empty list comes first, then the registry returns schema ID 1. Schemas live in a compacted _schemas topic, which the registry created with replication factor 3, so they survive the loss of a broker like any other topic.
[]
{"id":1,"version":1,"guid":"7e3fc17b-d515-662b-f01b-5fadabf35c33","schemaType":"AVRO","schema":"{\"type\":\"record\",\"name\":\"Event\",\"fields\":[{\"name\":\"event\",\"type\":\"string\"},{\"name\":\"ts\",\"type\":\"double\"}]}"}On our box the registry container used 281 MiB. If no one in your setup validates schemas yet, leave it out.
Upgrade with a rolling restart
You upgrade a Compose cluster one broker at a time, wait for a full ISR between each, and raise the metadata version only once every broker runs the new image. We rehearsed it twice on a separate cluster that started on 4.2.2, with a producer writing 500 records a second the whole time. When a new release appears, re-run the KAFKA_VERSION export from the variables step, then write the tag and pull.
cd ~/kafka-cluster
sed -i "s/^KAFKA_VERSION=.*/KAFKA_VERSION=${KAFKA_VERSION}/" .env
docker compose pullThen recreate the brokers in turn. --no-deps --wait restarts only the named service and returns when it is healthy, and the until loop holds the next broker back until the describe command succeeds and reports no under-replicated partition, so an admin error never counts as caught up. If Schema Registry is running, Compose warns about an orphan schema-registry container on each step, because that service lives in the override file, and the warning is harmless.
for b in kafka3 kafka2 kafka1; do
docker compose up -d --no-deps --wait "$b"
until out=$(docker compose exec -T kafka1 /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --describe --under-replicated-partitions < /dev/null) && [ -z "$out" ]; do sleep 2; done
doneIn the first drill each broker came back with a full ISR after 36, 53 and 53 seconds. At that point the brokers run the new binaries but still speak the old metadata version, which kafka-features.sh describe showed as 4.2-IV1 on our drill cluster. Raise only the metadata version, derived from the tag you deployed:
docker compose exec kafka1 /opt/kafka/bin/kafka-features.sh --bootstrap-server kafka1:9092 upgrade --metadata "${KAFKA_VERSION%.*}"Kafka 4.3 prints a note that the --metadata flag is deprecated, then confirms the change:
`metadata` flag is deprecated and may be removed in a future release.
metadata.version was upgraded to 30.Then check every feature level, because --metadata raises only the metadata version and a newer release may also add or raise others:
docker compose exec kafka1 /opt/kafka/bin/kafka-features.sh --bootstrap-server kafka1:9092 describeOn our cluster metadata.version read 4.3-IV0 and kraft.version stayed at 0. The other features (group, share, streams, transaction and eligible leader replicas) were already at their 4.3 levels, so nothing else needed raising:
Feature: eligible.leader.replicas.version SupportedMinVersion: 0 SupportedMaxVersion: 1 FinalizedVersionLevel: 1 Epoch: 458
Feature: group.version SupportedMinVersion: 0 SupportedMaxVersion: 1 FinalizedVersionLevel: 1 Epoch: 458
Feature: kraft.version SupportedMinVersion: 0 SupportedMaxVersion: 1 FinalizedVersionLevel: 0 Epoch: 458
Feature: metadata.version SupportedMinVersion: 3.3-IV3 SupportedMaxVersion: 4.3-IV0 FinalizedVersionLevel: 4.3-IV0 Epoch: 458
Feature: share.version SupportedMinVersion: 0 SupportedMaxVersion: 1 FinalizedVersionLevel: 1 Epoch: 458
Feature: streams.version SupportedMinVersion: 0 SupportedMaxVersion: 1 FinalizedVersionLevel: 1 Epoch: 458
Feature: transaction.version SupportedMinVersion: 0 SupportedMaxVersion: 2 FinalizedVersionLevel: 2 Epoch: 458If a line shows FinalizedVersionLevel below SupportedMaxVersion after a future upgrade, raise it with --feature <name>=<level>, which can sit on the same command as --metadata; a dry run of upgrade --metadata 4.3 --feature group.version=1 --dry-run accepted both together. The non-deprecated form for the metadata version is --feature metadata.version=<level>, but it takes the numeric level (30 for 4.3, the number the upgrade printed above), not 4.3-IV0, which the tool rejected with unable to parse 4.3-IV0 as a short. Leave kraft.version at 0 unless you also migrate the quorum configuration, as the next paragraph explains.
The upgrade notes use upgrade --release-version 4.3 instead, and we ran that too in the first drill. It raised kraft.version from 0 to 1 as well as the metadata version; the other features were already at their 4.3 levels and the tool simply reports them again. According to the Kafka KRaft operations guide, kraft.version 1 means a dynamic quorum, where controller.quorum.voters (the KAFKA_CONTROLLER_QUORUM_VOTERS line in this Compose file) is deprecated and must be replaced by controller.quorum.bootstrap.servers on every node. The metadata-only command avoids that migration and keeps the Compose file valid as written, which is why we use it here.
Both drills kept every record: 90,000 sent and 90,000 in the topic the first time, 60,000 and 60,000 the second. The producer’s worst latency was 1,219 ms in the first drill and 16.6 seconds in the second, and the first run’s log had 25 lines mentioning an error or exception while leaders moved, so a client with retries and idempotence enabled rides through a rolling upgrade without losing data. Treat the metadata bump as one-way. The Kafka 4.3 upgrade notes say cluster metadata downgrade is not supported in this version, so raise it only once you are sure you will not roll the images back.
Fix client connection errors
WARN Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Node may not be available.
We reproduced the classic advertised.listeners mistake with a single broker that advertised PLAINTEXT://localhost:9092. A client on the host worked and created a topic. A producer in another container on the same Compose network bootstrapped from kafka:9092, received localhost:9092 in the metadata and then tried to connect to itself.
[2026-10-18 12:48:58,830] WARN [Producer clientId=console-producer] Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
[2026-10-18 12:48:58,947] WARN [Producer clientId=console-producer] Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)Two listeners, one per audience, fix it, and the Compose files above already use that layout. The name in the error tells you which side is wrong. Seeing localhost from inside a container means the internal advertised address is missing, while a container hostname such as kafka1 showing up on your laptop means the external one is.
Error while executing topic command : Topic ‘payments’ already exists.
We hit this on a cluster that had been created seconds earlier, with no topic of that name ever made by hand. All three brokers reported healthy, but the metadata quorum had already gone through seven leader epochs, and the most likely explanation is that the admin client retried a create request whose first attempt had already succeeded. Describe the topic before doing anything else:
docker compose exec kafka1 /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka1:9092 --describe --topic paymentsIn our case it showed six partitions with three in-sync replicas each, exactly what the create asked for, so nothing needed fixing. If the partition count or replication factor differs from what you asked for, the topic really did exist before, and you should delete it or pick another name. The create commands in this guide carry --if-not-exists for this reason: in a later run the same command, issued twice, exited with status 0 both times and created the topic once.
Clients outside the host can reach Kafka although UFW blocks 9092
No error message here, just a surprise. With UFW active and only SSH, 80 and 443 allowed, a second machine on our LAN still opened a TCP connection to port 9092, because Docker inserts its own iptables rules for published ports ahead of UFW. Bind the ports to a specific address in Compose, for example "10.0.1.50:19092:19092" (we didn’t test that variant), or filter in the DOCKER-USER chain as the Docker CE guide shows.
Harden the cluster before production
Everything above runs PLAINTEXT on one host, which is fine for development and wrong for anything holding real data. Before you put production traffic on it, change these:
- Brokers on separate hosts and disks, identical in size
- SASL or mTLS on the external listener, and TLS between brokers
- Published ports bound to the client network only
stop_grace_periodlong enough for a clean shutdown- Broker metrics in Prometheus with alerts on under-replicated partitions
Metrics come first on that list because an under-replicated partition is silent until a second broker fails. The Kafka monitoring guide with Prometheus and Grafana covers getting broker metrics into Prometheus and onto dashboards, and the alert you want before anything else is under-replicated partitions above zero for more than a few minutes.
If the JVM footprint is the real problem rather than security, the Kafka vs Redpanda benchmarks compare both on identical hardware. The figure to compare is the one we measured here: each JVM broker grew to 1.1 GiB after the benchmark, with a 1 GB heap.
Once topics need processing rather than just storage, a Spark standalone cluster on Ubuntu 26.04 is the usual next stop. For a broker without Docker at all, the Kafka install guide for Ubuntu runs it under systemd instead, which some teams prefer when the broker is the only service on the host.
