dozeGitHubGet doze
doze Kafka

Wakes a Kafka broker. No JVM to put to bed.

Part of the doze family. Works alone.

A native broker that boots in milliseconds, sits quiet until a client connects, keeps your data on disk, and shuts down cleanly. One Go binary, single node: no JVM, no ZooKeeper, no KRaft quorum, no Docker.

-Xmx1Gwaiting for controller quorum…all you wanted was one topic
Install doze KafkaGetting started
on its ownloopback only
$ doze-kafka
  broker   127.0.0.1:9092
  console  http://127.0.0.1:9093/_console/

$ kcat -b 127.0.0.1:9092 -t orders.placed -P
$ kcat -b 127.0.0.1:9092 -t orders.placed -C -e
{"order_id": "[id]", "total": "[amount]"}
The console

See every record. And why it's there.

Topics, the tape of records on disk, messages decoded, and underneath, the requests that touched them. On by default.

The doze Kafka console: the orders topic, its tape of 40 records on disk, the messages, and the requests underneath
The tapeEach mark is one record on disk, taller is bigger, newest on the right.

The real console, from its demo stack: 40 records produced with kcat. re-capture with Larder

What it does

If your client can't tell, it's Kafka.

Built to behave like Kafka to any client on the socket, and to leave out what no client can observe. Five things, one at a time.

01 / 05 · Clients

Tested against your client, in CI, on every change.

One binary speaks the Kafka wire protocol 1 to 4; pick which with --kafka-version (default 4). Consumer groups use the classic protocol or KIP-848, chosen by the client.

Clients in CI
  • franz-go
  • Sarama
  • Java client
  • Kafka Streams
  • Spring Kafka
  • librdkafka
  • kcat
  • confluent-kafka
  • KafkaJS
Wire protocolKafka 1 – 4
Consumer groupsclassic · KIP-848
Default--kafka-version 4
02 / 05 · The console

A console that explains.

Topics, messages, lag, and why a group keeps rebalancing, with the Kafka error names. Here kcat produces records: each request shows up in the traffic, then on the topic's tape.

the real console, from its demo stack · re-capture with Larder
03 / 05 · Schema registry

A schema registry, built in.

Confluent-compatible, for Avro, JSON Schema and Protobuf. Stock serializers need only schema.registry.url, and doze-kafka env prints it for you.

  • Avro
  • JSON Schema
  • Protobuf
$ eval "$(doze-kafka env)"
$ env | grep -E 'KAFKA|SCHEMA'
KAFKA_BOOTSTRAP_SERVERS=127.0.0.1:9092
KAFKA_BROKERS=127.0.0.1:9092
SCHEMA_REGISTRY_URL=[the registry's address]

# your serializer's config, and nothing else
schema.registry.url = $SCHEMA_REGISTRY_URL
04 / 05 · Transactions

Transactions, SASL and TLS.

Transactional producers and read-committed consumers, so the exactly-once code you ship runs here first. SASL and TLS when you want to test them, off when you don't.

Producerstransactional
Consumersread_committed
SASLPLAIN · SCRAM-256 · SCRAM-512
TLSwhen you want it
Under SASLdoze-kafka envprints KAFKA_SASL_MECHANISM, never a password
05 / 05 · Durable and small

Durable, flat, embeddable.

Stop it with Ctrl-C: topics, messages and committed offsets are all there next time. And in Go tests it is a library.

Durableacks=all

An acknowledged write is fsynced before the ack.

Flat memorytens of MB

Idle memory stays there however much data sits on disk.

In Go testsserver.New(...)

Starts a broker on a free port, with nothing to clean up.

Works alone

On its own, or inside a doze stack.

The same broker either way. On its own it answers on loopback, so nothing on your network can reach it. Inside doze it gets a name, so two projects can both have their 9092.

On its own

One command, on loopback.

Run doze-kafka: the broker listens on 127.0.0.1:9092 and its console on 127.0.0.1:9093/_console/. Linux and macOS, amd64 and arm64, Intel Macs included.

$ doze-kafka
  broker   127.0.0.1:9092
  console  http://127.0.0.1:9093/_console/
^C
# topics, messages and offsets are there next time
Inside doze

The topics are the contracts.

doze add kafka, and in Larder, our made-up example project, doze names it kafka.larder.doze. Services in Go, Python and TypeScript (franz-go, confluent-kafka, KafkaJS) talk through four topics declared in doze.hcl, not created by whichever client writes first. doze itself needs Apple silicon on a Mac.

# doze.hcl, from the Larder example
module kafka "kafka" {
  version            = 4
  port               = 9092
  auto_create_topics = false

  topic "users.signed-up"  { partitions = 3 }
  topic "orders.placed"    { partitions = 3 }
  topic "payments.settled" { partitions = 3 }
  topic "payments.failed"  { partitions = 3 }
}

# and in each app that uses it
KAFKA_BROKERS = module.kafka.address
Honest by design

What it isn't, on purpose.

A single node for development, not production. Anything a client can't observe through the socket is left out.

  • No replication, ISR or controller quorum
  • No byte-for-byte on-disk segment compatibility
  • No tiered storage or quotas

Kafka for your laptop. Hold the JVM.

On its own: Linux and macOS, amd64 and arm64
$ curl -fsSL https://raw.githubusercontent.com/doze-dev/doze-kafka/main/install.sh | sh
In a doze stack (Apple silicon on a Mac)
$ doze add kafka
More ways, and doze itself: Download · Docs

Apache Kafka is a trademark of the Apache Software Foundation. doze Kafka is not affiliated with it.

Part of the doze family. Works alone.

Three products, each its own download, each happy on its own. Together, doze runs the other two inside your stack and gives them names.

dozeWakes your stack when you start. Sleeps it when you stop.doze AWSWakes AWS on your laptop. No account, no bill.doze Kafkayou're here · docs →Wakes a Kafka broker. No JVM to put to bed.