Configuration

doze-kafka is configured with command-line flags or a TOML config file — every setting is available either way, under the same name (with two exceptions, the [credentials] and [tls] tables, shown below). Every setting has a sensible default, so ./doze-kafka with no arguments is a complete, working broker. Run ./doze-kafka --help to see the flags, grouped, at any time.

Config file

Drop a doze-kafka.toml in your working directory and doze-kafka loads it automatically; point elsewhere with --config path/to/file.toml, or run without any file with --config "". A key is its flag’s name without the dashes in front. The two that differ are tables: SASL users (--sasl-user alice:s3cret) are a [credentials] table, and --tls-cert / --tls-key are cert and key under [tls]. Put tables last: in TOML, every key after a [table] line belongs to that table.

name = "harbour"            # answers on kafka.harbour.doze (default: this directory's name)
data-dir = "./data"
retention-ms = 604800000   # 7 days
fsync-interval = "500ms"

sasl-mechanisms = ["SCRAM-SHA-256"]

[credentials]
alice = "s3cret"

[tls]
cert = "./certs/server.pem"
key  = "./certs/server-key.pem"

Copy doze-kafka.example.toml for a fully commented starting point.

Precedence is flags > file > defaults. The file overlays only the keys it names — everything else stays at its default — and any flag still wins for a one-off override, so you can commit a doze-kafka.toml for your project and still run ./doze-kafka --kafka-version 3 for a quick experiment. A mistyped key is rejected rather than silently ignored.

An error about the file names the line and the key and never repeats what is written there, because the file that fails to load is often one with a password in the wrong place:

doze-kafka: read doze-kafka.toml: toml: line 7 (key "tls.key"): this is not valid TOML (a string is written in quotes); what was written is not repeated, because this file can hold passwords

Under [credentials] the key is not named either (it is where a user’s name goes, and a password on a line of its own is read as one): the error gives the line. credentials written as anything but a table (credentials = "...") is refused.

Two things a file does better than flags: it keeps a list of SASL users readable (and keeps passwords out of your shell history and ps output), and it’s committable — check doze-kafka.toml into your repo and the whole team gets the same broker.

doze-kafka config print writes your effective configuration — defaults, then the file, then any flags — as TOML, so you can see exactly what the broker will run with, or generate a starter file. Every setting is printed, defaults included; meta-db, unless you set it, is printed as a comment because it follows data-dir. It validates first, so what it prints is something the broker would accept, and starting from the printed file gives the same configuration:

doze-kafka config print > doze-kafka.toml

Passwords are never printed: each [credentials] entry is written as "<redacted>", so the output is safe to paste into an issue or a chat. A file with the placeholder still in it is refused when it is loaded, by user (it would otherwise be the password), so put the passwords back, or print with config print --show-secrets to write a file the broker can start from. The same rule holds everywhere else: no password and no key material is in the start-up summary, the log (at any level), the console, a DescribeConfigs answer or an error message (a malformed --sasl-user or credentials line names the flag or the line and not the value).

A bad setting is reported once, before anything starts, as a single line, with exit code 2 (a broker that fails while running exits 1). That covers a mistyped flag, a value out of range, a TLS certificate that is missing or does not match its key, and --zone=false with no --listen:

$ doze-kafka --listen nonsense
doze-kafka: listen address "nonsense" is not host:port (for example 127.0.0.1:9092 or :9092)
$ doze-kafka --listn :9092
doze-kafka: unknown flag --listn — did you mean --listen?

All flags

Every flag below has the same name as its config-file key, apart from the two tables above (durations like --fsync-interval are written as strings, e.g. fsync-interval = "500ms"). A file written for an earlier version may say version where this one says kafka-version; that spelling is still read.

Network

Flag	Default	Meaning
--name	this directory’s name	This instance’s name: it answers on kafka.<name>.doze:9092, on a loopback address of its own, so two projects can both be on port 9092 and nothing on your network can reach it.
--listen	(unset)	Serve this host:port instead of the name: no name is claimed, no DNS check runs, nothing is written to the registry. For a case a name cannot serve, such as a sibling container. :9092 accepts other machines and containers (read the note on --advertised-host below); 127.0.0.1:9092 restores the old loopback default; 127.0.0.1:0 is a free port. On Windows, which has no names, the default is 127.0.0.1:9092.
--advertised-host	derived	Host returned to clients in Metadata: a host name or an IP address, without a port. This is the address clients actually reconnect to — see the note below. Unset, it is the instance name (or its address where the name does not resolve on this machine); under --listen it follows the address: a specific one is advertised as itself, and a wildcard (:9092) as localhost.
--advertised-port	0	Port advertised to clients. 0 means the port actually bound, which is right even with :0.

Storage

Flag	Default	Meaning
--data-dir	./data	Directory holding per-partition log segments.
--meta-db	<data-dir>/meta.db	File holding the topic registry, committed offsets, and producer-id counter. It lives inside the data directory by default, so the directory is self-contained: move it or delete it and the topics and offsets go with it. Earlier versions kept it at ./meta.db in the working directory. That file is still used (with a notice) when the data directory has partition data but no database of its own, however the data directory is spelled; run from a different working directory and it is not found, so move it into the data directory when you can.
--segment-bytes	134217728 (128 MiB)	Roll to a new log segment once the active one exceeds this.
--index-interval-bytes	4096	Add one sparse index entry per this many bytes of log.

Retention (both default to unbounded)

Flag	Default	Meaning
--retention-bytes	0	Delete oldest segments once a partition exceeds this size. 0 = keep everything.
--retention-ms	0	Delete segments older than this many milliseconds. 0 = keep everything. (The most is about 292 years; for “for ever”, use 0.)

Durability

Flag	Default	Meaning
--fsync-interval	500ms	How often active segments are flushed to disk, at least 1ms. 0 disables the timer (the OS still flushes eventually; a clean shutdown always syncs). acks=all always fsyncs before it is acknowledged.

Topics and groups

Flag	Default	Meaning
--kafka-version	4	Which Kafka major (1–4) to emulate. Sets the API versions advertised and whether the new KIP-848 group protocol is offered. See API support. (This was --version; --version now prints doze-kafka’s own version. --version 3 is still understood, with a notice.)
--auto-create-topics	true	Create a topic the first time a client references it.
--default-partitions	1	Partition count for auto-created topics.
--max-partitions	1000	Most partitions one topic may have. Each is a directory and open files, so a typo like --default-partitions 100000 is refused rather than taking the machine down.
--message-max-bytes	1048588	Largest record batch a topic accepts unless the topic sets max.message.bytes: Kafka’s message.max.bytes. A larger batch is refused MESSAGE_TOO_LARGE, as a production broker refuses it.
--consumer-session-timeout	45s	KIP-848 group session timeout: evict a member after this long without a heartbeat. (Classic groups use the timeout the client requests.)
--group-initial-rebalance-delay	200ms	How long the first rebalance of an empty classic group waits for more members before it completes, and waits again while more keep joining, up to the rebalance timeout: Kafka’s group.initial.rebalance.delay.ms, whose default is 3s (see limitations). 0 completes it as soon as everyone has joined.
--offsets-retention	168h	How long a group’s committed offsets are kept once nothing is reading them: Kafka’s offsets.retention.minutes (7 days). 0 keeps them for ever. The rules are in consumer groups.
--client-telemetry	true	Accept the metrics clients push about themselves (KIP-714: Java 3.7+, librdkafka 2.5+) and show the last few minutes in the console. --client-telemetry=false stops advertising the two APIs, so clients stop sending. See Observability.
--zone	true	Join the shared .doze zone: claim kafka.<name>.doze and route the console through the front door. --zone=false is the same as DOZE_ZONE=off and needs --listen, since there is then no name to listen on: without it, either is a configuration error (one line, exit 2).

Authentication (all optional; see Auth & TLS)

Flag	Default	Meaning
--sasl-mechanisms	(none)	Comma-separated SASL mechanisms to require: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512. Enabling any turns SASL on. A flag replaces the file’s list.
--sasl-user	(none)	A user:password credential. Repeatable. Needs --sasl-mechanisms: users without a mechanism are an error, not silently ignored.
--tls-cert	(none)	PEM certificate file. With --tls-key, terminates TLS on the Kafka listener and on the schema registry (see --schema-registry-tls). The console stays plain HTTP.
--tls-key	(none)	PEM private key file.
--acl-default	allow	What a resource with no ACL allows: allow, or deny (Kafka’s own default). ACLs you create are enforced either way. See Authorization.
--acl-super-user	(none)	A principal ACLs never refuse, as User:admin. Repeatable.

Console (see Console)

Flag	Default	Meaning
--console	true	Serve the web console. --console=false to turn it off.
--console-addr	beside the broker	host:port for the console’s HTTP listener. Unset, it is port 9093 on the name’s own address (and http://kafka.<name>.doze, port-less, through the shared front door); under --listen, 127.0.0.1:9093, or a free loopback port when --listen is on port 0. A taken port is an error naming this flag.
--console-prefix	/_console	The URL path the console is served under, for a reverse proxy that mounts it below a path of its own. / serves it at the root of its port.
--console-allow-host	(none)	Extra Host names the console answers to, besides localhost, IP addresses and *.doze. Comma-separated or repeated. A guard against DNS rebinding.
--console-unprotected	false	Serve the console beyond loopback although SASL or TLS is on. The console has no login, is plain HTTP and, with SASL, acts as a super user, so without this flag that combination is refused at start-up. With it, a warning says what is open. See What SASL and TLS cover.
--trace-header	(none)	A record header whose whole value is a trace id (a request or correlation id). The console’s Follow this trace looks for it in addition to W3C traceparent and B3. See Observability.

Schema registry (see Schema Registry)

Flag	Default	Meaning
--schema-registry	true	Serve the built-in Confluent-compatible schema registry. --schema-registry=false to turn it off.
--schema-registry-addr	beside the broker	host:port for the registry’s HTTP listener. Unset, it is port 8081 on the name’s own address (and http://schema.<name>.doze, port-less, through the shared front door); under --listen, 127.0.0.1:8081, or a free loopback port when --listen is on port 0. A taken port is an error naming this flag.
--schema-registry-tls	true	Serve the registry over HTTPS, with the broker’s certificate, when --tls-cert and --tls-key are set; without them it changes nothing. --schema-registry-tls=false keeps the registry plain HTTP beside a TLS broker, and warns at start-up if it is then on an address other machines can reach.

Process

Flag	Default	Meaning
--config	./doze-kafka.toml if present	Path to a TOML config file. --config "" uses none, ignoring a doze-kafka.toml in the working directory.
--log-level	info	debug, info, warn or error. debug logs every request (client id, API, version).
--log-format	text	text or json.
--pprof-addr	(none)	If set, serve net/http/pprof there (for soak and debugging).

Commands

Command	Does
doze-kafka version / --version / -v	Print doze-kafka’s version and the Kafka range it speaks.
doze-kafka env	Print the shell block that points a client at this instance (eval "$(doze-kafka env)"): KAFKA_BOOTSTRAP_SERVERS and KAFKA_BROKERS, SCHEMA_REGISTRY_URL while the registry is on, and KAFKA_SASL_MECHANISM (the first of --sasl-mechanisms) under SASL; never a password. It reads its own flags and doze-kafka.toml, not a running broker: under --listen, run the one the start-up summary prints.
doze-kafka doctor	Report what is set up for .doze names and what is not, and which process holds which name. Needs no privilege.
doze-kafka config print [--show-secrets] [flags]	Print the effective configuration as TOML, passwords as "<redacted>" unless --show-secrets.
doze-kafka dns-setup	One-time setup for the shared .doze names. See Endpoints.
doze-kafka help	The grouped help.

A word that is not a flag or a command is an error (did you mean --listen?), not ignored — doze-kafka listen :9092 used to start the broker on the default port.

Worked examples

Persistent broker in a fixed location:

./doze-kafka --data-dir /var/tmp/kafka/data     # meta.db is in there too

Bounded disk use — keep at most ~1 GiB per partition and nothing older than a day:

./doze-kafka --retention-bytes 1073741824 --retention-ms 86400000

Multi-partition topics by default:

./doze-kafka --default-partitions 6

Emulate an older Kafka for a client that expects it (e.g. to exercise classic consumer groups only):

./doze-kafka --kafka-version 3

Throwaway broker on a random port (handy for scripts and CI). With --listen on port 0 the console and the registry take free ports too, so two of these run at once:

./doze-kafka --listen 127.0.0.1:0 --data-dir "$(mktemp -d)"
# the summary printed at start-up has the bootstrap address, the console and the registry URL

Reachable from a container on this machine:

./doze-kafka --listen :9092 --advertised-host host.docker.internal

Locked down with SCRAM + TLS:

./doze-kafka \
  --sasl-mechanisms SCRAM-SHA-256 --sasl-user alice:s3cret \
  --tls-cert cert.pem --tls-key key.pem

What start-up prints

Once everything is bound — and only then — doze-kafka prints where it is:

  doze-kafka is up.

    kafka     kafka.project.doze:9092
    console   http://kafka.project.doze   topics, messages and consumer groups, live
    registry  http://schema.project.doze   Confluent-compatible schema registry
    data      /home/you/project/data
    serving   Kafka 4 protocol · no auth · plaintext

    eval "$(doze-kafka env)" point this shell at it
    doze-kafka doctor        when something looks wrong

The summary goes to stdout and the log lines to stderr, so doze-kafka 2>/dev/null shows just the summary. Under --listen its last lines are point this shell at it: and the whole eval "$(doze-kafka env --listen … --config '')" command, carrying this broker’s bound addresses, since a bare doze-kafka env knows only its own flags and doze-kafka.toml. The kafka line is the name, or the address where the name does not resolve on this machine. With --listen :9092 it gives the advertised address (localhost:9092) and notes that the broker is listening on every interface, since [::]:9092 is not something to paste into a client. Ctrl-C, SIGTERM and closing the terminal (SIGHUP) all stop the broker cleanly: it stops accepting, flushes the logs and exits 0. With --log-format json it is a single ready record instead, and with --log-level warn or error it is not printed. The first run on a machine without .doze set up asks once: root installs without asking; a terminal (both stdin and stderr) gets a Y/n prompt, where Enter, y or yes runs one sudo and anything else, end of input included, changes nothing; with neither, it prints the command and exits 1.

A note on --advertised-host

Kafka clients connect to a bootstrap address, ask the broker for Metadata, and then reconnect to whatever host and port the broker advertises. By default that is the instance name (or its address where the name does not resolve here), which is right for a client on the same machine. Under --listen it follows the address: 127.0.0.1:<bound-port> for a loopback listener, and localhost:<port> when listening on every interface. (It is the address and not the name localhost for a loopback listener because that name can resolve to ::1 first, where nothing is listening, and not every client goes on to try the other address: kcat stops at the first refusal.)

If a client connects from elsewhere — another container, another host, a VM — it must be able to reach the advertised address, not just the bootstrap one. Listen beyond loopback (--listen :9092), then set --advertised-host (and --advertised-port if it’s remapped) to an address that client can actually reach. This is the single most common reason a client connects but then hangs or fails on the next request. See Troubleshooting.