Producers

Architecture

Application threads call send(), the Sender thread drains batches to brokers.

Key Numbers

Batching: linger.ms and batch.size

A batch is sent when either it fills up to batch.size, or linger.ms passes since the first record was added.

# Latency-optimized (default)
linger.ms=0
batch.size=16384
# Effect: send immediately, batches are tiny under low load

# Throughput-optimized
linger.ms=20
batch.size=131072        # 128 KB
# Effect: wait up to 20ms for more records, batches up to 128 KB
# Adds <= 20ms latency, 5-10x throughput improvement

# Extreme throughput (bulk loaders)
linger.ms=100
batch.size=1048576       # 1 MB
compression.type=zstd
# Adds 100ms latency, often 50x throughput vs default

The interaction with compression is multiplicative: bigger batches compress much better because the LZ4/zstd dictionary has more redundancy to exploit. A 16 KB batch of similar JSON records typically compresses 2-3x; a 1 MB batch of the same records often hits 8-10x.

Compression Codecs

Compression happens once on the producer. Brokers store batches as-is (the log format includes a compression byte) and consumers decompress. This means the broker does not pay CPU for compression — but it also means the broker cannot recompress with a better codec, so producer choice persists.

Acks Levels

# acks=0  — fire and forget
acks=0
# Producer does not wait for any ack. Lowest latency, can lose data on broker crash.
# Use only for metrics/telemetry where loss is acceptable.

# acks=1  — leader ack
acks=1
# Producer waits for leader to write to its log (page cache, not necessarily disk).
# Loses data if leader crashes before followers replicate.

# acks=all — full ISR ack (recommended)
acks=all
min.insync.replicas=2     # broker-side: configured per topic
replication.factor=3
# Leader waits for all in-sync followers to ack. With min.insync.replicas=2,
# the leader rejects writes if ISR drops below 2, surfacing the durability problem.

The subtle gotcha: acks=all alone is not enough. If two of your three replicas die and the leader is the only ISR member, acks=all degrades to acks=1 silently. min.insync.replicas=2 makes that case fail fast with NotEnoughReplicasException, which the producer can retry or surface as an alert.

max.in.flight.requests.per.connection

How many ProduceRequests can be outstanding to one broker at once. Affects ordering and throughput.

# Strict ordering (legacy non-idempotent)
max.in.flight.requests.per.connection=1
# Guarantees in-order delivery even on retry, but kills pipelining.
# Throughput drops 3-5x.

# Idempotent (recommended)
enable.idempotence=true
max.in.flight.requests.per.connection=5
# Broker dedupes by sequence number, so reordering is detected and corrected.
# Maintains per-key ordering with full pipelining.

# Non-idempotent, high throughput, no ordering needed
max.in.flight.requests.per.connection=100
acks=1
# Risky: a retry of an older batch can land after a newer batch.

Retries and Timeouts

# How many times to retry on retriable errors (default Integer.MAX_VALUE)
retries=2147483647

# Total time a record can spend in the producer before failing
delivery.timeout.ms=120000  # 2 min default

# Per-request timeout
request.timeout.ms=30000

# How long to wait between retry attempts
retry.backoff.ms=100

# Effect: producer keeps retrying until delivery.timeout.ms expires.
# Common retriable errors: NotEnoughReplicas, NotLeaderForPartition, RequestTimedOut.
# Non-retriable: SerializationException, RecordTooLargeException, AuthenticationException.

Partitioner: Sticky vs Hash

# Default since 2.4: built-in sticky partitioner
# - If record has a key: hash(key) % numPartitions  (consistent per-key routing)
// - If record has no key: stick to one partition until batch is full, then switch

# Custom partitioner (rare)
public class MyPartitioner implements Partitioner {
    public int partition(String topic, Object key, byte[] keyBytes,
                         Object value, byte[] valueBytes, Cluster cluster) {
        List<PartitionInfo> parts = cluster.partitionsForTopic(topic);
        // route premium tenants to partitions 0-3, others to 4-N
        if (((String)key).startsWith("premium-")) {
            return Math.abs(key.hashCode()) % 4;
        }
        return 4 + (Math.abs(key.hashCode()) % (parts.size() - 4));
    }
}

The sticky partitioner improved tail latency dramatically when KIP-480 landed. With round-robin and no key, a producer with 100 partitions and 100 messages per second made 100 batches of 1 record each. With sticky, the same workload makes one batch of 100 records, which compresses, sends, and acks ~10x faster — and partition load remains balanced over time.

Idempotent vs Transactional

Tradeoffs and When Not to Use Defaults

Frequently Asked Questions