Kafka: How to Produce Data Reliably

The first lever for reliable producing is the acknowledgment strategy (acks). This parameter defines how many acknowledgments the producer waits for before considering a message successfully sent.

acks=0: The producer sends the message and does not wait for any acknowledgment. Maximum performance, but high risk of data loss.

acks=1: The producer waits only for the acknowledgment from the leader broker of the respective partition. This was the default until Kafka 3.4. The message is safe as long as the leader doesn’t fail exactly after acknowledgment but before replication.

acks=all (or -1): The producer waits until the leader and all In-Sync Replicas (ISRs) have received the message. This is the new default since Kafka 3.4. This provides the highest guarantee against data loss.

For maximum reliability, acks=all is therefore mandatory. But this alone is not enough. What happens when the producer doesn’t receive an acknowledgment due to a temporary network problem and resends the message? Without further precautions, you would have the message duplicated in the topic.

This is where idempotence comes into play. By setting enable.idempotence=true (default since Kafka 3.4), the producer assigns a sequence number to each message. The broker stores the last sequence number per producer and partition and automatically discards duplicates.

With this combination, you ensure that messages are guaranteed to arrive and are not duplicated.

These two settings are the foundation for "Exactly-Once" semantics on the producer side.

We’ve set acks=all, which means the producer waits for acknowledgment from all In-Sync Replicas. But what exactly does "in-sync" mean and how many replicas do we need?

A replica is considered "in-sync" if it’s not too far behind the leader. If a broker fails or becomes too slow, the controller removes it from the ISR list for the affected partitions. So acks=all doesn’t wait for all configured replicas, but only for those currently considered healthy. This is a clever mechanism, because otherwise the failure of a single broker would block the entire write operation.

Now comes the crucial question: What happens when too many brokers fail and only the leader remains? acks=all would then be identical to acks=1, and we lose our safety guarantee.

To prevent this scenario, we configure the min.insync.replicas parameter at the broker or topic level. This value determines how many replicas must be in the ISR list for a produce request to be accepted at all.

Our recommendation for topics:

With this configuration, one broker can fail without affecting production. If two brokers fail, the producer stops production for the affected partitions, since the min.insync.replicas=2 condition is no longer met. The broker throws a NotEnoughReplicasException. This is exactly what we want: Better to stop production than risk potential data loss.

Even with perfect configuration, errors can occur. The producer could crash or timeout (delivery.timeout.ms) before receiving acknowledgment from the broker. How do you handle this?

Wrong approach: Blind resending Simply resending the message in a callback is a bad idea. The internal retry logic of the Kafka client library is much more intelligent and respects the idempotence guarantee. A manual retry bypasses these guarantees and leads to duplicates or ordering problems.

Better ideas:

It’s important to understand what our configuration achieves – and what it doesn’t.

What if multiple messages need to be written atomically? If you need to write multiple messages, possibly even to different topics, as a single, indivisible unit ("all or nothing"), you need Producer Transactions. This is an advanced topic that would exceed the scope of this article.

For consume-process-produce scenarios, Kafka Streams with the setting processing.guarantee=exactly_once_v2 is often the best and simplest solution, as it manages transactions internally.

Reliability in distributed systems is not a coincidence, but the result of conscious architectural decisions and careful configuration. To ensure your producers don’t lose messages, you’ve learned to use the three most important levers:

For error cases, we’ve seen that a controlled restart through infrastructure is often the cleanest solution. Remember that these guarantees are limited to the producer. For end-to-end scenarios, advanced concepts like Kafka Streams or Producer Transactions are the next logical step.