Kafka - Understanding Offset Commits

[Last Updated: May 9, 2020]

Current offsets vs Committed offsets

Current offset (position) is the offset from which next new record will be fetched (when it's available).

Committed offsets is the last committed offset for the given partition.

Committing an offset for a partition is the action of saying that the offset has been processed so that Kafka cluster won't send the committed records for the same partition. Committed offset is important in case of a consumer recovery or rebalancing (we will learn more about rebalancing in a next tutorial).

Auto Commit

For a consumer, we can enable auto commit by setting property to true. The default is true. In that case the consumer's offset will be periodically committed in the background. If this property is set to false then no offsets are committed

For a consumer, the property, specifies the frequency in milliseconds that the consumer offsets are auto-committed to Kafka if is set to true.

Also see auto commit examples here

Manually Committing offsets

By setting auto.commit.offset=false, offsets can only be committed when the application explicitly chooses to do so. This can be done by using KafkaConsumer#commitSync() and KafkaConsumer#CommitAsync()

See Also