Apache Kafka 4.4.0 adds share group dead-letter queues and transactional state stores

Apache Kafka 4.4.0 is out, bringing dead-letter queues for share groups, optional transactional state stores in Kafka Streams and a set of deprecations. According to the release announcement, the version contains 25 KIPs and 963 commits since 4.3.0.
- Apache Kafka 4.4.0 contains 25 KIPs and 963 commits since 4.3.0, according to the release announcement.
- Share groups gain dead-letter queues (KIP-1191) once share.version is upgraded to 2.
- Kafka Streams adds optional transactional state stores (KIP-892), disabled by default.
- broker.id and kafka-broker-api-versions.sh are among the deprecated items.
- Storage directories formatted by kafka-storage are no longer forward-compatible (KIP-1170).
Broker, client and tooling changes
The headline item for consumers is KIP-1191: share groups now support dead-letter queues. Records that are released beyond the max delivery count, or rejected by the share consumer, become eligible for the DLQ once the share.version feature is upgraded to 2. Behavior is configured dynamically through a set of errors.deadletterqueue settings, including the topic name, auto-creation of topics, a group-level switch and a copy-record option.
KIP-909 adds the client setting bootstrap.resolve.timeout.ms, which controls how DNS resolution of bootstrap.servers is done. The default of 0 keeps the historical synchronous behavior. A positive value opts into asynchronous resolution, where the client constructor returns immediately and failures appear later as BootstrapResolutionException. The announcement describes this as an evolving feature that may change in a minor release, and says Kafka Streams and Kafka Connect do not yet support it.
Other changes in this area include:
- KIP-1312: controllers can be unregistered from cluster metadata, using a new kafka-cluster.sh unregister-controller command or an --unregister flag on kafka-metadata-quorum.sh remove-controller. It requires metadata version 4.4-IV2.
- KIP-1241: new remote.copy.lag.bytes and remote.copy.lag.ms configs allow tiered storage redundancy to be reduced with delayed upload.
- KIP-1242: clients now include cluster and node information when connecting, so misrouted connections can be detected and handled.
- KIP-1291: the Linux I/O metrics collector exposes all 7 fields from /proc/self/io via JMX. According to the announcement, operators can use them to calculate page-cache hit ratios, detect write amplification and monitor log-compaction I/O activity.
- KIP-1299: kafka-producer-perf-test.sh gains --record-key-range, --key-distribution and --random-seed options for reproducible benchmark runs.
- KIP-1323: kafka-share-groups.sh can initialize share group offsets from a file, allowing a share group to start from the committed offsets of a consumer group.
Kafka Streams updates
The most significant Streams change is KIP-892, which adds transactional state stores. They are enabled with enable.transactional.statestores, which defaults to false. When on, state stores are no longer wiped and rebuilt on error under exactly-once processing. Only writes since the last successful commit are lost and replayed. The configs statestore.uncommitted.max.bytes and default.interactive.query.isolation.level control the uncommitted buffer size and the isolation level of interactive queries.
KIP-1331 lets brokers record a human-readable description of a Streams group's processing topology through a pluggable backend. When group.streams.topology.description.plugin.class is set, clients push a description equivalent to Topology#describe() to the group coordinator. It can be retrieved with Admin#describeStreamsGroups or kafka-streams-groups.sh --describe --topology, without access to the application's source code. KIP-1357 adds broker-side custom task assignors, registered with group.streams.assignors and selected per group with streams.assignor.name.
Further additions:
- KIP-1340: a new ValueJoinerWithStreamAndMappedKey interface and KStream join overloads for KStream-GlobalKTable joins expose both the mapped key and the original stream key. The older ValueJoinerWithKey-based overloads are deprecated but unchanged in behavior.
- KIP-1356: four new IQv2 query types return record headers from headers-aware state stores.
- KIP-1321: StreamPartitioner gets a new method that receives record headers, and the version without headers is deprecated.
- KIP-1245: application.server is validated when StreamsConfig is created and must be empty or a valid host:port endpoint, otherwise a ConfigException is thrown.
Deprecations and upgrade notes
Several items are deprecated. KIP-1232 deprecates the broker.id broker configuration. KIP-1200 deprecates ClientQuotaCallback#updateClusterMetadata, to be removed in Kafka 5.0, and custom implementations no longer need to override it because a default no-op is provided. KIP-1301 deprecates the remaining Yammer-based classic-group metrics in GroupCoordinatorMetrics, replacing them with equivalent Kafka Metrics gauges. KIP-1220 adds an api-versions command to kafka-cluster.sh, so kafka-broker-api-versions.sh is deprecated and will be removed in the next major release.
One change affects operations directly. Under KIP-1170, storage directories formatted by the kafka-storage tool are no longer forward-compatible. A broker must now be the same version as, or newer than, the tool that formatted its directory, regardless of the --release-version chosen at format time.
Also, KIP-1319 introduces TxnOffsetCommit v6, aligning it with OffsetCommit by carrying topic IDs instead of topic names and returning more granular error codes. The announcement points readers to the Upgrading to 4.4 section of the documentation for notable changes and detailed upgrade steps, and to the release notes for the full list.
What it means for Kafka users
Teams running share groups gain a standard route for handling records that cannot be processed. Streams users who need stronger recovery behavior under exactly-once processing can opt into transactional state stores, which are off by default. Operators should review the deprecations and the kafka-storage compatibility rule before upgrading, using the upgrade section of the documentation.
Confluent: Introducing Apache Kafka 4.4.0


