With legacy KafkaConsumer, achieving at-least-once processing for long-running tasks often leads to a costly operational failure: if processing time exceeds max.poll.interval.ms, the consumer is flagged as dead and ejected from the consumer group. Any offsets committed for that offset by that consumer instance are rejected by the broker, leaving the partition stuck and requiring manual intervention from the ops team to reset offsets. KIP-932 eliminates this headache through KafkaShareConsumer and AcknowledgeType.RENEW. Here's how it works.

Workflow Comparison: KafkaConsumer vs KafkaShareConsumer

Legacy KafkaConsumer workflow, illustrating the flow which can lead to pitfall if process(record) takes longer time than that of max.poll.interval.ms (default 5 minutes).

Let's see how this pitfall can be avoided using KafkaShareConsumer.

The key idea is offloading work from the main thread. By delegating message processing to background worker threads, the main thread remains free to poll the broker well before max.poll.interval.ms expires. However, this decoupling only works because KafkaShareConsumer allows fine-grained, record-level acknowledgments. Using acknowledge(record, AcknowledgeType.RENEW), worker threads can extend their locks on long-running records while the main thread signals ongoing liveliness back to the broker with every poll() call.

When poll() is called, the broker re-delivers any records previously acknowledged with AcknowledgeType.RENEW so the consumer instance can maintain its active locks on them. However, because poll() returns these same records on subsequent calls until they are finalized with ACCEPT, REJECT, or RELEASE, it is must to prevent worker threads from processing them twice. Maintaining an in-memory ConcurrentHashMap tracks active record states, ensuring newly polled records that are already in flight get ignored while background threads finish execution.

Once all the ConsumerRecords are processed and finalized with ACCEPT, REJECT, or RELEASE, then poll() returns next batch of records.

The Lock Concept

share.record.lock.duration.ms (default 30 seconds) is a configuration introduced with KIP-932 defining the ownership of ConsumerRecords during that duration of time window.
Lets understand with simple example, if Instance1 and Instance2 of share-group-A are subscribed to single partitioned topic-A, polling offset 0 to 4 and offset 5-9 respectively. As per the default value, Instance1 and Instance2 have exactly 30 seconds to commit those records.

If an instance fails to finalize processing within this duration, the broker releases the record lock, making those messages available to other instances in the share group, which can cause duplicate processing. Periodically committing with ACKTYPE.RENEW extends lock ownership notifying broker and keeps long-running tasks safely in flight.

Putting It All Together in the Code

Check out ShareConsumerRenewAcknowledgeTypeExample on GitHub to test this locally with Apache Kafka 4.2.0

    private static final ConcurrentHashMap<ConsumerRecord<String, EventMessage>, Future<Boolean>> underExecutionRecords = new ConcurrentHashMap<>(5);

    private static final ExecutorService processExecutorService = Executors.newFixedThreadPool(5);

The underExecutionRecords map acts as an in-flight state registry to track active tasks and prevent duplicate processing; when SHARE_ACQUIRE_MODE_CONFIG is set to record_limit, its size is strictly bounded by max.poll.records.

Pairing this with processExecutorService capped at or below max.poll.records ensures optimal concurrency while preventing thread starvation and maximizing efficient CPU utilization across worker threads.

    Properties properties = new Properties();
    properties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    properties.put(ConsumerConfig.GROUP_ID_CONFIG, "offsetzero-acknowledge-type-example");
    properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
    properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, EventMessageDeserializer.class.getName());
    properties.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "5");
    properties.put(ConsumerConfig.SHARE_ACKNOWLEDGEMENT_MODE_CONFIG, "explicit");
    properties.put(ConsumerConfig.SHARE_ACQUIRE_MODE_CONFIG, "record_limit");

    KafkaShareConsumer<String, EventMessage> consumer = new KafkaShareConsumer<>(properties);

Setting SHARE_ACKNOWLEDGEMENT_MODE_CONFIG to explicit disables automatic record commits, giving direct control to manually issue lock renewals RENEW, acceptances ACCEPT, or rejections REJECT.

Setting SHARE_ACQUIRE_MODE_CONFIG to record_limit ensures that poll() strictly respects max.poll.records.

consumer.subscribe(List.of(TOPIC));
            while (true) {
                ConsumerRecords<String, EventMessage> records;
                try {
                    records = consumer.poll(Duration.ofMillis(500));
                    System.out.println("-------------------------POLLED------------------------------");
                    for (ConsumerRecord<String, EventMessage> record : records) {
                        if(!underExecutionRecords.containsKey(record)){
                            //assign records to different threads for concurrent processing
                            Future<Boolean> futureProcessor = processExecutorService.submit(() -> process(record, consumer));
                            System.out.println(Instant.now()+" Adding record "+record.value().getId()+" to underExecutionRecords");
                            underExecutionRecords.put(record, futureProcessor);
                        }
                    }
                    //collect acknowledgements from childrent threads
                    handleAcknowledgements(underExecutionRecords, consumer);
                } catch (RecordDeserializationException exception) {
                    rejectDeserializationFailure(consumer, exception);
                    continue;
                }
                consumer.commitSync();
                System.out.println(Instant.now()+" Committed for batch of records " + records.count());
            }

When poll() retrieves a new batch of ConsumerRecords, the main thread first checks if ConsumerRecord is present in underExcecutionRecords in-memory state from previous poll(). If a record is new, it is submitted to the ExecutorService for concurrent background processing and registered in underExecutionRecords.

Because the main thread isn't blocked by execution logic, it immediately transitions to handleAcknowledgements() to check worker thread statuses and assign appropriate acknowledgment types (ACCEPT, REJECT, or RENEW).It's important to note that KafkaShareConsumer is not thread-safe. Hence, worker threads cannot invoke acknowledge() directly upon completing a task. All acknowledgments must be routed back and managed centrally by the main polling thread / single thread.

Finally, if any record triggers a DeserializationException due to malformed payload data, it is safely marked with AcknowledgeType.REJECT before committing the batch.

private static boolean process(ConsumerRecord<String,EventMessage> record, KafkaShareConsumer<String,EventMessage> consumer) {
        Random random = new Random();
        try{
            if(random.nextBoolean()){ // Simulating long process time of 15sec for few records
                System.out.println(Instant.now()+" Long Processing started for record id "+record.value().getId()+" by thread "+Thread.currentThread().getName());
                Thread.sleep(15000);
                System.out.println(Instant.now()+" Long Processing completed for record id "+record.value().getId()+" by thread "+Thread.currentThread().getName());
            }else{ // Simulating short process time of 1sec for few records
                Thread.sleep(1000);
                System.out.println(Instant.now()+" Quickly Processed for record id "+record.value().getId()+" by thread "+Thread.currentThread().getName());
            }
            return true;
        }catch(InterruptedException ex){
            System.out.println(Instant.now()+" Error while processing record id "+ record.value().getId()+" by thread "+Thread.currentThread().getName());
            ex.printStackTrace();
            return false;
        }
    }

This method simulates realistic business logic execution times by introducing random processing delays. Long-running ConsumerRecord take approximately 15 seconds to finish, while faster records complete in under 1 second. Once execution wraps up, process() returns true on success, or false if an exception occurs during execution.

    private static void handleAcknowledgements(
            ConcurrentHashMap<ConsumerRecord<String,EventMessage>,Future<Boolean>> underExecutionRecords, KafkaShareConsumer<String, EventMessage> consumer) {
        
        underExecutionRecords.forEach((record, futureProcessor) -> {
            try {
                    System.out.println(Instant.now()+" Calling future.get() for "+record.value().getId());
                    Boolean isDone = futureProcessor.get(2, TimeUnit.SECONDS);
                    
                    if(isDone){
                        consumer.acknowledge(record, AcknowledgeType.ACCEPT);
                        System.out.println(Instant.now()+" ACCEPT ACK for "+record.value().getId());
                    }else{
                        consumer.acknowledge(record, AcknowledgeType.REJECT);
                        System.out.println(Instant.now()+" REJECT ACK for "+record.value().getId());
                    }
                        
                System.out.println(Instant.now()+" Removing record "+record.value().getId()+" from underExecutionRecords");
                underExecutionRecords.remove(record); //either ACCEPT / REJECT in both cases record won't be polled again hence remove
           
            }catch(TimeoutException e){ 
                //If TimedOut after 2 seconds, it means record is still being processed hence renew the lock 
                consumer.acknowledge(record, AcknowledgeType.RENEW);
                System.out.println(Instant.now()+" Renewed lock for record id "+record.value().getId());
            
            }catch(Exception e){
                //Release record for reattempt as interrupted by unexpected exception
                consumer.acknowledge(record, AcknowledgeType.RELEASE);
                System.out.println(Instant.now()+" Release for record id "+record.value().getId());
                e.printStackTrace();
            }
        });
        
    }

Calling future.get() without a timeout blocks indefinitely until the worker thread returns a result. If a task exceeds max.poll.interval.ms, this blocking behavior recreates the exact main-thread starvation issue we set out to solve.

To keep the consumer responsive, always call future.get(timeout, unit) with a strict time cap. When future.get() throws a TimeoutException, it signals that the background thread is still actively processing the record—allowing you to catch the exception and acknowledge that record with AcknowledgeType.RENEW. For completed tasks, inspect the returned boolean value to acknowledge with ACCEPT (on true) or REJECT (on false). In both cases, remove record from underExecutionRecords as those need not be re-processed.

It is crucial that wait cap time in future.get(capTime, TimeUnit) is strictly less than share.record.lock.duration.ms. When iterating over a underExecutionRecords for the output, the total inspection time accumulates across all items. To prevent lock expiration and unwanted redeliveries to other share group instances, calculate the individual timeout cap using the following constraint:

(capTime x max.poll.records) < share.record.lock.duration.ms

Budgeting this time ensures that the main polling thread evaluates every in-flight task, issues all required AcknowledgeType.RENEW extensions, and flushes commitSync() well before the broker's default 30-second lock window expires.

After commitSync() extends record locks for subset of ConsumerRecord from current poll() using AcknowledgeType.RENEW, subsequent poll() calls return those same in-flight records, until those are finalized. The main thread detects them in underExecutionRecords, skips re-submitting them to worker threads, and immediately calls handleAcknowledgements(). If processing completes within the capTime window, the record is finalized with ACCEPT or REJECT; otherwise, it remains in underExecutionRecords to repeat the renewal cycle on the next poll.

Key Takeaways

1. Single-Threaded Safety: KafkaShareConsumer is not thread-safe. Worker threads cannot call acknowledge() directly; the main thread must collect results and issue acknowledgments centrally.

2. All-or-Nothing Commits: You must assign an AcknowledgeType to every record in a polled batch before invoking commitSync(). Partial commits are not allowed.

3. Lock Extension Loop: Records acknowledged with AcknowledgeType.RENEW are re-delivered on subsequent poll() calls until explicitly finalized with ACCEPT, REJECT, or RELEASE.

4. Broker Lock Timeouts: Failing to commit within share.record.lock.duration.ms triggers broker-side redelivery up to group.share.delivery.count.limit times (default: 5) before the message is treated as rejected.

Last Words !

By combining KafkaShareConsumer with asynchronous worker threads and periodic AcknowledgeType.RENEW lock extensions, you overcome the traditional max.poll.interval.ms limit. This design allows you to handle long-running, unpredictable workloads reliably without sacrificing consumer group stability or risking duplicate execution.

Apache Kafka is undergoing one of its most significant architectural shifts yet with KIP-932. By introducing a new abstraction called Share Groups, KIP-932 bridges the long-standing gap between log-based event streaming and message queuing. Not exactly the Queue, but it implements the concept of cooperative consumption to accommodate queuing mechanism using KafkaTopics. Kafka can now manage record-level acknowledgments and dynamic message delivery. Here is a breakdown of how KIP-932 works, why it matters, and how it transforms Kafka’s capabilities.

Why Does KIP-932 Matter?

Operations teams frequently run into severe operational friction when using traditional Kafka consumer groups:

1. Session Timeouts & Rebalance Storms: When a few records require mroe computation time, processing can exceed max.poll.interval.ms. The group coordinator assumes the consumer is dead, triggers a rebalance, and cascades delays across the entire consumer group - all because a couple of messages took longer than expected.

2. Scaling Bottlenecks & Static Tuning: For workloads with unpredictable payload spikes, teams are forced to continuously tune max.poll.records. Because traditional consumer groups enforce a strict 1-to-1 limit between a partition and a consumer instance, you cannot scale consumer instances to process messages from a single partition concurrently.

3. Poison Pills & Head-of-Line Blocking: In environments without strict schema governance, a single malformed or un-processable message will halt an entire partition. Because offset commits move sequentially, subsequent valid records are blocked until someone manually resets the offset in the morning after upstream team reported a problem.

These are daily headaches for Kafka developers and platform engineers. KIP-932 directly solves them by introducing native point-to-point queuing semantics with record-level acknowledgments and cooperative message processing.

KIP-932 into Practice

Instantiating a KafkaShareConsumer is similar to configuring a traditional Kafka consumer, but it introduces specific properties to control message delivery and acquisition:

    Properties properties = new Properties();
    properties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    properties.put(ConsumerConfig.GROUP_ID_CONFIG, "offsetzero-share-consumer-group");
    properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
    properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
    properties.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "5");
    properties.put(ConsumerConfig.SHARE_ACKNOWLEDGEMENT_MODE_CONFIG, "explicit");
    properties.put(ConsumerConfig.SHARE_ACQUIRE_MODE_CONFIG, "record_limit");

KafkaShareConsumer<String, EventMessage> consumer = new KafkaShareConsumer<>(properties);

Key configurations to note:

1. SHARE_ACKNOWLEDGEMENT_MODE_CONFIG: Defines how record processing status is committed to the broker.

   1. explicit(Recommended): Gives you fine-grained control by requiring manual acknowledgments (ACCEPT, RELEASE, REJECT, RENEW) per record.

   2. implicit: Automatically acknowledges fetched batches on subsequent calls to poll().

2. SHARE_ACQUIRE_MODE_CONFIG: Dictates how the broker locks and delivers messages to consumer instances.

   1. record_limit: Acquires records up to the batch limit set by MAX_POLL_RECORDS_CONFIG, ensuring controlled memory footprint and predictable lock acquisition across instances.

   2. batch_optimized: Optimizes for raw throughput by fetching full message batches from the broker, regardless of the limit set in MAX_POLL_RECORDS_CONFIG.

try {
            consumer.subscribe(List.of(TOPIC));
            while (true) {
                ConsumerRecords<String, EventMessage> records;
                try {
                    records = consumer.poll(Duration.ofMillis(500));
                } catch (RecordDeserializationException exception) {
                    //Handle poison pills / improper records separately
                    rejectDeserializationFailure(consumer, exception);
                    continue;
                }

                for (ConsumerRecord<String, EventMessage> record : records) {
                    Thread.sleep(3000);
                    //Process records
                    restBackendProcess(record, consumer);
                }
                //Report all the individual acknowledgements (ACCEPT, REJECT, RELEASE, RENEW) for this batch of messages to broker
                consumer.commitSync();
                System.out.println("Committed for batch of records " + records.count());
            }
        } catch (WakeupException exception) {
            consumer.commitSync();
            System.out.println("Wakeup exception caught, consumer shutdown initiated");
        } catch (Exception exception) {
            System.err.println("Error closing consumer: " + exception.getMessage());
        } finally {
            consumer.close();
            System.out.println("Consumer closed");
        }

The consumer is subscribed to the target topic, configured to deserialize record keys as String and values as EventMessage. Notice how deserialization errors (poison pills) are intercepted and rejected immediately, preventing them from blocking subsequent records.

private static void rejectDeserializationFailure(KafkaShareConsumer<String, EventMessage> consumer,
                                                     RecordDeserializationException exception) {
        String topic = exception.topicPartition().topic();
        int partition = exception.topicPartition().partition();
        long offset = exception.offset();
        System.err.println("Rejecting deserialization failure record id =" + topic
                + ", partition=" + partition + ", offset=" + offset
                + ": " + exception.getMessage());
        consumer.acknowledge(topic, partition, offset, AcknowledgeType.REJECT);
    }

When a poison pill record fails deserialization, consumer.poll() throws a RecordDeserializationException. Because a ConsumerRecord object is never constructed, you cannot pass a record instance to consumer.acknowledge().

KIP-932 solves this by providing an overloaded acknowledge() method accepting the specific topic, partition, offset, and AcknowledgeType.

private static void restBackendProcess(ConsumerRecord<String, EventMessage> record,
                                            KafkaShareConsumer<String, EventMessage> consumer) {
        //Simulate REST API throttling messages, hence release to re-attempt processing
        if (ThreadLocalRandom.current().nextBoolean()) {
            System.out.println("REST backend throttled record id = " + record.value().getId() + "; releasing it for redelivery");
            consumer.acknowledge(record, AcknowledgeType.RELEASE);
            return;
        }

        //Successful processing
        System.out.println("REST backend processed record id = " +record.value().getId());
        consumer.acknowledge(record, AcknowledgeType.ACCEPT);
    }

The restBackendProcess method demonstrates dynamic message acknowledgment by simulating a REST API call that occasionally encounters rate limits. When a downstream service returns a throttling error, the consumer calls consumer.acknowledge(record, AcknowledgeType.RELEASE)—releasing the message back to the broker so it can be re-attempted later or processed by another available worker. Conversely, upon successful execution, it issues an AcknowledgeType.ACCEPT to finalize record processing without blocking the rest of the partition.

Try it yourself:

To experiment with KafkaShareConsumer, clone the demo repository at shubhamsworking/OffsetZero, start a local Kafka cluster, and test these scenarios:

1. SimpleShareKafkaConsumer: Run multiple instances and see how it demonstrates multi-consumer concurrency on a single-partition topic using produce-events.sh.

2. ShareConsumerAcknowledgeTypeExample: Demonstrates record-level acknowledgments on a multi-partition topic using produce-json-events.sh. Highlights how KIP-932 handles poison pills, record releases/retries, and backend throttling.

Good to know

Unlike traditional Kafka consumer groups where offset and partition management are heavily driven by the client, Share Groups rely on the Group Coordinator on the broker as the central point of management. Because of this server-centric architecture, client-side overrides don't apply to key queue behaviors.

1. Key server-side and group configurations to keep in mind:

   1. Starting offset to read (share.auto.offset.reset): Setting this in client-side code will not work. Group offsets must be managed centrally on the broker using the kafka-configs.sh CLI tool.

      kafka-configs.sh \
          --bootstrap-server localhost:9092 \
          --entity-type groups \
          --entity-name offsetzero-share-consumer-group \
          --add-config share.auto.offset.reset=earliest \
          --alter
      

   2. Record Lock Duration (share.record.lock.duration.ms): Defines how long a broker locks a record to a consumer instance after it is fetched (defaults to 30s). This lock mechanism allows multiple concurrent consumers—or multiple threads within a single consumer—to process records in parallel without duplicate delivery.

   3. Delivery Count Limit (share.delivery.record.limit): Controls redelivery tolerance (defaults to 5). If a record is continuously released or times out without an explicit ACCEPT, the record is rejected once this retry limit is reached, preventing infinite retries.

   4. Lock Extension (share.renew.acknowledge.enable): If a worker needs more time for a long-running task, issuing a RENEW acknowledgment extends the record lock. Brokers can enable or enforce this behavior using the share.renew.acknowledge.enable config.

2. Management operations for Share Groups - such as describing group status or resetting offsets are performed using the dedicated kafka-share-groups.sh CLI tool rather than the traditional kafka-consumer-groups.sh.

Final Thoughts

With good JMX monitoring metrices for share groups, Apache Kafka’s Share Groups redefine how we think about work-queue patterns in event-driven systems. By combining record-level acknowledgments, automated handling for poison pills via ARCHIVED states, and centralized broker configuration, KIP-932 gives platform engineers the best of both worlds: high-throughput stream processing and queue mechanism.