Kafka源码之KafkaConsumer分析之offset操作

当消费者正常消费过程中以及Rebalance操作开始之前，都会提交一次Offset记录Consumer当前的消费位置。在SubscriptionState中使用TopicPartitionState记录了每个TopicPartition的消费状况，TopicPartitionState.position字段则记录了消费者下次要从服务端获取的消息的offset。

public void commitOffsetsAsync(final Map<TopicPartition, OffsetAndMetadata> offsets, OffsetCommitCallback callback) {
		//将needsFetchCommittedOffsets设置为true
        this.subscriptions.needRefreshCommits();
        //创建OffsetCommitRequest并添加到unsent队列中
        RequestFuture<Void> future = sendOffsetCommitRequest(offsets);
        //选择回调函数
        final OffsetCommitCallback cb = callback == null ? defaultOffsetCommitCallback : callback;
        //添加监听器
        future.addListener(new RequestFutureListener<Void>() {
            @Override
            public void onSuccess(Void value) {
                if (interceptors != null)
                    interceptors.onCommit(offsets);
                cb.onComplete(offsets, null);
            }

            @Override
            public void onFailure(RuntimeException e) {
                if (e instanceof RetriableException) {
                    cb.onComplete(offsets, new RetriableCommitFailedException("Commit offsets failed with retriable exception. You should retry committing offsets.", e));
                } else {
                    cb.onComplete(offsets, e);
                }
            }
        });

        //以非中断的方式发出去
        client.pollNoWakeup();
    }

这个是以异步的方式提交offsetcommitRequest，而同步的方式和它由两点不同，一是同步的方式通过阻塞poll方法获取响应，二是如果检测到RetriableException时会进行重试。
AutoCommitTask是一个定时任务，它周期性地调用commitOffsetsAsync方法，实现自动提交offset功能。
接下来我们看一下对OffsetCommitResponse的处理流程:

public void handle(OffsetCommitResponse commitResponse, RequestFuture<Void> future) {
            sensors.commitLatency.record(response.requestLatencyMs());
            Set<String> unauthorizedTopics = new HashSet<>();
			//遍历待提交的所有offset信息
            for (Map.Entry<TopicPartition, Short> entry : commitResponse.responseData().entrySet()) {
                TopicPartition tp = entry.getKey();
                OffsetAndMetadata offsetAndMetadata = this.offsets.get(tp);
                long offset = offsetAndMetadata.offset();
				//获取错误码
                Errors error = Errors.forCode(entry.getValue());
                if (error == Errors.NONE) {
                    log.debug("Group {} committed offset {} for partition {}", groupId, offset, tp);
                    if (subscriptions.isAssigned(tp))
                        // update the local cache only if the partition is still assigned
                        subscriptions.committed(tp, offsetAndMetadata);
                } 
                //错误码的处理
               ...

            if (!unauthorizedTopics.isEmpty()) {
                log.error("Not authorized to commit to topics {} for group {}", unauthorizedTopics, groupId);
                future.raise(new TopicAuthorizationException(unauthorizedTopics));
            } else {
                future.complete(null);
            }
        }

在Rebalance操作结束之后，每个消费者都缺定了其需要消费的分区。在开始消费之前，消费者需要确定拉去消息的其实位置。假设之前已经将最后的消费位置提交到了GroupCoordinator，GroupCoordinator将其保存到了Kafka内部的Offsets Topic中，此时消费者可以通过OffsetFetchRequest请求获取上次提交offset并从次开始消费

public void refreshCommittedOffsetsIfNeeded() {
		//检查needsFetchCommittedOffsets
        if (subscriptions.refreshCommitsNeeded()) {
        	//发送OffsetFetchRequest并处理OffsetFetchResponse响应，返回值时最近提交的offset集合
            Map<TopicPartition, OffsetAndMetadata> offsets = fetchCommittedOffsets(subscriptions.assignedPartitions());
            //遍历每个分区，更新committed字段
            for (Map.Entry<TopicPartition, OffsetAndMetadata> entry : offsets.entrySet()) {
                TopicPartition tp = entry.getKey();
                // verify assignment is still active
                if (subscriptions.isAssigned(tp))
                    this.subscriptions.committed(tp, entry.getValue());
            }
            //将needsFetchCommittedOffsets设置为false
            this.subscriptions.commitsRefreshed();
        }
    }

这个方法的主要功能时发送OffsetFetchRequest请求从服务端拉去最近提交的offset集合，并更新到Subscriptions集合中。

public Map<TopicPartition, OffsetAndMetadata> fetchCommittedOffsets(Set<TopicPartition> partitions) {
        while (true) {
        	//确保和GroupCoordinator处于连接状态
            ensureCoordinatorReady();

            // 创建并缓存请求
            RequestFuture<Map<TopicPartition, OffsetAndMetadata>> future = sendOffsetFetchRequest(partitions);
            //阻塞等待
            client.poll(future);
			//如果成功返回结果
            if (future.succeeded())
                return future.value();

            if (!future.isRetriable())
                throw future.exception();
			//如果时Retriable异常，等待一段时间后重试
            time.sleep(retryBackoffMs);
        }
    }

下面看一下OffsetFetchResponse的处理方法：

public void handle(OffsetFetchResponse response, RequestFuture<Map<TopicPartition, OffsetAndMetadata>> future) {
            Map<TopicPartition, OffsetAndMetadata> offsets = new HashMap<>(response.responseData().size());
            //遍历从服务端获取到的offset集合
            for (Map.Entry<TopicPartition, OffsetFetchResponse.PartitionData> entry : response.responseData().entrySet()) {
                TopicPartition tp = entry.getKey();
                OffsetFetchResponse.PartitionData data = entry.getValue();
                if (data.hasError()) {
                    Errors error = Errors.forCode(data.errorCode);
                    log.debug("Group {} failed to fetch offset for partition {}: {}", groupId, tp, error.message());
                    return;
                } else if (data.offset >= 0) {
                    // 记录正常的数据
                    offsets.put(tp, new OffsetAndMetadata(data.offset, data.metadata));
                } else {
                    log.debug("Group {} has no committed offset for partition {}", groupId, tp);
                }
            }
			//将保存的数据传播下去
            future.complete(offsets);
        }