max.request.size Kafka in Kubernetes

Question

我正在尝试在 kubernetes 中将 max.request.size 提供给 Kafka，这似乎不起作用。怎么做？

我也尝试了KAFKA_MAX_REQUEST_SIZE再次没有积极的结果：

spec:
containers:
- name: kafka
image: wurstmeister/kafka
...
env:
- name: KAFKA_ZOOKEEPER_CONNECT
value: "zookeeper:2181"
- name: KAFKA_ADVERTISED_PORT
value: "9092"
- name: KAFKA_DELETE_TOPIC_ENABLE
value: "true"
- name: KAFKA_CREATE_TOPICS
value: "tpinput:1:1,tpoutput:1:1,operinput:1:1,operoutput:1:1,authoutput:1:1"  
- name: KAFKA_JMX_PORT
value: "7071"             
- name: KAFKA_ZOOKEEPER_TIMEOUT_MS
value: "16000"
- name: KAFKA_MESSAGE_MAX_BYTES
value: "209715200"
- name: KAFKA_FETCH_MESSAGE_MAX_BYTES
value: "209715200"  
- name: KAFKA_REPLICA_FETCH_MAX_BYTES
value: "209715200" 
- name: KAFKA_PRODUCER_MAX_REQUEST_SIZE
value: "9651322321"
- name: KAFKA_ADVERTISED_HOST_NAME
valueFrom:
fieldRef:
fieldPath: status.podIP        
- name: KUBE_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
imagePullPolicy: Always
restartPolicy: Always
envFrom:
- configMapRef:
name: env-config-for-pods

Answer 1

基于此处的 Docker 入口点脚本和该文件中的这一部分：

EXCLUSIONS="|KAFKA_VERSION|KAFKA_HOME|KAFKA_DEBUG|KAFKA_GC_LOG_OPTS|KAFKA_HEAP_OPTS|KAFKA_JMX_OPTS|KAFKA_JVM_PERFORMANCE_OPTS|KAFKA_LOG|KAFKA_OPTS|"
# Read in env as a new-line separated array. This handles the case of env variables have spaces and/or carriage returns. See #313
IFS=$'n'
for VAR in $(env)
do
env_var=$(echo "$VAR" | cut -d= -f1)
if [[ "$EXCLUSIONS" = *"|$env_var|"* ]]; then
echo "Excluding $env_var from broker config"
continue
fi
if [[ $env_var =~ ^KAFKA_ ]]; then
kafka_name=$(echo "$env_var" | cut -d_ -f2- | tr '[:upper:]' '[:lower:]' | tr _ .)
updateConfig "$kafka_name" "${!env_var}" "$KAFKA_HOME/config/server.properties"
fi
if [[ $env_var =~ ^LOG4J_ ]]; then
log4j_name=$(echo "$env_var" | tr '[:upper:]' '[:lower:]' | tr _ .)
updateConfig "$log4j_name" "${!env_var}" "$KAFKA_HOME/config/log4j.properties"
fi
done

KAFKA_MAX_REQUEST_SIZE应作为max.request.size包含在$KAFKA_HOME/config/server.properties文件中。如果 docker 映像中存在错误，我不会感到惊讶。

您可以随时 shell 进入 Kafka pod 并检查$KAFKA_HOME/config/server.properties配置文件。

kubectl exec -it <kafka-pod> -c <kafka-container> sh

Answer 2

Kafka 代理没有max.request.size配置属性。类似的将在代理级别message.max.bytes，max.message.bytes在主题级别(请注意，单词的顺序不同(。

属性max.request.size实际上是卡夫卡生产者的。