Skip to content

Commit 2af0353

Browse files
jianxiaoguojianxiaoguo
committed
chore(kafka): kafka without zookeeper and up README
1 parent 2715230 commit 2af0353

32 files changed

Lines changed: 910 additions & 106 deletions

‎README.md‎

Lines changed: 22 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -30,39 +30,32 @@ The repository has the following service classes:
3030
```
3131
CLASS NAME | PLANS NAME | DESCRIPTION
3232
-----------|---------------|------------------------------------
33-
memcached | 128 | Memcached 128 plan which which limit resources memory size 128Mi.
34-
memcached | 250 | Memcached 250 plan which which limit resources memory size 250Mi.
35-
memcached | 500 | Memcached 1000 plan which which limit resources memory size 500Mi.
36-
memcached | 1000 | Memcached 1000 plan which which limit resources memory size 1Gi.
37-
memcached | 2500 | Memcached 2500 plan which which limit resources memory size 2.5Gi.
38-
memcached | 5000 | Memcached 5000 plan which which limit resources memory size 5Gi.
39-
memcached | 10000 | Memcached 10000 plan which which limit resources memory size 10Gi.
40-
memcached | 20000 | Memcached 20000 plan which which limit resources memory size 20Gi.
41-
memcached | 30000 | Memcached 30000 plan which which limit resources memory size 30Gi.
42-
memcached | 40000 | Memcached 40000 plan which which limit resources memory size 40Gi.
43-
memcached | 50000 | Memcached 50000 plan which which limit resources memory size 50Gi.
44-
mariadb | standard-25 | mysql standard-25 plan which limit persistence size 25Gi.
45-
mariadb | standard-50 | mysql standard-50 plan which limit persistence size 50Gi.
46-
mariadb | standard-100 | mysql standard-100 plan which limit persistence size 100Gi.
47-
mariadb | standard-200 | mysql standard-200 plan which limit persistence size 200Gi.
48-
mariadb | standard-400 | mysql standard-400 plan which limit persistence size 400Gi.
49-
mariadb | standard-800 | mysql standard-800 plan which limit persistence size 800Gi.
50-
mariadb | standard-1600| mysql standard-1600 plan which limit persistence size 1600Gi.
33+
mariadb | standard-16 | mysql standard-25 plan which limit persistence size 16Gi.
34+
mariadb | standard-64 | mysql standard-50 plan which limit persistence size 64Gi.
35+
mariadb | standard-256 | mysql standard-100 plan which limit persistence size 256Gi.
36+
mariadb | standard-512 | mysql standard-200 plan which limit persistence size 512Gi.
37+
mariadb | standard-768 | mysql standard-400 plan which limit persistence size 768Gi.
38+
mariadb | standard-1024| mysql standard-1600 plan which limit persistence size 1Ti.
5139
postgresql | standard-16 | postgresql standard-16 plan which limit persistence size 16Gi.
5240
postgresql | standard-64 | postgresql standard-64 plan which limit persistence size 64Gi.
5341
postgresql | standard-256 | postgresql standard-256 plan which limit persistence size 256Gi.
5442
postgresql | standard-512 | postgresql standard-512 plan which limit persistence size 512Gi.
5543
postgresql | standard-768 | postgresql standard-768 plan which limit persistence size 768Gi.
56-
postgresql | standard-1000| postgresql standard-1000 plan which limit persistence size 1Ti.
44+
postgresql | standard-1024| postgresql standard-1000 plan which limit persistence size 1Ti.
45+
kafka | standard-16 | kafka standard-16 plan which limit persistence size 16Gi.
46+
kafka | standard-64 | kafka standard-64 plan which limit persistence size 64Gi.
47+
kafka | standard-256 | kafka standard-256 plan which limit persistence size 256Gi.
48+
kafka | standard-512 | kafka standard-512 plan which limit persistence size 512Gi.
49+
kafka | standard-768 | kafka standard-768 plan which limit persistence size 768Gi.
50+
kafka | standard-1024| kafka standard-1000 plan which limit persistence size 1Ti.
5751
redis | 128 | Redis 128 plan which limit resources memory size 128Mi.
58-
redis | 250 | Redis 250 plan which limit resources memory size 250Mi.
59-
redis | 500 | Redis 500 plan which limit resources memory size 500Mi.
60-
redis | 1000 | Redis 1000 plan which limit resources memory size 1Gi.
61-
redis | 2500 | Redis 2500 plan which limit resources memory size 2.5Gi.
62-
redis | 5000 | Redis 5000 plan which limit resources memory size 5Gi.
63-
redis | 10000 | Redis 10000 plan which limit resources memory size 10Gi.
64-
redis | 20000 | Redis 20000 plan which limit resources memory size 20Gi.
65-
redis | 30000 | Redis 30000 plan which limit resources memory size 30Gi.
66-
redis | 40000 | Redis 40000 plan which limit resources memory size 40Gi.
67-
redis | 50000 | Redis 50000 plan which limit resources memory size 50Gi
52+
redis | 256 | Redis 250 plan which limit resources memory size 256Mi.
53+
redis | 512 | Redis 500 plan which limit resources memory size 512Mi.
54+
redis | 1024 | Redis 1000 plan which limit resources memory size 1Gi.
55+
redis | 2048 | Redis 2500 plan which limit resources memory size 2Gi.
56+
redis | 4096 | Redis 5000 plan which limit resources memory size 4Gi.
57+
redis | 8192 | Redis 10000 plan which limit resources memory size 8Gi.
58+
redis | 16384 | Redis 20000 plan which limit resources memory size 16Gi.
59+
redis | 32768 | Redis 30000 plan which limit resources memory size 32Gi.
60+
redis | 65536 | Redis 40000 plan which limit resources memory size 64Gi
6861
```

‎_dist/index.yaml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
apiVersion: v1
2+
entries:
3+
mariadb:
4+
- name: mariadb
5+
description: "MariaDB Server is one of the most popular open source relational databases. It’s made by the original developers of MySQL and guaranteed to stay open source. "
6+
version: 1.1.0
7+
postgresql:
8+
- name: postgresql
9+
description: "PostgreSQL is an object-relational database management system (ORDBMS) with an emphasis on extensibility and on standards-compliance."
10+
version: 1.1.0
11+
redis:
12+
- name: redis
13+
description: "Redis is an advanced key-value cache and store."
14+
version: 1.1.0

‎_dist/mariadb-1.1.0.tgz‎

50.9 KB
Binary file not shown.

‎_dist/postgresql-1.1.0.tgz‎

58.7 KB
Binary file not shown.

‎_dist/redis-1.1.0.tgz‎

90.3 KB
Binary file not shown.

‎addons/kafka-1.1.0/chart/kafka/templates/scripts-configmap.yaml‎

Lines changed: 15 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -14,8 +14,11 @@ data:
1414
{{- $fullname := include "common.names.fullname" . }}
1515
{{- $releaseNamespace := .Release.Namespace }}
1616
{{- $clusterDomain := .Values.clusterDomain }}
17+
{{- $interBrokerListenerName := .Values.interBrokerListenerName }}
1718
{{- $interBrokerPort := .Values.service.ports.internal }}
1819
{{- $clientPort := .Values.service.ports.client }}
20+
{{- $controllerPort := .Values.service.ports.controller }}
21+
{{- $externalPort := .Values.service.ports.external }}
1922
{{- $jksTruststoreSecret := .Values.auth.tls.jksTruststoreSecret -}}
2023
{{- $jksTruststore := .Values.auth.tls.jksTruststore -}}
2124
{{- $jksKeystoreSAN := .Values.auth.tls.jksKeystoreSAN -}}
@@ -76,11 +79,17 @@ data:
7679
#!/bin/bash
7780
7881
ID="${MY_POD_NAME#"{{ $fullname }}-"}"
79-
if [[ -f "/drycc/kafka/data/meta.properties" ]]; then
80-
export KAFKA_CFG_BROKER_ID="$(grep "broker.id" /drycc/kafka/data/meta.properties | awk -F '=' '{print $2}')"
81-
else
82-
export KAFKA_CFG_BROKER_ID="$((ID + {{ .Values.minBrokerId }}))"
83-
fi
82+
export KAFKA_CFG_BROKER_ID="$((ID + {{ .Values.minBrokerId }}))"
83+
CONTROLLER_QUORUM_VOTERS=""
84+
for i in $( seq 0 $REPLICAS); do
85+
if [[ $i != $REPLICAS ]]; then
86+
BROKER_ID="$((i + {{ .Values.minBrokerId }}))"
87+
CONTROLLER_QUORUM_VOTERS="$CONTROLLER_QUORUM_VOTERS$BROKER_ID@"{{ $fullname }}-$i".{{ $fullname }}-headless.{{ $releaseNamespace }}.svc.{{ $clusterDomain }}:{{ $controllerPort }},"
88+
else
89+
CONTROLLER_QUORUM_VOTERS=${CONTROLLER_QUORUM_VOTERS::-1}
90+
fi
91+
done
92+
export KAFKA_CFG_CONTROLLER_QUORUM_VOTERS="$CONTROLLER_QUORUM_VOTERS"
8493
8594
{{- if .Values.externalAccess.enabled }}
8695
# Configure external ip and port
@@ -112,7 +121,7 @@ data:
112121
{{- if .Values.advertisedListeners }}
113122
export KAFKA_CFG_ADVERTISED_LISTENERS={{ join "," .Values.advertisedListeners }}
114123
{{- else }}
115-
export KAFKA_CFG_ADVERTISED_LISTENERS="INTERNAL://${MY_POD_NAME}.{{ $fullname }}-headless.{{ $releaseNamespace }}.svc.{{ $clusterDomain }}:{{ $interBrokerPort }},CLIENT://${MY_POD_NAME}.{{ $fullname }}-headless.{{ $releaseNamespace }}.svc.{{ $clusterDomain }}:{{ $clientPort }},EXTERNAL://${EXTERNAL_ACCESS_HOST}:${EXTERNAL_ACCESS_PORT}"
124+
export KAFKA_CFG_ADVERTISED_LISTENERS="{{ $interBrokerListenerName }}://${MY_POD_NAME}.{{ $fullname }}-headless.{{ $releaseNamespace }}.svc.{{ $clusterDomain }}:{{ $interBrokerPort }},CLIENT://${MY_POD_NAME}.{{ $fullname }}-headless.{{ $releaseNamespace }}.svc.{{ $clusterDomain }}:{{ $clientPort }},EXTERNAL://${EXTERNAL_ACCESS_HOST}:${EXTERNAL_ACCESS_PORT}"
116125
{{- end }}
117126
{{- end }}
118127

‎addons/kafka-1.1.0/chart/kafka/templates/statefulset.yaml‎

Lines changed: 40 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,12 @@
33
{{- $releaseNamespace := .Release.Namespace }}
44
{{- $clusterDomain := .Values.clusterDomain }}
55
{{- $interBrokerPort := .Values.service.ports.internal }}
6+
{{- $controllerPort := .Values.service.ports.controller }}
67
{{- $clientPort := .Values.service.ports.client }}
8+
{{- $externalPort := .Values.service.ports.external }}
79
{{- $interBrokerProtocol := include "kafka.listenerType" (dict "protocol" .Values.auth.interBrokerProtocol) -}}
810
{{- $clientProtocol := include "kafka.listenerType" (dict "protocol" .Values.auth.clientProtocol) -}}
11+
{{- $controllerProtocol := include "kafka.listenerType" (dict "protocol" .Values.auth.controllerProtocol) -}}
912
{{- $externalClientProtocol := include "kafka.listenerType" (dict "protocol" (include "kafka.externalClientProtocol" . )) -}}
1013
apiVersion: {{ include "common.capabilities.statefulset.apiVersion" . }}
1114
kind: StatefulSet
@@ -178,21 +181,43 @@ spec:
178181
valueFrom:
179182
fieldRef:
180183
fieldPath: metadata.name
184+
{{- if .Values.kraft.enabled }}
185+
- name: KAFKA_ENABLE_KRAFT
186+
value: "true"
187+
- name: KAFKA_KRAFT_CLUSTER_ID
188+
{{- if .Values.kraft.cluster_id }}
189+
value: {{ .Values.kraft.cluster_id }}
190+
{{- else}}
191+
value: {{ (randAlphaNum 22) | quote }}
192+
{{- end }}
193+
- name: KAFKA_CFG_PROCESS_ROLES
194+
value: "broker,controller"
195+
- name: KAFKA_CFG_CONTROLLER_LISTENER_NAMES
196+
value: "CONTROLLER"
197+
- name: REPLICAS
198+
value: {{ .Values.replicaCount | quote }}
199+
{{- end}}
200+
{{- if .Values.zookeeper.enabled }}
201+
- name: KAFKA_ZOOKEEPER_PROTOCOL
202+
value: {{ include "kafka.zookeeper.protocol" . }}
203+
{{- end }}
204+
{{- if and (.Values.zookeeper.enabled) (.Values.externalZookeeper.servers) }}
181205
- name: KAFKA_CFG_ZOOKEEPER_CONNECT
182206
{{- if .Values.zookeeper.enabled }}
183207
value: {{ printf "%s%s" (include "kafka.zookeeper.fullname" .) (tpl .Values.zookeeperChrootPath .) | quote }}
184208
{{- else }}
185209
value: {{ include "common.tplvalues.render" (dict "value" (printf "%s%s" (join "," .Values.externalZookeeper.servers) (tpl .Values.zookeeperChrootPath .)) "context" $) }}
186210
{{- end }}
211+
{{- end }}
187212
- name: KAFKA_INTER_BROKER_LISTENER_NAME
188213
value: {{ .Values.interBrokerListenerName | quote }}
189214
- name: KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP
190215
{{- if .Values.listenerSecurityProtocolMap }}
191216
value: {{ .Values.listenerSecurityProtocolMap | quote }}
192217
{{- else if .Values.externalAccess.enabled }}
193-
value: "INTERNAL:{{ $interBrokerProtocol }},CLIENT:{{ $clientProtocol }},EXTERNAL:{{ $externalClientProtocol }}"
218+
value: "INTERNAL:{{ $interBrokerProtocol }},CLIENT:{{ $clientProtocol }},CONTROLLER:{{ $controllerProtocol }},EXTERNAL:{{ $externalClientProtocol }}"
194219
{{- else }}
195-
value: "INTERNAL:{{ $interBrokerProtocol }},CLIENT:{{ $clientProtocol }}"
220+
value: "INTERNAL:{{ $interBrokerProtocol }},CLIENT:{{ $clientProtocol }},CONTROLLER:{{ $controllerProtocol }}"
196221
{{- end }}
197222
{{- if or ($clientProtocol | regexFind "SASL") ($externalClientProtocol | regexFind "SASL") ($interBrokerProtocol | regexFind "SASL") .Values.auth.sasl.jaas.zookeeperUser }}
198223
- name: KAFKA_CFG_SASL_ENABLED_MECHANISMS
@@ -204,9 +229,9 @@ spec:
204229
{{- if .Values.listeners }}
205230
value: {{ join "," .Values.listeners }}
206231
{{- else if .Values.externalAccess.enabled }}
207-
value: "INTERNAL://:{{ $interBrokerPort }},CLIENT://:9092,EXTERNAL://:9094"
232+
value: "INTERNAL://0.0.0.0:{{ $interBrokerPort }},CONTROLLER://0.0.0.0:{{ $controllerPort }},CLIENT://0.0.0.0:{{ $clientPort }},EXTERNAL://0.0.0.0:{{ $externalPort }}"
208233
{{- else }}
209-
value: "INTERNAL://:{{ $interBrokerPort }},CLIENT://:9092"
234+
value: "INTERNAL://0.0.0.0:{{ $interBrokerPort }},CLIENT://0.0.0.0:{{ $clientProtocol }}"
210235
{{- end }}
211236
{{- if .Values.externalAccess.enabled }}
212237
{{- if .Values.externalAccess.autoDiscovery.enabled }}
@@ -260,8 +285,6 @@ spec:
260285
name: {{ include "kafka.jaasSecretName" . }}
261286
key: zookeeper-password
262287
{{- end }}
263-
- name: KAFKA_ZOOKEEPER_PROTOCOL
264-
value: {{ include "kafka.zookeeper.protocol" . }}
265288
{{- if .Values.auth.zookeeper.tls.enabled }}
266289
- name: KAFKA_ZOOKEEPER_TLS_TYPE
267290
value: {{ upper .Values.auth.zookeeper.tls.type | quote }}
@@ -349,8 +372,10 @@ spec:
349372
value: {{ .Values.socketRequestMaxBytes | replace "_" "" | quote }}
350373
- name: KAFKA_CFG_SOCKET_SEND_BUFFER_BYTES
351374
value: {{ .Values.socketSendBufferBytes | quote }}
375+
{{- if .Values.zookeeper.enabled }}
352376
- name: KAFKA_CFG_ZOOKEEPER_CONNECTION_TIMEOUT_MS
353377
value: {{ .Values.zookeeperConnectionTimeoutMs | quote }}
378+
{{- end }}
354379
- name: KAFKA_CFG_AUTHORIZER_CLASS_NAME
355380
value: {{ .Values.authorizerClassName | quote }}
356381
- name: KAFKA_CFG_ALLOW_EVERYONE_IF_NO_ACL_FOUND
@@ -372,11 +397,19 @@ spec:
372397
{{- end }}
373398
{{- end }}
374399
ports:
400+
{{- if .Values.containerPorts.client }}
375401
- name: kafka-client
376402
containerPort: {{ .Values.containerPorts.client }}
403+
{{- end }}
404+
{{- if .Values.containerPorts.internal }}
377405
- name: kafka-internal
378406
containerPort: {{ .Values.containerPorts.internal }}
379-
{{- if .Values.externalAccess.enabled }}
407+
{{- end }}
408+
{{- if and .Values.kraft.enabled .Values.containerPorts.controller }}
409+
- name: kafka-control
410+
containerPort: {{ .Values.containerPorts.controller }}
411+
{{- end }}
412+
{{- if and .Values.externalAccess.enabled .Values.containerPorts.external }}
380413
- name: kafka-external
381414
containerPort: {{ .Values.containerPorts.external }}
382415
{{- end }}

‎addons/kafka-1.1.0/chart/kafka/templates/svc-headless.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,5 +32,9 @@ spec:
3232
port: {{ .Values.service.ports.internal }}
3333
protocol: TCP
3434
targetPort: kafka-internal
35+
- name: tcp-controller
36+
port: {{ .Values.service.ports.controller }}
37+
protocol: TCP
38+
targetPort: kafka-control
3539
selector: {{- include "common.labels.matchLabels" . | nindent 4 }}
3640
app.kubernetes.io/component: kafka

‎addons/kafka-1.1.0/chart/kafka/values.yaml‎

Lines changed: 21 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -229,6 +229,7 @@ auth:
229229
## @param auth.interBrokerProtocol Authentication protocol for inter-broker communications. Allowed protocols: `plaintext`, `tls`, `mtls`, `sasl` and `sasl_tls`
230230
##
231231
clientProtocol: plaintext
232+
controllerProtocol: plaintext
232233
# Note: empty by default for backwards compatibility reasons, find more information at
233234
# https://github.com/bitnami/charts/pull/8902/
234235
externalClientProtocol: ""
@@ -422,7 +423,7 @@ extraEnvVarsSecret: ""
422423

423424
## @param replicaCount Number of Kafka nodes
424425
##
425-
replicaCount: 1
426+
replicaCount: 3
426427
## @param minBrokerId Minimal broker.id value, nodes increment their `broker.id` respectively
427428
## Brokers increment their ID starting at this minimal value.
428429
## E.g., with `minBrokerId=100` and 3 nodes, IDs will be 100, 101, 102 for brokers 0, 1, and 2, respectively.
@@ -434,8 +435,9 @@ minBrokerId: 0
434435
##
435436
containerPorts:
436437
client: 9092
437-
internal: 9093
438-
external: 9094
438+
controller: 9093
439+
internal: 9094
440+
external: 9095
439441
## Configure extra options for Kafka containers' liveness, readiness and startup probes
440442
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes
441443
## @param livenessProbe.enabled Enable livenessProbe on Kafka containers
@@ -447,7 +449,7 @@ containerPorts:
447449
##
448450
livenessProbe:
449451
enabled: true
450-
initialDelaySeconds: 60
452+
initialDelaySeconds: 10
451453
timeoutSeconds: 5
452454
failureThreshold: 3
453455
periodSeconds: 10
@@ -461,7 +463,7 @@ livenessProbe:
461463
##
462464
readinessProbe:
463465
enabled: true
464-
initialDelaySeconds: 30
466+
initialDelaySeconds: 5
465467
failureThreshold: 6
466468
timeoutSeconds: 5
467469
periodSeconds: 10
@@ -671,8 +673,9 @@ service:
671673
##
672674
ports:
673675
client: 9092
674-
internal: 9093
675-
external: 9094
676+
controller: 9093
677+
internal: 9094
678+
external: 9095
676679
## @param service.nodePorts.client Node port for the Kafka client connections
677680
## @param service.nodePorts.external Node port for the Kafka external connections
678681
## NOTE: choose port between <30000-32767>
@@ -782,7 +785,7 @@ externalAccess:
782785
## @param externalAccess.service.ports.external Kafka port used for external access when service type is LoadBalancer
783786
##
784787
ports:
785-
external: 9094
788+
external: 9095
786789
## @param externalAccess.service.loadBalancerIPs Array of load balancer IPs for each Kafka broker. Length must be the same as replicaCount
787790
## e.g:
788791
## loadBalancerIPs:
@@ -1629,6 +1632,14 @@ provisioning:
16291632
##
16301633
waitForKafka: true
16311634

1635+
kraft:
1636+
## @param kraft.enabled for kafka without Zookeeper
1637+
##
1638+
enabled: true
1639+
## @param kraft.cluster_id for Kafka Raft metadata mode.
1640+
##
1641+
cluster_id: ""
1642+
16321643
## @section ZooKeeper chart parameters
16331644

16341645
## ZooKeeper chart configuration
@@ -1637,7 +1648,7 @@ provisioning:
16371648
zookeeper:
16381649
## @param zookeeper.enabled Switch to enable or disable the ZooKeeper helm chart
16391650
##
1640-
enabled: true
1651+
enabled: false
16411652
## @param zookeeper.replicaCount Number of ZooKeeper nodes
16421653
##
16431654
replicaCount: 1
@@ -1674,7 +1685,7 @@ zookeeper:
16741685
size: 8Gi
16751686

16761687
## External Zookeeper Configuration
1677-
## All of these values are only used if `zookeeper.enabled=false`
1688+
## All of these values are only used if `zookeeper.enabled=false` and `kraft.enabled=false`
16781689
##
16791690
externalZookeeper:
16801691
## @param externalZookeeper.servers List of external zookeeper servers to use. Typically used in combination with 'zookeeperChrootPath'.

0 commit comments

Comments
 (0)