Skip to content

Commit 1e4973d

Browse files
EamonZhangEamonZhang
committed
2 parents 6e6c354 + aec7a56 commit 1e4973d

70 files changed

Lines changed: 1723 additions & 549 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎addons/airflow/2/chart/airflow/README.md‎

Lines changed: 2 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -125,7 +125,7 @@ The command removes all the Kubernetes components associated with the chart and
125125
| `web.containerPorts.http` | Airflow web HTTP container port | `8080` |
126126
| `web.replicaCount` | Number of Airflow web replicas | `1` |
127127
| `web.livenessProbe.enabled` | Enable livenessProbe on Airflow web containers | `true` |
128-
| `web.livenessProbe.initialDelaySeconds` | Initial delay seconds for livenessProbe | `180` |
128+
| `web.livenessProbe.initialDelaySeconds` | Initial delay seconds for livenessProbe | `360` |
129129
| `web.livenessProbe.periodSeconds` | Period seconds for livenessProbe | `20` |
130130
| `web.livenessProbe.timeoutSeconds` | Timeout seconds for livenessProbe | `5` |
131131
| `web.livenessProbe.failureThreshold` | Failure threshold for livenessProbe | `6` |
@@ -258,7 +258,7 @@ The command removes all the Kubernetes components associated with the chart and
258258
| `worker.containerPorts.http` | Airflow worker HTTP container port | `8793` |
259259
| `worker.replicaCount` | Number of Airflow worker replicas | `1` |
260260
| `worker.livenessProbe.enabled` | Enable livenessProbe on Airflow worker containers | `true` |
261-
| `worker.livenessProbe.initialDelaySeconds` | Initial delay seconds for livenessProbe | `180` |
261+
| `worker.livenessProbe.initialDelaySeconds` | Initial delay seconds for livenessProbe | `360` |
262262
| `worker.livenessProbe.periodSeconds` | Period seconds for livenessProbe | `20` |
263263
| `worker.livenessProbe.timeoutSeconds` | Timeout seconds for livenessProbe | `5` |
264264
| `worker.livenessProbe.failureThreshold` | Failure threshold for livenessProbe | `6` |
@@ -412,59 +412,6 @@ The command removes all the Kubernetes components associated with the chart and
412412
| `rbac.create` | Create Role and RoleBinding | `false` |
413413
| `rbac.rules` | Custom RBAC rules to set | `[]` |
414414

415-
### Airflow metrics parameters
416-
417-
| Name | Description | Value |
418-
| ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | ---------------------------------- |
419-
| `metrics.enabled` | Whether or not to create a standalone Airflow exporter to expose Airflow metrics | `false` |
420-
| `metrics.image.registry` | Airflow exporter image registry | `REGISTRY_NAME` |
421-
| `metrics.image.repository` | Airflow exporter image repository | `REPOSITORY_NAME/airflow-exporter` |
422-
| `metrics.image.digest` | Airflow exporter image digest in the way sha256:aa.... Please note this parameter, if set, will override the tag | `""` |
423-
| `metrics.image.pullPolicy` | Airflow exporter image pull policy | `IfNotPresent` |
424-
| `metrics.image.pullSecrets` | Airflow exporter image pull secrets | `[]` |
425-
| `metrics.extraEnvVars` | Array with extra environment variables to add Airflow exporter pods | `[]` |
426-
| `metrics.extraEnvVarsCM` | ConfigMap containing extra environment variables for Airflow exporter pods | `""` |
427-
| `metrics.extraEnvVarsSecret` | Secret containing extra environment variables (in case of sensitive data) for Airflow exporter pods | `""` |
428-
| `metrics.containerPorts.http` | Airflow exporter metrics container port | `9112` |
429-
| `metrics.resources.limits` | The resources limits for the container | `{}` |
430-
| `metrics.resources.requests` | The requested resources for the container | `{}` |
431-
| `metrics.podSecurityContext.enabled` | Enable security context for the pods | `true` |
432-
| `metrics.podSecurityContext.fsGroup` | Set Airflow exporter pod's Security Context fsGroup | `1001` |
433-
| `metrics.containerSecurityContext.enabled` | Enable Airflow exporter containers' Security Context | `true` |
434-
| `metrics.containerSecurityContext.runAsUser` | Set Airflow exporter containers' Security Context runAsUser | `1001` |
435-
| `metrics.containerSecurityContext.runAsNonRoot` | Set Airflow exporter containers' Security Context runAsNonRoot | `true` |
436-
| `metrics.containerSecurityContext.privileged` | Set metrics container's Security Context privileged | `false` |
437-
| `metrics.containerSecurityContext.allowPrivilegeEscalation` | Set metrics container's Security Context allowPrivilegeEscalation | `false` |
438-
| `metrics.containerSecurityContext.capabilities.drop` | List of capabilities to be dropped | `["ALL"]` |
439-
| `metrics.containerSecurityContext.seccompProfile.type` | Set container's Security Context seccomp profile | `RuntimeDefault` |
440-
| `metrics.lifecycleHooks` | for the Airflow exporter container(s) to automate configuration before or after startup | `{}` |
441-
| `metrics.hostAliases` | Airflow exporter pods host aliases | `[]` |
442-
| `metrics.podLabels` | Extra labels for Airflow exporter pods | `{}` |
443-
| `metrics.podAnnotations` | Extra annotations for Airflow exporter pods | `{}` |
444-
| `metrics.podAffinityPreset` | Pod affinity preset. Ignored if `metrics.affinity` is set. Allowed values: `soft` or `hard` | `""` |
445-
| `metrics.podAntiAffinityPreset` | Pod anti-affinity preset. Ignored if `metrics.affinity` is set. Allowed values: `soft` or `hard` | `soft` |
446-
| `metrics.nodeAffinityPreset.type` | Node affinity preset type. Ignored if `metrics.affinity` is set. Allowed values: `soft` or `hard` | `""` |
447-
| `metrics.nodeAffinityPreset.key` | Node label key to match Ignored if `metrics.affinity` is set. | `""` |
448-
| `metrics.nodeAffinityPreset.values` | Node label values to match. Ignored if `metrics.affinity` is set. | `[]` |
449-
| `metrics.affinity` | Affinity for pod assignment | `{}` |
450-
| `metrics.nodeSelector` | Node labels for pod assignment | `{}` |
451-
| `metrics.tolerations` | Tolerations for pod assignment | `[]` |
452-
| `metrics.schedulerName` | Name of the k8s scheduler (other than default) for Airflow exporter | `""` |
453-
| `metrics.service.ports.http` | Airflow exporter metrics service port | `9112` |
454-
| `metrics.service.clusterIP` | Static clusterIP or None for headless services | `""` |
455-
| `metrics.service.sessionAffinity` | Control where client requests go, to the same pod or round-robin | `None` |
456-
| `metrics.service.annotations` | Annotations for the Airflow exporter service | `{}` |
457-
| `metrics.serviceMonitor.enabled` | if `true`, creates a Prometheus Operator ServiceMonitor (requires `metrics.enabled` to be `true`) | `false` |
458-
| `metrics.serviceMonitor.namespace` | Namespace in which Prometheus is running | `""` |
459-
| `metrics.serviceMonitor.interval` | Interval at which metrics should be scraped | `""` |
460-
| `metrics.serviceMonitor.scrapeTimeout` | Timeout after which the scrape is ended | `""` |
461-
| `metrics.serviceMonitor.labels` | Additional labels that can be used so ServiceMonitor will be discovered by Prometheus | `{}` |
462-
| `metrics.serviceMonitor.selector` | Prometheus instance selector labels | `{}` |
463-
| `metrics.serviceMonitor.relabelings` | RelabelConfigs to apply to samples before scraping | `[]` |
464-
| `metrics.serviceMonitor.metricRelabelings` | MetricRelabelConfigs to apply to samples before ingestion | `[]` |
465-
| `metrics.serviceMonitor.honorLabels` | Specify honorLabels parameter to add the scrape endpoint | `false` |
466-
| `metrics.serviceMonitor.jobLabel` | The name of the label on the target service to use as the job name in prometheus. | `""` |
467-
468415
### Airflow database parameters
469416

470417
| Name | Description | Value |

‎addons/airflow/2/chart/airflow/templates/_helpers.tpl‎

Lines changed: 18 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -86,17 +86,17 @@ Return the proper Airflow Worker image name
8686
{{- end -}}
8787

8888
{{/*
89-
Return the proper git image name
89+
Return the proper Airflow Worker image name
9090
*/}}
91-
{{- define "git.image" -}}
92-
{{- include "common.images.image" (dict "imageRoot" .Values.git.image "global" .Values.global) -}}
91+
{{- define "airflow.statsdImage" -}}
92+
{{- include "common.images.image" (dict "imageRoot" .Values.statsd.image "global" .Values.global) -}}
9393
{{- end -}}
9494

9595
{{/*
96-
Return the proper Airflow Metrics image name
96+
Return the proper git image name
9797
*/}}
98-
{{- define "airflow.metrics.image" -}}
99-
{{- include "common.images.image" (dict "imageRoot" .Values.metrics.image "global" .Values.global) -}}
98+
{{- define "git.image" -}}
99+
{{- include "common.images.image" (dict "imageRoot" .Values.git.image "global" .Values.global) -}}
100100
{{- end -}}
101101

102102
{{/*
@@ -110,7 +110,7 @@ Return the proper load Airflow DAGs image name
110110
Return the proper Docker Image Registry Secret Names
111111
*/}}
112112
{{- define "airflow.imagePullSecrets" -}}
113-
{{- include "common.images.pullSecrets" (dict "images" (list .Values.web.image .Values.scheduler.image .Values.worker.image .Values.git .Values.metrics.image) "global" .Values.global) -}}
113+
{{- include "common.images.pullSecrets" (dict "images" (list .Values.web.image .Values.scheduler.image .Values.worker.image .Values.git .Values.statsd.image) "global" .Values.global) -}}
114114
{{- end -}}
115115

116116
{{/*
@@ -349,19 +349,18 @@ Add environment variables to configure redis values
349349
*/}}
350350
{{- define "airflow.configure.redis" -}}
351351
{{- if (not (or (eq .Values.executor "KubernetesExecutor" ) (eq .Values.executor "LocalKubernetesExecutor" ))) }}
352-
- name: REDIS_HOST
353-
value: {{ ternary (include "airflow.redis.fullname" .) .Values.externalRedis.host .Values.redis.enabled | quote }}
354-
- name: REDIS_PORT_NUMBER
355-
value: {{ ternary "6379" .Values.externalRedis.port .Values.redis.enabled | quote }}
356-
{{- if and (not .Values.redis.enabled) .Values.externalRedis.username }}
357-
- name: REDIS_USER
358-
value: {{ .Values.externalRedis.username | quote }}
359-
{{- end }}
360-
- name: REDIS_PASSWORD
352+
- name: AIRFLOW_CELERY_BROKER_URL
361353
valueFrom:
362354
secretKeyRef:
363-
name: {{ include "airflow.redis.secretName" . }}
364-
key: redis-password
355+
name: {{ printf "%s-%s" .Release.Name "celerybroker" }}
356+
key: celery-broker-url
357+
{{- if .Values.celeryBrokerTransportOption }}
358+
- name: AIRFLOW_CELERY_BROKER_TRANSPORT_OPTIONS
359+
valueFrom:
360+
secretKeyRef:
361+
name: {{ printf "%s-%s" .Release.Name "celerybroker" }}
362+
key: celery-broker-transport-option
363+
{{- end }}
365364
{{- end }}
366365
{{- end -}}
367366

@@ -530,7 +529,7 @@ airflow: git.plugins.repositories[$index].branch
530529
{{- include "common.warnings.rollingTag" .Values.scheduler.image }}
531530
{{- include "common.warnings.rollingTag" .Values.worker.image }}
532531
{{- include "common.warnings.rollingTag" .Values.git.image }}
533-
{{- include "common.warnings.rollingTag" .Values.metrics.image }}
532+
{{- include "common.warnings.rollingTag" .Values.statsd.image }}
534533
{{- end -}}
535534

536535
{{/*

‎addons/airflow/2/chart/airflow/templates/_statsd_helpers.tpl‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
{{/*
2+
Copyright Drycc Community.
3+
SPDX-License-Identifier: APACHE-2.0
4+
*/}}
5+
6+
{{/*
7+
Returns the container that will statsd metrics
8+
Usage:
9+
{{ include "airflow.statsd.container" ( dict "securityContext" .Values.path.to.the.component.securityContext "context" $ ) }}
10+
*/}}
11+
{{- define "airflow.statsd.container" -}}
12+
- name: statsd
13+
image: {{ include "airflow.statsdImage" .context | quote }}
14+
imagePullPolicy: {{ .context.Values.statsd.image.pullPolicy | quote }}
15+
{{- if .securityContext.enabled }}
16+
securityContext: {{- omit .securityContext "enabled" | toYaml | nindent 4 }}
17+
{{- end }}
18+
{{- if .context.Values.statsd.lifecycleHooks }}
19+
lifecycle: {{- include "common.tplvalues.render" (dict "value" .context.Values.statsd.lifecycleHooks "context" $) | nindent 4 }}
20+
{{- end }}
21+
{{- if .context.Values.statsd.resources }}
22+
resources: {{- toYaml .context.Values.statsd.resources | nindent 4 }}
23+
{{- end }}
24+
{{- with .context.Values.statsd.env }}
25+
env: {{- toYaml . | nindent 4 }}
26+
{{- end }}
27+
ports:
28+
- name: statsd-scrape
29+
containerPort: {{ .context.Values.statsd.ports.scrape }}
30+
livenessProbe:
31+
httpGet:
32+
path: /metrics
33+
port: {{ .context.Values.statsd.ports.scrape }}
34+
initialDelaySeconds: 10
35+
periodSeconds: 10
36+
timeoutSeconds: 5
37+
readinessProbe:
38+
httpGet:
39+
path: /metrics
40+
port: {{ .context.Values.statsd.ports.scrape }}
41+
initialDelaySeconds: 10
42+
periodSeconds: 10
43+
timeoutSeconds: 5
44+
{{- end -}}

0 commit comments

Comments
 (0)