With the release of Workflow Beta4 we now include a monitoring stack for introspection on a running Kubernetes cluster. The stack includes 4 components:
- Telegraf - Metrics collection daemon written by team behind InfluxDB.
- InfluxDB - Time series database
- Grafana - Graphing tool for time series data
- Stdout-Metrics - Tool for consuming metrics via standard out and forwards them to InfluxDB
┌────────┐
│ Router │
└────────┘
│
│
▼ ┌──────────┐
┌─────────────┐ ┌─────────┐ │ stdout │
│ HOST │ │ fluentd │────▶│ metrics │
│ Telegraf │───┐ └─────────┘ └──────────┘
└─────────────┘ │ │
│ │
┌─────────────┐ │ │
│ HOST │ │ ┌───────────┐ │
│ Telegraf │───┼───▶│ InfluxDB │◀─────────┘
└─────────────┘ │ └───────────┘
│ │
┌─────────────┐ │ │
│ HOST │ │ ▼
│ Telegraf │───┘ ┌──────────┐
└─────────────┘ │ Grafana │
└──────────┘
We expose Grafana through the router using service annotations. This
allows users to access the Grafana UI by accessing grafana.mydomain.com. While we provide a default username/password
of admin/admin this can be overridden at any time by setting the following environment variables in
$CHART_HOME/workspace/workflow-$WORKFLOW_RELEASE/manifests/deis-monitor-grafana-rc.yaml: GRAFANA_USER and
GRAFANA_PASSWD.
It will preload several dashboards that we've created to help operators get started with monitoring their Kubernetes and Workflow installations. Each dashboard is meant to be a starting place for the operator and is not representative of all the dashboards needed to monitor a production installation.
We are currently not writing the data to the host file system or to long term storage. Therefore, if the Grafana instance dies you will lose all custom and modified dashboards. It is recommended that you export your dashboards and store them in version control until a solution is implemented for long term storage.
As of the Beta4 release InfluxDB is writing data to the host disk, however, if the InfluxDB pod dies and comes back on
another host the data will not be recovered. We intend to fix this in a future release. The InfluxDB Admin UI is also
exposed through the router allowing users to access the query engine by going to influx.mydomain.com. You will need to
configure where to find the influx-api endpoint by clicking the "gear" icon at the top right and changing the host to
influxapi.mydomain.com and port to 80.
** Note: Each user accessing the Influx UI will need to make this change. **
You can choose to not expose the Influx UI and API to the world by updating
$CHART_HOME/workspace/workflow-$WORKFLOW_RELEASE/manifests/deis-monitor-influxdb-api-svc.yaml and
$CHART_HOME/workspace/workflow-$WORKFLOW_RELEASE/manifests/deis-monitor-influxdb-ui-svc.yaml and removing the
following line - router.deis.io/routable: "true".
Telegraf is the metrics collection daemon used within the monitoring stack. It will collect and send the following metrics to InfluxDB:
- System level metrics such as CPU, Load Average, Memory, Disk, and Network stats
- Container level memtrics such as CPU and Memory
- Kubernetes metrics such as API request latency, Pod Startup Latency, and number of running pods
It is possible to send these metrics to other endpoints besides InfluxDB. For more information please consult the following file
Stdout-Metrics is a custom tool built by the Deis team to provide metrics that are reported via standard out - like Nginx. It consumes the log stream from FluentD filtering out messages that are not from the Deis Router. Once it finds a message it can parse it will turn that into a metric and send it directly to InfluxDB.
Each of these components allows for customization via environment variables. If you would like to learn more please visit the following github repositories: