Skip to content

platform-monitoring.md

Latest commit

 

History

History
100 lines (76 loc) · 6.47 KB

platform-monitoring.md

File metadata and controls

100 lines (76 loc) · 6.47 KB

Platform Monitoring

Description

We now include a monitoring stack for introspection on a running Kubernetes cluster. The stack includes 3 components:

  • Telegraf - Metrics collection daemon written by team behind InfluxDB.
  • InfluxDB - Time series database
  • Grafana - Graphing tool for time series data

Architecture Diagram

                        ┌────────┐                                        
                        │ Router │                  ┌────────┐     ┌─────┐
                        └────────┘                  │ Logger │◀───▶│Redis│
                            │                       └────────┘     └─────┘
                        Log file                        ▲                
                            │                           │                
                            ▼                           │                
┌────────┐             ┌─────────┐    logs/metrics   ┌─────┐             
│App Logs│──Log File──▶│ fluentd │───────topics─────▶│ NSQ │             
└────────┘             └─────────┘                   └─────┘             
                                                        │                
                                                        │                
┌─────────────┐                                         │                
│ HOST        │                                         ▼                
│  Telegraf   │───┐                                 ┌────────┐            
└─────────────┘   │                                 │Telegraf│            
                  │                                 └────────┘            
┌─────────────┐   │                                      │                
│ HOST        │   │    ┌───────────┐                     │                
│  Telegraf   │───┼───▶│ InfluxDB  │◀────Wire ───────────┘                
└─────────────┘   │    └───────────┘   Protocol                   
                  │          ▲                                    
┌─────────────┐   │          │                                    
│ HOST        │   │          ▼                                    
│  Telegraf   │───┘    ┌──────────┐                               
└─────────────┘        │ Grafana  │                               
                       └──────────┘

Grafana

Deis Workflow exposes Grafana through the router using service annotations. This allows users to access the Grafana UI at http://grafana.mydomain.com. The default username/password of admin/admin can be overridden at any time by setting the following environment variables in $CHART_HOME/workspace/workflow-$WORKFLOW_RELEASE/manifests/deis-monitor-grafana-deployment.yaml: GRAFANA_USER and GRAFANA_PASSWD.

Grafana will preload several dashboards to help operators get started with monitoring Kubernetes and Deis Workflow. These dashboards are meant as starting points and don't include every item that might be desirable to monitor in a production installation.

Deis Workflow monitoring does not currently write data to the host filesystem or to long-term storage. If the Grafana instance fails, modified dashboards are lost. Until there is a solution to persist this, export dashboards and store them separately in version control.

Off Cluster Grafana

It is recommended that users provide their own installation for Grafana if possible. The current deployment of Grafana within Workflow is not durable across pod restarts which means custom dashboards that are created after startup will not be restored when the pod comes back up. If you wish to provide your own Grafana instance you can set grafana_location in the values.yaml file before running helm install.

InfluxDB

InfluxDB writes data to the host disk, however, if the InfluxDB pod dies and comes back on another host the data will not be recovered. We intend to fix this in a future release. The InfluxDB Admin UI is also exposed through the router allowing users to access the query engine by going to influx.mydomain.com. You will need to configure where to find the influx-api endpoint by clicking the "gear" icon at the top right and changing the host to influxapi.mydomain.com and port to 80.

** Note: Each user accessing the Influx UI will need to make this change. **

You can choose to not expose the Influx UI and API to the world by updating $CHART_HOME/workspace/workflow-$WORKFLOW_RELEASE/manifests/deis-monitor-influxdb-api-svc.yaml and $CHART_HOME/workspace/workflow-$WORKFLOW_RELEASE/manifests/deis-monitor-influxdb-ui-svc.yaml and removing the following line - router.deis.io/routable: "true".

Off Cluster Influxdb

To use off-cluster Influx, please provide the following values in the values.yaml file before running helm install.

  • influxdb_location=off-cluster
  • url = "http://my-influxhost.com:8086"
  • database = "metrics"
  • user = "InfluxUser"
  • password = "MysuperSecurePassword"

Telegraf

Telegraf is the metrics collection daemon used within the monitoring stack. It will collect and send the following metrics to InfluxDB:

  • System level metrics such as CPU, Load Average, Memory, Disk, and Network stats
  • Container level metrics such as CPU and Memory
  • Kubernetes metrics such as API request latency, Pod Startup Latency, and number of running pods

It is possible to send these metrics to other endpoints besides InfluxDB. For more information please consult the following file

Customizing

To learn more about customizing each of the above components please visit the monitor repository for more information.