docs(platform): document how to isolate etcd

Author: krancour

docs/_includes/_isolating-etcd-description.rst

@@ -0,0 +1,25 @@
+The Deis Control Plane, Data Plane, and Router Mesh components all depend on an
+etcd cluster for service discovery and configuration.
+
+Whether built for evaluation or to host production applications, when managing a
+small Deis cluster (three to five nodes), it is reasonable to accept the
+platform's default behavior wherein etcd runs on every node within the cluster.
+
+In larger Deis clusters however, running etcd on every node can have a
+deleterious effect on overall cluster performance since it increases the time
+required for nodes to reach consensus on writes and leader elections. In such
+cases, it is beneficial to isolate etcd to a small, fixed number of nodes.  All
+other nodes in the Deis cluster may run an etcd proxy.  Proxies will forward
+read and write requests to active participants in the etcd cluster (leader or
+followers) without affecting the time required for etcd nodes to reach consensus
+on writes or leader elections.
+
+.. note::
+
+    The benefit of running an etcd proxy on any node not running a full etcd
+    process is that any container or service depending on etcd can connect to
+    etcd easily via ``localhost`` from any node in the Deis cluster.
+
+    Also see `CoreOS cluster architecture documentation`_ for further details.
+
+.. _`CoreOS cluster architecture documentation`: https://coreos.com/os/docs/latest/cluster-architectures.html#production-cluster-with-central-services

docs/managing_deis/index.rst

@@ -20,6 +20,7 @@ Managing Deis
     platform_logging
     platform_monitoring
     production_deployments
+    isolating-etcd
     recovering-ceph-quorum
     running-deis-without-ceph
     security_considerations

docs/managing_deis/isolating-etcd.rst

@@ -0,0 +1,59 @@
+:title: Isolating etcd
+:description: Configuring the cluster to isolate etcd
+
+.. _isolating-etcd:
+
+Isolating etcd
+==============
+
+.. include:: ../_includes/_isolating-etcd-description.rst
+
+.. note::
+
+    The approach documented here works as of Deis 1.9.  Older versions of Deis
+    utilize an older version of etcd that did not include the proxy
+    functionality.
+
+cloud-config
+------------
+
+To realize the topology described above, it is necessary, at the time of
+provisioning, to provide different cloud-config for those hosts that will run
+etcd and for those that will only run an etcd proxy.
+
+For the small, fixed number of hosts running full etcd and satisfying the
+"central services" role (as described in the CoreOS documentation), the
+cloud-config provided with Deis is sufficient.
+
+For hosts running only an etcd proxy, satisfying the "worker" role (as described
+in the CoreOS documentation), cloud-config must be tweaked slightly to include
+the ``-proxy on`` flag. For example:
+
+.. code-block:: yaml
+
+    #cloud-config
+    ---
+    coreos:
+      # ...
+      - name: etcd.service
+        command: start
+        content: |
+          # ...
+          [Service]
+          # ...
+          ExecStart=/usr/bin/docker run --net=host --rm \
+            # ...
+            -listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001 \
+            # ...
+            --discovery <discovery url here> \
+            -proxy on
+          # ...
+        # ...
+      # ...
+    # ...
+
+Isolating etcd as described here requires subsets of a cluster's hosts to be
+configured differently from one another (including or excluding the
+``-proxy on`` flag). Deis provisioning scripts do not currently account for
+this, so managing separate cloud-config for each subset of nodes in the cluster
+is left as an exercise for the advanced operator.

docs/managing_deis/production_deployments.rst

@@ -9,6 +9,13 @@ Production deployments
 Many Deis users are running Deis quite successfully in production. When readying a Deis deployment
 for production workloads, there are some additional (but optional) recommendations.
 
+Isolating etcd
+--------------
+
+.. include:: ../_includes/_isolating-etcd-description.rst
+
+See :ref:`isolating-etcd` for further details.
+
 Running Deis without Ceph
 -------------------------
 

docs/understanding_deis/architecture.rst

@@ -72,3 +72,10 @@ For small deployments you can run the entire platform
 -- Control Plane, Data Plane and Router Mesh -- on just 3 servers.
 For larger deployments, you'll want to isolate the Control Plane and Router Mesh,
 then scale your data plane out to as many servers as you need.
+
+The Deis Control Plane, Data Plane, and Router Mesh components all depend on an
+etcd cluster for service discovery and configuration. For larger deployments,
+you'll want to isolate this as well.
+
+Those building larger clusters should reference :ref:`isolating-etcd` for
+further details.