Merge pull request #2450 from gabrtv/understanding-deis

docs(understanding_deis): refactor architecture, concepts, components

Author: Gabriel Monroy

docs/understanding_deis/DeisArchitecturalDiagram.png

@@ -6,103 +6,70 @@
 Architecture
 ============
 
-.. image:: DeisArchitecturalDiagram.png
-    :alt: Draft Architectural Diagram
+Deis uses a service oriented architecture with :ref:`components`
+grouped into a Control Plane, Data Plane and Router Mesh.
 
-.. TODO: Need a caption for the Deis architectural diagram
+.. _system-diagram:
 
-Deis consists of 7 components that combine to create a distributed PaaS.
-Each Deis component is deployed as a :ref:`Container`.
+System Diagram
+--------------
 
-.. _arch_controller:
+.. image:: DeisSystemDiagram.png
+    :alt: Deis System Diagram
 
-Controller
-----------
-The :ref:`controller <Controller>` component is a RESTful API server
-written with `Django`_ and `Celery`_. Command-line clients interact with
-this component.
+End-users of the platform interact with the Control Plane using the ``Deis API``.
+Operators use the ``Deisctl API`` to stand up the cluster's Control Plane, Data Plane
+and Router Mesh.
+
+The Control Plane dispatches work to the Data Plane via a scheduler.
+The Router Mesh is used to route traffic to both the Control Plane and Data Plane.
+Because the router mesh is usually connected to the public Internet,
+it is often connected to a front-end load balancer.
 
-.. _database:
+.. _control-plane:
 
-Database
---------
-The database component is a `PostgreSQL`_ server used to store durable
-platform state. Backups and WAL logs are pushed to :ref:`Store`.
+Control Plane
+-------------
 
-.. _cache:
+.. image:: DeisControlPlane.png
+    :alt: Deis Control Plane Architecture
 
-Cache
------
-The cache component uses `Redis`_ to:
+The Control Plane performs management functions for the platform.
+Control plane components (in blue) are all implemented as Docker containers.
 
- * Store work queue data for `Celery`_
- * Cache sessions and synchronize locks for `Django`_
- * Store recent log data for the :ref:`Controller`
+The :ref:`store` component consists of a number of smaller components that represent a
+containerized Ceph cluster which provides a blob storage API and POSIX filesystem API
+for control plane's stateful components:
 
-.. _builder:
+ * :ref:`registry` - a Docker registry used to hold images and configuration data
+ * :ref:`database` - a Postgres database used to store platform state
+ * :ref:`logger` - a syslog log server that holds aggregated logs from the data plane
 
-Builder
--------
-The builder component uses a `Git`_ server to process
-:ref:`Application` builds. The builder:
+End-users interact primarily with the :ref:`comp_controller` which exposes an
+HTTP API. They can also interact with the :ref:`builder` via ``git push``.
 
- #. Receives incoming ``git push`` requests over SSH
- #. Authenticates the user via SSH key fingerprint
- #. Authorizes the user's access to write to the Git repository
- #. Builds a new `Docker` image from the updated git repository
- #. Adds the latest :ref:`Config` to the resulting Docker image
- #. Pushes the new Docker image to the platform's :ref:`Registry`
- #. Creates a new :ref:`Release` on the :ref:`Controller`
+.. _data-plane:
 
-Once a new :ref:`Release` is generated, a new set of containers
-is deployed across the platform automatically.
+Data Plane
+----------
 
-.. _registry:
+.. image:: DeisDataPlane.png
+    :alt: Deis Data Plane Architecture
 
-Registry
---------
-The registry component hosts `Docker`_ images on behalf of the platform.
-Image data is stored by :ref:`Store`.
+The Data Plane is where :ref:`Containers <container>` (in blue) are run on behalf of end-users.
 
-.. _logspout:
+The platform scheduler is charge of placing containers on hosts in the data plane.
+Deis also requires a few lightweight components on these hosts:
 
-Logspout
---------
-The logspout component is a customized version of `progrium's logspout`_ that runs
-on all CoreOS hosts in the cluster and collects logs from running containers. It sends the logs
-to the :ref:`logger` component.
+ * :ref:`publisher` - publishes end-user containers to the :ref:`router`
+ * :ref:`logspout` - feeds log data to the Control Plane :ref:`logger`
 
-.. _logger:
+.. _topologies:
 
-Log Server
+Topologies
 ----------
-The log server component collects logs from the :ref:`logspout` component.
-This data can then be queried by the :ref:`Controller`.
-
-.. _router:
-
-Router
-------
-The router component uses `Nginx`_ to route traffic to
-application containers.
-
-.. _store:
-
-Store
-------
-The store component uses `Ceph`_ to store data for Deis components
-which need to store state (namely :ref:`Registry` and :ref:`Database`).
-
-.. _`Amazon S3`: http://aws.amazon.com/s3/
-.. _`Celery`: http://www.celeryproject.org/
-.. _`Ceph`: http://ceph.com
-.. _`Django`: https://www.djangoproject.com/
-.. _`Docker`: http://docker.io/
-.. _`etcd`: https://github.com/coreos/etcd
-.. _`Git`: http://git-scm.com/
-.. _`Nginx`: http://nginx.org/
-.. _`OpenStack Storage`: http://www.openstack.org/software/openstack-storage/
-.. _`PostgreSQL`: http://www.postgresql.org/
-.. _`progrium's logspout`: https://github.com/progrium/logspout
-.. _`Redis`: http://redis.io/
-.. _`rsyslog`: http://www.rsyslog.com/
+
+For small deployments you can run the entire platform
+-- Control Plane, Data Plane and Router Mesh -- on just 3 servers.
+For larger deployments, you'll want to isolate the Control Plane and Router Mesh,
+then scale your data plane out to as many servers as you need.

docs/understanding_deis/DeisControlPlane.png

@@ -0,0 +1,105 @@
+:title: Components
+:description: Components of the Deis application platform (PaaS)
+
+.. _components:
+
+Components
+==========
+
+Deis consists of a number of components that combine to create a distributed PaaS.
+Each Deis component is deployed as a container or set of containers.
+
+.. _comp_controller:
+
+Controller
+----------
+The controller component is an HTTP API server.
+The ``deis`` command-line client interacts with this component.
+
+.. _database:
+
+Database
+--------
+The database component is a `PostgreSQL`_ server used to store durable
+platform state. Backups and WAL logs are pushed to :ref:`Store`.
+
+.. _cache:
+
+Cache
+-----
+The cache is an instance of `Redis`_ used by the controller
+and registry.
+
+.. _builder:
+
+Builder
+-------
+The builder component uses a `Git`_ server to process
+:ref:`Application` builds. The builder:
+
+ #. Receives incoming ``git push`` requests over SSH
+ #. Authenticates the user via SSH key fingerprint
+ #. Authorizes the user's access to write to the Git repository
+ #. Builds a new `Docker` image from the updated git repository
+ #. Adds the latest :ref:`Config` to the resulting Docker image
+ #. Pushes the new Docker image to the platform's :ref:`Registry`
+ #. Creates a new :ref:`Release` on the :ref:`Controller`
+
+Once a new :ref:`Release` is generated, a new set of containers
+is deployed across the platform automatically.
+
+.. _registry:
+
+Registry
+--------
+The registry component hosts `Docker`_ images on behalf of the platform.
+Image data is stored by :ref:`Store`.
+
+.. _logspout:
+
+Logspout
+--------
+The logspout component is a customized version of `progrium's logspout`_ that runs
+on all CoreOS hosts in the cluster and collects logs from running containers.
+It sends the logs to the :ref:`logger` component.
+
+.. _logger:
+
+Logger
+------
+The logger component is a syslog server that collects logs from :ref:`logspout`
+components spread across the platform.
+This data can then be queried by the :ref:`Controller`.
+
+.. _publisher:
+
+Publisher
+---------
+The publisher component is a microservice written in Go that publishes
+containers to etcd so they can be exposed by the platform :ref:`router`.
+
+.. _router:
+
+Router
+------
+The router component uses `Nginx`_ to route traffic to application containers.
+
+.. _store:
+
+Store
+------
+The store component uses `Ceph`_ to store data for Deis components
+which need to store state, including :ref:`Registry`, :ref:`Database`
+and :ref:`Logger`.
+
+.. _`Amazon S3`: http://aws.amazon.com/s3/
+.. _`Celery`: http://www.celeryproject.org/
+.. _`Ceph`: http://ceph.com
+.. _`Docker`: http://docker.io/
+.. _`etcd`: https://github.com/coreos/etcd
+.. _`Git`: http://git-scm.com/
+.. _`Nginx`: http://nginx.org/
+.. _`OpenStack Storage`: http://www.openstack.org/software/openstack-storage/
+.. _`PostgreSQL`: http://www.postgresql.org/
+.. _`progrium's logspout`: https://github.com/progrium/logspout
+.. _`Redis`: http://redis.io/

docs/understanding_deis/DeisDataPlane.png

@@ -5,25 +5,23 @@
 
 Concepts
 ========
-Deis is a lightweight, flexible and powerful application platform that
-deploys and scales :ref:`concepts_twelve_factor` apps as
-:ref:`concepts_docker` containers across a cluster of
-:ref:`concepts_coreos` machines.
+Deis is a lightweight application platform that deploys and scales
+:ref:`concepts_twelve_factor` apps as :ref:`concepts_docker` containers
+across a cluster of :ref:`concepts_coreos` machines.
 
 .. _concepts_twelve_factor:
 
 Twelve-Factor
 -------------
-The `Twelve-Factor App`_ is a DevOps manifesto for building and
-deploying scalable, modern applications and services.
+The `Twelve-Factor App`_ is a methodology for building modern
+applications that can be scaled across a distributed system.
 
 We consider it an invaluable synthesis of much experience with
 software-as-a-service apps in the wild, especially on the
 Heroku platform.
 
-Deis works best with applications in a `Twelve-Factor App`_ style.
-Following the twelve-factor model, Deis enforces a strict separation of
-the :ref:`Build and Run <concepts_build_release_run>` stages.
+Deis is designed to run applications that adhere to `Twelve-Factor App`_
+methodology and best practices.
 
 .. _concepts_docker:
 
@@ -32,57 +30,54 @@ Docker
 `Docker`_ is an open source project to pack, ship and run any
 application as a lightweight, portable, self-sufficient container.
 
-When you deploy an app with ``git push deis master``, Deis builds and
-packages it as a Docker image, then distributes it as Docker containers
-across your cluster.
+Deis curates your applications as Docker images, which are then
+distributed across your cluster as Docker containers.
 
 (Deis itself is also a set of coordinated Docker containers.)
 
 .. _concepts_coreos:
 
 CoreOS
 ------
-`CoreOS`_ is a lean new Linux distribution, rearchitected for features
-needed by modern infrastructure stacks and targeted at massive
-server deployments.
+`CoreOS`_ is a new, minimal Linux distribution, rearchitected for
+running modern, containerized infrastructure stacks.
 
-Deis applications are processes running on CoreOS machines, which can
-be private or public cloud instances, or bare metal. CoreOS clusters
-allow Deis to host applications and services at scale with
-high resilience.
+Deis runs on CoreOS machines that be hosted anywhere -- public cloud,
+private cloud, bare metal or even your workstation.
 
-Yet Deis and CoreOS run identically in a Vagrant virtual machine on
-your laptop, for convenient testing and rapid development.
+CoreOS allows Deis to host applications and services at scale with
+high resilience, in a way that is simple to operate.
 
 .. _concepts_applications:
 
 Applications
 ------------
-An :ref:`application`, or app, lives on a cluster where it uses
-:ref:`Containers <container>` to process requests and run tasks for a
-deployed git repository.
+Deis is designed around the concept of an :ref:`application`, or app.
+Applications live on a cluster where they use :ref:`Containers <container>`
+to service requests.
 
-Developers use :ref:`Applications <application>` to push code, change
-configuration, scale processes, view logs, or run admin commands --
-regardless of the cluster's underlying infrastructure.
+Developers use applications to push code, change configuration, scale processes,
+view logs, run admin commands and much more.
 
 .. _concepts_build_release_run:
 
 Build, Release, Run
 -------------------
 
+.. image:: DeisGitPushWorkflow.png
+    :alt: Deis Git Push Workflow
+
 Build Stage
 ^^^^^^^^^^^
-The :ref:`Controller` includes a *gitreceive* hook that receives incoming git push requests over
-SSH and builds applications inside ephemeral Docker containers. Tarballs of the /app directory are
-extracted into a slug and is injected into another container, which will create the app image. The
-image is then pushed to a private registry for later execution.
+The :ref:`builder` processes incoming ``git push`` requests builds applications
+inside ephemeral Docker containers, creating a new Docker image.
 
 Release Stage
 ^^^^^^^^^^^^^
 During the release stage, a :ref:`build` is combined with :ref:`config` to create a new numbered
-:ref:`release`. The release stage is triggered any time a new build is created or config is
-changed, making it easy to rollback code and configuration.
+:ref:`release`. This release is then pushed to a Docker registry for later execution.
+The release stage is triggered any time a new build is created or config is
+changed, making it easy to rollback code and configuration changes.
 
 Run Stage
 ^^^^^^^^^
@@ -99,14 +94,13 @@ Deis treats databases, caches, storage, messaging systems, and other
 `backing services`_ as attached resources, in keeping with Twelve-Factor
 best practices.
 
-Applications can be decoupled this way, using simple
-`environment variables`_ to configure and attach to any services needed.
-Apps are then free to scale up independently, to use services provided
-by other apps, or to switch easily to external or third-party vendor
-services.
+Applications are attached to backing services using `environment variables`_.
+Because applications are decoupled from backing services, apps are free to scale up independently,
+to swap services provided by other apps, or to switch to external or third-party vendor services.
 
 See Also
 --------
+* :ref:`Architecture`
 * :ref:`Using Deis <using_deis>`
 * :ref:`Managing Deis <managing_deis>`
 * The `Twelve-Factor App`_

docs/understanding_deis/DeisGitPushWorkflow.png

@@ -10,3 +10,4 @@ Understanding Deis
 
     concepts
     architecture
+    components