docs(*): flesh out platform logging docs; add logspout docs

Author: carmstrong

docs/managing_deis/controller_settings.rst

@@ -13,7 +13,7 @@ Requires: :ref:`cache <cache_settings>`, :ref:`database <database_settings>`, :r
 
 Required by: :ref:`router <router_settings>`
 
-Considerations: must live on the same host as logger (see `#985`_)
+Considerations: none
 
 Settings set by controller
 --------------------------
@@ -73,4 +73,3 @@ Be sure that your custom image functions in the same way as the `stock controlle
 Deis. Specifically, ensure that it sets and reads appropriate etcd keys.
 
 .. _`stock controller image`: https://github.com/deis/deis/tree/master/controller
-.. _`#985`: https://github.com/deis/deis/issues/985

docs/managing_deis/logger_settings.rst

@@ -13,7 +13,7 @@ Requires: none
 
 Required by: :ref:`controller <controller_settings>`
 
-Considerations: must live on the same host as controller (see `#985`_)
+Considerations: none
 
 Settings set by logger
 ------------------------
@@ -32,6 +32,12 @@ The logger component uses no keys from etcd.
 
 Using a custom logger image
 ---------------------------
+
+.. note::
+
+  Instead of using a custom logger image, it is possible to redirect Deis logs to an external location.
+  For more details, see :ref:`platform_logging`.
+
 You can use a custom Docker image for the logger component instead of the image
 supplied with Deis:
 
@@ -50,4 +56,3 @@ Be sure that your custom image functions in the same way as the `stock logger im
 Deis. Specifically, ensure that it sets and reads appropriate etcd keys.
 
 .. _`stock logger image`: https://github.com/deis/deis/tree/master/logger
-.. _`#985`: https://github.com/deis/deis/issues/985

docs/managing_deis/platform_logging.rst

@@ -4,16 +4,53 @@
 .. _platform_logging:
 
 Platform logging
-=========================
+================
 
-Comprehensive platform logging is a goal for Deis 1.0. We are currently investigating solutions for
-this, and progress can be tracked in GitHub issue `#980`_.
+Logging for Deis components and deployed applications is handled by two components:
+:ref:`logger` and :ref:`logspout`.
 
-In the meantime, however, ``journalctl`` can be used to pipe log output to a remote host. For example,
-to send service log output to Papertrail:
+``deis-logspout`` is based on progrium's `logspout`_ project and runs on all CoreOS hosts. It
+collects logs from running containers and sends their logs to ``/deis/logs/host`` and
+``/deis/logs/port``.
+
+``deis-logger`` collects the logs sent by logspout and archives them for use by :ref:`Controller`
+when a client runs ``deis logs``. This component publishes its host and port to ``/deis/logs/host``
+and ``/deis/logs/port``, and is typically the service which consumes logs from ``deis-logspout``.
+
+Routing logs to a custom location
+---------------------------------
+
+Logging to an external location can be achieved without modifying the log flow within Deis -
+we can simply send the master journal on a CoreOS host using ``ncat``. For example, if I'm using the
+`Papertrail`_ hosted log service, I can forward all logs on a host to Papertrail using the host
+and port provided to me by Papertrail:
+
+.. code-block:: console
+
+    $ journalctl -o short -f | ncat --udp logs2.papertrailapp.com 23654
+
+This is really only useful when shipped as a service that we don't have to run ourselves in
+a shell. We can use a fleet service for this:
 
 .. code-block:: console
 
-    journalctl -o short -f | ncat --udp logs.papertrailapp.com 34000
+    [Unit]
+    Description=Log forwarder
+
+    [Service]
+    ExecStart=/bin/sh -c "journalctl -o short -f | ncat --udp logs2.papertrailapp.com 23654"
+
+    [Install]
+    WantedBy=multi-user.target
+
+    [X-Fleet]
+    Global=true
+
+Save the file as ``log-forwarder.service``. Load and start the service with
+``fleetctl load log-forwarder.service && fleetctl start log-forwarder.service``.
+
+Shortly thereafter, you should start to see logs from every host in your cluster appear in the
+Papertrail dashboard.
 
-.. _`#980`: https://github.com/deis/deis/issues/980
+.. _`logspout`: https://github.com/progrium/logspout
+.. _`papertrail`: https://papertrailapp.com/

docs/understanding_deis/architecture.rst

@@ -64,12 +64,19 @@ Registry
 The registry component hosts `Docker`_ images on behalf of the platform.
 Image data is stored by :ref:`Store`.
 
+.. _logspout:
+
+Logspout
+--------
+The logspout component is a customized version of `progrium's logspout`_ that runs
+on all CoreOS hosts in the cluster and collects logs from running containers. It sends the logs
+to the :ref:`logger` component.
+
 .. _logger:
 
 Log Server
 ----------
-The log server component uses `rsyslog`_ to aggregate log data from
-across the platform.
+The log server component collects logs from the :ref:`logspout` component.
 This data can then be queried by the :ref:`Controller`.
 
 .. _router:
@@ -96,5 +103,6 @@ which need to store state (namely :ref:`Registry` and :ref:`Database`).
 .. _`Nginx`: http://nginx.org/
 .. _`OpenStack Storage`: http://www.openstack.org/software/openstack-storage/
 .. _`PostgreSQL`: http://www.postgresql.org/
+.. _`progrium's logspout`: https://github.com/progrium/logspout
 .. _`Redis`: http://redis.io/
 .. _`rsyslog`: http://www.rsyslog.com/