Merge pull request #2779 from carmstrong/feat-debug_etcd

feat(contrib/coreos): add etcd debug unit

Author: carmstrong

contrib/coreos/user-data.example

@@ -56,6 +56,16 @@ coreos:
       ExecStart=/sbin/hwclock --systohc --utc
       RemainAfterExit=yes
       Type=oneshot
+  - name: debug-etcd.service
+    command: enable
+    content: |
+      [Unit]
+      Description=etcd debugging service
+
+      [Service]
+      ExecStartPre=/usr/bin/curl -sSL -o /opt/bin/jq http://stedolan.github.io/jq/download/linux64/jq
+      ExecStartPre=/usr/bin/chmod +x /opt/bin/jq
+      ExecStart=/usr/bin/bash -c "while true; do curl -sL http://127.0.0.1:4001/v2/stats/leader | /opt/bin/jq . ; sleep 1 ; done"
 write_files:
   - path: /etc/deis-release
     content: |

docs/troubleshooting_deis/index.rst

@@ -22,6 +22,18 @@ Connect to the public IP address of one of your nodes (or use "convenience" DNS
     $ ssh core@deis-1.example.com -i ~/.ssh/deis.pub
 
 
+Troubleshooting etcd
+--------------------
+
+Sometimes issues with Deis are caused by latency between CoreOS hosts. A telltale sign of this is
+if all of the Deis components on a single machine crash. To aid in debugging etcd, we've created
+a system service that is installed but not started when you deploy CoreOS using our provision scripts.
+
+To start this service, run ``sudo systemctl start debug-etcd`` on a CoreOS machine in your cluster.
+This starts a service which queries etcd's state once per second. Watching this output with
+``journalctl -fu debug-etcd`` makes it easy to spot heartbeat timeouts or other abnormalities
+which will lead to issues running Deis successfully.
+
 A deis-store component fails to start
 -------------------------------------