feat(contrib/ec2): use separate volume for /var/lib/docker

We've seen fewer performance issues when machines use a separate
volume for /var/lib/docker. This commit changes the AWS provision
script to automatically prepare and mount a separate volume for
Docker.

We have to touch `/etc/docker-volume-formatted` because the
user-data is applied on every reboot, and we don't want
the Docker graph getting destroyed.

Note that we use the device name `/dev/xvdf` irrespective
of the instance virtualization type (PV or HVM). This seems to be
the way the kernel maps the block device, and isn't necessarily
unusual - see http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html

Amazon documentation suggests using the letter 'f' as the first name
of a non-root EBS-backed device: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/block-device-mapping-concepts.html

I manually tested this on both PV and HVM instances and had no issues.
There is no change in UX other than the new requirement of installing
the PyYAML Python library.

Author: carmstrong

contrib/ec2/deis.template.json

@@ -251,6 +251,10 @@
         "BlockDeviceMappings" : [
           {
             "DeviceName" : { "Fn::FindInMap": [ "RootDevices", { "Ref": "EC2VirtualizationType" }, "Name" ] },
+            "Ebs" : { "VolumeSize" : "50" }
+          },
+          {
+            "DeviceName" : "/dev/xvdf",
             "Ebs" : { "VolumeSize" : "100" }
           }
         ]

contrib/ec2/gen-json.py

@@ -1,13 +1,42 @@
 #!/usr/bin/env python
 import json
 import os
+import yaml
 
 template = json.load(open("deis.template.json",'r'))
 
-with open('../coreos/user-data','r') as f:
-  lines = f.readlines()
+# Add EC2-specific units to the shared user-data
+FORMAT_EPHEMERAL = '''
+  [Unit]
+  Description=Formats the ephemeral drive
+  ConditionPathExists=!/etc/docker-volume-formatted
+  [Service]
+  Type=oneshot
+  RemainAfterExit=yes
+  ExecStart=/usr/sbin/wipefs -f /dev/xvdf
+  ExecStart=/usr/sbin/mkfs.btrfs -f /dev/xvdf
+  ExecStart=/bin/touch /etc/docker-volume-formatted
+'''
+DOCKER_MOUNT = '''
+  [Unit]
+  Description=Mount ephemeral to /var/lib/docker
+  Requires=format-ephemeral.service
+  After=format-ephemeral.service
+  Before=docker.service
+  [Mount]
+  What=/dev/xvdf
+  Where=/var/lib/docker
+  Type=btrfs
+'''
 
-template['Resources']['CoreOSServerLaunchConfig']['Properties']['UserData']['Fn::Base64']['Fn::Join'] = [ '', lines ]
+data = yaml.load(file('../coreos/user-data', 'r'))
+data['coreos']['units'].append(dict({'name': 'format-ephemeral.service', 'command': 'start', 'content': FORMAT_EPHEMERAL}))
+data['coreos']['units'].append(dict({'name': 'var-lib-docker.mount', 'command': 'start', 'content': DOCKER_MOUNT}))
+
+header = ["#cloud-config", "---"]
+dump = yaml.dump(data, default_flow_style=False)
+
+template['Resources']['CoreOSServerLaunchConfig']['Properties']['UserData']['Fn::Base64']['Fn::Join'] = [ "\n", header + dump.split("\n") ]
 template['Parameters']['ClusterSize']['Default'] = str(os.getenv('DEIS_NUM_INSTANCES', 3))
 
 VPC_ID = os.getenv('VPC_ID', None)

docs/installing_deis/aws.rst

@@ -20,10 +20,12 @@ In order to start working with Amazon's API, let's install `awscli`_:
 .. code-block:: console
 
     $ pip install awscli
-    Downloading/unpacking awscli
-      Downloading awscli-1.5.0.tar.gz (248kB): 248kB downloaded
-    ...
-    Successfully installed awscli
+
+We'll also need `PyYAML`_ for the Deis EC2 provision script to run:
+
+.. code-block:: console
+
+    $ pip install pyyaml
 
 
 Configure aws-cli
@@ -177,3 +179,4 @@ start installing the platform.
 .. _`cloudformation.json`: https://github.com/deis/deis/blob/master/contrib/ec2/cloudformation.json
 .. _`etcd`: https://github.com/coreos/etcd
 .. _`optimal etcd cluster size`: https://github.com/coreos/etcd/blob/master/Documentation/optimal-cluster-size.md
+.. _`PyYAML`: http://pyyaml.org/