fix(contrib/ec2): use shared user-data

Previously, contrib/ec2 used its own stripped-down user-data file.
We have customizations in contrib/coreos/user-data, and we want all
providers using the same user-data file.

To do this, we introduce a Python script which will dynamically generate
the appropriate JSON payload from contrib/coreos/user-data when running
the provision-ec2-cluster script. While ideally this would be achieved without
requiring an external scripting language, this is very easy to achieve in Python,
and we already require Python as a dependency (since the AWS CLI is a Python package).

Also, have EC2 respect DEIS_NUM_INSTANCES like every other provider does.

closes #858, #826

Author: carmstrong

contrib/ec2/README.md

@@ -25,19 +25,29 @@ $ ssh-keygen -q -t rsa -f ~/.ssh/deis -N '' -C deis
 $ aws ec2 import-key-pair --key-name deis --public-key-material file://~/.ssh/deis.pub
 ```
 
-## Customize cloudformation.json
-Edit [cloudformation.json][cf-params], ensuring to add a new discovery URL.
-You can get a new one by sending a new request to http://discovery.etcd.io/new.
+## Choose number of instances
+By default, the script will provision 3 servers. You can override this by setting `DEIS_NUM_INSTANCES`:
 ```console
+$ export DEIS_NUM_INSTANCES=5
+```
+
+## Customize user-data
+Edit [user-data](../coreos/user-data) and add a new discovery URL.
+You can get a new one by sending a request to http://discovery.etcd.io/new.
+
+## Customize cloudformation.json
+By default, this script spins up m3.large instances. You can override this
+by adding a new entry to [cloudformation.json](cloudformation.json) like so:
+
+```
     {
-        "ParameterKey":     "DiscoveryURL",
-        "ParameterValue":   "https://discovery.etcd.io/40826e8da55f4d9026935ab67b243c6a"
+        "ParameterKey":     "InstanceType",
+        "ParameterValue":   "m3.xlarge"
     }
 ```
-NOTE: If you're interested in running your own discovery endpoint or want to know more
-about the discovery URL, see http://discovery.etcd.io for more information. You can also
-read more on how you can customize this cluster by looking at the
-[CoreOS EC2 template][template] and applying it to [cloudformation.json][cf-params].
+
+The only entry in cloudformation.json required to launch your cluster is `KeyPair`,
+which is already filled out. The defaults will be applied for the other settings.
 
 ## Run the provision script
 Run the [cloudformation provision script][pro-script] to spawn a new CoreOS cluster:
@@ -77,5 +87,4 @@ email: info@opdemand.com
 
 [aws-cli]: https://github.com/aws/aws-cli
 [template]: https://s3.amazonaws.com/coreos.com/dist/aws/coreos-alpha.template
-[cf-params]: cloudformation.json
 [pro-script]: provision-ec2-cluster.sh

contrib/ec2/cloudformation.json

@@ -2,9 +2,5 @@
     {
         "ParameterKey":     "KeyPair",
         "ParameterValue":   "deis"
-    },
-    {
-        "ParameterKey":     "DiscoveryURL",
-        "ParameterValue":   "https://discovery.etcd.io/06f9b54a2ffe65f072109924604b4cd7"
     }
 ]

contrib/ec2/deis.template

@@ -44,16 +44,6 @@
       "Description": "Number of nodes in cluster (3-12).",
       "Type": "Number"
     },
-    "DiscoveryURL": {
-      "Description": "An unique etcd cluster discovery URL. Grab a new token from https://discovery.etcd.io/new",
-      "Type": "String"
-    },
-    "AdvertisedIPAddress": {
-      "Description": "Use 'private' if your etcd cluster is within one region or 'public' if it spans regions or cloud providers.",
-      "Default": "private",
-      "AllowedValues": ["private", "public"],
-      "Type": "String"
-    },
     "AllowSSHFrom": {
       "Description": "The net block (CIDR) that SSH is available to.",
       "Default": "0.0.0.0/0",
@@ -169,21 +159,7 @@
         "InstanceType": {"Ref": "InstanceType"},
         "KeyName": {"Ref": "KeyPair"},
         "SecurityGroups": [{"Ref": "CoreOSSecurityGroup"}, {"Ref": "DeisSecurityGroup"}],
-        "UserData" : { "Fn::Base64":
-          { "Fn::Join": [ "", [
-            "#cloud-config\n\n",
-            "coreos:\n",
-            "  etcd:\n",
-            "    discovery: ", { "Ref": "DiscoveryURL" }, "\n",
-            "    addr: $", { "Ref": "AdvertisedIPAddress" }, "_ipv4:4001\n",
-            "    peer-addr: $", { "Ref": "AdvertisedIPAddress" }, "_ipv4:7001\n",
-            "  units:\n",
-            "    - name: etcd.service\n",
-            "      command: start\n",
-            "    - name: fleet.service\n",
-            "      command: start\n"
-            ] ]
-          }
+        "UserData" : { "Fn::Base64": { "Fn::Join": [ "", [ ] ] }
         }
       }
     }

contrib/ec2/gen-json.py

@@ -0,0 +1,13 @@
+#!/usr/bin/env python
+import json
+import os
+
+template = json.load(open("deis.template",'r'))
+
+with open('../coreos/user-data','r') as f:
+  lines = f.readlines()
+
+template['Resources']['CoreOSServerLaunchConfig']['Properties']['UserData']['Fn::Base64']['Fn::Join'] = [ '', lines ]
+template['Parameters']['ClusterSize']['Default'] = str(os.getenv('DEIS_NUM_INSTANCES', 3))
+
+print json.dumps(template)

contrib/ec2/provision-ec2-cluster.sh

@@ -18,7 +18,7 @@ fi
 
 # create an EC2 cloudformation stack based on CoreOS's default template
 aws cloudformation create-stack \
-    --template-body "$(<deis.template)" \
+    --template-body "$(./gen-json.py)" \
     --stack-name deis \
     --parameters "$(<cloudformation.json)"