feat(contrib): add DigitalOcean provisioner booting CoreOS via kexec

This introduces a provisioning script for Deis on DigitalOcean droplets. As DigitalOcean
does not (currently) support CoreOS, we deploy CoreOS on top of Ubuntu 14.04 using kexec
on boot to switch to the CoreOS kernel and initrd. The state filesystem is a loop
mounted btrfs.

The CoreOS bootstrapping is heavily based on Levi Aul's code:

  https://gist.github.com/tsutsu/490f35f48897df0f5173

We don't use Debian 7.0 as base distribution because kexec is unstable on the Debian
kernel.

Author: sttts

contrib/digitalocean/README.md

@@ -1,9 +1,69 @@
-Deis on DigitalOcean
-====================
+# How to Provision a Deis Controller on Digital Ocean
 
-Unfortunately, DigitalOcean does not yet provide CoreOS images. This
-prevents Deis from being deployed on DigitalOcean.
+Here are the steps to get started on Digital Ocean:
 
-If you use DigitalOcean, please
-[show your support](http://digitalocean.uservoice.com/forums/136585-digital-ocean/suggestions/4250154-suport-coreos-as-a-deployment-platform)
-for CoreOS and help us to support Deis on DO.
+## Customize cloud-config.yml
+Edit [user-data](../coreos/user-data) and add a discovery URL. This URL will be used by all nodes in this Deis cluster. You can get a new discovery URL by sending a request to http://discovery.etcd.io/new.
+
+## Install DO command line client and authorize:
+```console
+$ gem install tugboat
+$ tugboat authorize
+```
+You can leave all but the client and API keys as the defaults.
+
+Find out about the ID of your ssh key (import it into DO if it's not listed):
+```console
+$ tugboat keys
+```
+
+## Create a controller image:
+```console
+$ ./provision-digitalocean-controller-image.sh <YOU SSH KEY ID>
+```
+
+## Deploy controllers
+Use the created image to launch (an odd number) controller droplets, either via UI
+or on the command line using tugboat:
+```console
+$ tugboat create deis1 -r <REGION_ID> -i <IMAGE_ID> -p true -k <SSH_ID> -s 65
+```
+
+Not all regions allow private networks. Choose one which does, e.g. NY 2, Amsterdam 2 or
+Singapore 1 at the time of this writing (check the web UI for the current private network
+support).
+
+The provisioning script uses a 512 MB droplet by default because for iamge creation
+more memory is not needed. Deis controller nodes will need at least 2 GB to even start all
+the services. Add the memory requirements of deployed applications and choose an adequat
+droplet size, e.g. 8 GB (ID "65").
+
+## Initialize the cluster
+Once the cluster is up, get the IPs of any of the machines using `tugboat droplets`, set
+FLEETCTL_TUNNEL to one of these IPs:
+```console
+$ export FLEETCTL_TUNNEL=23.253.219.94
+$ cd ../.. && make run
+```
+The script will deploy Deis and make sure the services start properly.
+
+### Configure DNS
+You'll need to configure DNS records so you can access applications hosted on Deis. See [Configuring DNS](http://docs.deis.io/en/latest/operations/configure-dns/) for details.
+
+### Use Deis!
+After that, register with Deis!
+```console
+$ deis register http://deis.example.org
+username: deis
+password:
+password (confirm):
+email: info@opdemand.com
+```
+
+## Hack on Deis
+If you'd like to use this deployment to build Deis, you'll need to set `DEIS_HOSTS` to an array of your cluster hosts:
+```console
+$ export DEIS_HOSTS=10.21.12.1 10.21.12.2 10.21.12.3
+```
+
+This variable is used in the `make build` command.

contrib/digitalocean/cloud-config.yml

@@ -0,0 +1,76 @@
+#cloud-config
+
+hostname: HOSTNAME
+
+users:
+  - name: core
+    groups:
+      - sudo
+      - docker
+    ssh-authorized-keys:
+      - SSH_KEY
+
+coreos:
+  units:
+    - name: public.network
+      content: |
+        [Match]
+        Name=ens3
+
+        [Network]
+        Address=PUBLIC_IP
+        Gateway=GATEWAY
+        DNS=8.8.8.8
+        DNS=8.8.4.4
+    - name: private.network
+      content: |
+        [Match]
+        Name=ens4v1
+
+        [Network]
+        Address=PRIVATE_IP
+    - name: media-doroot.mount
+      command: start
+      content: |
+        [Mount]
+        What=/dev/vda
+        Where=/media/doroot
+        Type=ext4
+    - name: format-docker-store.service
+      command: start
+      content: |
+        [Unit]
+        Requires=media-doroot.mount
+        [Service]
+        Type=oneshot
+        RemainAfterExit=yes
+        ExecStart=/usr/share/oem/bin/create-coreos-docker-store
+    - name: var-lib-docker.mount
+      command: start
+      content: |
+        [Unit]
+        Requires=format-docker-store.service
+        Before=docker.service
+        [Mount]
+        What=/dev/disk/by-label/docker
+        Where=/var/lib/docker
+        Type=btrfs
+    - name: coreos-setup-environment.service
+      command: restart
+      runtime: yes
+      content: |
+        [Unit]
+        Before=docker.service
+        [Service]
+        Type=oneshot
+        ExecStart=/usr/share/oem/bin/coreos-setup-environment /etc/environment
+    - name: coreos-apply-user-data.service
+      command: restart
+      runtime: yes
+      content: |
+        [Unit]
+        After=coreos-setup-environment.service
+        [Service]
+        EnvironmentFile=/etc/environment
+        Type=oneshot
+        ExecStart=/usr/share/oem/bin/coreos-apply-user-data

contrib/digitalocean/coreos-apply-user-data

@@ -0,0 +1,2 @@
+#!/bin/bash
+coreos-cloudinit --from-file=/usr/share/oem/user-data.yml

contrib/digitalocean/coreos-setup-environment

@@ -0,0 +1,21 @@
+#!/bin/bash -e
+
+ENV=$1
+
+if [ -z "$ENV" ]; then
+        echo usage: $0 /etc/environment
+        exit 1
+fi
+
+touch $ENV
+if [ $? -ne 0 ]; then
+        echo exiting, unable to modify: $ENV
+        exit 1
+fi
+
+sed -i -e '/^COREOS_PUBLIC_IPV4=/d;/^COREOS_PRIVATE_IPV4=/d' $ENV
+
+COREOS_PUBLIC_IPV4=$(ip -4 -o addr show dev ens3 | awk '{ print $4; }' | cut -d / -f1)
+COREOS_PRIVATE_IPV4=$(ip -4 -o addr show dev ens4v1 | awk '{ print $4; }' | cut -d / -f1)
+echo COREOS_PUBLIC_IPV4=$COREOS_PUBLIC_IPV4 >> $ENV
+echo COREOS_PRIVATE_IPV4=$COREOS_PRIVATE_IPV4 >> $ENV

contrib/digitalocean/create-coreos-docker-store

@@ -0,0 +1,13 @@
+#!/bin/sh
+mkdir -p /media/doroot/var/lib/coreos
+
+if [ ! -f "/media/doroot/var/lib/coreos/docker.img" ]; then
+  DOFORMAT=1
+fi
+
+truncate -s 10G /media/doroot/var/lib/coreos/docker.img
+LODEV=`losetup -f --show /media/doroot/var/lib/coreos/docker.img`
+
+if [ -n "$DOFORMAT" ]; then
+  mkfs.btrfs -L docker "$LODEV"
+fi

contrib/digitalocean/provision-digitalocean-controller-image.sh

@@ -0,0 +1,106 @@
+#!/bin/bash
+
+set -e
+
+THIS_DIR=$(cd $(dirname $0); pwd) # absolute path
+CONTRIB_DIR=$(dirname $THIS_DIR)
+
+source $CONTRIB_DIR/utils.sh
+
+if ! which tugboat &>/dev/null; then
+	echo_red 'Digital Ocean command line client tugboat not found.'
+	exit 1
+fi
+
+function wait_ssh () {
+	echo -n "Trying to ssh into $1@$2..."
+	while ! $SSH -o ConnectTimeout=10 $1@$2 hostname &>/dev/null; do
+		echo -n "."
+		sleep 1
+	done
+	echo_green "done"
+}
+
+# parse parameters
+if [ -z "$1" ]; then
+	echo_red 'Usage: $0 SSH_ID [REGION_ID]'
+	echo
+	echo 'Use `tugboat keys` to list available SSH_IDs.'
+	exit 1
+fi
+
+SSH_ID="$1"
+REGION="${2:-5}" # Amsterdam 1 by default
+SIZE="66" # 512 MB
+NAME="deis-controller-image-$(date +%Y%m%d%H%M%S)"
+
+BASE_IMAGE='Ubuntu 14.04 x64'
+BASE_IMAGE_ID=$(tugboat images -g | grep "$BASE_IMAGE" | sed 's/.*id: \([0-9]*\).*/\1/')
+SSH_OPTIONS="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=1 -o ConnectionAttempts=10"
+SSH="ssh $SSH_OPTIONS"
+SCP="scp $SSH_OPTIONS -q"
+
+# create droplet
+tugboat create "$NAME" -i $BASE_IMAGE_ID -p true -k $SSH_ID -s $SIZE -r $REGION
+
+# destroy droplets on error
+function cleanup () {
+	set +e
+	tugboat destroy -c -n "$NAME"
+	exit 1
+}
+trap cleanup ERR
+trap cleanup SIGINT
+
+# wait for droplet to come up with ssh login
+tugboat wait -n "$NAME" --state active
+IP=$(tugboat info -n "$NAME" | egrep '^IP' | awk '{ print $2; }')
+wait_ssh root $IP
+
+# bootstrap
+echo "Deploying CoreOS on top of $BASE_IMAGE..."
+(
+	set -ex
+	$SCP update-coreos root@$IP:/usr/sbin
+	$SSH root@$IP mkdir -p /usr/share/oem/bin
+	$SCP cloud-config.yml root@$IP:/usr/share/oem/cloud-config.yml.template
+	$SCP ../coreos/user-data root@$IP:/usr/share/oem/user-data.yml
+	$SCP create-coreos-docker-store coreos-setup-environment coreos-apply-user-data root@$IP:/usr/share/oem/bin
+	$SCP rc.local root@$IP:/etc
+	$SSH root@$IP <<EOF
+set -xe
+
+DEBIAN_FRONTEND=noninteractive
+apt-get install debconf-utils -y
+echo "kexec-tools kexec-tools/load_kexec boolean false" | debconf-set-selections
+apt-get install squashfs-tools kexec-tools -y
+shutdown -h now
+EOF
+) 2>&1 | sed 's/^/    /'; test ${PIPESTATUS[0]} -eq 0
+
+# switch off and make snapshot
+tugboat wait -n "$NAME" --state off
+tugboat snapshot $NAME -n $NAME
+
+# wait for snapshot to finish
+echo -n "Waiting for snapsnot to appear..."
+while ! tugboat images | grep $NAME &>/dev/null; do
+	sleep 5
+	echo -n "."
+done
+IMAGE_ID=$(tugboat images | grep $NAME | awk '{print $3;}' | sed 's/,//')
+echo_green "done"
+
+echo -n "Trying to destroy droplet..."
+while ! tugboat destroy -c -n $NAME &>/dev/null; do
+	sleep 5
+	echo -n "."
+done
+echo_green "done"
+
+echo
+echo_green "Congratulations: The Deis controller image $NAME was created with ID $IMAGE_ID."
+echo
+echo "Spawn controllers with:"
+echo
+echo "  tugboat create deis1 -i $IMAGE_ID -r $REGION -p true -k $SSH_ID -s 65"

contrib/digitalocean/rc.local

@@ -0,0 +1,29 @@
+#!/bin/bash
+
+set -ex
+
+function echo_red {
+  echo -e "\033[0;31m$1\033[0m"
+}
+
+function error_handler () {
+	set +ex
+	echo_red "An error occured while bootstrapping CoreOS."
+	while true; do sleep 1; done
+	exit 1
+}
+trap error_handler ERR
+
+# create coreos cloud-config
+PUBLIC_IP=$(/sbin/ip -4 -o addr show dev eth0 | awk '{ print $4; }')
+PRIVATE_IP=$(/sbin/ip -4 -o addr show dev eth1 | awk '{ print $4; }')
+GATEWAY=$(/sbin/ip route | awk '/default/ { print $3 }')
+SSH_KEY="$(cat /root/.ssh/authorized_keys | head -n1)"
+sed "s,HOSTNAME,$HOSTNAME,;s,PUBLIC_IP,$PUBLIC_IP,;s,PRIVATE_IP,$PRIVATE_IP,;s,GATEWAY,$GATEWAY,;s,SSH_KEY,$SSH_KEY," /usr/share/oem/cloud-config.yml.template > /usr/share/oem/cloud-config.yml
+
+# download coreos, kernel and create initrd
+/usr/sbin/update-coreos
+
+# run CoreOS kernel
+/sbin/kexec --load /boot/coreos/vmlinuz --initrd /boot/coreos/initrd.cpio.gz
+/sbin/kexec --exec

contrib/digitalocean/update-coreos

@@ -0,0 +1,55 @@
+#!/bin/bash
+
+set -e
+
+COREOS_VERSION=349.0.0
+BASE_URL="http://storage.core-os.net/coreos/amd64-usr/$COREOS_VERSION"
+
+KERNEL="/boot/coreos/vmlinuz"
+INITRD_IMAGE="/boot/coreos/initrd.cpio.gz"
+INITRD_TEMPLATE="/boot/coreos/initrd.template"
+CONFIG_LINK_DIR="/boot/coreos/initrd.template/usr/share/oem"
+
+if [ "$1" = "-d" -o "$1" = "--download" ]; then
+  shift
+  REDOWNLOAD_KERNEL=1
+  REDOWNLOAD_INITRD=1
+fi
+
+mkdir -p "$CONFIG_LINK_DIR"
+
+cp -avx /usr/share/oem/* "$CONFIG_LINK_DIR"
+
+if [ ! -s "$KERNEL" ]; then
+  REDOWNLOAD_KERNEL=1
+fi
+
+if [ "$(find "$INITRD_TEMPLATE" -name "*.squashfs" -type f -size +0 | wc -l)" -eq 0 ]; then
+  REDOWNLOAD_INITRD=1
+fi
+
+if [ -n "$REDOWNLOAD_KERNEL" ]; then
+  curl --retry 10 -f "$BASE_URL/coreos_production_pxe.vmlinuz" > "$KERNEL"
+fi
+
+if [ -n "$REDOWNLOAD_INITRD" ]; then
+  find "$INITRD_TEMPLATE" -name "*.squashfs" -type f -delete
+
+  TMP=`mktemp -d`
+  pushd "$TMP" >/dev/null
+    curl --retry 10 -f "$BASE_URL/coreos_production_pxe_image.cpio.gz" | zcat | cpio -id
+    test ${PIPESTATUS[0]} -eq 0
+    find "$TMP" -name "*.squashfs" -type f -exec 'mv' '{}' "${INITRD_TEMPLATE}/" ';'
+  popd >/dev/null
+  rm -rf "$TMP"
+fi
+
+
+rm -f "$INITRD_IMAGE"
+
+pushd "$INITRD_TEMPLATE" >/dev/null
+  rm -f ./manifest
+  find . -name "*.squashfs" >> ./manifest
+  find ./usr >> ./manifest
+  cat ./manifest | cpio -o -H newc -L | gzip -nc - > "$INITRD_IMAGE"
+popd >/dev/null