Skip to content

Commit 461bbd2

Browse files
carmstrongcarmstrong
committed
Merge pull request #1217 from deis/do
feat(contrib): add DigitalOcean provisioner booting CoreOS via kexec
2 parents 9e341a6 + 5555648 commit 461bbd2

9 files changed

Lines changed: 440 additions & 7 deletions

‎contrib/digitalocean/README.md‎

Lines changed: 105 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,107 @@
1-
Deis on DigitalOcean
2-
====================
1+
# Provision a Deis Cluster on Digital Ocean
32

4-
Unfortunately, DigitalOcean does not yet provide CoreOS images. This
5-
prevents Deis from being deployed on DigitalOcean.
3+
Note that Digital Ocean does not support CoreOS images natively. To work around this, the provision
4+
scripts for Digital Ocean first create a CoreOS image which will be used for provisioning the cluster.
65

7-
If you use DigitalOcean, please
8-
[show your support](http://digitalocean.uservoice.com/forums/136585-digital-ocean/suggestions/4250154-suport-coreos-as-a-deployment-platform)
9-
for CoreOS and help us to support Deis on DO.
6+
Digital Ocean support was contributed by [sttts](https://github.com/sttts). The CoreOS bootstrapping
7+
is heavily based on [Levi Aul's code](https://gist.github.com/tsutsu/490f35f48897df0f5173).
8+
9+
To deploy Deis to Digital Ocean:
10+
11+
## Customize cloud-config.yml
12+
Edit [user-data](../coreos/user-data) and add a discovery URL. This URL will be used by all nodes in this Deis cluster. You can get a new discovery URL by sending a request to http://discovery.etcd.io/new.
13+
14+
## Install tugboat and authorize:
15+
The tugboat gem consumes the Digital Ocean API.
16+
```console
17+
$ gem install tugboat
18+
$ tugboat authorize
19+
```
20+
You can leave all but the client and API keys as the defaults.
21+
22+
## Upload keys
23+
Choose an SSH keypair to use for Deis and import it to Digital Ocean:
24+
```console
25+
$ tugboat add-key deis
26+
```
27+
28+
Then, get the ID of the key:
29+
```console
30+
$ tugboat keys
31+
```
32+
33+
## Create a Deis image:
34+
```console
35+
$ ./provision-digitalocean-deis-image.sh <SSH KEY ID>
36+
```
37+
38+
## Choose number of instances
39+
By default, the script will provision 3 servers. You can override this by setting `DEIS_NUM_INSTANCES`:
40+
```console
41+
$ export DEIS_NUM_INSTANCES=5
42+
```
43+
44+
Note that for scheduling to work properly, clusters must consist of at least 3 nodes and always have an odd number of members.
45+
For more information, see [optimal etcd cluster size](https://github.com/coreos/etcd/blob/master/Documentation/optimal-cluster-size.md).
46+
47+
Deis clusters of less than 3 nodes are unsupported.
48+
49+
## Deploy cluster
50+
Run the provision script:
51+
```console
52+
$ ./provision-do-cluster.sh <REGION_ID> <IMAGE_ID> <SSH_ID> <SIZE>
53+
```
54+
55+
Not all regions allow private networks. Choose one which does (at the time of this writing, NY 2,
56+
Amsterdam 2, Singapore 1 or London 1) - check the web UI for the current private network support.
57+
58+
You can enumerate all the regions with:
59+
60+
```console
61+
$ tugboat regions
62+
```
63+
64+
The provisioning script uses a 512 MB droplet by default because for image creation
65+
more memory is not needed. Deis controller nodes will need at least 2 GB to even start all
66+
the services. Add the memory requirements of deployed applications and choose an adequate
67+
droplet size. The default is 8 GB (ID "65"). You can enumerate all sizes with:
68+
69+
```console
70+
$ tugboat sizes
71+
```
72+
73+
## Choose number of routers
74+
By default, the Makefile will provision 1 router. You can override this by setting `DEIS_NUM_ROUTERS`:
75+
```console
76+
$ export DEIS_NUM_ROUTERS=2
77+
```
78+
79+
## Initialize the cluster
80+
Once the cluster is up, get the IPs of any of the machines using `tugboat droplets`, set
81+
FLEETCTL_TUNNEL to one of these IPs:
82+
```console
83+
$ export FLEETCTL_TUNNEL=23.253.219.94
84+
$ cd ../.. && make run
85+
```
86+
The script will deploy Deis and make sure the services start properly.
87+
88+
### Configure DNS
89+
You'll need to configure DNS records so you can access applications hosted on Deis. See [Configuring DNS](http://docs.deis.io/en/latest/operations/configure-dns/) for details.
90+
91+
### Use Deis!
92+
After that, register with Deis!
93+
```console
94+
$ deis register http://deis.example.org
95+
username: deis
96+
password:
97+
password (confirm):
98+
email: info@opdemand.com
99+
```
100+
101+
## Hack on Deis
102+
If you'd like to use this deployment to build Deis, you'll need to set `DEIS_HOSTS` to an array of your cluster hosts:
103+
```console
104+
$ DEIS_HOSTS="1.2.3.4 2.3.4.5 3.4.5.6" make build
105+
```
106+
107+
This variable is used in the `make build` command.

‎contrib/digitalocean/cloud-config.yml‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
#cloud-config
2+
3+
hostname: HOSTNAME
4+
5+
users:
6+
- name: core
7+
groups:
8+
- sudo
9+
- docker
10+
ssh-authorized-keys:
11+
- SSH_KEY
12+
13+
coreos:
14+
units:
15+
- name: public.network
16+
content: |
17+
[Match]
18+
Name=ens3
19+
20+
[Network]
21+
Address=PUBLIC_IP
22+
Gateway=GATEWAY
23+
DNS=8.8.8.8
24+
DNS=8.8.4.4
25+
- name: private.network
26+
content: |
27+
[Match]
28+
Name=ens4v1
29+
30+
[Network]
31+
Address=PRIVATE_IP
32+
- name: media-doroot.mount
33+
command: start
34+
content: |
35+
[Mount]
36+
What=/dev/vda
37+
Where=/media/doroot
38+
Type=ext4
39+
- name: format-docker-store.service
40+
command: start
41+
content: |
42+
[Unit]
43+
Requires=media-doroot.mount
44+
[Service]
45+
Type=oneshot
46+
RemainAfterExit=yes
47+
ExecStart=/usr/share/oem/bin/create-coreos-docker-store
48+
- name: var-lib-docker.mount
49+
command: start
50+
content: |
51+
[Unit]
52+
Requires=format-docker-store.service
53+
Before=docker.service
54+
[Mount]
55+
What=/dev/disk/by-label/docker
56+
Where=/var/lib/docker
57+
Type=btrfs
58+
- name: coreos-setup-environment.service
59+
command: restart
60+
runtime: yes
61+
content: |
62+
[Unit]
63+
Before=docker.service
64+
[Service]
65+
Type=oneshot
66+
ExecStart=/usr/share/oem/bin/coreos-setup-environment /etc/environment
67+
- name: coreos-apply-user-data.service
68+
command: restart
69+
runtime: yes
70+
content: |
71+
[Unit]
72+
After=coreos-setup-environment.service
73+
[Service]
74+
EnvironmentFile=/etc/environment
75+
Type=oneshot
76+
ExecStart=/usr/share/oem/bin/coreos-apply-user-data

‎contrib/digitalocean/coreos-apply-user-data‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
#!/bin/bash
2+
coreos-cloudinit --from-file=/usr/share/oem/user-data.yml

‎contrib/digitalocean/coreos-setup-environment‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
#!/bin/bash -e
2+
3+
ENV=$1
4+
5+
if [ -z "$ENV" ]; then
6+
echo usage: $0 /etc/environment
7+
exit 1
8+
fi
9+
10+
touch $ENV
11+
if [ $? -ne 0 ]; then
12+
echo exiting, unable to modify: $ENV
13+
exit 1
14+
fi
15+
16+
sed -i -e '/^COREOS_PUBLIC_IPV4=/d;/^COREOS_PRIVATE_IPV4=/d' $ENV
17+
18+
COREOS_PUBLIC_IPV4=$(ip -4 -o addr show dev ens3 | awk '{ print $4; }' | cut -d / -f1)
19+
COREOS_PRIVATE_IPV4=$(ip -4 -o addr show dev ens4v1 | awk '{ print $4; }' | cut -d / -f1)
20+
echo COREOS_PUBLIC_IPV4=$COREOS_PUBLIC_IPV4 >> $ENV
21+
echo COREOS_PRIVATE_IPV4=$COREOS_PRIVATE_IPV4 >> $ENV

‎contrib/digitalocean/create-coreos-docker-store‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
#!/bin/sh
2+
mkdir -p /media/doroot/var/lib/coreos
3+
4+
if [ ! -f "/media/doroot/var/lib/coreos/docker.img" ]; then
5+
DOFORMAT=1
6+
fi
7+
8+
truncate -s 10G /media/doroot/var/lib/coreos/docker.img
9+
LODEV=`losetup -f --show /media/doroot/var/lib/coreos/docker.img`
10+
11+
if [ -n "$DOFORMAT" ]; then
12+
mkfs.btrfs -L docker "$LODEV"
13+
fi

‎contrib/digitalocean/provision-digitalocean-deis-image.sh‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
#!/bin/bash
2+
3+
set -e
4+
5+
THIS_DIR=$(cd $(dirname $0); pwd) # absolute path
6+
CONTRIB_DIR=$(dirname $THIS_DIR)
7+
8+
source $CONTRIB_DIR/utils.sh
9+
10+
if ! which tugboat &>/dev/null; then
11+
echo_red 'Digital Ocean command line client tugboat not found.'
12+
exit 1
13+
fi
14+
15+
function wait_ssh () {
16+
echo -n "Trying to ssh into $1@$2..."
17+
while ! $SSH -o ConnectTimeout=10 $1@$2 hostname &>/dev/null; do
18+
echo -n "."
19+
sleep 1
20+
done
21+
echo_green "done"
22+
}
23+
24+
# parse parameters
25+
if [ -z "$1" ]; then
26+
echo_red 'Usage: $0 SSH_ID [REGION_ID]'
27+
echo
28+
echo 'Use `tugboat keys` to list available SSH_IDs.'
29+
exit 1
30+
fi
31+
32+
SSH_ID="$1"
33+
REGION="${2:-4}" # NYC 2 by default
34+
SIZE="66" # 512 MB
35+
NAME="deis-controller-image-$(date +%Y%m%d%H%M%S)"
36+
37+
BASE_IMAGE='Ubuntu 14.04 x64'
38+
BASE_IMAGE_ID=$(tugboat images -g | grep "$BASE_IMAGE" | sed 's/.*id: \([0-9]*\).*/\1/')
39+
SSH_OPTIONS="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=1 -o ConnectionAttempts=10"
40+
SSH="ssh $SSH_OPTIONS"
41+
SCP="scp $SSH_OPTIONS -q"
42+
43+
# create droplet
44+
tugboat create "$NAME" -i $BASE_IMAGE_ID -p true -k $SSH_ID -s $SIZE -r $REGION
45+
46+
# destroy droplets on error
47+
function cleanup () {
48+
set +e
49+
tugboat destroy -c -n "$NAME"
50+
exit 1
51+
}
52+
trap cleanup ERR
53+
trap cleanup SIGINT
54+
55+
# wait for droplet to come up with ssh login
56+
tugboat wait -n "$NAME" --state active
57+
IP=$(tugboat info -n "$NAME" | egrep '^IP' | awk '{ print $2; }')
58+
wait_ssh root $IP
59+
60+
# bootstrap
61+
echo "Deploying CoreOS on top of $BASE_IMAGE..."
62+
(
63+
set -ex
64+
$SCP update-coreos root@$IP:/usr/sbin
65+
$SSH root@$IP mkdir -p /usr/share/oem/bin
66+
$SCP cloud-config.yml root@$IP:/usr/share/oem/cloud-config.yml.template
67+
$SCP ../coreos/user-data root@$IP:/usr/share/oem/user-data.yml
68+
$SCP create-coreos-docker-store coreos-setup-environment coreos-apply-user-data root@$IP:/usr/share/oem/bin
69+
$SCP rc.local root@$IP:/etc
70+
$SSH root@$IP <<EOF
71+
set -xe
72+
73+
DEBIAN_FRONTEND=noninteractive
74+
apt-get install debconf-utils -y
75+
echo "kexec-tools kexec-tools/load_kexec boolean false" | debconf-set-selections
76+
apt-get install squashfs-tools kexec-tools -y
77+
shutdown -h now
78+
EOF
79+
) 2>&1 | sed 's/^/ /'; test ${PIPESTATUS[0]} -eq 0
80+
81+
# switch off and make snapshot
82+
tugboat wait -n "$NAME" --state off
83+
tugboat snapshot $NAME -n $NAME
84+
85+
# wait for snapshot to finish
86+
echo -n "Waiting for snapsnot to appear..."
87+
while ! tugboat images | grep $NAME &>/dev/null; do
88+
sleep 5
89+
echo -n "."
90+
done
91+
IMAGE_ID=$(tugboat images | grep $NAME | awk '{print $3;}' | sed 's/,//')
92+
echo_green "done"
93+
94+
echo -n "Trying to destroy droplet..."
95+
while ! tugboat destroy -c -n $NAME &>/dev/null; do
96+
sleep 5
97+
echo -n "."
98+
done
99+
echo_green "done"
100+
101+
echo
102+
echo_green "Congratulations: The Deis image $NAME was created with ID $IMAGE_ID."
103+
echo
104+
echo_yellow "Launch the cluster with: ./provision-do-cluster.sh $REGION $IMAGE_ID $SSH_ID 65"

‎contrib/digitalocean/provision-do-cluster.sh‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
#!/usr/bin/env bash
2+
#
3+
# Usage: ./provision-do-cluster.sh <REGION_ID> <IMAGE_ID> <SSH_ID> <SIZE>
4+
#
5+
6+
set -e
7+
8+
THIS_DIR=$(cd $(dirname $0); pwd) # absolute path
9+
CONTRIB_DIR=$(dirname $THIS_DIR)
10+
11+
source $CONTRIB_DIR/utils.sh
12+
13+
# check for DO tools in $PATH
14+
if ! which tugboat > /dev/null; then
15+
echo_red 'Please install the tugboat gem and ensure it is in your $PATH.'
16+
exit 1
17+
fi
18+
19+
if [ -z "$DEIS_NUM_INSTANCES" ]; then
20+
DEIS_NUM_INSTANCES=3
21+
fi
22+
23+
# check that the CoreOS user-data file is valid
24+
$CONTRIB_DIR/util/check-user-data.sh
25+
26+
# launch the Deis cluster on Digital Ocean
27+
i=1 ; while [[ $i -le $DEIS_NUM_INSTANCES ]] ; do \
28+
NAME=deis-$i
29+
echo_yellow "Provisioning ${NAME}..."
30+
tugboat create $NAME -r $1 -i $2 -p true -k $3 -s $4
31+
((i = i + 1)) ; \
32+
done
33+
34+
echo_green "Your Deis cluster has successfully deployed to Digital Ocean."
35+
echo_green "Please continue to follow the instructions in the README."

0 commit comments

Comments
 (0)