Skip to content

Commit c658148

Browse files
author
Matthew Fisher
committed
Merge pull request #1707 from bacongobbler/import-builder-components
chore(builder): import slug* components
2 parents 35388ce + 8c72985 commit c658148

12 files changed

Lines changed: 468 additions & 0 deletions

File tree

‎builder/slugbuilder/Dockerfile‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
FROM progrium/cedarish
2+
MAINTAINER OpDemand <info@opdemand.com>
3+
4+
RUN useradd slugbuilder --home-dir /app
5+
6+
ADD ./builder/ /tmp/builder
7+
RUN /tmp/builder/install-buildpacks
8+
ENTRYPOINT ["/tmp/builder/build.sh"]

‎builder/slugbuilder/LICENSE‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
Flynn is a trademark of Prime Directive, Inc.
2+
3+
Copyright (c) 2013-2014 Prime Directive, Inc. All rights reserved.
4+
5+
Redistribution and use in source and binary forms, with or without
6+
modification, are permitted provided that the following conditions are
7+
met:
8+
9+
* Redistributions of source code must retain the above copyright
10+
notice, this list of conditions and the following disclaimer.
11+
* Redistributions in binary form must reproduce the above
12+
copyright notice, this list of conditions and the following disclaimer
13+
in the documentation and/or other materials provided with the
14+
distribution.
15+
* Neither the name of Prime Directive, Inc. nor the names of its
16+
contributors may be used to endorse or promote products derived from
17+
this software without specific prior written permission.
18+
19+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
20+
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
21+
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
22+
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
23+
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24+
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
25+
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
26+
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
27+
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
28+
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
29+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

‎builder/slugbuilder/Makefile‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
all:
2+
docker build -t deis/slugbuilder .
3+
4+
.PHONY: clean
5+
clean:
6+
docker rmi -f deis/slugbuilder

‎builder/slugbuilder/README.md‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# (Heroku-ish) Slug Builder
2+
3+
A tool using [Docker](http://docker.io) and
4+
[Buildpacks](https://devcenter.heroku.com/articles/buildpacks) to produce a Heroku-like
5+
[slug](https://devcenter.heroku.com/articles/slug-compiler) given some application source.
6+
7+
## What does it do exactly?
8+
9+
It's a Docker container that takes an uncompressed tarball of an application source piped to it.
10+
The source is run through buildpacks, then if it's detected as a supported app it will be compiled
11+
into a gzipped tarball ready to be run somewhere.
12+
13+
## Using Slug Builder
14+
15+
First, you need Docker. Then you can either pull the image from the public index:
16+
17+
$ docker pull deis/slugbuilder
18+
19+
Or you can build from this source:
20+
21+
$ cd slugbuilder
22+
$ make
23+
24+
When you run the container, it always expects a tar of your app source to be passed via stdin. So
25+
let's run it from a git repo and use `git archive` to produce a tar:
26+
27+
$ id=$(git archive master | docker run -i -a stdin deis/slugbuilder)
28+
$ docker wait $id
29+
$ docker cp $id:/tmp/slug.tgz .
30+
31+
We run slugbuilder, wait for it to finish using the id it gave us, then copies out the slug
32+
artifact into the current directory. If we attached to the container with `docker attach` we could
33+
also see the build output as you would with Heroku. We can also *just* see the build output by
34+
running it with stdout:
35+
36+
$ git archive master | docker run -i -a stdin -a stdout deis/slugbuilder
37+
38+
We still have to look up the id and copy the slug out of the container, but there's an easier way!
39+
40+
$ git archive master | docker run -i -a stdin -a stdout deis/slugbuilder - > myslug.tgz
41+
42+
By running with the `-` argument, it will send all build output to stderr (which we didn't attach
43+
here) and then spit out the slug to stdout, which as you can see we can easily redirect into a
44+
file.
45+
46+
Lastly, you can also have it PUT the slug somewhere via HTTP if you give it a URL as an argument.
47+
This lets us specify a place to put it *and* get the build output via stdout:
48+
49+
$ git archive master | docker run -i -a stdin -a stdout deis/slugbuilder http://fileserver/path/for/myslug.tgz
50+
51+
## Caching
52+
53+
To speed up slug building, it's best to mount a volume specific to your app at `/tmp/cache`. For
54+
example, if you wanted to keep the cache for this app on your host at `/tmp/app-cache`, you'd mount
55+
a read-write volume by running docker with this added `-v /tmp/app-cache:/tmp/cache:rw` option:
56+
57+
docker run -v /tmp/app-cache:/tmp/cache:rw -i -a stdin -a stdout deis/slugbuilder
58+
59+
60+
## Buildpacks
61+
62+
As you can see, slugbuilder supports a number of official and third-party Heroku buildpacks. You
63+
can change the buildpacks.txt file and rebuild the container to create a version that supports
64+
more/less buildpacks than we do here. You can also bind mount your own directory of buildpacks if
65+
you'd like:
66+
67+
docker run -v /my/buildpacks:/tmp/buildpacks:ro -i -a stdin -a stdout deis/slugbuilder
68+
69+
## Base Environment
70+
71+
The Docker image here is based on [cedarish](https://github.com/progrium/cedarish), an image that
72+
emulates the Heroku Cedar stack environment. All buildpacks should have everything they need to run
73+
in this environment, but if something is missing it should be added upstream to cedarish.

‎builder/slugbuilder/builder/build.sh‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
#!/bin/bash
2+
set -eo pipefail
3+
4+
slug_file=/tmp/slug.tgz
5+
app_dir=/app
6+
build_root=/tmp/build
7+
cache_root=/tmp/cache
8+
buildpack_root=/tmp/buildpacks
9+
10+
mkdir -p $app_dir
11+
mkdir -p $cache_root
12+
mkdir -p $buildpack_root
13+
mkdir -p $build_root/.profile.d
14+
15+
function output_redirect() {
16+
if [[ "$slug_file" == "-" ]]; then
17+
cat - 1>&2
18+
else
19+
cat -
20+
fi
21+
}
22+
23+
function echo_title() {
24+
echo $'\e[1G----->' $* | output_redirect
25+
}
26+
27+
function echo_normal() {
28+
echo $'\e[1G ' $* | output_redirect
29+
}
30+
31+
function ensure_indent() {
32+
while read line; do
33+
if [[ "$line" == --* ]]; then
34+
echo $'\e[1G'$line | output_redirect
35+
else
36+
echo $'\e[1G ' "$line" | output_redirect
37+
fi
38+
39+
done
40+
}
41+
42+
## Copy application code over
43+
if [ -d "/tmp/app" ]; then
44+
cp -rf /tmp/app/. $app_dir
45+
else
46+
cat | tar -xmC $app_dir
47+
fi
48+
49+
# In heroku, there are two separate directories, and some
50+
# buildpacks expect that.
51+
cp -r $app_dir/. $build_root
52+
53+
# Grant the slugbuilder user access to all relevant
54+
# directories, then run the rest as slugbuilder
55+
chown -R slugbuilder:slugbuilder $app_dir \
56+
$cache_root \
57+
$buildpack_root \
58+
$build_root
59+
su slugbuilder
60+
61+
## Buildpack fixes
62+
63+
export APP_DIR="$app_dir"
64+
export HOME="$app_dir"
65+
export REQUEST_ID=$(openssl rand -base64 32)
66+
67+
## Buildpack detection
68+
69+
buildpacks=($buildpack_root/*)
70+
selected_buildpack=
71+
72+
if [[ -n "$BUILDPACK_URL" ]]; then
73+
echo_title "Fetching custom buildpack"
74+
75+
buildpack="$buildpack_root/custom"
76+
rm -fr "$buildpack"
77+
git clone --quiet --depth=1 "$BUILDPACK_URL" "$buildpack"
78+
selected_buildpack="$buildpack"
79+
buildpack_name=$($buildpack/bin/detect "$build_root") && selected_buildpack=$buildpack
80+
else
81+
for buildpack in "${buildpacks[@]}"; do
82+
buildpack_name=$($buildpack/bin/detect "$build_root") && selected_buildpack=$buildpack && break
83+
done
84+
fi
85+
86+
if [[ -n "$selected_buildpack" ]]; then
87+
echo_title "$buildpack_name app detected"
88+
else
89+
echo_title "Unable to select a buildpack"
90+
exit 1
91+
fi
92+
93+
## Buildpack compile
94+
95+
$selected_buildpack/bin/compile "$build_root" "$cache_root" | ensure_indent
96+
97+
$selected_buildpack/bin/release "$build_root" "$cache_root" > $build_root/.release
98+
99+
## Display process types
100+
101+
echo_title "Discovering process types"
102+
if [[ -f "$build_root/Procfile" ]]; then
103+
types=$(ruby -e "require 'yaml';puts YAML.load_file('$build_root/Procfile').keys().join(', ')")
104+
echo_normal "Procfile declares types -> $types"
105+
fi
106+
default_types=""
107+
if [[ -s "$build_root/.release" ]]; then
108+
default_types=$(ruby -e "require 'yaml';puts (YAML.load_file('$build_root/.release')['default_process_types'] || {}).keys().join(', ')")
109+
[[ $default_types ]] && echo_normal "Default process types for $buildpack_name -> $default_types"
110+
fi
111+
112+
113+
## Produce slug
114+
115+
if [[ -f "$build_root/.slugignore" ]]; then
116+
tar --exclude='.git' --use-compress-program=pigz -X "$build_root/.slugignore" -C $build_root -cf $slug_file . | cat
117+
else
118+
tar --exclude='.git' --use-compress-program=pigz -C $build_root -cf $slug_file . | cat
119+
fi
120+
121+
if [[ "$slug_file" != "-" ]]; then
122+
slug_size=$(du -Sh "$slug_file" | cut -f1)
123+
echo_title "Compiled slug size is $slug_size"
124+
fi

‎builder/slugbuilder/builder/install-buildpacks‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
#!/usr/bin/env bash
2+
set -eo pipefail
3+
4+
BUILDPACK_INSTALL_PATH="/tmp/buildpacks"
5+
6+
download_buildpack() {
7+
buildpack_url="$1"
8+
buildpack_name=$(basename $buildpack_url)
9+
buildpack_commit="$2"
10+
11+
git clone $buildpack_url $BUILDPACK_INSTALL_PATH/$buildpack_name
12+
pushd $BUILDPACK_INSTALL_PATH/$buildpack_name 2>&1 >/dev/null
13+
git checkout --quiet $buildpack_commit
14+
popd 2>&1 >/dev/null
15+
}
16+
17+
mkdir -p $BUILDPACK_INSTALL_PATH
18+
19+
download_buildpack https://github.com/ddollar/heroku-buildpack-multi.git d5b4b97
20+
download_buildpack https://github.com/heroku/heroku-buildpack-ruby.git v121
21+
download_buildpack https://github.com/heroku/heroku-buildpack-nodejs.git v60
22+
download_buildpack https://github.com/heroku/heroku-buildpack-java.git 586a075
23+
download_buildpack https://github.com/heroku/heroku-buildpack-gradle.git 0d14592
24+
download_buildpack https://github.com/heroku/heroku-buildpack-grails.git 1ef927d
25+
download_buildpack https://github.com/heroku/heroku-buildpack-play.git 420fb6a
26+
download_buildpack https://github.com/heroku/heroku-buildpack-python.git v41
27+
download_buildpack https://github.com/deis/heroku-buildpack-php.git 14ee155
28+
download_buildpack https://github.com/heroku/heroku-buildpack-clojure.git v7
29+
download_buildpack https://github.com/kr/heroku-buildpack-go.git 98109e2
30+
download_buildpack https://github.com/oortcloud/heroku-buildpack-meteorite.git 5fea1e3
31+
download_buildpack https://github.com/miyagawa/heroku-buildpack-perl.git 2da7480
32+
download_buildpack https://github.com/heroku/heroku-buildpack-scala.git 2ece3bd
33+
download_buildpack https://github.com/igrigorik/heroku-buildpack-dart.git c492c52
34+
download_buildpack https://github.com/rhy-jot/buildpack-nginx.git 0433970
35+
download_buildpack https://github.com/Kloadut/heroku-buildpack-static-apache.git f65c835

‎builder/slugrunner/.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
build

‎builder/slugrunner/Dockerfile‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
FROM progrium/cedarish
2+
MAINTAINER Jonathan Rudenberg <jonathan@titanous.com>
3+
4+
ADD ./runner /runner
5+
ENTRYPOINT ["/runner/init"]
6+
7+
# add default port to expose (can be overridden)
8+
ENV PORT 5000
9+
EXPOSE 5000

‎builder/slugrunner/LICENSE‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
Flynn is a trademark of Apollic Software, LLC.
2+
3+
Copyright (c) 2013-2014 Apollic Software, LLC. All rights reserved.
4+
5+
Redistribution and use in source and binary forms, with or without
6+
modification, are permitted provided that the following conditions are
7+
met:
8+
9+
* Redistributions of source code must retain the above copyright
10+
notice, this list of conditions and the following disclaimer.
11+
* Redistributions in binary form must reproduce the above
12+
copyright notice, this list of conditions and the following disclaimer
13+
in the documentation and/or other materials provided with the
14+
distribution.
15+
* Neither the name of Apollic Software, LLC nor the names of its
16+
contributors may be used to endorse or promote products derived from
17+
this software without specific prior written permission.
18+
19+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
20+
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
21+
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
22+
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
23+
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24+
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
25+
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
26+
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
27+
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
28+
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
29+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

‎builder/slugrunner/Makefile‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
build/container: build/sdutil Dockerfile runner/*
2+
docker build -t flynn/slugrunner .
3+
touch build/container
4+
5+
build/sdutil:
6+
mkdir -p tmp
7+
cd tmp && git clone https://github.com/flynn/sdutil.git
8+
GOPATH=. cd tmp/sdutil && godep go build -o ../../build/sdutil
9+
rm -rf tmp
10+
11+
.PHONY: clean
12+
clean:
13+
rm -rf tmp build

0 commit comments

Comments
 (0)