Skip to content

Commit 30228c9

Browse files
author
Matthew Fisher
committed
chore(builder): import deis/slugbuilder
1 parent d7f7118 commit 30228c9

9 files changed

Lines changed: 394 additions & 0 deletions

File tree

‎builder/slugbuilder/CONTRIBUTING.md‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
# How to Contribute
2+
3+
The Deis project is Apache 2.0 licensed and accepts contributions via Github pull
4+
requests. This document outlines some of the conventions on commit message formatting,
5+
contact points for developers and other resources to make getting your contribution
6+
accepted.
7+
8+
# Certificate of Origin
9+
10+
By contributing to this project you agree to the
11+
[Developer Certificate of Origin (DCO)][dco]. This document was created by the Linux
12+
Kernel community and is a simple statement that you, as a contributor, have the legal
13+
right to make the contribution.
14+
15+
# Support Channels
16+
17+
- IRC: #[deis](irc://irc.freenode.org:6667/#deis) IRC channel on freenode.org
18+
19+
## Getting Started
20+
21+
- Fork the repository on GitHub
22+
- Read the README.md for build instructions
23+
24+
## Contribution Flow
25+
26+
This is a rough outline of what a contributor's workflow looks like:
27+
28+
- Create a topic branch from where you want to base your work. This is usually master.
29+
- Make commits of logical units.
30+
- Make sure your commit messages are in the proper format, see below
31+
- Push your changes to a topic branch in your fork of the repository.
32+
- Submit a pull request
33+
34+
Thanks for your contributions!
35+
36+
### Format of the Commit Message
37+
38+
We follow a rough convention for commit messages borrowed from CoreOS, who borrowed theirs
39+
from AngularJS. This is an example of a commit:
40+
41+
```
42+
feat(scripts/test-cluster): add a cluster test command
43+
44+
this uses tmux to setup a test cluster that you can easily kill and
45+
start for debugging.
46+
```
47+
48+
To make it more formal it looks something like this:
49+
50+
```
51+
{type}({scope}): {subject}
52+
<BLANK LINE>
53+
{body}
54+
<BLANK LINE>
55+
{footer}
56+
```
57+
58+
Any line of the commit message cannot be longer than 90 characters, with the subject line
59+
limited to 70 characters. This allows the message to be easier to read on github as well
60+
as in various git tools.
61+
62+
The allowed {types} are as follows:
63+
64+
```
65+
feat -> feature
66+
fix -> bug fix
67+
docs -> documentation
68+
style -> formatting
69+
refactor
70+
test -> adding missing tests
71+
chore -> maintenance
72+
```
73+
74+
### More Details on Commits
75+
76+
For more details see the [commit style guide][style-guide].
77+
78+
[dco]: DCO
79+
[style-guide]: http://docs.deis.io/en/latest/contributing/standards/#commit-style-guide

‎builder/slugbuilder/DCO‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
Developer Certificate of Origin
2+
Version 1.1
3+
4+
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
5+
660 York Street, Suite 102,
6+
San Francisco, CA 94110 USA
7+
8+
Everyone is permitted to copy and distribute verbatim copies of this
9+
license document, but changing it is not allowed.
10+
11+
12+
Developer's Certificate of Origin 1.1
13+
14+
By making a contribution to this project, I certify that:
15+
16+
(a) The contribution was created in whole or in part by me and I
17+
have the right to submit it under the open source license
18+
indicated in the file; or
19+
20+
(b) The contribution is based upon previous work that, to the best
21+
of my knowledge, is covered under an appropriate open source
22+
license and I have the right under that license to submit that
23+
work with modifications, whether created in whole or in part
24+
by me, under the same open source license (unless I am
25+
permitted to submit under a different license), as indicated
26+
in the file; or
27+
28+
(c) The contribution was provided directly to me by some other
29+
person who certified (a), (b) or (c) and I have not modified
30+
it.
31+
32+
(d) I understand and agree that this project and the contribution
33+
are public and that a record of the contribution (including all
34+
personal information I submit with it, including my sign-off) is
35+
maintained indefinitely and may be redistributed consistent with
36+
this project or the open source license(s) involved.

‎builder/slugbuilder/Dockerfile‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
FROM progrium/cedarish
2+
MAINTAINER OpDemand <info@opdemand.com>
3+
4+
RUN useradd slugbuilder --home-dir /app
5+
6+
ADD ./builder/ /tmp/builder
7+
RUN /tmp/builder/install-buildpacks
8+
ENTRYPOINT ["/tmp/builder/build.sh"]

‎builder/slugbuilder/LICENSE‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
Flynn is a trademark of Prime Directive, Inc.
2+
3+
Copyright (c) 2013-2014 Prime Directive, Inc. All rights reserved.
4+
5+
Redistribution and use in source and binary forms, with or without
6+
modification, are permitted provided that the following conditions are
7+
met:
8+
9+
* Redistributions of source code must retain the above copyright
10+
notice, this list of conditions and the following disclaimer.
11+
* Redistributions in binary form must reproduce the above
12+
copyright notice, this list of conditions and the following disclaimer
13+
in the documentation and/or other materials provided with the
14+
distribution.
15+
* Neither the name of Prime Directive, Inc. nor the names of its
16+
contributors may be used to endorse or promote products derived from
17+
this software without specific prior written permission.
18+
19+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
20+
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
21+
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
22+
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
23+
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24+
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
25+
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
26+
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
27+
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
28+
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
29+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

‎builder/slugbuilder/MAINTAINERS‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
Matthew Fisher <matthrewf@opdemand.com> (github: bacongobbler)
2+
Matt Boersma <matt@opdemand.com> (github: mboersma)
3+
Chris Armstrong <chris@opdemand.com> (github: carmstrong)
4+
Gabriel Monroy <gabriel@opdemand.com> (github: gabrtv)

‎builder/slugbuilder/Makefile‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
all:
2+
docker build -t deis/slugbuilder .
3+
4+
.PHONY: clean
5+
clean:
6+
docker rmi -f deis/slugbuilder

‎builder/slugbuilder/README.md‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# (Heroku-ish) Slug Builder
2+
3+
A tool using [Docker](http://docker.io) and
4+
[Buildpacks](https://devcenter.heroku.com/articles/buildpacks) to produce a Heroku-like
5+
[slug](https://devcenter.heroku.com/articles/slug-compiler) given some application source.
6+
7+
## What does it do exactly?
8+
9+
It's a Docker container that takes an uncompressed tarball of an application source piped to it.
10+
The source is run through buildpacks, then if it's detected as a supported app it will be compiled
11+
into a gzipped tarball ready to be run somewhere.
12+
13+
## Using Slug Builder
14+
15+
First, you need Docker. Then you can either pull the image from the public index:
16+
17+
$ docker pull deis/slugbuilder
18+
19+
Or you can build from this source:
20+
21+
$ cd slugbuilder
22+
$ make
23+
24+
When you run the container, it always expects a tar of your app source to be passed via stdin. So
25+
let's run it from a git repo and use `git archive` to produce a tar:
26+
27+
$ id=$(git archive master | docker run -i -a stdin deis/slugbuilder)
28+
$ docker wait $id
29+
$ docker cp $id:/tmp/slug.tgz .
30+
31+
We run slugbuilder, wait for it to finish using the id it gave us, then copies out the slug
32+
artifact into the current directory. If we attached to the container with `docker attach` we could
33+
also see the build output as you would with Heroku. We can also *just* see the build output by
34+
running it with stdout:
35+
36+
$ git archive master | docker run -i -a stdin -a stdout deis/slugbuilder
37+
38+
We still have to look up the id and copy the slug out of the container, but there's an easier way!
39+
40+
$ git archive master | docker run -i -a stdin -a stdout deis/slugbuilder - > myslug.tgz
41+
42+
By running with the `-` argument, it will send all build output to stderr (which we didn't attach
43+
here) and then spit out the slug to stdout, which as you can see we can easily redirect into a
44+
file.
45+
46+
Lastly, you can also have it PUT the slug somewhere via HTTP if you give it a URL as an argument.
47+
This lets us specify a place to put it *and* get the build output via stdout:
48+
49+
$ git archive master | docker run -i -a stdin -a stdout deis/slugbuilder http://fileserver/path/for/myslug.tgz
50+
51+
## Caching
52+
53+
To speed up slug building, it's best to mount a volume specific to your app at `/tmp/cache`. For
54+
example, if you wanted to keep the cache for this app on your host at `/tmp/app-cache`, you'd mount
55+
a read-write volume by running docker with this added `-v /tmp/app-cache:/tmp/cache:rw` option:
56+
57+
docker run -v /tmp/app-cache:/tmp/cache:rw -i -a stdin -a stdout deis/slugbuilder
58+
59+
60+
## Buildpacks
61+
62+
As you can see, slugbuilder supports a number of official and third-party Heroku buildpacks. You
63+
can change the buildpacks.txt file and rebuild the container to create a version that supports
64+
more/less buildpacks than we do here. You can also bind mount your own directory of buildpacks if
65+
you'd like:
66+
67+
docker run -v /my/buildpacks:/tmp/buildpacks:ro -i -a stdin -a stdout deis/slugbuilder
68+
69+
## Base Environment
70+
71+
The Docker image here is based on [cedarish](https://github.com/progrium/cedarish), an image that
72+
emulates the Heroku Cedar stack environment. All buildpacks should have everything they need to run
73+
in this environment, but if something is missing it should be added upstream to cedarish.

‎builder/slugbuilder/builder/build.sh‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
#!/bin/bash
2+
set -eo pipefail
3+
4+
slug_file=/tmp/slug.tgz
5+
app_dir=/app
6+
build_root=/tmp/build
7+
cache_root=/tmp/cache
8+
buildpack_root=/tmp/buildpacks
9+
10+
mkdir -p $app_dir
11+
mkdir -p $cache_root
12+
mkdir -p $buildpack_root
13+
mkdir -p $build_root/.profile.d
14+
15+
function output_redirect() {
16+
if [[ "$slug_file" == "-" ]]; then
17+
cat - 1>&2
18+
else
19+
cat -
20+
fi
21+
}
22+
23+
function echo_title() {
24+
echo $'\e[1G----->' $* | output_redirect
25+
}
26+
27+
function echo_normal() {
28+
echo $'\e[1G ' $* | output_redirect
29+
}
30+
31+
function ensure_indent() {
32+
while read line; do
33+
if [[ "$line" == --* ]]; then
34+
echo $'\e[1G'$line | output_redirect
35+
else
36+
echo $'\e[1G ' "$line" | output_redirect
37+
fi
38+
39+
done
40+
}
41+
42+
## Copy application code over
43+
if [ -d "/tmp/app" ]; then
44+
cp -rf /tmp/app/. $app_dir
45+
else
46+
cat | tar -xmC $app_dir
47+
fi
48+
49+
# In heroku, there are two separate directories, and some
50+
# buildpacks expect that.
51+
cp -r $app_dir/. $build_root
52+
53+
# Grant the slugbuilder user access to all relevant
54+
# directories, then run the rest as slugbuilder
55+
chown -R slugbuilder:slugbuilder $app_dir \
56+
$cache_root \
57+
$buildpack_root \
58+
$build_root
59+
su slugbuilder
60+
61+
## Buildpack fixes
62+
63+
export APP_DIR="$app_dir"
64+
export HOME="$app_dir"
65+
export REQUEST_ID=$(openssl rand -base64 32)
66+
67+
## Buildpack detection
68+
69+
buildpacks=($buildpack_root/*)
70+
selected_buildpack=
71+
72+
if [[ -n "$BUILDPACK_URL" ]]; then
73+
echo_title "Fetching custom buildpack"
74+
75+
buildpack="$buildpack_root/custom"
76+
rm -fr "$buildpack"
77+
git clone --quiet --depth=1 "$BUILDPACK_URL" "$buildpack"
78+
selected_buildpack="$buildpack"
79+
buildpack_name=$($buildpack/bin/detect "$build_root") && selected_buildpack=$buildpack
80+
else
81+
for buildpack in "${buildpacks[@]}"; do
82+
buildpack_name=$($buildpack/bin/detect "$build_root") && selected_buildpack=$buildpack && break
83+
done
84+
fi
85+
86+
if [[ -n "$selected_buildpack" ]]; then
87+
echo_title "$buildpack_name app detected"
88+
else
89+
echo_title "Unable to select a buildpack"
90+
exit 1
91+
fi
92+
93+
## Buildpack compile
94+
95+
$selected_buildpack/bin/compile "$build_root" "$cache_root" | ensure_indent
96+
97+
$selected_buildpack/bin/release "$build_root" "$cache_root" > $build_root/.release
98+
99+
## Display process types
100+
101+
echo_title "Discovering process types"
102+
if [[ -f "$build_root/Procfile" ]]; then
103+
types=$(ruby -e "require 'yaml';puts YAML.load_file('$build_root/Procfile').keys().join(', ')")
104+
echo_normal "Procfile declares types -> $types"
105+
fi
106+
default_types=""
107+
if [[ -s "$build_root/.release" ]]; then
108+
default_types=$(ruby -e "require 'yaml';puts (YAML.load_file('$build_root/.release')['default_process_types'] || {}).keys().join(', ')")
109+
[[ $default_types ]] && echo_normal "Default process types for $buildpack_name -> $default_types"
110+
fi
111+
112+
113+
## Produce slug
114+
115+
if [[ -f "$build_root/.slugignore" ]]; then
116+
tar --exclude='.git' --use-compress-program=pigz -X "$build_root/.slugignore" -C $build_root -cf $slug_file . | cat
117+
else
118+
tar --exclude='.git' --use-compress-program=pigz -C $build_root -cf $slug_file . | cat
119+
fi
120+
121+
if [[ "$slug_file" != "-" ]]; then
122+
slug_size=$(du -Sh "$slug_file" | cut -f1)
123+
echo_title "Compiled slug size is $slug_size"
124+
fi

0 commit comments

Comments
 (0)