Skip to content

Commit b5cb742

Browse files
carmstrongcarmstrong
committed
feat(store): add deis-store component
This is the deis-store component described in deis/deis#1384. It uses Ceph as a backing service, and features a blob store gateway that is both Swift-compatible and S3-compatible. Note that to test this, you will need Docker 1.2.0 and fleet 0.8.0+. TESTING: build, install, and start the stores and the routers ```console $ make -C store build $ make -C router build $ make install-routers install-stores start-routers start-stores ``` If you make changes and want to restart stores: ```console $ make -C store stop build $ make start-stores ``` After this, your cluster should look like the following: ```console $ fleetctl --strict-host-key-checking=false list-units --fields unit,load,active,sub,machine UNIT LOAD ACTIVE SUB MACHINE deis-router@1.service loaded active running 442acc93.../172.17.8.102 deis-store-daemon@1.service loaded active running a852dd23.../172.17.8.100 deis-store-daemon@2.service loaded active running a3b47562.../172.17.8.101 deis-store-daemon@3.service loaded active running 442acc93.../172.17.8.102 deis-store-data@1.service loaded active exited a852dd23.../172.17.8.100 deis-store-data@2.service loaded active exited a3b47562.../172.17.8.101 deis-store-data@3.service loaded active exited 442acc93.../172.17.8.102 deis-store-gateway.service loaded active running a3b47562.../172.17.8.101 deis-store-monitor@1.service loaded active running a852dd23.../172.17.8.100 deis-store-monitor@2.service loaded active running a3b47562.../172.17.8.101 deis-store-monitor@3.service loaded active running 442acc93.../172.17.8.102 ``` Once the services are running, you can enter one of the `deis-store-monitor` or `deis-store-daemon` services and `ceph -s` to see the cluster health. To access the object store endpoint, hit `http://store.local3.deisapp.com`. Your object store requests go through the router! fix(store): try and try again until OSD creation succeeds chore(store/systemd): remove unit files in favor of deisctl chore(store): refactor for new dev environment; bump to Ubuntu 14.04 fix(store): check OSD_ID type rather than return value fix(store/Makefile): fix start/stop, etc. fix(Makefile): add store to main Makefile fix(store): strip newlines from secret key / access key ref(store): use EXTERNAL_PORT like the other components feat(store): add functional tests fix(store): use multiple Docker CLI handles in tests This is necessary because the PrintToStdout function closes the pipes after it matches output. fix(store): finish tests, skipping if we're on boot2docker ref(store): change min_size from 1 to 2 ref(store): no longer use a data volume ref(store): clean up Dockerfiles fix(store): add dev-release and release targets to Makefile fix(store): add deploy target for hacking fix(store): invert docker start pattern, dont remove database ref(store): template ceph.conf ref(store): change min_size back to 1 fix(store): sleep before trying to create OSD ref(store): expose all OSD ports This has no effect since we use `--net host` on the store containers, but it's good to document that OSDs actually use four ports. feat(store): exit if OSD ID isn't valid fix(store): passing component tests, go fmt fix(store): correct log message fix(store): mock ceph for database/registry tests fix(store): remove /var/lib/ceph/osd from copy-on-write fix(store): cleanup osd vfs on container removal fix(store): use portable sed strategy fix(store): change pg_num for default pools
1 parent e05478d commit b5cb742

32 files changed

Lines changed: 930 additions & 20 deletions

File tree

‎Makefile‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,8 @@
44

55
include includes.mk
66

7-
COMPONENTS=builder cache controller database logger logspout publisher registry router
8-
START_ORDER=publisher logger logspout database cache registry controller builder router
7+
COMPONENTS=builder cache controller database logger logspout publisher registry router store
8+
START_ORDER=publisher store logger logspout database cache registry controller builder router
99
CLIENTS=client deisctl
1010

1111
all: build run

‎contrib/docker-registry/user-data‎

Lines changed: 34 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,37 @@
22
---
33
coreos:
44
units:
5-
- name: docker-registry.service
6-
command: start
7-
content: |
8-
[Unit]
9-
Description=Docker Registry server
10-
11-
[Service]
12-
TimeoutStartSec=10min
13-
ExecStartPre=/bin/sh -c "docker history registry:latest >/dev/null || docker pull registry:latest"
14-
ExecStart=/usr/bin/docker run --name docker-registry -e STORAGE_PATH=/registry -e SEARCH_BACKEND=sqlalchemy -p 5000:5000 registry:latest
5+
- name: stop-docker-socket.service
6+
command: start
7+
content: |
8+
[Unit]
9+
Description=stop docker.socket
10+
11+
[Service]
12+
Type=oneshot
13+
ExecStart=/usr/bin/systemctl stop docker.socket
14+
ExecStartPost=/usr/bin/systemctl mask docker.socket
15+
- name: docker-registry.service
16+
command: start
17+
content: |
18+
[Unit]
19+
Description=Docker Registry server
20+
21+
[Service]
22+
TimeoutStartSec=10min
23+
ExecStartPre=/bin/sh -c "docker history registry:latest >/dev/null || docker pull registry:latest"
24+
ExecStart=/usr/bin/docker run --name docker-registry -e STORAGE_PATH=/registry -e SEARCH_BACKEND=sqlalchemy -p 5000:5000 registry:latest
25+
- name: docker-tcp.socket
26+
command: start
27+
enable: true
28+
content: |
29+
[Unit]
30+
Description=Docker Socket for the API
31+
32+
[Socket]
33+
ListenStream=2375
34+
Service=docker.service
35+
BindIPv6Only=both
36+
37+
[Install]
38+
WantedBy=sockets.target

‎database/tests/database_test.go‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,19 +7,28 @@ import (
77

88
"github.com/deis/deis/tests/dockercli"
99
"github.com/deis/deis/tests/etcdutils"
10+
"github.com/deis/deis/tests/mock"
1011
"github.com/deis/deis/tests/utils"
1112
)
1213

1314
func TestDatabase(t *testing.T) {
1415
var err error
1516
tag, etcdPort := utils.BuildTag(), utils.RandomPort()
16-
etcdName := "deis-etcd-" + tag
1717
cli, stdout, stdoutPipe := dockercli.NewClient()
18+
19+
// start etcd container
20+
etcdName := "deis-etcd-" + tag
1821
dockercli.RunTestEtcd(t, etcdName, etcdPort)
1922
defer cli.CmdRm("-f", etcdName)
20-
dataName := "deis-database-data-" + tag
21-
dockercli.RunDeisDataTest(t, "--name", dataName,
22-
"-v", "/var/lib/postgresql", "ubuntu:14.04", "/bin/true")
23+
24+
// run mock ceph containers
25+
cephName := "deis-ceph-" + tag
26+
mock.RunMockCeph(t, cephName, cli, etcdPort)
27+
defer cli.CmdRm("-f", cephName+"-monitor")
28+
defer cli.CmdRm("-f", "-v", cephName+"-daemon")
29+
defer cli.CmdRm("-f", cephName+"-gateway")
30+
31+
// run database container
2332
host, port := utils.HostAddress(), utils.RandomPort()
2433
fmt.Printf("--- Run deis/database:%s at %s:%s\n", tag, host, port)
2534
name := "deis-database-" + tag

‎deisctl/units/deis-database.service‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ EnvironmentFile=/etc/environment
88
TimeoutStartSec=20m
99
ExecStartPre=/bin/sh -c "IMAGE=`/run/deis/bin/get_image /deis/database`; docker history $IMAGE >/dev/null || docker pull $IMAGE"
1010
ExecStartPre=/bin/sh -c "docker inspect deis-database >/dev/null && docker rm -f deis-database || true"
11-
ExecStart=/bin/sh -c "IMAGE=`/run/deis/bin/get_image /deis/database` && docker run --name deis-database --rm -p 5432:5432 -e EXTERNAL_PORT=5432 -e HOST=$COREOS_PRIVATE_IPV4 --volumes-from deis-database-data $IMAGE"
11+
ExecStart=/bin/sh -c "IMAGE=`/run/deis/bin/get_image /deis/database`; docker start -a deis-database >/dev/null || docker run --name deis-database -p 5432:5432 -e EXTERNAL_PORT=5432 -e HOST=$COREOS_PRIVATE_IPV4 $IMAGE"
1212
ExecStop=/bin/bash -c "nsenter --pid --uts --mount --ipc --net --target $(docker inspect --format='{{ .State.Pid }}' deis-database) sudo service postgresql stop"
1313
ExecStopPost=/usr/bin/docker stop deis-database
1414

‎registry/tests/registry_test.go‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ import (
77

88
"github.com/deis/deis/tests/dockercli"
99
"github.com/deis/deis/tests/etcdutils"
10+
"github.com/deis/deis/tests/mock"
1011
"github.com/deis/deis/tests/utils"
1112
)
1213

@@ -26,9 +27,14 @@ func TestRegistry(t *testing.T) {
2627
defer cli.CmdRm("-f", etcdName)
2728
handler := etcdutils.InitEtcd(setdir, setkeys, etcdPort)
2829
etcdutils.PublishEtcd(t, handler)
29-
dataName := "deis-registry-data-" + tag
30-
dockercli.RunDeisDataTest(t, "--name", dataName,
31-
"-v", "/data", "ubuntu:14.04", "/bin/true")
30+
31+
// run mock ceph containers
32+
cephName := "deis-ceph-" + tag
33+
mock.RunMockCeph(t, cephName, cli, etcdPort)
34+
defer cli.CmdRm("-f", cephName+"-monitor")
35+
defer cli.CmdRm("-f", "-v", cephName+"-daemon")
36+
defer cli.CmdRm("-f", cephName+"-gateway")
37+
3238
host, port := utils.HostAddress(), utils.RandomPort()
3339
fmt.Printf("--- Run deis/registry:%s at %s:%s\n", tag, host, port)
3440
name := "deis-registry-" + tag

‎router/conf.d/nginx.conf.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ keys = [
1010
"/deis/domains",
1111
"/deis/controller",
1212
"/deis/builder",
13+
"/deis/store/gateway",
1314
]
1415
check_cmd = "/app/bin/check {{ .src }}"
1516
reload_cmd = "/opt/nginx/sbin/nginx -s reload"

‎router/templates/nginx.conf‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,31 @@ http {
6767
}{{ end }}
6868
## end deis-controller
6969

70+
## start deis-store-gateway
71+
{{ if .deis_store_gateway_host }}
72+
upstream deis-store-gateway {
73+
server {{ .deis_store_gateway_host }}:{{ .deis_store_gateway_port }};
74+
}
75+
76+
server {
77+
server_name ~^store\.(?<domain>.+)$;
78+
server_name_in_redirect off;
79+
port_in_redirect off;
80+
81+
location / {
82+
proxy_buffering off;
83+
proxy_set_header Host $host;
84+
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
85+
proxy_redirect off;
86+
proxy_connect_timeout 10s;
87+
proxy_send_timeout 1200s;
88+
proxy_read_timeout 1200s;
89+
90+
proxy_pass http://deis-store-gateway;
91+
}
92+
}{{ end }}
93+
## end deis-store-gateway
94+
7095
## start service definitions for each application
7196
{{ $domains := .deis_domains }}{{ range $service := .deis_services }}{{ if $service.Nodes }}
7297
upstream {{ Base $service.Key }} {

‎store/LICENSE‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
Copyright 2014 OpDemand LLC
2+
3+
Licensed under the Apache License, Version 2.0 (the "License");
4+
you may not use this file except in compliance with the License.
5+
You may obtain a copy of the License at
6+
7+
http://www.apache.org/licenses/LICENSE-2.0
8+
9+
Unless required by applicable law or agreed to in writing, software
10+
distributed under the License is distributed on an "AS IS" BASIS,
11+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
See the License for the specific language governing permissions and
13+
limitations under the License.

‎store/Makefile‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
include ../includes.mk
2+
3+
TEMPLATE_IMAGES=daemon monitor gateway
4+
BUILT_IMAGES=base $(TEMPLATE_IMAGES)
5+
6+
DAEMON_IMAGE = $(IMAGE_PREFIX)store-daemon:$(BUILD_TAG)
7+
DAEMON_DEV_IMAGE = $(DEV_REGISTRY)/$(DAEMON_IMAGE)
8+
MONITOR_IMAGE = $(IMAGE_PREFIX)store-monitor:$(BUILD_TAG)
9+
MONITOR_DEV_IMAGE = $(DEV_REGISTRY)/$(MONITOR_IMAGE)
10+
GATEWAY_IMAGE = $(IMAGE_PREFIX)store-gateway:$(BUILD_TAG)
11+
GATEWAY_DEV_IMAGE = $(DEV_REGISTRY)/$(GATEWAY_IMAGE)
12+
13+
build: check-docker
14+
@# Build base as normal
15+
docker build -t deis/store-base:$(BUILD_TAG) base/
16+
$(foreach I, $(TEMPLATE_IMAGES), \
17+
sed -e "s/#FROM is generated dynamically by the Makefile/FROM deis\/store-base:${BUILD_TAG}/" $(I)/Dockerfile.template > $(I)/Dockerfile ; \
18+
docker build -t deis/store-$(I):$(BUILD_TAG) $(I)/ ; \
19+
rm $(I)/Dockerfile ; \
20+
)
21+
22+
push: check-docker check-registry check-deisctl
23+
$(foreach I, $(BUILT_IMAGES), \
24+
docker tag deis/store-$(I):$(BUILD_TAG) $(REGISTRY)/deis/store-$(I):$(BUILD_TAG) ; \
25+
docker push $(REGISTRY)/deis/store-$(I):$(BUILD_TAG) ; \
26+
)
27+
28+
$(foreach I, $(TEMPLATE_IMAGES), \
29+
deisctl config store-$(I) set image=$(REGISTRY)/deis/store-$(I):$(BUILD_TAG) ; \
30+
)
31+
32+
clean: check-docker check-registry
33+
$(foreach I, $(BUILT_IMAGES), \
34+
docker rmi deis/store-$(I):$(BUILD_TAG) ; \
35+
docker rmi $(REGISTRY)/deis/store-$(I):$(BUILD_TAG) ; \
36+
)
37+
38+
full-clean: check-docker check-registry
39+
$(foreach I, $(BUILT_IMAGES), \
40+
docker images -q deis/store-$(I) | xargs docker rmi -f ; \
41+
docker images -q $(REGISTRY)/deis/store-$(I) | xargs docker rmi -f ; \
42+
)
43+
44+
install: check-deisctl
45+
deisctl scale store-monitor=3
46+
deisctl scale store-daemon=3
47+
deisctl scale store-gateway=1
48+
49+
uninstall: check-deisctl
50+
deisctl scale store-gateway=0
51+
deisctl scale store-daemon=0
52+
deisctl scale store-monitor=0
53+
54+
start: check-deisctl
55+
deisctl start store-monitor@1
56+
deisctl start store-monitor@2
57+
deisctl start store-monitor@3
58+
deisctl start store-daemon@1
59+
deisctl start store-daemon@2
60+
deisctl start store-daemon@3
61+
deisctl start store-gateway@1
62+
63+
stop: check-deisctl
64+
deisctl stop store-gateway@1
65+
deisctl stop store-daemon@1
66+
deisctl stop store-daemon@2
67+
deisctl stop store-daemon@3
68+
deisctl stop store-monitor@1
69+
deisctl stop store-monitor@2
70+
deisctl stop store-monitor@3
71+
72+
restart: stop start
73+
74+
run: install start
75+
76+
dev-release: check-registry check-deisctl
77+
docker tag $(DAEMON_IMAGE) $(DAEMON_DEV_IMAGE)
78+
docker push $(DAEMON_DEV_IMAGE)
79+
deisctl config store-daemon set image=$(DAEMON_DEV_IMAGE)
80+
docker tag $(MONITOR_IMAGE) $(MONITOR_DEV_IMAGE)
81+
docker push $(MONITOR_DEV_IMAGE)
82+
deisctl config store-monitor set image=$(MONITOR_DEV_IMAGE)
83+
docker tag $(GATEWAY_IMAGE) $(GATEWAY_DEV_IMAGE)
84+
docker push $(GATEWAY_DEV_IMAGE)
85+
deisctl config store-gateway set image=$(GATEWAY_DEV_IMAGE)
86+
87+
release:
88+
docker push $(DAEMON_IMAGE)
89+
docker push $(MONITOR_IMAGE)
90+
docker push $(GATEWAY_IMAGE)
91+
92+
deploy: build dev-release restart
93+
94+
test: test-unit test-functional
95+
96+
test-unit:
97+
@echo no unit tests
98+
99+
test-functional:
100+
@docker history deis/test-etcd >/dev/null 2>&1 || docker pull deis/test-etcd:latest
101+
GOPATH=$(CURDIR)/../tests/_vendor:$(GOPATH) go test -v ./tests/...

‎store/README.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# Deis Store
2+
3+
A backing store built on [Ceph](http://ceph.com) for use in the [Deis](http://deis.io) open
4+
source PaaS.
5+
6+
The bin/boot scripts and Dockerfiles were inspired by
7+
Seán C. McCord's [docker-ceph](https://github.com/Ulexus/docker-ceph) repository.
8+
9+
This Docker image is based on the official
10+
[ubuntu:14.04](https://registry.hub.docker.com/_/ubuntu/) image.
11+
12+
Please add any issues you find with this software to the
13+
[Deis project](https://github.com/deis/deis/issues).
14+
15+
## Containers
16+
17+
The store component is comprised of four containers:
18+
19+
* [store-daemon](https://index.docker.io/u/deis/store-daemon/) - the daemon which serves data
20+
(in Ceph, this is an object store daemon, or OSD)
21+
* [store-monitor](https://index.docker.io/u/deis/store-monitor/) - the service responsible for
22+
keeping track of the cluster state (this is also called a monitor in Ceph)
23+
* [store-gateway](https://index.docker.io/u/deis/store-gateway/) - the blob store gateway,
24+
offering Swift and S3-compatible bucket APIs
25+
* [store-data](systemd/deis-store-data@.service.template) - a container which exposes volumes
26+
to be used by the other containers
27+
28+
These are all based upon the [store-base](https://index.docker.io/u/deis/store-base/) image,
29+
which is a Docker container that preinstalls Ceph.
30+
31+
## Usage
32+
33+
Please consult the [Makefile](Makefile) for current instructions on how to build, test, push,
34+
install, and start **deis/database**.
35+
36+
## License
37+
38+
© 2014 OpDemand LLC
39+
40+
Licensed under the Apache License, Version 2.0 (the "License"); you may
41+
not use this file except in compliance with the License. You may obtain
42+
a copy of the License at <http://www.apache.org/licenses/LICENSE-2.0>
43+
44+
Unless required by applicable law or agreed to in writing, software
45+
distributed under the License is distributed on an "AS IS" BASIS,
46+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
47+
See the License for the specific language governing permissions and
48+
limitations under the License.

0 commit comments

Comments
 (0)