Skip to content

Commit 5fdf2da

Browse files
slackslack
authored
doc(storage): cleanup storage docs (#454)
* doc(storage): reorganize object storage configuration * doc(storage): light edits to storage docs
1 parent 8befc2a commit 5fdf2da

1 file changed

Lines changed: 70 additions & 42 deletions

File tree

‎src/installing-workflow/configuring-object-storage.md‎

Lines changed: 70 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -13,47 +13,74 @@ Every component that relies on object storage uses two inputs for configuration:
1313

1414
The helm classic chart for Deis Workflow can be easily configured to connect Workflow components to off-cluster object storage. Deis Workflow currently supports Google Compute Storage, Amazon S3, Azure Blob Storage and OpenStack Swift Storage.
1515

16-
* **Step 1:** Create storage buckets for each of the Workflow subsystems: builder, registry and database
17-
* Note: Depending on your chosen object storage you may need to provide globally unique bucket names.
18-
* Note: If you provide credentials with sufficient access to the underlying storage, Workflow components will create the buckets if they do not exist.
19-
* **Step 2:** If applicable, generate credentials that have write access to the storage buckets created in Step 1
20-
* **Step 3:** If you haven't already fetched the helm classic chart, do so with `helmc fetch deis/workflow-v2.3.0`
21-
* **Step 4:** Update storage details either by setting the appropriate environment variables or by modifying the template file `tpl/generate_params.toml`
22-
* **1.** Using environment variables:
23-
* Set `STORAGE_TYPE` to `s3`, `azure` or `gcs`, then set the following environment variables accordingly.
24-
* For `STORAGE_TYPE=gcs`:
25-
26-
```
27-
export GCS_KEY_JSON, GCS_REGISTRY_BUCKET, GCS_DATABASE_BUCKET, GCS_BUILDER_BUCKET
28-
```
29-
30-
* For `STORAGE_TYPE=s3`:
31-
32-
```
33-
export AWS_ACCESS_KEY, AWS_SECRET_KEY, AWS_REGISTRY_BUCKET, AWS_DATABASE_BUCKET, AWS_BUILDER_BUCKET, S3_REGION
34-
```
35-
* To use [IAM credentials][aws-iam], it is not necessary to set `AWS_ACCESS_KEY` or `AWS_SECRET_KEY`.
36-
37-
* For `STORAGE_TYPE=azure`:
38-
39-
```
40-
export AZURE_ACCOUNT_NAME, AZURE_ACCOUNT_KEY, AZURE_REGISTRY_CONTAINER, AZURE_DATABASE_CONTAINER, AZURE_BUILDER_CONTAINER
41-
```
42-
43-
* For `STORAGE_TYPE=swift`:
44-
45-
```
46-
export SWIFT_USERNAME, SWIFT_PASSWORD, SWIFT_AUTHURL, SWIFT_AUTHVERSION, SWIFT_REGISTRY_CONTAINER, SWIFT_DATABASE_CONTAINER, SWIFT_BUILDER_CONTAINER
47-
```
48-
* To specify tenant set `SWIFT_TENANT` if the auth version is 2 or more.
49-
50-
* **2.** Using template file `tpl/generate_params.toml`:
51-
* Open the helm classic chart with `helmc edit workflow-v2.3.0` and look for the template file `tpl/generate_params.toml`
52-
* Update the `storage` parameter to reference the storage platform you are using: `s3`, `azure`, `gcs`
53-
* Update the values in the section which corresponds to your storage type, including region, bucket names and access credentials
54-
* Note: you do not need to base64 encode any of these values as Helm Classic will handle encoding automatically
55-
* **Step 5:** Save your changes and re-generate the helm classic chart by running `helmc generate -x manifests workflow-v2.3.0`
56-
* **Step 6:** Check the generated file in your manifests directory, you should see `deis-objectstorage-secret.yaml`
16+
### Step 1: Create storage buckets
17+
18+
Create storage buckets for each of the Workflow subsystems: `builder`, `registry`, and `database`.
19+
20+
Depending on your chosen object storage you may need to provide globally unique bucket names.
21+
22+
If you provide credentials with sufficient access to the underlying storage, Workflow components will create the buckets if they do not exist.
23+
24+
### Step 2: Generate storage credentials
25+
26+
If applicable, generate credentials that have create and write access to the storage buckets created in Step 1.
27+
28+
If you are using AWS S3 and your Kubernetes nodes are configured with appropriate IAM API keys via InstanceRoles, you do not need to create API credentials. Do, however, validate that the InstanceRole has appropriate permissions to the configured buckets!
29+
30+
### Step 3: Fetch Workflow charts
31+
32+
If you haven't already fetched the Helm Classic chart, do so with `helmc fetch deis/workflow-v2.3.0`
33+
34+
### Step 4: Configure Workflow charts
35+
36+
Operators should configure object storage by either populating a set of environment variables or editing the the Helm Classic parameters file before running `helmc generate`. Both options are documented below:
37+
38+
**Option 1:** Using environment variables
39+
40+
| Storage Type | Required Variables | Notes |
41+
| --- | --- | --- |
42+
| s3 | `AWS_ACCESS_KEY`, `AWS_SECRET_KEY`, `AWS_REGISTRY_BUCKET`, `AWS_DATABASE_BUCKET`, `AWS_BUILDER_BUCKET`, `S3_REGION` | To use [IAM credentials][aws-iam], it is not necessary to set `AWS_ACCESS_KEY` or `AWS_SECRET_KEY`. |
43+
| gcs | `GCS_KEY_JSON`, `GCS_REGISTRY_BUCKET`, `GCS_DATABASE_BUCKET`, `GCS_BUILDER_BUCKET` | |
44+
| azure | `AZURE_ACCOUNT_NAME`, `AZURE_ACCOUNT_KEY`, `AZURE_REGISTRY_CONTAINER`, `AZURE_DATABASE_CONTAINER`, `AZURE_BUILDER_CONTAINER` | |
45+
| swift | `SWIFT_USERNAME`, `SWIFT_PASSWORD`, `SWIFT_AUTHURL`, `SWIFT_AUTHVERSION`, `SWIFT_REGISTRY_CONTAINER`, `SWIFT_DATABASE_CONTAINER`, `SWIFT_BUILDER_CONTAINER` | To specify tenant set `SWIFT_TENANT` if the auth version is 2 or later. |
46+
47+
!!! note
48+
These environment variables should be set **before** running `helmc generate` in Step 5.
49+
50+
**Option 2:** Using template file `tpl/generate_params.toml`
51+
52+
* Edit Helm Classic chart by running `helmc edit workflow-v2.3.0` and look for the template file `tpl/generate_params.toml`
53+
* Update the `storage` parameter to reference the platform you are using, e.g. `s3`, `azure`, `gcs`, or `swift`
54+
* Find the corresponding section for your storage type and provide appropriate values including region, bucket names, and access credentials.
55+
* Save your changes to `tpl/generate_params.toml`.
56+
57+
!!! note
58+
You do not need to base64 encode any of these values as Helm Classic will handle encoding automatically.
59+
60+
### Step 5: Generate manifests
61+
62+
Generate the Workflow chart by running `helmc generate -x manifests workflow-v2.3.0`.
63+
64+
### Step 6: Verify credentials
65+
66+
Helm Classic stores the object storage configuration as a Kubernetes secret.
67+
68+
You may check the contents of the generated file named `deis-objectstorage-secret.yaml` in the `helmc` workspace directory:
69+
```
70+
$ cat $(helmc home)/workspace/charts/workflow-v2.3.0/manifests/deis-objectstorage-secret.yaml
71+
apiVersion: v1
72+
kind: Secret
73+
metadata:
74+
name: objectstorage-keyfile
75+
...
76+
data:
77+
accesskey: bm9wZSBub3BlCg==
78+
secretkey: c3VwZXIgbm9wZSBub3BlIG5vcGUgbm9wZSBub3BlCg==
79+
region: ZWFyZgo=
80+
registry-bucket: bXlmYW5jeS1yZWdpc3RyeS1idWNrZXQK
81+
database-bucket: bXlmYW5jeS1kYXRhYmFzZS1idWNrZXQK
82+
builder-bucket: bXlmYW5jeS1idWlsZGVyLWJ1c2tldAo=
83+
```
5784

5885
You are now ready to `helmc install workflow-v2.3.0` using your desired object storage.
5986

@@ -90,7 +117,8 @@ If you are using slugbuilder as a standalone component the following configurati
90117
- `TAR_PATH` - The location of the application `.tar` archive, relative to the configured bucket for builder e.g. `home/burley-yeomanry:git-3865c987/tar`
91118
- `PUT_PATH` - The location to upload the finished slug, relative to the configured bucket fof builder e.g. `home/burley-yeomanry:git-3865c987/push`
92119

93-
**Note: these environment variables are case-sensitive**
120+
!!! note
121+
These environment variables are case-sensitive.
94122

95123
### [deis/slugrunner](https://github.com/deis/slugrunner)
96124

0 commit comments

Comments
 (0)