feat(platform): support placement options for each plane and router mesh

This updates deisctl code that creates and loads system units such that
platform components belonging to each of the control plane, data plane, and
router mesh can OPTIONALLY be scheduled only where fleet machine metadata
indicates.

The means of achieving this are that a "decorator" is applied to any system
unit just before scheduling if and only if this option is enabled by means
of the etcd key /deis/platform/enablePlacementOptions having been set to the
value true.  The decorator adds the correct metadata.

Nodes in existing clusters will lack the machine metadata referenced by the
system unit decorators.  That would make this a breaking change except for
the fact that this is entirely OPT-IN.

Users should be encouraged to begin adopting metadata that will permit them
to isolate their planes more easily in the future.  As such, user-data.example
is modified such that machines configured from that cloud-config are eligible
to host both planes and the router mesh.  This permits NEW small clusters
without any isolation to be constructed as easily as ever before and permits
the platform to be installed to such a cluster without complication.

The main benefit of this change is that advanced users can use custom cloud
config to isolate their planes using ONLY metadata and without also needing
to also manually customize their system units, as would have been required
in the past.

Author: krancour

scheduler/fleet.py

@@ -150,10 +150,10 @@ def _create_container(self, name, image, command, unit, **kwargs):
             f['value'] = f['value'].format(**l)
         # prepare tags only if one was provided
         tags = kwargs.get('tags', {})
-        if tags:
-            tagset = ' '.join(['"{}={}"'.format(k, v) for k, v in tags.viewitems()])
+        tagset = ' '.join(['"{}={}"'.format(k, v) for k, v in tags.viewitems()])
+        if settings.ENABLE_PLACEMENT_OPTIONS in ['true', 'True', 'TRUE', '1']:
             unit.append({"section": "X-Fleet", "name": "MachineMetadata",
-                         "value": tagset})
+                         "value": tagset + ' "dataPlane=true"'})
         # post unit to fleet
         self._put_unit(name, {"desiredState": "loaded", "options": unit})
 

templates/confd_settings.py

@@ -24,6 +24,8 @@
 # platform domain must be provided
 DEIS_DOMAIN = '{{ getv "/deis/platform/domain" }}'
 
+ENABLE_PLACEMENT_OPTIONS = """{{ if exists "/deis/platform/enablePlacementOptions" }}{{ getv "/deis/platform/enablePlacementOptions" }}{{ else }}false{{end}}"""
+
 # use the private registry module
 REGISTRY_MODULE = 'registry.private'
 REGISTRY_URL = '{{ getv "/deis/registry/protocol" }}://{{ getv "/deis/registry/host" }}:{{ getv "/deis/registry/port" }}'  # noqa