feat(healthchecks): add DB health check to /healthz

On boot see if the DB is connectable *before* hitting the migration step and in /healthz connect to the DB and run a simple dumb SELECT statement

Note that this will cause k8s to restart the container after a few tries, if the DB is not available then it will go into a crash loop. This is the nature of the beast:
http://kubernetes.io/v1.1/docs/user-guide/production-pods.html#liveness-and-readiness-probes-aka-health-checks and http://kubernetes.io/v1.1/docs/user-guide/walkthrough/k8s201.html#health-checking

'if the Kubelet discovers a failure the container is restarted.'

To test deploy deis-workflow RC and then scale deis-database to 0

We will have more control over this when #251 lands

Fixes #305

Author: helgi

Makefile

@@ -80,7 +80,10 @@ setup-venv:
 	@if [ ! -d venv ]; then virtualenv venv; fi
 	venv/bin/pip install --disable-pip-version-check -q -r rootfs/requirements.txt -r rootfs/dev_requirements.txt
 
-test: test-style test-unit test-functional
+test: test-style test-check test-unit test-functional
+
+test-check:
+	cd rootfs && python manage.py check
 
 test-style:
 	cd rootfs && flake8 --show-pep8 --show-source

rootfs/api/management/commands/healthchecks.py

@@ -0,0 +1,20 @@
+from django.core.management.base import BaseCommand
+from django.conf import settings
+import django.db
+import sys
+
+
+class Command(BaseCommand):
+    """Management command for healthchecks"""
+    def handle(self, *args, **options):
+        """Ensure DB and other things are alive"""
+        print("Checking if database is alive")
+        try:
+            django.db.connection.cursor()
+            print("Database is alive!")
+        except Exception as e:
+            print("There was a problem connecting to the database")
+            if settings.DEBUG:
+                print(str(e))
+
+            sys.exit(1)

rootfs/api/views.py

@@ -19,14 +19,31 @@
 from api import authentication, models, permissions, serializers, viewsets
 
 import requests
+import logging
+
+logger = logging.getLogger(__name__)
 
 
 class HealthCheckView(View):
-    """Simple health check view to determine if the server
-       is responding to HTTP requests.
+    """
+    Simple health check view to determine if the server
+    is responding to HTTP requests and DB connection / query.
     """
 
     def get(self, request):
+        try:
+            import django.db
+            with django.db.connection.cursor() as c:
+                c.execute("SELECT 0")
+        except django.db.Error as e:
+            logger.critical("Database health check failed")
+            logger.debug(str(e))
+
+            return HttpResponse(
+                "Database health check failed",
+                status=status.HTTP_503_SERVICE_UNAVAILABLE
+            )
+
         return HttpResponse("OK")
     head = get
 

rootfs/bin/boot

@@ -77,7 +77,10 @@ else
 	addgroup deis "$(stat -c "%G" /var/run/docker.sock)"
 fi
 
-# run an idempotent database migration
+echo "Health Checks:"
+./manage.py healthchecks
+
+echo "Database Migrations:"
 sudo -E -u deis ./manage.py migrate --noinput
 
 # spawn a gunicorn server in the background