|
| 1 | +:title: Disk usage |
| 2 | +:description: Understanding disk usage for CoreOS and Deis. |
| 3 | + |
| 4 | +.. _disk_usage: |
| 5 | + |
| 6 | +Disk Usage |
| 7 | +========== |
| 8 | + |
| 9 | +When planning a Deis deployment, it's helpful to understand how both Deis and CoreOS utilize |
| 10 | +local storage on a machine. |
| 11 | + |
| 12 | +The following filesystem paths are the most important to consider: |
| 13 | + |
| 14 | +=================== ============================= ============================================================================================================================================ |
| 15 | +location purpose considerations |
| 16 | +=================== ============================= ============================================================================================================================================ |
| 17 | +/var/lib/etcd etcd snapshot data etcd writes a relatively small amount of snapshot data here, so access should be as fast as possible (cloud providers use fast, local disks) |
| 18 | +/var/lib/docker Docker image/volume storage should be large - on cloud providers with external storage (AWS, GCE, Azure) this is a separate 100GB volume |
| 19 | +/var/lib/deis/store mounted CephFS for deis-store none - this is a virtually-mounted filesystem, and the "real" Ceph data lives in a Docker volume (so it's stored in /var/lib/docker) |
| 20 | +/ everything else (logs, etc.) should be adequately large enough to prevent out-of-space issues causing service failure (on EC2 this is a 50GB volume) |
| 21 | +=================== ============================= ============================================================================================================================================ |
| 22 | + |
| 23 | +Identifying low disk space |
| 24 | +~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 25 | + |
| 26 | +Usually, errors in component logs like "No space left on device" will clearly indicate that a |
| 27 | +low disk space condition is the culprit of operational issues. Upon investigation, ``df -h`` should |
| 28 | +reveal a filesystem with low free disk space. |
| 29 | + |
| 30 | +In some cases, however, the output from ``df -h`` doesn't show any volume having low free space. |
| 31 | +This typically points to a btrfs issue - see `btrfs troubleshooting`_ for more information. |
| 32 | + |
| 33 | +Recovering disk space |
| 34 | +~~~~~~~~~~~~~~~~~~~~~ |
| 35 | + |
| 36 | +If a volume is nearly full, it may be necessary to prune old data from it to ensure the cluster |
| 37 | +remains operational. |
| 38 | + |
| 39 | +The root volume should rarely become full. If it does, explore pruning old log files (or look for |
| 40 | +a forgotten backup or download in the ``core`` user's home directory). |
| 41 | + |
| 42 | +The most alarming low-disk-space condition is when the Docker volume is nearly full. The ``builder`` |
| 43 | +component should remove unnecessary images after a build, and will also remove images after |
| 44 | +an application has been deleted. |
| 45 | + |
| 46 | +However, it some cases it may be necessary to manually prune old images using ``docker rmi``: |
| 47 | + |
| 48 | +.. code-block:: console |
| 49 | +
|
| 50 | + $ docker images -aq | xargs -l10 docker rmi |
| 51 | +
|
| 52 | +.. note:: |
| 53 | + |
| 54 | + This command actually instructs Docker to remove **all** images, and relies on the daemon's |
| 55 | + refusal to remove images which are in-use (errors will be emitted for running images). |
| 56 | + |
| 57 | + |
| 58 | +.. _`btrfs troubleshooting`: https://coreos.com/docs/cluster-management/debugging/btrfs-troubleshooting/ |
0 commit comments