Docker Images, Containers, and Storage Drivers

To use storage drivers effectively, you must understand how Docker builds and stores images. Then, you need an understanding of how these images are used by containers. Finally, you’ll need a short introduction to the technologies that enable both images and container operations.

Images and layers

Each Docker image references a list of read-only layers that represent filesystem differences. Layers are stacked on top of each other to form a base for a container’s root filesystem. The diagram below shows the Ubuntu 15.04 image comprising 4 stacked image layers.

image

The Docker storage driver is responsible for stacking these layers and providing a single unified view.

When you create a new container, you add a new, thin, writable layer on top of the underlying stack. This layer is often called the “container layer”. All changes made to the running container - such as writing new files, modifying existing files, and deleting files - are written to this thin writable container layer. The diagram below shows a container based on the Ubuntu 15.04 image.

container

Content Addressable Storage

Docker 1.10 introduced a new content addressable storage model. This is a completely new way to address image and layer data on disk. Previously, image and layer data was referenced and stored using a randomly generated UUID. In the new model this is replaced by a secure content hash.

The new model improves security, provides a built-in way to avoid ID collisions, and guarantees data integrity after pull, push, load, and save operations. It also enables better sharing of layers by allowing many images to freely share their layers even if they didn’t come from the same build.

The diagram below shows an updated version of the previous diagram, highlighting the changes implemented by Docker 1.10.

container

As can be seen, all image layer IDs are cryptographic hashes, whereas the container ID is still a randomly generated UUID.

Container and Layers

The major difference between a container and an image is the top writable layer. All writes to the container that add new or modify existing data are stored in this writable layer. When the container is deleted the writable layer is also deleted. The underlying image remains unchanged.

Because each container has its own thin writable container layer, and all changes are stored in this container layer, this means that multiple containers can share access to the same underlying image and yet have their own data state.

The diagram below shows multiple containers sharing the same Ubuntu 15.04 image.

shared image layer

The Docker storage driver is responsible for enabling and managing both the image layers and the writable container layer. How a storage driver accomplishes these can vary between drivers. Two key technologies behind Docker image and container management are stackable image layers and copy-on-write (CoW).

The copy-on-write strategy

Sharing is a good way to optimize resources.

People do this instinctively in daily life. For example, twins Jane and Joseph taking an Algebra class at different times from different teachers can share the same exercise book by passing it between each other. Now, suppose Jane gets an assignment to complete the homework on page 11 in the book. At that point, Jane copies page 11, completes the homework, and hands in her copy. The original exercise book is unchanged and only Jane has a copy of the changed page 11.

Copy-on-write is a similar strategy of sharing and copying. In this strategy, system processes that need the same data share the same instance of that data rather than having their own copy. At some point, if one process needs to modify or write to the data, only then does the operating system make a copy of the data for that process to use. Only the process that needs to write has access to the data copy. All the other processes continue to use the original data.

Docker uses a copy-on-write technology with both images and containers. This CoW strategy optimizes both image disk space usage and the performance of container start times.

The next sections look at how copy-on-write is leveraged with images and containers through sharing and copying.

Sharing promotes smaller images

This section looks at image layers and copy-on-write technology. All image and container layers exist inside the Docker host’s local storage area and are managed by the storage driver. On Linux-based Docker hosts this is usually located under /var/lib/docker/.

The Docker client reports on image layers when instructed to pull and push images with docker pull and docker push.

The command below pulls the ubuntu:15.04 Docker image from Docker Hub.

1
2
3
4
5
6
7
8
9
$ docker pull ubuntu:15.04

15.04: Pulling from library/ubuntu
1ba8ac955b97: Pull complete
f157c4e5ede7: Pull complete
0b7e98f84c4c: Pull complete
a3ed95caeb02: Pull complete
Digest: sha256:5e279a9df07990286cce22e1b0f5b0490629ca6d187698746ae5e28e604a640e
Status: Downloaded newer image for ubuntu:15.04

From the output, you’ll see that the command actually pulls 4 image layers. Each of the above lines lists an image layer and its UUID or cryptographic hash. The combination of these four layers makes up the ubuntu:15.04 Docker image.

Each of these layers is stored in its own directory inside the Docker host’s local storage are.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$  docker pull ubuntu:15.04

15.04: Pulling from library/ubuntu
47984b517ca9: Pull complete
df6e891a3ea9: Pull complete
e65155041eed: Pull complete
c8be1ac8145a: Pull complete
Digest: sha256:5e279a9df07990286cce22e1b0f5b0490629ca6d187698746ae5e28e604a640e
Status: Downloaded newer image for ubuntu:15.04

$ ls /var/lib/docker/aufs/layers

47984b517ca9ca0312aced5c9698753ffa964c2015f2a5f18e5efa9848cf30e2
c8be1ac8145a6e59a55667f573883749ad66eaeef92b4df17e5ea1260e2d7356
df6e891a3ea9cdce2a388a2cf1b1711629557454fd120abd5be6d32329a0e0ac
e65155041eed7ec58dea78d90286048055ca75d41ea893c7246e794389ecf203

Notice how the four directories match up with the layer IDs of the downloaded image.

All versions of Docker still allow images to share layers. For example, If you pull an image that shares some of the same image layers as an image that has already been pulled, the Docker daemon recognizes this, and only pulls the layers it doesn’t already have stored locally. After the second pull, the two images will share any common image layers.

shared image layer

Copying makes containers efficient

You learned earlier that a container is a Docker image with a thin writable, container layer added.

All writes made to a container are stored in the thin writable container layer. The other layers are read-only (RO) image layers and can’t be changed. This means that multiple containers can safely share a single underlying image. The diagram below shows multiple containers sharing a single copy of the ubuntu:15.04 image. Each container has its own thin RW layer, but they all share a single instance of the ubuntu:15.04 image:

When an existing file in a container is modified, Docker uses the storage driver to perform a copy-on-write operation. The specifics of operation depends on the storage driver. For the AUFS and OverlayFS storage drivers, the copy-on-write operation is pretty much as follows:

  • Search through the image layers for the file to update. The process starts at the top, newest layer and works down to the base layer one layer at a time.

  • Perform a “copy-up” operation on the first copy of the file that is found. A “copy up” copies the file up to the container’s own thin writable layer.

  • Modify the copy of the file in container’s thin writable layer.

Containers that write a lot of data will consume more space than containers that do not. This is because most write operations consume new space in the container’s thin writable top layer. If your container needs to write a lot of data, you should consider using a data volume.

A copy-up operation can incur a noticeable performance overhead. This overhead is different depending on which storage driver is in use. However, large files, lots of layers, and deep directory trees can make the impact more noticeable. Fortunately, the operation only occurs the first time any particular file is modified. Subsequent modifications to the same file do not cause a copy-up operation and can operate directly on the file’s existing copy already present in the container layer.

Let’s see what happens if we spin up 5 containers based on our changed-ubuntu image we built earlier:

  1. From a terminal on your Docker host, run the following docker run command 5 times.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    $ docker run -dit changed-ubuntu bash

    75bab0d54f3cf193cfdc3a86483466363f442fba30859f7dcd1b816b6ede82d4

    $ docker run -dit changed-ubuntu bash

    9280e777d109e2eb4b13ab211553516124a3d4d4280a0edfc7abf75c59024d47

    $ docker run -dit changed-ubuntu bash

    a651680bd6c2ef64902e154eeb8a064b85c9abf08ac46f922ad8dfc11bb5cd8a

    $ docker run -dit changed-ubuntu bash

    8eb24b3b2d246f225b24f2fca39625aaad71689c392a7b552b78baf264647373

    $ docker run -dit changed-ubuntu bash

    0ad25d06bdf6fca0dedc38301b2aff7478b3e1ce3d1acd676573bba57cb1cfef
This launches 5 containers based on the changed-ubuntu image. As each container is created, Docker adds a writable layer and assigns it a random UUID. This is the value returned from the docker run command.
  1. Run the docker ps command to verify the 5 containers are running.

    1
    2
    3
    4
    5
    6
    7
    $ docker ps
    CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
    0ad25d06bdf6 changed-ubuntu "bash" About a minute ago Up About a minute stoic_ptolemy
    8eb24b3b2d24 changed-ubuntu "bash" About a minute ago Up About a minute pensive_bartik
    a651680bd6c2 changed-ubuntu "bash" 2 minutes ago Up 2 minutes hopeful_turing
    9280e777d109 changed-ubuntu "bash" 2 minutes ago Up 2 minutes backstabbing_mahavira
    75bab0d54f3c changed-ubuntu "bash" 2 minutes ago Up 2 minutes boring_pasteur
The output above shows 5 running containers, all sharing the changed-ubuntu image. Each CONTAINER ID is derived from the UUID when creating each container.
  1. List the contents of the local storage area.

    1
    2
    3
    4
    5
    6
    7
    $ sudo ls /var/lib/docker/containers

    0ad25d06bdf6fca0dedc38301b2aff7478b3e1ce3d1acd676573bba57cb1cfef
    9280e777d109e2eb4b13ab211553516124a3d4d4280a0edfc7abf75c59024d47
    75bab0d54f3cf193cfdc3a86483466363f442fba30859f7dcd1b816b6ede82d4
    a651680bd6c2ef64902e154eeb8a064b85c9abf08ac46f922ad8dfc11bb5cd8a
    8eb24b3b2d246f225b24f2fca39625aaad71689c392a7b552b78baf264647373
![](https://docs.docker.com/engine/userguide/storagedriver/images/shared-uuid.jpg)

Docker’s copy-on-write strategy not only reduces the amount of space consumed by containers, it also reduces the time required to start a container. At start time, Docker only has to create the thin writable layer for each container. The diagram below shows these 5 containers sharing a single read-only (RO) copy of the changed-ubuntu image.

If Docker had to make an entire copy of the underlying image stack each time it started a new container, container start times and disk space used would be significantly increased.

Data volumes and the storage driver

When a container is deleted, any data written to the container that is not stored in a data volume is deleted along with the container.

A data volume is a directory or file in the Docker host’s filesystem that is mounted directly into a container. Data volumes are not controlled by the storage driver. Reads and writes to data volumes bypass the storage driver and operate at native host speeds. You can mount any number of data volumes into a container. Multiple containers can also share one or more data volumes.

The diagram below shows a single Docker host running two containers. Each container exists inside of its own address space within the Docker host’s local storage area (/var/lib/docker/...). There is also a single shared data volume located at /data on the Docker host. This is mounted directly into both containers.

Data volumes reside outside of the local storage area on the Docker host, further reinforcing their independence from the storage driver’s control. When a container is deleted, any data stored in data volumes persists on the Docker host.


References: