Docker Volumes Deep Dive

-
Different --mount Options
This tutorial helps you to understand different mount options, storage drivers, volume drivers(plugins) and troubleshooting issues with Storage Driver(Overlay2).
— Thiru

Volumes are the preferred mechanism to persist container data. bind mounts are dependant on host machine directory structure but volumes are completely managed by docker. more on volumes…​

Tip
 You can use --mount or --volume to create volume , --mount is recommended for newer docker(>17.06).

Different --mount Options

The --mount option allows us to create different kinds of volumes, below is syntax to use --mount option.

mount.sh
docker run --mount
                type=<bind|volume|tmpfs>,
                source=<NameOfVolume/hostDirectory>,
                destination=<NameInsideContainer>,
                readOnly,
                volume-opt k=v,
                volume-opt k=v,
                <docker-image:tag>

  • type : This attribute supports three values, if not specified it is volume.

    1. bind

    2. tmpfs

    3. volume (default)

bind mount

  • bind: Bind mounts have limited functionality compared to volumes. When you use a bind mount, a file or directory on the host machine is mounted into a container.

Caution
 make sure source directory must exists on host machine , before executing bind mounts.
bind.sh
docker run --rm -it --mount type=bind,source="$(pwd)",target=/app nginx bash

# bind mount with readOnly
docker run --rm -it --mount type=bind,source="$(pwd)",target=/app,readOnly nginx bash

tmpfs mount

  • tmpfs: When you create a container with a tmpfs mount, the container can create files outside the container’s writable layer. more…​

tmpfs.sh
# direct tmpfs (standalone containers)
tvajjala$ docker run -it --rm --mount type=tmpfs,destination=/app nginx bash

# using mount we can specify filesystem, size and permission-mode
tvajjala$ docker run --rm -it --mount type=tmpfs,destination=/mycache,tmpfs-size=1.1M,tmpfs-mode=777 nginx bash
root@fa83e775ce51:/# df -kh /mycache/
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           1.2M     0  1.2M   0% /mycache

volume mount

  • volume: Volumes are the preferred mechanism for persisting data generated by and used by Docker containers.

if source volume doesn’t exist while running container it will create it for you.

volume.sh
# using --mount option (recommended)
tvajjala-mac:~ tvajjala$ docker run --rm -it --mount source=thiruVol,destination=/thiruDir nginx bash
root@297e72a94113:/# df -kh thiruDir/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        59G  2.7G   53G   5% /thiruDir

# using -v/--volume option with `ro` option
docker run --rm -it -v myvolume1:/usr/share/nginx/html:ro nginx bash
Note
 you can create named volume before execute above command.
volume-create.sh
docker volume create mySecretVolume

Running docker container same user as host

Below command demonstrates running docker container same user as your host and mount volume(bind type) to see host files inside container.

docker-container-run-as.sh
docker run --rm -it -v $(pwd):/ws --user `id -u`:`id -g` busybox sh

docker run --rm -it -v $(pwd):/ws --user `id -u tvajjala`:`id -g tvajjala` busybox sh

Bind Propagation

for given bind-mount /named volume can be propagated to replicas of that mount. for a mount point /mnt on /tmp directory, the propagation settings controls /tmp/a would be available on /mnt/a.

progrpgation.sh
tvajjala-mac:bindPropgation tvajjala$ docker run --rm -it --mount type=bind,source=$(pwd),target=/bpmnt nginx bash

tvajjala-mac:~ tvajjala$ docker container inspect 55faf5da0652 --format '{{json .Mounts}}' | python -m json.tool
[
    {
        "Destination": "/bpmnt",
        "Mode": "",
        "Propagation": "rprivate", (1)
        "RW": true,
        "Source": "/Users/tvajjala/Documents/Personal/bindPropgation",
        "Type": "bind"
    }
]

  1. default Propagation is rprivate

bind-propagation with private mode

bind.sh
docker run --rm -it --mount type=bind,source="$(pwd)"/src,target=/bpmnt,bind-propagation=private \
                    --mount type=bind,source="$(pwd)"/src,target=/bpmnt2,bind-propagation=private \
                    nginx bash

SELinux label

if you use selinux you can add z or Z option to modify the selinux label of the host file.

  • The z option indicates that the bind mount content is shared among multiple containers.

  • The Z option indicates that bind mount content is private and unshared

selinux.sh
docker run --rm -it -v "$(pwd)"/src:/app:z nginx bash

Consistency Option (MacOS)

on macOS every time write happens on host or through mount in a container, the changes are flushed to disk. latest docker version on macOS introduced new flag for mount

consistency.sh
docker run --rm -it --mount type=bind,source="$(pwd)",destination=/app,consistency=cached nginx bash

  • consistent/default : fully consistent

  • delegated: delay in container changes visible on host

  • cached: delay in host changes visible on container

Directory Type

The overlay storage driver relies on a technology called "directory entry type" (d_type) and is used to describe information of a directory on the filesystem This can cause some issues with chowning and chmoding, deleting and creating files and directories because the overlay storage driver can’t find directory type (d_type) entries in the Linux kernel.

You can check if your existing XFS filesystem has d_type enabled by running the xfs_info. more…​

d_type.sh
[root@docker]# xfs_info /
meta-data=/dev/sda3              isize=256    agcount=4, agsize=2515200 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=0        finobt=0 spinodes=0 rmapbt=0
         =                       reflink=0
data     =                       bsize=4096   blocks=10060800, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1 (1)
log      =internal               bsize=4096   blocks=4912, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

  1. When ftype is 0, d_type support is disabled. When it is 1, d_type support is enabled and you’re safe to use the overlay(2) storage driver with Docker on an XFS filesystem.

enable_d_type.sh
mkfs.xfs -n ftype=1 /mount-point
Note
 it isn’t possible to enable d_type support on an existing filesystem

Container Size on disk

To view approximate size of running container, run below command

size.sh
docker ps -s
CONTAINER   Size
A           1 MB (virtual 10 MB)
B           1 MB (virtual 10 MB)

  • Size: amount of data(on disk) that is used for writable layer of each container.

  • virtual size: amount of data used for the read-only image data used by the container plus size

If A, B containers created from same image, Total Disk space used by containers (A+B) = SUM(Size(A)+ Size(B)) + (Virtual Size of (A or B) - Size(A | B))

Ex: (1 MB + 1 MB ) + (10 MB -1 MB) = 2 MB + 9 MB = 11 MB on disk

nginx.sh
tvajjala-mac:.docker tvajjala$ docker ps -as
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                NAMES               SIZE
becb5a407977        nginx               "/docker-entrypoint.…"   16 minutes ago      Up 16 minutes       0.0.0.0:80->80/tcp   trusting_gould      1.12kB (virtual 133MB)

tvajjala-mac:.docker tvajjala$ docker images
REPOSITORY               TAG                 IMAGE ID            CREATED             SIZE
nginx                    latest              4bb46517cac3        13 days ago         133MB
Note
 nginx image size is equals to virtual size of container.

CoW Strategy

If a file or directory exists in a lower layer within the image, and another layer (including the writable layer) needs read access to it, it just uses the existing file. The first time another layer needs to modify the file (when building the image or running the container), the file is copied into that layer and modified.

Tip
 Docker volumes not accessible directly on MacOS as It’s hidden inside the xhyve virtual machine.
list_volumes.sh
$ docker volume ls

Storage Driver Vs Volume Plugins

Docker uses storage drivers to manage the contents of the image layers and the writable container layer. Each storage driver handles the implementation differently, but all drivers use stackable image layers and the copy-on-write (CoW) strategy. more…​

Storage driver controls how images and containers are stored and managed on your docker host. whereas volume driver helps to manage volume mounts. (i.e to create volume on remote host/cloud encrypt volume data)

Storage Driver

Storage Drivers

  • overlay2 is the preferred (default) storage driver that supports all linux distributions.

  • aufs is the preferred storage driver for docker 18.06 and older.

  • devicemapper is recommended storage driver for CentOS and RHEL , as the old kernal version did not support overlay2. requires direct-lvm for production environments, because loopback-lvm has very poor performance.

  • btrfs/zfs used for the host file system is btrfs/zfs.

  • vfs storage driver intended for testing purposes, performance of this driver is poor not recommended for production.

How to change Storage Driver

You can change storage driver in different ways as shown below. more…​

  • You can pass docker option to dockerd while running docker engine. more…​

dockerd.sh
 dockerd --storage-driver=devicemapper

  • You can pass driver under OPTIONS in /etc/sysconfig/docker file.

/etc/sysconfig/docker
OPTIONS="--storage-driver=devicemapper"

  • Add entry in /etc/docker/daemon.json

/etc/docker/daemon.json
{
  "storage-driver": "devicemapper"
}
Tip
 container writable layers store at /var/lib/docker/<storage-driver>/ location.

Volume Plugins(Drivers)

Volume plugins enable Engine deployments to be integrated with external storage systems such as Amazon EBS, and enable data volumes to persist beyond the lifetime of a single Docker host.

How to change Volume Plugins

if you want to use different volume plugin(driver), you need to install driver plugin as show below. more…​

Tip
 you can explore available volume plugins in docker hub
volume-creation.sh
# install your driver plugin
docker plugin install store/hypergrid/hypercloud:1.5 --alias hypercloud

# create volume with driver
docker volume create -d hypercloud --name vol-100

Troubleshooting

Inspect Volumes

When you run inspect command on container you can find StorageDriver(Overlay2) and Volume Driver(local) and corresponding layers on host

layers.sh
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/333/diff",
                "MergedDir": "/var/lib/docker/overlay2/333/merged",
                "UpperDir": "/var/lib/docker/overlay2/333/diff",
                "WorkDir": "/var/lib/docker/overlay2/333/work"
            },
            "Name": "overlay2" (1)
        },

        "Mounts": [
            {
                "Type": "volume",
                "Name": "9d8aa8ef5c8d26a4491697b1ea8959bd41bb3b083ac342a5a578a82f13f93470",
                "Source": "/var/lib/docker/volumes/9d8aa8ef5c8d26a4491697b1ea8959bd41bb3b083ac342a5a578a82f13f93470/_data",
                "Destination": "/CERTS",
                "Driver": "local", (2)
                "Mode": "",
                "RW": true,
                "Propagation": ""
            }
        ],

  1. is the storage driver name which is specific docker engine(daemon)

  2. is the volume driver(plugin) that is specific to volume mounted to the container.

Changing Volume Default Location

Default location where docker stores these drivers is /var/lib/docker on linux.

  • you change this location in /etc/sysconfig/docker file and restart docker

/etc/sysconfg/docker
# Modify these options if you want to change the way the docker daemon runs
OPTIONS='-g /new/path/docker'

  • You can also change in /lib/systemd/system/docker.service

/lib/systemd/system/docker.service
$ cat /lib/systemd/system/docker.service

[Service]
Type=notify
# for containers run by docker
#ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecStart=/usr/bin/dockerd  -g /new/path/docker -H fd:// --containerd=/run/containerd/containerd.sock

Docker Deamon Flow

Overlay2 Storage Driver

By default, newer versions of Docker will prefer overlay2 if it is available and has the necessary filesystem to support it. Technically overlay2 works on both ext4 and xfs filesystems. Using xfs is preferred over ext4 but the main thing you need to know about your xfs filesystem would be that it must be formatted correctly with ftype=1.

For example, if I had a partition I was going to use for /var/lib/docker named sdb1, I should have formatted it with:

mkfs.sh
mkfs.xfs -n ftype=1 /dev/sdb1
mkfs.ext4 -n ftype=1 /dev/mapper/volgroup01-scratch

Then just mount /dev/sdb1 as you normally would in your /etc/fstab (or whatever syntax tickles your fancy):

UUID=173d6966-4df5-4417-9a55-db9f8cac8cde /var/lib/docker xfs defaults 0 0 I can use a basic daemon.json to set overlay2 as the storage driver:

/etc/docker/daemon.json
{
    "storage-driver": "overlay2"
}
Note
 The biggest difference between overlay2 and devicemapper is that overlay2 uses a formatted and mounted filesystem vs using a block device.

When you run systemctl status docker first loads content /usr/lib/systemd/system/docker.service

then loads drops-In content from /etc/systemd/system/docker.service.d directory

deamon.sh
[root@docker.service.d]# systemctl status docker -l
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/docker.service.d
           └─docker-sysconfig.conf, http-proxy.conf
   Active: active (running) since Thu 2020-08-27 17:47:34 PDT; 6min ago
     Docs: https://docs.docker.com
 Main PID: 21947 (dockerd)
    Tasks: 9
   Memory: 37.2M
   CGroup: /system.slice/docker.service
           └─21947 /usr/bin/dockerd -g /var/lib/docker

level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc000512340, READY" module=grpc
level=info msg="[graphdriver] using prior storage driver: overlay2"
level=info msg="Loading containers: start."
level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
level=info msg="Loading containers: done."
level=warning msg="Not using native diff for overlay2, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled" storage-driver=overlay2
level=info msg="Docker daemon" commit=ead9442 graphdriver(s)=overlay2 version=19.03.1-ol
level=info msg="Daemon has completed initialization"
level=info msg="API listen on /var/run/docker.sock"
systemd[1]: Started Docker Application Container Engine.

Troubleshooting Overlay2

When you run docker on latest linux kernels with Overlay2 storage driver you will see below warning message

Not using native diff for overlay2, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled

To fix this follow these steps

filename.sh
# make directory if not exists
sudo mkdir -p /etc/modprobe.d


#create file with name overlay-redirect-dir.conf

# add below content into it
echo 'options overlay redirect_dir=off' > overlay-redirect-dir.conf

#Unload and reload the overlay module with the command
sudo modprobe -r overlay && sudo modprobe overlay

#reboot system
sudo reboot

devicemapper Storage Driver

Refer section to change devicemapper more…​

/etc/docker/daemon.json
{
    "storage-driver": "devicemapper",
    "storage-opts": [
    "dm.thinpooldev=/dev/mapper/docker-252:0-52953143-pool",
    "dm.use_deferred_removal=true",
    "dm.use_deferred_deletion=true"
    ]
}

  • STEP-1: change storage-driver to devicemapper in /etc/sysconfig/docker or daemon.json

  • STEP-2: docker will create new volume you check with blkid sth like /dev/mapper/docker-252:0-52953143-pool

blkid.sh
[tvajjala]# blkid
/dev/mapper/docker-252:0-52953143-pool: UUID="ffffe1ae-ea67-4912-913b-6bce1b779ed0" TYPE="xfs"

Execution order

You can specify docker runtime parameter in different ways.

Below docker-sysconfig.conf file has execution order defined.

docker-sysconfig.conf
[root@lcm-0005 docker.service.d]# cat /etc/systemd/system/docker.service.d/docker-sysconfig.conf
[Service]
ExecStart=
MountFlags=shared
EnvironmentFile=-/etc/sysconfig/docker
EnvironmentFile=-/etc/sysconfig/docker-storage
EnvironmentFile=-/etc/sysconfig/docker-network
ExecStart=/usr/bin/dockerd \
$OPTIONS \
$DOCKER_STORAGE_OPTIONS \
$DOCKER_NETWORK_OPTIONS \
$INSECURE_REGISTRY

You can create file /etc/sysconfig/docker-storage and specify storage options as show below.

/etc/sysconfig/docker-storage
DOCKER_STORAGE_OPTIONS=-s devicemapper --storage-opt dm.datadev=/dev/my-vg/docker-data --storage-opt dm.metadatadev=/dev/my-vg/docker-metadata

Comments

Post a Comment

Popular posts from this blog

IBM Datapower GatewayScript

-
Introduction IBM API Assembly Language Introduction GatewayScript is a new transformation technology for API, Web, and Mobile and is available in DataPower. GatewayScript is based on the JavaScript programming language. More precisely, it supports the ECMAScript 5.1 version of the specification. On the DataPower appliance, JavaScript runs in ‘strict’ mode only. The strict mode provides a more consistent and more disciplined behavior. For example, if a script attempts to write a property of a DataPower API object and that object is frozen, an immediate exception is thrown. The ‘strict’ mode behavior supports stronger error checking and handling and increases security. DataPower Playground (GatewayScript Fiddle) IBM DataPower Playground is an interactive website that lets you write GatewayScript code and execute it on a cloud-hosted DataPower Gateway for learning purposes. Available at : https://datapower-playground.mybluemix.net/ GatewayScript provides
Read more

Spring boot Kafka Integration

-
Image
Introduction Introduction K afka is an open source streaming platform which scales very well in a horizontal way without compromising speed and efficiency. The Kafka core is written in Scala , and Kafka Streams and KSQL are written in Java . Features Kafka is a service bus To connect heterogeneous applications, we need to implement a message publication mechanism to send and receive messages among them. A message router is known as message broker . Kafka is a message broker , a solution to deal with routing messages among clients in a quick way. Kafka architecture has two directives The first is to not block the producers (to deal with back pressure). The second is to isolate producers and consumers . The producers should not know who their consumers are, hence the kafka follows the dumb broker and smart clients model. Tip Back pressure is to make applications robust against data surges. when producer generates data higher rate tha
Read more

Spring boot SOAP Web Service Performance

-
Image
Consumng This tutorial walks you through the process of consuming a SOAP-based web service with Spring. It elucidates following items Making parallel calls with RxJava (or WebFlux ) Sending large SOAP message payloads using AXIOM( AXis Object Model) AxiomSoapMessageFactory .(which improves performance) Marshalling( is the process of transforming the memory representation of an object to a data format suitable for storage or transmission) Marshalling (XML Serialization) done with O/X Mappers, different types of mappers out there and choose appropriate one for your need. Castor XML mapping is an open source XML binding framework. It allows you to transform the data contained in a java object model into/from an XML document. By default, it does not require any further configuration, though a mapping file can be used to have more control over the behavior of Castor. The Spring integration classes reside in the org.springframework.oxm.castor pa
Read more