The Atomic Host (CentOS Linux release 7.3.1611 (Core)), the Docker Engine still writes its images to /var/lib/docker even with DeviceMapper as the storage driver.

Perplexing.

We performed a "rm -rf /var/lib/docker/*" and rebuilt the docker thinpool (lvcreate.... etc.) after the Docker Engine complained.

# lvremove cah/docker-pool # lvcreate --wipesignatures y -n docker-pool cah -l 50%VG # lvcreate --wipesignatures y -n docker-poolmeta cah -l 1%VG # lvconvert -y --zero n -c 512K --thinpool cah/docker-pool --poolmetadata cah/docker-poolmeta # lvchange --metadataprofile docker-pool-extend cah/docker-pool # lvchange --metadataprofile cah--docker-pool-extend cah/docker-pool

b132d0cfa1a: Waiting 64afa62fe941: Waiting c45ccadbbd13: Waiting 025d185dba2b: Waiting f4a0ea1a1da2: Waiting write /var/lib/docker/tmp/GetImageBlob531020381: no space left on device

[root@atomic1 docker]# docker info Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 0 Server Version: 1.10.3 Storage Driver: devicemapper Pool Name: cah-docker--pool Pool Blocksize: 524.3 kB Base Device Size: 10.74 GB Backing Filesystem: xfs Data file: Metadata file: Data Space Used: 19.92 MB Data Space Total: 499.4 GB Data Space Available: 499.4 GB Metadata Space Used: 888.8 kB Metadata Space Total: 9.987 GB Metadata Space Available: 9.986 GB Udev Sync Supported: true Deferred Removal Enabled: false Deferred Deletion Enabled: false Deferred Deleted Device Count: 0 Library Version: 1.02.135-RHEL7 (2016-09-28) Execution Driver: native-0.2 Logging Driver: json-file Plugins: Volume: local Network: bridge null host Kernel Version: 3.10.0-514.2.2.el7.x86_64 Operating System: CentOS Linux 7 (Core) OSType: linux Architecture: x86_64 Number of Docker Hooks: 2 CPUs: 8 Total Memory: 31.26 GiB Name: atomic1.eupraxialabs.com ID: CLGC:SBXV:HWBO:ZA6X:BYIS:RU4D:LIII:2GEJ:ZR4K:

As an added measure, we even did an override:

[root@atomic1 docker]# cat /etc/systemd/system/docker.service.d/override.conf [Service] ExecStart= ExecStart=/usr/bin/docker daemon --storage-opt dm.thinpooldev=cah-docker--pool

Same result. The graphDriver is clearly DeviceMapper.

This looks like a bug.

[root@atomic1 docker]# systemctl status -l docker ● docker.service - Docker Application Container Engine Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/docker.service.d └─flannel.conf /etc/systemd/system/docker.service.d └─override.conf Active: active (running) since Wed 2017-02-08 08:36:32 CST; 1h 22min ago Docs: http://docs.docker.com Main PID: 4626 (docker-current) Memory: 21.2M CGroup: /system.slice/docker.service └─4626 /usr/bin/docker-current daemon --storage-opt dm.thinpooldev=cah-docker--pool

Feb 08 08:36:32 atomic1.eupraxialabs.com docker[4626]: time="2017-02-08T08:36:32.271317883-06:00" level=info msg="Firewalld running: false" Feb 08 08:36:32 atomic1.eupraxialabs.com docker[4626]: time="2017-02-08T08:36:32.337846499-06:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address" Feb 08 08:36:32 atomic1.eupraxialabs.com docker[4626]: time="2017-02-08T08:36:32.408356956-06:00" level=info msg="Loading containers: start." Feb 08 08:36:32 atomic1.eupraxialabs.com docker[4626]: time="2017-02-08T08:36:32.408635390-06:00" level=info msg="Loading containers: done." Feb 08 08:36:32 atomic1.eupraxialabs.com docker[4626]: time="2017-02-08T08:36:32.408684424-06:00" level=info msg="Daemon has completed initialization" Feb 08 08:36:32 atomic1.eupraxialabs.com docker[4626]: time="2017-02-08T08:36:32.408740148-06:00" level=info msg="Docker daemon" commit=3999ccb-unsupported execdriver=native-0.2 graphdriver=devicemapper version=1.10.3 Feb 08 08:36:32 atomic1.eupraxialabs.com systemd[1]: Started Docker Application Container Engine. Feb 08 08:36:32 atomic1.eupraxialabs.com docker[4626]: time="2017-02-08T08:36:32.421796182-06:00" level=info msg="API listen on /var/run/docker.sock" Feb 08 08:36:52 atomic1.eupraxialabs.com docker[4626]: time="2017-02-08T08:36:52.352294281-06:00" level=info msg="{Action=create, Username=centos, LoginUID=1001, PID=4701}" Feb 08 08:37:52 atomic1.eupraxialabs.com docker[4626]: time="2017-02-08T08:37:52.430393646-06:00" level=error msg="Download failed: write /var/lib/docker/tmp/GetImageBlob531020381: no space left on device"

This looks like a bug in the Docker Storage Driver.

Is anyone else experiencing a similar issue?

Dave