The Atomic Host (CentOS Linux release 7.3.1611 (Core)), the Docker
Engine still writes its images to /var/lib/docker even with
DeviceMapper as the storage driver.
Perplexing.
We performed a "rm -rf /var/lib/docker/*" and rebuilt the docker
thinpool (lvcreate.... etc.) after the Docker Engine complained.
# lvremove cah/docker-pool
# lvcreate --wipesignatures y -n docker-pool cah -l 50%VG
# lvcreate --wipesignatures y -n docker-poolmeta cah -l 1%VG
# lvconvert -y --zero n -c 512K --thinpool cah/docker-pool
--poolmetadata cah/docker-poolmeta
# lvchange --metadataprofile docker-pool-extend cah/docker-pool
# lvchange --metadataprofile cah--docker-pool-extend
cah/docker-pool
b132d0cfa1a: Waiting 64afa62fe941: Waiting c45ccadbbd13: Waiting
025d185dba2b: Waiting f4a0ea1a1da2: Waiting write
/var/lib/docker/tmp/GetImageBlob531020381: no space left on device
[root@atomic1 docker]# docker info
Containers: 0 Running: 0 Paused: 0 Stopped: 0
Images: 0
Server Version: 1.10.3
Storage Driver: devicemapper Pool Name: cah-docker--pool Pool
Blocksize: 524.3 kB Base Device Size: 10.74 GB Backing Filesystem:
xfs Data file: Metadata file: Data Space Used: 19.92 MB Data Space
Total: 499.4 GB Data Space Available: 499.4 GB Metadata Space
Used: 888.8 kB Metadata Space Total: 9.987 GB Metadata Space
Available: 9.986 GB Udev Sync Supported: true Deferred Removal
Enabled: false Deferred Deletion Enabled: false Deferred Deleted
Device Count: 0 Library Version: 1.02.135-RHEL7 (2016-09-28)
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: Volume: local Network: bridge null host
Kernel Version: 3.10.0-514.2.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 8
Total Memory: 31.26 GiB
Name: atomic1.eupraxialabs.com
ID: CLGC:SBXV:HWBO:ZA6X:BYIS:RU4D:LIII:2GEJ:ZR4K:
As an added measure, we even did an override:
[root@atomic1 docker]# cat
/etc/systemd/system/docker.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon --storage-opt
dm.thinpooldev=cah-docker--pool
Same result. The graphDriver is clearly DeviceMapper.
This looks like a bug.
[root@atomic1 docker]# systemctl status -l docker
● docker.service - Docker Application Container Engine Loaded:
loaded (/usr/lib/systemd/system/docker.service; enabled; vendor
preset: disabled) Drop-In:
/usr/lib/systemd/system/docker.service.d └─flannel.conf
/etc/systemd/system/docker.service.d └─override.conf Active:
active (running) since Wed 2017-02-08 08:36:32 CST; 1h 22min ago
Docs: http://docs.docker.com Main PID: 4626 (docker-current)
Memory: 21.2M CGroup: /system.slice/docker.service └─4626
/usr/bin/docker-current daemon --storage-opt
dm.thinpooldev=cah-docker--pool
Feb 08 08:36:32 atomic1.eupraxialabs.com docker[4626]:
time="2017-02-08T08:36:32.271317883-06:00" level=info
msg="Firewalld running: false"
Feb 08 08:36:32 atomic1.eupraxialabs.com docker[4626]:
time="2017-02-08T08:36:32.337846499-06:00" level=info
msg="Default bridge (docker0) is assigned with an IP address
172.17.0.0/16. Daemon option --bip can be used to set a
preferred IP address"
Feb 08 08:36:32 atomic1.eupraxialabs.com docker[4626]:
time="2017-02-08T08:36:32.408356956-06:00" level=info
msg="Loading containers: start."
Feb 08 08:36:32 atomic1.eupraxialabs.com docker[4626]:
time="2017-02-08T08:36:32.408635390-06:00" level=info
msg="Loading containers: done."
Feb 08 08:36:32 atomic1.eupraxialabs.com docker[4626]:
time="2017-02-08T08:36:32.408684424-06:00" level=info
msg="Daemon has completed initialization"
Feb 08 08:36:32 atomic1.eupraxialabs.com docker[4626]:
time="2017-02-08T08:36:32.408740148-06:00" level=info
msg="Docker daemon" commit=3999ccb-unsupported
execdriver=native-0.2 graphdriver=devicemapper version=1.10.3
Feb 08 08:36:32 atomic1.eupraxialabs.com systemd[1]: Started
Docker Application Container Engine.
Feb 08 08:36:32 atomic1.eupraxialabs.com docker[4626]:
time="2017-02-08T08:36:32.421796182-06:00" level=info msg="API
listen on /var/run/docker.sock"
Feb 08 08:36:52 atomic1.eupraxialabs.com docker[4626]:
time="2017-02-08T08:36:52.352294281-06:00" level=info
msg="{Action=create, Username=centos, LoginUID=1001, PID=4701}"
Feb 08 08:37:52 atomic1.eupraxialabs.com docker[4626]:
time="2017-02-08T08:37:52.430393646-06:00" level=error
msg="Download failed: write
/var/lib/docker/tmp/GetImageBlob531020381: no space left on
device"
This looks like a bug in the Docker Storage Driver.
Is anyone else experiencing a similar issue?
Dave
--
As an added measure, we even did an override:
[root@atomic1 docker]# cat /etc/systemd/system/docker.service.d/override.conf [Service] ExecStart= ExecStart=/usr/bin/docker daemon --storage-opt dm.thinpooldev=cah-docker--pool
Same result. The graphDriver is clearly DeviceMapper.
This looks like a bug.
[root@atomic1 docker]# systemctl status -l docker ● docker.service - Docker Application Container Engine Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/docker.service.d └─flannel.conf /etc/systemd/system/docker.service.d └─override.conf Active: active (running) since Wed 2017-02-08 08:36:32 CST; 1h 22min ago Docs: http://docs.docker.com Main PID: 4626 (docker-current) Memory: 21.2M CGroup: /system.slice/docker.service └─4626 /usr/bin/docker-current daemon --storage-opt dm.thinpooldev=cah-docker--pool
Feb 08 08:36:32 atomic1.eupraxialabs.com docker[4626]: time="2017-02-08T08:36:32.271317883-06:00" level=info msg="Firewalld running: false" Feb 08 08:36:32 atomic1.eupraxialabs.com docker[4626]: time="2017-02-08T08:36:32.337846499-06:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address" Feb 08 08:36:32 atomic1.eupraxialabs.com docker[4626]: time="2017-02-08T08:36:32.408356956-06:00" level=info msg="Loading containers: start." Feb 08 08:36:32 atomic1.eupraxialabs.com docker[4626]: time="2017-02-08T08:36:32.408635390-06:00" level=info msg="Loading containers: done." Feb 08 08:36:32 atomic1.eupraxialabs.com docker[4626]: time="2017-02-08T08:36:32.408684424-06:00" level=info msg="Daemon has completed initialization" Feb 08 08:36:32 atomic1.eupraxialabs.com docker[4626]: time="2017-02-08T08:36:32.408740148-06:00" level=info msg="Docker daemon" commit=3999ccb-unsupported execdriver=native-0.2 graphdriver=devicemapper version=1.10.3 Feb 08 08:36:32 atomic1.eupraxialabs.com systemd[1]: Started Docker Application Container Engine. Feb 08 08:36:32 atomic1.eupraxialabs.com docker[4626]: time="2017-02-08T08:36:32.421796182-06:00" level=info msg="API listen on /var/run/docker.sock" Feb 08 08:36:52 atomic1.eupraxialabs.com docker[4626]: time="2017-02-08T08:36:52.352294281-06:00" level=info msg="{Action=create, Username=centos, LoginUID=1001, PID=4701}" Feb 08 08:37:52 atomic1.eupraxialabs.com docker[4626]: time="2017-02-08T08:37:52.430393646-06:00" level=error msg="Download failed: write /var/lib/docker/tmp/GetImageBlob531020381: no space left on device"