[CentOS-devel] DeviceMapper Storage Drive on CentOS7 Atomic Host

David J. Brewer

davidjbrewer at eupraxialabs.com
Wed Feb 8 16:41:54 UTC 2017


The Atomic Host (CentOS Linux release 7.3.1611 (Core)), the Docker 
Engine still writes its images to /var/lib/docker even with DeviceMapper 
as the storage driver.

Perplexing.

We performed a "rm -rf /var/lib/docker/*" and rebuilt the docker 
thinpool (lvcreate.... etc.) after the Docker Engine complained.

# lvremove cah/docker-pool
# lvcreate --wipesignatures y -n docker-pool cah -l 50%VG
# lvcreate --wipesignatures y -n docker-poolmeta cah -l 1%VG
# lvconvert -y --zero n -c 512K --thinpool cah/docker-pool 
--poolmetadata cah/docker-poolmeta
# lvchange --metadataprofile docker-pool-extend cah/docker-pool
# lvchange --metadataprofile cah--docker-pool-extend cah/docker-pool

b132d0cfa1a: Waiting 64afa62fe941: Waiting c45ccadbbd13: Waiting 
025d185dba2b: Waiting f4a0ea1a1da2: Waiting write 
/var/lib/docker/tmp/GetImageBlob531020381: no space left on device

[root at atomic1 docker]# docker info Containers: 0 Running: 0 Paused: 0 
Stopped: 0 Images: 0 Server Version: 1.10.3 Storage Driver: devicemapper 
Pool Name: cah-docker--pool Pool Blocksize: 524.3 kB Base Device Size: 
10.74 GB Backing Filesystem: xfs Data file: Metadata file: Data Space 
Used: 19.92 MB Data Space Total: 499.4 GB Data Space Available: 499.4 GB 
Metadata Space Used: 888.8 kB Metadata Space Total: 9.987 GB Metadata 
Space Available: 9.986 GB Udev Sync Supported: true Deferred Removal 
Enabled: false Deferred Deletion Enabled: false Deferred Deleted Device 
Count: 0 Library Version: 1.02.135-RHEL7 (2016-09-28) Execution Driver: 
native-0.2 Logging Driver: json-file Plugins: Volume: local Network: 
bridge null host Kernel Version: 3.10.0-514.2.2.el7.x86_64 Operating 
System: CentOS Linux 7 (Core) OSType: linux Architecture: x86_64 Number 
of Docker Hooks: 2 CPUs: 8 Total Memory: 31.26 GiB Name: 
atomic1.eupraxialabs.com ID: CLGC:SBXV:HWBO:ZA6X:BYIS:RU4D:LIII:2GEJ:ZR4K:

As an added measure, we even did an override:

[root at atomic1 docker]# cat 
/etc/systemd/system/docker.service.d/override.conf [Service] ExecStart= 
ExecStart=/usr/bin/docker daemon --storage-opt 
dm.thinpooldev=cah-docker--pool

Same result. The graphDriver is clearly DeviceMapper.

This looks like a bug.

[root at atomic1 docker]# systemctl status -l docker ● docker.service - 
Docker Application Container Engine Loaded: loaded 
(/usr/lib/systemd/system/docker.service; enabled; vendor preset: 
disabled) Drop-In: /usr/lib/systemd/system/docker.service.d 
└─flannel.conf /etc/systemd/system/docker.service.d └─override.conf 
Active: active (running) since Wed 2017-02-08 08:36:32 CST; 1h 22min ago 
Docs: http://docs.docker.com Main PID: 4626 (docker-current) Memory: 
21.2M CGroup: /system.slice/docker.service └─4626 
/usr/bin/docker-current daemon --storage-opt dm.thinpooldev=cah-docker--pool

Feb 08 08:36:32 atomic1.eupraxialabs.com docker[4626]: 
time="2017-02-08T08:36:32.271317883-06:00" level=info msg="Firewalld 
running: false" Feb 08 08:36:32 atomic1.eupraxialabs.com docker[4626]: 
time="2017-02-08T08:36:32.337846499-06:00" level=info msg="Default 
bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon 
option --bip can be used to set a preferred IP address" Feb 08 08:36:32 
atomic1.eupraxialabs.com docker[4626]: 
time="2017-02-08T08:36:32.408356956-06:00" level=info msg="Loading 
containers: start." Feb 08 08:36:32 atomic1.eupraxialabs.com 
docker[4626]: time="2017-02-08T08:36:32.408635390-06:00" level=info 
msg="Loading containers: done." Feb 08 08:36:32 atomic1.eupraxialabs.com 
docker[4626]: time="2017-02-08T08:36:32.408684424-06:00" level=info 
msg="Daemon has completed initialization" Feb 08 08:36:32 
atomic1.eupraxialabs.com docker[4626]: 
time="2017-02-08T08:36:32.408740148-06:00" level=info msg="Docker 
daemon" commit=3999ccb-unsupported execdriver=native-0.2 
graphdriver=devicemapper version=1.10.3 Feb 08 08:36:32 
atomic1.eupraxialabs.com systemd[1]: Started Docker Application 
Container Engine. Feb 08 08:36:32 atomic1.eupraxialabs.com docker[4626]: 
time="2017-02-08T08:36:32.421796182-06:00" level=info msg="API listen on 
/var/run/docker.sock" Feb 08 08:36:52 atomic1.eupraxialabs.com 
docker[4626]: time="2017-02-08T08:36:52.352294281-06:00" level=info 
msg="{Action=create, Username=centos, LoginUID=1001, PID=4701}" Feb 08 
08:37:52 atomic1.eupraxialabs.com docker[4626]: 
time="2017-02-08T08:37:52.430393646-06:00" level=error msg="Download 
failed: write /var/lib/docker/tmp/GetImageBlob531020381: no space left 
on device"

This looks like a bug in the Docker Storage Driver.

Is anyone else experiencing a similar issue?

Dave
-- 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.centos.org/pipermail/centos-devel/attachments/20170208/8629867e/attachment-0003.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: davidjbrewerSignatureBlock_V8.png
Type: image/png
Size: 14991 bytes
Desc: not available
URL: <http://lists.centos.org/pipermail/centos-devel/attachments/20170208/8629867e/attachment-0003.png>


More information about the CentOS-devel mailing list