hey guys,
I need to mount a different volume onto /var so we have more room to breathe. I'll be turning 3 servers into an elasticsearch cluster. And for some reason when the servers were ordered the large local volume ended up being /usr when the ES rpm likes to store it's indexes on /var.
So I'm syncing the contents of both directories to a different place, and I'm going swap the large local volume from /usr to /var.
It looked like /opt had more than enough space to hold both directories. /opt was 6GB and I successfully synced /usr to it. /usr was 2.5GB.
Then I went to sync /var to a temp folder in /opt. Checking I see that it still has 1/6GB available after the first sync.
# df -h /opt Filesystem Size Used *Avail* Use% Mounted on /dev/mapper/SysVG-OptVol 6.0G 4.1G *1.6G* 72% /opt
The entire /var partition is only using 549MB:
# df -h /var Filesystem Size *Used* Avail Use% Mounted on /dev/mapper/SysVG-VarVol 6.0G *549M* 5.1G 10% /var
So that being the case, if I make a temp directory in /opt called /opt/var, how come I am running out of space in doing my rsync? It fails at the end and the /opt volume is filled up to 100%. Even tho I only have 549MB to sync.
rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32) rsync: write failed on "/opt/var/log/lastlog": No space left on device (28) rsync error: error in file IO (code 11) at receiver.c(301) [receiver=3.0.6] rsync: recv_generator: mkdir "/opt/var/www/manual/developer" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/opt/var/www/manual/faq" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/opt/var/www/manual/howto" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/opt/var/www/manual/images" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/opt/var/www/manual/misc" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/opt/var/www/manual/mod" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: connection unexpectedly closed (148727 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
And if I do a df of the entire system, it looks like everything is still ok:
# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/SysVG-RootVol 2.0G 872M 1.1G 46% / tmpfs 4.0G 0 4.0G 0% /dev/shm /dev/sda1 486M 87M 375M 19% /boot /dev/mapper/SysVG-HomeVol 4.0G 137M 3.7G 4% /home /dev/mapper/SysVG-OptVol 6.0G 4.3G 1.4G 76% /opt /dev/mapper/SysVG-TmpVol 2.0G 130M 1.8G 7% /tmp /dev/mapper/SysVG-UsrVol 197G 2.8G 185G 2% /usr /dev/mapper/SysVG-VarVol 6.0G 549M 5.1G 10% /var
Does anyone have a good guess as to why these 'out of space' failures are occurring?
Thanks, Tim