hey guys,
I need to mount a different volume onto /var so we have more room to breathe. I'll be turning 3 servers into an elasticsearch cluster. And for some reason when the servers were ordered the large local volume ended up being /usr when the ES rpm likes to store it's indexes on /var.
So I'm syncing the contents of both directories to a different place, and I'm going swap the large local volume from /usr to /var.
It looked like /opt had more than enough space to hold both directories. /opt was 6GB and I successfully synced /usr to it. /usr was 2.5GB.
Then I went to sync /var to a temp folder in /opt. Checking I see that it still has 1/6GB available after the first sync.
# df -h /opt Filesystem Size Used *Avail* Use% Mounted on /dev/mapper/SysVG-OptVol 6.0G 4.1G *1.6G* 72% /opt
The entire /var partition is only using 549MB:
# df -h /var Filesystem Size *Used* Avail Use% Mounted on /dev/mapper/SysVG-VarVol 6.0G *549M* 5.1G 10% /var
So that being the case, if I make a temp directory in /opt called /opt/var, how come I am running out of space in doing my rsync? It fails at the end and the /opt volume is filled up to 100%. Even tho I only have 549MB to sync.
rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32) rsync: write failed on "/opt/var/log/lastlog": No space left on device (28) rsync error: error in file IO (code 11) at receiver.c(301) [receiver=3.0.6] rsync: recv_generator: mkdir "/opt/var/www/manual/developer" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/opt/var/www/manual/faq" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/opt/var/www/manual/howto" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/opt/var/www/manual/images" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/opt/var/www/manual/misc" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/opt/var/www/manual/mod" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: connection unexpectedly closed (148727 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
And if I do a df of the entire system, it looks like everything is still ok:
# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/SysVG-RootVol 2.0G 872M 1.1G 46% / tmpfs 4.0G 0 4.0G 0% /dev/shm /dev/sda1 486M 87M 375M 19% /boot /dev/mapper/SysVG-HomeVol 4.0G 137M 3.7G 4% /home /dev/mapper/SysVG-OptVol 6.0G 4.3G 1.4G 76% /opt /dev/mapper/SysVG-TmpVol 2.0G 130M 1.8G 7% /tmp /dev/mapper/SysVG-UsrVol 197G 2.8G 185G 2% /usr /dev/mapper/SysVG-VarVol 6.0G 549M 5.1G 10% /var
Does anyone have a good guess as to why these 'out of space' failures are occurring?
Thanks, Tim
On Wed, 24 Jun 2015 12:42:19 -0400 Tim Dunphy wrote:
Does anyone have a good guess as to why these 'out of space' failures are occurring?
Hi Tim,
At first glance, I don't see anything obvious, but if it were me, I'd do the following:
a) add the 'n' flag to do a dry run (no actual copying)
b) increase rsync's verbosity (A single -v will give you information about what files are being transferred and a brief summary at the end. Two -v options (-vv) will give you information on what files are being skipped and slightly more information at the end. A third 'v' is insanely verbose.)
c) redirect standard out to a text file that you can examine for more clues.
hth & regards,
Carl
Hey Carl,
Hi Tim,
At first glance, I don't see anything obvious, but if it were me, I'd do the following: a) add the 'n' flag to do a dry run (no actual copying) b) increase rsync's verbosity (A single -v will give you information about what files are being transferred and a brief summary at the end. Two -v options (-vv) will give you information on what files are being skipped and slightly more information at the end. A third 'v' is insanely verbose.) c) redirect standard out to a text file that you can examine for more clues. hth & regards,
Good suggestions! Thanks!
Tim
On Wed, Jun 24, 2015 at 1:05 PM, Carl E. Hartung carlh04426@gmail.com wrote:
On Wed, 24 Jun 2015 12:42:19 -0400 Tim Dunphy wrote:
Does anyone have a good guess as to why these 'out of space' failures are occurring?
Hi Tim,
At first glance, I don't see anything obvious, but if it were me, I'd do the following:
a) add the 'n' flag to do a dry run (no actual copying)
b) increase rsync's verbosity (A single -v will give you information about what files are being transferred and a brief summary at the end. Two -v options (-vv) will give you information on what files are being skipped and slightly more information at the end. A third 'v' is insanely verbose.)
c) redirect standard out to a text file that you can examine for more clues.
hth & regards,
Carl _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Wed, 24 Jun 2015 12:42:19 -0400 Tim Dunphy wrote:
how come I am running out of space in doing my rsync?
Are you running out of room for file and directory names, which is a different thing than simple free disk space?
On 06/24/2015 09:42 AM, Tim Dunphy wrote:
And for some reason when the servers were ordered the large local volume ended up being /usr when the ES rpm likes to store it's indexes on /var.
So I'm syncing the contents of both directories to a different place, and I'm going swap the large local volume from /usr to /var.
Have you considered just resizing the volumes? If you're trying to swap them with rsync, you're going to have to reboot anyway, and relabel your system. If any daemons are running, you might also corrupt their data this way.
The entire /var partition is only using 549MB:
rsync: write failed on "/opt/var/log/lastlog": No space left on device (28)
Depending on what UIDs are allocated to your users, lastlog can be an enormous sparse file. You would need to use rsync's -S flag to copy it.
Gordon Messmer wrote:
On 06/24/2015 09:42 AM, Tim Dunphy wrote:
And for some reason when the servers were ordered the large local volume ended up being /usr when the ES rpm likes to store it's indexes on /var.
So I'm syncing the contents of both directories to a different place, and I'm going swap the large local volume from /usr to /var.
Have you considered just resizing the volumes? If you're trying to swap them with rsync, you're going to have to reboot anyway, and relabel your system. If any daemons are running, you might also corrupt their data this way.
The entire /var partition is only using 549MB:
rsync: write failed on "/opt/var/log/lastlog": No space left on device (28)
Depending on what UIDs are allocated to your users, lastlog can be an enormous sparse file. You would need to use rsync's -S flag to copy it.
Um, I've not been following this closely, but /var is 549M? And a separate partition? I haven't had /var and /usr as separate partitions in almost 10 years. Nor have I had a drive smaller than, um, 160G in about the same.
That being said, why not simply mount the additional partition under var, for the directory that's running out of space?
mark
Have you considered just resizing the volumes?
That'd probably be my preference. But in my role at this company I don't have the direct access to do that. I'd probably have to open up a ticket to another department and have it done when 'they get around to it'. In say 3 or 4 weeks. On my own servers no sweat. But at work. nah. not really practical.
Thanks for the suggestion anyway!
On Wed, Jun 24, 2015 at 2:33 PM, Gordon Messmer gordon.messmer@gmail.com wrote:
On 06/24/2015 09:42 AM, Tim Dunphy wrote:
And for some reason when the servers were ordered the large local volume ended up being /usr when the ES rpm likes to store it's indexes on /var.
So I'm syncing the contents of both directories to a different place, and I'm going swap the large local volume from /usr to /var.
Have you considered just resizing the volumes? If you're trying to swap them with rsync, you're going to have to reboot anyway, and relabel your system. If any daemons are running, you might also corrupt their data this way.
The entire /var partition is only using 549MB:
rsync: write failed on "/opt/var/log/lastlog": No space left on device (28)
Depending on what UIDs are allocated to your users, lastlog can be an enormous sparse file. You would need to use rsync's -S flag to copy it.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Hello Tim,
On 06/24/2015 07:42 PM, Tim Dunphy wrote:
rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32) rsync: write failed on "/opt/var/log/lastlog": No space left on device (28)
lastlog is a VERY large SPARSE file and when you rsync it it looses the sparsity and tries to copy all the data to /opt
ls -al -h /var/log/lastlog -rw-r--r--. 1 root root *94G* Jun 25 09:10 /var/log/lastlog
Real space on disk du -h /var/log/lastlog *60K* /var/log/lastlog
Lec
-
Alexandru Chiscan -
Carl E. Hartung -
Frank Cox -
Gordon Messmer -
m.roth＠5-cent.us -
Tim Dunphy -
Александр Кириллов