I have an ext4 filesystem for which the reported disk usage is not correct. I have noticed the discrepancy after I rsync-ed the content to another filesystem and noticed that the used space on the target is almost double of the size reported on the source. Both machines are running the same software - with the same kernel version and same coreutils version (which I later upgraded to latest available version). Both filesystems are clean (verified with fsck.ext4). No sparse files. After further investigation I think that the problem is most likely on the source machine. Here is the du output for for one directory exhibiting the problem:
#du -h |grep /51 201M ./51/msg/8 567M ./51/msg/9 237M ./51/msg/6 279M ./51/msg/0 174M ./51/msg/10 273M ./51/msg/2 341M ./51/msg/7 408M ./51/msg/4 222M ./51/msg/11 174M ./51/msg/5 238M ./51/msg/1 271M ./51/msg/3 3.3G ./51/msg 3.3G ./51
after changing the directory and running du again I get different numbers
#cd 51 du -h 306M ./msg/8 676M ./msg/9 351M ./msg/6 338M ./msg/0 347M ./msg/10 394M ./msg/2 480M ./msg/7 544M ./msg/4 407M ./msg/11 312M ./msg/5 326M ./msg/1 377M ./msg/3 4.8G ./msg 4.8G .
Do you have any idea what could cause this behaviour?
On Wed, Mar 19, 2014 at 8:14 AM, Radu Radutiu rradutiu@gmail.com wrote:
I have an ext4 filesystem for which the reported disk usage is not correct. I have noticed the discrepancy after I rsync-ed the content to another filesystem and noticed that the used space on the target is almost double of the size reported on the source. Both machines are running the same software - with the same kernel version and same coreutils version (which I later upgraded to latest available version). Both filesystems are clean (verified with fsck.ext4). No sparse files. After further investigation I think that the problem is most likely on the source machine. Here is the du output for for one directory exhibiting the problem:
#du -h |grep /51 201M ./51/msg/8 567M ./51/msg/9 237M ./51/msg/6 279M ./51/msg/0 174M ./51/msg/10 273M ./51/msg/2 341M ./51/msg/7 408M ./51/msg/4 222M ./51/msg/11 174M ./51/msg/5 238M ./51/msg/1 271M ./51/msg/3 3.3G ./51/msg 3.3G ./51
after changing the directory and running du again I get different numbers
#cd 51 du -h 306M ./msg/8 676M ./msg/9 351M ./msg/6 338M ./msg/0 347M ./msg/10 394M ./msg/2 480M ./msg/7 544M ./msg/4 407M ./msg/11 312M ./msg/5 326M ./msg/1 377M ./msg/3 4.8G ./msg 4.8G .
Do you have any idea what could cause this behaviour? _______________________________________________
so you have software creating file on machines A & B, synchronized from A to B and now B is using 2x the space; was the software running on B when you did the sync? I've seen similar things happen on all unix systems when you don't close out the file handles on running programs but then overwrite their opened files. to fix it you have to have make the programs close and re-open their files. with well written programs you can do that via a signal or some other trigger mechanism, others will need to be restarted. often it's easier to just schedule a reboot and restart everything rather than wade through all the individual process shutdowns, restarts and time that you'll take affecting production processes, but YMMV.
No process is reading or writing to the target filesytem (it is a backup machine) or the source machine (I am working on a LVM snapshot but the problem exists for the source filesytem as well). The problem I describe is on the same machine (the source).
On Wed, Mar 19, 2014 at 2:33 PM, zGreenfelder zgreenfelder@gmail.comwrote:
On Wed, Mar 19, 2014 at 8:14 AM, Radu Radutiu rradutiu@gmail.com wrote:
I have an ext4 filesystem for which the reported disk usage is not correct. I have noticed the discrepancy after I rsync-ed the content to another filesystem and noticed that the used space on the target is
almost
double of the size reported on the source. Both machines are running the same software - with the same kernel
version
and same coreutils version (which I later upgraded to latest available version). Both filesystems are clean (verified with fsck.ext4). No sparse files. After further investigation I think that the problem is most likely on
the
source machine. Here is the du output for for one directory exhibiting the problem:
#du -h |grep /51 201M ./51/msg/8 567M ./51/msg/9 237M ./51/msg/6 279M ./51/msg/0 174M ./51/msg/10 273M ./51/msg/2 341M ./51/msg/7 408M ./51/msg/4 222M ./51/msg/11 174M ./51/msg/5 238M ./51/msg/1 271M ./51/msg/3 3.3G ./51/msg 3.3G ./51
after changing the directory and running du again I get different numbers
#cd 51 du -h 306M ./msg/8 676M ./msg/9 351M ./msg/6 338M ./msg/0 347M ./msg/10 394M ./msg/2 480M ./msg/7 544M ./msg/4 407M ./msg/11 312M ./msg/5 326M ./msg/1 377M ./msg/3 4.8G ./msg 4.8G .
Do you have any idea what could cause this behaviour? _______________________________________________
so you have software creating file on machines A & B, synchronized from A to B and now B is using 2x the space; was the software running on B when you did the sync? I've seen similar things happen on all unix systems when you don't close out the file handles on running programs but then overwrite their opened files. to fix it you have to have make the programs close and re-open their files. with well written programs you can do that via a signal or some other trigger mechanism, others will need to be restarted. often it's easier to just schedule a reboot and restart everything rather than wade through all the individual process shutdowns, restarts and time that you'll take affecting production processes, but YMMV.
-- Even the Magic 8 ball has an opinion on email clients: Outlook not so good. _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Wed, 19 Mar 2014 14:14:04 +0200 Radu Radutiu rradutiu@gmail.com wrote:
Do you have any idea what could cause this behaviour?
http://mradomski.wordpress.com/2007/01/08/finding-an-unlinked-open-file-and-...
http://mradomski.wordpress.com/2007/01/08/finding-an-unlinked-open-file-and-...
There are no open files. The filesystem was unmounted, verified (fsck) , mounted again - the behavior remains.
On 03/19/2014 07:14 AM, Radu Radutiu wrote:
I have an ext4 filesystem for which the reported disk usage is not correct.
...
Here is the du output for for one directory exhibiting the problem:
#du -h |grep /51 201M ./51/msg/8 567M ./51/msg/9 237M ./51/msg/6 279M ./51/msg/0 174M ./51/msg/10 273M ./51/msg/2 341M ./51/msg/7 408M ./51/msg/4 222M ./51/msg/11 174M ./51/msg/5 238M ./51/msg/1 271M ./51/msg/3 3.3G ./51/msg 3.3G ./51
after changing the directory and running du again I get different numbers
#cd 51 du -h 306M ./msg/8 676M ./msg/9 351M ./msg/6 338M ./msg/0 347M ./msg/10 394M ./msg/2 480M ./msg/7 544M ./msg/4 407M ./msg/11 312M ./msg/5 326M ./msg/1 377M ./msg/3 4.8G ./msg 4.8G .
The space used by hard-linked files will be included only in the first directory where they are encountered. In your first case, linked files seen prior to the /51 directory would not have had their space included again under that directory. In the second case, _only_ the /51 directory is being examined, so all space will be included.
The space used by hard-linked files will be included only in the first
directory where they are encountered. In your first case, linked files seen prior to the /51 directory would not have had their space included again under that directory. In the second case, _only_ the /51 directory is being examined, so all space will be included.
-- Bob Nichols "NOSPAM" is really part of my email address. Do NOT delete it.
Thank you, this makes sense. I was starting to worry that the filesystem is broken. I'll modify my rsync command to preserve hard links.
Radu
On 3/19/2014 6:36 AM, Radu Radutiu wrote:
I'll modify my rsync command to preserve hard links.
note that on a large file system with a large number of files, thats VERY expensive, as rsync has to keep a list of every inode number on the whole file system and verify each directory entry isn't pointing to an inode its already linked. if there's a few million files, this data structure gets HUGE in memory.
note that on a large file system with a large number of files, thats
VERY expensive, as rsync has to keep a list of every inode number on the whole file system and verify each directory entry isn't pointing to an inode its already linked. if there's a few million files, this data structure gets HUGE in memory.
Thanks for the tip. I will keep an eye on the memory usage. For the filesystem in question with around 0.8 million files the memory usage of rsync is acceptable (around 160MB as reported by top RES column and 330 MB for VIRT).
On 03/19/2014 01:37 PM, John R Pierce wrote:
On 3/19/2014 6:36 AM, Radu Radutiu wrote:
I'll modify my rsync command to preserve hard links.
note that on a large file system with a large number of files, thats VERY expensive, as rsync has to keep a list of every inode number on the whole file system and verify each directory entry isn't pointing to an inode its already linked. if there's a few million files, this data structure gets HUGE in memory.
I would hope that rsync is clever enough to do this only for non-directory files with a link count greater than 1. Yes, for some backup archives that still could be the vast majority of the files.
-
John R Pierce -
Mr Queue -
Radu Radutiu -
Robert Nichols -
zGreenfelder