Hi all,
Curious issue.. looking in to how much disk space is being used on a machine (CentOS 5.3). When I compare the output of du vs df, I am seeing a 12GB difference with du saying 8G used and df saying 20G used.
# du -hcx / 8.0G total
# df -h / Filesystem Size Used Avail Use% Mounted on /dev/xvda3 22G 20G 637M 97% /
I recognize that in most cases du and df are not going to report the same but I am concerned about having a 12GB disparity. Does anyone have any thoughts about this or reason as to why there is a big difference? I have read a few articles online about it and none have really shown such a large difference.
Thanks
Ryan Pugatch Systems Administrator, TripAdvisor
On Wed, Sep 30, 2009 at 04:59:25PM -0400, Ryan Pugatch wrote:
Hi all,
Curious issue.. looking in to how much disk space is being used on a machine (CentOS 5.3). When I compare the output of du vs df, I am seeing a 12GB difference with du saying 8G used and df saying 20G used.
# du -hcx / 8.0G total
# df -h / Filesystem Size Used Avail Use% Mounted on /dev/xvda3 22G 20G 637M 97% /
I recognize that in most cases du and df are not going to report the same but I am concerned about having a 12GB disparity. Does anyone have any thoughts about this or reason as to why there is a big difference? I have read a few articles online about it and none have really shown such a large difference.
Do this: mount /dev/xvda3 /mnt du -hc /mnt
And see if you can find the other 12GB.
I usually do: du -mc --max-depth 2 /mnt | sort -n
Though I've recently learned: du -hc --max-depth 2 /mnt | sort --human-readable or some such, but that requires a very recent coreutils installation.
Luciano Rocha wrote:
Do this: mount /dev/xvda3 /mnt du -hc /mnt
And see if you can find the other 12GB.
I usually do: du -mc --max-depth 2 /mnt | sort -n
Though I've recently learned: du -hc --max-depth 2 /mnt | sort --human-readable or some such, but that requires a very recent coreutils installation.
That's interesting.. du is showing the same amount as df when I do this. I wonder why..
Ryan
On Wed, Sep 30, 2009 at 06:30:08PM -0400, Ryan Pugatch wrote:
Luciano Rocha wrote:
Do this: mount /dev/xvda3 /mnt du -hc /mnt
And see if you can find the other 12GB.
I usually do: du -mc --max-depth 2 /mnt | sort -n
Though I've recently learned: du -hc --max-depth 2 /mnt | sort --human-readable or some such, but that requires a very recent coreutils installation.
That's interesting.. du is showing the same amount as df when I do this. I wonder why..
There's some data hidden by mount points. Check your mount points, /home, etc., and there's probably old data there that may be removed.
Cut'n'paste from a tutorial I'm writing right now:
Check with lsof if there are any very large files that are already deleted but are still open by some processes.
# df -h . Filesystem Size Used Avail Use% Mounted on /dev/md0 194M 32M 153M 18% /boot # dd if=/dev/zero of=testfile bs=1024000 count=50 50+0 records in 50+0 records out 51200000 bytes (51 MB) copied, 0.370919 s, 138 MB/s # df -h . Filesystem Size Used Avail Use% Mounted on /dev/md0 194M 81M 104M 44% /boot # vi testfile
<wait till vi comes up, then crtl-z and bg the process>
# rm testfile # df -h . Filesystem Size Used Avail Use% Mounted on /dev/md0 194M 81M 104M 44% /boot
< quit vi>
# df -h . Filesystem Size Used Avail Use% Mounted on /dev/md0 194M 32M 153M 18% /boot
Also verify that your fs doesn't have a insane root reserved space set. # df -h . Filesystem Size Used Avail Use% Mounted on /dev/md0 194M 32M 153M 18% /boot # tune2fs -m 50 /dev/md0 tune2fs 1.41.4 (27-Jan-2009) Setting reserved blocks percentage to 50% (102368 blocks) # df -h . Filesystem Size Used Avail Use% Mounted on /dev/md0 194M 32M 63M 34% /boot
THere are of course other reasons but those two are a common problem.
Peter.
On Wednesday 30 September 2009 16:59:25 Ryan Pugatch wrote:
Hi all,
Curious issue.. looking in to how much disk space is being used on a machine (CentOS 5.3). When I compare the output of du vs df, I am seeing a 12GB difference with du saying 8G used and df saying 20G used.
# du -hcx / 8.0G total
# df -h / Filesystem Size Used Avail Use% Mounted on /dev/xvda3 22G 20G 637M 97% /
I recognize that in most cases du and df are not going to report the same but I am concerned about having a 12GB disparity. Does anyone have any thoughts about this or reason as to why there is a big difference? I have read a few articles online about it and none have really shown such a large difference.
Thanks
Ryan Pugatch Systems Administrator, TripAdvisor _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Ryan Pugatch wrote:
I recognize that in most cases du and df are not going to report the same but I am concerned about having a 12GB disparity. Does anyone have any thoughts about this or reason as to why there is a big difference? I have read a few articles online about it and none have really shown such a large difference.
du won't look at deleted files, while df will.
run
lsof | grep deleted
and see if any files are deleted are still open, space from them will not be freed until the process releases the file handle.
nate
At Wed, 30 Sep 2009 14:10:46 -0700 (PDT) CentOS mailing list centos@centos.org wrote:
Ryan Pugatch wrote:
I recognize that in most cases du and df are not going to report the same but I am concerned about having a 12GB disparity. Does anyone have any thoughts about this or reason as to why there is a big difference? I have read a few articles online about it and none have really shown such a large difference.
du won't look at deleted files, while df will.
df does not look at files (deleted or not) at all! It just looks at allocated inodes.
run
lsof | grep deleted
and see if any files are deleted are still open, space from them will not be freed until the process releases the file handle.
nate
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Wed, Sep 30, 2009 at 2:59 PM, Ryan Pugatch rpug@tripadvisor.com wrote:
Hi all,
Curious issue.. looking in to how much disk space is being used on a machine (CentOS 5.3). When I compare the output of du vs df, I am seeing a 12GB difference with du saying 8G used and df saying 20G used.
# du -hcx / 8.0G total
# df -h / Filesystem Size Used Avail Use% Mounted on /dev/xvda3 22G 20G 637M 97% /
I recognize that in most cases du and df are not going to report the same but I am concerned about having a 12GB disparity. Does anyone have any thoughts about this or reason as to why there is a big difference? I have read a few articles online about it and none have really shown such a large difference.
One of the things I run into are either hidden files or leaked files where a process is still talking to a file but the directory no longer sees it so du doesn't catch it.
ls -l /proc/[0-9]*/fd/| grep delete
will show those up. Then its a matter if you want to keep that file around or not.
also du / did not look for files in / that were starting with a .
ls -la / and see if there are hidden directories or files taking up space.
Finally sparse files can give odd readings at time.. but that is the least likely reason.
Thanks
Ryan Pugatch Systems Administrator, TripAdvisor _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Stephen John Smoogen wrote:
ls -l /proc/[0-9]*/fd/| grep delete
will show those up. Then its a matter if you want to keep that file around or not.
also du / did not look for files in / that were starting with a .
ls -la / and see if there are hidden directories or files taking up space.
Finally sparse files can give odd readings at time.. but that is the least likely reason.
Nothing for deleted files and no large . files found.
Thanks,
Ryan
Ryan Pugatch wrote:
Oh, and no sparse files either :)
Last time I saw this issue, no sparse files, nothing legit, it was a corrupted FS. :(
On Thursday 01 October 2009, Ryan Pugatch wrote:
Florin Andrei wrote:
Last time I saw this issue, no sparse files, nothing legit, it was a corrupted FS. :(
Well, if I mount to another directory the size is right. My next step will be to fsck probably.
One possibility is that the missing data is hiding under a mount-point in the normal case.
/Peter
Peter Kjellstrom wrote:
One possibility is that the missing data is hiding under a mount-point in the normal case.
/Peter
So what you're saying is something is mounted on to a directory that had data in it before the mount. How do I see the data being hidden without unmounting the point?
Thanks,
Ryan
Ryan Pugatch wrote:
So what you're saying is something is mounted on to a directory that had data in it before the mount. How do I see the data being hidden without unmounting the point?
Thanks,
Ryan
After thinking about this, I realized I could mount the partition to another point and then see what was being hidden under the mount point. I found 12G of data under one point. This explains the discrepancy. That solves the problem.
Thank you all for your advice.
Ryan
At Thu, 01 Oct 2009 10:13:26 -0400 CentOS mailing list centos@centos.org wrote:
Peter Kjellstrom wrote:
One possibility is that the missing data is hiding under a mount-point in the normal case.
/Peter
So what you're saying is something is mounted on to a directory that had data in it before the mount. How do I see the data being hidden without unmounting the point?
You can't. You must unmount. You should be able to do this from single user mode if the file system cannot be unmounted under multiuser mode (eg /usr, /var, etc.). Usually other mount points can be unmounted, but depends on what is running on the system at the time. Unmounting /home would require that you kick all users off for example, unmounting /var/www would require stopping apache, etc.
Thanks,
Ryan _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Robert Heller wrote:
You can't. You must unmount. You should be able to do this from single user mode if the file system cannot be unmounted under multiuser mode (eg /usr, /var, etc.). Usually other mount points can be unmounted, but depends on what is running on the system at the time. Unmounting /home would require that you kick all users off for example, unmounting /var/www would require stopping apache, etc.
I mounted the partition to a directory under /mnt (simultaneously) and was able to see the hidden files.
Ryan
Robert Heller heller@deepsoft.com schrieb am 01.10.2009 19:35:20:
At Thu, 01 Oct 2009 10:13:26 -0400 CentOS mailing list centos@centos.org wrote:
Peter Kjellstrom wrote:
One possibility is that the missing data is hiding under a
mount-point in the
normal case.
/Peter
So what you're saying is something is mounted on to a directory that
had
data in it before the mount. How do I see the data being hidden
without
unmounting the point?
You can't. You must unmount. You should be able to do this from single user mode if the file system cannot be unmounted under multiuser mode (eg /usr, /var, etc.). Usually other mount points can be unmounted, but depends on what is running on the system at the time. Unmounting /home would require that you kick all users off for example, unmounting /var/www would require stopping apache, etc.
This is absolutely untested but it could work: mount / to /mnt and delete the data from there. Dig into mount(8) and test this somewhere outside the production area:
--bind Remount a subtree somewhere else (so that its contents are available in both places). See above.
But again, this is untested and nothing more but a wild guess.
Frank.
This is quite possible, and having just gone thru this recently a few weeks ago, I thought I post a warning here to hopefully save someone else from the brain-fart I suffered a few weeks ago. I needed to change the mount point permissions w/out umounting the filesystems in a few places, so I remounted / over /tmp/mnt (that was my brain fart, should have used /mnt/tmp). Did my thing, changed what I needed to and then got sidetracked on other things and never got around to umount'ing /tmp/mnt. Next day I started noticing weirdness, and then realized cron.daily fired off it's tmpwatch run and it happily started cleaning older things from /tmp/mnt/... :-(
- Jerry
Frank.Brodbeck@klingel.de wrote:
This is absolutely untested but it could work: mount / to /mnt and delete the data from there. Dig into mount(8) and test this somewhere outside the production area:
--bind Remount a subtree somewhere else (so that its contents are available in both places). See above.
But again, this is untested and nothing more but a wild guess.
On Thu, Oct 01, 2009 at 01:35:20PM -0400, Robert Heller wrote:
At Thu, 01 Oct 2009 10:13:26 -0400 CentOS mailing list centos@centos.org wrote:
So what you're saying is something is mounted on to a directory that had data in it before the mount. How do I see the data being hidden without unmounting the point?
You can't. You must unmount. You should be able to do this from
Two options: 1) Try a bind mount to rebind the parent 2) NFS export the parent; NFS exports typically don't cross mount points and show the underlying filesystem
At Wed, 30 Sep 2009 16:59:25 -0400 CentOS mailing list centos@centos.org wrote:
Hi all,
Curious issue.. looking in to how much disk space is being used on a machine (CentOS 5.3). When I compare the output of du vs df, I am seeing a 12GB difference with du saying 8G used and df saying 20G used.
# du -hcx / 8.0G total
# df -h / Filesystem Size Used Avail Use% Mounted on /dev/xvda3 22G 20G 637M 97% /
I recognize that in most cases du and df are not going to report the same but I am concerned about having a 12GB disparity. Does anyone have any thoughts about this or reason as to why there is a big difference? I have read a few articles online about it and none have really shown such a large difference.
Somewhere you have an open, but deleted file. Maybe someone deleted a log file and did NOT do a SIGHUP (or restart) of the corresponding deamon? Or some similar weirdness. Some running process is hanging onto a an open file descriptor of a file that has been deleted. Note that some program create 'unnamed' temp files -- opened, then deleted -- as soon as the program exits, the temp file space gets recycled, without the program 'remembering' to delete it, since it is already deleted. This saves having to deal with an exit handler in case the program crashes.
Thanks
Ryan Pugatch Systems Administrator, TripAdvisor _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Ryan Pugatch wrote:
I recognize that in most cases du and df are not going to report the same but I am concerned about having a 12GB disparity. Does anyone have any thoughts about this or reason as to why there is a big difference?
Sparse files?
On Wed, Sep 30, 2009 at 17:59, Ryan Pugatch rpug@tripadvisor.com wrote:
Hi all,
Curious issue.. looking in to how much disk space is being used on a machine (CentOS 5.3). When I compare the output of du vs df, I am seeing a 12GB difference with du saying 8G used and df saying 20G used.
Maybe you have a mount point overlaping big files... du -x will not find them...
Ryan Pugatch wrote:
Marcelo Roccasalva wrote:
Maybe you have a mount point overlaping big files... du -x will not find them...
Hey Marcelo,
I am not sure what you mean.. can you give me an example?
Thanks
Ryan
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Hi,
He means a situation where you have something this this
You create a partion lets say /dev/hda1 you use it as you / partition you create a directory called /data and copy some data into it You then have a second partiton /dev/hda2 and you mount /dev/hda2 off of /data
This mean that the data originally copied to the /data directory on hda1 is still taking up space on hda1 but you cant see it.
Clint Dilks wrote:
Hi,
He means a situation where you have something this this
You create a partion lets say /dev/hda1 you use it as you / partition you create a directory called /data and copy some data into it You then have a second partiton /dev/hda2 and you mount /dev/hda2 off of /data
This mean that the data originally copied to the /data directory on hda1 is still taking up space on hda1 but you cant see it.
In this case, if I mount my partition to, say, /mnt/tmp, it would still show the 'wrong' size. In my situation, it did not, so that indicates this is not my issue.
Ryan
Ryan Pugatch wrote:
Hi all,
Curious issue.. looking in to how much disk space is being used on a machine (CentOS 5.3). When I compare the output of du vs df, I am seeing a 12GB difference with du saying 8G used and df saying 20G used.
# du -hcx / 8.0G total
# df -h / Filesystem Size Used Avail Use% Mounted on /dev/xvda3 22G 20G 637M 97% /
I recognize that in most cases du and df are not going to report the same but I am concerned about having a 12GB disparity. Does anyone have any thoughts about this or reason as to why there is a big difference? I have read a few articles online about it and none have really shown such a large difference.
I see similar differences even when I:
a) Boot from a rescue CD, b) Freshly fsck the file system to be tested, and c) Mount that file system read-only.
I suspected the discrepancy might be due to the space used for the ext3 journal, but I also see it on a freshly created ext2 file system:
# mount -r /dev/hda8 /mnt/tmp # du -s /mnt/tmp; df /mnt/tmp 20 /mnt/tmp Filesystem 1K-blocks Used Available Use% Mounted on /dev/hda8 13638436 33824 12911812 1% /mnt/tmp
So, there's a 33+MB difference on a fresh, empty ext2 file system.
Looking at the file system with debugfs, I find inode 7 is a regular file of size 4299210752 and a block count of 67608. That's a huge sparse file. A little research shows that this is the "resize inode" that reserves space for future GDT blocks so that the file system can be expanded in place.
-
Clint Dilks -
Florin Andrei -
Frank.Brodbeck＠klingel.de -
Jerry Queirolo -
Luciano Rocha -
Marcelo Roccasalva -
nate -
Peter A -
Peter Kjellstrom -
Robert Heller -
Robert Nichols -
Ryan Pugatch -
Stephen Harris -
Stephen John Smoogen