Hi,
While checking with df -h, it's showing the used space is 94% on root (/). If checked with du -sh, it's not showing the used space.
# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 7.8G 0 7.8G 0% /dev tmpfs 7.8G 0 7.8G 0% /dev/shm tmpfs 7.8G 857M 7.0G 11% /run tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup /dev/mapper/centos-root 50G 47G 3.4G 94% / /dev/mapper/centos-home 241G 47G 195G 20% /var/log /dev/sda1 1014M 189M 826M 19% /boot tmpfs 1.6G 0 1.6G 0% /run/user/0 tmpfs 1.6G 0 1.6G 0% /run/user/1002
# du -sh /* 0 /bin 156M /boot 0 /dev 33M /etc 388K /home 0 /lib 0 /lib64 0 /media 0 /mnt 0 /opt du: cannot access ‘/proc/21489/task/21489/fd/4’: No such file or directory du: cannot access ‘/proc/21489/task/21489/fdinfo/4’: No such file or directory du: cannot access ‘/proc/21489/fd/4’: No such file or directory du: cannot access ‘/proc/21489/fdinfo/4’: No such file or directory 0 /proc 6.1M /root 857M /run 0 /sbin 0 /srv 0 /sys 0 /tmp 2.8G /usr 62G /var
# du -sh /var/* 0 /var/adm 89M /var/cache 0 /var/crash 8.0K /var/db 0 /var/empty 0 /var/games 0 /var/gopher 0 /var/kerberos 16G /var/lib 0 /var/local 0 /var/lock 47G /var/log 0 /var/mail 0 /var/nis 0 /var/opt 0 /var/preserve 0 /var/run 98M /var/spool 0 /var/tmp 499M /var/www 0 /var/yp
How can i find this hidden space?
Hi Sachchidanand,
On Unix-like operating systems, if a process has a file open, and you delete the file, it will not be removed from disk immediately. That will only happen when the process closes the file descriptor, or exits. People new to Unix-like operating systems often don't know this.
So you may have one or more such files on your / partition. Install a utility called "lsof", run "lsof -n" as root, and grep the output for the word "deleted", and examine those files, and see which processes are holding them open. Then either restart those processes, or reboot the server.
Regards, Anand Buddhdev
On 29/06/2020 11:51, Sachchidanand Upadhyay via CentOS wrote:
Hi,
While checking with df -h, it's showing the used space is 94% on root (/). If checked with du -sh, it's not showing the used space.
# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 7.8G 0 7.8G 0% /dev tmpfs 7.8G 0 7.8G 0% /dev/shm tmpfs 7.8G 857M 7.0G 11% /run tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup /dev/mapper/centos-root 50G 47G 3.4G 94% / /dev/mapper/centos-home 241G 47G 195G 20% /var/log /dev/sda1 1014M 189M 826M 19% /boot tmpfs 1.6G 0 1.6G 0% /run/user/0 tmpfs 1.6G 0 1.6G 0% /run/user/1002
# du -sh /* 0 /bin 156M /boot 0 /dev 33M /etc 388K /home 0 /lib 0 /lib64 0 /media 0 /mnt 0 /opt du: cannot access ‘/proc/21489/task/21489/fd/4’: No such file or directory du: cannot access ‘/proc/21489/task/21489/fdinfo/4’: No such file or directory du: cannot access ‘/proc/21489/fd/4’: No such file or directory du: cannot access ‘/proc/21489/fdinfo/4’: No such file or directory 0 /proc 6.1M /root 857M /run 0 /sbin 0 /srv 0 /sys 0 /tmp 2.8G /usr 62G /var
# du -sh /var/* 0 /var/adm 89M /var/cache 0 /var/crash 8.0K /var/db 0 /var/empty 0 /var/games 0 /var/gopher 0 /var/kerberos 16G /var/lib 0 /var/local 0 /var/lock 47G /var/log 0 /var/mail 0 /var/nis 0 /var/opt 0 /var/preserve 0 /var/run 98M /var/spool 0 /var/tmp 499M /var/www 0 /var/yp
How can i find this hidden space?
# du -sh /*
Use 'du -xh --max-depth=1 /' it will clean up your output and show you only things on the root partition.
And as someone else said, deleted but open files are not removed until the file handle is closed. This is used by some applications to "hide" totally temporary files. Do 'lsof | grep delete' to see such files. (This technique is also used by malware to hide their files.)
P.
On Mon, 29 Jun 2020, Pete Biggs wrote:
# du -sh /*
Use 'du -xh --max-depth=1 /' it will clean up your output and show you only things on the root partition.
Note the reason for -x . -x is equivalent to --one-file-system . It says that when searching from a directory, include only descendants in the same filesystem.
Though rather unlikely, it is possible that none of /* are in the same filesystem as / , hence the need for / rather than /* .
On Mon, Jun 29, 2020 at 1:23 PM Michael Hennebry < hennebry@web.cs.ndsu.nodak.edu> wrote:
Though rather unlikely, it is possible that none of /* are in the same filesystem as / , hence the need for / rather than /* .
as /dev, /proc, and /sys are mounted on nearly every linux system, and those are places you don't want du poking around, you do need to avoid that.
On Mon, 29 Jun 2020 15:21:12 +0530 Sachchidanand Upadhyay via CentOS centos@centos.org wrote:
Hi,
While checking with df -h, it's showing the used space is 94% on root (/). If checked with du -sh, it's not showing the used space.
# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 7.8G 0 7.8G 0% /dev tmpfs 7.8G 0 7.8G 0% /dev/shm tmpfs 7.8G 857M 7.0G 11% /run tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup /dev/mapper/centos-root 50G 47G 3.4G 94% / /dev/mapper/centos-home 241G 47G 195G 20% /var/log /dev/sda1 1014M 189M 826M 19% /boot tmpfs 1.6G 0 1.6G 0% /run/user/0 tmpfs 1.6G 0 1.6G 0% /run/user/1002
As an addition to what others have already said. You'll also miss things "hidden under mounts". That is, if you had 5G in /var/log on the root file system and then mounted a different device on /var/log, then that 5G would still be there but invisible.
Also, /dev/mapper/centos-home, HOME?!, on /var/log?
/Peter
Peter Kjellström wrote:
As an addition to what others have already said. You'll also miss things "hidden under mounts". That is, if you had 5G in /var/log on the root file system and then mounted a different device on /var/log, then that 5G would still be there but invisible.
What I usually do in cases like this, is to bind mount root to somewhere else - and then run 'du' or whatever using the bind mount point - this will show up any 'hidden under mounts' data - something like:
mkdir /var/run/mnt mount --bind / /var/run/mnt du -sh /var/run/mnt/*
James Pearson
-
Anand Buddhdev -
James Pearson -
John Pierce -
Michael Hennebry -
Pete Biggs -
Peter Kjellström -
Sachchidanand Upadhyay