Hey all,
Ok, so I've been having some trouble for a while with an EC2 instance running CentOS 5.11 with a disk volume reporting 100% usage. Root is on an EBS volume.
So I've tried the whole 'du -sk | sort -nr | head -10' routine all around this volume getting rid of files. At first I was getting rid of about 50MB of files. Yet the volume remains at 100% capacity.
Thinking that maybe the OS was just not letting go of the inodes for the files on the disk, I attempted rebooting the instance. After logging in again I did a df -h / on the root volume. And look! Still at 100% capcity used. Grrr....
Ok so I then did a du -h on the /var/www directory, which was mounted on the root volume. And saw that it was gobbling up 190MB of disk space.
So then I reasoned that I could create an EBS volume, rsync the data there, blow away the contents of /var/www/* and then mount the EBS volume on the /var/www directory. So I went through that exercise and lo and behold. Still at 100% capacity. Rebooted the instance again. Logged in and.. still at 100% capacity.
Here's how the volumes are looking now.
[root@ops:~] #df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 9.9G 9.3G 49M 100% / none 312M 0 312M 0% /dev/shm /dev/sdi 148G 116G 25G 83% /backup/tapes /dev/sdh 9.9G 385M 9.0G 5% /backup/tapes/bacula-restores /dev/sdf 9.9G 2.1G 7.4G 22% /var/lib/mysql fuse 256T 0 256T 0% /backup/mysql fuse 256T 0 256T 0% /backup/svn /dev/sdg 197G 377M 187G 1% /var/www
There are some really important functions I need this volume to perform that it simply can't because the root volume is at 100% capacity. Like the fact that neither mysql nor my backup program - bacula will even think of starting up and functioning!
I'm at a loss to explain how I can delete 190MB worth of data, reboot the instance and still be at 100% usage.
I'm at my wits end over this. Can someone please offer some advice on how to solve this problem?
Thanks Tim