[CentOS] disk space trouble on ec2 instance

Sat Feb 28 06:46:15 UTC 2015
Tim Dunphy <bluethundr at gmail.com>

Hey all,

 Ok, so I've been having some trouble for a while with an EC2 instance
running CentOS 5.11 with  a disk volume reporting 100% usage. Root is on an
EBS volume.

 So I've tried the whole 'du -sk | sort -nr | head -10' routine all around
this volume getting rid of files.  At first I was getting rid of about 50MB
of files. Yet the volume remains at 100% capacity.

 Thinking that maybe the OS was just not letting go of the inodes for the
files on the disk, I attempted rebooting the instance. After logging in
again I did a df -h / on the root volume. And look! Still at 100% capcity
used. Grrr....

Ok so I then did a du -h on the /var/www directory, which was mounted on
the root volume. And saw that it was gobbling up 190MB of disk space.

So then I reasoned that I could create an EBS volume, rsync the data there,
blow away the contents of /var/www/* and then mount the EBS volume on the
/var/www directory. So I went through that exercise and lo and behold.
Still at 100% capacity. Rebooted the instance again. Logged in and.. still
at 100% capacity.

Here's how the volumes are looking now.

[root at ops:~] #df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             9.9G  9.3G   49M 100% /
none                  312M     0  312M   0% /dev/shm
/dev/sdi              148G  116G   25G  83% /backup/tapes
/dev/sdh              9.9G  385M  9.0G   5% /backup/tapes/bacula-restores
/dev/sdf              9.9G  2.1G  7.4G  22% /var/lib/mysql
fuse                  256T     0  256T   0% /backup/mysql
fuse                  256T     0  256T   0% /backup/svn
/dev/sdg              197G  377M  187G   1% /var/www

There are some really important functions I need this volume to perform
that it simply can't because the root volume is at 100% capacity. Like the
fact that neither mysql nor my backup program - bacula will even think of
starting up and functioning!

I'm at a loss to explain how I can delete 190MB worth of data, reboot the
instance and still be at 100% usage.

I'm at my wits end over this. Can someone please offer some advice on how
to solve this problem?

Thanks
Tim





-- 
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B