I have a CentOS 5.2 box that every few months runs out of drivespace on its root filesystem. Last time I manually searched and deleted some big files, but don't remember what they were or what wrote to them. The applications I'm aware of on the box don't write to /.
Is there a way to find the files that get written to the most, or grow the most over time? Doing a df gives me a snapshot, but it seems clunky to keep track of the diff on that output over time. I can then see what processes write to them. Any other ideas on how to investigate this are welcome.
Thanks much.
Jorge
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of JCARRIZOSA@Crutchfield.com Sent: Thursday, April 09, 2009 18:52 To: CentOS@centos.org Subject: [CentOS] What is writing to my filesystem
I have a CentOS 5.2 box that every few months runs out of drivespace on its root filesystem. Last time I manually searched and deleted some big files, but don't remember what they were or what wrote to them. The applications I'm aware of on the box don't write to /.
Not much help, but every time we have "this" problem, it is in /var/log
Is there a way to find the files that get written to the most, or grow the most over time? Doing a df gives me a snapshot, but it seems clunky to keep track of the diff on that output over time. I can then see what processes write to them. Any other ideas on how to investigate this are welcome.
du -s /*
Thanks much.
Jorge
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - - - Jason Pyeron PD Inc. http://www.pdinc.us - - Principal Consultant 10 West 24th Street #100 - - +1 (443) 269-1555 x333 Baltimore, Maryland 21218 - - - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- This message is copyright PD Inc, subject to license 20080407P00.
On Thu, 9 Apr 2009, JCARRIZOSA@Crutchfield.com wrote:
I have a CentOS 5.2 box that every few months runs out of drivespace on its root filesystem. Last time I manually searched and deleted some big files, but don't remember what they were or what wrote to them. The applications I'm aware of on the box don't write to /.
Is there a way to find the files that get written to the most, or grow the most over time? Doing a df gives me a snapshot, but it seems clunky to keep track of the diff on that output over time. I can then see what processes write to them. Any other ideas on how to investigate this are welcome.
I like filelight for finding big consumers, unfortunately it requires Qt and X.
What is possible as well is to go to your mountpoint that is problematic and then use:
du -xh --max-depth=1
and then follow your gut-feeling in what you think is normal, and what is not :)
Doing this on my own laptop just to show, I found out this:
---- [root@moria var]# du -xh --max-depth=1 554M ./lib 57M ./log 16K ./ftp 942M ./cache 16K ./games 24K ./yp 8.0K ./nis 508K ./run 8.0K ./tux 8.0K ./cvs 8.0K ./preserve 12K ./account 32K ./empty 1.1M ./spool 64M ./clamav 8.0K ./tmp 8.0K ./racoon 168K ./lock 8.6M ./www 16K ./mrepo 76K ./gdm 8.0K ./local 7.3G ./crash 140K ./named 8.0K ./opt 28K ./db 8.9G . ----
./crash has size 7.3G ??
---- [root@moria crash]# du -xh --max-depth=1 8.0K ./2009-02-27-11:59 3.7G ./2009-04-06-19:37 8.0K ./2009-01-23-16:56 3.7G ./2009-04-08-19:55 7.3G . [root@moria crash]# ls -l */* -r-------- 1 root root 4054784396 Apr 6 19:38 2009-04-06-19:37/vmcore -r-------- 1 root root 4054784396 Apr 8 19:57 2009-04-08-19:55/vmcore ----
So even when I didn't know anything was wrong, it helped me :)
JCARRIZOSA@Crutchfield.com wrote:
I have a CentOS 5.2 box that every few months runs out of drivespace on its root filesystem. Last time I manually searched and deleted some big files, but don't remember what they were or what wrote to them. The applications I'm aware of on the box don't write to /.
Is there a way to find the files that get written to the most, or grow the most over time? Doing a df gives me a snapshot, but it seems clunky to keep track of the diff on that output over time. I can then see what processes write to them. Any other ideas on how to investigate this are welcome.
find / -mount -ctime -1 #files that changed today find / -mount -size +50M # files larger than 50 megs.
In article 49DE80CC.4040004@gmail.com, Les Mikesell lesmikesell@gmail.com wrote:
JCARRIZOSA@Crutchfield.com wrote:
I have a CentOS 5.2 box that every few months runs out of drivespace on its root filesystem. Last time I manually searched and deleted some big files, but don't remember what they were or what wrote to them. The applications I'm aware of on the box don't write to /.
Is there a way to find the files that get written to the most, or grow the most over time? Doing a df gives me a snapshot, but it seems clunky to keep track of the diff on that output over time. I can then see what processes write to them. Any other ideas on how to investigate this are welcome.
find / -mount -ctime -1 #files that changed today
ITYM -mtime
Cheers Tony
Tony Mountifield wrote:
In article 49DE80CC.4040004@gmail.com, Les Mikesell lesmikesell@gmail.com wrote:
JCARRIZOSA@Crutchfield.com wrote:
I have a CentOS 5.2 box that every few months runs out of drivespace on its root filesystem. Last time I manually searched and deleted some big files, but don't remember what they were or what wrote to them. The applications I'm aware of on the box don't write to /.
Is there a way to find the files that get written to the most, or grow the most over time? Doing a df gives me a snapshot, but it seems clunky to keep track of the diff on that output over time. I can then see what processes write to them. Any other ideas on how to investigate this are welcome.
find / -mount -ctime -1 #files that changed today
ITYM -mtime
In this case it doesn't matter but ctime can never be earlier than mtime and catches some other types of changes like renames, ownership, etc. Ctime is the time of the last change of the inode, and updating mtime would be an inode change.
At Thu, 9 Apr 2009 18:51:42 -0400 CentOS mailing list centos@centos.org wrote:
I have a CentOS 5.2 box that every few months runs out of drivespace on its root filesystem. Last time I manually searched and deleted some big files, but don't remember what they were or what wrote to them. The applications I'm aware of on the box don't write to /.
You have /var and/or /tmp on separate filesystems? If not, that is where things are being written to. Specificly, look in /tmp, /var/log/, and under /var/spool. Is /var/tmp separate from /tmp? What about /usr/tmp? If you don't run the machine 24/7, logrotate might not be run, this would mean your log files (under /var/log) will grow and grow and grow ... If you don't have an email alias for root (I assume you don't login as root!), then /var/spool/mail/root will grow and grow and grow ... as well (all sorts of silly messages sent to root). It is also possible that stuff is accumulating under /var/spool/lpd as well, although this will be small files.
Is there a way to find the files that get written to the most, or grow the most over time? Doing a df gives me a snapshot, but it seems clunky to keep track of the diff on that output over time. I can then see what processes write to them. Any other ideas on how to investigate this are welcome.
Thanks much.
Jorge
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-----Original Message----- You have /var and/or /tmp on separate filesystems? If not, that is where things are being written to. Specificly, look in /tmp, /var/log/, and under /var/spool. Is /var/tmp separate from /tmp?
What
about /usr/tmp? If you don't run the machine 24/7, logrotate might
not
be run, this would mean your log files (under /var/log) will grow and grow and grow ... If you don't have an email alias for root (I assume you don't login as root!), then /var/spool/mail/root will grow and
grow
and grow ... as well (all sorts of silly messages sent to root). It
is
also possible that stuff is accumulating under /var/spool/lpd as well, although this will be small files.
The machine is up 24/7. The different var, tmps and spool folders didn't look too bad, but did find some large files I had copied a couple of months ago and forgotten. I have cricket monitoring drive space and email if it's above 75% usage. It didn't complain back then, so I'm thinking it's been just below the threshold for a while, and went over the hump just this week with recent smaller files.
Thanks!
The only thing I know that will give you a instant snapshot of what's happening is "mtop". It shows you whayt apps are using your harddrive very similar to top with processes.
David
On 4/9/09, JCARRIZOSA@crutchfield.com JCARRIZOSA@crutchfield.com wrote:
I have a CentOS 5.2 box that every few months runs out of drivespace on its root filesystem. Last time I manually searched and deleted some big files, but don't remember what they were or what wrote to them. The applications I'm aware of on the box don't write to /.
Is there a way to find the files that get written to the most, or grow the most over time? Doing a df gives me a snapshot, but it seems clunky to keep track of the diff on that output over time. I can then see what processes write to them. Any other ideas on how to investigate this are welcome.
Thanks much.
Jorge
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Thu, Apr 09, 2009 at 08:12:47PM -0400, David Lemcoe wrote:
The only thing I know that will give you a instant snapshot of what's happening is "mtop". It shows you whayt apps are using your harddrive very similar to top with processes.
"mtop" is MySQL-Top; how will that help the original poster?
John
Oh my. I apologize so much. It's iotop I was looking to refer to. They both operate so similarly.
Sorry again, David
On 4/9/09, John R. Dennison jrd@gerdesas.com wrote:
On Thu, Apr 09, 2009 at 08:12:47PM -0400, David Lemcoe wrote:
The only thing I know that will give you a instant snapshot of what's happening is "mtop". It shows you whayt apps are using your harddrive very similar to top with processes.
"mtop" is MySQL-Top; how will that help the original poster?
John-- "I'm sorry but our engineers do not have phones." As stated by a Network Solutions Customer Service representative when asked to be put through to an engineer.
"My other computer is your windows box." Ralf Hildebrandt <sxem> trying to play sturgeon while it's under attack is apparently not fun.
From the iotop site at http://guichaz.free.fr/iotop/ - Iotop is a Python
program with a top like UI used to show of behalf of which process is the I/O going on. It requires Python ≥ 2.5 and a Linux kernel ≥ 2.6.20.
Neither of which are standard in CentOS...
d
2009/4/10 David Lemcoe forum@lemcoe.com
Oh my. I apologize so much. It's iotop I was looking to refer to. They both operate so similarly.
Sorry again, David
On 4/9/09, John R. Dennison jrd@gerdesas.com wrote:
On Thu, Apr 09, 2009 at 08:12:47PM -0400, David Lemcoe wrote:
The only thing I know that will give you a instant snapshot of what's happening is "mtop". It shows you whayt apps are using your harddrive very similar to top with processes.
"mtop" is MySQL-Top; how will that help the original poster? John-- "I'm sorry but our engineers do not have phones." As stated by a Network Solutions Customer Service representative when
asked
to be put through to an engineer.
"My other computer is your windows box." Ralf Hildebrandt <sxem> trying to play sturgeon while it's under attack is apparently not fun.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
True, but I got mine from rpmforge. Yum install iotop.i386 I believe.
On 4/9/09, D Tucny d@tucny.com wrote:
From the iotop site at http://guichaz.free.fr/iotop/ - Iotop is a Python
program with a top like UI used to show of behalf of which process is the I/O going on. It requires Python ≥ 2.5 and a Linux kernel ≥ 2.6.20.
Neither of which are standard in CentOS...
d
2009/4/10 David Lemcoe forum@lemcoe.com
Oh my. I apologize so much. It's iotop I was looking to refer to. They both operate so similarly.
Sorry again, David
On 4/9/09, John R. Dennison jrd@gerdesas.com wrote:
On Thu, Apr 09, 2009 at 08:12:47PM -0400, David Lemcoe wrote:
The only thing I know that will give you a instant snapshot of what's happening is "mtop". It shows you whayt apps are using your harddrive very similar to top with processes.
"mtop" is MySQL-Top; how will that help the original poster? John-- "I'm sorry but our engineers do not have phones." As stated by a Network Solutions Customer Service representative when
asked
to be put through to an engineer.
"My other computer is your windows box." Ralf Hildebrandt <sxem> trying to play sturgeon while it's under attack is apparently not fun.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Odd... I did check rpmforge and that only had a buildlog of a failed build from 2007... fedora epel had a number of bugzilla requests closed with the requirements being restated...
Are you sure it came from rpmforge?
d
2009/4/10 David Lemcoe forum@lemcoe.com
True, but I got mine from rpmforge. Yum install iotop.i386 I believe.
On 4/9/09, D Tucny d@tucny.com wrote:
From the iotop site at http://guichaz.free.fr/iotop/ - Iotop is a
Python
program with a top like UI used to show of behalf of which process is the I/O going on. It requires Python ≥ 2.5 and a Linux kernel ≥ 2.6.20.
Neither of which are standard in CentOS...
d
2009/4/10 David Lemcoe forum@lemcoe.com
Oh my. I apologize so much. It's iotop I was looking to refer to. They both operate so similarly.
Sorry again, David
On 4/9/09, John R. Dennison jrd@gerdesas.com wrote:
On Thu, Apr 09, 2009 at 08:12:47PM -0400, David Lemcoe wrote:
The only thing I know that will give you a instant snapshot of what's happening is "mtop". It shows you whayt apps are using your harddrive very similar to top with processes.
"mtop" is MySQL-Top; how will that help the original poster? John-- "I'm sorry but our engineers do not have phones." As stated by a Network Solutions Customer Service representative when
asked
to be put through to an engineer.
"My other computer is your windows box." Ralf Hildebrandt <sxem> trying to play sturgeon while it's under attack is apparently
not
fun.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Very odd. I know I got it from somewhere. I may have built it from an older version. I have no idea exactly though.
On 4/9/09, D Tucny d@tucny.com wrote:
Odd... I did check rpmforge and that only had a buildlog of a failed build from 2007... fedora epel had a number of bugzilla requests closed with the requirements being restated...
Are you sure it came from rpmforge?
d
2009/4/10 David Lemcoe forum@lemcoe.com
True, but I got mine from rpmforge. Yum install iotop.i386 I believe.
On 4/9/09, D Tucny d@tucny.com wrote:
From the iotop site at http://guichaz.free.fr/iotop/ - Iotop is a
Python
program with a top like UI used to show of behalf of which process is the I/O going on. It requires Python ≥ 2.5 and a Linux kernel ≥ 2.6.20.
Neither of which are standard in CentOS...
d
2009/4/10 David Lemcoe forum@lemcoe.com
Oh my. I apologize so much. It's iotop I was looking to refer to. They both operate so similarly.
Sorry again, David
On 4/9/09, John R. Dennison jrd@gerdesas.com wrote:
On Thu, Apr 09, 2009 at 08:12:47PM -0400, David Lemcoe wrote:
The only thing I know that will give you a instant snapshot of what's happening is "mtop". It shows you whayt apps are using your harddrive very similar to top with processes.
"mtop" is MySQL-Top; how will that help the original poster? John-- "I'm sorry but our engineers do not have phones." As stated by a Network Solutions Customer Service representative when
asked
to be put through to an engineer.
"My other computer is your windows box." Ralf Hildebrandt <sxem> trying to play sturgeon while it's under attack is apparently
not
fun.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
2009/4/10 JCARRIZOSA@crutchfield.com:
I have a CentOS 5.2 box that every few months runs out of drivespace on its root filesystem. Last time I manually searched and deleted some big files, but don't remember what they were or what wrote to them. The applications I'm aware of on the box don't write to /.
I have a Centos 3 system, which has its /boot in the / filesystem, and after several kernel upgrades it gets full, so I have to deinstall old kernels.
On Friday 10 April 2009 08:11:32 Alexander Georgiev wrote:
2009/4/10 JCARRIZOSA@crutchfield.com:
I have a CentOS 5.2 box that every few months runs out of drivespace on its root filesystem. Last time I manually searched and deleted some big files, but don't remember what they were or what wrote to them. The applications I'm aware of on the box don't write to /.
I have a Centos 3 system, which has its /boot in the / filesystem, and after several kernel upgrades it gets full, so I have to deinstall old kernels.
Do these systems run 24/7? If they don't, the housekeeping jobs will be missed unless anacron is running them. Certainly I have been running CentOS since 5.1 and have never seen this problem.
Anne
From: "JCARRIZOSA@Crutchfield.com" JCARRIZOSA@Crutchfield.com
I have a CentOS 5.2 box that every few months runs out of drivespace on its root filesystem. Last time I manually searched and deleted some big files, but don't remember what they were or what wrote to them. The applications I'm aware of on the box don't write to /. Is there a way to find the files that get written to the most, or grow the most over time? Doing a df gives me a snapshot, but it seems clunky to keep track of the diff on that output over time. I can then see what processes write to them. Any other ideas on how to investigate this are welcome.
Do you use logrotate? Do you use logrotate's compression?
Maybe use some kind of snapshots, like:
#!/bin/bash find /var -type f -printf "%k %p\n" > /tmp/usedspace.new if [ -f /tmp/usedspace.old ]; then cat /tmp/usedspace.old | while read LINE do set $LINE OLDSIZE=$1 OLDFILE=$2 NEWSIZE=`grep "$OLDFILE$" /tmp/usedspace.new | cut -d " " -f1` if [ -n "$NEWSIZE" -a "$OLDSIZE" != "$NEWSIZE" ]; then echo "$OLDFILE: $OLDSIZE => $NEWSIZE" fi done fi mv -f /tmp/usedspace.new /tmp/usedspace.old
JD
-
Alexander Georgiev -
Anne Wilson -
D Tucny -
Dag Wieers -
David Lemcoe -
Jason Pyeron -
JCARRIZOSA＠Crutchfield.com -
John Doe -
John R. Dennison -
Les Mikesell -
Robert Heller -
tony＠softins.clara.co.uk