On Wednesday 04 May 2011 21:01:03 Jorge Fábregas wrote:

On 05/04/2011 12:17 PM, Nicolas Ross wrote:

...

iotop can points me to wich process, but that doesn't points me to what files are the culprits...

A rough way would be to change to the top-level directory where you suspect the files are being written and perform:

find . -type f -mmin -1 (that would search for all files modified within the last minute)

A more elegant way would be:

lsof -p PID (where PID is the process ID...of the process iotop showed you)

Just out of curiosity I decided to write a simple script which checks all the files from all pids on the system.

Here is what I got: http://hydra.azilian.net/scripts/read_fds.pl

The idea is to read all the /proc/PID/fdinfo/ files and check the difference in the pos lines (the position in the file descriptor). This is both write and read position depending on how the application has opened the file. So in the end it lists all pids and the respective FDs which have changes:

hackman@gamelon:~$ sudo ./read_fds.pl 4 Pid: 14229 Position change: 22 blocks FD: 4(/home/hackman/f2.tst) Pid: 14229 Position change: 12 blocks FD: 3(/home/hackman/f1.tst)

The argument to the script is the sleep between the two checks. I have tested the script on a few production servers... It works as a charm :)

Thank you for the good question... now I have one good tool in my arsenal :)

-- Best regards, Marian Marinov

(...)

This is excellent, and sooo clever... Except that I don't have the /proc/*/fdinfo directories. It seems that theses directories appeared in 2.6.22, and, since I am in centos5, I only have 2.6.18...

I tested it on SL6 machine, and it works perfectly... Upgrade is not an option for the moment for the machine I have the problem with.

It appears that RHEL has back-ported some featears in 2.6.18 kernels. My service is now at 5.6 (was 5.5). Now I have the /proc/*/fdinfo directories, and can use your tool.

Best regards,

On Thursday 05 May 2011 05:24:10 Marcelo Beckmann wrote:

2011/5/4 Nicolas Ross rossnick-lists@cybercat.ca:

...

Hi !

I have a server (Centos 5) that is using a pair of SAS drives to store the data. (Mail server) They are on an adaptec raid controler with a battery backup and write back cache active.

...

From time to time, I have sever peak io to those data disks (> 400 to 500

iops, > 70 to 100 megs/sec).

With iostat, I find that it's almost a write i/o problem. How can I find to which files the OS writes ? On OSX boxes, there is a utility called fs_usage that can reports any disk activity for a particular process or all processes. Is there any utility like this on Centos ?

iotop can points me to wich process, but that doesn't points me to what files are the culprits...

I sugest a look for tools like this http://freshmeat.net/projects/fsniper

it helps to make a script to watch file activities, and it uses a kernel feature

I discovered inotify some months ago when I looked into every initscript in init.d

[23:13:35 root@gw init.d]# cat /etc/redhat-release CentOS release 5.3 (Final) [23:13:45 root@gw init.d]# head restorecond #!/bin/sh # # restorecond: Daemon used to maintain path file context # # chkconfig: 2345 12 87 # description: restorecond uses inotify to look for creation of new files \ # listed in the /etc/selinux/restorecond.conf file, and restores the \ # correct security context.

more about inotify: http://linux.die.net/man/7/inotify

http://www.linuxjournal.com/article/8478 What Is inotify?

inotify is a file change notification system—a kernel feature that allows applications to request the monitoring of a set of files against a list of events. When the event occurs, the application is notified. To be useful, such a feature must be simple to use, lightweight with little overhead and flexible. It should be easy to add new watches and painless to receive notification of events.

If you go the inotify route, do keep in mind that you need to monitor for modify events, otherwise you would not see the file changes before the applications finish with the files.

Regards, Marian

Greetings,

On Thu, May 5, 2011 at 2:21 PM, przemolicc@poczta.fm wrote:

On Wed, May 04, 2011 at 12:17:15PM -0400, Nicolas Ross wrote:

...

Hi !

With iostat, I find that it's almost a write i/o problem. How can I find to which files the OS writes ? On OSX boxes, there is a utility called fs_usage that can reports any disk activity for a particular process or all processes. Is there any utility like this on Centos ?

dunno if inotify et al helps your cause