[CentOS] GFS and Small Files

Wed Apr 29 19:35:18 UTC 2009
Filipe Brandenburger <filbranden at gmail.com>

Hi,

On Wed, Apr 29, 2009 at 08:35, William L. Maltby
<CentOS4Bill at triad.rr.com> wrote:
> One thing to keep in mind is that ls must sort the file list.

Not only sorting, but usually "ls" ends up trying to find out if the
file is a directory, which uses a "stat" syscall for each of the
files.

This is always expensive on remote filesystems (e.g., NFS) and I would
expect it to be also the case in GFS.

Besides the "-U" option of "ls", you might want to look at other
options that would make "ls" not try to "stat" each file. I'm not sure
exactly which of them, but one I can think of is disabling color
(since color will trigger a "stat" to find if the file is a
directory).

In general, having directories with a huge number of files tends to be
a bad idea, you will most likely have performance bottlenecks with
specific filesystems or tools. If possible, try to change the
application to create two or three levels of directories using a hash
on the filename and then creating directories with a small number of
files on each of them.

HTH,
Filipe