[CentOS] GFS and Small Files

Wed Apr 29 11:01:17 UTC 2009
Hairul Ikmal Mohamad Fuzi <hairul.ikmal at gmail.com>

Hi all,

We are running CentOS 5.2 64bit as our file server.
Currently, we used GFS (with CLVM underneath it) as our filesystem
(for our multiple 2TB SAN volume exports) since we plan to add more
file servers (serving the same contents) later on.

The issue we are facing at the moment is we found out that command
such as 'ls' gives a very slow response.(e.g 3-4minutes for the
outputs of ls to be printed out, or in certain cases, 20minutes or so)
This is completely true especially in directories containing "large
number" of small files (e.g 90000+ of 1-4kb files). The thing is, most
of system users are generating these small files frequently as part of
their workflow.

We tried emulating the same scenario (90000+ of small files) on a ext3
partition and it gives almost the same result.

I believe most of the CLVM/GFS settings done are using the defaults
parameters. Additionally, we would prefer to stick to GFS (or at least
ext3) as it is part of CentOS / RHEL distribution rather than changing
into other small-files 'friendly' filesystems (such as XFS, ReiserFS).

I'm exploring whether is there anyway we can tune the GFS parameters
to make the system more responsive?
I have read that we can apply 'dir_index' option to ext3 partition to
speedup things, but I'm not so sure about GFS.

Below are the output from "gfs_tool gettune /export/gfs" :

ilimit1 = 100
ilimit1_tries = 3
ilimit1_min = 1
ilimit2 = 500
ilimit2_tries = 10
ilimit2_min = 3
demote_secs = 300
incore_log_blocks = 1024
jindex_refresh_secs = 60
depend_secs = 60
scand_secs = 5
recoverd_secs = 60
logd_secs = 1
quotad_secs = 5
inoded_secs = 15
glock_purge = 0
quota_simul_sync = 64
quota_warn_period = 10
atime_quantum = 3600
quota_quantum = 60
quota_scale = 1.0000   (1, 1)
quota_enforce = 1
quota_account = 1
new_files_jdata = 0
new_files_directio = 0
max_atomic_write = 4194304
max_readahead = 262144
lockdump_size = 131072
stall_secs = 600
complain_secs = 10
reclaim_limit = 5000
entries_per_readdir = 32
prefetch_secs = 10
statfs_slots = 64
max_mhc = 10000
greedy_default = 100
greedy_quantum = 25
greedy_max = 250
rgrp_try_threshold = 100
statfs_fast = 0


TIA.

.ikmal