[CentOS] Question about optimal filesystem with many small files.

Mon Jul 13 05:49:11 UTC 2009
oooooooooooo ooooooooooooo <hhh735 at hotmail.com>

>How many files per directory do you have?

I have 4 directory levels, 65536 leaves directories and around 200 files per dir (15M in total)- 
 
>Something is wrong. Got to figure this out.  Where did this RAM go?

Thanks I reduced the memory usage of mysql and my app it and I got around a 15% performance increase. Now my atop looks like this (currently reading only cached files from disk).

PRC | sys   0.51s | user   9.29s | #proc    114 | #zombie    0 | #exit      0 |
CPU | sys      4% | user     93% | irq       1% | idle    208% | wait     94% |
cpu | sys      2% | user     48% | irq       1% | idle     21% | cpu001 w 28% |
cpu | sys      1% | user     17% | irq       0% | idle     41% | cpu000 w 40% |
cpu | sys      1% | user     14% | irq       0% | idle     74% | cpu003 w 12% |
cpu | sys      1% | user     13% | irq       0% | idle     72% | cpu002 w 14% |
CPL | avg1   3.45 | avg5    7.42 | avg15  10.76 | csw    15891 | intr   11695 |
MEM | tot    2.0G | free   51.2M | cache 587.8M | buff    1.0M | slab  281.2M |
SWP | tot    1.9G | free    1.9G |              | vmcom   1.6G | vmlim   2.9G |
PAG | scan   3072 | stall      0 |              | swin       0 | swout      0 |
DSK |         sdb | busy     89% | read    1451 | write      0 | avio    6 ms |
DSK |         sda | busy      6% | read     178 | write     54 | avio    2 ms |
NET | transport   | tcpi    3631 | tcpo    3629 | udpi       0 | udpo       0 |
NET | network     | ipi     3632 | ipo     3630 | ipfrw      0 | deliv   3632 |
NET | eth0     0% | pcki       5 | pcko       3 | si    0 Kbps | so    1 Kbps |
NET | lo     ---- | pcki    3627 | pcko    3627 | si  775 Kbps | so  775 Kbps |

>It is 1024 chars long. Witch want still help.
I'm usng mysam and according to: http://dev.mysql.com/doc/refman/5.1/en/myisam-storage-engine.html
"The maximum key length is 1000 bytes. This can also be changed by changing the source and recompiling. For the case of a key longer than 250 bytes, a larger key block size than the default of 1024 bytes is used. "

>I would not store images in either one
as your SELECT LIKE and Random will kill it. 

Well, I think that this can be avoided, using just searches in teh key fields should not give these issues. Does somebody have experience storing a large amount of medium (1KB-150KB) blob objects in mysql?

>However I have not a clue that this is even doable in MySQL.

In mysql there is already a MD5 funtion: http://dev.mysql.com/doc/refman/5.1/en/encryption-functions.html#function_md5

Thanks for the help.

_________________________________________________________________
Connect to the next generation of MSN Messenger 
http://imagine-msn.com/messenger/launch80/default.aspx?locale=en-us&source=wlmailtagline