[CentOS] wich filesystem to store > 250E6 small files in same or hashed dire

Alain Spineux aspineux at gmail.com
Mon Mar 14 17:33:15 UTC 2011

On Sat, Mar 12, 2011 at 10:14 PM, Nico Kadel-Garcia <nkadel at gmail.com> wrote:
> On Sat, Mar 12, 2011 at 4:04 PM, Alain Spineux <aspineux at gmail.com> wrote:
>> Thanks to everybody for answering.
>> I thing >250E6 is a lot and keep decent read and write access speed is unreal
>> using mutli-purpose filesystems like ext? and other ?FS.
>> I would need a dedicated filesystem for that.
>> This problem was only a possible solution to another problem.
>> I will solve the original problem using another way.
> Why, exactly, are you doing this? The normal approach for such dense
> repositories is to create a hierarchy of subdirectories.
> File aaa12345 goes in
>   $DIR/a/a/a/12345
> File abc6789 goes in
>    $DIR/a/b/c/6789

Try to create this king of tree yourself and when done, remove it.
I took hours on my box and it is even faster to keep all files in the
same diretory.
I looks like working in multiple directories slow down the process also

I read other posts and articles and handle more than 100M files become
a problem !
256M is a problem and more than 1G files is a big problem.

I was splitting data into files to help me. I will keep them in big files.


Anyway it was too slow.

> And whatever is accessing or creating the files is taught the
> algorithm used. This requires some programming up front, but helps
> prevent precisely the outrageous directory size you describe.
> Handling, sorting, and reporting on that many files in one directory
> is an old and painful problem.
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos

Alain Spineux                   |  aspineux gmail com
Monitor your iT & Backups |  http://www.magikmon.com
Free Backup front-end       | http://www.magikmon.com/mksbackup
Your email 100% available |  http://www.emailgency.com

More information about the CentOS mailing list