[CentOS] wich filesystem to store > 250E6 small files in same or hashed dire
aspineux at gmail.com
Mon Mar 14 17:33:15 UTC 2011
On Sat, Mar 12, 2011 at 10:14 PM, Nico Kadel-Garcia <nkadel at gmail.com> wrote:
> On Sat, Mar 12, 2011 at 4:04 PM, Alain Spineux <aspineux at gmail.com> wrote:
>> Thanks to everybody for answering.
>> I thing >250E6 is a lot and keep decent read and write access speed is unreal
>> using mutli-purpose filesystems like ext? and other ?FS.
>> I would need a dedicated filesystem for that.
>> This problem was only a possible solution to another problem.
>> I will solve the original problem using another way.
> Why, exactly, are you doing this? The normal approach for such dense
> repositories is to create a hierarchy of subdirectories.
> File aaa12345 goes in
> File abc6789 goes in
Try to create this king of tree yourself and when done, remove it.
I took hours on my box and it is even faster to keep all files in the
I looks like working in multiple directories slow down the process also
I read other posts and articles and handle more than 100M files become
a problem !
256M is a problem and more than 1G files is a big problem.
I was splitting data into files to help me. I will keep them in big files.
Anyway it was too slow.
> And whatever is accessing or creating the files is taught the
> algorithm used. This requires some programming up front, but helps
> prevent precisely the outrageous directory size you describe.
> Handling, sorting, and reporting on that many files in one directory
> is an old and painful problem.
> CentOS mailing list
> CentOS at centos.org
Alain Spineux | aspineux gmail com
Monitor your iT & Backups | http://www.magikmon.com
Free Backup front-end | http://www.magikmon.com/mksbackup
Your email 100% available | http://www.emailgency.com
More information about the CentOS