On Sat, Mar 12, 2011 at 10:14 PM, Nico Kadel-Garcia nkadel@gmail.com wrote:
On Sat, Mar 12, 2011 at 4:04 PM, Alain Spineux aspineux@gmail.com wrote:
Thanks to everybody for answering.
I thing >250E6 is a lot and keep decent read and write access speed is unreal using mutli-purpose filesystems like ext? and other ?FS. I would need a dedicated filesystem for that.
This problem was only a possible solution to another problem. I will solve the original problem using another way.
Why, exactly, are you doing this? The normal approach for such dense repositories is to create a hierarchy of subdirectories.
File aaa12345 goes in
$DIR/a/a/a/12345
File abc6789 goes in
$DIR/a/b/c/6789
Try to create this king of tree yourself and when done, remove it. I took hours on my box and it is even faster to keep all files in the same diretory. I looks like working in multiple directories slow down the process also
I read other posts and articles and handle more than 100M files become a problem ! 256M is a problem and more than 1G files is a big problem.
I was splitting data into files to help me. I will keep them in big files.
Regards
Anyway it was too slow.
And whatever is accessing or creating the files is taught the algorithm used. This requires some programming up front, but helps prevent precisely the outrageous directory size you describe. Handling, sorting, and reporting on that many files in one directory is an old and painful problem. _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos