Hi
I need to store about 250.000.000 files. Files are less than 4k.
On a ext4 (fedora 14) the system crawl at 10.000.000 in the same directory.
I tried to create hash directories, two level of 4096 dir = 16.000.000 but I had to stop the script to create these dir after hours and "rm -rf" would have taken days ! mkfs was my friend
I tried two levels, first of 4096 dir, second of 64 dir. The creation of the hash dir took "only" few minutes, but copying 10000 files make my HD scream for 120s ! I take only 10s when working in the same directory.
The filenames are all 27 chars and the first chars can be used to hash the files.
My question is : Which filesystem and how to store these files ?
Regards
I haven't tried it but could you possibly use a database to hold all those files instead? At less than 4K per "row", performance from an indexed database might be faster.
On 3/12/11, Alain Spineux aspineux@gmail.com wrote:
Hi
I need to store about 250.000.000 files. Files are less than 4k.
On a ext4 (fedora 14) the system crawl at 10.000.000 in the same directory.
I tried to create hash directories, two level of 4096 dir = 16.000.000 but I had to stop the script to create these dir after hours and "rm -rf" would have taken days ! mkfs was my friend
I tried two levels, first of 4096 dir, second of 64 dir. The creation of the hash dir took "only" few minutes, but copying 10000 files make my HD scream for 120s ! I take only 10s when working in the same directory.
The filenames are all 27 chars and the first chars can be used to hash the files.
My question is : Which filesystem and how to store these files ?
Regards
-- Alain Spineux | aspineux gmail com Monitor your iT & Backups | http://www.magikmon.com Free Backup front-end | http://www.magikmon.com/mksbackup Your email 100% available | http://www.emailgency.com _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Hi
I need to store about 250.000.000 files. Files are less than 4k.
On a ext4 (fedora 14) the system crawl at 10.000.000 in the same directory.
I tried to create hash directories, two level of 4096 dir = 16.000.000 but I had to stop the script to create these dir after hours and "rm -rf" would have taken days ! mkfs was my friend
I tried two levels, first of 4096 dir, second of 64 dir. The creation of the hash dir took "only" few minutes, but copying 10000 files make my HD scream for 120s ! I take only 10s when working in the same directory.
The filenames are all 27 chars and the first chars can be used to hash the files.
My question is : Which filesystem and how to store these files ?
Did you try XFS? Deletes may be slow but apart from that it did a nice jobs when I last used it. But we had only around 50.000.000 files at the time. However, also ext3 worked quite well after *removing* dir_index.
Also, did you run a x86_64 kernel? We were having all kind of troubles with big boxes and i686-PAE kernel, because direntry and inode caches were very small.
Simon
Thanks to everybody for answering.
I thing >250E6 is a lot and keep decent read and write access speed is unreal using mutli-purpose filesystems like ext? and other ?FS. I would need a dedicated filesystem for that.
This problem was only a possible solution to another problem. I will solve the original problem using another way.
Regards
On Sat, Mar 12, 2011 at 3:12 PM, Simon Matter simon.matter@invoca.ch wrote:
Hi
I need to store about 250.000.000 files. Files are less than 4k.
On a ext4 (fedora 14) the system crawl at 10.000.000 in the same directory.
I tried to create hash directories, two level of 4096 dir = 16.000.000 but I had to stop the script to create these dir after hours and "rm -rf" would have taken days ! mkfs was my friend
I tried two levels, first of 4096 dir, second of 64 dir. The creation of the hash dir took "only" few minutes, but copying 10000 files make my HD scream for 120s ! I take only 10s when working in the same directory.
The filenames are all 27 chars and the first chars can be used to hash the files.
My question is : Which filesystem and how to store these files ?
Did you try XFS? Deletes may be slow but apart from that it did a nice jobs when I last used it. But we had only around 50.000.000 files at the time. However, also ext3 worked quite well after *removing* dir_index.
Also, did you run a x86_64 kernel? We were having all kind of troubles with big boxes and i686-PAE kernel, because direntry and inode caches were very small.
Simon
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Sat, Mar 12, 2011 at 4:04 PM, Alain Spineux aspineux@gmail.com wrote:
Thanks to everybody for answering.
I thing >250E6 is a lot and keep decent read and write access speed is unreal using mutli-purpose filesystems like ext? and other ?FS. I would need a dedicated filesystem for that.
This problem was only a possible solution to another problem. I will solve the original problem using another way.
Why, exactly, are you doing this? The normal approach for such dense repositories is to create a hierarchy of subdirectories.
File aaa12345 goes in
$DIR/a/a/a/12345
File abc6789 goes in
$DIR/a/b/c/6789
And whatever is accessing or creating the files is taught the algorithm used. This requires some programming up front, but helps prevent precisely the outrageous directory size you describe. Handling, sorting, and reporting on that many files in one directory is an old and painful problem.
On Sat, Mar 12, 2011 at 10:14 PM, Nico Kadel-Garcia nkadel@gmail.com wrote:
On Sat, Mar 12, 2011 at 4:04 PM, Alain Spineux aspineux@gmail.com wrote:
Thanks to everybody for answering.
I thing >250E6 is a lot and keep decent read and write access speed is unreal using mutli-purpose filesystems like ext? and other ?FS. I would need a dedicated filesystem for that.
This problem was only a possible solution to another problem. I will solve the original problem using another way.
Why, exactly, are you doing this? The normal approach for such dense repositories is to create a hierarchy of subdirectories.
File aaa12345 goes in
$DIR/a/a/a/12345
File abc6789 goes in
$DIR/a/b/c/6789
Try to create this king of tree yourself and when done, remove it. I took hours on my box and it is even faster to keep all files in the same diretory. I looks like working in multiple directories slow down the process also
I read other posts and articles and handle more than 100M files become a problem ! 256M is a problem and more than 1G files is a big problem.
I was splitting data into files to help me. I will keep them in big files.
Regards
Anyway it was too slow.
And whatever is accessing or creating the files is taught the algorithm used. This requires some programming up front, but helps prevent precisely the outrageous directory size you describe. Handling, sorting, and reporting on that many files in one directory is an old and painful problem. _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On 3/14/2011 12:33 PM, Alain Spineux wrote:
File aaa12345 goes in
$DIR/a/a/a/12345
File abc6789 goes in
$DIR/a/b/c/6789Try to create this king of tree yourself and when done, remove it. I took hours on my box and it is even faster to keep all files in the same diretory. I looks like working in multiple directories slow down the process also
Normally, directory/inode caching would help but you are probably exceeding any reasonable caching attempt and making it thresh.
I read other posts and articles and handle more than 100M files become a problem ! 256M is a problem and more than 1G files is a big problem.
I was splitting data into files to help me. I will keep them in big files.
Depending on your access needs you might investigate some of the scalable nosql databases like riak or cassandra. These would let you distribute the data (and access contention) across a cluster of machines. Riak has an extension called luwak that handles large data streams by chunking into smaller key/value sets which are then distributed over the cluster. The way implementation works, you also get de-duplication of chunks - with the down side that you can't really delete anything. And riak in general wants to keep all the keys in RAM which might be an issue for your use unless you can spread it over several machines.
on 13:10 Sat 12 Mar, Alain Spineux (aspineux@gmail.com) wrote:
Hi
I need to store about 250.000.000 files. Files are less than 4k.
On a ext4 (fedora 14) the system crawl at 10.000.000 in the same directory.
I tried to create hash directories, two level of 4096 dir = 16.000.000 but I had to stop the script to create these dir after hours and "rm -rf" would have taken days ! mkfs was my friend
I tried two levels, first of 4096 dir, second of 64 dir. The creation of the hash dir took "only" few minutes, but copying 10000 files make my HD scream for 120s ! I take only 10s when working in the same directory.
The filenames are all 27 chars and the first chars can be used to hash the files.
My question is : Which filesystem and how to store these files ?
I'd also question the architecture and suggest an alternate approach: hierarchical directory tree, database, "nosql" hashing lookup, or other approach. See squid for an example of using directory trees to handle very large numbers of objects. In fact, if you wired things up right, you could probably use squid as a proxy back-end.
In general, I'd say a filesystem is the wrong approach to this problem.
What's the creation/deletion/update/lifecycle of these objects? Are they all created at once? A few at a time? Are they ever updated? Are they expired and/or deleted?
Otherwise, reiserfs and its hashed directory indexes scales well, though I've only pushed it to about 125,000 entries in a single node. There is the usual comment about viability of a filesystem whose principle architect is in jail on a murder charge.
It's possible XFS/JFS might also work. I'd suggest you test building and deleting large directories.
Incidentally, for testing, 'make -J' can be useful for parallelizing processing, which would also test whether or not locking/contention on the directory entry itself is going to be a bottleneck (I suspect it may be).
You might also find that GNU 'find's "-depth" argument is useful for deleting deep/large trees.
On Mon, 2011-03-14 at 13:10 -0700, Dr. Ed Morbius wrote:
on 13:10 Sat 12 Mar, Alain Spineux (aspineux@gmail.com) wrote:
Hi I need to store about 250.000.000 files. Files are less than 4k. On a ext4 (fedora 14) the system crawl at 10.000.000 in the same directory. I tried to create hash directories, two level of 4096 dir = 16.000.000 but I had to stop the script to create these dir after hours and "rm -rf" would have taken days ! mkfs was my friend I tried two levels, first of 4096 dir, second of 64 dir. The creation of the hash dir took "only" few minutes, but copying 10000 files make my HD scream for 120s ! I take only 10s when working in the same directory. The filenames are all 27 chars and the first chars can be used to hash the files.
Exactly {XY}/{XY}/{ABCDEFGHIJKLMNOPQRSTUVW} will probably work just fine. Two characters is 676 combinations, hardly a large directory, and that puts less than 1,000 entries in a folder.
My question is : Which filesystem and how to store these files ?
I'd also question the architecture and suggest an alternate approach: hierarchical directory tree, database, "nosql" hashing lookup, or other approach. See squid for an example of using directory trees to handle very large numbers of objects.
Exactly. Squid and Cyrus IMAPd both manage to store massive number of objects in a filesystem using hashing. It is simple and reliable.
I'd wonder if that is really an issue if you just don't have an actual I/O throughput problem. Certainly trying to do any solution on a single-disk is going to be awful.
-
Adam Tauno Williams -
Alain Spineux -
Dr. Ed Morbius -
Emmanuel Noobadmin -
Les Mikesell -
Nico Kadel-Garcia -
Simon Matter