Thank you for your advice. I know the issue depends on a lot of factors. Would you please give me some detail information about how to tune these parameters such as the size of cache,the type of cpu? I am not quite familiar with these detail.
At 2018-11-03 22:39:55, "Stephen John Smoogen" smooge@gmail.com wrote:
On Sat, 3 Nov 2018 at 04:17, yf chu cyflhn@163.com wrote:
Thank you for your hint. I really mean I am planning to store millions of files on the file system. Then may I ask that what is the maximum number of files which could be stored in one directory without affecting the performance of web server?
There is no simple answer to that. It will depend on everything from the physical drives used, the hardware that connects the motherboard to the drives, the size of the cache and type of CPU on the system, any low level filesystem items (software/hardware raid, type of raid, redundancy of the raid, etc), the type of the file system, the size of the files, the layout of directory structure, and the metadata connected to those files and needing to be checked.
Any one of those can severely affect partially performance of the web-server, and multiple combinations of them can severely affect it. This means a lot of benchmarking for the hardware and os are needed to get an idea if any of the tuning of number of files per directory will make things better or not. I have seen many systems where the hardware worked better with a certain type of RAID and it didn't matter if you had 10,000 or 100 files in each directory.. the changes in performance were minimal but moving from RAID10 to RAID6 or vice versa sped things up much more.. or adding more cache to the hardware controller etc etc.
Assuming you have tuned all of that, then the number of files in the directory comes down to a 'gut' check. I have seen some people do some sort of power of 2 per directory but rarely go over 1024. if you do a 3 level double hex tree <[0-f][0-f]>/<[0-f][0-f]>/<[0-f][0-f]>/ and lay them out using some sort of file hash method.. you can easily sit 256 files in each directory and have 2^32 files.. You will probably end up with some hot spots depending on the hash method so it would be good to test that first.
At 2018-11-03 16:03:56, "Walter H." Walter.H@mathemainzel.info wrote:
On 03.11.2018 08:44, yf chu wrote:
I have a website with millions of pages.
does 'millions of pages' also mean 'millions of files on the file system'?
just a hint - has nothing to do with any file system as its universal: e.g. when you have 10000 files don't store them in one folder, create 100 folders with 100 files in each;
there is no file system that handles millions of files in one folder or with limited resources (e.g. RAM)
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
-- Stephen J Smoogen. _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos