[CentOS] inquiry about limitation of file system

[yf chu]

Thank you for your advice. I know the issue depends on  a lot of factors. Would you please give me some detail information about how to tune these parameters such as the size of cache,the type of cpu? I am not quite familiar with these detail.








At 2018-11-03 22:39:55, "Stephen John Smoogen" <smooge at gmail.com> wrote:
>On Sat, 3 Nov 2018 at 04:17, yf chu <cyflhn at 163.com> wrote:
>>
>> Thank you for your hint.
>> I really mean I am planning to  store  millions of files on the file system.
>> Then may I ask that what is the maximum number of files which could be stored in one directory without affecting the performance of web server?
>>
>>
>
>There is no simple answer to that. It will depend on everything from
>the physical drives used, the hardware that connects the motherboard
>to the drives, the size of the cache and type of CPU on the system,
>any low level filesystem items (software/hardware raid, type of raid,
>redundancy of the raid, etc), the type of the file system, the size of
>the files, the layout of directory structure, and the metadata
>connected to those files and needing to be checked.
>
>Any one of those can severely affect partially performance of the
>web-server, and multiple combinations of them can severely affect it.
>This means a lot of benchmarking for the hardware and os are needed to
>get an idea if any of the tuning of number of files per directory will
>make things better or not. I have seen many systems where the hardware
>worked better with a certain type of RAID and it didn't matter if you
>had 10,000 or 100 files in each directory.. the changes in performance
>were minimal but moving from RAID10 to RAID6 or vice versa sped things
>up much more.. or adding more cache to the hardware controller etc
>etc.
>
>Assuming you have tuned all of that, then the number of files in the
>directory comes down to a 'gut' check. I have seen some people do some
>sort of power of 2 per directory but rarely go over 1024. if you do a
>3 level double hex tree <[0-f][0-f]>/<[0-f][0-f]>/<[0-f][0-f]>/ and
>lay them out using some sort of file hash method.. you can easily sit
>256 files in each directory and have 2^32 files.. You will probably
>end up with some hot spots depending on the hash method so it would be
>good to test that first.
>
>>
>>
>>
>>
>>
>>
>> At 2018-11-03 16:03:56, "Walter H." <Walter.H at mathemainzel.info> wrote:
>> >On 03.11.2018 08:44, yf chu wrote:
>> >> I have a website with millions of pages.
>> >>
>> >does 'millions of pages' also mean 'millions of files on the file system'?
>> >
>> >just a hint - has nothing to do with any file system as its universal:
>> >e.g. when you have 10000 files
>> >don't store them in one folder, create 100 folders with 100 files in each;
>> >
>> >there is no file system that handles millions of files in one folder
>> >or with limited resources (e.g. RAM)
>> >
>> >_______________________________________________
>> >CentOS mailing list
>> >CentOS at centos.org
>> >https://lists.centos.org/mailman/listinfo/centos
>> _______________________________________________
>> CentOS mailing list
>> CentOS at centos.org
>> https://lists.centos.org/mailman/listinfo/centos
>
>
>
>-- 
>Stephen J Smoogen.
>_______________________________________________
>CentOS mailing list
>CentOS at centos.org
>https://lists.centos.org/mailman/listinfo/centos