We use ext3 for maildir. I have not had any issues to date. This is also on fibre SAN drives, not ATA.
On Nov 26, 2007, at 12:22 PM, Heitor Augusto M Cardozo wrote:
Hi all,
In last year, i had made some research and benchmarks based on CentOS 4 to know which filesystem is better for Maildir: ReiserFS, XFS or EXT3.
My conclusion was as follows:
- EXT3: reliable but very slow to read many small files.
- ReiserFS: best performance but unreliable and bad recovery tools.
- XFS: My choice, good performance and reliability.
On CentOS 5.0, a had the same benchmarks and now, EXT3 and XFS seems to had better or equivalent performance on Read and Create Random files. One of this tests, using bonnie++, show this:
# bonnie++ -d /mnt/sdc1/testfile -s 8192 -m `hostname` -n 50:150000:5000:1000
XFS: Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- -- Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP localhost 8G 74048 99 201584 32 74014 12 61610 92 228977 18 623.5 0 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- -- Read--- -Delete-- files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 50:158999:5125/1000 692 12 82 0 3502 28 529 7 107 0 1149 11 localhost,8G, 74048,99,201584,32,74014,12,61610,92,228977,18,623.5,0,50 :158999:5125/1000,692,12,82,0,3502,28,529,7,107,0,1149,11
EXT3: Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- -- Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP localhost 8G 54569 99 249788 68 75268 11 59128 91 211926 15 587.2 0 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- -- Read--- -Delete-- files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 50:158999:5125/1000 748 15 256 1 1081 5 713 15 108 0 207 1 localhost,8G, 54569,99,249788,68,75268,11,59128,91,211926,15,587.2,0,50 :158999:5125/1000,748,15,256,1,1081,5,713,15,108,0,207,1
What i want to know is: Anyone use or recommend EXT3 for Maildir?
My configuration: 3Ware 9650SE-8LPML, 8 drives SATA2 ST3500630AS 500GB on RAID 10.
Regards,
Heitor A.M. Cardozo _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
on 11/26/2007 11:22 AM Heitor Augusto M Cardozo spake the following:
Hi all,
In last year, i had made some research and benchmarks based on CentOS 4 to know which filesystem is better for Maildir: ReiserFS, XFS or EXT3.
My conclusion was as follows:
- EXT3: reliable but very slow to read many small files.
- ReiserFS: best performance but unreliable and bad recovery tools.
- XFS: My choice, good performance and reliability.
On CentOS 5.0, a had the same benchmarks and now, EXT3 and XFS seems to had better or equivalent performance on Read and Create Random files. One of this tests, using bonnie++, show this:
# bonnie++ -d /mnt/sdc1/testfile -s 8192 -m `hostname` -n 50:150000:5000:1000
XFS: Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP localhost 8G 74048 99 201584 32 74014 12 61610 92 228977 18 623.5 0 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 50:158999:5125/1000 692 12 82 0 3502 28 529 7 107 0 1149 11 localhost,8G,74048,99,201584,32,74014,12,61610,92,228977,18,623.5,0,50:158999:5125/1000,692,12,82,0,3502,28,529,7,107,0,1149,11
EXT3: Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP localhost 8G 54569 99 249788 68 75268 11 59128 91 211926 15 587.2 0 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 50:158999:5125/1000 748 15 256 1 1081 5 713 15 108 0 207 1 localhost,8G,54569,99,249788,68,75268,11,59128,91,211926,15,587.2,0,50:158999:5125/1000,748,15,256,1,1081,5,713,15,108,0,207,1
What i want to know is: Anyone use or recommend EXT3 for Maildir?
My configuration: 3Ware 9650SE-8LPML, 8 drives SATA2 ST3500630AS 500GB on RAID 10.
Ext3 is pretty good if directory indexes are on.; See http://wiki.archlinux.org/index.php/Ext3_Filesystem_Tips
Scott Silva wrote:
Ext3 is pretty good if directory indexes are on.; See http://wiki.archlinux.org/index.php/Ext3_Filesystem_Tips
and, disable 'access time', which is almost always a good idea.
Scott Silva schrieb:
What i want to know is: Anyone use or recommend EXT3 for Maildir?
My configuration: 3Ware 9650SE-8LPML, 8 drives SATA2 ST3500630AS 500GB on RAID 10.
Ext3 is pretty good if directory indexes are on.; See http://wiki.archlinux.org/index.php/Ext3_Filesystem_Tips
If the ext3 fs was made with anaconda it has this feature anyway (CMIIW since centos 4.4 and 5.0).
You can check with:
# tune2fs -l /dev/mapper/VG00-LVvar|grep features Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
Rainer
Heitor Augusto M Cardozo wrote:
Hi all,
In last year, i had made some research and benchmarks based on CentOS 4 to know which filesystem is better for Maildir: ReiserFS, XFS or EXT3.
My conclusion was as follows:
- EXT3: reliable but very slow to read many small files.
- ReiserFS: best performance but unreliable and bad recovery tools.
- XFS: My choice, good performance and reliability.
I would contest the last two.
On CentOS 5.0, a had the same benchmarks and now, EXT3 and XFS seems to had better or equivalent performance on Read and Create Random files. One of this tests, using bonnie++, show this:
# bonnie++ -d /mnt/sdc1/testfile -s 8192 -m `hostname` -n 50:150000:5000:1000
bonnie++? Not appropriate. Try this: http://untroubled.org/benchmarking/2004-04/
And add JFS to the mix. You will be surprised.
What i want to know is: Anyone use or recommend EXT3 for Maildir?
If you do not have full blown battery back for write caches yes.
My configuration: 3Ware 9650SE-8LPML, 8 drives SATA2 ST3500630AS 500GB on RAID 10.
Add BBU and XFS or JFS should do.
Christopher Chan wrote:
Heitor Augusto M Cardozo wrote:
Hi all,
In last year, i had made some research and benchmarks based on CentOS 4 to know which filesystem is better for Maildir: ReiserFS, XFS or EXT3.
My conclusion was as follows:
- EXT3: reliable but very slow to read many small files.
- ReiserFS: best performance but unreliable and bad recovery tools.
- XFS: My choice, good performance and reliability.
I would contest the last two.
I had two bad experiences with ReiserFS in our Mail Server, reiserfsck is too slow and lost data.
IMHO ReiserFS have the best performance for Maildir but its only safe on production if you´re sure that the system I/O will never fail.
On CentOS 5.0, a had the same benchmarks and now, EXT3 and XFS seems to had better or equivalent performance on Read and Create Random files. One of this tests, using bonnie++, show this:
# bonnie++ -d /mnt/sdc1/testfile -s 8192 -m `hostname` -n 50:150000:5000:1000
bonnie++? Not appropriate. Try this: http://untroubled.org/benchmarking/2004-04/
And add JFS to the mix. You will be surprised.
I already done tests with fsbench and the results on CentOS 4.5 were equivalent: the performance of XFS was much higher than EXT3.
Then, i retest using fsbench, bonnie++ and iozone on CentOS 5.0, and the results now show the EXT3 (dir_index, noatime) with performance similar to XFS.
What i want to know is: Anyone use or recommend EXT3 for Maildir?
If you do not have full blown battery back for write caches yes.
My configuration: 3Ware 9650SE-8LPML, 8 drives SATA2 ST3500630AS 500GB on RAID 10.
Add BBU and XFS or JFS should do.
Yes, the BBU is installed and write_cache is enable. I will test JFS to compare.
Thanks for your help.
Heitor A. M. Cardozo
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Tue, Nov 27, 2007 at 11:28:58AM -0200, Heitor Augusto M Cardozo wrote:
- EXT3: reliable but very slow to read many small files.
- ReiserFS: best performance but unreliable and bad recovery tools.
- XFS: My choice, good performance and reliability.
I would contest the last two.
I had two bad experiences with ReiserFS in our Mail Server, reiserfsck is too slow and lost data.
IMHO ReiserFS have the best performance for Maildir but its only safe on production if you´re sure that the system I/O will never fail.
nullfs (mount -t nullfs /dev/null /var/spool/mail) is even faster than ReiserFS, and is just slightly more likely to loose your data. :)
[]s
- -- Rodrigo Barbosa "Quid quid Latine dictum sit, altum viditur" "Be excellent to each other ..." - Bill & Ted (Wyld Stallyns)
on 11/27/2007 10:41 AM Rodrigo Barbosa spake the following:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Tue, Nov 27, 2007 at 11:28:58AM -0200, Heitor Augusto M Cardozo wrote:
- EXT3: reliable but very slow to read many small files.
- ReiserFS: best performance but unreliable and bad recovery tools.
- XFS: My choice, good performance and reliability.
I would contest the last two.
I had two bad experiences with ReiserFS in our Mail Server, reiserfsck is too slow and lost data.
IMHO ReiserFS have the best performance for Maildir but its only safe on production if you´re sure that the system I/O will never fail.
nullfs (mount -t nullfs /dev/null /var/spool/mail) is even faster than ReiserFS, and is just slightly more likely to loose your data. :)
But it is way more predictable at WHEN it will lose data.
Rodrigo Barbosa wrote:
On Tue, Nov 27, 2007 at 11:28:58AM -0200, Heitor Augusto M Cardozo wrote:
- EXT3: reliable but very slow to read many small files.
- ReiserFS: best performance but unreliable and bad recovery tools.
- XFS: My choice, good performance and reliability.
I would contest the last two.
I had two bad experiences with ReiserFS in our Mail Server, reiserfsck is too slow and lost data.
IMHO ReiserFS have the best performance for Maildir but its only safe on production if you´re sure that the system I/O will never fail.
nullfs (mount -t nullfs /dev/null /var/spool/mail) is even faster than ReiserFS, and is just slightly more likely to loose your data. :)
I have /var on reiserfs on a busy RH 7.3 machine that hasn't had problems in however many years that has been (2001?). It wasn't rebooted in the last 4 of those years until just recently when I had to move it to a new location, though - but it's on raid1 and I've hot-swapped replacement drives into it.
Heitor Augusto M Cardozo wrote:
Christopher Chan wrote:
Heitor Augusto M Cardozo wrote:
Hi all,
In last year, i had made some research and benchmarks based on CentOS 4 to know which filesystem is better for Maildir: ReiserFS, XFS or EXT3.
My conclusion was as follows:
- EXT3: reliable but very slow to read many small files.
- ReiserFS: best performance but unreliable and bad recovery tools.
- XFS: My choice, good performance and reliability.
I would contest the last two.
I had two bad experiences with ReiserFS in our Mail Server, reiserfsck is too slow and lost data.
Well, not on that...reiserfs assumes perfect media according to the complaints of some who use reiser so bad blocks could cause even the entire loss of the filesystem...not to mention that RH and therefore Centos (I wonder about plus...) does not support/maintain reiserfs.
IMHO ReiserFS have the best performance for Maildir but its only safe on production if you´re sure that the system I/O will never fail.
What does fsbench say? It has the best writing performance too?!?
On CentOS 5.0, a had the same benchmarks and now, EXT3 and XFS seems to had better or equivalent performance on Read and Create Random files. One of this tests, using bonnie++, show this:
# bonnie++ -d /mnt/sdc1/testfile -s 8192 -m `hostname` -n 50:150000:5000:1000
bonnie++? Not appropriate. Try this: http://untroubled.org/benchmarking/2004-04/
And add JFS to the mix. You will be surprised.
I already done tests with fsbench and the results on CentOS 4.5 were equivalent: the performance of XFS was much higher than EXT3.
Then, i retest using fsbench, bonnie++ and iozone on CentOS 5.0, and the results now show the EXT3 (dir_index, noatime) with performance similar to XFS.
Now that is very interesting.
What i want to know is: Anyone use or recommend EXT3 for Maildir?
If you do not have full blown battery back for write caches yes.
My configuration: 3Ware 9650SE-8LPML, 8 drives SATA2 ST3500630AS 500GB on RAID 10.
Add BBU and XFS or JFS should do.
Yes, the BBU is installed and write_cache is enable. I will test JFS to compare.
Thanks for your help.
Please post your findings. :-)
Christopher
Christopher Chan wrote:
Heitor Augusto M Cardozo wrote:
Christopher Chan wrote:
Heitor Augusto M Cardozo wrote:
Hi all,
In last year, i had made some research and benchmarks based on CentOS 4 to know which filesystem is better for Maildir: ReiserFS, XFS or EXT3.
My conclusion was as follows:
- EXT3: reliable but very slow to read many small files.
- ReiserFS: best performance but unreliable and bad recovery tools.
- XFS: My choice, good performance and reliability.
I would contest the last two.
I had two bad experiences with ReiserFS in our Mail Server, reiserfsck is too slow and lost data.
Well, not on that...reiserfs assumes perfect media according to the complaints of some who use reiser so bad blocks could cause even the entire loss of the filesystem...not to mention that RH and therefore Centos (I wonder about plus...) does not support/maintain reiserfs.
IMHO ReiserFS have the best performance for Maildir but its only safe on production if you´re sure that the system I/O will never fail.
What does fsbench say? It has the best writing performance too?!?
No, according to the fsbench results, ReiserFS wins on Read Performance, but XFS is, approximately, four times more faster on write.
I said that the ReiserFS have the best performance based on my read/write server statics, where read requests are 70% of total I/O requests.
In production, with ReiserFS, the server load average was around 30% lower than XFS.
On CentOS 5.0, a had the same benchmarks and now, EXT3 and XFS seems to had better or equivalent performance on Read and Create Random files. One of this tests, using bonnie++, show this:
# bonnie++ -d /mnt/sdc1/testfile -s 8192 -m `hostname` -n 50:150000:5000:1000
bonnie++? Not appropriate. Try this: http://untroubled.org/benchmarking/2004-04/
And add JFS to the mix. You will be surprised.
I already done tests with fsbench and the results on CentOS 4.5 were equivalent: the performance of XFS was much higher than EXT3.
Then, i retest using fsbench, bonnie++ and iozone on CentOS 5.0, and the results now show the EXT3 (dir_index, noatime) with performance similar to XFS.
Now that is very interesting.
What i want to know is: Anyone use or recommend EXT3 for Maildir?
If you do not have full blown battery back for write caches yes.
My configuration: 3Ware 9650SE-8LPML, 8 drives SATA2 ST3500630AS 500GB on RAID 10.
Add BBU and XFS or JFS should do.
Yes, the BBU is installed and write_cache is enable. I will test JFS to compare.
Thanks for your help.
Please post your findings. :-)
I'm doing new tests with ReiserFS, XFS, EXT3 and JFS in CentOS 5. I will post soon as possible.
And sorry for my english...
Heitor A.M. Cardozo
What does fsbench say? It has the best writing performance too?!?
No, according to the fsbench results, ReiserFS wins on Read Performance, but XFS is, approximately, four times more faster on write.
I said that the ReiserFS have the best performance based on my read/write server statics, where read requests are 70% of total I/O requests.
Ah. Too bad reiserfs is not stable enough for you.
In production, with ReiserFS, the server load average was around 30% lower than XFS.
I guess Hans got something right with his reiserfs.
Please post your findings. :-)
I'm doing new tests with ReiserFS, XFS, EXT3 and JFS in CentOS 5. I will post soon as possible.
Thank you very much in advance.
And sorry for my english...
No need to be and it is not bad at all.
On Wed, Nov 28, 2007, Christopher Chan wrote:
What does fsbench say? It has the best writing performance too?!?
No, according to the fsbench results, ReiserFS wins on Read Performance, but XFS is, approximately, four times more faster on write.
I said that the ReiserFS have the best performance based on my read/write server statics, where read requests are 70% of total I/O requests.
Ah. Too bad reiserfs is not stable enough for you.
I've lost several file systems to reiserfs, originally figuring that they were safe since SuSE has used them as their default for years.
We're using ext3 now as it appears to be rock-solid, is supported out of the box by every Linux I've used, and I've never lost one.
We haven't had any notable performance problems using this at a regional ISP customer's site with about 10,000 e-mail users and several machines in a cluster delivering mail to Maildir folders that are NFS mounted to the central server.
Bill -- INTERNET: bill@celestial.com Bill Campbell; Celestial Software LLC URL: http://www.celestial.com/ PO Box 820; 6641 E. Mercer Way FAX: (206) 232-9186 Mercer Island, WA 98040-0820; (206) 236-1676
Many companies that have made themselves dependent on [the equipment of a certain major manufacturer] (and in doing so have sold their soul to the devil) will collapse under the sheer weight of the unmastered complexity of their data processing systems. -- Edsger W. Dijkstra, SIGPLAN Notices, Volume 17, Number 5
Bill Campbell wrote:
On Wed, Nov 28, 2007, Christopher Chan wrote:
What does fsbench say? It has the best writing performance too?!?
No, according to the fsbench results, ReiserFS wins on Read Performance, but XFS is, approximately, four times more faster on write.
I said that the ReiserFS have the best performance based on my read/write server statics, where read requests are 70% of total I/O requests.
Ah. Too bad reiserfs is not stable enough for you.
I've lost several file systems to reiserfs, originally figuring that they were safe since SuSE has used them as their default for years.
The thing is, people swear by reiserfs. They probably used it after 2.4.18 after the vfs layer in the linux kernel stabilized.
We're using ext3 now as it appears to be rock-solid, is supported out of the box by every Linux I've used, and I've never lost one.
Well I have. No, I do not intend to use anything other than ext3 as it is still the most stable of the lot unfortunately.
We haven't had any notable performance problems using this at a regional ISP customer's site with about 10,000 e-mail users and several machines in a cluster delivering mail to Maildir folders that are NFS mounted to the central server.
I did a lot of tweaking to get the best out of ext3, xfs and Linux when I worked for an email service provider that handled 30 million mailboxes and had hundreds of machines but that was on mta machines. ext3 was not used for maildir until a few years after I started working there presumably after I demonstrated that Linux and ext3/xfs have acceptable stability. When I started, most machines were running FreeBSD including the mta ones where my primary responsibilities laid.
ext3 is (was with Centos5?) the slowest of all filesystems available for Linux.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Wed, Nov 28, 2007 at 08:51:25AM -0800, Bill Campbell wrote:
We haven't had any notable performance problems using this at a regional ISP customer's site with about 10,000 e-mail users and several machines in a cluster delivering mail to Maildir folders that are NFS mounted to the central server.
I've been using ext3 on server with 20000+ boxes for quite some time now, without any performance problems.
I'm using the same kind of setup you use. Cluster, Maildir, NFS.
Works quite nice, doesn't it ?
Exim + Courier-imap here, MySQL backend.
[]s
- -- Rodrigo Barbosa "Quid quid Latine dictum sit, altum viditur" "Be excellent to each other ..." - Bill & Ted (Wyld Stallyns)
On Thu, Nov 29, 2007, Rodrigo Barbosa wrote:
On Wed, Nov 28, 2007 at 08:51:25AM -0800, Bill Campbell wrote:
We haven't had any notable performance problems using this at a regional ISP customer's site with about 10,000 e-mail users and several machines in a cluster delivering mail to Maildir folders that are NFS mounted to the central server.
I've been using ext3 on server with 20000+ boxes for quite some time now, without any performance problems.
I'm using the same kind of setup you use. Cluster, Maildir, NFS.
Works quite nice, doesn't it ?
Very. We have a single Linux box facing the Internet which runs everything through postfix, amavisd, and clamav to weed out the phishing and worms that attack the Microsoft virus, Windows, then hands off messages that pass to the internal cluster using round-robin DNS as the poor-mans load balancer. This box runs with a load average less than 1.00 most of the time, rejects close to 2 million messages a day on IP related tests, passing about a half-million through to the internal servers which do the spamassassin checking and delivery to the user's mail stores.
Bill -- INTERNET: bill@celestial.com Bill Campbell; Celestial Software LLC URL: http://www.celestial.com/ PO Box 820; 6641 E. Mercer Way FAX: (206) 232-9186 Mercer Island, WA 98040-0820; (206) 236-1676
We shouldn't elect a President; we should elect a magician. Will Rogers
Bill Campbell wrote:
On Thu, Nov 29, 2007, Rodrigo Barbosa wrote:
On Wed, Nov 28, 2007 at 08:51:25AM -0800, Bill Campbell wrote:
We haven't had any notable performance problems using this at a regional ISP customer's site with about 10,000 e-mail users and several machines in a cluster delivering mail to Maildir folders that are NFS mounted to the central server.
I've been using ext3 on server with 20000+ boxes for quite some time now, without any performance problems.
I'm using the same kind of setup you use. Cluster, Maildir, NFS.
Works quite nice, doesn't it ?
Very. We have a single Linux box facing the Internet which runs everything through postfix, amavisd, and clamav to weed out the phishing and worms that attack the Microsoft virus, Windows, then hands off messages that pass to the internal cluster using round-robin DNS as the poor-mans load balancer. This box runs with a load average less than 1.00 most of the time, rejects close to 2 million messages a day on IP related tests, passing about a half-million through to the internal servers which do the spamassassin checking and delivery to the user's mail stores.
What processing and i/o power do you have on that box and how much RAM? For the front end boxes, I had about 20-30 dual PIII 800Mhz boxes with two SCSI disks and 1GB worth of RAM. They reject close to 180 million messages based on access and ip rules, header and body checks (so nothing cpu heavy) and they pass on about 3 million for routing or further processing.
On Fri, Nov 30, 2007, Christopher Chan wrote:
Bill Campbell wrote:
On Thu, Nov 29, 2007, Rodrigo Barbosa wrote:
On Wed, Nov 28, 2007 at 08:51:25AM -0800, Bill Campbell wrote:
We haven't had any notable performance problems using this at a regional ISP customer's site with about 10,000 e-mail users and several machines in a cluster delivering mail to Maildir folders that are NFS mounted to the central server.
I've been using ext3 on server with 20000+ boxes for quite some time now, without any performance problems.
I'm using the same kind of setup you use. Cluster, Maildir, NFS.
Works quite nice, doesn't it ?
Very. We have a single Linux box facing the Internet which runs everything through postfix, amavisd, and clamav to weed out the phishing and worms that attack the Microsoft virus, Windows, then hands off messages that pass to the internal cluster using round-robin DNS as the poor-mans load balancer. This box runs with a load average less than 1.00 most of the time, rejects close to 2 million messages a day on IP related tests, passing about a half-million through to the internal servers which do the spamassassin checking and delivery to the user's mail stores.
What processing and i/o power do you have on that box and how much RAM? For the front end boxes, I had about 20-30 dual PIII 800Mhz boxes with two SCSI disks and 1GB worth of RAM. They reject close to 180 million messages based on access and ip rules, header and body checks (so nothing cpu heavy) and they pass on about 3 million for routing or further processing.
The border MX machine is running a Intel(R) Pentium(R) 4 CPU 3.20GHz, seen as two processors in /proc/cpuinfo with 6389.76 bogomips. It has 2GB RAM, and currently has a load average of 0.24 reported by top.
The hard drive is a 40GB WDC WD400JD-19LS SATA which isn't anything special by any means. It's running SLES9, installed in February 2006. Uptime is only 356 days as it had to be rebooted to move things around in the rack.
The machines handling mail deliver in the cluster vary. The first one I checked has an Intel(R) Celeron(R) CPU 2.66GHz with 1GB of RAM. These too have pretty vanilla SATA drives.
The main server with the home directories has an Intel(R) Pentium(R) 4 CPU 3.00GHz with no SMP, 2GB RAM, and several SATA drives.
The border MX isn't beginning to breath hard handling the IP access rules, postfix, amavisd, and clamav. We have seen very even distribution amongst the delivery machines in the cluster using nothing more for load balancing than dnscache from djbdns for a single hostname on the private internal 10/100 LAN.
The attached image shows the size of the mail queues on each of the 4 machines every fifteen minutes since midnight yesterday. This peaks shortly after midnight when daily security scans and other maintenance jobs are running.
The load averages on these cluster machines rarely gets over 1.00.
The primary limiting factor seems to be the time spamassassin takes to process messages. This is typically measured in seconds per message on commodity hardware.
Bill -- INTERNET: bill@celestial.com Bill Campbell; Celestial Software LLC URL: http://www.celestial.com/ PO Box 820; 6641 E. Mercer Way FAX: (206) 232-9186 Mercer Island, WA 98040-0820; (206) 236-1676
Do not meddle in the affairs of dragons, for you are crunchy and taste good with ketchup.
Very. We have a single Linux box facing the Internet which runs everything through postfix, amavisd, and clamav to weed out the phishing and worms that attack the Microsoft virus, Windows, then hands off messages that pass to the internal cluster using round-robin DNS as the poor-mans load balancer. This box runs with a load average less than 1.00 most of the time, rejects close to 2 million messages a day on IP related tests, passing about a half-million through to the internal servers which do the spamassassin checking and delivery to the user's mail stores.
What processing and i/o power do you have on that box and how much RAM? For the front end boxes, I had about 20-30 dual PIII 800Mhz boxes with two SCSI disks and 1GB worth of RAM. They reject close to 180 million messages based on access and ip rules, header and body checks (so nothing cpu heavy) and they pass on about 3 million for routing or further processing.
The border MX machine is running a Intel(R) Pentium(R) 4 CPU 3.20GHz, seen as two processors in /proc/cpuinfo with 6389.76 bogomips. It has 2GB RAM, and currently has a load average of 0.24 reported by top.
I guess that is plenty for one third the volume of rejects one of those dual PIII boxes handled...but then your box handles five times as many deliveries and therefore scans...
The hard drive is a 40GB WDC WD400JD-19LS SATA which isn't anything special by any means. It's running SLES9, installed in February 2006. Uptime is only 356 days as it had to be rebooted to move things around in the rack.
You make sure emails never queue eh?
The machines handling mail deliver in the cluster vary. The first one I checked has an Intel(R) Celeron(R) CPU 2.66GHz with 1GB of RAM. These too have pretty vanilla SATA drives.
The main server with the home directories has an Intel(R) Pentium(R) 4 CPU 3.00GHz with no SMP, 2GB RAM, and several SATA drives.
The border MX isn't beginning to breath hard handling the IP access rules, postfix, amavisd, and clamav. We have seen very even distribution amongst the delivery machines in the cluster using nothing more for load balancing than dnscache from djbdns for a single hostname on the private internal 10/100 LAN.
Heh, I cannot imagine any other software for dns caching.
The attached image shows the size of the mail queues on each of the 4 machines every fifteen minutes since midnight yesterday. This peaks shortly after midnight when daily security scans and other maintenance jobs are running.
The load averages on these cluster machines rarely gets over 1.00.
The only times I had high load averages was when sendmail was in and when we are under bounce floods/ddos.
I guess this is the cheapest and most efficient way to do email.
Hi,
A draft with results of my benchmark based on fsbench is available in http://www.htiweb.inf.br/benchmark/fsbench.htm.
The methodology and the conclusion i will publish later, however, it shows that the XFS obtained better performance and EXT3 had results that can now compete in this environment.
Regards,
Heitor A.M. Cardozo
Christopher Chan wrote:
What does fsbench say? It has the best writing performance too?!?
No, according to the fsbench results, ReiserFS wins on Read Performance, but XFS is, approximately, four times more faster on write.
I said that the ReiserFS have the best performance based on my read/write server statics, where read requests are 70% of total I/O requests.
Ah. Too bad reiserfs is not stable enough for you.
In production, with ReiserFS, the server load average was around 30% lower than XFS.
I guess Hans got something right with his reiserfs.
Please post your findings. :-)
I'm doing new tests with ReiserFS, XFS, EXT3 and JFS in CentOS 5. I will post soon as possible.
Thank you very much in advance.
And sorry for my english...
No need to be and it is not bad at all. _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Heitor A. M. Cardozo wrote:
Hi,
A draft with results of my benchmark based on fsbench is available in http://www.htiweb.inf.br/benchmark/fsbench.htm.
The methodology and the conclusion i will publish later, however, it shows that the XFS obtained better performance and EXT3 had results that can now compete in this environment.
Thank you very much Heitor. May I trouble you to publish the files that fsbench outputs or at least the summary files?
best regards,
Christopher
Christopher Chan wrote:
Heitor A. M. Cardozo wrote:
Hi,
A draft with results of my benchmark based on fsbench is available in http://www.htiweb.inf.br/benchmark/fsbench.htm.
The methodology and the conclusion i will publish later, however, it shows that the XFS obtained better performance and EXT3 had results that can now compete in this environment.
Thank you very much Heitor. May I trouble you to publish the files that fsbench outputs or at least the summary files?
Ok Christopher, now the tests are available for download on site.
Any suggestions you may have to improve this benchmark are much appreciated.
Regards,
Heitor A. M. Cardozo
Heitor A. M. Cardozo wrote:
Christopher Chan wrote:
Heitor A. M. Cardozo wrote:
Hi,
A draft with results of my benchmark based on fsbench is available in http://www.htiweb.inf.br/benchmark/fsbench.htm.
The methodology and the conclusion i will publish later, however, it shows that the XFS obtained better performance and EXT3 had results that can now compete in this environment.
Thank you very much Heitor. May I trouble you to publish the files that fsbench outputs or at least the summary files?
Ok Christopher, now the tests are available for download on site.
Any suggestions you may have to improve this benchmark are much appreciated.
Well...creating graphs like the ones Bruce made would be nice...
I am writing an awk script to pull out the averages from the summary file. I already have the reader times done, all I need to do is get the averages for the writers and then calculate the deliveries per second for the different number of writers being invoked.
One thing that I do have in mind due to curiosity is what ext3j would look like...
regards,
Christopher
Christopher Chan wrote:
Heitor A. M. Cardozo wrote:
Christopher Chan wrote:
Heitor A. M. Cardozo wrote:
Hi,
A draft with results of my benchmark based on fsbench is available in http://www.htiweb.inf.br/benchmark/fsbench.htm.
The methodology and the conclusion i will publish later, however, it shows that the XFS obtained better performance and EXT3 had results that can now compete in this environment.
Thank you very much Heitor. May I trouble you to publish the files that fsbench outputs or at least the summary files?
Ok Christopher, now the tests are available for download on site.
Any suggestions you may have to improve this benchmark are much appreciated.
Well...creating graphs like the ones Bruce made would be nice...
I am writing an awk script to pull out the averages from the summary file. I already have the reader times done, all I need to do is get the averages for the writers and then calculate the deliveries per second for the different number of writers being invoked.
I agree and thank you if send me the average values or even the graphs.
Any suggestions to publish the results? wiki.centos.org?
One thing that I do have in mind due to curiosity is what ext3j would look like...
Ok, I added the log for ext3j in file log.tar.gz available on site.
Heitor A. M. Cardozo
Heitor A.M. Cardozo wrote:
Christopher Chan wrote:
Heitor A. M. Cardozo wrote:
Christopher Chan wrote:
Heitor A. M. Cardozo wrote:
Hi,
A draft with results of my benchmark based on fsbench is available in http://www.htiweb.inf.br/benchmark/fsbench.htm.
The methodology and the conclusion i will publish later, however, it shows that the XFS obtained better performance and EXT3 had results that can now compete in this environment.
Thank you very much Heitor. May I trouble you to publish the files that fsbench outputs or at least the summary files?
Ok Christopher, now the tests are available for download on site.
Any suggestions you may have to improve this benchmark are much appreciated.
Well...creating graphs like the ones Bruce made would be nice...
I am writing an awk script to pull out the averages from the summary file. I already have the reader times done, all I need to do is get the averages for the writers and then calculate the deliveries per second for the different number of writers being invoked.
I agree and thank you if send me the average values or even the graphs.
Here they are: The reader/writer times are in milliseconds and they are the amount of time needed to read/write one message.
jfs filesystem results: Reader time Writer time Deliveries per second No. of writers: one 0.058 6.339 157.754 No. of writers: two 0.102 19.12 104.603 No. of writers: four 0.636 122.947 32.5343 No. of writers: eight 1.782 867.593 9.22091 No. of writers: sixteen 6.744 2917.31 5.4845
reiser filesystem results: Reader time Writer time Deliveries per second No. of writers: one 0.154 20.829 48.01 No. of writers: two 0.223 63.141 31.6751 No. of writers: four 0.373 173.847 23.0087 No. of writers: eight 0.576 945.43 8.46176 No. of writers: sixteen 0.795 3812.84 4.19635
ext3o+htree filesystem results: Reader time Writer time Deliveries per second No. of writers: one 0.059 16.149 61.9233 No. of writers: two 0.087 87.719 22.8001 No. of writers: four 0.255 237.293 16.8568 No. of writers: eight 0.536 1184.24 6.75538 No. of writers: sixteen 0.753 4296.05 3.72435
ext3w+htree filesystem results: Reader time Writer time Deliveries per second No. of writers: one 0.059 14.538 68.7853 No. of writers: two 0.088 61.856 32.3332 No. of writers: four 0.364 208.894 19.1485 No. of writers: eight 0.815 1142.34 7.00315 No. of writers: sixteen 1.692 4385.77 3.64816
xfs filesystem results: Reader time Writer time Deliveries per second No. of writers: one 0.04 4.662 214.5 No. of writers: two 0.046 9.818 203.707 No. of writers: four 0.103 38.783 103.138 No. of writers: eight 0.277 301.13 26.5666 No. of writers: sixteen 2.038 1716.02 9.32388
ext3 again takes the slowest performing title overall as expected...in fact it appears not much as changed fs vs fs wise since Bruce Guenter's tests. But I am surprised at the overall performance regressions in comparison to 2.6.5/6 kernels with regards to deliveries vs amount of writers. Heitor, you are using a 3ware 95xx or 96xx with BBU write cache and write caching on right? How much RAM do you have for your cache? How is your raid10 configured? I cannot believe a four disk raid0 array can beat a software raid mirror of scsi disks as used by Bruce Guenter.
Any suggestions to publish the results? wiki.centos.org?
I'll ask on the docs list.
One thing that I do have in mind due to curiosity is what ext3j would look like...
Ok, I added the log for ext3j in file log.tar.gz available on site.
Thanks Heitor. Is the site down or something? I cannot access the page....it is timing out.
ext3 again takes the slowest performing title overall as expected...in fact it appears not much as changed fs vs fs wise since Bruce Guenter's tests. But I am surprised at the overall performance regressions in comparison to 2.6.5/6 kernels with regards to deliveries vs amount of writers. Heitor, you are using a 3ware 95xx or 96xx with BBU write cache and write caching on right? How much RAM do you have for your cache? How is your raid10 configured? I cannot believe a four disk raid0 array can beat a software raid mirror of scsi disks as used by Bruce Guenter.
that last sentence should have been 'cannot beat'
On Tue, 4 Dec 2007, Christopher Chan wrote:
ext3 again takes the slowest performing title overall as expected...in fact it appears not much as changed fs vs fs wise since Bruce Guenter's tests. But I am surprised at the overall performance regressions in comparison to 2.6.5/6 kernels with regards to deliveries vs amount of writers. Heitor, you are using a 3ware 95xx or 96xx with BBU write cache and write caching on right? How much RAM do you have for your cache? How is your raid10 configured? I cannot believe a four disk raid0 array can beat a software raid mirror of scsi disks as used by Bruce Guenter.
that last sentence should have been 'cannot beat'
I can believe it. Linux SW RAID is _VERY_ fast. The advantage of 3ware HW RAID is in its convienence and robustness when Bad Things (tm) happen, not its performance, in my experience.
that last sentence should have been 'cannot beat'
I can believe it. Linux SW RAID is _VERY_ fast. The advantage of 3ware HW RAID is in its convienence and robustness when Bad Things (tm) happen, not its performance, in my experience.
I have used both 3ware, linux software raid and others.
If you are comparing linux software raid versus 3ware 75xx or 85xx for raid5, yes, linux software raid beats 3ware hands down. For raid0 or raid1, 3ware is on par if not better than linux software raid (okay, this was about 2-3 years ago, maybe the mirroring code on linux software raid has been improved) given that both relied on disk speed.
However, since the 95xx series, 3ware now comes with RAM caches which are orders of magnitude faster than disk speed. It is impossible to compare RAM vs disk and so even raid5 on 3ware 95xx or above with a RAM cache is going to beat linux software raid.
Man, I wish I had a spare box to run fsbench and do a centos 4 vs 5 comparison.
Christopher Chan wrote:
ext3 again takes the slowest performing title overall as expected...in fact it appears not much as changed fs vs fs wise since Bruce Guenter's tests.
I agree but the values are more "acceptable" in comparision with others filesystems. On Bruce tests it shows a very bad performance for reading.
But I am surprised at the overall performance regressions in comparison to 2.6.5/6 kernels with regards to deliveries vs amount of writers.Heitor, you are using a 3ware 95xx or 96xx with BBU write cache and write caching on right? How much RAM do you have for your cache? How is your raid10 configured? I cannot believe a four disk raid0 array can beat a software raid mirror of scsi disks as used by Bruce Guenter.
3ware 9650SE with BBU and write cache on. Available memory: 224 MB Bus Type/Speed: PCIe/2.5 Gbps RAID10: 4 RAID1 subunits with MAXTOR STM3500630AS 500GB SATA2
Thanks Heitor. Is the site down or something? I cannot access the page....it is timing out.
The site is online now.
Regards,
Heitor A. M. Cardozo
Heitor A.M. Cardozo wrote:
Christopher Chan wrote:
ext3 again takes the slowest performing title overall as expected...in fact it appears not much as changed fs vs fs wise since Bruce Guenter's tests.
I agree but the values are more "acceptable" in comparision with others filesystems. On Bruce tests it shows a very bad performance for reading.
Yes, reads are vastly improved at the cost of write performance. Weird. XFS has like the best read response times too. XFS is looking very good at the moment with just about the fastest performance in everything. What io-scheduler is default on Centos 5? I assume you prefer read performance to write performance. After all, it is for maildir use. Have you tuned the box for read performance?
But I am surprised at the overall performance regressions in comparison to 2.6.5/6 kernels with regards to deliveries vs amount of writers.Heitor, you are using a 3ware 95xx or 96xx with BBU write cache and write caching on right? How much RAM do you have for your cache? How is your raid10 configured? I cannot believe a four disk raid0 array can beat a software raid mirror of scsi disks as used by Bruce Guenter.
3ware 9650SE with BBU and write cache on. Available memory: 224 MB Bus Type/Speed: PCIe/2.5 Gbps RAID10: 4 RAID1 subunits with MAXTOR STM3500630AS 500GB SATA2
Yup, that is four disks versus a single linux mirrored scsi array. Write performance cannot be that horrible now can it?
Thanks Heitor. Is the site down or something? I cannot access the page....it is timing out.
The site is online now.
thanks.
-------- Mensagem original -------- Assunto: Re:[CentOS] Filesystem for Maildir De: Christopher Chan christopher@ias.com.hk Para: CentOS mailing list centos@centos.org Data: terça-feira, 04 de dezembro de 2007 12:06:43
Heitor A.M. Cardozo wrote:
Christopher Chan wrote:
ext3 again takes the slowest performing title overall as expected...in fact it appears not much as changed fs vs fs wise since Bruce Guenter's tests.
I agree but the values are more "acceptable" in comparision with others filesystems. On Bruce tests it shows a very bad performance for reading.
Yes, reads are vastly improved at the cost of write performance. Weird. XFS has like the best read response times too. XFS is looking very good at the moment with just about the fastest performance in everything. What io-scheduler is default on Centos 5? I assume you prefer read performance to write performance. After all, it is for maildir use. Have you tuned the box for read performance?
Initially this box is not tuned for read because I would to compare the results of tests on default configuration with other configurations.
The default io-scheduler on CentOS 5 is CFQ.
But I am surprised at the overall performance regressions in comparison to 2.6.5/6 kernels with regards to deliveries vs amount of writers.Heitor, you are using a 3ware 95xx or 96xx with BBU write cache and write caching on right? How much RAM do you have for your cache? How is your raid10 configured? I cannot believe a four disk raid0 array can beat a software raid mirror of scsi disks as used by Bruce Guenter.
3ware 9650SE with BBU and write cache on. Available memory: 224 MB Bus Type/Speed: PCIe/2.5 Gbps RAID10: 4 RAID1 subunits with MAXTOR STM3500630AS 500GB SATA2
Yup, that is four disks versus a single linux mirrored scsi array. Write performance cannot be that horrible now can it?
Sorry, i forgot to say that the stripe size is set at 64Kb, not that this explain the bad write performance. I will configure and initilize array again and repeat one test to check.
Thanks Heitor. Is the site down or something? I cannot access the page....it is timing out.
The site is online now.
thanks. _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
ext3 again takes the slowest performing title overall as expected...in fact it appears not much as changed fs vs fs wise since Bruce Guenter's tests.
I agree but the values are more "acceptable" in comparision with others filesystems. On Bruce tests it shows a very bad performance for reading.
Yes, reads are vastly improved at the cost of write performance. Weird. XFS has like the best read response times too. XFS is looking very good at the moment with just about the fastest performance in everything. What io-scheduler is default on Centos 5? I assume you prefer read performance to write performance. After all, it is for maildir use. Have you tuned the box for read performance?
Initially this box is not tuned for read because I would to compare the results of tests on default configuration with other configurations.
The default io-scheduler on CentOS 5 is CFQ.
Ah, thank you for doing all that testing.
Here they are: The reader/writer times are in milliseconds and they are the amount of time needed to read/write one message.
jfs filesystem results: Reader time Writer time Deliveries per second No. of writers: one 0.058 6.339 157.754 No. of writers: two 0.102 19.12 104.603 No. of writers: four 0.636 122.947 32.5343 No. of writers: eight 1.782 867.593 9.22091 No. of writers: sixteen 6.744 2917.31 5.4845
reiser filesystem results: Reader time Writer time Deliveries per second No. of writers: one 0.154 20.829 48.01 No. of writers: two 0.223 63.141 31.6751 No. of writers: four 0.373 173.847 23.0087 No. of writers: eight 0.576 945.43 8.46176 No. of writers: sixteen 0.795 3812.84 4.19635
ext3o+htree filesystem results: Reader time Writer time Deliveries per second No. of writers: one 0.059 16.149 61.9233 No. of writers: two 0.087 87.719 22.8001 No. of writers: four 0.255 237.293 16.8568 No. of writers: eight 0.536 1184.24 6.75538 No. of writers: sixteen 0.753 4296.05 3.72435
ext3w+htree filesystem results: Reader time Writer time Deliveries per second No. of writers: one 0.059 14.538 68.7853 No. of writers: two 0.088 61.856 32.3332 No. of writers: four 0.364 208.894 19.1485 No. of writers: eight 0.815 1142.34 7.00315 No. of writers: sixteen 1.692 4385.77 3.64816
xfs filesystem results: Reader time Writer time Deliveries per second No. of writers: one 0.04 4.662 214.5 No. of writers: two 0.046 9.818 203.707 No. of writers: four 0.103 38.783 103.138 No. of writers: eight 0.277 301.13 26.5666 No. of writers: sixteen 2.038 1716.02 9.32388
ext3j+htree: Reader time Writer time Deliveries per second No. of writers: one 0.071 18.093 55.27 No. of writers: two 0.115 79.087 25.2886 No. of writers: four 0.413 276.66 14.4582 No. of writers: eight 0.924 1592.82 5.02255 No. of writers: sixteen 1.56 6439.78 2.48456
On CentOS 5.0, a had the same benchmarks and now, EXT3 and XFS seems to had better or equivalent performance on Read and Create Random files. One of this tests, using bonnie++, show this:
# bonnie++ -d /mnt/sdc1/testfile -s 8192 -m `hostname` -n 50:150000:5000:1000
bonnie++? Not appropriate. Try this: http://untroubled.org/benchmarking/2004-04/
And add JFS to the mix. You will be surprised.
I already done tests with fsbench and the results on CentOS 4.5 were equivalent: the performance of XFS was much higher than EXT3.
Then, i retest using fsbench, bonnie++ and iozone on CentOS 5.0, and the results now show the EXT3 (dir_index, noatime) with performance similar to XFS.
Say....how do the numbers on Centos 5 compare with Centos 4.5? Did XFS loose performance?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Mon, Nov 26, 2007 at 05:22:42PM -0200, Heitor Augusto M Cardozo wrote:
What i want to know is: Anyone use or recommend EXT3 for Maildir?
That is what I use.
If performance becames an issue, I just add another server. "Customizing" the OS adds up to more maintenance work than splitting the job between servers (disks/NAS etc).
- -- Rodrigo Barbosa "Quid quid Latine dictum sit, altum viditur" "Be excellent to each other ..." - Bill & Ted (Wyld Stallyns)
-
Benjamin Franz -
Bill Campbell -
Christopher Chan -
David Hrbác( -
Heitor A. M. Cardozo -
Heitor A.M. Cardozo
-
Heitor Augusto M Cardozo
-
John R Pierce -
Joshua Gimer -
Les Mikesell -
Rainer Traut -
Rodrigo Barbosa -
Scott Silva