I had my eye on the Tyan dual-Opteron mobos for awhile. I tried to find
a posting *anywhere* sharing experiences with these boards under Linux.
No such luck. So placing myself under the heading "Where Angles Fear to
Tread," I went ahead and built a system anyway. Here's what I've learned.
The specs:
Tyan Thunder K8SE S2892, BIOS 1.01
2x Opteron 270, 2Ghz Dual-Core, retail package with fan
2GB RAM (4x SuperTalent 512MB P/N D32RB12P3)
3Ware 9500S-4LP 4 port SATA controller
4x 200GB Segate ST3200826AS 7200.8 SATA (attached to 3Ware controller)
1x 80GB Wester Digital WD800JD-00LSA0 SATA (attached to onboard controller)
Antec Truepower 2.0 550W PSU
Chenming 901A-0-0 RTL Case
Supermicro 5-drive SATA HD Cage
CentOS 4.1 x86_64
I thought I bought a plenty large case. Turns out the CPU heat sink at
the front of the motherboard sticks up far enough to interfere with the
drive bays at the bottom of the case. I had carefully mounted the CPUs,
heat sinks and memory before mounting the motherboard. When I tried to
insert the motherboard, the front heat sink hit the shelf that supports
one of the two internal hard drive cages that come with the Chenming
case. I had to peel off the heat sink to get the board in then reinstall
it -- something I didn't want to do. If the case was 9" wide instead of
8", this probably wouldn't be an issue.
The heat sink sticks up far enough to keep me from reinstalling the
internal hard drive cages. Not a problem since I'm using the Supermicro
SATA cage, but it bugs me to lose expansion possibilities. Can anyone
recommend a "low-rise" Opteron heat sink?
I hadn't yet purchased an optical drive and so was planning on
booting the install program from a USB key. However I could not get the
BIOS to recognize the key. I put 'removable drives' at the top of the
boot preferences but it refused to recognize the key. I ended up doing a
PXE boot install instead.
The install hung once forcing me to restart. But after that the install
process was the fastest I've ever seen, taking about seven minutes to
copy everything.
Booting the first time, I was happy to see that everything was
recognized by the system, all three network ports, the Nvidia SATA
controller and even the 3Ware controller.
However, the boot messages showed:
warning: many lost ticks.
Your time source seems to be instable or some driver is hogging
interrupts
and several repeating messages:
powernow-k8: error - out of sync...
Also, /proc/cpuinfo showed each cpu as running at about 1004Mhz with
about 900 bogomips. Not what I expected.
I decided to press ahead and added a couple of drives the the 3Ware,
created a unit, formatted it and began benchmarking the array. The
"powernow" messages continued to occur and then the 3Ware driver started
issuing error messages. Eventually the system crashed and stopped
responding.
Googling on the powernow error message let me know that the kernel
wizards had started to fix some powernow bugs starting with about kernel
2.6.10. Of course the stock CentOS 4.1 kernel is 2.6.9-11.
So I downloaded, compiled and installed kernel 2.6.12.3. On rebooting,
the error messages went away and the 3Ware worked without complaint. Now
/proc/cpuinfo showed each cpu running at 2009.267 Mhz with 4014.08
bogomips. Ah, much better! In addition, the system came up in with NUMA
enabled. Looks like the Red Hat kernel had it turned off by default.
I benchmarked disk array performance using Bonnie++ version 1.03a. I ran
six benchmarks using 50GB of data, three using 16k blocks and three
using 64k blocks. During the raid 0 testing I had two instances of
Setiathome running. During the raid 5 testing I had three instances of
Setiathome running. Here are the results:
Raid 0, 64k Stripes:
bonnie++ 1.03a ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
Tyan 50G:16k 49317 99 138307 62 +++++ +++ 46765 80 183180 22 88.9 0
Tyan 50G:16k 48911 98 148421 67 77099 21 46018 79 182136 21 89.8 0
Tyan 50G:16k 49188 98 143615 65 77381 21 46158 78 181181 21 90.5 0
Tyan 50G:64k 49372 99 146417 67 77185 21 45828 78 179758 21 76.7 0
Tyan 50G:64k 48585 98 145376 66 76580 21 45609 78 171888 21 76.3 0
Tyan 50G:64k 46093 92 134903 58 67851 18 45200 77 172103 21 75.6 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
Tyan 16 2524 97 +++++ +++ +++++ +++ 2542 97 +++++ +++ 8670 99
Tyan 16 2385 98 +++++ +++ +++++ +++ 2579 97 +++++ +++ 8408 98
Tyan 16 2627 97 +++++ +++ +++++ +++ 2437 92 +++++ +++ 8671 100
Tyan 16 1582 98 +++++ +++ +++++ +++ 1647 98 +++++ +++ 8488 100
Tyan 16 1399 85 +++++ +++ +++++ +++ 1632 98 +++++ +++ 8476 99
Tyan 16 1654 95 +++++ +++ +++++ +++ 1636 98 +++++ +++ 8405 99
Writes at about 140MB/Sec, reads at about 180MB/Sec. Very nice.
Raid 5, 64k Stripes:
bonnie++ 1.03a ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
Tyan 50G:16k 33394 68 35889 16 26513 7 44096 75 157248 19 84.1 0
Tyan 50G:16k 32601 67 36058 16 26623 7 43964 75 156970 19 83.3 0
Tyan 50G:16k 37892 77 35702 16 26960 7 44390 75 157084 19 83.5 0
Tyan 50G:64k 36168 74 35560 17 27003 8 43463 75 155179 20 69.1 0
Tyan 50G:64k 32713 68 36187 16 26580 7 43922 76 156752 20 69.2 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
Tyan 16 1561 98 +++++ +++ +++++ +++ 1694 97 +++++ +++ 8428 97
Tyan 16 1672 98 +++++ +++ +++++ +++ 1712 97 +++++ +++ 8751 99
Tyan 16 1542 98 +++++ +++ +++++ +++ 1355 81 +++++ +++ 8986 100
Tyan 16 2534 96 +++++ +++ +++++ +++ 2408 96 +++++ +++ 7945 98
Tyan 16 2382 97 +++++ +++ +++++ +++ 2380 97 +++++ +++ 7989 99
Whoa! Writes drop to about 35MB/Sec under raid 5, about 25% of the raid 0
performance. In fact, it was taking so long that I aborted the last test. Just to be
sure that setiathome wasn't interfering, I killed seti and reran a smaller test.
Writes increased to about 50MB/Sec without seti running. I was expecting some hit for
raid 5 but not this much. Guess I'll stick with the hot rsync backups to other hosts
scheme that I'm using now.
As a final test, I restarted the 50GB benchmark under raid5 and popped out one of the
drives. There was a pause while I guess the 3Ware controller decided that the drive
actually went off-line. The writing then continued to the remaining three disks.
Checking the controller showed that both the drive and the unit were 'DEGRADED.'
After popping the disk back in, I had to remove and re-add the drive to the array
configuration then start the rebuild process. At this point the drive showed
'DEGRADED' and the unit showed 'REBUILDING.' The 3Ware drive logged appropriate
messages when the disk was removed and replaced. I didn't bother finishing the
rebuild, but it looked like it would take a couple of hours to finish if I let the
benchmark continue to run.
The 3Ware controller is pretty cool. As I said, the driver (3w-9xxx) is
included in the 2.6 kernel. 3Ware provides a simple CLI utility for
management. They also have a GUI tool but I didn't bother running it.
You can create, remove and verify 'units' on a running system. The
associated /dev entries are dynamically added and removed as you make
changes. Very nice.
In summary, the Tyan S2892 plus the 3Ware controller runs well under CentOS 4.1
although you will have to update to a more current kernel than the one provided by
Red Hat. Watch your clearance at the front of the motherboard when selecting a case.
Comments and corrections are encouraged.
Kirk Bocek