I had my eye on the Tyan dual-Opteron mobos for awhile. I tried to find a posting *anywhere* sharing experiences with these boards under Linux. No such luck. So placing myself under the heading "Where Angles Fear to Tread," I went ahead and built a system anyway. Here's what I've learned.
The specs: Tyan Thunder K8SE S2892, BIOS 1.01 2x Opteron 270, 2Ghz Dual-Core, retail package with fan 2GB RAM (4x SuperTalent 512MB P/N D32RB12P3) 3Ware 9500S-4LP 4 port SATA controller 4x 200GB Segate ST3200826AS 7200.8 SATA (attached to 3Ware controller) 1x 80GB Wester Digital WD800JD-00LSA0 SATA (attached to onboard controller) Antec Truepower 2.0 550W PSU Chenming 901A-0-0 RTL Case Supermicro 5-drive SATA HD Cage CentOS 4.1 x86_64
I thought I bought a plenty large case. Turns out the CPU heat sink at the front of the motherboard sticks up far enough to interfere with the drive bays at the bottom of the case. I had carefully mounted the CPUs, heat sinks and memory before mounting the motherboard. When I tried to insert the motherboard, the front heat sink hit the shelf that supports one of the two internal hard drive cages that come with the Chenming case. I had to peel off the heat sink to get the board in then reinstall it -- something I didn't want to do. If the case was 9" wide instead of 8", this probably wouldn't be an issue.
The heat sink sticks up far enough to keep me from reinstalling the internal hard drive cages. Not a problem since I'm using the Supermicro SATA cage, but it bugs me to lose expansion possibilities. Can anyone recommend a "low-rise" Opteron heat sink?
I hadn't yet purchased an optical drive and so was planning on booting the install program from a USB key. However I could not get the BIOS to recognize the key. I put 'removable drives' at the top of the boot preferences but it refused to recognize the key. I ended up doing a PXE boot install instead.
The install hung once forcing me to restart. But after that the install process was the fastest I've ever seen, taking about seven minutes to copy everything.
Booting the first time, I was happy to see that everything was recognized by the system, all three network ports, the Nvidia SATA controller and even the 3Ware controller.
However, the boot messages showed:
warning: many lost ticks. Your time source seems to be instable or some driver is hogging interrupts
and several repeating messages:
powernow-k8: error - out of sync...
Also, /proc/cpuinfo showed each cpu as running at about 1004Mhz with about 900 bogomips. Not what I expected.
I decided to press ahead and added a couple of drives the the 3Ware, created a unit, formatted it and began benchmarking the array. The "powernow" messages continued to occur and then the 3Ware driver started issuing error messages. Eventually the system crashed and stopped responding.
Googling on the powernow error message let me know that the kernel wizards had started to fix some powernow bugs starting with about kernel 2.6.10. Of course the stock CentOS 4.1 kernel is 2.6.9-11.
So I downloaded, compiled and installed kernel 2.6.12.3. On rebooting, the error messages went away and the 3Ware worked without complaint. Now /proc/cpuinfo showed each cpu running at 2009.267 Mhz with 4014.08 bogomips. Ah, much better! In addition, the system came up in with NUMA enabled. Looks like the Red Hat kernel had it turned off by default.
I benchmarked disk array performance using Bonnie++ version 1.03a. I ran six benchmarks using 50GB of data, three using 16k blocks and three using 64k blocks. During the raid 0 testing I had two instances of Setiathome running. During the raid 5 testing I had three instances of Setiathome running. Here are the results:
Raid 0, 64k Stripes:
bonnie++ 1.03a ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP Tyan 50G:16k 49317 99 138307 62 +++++ +++ 46765 80 183180 22 88.9 0 Tyan 50G:16k 48911 98 148421 67 77099 21 46018 79 182136 21 89.8 0 Tyan 50G:16k 49188 98 143615 65 77381 21 46158 78 181181 21 90.5 0 Tyan 50G:64k 49372 99 146417 67 77185 21 45828 78 179758 21 76.7 0 Tyan 50G:64k 48585 98 145376 66 76580 21 45609 78 171888 21 76.3 0 Tyan 50G:64k 46093 92 134903 58 67851 18 45200 77 172103 21 75.6 0 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP Tyan 16 2524 97 +++++ +++ +++++ +++ 2542 97 +++++ +++ 8670 99 Tyan 16 2385 98 +++++ +++ +++++ +++ 2579 97 +++++ +++ 8408 98 Tyan 16 2627 97 +++++ +++ +++++ +++ 2437 92 +++++ +++ 8671 100 Tyan 16 1582 98 +++++ +++ +++++ +++ 1647 98 +++++ +++ 8488 100 Tyan 16 1399 85 +++++ +++ +++++ +++ 1632 98 +++++ +++ 8476 99 Tyan 16 1654 95 +++++ +++ +++++ +++ 1636 98 +++++ +++ 8405 99
Writes at about 140MB/Sec, reads at about 180MB/Sec. Very nice.
Raid 5, 64k Stripes:
bonnie++ 1.03a ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP Tyan 50G:16k 33394 68 35889 16 26513 7 44096 75 157248 19 84.1 0 Tyan 50G:16k 32601 67 36058 16 26623 7 43964 75 156970 19 83.3 0 Tyan 50G:16k 37892 77 35702 16 26960 7 44390 75 157084 19 83.5 0 Tyan 50G:64k 36168 74 35560 17 27003 8 43463 75 155179 20 69.1 0 Tyan 50G:64k 32713 68 36187 16 26580 7 43922 76 156752 20 69.2 0 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP Tyan 16 1561 98 +++++ +++ +++++ +++ 1694 97 +++++ +++ 8428 97 Tyan 16 1672 98 +++++ +++ +++++ +++ 1712 97 +++++ +++ 8751 99 Tyan 16 1542 98 +++++ +++ +++++ +++ 1355 81 +++++ +++ 8986 100 Tyan 16 2534 96 +++++ +++ +++++ +++ 2408 96 +++++ +++ 7945 98 Tyan 16 2382 97 +++++ +++ +++++ +++ 2380 97 +++++ +++ 7989 99
Whoa! Writes drop to about 35MB/Sec under raid 5, about 25% of the raid 0 performance. In fact, it was taking so long that I aborted the last test. Just to be sure that setiathome wasn't interfering, I killed seti and reran a smaller test. Writes increased to about 50MB/Sec without seti running. I was expecting some hit for raid 5 but not this much. Guess I'll stick with the hot rsync backups to other hosts scheme that I'm using now.
As a final test, I restarted the 50GB benchmark under raid5 and popped out one of the drives. There was a pause while I guess the 3Ware controller decided that the drive actually went off-line. The writing then continued to the remaining three disks. Checking the controller showed that both the drive and the unit were 'DEGRADED.'
After popping the disk back in, I had to remove and re-add the drive to the array configuration then start the rebuild process. At this point the drive showed 'DEGRADED' and the unit showed 'REBUILDING.' The 3Ware drive logged appropriate messages when the disk was removed and replaced. I didn't bother finishing the rebuild, but it looked like it would take a couple of hours to finish if I let the benchmark continue to run.
The 3Ware controller is pretty cool. As I said, the driver (3w-9xxx) is included in the 2.6 kernel. 3Ware provides a simple CLI utility for management. They also have a GUI tool but I didn't bother running it. You can create, remove and verify 'units' on a running system. The associated /dev entries are dynamically added and removed as you make changes. Very nice.
In summary, the Tyan S2892 plus the 3Ware controller runs well under CentOS 4.1 although you will have to update to a more current kernel than the one provided by Red Hat. Watch your clearance at the front of the motherboard when selecting a case.
Comments and corrections are encouraged.
Kirk Bocek
On Thu, 2005-07-28 at 13:30 -0700, Kirk Bocek wrote:
Tyan Thunder K8SE S2892, BIOS 1.01 ... Antec Truepower 2.0 550W PSU
Is that the 550EPS12V? Or just the regular 550?
The former has an 8-pin (4x2) SSI server connector. The latter has only the 4-pin (2x2) P4 connector.
Don't confuse either with the 6-pin (3x2) SSI Workstation (WS) connector, now known as a PCIe connector. And no, that's not the 6-pin (6x1) "AUX" connector.
[ Confused yet? ;-]
Chenming 901A-0-0 RTL Case
Isn't that an ATX case, and _not_ an EATX/SSI EEB case?
I thought I bought a plenty large case.
There are (not including newer BTX, or older formats): FlexATX: 9.0" x 7.5" MicroATX: 9.6" x 9.6" ATX: 12.0" x 9.6" Ext ATX: 12.0" x 13.0" SSI EEB: 12.0" x 13.0" + additional height/spacing/thermal
Most of the time, an EATX case will fit an SSI EEB mainboard, but sometimes you don't have the headroom. Although some ATX cases may physically fit an EATX, they often don't have the stand-offs required, and definitely not the height required.
Turns out the CPU heat sink at the front of the motherboard sticks up far enough to interfere with the drive bays at the bottom of the case. I had carefully mounted the CPUs, heat sinks and memory before mounting the motherboard. When I tried to insert the motherboard, the front heat sink hit the shelf that supports one of the two internal hard drive cages that come with the Chenming case. I had to peel off the heat sink to get the board in then reinstall it -- something I didn't want to do. If the case was 9" wide instead of 8", this probably wouldn't be an issue.
Welcome to SSI EEB. ;->
"Drive overhang" is typical of most ATX cases, and will quickly interfere with EATX and especially SSI EEB mainboards.
The heat sink sticks up far enough to keep me from reinstalling the internal hard drive cages. Not a problem since I'm using the Supermicro SATA cage, but it bugs me to lose expansion possibilities. Can anyone recommend a "low-rise" Opteron heat sink?
The problem is that you need to use a case designed for SSI EEB mainboards. You have a case designed for ATX, possibly EATX. You can tell if its EATX if you're last 2-3" of your mainboard has standoffs/screws. If it's "overhanging" without them, then you only have an ATX case.
Booting the first time, I was happy to see that everything was recognized by the system, all three network ports, the Nvidia SATA controller and even the 3Ware controller. However, the boot messages showed: warning: many lost ticks. Your time source seems to be instable or some driver is hogging interrupts and several repeating messages:
powernow-k8: error - out of sync... Also, /proc/cpuinfo showed each cpu as running at about 1004Mhz with about 900 bogomips. Not what I expected.
Turn off PowerNOW in the BIOS. It's what's slowing down your CPUs. Everything I've read says do _not_ use it on a server.
So I downloaded, compiled and installed kernel 2.6.12.3. On rebooting, the error messages went away and the 3Ware worked without complaint. Now /proc/cpuinfo showed each cpu running at 2009.267 Mhz with 4014.08 bogomips. Ah, much better! In addition, the system came up in with NUMA enabled. Looks like the Red Hat kernel had it turned off by default. I benchmarked disk array performance using Bonnie++ version 1.03a. I ran six benchmarks using 50GB of data, three using 16k blocks and three using 64k blocks. During the raid 0 testing I had two instances of Setiathome running. During the raid 5 testing I had three instances of Setiathome running. Here are the results:
Raid 0, 64k Stripes:
bonnie++ 1.03a ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP Tyan 50G:16k 49317 99 138307 62 +++++ +++ 46765 80 183180 22 88.9 0 Tyan 50G:16k 48911 98 148421 67 77099 21 46018 79 182136 21 89.8 0 Tyan 50G:16k 49188 98 143615 65 77381 21 46158 78 181181 21 90.5 0 Tyan 50G:64k 49372 99 146417 67 77185 21 45828 78 179758 21 76.7 0 Tyan 50G:64k 48585 98 145376 66 76580 21 45609 78 171888 21 76.3 0 Tyan 50G:64k 46093 92 134903 58 67851 18 45200 77 172103 21 75.6 0 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP Tyan 16 2524 97 +++++ +++ +++++ +++ 2542 97 +++++ +++ 8670 99 Tyan 16 2385 98 +++++ +++ +++++ +++ 2579 97 +++++ +++ 8408 98 Tyan 16 2627 97 +++++ +++ +++++ +++ 2437 92 +++++ +++ 8671 100 Tyan 16 1582 98 +++++ +++ +++++ +++ 1647 98 +++++ +++ 8488 100 Tyan 16 1399 85 +++++ +++ +++++ +++ 1632 98 +++++ +++ 8476 99 Tyan 16 1654 95 +++++ +++ +++++ +++ 1636 98 +++++ +++ 8405 99
Writes at about 140MB/Sec, reads at about 180MB/Sec. Very nice.
Raid 5, 64k Stripes:
bonnie++ 1.03a ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP Tyan 50G:16k 33394 68 35889 16 26513 7 44096 75 157248 19 84.1 0 Tyan 50G:16k 32601 67 36058 16 26623 7 43964 75 156970 19 83.3 0 Tyan 50G:16k 37892 77 35702 16 26960 7 44390 75 157084 19 83.5 0 Tyan 50G:64k 36168 74 35560 17 27003 8 43463 75 155179 20 69.1 0 Tyan 50G:64k 32713 68 36187 16 26580 7 43922 76 156752 20 69.2 0 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP Tyan 16 1561 98 +++++ +++ +++++ +++ 1694 97 +++++ +++ 8428 97 Tyan 16 1672 98 +++++ +++ +++++ +++ 1712 97 +++++ +++ 8751 99 Tyan 16 1542 98 +++++ +++ +++++ +++ 1355 81 +++++ +++ 8986 100 Tyan 16 2534 96 +++++ +++ +++++ +++ 2408 96 +++++ +++ 7945 98 Tyan 16 2382 97 +++++ +++ +++++ +++ 2380 97 +++++ +++ 7989 99
Whoa! Writes drop to about 35MB/Sec under raid 5, about 25% of the raid 0 performance. In fact, it was taking so long that I aborted the last test. Just to be sure that setiathome wasn't interfering, I killed seti and reran a smaller test. Writes increased to about 50MB/Sec without seti running. I was expecting some hit for raid 5 but not this much. Guess I'll stick with the hot rsync backups to other hosts scheme that I'm using now.
Or consider RAID-10. Send some benchmarks of that. ;->
The 3Ware controller is pretty cool. As I said, the driver (3w-9xxx) is included in the 2.6 kernel. 3Ware provides a simple CLI utility for management. They also have a GUI tool but I didn't bother running it. You can create, remove and verify 'units' on a running system. The associated /dev entries are dynamically added and removed as you make changes. Very nice.
Make sure you have the latest 3Ware driver and firmware for the 9500S series. Also consider the "tweaks" on 3Ware's site.
In summary, the Tyan S2892 plus the 3Ware controller runs well under CentOS 4.1 although you will have to update to a more current kernel than the one provided by Red Hat. Watch your clearance at the front of the motherboard when selecting a case.
Or just get an SSI EEB case in the first place for an SSI EEB mainboard. ;->
Bryan J. Smith wrote:
Is that the 550EPS12V? Or just the regular 550?
It's the EPS 12V. Tyan's website is *very* clear about which PSU is required.
Isn't that an ATX case, and _not_ an EATX/SSI EEB case? There are (not including newer BTX, or older formats): FlexATX: 9.0" x 7.5" MicroATX: 9.6" x 9.6" ATX: 12.0" x 9.6" Ext ATX: 12.0" x 13.0" SSI EEB: 12.0" x 13.0" + additional height/spacing/thermal
Its pretty clear that it's an EATX case. My bad. Do you have an SSI EEB case you like?
Turn off PowerNOW in the BIOS. It's what's slowing down your CPUs. Everything I've read says do _not_ use it on a server.
I'll go back and look, but I didn't see any entry specifically for PowerNow. Plus as I said, installing kernel 2.6.12.3 fixed the errors.
Or consider RAID-10. Send some benchmarks of that. ;-> Make sure you have the latest 3Ware driver and firmware for the 9500S series. Also consider the "tweaks" on 3Ware's site.
Will do. I'll run the benchmarks tonight.
On Thu, 2005-07-28 at 15:37 -0700, Kirk Bocek wrote:
It's the EPS 12V. Tyan's website is *very* clear about which PSU is required.
Correct. Some people claim that some of the ATX2.0's work with the EPS12V pin-out. I doubt that, but I do know there _are_ "universal" ATX, ATX12V, ATX2.0, EPS12V power supplies from Enermax and others. I was curious if Antec's was as well.
Other than the 24-pin connector, the 8-pin (4x2) SSI Server connector is the biggie. That's what gives you not only extra +12V rails, but more importantly, +3.3V rails -- something the P4 doesn't.
Its pretty clear that it's an EATX case. My bad. Do you have an SSI EEB case you like?
In the rackmount space there are plenty. The 3U route seems to be best, because:
1) They typically have the best price for the expansion point
2) You can typically use standard ATX/EPS12V power supplies for non- redundant (as well as pay more for a 1+1 or 2+1 setup)
3) You can typically fit full height expansion cards
4) You typically get 1-3 sets of 3U x 3U (5.25") bays
If you have even more drives (16+), I'd go for a full pedestal setup. Just get one that you can always rackmount later (as 6-8U).
I'll go back and look, but I didn't see any entry specifically for PowerNow. Plus as I said, installing kernel 2.6.12.3 fixed the errors.
And that's great news. I've been installing the older Opterons in the past. I actually haven't setup an Opteron server in over 3 months now (I believe?), only Athlon 64s in desktops.
Will do. I'll run the benchmarks tonight.
Well, don't do it for my benefit. I don't want to create work for you. But if you're still testing, it's always a consideration.
Especially with 4 disks, which is ideal for RAID-10 without losing much usable storage over RAID-5 (only 33%).
Just remember to update your 3Ware 9w-xxxx driver and firmware, as well as 3DM2.
Bryan J. Smith wrote:
Correct. Some people claim that some of the ATX2.0's work with the EPS12V pin-out. I doubt that, but I do know there _are_ "universal" ATX, ATX12V, ATX2.0, EPS12V power supplies from Enermax and others. I was curious if Antec's was as well.
This PSU does have a flexible setup. The "24" pin connector actually separates into 20-pin and 4-pin connectors. I believe that's what you'd use on a regular ATX board. This is in addition to the EPS 8-pin connector.
In the rackmount space there are plenty.
At this point we're doing tower cases (and UPS's) sitting on shelves. I'd love to have a pretty rack, but I wanted the dual-Opteron rig more. :)
Well, don't do it for my benefit. I don't want to create work for you. But if you're still testing, it's always a consideration.
Especially with 4 disks, which is ideal for RAID-10 without losing much usable storage over RAID-5 (only 33%).
Just remember to update your 3Ware 9w-xxxx driver and firmware, as well as 3DM2.
Looks like the 2.6.12.3 kernel has the latest driver. And if I read 3Ware's site correctly, the 2.26 driver includes the firmware. I'm not using the GUI tool at all. Come on! I'm a *real* man! :)
On Thu, 2005-07-28 at 16:08 -0700, Kirk Bocek wrote:
This PSU does have a flexible setup. The "24" pin connector actually separates into 20-pin and 4-pin connectors. I believe that's what you'd use on a regular ATX board.
Not exactly. While 20-pin ATX[1.0] and ATX12V (aka P4) as well as 24- pin ATX2.0 all share the same, base 20-pin, I'm not entirely sure EPS12V is the same as ATX2.0.
I believe there is a converter or partial cable used for EPS12V to ATX [1.0|12V|2.0]. I could be wrong though. Never tried it.
This is in addition to the EPS 8-pin connector.
Yep, the 2x4 SSI Server pin-out.
At this point we're doing tower cases (and UPS's) sitting on shelves. I'd love to have a pretty rack, but I wanted the dual-Opteron rig more. :)
I still deliver rackmounts even when I don't mount them in a rack so the option is always there for the client. I won't worry about mounting or rails, but at least it's already in a rack.
Looks like the 2.6.12.3 kernel has the latest driver. And if I read 3Ware's site correctly, the 2.26 driver includes the firmware.
That depends. One way you can check and compare is to compare your POST string to the versions listed in the README of release 9.2.0: http://www.3ware.com/download/Escalade9000Series/9.2/9.2_Release_Notes_Web.p...
I see the "3w-9xxx_fw.h" included in the EL kernels, but that doesn't mean it's getting built with it. I'll have to check the config defaults. I always just upgrade my manually so there's no issue. If your POST looks good, then that's all you need.
I'm not using the GUI tool at all. Come on! I'm a *real* man! :)
The 3DM2 is a nice monitoring tool. I can be setup to page you via e- mail, send out alerts, etc... when you have a failure.
On Thu, 2005-07-28 at 18:26 -0500, Bryan J. Smith wrote:
Not exactly. While 20-pin ATX[1.0] and ATX12V (aka P4) as well as 24- pin ATX2.0 all share the same, base 20-pin, I'm not entirely sure EPS12V is the same as ATX2.0. I believe there is a converter or partial cable used for EPS12V to ATX [1.0|12V|2.0]. I could be wrong though. Never tried it.
Okay, here's the deal. EPS12V 2.0 and ATX 2.0 are compatible through 24-pins. EPS12V 1.x is not from what I read, but the power supplies and mainboards were rare.
I'm still searching for more technical information, especially pin-outs and expected tolerances.
Bryan, here's the page on the Antec TruePower 2.0 PSU:
http://www.antec.com/us/productDetails.php?ProdID=20551
Bryan J. Smith wrote:
On Thu, 2005-07-28 at 16:08 -0700, Kirk Bocek wrote:
This PSU does have a flexible setup. The "24" pin connector actually separates into 20-pin and 4-pin connectors. I believe that's what you'd use on a regular ATX board.
Not exactly. While 20-pin ATX[1.0] and ATX12V (aka P4) as well as 24- pin ATX2.0 all share the same, base 20-pin, I'm not entirely sure EPS12V is the same as ATX2.0.
I believe there is a converter or partial cable used for EPS12V to ATX [1.0|12V|2.0]. I could be wrong though. Never tried it.
This is in addition to the EPS 8-pin connector.
Yep, the 2x4 SSI Server pin-out.
At this point we're doing tower cases (and UPS's) sitting on shelves. I'd love to have a pretty rack, but I wanted the dual-Opteron rig more. :)
I still deliver rackmounts even when I don't mount them in a rack so the option is always there for the client. I won't worry about mounting or rails, but at least it's already in a rack.
Looks like the 2.6.12.3 kernel has the latest driver. And if I read 3Ware's site correctly, the 2.26 driver includes the firmware.
That depends. One way you can check and compare is to compare your POST string to the versions listed in the README of release 9.2.0: http://www.3ware.com/download/Escalade9000Series/9.2/9.2_Release_Notes_Web.p...
I see the "3w-9xxx_fw.h" included in the EL kernels, but that doesn't mean it's getting built with it. I'll have to check the config defaults. I always just upgrade my manually so there's no issue. If your POST looks good, then that's all you need.
I'm not using the GUI tool at all. Come on! I'm a *real* man! :)
The 3DM2 is a nice monitoring tool. I can be setup to page you via e- mail, send out alerts, etc... when you have a failure.
On Thu, 28 Jul 2005 at 4:18pm, Bryan J. Smith wrote
On Thu, 2005-07-28 at 13:30 -0700, Kirk Bocek wrote:
The 3Ware controller is pretty cool. As I said, the driver (3w-9xxx) is included in the 2.6 kernel. 3Ware provides a simple CLI utility for management. They also have a GUI tool but I didn't bother running it. You can create, remove and verify 'units' on a running system. The associated /dev entries are dynamically added and removed as you make changes. Very nice.
Make sure you have the latest 3Ware driver and firmware for the 9500S series. Also consider the "tweaks" on 3Ware's site.
Do *not* use the latest 3ware driver/firmware (i.e. the 9.2 codeset) if you have more than unit per controller. There is a bad cache allocation bug in that codeset that kills performance if there is more than unit present on a controller when it boots. And, yes, this holds even if the 2nd unit is merely a hot spare.
Of course, the 9.2 codeset is the only one that properly handles BBUs, which presents a bit of a catch-22 if your controllers have 'em. The current kludge^Wfix for that situation is to boot with the hot spare drives *physically* disconnected from the controller, plug them in after the system is up, and then go into 3dm2/tw_cli and add them as hot spares.
There's a beta of 9.2.1 on 3ware's site, but my only 9500 based system is in production now (using the above kludge), so I won't be testing it until after it's officially released.
On Fri, 2005-07-29 at 08:59 -0400, Joshua Baker-LePain wrote:
Do *not* use the latest 3ware driver/firmware (i.e. the 9.2 codeset) if you have more than unit per controller. There is a bad cache allocation bug in that codeset that kills performance if there is more than unit present on a controller when it boots. And, yes, this holds even if the 2nd unit is merely a hot spare.
Yep, I just forwarded that.
Of course, the 9.2 codeset is the only one that properly handles BBUs, which presents a bit of a catch-22 if your controllers have 'em. The current kludge^Wfix for that situation is to boot with the hot spare drives *physically* disconnected from the controller, plug them in after the system is up, and then go into 3dm2/tw_cli and add them as hot spares. There's a beta of 9.2.1 on 3ware's site, but my only 9500 based system is in production now (using the above kludge), so I won't be testing it until after it's officially released.
Yep, I saw that too.
I'm probably going to go back to my previous instinct, until the 9500S has been out for awhile, it's probably safest to stick with the 8506 series. Just don't do RAID-5 on it though. ;-ppp
Bryan J. Smith wrote:
Or consider RAID-10. Send some benchmarks of that. ;->
Here are the Raid-10 benchmarks. Three instances of Setiathome were running during the benchmark.
Bonnie++ 1.03a ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP Tyan 50G:16k 44842 89 79313 35 44383 12 44677 77 111266 13 94.5 0 Tyan 50G:16k 44597 89 77035 35 44487 12 44644 77 111081 13 94.1 0 Tyan 50G:16k 42774 85 77262 34 44663 12 44891 77 111075 13 94.6 0 Tyan 50G:64k 42994 86 76934 34 43913 12 44790 77 110579 13 79.5 0 Tyan 50G:64k 43919 88 76894 34 44073 12 44879 77 110929 13 78.6 0 Tyan 50G:64k 43699 87 76476 34 44436 12 44902 77 110922 13 79.0 0 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP Tyan 16 2420 89 +++++ +++ +++++ +++ 2511 97 +++++ +++ 8720 99 Tyan 16 2490 97 +++++ +++ +++++ +++ 2261 97 +++++ +++ 8742 99 Tyan 16 2475 97 +++++ +++ +++++ +++ 2695 97 +++++ +++ 8594 99 Tyan 16 2500 96 +++++ +++ +++++ +++ 2183 80 +++++ +++ 8287 98 Tyan 16 2429 97 +++++ +++ +++++ +++ 2519 97 +++++ +++ 8166 98 Tyan 16 2404 97 +++++ +++ +++++ +++ 2399 91 +++++ +++ 8657 99
A bit over half the write speeds of Raid-0. Big surprise. Seems like this is the best combination of performance and reliability.
Kirk Bocek
Kirk Bocek wrote:
I had my eye on the Tyan dual-Opteron mobos for awhile. I tried to find a posting *anywhere* sharing experiences with these boards under Linux. No such luck. So placing myself under the heading "Where Angles Fear to Tread," I went ahead and built a system anyway. Here's what I've learned.
The specs: Tyan Thunder K8SE S2892, BIOS 1.01 2x Opteron 270, 2Ghz Dual-Core, retail package with fan 2GB RAM (4x SuperTalent 512MB P/N D32RB12P3) 3Ware 9500S-4LP 4 port SATA controller 4x 200GB Segate ST3200826AS 7200.8 SATA (attached to 3Ware controller) 1x 80GB Wester Digital WD800JD-00LSA0 SATA (attached to onboard controller) Antec Truepower 2.0 550W PSU Chenming 901A-0-0 RTL Case Supermicro 5-drive SATA HD Cage CentOS 4.1 x86_64
I thought I bought a plenty large case. Turns out the CPU heat sink at the front of the motherboard sticks up far enough to interfere with the drive bays at the bottom of the case. I had carefully mounted the CPUs, heat sinks and memory before mounting the motherboard. When I tried to insert the motherboard, the front heat sink hit the shelf that supports one of the two internal hard drive cages that come with the Chenming case. I had to peel off the heat sink to get the board in then reinstall it -- something I didn't want to do. If the case was 9" wide instead of 8", this probably wouldn't be an issue.
The heat sink sticks up far enough to keep me from reinstalling the internal hard drive cages. Not a problem since I'm using the Supermicro SATA cage, but it bugs me to lose expansion possibilities. Can anyone recommend a "low-rise" Opteron heat sink?
Monarch lists 3 1U all-copper-no-fan heat sinks that I bet would work. Please note the speculative nature of that recommendation !!!! I have been aiming for an Opteron based system for a while now, that's why I joined this list :-).
<snip>
On Thu, 2005-07-28 at 18:10 -0500, William A. Mahaffey III wrote:
Monarch lists 3 1U all-copper-no-fan heat sinks that I bet would work. Please note the speculative nature of that recommendation !!!! I have been aiming for an Opteron based system for a while now, that's why I joined this list :-).
If you go component, I recommend Monarch Computer. They give you close to NewEgg price, while assembling systems proper, and they'll sell you RHEL with a full hardware/software SLA if you want or plunk Fedora, Debian, etc... if you don't.
William A. Mahaffey III wrote:
Monarch lists 3 1U all-copper-no-fan heat sinks that I bet would work. Please note the speculative nature of that recommendation !!!! I have been aiming for an Opteron based system for a while now, that's why I joined this list :-).
I didn't find anything on NewEgg. I'll see what Monarch has. AMD's stock fan and heat sink is about 50mm high. If I can get it down to about 25mm I should be good to go.
Does anyone have any notes on heat sink-only cooling. I'm sure it depends on airflow in the case (which is good in my case) but I'd like to hear any experiences.
Kirk Bocek wrote:
William A. Mahaffey III wrote:
Monarch lists 3 1U all-copper-no-fan heat sinks that I bet would work. Please note the speculative nature of that recommendation !!!! I have been aiming for an Opteron based system for a while now, that's why I joined this list :-).
I didn't find anything on NewEgg. I'll see what Monarch has. AMD's stock fan and heat sink is about 50mm high. If I can get it down to about 25mm I should be good to go.
Does anyone have any notes on heat sink-only cooling. I'm sure it depends on airflow in the case (which is good in my case) but I'd like to hear any experiences.
Roger that, extremely important observation. Monarch lists 1U heat sinks with fins aligned both ways, for about the same price. Pick the one with fins aligned with your case fans & you would probably be well off. *Speculation again* :-).
Kirk Bocek wrote: ...
However, the boot messages showed:
warning: many lost ticks. Your time source seems to be instable or some driver is hogging interrupts
and several repeating messages:
powernow-k8: error - out of sync...
Also, /proc/cpuinfo showed each cpu as running at about 1004Mhz with about 900 bogomips. Not what I expected.
I decided to press ahead and added a couple of drives the the 3Ware, created a unit, formatted it and began benchmarking the array. The "powernow" messages continued to occur and then the 3Ware driver started issuing error messages. Eventually the system crashed and stopped responding.
Googling on the powernow error message let me know that the kernel wizards had started to fix some powernow bugs starting with about kernel 2.6.10. Of course the stock CentOS 4.1 kernel is 2.6.9-11.
So I downloaded, compiled and installed kernel 2.6.12.3. On rebooting, the error messages went away and the 3Ware worked without complaint. Now /proc/cpuinfo showed each cpu running at 2009.267 Mhz with 4014.08 bogomips. Ah, much better! In addition, the system came up in with NUMA enabled. Looks like the Red Hat kernel had it turned off by default.
...
Comments and corrections are encouraged.
Kirk Bocek
This is what I get with the standard (2.6.9-11ELsmp) kernel on a quad-opteron (2.6GHz) Tyan S4882 - actually the Tyan TX46 ( http://www.tyan.com/products/html/tx46b4882.html ):
powernow-k8: Found 4 AMD Athlon 64 / Opteron processors (version 1.00.09b) powernow-k8: 0 : fid 0x12 (2600 MHz), vid 0x6 (1400 mV) powernow-k8: 1 : fid 0x10 (2400 MHz), vid 0x8 (1350 mV) powernow-k8: 2 : fid 0xe (2200 MHz), vid 0xa (1300 mV) powernow-k8: 3 : fid 0xc (2000 MHz), vid 0xc (1250 mV) powernow-k8: 4 : fid 0xa (1800 MHz), vid 0xe (1200 mV) powernow-k8: 5 : fid 0x2 (1000 MHz), vid 0x12 (1100 mV) powernow-k8: cpu_init done, current fid 0x12, vid 0x6 powernow-k8: 0 : fid 0x12 (2600 MHz), vid 0x6 (1400 mV) powernow-k8: 1 : fid 0x10 (2400 MHz), vid 0x8 (1350 mV) powernow-k8: 2 : fid 0xe (2200 MHz), vid 0xa (1300 mV) powernow-k8: 3 : fid 0xc (2000 MHz), vid 0xc (1250 mV) powernow-k8: 4 : fid 0xa (1800 MHz), vid 0xe (1200 mV) powernow-k8: 5 : fid 0x2 (1000 MHz), vid 0x12 (1100 mV) powernow-k8: cpu_init done, current fid 0x12, vid 0x6 powernow-k8: 0 : fid 0x12 (2600 MHz), vid 0x6 (1400 mV) powernow-k8: 1 : fid 0x10 (2400 MHz), vid 0x8 (1350 mV) powernow-k8: 2 : fid 0xe (2200 MHz), vid 0xa (1300 mV) powernow-k8: 3 : fid 0xc (2000 MHz), vid 0xc (1250 mV) powernow-k8: 4 : fid 0xa (1800 MHz), vid 0xe (1200 mV) powernow-k8: 5 : fid 0x2 (1000 MHz), vid 0x12 (1100 mV) powernow-k8: cpu_init done, current fid 0x12, vid 0x6 powernow-k8: 0 : fid 0x12 (2600 MHz), vid 0x6 (1400 mV) powernow-k8: 1 : fid 0x10 (2400 MHz), vid 0x8 (1350 mV) powernow-k8: 2 : fid 0xe (2200 MHz), vid 0xa (1300 mV) powernow-k8: 3 : fid 0xc (2000 MHz), vid 0xc (1250 mV) powernow-k8: 4 : fid 0xa (1800 MHz), vid 0xe (1200 mV) powernow-k8: 5 : fid 0x2 (1000 MHz), vid 0x12 (1100 mV) powernow-k8: cpu_init done, current fid 0x12, vid 0x6 ACPI: (supports S0 S1 S4 S5)
which clearly shows that the kernel detects the powernow features nicely. Actually powernow adjusts the CPU speed dynamically according to the requirements - an idle server runs all CPUs at 1000MHz; whereas at higher loads the speed goes up to 2600MHz automagically (I grepped MHz in /proc/cpuinfo).
All of this is no different when using the 2.6.12 kernel. I have not found a single problem with the system.
One question of my own: I am pretty much confused by the multitude of BIOS options which influence the handling of the memory: memory hole remapping, SRAT table and such. Is there anybody who can say definitely what works best on the standard RH kernel? I tried a number of possibilities but did not find any differences in lmbench benchmarks. The existence of "NUMA" is only reported by the 2.6.12 kernel, not by the RH one. Currently running with memory remapping "SOFTWARE", SRAT table "DISABLED" and everything else at "AUTO".
Kay
-
Bryan J. Smith -
Joshua Baker-LePain -
Kay Diederichs -
Kirk Bocek -
William A. Mahaffey III