Hi All,
I've just started caring for a dual Opteron machine. It has a 3Ware 9500S-12 RAID controller, a Tyan S2882G3NR motherboard, 10 IBM 400GB disk drives, and a 650 watt power supply.
It has been behaving itself quite nicely until recently. The poor behavior started when I upgraded the OS from CentOS 4.1 (32 bit) to Centos 4.1 (x86_64). When running in uniprocessor mode, the system appears to work properly. However, when running in SMP mode I get a lot of SCSI disk errors in /var/log/messages. I've upgraded the 3w_9xxx driver and firmware, but no joy (actually, I did get some joy out of recompiling the kernel. Its been several years since I last compiled a kernel and the process has changed for the better).
I've been in contact with 3Ware. From the error logs I supplied to them, it looks like the issue is an insufficient power supply. Their rule of thumb is 300W for the motherboard and 35 W for each disk drive. That means that I would need (at a theoretical minimum) a 650 W power supply - which I have.
I only notice the problem in SMP mode, not in uniprocessor mode. That suggests to me that (1) I have a boarderline-sufficient power supply that can support one but not two CPUs or (2) there is a problem with the combination of Opteron/3Ware 9500S-12 hardware and the x86_64 SMP kernel. I suppose there are even worse possibilities, but I'd rather not think about those right now.
What do you think?
Does anyone have any experience with this sort of hardware on x86_64 kernels?
Does the 3w_9xxx driver work well on a x86_64 machine?
Does anyone have a suggestion as to what capacity power supply I should have?
Thank you, Evan
Evan Bursey ehbursey@lbl.gov wrote:
I've just started caring for a dual Opteron machine. It has a 3Ware 9500S-12 RAID controller, a Tyan S2882G3NR motherboard, 10 IBM 400GB disk drives, and a 650 watt power supply.
Ouch, that's pushing it bad. I use a 550W with an old dual-P3 (sub-20W/each) and 3Ware Escalade 7800 with 8 x PATA at home.
Most of my other installations that have 8-12 drives are typically 6U or pedestal with 850W or 1000W 2+1 2xSSI power supplies.
(actually, I did get some joy out of recompiling the
kernel.
Its been several years since I last compiled a kernel and the process has changed for the better).
Actually, you shouldn't have to rebuild the kernel. You just: 1. Run make in the 3Ware source directory (the Makefile 3Ware includes does the job) 2. Copy the new 3w-9xxx.ko module in the appropriate /lib/modules/x.x.x-x directory 3. Run "depmod -a" to rebuild dependencies 4. Run "mkinitrd /boot/x.x.x-x-new.img x.x.x-x" to build a new initrd 5. Edit /etc/grub.conf to add a new entry with the new initrd
I've been in contact with 3Ware. From the error logs I supplied to them, it looks like the issue is an
insufficient
power supply.
Would not surprise me one bit. Although the 3Ware can stage spin-ups at boot, if you have enough power usage, the drives can still stall due to lack of power in full seek.
Their rule of thumb is 300W for the motherboard
You'll easily need 400+W with dual-Opterons. I'm sure your voltage is dropping on the +12V due to inadequate current.
and 35 W for each disk drive.
Yep, sounds typical at 3A@12V ~ 36W.
That means that I would need (at a theoretical minimum) a 650 W power supply - which I have.
I'd say you're way over budget.
I only notice the problem in SMP mode, not in uniprocessor mode.
Correct because even though the CPU is plugged in, it's probably idle and not sucking up any juice.
That suggests to me that (1) I have a boarderline-sufficient power supply that can support one but not two CPUs
I concur, it's definitely power. Unless you're staging your drive startup, I'm surprised you don't have an issue at boot.
or (2) there is a problem with the combination of Opteron/3Ware 9500S-12 hardware and the x86_64 SMP kernel. I suppose there are even worse possibilities, but I'd rather not think about those right now. What do you think?
Definitely power. You are way over budget.
You should either consider a 2+1 PS design (3 units, 2 required to run), or use a separate power supply for the drives. What kind of enclosure do you have currently?
Does anyone have any experience with this sort of hardware on x86_64 kernels?
I don't think it's a kernel issue. I think the x86-64 kernel is making far greater use of your Opteron's capabilities. I mean, there are literally units that get "turned on" when the x86-64 kernel is running -- things the i686/Athlon kernel does not enable at all.
Does the 3w_9xxx driver work well on a x86_64 machine? Does anyone have a suggestion as to what capacity power supply I should have?
Yes, 850W, easily. Power PC & Cooling has a non-redundant SSI one for just over $400. I know there are a few other vendors that have one for about $300.
I'd be interested in what your enclosure is right now.
On Fri, 2005-08-26 at 15:52, Bryan J. Smith wrote:
Evan Bursey ehbursey@lbl.gov wrote:
I've just started caring for a dual Opteron machine. It has a 3Ware 9500S-12 RAID controller, a Tyan S2882G3NR motherboard, 10 IBM 400GB disk drives, and a 650 watt power supply.
Ouch, that's pushing it bad. I use a 550W with an old dual-P3 (sub-20W/each) and 3Ware Escalade 7800 with 8 x PATA at home.
Most of my other installations that have 8-12 drives are typically 6U or pedestal with 850W or 1000W 2+1 2xSSI power supplies.
(actually, I did get some joy out of recompiling the
kernel.
Its been several years since I last compiled a kernel and the process has changed for the better).
Actually, you shouldn't have to rebuild the kernel. You just:
- Run make in the 3Ware source directory (the Makefile
3Ware includes does the job) 2. Copy the new 3w-9xxx.ko module in the appropriate /lib/modules/x.x.x-x directory 3. Run "depmod -a" to rebuild dependencies 4. Run "mkinitrd /boot/x.x.x-x-new.img x.x.x-x" to build a new initrd 5. Edit /etc/grub.conf to add a new entry with the new initrd
I've been in contact with 3Ware. From the error logs I supplied to them, it looks like the issue is an
insufficient
power supply.
Would not surprise me one bit. Although the 3Ware can stage spin-ups at boot, if you have enough power usage, the drives can still stall due to lack of power in full seek.
Their rule of thumb is 300W for the motherboard
You'll easily need 400+W with dual-Opterons. I'm sure your voltage is dropping on the +12V due to inadequate current.
and 35 W for each disk drive.
Yep, sounds typical at 3A@12V ~ 36W.
That means that I would need (at a theoretical minimum) a 650 W power supply - which I have.
I'd say you're way over budget.
I only notice the problem in SMP mode, not in uniprocessor mode.
Correct because even though the CPU is plugged in, it's probably idle and not sucking up any juice.
That suggests to me that (1) I have a boarderline-sufficient power supply that can support one but not two CPUs
I concur, it's definitely power. Unless you're staging your drive startup, I'm surprised you don't have an issue at boot.
or (2) there is a problem with the combination of Opteron/3Ware 9500S-12 hardware and the x86_64 SMP kernel. I suppose there are even worse possibilities, but I'd rather not think about those right now. What do you think?
Definitely power. You are way over budget.
You should either consider a 2+1 PS design (3 units, 2 required to run), or use a separate power supply for the drives. What kind of enclosure do you have currently?
Does anyone have any experience with this sort of hardware on x86_64 kernels?
I don't think it's a kernel issue. I think the x86-64 kernel is making far greater use of your Opteron's capabilities. I mean, there are literally units that get "turned on" when the x86-64 kernel is running -- things the i686/Athlon kernel does not enable at all.
Does the 3w_9xxx driver work well on a x86_64 machine? Does anyone have a suggestion as to what capacity power supply I should have?
Yes, 850W, easily. Power PC & Cooling has a non-redundant SSI one for just over $400. I know there are a few other vendors that have one for about $300.
I'd be interested in what your enclosure is right now.
Hi Bryan, Thanks very much for your reply. The machine uses what looks to me to be a large PC case. Its a SuperMicro 942i single power supply tower. It has two SuperMicro 5 bay SATA hot swap modules.
Evan
Evan Bursey wrote:
On Fri, 2005-08-26 at 15:52, Bryan J. Smith wrote:
Evan Bursey ehbursey@lbl.gov wrote:
I've just started caring for a dual Opteron machine. It has a 3Ware 9500S-12 RAID controller, a Tyan S2882G3NR motherboard, 10 IBM 400GB disk drives, and a 650 watt power supply.
Ouch, that's pushing it bad. I use a 550W with an old dual-P3 (sub-20W/each) and 3Ware Escalade 7800 with 8 x PATA at home.
Most of my other installations that have 8-12 drives are typically 6U or pedestal with 850W or 1000W 2+1 2xSSI power supplies.
Hmmm...my mail filter must be malfunctioning... 8-)
Bryan is right. Your machine doesn't have nearly a large enough power supply. Between the drives/processors/fans/fancy mood lighting/etc I'm surprised that it even boots. As Bryan mentioned, PC Power and Cooling (also my preferred vendor for power supplies) has what you need. The turbocool 850SSI will probably solve your problem. PCPC tends to be very conservative when rating their supplies unlike most of the bottom feeder Taiwanese/Chinese suppliers (which is undoubtedly where the supply in your supermicro case came from).
Also, if that machine is doing anything remotely important, do yourself a favor and buy some spare drives now and run it in RAID5 with some hot spares. You WILL lose drives. The Hitachi/IBM drives have reliability issues. Going forward, I'd recommend you stick with the equivalent Seagate parts if that's an option for you.
Cheers,
Evan Bursey wrote:
Hi All,
I've just started caring for a dual Opteron machine. It has a 3Ware 9500S-12 RAID controller, a Tyan S2882G3NR motherboard, 10 IBM 400GB disk drives, and a 650 watt power supply.
6x200GB IDE, 3ware 7056-8 + 2x18GB SCSI on a Tyan S2881 motherboard, 2GB RAM + two 242 Opterons. 350 Watt EPS12V redundant power supply.
What do you think?
Power.
Does anyone have any experience with this sort of hardware on x86_64 kernels?
Mine is similar but then that was with year old hardware.
Does the 3w_9xxx driver work well on a x86_64 machine?
Does anyone have a suggestion as to what capacity power supply I should have?
Just the capacity of the power supply is not enough to go on. Every power supply will also list how much current they can supply at different voltages (5V, 12V, 3.3V etc)
Your power supply also needs to be one of those EPS12V versions which comes with a 8-pin 12V power connector. So you should get a power supply that can supply more than 30A at 12V (not sure how much current you need for the cpus...been a while so you're looking for something that at least does 40A? on 12V). That is how I managed a dual Opteron + 2 SCSI and a 3ware 7056-8 + 6 IDE disks on a 350 watt EPS12V power supply.
-
Bryan J. Smith -
Chris Mauritz -
Evan Bursey -
Feizhou