[CentOS] Raid / dual Opteron power issues

Fri Aug 26 22:52:37 UTC 2005
Bryan J. Smith <b.j.smith at ieee.org>

Evan Bursey <ehbursey at lbl.gov> wrote:
> I've just started caring for a dual Opteron machine.  It
> has a 3Ware 9500S-12 RAID controller, a Tyan S2882G3NR
> motherboard, 10 IBM 400GB disk drives, and a 650 watt power
> supply.

Ouch, that's pushing it bad.  I use a 550W with an old
dual-P3 (sub-20W/each) and 3Ware Escalade 7800 with 8 x PATA
at home.

Most of my other installations that have 8-12 drives are
typically 6U or pedestal with 850W or 1000W 2+1 2xSSI power

> (actually, I did get some joy out of recompiling the
> Its been several years since I last compiled a kernel and
> the process has changed for the better).

Actually, you shouldn't have to rebuild the kernel.
You just:
1.  Run make in the 3Ware source directory (the Makefile
3Ware includes does the job)
2.  Copy the new 3w-9xxx.ko module in the appropriate
/lib/modules/x.x.x-x directory
3.  Run "depmod -a" to rebuild dependencies
4.  Run "mkinitrd /boot/x.x.x-x-new.img x.x.x-x" to build a
new initrd
5.  Edit /etc/grub.conf to add a new entry with the new

> I've been in contact with 3Ware.  From the error logs I
> supplied to them, it looks like the issue is an
> power supply.

Would not surprise me one bit.  Although the 3Ware can stage
spin-ups at boot, if you have enough power usage, the drives
can still stall due to lack of power in full seek.

> Their rule of thumb is 300W for the motherboard

You'll easily need 400+W with dual-Opterons.  I'm sure your
voltage is dropping on the +12V due to inadequate current.

> and 35 W for each disk drive.

Yep, sounds typical at 3A at 12V ~ 36W.

> That means that I would need (at a theoretical minimum) a
> 650 W power supply - which I have.

I'd say you're way over budget.

> I only notice the problem in SMP mode, not in uniprocessor
> mode.

Correct because even though the CPU is plugged in, it's
probably idle and not sucking up any juice.

> That suggests to me that (1) I have a
> boarderline-sufficient power supply that can support one
> but not two CPUs

I concur, it's definitely power.  Unless you're staging your
drive startup, I'm surprised you don't have an issue at boot.

> or (2) there is a problem with the combination of
> Opteron/3Ware 9500S-12 hardware and the x86_64 SMP 
> kernel.  I suppose there are even worse possibilities,
> but I'd rather not think about those right now.
> What do you think?

Definitely power.  You are way over budget.

You should either consider a 2+1 PS design (3 units, 2
required to run), or use a separate power supply for the
drives.  What kind of enclosure do you have currently?

> Does anyone have any experience with this sort of hardware
> on x86_64 kernels?

I don't think it's a kernel issue.  I think the x86-64 kernel
is making far greater use of your Opteron's capabilities.  I
mean, there are literally units that get "turned on" when the
x86-64 kernel is running -- things the i686/Athlon kernel
does not enable at all.

> Does the 3w_9xxx driver work well on a x86_64 machine?
> Does anyone have a suggestion as to what capacity power
> supply I should have?

Yes, 850W, easily.  Power PC & Cooling has a non-redundant
SSI one for just over $400.  I know there are a few other
vendors that have one for about $300.

I'd be interested in what your enclosure is right now.

Bryan J. Smith                | Sent from Yahoo Mail
mailto:b.j.smith at ieee.org     |  (please excuse any
http://thebs413.blogspot.com/ |   missing headers)