[CentOS] Raid / dual Opteron power issues

On Fri, 2005-08-26 at 15:52, Bryan J. Smith wrote:
> Evan Bursey <ehbursey at lbl.gov> wrote:
> > I've just started caring for a dual Opteron machine.  It
> > has a 3Ware 9500S-12 RAID controller, a Tyan S2882G3NR
> > motherboard, 10 IBM 400GB disk drives, and a 650 watt power
> > supply.
> 
> Ouch, that's pushing it bad.  I use a 550W with an old
> dual-P3 (sub-20W/each) and 3Ware Escalade 7800 with 8 x PATA
> at home.
> 
> Most of my other installations that have 8-12 drives are
> typically 6U or pedestal with 850W or 1000W 2+1 2xSSI power
> supplies.
> 
> > (actually, I did get some joy out of recompiling the
> kernel.
> > Its been several years since I last compiled a kernel and
> > the process has changed for the better).
> 
> Actually, you shouldn't have to rebuild the kernel.
> You just:
> 1.  Run make in the 3Ware source directory (the Makefile
> 3Ware includes does the job)
> 2.  Copy the new 3w-9xxx.ko module in the appropriate
> /lib/modules/x.x.x-x directory
> 3.  Run "depmod -a" to rebuild dependencies
> 4.  Run "mkinitrd /boot/x.x.x-x-new.img x.x.x-x" to build a
> new initrd
> 5.  Edit /etc/grub.conf to add a new entry with the new
> initrd
> 
> > I've been in contact with 3Ware.  From the error logs I
> > supplied to them, it looks like the issue is an
> insufficient
> > power supply.
> 
> Would not surprise me one bit.  Although the 3Ware can stage
> spin-ups at boot, if you have enough power usage, the drives
> can still stall due to lack of power in full seek.
> 
> > Their rule of thumb is 300W for the motherboard
> 
> You'll easily need 400+W with dual-Opterons.  I'm sure your
> voltage is dropping on the +12V due to inadequate current.
> 
> > and 35 W for each disk drive.
> 
> Yep, sounds typical at 3A at 12V ~ 36W.
> 
> > That means that I would need (at a theoretical minimum) a
> > 650 W power supply - which I have.
> 
> I'd say you're way over budget.
> 
> > I only notice the problem in SMP mode, not in uniprocessor
> > mode.
> 
> Correct because even though the CPU is plugged in, it's
> probably idle and not sucking up any juice.
> 
> > That suggests to me that (1) I have a
> > boarderline-sufficient power supply that can support one
> > but not two CPUs
> 
> I concur, it's definitely power.  Unless you're staging your
> drive startup, I'm surprised you don't have an issue at boot.
> 
> > or (2) there is a problem with the combination of
> > Opteron/3Ware 9500S-12 hardware and the x86_64 SMP 
> > kernel.  I suppose there are even worse possibilities,
> > but I'd rather not think about those right now.
> > What do you think?
> 
> Definitely power.  You are way over budget.
> 
> You should either consider a 2+1 PS design (3 units, 2
> required to run), or use a separate power supply for the
> drives.  What kind of enclosure do you have currently?
> 
> > Does anyone have any experience with this sort of hardware
> > on x86_64 kernels?
> 
> I don't think it's a kernel issue.  I think the x86-64 kernel
> is making far greater use of your Opteron's capabilities.  I
> mean, there are literally units that get "turned on" when the
> x86-64 kernel is running -- things the i686/Athlon kernel
> does not enable at all.
> 
> > Does the 3w_9xxx driver work well on a x86_64 machine?
> > Does anyone have a suggestion as to what capacity power
> > supply I should have?
> 
> Yes, 850W, easily.  Power PC & Cooling has a non-redundant
> SSI one for just over $400.  I know there are a few other
> vendors that have one for about $300.
> 
> I'd be interested in what your enclosure is right now.
> 

Hi Bryan,  Thanks very much for your reply.  The machine uses what looks
to me to be a large PC case.  Its a SuperMicro 942i single power supply
tower.  It has two SuperMicro 5 bay SATA hot swap modules.

Evan