On Fri, 2005-08-26 at 15:52, Bryan J. Smith wrote:
Evan Bursey ehbursey@lbl.gov wrote:
I've just started caring for a dual Opteron machine. It has a 3Ware 9500S-12 RAID controller, a Tyan S2882G3NR motherboard, 10 IBM 400GB disk drives, and a 650 watt power supply.
Ouch, that's pushing it bad. I use a 550W with an old dual-P3 (sub-20W/each) and 3Ware Escalade 7800 with 8 x PATA at home.
Most of my other installations that have 8-12 drives are typically 6U or pedestal with 850W or 1000W 2+1 2xSSI power supplies.
(actually, I did get some joy out of recompiling the
kernel.
Its been several years since I last compiled a kernel and the process has changed for the better).
Actually, you shouldn't have to rebuild the kernel. You just:
- Run make in the 3Ware source directory (the Makefile
3Ware includes does the job) 2. Copy the new 3w-9xxx.ko module in the appropriate /lib/modules/x.x.x-x directory 3. Run "depmod -a" to rebuild dependencies 4. Run "mkinitrd /boot/x.x.x-x-new.img x.x.x-x" to build a new initrd 5. Edit /etc/grub.conf to add a new entry with the new initrd
I've been in contact with 3Ware. From the error logs I supplied to them, it looks like the issue is an
insufficient
power supply.
Would not surprise me one bit. Although the 3Ware can stage spin-ups at boot, if you have enough power usage, the drives can still stall due to lack of power in full seek.
Their rule of thumb is 300W for the motherboard
You'll easily need 400+W with dual-Opterons. I'm sure your voltage is dropping on the +12V due to inadequate current.
and 35 W for each disk drive.
Yep, sounds typical at 3A@12V ~ 36W.
That means that I would need (at a theoretical minimum) a 650 W power supply - which I have.
I'd say you're way over budget.
I only notice the problem in SMP mode, not in uniprocessor mode.
Correct because even though the CPU is plugged in, it's probably idle and not sucking up any juice.
That suggests to me that (1) I have a boarderline-sufficient power supply that can support one but not two CPUs
I concur, it's definitely power. Unless you're staging your drive startup, I'm surprised you don't have an issue at boot.
or (2) there is a problem with the combination of Opteron/3Ware 9500S-12 hardware and the x86_64 SMP kernel. I suppose there are even worse possibilities, but I'd rather not think about those right now. What do you think?
Definitely power. You are way over budget.
You should either consider a 2+1 PS design (3 units, 2 required to run), or use a separate power supply for the drives. What kind of enclosure do you have currently?
Does anyone have any experience with this sort of hardware on x86_64 kernels?
I don't think it's a kernel issue. I think the x86-64 kernel is making far greater use of your Opteron's capabilities. I mean, there are literally units that get "turned on" when the x86-64 kernel is running -- things the i686/Athlon kernel does not enable at all.
Does the 3w_9xxx driver work well on a x86_64 machine? Does anyone have a suggestion as to what capacity power supply I should have?
Yes, 850W, easily. Power PC & Cooling has a non-redundant SSI one for just over $400. I know there are a few other vendors that have one for about $300.
I'd be interested in what your enclosure is right now.
Hi Bryan, Thanks very much for your reply. The machine uses what looks to me to be a large PC case. Its a SuperMicro 942i single power supply tower. It has two SuperMicro 5 bay SATA hot swap modules.
Evan