Joshua Baker-LePain wrote:
On Wed, 12 Jul 2006 at 10:16am, Mark Schoonover wrote
I just built up a server running CentOS 4.3, and trying to boot from a 3Ware 9550SX-16ML, 16 port card. This system has 9x500GB drives, AMD Opteron, Tyan K8SD mobo and 1 GB of ram. I downloaded the latest (9.3.0.4) driver from 3Ware, and after a couple of botched attempts, I got CentOS installed. In my two botched attempts, the problem was X hanging during the formatting of the drives, so I switched to another virtual screen while formatting took place, and the system didn't freeze.
My problem now is the system won't boot. I know this is not exactly a CentOS problem, I'm just hoping some kind sole would help me outta this overworked fog I'm in. When the system tries to boot, the cursor just sits there in the left corner of the screen flashing. Nothing happens. I am probably missing something very simple, that will cause me great public embarrassment, but I really don't care right now.
How is your array setup? You can *not* boot from a device >4TiB. On such a device, you must use a gpt disklabel and neither grub nor LILO support such.
My array size is 4.5TB. It's just one large single RAID5, so it's slightly less than 4.5TB. So, what you're saying then is to either put a small single HD to install CentOS, or make the array smaller than 4.5TB. I could easily make something much smaller, say ~1TB, install CentOS on that. Once that's finished, create the rest of the array as another array at the size that I need. Could use LVM to stitch it all together...
I'll give this a try, thanks for the help.
Mark
On Wed, 12 Jul 2006 at 11:00am, Mark Schoonover wrote
Joshua Baker-LePain wrote:
How is your array setup? You can *not* boot from a device >4TiB. On such a device, you must use a gpt disklabel and neither grub nor LILO support such.
My array size is 4.5TB. It's just one large single RAID5, so it's slightly less than 4.5TB. So, what you're saying then is to either put a small single HD to install CentOS, or make the array smaller than 4.5TB. I could easily make something much smaller, say ~1TB, install CentOS on that. Once that's finished, create the rest of the array as another array at the size that I need. Could use LVM to stitch it all together...
What I tend to do is have a secondary controller (e.g. a 8006-2) with 2 drives as a simple mirror for the OS, and then use all the drives on the 9550 for the data partition. My latest system even has those 2 drives in a hot-swap carrier accessible from the back (in addition to the 24 drives accessible from the front).
But, yes, creating multiple arrays (or enabling auto-carving on the 3ware) will work as well.
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Joshua Baker-LePain Sent: Wednesday, July 12, 2006 2:08 PM To: CentOS mailing list Subject: RE: [CentOS] Speaking of 3Ware/RAID/SATA/9550SX...
On Wed, 12 Jul 2006 at 11:00am, Mark Schoonover wrote
Joshua Baker-LePain wrote:
How is your array setup? You can *not* boot from a device
4TiB. On
such a device, you must use a gpt disklabel and neither
grub nor LILO
support such.
My array size is 4.5TB. It's just one large single RAID5,
so it's slightly
less than 4.5TB. So, what you're saying then is to either
put a small single
HD to install CentOS, or make the array smaller than 4.5TB.
SNIP
Ladies and Gentlemen,
Using the 3ware 9550 or 9500 series controllers for any array larger than 2 TB requires that you set up the controller to use "Auto-Carving", set up and build the kernel to enable multiple LUNs, rebuild initrd, reboot, and fdisk / mkfs the unused areas.
This forces the controller to "Auto-Carve" multiple 2 TB devices on different LUNS of the same SCSI ID. You also need to enable multiple LUN support in the kernel when you build a new kernel - which is required.
So .... the install will go like this:
1) Enable Auto-carving in the 3ware BIOS. 1A) Build 3ware boot driver floppy / CDROM for install
2) Install Centos with: 100 to 200 MB for /boot, 2 to 4 GB for swap, 50 to 100 GB for / - leave rest of array empty. Do rest of install as normal.
3) Install kernel source rpm and rebuild kernel from existing .config file BUT turn on multiple LUN function in the SCSI functions of the kernel.
Rebuild kernel and initrd files and test. Reboot with new kernel.
4) fdisk and mkfs the rest of sda
5) fdisk and mkfs sdb (second LUN of first device - which you will now be able to see)
6) If you want to span the devices use LVM or md or whatever.
We have done this and getting a system to this point took some time.
Once we had the magic formula it went smoothly.
Hope this helps...........
Seth Bardash
Integrated Solutions and Systems 1510 Old North Gate Road Colorado Springs, CO 80921
719-495-5866 719-495-5870 Fax 719-337-4779 Cell
http://www.integratedsolutions.org
Failure can not cope with knowledge and perseverance!
On Wed, 12 Jul 2006 at 2:31pm, Seth Bardash wrote
Using the 3ware 9550 or 9500 series controllers for any array larger than 2 TB requires that you set up the controller to use "Auto-Carving", set up and build the kernel to enable multiple LUNs, rebuild initrd, reboot, and fdisk / mkfs the unused areas.
This forces the controller to "Auto-Carve" multiple 2 TB devices on different LUNS of the same SCSI ID. You also need to enable multiple LUN support in the kernel when you build a new kernel - which is required.
This is absolutely *not* required. RHEL4 (and thus CentOS-4) explicitly support devices and filesystems up to 8TiB. As I mentioned before, however, one must use gpt disklabels on any device larger than 2TiB. And this precludes one from booting from such a large device, as neither grub nor lilo understand gpt disk labels.
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Joshua Baker-LePain Sent: Wednesday, July 12, 2006 2:36 PM To: CentOS mailing list Subject: RE: [CentOS] Speaking of 3Ware/RAID/SATA/9550SX...
On Wed, 12 Jul 2006 at 2:31pm, Seth Bardash wrote
Using the 3ware 9550 or 9500 series controllers for any array
larger
than 2 TB requires that you set up the controller to use "Auto-Carving", set up and build the kernel to enable multiple
LUNs,
rebuild initrd, reboot, and fdisk / mkfs the unused areas.
This forces the controller to "Auto-Carve" multiple 2 TB devices
on
different LUNS of the same SCSI ID. You also need to enable multiple LUN support in the kernel when
you
build a new kernel - which is required.
This is absolutely *not* required. RHEL4 (and thus CentOS-4) explicitly support devices and filesystems up to 8TiB. As I mentioned before, however, one must use gpt disklabels on any device larger than 2TiB. And this precludes one from booting from such a large device, as neither grub nor lilo understand gpt disk labels.
-- Joshua Baker-LePain Department of Biomedical Engineering Duke University
Mr. Joshua Baker-LePain,
You are absolutely correct - BUT I answered under the assumption that the original poster wants to boot from this large device and the only way to do that is as I explained.
You wrote " And this precludes one from booting from such a large device, as neither grub nor lilo understand gpt disk labels."
Exactly. So, if one wants to boot using the standard grub function AND wants to install Centos with minimal changes, the sequence I describe works without any additional controllers or disks.
Adding mirrored boot disks with an 8006-2 as mentioned by someone else is another way to go to get around the 2 TB BOOT limitation. I stress that this is a BOOT limitation.
Seth Bardash
Integrated Solutions and Systems 1510 Old North Gate Road Colorado Springs, CO 80921
719-495-5866 719-495-5870 Fax 719-337-4779 Cell
http://www.integratedsolutions.org
Failure can not cope with knowledge and perseverance!
Seth Bardash wrote:
- Install kernel source rpm and rebuild kernel from existing .config
file BUT turn on multiple LUN function in the SCSI functions of the kernel.
Hmmm... You don't need to rebuild kernel to get multiple LUNs support. You just need to use max_luns option of scsi_mod module. The default value for max_luns is 1, hence you see only first LUN. For example, to be able to see first 8 LUNs, simply place this into /etc/modprobe.conf and rebuild initrd image(s) in /boot:
options scsi_mod max_luns=8
If using LILO as boot loader, you'd need to rerun it after rebuilding initrd images. If using Grub no special action is needed for it. Reboot and you are done.
-
Aleksandar Milivojevic -
Joshua Baker-LePain -
Mark Schoonover -
Seth Bardash