hi, we've got a server with a 8 port 3ware card and 2 ide system disks. now we'd like to replace the ide disks with scsi disks or sata disks (these also recognized as scsi in the kernel). but we can't boot from it. the problem are twofold. first in the normal case the first scsi host scsi0 id the 3ware card, but grub only see the first 8 disk so if the system disk are sdi and sdj the one can't boot from it. on the other hand if one disk is removed from the 3ware card or the kernel reorder the scsi disk and sdi become sdh which is also a problem both for the bios and for the kernel. if i change the order in bios, the kernel still use as 3ware scsi0. if i use scsihosts kernel param or load the scsi driver (through modprobe.conf) in a reverse order then the kernel hang forever during the initialization of the 3ware driver. any tip? thanks in advance. yours.
centos 4.2 kernel-2.6.9-22.0.1.EL 3w_xxxx 1.26.00.039
On Thu, 2005-11-24 at 14:51 +0100, Farkas Levente wrote:
hi, we've got a server with a 8 port 3ware card and 2 ide system disks.
Are you saying you have 2 IDE disks on the 3Ware? Or they are connected to the mainboard's ATA channels?
now we'd like to replace the ide disks with scsi disks or sata disks (these also recognized as scsi in the kernel).
On which device? Are you saying you want to remove the IDE disks from the mainboard's ATA channels, and use the 3Ware volumes to boot?
Or other?
but we can't boot from it.
If its the case where you want to remove the IDE disks from the mainboard's ATA channels and let the 3Ware boot, then your problem is the BIOS Fixed Disk mapping to Linux device.
1) That is handled by the file "/boot/grub/device.map".
E.g., your current /boot/grub/device.map probably has something like: (fd0) /dev/fd0 (hd0) /dev/hda (hd1) /dev/hdc (hd2) /dev/sda
BIOS Disk 80h is /dev/hda BIOS Disk 81h is /dev/hdc BIOS Disk 82h is /dev/sda
If you remove /dev/hda and /dev/hdc -- *OR* tell your BIOS to boot off- chipset channels/cards first (like the 3Ware card) -- then the BIOS Disk 80h will become /dev/sda. You have to tell GRUB this in its mapping or it won't boot.
So you need to change your file "/boot/grub/device.map" to: (fd0) /dev/fd0 (hd0) /dev/sda And, optionally if you still leave the IDE disks attached to the ATA channels, but just changed the boot order in the BIOS: (hd1) /dev/hda (hd2) /dev/hdc
2) You'll _also_ need to re-install the GRUB loader into /dev/sda with: grub-install /dev/sda
The easiest way to do that is to use a Rescue disk (e.g., CD #1 of your CentOS release).
3) Lastly, you'll need to modify your main /boot/grub/grub.conf entries to reflect that root is in /dev/sda now, instead of /dev/hda -- as well as in /etc/fstab (if you're not using Ext2/3 filesystem labels).
the problem are twofold. first in the normal case the first scsi host scsi0 id the 3ware card, but grub only see the first 8 disk so if the system disk are sdi and sdj the one can't boot from it. on the other hand if one disk is removed from the 3ware card or the kernel reorder the scsi disk and sdi become sdh which is also a problem both for the bios and for the kernel.
If you're going to use the 3Ware card, why not put the 2 disks on port 0 and 1 -- instead of port 6 and 7?
But even then, the BIOS _should_ enumerate the first 2 disks as BIOS disk 80h and 81h _if_ no other drives are in the system. In that case, it's just a matter of editing the /boot/grub/device.map with: (fd0) /dev/fd0 (hd0) /dev/sdi (hd1) /dev/sdj And running: grub-install /dev/sdi And well as /boot/grub/grub.conf and /etc/fstab changes as appropriate.
Or do you already have the first port 0-5 populated? If so, then let's assume they are a 6 disk RAID-5 volume of /dev/sda, and then you have the port 6 and 7 disks as /dev/sdi and /dev/sdj: (fd0) /dev/fd0 (hd0) /dev/sda (hd1) /dev/sdi (hd2) /dev/sdj Running: grub-install /dev/sda While putting in hd(1,x) and /dev/sdi in /boot/grub/grub.conf because it's booting from /dev/sdi (even though GRUB is on /dev/sda for the MBR).
if i change the order in bios, the kernel still use as 3ware scsi0. if i use scsihosts kernel param or load the scsi driver (through modprobe.conf) in a reverse order then the kernel hang forever during the initialization of the 3ware driver.
It hangs? Or you get a kernel panic?
You should get a kernel panic because the BIOS disk doesn't map to the proper Linux /dev/.
Again, I need you to _clarify_ and give _exact_info_ of ...
1. What is your _exact_ configuration of your mainboard ATA channels (channel, disk, etc...) -- current as well as what you want to change?
2. What is your _exact_ configuration of your 3Ware card (ports, disk, etc...) -- again, current as well as what you want to change?
3. Where is your / and/or /boot filesystems, on what disk?
4. What you have setup to boot first in your BIOS -- current as well as what you want to change?
From that, I will give you an _exact_ /boot/grub/device.map file, and
tell you how to configure /boot/grub/grub.conf. It also helps to have a Rescue Disk (or CentOS CD #1) around.
any tip?
You have to get: A) The BIOS order B) The /boot/grub/device.map and /boot/grub/grub.conf C) The GRUB MBR/bootstrap
_Exactly_ mapping correctly. That means if you change just 1, you have to change _all_ 3. Otherwise, it won't work.
thanks in advance. yours.
centos 4.2 kernel-2.6.9-22.0.1.EL 3w_xxxx 1.26.00.039
Bryan J. Smith wrote:
On Thu, 2005-11-24 at 14:51 +0100, Farkas Levente wrote:
hi, we've got a server with a 8 port 3ware card and 2 ide system disks.
Are you saying you have 2 IDE disks on the 3Ware? Or they are connected to the mainboard's ATA channels?
two more on the mainboard.
now we'd like to replace the ide disks with scsi disks or sata disks (these also recognized as scsi in the kernel).
On which device? Are you saying you want to remove the IDE disks from the mainboard's ATA channels, and use the 3Ware volumes to boot?
Or other?
leave 8 on the 3ware (these are the data disks) and replace the mainboard 2.
if i change the order in bios, the kernel still use as 3ware scsi0. if i use scsihosts kernel param or load the scsi driver (through modprobe.conf) in a reverse order then the kernel hang forever during the initialization of the 3ware driver.
It hangs? Or you get a kernel panic?
hangs, there is no opps:-(
On Fri, 2005-11-25 at 00:22 +0100, Farkas Levente wrote:
two more on the mainboard. leave 8 on the 3ware (these are the data disks) and replace the mainboard 2.
So you're going to take the 2 IDE disks attached to mainboard ATA channels and put in SATA or SCSI disks. Thanx for clarifying what you meant by SATA or SCSI (I wasn't sure if you mean SATA devices on the 3Ware, or SCSI meaning the 3Ware volumes appear as SCSI). Sorry I misread you.
hangs, there is no opps:-(
If you're putting in a SATA or SCSI that uses a SCSI driver, then your "hang" is due to the fact that you don't have the booting SATA or SCSI driver in the initrd. You'll need to generate a _new_ initrd with the drivers.
So now I understand what you meant by ... (from your previous e-mail)
if i change the order in bios, the kernel still use as 3ware scsi0. if i use scsihosts kernel param or load the scsi driver (through modprobe.conf) in a reverse order then the kernel hang forever during the initialization of the 3ware driver.
So you were talking about the "alias scsi_hostadapter" lines in your /etc/modules.conf (2.4/CentOS3) or /etc/modprobe.conf (2.6/CentOS4). Okay, now I got you.
Remember, the _primary_ reason for the existence of these lines is to tell "mkinitrd" what order to load the drivers in. So you'll want something like ...
alias scsi_hostadapter (your booting SATA or SCSI card/controller) alias scsi_hostadapter1 3w-xxxx
_Unless_ the 3Ware cards are hosting your /boot where your initrd is, you don't even need the 2nd line. You _only_ need the first line which tells initrd to put the proper SCSI driver (and resolve related module dependencies) in the initrd.
Once you have this correct, you'll need to generate a new initrd: mkinitrd /boot/initrd-`uname -r`-SCSI.img `uname -r`
Then create a new entry in grub.conf that loads this initrd instead of your old one. Try booting it and it should load what you need.
Right now your initrd either has nothing, because it was using ATA to boot, or possibly just the 3Ware driver (which was never used to boot).
-- Bryan
P.S., I would still need to know the _exact_ configuration of your _new_ SATA or SCSI disk to help ensure _all_ files are correct (especially the device.map).
1. What's the _exact_ boot order in the BIOS of the fixed disks
Some newer BIOSes let you re-arrange the exact order of each type of device (hard drives, optical, floppy, USB, network, etc...) under a special page.
2. The card/controller, what driver, etc... of _each_ SATA or SCSI you are putting in?
3. What order those devices are showing up as in Linux -- /dev/sda, etc...
Without that _complete_ picture, I can't begin to help you with a correct /boot/grub/device.map, /boot/grub/grub.conf, etc...
-
Bryan J. Smith -
Farkas Levente