I have a 3.5 TB RAID 5 Array through a 3ware 9590SE 8 port card, and am running into what seems like quite a lot of problems trying to get the system installed correctly. The system is a running dual Opterons with 2 GB of RAM, and I'm attempting to install through the x86_64 ServerCD.
First, grub refuses to install on the disk, as it's too large. I can get around this though by booting Fedora Core 6 in rescue mode and running grub, as it can deal with the disk size correctly. This might only be working when I attempt installs that don't utilize as much of the array as possible though (100 MB /boot, 2GB swap, 10Gb /, the rest left unallocated).
Secondly, there seems to be a problem using msdos disk labels with large partitions, and I can't seem to find any solutions to this in the manuals or on the net. GPT labels are offered as a solution to this but only for Itanium systems.
Am I missing some obvious solution to this? Are there some best practices for dealing with large disks anyone can share that may save me problems now or later? I'm open to suggestions, as all I've encountered so far are problems.
On Tue, 28 Nov 2006 at 12:56pm, Kevan Benson wrote
I have a 3.5 TB RAID 5 Array through a 3ware 9590SE 8 port card, and am running into what seems like quite a lot of problems trying to get the system installed correctly. The system is a running dual Opterons with 2 GB of RAM, and I'm attempting to install through the x86_64 ServerCD.
First, grub refuses to install on the disk, as it's too large. I can get around this though by booting Fedora Core 6 in rescue mode and running grub, as it can deal with the disk size correctly. This might only be working when I attempt installs that don't utilize as much of the array as possible though (100 MB /boot, 2GB swap, 10Gb /, the rest left unallocated).
Secondly, there seems to be a problem using msdos disk labels with large partitions, and I can't seem to find any solutions to this in the manuals or on the net. GPT labels are offered as a solution to this but only for Itanium systems.
Am I missing some obvious solution to this? Are there some best practices for dealing with large disks anyone can share that may save me problems now or later? I'm open to suggestions, as all I've encountered so far are problems.
1) You cannot use a msdos disk label on a disk >2TiB. You must use GPT.
2) No current boot loader understands GPT disk labels.
3) Therefore, you cannot boot from a disk >2TiB.
Possible solutions are:
1) Install a 2nd disk/array for the OS, and use the big array for data.
2) Turn on the 3ware's auto-carving feature, which will divide the single RAID volume into multiple disks, none larger than 2TiB.
Possible solutions are:
Install a 2nd disk/array for the OS, and use the big array for data.
Turn on the 3ware's auto-carving feature, which will divide the single RAID volume into multiple disks, none larger than 2TiB.
Here's how I did it:
Create a big 3TB RAID5 array.
Create an 80GB virtual disk of the OS and applications (sda1)
Create as many other virtual disks as you want/need. My raid controller required me to make 2 as it couldn't create a disk that big. (sdb)
Install the OS as per normal onto sda1. Should work just fine.
Manually create an LV directy onto sdb. No partitions.
Set up any LV partitions you might want. Format and mount the drive.
I chose to use JFS which has been fast for disk checks, formatting and overall performance seems good.
Shawn
Kevan Benson wrote:
I have a 3.5 TB RAID 5 Array through a 3ware 9590SE 8 port card, and am running into what seems like quite a lot of problems trying to get the system installed correctly. The system is a running dual Opterons with 2 GB of RAM, and I'm attempting to install through the x86_64 ServerCD.
First, grub refuses to install on the disk, as it's too large. I can get around this though by booting Fedora Core 6 in rescue mode and running grub, as it can deal with the disk size correctly. This might only be working when I attempt installs that don't utilize as much of the array as possible though (100 MB /boot, 2GB swap, 10Gb /, the rest left unallocated).
Secondly, there seems to be a problem using msdos disk labels with large partitions, and I can't seem to find any solutions to this in the manuals or on the net. GPT labels are offered as a solution to this but only for Itanium systems.
Am I missing some obvious solution to this? Are there some best practices for dealing with large disks anyone can share that may save me problems now or later? I'm open to suggestions, as all I've encountered so far are problems.
On two different systems with hardware similar to yours, I ended up using 2 different solutions. :) On one, I simply replaced 2 of the 750gig 'cudas in an 8 disk array with 160gig drives and then created a RAID1 on those 2 drives for the OS. After the OS was installed, I then used parted to create the gpt label on the large array and created the filesystem on it.
On another system, I really needed the ports on the 3ware for the array so I installed a pair of 160gig drives using a cheapo dual port SATA card and used software RAID 1 on those two drives for the OS install. After the system was installed, I then used parted to create the GPT label on the large array and created the filesystem on it.
If you go back through the list archives a bit, you'll see a bunch of very useful information from a couple of months ago where a number of people offered excellent tuning suggestions. If you get stuck, shoot me some email off list and I'll see if I can give you a hand.
Cheers,
On Tuesday 28 November 2006 13:07, chrism@imntv.com wrote:
Kevan Benson wrote:
I have a 3.5 TB RAID 5 Array through a 3ware 9590SE 8 port card, and am running into what seems like quite a lot of problems trying to get the system installed correctly. The system is a running dual Opterons with 2 GB of RAM, and I'm attempting to install through the x86_64 ServerCD.
First, grub refuses to install on the disk, as it's too large. I can get around this though by booting Fedora Core 6 in rescue mode and running grub, as it can deal with the disk size correctly. This might only be working when I attempt installs that don't utilize as much of the array as possible though (100 MB /boot, 2GB swap, 10Gb /, the rest left unallocated).
Secondly, there seems to be a problem using msdos disk labels with large partitions, and I can't seem to find any solutions to this in the manuals or on the net. GPT labels are offered as a solution to this but only for Itanium systems.
Am I missing some obvious solution to this? Are there some best practices for dealing with large disks anyone can share that may save me problems now or later? I'm open to suggestions, as all I've encountered so far are problems.
On two different systems with hardware similar to yours, I ended up using 2 different solutions. :) On one, I simply replaced 2 of the 750gig 'cudas in an 8 disk array with 160gig drives and then created a RAID1 on those 2 drives for the OS. After the OS was installed, I then used parted to create the gpt label on the large array and created the filesystem on it.
On another system, I really needed the ports on the 3ware for the array so I installed a pair of 160gig drives using a cheapo dual port SATA card and used software RAID 1 on those two drives for the OS install. After the system was installed, I then used parted to create the GPT label on the large array and created the filesystem on it.
If you go back through the list archives a bit, you'll see a bunch of very useful information from a couple of months ago where a number of people offered excellent tuning suggestions. If you get stuck, shoot me some email off list and I'll see if I can give you a hand.
Thanks for all the suggestions, and to Joshua for mentioning 3ware's carving, which I wasn't aware of before this.
Carving with LVM to append portions into a large LV seems to be the solution I'm leaning towards, as it seems to be the easiest to implement "out of the box" and I expect it will cause the least issues in the long run as there's no room in the chassis for more drives (it also has a tape drive) and dedicating one or two 500 GB drives to an install array seems wasteful.
On Tuesday 28 November 2006 15:04, Kevan Benson wrote:
Carving with LVM to append portions into a large LV seems to be the solution I'm leaning towards, as it seems to be the easiest to implement "out of the box" and I expect it will cause the least issues in the long run as there's no room in the chassis for more drives (it also has a tape drive) and dedicating one or two 500 GB drives to an install array seems wasteful.
Archival note for anyone searching this later:
I noticed that 3ware also provides an option to create a "boot" portion of the array, with a predefined amount of space assigned to it, and am using that to assign 80 GB to /dev/sda with the remaining in /dev/sdb, to be used by LVM. Unfortunately I wasn't the one to initially set up the array, or I might have noticed this and saved myself a few days of frustration.
I'll go punish myself accordingly now for responding to my own post.
It looks like you already have a solution, but you could always create two arrays, mirroring the 500GB drives. Then, create / and /boot partitions from the 500GB mirror and put the rest of the 500GB + the remaining space in the array into an LVM and carve it up however you see fit. I suppose that would could work if you weren't using a 3ware card..
On 11/28/06, Kevan Benson kbenson@a-1networks.com wrote:
On Tuesday 28 November 2006 13:07, chrism@imntv.com wrote:
Kevan Benson wrote:
I have a 3.5 TB RAID 5 Array through a 3ware 9590SE 8 port card, and
am
running into what seems like quite a lot of problems trying to get the system installed correctly. The system is a running dual Opterons
with 2
GB of RAM, and I'm attempting to install through the x86_64 ServerCD.
First, grub refuses to install on the disk, as it's too large. I can
get
around this though by booting Fedora Core 6 in rescue mode and running grub, as it can deal with the disk size correctly. This might only be working when I attempt installs that don't utilize as much of the
array
as possible though (100 MB /boot, 2GB swap, 10Gb /, the rest left unallocated).
Secondly, there seems to be a problem using msdos disk labels with
large
partitions, and I can't seem to find any solutions to this in the
manuals
or on the net. GPT labels are offered as a solution to this but only
for
Itanium systems.
Am I missing some obvious solution to this? Are there some best practices for dealing with large disks anyone can share that may save
me
problems now or later? I'm open to suggestions, as all I've
encountered
so far are problems.
On two different systems with hardware similar to yours, I ended up using 2 different solutions. :) On one, I simply replaced 2 of the 750gig 'cudas in an 8 disk array with 160gig drives and then created a RAID1 on those 2 drives for the OS. After the OS was installed, I then used parted to create the gpt label on the large array and created the filesystem on it.
On another system, I really needed the ports on the 3ware for the array so I installed a pair of 160gig drives using a cheapo dual port SATA card and used software RAID 1 on those two drives for the OS install. After the system was installed, I then used parted to create the GPT label on the large array and created the filesystem on it.
If you go back through the list archives a bit, you'll see a bunch of very useful information from a couple of months ago where a number of people offered excellent tuning suggestions. If you get stuck, shoot me some email off list and I'll see if I can give you a hand.
Thanks for all the suggestions, and to Joshua for mentioning 3ware's carving, which I wasn't aware of before this.
Carving with LVM to append portions into a large LV seems to be the solution I'm leaning towards, as it seems to be the easiest to implement "out of the box" and I expect it will cause the least issues in the long run as there's no room in the chassis for more drives (it also has a tape drive) and dedicating one or two 500 GB drives to an install array seems wasteful.
--
- Kevan Benson
- A-1 Networks
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-
chrism＠imntv.com -
Gary Richardson -
Joshua Baker-LePain -
Kevan Benson -
Shawn Everett