Has anyone tried to use these yet?
For hysterical raisons, my primary machine is CentOS5 32bit (up to date).
In the machine I have a SiI 3132 dual port SATA controller with port multiplier. Both ports are plugged into an external storage device which presents 4 SATA disks on each port, for a total of 8 possible disks.
Currently I have 5*1Tb in there, configured as RAID5 via mdadm
I boot off an internal SATA disk; I don't see a need to change that.
But space is running out; I wondering if the new 3Tb disks would work in this scenario; most importantly will 2.6.18-238.9.1.el5PAE see the whole disk? And also whether I need to be worried about 4Kbyte sectors.
These new disks are confusing me!
On Saturday, June 18, 2011 10:00:25 PM Stephen Harris wrote:
But space is running out; I wondering if the new 3Tb disks would work in this scenario; most importantly will 2.6.18-238.9.1.el5PAE see the whole disk? And also whether I need to be worried about 4Kbyte sectors.
You'll have to change to GPT partitioning to use >2TB; for instance: ++++++++++ [root@www ~]# uname -srvmpio Linux 2.6.32-131.2.1.el6.i686 #1 SMP Wed May 18 07:07:27 EDT 2011 i686 i686 i386 GNU/Linux [root@www ~]# parted /dev/sdn print Model: DGC RAID 5 (scsi) Disk /dev/sdn: 6893GB Sector size (logical/physical): 512B/512B Partition Table: gpt
Number Start End Size File system Name Flags 1 1049kB 6893GB 6893GB ext4 lomover
[root@www ~]# +++++++++++
You will need to align your partitions to the 4KB sector boundaries; parted is doing this automatically these days; not sure about C5, though, as I don't currently have a LUN on a storage group with a hardware (non-VMware) C5 i686 box (using VMware ESX 3.5U5 limits LUNs to 2TB, so can't test there). The above machine is a real RHEL 6.1 box on an i686 processor system (dual Intel Xeon 2.8GHz; generation prior to EM64T's introduction). Works fine, just have to remember to partition GPT.
On Sat, Jun 18, 2011 at 10:13:00PM -0400, Lamar Owen wrote:
On Saturday, June 18, 2011 10:00:25 PM Stephen Harris wrote:
But space is running out; I wondering if the new 3Tb disks would work in this scenario; most importantly will 2.6.18-238.9.1.el5PAE see the whole disk? And also whether I need to be worried about 4Kbyte sectors.
You'll have to change to GPT partitioning to use >2TB; for instance:
Only if I'm partitioning the disk, surely. If I just throw the whole raw disk into mdadm (sdd, sde, sdf...) then isn't partitioning irrelevant?
Linux 2.6.32-131.2.1.el6.i686 #1 SMP Wed May 18 07:07:27 EDT 2011 i686 i686 i386 GNU/Linux
Hmm, el6. Nice to know that will work when CentOS6 comes out; maybe I'll wait until then and rebuild my machine with 64bit mode.
You will need to align your partitions to the 4KB sector boundaries;
What if I don't partition?
Thanks,
On 6/18/2011 8:51 PM, Stephen Harris wrote:
On Sat, Jun 18, 2011 at 10:13:00PM -0400, Lamar Owen wrote:
On Saturday, June 18, 2011 10:00:25 PM Stephen Harris wrote:
But space is running out; I wondering if the new 3Tb disks would work in this scenario; most importantly will 2.6.18-238.9.1.el5PAE see the whole disk? And also whether I need to be worried about 4Kbyte sectors.
You'll have to change to GPT partitioning to use>2TB; for instance:
Only if I'm partitioning the disk, surely. If I just throw the whole raw disk into mdadm (sdd, sde, sdf...) then isn't partitioning irrelevant?
So...you were planning on treating the RAID as a raw block device, too, then? :) Otherwise, when you go to partition *it*, I think you'll find that you need to use GPT.
Linux 2.6.32-131.2.1.el6.i686 #1 SMP Wed May 18 07:07:27 EDT 2011 i686 i686 i386 GNU/Linux
Hmm, el6. Nice to know that will work when CentOS6 comes out; maybe I'll wait until then and rebuild my machine with 64bit mode.
It's a good idea. These many-TB filesystems really push the limits of a 32-bit system.
On Mon, Jun 20, 2011 at 12:29:50PM -0600, Warren Young wrote:
On 6/18/2011 8:51 PM, Stephen Harris wrote:
Only if I'm partitioning the disk, surely. If I just throw the whole raw disk into mdadm (sdd, sde, sdf...) then isn't partitioning irrelevant?
So...you were planning on treating the RAID as a raw block device, too, then? :) Otherwise, when you go to partition *it*, I think you'll find that you need to use GPT.
LVM; pvcreate, vgcreate, lvcreate. What's a partition? :-)
On 20.6.2011 21:19, Stephen Harris wrote:
On Mon, Jun 20, 2011 at 12:29:50PM -0600, Warren Young wrote:
On 6/18/2011 8:51 PM, Stephen Harris wrote:
Only if I'm partitioning the disk, surely. If I just throw the whole raw disk into mdadm (sdd, sde, sdf...) then isn't partitioning irrelevant?
So...you were planning on treating the RAID as a raw block device, too, then? :) Otherwise, when you go to partition *it*, I think you'll find that you need to use GPT.
LVM; pvcreate, vgcreate, lvcreate. What's a partition? :-)
But your filesystem has still to be aligned correctly. Is lvm 4k friendly ? Is md ?
On 6/20/2011 2:32 PM, Markus Falb wrote:
On 20.6.2011 21:19, Stephen Harris wrote:
On Mon, Jun 20, 2011 at 12:29:50PM -0600, Warren Young wrote:
On 6/18/2011 8:51 PM, Stephen Harris wrote:
Only if I'm partitioning the disk, surely. If I just throw the whole raw disk into mdadm (sdd, sde, sdf...) then isn't partitioning irrelevant?
So...you were planning on treating the RAID as a raw block device, too, then? :) Otherwise, when you go to partition *it*, I think you'll find that you need to use GPT.
LVM; pvcreate, vgcreate, lvcreate. What's a partition? :-)
But your filesystem has still to be aligned correctly. Is lvm 4k friendly ? Is md ?
And in the scenario of the whole disk as an md member, will that auto-assemble at boot time like partitions marked as type FD?
On Mon, Jun 20, 2011 at 09:32:03PM +0200, Markus Falb wrote:
On 20.6.2011 21:19, Stephen Harris wrote:
LVM; pvcreate, vgcreate, lvcreate. What's a partition? :-)
But your filesystem has still to be aligned correctly. Is lvm 4k friendly ? Is md ?
That is a very good question, and one of the underlying aspects of my original question :-)
At Mon, 20 Jun 2011 16:46:18 -0400 CentOS mailing list centos@centos.org wrote:
On Mon, Jun 20, 2011 at 09:32:03PM +0200, Markus Falb wrote:
On 20.6.2011 21:19, Stephen Harris wrote:
LVM; pvcreate, vgcreate, lvcreate. What's a partition? :-)
But your filesystem has still to be aligned correctly. Is lvm 4k friendly ? Is md ?
That is a very good question, and one of the underlying aspects of my original question :-)
From 'man vgcreate':
-s, --physicalextentsize PhysicalExtentSize[bBsSkKmMgGtTpPeE] Sets the physical extent size on physical volumes of this volume group. A size suffix (k for kilobytes up to t for terabytes) is optional, megabytes is the default if no suffix is present. The default is 4 MB and it must be at least 1 KB and a power of 2.
This suggests that so long as the Volume Group is properly aligned and the PE's are defined as a multiple of 4KB, logical volumes will be properly aligned. At least that is how I read it.
From 'man mdadm':
-c, --chunk= Specify chunk size in kibibytes. The default is 64.
This also suggests that so long as the physical partitions are properly aligned and so long as one does not specify something strange for this option (eg something other than a multiple of 4), then the stripes in the RAID array will be properly aligned. I would *guess* that using a whole bare disk would be aligned (at LBA 0) and any partitions created with a proper partitioning tool (GNU Parted) will be properly aligned.
On Jun 20, 2011, at 4:46 PM, Stephen Harris lists@spuddy.org wrote:
On Mon, Jun 20, 2011 at 09:32:03PM +0200, Markus Falb wrote:
On 20.6.2011 21:19, Stephen Harris wrote:
LVM; pvcreate, vgcreate, lvcreate. What's a partition? :-)
But your filesystem has still to be aligned correctly. Is lvm 4k friendly ? Is md ?
That is a very good question, and one of the underlying aspects of my original question :-)
4K, yes, but you also need to make sure it is aligned with the RAID chunk size.
I typically use pvcreate --metadatasize option to make sure it is padded enough so LV 1 has the desired offset. A PV meta data size of 960K, which combined with the 64K VG meta data will make sure LV 1 starts at sector 2048 (1MB offset) which will align with RAID chunk sizes up to 1MB.
The default PV meta data size is 128K, plus the 64K VG meta data, puts LV 1 with a 192K offset, which works for 64K chunk sizes, but not 128K chunk sizes that I typically use.
That is for whole disks, if you are building PVs from partitions you should take the partition offset into account when choosing the PV meta data padding.
Always specify the partition offset, don't take the default which is sector 63, which is useless and hard to fix later. You can't go wrong with starting at sector 2048 and if that becomes the future default it leaves some nice room up front for a fancy boot loader.
-Ross
On 6/18/2011 8:00 PM, Stephen Harris wrote:
I wondering if the new 3Tb disks would work in this scenario;
I've tried 3 TB disks on a 32-bit CentOS 5 box, plugged into generic Intel PIIX on-board SATA ports. It seemed to work fine.
I only did it to see if it would work, so I currently have no systems deployed like that. Everything using 3 TB disks in production right now does so via a 3ware RAID controller. The big gotcha there is that only their newest controllers (9700 series) support 3 TB disks.
On 6/20/2011 1:34 PM, Warren Young wrote:
On 6/18/2011 8:00 PM, Stephen Harris wrote:
I wondering if the new 3Tb disks would work in this scenario;
I've tried 3 TB disks on a 32-bit CentOS 5 box, plugged into generic Intel PIIX on-board SATA ports. It seemed to work fine.
I only did it to see if it would work, so I currently have no systems deployed like that. Everything using 3 TB disks in production right now does so via a 3ware RAID controller. The big gotcha there is that only their newest controllers (9700 series) support 3 TB disks.
If they are 4k sector devices you need to be sure the partitions are aligned correctly or you will have a huge hit in write performance (although everything will still appear to be working). I don't think the tools in 5.x know how to do this automatically yet.
-
Lamar Owen -
Les Mikesell -
Markus Falb -
Robert Heller -
Ross Walker -
Stephen Harris -
Warren Young