On Mon, 2011-07-25 at 13:23 -0400, R P Herrold wrote:

On Mon, 25 Jul 2011, Les Mikesell wrote:

...

My questions for any filesystem experts are:

Is there a way to adjust the existing md partitions to get the right alignment for 4k sectors without having to do a file-oriented copy to new partitions? A resize + a dd copy to shift the position might be feasible time-wise if that would work.

no expert here, but I have the scars across my back from pulling arrows out, as a pioneer

We have hit the issue on our storage backend which runs ext4, and on some of our dom0 built before the 4k sector alignment was generally acknowledged and known to be potentially in play

We have some non-conformant units, and after seaching, concluded that a 'wipe and rebuild' was the most time efficient process for us -- YMMV

...

Is it worth converting to ext4?

ext4 is pleasant in some large filesystem cases, but probably overkill as a blanket option.

Certainly it is 'wayy overkill for domU as a general rule, as it makes for a more fragile image in the sense that generic tools are less likely to work without higher version and skill levels when a filesystem gets horked up and a repair expedition has to be mounted ... we had an issue that a 'dirty' filesystem that would not fsck kept showing up in a nightly backup exception report, and ended up manually repairing what should have been able to be repaired automatically

...

Is there a difference between doing this on 5.6 or 6.x?

in C5, it took extra effort to use the technology preview; in C6 it is natively available

...

If I start over from scratch with 6.x, will the partitioning tools automatically align for 4k sector drives (with/without lvm?)?

no idea if gparted does this by default -- it does not in all versions; certainly fdisk did not -- 4k alignment is on our deployment checklist, and we are manually checking partitioning to make sure, when we are rebuilding boxes

I can only comment on the last section

I have built Centos 6.0 on a SSD using F15 version of gdisk man gdisk shows for the l option "Change the sector alignment value. Disks with more logical sectors per physical sectors (such as some Western Digital models introduced in December of 2009) and some RAID configurations can suffer performance problems if partitions are not aligned properly for their internal data structures. On new disks, GPT fdisk attempts to align partitions on 2048-sector (1MiB) boundaries by default, which optimizes performance for both of these disk types."

Only straight ext4 parttions (No lvm) I have seen no problems so far ...

Number Start (sector) End (sector) Size Code Name 1 2048 4095 1024.0 KiB EF02 BIOS boot partition 2 4096 2101247 1024.0 MiB 0700 Linux/Windows data 3 2101248 6295551 2.0 GiB 8200 Linux swap 4 6295552 69210111 30.0 GiB 0700 Linux/Windows data 5 69210112 132124671 30.0 GiB 0700 Linux/Windows data 6 132124672 174067711 20.0 GiB 0700 Linux/Windows data 7 174067712 468862094 140.6 GiB 0700 Linux/Windows data

John

--On Monday, July 25, 2011 01:56:38 PM -0400 m.roth@5-cent.us wrote:

I think it was when I was building a 6.0 box a couple weeks ago, but I'd partition, it would do an mkfs... and *then* tell me it wasn't aligned, and I played with it several times, and it absolutely would NOT align it, nor offer to do so.

When I was building out a 6.0 box a few days ago using 4k sector drives, I first booted into rescue mode and partitioned using fdisk via: fdisk -uc -H 224 -S 56 /dev/sd{a,b,c,d} (I'm not sure, but the -H and -S might be irrelevent due to the -uc.)

The first partition then defaulted to starting at sector 2048 (one MB), size of 200MB. On all disks, one other partition was created holding the remainder of the disk. All partition types were set to 0xfd.

I then booted the install disk normally and eventually things got configured so that partion 1 on all drives makes up a 200MB mirrored /dev/md0 for /boot, and everthing else went into /dev/md1 as RAID6.

As far as I can tell, I've not buggered things up ...

Devin

--On Monday, July 25, 2011 02:19:16 PM -0400 m.roth@5-cent.us wrote:

No joy - I think I have to use parted - the drive was too big for fdisk.

I should have mentioned that this was with 1.5TB disks. I think there's a limit somewhere beyond 2TB for fdisk.

Devin

On 7/25/2011 1:42 PM, Brian Mathis wrote:

I've wondered many times, though haven't tried it, if the issues with hard links and backuppc could be solved by using a container file with a loopback mount, and then that file could be moved around as needed without running into hard-link issues.

In this case, you could format the external drive in the optimal mode for 4k sectors, then create a container file and mount it using loopback. Then add the loopback device to the mdraid and have it sync.

It doesn't really help with the problem as it stands, which is that the target disk (a swappable sata, not really external) has no extra space that would permit shifting the alignment. It might work to shrink the existing size, then partition the new drives with the right offset, but I may just start from scratch and keep the old drives around in case I need the old history.

On 7/25/2011 4:05 PM, Brian Mathis wrote:

...

On 7/25/2011 1:42 PM, Brian Mathis wrote:

...

I've wondered many times, though haven't tried it, if the issues with hard links and backuppc could be solved by using a container file with a loopback mount, and then that file could be moved around as needed without running into hard-link issues.

In this case, you could format the external drive in the optimal mode for 4k sectors, then create a container file and mount it using loopback. Then add the loopback device to the mdraid and have it sync.

It doesn't really help with the problem as it stands, which is that the target disk (a swappable sata, not really external) has no extra space that would permit shifting the alignment. It might work to shrink the existing size, then partition the new drives with the right offset, but I may just start from scratch and keep the old drives around in case I need the old history.

I thought this was a 3-disk RAID1? Can't you repartition the hotswap disk and still have the data on the other 2? Why would you need to shrink the existing partition? Just blow it away and resync the data once you rebuild the disk.

The disk I want to add is the same size as the existing disks if expressed in 512 byte sectors - and they have one partition taking all of the disk space. If I add a leading offset to get the 4k alignment, there won't be enough room for the existing partition size.

On 7/25/2011 4:26 PM, John R Pierce wrote:

On 07/25/11 2:17 PM, Les Mikesell wrote:

...

The disk I want to add is the same size as the existing disks if expressed in 512 byte sectors - and they have one partition taking all of the disk space. If I add a leading offset to get the 4k alignment, there won't be enough room for the existing partition size.

you sure its that tight? different brand and model 1TB (or whatever) drives vary all over the place in actual size. generally newer ones are a hair bigger than older ones. you need at most 7 sectors to achieve 4 kilobyte alignment.

The full sized disks are Seagates:

Host: scsi7 Channel: 00 Id: 00 Lun: 00 Vendor: ATA Model: ST3750640NS Rev: 3.AE and fdisk sees this: Disk /dev/sdh: 750.1 GB, 750156374016 bytes 255 heads, 63 sectors/track, 91201 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

The 2.5" ones are WD's: Host: scsi9 Channel: 00 Id: 00 Lun: 00 Vendor: ATA Model: WDC WD7500BPVT-0 Rev: 01.0 Type: Direct-Access ANSI SCSI revision: 05 fdisk: Disk /dev/sdi: 750.1 GB, 750156374016 bytes 255 heads, 63 sectors/track, 91201 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Don't see any extra space there unless you can shift the partition start forwards.

There's a very new 1 TB drive that might fit in the swappable bays (a cute little thing that fits 2 in a floppy drive space), but when I got these 750Gb was as large as you could go without adding extra height.

On 7/25/2011 5:33 PM, John R Pierce wrote:

On 07/25/11 2:44 PM, Les Mikesell wrote:

...

255 heads, 63 sectors/track, 91201 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

where is your existing partition starting?

if its on a track or cylinder boundary... then sure, you can move it forward by using something that will let you partition by sectors.

Device Boot Start End Blocks Id System /dev/sdh1 1 91201 732572001 fd Linux raid autodetect

It doesn't need to boot. And the 3rd member doesn't need to autodetect, although I do want to be able to mount it independently if needed. Should it work to use the raw disk instead of a partition?

On 07/25/2011 10:10 AM, Les Mikesell wrote:

I've mentioned this problem before but put off doing anything about it and maybe now someone can suggest the best solution.

I have a 3-member RAID1 set where one of the members is periodically swapped and rotated offsite. The filesystem contains a backuppc archive which has millions of hardlinks that make it impractical to copy with a file-oriented approach. The current filesystem is ext3 with one partition that uses the entire disk capacity (no lvm). It works as is, but...

I'd like to use a laptop size drive for the swapped member and the only ones available that match the size have 4k sectors. I have swappable, trayless SATA bays available for both drive sizes. The problem is that with the current partition layout, the drive with 4k sectors takes more than a day to re-sync even though on read access the speed is a match for the full sized drives that sync in a few hours.

My questions for any filesystem experts are:

Is there a way to adjust the existing md partitions to get the right alignment for 4k sectors without having to do a file-oriented copy to new partitions? A resize + a dd copy to shift the position might be feasible time-wise if that would work.

Is it worth converting to ext4?

Is there a difference between doing this on 5.6 or 6.x?

If I start over from scratch with 6.x, will the partitioning tools automatically align for 4k sector drives (with/without lvm?)?

For LVM's see the --dataalignment and --dataalignmentoffset options. For md devices, my understanding is that the raid superblock is at the end of the partition, so the data is aligned with wherever the partition starts. I verified this using: hexdump /dev/md1 | head -6 hexdump /dev/sda4 | head -6

Nataraj

From: John R Pierce pierce@hogranch.com

thats by "cylinder", which is an old MSDOS legacy thing.  I believe parted and probably some other programs let you partition by sector instead.

In my kickstart pre script, I use: ... | sfdisk -H $HEADS -S $SECTORS -uS --force -L $DEVICE For SSDs, I saw the recommended respective values: 224 56 (or 32 32) fdisk also has -u for sectors unit, and -H/-S to force a fake geometry. But, from fdisk man page: "-b sectorsize       Specify the sector size of the disk. Valid values are 512, 1024,       or 2048.  (Recent kernels know the sector size. Use this only on       old kernels or to override the kernel’s ideas.)" Which would seem to imply that fdisk is limited to 2K sector sizes?

I do not deal with drives above 2GB though... But I must admit that I am still a bit confused with all these alignments...

JD

On 7/26/2011 3:11 AM, Andrzej Szymanski wrote:

On 2011-07-25 19:10, Les Mikesell wrote:

...

My questions for any filesystem experts are:

Is there a way to adjust the existing md partitions to get the right alignment for 4k sectors without having to do a file-oriented copy to new partitions? A resize + a dd copy to shift the position might be feasible time-wise if that would work.

I think so. Your partition starts at sector 63, you need anything divisible by 8, so:

• 64 is my expectation,
• 56 is a fallback solution if the partition does not fit on the disk

with 64 sector offset

• 2048 would be perfect (1M alignment is currently preferred by Centos 6

and many other OSs)

To be on the safe side, take the disk out of the array (mdadm -f /dev/md0 /dev/sdX1 ; mdadm -r /dev/md0 /dev/sdX1) and clear superblock using mdadm --zero-superblock /dev/sdX1.

Then repartition the disk using fdisk using the following commands: fdisk /dev/sdX u -- display units are sectors c -- no DOS compatibility (== no cyllinder rounding, you definitely want that) o -- new dos partition table n -- new partition p -- primary 1 -- partition 1 64 -- starting offset 1465144065 -- exact size here, because (just to be on the safe side) you do not want to have a larger partition on a rescue disk than on a base disk. Your partition sdh1 has 732572001 1k-blocks, as you wrote in one e-mail, multiply this by 2 (sectors) add 64, (starting offset) subtract 1 because the offset is inclusive. You get 2*732572001+64-1=1465144065. If fdisk complains that this is too much then offset 64 cannot be used and you need to repeat the procedure using offset 56 (don't forget to recalculate ending sector). t -- type fd -- linux raid autodetect w

mdadm -a /dev/mdX /dev/sdX1

And everything should be fine.

Thank you! That seems to have worked, but now I'm curious as to why the partition on the old drives didn't go to the end of the disk - which I had expected would have left no extra room. Was the dos style rounding computing the end of a cylinder wrong?

fdisk -lu /dev/sdh (old 3.5") Disk /dev/sdh: 750.1 GB, 750156374016 bytes 255 heads, 63 sectors/track, 91201 cylinders, total 1465149168 sectors Units = sectors of 1 * 512 = 512 bytes Device Boot Start End Blocks Id System /dev/sdh1 63 1465144064 732572001 fd Linux raid autodetect

fdisk -lu /dev/sdi (new 2.5") Disk /dev/sdi: 750.1 GB, 750156374016 bytes 1 heads, 1 sectors/track, 1465149168 cylinders, total 1465149168 sectors Units = sectors of 1 * 512 = 512 bytes Device Boot Start End Blocks Id System /dev/sdi1 64 1465144065 732572001 fd Linux raid autodetect

The laptop drive is still slower, but not 10x slower like before. I did try something like this earlier trying for a 56 sector offset but must have done something wrong.

Are there differences in the way CentOS 6.0 handles md raid1 arrays compared to earlier versions? After getting my drive with 4k sectors partitioned with a 64-sector starting offset and working under 5.6 with about a 20Mb/sec sync rate, I booted with the 6.0 livecd to see if there would be any improvement, but instead of autodetecting the pairing, it assembled each partition into its own md device with missing members. And the sync of the drive with the 4k sectors would only go at about 4Mb/sec (regardless of what I put in /proc/sys/dev/raid/speed_limit_min and _max).

What was worse was that booting back to 5.6 left most of the raid pairs broken, some still having numbers in the md120 range, and one of the drives with an unrecognizable partition table. I think I can repair everything, but did I miss something about this in release notes somewhere?

On 7/26/2011 3:11 AM, Andrzej Szymanski wrote:

On 2011-07-25 19:10, Les Mikesell wrote:

...

My questions for any filesystem experts are:

Is there a way to adjust the existing md partitions to get the right alignment for 4k sectors without having to do a file-oriented copy to new partitions? A resize + a dd copy to shift the position might be feasible time-wise if that would work.

I think so. Your partition starts at sector 63, you need anything divisible by 8, so:

• 64 is my expectation,
• 56 is a fallback solution if the partition does not fit on the disk

with 64 sector offset

• 2048 would be perfect (1M alignment is currently preferred by Centos 6

and many other OSs)

To be on the safe side, take the disk out of the array (mdadm -f /dev/md0 /dev/sdX1 ; mdadm -r /dev/md0 /dev/sdX1) and clear superblock using mdadm --zero-superblock /dev/sdX1.

Then repartition the disk using fdisk using the following commands: fdisk /dev/sdX u -- display units are sectors c -- no DOS compatibility (== no cyllinder rounding, you definitely want that) o -- new dos partition table n -- new partition p -- primary 1 -- partition 1 64 -- starting offset 1465144065 -- exact size here, because (just to be on the safe side) you do not want to have a larger partition on a rescue disk than on a base disk. Your partition sdh1 has 732572001 1k-blocks, as you wrote in one e-mail, multiply this by 2 (sectors) add 64, (starting offset) subtract 1 because the offset is inclusive. You get 2*732572001+64-1=1465144065. If fdisk complains that this is too much then offset 64 cannot be used and you need to repeat the procedure using offset 56 (don't forget to recalculate ending sector). t -- type fd -- linux raid autodetect w

mdadm -a /dev/mdX /dev/sdX1

And everything should be fine.

One more follow-up on this. This did work fine but it turned out that the disk I was using was defective which is probably what threw off my earlier attempts to get the alignment right. It would start with some promising values for the resync speed, but would keep slowing down more and more as it went and eventually it got to the point where there were errors reported. I returned it and the replacement is a pretty close match to the 3.5" drives in speed.