Has anything changed in updates that would affect md raid1 resync speed? I regularly swap a 750G drive and resync to keep an offsite copy and haven't paid enough attention to known when things changed but it seems to take much longer to sync than it did months ago, even if I unmount the partition and stop most other processes that might compete with it.
2010/5/13 Les Mikesell lesmikesell@gmail.com:
Has anything changed in updates that would affect md raid1 resync speed? I regularly swap a 750G drive and resync to keep an offsite copy and haven't paid enough attention to known when things changed but it seems to take much longer to sync than it did months ago, even if I unmount the partition and stop most other processes that might compete with it.
well, try this:
http://www.ducea.com/2006/06/25/increase-the-speed-of-linux-software-raid-re...
-- Eero, RHCE
On Thu, 13 May 2010, Eero Volotinen wrote:
well, try this:
http://www.ducea.com/2006/06/25/increase-the-speed-of-linux-software-raid-re...
Interesting link. Thanks. I'll be interested to try it out next time I swap out a RAID disk.
On 5/13/2010 1:42 PM, Eero Volotinen wrote:
2010/5/13 Les Mikeselllesmikesell@gmail.com:
Has anything changed in updates that would affect md raid1 resync speed? I regularly swap a 750G drive and resync to keep an offsite copy and haven't paid enough attention to known when things changed but it seems to take much longer to sync than it did months ago, even if I unmount the partition and stop most other processes that might compete with it.
well, try this:
http://www.ducea.com/2006/06/25/increase-the-speed-of-linux-software-raid-re...
OK, I can get a full-size Seagate 750G to resync at about 40M/s which easily completes in a workday. But now what I really want to do is use a laptop size 'WD Scorpio blue' drive which claims to have the same sector count but will only sync at about a tenth of the speed. It does say it uses 'advanced format', which I think means 4k sectors. Should that make a difference? Read speed tests show about the same as the desktop drives but writes are much slower.
Les Mikesell wrote:
OK, I can get a full-size Seagate 750G to resync at about 40M/s which easily completes in a workday. But now what I really want to do is use a laptop size 'WD Scorpio blue' drive which claims to have the same sector count but will only sync at about a tenth of the speed. It does say it uses 'advanced format', which I think means 4k sectors. Should that make a difference? Read speed tests show about the same as the desktop drives but writes are much slower.
From what I've read, you have to 'trim' those drives much like SSD so the partition starts on a 4K boundary. I understand you can do this with parted.
On 5/21/2010 12:13 PM, John R Pierce wrote:
Les Mikesell wrote:
OK, I can get a full-size Seagate 750G to resync at about 40M/s which easily completes in a workday. But now what I really want to do is use a laptop size 'WD Scorpio blue' drive which claims to have the same sector count but will only sync at about a tenth of the speed. It does say it uses 'advanced format', which I think means 4k sectors. Should that make a difference? Read speed tests show about the same as the desktop drives but writes are much slower.
From what I've read, you have to 'trim' those drives much like SSD so the partition starts on a 4K boundary. I understand you can do this with parted.
But it is just a match for the Seagate drives with the default layout using one partition that fills the disk. If I have to skip some amount at the start of the partition I think that will make the partition size not match, making it impossible to add as a raid member.
Les Mikesell wrote:
But it is just a match for the Seagate drives with the default layout using one partition that fills the disk. If I have to skip some amount at the start of the partition I think that will make the partition size not match, making it impossible to add as a raid member.
750GB drives can vary quite a bit in size +/- depending on the exact model. The required trim is 0-3.5K, and its due to the MBR on the first "track" of the drive before the first/primary partition starts.
On 5/21/2010 4:19 PM, John R Pierce wrote:
Les Mikesell wrote:
But it is just a match for the Seagate drives with the default layout using one partition that fills the disk. If I have to skip some amount at the start of the partition I think that will make the partition size not match, making it impossible to add as a raid member.
750GB drives can vary quite a bit in size +/- depending on the exact model. The required trim is 0-3.5K, and its due to the MBR on the first "track" of the drive before the first/primary partition starts.
The 3.5" Seagates look like: Disk /dev/sdc: 750.1 GB, 750156374016 bytes 255 heads, 63 sectors/track, 91201 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sdc1 1 91201 732572001 fd Linux raid autodetect
And the 2.5 WD looks the same to me:
Disk /dev/sdh: 750.1 GB, 750156374016 bytes 255 heads, 63 sectors/track, 91201 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sdh1 1 91201 732572001 fd Linux raid autodetect
These is a backuppc archive with millions of hardlinks that will take forever to copy if I have to do a file-oriented copy onto a different partition size.
On 05/21/2010 02:32 PM, Les Mikesell wrote: [..]
Disk /dev/sdh: 750.1 GB, 750156374016 bytes 255 heads, 63 sectors/track, 91201 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sdh1 1 91201 732572001 fd Linux raid autodetect
These is a backuppc archive with millions of hardlinks that will take forever to copy if I have to do a file-oriented copy onto a different partition size.
You can cheat. Remove partition 1 and use the *entire drive* as a RAID volume (no partition table at all).
On 5/21/2010 4:37 PM, Benjamin Franz wrote:
On 05/21/2010 02:32 PM, Les Mikesell wrote: [..]
Disk /dev/sdh: 750.1 GB, 750156374016 bytes 255 heads, 63 sectors/track, 91201 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sdh1 1 91201 732572001 fd Linux raid autodetect
These is a backuppc archive with millions of hardlinks that will take forever to copy if I have to do a file-oriented copy onto a different partition size.
You can cheat. Remove partition 1 and use the *entire drive* as a RAID volume (no partition table at all).
But can I change the existing setup to do that without losing the contents? (It's a 3-member raid1 where one member is only added long enough to sync, then removed). And if you raid at the disk level, how do you tell it to reconnect at startup (the equivalent of the FD partition type)?
Les Mikesell wrote:
These is a backuppc archive with millions of hardlinks that will take forever to copy if I have to do a file-oriented copy onto a different partition size.
have you tried a dump | restore style Ext{3|2}FS replica? that works by inode and does it pretty efficiently. catch22, the source shouldn't be mounted.
On 5/21/2010 5:46 PM, John R Pierce wrote:
Les Mikesell wrote:
These is a backuppc archive with millions of hardlinks that will take forever to copy if I have to do a file-oriented copy onto a different partition size.
have you tried a dump | restore style Ext{3|2}FS replica? that works by inode and does it pretty efficiently. catch22, the source shouldn't be mounted.
I haven't tried on this machine, but with an earlier 250 gig setup I was never able to complete a copy over a weekend with any of the file-oriented approaches. Dump can dump the inode numbers but restore will remap them to different numbers, needing a lookup table for all files with multiple hard links - which with backuppc is all of them.
On 05/21/2010 04:32 PM, Les Mikesell wrote:
On 5/21/2010 4:19 PM, John R Pierce wrote:
Les Mikesell wrote:
But it is just a match for the Seagate drives with the default layout using one partition that fills the disk. If I have to skip some amount at the start of the partition I think that will make the partition size not match, making it impossible to add as a raid member.
750GB drives can vary quite a bit in size +/- depending on the exact model. The required trim is 0-3.5K, and its due to the MBR on the first "track" of the drive before the first/primary partition starts.
The 3.5" Seagates look like: Disk /dev/sdc: 750.1 GB, 750156374016 bytes 255 heads, 63 sectors/track, 91201 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sdc1 1 91201 732572001 fd Linux raid autodetect
And the 2.5 WD looks the same to me:
Disk /dev/sdh: 750.1 GB, 750156374016 bytes 255 heads, 63 sectors/track, 91201 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sdh1 1 91201 732572001 fd Linux raid autodetect
You have another way out. By my calculation, that drive is partitioned in DOS compatibility mode, which leaves the remainder of the MBR track unused. Running fdisk in expert mode ("x" command), you can move the partition's beginning of data ("b" command) from sector 63 back to sector 56. That will give you the needed 4K alignment and a partition that is no smaller than what it was before.
On 05/21/2010 07:39 PM, Robert Nichols wrote:
You have another way out. By my calculation, that drive is partitioned in DOS compatibility mode, which leaves the remainder of the MBR track unused. Running fdisk in expert mode ("x" command), you can move the partition's beginning of data ("b" command) from sector 63 back to sector 56. That will give you the needed 4K alignment and a partition that is no smaller than what it was before.
Right idea, not the right procedure. You'll need to turn off DOS compatibility mode, then create the partition, and then go into expert mode and move the beginning of data from sector 1 to sector 56.
Robert Nichols wrote:
On 05/21/2010 07:39 PM, Robert Nichols wrote:
You have another way out. By my calculation, that drive is partitioned in DOS compatibility mode, which leaves the remainder of the MBR track unused. Running fdisk in expert mode ("x" command), you can move the partition's beginning of data ("b" command) from sector 63 back to sector 56. That will give you the needed 4K alignment and a partition that is no smaller than what it was before.
Right idea, not the right procedure. You'll need to turn off DOS compatibility mode, then create the partition, and then go into expert mode and move the beginning of data from sector 1 to sector 56.
It ended up like this, but still sync'ing at about 4M/sec instead of 40.
Expert command (m for help): p
Disk /dev/sdh: 255 heads, 63 sectors, 91201 cylinders
Nr AF Hd Sec Cyl Hd Sec Cyl Start Size ID 1 00 0 2 0 254 63 64 56 1465144009 fd
On 05/22/2010 11:29 AM, Les Mikesell wrote:
Robert Nichols wrote:
On 05/21/2010 07:39 PM, Robert Nichols wrote:
You have another way out. By my calculation, that drive is partitioned in DOS compatibility mode, which leaves the remainder of the MBR track unused. Running fdisk in expert mode ("x" command), you can move the partition's beginning of data ("b" command) from sector 63 back to sector 56. That will give you the needed 4K alignment and a partition that is no smaller than what it was before.
Right idea, not the right procedure. You'll need to turn off DOS compatibility mode, then create the partition, and then go into expert mode and move the beginning of data from sector 1 to sector 56.
It ended up like this, but still sync'ing at about 4M/sec instead of 40.
Expert command (m for help): p
Disk /dev/sdh: 255 heads, 63 sectors, 91201 cylinders
Nr AF Hd Sec Cyl Hd Sec Cyl Start Size ID 1 00 0 2 0 254 63 64 56 1465144009 fd
Is that one of those WD drives that falsely reports its physical sector size as 512 bytes? I don't know if the kernel can always do the right thing when that happens, but all the reports I've seen say that getting the start of the partition aligned properly is sufficient.
What does "hdparm -I /dev/sdh | grep 'Sector size'" show?
Robert Nichols wrote:
On 05/22/2010 11:29 AM, Les Mikesell wrote:
Robert Nichols wrote:
On 05/21/2010 07:39 PM, Robert Nichols wrote:
You have another way out. By my calculation, that drive is partitioned in DOS compatibility mode, which leaves the remainder of the MBR track unused. Running fdisk in expert mode ("x" command), you can move the partition's beginning of data ("b" command) from sector 63 back to sector 56. That will give you the needed 4K alignment and a partition that is no smaller than what it was before.
Right idea, not the right procedure. You'll need to turn off DOS compatibility mode, then create the partition, and then go into expert mode and move the beginning of data from sector 1 to sector 56.
It ended up like this, but still sync'ing at about 4M/sec instead of 40.
Expert command (m for help): p
Disk /dev/sdh: 255 heads, 63 sectors, 91201 cylinders
Nr AF Hd Sec Cyl Hd Sec Cyl Start Size ID 1 00 0 2 0 254 63 64 56 1465144009 fd
Is that one of those WD drives that falsely reports its physical sector size as 512 bytes? I don't know if the kernel can always do the right thing when that happens, but all the reports I've seen say that getting the start of the partition aligned properly is sufficient.
What does "hdparm -I /dev/sdh | grep 'Sector size'" show?
It doesn't mention sector size. All of the size related options seem to match the Seagate desktop drive.
Les Mikesell wrote:
Robert Nichols wrote:
On 05/22/2010 11:29 AM, Les Mikesell wrote:
Robert Nichols wrote:
On 05/21/2010 07:39 PM, Robert Nichols wrote:
You have another way out. By my calculation, that drive is partitioned in DOS compatibility mode, which leaves the remainder of the MBR track unused. Running fdisk in expert mode ("x" command), you can move the partition's beginning of data ("b" command) from sector 63 back to sector 56. That will give you the needed 4K alignment and a partition that is no smaller than what it was before.
Right idea, not the right procedure. You'll need to turn off DOS compatibility mode, then create the partition, and then go into expert mode and move the beginning of data from sector 1 to sector 56.
It ended up like this, but still sync'ing at about 4M/sec instead of 40.
Expert command (m for help): p
Disk /dev/sdh: 255 heads, 63 sectors, 91201 cylinders
Nr AF Hd Sec Cyl Hd Sec Cyl Start Size ID 1 00 0 2 0 254 63 64 56 1465144009 fd
Is that one of those WD drives that falsely reports its physical sector size as 512 bytes? I don't know if the kernel can always do the right thing when that happens, but all the reports I've seen say that getting the start of the partition aligned properly is sufficient.
What does "hdparm -I /dev/sdh | grep 'Sector size'" show?
It doesn't mention sector size. All of the size related options seem to match the Seagate desktop drive.
Does the 4K sector size mean that the drive is going to read the 4K chunk then merge in the 512 bytes you wrote, the wait for the sector come around again to write it back? I guess that could explain the 10x write speed difference regardless of cylinder alignment. Read speed doesn't seem that much different.
On 05/22/2010 05:46 PM, Les Mikesell wrote:
Les Mikesell wrote:
Robert Nichols wrote:
On 05/22/2010 11:29 AM, Les Mikesell wrote:
Robert Nichols wrote:
On 05/21/2010 07:39 PM, Robert Nichols wrote:
You have another way out. By my calculation, that drive is partitioned in DOS compatibility mode, which leaves the remainder of the MBR track unused. Running fdisk in expert mode ("x" command), you can move the partition's beginning of data ("b" command) from sector 63 back to sector 56. That will give you the needed 4K alignment and a partition that is no smaller than what it was before.
Right idea, not the right procedure. You'll need to turn off DOS compatibility mode, then create the partition, and then go into expert mode and move the beginning of data from sector 1 to sector 56.
It ended up like this, but still sync'ing at about 4M/sec instead of 40.
Expert command (m for help): p
Disk /dev/sdh: 255 heads, 63 sectors, 91201 cylinders
Nr AF Hd Sec Cyl Hd Sec Cyl Start Size ID 1 00 0 2 0 254 63 64 56 1465144009 fd
Is that one of those WD drives that falsely reports its physical sector size as 512 bytes? I don't know if the kernel can always do the right thing when that happens, but all the reports I've seen say that getting the start of the partition aligned properly is sufficient.
What does "hdparm -I /dev/sdh | grep 'Sector size'" show?
It doesn't mention sector size. All of the size related options seem to match the Seagate desktop drive.
Does the 4K sector size mean that the drive is going to read the 4K chunk then merge in the 512 bytes you wrote, the wait for the sector come around again to write it back? I guess that could explain the 10x write speed difference regardless of cylinder alignment. Read speed doesn't seem that much different.
Yes, that's exactly what it means. Every unaligned write or write that is not a multiple of the 4KB sector size becomes a read-modify-write within the drive, and a 10X reduction in write throughput is typical.
On 05/22/2010 07:39 PM, Robert Nichols wrote:
On 05/22/2010 05:46 PM, Les Mikesell wrote:
Does the 4K sector size mean that the drive is going to read the 4K chunk then merge in the 512 bytes you wrote, the wait for the sector come around again to write it back? I guess that could explain the 10x write speed difference regardless of cylinder alignment. Read speed doesn't seem that much different.
Yes, that's exactly what it means. Every unaligned write or write that is not a multiple of the 4KB sector size becomes a read-modify-write within the drive, and a 10X reduction in write throughput is typical.
I should add that the kernel normally will do I/O in multiples of its 4KB (typical) page size where possible, but I have no idea whether any effort is made to align those writes if the drive does not report a 4KB physical sector size, or whether it even makes sense to try beyond what the elevator algorithm does for coalescing sequential writes.
I don't currently have any of these "enhanced format" drives, nor am I using RAID, so all I can report is the collected experience of others.
Robert Nichols wrote:
On 05/22/2010 07:39 PM, Robert Nichols wrote:
On 05/22/2010 05:46 PM, Les Mikesell wrote:
Does the 4K sector size mean that the drive is going to read the 4K chunk then merge in the 512 bytes you wrote, the wait for the sector come around again to write it back? I guess that could explain the 10x write speed difference regardless of cylinder alignment. Read speed doesn't seem that much different.
Yes, that's exactly what it means. Every unaligned write or write that is not a multiple of the 4KB sector size becomes a read-modify-write within the drive, and a 10X reduction in write throughput is typical.
I should add that the kernel normally will do I/O in multiples of its 4KB (typical) page size where possible, but I have no idea whether any effort is made to align those writes if the drive does not report a 4KB physical sector size, or whether it even makes sense to try beyond what the elevator algorithm does for coalescing sequential writes.
I don't currently have any of these "enhanced format" drives, nor am I using RAID, so all I can report is the collected experience of others.
Well, the form factor is certainly nice. I got a hot-swap carrier with 2 slots that fits in a floppy bay and the drives themselves are tiny so it seemed ideal for copies of data to go offsite. I just wish it would work... Even a dd at the disk level seems slow so I'm not sure the writes are being aggregated even if you ignore partitioning and offsets.
On 05/22/2010 08:40 PM, Les Mikesell wrote:
Robert Nichols wrote:
On 05/22/2010 07:39 PM, Robert Nichols wrote: I should add that the kernel normally will do I/O in multiples of its 4KB (typical) page size where possible, but I have no idea whether any effort is made to align those writes if the drive does not report a 4KB physical sector size, or whether it even makes sense to try beyond what the elevator algorithm does for coalescing sequential writes.
I don't currently have any of these "enhanced format" drives, nor am I using RAID, so all I can report is the collected experience of others.
Well, the form factor is certainly nice. I got a hot-swap carrier with 2 slots that fits in a floppy bay and the drives themselves are tiny so it seemed ideal for copies of data to go offsite. I just wish it would work... Even a dd at the disk level seems slow so I'm not sure the writes are being aggregated even if you ignore partitioning and offsets.
Another thing to keep in mind is that the SATA spec. only requires the internal SATA connector to withstand 50 insertions. I picked up some nice acomdata (TM) eSATA housings for the drives (512-byte sectors, thankfully) I use for my offsite backup copies. The eSATA connector is spec-ed for 50,000 insertions. My laptop came with a warning about the life of that connector, and I found those ratings mentioned at http://www.serialata.org/technology/esata.asp .
Robert Nichols wrote:
On 05/22/2010 08:40 PM, Les Mikesell wrote:
Robert Nichols wrote:
On 05/22/2010 07:39 PM, Robert Nichols wrote: I should add that the kernel normally will do I/O in multiples of its 4KB (typical) page size where possible, but I have no idea whether any effort is made to align those writes if the drive does not report a 4KB physical sector size, or whether it even makes sense to try beyond what the elevator algorithm does for coalescing sequential writes.
I don't currently have any of these "enhanced format" drives, nor am I using RAID, so all I can report is the collected experience of others.
Well, the form factor is certainly nice. I got a hot-swap carrier with 2 slots that fits in a floppy bay and the drives themselves are tiny so it seemed ideal for copies of data to go offsite. I just wish it would work... Even a dd at the disk level seems slow so I'm not sure the writes are being aggregated even if you ignore partitioning and offsets.
Another thing to keep in mind is that the SATA spec. only requires the internal SATA connector to withstand 50 insertions. I picked up some nice acomdata (TM) eSATA housings for the drives (512-byte sectors, thankfully) I use for my offsite backup copies. The eSATA connector is spec-ed for 50,000 insertions. My laptop came with a warning about the life of that connector, and I found those ratings mentioned at http://www.serialata.org/technology/esata.asp .
The trayless internal hotswap enclosures claim to be good for 10,000+ insertions and I'm using larger ones for the desktop drives I had been using without any problems. I have seen some postings to the effect that I need a newer kernel to recognize the 4k sectors besides doing the partition alignment. Maybe I can boot the RHEL 6 beta or a fedora iso and see if they are faster.
On 05/23/2010 01:57 PM, Les Mikesell wrote:
Robert Nichols wrote:
Another thing to keep in mind is that the SATA spec. only requires the internal SATA connector to withstand 50 insertions. I picked up some nice acomdata (TM) eSATA housings for the drives (512-byte sectors, thankfully) I use for my offsite backup copies. The eSATA connector is spec-ed for 50,000 insertions. My laptop came with a warning about the life of that connector, and I found those ratings mentioned at http://www.serialata.org/technology/esata.asp .
The trayless internal hotswap enclosures claim to be good for 10,000+ insertions and I'm using larger ones for the desktop drives I had been using without any problems. I have seen some postings to the effect that I need a newer kernel to recognize the 4k sectors besides doing the partition alignment. Maybe I can boot the RHEL 6 beta or a fedora iso and see if they are faster.
Sounds good, unless that 10,000 insertion spec is predicated on spreading the usage among 200 drives such that no individual drives has more than 50 insertions. ;-(
BTW, I got carried away typing zeros in that spec. for the eSATA connector. The right number is 5,000 insertions, not 50,000.
If you want a good write-up on what the kernel developers have to contend with to handle these 4KiB sector sizes, see https://ata.wiki.kernel.org/index.php/ATA_4_KiB_sector_issues
For those with a strong stomach for such things, a nice historical perspective on the horrors of ever-increasing drive sizes can be found at http://www.win.tue.nl/~aeb/linux/Large-Disk.html . Just be sure to keep the 2004 date in mind when it mentions "recent kernels."
On 5/24/2010 6:56 AM, Robert Nichols wrote:
The trayless internal hotswap enclosures claim to be good for 10,000+ insertions and I'm using larger ones for the desktop drives I had been using without any problems. I have seen some postings to the effect that I need a newer kernel to recognize the 4k sectors besides doing the partition alignment. Maybe I can boot the RHEL 6 beta or a fedora iso and see if they are faster.
Sounds good, unless that 10,000 insertion spec is predicated on spreading the usage among 200 drives such that no individual drives has more than 50 insertions. ;-(
BTW, I got carried away typing zeros in that spec. for the eSATA connector. The right number is 5,000 insertions, not 50,000.
If you want a good write-up on what the kernel developers have to contend with to handle these 4KiB sector sizes, see https://ata.wiki.kernel.org/index.php/ATA_4_KiB_sector_issues
For those with a strong stomach for such things, a nice historical perspective on the horrors of ever-increasing drive sizes can be found at http://www.win.tue.nl/~aeb/linux/Large-Disk.html . Just be sure to keep the 2004 date in mind when it mentions "recent kernels."
I booted a RHEL6 beta DVD in rescue mode, but it wasn't any better. That version of fdisk does show: Logical/Physical Sector size 512 bytes though, so it's probably hopeless. Oddly, that version of fdisk wouldn't let me move the beginning back below 63 either - but I did that with the 5.x fdisk.
On 05/24/2010 12:41 PM, Les Mikesell wrote:
On 5/24/2010 6:56 AM, Robert Nichols wrote:
The trayless internal hotswap enclosures claim to be good for 10,000+ insertions and I'm using larger ones for the desktop drives I had been using without any problems. I have seen some postings to the effect that I need a newer kernel to recognize the 4k sectors besides doing the partition alignment. Maybe I can boot the RHEL 6 beta or a fedora iso and see if they are faster.
Sounds good, unless that 10,000 insertion spec is predicated on spreading the usage among 200 drives such that no individual drives has more than 50 insertions. ;-(
BTW, I got carried away typing zeros in that spec. for the eSATA connector. The right number is 5,000 insertions, not 50,000.
If you want a good write-up on what the kernel developers have to contend with to handle these 4KiB sector sizes, see https://ata.wiki.kernel.org/index.php/ATA_4_KiB_sector_issues
For those with a strong stomach for such things, a nice historical perspective on the horrors of ever-increasing drive sizes can be found at http://www.win.tue.nl/~aeb/linux/Large-Disk.html . Just be sure to keep the 2004 date in mind when it mentions "recent kernels."
I booted a RHEL6 beta DVD in rescue mode, but it wasn't any better. That version of fdisk does show: Logical/Physical Sector size 512 bytes though, so it's probably hopeless. Oddly, that version of fdisk wouldn't let me move the beginning back below 63 either - but I did that with the 5.x fdisk.
The only other thing I can think of to try would be to use 'dd' with a blocksize that is a multiple of 4K directly into a raw partition located at each of the 8 possible alignments and see if any of them gives better throughput. I suppose it's possible that something in the USB bridge chip (guessing that the enclosure presents a USB interface) won't allow the 4K write to pass. Any chance of experimenting with one of those drives hooked directly to an SATA port?
From: Robert Nichols rnicholsNOSPAM@comcast.net
Is that one of those WD drives that falsely reports its physical sector size as 512 bytes?
From the Scorpio blue specs, if I divide the capacity by the number of sectors, I get 512...
JD
John Doe wrote:
From: Robert Nichols rnicholsNOSPAM@comcast.net
Is that one of those WD drives that falsely reports its physical sector size as 512 bytes?
From the Scorpio blue specs, if I divide the capacity by the number of sectors, I get 512...
So is there any way to tell the kernel to write 4k at once even after the partitions are aligned right? The size of this thing is really attractive for carrying data offsite and it seemed to "just work" in a windows 2003 server. I don't know the write speed it gets there, but the backup run to it completes overnight.
John Doe wrote:
From: Robert Nichols rnicholsNOSPAM@comcast.net
Is that one of those WD drives that falsely reports its physical sector size as 512 bytes?
From the Scorpio blue specs, if I divide the capacity by the number of sectors, I get 512...
all these new 'advanced' drives look to the host like they have 512 byte sectors.... its just that they pack 8 of them internally into a single 4K sector. if you don't ensure that your partitions start on a 4K (8 sector) boundary, then committed random writes are very slow as your logical file system blocks will span multiple physical sectors
On 5/26/2010 9:52 AM, John R Pierce wrote:
Is that one of those WD drives that falsely reports its physical sector size as 512 bytes?
From the Scorpio blue specs, if I divide the capacity by the number of sectors, I get 512...
all these new 'advanced' drives look to the host like they have 512 byte sectors.... its just that they pack 8 of them internally into a single 4K sector. if you don't ensure that your partitions start on a 4K (8 sector) boundary, then committed random writes are very slow as your logical file system blocks will span multiple physical sectors
How can they ever be fast if the OS is writing 512 byte sectors? The drive is going to have to read the 4k sector, merge the update, wait for the disk to spin around and write it back. The read speed seems a match for a desktop Seagate with the same capacity, but writes are about 10x slower, even if I dd to the raw disk which should bypass any partition alignment issues. And unfortunately since I want to store backups on it, the write speed is what matters.
Les Mikesell wrote:
On 5/26/2010 9:52 AM, John R Pierce wrote:
Is that one of those WD drives that falsely reports its physical sector size as 512 bytes?
From the Scorpio blue specs, if I divide the capacity by the number of sectors, I get 512...
all these new 'advanced' drives look to the host like they have 512 byte sectors.... its just that they pack 8 of them internally into a single 4K sector. if you don't ensure that your partitions start on a 4K (8 sector) boundary, then committed random writes are very slow as your logical file system blocks will span multiple physical sectors
How can they ever be fast if the OS is writing 512 byte sectors? The drive is going to have to read the 4k sector, merge the update, wait for the disk to spin around and write it back. The read speed seems a match for a desktop Seagate with the same capacity, but writes are about 10x slower, even if I dd to the raw disk which should bypass any partition alignment issues. And unfortunately since I want to store backups on it, the write speed is what matters.
well, as long as the writes aren't being 'committed' on every sector, they should be cached long enough for the full 4K block to be filed prior to actually writing to disk.
your `dd` command, what did you specify for the block size? try something realistic like 32768
On 05/26/2010 10:12 AM, Les Mikesell wrote:
On 5/26/2010 9:52 AM, John R Pierce wrote:
Is that one of those WD drives that falsely reports its physical sector size as 512 bytes?
From the Scorpio blue specs, if I divide the capacity by the number of sectors, I get 512...
all these new 'advanced' drives look to the host like they have 512 byte sectors.... its just that they pack 8 of them internally into a single 4K sector. if you don't ensure that your partitions start on a 4K (8 sector) boundary, then committed random writes are very slow as your logical file system blocks will span multiple physical sectors
How can they ever be fast if the OS is writing 512 byte sectors? The drive is going to have to read the 4k sector, merge the update, wait for the disk to spin around and write it back. The read speed seems a match for a desktop Seagate with the same capacity, but writes are about 10x slower, even if I dd to the raw disk which should bypass any partition alignment issues. And unfortunately since I want to store backups on it, the write speed is what matters.
According to that page I referenced on ata.wiki.kernel.org, "Most modern filesystems generate 4KiB aligned accesses from the partition it is in." If you get the partition alignment correct, it should "just work" unless something else is getting in the way. When I look at the counts from "vmstat -d" I see that the sector count is indeed 8X the number of I/O requests, but I have no way to see the alignment of those.
https://ata.wiki.kernel.org/index.php/ATA_4_KiB_sector_issues
-
Benjamin Franz -
Eero Volotinen -
John Doe -
John R Pierce -
Les Mikesell -
Paul Heinlein -
Robert Nichols