If you have a disk with several partitions set up as members of a raid1 md devices, can you make a dd image of that disk to replace its matching drive with identical partitions or are there differences between the mirrored partitions?
Les Mikesell wrote:
If you have a disk with several partitions set up as members of a raid1 md devices, can you make a dd image of that disk to replace its matching drive with identical partitions or are there differences between the mirrored partitions?
you can 'dd' the MBR and then re-add the partitions to the raid for resyncing with 'mdadm'.
# dd if=/dev/sda of=/dev/sdb bs=512 count=1 # mdadm /dev/md0 --add /dev/sdb1 # mdadm /dev/md1 --add /dev/sdb2
If you want to really make sure you got everything you could dd the whole first track with:
# dd if=/dev/sda of=/dev/sdb bs=512 count=63
-Ross
______________________________________________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof.
on 6-10-2008 9:59 AM Ross S. W. Walker spake the following:
Les Mikesell wrote:
If you have a disk with several partitions set up as members of a raid1 md devices, can you make a dd image of that disk to replace its matching drive with identical partitions or are there differences between the mirrored partitions?
you can 'dd' the MBR and then re-add the partitions to the raid for resyncing with 'mdadm'.
# dd if=/dev/sda of=/dev/sdb bs=512 count=1 # mdadm /dev/md0 --add /dev/sdb1 # mdadm /dev/md1 --add /dev/sdb2
If you want to really make sure you got everything you could dd the whole first track with:
# dd if=/dev/sda of=/dev/sdb bs=512 count=63
-Ross
Or sfdisk -d /dev/sdX | sfdisk /dev/sdY where x is source and y is the target. This will work across drives that have slight geometry differences.
Scott Silva wrote:
If you have a disk with several partitions set up as members of a raid1 md devices, can you make a dd image of that disk to replace its matching drive with identical partitions or are there differences between the mirrored partitions?
you can 'dd' the MBR and then re-add the partitions to the raid for resyncing with 'mdadm'.
# dd if=/dev/sda of=/dev/sdb bs=512 count=1 # mdadm /dev/md0 --add /dev/sdb1 # mdadm /dev/md1 --add /dev/sdb2
If you want to really make sure you got everything you could dd the whole first track with:
# dd if=/dev/sda of=/dev/sdb bs=512 count=63
-Ross
Or sfdisk -d /dev/sdX | sfdisk /dev/sdY where x is source and y is the target. This will work across drives that have slight geometry differences.
What I was hoping to do was to take the grub setup, the partitioning info and the contents in one shot and have the disks pair automatically when booted. They didn't - but I think the other parts worked.
Now, is there a way to change the uuid on a running raid1 set? I'd prefer that if the split and re-paired disks ever find their way back to the same machine that they not sync again.
Les Mikesell wrote:
Scott Silva wrote:
If you have a disk with several partitions set up as members of a raid1 md devices, can you make a dd image of that disk to replace its matching drive with identical partitions or are there differences between the mirrored partitions?
you can 'dd' the MBR and then re-add the partitions to the raid for resyncing with 'mdadm'.
# dd if=/dev/sda of=/dev/sdb bs=512 count=1 # mdadm /dev/md0 --add /dev/sdb1 # mdadm /dev/md1 --add /dev/sdb2
If you want to really make sure you got everything you could dd the whole first track with:
# dd if=/dev/sda of=/dev/sdb bs=512 count=63
-Ross
Or sfdisk -d /dev/sdX | sfdisk /dev/sdY where x is source and y is the target. This will work across drives that have slight geometry differences.
What I was hoping to do was to take the grub setup, the partitioning info and the contents in one shot and have the disks pair automatically when booted. They didn't - but I think the other parts worked.
Now, is there a way to change the uuid on a running raid1 set? I'd prefer that if the split and re-paired disks ever find their way back to the same machine that they not sync again.
'mdadm' writes a listing of the devices in the array to the md superblock and orders them by number,major,minor. You cannot add another device to the array with the same tuple.
If you dd the first sector of the drive though you will duplicate the partition table and grub boot loader to the other drive. Then md-device-mapper will take care of copying the data over.
-Ross
______________________________________________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof.
Ross S. W. Walker wrote:
If you have a disk with several partitions set up as members of a raid1 md devices, can you make a dd image of that disk to replace its matching drive with identical partitions or are there differences between the mirrored partitions?
you can 'dd' the MBR and then re-add the partitions to the raid for resyncing with 'mdadm'.
# dd if=/dev/sda of=/dev/sdb bs=512 count=1 # mdadm /dev/md0 --add /dev/sdb1 # mdadm /dev/md1 --add /dev/sdb2
If you want to really make sure you got everything you could dd the whole first track with:
# dd if=/dev/sda of=/dev/sdb bs=512 count=63
-Ross
Or sfdisk -d /dev/sdX | sfdisk /dev/sdY where x is source and y is the target. This will work across drives that have slight geometry differences.
What I was hoping to do was to take the grub setup, the partitioning info and the contents in one shot and have the disks pair automatically when booted. They didn't - but I think the other parts worked.
Now, is there a way to change the uuid on a running raid1 set? I'd prefer that if the split and re-paired disks ever find their way back to the same machine that they not sync again.
'mdadm' writes a listing of the devices in the array to the md superblock and orders them by number,major,minor. You cannot add another device to the array with the same tuple.
Isn't this updated at detect time so the device minor's should always be different?
If you dd the first sector of the drive though you will duplicate the partition table and grub boot loader to the other drive. Then md-device-mapper will take care of copying the data over.
I'm curious as to why 2 complete dd'd copies don't pair at boot. One comes up running and it does work to mdadm --add the partner partitions and after the resync they do automatically pair at boot.
on 6-11-2008 10:41 AM Les Mikesell spake the following:
Ross S. W. Walker wrote:
If you have a disk with several partitions set up as members of a raid1 md devices, can you make a dd image of that disk to replace its matching drive with identical partitions or are there differences between the mirrored partitions?
you can 'dd' the MBR and then re-add the partitions to the raid for resyncing with 'mdadm'.
# dd if=/dev/sda of=/dev/sdb bs=512 count=1 # mdadm /dev/md0 --add /dev/sdb1 # mdadm /dev/md1 --add /dev/sdb2
If you want to really make sure you got everything you could dd the whole first track with:
# dd if=/dev/sda of=/dev/sdb bs=512 count=63
-Ross
Or sfdisk -d /dev/sdX | sfdisk /dev/sdY where x is source and y is the target. This will work across drives that have slight geometry differences.
What I was hoping to do was to take the grub setup, the partitioning info and the contents in one shot and have the disks pair automatically when booted. They didn't - but I think the other parts worked.
Now, is there a way to change the uuid on a running raid1 set? I'd prefer that if the split and re-paired disks ever find their way back to the same machine that they not sync again.
'mdadm' writes a listing of the devices in the array to the md superblock and orders them by number,major,minor. You cannot add another device to the array with the same tuple.
Isn't this updated at detect time so the device minor's should always be different?
If you dd the first sector of the drive though you will duplicate the partition table and grub boot loader to the other drive. Then md-device-mapper will take care of copying the data over.
I'm curious as to why 2 complete dd'd copies don't pair at boot. One comes up running and it does work to mdadm --add the partner partitions and after the resync they do automatically pair at boot.
Are they dd copies of the different nodes (IE.. dd copy of sda paired with dd copy of sdb)?
Scott Silva wrote:
I'm curious as to why 2 complete dd'd copies don't pair at boot. One comes up running and it does work to mdadm --add the partner partitions and after the resync they do automatically pair at boot.
Are they dd copies of the different nodes (IE.. dd copy of sda paired with dd copy of sdb)?
No - what was sda was dd'd twice with the dups put in sda and sdb.
on 6-11-2008 11:36 PM Les Mikesell spake the following:
Scott Silva wrote:
I'm curious as to why 2 complete dd'd copies don't pair at boot. One comes up running and it does work to mdadm --add the partner partitions and after the resync they do automatically pair at boot.
Are they dd copies of the different nodes (IE.. dd copy of sda paired with dd copy of sdb)?
No - what was sda was dd'd twice with the dups put in sda and sdb.
That is why it didn't work. The superblock knows which drive it was on, and knows if there are dupes.
Scott Silva wrote:
I'm curious as to why 2 complete dd'd copies don't pair at boot. One comes up running and it does work to mdadm --add the partner partitions and after the resync they do automatically pair at boot.
Are they dd copies of the different nodes (IE.. dd copy of sda paired with dd copy of sdb)?
No - what was sda was dd'd twice with the dups put in sda and sdb.
That is why it didn't work. The superblock knows which drive it was on, and knows if there are dupes.
But I thought the locations were re-detected at boot/assembly time.
The other question is whether it is possible to change the uuid while the system is running or if it would have to be done from a CD boot. I've cloned several machines from one initial setup and if copies of the disks ever find their way back into one box I'd prefer not to have the wrong set try to re-sync.
on 6-12-2008 10:28 AM Les Mikesell spake the following:
Scott Silva wrote:
I'm curious as to why 2 complete dd'd copies don't pair at boot. One comes up running and it does work to mdadm --add the partner partitions and after the resync they do automatically pair at boot.
Are they dd copies of the different nodes (IE.. dd copy of sda paired with dd copy of sdb)?
No - what was sda was dd'd twice with the dups put in sda and sdb.
That is why it didn't work. The superblock knows which drive it was on, and knows if there are dupes.
But I thought the locations were re-detected at boot/assembly time.
The other question is whether it is possible to change the uuid while the system is running or if it would have to be done from a CD boot. I've cloned several machines from one initial setup and if copies of the disks ever find their way back into one box I'd prefer not to have the wrong set try to re-sync.
The locations can re-detect, but I still think it will stop if it detects 2 identical superblocks. I don't think you can change the UUID of a running array, and I'm not sure if you can do it easily on a stopped array. I think the safest thing is to use the --add to join the pair. Since they are dd clones, the sync should be fairly fast. Just be careful.
I know that it seems like an easy way to clone machines, but I think there are better and safer ways to clone machines.
Scott Silva wrote:
I'm curious as to why 2 complete dd'd copies don't pair at boot. One comes up running and it does work to mdadm --add the partner partitions and after the resync they do automatically pair at boot.
Are they dd copies of the different nodes (IE.. dd copy of sda paired with dd copy of sdb)?
No - what was sda was dd'd twice with the dups put in sda and sdb.
That is why it didn't work. The superblock knows which drive it was on, and knows if there are dupes.
But I thought the locations were re-detected at boot/assembly time.
The other question is whether it is possible to change the uuid while the system is running or if it would have to be done from a CD boot. I've cloned several machines from one initial setup and if copies of the disks ever find their way back into one box I'd prefer not to have the wrong set try to re-sync.
The locations can re-detect, but I still think it will stop if it detects 2 identical superblocks. I don't think you can change the UUID of a running array, and I'm not sure if you can do it easily on a stopped array. I think the safest thing is to use the --add to join the pair. Since they are dd clones, the sync should be fairly fast. Just be careful.
Yes, the sync does work fine and once finished the set will always start itself.
I know that it seems like an easy way to clone machines, but I think there are better and safer ways to clone machines.
You can't beat dd for getting everything exactly the same regardless of what you changed - or just splitting the mirrors and letting each sync to new partners but then you have to reinstall grub. I prefer clonezilla for non-raid configurations but most of the machines I care about are configured with raid1.
Les Mikesell wrote:
You can't beat dd for getting everything exactly the same regardless of what you changed - or just splitting the mirrors and letting each sync to new partners but then you have to reinstall grub. I prefer clonezilla for non-raid configurations but most of the machines I care about are configured with raid1.
Well, actually dd isn't so good in this area. dd will do the whole disk no matter how much data is actually stored on it and for a 500GB disk that can take a lot of time. It also doesn't take into consideration any disk geometry differences.
A better way is to use kickstart script to automate a network install and then to use dump/restore to load the user/application data back.
With remote access cards and vnc kickstart installs this can even be done remotely on a headless server even without a technician present to power it on or off.
-Ross
______________________________________________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof.
Ross S. W. Walker wrote:
You can't beat dd for getting everything exactly the same regardless of what you changed - or just splitting the mirrors and letting each sync to new partners but then you have to reinstall grub. I prefer clonezilla for non-raid configurations but most of the machines I care about are configured with raid1.
Well, actually dd isn't so good in this area. dd will do the whole disk no matter how much data is actually stored on it and for a 500GB disk that can take a lot of time.
Sure, but it isn't human time. I just give the command and come back later.
It also doesn't take into consideration any disk geometry differences.
I happen to have a lot of identical disks in swappable carriers.
A better way is to use kickstart script to automate a network install and then to use dump/restore to load the user/application data back.
Sounds like more work to me. Besides figuring out the kickstart options you need an up-to-the-minute dump made beforehand and there's always some risk in restoring running programs unless you use a program like rsync that creates a tmp name, then renames when complete. And there's some work to sort out what you can restore and what you can't.
With remote access cards and vnc kickstart installs this can even be done remotely on a headless server even without a technician present to power it on or off.
Shipping a box of disks for someone else to swap in works for me, and it doesn't matter if you decide to drastically change OS's between swaps. You do have to deal with the way NIC detection has changed across Centos versions, though.
On Thu, 2008-06-12 at 15:14 -0400, Ross S. W. Walker wrote:
Les Mikesell wrote:
You can't beat dd for getting everything exactly the same regardless of what you changed - or just splitting the mirrors and letting each sync to new partners but then you have to reinstall grub. I prefer clonezilla for non-raid configurations but most of the machines I care about are configured with raid1.
Well, actually dd isn't so good in this area. dd will do the whole disk no matter how much data is actually stored on it and for a 500GB disk that can take a lot of time. It also doesn't take into consideration any disk geometry differences.
True. But with a small amount of scripting (assuming some experience like I had at the time I did this professionally), you can quickly produce a fairly flexible, automated, fast and reliable process that accomplishes the task.
I'll elaborate a little below.
<snip>
-Ross
<snip sig stuff>
First, as to speed. Using a blksize= parameter (I used a cyl size as my "standard" unit of transfer), the number of system calls is reduced and the speed of the hardware becomes the limiting factor. Back when I tested this (old slower low-single-digit GB drives, circa 2000-2002), very large speedups were seen. I don't recall the percentages.
Second, "copies the whole disk ...". Here is where a small amount of scripting becomes useful. You can copy only specific partitions. If the whole disk is a single partition, some stats gathered by various utilities can determine used counts, and a combination of shrinking the file system (didn't have a shrink ability back when) and (if desired) shrinking the partition can be used to "compact" the source.
In conjunction with sfdisk to both gather configuration information and generate new configuration (via scripts), you can reduce the source copied to very close to just that needed.
In the final step, this is engendered via the "count=", "skip=" and "seek=" parameters to dd.
This was implemented in a NAS product as part of the RAS process that could not predict the specifications of HDs that might be replaced in the field.
Lastly, as to geometry differences, again sfdisk is your friend. What I can not address is how to integrate this with raid - no experience there. I do presume that one knowledgeable in that area could also automate that.
If this sounds like a lot of work, it's not really. This part of my effort was minimal. The large part was creation of the boot CD, interfacing with the custom hardware timeout facility for auto-reboot to a fallback device and implementing and testing the software install and rejoin with the cluster.
Oh... and the constantly changing specs (list of requirements kept changing as they saw opportunities I brought to the table - their *IX experience was quite limited).
HTH
Les Mikesell wrote:
<snip>
'mdadm' writes a listing of the devices in the array to the md superblock and orders them by number,major,minor. You cannot add another device to the array with the same tuple.
Isn't this updated at detect time so the device minor's should always be different?
If you dd the first sector of the drive though you will duplicate the partition table and grub boot loader to the other drive. Then md-device-mapper will take care of copying the data over.
I'm curious as to why 2 complete dd'd copies don't pair at boot. One comes up running and it does work to mdadm --add the partner partitions and after the resync they do automatically pair at boot.
Gosh darn it Les your just too damn inquisitive!
Well for the sake of the truth, the whole truth, and you know the rest.
Let me dig around here, hmmm, ok, not here, ok here we go:
/usr/src/kernels/2.6.18-53.1.21.el5-x86_64/include/linux/raid/md_p.h
Ok, looking at the comments, it seems...
/* * RAID superblock. * * The RAID superblock maintains some statistics on each RAID configuration. * Each real device in the RAID set contains it near the end of the device. * Some of the ideas are copied from the ext2fs implementation. * * We currently use 4096 bytes as follows: * * word offset function * * 0 - 31 Constant generic RAID device information. * 32 - 63 Generic state information. * 64 - 127 Personality specific information. * 128 - 511 12 32-words descriptors of the disks in the raid set. * 512 - 911 Reserved. * 912 - 1023 Disk specific descriptor. */
The last part of the superblock contains a disk specific descriptor, identifier (whatever), and the middle contains a list of all the disk descriptors participating in the RAID set. (there can only be 12 disks max in a raid set? that's news to me, maybe the comment is old, if you combine the reserved area you can get 24 disks).
From this we can observe that when the disks are identified as
auto-raid and their superblocks are read it tells them what RAID they belong to and their order in the array.
What happens if a duplicate descriptor is encountered? And how does it determine which disk is the "official" disk?
I have yet to find those answers, but let me hypothesis that it ejects either a) the disk with the oldest timestamp, or b) the disk with the odd checksum out of the array.
To find out the real truth will need some detailed MD RAID docs which I am having trouble finding or the sources which I cannot be bothered/don't have time to download and audit right now.
-Ross
______________________________________________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof.
-
Les Mikesell -
Ross S. W. Walker -
Scott Silva -
William L. Maltby