Looks like the chum did not have to lose any data.
Wiping out the MBR and the next 63 blocks apparently only wiped out grub stage1, partition table, and part of the lvm config data.
I get to try to do a lvm 'recovery' at his expense now but this is my first time...has anybody ever tried restoring lvm config data back to the md/pv device?
Cheers,
Christopher
Chan Chung Hang Christopher wrote:
Looks like the chum did not have to lose any data.
I cannot believe he actually tried to create a new filesystem on sda according to the .bash_history file after the dd commands. I think I need a titanium clueby4. Anybody know where I can get one?
Chan Chung Hang Christopher wrote:
Chan Chung Hang Christopher wrote:
Looks like the chum did not have to lose any data.
I cannot believe he actually tried to create a new filesystem on sda according to the .bash_history file after the dd commands. I think I need a titanium clueby4. Anybody know where I can get one?
Shouldn't be a problem, just recover from the backup right. Surely he does backups :D
Johnny Hughes wrote:
Chan Chung Hang Christopher wrote:
Chan Chung Hang Christopher wrote:
Looks like the chum did not have to lose any data.
I cannot believe he actually tried to create a new filesystem on sda according to the .bash_history file after the dd commands. I think I need a titanium clueby4. Anybody know where I can get one?
Shouldn't be a problem, just recover from the backup right. Surely he does backups :D
Nope...he is stilling working on that one...
XD
The company fried the disks for the previous backup server solution shortly after I left. I kept telling them that they should first fix up the new 'server room' before anything else for the office renovation. No, the boss would not listen. Under powering the disks (external disk cage connected via eSata) due to too many devices on the same power socket did the rest.
Good thing I left before that disaster.
----- Original Message ----
From: Chan Chung Hang Christopher christopher.chan@bradbury.edu.hk To: CentOS mailing list centos@centos.org Sent: Friday, 14 August, 2009 3:31:32 Subject: [CentOS] OT: Fortunate clueless dd chum - lvm recovery
Looks like the chum did not have to lose any data.
Wiping out the MBR and the next 63 blocks apparently only wiped out grub stage1, partition table, and part of the lvm config data.
I get to try to do a lvm 'recovery' at his expense now but this is my first time...has anybody ever tried restoring lvm config data back to the md/pv device?
Cheers,
Christopher _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
First of all, I would dd a copy of the whole drive off to another drive, so you can have a few goes at this.
How do you know only those bits where lost?
First of all, I would dd a copy of the whole drive off to another drive, so you can have a few goes at this.
How do you know only those bits where lost?
The dd command zeros the first 64 sectors, that is, the mbr and then the next 63 sectors which would the bootsector of the first partition and the next 62 sectors following that. The first partition on both disks belong to the md device that is the basis for the physical volume for the system. And if it had not, it would have belonged to the md device for the /boot partition which is not a great loss. Default Redhat layout this.
The box is still live and I am glad he was not clueless enough to say yes to the mkefs2 command he was following from whatever howto he had been looking at. It looks like the lvm survived having the first 62 sectors being zeroed. Apparently lvm uses the first 255 sectors/blocks for lvm config data. No alarms/warnings in the logs. Certainly no panics otherwise I would not even be able to get in.
I get to learn something new at his expense, (which is now just a scare) nice successor eh? :-D
Absolutely zero data loss. What can I say?
----- Original Message ----
From: Chan Chung Hang Christopher christopher.chan@bradbury.edu.hk To: CentOS mailing list centos@centos.org Sent: Friday, 14 August, 2009 10:00:41 Subject: Re: [CentOS] OT: Fortunate clueless dd chum - lvm recovery
First of all, I would dd a copy of the whole drive off to another drive, so
you can have a few goes at this.
How do you know only those bits where lost?
The dd command zeros the first 64 sectors, that is, the mbr and then the next 63 sectors which would the bootsector of the first partition and the next 62 sectors following that. The first partition on both disks belong to the md device that is the basis for the physical volume for the system. And if it had not, it would have belonged to the md device for the /boot partition which is not a great loss. Default Redhat layout this.
The box is still live and I am glad he was not clueless enough to say yes to the mkefs2 command he was following from whatever howto he had been looking at. It looks like the lvm survived having the first 62 sectors being zeroed. Apparently lvm uses the first 255 sectors/blocks for lvm config data. No alarms/warnings in the logs. Certainly no panics otherwise I would not even be able to get in.
I get to learn something new at his expense, (which is now just a scare) nice successor eh? :-D
Absolutely zero data loss. What can I say? _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
I think if you can access the data at the moment, I would back it up and set the LVM stuff up and mbr up again, unless you are really sure you can fix the zero'd info. Either way, if the data is accessible, back it up now!
On Fri, Aug 14, 2009 at 8:03 AM, Kristopher Kanekristopher.kane@gmail.com wrote:
I get to learn something new at his expense, (which is now just a scare) nice successor eh? :-D
Maybe you could point him to this list for lunch time lesson reading, however, you won't be able to talk about him behind his back anymore.
If your former employer is going to continue to employ him in that position, possibly he should be using webmin to do sys admin? That might prevent some disaster(s). Or, at least, he might get a warning from webmin, before he destroys something.
Lanny Marcus wrote:
On Fri, Aug 14, 2009 at 8:03 AM, Kristopher Kanekristopher.kane@gmail.com wrote:
I get to learn something new at his expense, (which is now just a scare) nice successor eh? :-D
Maybe you could point him to this list for lunch time lesson reading, however, you won't be able to talk about him behind his back anymore.
If your former employer is going to continue to employ him in that position, possibly he should be using webmin to do sys admin? That might prevent some disaster(s). Or, at least, he might get a warning from webmin, before he destroys something.
Not possible...he has to access a postgresql server via console to run sql commands or fix up some other access. Likewise with vpopmail admin. He has to add new disks and (sigh) move the vpopmail stuff at the very least over to the filesystems. You guys can help him if he gets himself on the list. Doing final data checking on the disks. I got the partition layout wrong in the end but I have that sorted now. Got both md devices back online and pvscan checked out during the rebuild of the md array that is the basis for the pv so I guess things will be okay later too. Just got a final fsck to run on the logvols and I am tossing this back in his hands. (Bored yet?)
Things have not turned out as bad as I thought they might have. Ah well, at least I know there is way to check lvm metadata now.
Link here just in case someone can point out any errors as I have not quite have to opportunity to put the information to the test. http://anthonydawson.thelasis.com/LVRecovery/index.htm
Kristopher Kane wrote:
I get to learn something new at his expense, (which is now just a scare) nice successor eh? :-D
Maybe you could point him to this list for lunch time lesson reading, however, you won't be able to talk about him behind his back anymore.
Haha, I am not worried about that at all. I will point him over here. He should have clear up the piled up work (Windows related, gah) by now. Not naming him but please be good to him...he may not be up on mailing list etiquette and other stuff like RTM. No, I ain't teaching him all that. I have babysat him for long enough.
:-/
8-|
On Fri, Aug 14, 2009 at 5:00 AM, Chan Chung Hang Christopherchristopher.chan@bradbury.edu.hk wrote:
First of all, I would dd a copy of the whole drive off to another drive, so you can have a few goes at this.
How do you know only those bits where lost?
The dd command zeros the first 64 sectors, that is, the mbr and then the next 63 sectors which would the bootsector of the first partition and the next 62 sectors following that. The first partition on both disks belong to the md device that is the basis for the physical volume for the system. And if it had not, it would have belonged to the md device for the /boot partition which is not a great loss. Default Redhat layout this.
The box is still live and I am glad he was not clueless enough to say yes to the mkefs2 command he was following from whatever howto he had been looking at. It looks like the lvm survived having the first 62 sectors being zeroed. Apparently lvm uses the first 255 sectors/blocks for lvm config data. No alarms/warnings in the logs. Certainly no panics otherwise I would not even be able to get in.
I get to learn something new at his expense, (which is now just a scare) nice successor eh? :-D
Absolutely zero data loss. What can I say?
Well if he dd'd 64 sectors instead of 63 then the first sector of the first partition is going to be zero'd too.
Backup the data while the stale partition table is still in memory!
A reboot will make it inaccessible.
Try a:
# sfdisk -d /dev/sdX >/root/sdX.save
Look at it and see if it contains a valid partition table, if it does then do a:
# sfdisk --no-reread /dev/sdX </root/sdX.save
Question now is, was the first sector of partition 1 damaged (was it 63 or 64 sectors dd'd)?
If so it will require a more tricky procedure to fix.
-Ross
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Ross Walker Sent: Friday, August 14, 2009 10:30 To: CentOS mailing list Subject: Re: [CentOS] OT: Fortunate clueless dd chum - lvm recovery
On Fri, Aug 14, 2009 at 5:00 AM, Chan Chung Hang Christopherchristopher.chan@bradbury.edu.hk wrote:
First of all, I would dd a copy of the whole drive off to
another drive, so you can have a few goes at this.
How do you know only those bits where lost?
The dd command zeros the first 64 sectors, that is, the mbr
and then
the next 63 sectors which would the bootsector of the first
partition
and the next 62 sectors following that. The first partition on both disks belong to the md device that is the basis for the physical volume for the system. And if it had not, it would have belonged to the md device for the /boot partition which is not a great loss. Default Redhat layout this.
The box is still live and I am glad he was not clueless
enough to say
yes to the mkefs2 command he was following from whatever
howto he had
been looking at. It looks like the lvm survived having the first 62 sectors being zeroed. Apparently lvm uses the first 255
sectors/blocks
for lvm config data. No alarms/warnings in the logs. Certainly no panics otherwise I would not even be able to get in.
I get to learn something new at his expense, (which is now just a scare) nice successor eh? :-D
Absolutely zero data loss. What can I say?
Well if he dd'd 64 sectors instead of 63 then the first sector of the first partition is going to be zero'd too.
Backup the data while the stale partition table is still in memory!
A reboot will make it inaccessible.
Try a:
# sfdisk -d /dev/sdX >/root/sdX.save
ROTFLMAO, you should make sure, before copying and pasting that /root/sdX.save does not exist.
Look at it and see if it contains a valid partition table, if it does then do a:
# sfdisk --no-reread /dev/sdX </root/sdX.save
Question now is, was the first sector of partition 1 damaged (was it 63 or 64 sectors dd'd)?
If so it will require a more tricky procedure to fix.
-Ross _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - - - Jason Pyeron PD Inc. http://www.pdinc.us - - Principal Consultant 10 West 24th Street #100 - - +1 (443) 269-1555 x333 Baltimore, Maryland 21218 - - - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- This message is copyright PD Inc, subject to license 20080407P00.
Ross Walker wrote:
On Fri, Aug 14, 2009 at 5:00 AM, Chan Chung Hang Christopherchristopher.chan@bradbury.edu.hk wrote:
Question now is, was the first sector of partition 1 damaged (was it 63 or 64 sectors dd'd)?
If so it will require a more tricky procedure to fix.
No, the ext2 file system does not use the first 1K block of the partition. That space is left free for a boot loader. The first super block of the file system is the 2nd 1K block of the partition, and even if that gets overwritten it is easily recovered from one of the backup super blocks.
On Aug 14, 2009, at 12:48 PM, Robert Nichols rnicholsNOSPAM@comcast.net wrote:
Ross Walker wrote:
On Fri, Aug 14, 2009 at 5:00 AM, Chan Chung Hang Christopherchristopher.chan@bradbury.edu.hk wrote:
Question now is, was the first sector of partition 1 damaged (was it 63 or 64 sectors dd'd)?
If so it will require a more tricky procedure to fix.
No, the ext2 file system does not use the first 1K block of the partition. That space is left free for a boot loader. The first super block of the file system is the 2nd 1K block of the partition, and even if that gets overwritten it is easily recovered from one of the backup super blocks.
I did not know that, good information.
So stage2 is written in first 1k of the partition?
Ah but I believe the OP has the partition setup as a LVM PV. Question is does LVM leave the first 1k free for a stage2 boot loader?
And if the LVM metadata is corrupt is there a second copy at the end of the PV?
-Ross
Ross Walker wrote:
On Aug 14, 2009, at 12:48 PM, Robert Nichols rnicholsNOSPAM@comcast.net wrote:
Ross Walker wrote:
On Fri, Aug 14, 2009 at 5:00 AM, Chan Chung Hang Christopherchristopher.chan@bradbury.edu.hk wrote:
Question now is, was the first sector of partition 1 damaged (was it 63 or 64 sectors dd'd)?
If so it will require a more tricky procedure to fix.
No, the ext2 file system does not use the first 1K block of the partition. That space is left free for a boot loader. The first super block of the file system is the 2nd 1K block of the partition, and even if that gets overwritten it is easily recovered from one of the backup super blocks.
I did not know that, good information.
So stage2 is written in first 1k of the partition?
No, nowhere near enough room. Just stage 1 goes into the first sector. The second sector is unused. Stage 1 then loads the fs-specific stage 1.5 from absolute sector numbers on the disk. Stage 1.5 then loads stage 2 from the file system. (Yes, if the *_stage1_5 files get moved on the disk, you have to re-run the GRUB installer.)
The only time more than the 446-byte stage 1 gets installed is when installing GRUB in the MBR or in the boot sector of a logical drive in the extended partition. There, there can be a full track available, which is enough room for the appropriate stage 1.5, but not for the 100+ KB of stage 2.
Ah but I believe the OP has the partition setup as a LVM PV. Question is does LVM leave the first 1k free for a stage2 boot loader?
And if the LVM metadata is corrupt is there a second copy at the end of the PV?
No idea. I don't use LVM. Having a known recovery path if things get munged is more important to me than the features of LVM.
On Aug 14, 2009, at 9:22 PM, Robert Nichols rnicholsNOSPAM@comcast.net wrote:
Ross Walker wrote:
On Aug 14, 2009, at 12:48 PM, Robert Nichols rnicholsNOSPAM@comcast.net wrote:
Ross Walker wrote:
On Fri, Aug 14, 2009 at 5:00 AM, Chan Chung Hang Christopherchristopher.chan@bradbury.edu.hk wrote:
Question now is, was the first sector of partition 1 damaged (was it 63 or 64 sectors dd'd)?
If so it will require a more tricky procedure to fix.
No, the ext2 file system does not use the first 1K block of the partition. That space is left free for a boot loader. The first super block of the file system is the 2nd 1K block of the partition, and even if that gets overwritten it is easily recovered from one of the backup super blocks.
I did not know that, good information.
So stage2 is written in first 1k of the partition?
No, nowhere near enough room. Just stage 1 goes into the first sector. The second sector is unused. Stage 1 then loads the fs-specific stage 1.5 from absolute sector numbers on the disk. Stage 1.5 then loads stage 2 from the file system. (Yes, if the *_stage1_5 files get moved on the disk, you have to re-run the GRUB installer.)
The only time more than the 446-byte stage 1 gets installed is when installing GRUB in the MBR or in the boot sector of a logical drive in the extended partition. There, there can be a full track available, which is enough room for the appropriate stage 1.5, but not for the 100+ KB of stage 2.
Ah but I believe the OP has the partition setup as a LVM PV. Question is does LVM leave the first 1k free for a stage2 boot loader?
And if the LVM metadata is corrupt is there a second copy at the end of the PV?
No idea. I don't use LVM. Having a known recovery path if things get munged is more important to me than the features of LVM.
Since you don't know if LVM has a recovery path how can you imply it doesn't?
-Ross
Ross Walker wrote:
Since you don't know if LVM has a recovery path how can you imply it doesn't?
I've seen plenty of evidence that tools for LVM recovery are lacking. I see postings from people asking about recovery of damaged LVM volumes and not getting any reasonable answers about how to repair. Or, complexity of dealing with things like duplicate volume group names when physically moving disk drives. Conventional partitioning works just fine for me, and I'm confident that I can handle just about any situation that crops up. I don't need an additional layer of complexity.
Robert Nichols wrote:
Ross Walker wrote:
Since you don't know if LVM has a recovery path how can you imply it doesn't?
I've seen plenty of evidence that tools for LVM recovery are lacking. I see postings from people asking about recovery of damaged LVM volumes and not getting any reasonable answers about how to repair.
This might help in future. http://anthonydawson.thelasis.com/LVRecovery/index.htm
Or, complexity of dealing with things like duplicate volume group names when physically moving disk drives. Conventional partitioning works just fine for me, and I'm confident that I can handle just about any situation that crops up. I don't need an additional layer of complexity.
Which is why I do not use default names when creating volgroups and logvols.
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Chan Chung Hang Christopher Sent: Saturday, August 15, 2009 10:20 To: CentOS mailing list Subject: Re: [CentOS] OT: Fortunate clueless dd chum - lvm recovery
Robert Nichols wrote:
Ross Walker wrote:
Since you don't know if LVM has a recovery path how can
you imply it
doesn't?
I've seen plenty of evidence that tools for LVM recovery
are lacking.
I see postings from people asking about recovery of damaged LVM volumes and not getting any reasonable answers about how to repair.
This might help in future. http://anthonydawson.thelasis.com/LVRecovery/index.htm
Or, complexity of dealing with things like duplicate volume group names when physically moving disk drives. Conventional
partitioning
works just fine for me, and I'm confident that I can handle
just about
any situation that crops up. I don't need an additional layer of complexity.
Which is why I do not use default names when creating volgroups and logvols.
I always found it handy to use ASSET#_VG#_YYYYMMDDHHMM as a "default" pattern here.
-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - - - Jason Pyeron PD Inc. http://www.pdinc.us - - Principal Consultant 10 West 24th Street #100 - - +1 (443) 269-1555 x333 Baltimore, Maryland 21218 - - - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- This message is copyright PD Inc, subject to license 20080407P00.
-
Chan Chung Hang Christopher -
Ian Murray -
Jason Pyeron -
Johnny Hughes -
Kristopher Kane -
Lanny Marcus -
Robert Nichols -
Ross Walker