Hi All
I've been looking into some ways to break a CentOS system so I can perform some simulated disaster recovery and get some hands-on practice with rescue mode and other system recovery tools. I'm thinking to start off with things like corrupting the password file with random characters, breaking grub.conf.
I've been doing some research on the net, but would welcome any input from list members. Does anybody have any creative suggestions for ways to break the CentOS boot process?
Thanks in advance :)
Alex.
On 21/02/2012, Alex Walker tech@alexwalker.org.uk wrote:
Does anybody have any creative suggestions for ways to break the CentOS boot process?
I have *not* actually tried it myself, but this might help you: http://trouble-maker.sourceforge.net/
On 21/02/2012 10:14, David wrote:
On 21/02/2012, Alex Walkertech@alexwalker.org.uk wrote:
Does anybody have any creative suggestions for ways to break the CentOS boot process?
I have *not* actually tried it myself, but this might help you: http://trouble-maker.sourceforge.net/ _______________________________________________
Thanks, that looks to be a good start, will give it a try :)
Alex.
On Tue, Feb 21, 2012 at 3:45 AM, Alex Walker tech@alexwalker.org.uk wrote:
Hi All
I've been looking into some ways to break a CentOS system so I can perform some simulated disaster recovery and get some hands-on practice with rescue mode and other system recovery tools. I'm thinking to start off with things like corrupting the password file with random characters, breaking grub.conf.
I've been doing some research on the net, but would welcome any input from list members. Does anybody have any creative suggestions for ways to break the CentOS boot process?
Thanks in advance :)
Alex. _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Things like boot process rarely break. Try something like filling up your root or tmp partition. That just seems to be a bit more common as far as problem scenarios go.
Good luck:)
Boris.
On 2/21/2012 5:57 AM, Boris Epstein wrote:
Things like boot process rarely break.
I can't remember the last time I caused a system to outright fail to boot, but I *do* get unclean boots regularly.
Examples:
- Build and install some needed driver from source, yum upgrade repeatedly, implicitly upgrade kernel, forget to rebuild the driver against the new kernel, reboot, boom.
- Get asked to configure the foo service, get it all working, forget to add it to init.d, use it happily for months, reboot, fail to notice the service's absence until someone gives a misleading bug report. ("The foo service has crashed!") Then I have to go chasing it, handicapped by being half a year separated from the last time I looked at it.
What with the long uptimes on our servers and their many-hattedness, I'd say chances are actually fairly low that any given reboot will bring it 100% back to the state it was in when we shut it down. Something boot-related almost *always* changes between reboots.
I've made it a policy that if you get an unclean reboot, you fix it in such a way that it should come up cleanly, then test by rebooting. Repeat until all symptoms are fixed. But, never believe you've thereby guaranteed that it will come up cleanly the next time.
On 02/21/2012 04:00 PM, Warren Young wrote:
- Build and install some needed driver from source, yum upgrade
repeatedly, implicitly upgrade kernel, forget to rebuild the driver against the new kernel, reboot, boom.
Use ElRepo repository for drivers. They use kmod so it works on new kernel without recompilation, just works. kmod is based on symlinks/weak-updates.
If there is something ElRepo does not have, and it is open source, they will be happy to include it. Even if it is proprietary, they could build you spec (+ instructions ) for kmod package.
On Tue, Feb 21, 2012 at 08:00:17AM -0700, Warren Young (warren@etr-usa.com) wrote:
On 2/21/2012 5:57 AM, Boris Epstein wrote:
Things like boot process rarely break.
- Get asked to configure the foo service, get it all working, forget to
add it to init.d, use it happily for months, reboot, fail to notice the service's absence until someone gives a misleading bug report. ("The foo service has crashed!") Then I have to go chasing it, handicapped by being half a year separated from the last time I looked at it.
I am glad to see I am NOT the only one doing this ... ;-) You made my day ;-)
Jobst
Things like boot process rarely break. Try something like filling up your root or tmp partition. That just seems to be a bit more common as far as problem scenarios go.
Thanks for the reply. I've recently started working for a large hosting company, so there's a reasonable amount of scope for anything that can go wrong going wrong at some point, so I wanted to be prepared in advance for as many eventualities as possible :)
I've seen root partitions fill up a few times, but in my own experience we've always been able to get in and clear out some space before it's knocked the server over. I'll look into it a bit further tho.
Cheers.
A.
Alex Walker wrote:
Things like boot process rarely break. Try something like filling up your root or tmp partition. That just seems to be a bit more common as far as problem scenarios go.
Thanks for the reply. I've recently started working for a large hosting company, so there's a reasonable amount of scope for anything that can go wrong going wrong at some point, so I wanted to be prepared in advance for as many eventualities as possible :)
*sigh* Good luck with the new job.
I've seen root partitions fill up a few times, but in my own experience we've always been able to get in and clear out some space before it's knocked the server over. I'll look into it a bit further tho.
One trick my manager's shown me, that you can use to keep things going while dealing with a root filesystem full is to use tune2fs to lower the f/s reserved space.
mark
On Tue, Feb 21, 2012 at 2:45 AM, Alex Walker tech@alexwalker.org.uk wrote:
Hi All
I've been looking into some ways to break a CentOS system so I can perform some simulated disaster recovery and get some hands-on practice with rescue mode and other system recovery tools. I'm thinking to start off with things like corrupting the password file with random characters, breaking grub.conf.
I've been doing some research on the net, but would welcome any input from list members. Does anybody have any creative suggestions for ways to break the CentOS boot process?
If you want a real workout, try restoring your backups on a different machine - where more hardware differences or raid layout changes can make it harder. It is also a realistic thing to need to do. Grub isn't nearly as likely to fail as a disk drive, motherboard, or power supply.
Pull out one of the hard drives while it's still running and see if RAID keeps up. For more fun, see how the latencies increase!
On Tue, Feb 21, 2012 at 3:45 AM, Alex Walker tech@alexwalker.org.uk wrote:
Hi All
I've been looking into some ways to break a CentOS system so I can perform some simulated disaster recovery and get some hands-on practice with rescue mode and other system recovery tools. I'm thinking to start off with things like corrupting the password file with random characters, breaking grub.conf.
I've been doing some research on the net, but would welcome any input from list members. Does anybody have any creative suggestions for ways to break the CentOS boot process?
Thanks in advance :)
Alex. _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On 2/21/2012 1:45 AM, Alex Walker wrote:
I've been looking into some ways to break a CentOS system so I can perform some simulated disaster recovery
Bring up a fresh CentOS 6.0 system. Disable automatic updates. Add a bunch of third-party software. Install at least one bit of hardware so nonstandard you're forced to install a driver from source. Then do all the system updates, and immediately reboot.
Ideally, nothing should break, because all the upgrades will have proceeded cleanly and migrated everything forward nicely. Doesn't always happen. The more third-party software and nonstandard hardware you throw into the mix, the more likely you will run across one of these corner cases.
on 2/21/2012 12:45 AM Alex Walker spake the following:
Hi All
I've been looking into some ways to break a CentOS system so I can perform some simulated disaster recovery and get some hands-on practice with rescue mode and other system recovery tools. I'm thinking to start off with things like corrupting the password file with random characters, breaking grub.conf.
I've been doing some research on the net, but would welcome any input from list members. Does anybody have any creative suggestions for ways to break the CentOS boot process?
Thanks in advance :)
Alex.
You can always turn off the firewall and point it to the internet... Some kiddy will break it for you! ;)
-
Alex Walker -
Boris Epstein -
D. Marshall Lemcoe Jr. -
David -
Jobst Schmalenbach -
Les Mikesell -
Ljubomir Ljubojevic -
m.roth＠5-cent.us -
Marc Deop -
Scott Silva -
Warren Young