[CentOS] OT: SysAdmin Stories

[MHR]

On Sun, Jun 6, 2010 at 10:53 AM, Digimer <linux at alteeve.com> wrote:
>
> Under the category of "learn from the mistake of others..."
>
> About eight years ago, I was working on a program with tight deadlines.
> I'd worked through the night, only catching an hour or two of sleep in
> the office.
>
> The next morning, one of the servers remounted it's file systems
> read-only. Being a small shop, I decided to just take the server down to
> run a quick fsck.ext2. In my sleepiness though, I typed 'mkfs.ext2'.
>
> When people say that "root" is god, well, no one asks god "Are you sure?".
>

Way back in the stone age, I was a sys admin at my university, working
the graveyard (i.e., backup) shift two days a week and an occasional
Sunday.  On Sundays, we did the full backup and restore, but we
switched out the disk packs (I said this was a long time ago) so we
never lost more than a week's worth of data at the time.  Well, almost
never....

My last Sunday there, I accidentally reinitialized all the disks after
the backup but before I had switched them.  Then, I realized what I
did, switched them anyway, and reinitialized them again, then did a
full restore.

Everything would have been fine if the file system hadn't crashed that
Friday afternoon....

This was on a Xerox Sigma 7 (I'm dating myself).

UNIX horror story: 24 years ago, I was working on a development system
(i.e., nothing critical on it) and my latest build didn't work the way
I expected, so I erased it with an 'rm -rf *' - except that I was in
the root directory at the time, not my build directory.  By the time I
realized what I had done, it was too far gone to recover, so I wound
up reinstalling the whole system.  No harm done (I did things like
that sometimes on purpose, when it was *my* machine involved), but I
don't do 'rm -rf' of anything any more without double-checking where I
am FIRST, even if the default "-v" is set.

(unsigned confession)