On 8/20/2010 10:44 AM, Brunner, Brian T. wrote:
>
>>> 3: When you first build the system, ghost/image the boot/root/usr
>>> (bru) drive onto a spare backup, verify the backup boots
>>> the machine the same as the main drive.
>>> 4: have the backup bru drive mailed to you, dupe it, and rsync the
>>> remote bru to your local copy whenever you make a change to
>>> the remote bru.
>>
>> This part tends to be problematic when the system is remote
>> and you need hands-on access for the install. It would be
>> much nicer to build locally and ship the initial drives.
>
> Build-local-and-ship> build-remote! Agreed! I'm not assuming the
> remote personnel are competent in Linux. I think you are. I suggest
> that "remote hosting security" howtos address the question of
> "what can we trust the remote people to do right?" We had one where
> the only thing we could trust the remote folks to do was cash a check.
The bulk of our machines are in two large data centers of our own. The
people there are trusted and competent enough in linux to assign IPs and
help troubleshoot. A smaller number are in hosted data centers, but
almost none of them are linux and we tend to handle one-off problems
there by shipping whole configured servers to swap.
> My comments were thinking "if you don't own the remote server hardware
> but can own/update identical disk drives"
If you have a large number of servers you probably have something
identical locally. For a smaller number, it is probably time to be
using some flavor of virtual machine or cloud service to eliminate the
issues of hardware differences. In that case, copying the images around
is even easier than cloning drives.
>>> 5: In the event of fire, vandalism, or other urgent cause, your
>>> cluster can appear on a new server rapidly. Just FedEx
>>> ghosts of your locally stored bru drive rsynced from what were your
>>> remote machines, and (on similar hardware) they should turn-key boot
> and run.
>>
>> Try it - you won't like it. If the MAC addresses of the NICs
>> don't match what is configured, the network won't come up.
>> Have fun with that when you've broken the local
>> keyboard/monitor.
>
> I do try it (as INfrequently as possible), it's to address situations
> that nobody likes. Phone calls to establish IP/MAC numbers take care
> of most problems. When remote people can't be bothered to know how to
> spell IP, like on some offshore oil drilling rigs, we send out a
> disk-installer.
>
> I didn't point out the details of the unknown.
I thought you disabled all access except ssh - which would make it a big
problem if the network doesn't start.
> Fire might or might not move the machines to a new ISP or IP;
> Vandalism ditto;
> "Other urgent causes" has (in my experience) been remote
> operators who could not be relied on and therefore a new remote host was
> necessary.
> In some cases the old IPs work, new MACs are needed, in other cases both
> change or neither changes.
> So editing of your rsync'd drives before shipping them to the (new?)
> ISP/IP site is maybe necessary.
Needing new IPs is a different issue than then one I mentioned. If you
have a HWADDR specified in your ifcfg-eth? file, it won't come up if you
move the disk to a different machine where it doesn't match. If you
remove the HDWADDR line and have more than one interface they are
unlikely to be assigned in the right order. In some cases I've tried to
edit in the right values but as the machine came up it renamed all of
the ifcfg-eth? files with a .bak extension and created new ones using dhcp.
> If you can trust your remote operators (more than I can) then remote KVM
> is an affordable risk.
So far it hasn't been enough of a problem to be worth a huge effort to
solve, but I did prefer the predictable NIC assignement of the 2.4
kernel. And in a lab setting where things change often, I have someone
install VMware ESXi and assign it one IP address which is pretty simple.
Then even with the free version I can do everything else remotely
including installs or conversions of guest images.
--
Les Mikesell
lesmikesell at gmail.com