[CentOS] Anyway to ensure SSH availability?

[Devin Reade]

--On Thursday, June 30, 2011 04:15:07 AM +0800 Emmanuel Noobadmin
<centos.admin at gmail.com> wrote:

> Would ILO work on a server that's unresponsive due to heavy load?

ILO or any other OOB solution gives you the functionality of sitting
at the console.  So if the problem is one that would cause the console
to be unresponsive, you're still not going to be able to log in.
OTOH, if console is responsive but network-based access such as ssh
is not, then OOB may help.

As others have said, OOB is always a good idea.  When the money 
managers question the cost, compare it to the cost of servers 
becoming unavailable, someone having to travel to the site, etc.
For aftermarket stuff, in addition to what others have mentioned,
search the list archives for threads mentioning 'ipeps'.

Getting back to determining the source of the problem:

I would suggest that you might want to look at running sar.  It's
something that will collect various system statistics constantly,
so it's good for, when there's an event, going back after the fact
and showing what lead up to it.  It may not tell you the actual
problem, but the statistics can help isolate the cause.

Be aware of Heisenberg, though.  You suspect that your problem is
I/O based.  Sar is going to increase your I/O in order to log the
stats to disk.  If you have sar sampling too often, you're going
to increase the amount your problem is happening (or make it happen
faster).  If you don't sample often enough, the lack of resolution of
the data can hide what is actually going on with the system.  (Grabbing
a number out of the air, you might be able start sampling at once
per minute.)  If you can log sar stats to a different disk, it might
help.

Also be aware that when things grind to a halt, you're probably not 
going to get stats.  So what you see (after reboot) may just include
the *start* of the event.

sar is part of the sysstat rpm.

Devin