[CentOS] Cluster Broken Pipe error and Heartbeat configuration

Thu Nov 13 01:09:32 UTC 2008
Ian Forde <ian at duckland.org>

On Wed, 2008-11-12 at 08:41 -0800, nate wrote:
> lingu wrote:
> 
> >  Can any one guide me  what is this above error indicates and how to
> > troubleshoot.After a long google search i found the below link from
> > redhat that is matching my scenario.Can i follow the same because it
> > is my very critical production server.
> 
> I suggest you contact Red Hat support for this issue if it's
> such a critical server and sounds like a pretty fragile situation.
> That's what they are there for. And your running a really old version
> of RH.

I'm inclined to agree the RH is probably the fastest way to get this
resolved.  That isn't such an old version of RHEL, btw... current RHEL3
version is 3.9, but RH recommends sticking with particular versions when
using RHCS (Red Hat Cluster Server) as Cluster can often come with
replacement versions of stock rpms (including the kernel)... From
checking http://www.redhat.com/docs/manuals/csgfs/ we can see that RH
didn't update RHCS for 3.9, so RHEL 3.8 is the current version supported
for RHCS...

> If it were me I would upgrade the system to be fiber channel instead
> of SCSI, and update to all the latest patches for your version of
> RH. The bug mentions how using SCSI attached storage as your shared
> storage medium is not at all proven reliable. At least some MSAs
> out there you can get a fiber channel head unit and a few HBAs, and
> perhaps a switch and hook things up without too much downtime and
> have a better system as a result.

I wouldn't do that... not right away anyway... SCSI has proven itself
reliable over the years for clustering just fine in my experience.
However, it's how you've got it configured that may cause headaches...

You should definitely configure a private LAN for the heartbeat.  It's
as simple as editing /etc/sysconfig/network-scripts/ifcfg-eth3 on each
box and setting up the IP addresses.

But I wouldn't use a crossover cable for this - create a 2-port vlan on
your switch for it (or use a cheap switch or whatever else will work).
If you use a crossover, if either NIC or the cable fails, link state
will be down on both nodes and both nodes will attempt (and possibly
succeed) to fence each other.  I'm not making this up - I've seen it
happen on cluster deployments. (In a past life I used to deploy RHCS for
RH.)

Best advice? Call Red Hat and speak to them... they'll give you the
recommended config... also, check out the RHCS docs...
http://www.redhat.com/docs/manuals/csgfs/browse/rh-cs-en-3/index.html

I'd also recommend taking the RHCS class.  It's um... enlightening... ;)

	-I