[CentOS] Problem with CLVM (really openais)

[Digimer]

On 11/05/2012 02:04 PM, Cris Rhea wrote:
> On Sun, Nov 04, 2012 at 11:59:08AM -0500, Digimer wrote:
>> On 11/04/2012 10:48 AM, Cris Rhea wrote:
>>> One of the nodes will be barking about trying to fence the "failed" node
>>> (expected, as I don't have real fencing).  
>>
>> This is your problem. Without fencing, DLM (which is required for
>> clustered LVM, GFS2 and rgmanager) is designed to block when a fence is
>> called and stay blocked until the fence succeeds. Why it was called is
>> secondary, even if a human calls the fence and everything is otherwise
>> working fine, the cluster will hang.
>>
>> This is by design. "A hung cluster is better than a corrupt cluster".
> 
> I understand what you're saying, but I've got three concerns:
> 
> 1. Yes, I believe DLM is acting appropriately. It does "hang everything"
>    until fencing succeeds. If I manually fence the node (fence_ack_manual),
>    the remaining 3 nodes are fine.
> 
> 2. As a practical matter, I cannot enable real fencing at this point.
>    These nodes are running VMs that are doing real stuff. Fencing a "failed
>    node" would dump the VMs.   While I understand doing that in a real 
>    failure situation, I have no indications that anyting is wrong (other than
>    aisexec/TOTEM issues).
> 
> 3. At this point, this is NOT an HA cluster-- so I don't have VMs defined
>    as resources that need to be running someplace. All I'm tring to achieve
>    is to use CLVM (reliably) across a set of nodes.

As far as the cluster is concerned, it is HA. The cluster does not
understand the concept of unimportant things; It treats everything as
critical.

If you need CLVM (or anything else related to the cluster), then either
make the VMs HA resources or move them off. Until you use real fencing,
you will have problems. That said, some won't appear until it's too late
if you try to avoid this.

In short; use real fencing, period. Nothing else is supported or safe.

>>> 4. OK, now reboot the "failed" node. It reboots and rejoins the cluster. 
>>>    CLVM commands work, but are slow. Lots of these errors:
>>> 	openais[7154]: [TOTEM] Retransmit List: fe ff
>>
>> Only time I've seen this happen is when something starves a node (slow
>> network, loaded cpu, insufficient ram...).
> 
> What methods would you use to pin this down?  From my perspective, the
> machines have enough RAM (large blades with fairly small VMs), decent CPU
> (I can be logged into the node via SSH while this is happening and don't
> see a performance issue), and no network "glitches" (the aisexec failure
> happens well after machine has booted and come on-line on the network).

This requires the assistance of the devs/advanced support people. If you
have Red Hat support, please call them.

>>> 5. This goes on for about 10 minutes and the whole cycle repeats (one of 
>>>    the other nodes will "fail"...)
>>
>> If a totem packet fails to return from a node within a set period of
>> time more than a set number of times in a row, the node is declared lost
>> and a fence action is initiated.
> 
> Yup, got that. How can I debug this further? I have no indication (other
> than aisexec) that anything is wrong.  

Same as above comment.

>>> 1. Switches have IGMP snooping disabled. This is a simple config, so
>>>    no switch-to-switch multicast is needed (all cluster/multicast traffic
>>>    stays on the blade enclosure switch).  I've had the cluster
>>>    messages use the front-end net and the back-end net (different switch
>>>    model)-- no change in behavior.
>>
>> Are the multicast groups static? Is STP disabled?
> 
> I'm not a cisco command guru (so please provide real commands in any hints). 

Nor am I. Whenever I hear "cisco", it's along with problems cause by
Cisco doing non-standard things.

> I have not defined anything in the switch for MC groups. In speaking with 
> my local network team, turning off IGMP snooping should allow full/unlimited
> MC within that switch.   Again, only a single switch involved, so no
> switch-switch configs needed for MC.

As I understand it, Cisco periodically purges multicast groups, forcing
machine to resubscribe, as a way to clean out disused groups. This
breaks the cluster comms. This is just one example of what might be
happening. The take-away is that latency must remain below 2ms and
multicast messages must never be interrupted. Your network people should
be able to interpret that further.

> Do I need to do something else within the switch or cluster configs to 
> aid MC?  All the nodes are using the default MC address/port. 

Set a static multicast group, for one.

> STP is currently enabled (set to pvst).

This can cause problems. When STP tries to find loops, it can block a
port or ports. This can break the cluster as well. Disable switch-wide
STP and only enable it on outword-facing ports (if any at all).

>>> 3. Tested multicast by enabling multicast/ICMP and running multicast
>>>    pings. Ran with no data loss for > 10 minutes. (IBM Tech web site
>>>    article-- where the end of the article say it's almost always a network
>>>    problem.)
>>
>> I'm leaning to a network problem, too.
> 
> What else would you use to pinpoint the problem?  A Dell M6220 switch
> seems to be a Cisco clone, so any config suggestions are welcome.

Again, I am not a cisco user. Ask your network engineer(s) and/or Red
Hat for help specific to your environment.

>> I don't think RRP worked in EL5... Maybe it does now?
> 
> Good to know. As of 5.8, RRP doesn't seem to work for me (kernel faults).
> 
>> First and foremost; Get fencing working. At the very least, a lost node
>> will reboot and the cluster will recover as designed. It's amazing how
>> many problems "just go away" once fencing is properly configured. Please
>> read this:
>>
>> https://alteeve.com/w/2-Node_Red_Hat_KVM_Cluster_Tutorial#Concept.3B_Fencing
> 
> Sure... makes perfect sense if one has REAL node failures. In my case, all
> I'd have is a set of VMs crash (due to fencing) every 10 mins. The people
> using those VMs wouldn't be very happy with me.... :)

No, it makes sense whenever a node loses connection. The goal of fencing
is to ensure that two nodes don't try to both provide HA services. If
they can't talk to each other, then the *only* way to ensure that a node
is the only one providing services is to fence the lost node. It does
not matter at all that the rest of the machine is otherwise healthy.

> (I've done AIX and HP HA clusters before, so I understand the need/purpose
> for fencing failed nodes. In this case, I have real users using the VMs
> on the nodes-- the only thing that appears to fail is aisexec's communication.)

Then put your VMs under HA control. If the host node is lost, it will
restart on a healthy node. It's the *only* safe option.

-- 
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?