Hello all. I posted this in the forum and was told to instead post it to the mailing list. My apologies for the redundancy if you have already seen and been irritated by my blatherings.
Thanks. _________________________
I am working on a CentOS clustered LAMP stack and running into problems. I have searched extensively and have come up empty.
Here's my setup:
Two node cluster identical hardware. IBM x226 with RSAII adapters for fencing. Configured for Active/Passive failover - no load balancing. No shared storage - manual rsync of data (shared SSH keys, rsync over SSH, cron job). Single shared IP address
I used luci and ricci to configure the cluster. It's a bit confusing that there's an 'apache' script but you have to use the custom init script. I'm past that though.
The failover function is working when it's kicked off manually from the luci web interface. I can tell it to transfer the services (IP, httpd, msqld) to the secondary server and it works fine.
I run into problems when I attempt to simulate a failure (a pulled network cord for instance). The primary system recognizes the failure, shuts down it's services, attempts to inform the secondary server to take over and then it never does. Here is a log excerpt from a cable pull test:
Jun 16 15:33:27 flex kernel: tg3: eth0: Link is down. Jun 16 15:33:34 flex clurgmgrd: [2970]: <warning> Link for eth0: Not detected Jun 16 15:33:34 flex clurgmgrd: [2970]: <warning> No link on eth0... Jun 16 15:33:34 flex clurgmgrd[2970]: <notice> status on ip "10.6.2.25" returned 1 (generic error) Jun 16 15:33:34 flex clurgmgrd[2970]: <notice> Stopping service service:web Jun 16 15:33:35 flex proftpd[6321]: 10.6.2.47 - ProFTPD killed (signal 15) Jun 16 15:33:35 flex proftpd[6321]: 10.6.2.47 - ProFTPD 1.3.3c standalone mode SHUTDOWN Jun 16 15:33:39 flex avahi-daemon[2850]: Withdrawing address record for 10.6.2.25 on eth0. Jun 16 15:33:49 flex clurgmgrd[2970]: <notice> Service service:web is recovering Jun 16 15:33:49 flex clurgmgrd[2970]: <notice> Recovering failed service service:web Jun 16 15:33:49 flex clurgmgrd: [2970]: <warning> Link for eth0: Not detected Jun 16 15:33:49 flex clurgmgrd[2970]: <notice> start on ip "10.6.2.25" returned 1 (generic error) Jun 16 15:33:49 flex clurgmgrd[2970]: <warning> #68: Failed to start service:web; return value: 1 Jun 16 15:33:49 flex clurgmgrd[2970]: <notice> Stopping service service:web Jun 16 15:33:49 flex clurgmgrd: [2970]: <err> script:mysqld: stop of /etc/rc.d/init.d/mysqld failed (returned 1) Jun 16 15:33:49 flex clurgmgrd[2970]: <notice> stop on script "mysqld" returned 1 (generic error) Jun 16 15:33:49 flex clurgmgrd[2970]: <crit> #12: RG service:web failed to stop; intervention required Jun 16 15:33:49 flex clurgmgrd[2970]: <notice> Service service:web is failed Jun 16 15:33:49 flex clurgmgrd[2970]: <crit> #13: Service service:web failed to stop cleanly Jun 16 15:36:43 flex kernel: tg3: eth0: Link is up at 100 Mbps, full duplex. Jun 16 15:36:43 flex kernel: tg3: eth0: Flow control is off for TX and off for RX. Jun 16 16:04:52 flex luci[2904]: Unable to retrieve batch 306226694 status from web2:11111: Unable to disable failed service web before starting it:clusvcadm failed to stop web: Jun 16 16:05:28 flex clurgmgrd[2970]: <notice> Starting disabled service service:web Jun 16 16:05:31 flex avahi-daemon[2850]: Registering new address record for 10.6.2.25 on eth0. Jun 16 16:05:31 flex luci[2904]: Unable to retrieve batch 1997354692 status from web2:11111: module scheduled for execution Jun 16 16:05:33 flex proftpd[1926]: 10.6.2.47 - ProFTPD 1.3.3c (maint) (built Thu Nov 18 2010 03:38:57 CET) standalone mode STARTUP Jun 16 16:05:33 flex clurgmgrd[2970]: <notice> Service service:web started
I have followed the HowTos for setting up the cluster (with the exception of the shared storage) as closely as possible.
Here's what I've already troubleshot:
No IPTables running No SELinux running Hosts file resolves all IP address/host names properly.
I must say that I am less familiar with how all of the cluster components work together. All of the Linux clusters I have built thus far have been heartbeat+mon style clusters.
I'm looking to find out if there is an additional debug layer that I can put in place to get some more detailed information about what is transacting (or not) between the two cluster members.
Many thanks.
Ryan Bunce wrote:
I must say that I am less familiar with how all of the cluster components work together. All of the Linux clusters I have built thus far have been heartbeat+mon style clusters.
I'm looking to find out if there is an additional debug layer that I can put in place to get some more detailed information about what is transacting (or not) between the two cluster members.
Many thanks.
I never installed or used any Conga/lucci/ricci sistem.
But as far as I know and understand, you need to have a way for server failing to warn the rest of the nodes. Your log said it failed.
Some of the failover sistems need separate network connected to collective file systems. So when eth0 is not working, main node will use eth1(2,3,4) to report this event to all other nodes.
What comes to mind is that IP's set for interconnection (in lucci conf) must not be public IP's but of that separate/secundary network in order for main node to be able to contact the rest of the nodes.
I hope this helps.
Ljubomir
Ryan Bunce wrote: <snip>
I am working on a CentOS clustered LAMP stack and running into problems. I have searched extensively and have come up empty.
Here's my setup:
Two node cluster identical hardware. IBM x226 with RSAII adapters for fencing. Configured for Active/Passive failover - no load balancing. No shared storage - manual rsync of data (shared SSH keys, rsync over SSH, cron job). Single shared IP address
I used luci and ricci to configure the cluster. It's a bit confusing that there's an 'apache' script but you have to use the custom init script. I'm past that though.
<snip> I'm not sure if either of them install heartbeat.
mark
-
Ljubomir Ljubojevic -
m.roth＠5-cent.us -
Ryan Bunce