I'm running a pair of squids as an internal cache for some intermediate data used by a web server farm and currently doing failover by going through an F5 bigip. However, I'd like to avoid the bigip and use heartbeat since there are only two machines. I don't need to sync any content since it is a fast-changing cache and either machine can handle everything. Is it possible to add a floating IP with heartbeat and cut over to it slowly while also still using the existing IP's on that subnet? The documentation I've seen implies that it is bad to use the real IP's. If heartbeat can't do this, would it work to use vrrp?
Les Mikesell wrote:
I'm running a pair of squids as an internal cache for some intermediate data used by a web server farm and currently doing failover by going through an F5 bigip. However, I'd like to avoid the bigip and use heartbeat since there are only two machines. I don't need to sync any
I use bigip even if there is only 1 machine, using heartbeat sounds like it will make it quite a bit more complicated, load balancers make life easier..since you already have a good one(a pair hopefully), keep using it!
I use load balancers wherever possible even with 1 machine so I can make changes to the system(s) in the pool without impact. e.g. if I want to replace hardware, I can install the new hardware, add it to the pool and remove the old system. Of course it's ideal to always have at least 2.
nate
nate wrote:
I'm running a pair of squids as an internal cache for some intermediate data used by a web server farm and currently doing failover by going through an F5 bigip. However, I'd like to avoid the bigip and use heartbeat since there are only two machines. I don't need to sync any
I use bigip even if there is only 1 machine, using heartbeat sounds like it will make it quite a bit more complicated, load balancers make life easier..since you already have a good one(a pair hopefully), keep using it!
We have a lot of other equipment behind it and I want the capacity for growth in the other pools. I agree it does a good job and is hard to match where you need balancing, but this only needs failover.
I use load balancers wherever possible even with 1 machine so I can make changes to the system(s) in the pool without impact. e.g. if I want to replace hardware, I can install the new hardware, add it to the pool and remove the old system. Of course it's ideal to always have at least 2.
A floating IP should serve the same purpose if you can swap it cleanly.
Assuming server A with IP M, server B with IP N, and DNS entry X currently pointing at IP M:
1) Add heartbeat on servers A and B, with heartbeat managing a new IP address O (this is your virtual IP -- nothing to do with VRRP, that's for your routers to failover your gateway). 2) If you want active-active load sharing on servers A and B, install pound on both server A and B, and in your pound config, point pound to IP M and IP N (same pound config on both servers), 3) Update your DNS to point entry X to IP M.
If you want active-standby on your squids, then have both squids bind to 0.0.0.0 and you're done. The standby server will have squid listening to requests, but since standby won't have the VIP O, it'll just sit there. In this setup, heartbeat is only managing the VIP, but no services.
If you want active-active on your squids, then have squid on server A bind to only IP M, squid on server B bind to only IP N, and pound configured to bind to only IP O. Heartbeat will need to be configured to start pound on failover (since IP O will only exist on one box at a time, so pound can't bind to it unless the interface is up).
Make sure you test the case where squid (or pound in active-active) on the server running VIP O crashes.
-Jeff
J Potter wrote:
Assuming server A with IP M, server B with IP N, and DNS entry X currently pointing at IP M:
- Add heartbeat on servers A and B, with heartbeat managing a new IP
address O (this is your virtual IP -- nothing to do with VRRP, that's for your routers to failover your gateway).
Thanks - I know vrrp is different (and compatible with routers and other devices) but I thought it should work as an alternative.
If you want active-standby on your squids, then have both squids bind to 0.0.0.0 and you're done. The standby server will have squid listening to requests, but since standby won't have the VIP O, it'll just sit there. In this setup, heartbeat is only managing the VIP, but no services.
Yes, I normally want one server handling the full load to maximize the cache hits. But the other one should be up and running. And, because this is already in production as a mid-level cache working behind a loadbalancer, I would like to be able to keep answering on the M or N addresses, gradually reconfiguring the web servers in the farm to use address O instead of the loadbalancer VIP address. I got the impression from some of the docs/tutorials that it was a bad idea to access the M/N addresses directly. Or does that only apply to services where it is important to only have one instance alive at a time or where you are replicating data? Even after converting all of the farm to use the new address, I'll still want to be able to monitor the backup server to be sure it is still healthy.
Yes, I normally want one server handling the full load to maximize the cache hits. But the other one should be up and running.
So, active/standby. Easier config. Squid won't even be aware that heartbeat is running; just keep it running on both servers all the time.
See my install notes at bottom.
And, because this is already in production as a mid-level cache working behind a loadbalancer, I would like to be able to keep answering on the M or N addresses, gradually reconfiguring the web servers in the farm to use address O instead of the loadbalancer VIP address.
Go for it. It'll work fine. You could get fancy and switch primary interface from M to O, and make M the VIP. Depends if you can accept the ~30 seconds of downtime and your tolerance for risk.
I got the impression from some of the docs/tutorials that it was a bad idea to access the M/N addresses directly.
In your case, it's only "bad" if M/N go down.
Or does that only apply to services where it is important to only have one instance alive at a time or where you are replicating data?
Depends on the service and replication setup. If you had master/slave MySQL and connected to the slave, you'd see amnesia on the master. (That setup wouldn't allow for fail-back though, so, probably wouldn't see it.) Things like drbd protect you from concurrent mounting.
Even after converting all of the farm to use the new address, I'll still want to be able to monitor the backup server to be sure it is still healthy.
Yup. And you'll want to monitor the active node and force-failover if the service fails. My config below doesn't take this into consideration; maybe other list lurkers can correct it to be better. The quick and dirty fix is to for each node to check if it is active, and if it is, if squid is not active, to then run 'service heartbeat restart' to failover to the other node. (I.e. once-a-minute cron job.) Not as pretty as it should be.
best, Jeff
Replace 1.2.3.4 with your VIP ip address, and a.example.com and b.example.com with your FQDN hostnames.
server A ("a.example.com"): yum -y install heartbeat chkconfig --add heartbeat chkconfig --level 345 heartbeat on
echo 'a.example.com IPaddr::1.2.3.4' > /etc/ha.d/haresources echo "node a.example.com" > /etc/ha.d/ha.cf echo "node b.example.com" >> /etc/ha.d/ha.cf echo "udpport 9000" >> /etc/ha.d/ha.cf echo "bcast bond0" >> /etc/ha.d/ha.cf echo "auto_failback off" >> /etc/ha.d/ha.cf echo "logfile /var/log/ha-log" >> /etc/ha.d/ha.cf echo "logfacility local0" >> /etc/ha.d/ha.cf echo "auth 1" > /etc/ha.d/authkeys echo "1 crc" >> /etc/ha.d/authkeys chmod go-rwx /etc/ha.d/authkeys
server B ("b.example.com"): yum -y install heartbeat chkconfig --add heartbeat chkconfig --level 345 heartbeat on
echo 'a.example.com IPaddr::1.2.3.4' > /etc/ha.d/haresources # yes, "a" again - that's the default host to run the service echo "node a.example.com" > /etc/ha.d/ha.cf echo "node b.example.com" >> /etc/ha.d/ha.cf echo "udpport 9000" >> /etc/ha.d/ha.cf echo "bcast bond0" >> /etc/ha.d/ha.cf echo "auto_failback off" >> /etc/ha.d/ha.cf echo "logfile /var/log/ha-log" >> /etc/ha.d/ha.cf echo "logfacility local0" >> /etc/ha.d/ha.cf echo "auth 1" > /etc/ha.d/authkeys echo "1 crc" >> /etc/ha.d/authkeys chmod go-rwx /etc/ha.d/authkeys
# This assumes: # 1) your network is bond0, not eth0 # 2) you are on a private network where you don't care about security, otherwise see http://www.linux-ha.org/authkeys # Make sure udpport isn't in use by any other instances; or, use mcast.
On server A: service heartbeat start # Then, check your log files (/var/log/ha-log and /var/log/messages). # Ping the virtual IP. On server B: service heartbeat start # check your log files
On server A: service heartbeat restart
On server B: ifconfig -a # Check if the interface is now runing on server B.
You can monitor current active node with arp -- the mac address will switch to match the physical interface that the VIP is running on.
-
J Potter -
Les Mikesell -
nate