I'm interested in the idea of sharing a bunch of SAS JBOD devices between two CentOS servers in an active-standby HA cluster sort of arrangement, and found something about using scsi3 persistent reservations as a fencing method. I'm not finding a lot of specifics about how this works, or how you configure two initiator systems on a SAS chain. I don't have any suitable hardware for running any tests or evaluations yet.
general idea: 2 centos servers each with 8 port external SAS cards (2 x4), cascaded through a SAS box-o-disks with a whole bunch of SAS dual ported drives, to implement high availability large nearline storage. all storage configured as JBOD, using linux md raid or lvm mirroring. drives should only be accessible by the currently active server, with heartbeat managing the fencing.
here's a hardware block diagram http://freescruz.com/shared-sas.jpg
This is about the only details I've found on scsi persistant reservations and linux HA http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Configurat...
One question I have is: how well will this scale with several strings of 100 SAS drives on the same HA pair of servers?
Can SAS storage instead be fenced at the SES/expander level rather than having to use reservations with each separate drive?
You might be able to go a simpler route, try building a GFBD between them and sharing the storage that way. I personally have never tried it, but that might give you the HA you're looking for while staying withing the CentOS/Upstream packaging. One added benefit is the ability to scale vertically and horizontally with a setup like that with failover and quorum. Heck, toss in a little heartbeat scripting and you could call it an auto-healing storage cloud.
Good luck!
On Fri, May 20, 2011 at 5:28 PM, John R Pierce pierce@hogranch.com wrote:
I'm interested in the idea of sharing a bunch of SAS JBOD devices between two CentOS servers in an active-standby HA cluster sort of arrangement, and found something about using scsi3 persistent reservations as a fencing method. I'm not finding a lot of specifics about how this works, or how you configure two initiator systems on a SAS chain. I don't have any suitable hardware for running any tests or evaluations yet.
general idea: 2 centos servers each with 8 port external SAS cards (2 x4), cascaded through a SAS box-o-disks with a whole bunch of SAS dual ported drives, to implement high availability large nearline storage. all storage configured as JBOD, using linux md raid or lvm mirroring. drives should only be accessible by the currently active server, with heartbeat managing the fencing.
here's a hardware block diagram http://freescruz.com/shared-sas.jpg
This is about the only details I've found on scsi persistant reservations and linux HA http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Configurat...
One question I have is: how well will this scale with several strings of 100 SAS drives on the same HA pair of servers?
Can SAS storage instead be fenced at the SES/expander level rather than having to use reservations with each separate drive?
-- john r pierce N 37, W 123 santa cruz ca mid-left coast
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
--On Friday, May 20, 2011 02:28:18 PM -0700 John R Pierce pierce@hogranch.com wrote:
[snip]
One question I have is: how well will this scale with several strings of 100 SAS drives on the same HA pair of servers?
Can SAS storage instead be fenced at the SES/expander level rather than having to use reservations with each separate drive?
It's been a few years since I've used SCSI reservations as a fencing method (as in about the time that wide SCSI was taking over from narrow SCSI, and while it fenced nicely, it sometimes caused bus errors when you detached the dead node from the bus).
However if you are using a set of disks as a logical unit from the cluster's perspective (like having set of disks acting as a RAID device), then IIRC it is sufficient to use a reservation on a single disk as your fence for the entire disk set (rather than using a reservation on every single disk). Of course, of your storage cabinet has a few different disk sets in it, you'd need additional reservations (one per set).
But, as I said, it's been a few years so I'd investigate it more rather than depending on *just* the above comment. In recent years I've tended to use various mechanisms to kill power to the node as my preferred fencing method.
The linux-ha and pacemaker lists may be a better source of current information.
Devin
-
Devin Reade -
John R Pierce -
Steven Crothers