The reason for the behaviour observed below turned out to be that the device entry in /etc/multipath.conf was inadvertently appended *after* the devices section , rather than inside it - so that we had
#devices { # device { # blah blah # } (file has a bunch of defaults commented out) # etc #} # # device { our settings }
*rather than*
devices { device { our settings } }
Also - looking more closely at our multipath.conf.defaults, there is an entry for a product pattern HSV2.*
That would explain why the multipath settings for the HSV200 looked different to the HSV400 - the HSV200 was picking up a different set of defaults. However still not exactly what we had specified.
(sort of like CSS behaviour....."cascading multipath.conf defaults" ! :-) )
You also possibly need to pay attention to the basic whitespace formatting of this file - for example I had noticed the above early on and rebooted with it "fixed" , only to find that this meant the system came up unable to even recognise the ext3 filesystem, complaining about bad superblock etc etc.
After recovering from that ( see below) I went back ( a week later) and a) made sure that the whitespace - tabs etc - looked like the other commented out defaults and b) added in a second device entry (from the defaults section - not a device we even have) , just in case there was some bug relating to having only one device entry in the "devices" section.
Whether because of those or some other change, the system now comes up fine, and with multipath -ll now reporting the correct settings for the HSV400. So issue resolved finally. Touch wood.
Here is a useful tip (not news to gurus but was to me ) : when things turn to custard on reboot and everything including the root filesystem is mounted readonly - so you can't even restore multipath.conf to the backup you made before rebooting - you can
mount -n -o remount /
to remount / as writeable - then you can
cd /etc cp multipath.conf.bu1 multipath.conf
and breathe a big sigh of relief !
If there are any multipath developers reading this - it could be handy if multipath could log some diagnostic info about how it parses the conf file , and exactly what entry it ends up using , which would then appear in /var/log/messages and /var/log/dmesg, since it seems relatively easy to end up matching a device spec you didn't expect, and the effects on performance are subtle so easy to overlook this.
Cheers
AMcC
------------------------ original post ------------------------
hi all
We have a rh linux server connected to two HP SAN controllers, one an HSV200 (on the way out), the other an HSV400 (on the way in). (Via a Qlogic HBA).
/etc/multipath.conf contains this :
device { vendor "(COMPAQ|HP)" product "HSV1[01]1|HSV2[01]0|HSV300|HSV4[05]0" getuid_callout "/sbin/scsi_id -g -u -s /block/%n" prio_callout "/sbin/mpath_prio_alua /dev/%n" hardware_handler "0" path_selector "round-robin 0" path_grouping_policy group_by_prio failback immediate rr_weight uniform no_path_retry 18 rr_min_io 100 path_checker tur }
- but our actual multipathing as shown by multipath -ll , and multipath -ll -v 3 looks as though for the HSV400 it is using the defaults rather than these settings. The defaults are
#defaults { # udev_dir /dev # polling_interval 10 # selector "round-robin 0" # path_grouping_policy multibus # getuid_callout "/sbin/scsi_id -g -u -s /block/%n" # prio_callout /bin/true # path_checker readsector0 # rr_min_io 100 # rr_weight priorities # failback immediate # no_path_retry fail # user_friendly_name yes
and multipath -ll reports :
. . [snip other HSV400 paths - all similar] mpath12 (3600508b40007518f0000900000520000) dm-1 HP,HSV400 [size=150G][features=0][hwhandler=0] _ round-robin 0 [prio=1][active] _ 0:0:5:9 sdab 65:176 [active][ready] _ round-robin 0 [prio=1][enabled] _ 0:0:3:9 sdn 8:208 [active][ready] _ round-robin 0 [prio=1][enabled] _ 0:0:4:9 sdu 65:64 [active][ready] mpath11 (3600508b40007518f0000700000370000) dm-6 HP,HSV200 [size=200G][features=1 queue_if_no_path][hwhandler=0] _ round-robin 0 [prio=50][active] _ 0:0:1:7 sdd 8:48 [active][ready] _ round-robin 0 [prio=10][enabled] _ 0:0:2:7 sdh 8:112 [active][ready] . . [snip other HSV200 paths - all similar]
multipath -ll -v 3 includes explicit statements that defaults are being used for the HSV400
(long output snipped...)
sdaa: path checker = readsector0 (config file default)
versus
sda: path checker = tur (controller setting)
sdx: getprio = NULL (internal default)
versus
sdd: getprio = /sbin/mpath_prio_alua %n (controller setting)
- furthermore we see in the log file messages from both readsector0 *and* tur rather than just tur if the correct settings were used , which also backs that up.
My questions are basically - why is it happening , and how to fix it ?
The vendor and product regexps definitely do match both "HSP" and both "HSV200" and "HSV400" respectively so it doesn't seem that fiddling with the patterns will work , and I'm sure this config has been tested.
Its not due to this server having to deal with two controllers - we have a second server that only mounts from the HSV400, and its multipath settings appear to be entirely the defaults, and not what we have set.
(And conversely, its not due to the conf file not being read at all - since the server with two controllers is using the correct config for one of them , but not the other.)
thanks for any tips and I will summarise.
Cheers
AMcC
======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================