<html><body bgcolor="#FFFFFF"><div><br><div><span class="Apple-style-span" style="-webkit-tap-highlight-color: rgba(26, 26, 26, 0.296875); -webkit-composition-fill-color: rgba(175, 192, 227, 0.230469); -webkit-composition-frame-color: rgba(77, 128, 180, 0.230469); ">On Sep 14, 2009, at 10:25 PM, "McCulloch, Alan" <<a href="mailto:alan.mcculloch@agresearch.co.nz">alan.mcculloch@agresearch.co.nz</a>> wrote:</span><br></div></div><div><br></div><div></div><blockquote type="cite"><div>
<font face="Calibri, sans-serif" size="2">
<div>hi All,</div>
<div> </div>
<div>thanks for the responses.</div>
<div> </div>
<div>After being dropped into the </div>
<div> </div>
<div># Filesystem repair </div>
<div> </div>
<div>prompt, </div>
<div> </div>
<div>( on account of “inode 27344909 has illegal blocks” )</div>
<div> </div>
<div>following warm reboot (via “reboot”) after finding (SAN ) filesystem in read-only </div>
<div>mode yesterday morning (possibly because of HBA fault on SAN) , I ran </div>
<div> </div>
<div>fsck –r /data </div>
<div> </div>
<div>(Linux version 2.6.18-92.1.18.el5 , Red Hat 4.1.2-42 , ext3 filesystem)</div>
<div> </div>
<div>This took a couple of hours or so , prompting me for various changes </div>
<div>all of which I accepted. This appeared to complete OK, but then the </div>
<div>system would not boot, with the following error from the qla2xxx driver.</div>
<div> </div>
<div>.</div>
<div>.</div>
<div>qla2xxx 0000:05:0d.0: Mailbox command timeout occurred. Scheduling ISP abort.</div>
<div>qla2xxx 0000:05:0d.0: Mailbox command timeout occurred. Scheduling ISP abort.</div>
<div>.</div>
<div>etc</div>
<div> </div>
<div>However after powering down the system and cold-booting, the system was able</div>
<div>to boot up and mount the repaired filesystem without any obvious damage, but with </div>
<div>abnormal not to mention scary looking boot messages and ongoing warnings from </div>
<div>multipath.</div>
<div> </div>
<div>This morning (as I sort of expected) the filesystem had dropped back down to read-only mode, but meanwhile</div>
<div>the source of our woes was identified, a fibre port on the SAN controller which was degraded but not </div>
<div>completely failed, so that there had been no clean failover to the twin controller, and therefore a degraded</div>
<div>virtual device was presented to the O/S, with consequence for the filesystem.</div>
<div> </div>
<div>After that port and controller was quarantined, this time around I did a cold power-off reboot</div>
<div>of the server , and this time there was a more normal looking boot and the filesystem </div>
<div>came up normally without any repair being requested.</div>
<div> </div>
<div>(My hypothesis is that in this situation – i.e. ext3 filesystem has put itself in read-only mode – </div>
<div>a warm boot , via reboot, does not cleanly remount the filesystem and apply the journal </div>
<div>quite like a cold power-off reboot does. I think it is likely that the lengthy</div>
<div>session of me answering “yes” to fsck’s interactive repair, the first time around, simply applied all of the </div>
<div>fixes that would automatically have been done from the journal , had I cold-rebooted in the first place.</div>
<div>However that is only a hunch. But I will be making sure to do cold power-off reboots in general, in</div>
<div>future.) </div>
<div> </div>
<div>Another lesson is that a sophisticated system of twin SAN controllers with failover does not protect</div>
<div>against a situation where a device is degrading rather than failing completely. </div>
<div> </div>
<div>Thanks again for the responses and sorry if my questions were a bit basic but I have</div>
<div>been dropped in a little out of my depth with this system.</div></font></div></blockquote><br><div>I always prefer round-robin mpath versus fail-over if possible as a degraded or failed path simply is not used, then there is the twice the bandwidth factor when both paths are working which is nice.</div><div><br></div><div>-Ross</div><div><br></div></body></html>