Hi all,
We've having an issue at the moment where an iSCSI connection was temporarily lost on a few VMs running CentOS 6 on ESXi.
The problem is, now that the iSCSI connection has returned, we are not able to remount the drive.
At first the drive is read-only, so I tried '*mount -o remount,rw*' which didn't work (still read-only), so then I tried a '*umount*' (which worked), but now I get the following error when trying to mount it again:
root@server [~]# mount /backup/ mount: /dev/sdb1 already mounted or /backup busy root@server [~]# mount /dev/sdb1 mount: /dev/sdb1 already mounted or /backup busy
I have also tried '*echo "- - -" > /sys/class/scsi_host/host1/scan*' and checked to see if any processes were using /dev/sdb1 or /backup using ' *fuser*' and '*lsof*'.
I know a reboot will solve it, but I'm trying to avoid that as best I can, so I'm wondering if anyone else has any other ideas?
Any help would be greatly appreciated, thanks. :)
Regards,
Kyle Thorne
Any clue on dmesg? I'd remove de disk and rescan...
El jue., 5 de marzo de 2015 a las 7:40, Kyle Thorne (< kthorne@staff.ventraip.com>) escribió:
Hi all,
We've having an issue at the moment where an iSCSI connection was temporarily lost on a few VMs running CentOS 6 on ESXi.
The problem is, now that the iSCSI connection has returned, we are not able to remount the drive.
At first the drive is read-only, so I tried '*mount -o remount,rw*' which didn't work (still read-only), so then I tried a '*umount*' (which worked), but now I get the following error when trying to mount it again:
root@server [~]# mount /backup/ mount: /dev/sdb1 already mounted or /backup busy root@server [~]# mount /dev/sdb1 mount: /dev/sdb1 already mounted or /backup busy
I have also tried '*echo "- - -" > /sys/class/scsi_host/host1/scan*' and checked to see if any processes were using /dev/sdb1 or /backup using ' *fuser*' and '*lsof*'.
I know a reboot will solve it, but I'm trying to avoid that as best I can, so I'm wondering if anyone else has any other ideas?
Any help would be greatly appreciated, thanks. :)
Regards,
Kyle Thorne
-- *The contents of this email are strictly private and confidential unless otherwise noted and is intended for the marked recipients only. If you are not a marked recipient please disregard and delete this email.* _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
The most recent message is:
[3108269.919256] sd 2:0:1:0: timing out command, waited 1080s [3108269.919528] sd 2:0:1:0: [sdb] Unhandled error code [3108269.919535] sd 2:0:1:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OK [3108269.919540] sd 2:0:1:0: [sdb] CDB: Read(10): 28 00 00 01 21 47 00 00 08 00 [3108269.919586] EXT4-fs error (device sdb1): ext4_find_entry: reading directory #2 offset 0
Removing the device, rescanning, and then re-adding it worked, but that moved the device to /dev/sdc instead. Which is fine, but it would be much better if it was /dev/sdb.
Thanks for your help. :)
On 5 March 2015 at 23:17, Marcelo Roccasalva < marcelo-centos@irrigacion.gov.ar> wrote:
Any clue on dmesg? I'd remove de disk and rescan...
El jue., 5 de marzo de 2015 a las 7:40, Kyle Thorne (< kthorne@staff.ventraip.com>) escribió:
Hi all,
We've having an issue at the moment where an iSCSI connection was temporarily lost on a few VMs running CentOS 6 on ESXi.
The problem is, now that the iSCSI connection has returned, we are not
able
to remount the drive.
At first the drive is read-only, so I tried '*mount -o remount,rw*' which didn't work (still read-only), so then I tried a '*umount*' (which
worked),
but now I get the following error when trying to mount it again:
root@server [~]# mount /backup/ mount: /dev/sdb1 already mounted or /backup busy root@server [~]# mount /dev/sdb1 mount: /dev/sdb1 already mounted or /backup busy
I have also tried '*echo "- - -" > /sys/class/scsi_host/host1/scan*' and checked to see if any processes were using /dev/sdb1 or /backup using ' *fuser*' and '*lsof*'.
I know a reboot will solve it, but I'm trying to avoid that as best I
can,
so I'm wondering if anyone else has any other ideas?
Any help would be greatly appreciated, thanks. :)
Regards,
Kyle Thorne
-- *The contents of this email are strictly private and confidential unless otherwise noted and is intended for the marked recipients only. If you
are
not a marked recipient please disregard and delete this email.* _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
# cat /etc/redhat-release CentOS Linux release 7.1.1503 (Core) # uname -r 3.10.0-123.20.1.el7.x86_64
Hi,
We use iSCSI over a 10G Ethernet Adapter and SRP over an Infiniband adapter to provide multipathing to our storage:
# lspci | grep 10-Gigabit 81:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) 81:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) # lspci | grep Mellanox 06:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0)
Originally the HBAs came up as host0, host1 and host2. After recent reboots we get host1, host2 and host3:
# ls -l /sys/class/scsi_host/host* lrwxrwxrwx. 1 root root 0 Apr 9 19:10 /sys/class/scsi_host/host1 -> ../../devices/platform/host1/scsi_host/host1 lrwxrwxrwx. 1 root root 0 Apr 9 08:12 /sys/class/scsi_host/host2 -> ../../devices/pci0000:00/0000:00:03.2/0000:06:00.0/host2/scsi_host/host2 lrwxrwxrwx. 1 root root 0 Apr 9 08:12 /sys/class/scsi_host/host3 -> ../../devices/pci0000:00/0000:00:03.2/0000:06:00.0/host3/scsi_host/host3
I've observed that at different times during configuration of this host we have had varying combinations of the mapping of HBA to an enumerated /sys node. e.g.:
host0 host2 host3
or
host1 host3 host4
other combinations, etc.
This messes up our multipath filtering for weightedpath priority settings. From /etc/multipath.conf:
defaults { user_friendly_names no find_multipaths yes prio weightedpath prio_args "hbtl 0:.:.:.* 1 [1-9]:.:.:.* 10" path_grouping_policy group_by_prio path_selector "service-time 0" failback immediate polling_interval 5 no_path_retry 12 }
The setting for prio_args fails to set the correct path map topology and priority against an HBA as I can't predict what HBA is what. Consistently iSCSI claims the lowest number but I can't say if that's going to be 0 or 1 a this stage.
Does anyone know how to configure HBAs to stay on a fixed enumeration at boot?
Kind regards, Tom
Tom Robinson IT Manager/System Administrator
MoTeC Pty Ltd
121 Merrindale Drive Croydon South 3136 Victoria Australia
T: +61 3 9761 5050 F: +61 3 9761 5051 E: tom.robinson@motec.com.au
Hi,
I have had the same problem and I solved it with following commands :
iscsiadm -m node -o show (not sure this was useful) iscsiadm -m node --login iscsiadm -m session -o show
after, umount -f and mount each mountpoint and now I can show disks with command blkid.
Hope will help. Laurent
-- View this message in context: http://centos.1050465.n5.nabble.com/CentOS-Cannot-remount-drive-after-lost-i... Sent from the CentOS mailing list archive at Nabble.com.
-
Kyle Thorne -
Laurent31 -
Marcelo Roccasalva -
Tom Robinson