Hi all,
one of my lv has become completely unaccessible. Every read access results in a buffer io error:
Buffer I/O error on dev dm-13, logical block 0, async page read
this goes for every block in the lv. A ddrescue failed on every single block.
$ lvdisplay --- Logical volume --- LV Path /dev/vg0/lv-vm-tviewer LV Name lv-vm-tviewer VG Name vg0 LV UUID XdgHFs-RHVZ-9BAH-1ZSK-yiBX-qqf0-273CtT LV Write Access read/write LV Creation host, time host1, 2016-02-06 14:58:19 +0100 LV snapshot status INACTIVE destination for lv-vm-tviewer_vorigin LV Status available # open 0 LV Size 58.59 GiB Current LE 15000 COW-table size 5.86 GiB COW-table LE 1500 Snapshot chunk size 4.00 KiB Segments 1 Allocation inherit Read ahead sectors auto - currently set to 8192 Block device 253:13
Other lv on the same volume group are healthy. In fact the whole host runs on the same vg and does not show any problems. Physical volume is a md raid1 device which is also healthy, as well as its hard disks.
The faulty lv is used as the storage device for a qemu vm, which suddenly stalled when the io-errors appeared on the host.
Is the any way to diagnose this problem? (centos 7)
Regards .....Volker
On 2017-07-06 5:43 am, Volker wrote:
Hi all,
one of my lv has become completely unaccessible. Every read access results in a buffer io error:
Buffer I/O error on dev dm-13, logical block 0, async page read
this goes for every block in the lv. A ddrescue failed on every single block.
$ lvdisplay --- Logical volume --- LV Path /dev/vg0/lv-vm-tviewer LV Name lv-vm-tviewer VG Name vg0 LV UUID XdgHFs-RHVZ-9BAH-1ZSK-yiBX-qqf0-273CtT LV Write Access read/write LV Creation host, time host1, 2016-02-06 14:58:19 +0100 LV snapshot status INACTIVE destination for lv-vm-tviewer_vorigin LV Status available # open 0 LV Size 58.59 GiB Current LE 15000 COW-table size 5.86 GiB COW-table LE 1500 Snapshot chunk size 4.00 KiB Segments 1 Allocation inherit Read ahead sectors auto
- currently set to 8192
Block device 253:13
Other lv on the same volume group are healthy. In fact the whole host runs on the same vg and does not show any problems. Physical volume is a md raid1 device which is also healthy, as well as its hard disks.
The faulty lv is used as the storage device for a qemu vm, which suddenly stalled when the io-errors appeared on the host.
Is the any way to diagnose this problem? (centos 7)
Regards .....Volker
Hello, Volker.
Assuming that, at this point, everything that would have been running from this LV/FS is down, have you considered unmounting the filesystem and running a filesystem check against it?
If it's EXT2/3/4, you can run fsck against it, if it's XFS, you can run xfs_repair against it.
On 07/06/2017 04:43 AM, Volker wrote:
Hi all,
one of my lv has become completely unaccessible. Every read access results in a buffer io error:
Buffer I/O error on dev dm-13, logical block 0, async page read
this goes for every block in the lv. A ddrescue failed on every single block.
$ lvdisplay --- Logical volume --- LV Path /dev/vg0/lv-vm-tviewer LV Name lv-vm-tviewer VG Name vg0 LV UUID XdgHFs-RHVZ-9BAH-1ZSK-yiBX-qqf0-273CtT LV Write Access read/write LV Creation host, time host1, 2016-02-06 14:58:19 +0100 LV snapshot status INACTIVE destination for lv-vm-tviewer_vorigin LV Status available # open 0 LV Size 58.59 GiB Current LE 15000 COW-table size 5.86 GiB COW-table LE 1500 Snapshot chunk size 4.00 KiB Segments 1 Allocation inherit Read ahead sectors auto
- currently set to 8192
Block device 253:13
That looks like a snapshot volume that became invalid because it was filled to capacity. Such a snapshot is lost forever. It si your responsibility to monitor snapshot usage to make sure it does not run out of space. The base volume, lv-vm-tviewer_vorigin, should still have its original content.
On 06.07.2017 15:35, Robert Nichols wrote:
On 07/06/2017 04:43 AM, Volker wrote:
Hi all,
one of my lv has become completely unaccessible. Every read access results in a buffer io error:
Buffer I/O error on dev dm-13, logical block 0, async page read
this goes for every block in the lv. A ddrescue failed on every single block.
$ lvdisplay --- Logical volume --- LV Path /dev/vg0/lv-vm-tviewer LV Name lv-vm-tviewer VG Name vg0 LV UUID XdgHFs-RHVZ-9BAH-1ZSK-yiBX-qqf0-273CtT LV Write Access read/write LV Creation host, time host1, 2016-02-06 14:58:19 +0100 LV snapshot status INACTIVE destination for lv-vm-tviewer_vorigin LV Status available # open 0 LV Size 58.59 GiB Current LE 15000 COW-table size 5.86 GiB COW-table LE 1500 Snapshot chunk size 4.00 KiB Segments 1 Allocation inherit Read ahead sectors auto
- currently set to 8192
Block device 253:13
That looks like a snapshot volume that became invalid because it was filled to capacity. Such a snapshot is lost forever. It si your responsibility to monitor snapshot usage to make sure it does not run out of space. The base volume, lv-vm-tviewer_vorigin, should still have its original content.
That's it. Thanks for the information.
How would I remove the only the snapshot but not the base volume? Will
$ lvremove /dev/vg0/lv-vm-tviewer
leave the base volume untouched?
On 07/06/2017 10:47 AM, Volker wrote:
On 06.07.2017 15:35, Robert Nichols wrote:
That looks like a snapshot volume that became invalid because it was filled to capacity. Such a snapshot is lost forever. It si your responsibility to monitor snapshot usage to make sure it does not run out of space. The base volume, lv-vm-tviewer_vorigin, should still have its original content.
That's it. Thanks for the information.
How would I remove the only the snapshot but not the base volume? Will
$ lvremove /dev/vg0/lv-vm-tviewer
leave the base volume untouched?
That should do it. You will get one less warning if you deactivate that LV first: lvchange -an vg0/lv-vm-tviewer
You will still get an overly alarmist warning that removing the snapshot LV will change the source LV. Don't worry about it. The only "change" is that the source LV will no longer be flagged as a snapshot origin.
On 07/06/2017 01:31 PM, Robert Nichols wrote:
On 07/06/2017 10:47 AM, Volker wrote:
On 06.07.2017 15:35, Robert Nichols wrote:
That looks like a snapshot volume that became invalid because it was filled to capacity. Such a snapshot is lost forever. It si your responsibility to monitor snapshot usage to make sure it does not run out of space. The base volume, lv-vm-tviewer_vorigin, should still have its original content.
That's it. Thanks for the information.
How would I remove the only the snapshot but not the base volume? Will
$ lvremove /dev/vg0/lv-vm-tviewer
leave the base volume untouched?
That should do it. You will get one less warning if you deactivate that LV first: lvchange -an vg0/lv-vm-tviewer
You will still get an overly alarmist warning that removing the snapshot LV will change the source LV. Don't worry about it. The only "change" is that the source LV will no longer be flagged as a snapshot origin.
I should add, you may have to re-activate that base LV: lvchange -ay vg0/lv-vm-tviewer_vorigin
-
Mike Burger -
Robert Nichols -
Volker