[CentOS] LVM and hotswap (USB/iSCSI) devices?

Sun Aug 31 11:14:57 UTC 2008
William L. Maltby <CentOS4Bill at triad.rr.com>

On Sun, 2008-08-31 at 11:19 +0200, Simen Timian Thoresen wrote:
> Hi list,
> 
> I'm having one of those 'I'm stupid' -problems with LVM on CentOS 5.2. 
> I've been working with traditional partitions until now, but I've 
> finally been sold on the theoretical benefits of using LVM, but for now 
> I only have a huge pile of broken filesystems to show for my efforts.
> 
> My scenario;
> I attach a disk, either over USB or iSCSI.
> I create a PV on this device, create a VG using the device, and slice 
> off a tiny LV for tests.

Uh-oh! I hope these are for temporary usage only!

> 
> [root at kasse ~]# dmesg
> <snip>

> [root at kasse ~]# pvcreate /dev/sdg1
>    Physical volume "/dev/sdg1" successfully created
> [root at kasse ~]# vgcreate testgroup /dev/sdg1
>    Volume group "testgroup" successfully created
> [root at kasse ~]# lvcreate -L 200M testgroup -n testLV
>    Logical volume "testLV" created
> 
> 
> At this point, I have /dev/testgroup/testLV, which I can stick a fs on, 
> mount, store files on, unmount, remount and recover the files from. Just 
> as I'd expect.
> 
> [root at kasse ~]# lvdisplay
>    --- Logical volume ---
>    LV Name                /dev/testgroup/testLV
>    VG Name                testgroup
>    LV UUID                3KjDiZ-gTAD-sC9E-sOSA-772o-41Yk-ZKK7cx
>    LV Write Access        read/write
>    LV Status              available
>    # open                 0
>    LV Size                200.00 MB
>    Current LE             50
>    Segments               1
>    Allocation             inherit
>    Read ahead sectors     auto
>    - currently set to     256
>    Block device           253:1
> 
> 
> I have not explicitly set the vg active, as I understand that it 
> implicitly will be set active in non-clustered environments like mine. 
> LVM works as I'd expect it to up until here in any case.
> 
> So - with the LV unmounted, I power my USB-device down, and then back up;
> 
> [root at kasse ~]# dmesg
> <snip>


> (note that my disk is now sdi, not sdg, and connected on scsi28 and not 
> scsi27)

By now you realize that is a big problem?

> 
> If I now try to access the device, all falls appart;
> [root at kasse ~]# lvdisplay
>    /dev/testgroup/testLV: read failed after 0 of 4096 at 209649664: 
> Input/output error
> <snip>

>    --- Logical volume ---
>    LV Name                /dev/testgroup/testLV
>    VG Name                testgroup
>    LV UUID                3KjDiZ-gTAD-sC9E-sOSA-772o-41Yk-ZKK7cx
>    LV Write Access        read/write
>    LV Status              available
>    # open                 0
>    LV Size                200.00 MB
>    Current LE             50
>    Segments               1
>    Allocation             inherit
>    Read ahead sectors     auto
>    - currently set to     256
>    Block device           253:1
> 
> 
> Trying to mount or otherwise access the LV also shows the problems;
> 
> 
> scsi 27:0:0:0: rejecting I/O to dead device
> <snip>
> Buffer I/O error on device dm-1, logical block 7
> scsi 27:0:0:0: rejecting I/O to dead device
> scsi 27:0:0:0: rejecting I/O to dead device
> scsi 27:0:0:0: rejecting I/O to dead device
> scsi 27:0:0:0: rejecting I/O to dead device
> hfs: unable to find HFS+ superblock
> 
> So - LVM seems to believe my LV is still attached to scsi27, while it 
> now actually sits on scsi28.
> 
> What am I missing here?

Use the pvdisplay command and you'll see.

# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sdb2
  VG Name               VolGroup00
  PV Size               148.95 GB / not usable 11.37 MB
  Allocatable           yes (but full)
  PE Size (KByte)       32768
  Total PE              4766
  Free PE               0
  Allocated PE          4766
  PV UUID               eeqb0A-6EJV-VE3Q-paLi-kTXv-S46k-0xaeTy

Sit back a moment and think about it. You defined a PV on a specific
device. That device is no longer available. LVM has to track the
assignment, allocation and use of PVs. It must have a record of it
somewhere.

# ls -l /etc/lvm
total 44
drwx------ 2 root root  4096 May 25 00:41 archive
drwx------ 2 root root  4096 May 25 00:41 backup
drwx------ 2 root root  4096 Aug 30 08:38 cache
-rw-r--r-- 1 root root 15911 May 25 00:41 lvm.conf

Use the man command to read up on more stuff, like "man lvm.conf" and
"man lvm". Be sure and read some of the other refs at the bottom of the
man pages.

> 
> How do I get LVM to use a persistent device instead of a transient path?

Normally you give a persistent device instead of a transient path.  You
didn't do that.

Generally speaking, LVM is designed to use fixed devices - those that
always come up on the same address. Neither iscsi nor UHB devices do
this reliably. However, on a reboot, there is a liklihood that they will
be the same devices.

I've not tried to configure it on transient devices. There may be a way,
but I've not looked into it.

If you make them their own VG, rather than adding the PV to another VG,
you can use the export and import LVM commands, but then you really
accomplish nothing more than what could be done by mounting the
underlying devices in existing directory structures. No point in LVM
then unless you anticipate having to grow those mounted structures while
in use.

Even so, from my LFS days, I do recall a lot of flexibility in the udev
process. Maybe there is something there that will allow the assigned
device addresses to be fixed?

Do man for udev, udevd, udevinfo, udevmonitor. I'm guessing that you'll
want to run some utility from udev that will (maybe) look at a a volume
label (if one if available on the usb device - is there one on the iscsi
device?) and assign a predetermined name if a certain label is found.

I've never used iscsi, so I'm even less help on that. Maybe there is
something in its man pages that allow a persistent device name
assignment.

> 
> Yours,
> -S

Sorry I couldn't offer more.

-- 
Bill