<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Bryan J. Smith wrote:
<blockquote
cite="mid20051110021456.50355.qmail@web34111.mail.mud.yahoo.com"
type="cite">
<pre wrap="">kadafax <a class="moz-txt-link-rfc2396E" href="mailto:kadafax@gmail.com"><kadafax@gmail.com></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">(I found that the module used was qla6320, dont know if
there is an issue with this but I thought that the qla2200
would be more appropriate since the host card is a qla200.)
</pre>
</blockquote>
<pre wrap=""><!---->
There's many variants in the QLogic 200 series. I just
listed the common denominator that most people have. I
figured if you were messing with SAN solutions, you might
know the exact variant/driver you were already running.
Sorry about that. I probably should have told you to run
"lsmod" to verify first.
</pre>
</blockquote>
never too late:<br>
---<br>
<small>[root@X ~]# lsmod <br>
Module Size Used by<br>
qla6312 116545 0 <br>
qla2322 135745 0 <br>
qla2200 88769 0 <br>
qla2100 80961 0 <br>
qla2300 126017 0 <br>
ipt_REJECT 8641 1 <br>
ipt_state 3265 13 <br>
ip_conntrack 53657 1 ipt_state<br>
iptable_filter 4417 1 <br>
ip_tables 20289 3 ipt_REJECT,ipt_state,iptable_filter<br>
autofs4 23241 0 <br>
dcdipm 68980 2 <br>
dcdbas 50964 2 <br>
joydev 11841 0 <br>
button 9057 0 <br>
battery 11209 0 <br>
ac 6729 0 <br>
uhci_hcd 34665 0 <br>
ehci_hcd 33349 0 <br>
hw_random 7137 0 <br>
e1000 110381 0 <br>
floppy 65809 0 <br>
sg 42489 0 <br>
dm_snapshot 18561 0 <br>
dm_zero 3649 0 <br>
dm_mirror 28889 0 <br>
ext3 137681 6 <br>
jbd 68849 1 ext3<br>
dm_mod 66433 10 dm_snapshot,dm_zero,dm_mirror<br>
qla2xxx 178849 7 qla6312,qla2322,qla2200,qla2100,qla2300<br>
scsi_transport_fc 11201 1 qla2xxx<br>
megaraid_mbox 40017 2 <br>
megaraid_mm 15881 5 megaraid_mbox<br>
sd_mod 19392 5 <br>
scsi_mod 140177 5
sg,qla2xxx,scsi_transport_fc,megaraid_mbox,sd_mod</small><br>
---<br>
<small>[root@X ~]# modprobe -l | grep qla<br>
/lib/modules/2.6.9-22.0.1.ELsmp/kernel/drivers/scsi/qla2xxx/qla2300.ko<br>
/lib/modules/2.6.9-22.0.1.ELsmp/kernel/drivers/scsi/qla2xxx/qla2100.ko<br>
/lib/modules/2.6.9-22.0.1.ELsmp/kernel/drivers/scsi/qla2xxx/qla6312.ko<br>
/lib/modules/2.6.9-22.0.1.ELsmp/kernel/drivers/scsi/qla2xxx/qla2200.ko<br>
/lib/modules/2.6.9-22.0.1.ELsmp/kernel/drivers/scsi/qla2xxx/qla2322.ko<br>
/lib/modules/2.6.9-22.0.1.ELsmp/kernel/drivers/scsi/qla2xxx/qla2xxx.ko<br>
/lib/modules/2.6.9-22.0.1.ELsmp/kernel/drivers/scsi/qla1280.ko</small><br>
---<br>
<small>[root@X ~]# cat /proc/scsi/qla2xxx/4 <br>
QLogic PCI to Fibre Channel Host Adapter for QLA200:<br>
Firmware version 3.03.15 FLX, Driver version 8.01.00b5-rh2<br>
ISP: ISP6312, Serial# U93681<br>
Request Queue = 0x21980000, Response Queue = 0x33300000<br>
Request Queue count = 2048, Response Queue count = 512<br>
Total number of active commands = 0<br>
Total number of interrupts = 427<br>
Device queue depth = 0x20<br>
Number of free request entries = 1585<br>
Number of mailbox timeouts = 0<br>
Number of ISP aborts = 0<br>
Number of loop resyncs = 0<br>
Number of retries for empty slots = 0<br>
Number of reqs in pending_q= 0, retry_q= 0, done_q= 0, scsi_retry_q= 0<br>
Host adapter:loop state = <READY>, flags = 0x1a13<br>
Dpc flags = 0x80000<br>
MBX flags = 0x0<br>
Link down Timeout = 030<br>
Port down retry = 030<br>
Login retry count = 030<br>
Commands retried with dropped frame(s) = 0<br>
Product ID = 4953 5020 2020 0003</small><br>
<br>
<br>
<small>SCSI Device Information:<br>
scsi-qla3-adapter-node=200000e08b1f71f2;<br>
scsi-qla3-adapter-port=210000e08b1f71f2;<br>
scsi-qla3-target-0=5006016039201733;<br>
<br>
FC Port Information:<br>
scsi-qla3-port-0=50060160b9201733:5006016039201733:0000ef:0;<br>
<br>
SCSI LUN Information:<br>
(Id:Lun) * - indicates lun is not registered with the OS.<br>
( 0: 0): Total reqs 184, Pending reqs 0, flags 0x0, 3:0:00 00<br>
( 0: 1): Total reqs 183, Pending reqs 0, flags 0x0, 3:0:00 00<br>
---<br>
[root@X ~]# cat /proc/partitions <br>
major minor #blocks name<br>
<br>
8 0 143257600 sda<br>
8 1 104391 sda1<br>
8 2 143147182 sda2<br>
253 0 51216384 dm-0<br>
253 1 22216704 dm-1<br>
253 2 4096000 dm-2<br>
253 3 51216384 dm-3<br>
253 4 10256384 dm-4<br>
253 5 4096000 dm-5<br>
253 6 683667456 dm-6<br>
253 7 227536896 dm-7<br>
8 48 911212544 sdd<br>
8 64 227540992 sde</small>
<br>
---<br>
<small>[root@X ~]# cat /proc/scsi/scsi <br>
Attached devices:<br>
Host: scsi0 Channel: 00 Id: 06 Lun: 00<br>
Vendor: PE/PV Model: 1x6 SCSI BP Rev: 1.0 <br>
Type: Processor ANSI SCSI revision: 02<br>
Host: scsi0 Channel: 02 Id: 00 Lun: 00<br>
Vendor: MegaRAID Model: LD 0 RAID1 139G Rev: 521S<br>
Type: Direct-Access ANSI SCSI revision: 02<br>
Host: scsi4 Channel: 00 Id: 00 Lun: 00<br>
Vendor: DGC Model: RAID 5 Rev: 0217<br>
Type: Direct-Access ANSI SCSI revision: 04<br>
Host: scsi4 Channel: 00 Id: 00 Lun: 01<br>
Vendor: DGC Model: RAID 5 Rev: 0217<br>
Type: Direct-Access ANSI SCSI revision: 04</small><br>
---<br>
<br>
<small>[root@X ~]# ls /proc/<br>
1 1853 25048 2712 28654 55 9 filesystems
locks stat<br>
10 1854 25062 2721 2866 56 acpi fs
mdstat swaps<br>
11 2 2510 2738 2867 57 buddyinfo ide
meminfo sys<br>
12 227 258 2750 2868 58 bus interrupts
misc sysrq-trigger<br>
13 239 259 2790 3 59 cmdline iomem
modules sysvipc<br>
1355 23965 2596 2804 30357 6 cpuinfo ioports
mounts tty<br>
14 23968 2612 2805 30359 7 crypto irq
mtrr uptime<br>
15 2416 2631 2815 30360 75 devices kallsyms
net version<br>
152 2420 2660 2822 30386 76 diskstats kcore
partitions vmstat<br>
1666 2430 2670 2863 30387 77 dma keys pci<br>
1850 2470 2692 2864 30433 78 driver key-users scsi<br>
1851 2471 2702 2865 4 79 execdomains kmsg self<br>
1852 2500 271 28653 5 8 fb loadavg slabinfo</small><br>
---<br>
<br>
<br>
<blockquote
cite="mid20051110021456.50355.qmail@web34111.mail.mud.yahoo.com"
type="cite">
<pre wrap="">
Okay, next time I'll make that big, long, step-by-step
verbose post that makes the seasoned guys around here cringe
(just stick up for me if anyone complains ;-).
</pre>
</blockquote>
I'll make a summary once we sorted on this. I have another HD to add to
the SAN, so I will reboot the server (after putting a slave ldap
somewhere) and restart the procedure from the beginning. Note: the
process to add a HD to the disk pool takes more than one day (near two
days in fact, don't know how it is with other SAN, but I found this
quite long - maybe it is normal behavior for the system to take so much
time to re-create RAID 5, just don't know... hopefully I can work with
the mounted volume during the process).<br>
(btw, your long technical post are precious)<br>
<blockquote
cite="mid20051110021456.50355.qmail@web34111.mail.mud.yahoo.com"
type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">how to release device assignments? I didn't find any clue
on google.
</pre>
</blockquote>
<pre wrap=""><!---->
You have to go into the /proc filesystem. Location may
differ between kernel 2.4 and 2.6.
</pre>
<blockquote type="cite">
<pre wrap="">nope CentOS 4.2
</pre>
</blockquote>
<pre wrap=""><!---->
Hmmm, interesting. I guess when we loaded the qla2200
module, it went after the same path as another module already
loaded(?). I guess I'd have to see the output of your
"lsmod" and look around the /proc filesystem.
Eck. It's stuff like this that I just need to be in front of
the system, or I could be making things worse for you (and
probably already am ;-).
</pre>
</blockquote>
I have taken some precautions. no problem :)<br>
<blockquote
cite="mid20051110021456.50355.qmail@web34111.mail.mud.yahoo.com"
type="cite">
<pre wrap=""></pre>
<blockquote type="cite">
<pre wrap="">( come on :) )
Will this take off-line the physical extents along with the
VG? :
---
[root@X ~]# vgchange -a n /dev/VG-B/LV-B
---
</pre>
</blockquote>
<pre wrap=""><!---->
I'd rather not say either way without being right at your
setup. There are just so many factors. E.g., the fact that
your main /dev/sda is live (is it also a SAN connection?).
I'd really rather not tell you to do anything else at this
point.
</pre>
</blockquote>
no the /dev/sda is a RAID 1 virtual disk from 2 HD directly connected
to an internal PERC-4 RAID controller card.<br>
/dev/sda1 is the /boot and sda2 is the system Volume Group.<br>
<blockquote
cite="mid20051110021456.50355.qmail@web34111.mail.mud.yahoo.com"
type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">after the 'vgchange -a n', 'rmmod & modprobe' then the
'vgchange -a y':
---
[root@X ~]# vgscan
Reading all physical volumes. This may take a while...
Found volume group "VG-C" using metadata type lvm2
Found volume group "VG-B" using metadata type lvm2
Found volume group "VolGroup00" using metadata type lvm2
---
</pre>
</blockquote>
<pre wrap=""><!---->
Looks good. I'm just curious what we do about the SCSI
mapping though -- let alone the fact that we loaded that
other module.
</pre>
<blockquote type="cite">
<pre wrap="">but the /dev/sdb and sdc are gone, the PV have just
followed the shift of the device assignments and the new
available physical extents are not there.
</pre>
</blockquote>
<pre wrap=""><!---->
Hmmm, what does the raw "fdisk -l /dev/sdd" and "fdisk -l
/dev/sde" commands give you? Do you see the extra space (you
should)?
</pre>
</blockquote>
Yeah the extra space has showned up with fdisk (/dev/sdd, who was
/dev/sdb yesterday, growed up after the module reload):<br>
---<br>
<small>[root@X ~]# fdisk -l /dev/sdd<br>
<br>
Disk /dev/sdd: 933.0 GB, 933081645056 bytes<br>
255 heads, 63 sectors/track, 113440 cylinders<br>
Units = cylinders of 16065 * 512 = 8225280 bytes<br>
<br>
Disk /dev/sdd doesn't contain a valid partition table</small><br>
---<br>
<small>[root@X ~]# fdisk -l /dev/sde<br>
<br>
Disk /dev/sde: 233.0 GB, 233001975808 bytes<br>
255 heads, 63 sectors/track, 28327 cylinders<br>
Units = cylinders of 16065 * 512 = 8225280 bytes<br>
<br>
Disk /dev/sde doesn't contain a valid partition table</small><br>
---<br>
<br>
<blockquote
cite="mid20051110021456.50355.qmail@web34111.mail.mud.yahoo.com"
type="cite">
<pre wrap="">
I'm just trying to think how you expand the PVs (slice type
8Eh, I assume?) to the actual size of the partitions.
</pre>
</blockquote>
I think it's here I missed something. I have not created a partition
with fdisk before working with lvm. basically I've told lvm to use the
entire /dev/sdb (now /dev/sde with extra space). Maybe if I had first
created a partition (type LVM with fdisk) equal to HD size (~650GB)
let's say /dev/sdb1, then I would be able to create another partition
with the extra space (something like /dev/sdb2 (if there was no
shifting with the device assignment ) with the extra 230GB) and then I
would be able to create a new physical volume and benefit the new extra
physical extents.<br>
---<br>
<small>[root@X ~]# pvs<br>
PV VG Fmt Attr PSize PFree <br>
...<br>
/dev/sdd VG-B lvm2 a- 652.00G 0 <br>
...</small><br>
---<br>
BUT:<br>
---<br>
<small>[root@X ~]# fdisk -l /dev/sdd<br>
<br>
Disk /dev/sdd: 933.0 GB, 933081645056 bytes<br>
255 heads, 63 sectors/track, 113440 cylinders<br>
Units = cylinders of 16065 * 512 = 8225280 bytes<br>
<br>
Disk /dev/sdd doesn't contain a valid partition table</small><br>
---<br>
<br>
<blockquote
cite="mid20051110021456.50355.qmail@web34111.mail.mud.yahoo.com"
type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">After putting off-line the VG (vgchange -a n)
unloading then loading the qla module, the new correct
size is shown by fdisk but not with LVM (no new physical
extents appeared). How to gain the new extents
within LVM when the HD growed up?
</pre>
</blockquote>
<pre wrap=""><!---->
Okay, so you ran "fdisk -l" and see it -- good.
You probably need to add another slice (partition) to the
devices then. Type it as LVM (8Eh), then add that as another
PV.
</pre>
</blockquote>
May I create a new slice without erasing the data previously created
(again it is not a problem, nothing usefull yet on this volume)? I'm
quite confused here, there is no partition table for fdisk because I've
done all with lvm and it's working (ext3 fs). Can I create now a
partition table with fdisk?<br>
<blockquote
cite="mid20051110021456.50355.qmail@web34111.mail.mud.yahoo.com"
type="cite">
<blockquote type="cite">
<pre wrap="">(It's late here in europe, I will be back tomorrow. Thanks
again for support).
</pre>
</blockquote>
<pre wrap=""><!---->
I'll be here (up all night for the next 2 nights, long
story).
</pre>
</blockquote>
</body>
</html>