I have a dual xen/drbd cluster now broken after the 5.3 update Saturday. FATAL: Could not open '/lib/modules/2.6.18-128.1.6.el5xen/weak-updates/drbd82/drbd.ko': No such file or directory
I'll research further when I get to the office.
Thanks, jerry
Jerry Amundson wrote:
I have a dual xen/drbd cluster now broken after the 5.3 update Saturday. FATAL: Could not open '/lib/modules/2.6.18-128.1.6.el5xen/weak-updates/drbd82/drbd.ko': No such file or directory
I'll research further when I get to the office.
Thanks, jerry
Working here :
[root@helium ~]# modinfo drbd filename: /lib/modules/2.6.18-128.1.6.el5xen/weak-updates/drbd82/drbd.ko alias: block-major-147-* license: GPL description: drbd - Distributed Replicated Block Device v8.2.6 author: Philipp Reisner phil@linbit.com, Lars Ellenberg lars@linbit.com srcversion: 61C6A5A780BA9774A184BC1 depends: vermagic: 2.6.18-92.1.13.el5xen SMP mod_unload gcc-4.1 parm: minor_count:Maximum number of drbd devices (1-255) (int) parm: allow_oos:DONT USE! (bool) parm: enable_faults:int parm: fault_rate:int parm: fault_count:int parm: fault_devs:int parm: trace_level:int parm: trace_type:int parm: trace_devs:int parm: proc_details:int parm: usermode_helper:string
Have you removed manually the original drbd.ko ? it should be a symlink to /lib/modules/2.6.18-92.1.13.el5xen/extra/drbd82/drbd.ko
-- Fabian Arrotin idea=`grep -i clue /dev/brain` test -z "$idea" && echo "sorry, init 6 in progress" || sh ./answer.sh
On Mon, Apr 13, 2009 at 9:20 AM, Fabian Arrotin fabian.arrotin@arrfab.net wrote:
Jerry Amundson wrote:
I have a dual xen/drbd cluster now broken after the 5.3 update Saturday. FATAL: Could not open '/lib/modules/2.6.18-128.1.6.el5xen/weak-updates/drbd82/drbd.ko': No such file or directory
I'll research further when I get to the office.
Thanks, jerry
Working here :
[root@helium ~]# modinfo drbd filename: /lib/modules/2.6.18-128.1.6.el5xen/weak-updates/drbd82/drbd.ko alias: block-major-147-* license: GPL description: drbd - Distributed Replicated Block Device v8.2.6 author: Philipp Reisner phil@linbit.com, Lars Ellenberg lars@linbit.com srcversion: 61C6A5A780BA9774A184BC1 depends: vermagic: 2.6.18-92.1.13.el5xen SMP mod_unload gcc-4.1 parm: minor_count:Maximum number of drbd devices (1-255) (int) parm: allow_oos:DONT USE! (bool) parm: enable_faults:int parm: fault_rate:int parm: fault_count:int parm: fault_devs:int parm: trace_level:int parm: trace_type:int parm: trace_devs:int parm: proc_details:int parm: usermode_helper:string
Have you removed manually the original drbd.ko ? it should be a symlink to /lib/modules/2.6.18-92.1.13.el5xen/extra/drbd82/drbd.ko
No. Should I?
As yum sees things:
Installed Packages drbd82.i386 8.2.6-1.el5.centos installed kernel-headers.i386 2.6.18-128.1.6.el5 installed kernel-xen.i686 2.6.18-92.1.10.el5 installed kernel-xen.i686 2.6.18-92.1.22.el5 installed kernel-xen.i686 2.6.18-128.1.6.el5 installed kmod-drbd82-xen.i686 8.2.6-1.2.6.18_92.1.10.el5 installed kmod-drbd82-xen.i686 8.2.6-2 installed Available Packages drbd.i386 8.0.13-1.el5.centos extras kernel.i686 2.6.18-128.1.6.el5 updates kernel-PAE.i686 2.6.18-128.1.6.el5 updates kernel-PAE-devel.i686 2.6.18-128.1.6.el5 updates kernel-debug.i686 2.6.18-128.1.6.el5 updates kernel-debug-devel.i686 2.6.18-128.1.6.el5 updates kernel-devel.i686 2.6.18-128.1.6.el5 updates kernel-doc.noarch 2.6.18-128.1.6.el5 updates kernel-xen-devel.i686 2.6.18-128.1.6.el5 updates kmod-drbd.i686 8.0.13-2 extras kmod-drbd-PAE.i686 8.0.13-2 extras kmod-drbd-xen.i686 8.0.13-2 extras kmod-drbd82.i686 8.2.6-2 extras kmod-drbd82-PAE.i686 8.2.6-2 extras yum-kernel-module.noarch 1.1.16-13.el5.centos base
thanks, jerry
On Mon, Apr 13, 2009 at 6:56 AM, Jerry Amundson jamundso@gmail.com wrote:
I have a dual xen/drbd cluster now broken after the 5.3 update Saturday. FATAL: Could not open '/lib/modules/2.6.18-128.1.6.el5xen/weak-updates/drbd82/drbd.ko': No such file or directory
Could you post the output from:
uname -mr rpm -qa kernel* | sort rpm -qa drbd* ls -l `find /lib/modules -name drbd.ko`
Akemi
On Mon, Apr 13, 2009 at 9:20 AM, Akemi Yagi amyagi@gmail.com wrote:
On Mon, Apr 13, 2009 at 6:56 AM, Jerry Amundson jamundso@gmail.com wrote:
I have a dual xen/drbd cluster now broken after the 5.3 update Saturday. FATAL: Could not open '/lib/modules/2.6.18-128.1.6.el5xen/weak-updates/drbd82/drbd.ko': No such file or directory
Could you post the output from:
uname -mr rpm -qa kernel* | sort rpm -qa drbd* ls -l `find /lib/modules -name drbd.ko`
[root@xena ~]# uname -mr 2.6.18-128.1.6.el5xen i686 [root@xena ~]# rpm -qa kernel* | sort kernel-headers-2.6.18-128.1.6.el5 kernel-xen-2.6.18-128.1.6.el5 kernel-xen-2.6.18-92.1.10.el5 kernel-xen-2.6.18-92.1.22.el5 [root@xena ~]# rpm -qa drbd* drbd82-8.2.6-1.el5.centos [root@xena ~]# ls -l `find /lib/modules -name drbd.ko` lrwxrwxrwx 1 root root 54 Apr 11 03:36 /lib/modules/2.6.18-128.1.6.el5xen/weak-updates/drbd82/drbd.ko -> /lib/modules/2.6.18-92.1.6.el5xen/extra/drbd82/drbd.ko -rw-r--r-- 1 root root 2021323 Aug 7 2008 /lib/modules/2.6.18-92.1.10.el5xen/extra/drbd82/drbd.ko lrwxrwxrwx 1 root root 55 Jan 11 03:31 /lib/modules/2.6.18-92.1.10.el5xen/weak-updates/drbd82/drbd.ko -> /lib/modules/2.6.18-92.1.13.el5xen/extra/drbd82/drbd.ko -rw-r--r-- 1 root root 2021323 Oct 3 2008 /lib/modules/2.6.18-92.1.13.el5xen/extra/drbd82/drbd.ko lrwxrwxrwx 1 root root 55 Jan 11 03:31 /lib/modules/2.6.18-92.1.22.el5xen/weak-updates/drbd82/drbd.ko -> /lib/modules/2.6.18-92.1.13.el5xen/extra/drbd82/drbd.ko [root@xena ~]#
On Mon, Apr 13, 2009 at 7:49 AM, Jerry Amundson jamundso@gmail.com wrote:
[root@xena ~]# uname -mr 2.6.18-128.1.6.el5xen i686 [root@xena ~]# rpm -qa kernel* | sort kernel-headers-2.6.18-128.1.6.el5 kernel-xen-2.6.18-128.1.6.el5 kernel-xen-2.6.18-92.1.10.el5 kernel-xen-2.6.18-92.1.22.el5 [root@xena ~]# rpm -qa drbd* drbd82-8.2.6-1.el5.centos [root@xena ~]# ls -l `find /lib/modules -name drbd.ko` lrwxrwxrwx 1 root root 54 Apr 11 03:36 /lib/modules/2.6.18-128.1.6.el5xen/weak-updates/drbd82/drbd.ko -> /lib/modules/2.6.18-92.1.6.el5xen/extra/drbd82/drbd.ko
Is this symlink blinking in red by any chance? If not, you do seem to have that "missing" file. Just to double-check:
rpm -qf /lib/modules/2.6.18-92.1.6.el5xen/extra/drbd82/drbd.ko
should return the kmod-drbd82-xen 8.2.6-2 package.
Akemi
On Mon, Apr 13, 2009 at 10:06 AM, Akemi Yagi amyagi@gmail.com wrote:
On Mon, Apr 13, 2009 at 7:49 AM, Jerry Amundson jamundso@gmail.com wrote:
[root@xena ~]# uname -mr 2.6.18-128.1.6.el5xen i686 [root@xena ~]# rpm -qa kernel* | sort kernel-headers-2.6.18-128.1.6.el5 kernel-xen-2.6.18-128.1.6.el5 kernel-xen-2.6.18-92.1.10.el5 kernel-xen-2.6.18-92.1.22.el5 [root@xena ~]# rpm -qa drbd* drbd82-8.2.6-1.el5.centos [root@xena ~]# ls -l `find /lib/modules -name drbd.ko` lrwxrwxrwx 1 root root 54 Apr 11 03:36 /lib/modules/2.6.18-128.1.6.el5xen/weak-updates/drbd82/drbd.ko -> /lib/modules/2.6.18-92.1.6.el5xen/extra/drbd82/drbd.ko
Is this symlink blinking in red by any chance? If not, you do seem to have that "missing" file. Just to double-check:
rpm -qf /lib/modules/2.6.18-92.1.6.el5xen/extra/drbd82/drbd.ko
should return the kmod-drbd82-xen 8.2.6-2 package.
[root@xena ~]# rpm -qf /lib/modules/2.6.18-92.1.6.el5xen/extra/drbd82/drbd.ko error: file /lib/modules/2.6.18-92.1.6.el5xen/extra/drbd82/drbd.ko: No such file or directory
I'd guess that removing kmod-drbd82-xen*, and installing kmod-drbd82-xen will resolve this, but is there anything else you want me to look into beforehand? I 'd like to get the domu's running again... :-)
thanks again, jerry
On Mon, Apr 13, 2009 at 8:16 AM, Jerry Amundson jamundso@gmail.com wrote:
Is this symlink blinking in red by any chance? If not, you do seem to have that "missing" file. Just to double-check:
rpm -qf /lib/modules/2.6.18-92.1.6.el5xen/extra/drbd82/drbd.ko
should return the kmod-drbd82-xen 8.2.6-2 package.
[root@xena ~]# rpm -qf /lib/modules/2.6.18-92.1.6.el5xen/extra/drbd82/drbd.ko error: file /lib/modules/2.6.18-92.1.6.el5xen/extra/drbd82/drbd.ko: No such file or directory
I'd guess that removing kmod-drbd82-xen*, and installing kmod-drbd82-xen will resolve this, but is there anything else you want me to look into beforehand? I 'd like to get the domu's running again... :-)
I think you should go ahead and do it now. I don't understand why the install of the latest kmod-drbd82-xen did not work out. Maybe Fabian knows...
Akemi
On Mon, Apr 13, 2009 at 10:33 AM, Akemi Yagi amyagi@gmail.com wrote:
On Mon, Apr 13, 2009 at 8:16 AM, Jerry Amundson jamundso@gmail.com wrote:
Is this symlink blinking in red by any chance? If not, you do seem to have that "missing" file. Just to double-check:
rpm -qf /lib/modules/2.6.18-92.1.6.el5xen/extra/drbd82/drbd.ko
should return the kmod-drbd82-xen 8.2.6-2 package.
[root@xena ~]# rpm -qf /lib/modules/2.6.18-92.1.6.el5xen/extra/drbd82/drbd.ko error: file /lib/modules/2.6.18-92.1.6.el5xen/extra/drbd82/drbd.ko: No such file or directory
I'd guess that removing kmod-drbd82-xen*, and installing kmod-drbd82-xen will resolve this, but is there anything else you want me to look into beforehand? I 'd like to get the domu's running again... :-)
I think you should go ahead and do it now. I don't understand why the install of the latest kmod-drbd82-xen did not work out. Maybe Fabian knows...
Ok, I'll do that.
[root@xena ~]# grep drbd /var/log/yum.log.1 Jul 11 15:37:55 Installed: drbd - 8.0.12-1.el5.centos.i386 Jul 11 15:37:59 Installed: kmod-drbd-xen - 8.0.12-1.2.6.18_92.1.6.el5.i686 Jul 29 14:55:20 Erased: kmod-drbd-xen Jul 29 14:55:39 Installed: drbd82 - 8.2.6-1.el5.centos.i386 Jul 29 14:55:39 Erased: drbd Jul 29 14:56:38 Installed: kmod-drbd82-xen - 8.2.6-1.2.6.18_92.1.6.el5.i686 Aug 15 14:00:42 Installed: kmod-drbd82-xen - 8.2.6-1.2.6.18_92.1.10.el5.i686 Jan 11 03:32:10 Installed: kmod-drbd82-xen - 8.2.6-2.i686 Apr 11 03:37:31 Erased: kmod-drbd82-xen
I'm interested what was erased, when two are still installed... kmod-drbd82-xen.i686 8.2.6-1.2.6.18_92.1.10.el5 installed kmod-drbd82-xen.i686 8.2.6-2 installed
Anyway, I'll clean them off, then install.
Thanks, jerry
On Mon, Apr 13, 2009 at 10:33 AM, Akemi Yagi amyagi@gmail.com wrote:
On Mon, Apr 13, 2009 at 8:16 AM, Jerry Amundson jamundso@gmail.com wrote:
Is this symlink blinking in red by any chance? If not, you do seem to have that "missing" file. Just to double-check:
rpm -qf /lib/modules/2.6.18-92.1.6.el5xen/extra/drbd82/drbd.ko
should return the kmod-drbd82-xen 8.2.6-2 package.
[root@xena ~]# rpm -qf /lib/modules/2.6.18-92.1.6.el5xen/extra/drbd82/drbd.ko error: file /lib/modules/2.6.18-92.1.6.el5xen/extra/drbd82/drbd.ko: No such file or directory
I'd guess that removing kmod-drbd82-xen*, and installing kmod-drbd82-xen will resolve this, but is there anything else you want me to look into beforehand? I 'd like to get the domu's running again... :-)
I think you should go ahead and do it now. I don't understand why the install of the latest kmod-drbd82-xen did not work out. Maybe Fabian knows...
I've fixed this by removing kmod-drbd82-xen*, and installing kmod-drbd82-xen. Oddly, the erase process actually left another hanging symlink (below). ISTR that topic covered in a recent thread, so no big deal.
Thanks for listening, jerry
[root@xena ~]# yum erase kmod-drbd82-xen Loaded plugins: fastestmirror Setting up Remove Process Resolving Dependencies --> Running transaction check ---> Package kmod-drbd82-xen.i686 0:8.2.6-1.2.6.18_92.1.10.el5 set to be erased ---> Package kmod-drbd82-xen.i686 0:8.2.6-2 set to be erased --> Finished Dependency Resolution
Dependencies Resolved
================================================================================ Package Arch Version Repository Size ================================================================================ Removing: kmod-drbd82-xen i686 8.2.6-2 installed 1.9 M kmod-drbd82-xen i686 8.2.6-1.2.6.18_92.1.10.el5 installed 1.9 M
Transaction Summary ================================================================================ Install 0 Package(s) Update 0 Package(s) Remove 2 Package(s)
Is this ok [y/N]: y Downloading Packages: Running rpm_check_debug Running Transaction Test Finished Transaction Test Transaction Test Succeeded Running Transaction Erasing : kmod-drbd82-xen [1/2] WARNING: Can't read module /lib/modules/2.6.18-128.1.6.el5xen/weak-updates/drbd82/drbd.ko: No such file or directory WARNING: No module dm-mem-cache found for kernel 2.6.18-92.1.10.el5xen, continuing anyway WARNING: No module dm-region_hash found for kernel 2.6.18-92.1.10.el5xen, continuing anyway WARNING: No module dm-message found for kernel 2.6.18-92.1.10.el5xen, continuing anyway WARNING: No module dm-raid45 found for kernel 2.6.18-92.1.10.el5xen, continuing anyway WARNING: No module dm-mem-cache found for kernel 2.6.18-92.1.22.el5xen, continuing anyway WARNING: No module dm-region_hash found for kernel 2.6.18-92.1.22.el5xen, continuing anyway WARNING: No module dm-message found for kernel 2.6.18-92.1.22.el5xen, continuing anyway WARNING: No module dm-raid45 found for kernel 2.6.18-92.1.22.el5xen, continuing anyway Erasing : kmod-drbd82-xen [2/2]
Removed: kmod-drbd82-xen.i686 0:8.2.6-2 kmod-drbd82-xen.i686 0:8.2.6-1.2.6.18_92.1.10.el5 Complete! [root@xena ~]# ls -l `find /lib/modules -name drbd.ko` lrwxrwxrwx 1 root root 54 Apr 11 03:36 /lib/modules/2.6.18-128.1.6.el5xen/weak-updates/drbd82/drbd.ko -> /lib/modules/2.6.18-92.1.6.el5xen/extra/drbd82/drbd.ko lrwxrwxrwx 1 root root 55 Apr 13 10:43 /lib/modules/2.6.18-92.1.22.el5xen/weak-updates/drbd82/drbd.ko -> /lib/modules/2.6.18-92.1.10.el5xen/extra/drbd82/drbd.ko [root@xena ~]# rpm -qf /lib/modules/*/weak-updates/* file /lib/modules/2.6.18-128.1.6.el5xen/weak-updates/drbd82 is not owned by any package file /lib/modules/2.6.18-92.1.22.el5xen/weak-updates/drbd82 is not owned by any package [root@xena ~]# rm -rf /lib/modules/*/weak-updates/* [root@xena ~]# yum install kmod-drbd82-xen Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.fdcservers.net * updates: centos.mirror.nac.net * addons: pubmirrors.reflected.net * extras: pubmirrors.reflected.net Setting up Install Process Parsing package install arguments Resolving Dependencies --> Running transaction check ---> Package kmod-drbd82-xen.i686 0:8.2.6-2 set to be installed --> Finished Dependency Resolution
Dependencies Resolved
================================================================================ Package Arch Version Repository Size ================================================================================ Installing: kmod-drbd82-xen i686 8.2.6-2 extras 823 k
Transaction Summary ================================================================================ Install 1 Package(s) Update 0 Package(s) Remove 0 Package(s)
Total download size: 823 k Is this ok [y/N]: y Downloading Packages: kmod-drbd82-xen-8.2.6-2.i686.rpm | 823 kB 00:00 Running rpm_check_debug Running Transaction Test Finished Transaction Test Transaction Test Succeeded Running Transaction Installing : kmod-drbd82-xen [1/1] WARNING: No module dm-mem-cache found for kernel 2.6.18-92.1.10.el5xen, continuing anyway WARNING: No module dm-region_hash found for kernel 2.6.18-92.1.10.el5xen, continuing anyway WARNING: No module dm-message found for kernel 2.6.18-92.1.10.el5xen, continuing anyway WARNING: No module dm-raid45 found for kernel 2.6.18-92.1.10.el5xen, continuing anyway WARNING: No module dm-mem-cache found for kernel 2.6.18-92.1.22.el5xen, continuing anyway WARNING: No module dm-region_hash found for kernel 2.6.18-92.1.22.el5xen, continuing anyway WARNING: No module dm-message found for kernel 2.6.18-92.1.22.el5xen, continuing anyway WARNING: No module dm-raid45 found for kernel 2.6.18-92.1.22.el5xen, continuing anyway
Installed: kmod-drbd82-xen.i686 0:8.2.6-2 Complete!
On Mon, 13 Apr 2009, Jerry Amundson wrote:
I've fixed this by removing kmod-drbd82-xen*, and installing kmod-drbd82-xen. Oddly, the erase process actually left another hanging symlink (below). ISTR that topic covered in a recent thread, so no big deal.
FWIW: I upgraded two servers from 5.2 -> 5.3 a week ago. On reboot heartbeat didn't bring up all my services. A couple of dangling symlinks were in the Apache area which I had to fix manually, which sounds similar to your problem.
I don't know whether those bad symlinks were caused by my drbdlinks configuration or by the upgrade. I can look in my notes for the specifics on the bad symlinks if anyones cares about that.
Not complaining... Happily running here with 5.3!
On Mon, Apr 13, 2009 at 8:56 AM, Jerry Amundson jamundso@gmail.com wrote:
I have a dual xen/drbd cluster now broken after the 5.3 update Saturday. FATAL: Could not open '/lib/modules/2.6.18-128.1.6.el5xen/weak-updates/drbd82/drbd.ko': No such file or directory
Just to beat on this further, I ran into the the exact same problem when I upgraded the secondary server (note to self: update the secondary first next time :-).
I'll leave this one "unfixed" for now, for the sake of investigating what may have caused the original problem.
[root@xenb ~]# uname -mr 2.6.18-92.1.22.el5xen i686 [root@xenb ~]# rpm -qa kernel* | sort kernel-headers-2.6.18-128.1.6.el5 kernel-xen-2.6.18-128.1.6.el5 kernel-xen-2.6.18-92.1.10.el5 kernel-xen-2.6.18-92.1.22.el5 [root@xenb ~]# rpm -qa *drbd* | sort drbd82-8.2.6-1.el5.centos kmod-drbd82-xen-8.2.6-1.2.6.18_92.1.10.el5 kmod-drbd82-xen-8.2.6-2 [root@xenb ~]# ls -l `find /lib/modules -name drbd.ko` lrwxrwxrwx 1 root root 54 Apr 13 13:57 /lib/modules/2.6.18-128.1.6.el5xen/weak-updates/drbd82/drbd.ko -> /lib/modules/2.6.18-92.1.6.el5xen/extra/drbd82/drbd.ko -rw-r--r-- 1 root root 2021323 Aug 7 2008 /lib/modules/2.6.18-92.1.10.el5xen/extra/drbd82/drbd.ko lrwxrwxrwx 1 root root 55 Mar 19 14:41 /lib/modules/2.6.18-92.1.10.el5xen/weak-updates/drbd82/drbd.ko -> /lib/modules/2.6.18-92.1.13.el5xen/extra/drbd82/drbd.ko -rw-r--r-- 1 root root 2021323 Oct 3 2008 /lib/modules/2.6.18-92.1.13.el5xen/extra/drbd82/drbd.ko lrwxrwxrwx 1 root root 55 Mar 19 14:41 /lib/modules/2.6.18-92.1.22.el5xen/weak-updates/drbd82/drbd.ko -> /lib/modules/2.6.18-92.1.13.el5xen/extra/drbd82/drbd.ko [root@xenb ~]# find /lib/modules -name drbd.ko | xargs rpm -qf file /lib/modules/2.6.18-92.1.22.el5xen/weak-updates/drbd82/drbd.ko is not owned by any package file /lib/modules/2.6.18-92.1.10.el5xen/weak-updates/drbd82/drbd.ko is not owned by any package kmod-drbd82-xen-8.2.6-1.2.6.18_92.1.10.el5 file /lib/modules/2.6.18-128.1.6.el5xen/weak-updates/drbd82/drbd.ko is not owned by any package kmod-drbd82-xen-8.2.6-2
thanks, jerry
On Mon, Apr 13, 2009 at 1:20 PM, Jerry Amundson jamundso@gmail.com wrote:
On Mon, Apr 13, 2009 at 8:56 AM, Jerry Amundson jamundso@gmail.com wrote:
I have a dual xen/drbd cluster now broken after the 5.3 update Saturday. FATAL: Could not open '/lib/modules/2.6.18-128.1.6.el5xen/weak-updates/drbd82/drbd.ko': No such file or directory
Just to beat on this further, I ran into the the exact same problem when I upgraded the secondary server (note to self: update the secondary first next time :-).
I'll leave this one "unfixed" for now, for the sake of investigating what may have caused the original problem.
Thank you. That will help troubleshoot the current issue.
[root@xenb ~]# uname -mr 2.6.18-92.1.22.el5xen i686 [root@xenb ~]# rpm -qa kernel* | sort kernel-headers-2.6.18-128.1.6.el5 kernel-xen-2.6.18-128.1.6.el5 kernel-xen-2.6.18-92.1.10.el5 kernel-xen-2.6.18-92.1.22.el5 [root@xenb ~]# rpm -qa *drbd* | sort drbd82-8.2.6-1.el5.centos kmod-drbd82-xen-8.2.6-1.2.6.18_92.1.10.el5 kmod-drbd82-xen-8.2.6-2 [root@xenb ~]# ls -l `find /lib/modules -name drbd.ko` lrwxrwxrwx 1 root root 54 Apr 13 13:57 /lib/modules/2.6.18-128.1.6.el5xen/weak-updates/drbd82/drbd.ko -> /lib/modules/2.6.18-92.1.6.el5xen/extra/drbd82/drbd.ko
This is not right. kmod-drbd82-xen-8.2.6-2 is supposed to install:
/lib/modules/2.6.18-92.1.13.el5xen/extra/drbd82/drbd.ko
and the symlink should look like:
/lib/modules/2.6.18-128.1.6.el5xen/weak-updates/drbd82/drbd.ko -> /lib/modules/2.6.18-92.1.13.el5xen/extra/drbd82/drbd.ko
Could you show us the output from:
rpm -ql kmod-drbd82-xen
rpm -q --scripts kmod-drbd82-xen
Akemi
On Mon, Apr 13, 2009 at 4:05 PM, Akemi Yagi amyagi@gmail.com wrote:
On Mon, Apr 13, 2009 at 1:20 PM, Jerry Amundson jamundso@gmail.com wrote:
On Mon, Apr 13, 2009 at 8:56 AM, Jerry Amundson jamundso@gmail.com wrote:
I have a dual xen/drbd cluster now broken after the 5.3 update Saturday. FATAL: Could not open '/lib/modules/2.6.18-128.1.6.el5xen/weak-updates/drbd82/drbd.ko': No such file or directory
Just to beat on this further, I ran into the the exact same problem when I upgraded the secondary server (note to self: update the secondary first next time :-).
I'll leave this one "unfixed" for now, for the sake of investigating what may have caused the original problem.
Thank you. That will help troubleshoot the current issue.
[root@xenb ~]# uname -mr 2.6.18-92.1.22.el5xen i686 [root@xenb ~]# rpm -qa kernel* | sort kernel-headers-2.6.18-128.1.6.el5 kernel-xen-2.6.18-128.1.6.el5 kernel-xen-2.6.18-92.1.10.el5 kernel-xen-2.6.18-92.1.22.el5 [root@xenb ~]# rpm -qa *drbd* | sort drbd82-8.2.6-1.el5.centos kmod-drbd82-xen-8.2.6-1.2.6.18_92.1.10.el5 kmod-drbd82-xen-8.2.6-2 [root@xenb ~]# ls -l `find /lib/modules -name drbd.ko` lrwxrwxrwx 1 root root 54 Apr 13 13:57 /lib/modules/2.6.18-128.1.6.el5xen/weak-updates/drbd82/drbd.ko -> /lib/modules/2.6.18-92.1.6.el5xen/extra/drbd82/drbd.ko
This is not right. kmod-drbd82-xen-8.2.6-2 is supposed to install:
/lib/modules/2.6.18-92.1.13.el5xen/extra/drbd82/drbd.ko
and the symlink should look like:
/lib/modules/2.6.18-128.1.6.el5xen/weak-updates/drbd82/drbd.ko -> /lib/modules/2.6.18-92.1.13.el5xen/extra/drbd82/drbd.ko
Could you show us the output from:
rpm -ql kmod-drbd82-xen
rpm -q --scripts kmod-drbd82-xen
[root@xenb ~]# rpm -ql kmod-drbd82-xen /lib/modules/2.6.18-92.1.10.el5xen/extra/drbd82/drbd.ko /lib/modules/2.6.18-92.1.13.el5xen /lib/modules/2.6.18-92.1.13.el5xen/extra /lib/modules/2.6.18-92.1.13.el5xen/extra/drbd82 /lib/modules/2.6.18-92.1.13.el5xen/extra/drbd82/drbd.ko [root@xenb ~]# [root@xenb ~]# rpm -q --scripts kmod-drbd82-xen postinstall scriptlet (using /bin/sh): /sbin/depmod -aeF /boot/System.map-2.6.18-92.1.10.el5xen 2.6.18-92.1.10.el5xen > /dev/null || : postuninstall scriptlet (using /bin/sh): /sbin/depmod -aF /boot/System.map-2.6.18-92.1.10.el5xen 2.6.18-92.1.10.el5xen &> /dev/null || : postinstall scriptlet (using /bin/sh): if [ -e "/boot/System.map-2.6.18-92.1.13.el5xen" ]; then /sbin/depmod -aeF "/boot/System.map-2.6.18-92.1.13.el5xen" "2.6.18-92.1.13.el5xen" > /dev/null || : fi modules=( "/lib/modules/2.6.18-92.1.13.el5xen/extra/drbd82/drbd.ko" ) if [ -x "/sbin/weak-modules" ]; then printf '%s\n' "${modules[@]}" | /sbin/weak-modules --add-modules fi preuninstall scriptlet (using /bin/sh): rpm -ql kmod-drbd82-xen | grep '.ko$' > /var/run/rpm-kmod-drbd82-xen-modules postuninstall scriptlet (using /bin/sh): /sbin/depmod -aF /boot/System.map-2.6.18-92.1.13.el5xen 2.6.18-92.1.13.el5xen &> /dev/null || : modules=( $(cat /var/run/rpm-kmod-drbd82-xen-modules) ) #rm /var/run/rpm-kmod-drbd82-xen-modules if [ -x "/sbin/weak-modules" ]; then printf '%s\n' "${modules[@]}" | /sbin/weak-modules --remove-modules fi
On Mon, Apr 13, 2009 at 2:09 PM, Jerry Amundson jamundso@gmail.com wrote:
On Mon, Apr 13, 2009 at 4:05 PM, Akemi Yagi amyagi@gmail.com wrote:
[root@xenb ~]# rpm -qa kernel* | sort kernel-headers-2.6.18-128.1.6.el5 kernel-xen-2.6.18-128.1.6.el5 kernel-xen-2.6.18-92.1.10.el5 kernel-xen-2.6.18-92.1.22.el5 [root@xenb ~]# rpm -qa *drbd* | sort drbd82-8.2.6-1.el5.centos kmod-drbd82-xen-8.2.6-1.2.6.18_92.1.10.el5 kmod-drbd82-xen-8.2.6-2 [root@xenb ~]# ls -l `find /lib/modules -name drbd.ko` lrwxrwxrwx 1 root root 54 Apr 13 13:57 /lib/modules/2.6.18-128.1.6.el5xen/weak-updates/drbd82/drbd.ko -> /lib/modules/2.6.18-92.1.6.el5xen/extra/drbd82/drbd.ko
This is not right. kmod-drbd82-xen-8.2.6-2 is supposed to install:
/lib/modules/2.6.18-92.1.13.el5xen/extra/drbd82/drbd.ko
and the symlink should look like:
/lib/modules/2.6.18-128.1.6.el5xen/weak-updates/drbd82/drbd.ko -> /lib/modules/2.6.18-92.1.13.el5xen/extra/drbd82/drbd.ko
Could you show us the output from:
rpm -ql kmod-drbd82-xen
rpm -q --scripts kmod-drbd82-xen
[root@xenb ~]# rpm -ql kmod-drbd82-xen /lib/modules/2.6.18-92.1.10.el5xen/extra/drbd82/drbd.ko /lib/modules/2.6.18-92.1.13.el5xen /lib/modules/2.6.18-92.1.13.el5xen/extra /lib/modules/2.6.18-92.1.13.el5xen/extra/drbd82 /lib/modules/2.6.18-92.1.13.el5xen/extra/drbd82/drbd.ko
I suppose, at some point in time, you had kmod-drbd-xen-8.0.12-1.2.6.18_92.1.6.el5 which installed /lib/modules/2.6.18-92.1.6.el5xen/extra/drbd82/drbd.ko. [expected behavior]
When kmod-drbd82-xen-8.2.6-2 was installed later on, it installed /lib/modules/2.6.18-92.1.13.el5xen/extra/drbd82/drbd.ko and created symlinks through the action of /sbin/weak-modules. Kernels that existed at that moment now point to the drbd.ko in /lib/modules/2.6.18-92.1.13.el5xen. [expected behavior]
When the kernel was updated to -128 (upon 5.3 update), weak-modules was supposed to do the same (create a symlink to 2.6.18-92.1.13.el5xen). But instead, it made a link to 2.6.18-92.1.6.el5xen. This is the part I don't understand. [???]
Can someone figure out what is going on here? Fabian?
Akemi
On Mon, Apr 13, 2009 at 5:08 PM, Akemi Yagi amyagi@gmail.com wrote:
On Mon, Apr 13, 2009 at 2:09 PM, Jerry Amundson jamundso@gmail.com wrote:
On Mon, Apr 13, 2009 at 4:05 PM, Akemi Yagi amyagi@gmail.com wrote:
[root@xenb ~]# rpm -qa kernel* | sort kernel-headers-2.6.18-128.1.6.el5 kernel-xen-2.6.18-128.1.6.el5 kernel-xen-2.6.18-92.1.10.el5 kernel-xen-2.6.18-92.1.22.el5 [root@xenb ~]# rpm -qa *drbd* | sort drbd82-8.2.6-1.el5.centos kmod-drbd82-xen-8.2.6-1.2.6.18_92.1.10.el5 kmod-drbd82-xen-8.2.6-2 [root@xenb ~]# ls -l `find /lib/modules -name drbd.ko` lrwxrwxrwx 1 root root 54 Apr 13 13:57 /lib/modules/2.6.18-128.1.6.el5xen/weak-updates/drbd82/drbd.ko -> /lib/modules/2.6.18-92.1.6.el5xen/extra/drbd82/drbd.ko
This is not right. kmod-drbd82-xen-8.2.6-2 is supposed to install:
/lib/modules/2.6.18-92.1.13.el5xen/extra/drbd82/drbd.ko
and the symlink should look like:
/lib/modules/2.6.18-128.1.6.el5xen/weak-updates/drbd82/drbd.ko -> /lib/modules/2.6.18-92.1.13.el5xen/extra/drbd82/drbd.ko
Could you show us the output from:
rpm -ql kmod-drbd82-xen
rpm -q --scripts kmod-drbd82-xen
[root@xenb ~]# rpm -ql kmod-drbd82-xen /lib/modules/2.6.18-92.1.10.el5xen/extra/drbd82/drbd.ko /lib/modules/2.6.18-92.1.13.el5xen /lib/modules/2.6.18-92.1.13.el5xen/extra /lib/modules/2.6.18-92.1.13.el5xen/extra/drbd82 /lib/modules/2.6.18-92.1.13.el5xen/extra/drbd82/drbd.ko
I suppose, at some point in time, you had kmod-drbd-xen-8.0.12-1.2.6.18_92.1.6.el5 which installed /lib/modules/2.6.18-92.1.6.el5xen/extra/drbd82/drbd.ko. [expected behavior]
When kmod-drbd82-xen-8.2.6-2 was installed later on, it installed /lib/modules/2.6.18-92.1.13.el5xen/extra/drbd82/drbd.ko and created symlinks through the action of /sbin/weak-modules. Kernels that existed at that moment now point to the drbd.ko in /lib/modules/2.6.18-92.1.13.el5xen. [expected behavior]
Yes, I'm with you so far.
When the kernel was updated to -128 (upon 5.3 update), weak-modules was supposed to do the same (create a symlink to 2.6.18-92.1.13.el5xen). But instead, it made a link to 2.6.18-92.1.6.el5xen. This is the part I don't understand. [???]
It seems related to installation order. With the kernel not yet installed, weak-modules has nothing to pick up in /lib/modules for the new kernel...
[root@xenb ~]# rpm -qa --last | grep xen xen-3.0.3-80.el5_3.2 Mon 13 Apr 2009 01:58:11 PM CDT kernel-xen-2.6.18-128.1.6.el5 Mon 13 Apr 2009 01:57:51 PM CDT xen-libs-3.0.3-80.el5_3.2 Mon 13 Apr 2009 01:57:26 PM CDT kmod-drbd82-xen-8.2.6-2 Thu 19 Mar 2009 02:41:55 PM CDT kernel-xen-2.6.18-92.1.22.el5 Thu 19 Mar 2009 02:41:39 PM CDT kmod-drbd82-xen-8.2.6-1.2.6.18_92.1.10.el5 Fri 15 Aug 2008 03:19:31 PM CDT kernel-xen-2.6.18-92.1.10.el5 Fri 15 Aug 2008 03:19:09 PM CDT
jerry
On Tue, Apr 14, 2009 at 6:20 AM, Jerry Amundson jamundso@gmail.com wrote:
On Mon, Apr 13, 2009 at 5:08 PM, Akemi Yagi amyagi@gmail.com wrote:
I suppose, at some point in time, you had kmod-drbd-xen-8.0.12-1.2.6.18_92.1.6.el5 which installed /lib/modules/2.6.18-92.1.6.el5xen/extra/drbd82/drbd.ko. [expected behavior]
When kmod-drbd82-xen-8.2.6-2 was installed later on, it installed /lib/modules/2.6.18-92.1.13.el5xen/extra/drbd82/drbd.ko and created symlinks through the action of /sbin/weak-modules. Kernels that existed at that moment now point to the drbd.ko in /lib/modules/2.6.18-92.1.13.el5xen. [expected behavior]
Yes, I'm with you so far.
When the kernel was updated to -128 (upon 5.3 update), weak-modules was supposed to do the same (create a symlink to 2.6.18-92.1.13.el5xen). But instead, it made a link to 2.6.18-92.1.6.el5xen. This is the part I don't understand. [???]
It seems related to installation order. With the kernel not yet installed, weak-modules has nothing to pick up in /lib/modules for the new kernel...
[root@xenb ~]# rpm -qa --last | grep xen xen-3.0.3-80.el5_3.2 Mon 13 Apr 2009 01:58:11 PM CDT kernel-xen-2.6.18-128.1.6.el5 Mon 13 Apr 2009 01:57:51 PM CDT xen-libs-3.0.3-80.el5_3.2 Mon 13 Apr 2009 01:57:26 PM CDT kmod-drbd82-xen-8.2.6-2 Thu 19 Mar 2009 02:41:55 PM CDT kernel-xen-2.6.18-92.1.22.el5 Thu 19 Mar 2009 02:41:39 PM CDT kmod-drbd82-xen-8.2.6-1.2.6.18_92.1.10.el5 Fri 15 Aug 2008 03:19:31 PM CDT kernel-xen-2.6.18-92.1.10.el5 Fri 15 Aug 2008 03:19:09 PM CDT
Yes, that is the normal order. /sbin/weak-module is run during the kernel installation as you can see in the script (some messages ago). It surveys installed modules (not kernel built-in) and sets up symlinks for those that are kABI compatible.
When kernel-xen-2.6.18-128.1.6 was installed on your system, it was supposed to make a link to the drbd.ko in /lib/modules/2.6.18-92.1.13.el5xen which already existed at that time. But it didn't. Very strange.
Akemi
-
Akemi Yagi -
Curt Mills -
Fabian Arrotin -
Jerry Amundson