Hi Alexey,
On 12/4/06, Alexey Loukianov aloukianov@lavtech.ru wrote:
Hello all,
a couple of weeks ago I've been installing CentOS 4.2 on a very-old server machine with MSI server board based on Intel GX440 chipset, two Xeons 500Mhz and one 1Gig of RAM. There is an AMI MegaRaid 467 installed as a storage controller, which causes some troubles with installation, as stock CenOS4 install and production kernels doesn't have older megaraid.ko module compiled, but there are a lot of not so very difficult ways to overcome it. After a clean-and-relatively-fast install first of all I had up2date-d it to CentOS 4.4, did some basic initial reconfigurations, turned it off and left it lay around doing nothing.
Today its time has come, and I turned it on. While booting I've seen a message "Segmentation fault" just after a line about "Starting up LVM2". That's confused me a bit. Logged it as root, typed vgdisplay, got a normal output and a message "Segmentation fault" after it. lvdisplay performed just the same - normal display of all LVM logical volumes and a "Segmentation fault" at the bottom.
Next step was obvious: # rpm -Va
Huh, here we are. There's a bunch of RPMs with binary files in them changed since they've been installed. Just looks like it's a virus job I thinked. But, wait! That's very strange! This server has been laying around turned off and doing nothing since the moment I've done the installation of the system. There were NO possible time for a virus to infect a system. Well, in any case, I took on my special LiveCD with a ClamAV on it and a RamDisk for freshclam to store updated virus databases, booted it, mounted possibly infected system and checked it with clamscan. There were NO viruses found.
Well, I though that this might be caused by a faulty SCSI disk in array, that distort the data that's being written to it, instead of informing host that there's a bad block here. Ok, that's easy to check. Let me go to the single mode, reinstall distorted RPMs using rpm -Uvh --replacepkgs, do a couple of 'sync's, remount all filesystems with -O ro,sync, and check installed rpm's with a rpm -Va. Headed on, done all above, got nothing. After a reinstall all files became correct, and LVM tools got back to a correct behavior without "Segmentation fault". Hmm... that's strange, I thought. Well, at least ATM I've got a correctly functioning system without viruses. Huh, well, now it's time to reboot and check how does it performs. I'm going to do unattended reboots in future, it should reboot seamlessly without excess questions. # shutdown -r now Reboot went smoothly, but just as LVM2 was initializing, I've got "Segmentation fault" message again! Damn! What's wrong?! Logged in, rpm -Va - gotcha! Again, device-mapper RPM was broken. Well, let's reinstall it again, sync, remount root readonly, check with rpm -V device-mapper. Done that - all seems to be ok, no output from the rpm -V = files are intact. Rebooted again. Run: [root@omega MegaMgr5.20]# rpm -V device-mapper ..5..... /lib/libdevmapper.so.1.02
That's it. After each and every reboot I've got this file corrupted. Looks like it's not a faulty HDD trouble, and it's not a faulty RAID controller. Most likely something corrupts this file during shutdown process or during boot process. Haven't got enough time today to investigate more deeply, going to continue with it tomorrow. Will post here the results, if any.
-- Best regards, Alexey Loukianov mailto:aloukianov@lavtech.ru System Engineer, IT Department, Lavtech Corp
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[root@omega MegaMgr5.20]# rpm -V device-mapper ..5..... /lib/libdevmapper.so.1.02
Is that the only file consistently corrupted or are there others? I've seen similar "mysteries" before that turned out to be a memory issue, once system memory, once CPU cache (that was really weird). I'm not sure this is a memory problem, but wouldn't hurt to run a memory test.
Nick