[CentOS] Strange situation with openssl and kernel

Thu May 3 07:55:12 UTC 2012
Nikolaos Milas <nmilas at noa.gr>

Hello,

On a VM guest (running under KVM) with CentOS 5.8, I yesterday strangely 
got the following (see below).

The server runs apache/php/squirrelmail/postfix/dovecot/openldap and it 
is a production server.

Do you advise to do some particular file system checks or other 
settings? Any other advice?

Such errors have not appeared in the past.

Thanks,
Nick

System info:

=============================================================

[root at vmail log]# parted -l
Error: Unable to open /dev/md0 - unrecognised disk label.


Model: Unknown (unknown)
Disk /dev/vda: 623GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
  1      32.3kB  107MB   107MB   primary  ext3         boot
  2      107MB   85.9GB  85.8GB  primary               lvm
  3      85.9GB  623GB   537GB   primary               lvm


[root at vmail log]#
[root at vmail log]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       560G   72G  460G  14% /
/dev/vda1              99M   20M   75M  21% /boot
tmpfs                 1.5G     0  1.5G   0% /dev/shm
[root at vmail log]#
[root at vmail log]#
[root at vmail log]# uname -a
Linux vmail.noa.gr 2.6.18-308.4.1.el5 #1 SMP Tue Apr 17 17:08:00 EDT 
2012 x86_64 x86_64 x86_64 GNU/Linux
[root at vmail log]#
[root at vmail log]#
[root at vmail log]# vgdisplay
   --- Volume group ---
   VG Name               VolGroup00
   System ID
   Format                lvm2
   Metadata Areas        2
   Metadata Sequence No  5
   VG Access             read/write
   VG Status             resizable
   MAX LV                0
   Cur LV                2
   Open LV               2
   Max PV                0
   Cur PV                2
   Act PV                2
   VG Size               579.84 GB
   PE Size               32.00 MB
   Total PE              18555
   Alloc PE / Size       18555 / 579.84 GB
   Free  PE / Size       0 / 0
   VG UUID               fDg7W8-xe04-7OpQ-tFLe-Sioh-G2jb-dOuLGa

[root at vmail log]# lvdisplay
   --- Logical volume ---
   LV Name                /dev/VolGroup00/LogVol00
   VG Name                VolGroup00
   LV UUID                2Q26xO-HIdJ-UXAR-0zAM-yYFq-ysqu-bEs8J8
   LV Write Access        read/write
   LV Status              available
   # open                 1
   LV Size                577.88 GB
   Current LE             18492
   Segments               2
   Allocation             inherit
   Read ahead sectors     auto
   - currently set to     256
   Block device           252:0

   --- Logical volume ---
   LV Name                /dev/VolGroup00/LogVol01
   VG Name                VolGroup00
   LV UUID                Ceb2Qe-tnNk-te6R-cxdm-p0cK-SieL-0mqhHW
   LV Write Access        read/write
   LV Status              available
   # open                 1
   LV Size                1.97 GB
   Current LE             63
   Segments               1
   Allocation             inherit
   Read ahead sectors     auto
   - currently set to     256
   Block device           252:1

=============================================================

/var/log/messages:

May  2 20:48:24 vmail nrpe[25864]: Error: Could not complete SSL 
handshake. 5
May  2 20:48:24 vmail nrpe[25866]: Error: Could not complete SSL 
handshake. 5
May  2 21:00:31 vmail nrpe[26106]: Error: Could not complete SSL 
handshake. 5
May  2 21:20:34 vmail nrpe[26666]: Error: Could not complete SSL 
handshake. 5
May  2 21:21:32 vmail nrpe[26686]: Could not read request from client, 
bailing out...
May  2 21:21:32 vmail nrpe[26686]: INFO: SSL Socket Shutdown.
May  2 21:43:11 vmail kernel: INFO: task fail2ban-server:2589 blocked 
for more than 120 seconds.
May  2 21:45:32 vmail kernel: "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
May  2 21:45:32 vmail kernel: fail2ban-serv D 0000000000000000     0  
2589      1          2590  2585 (NOTLB)
May  2 21:47:25 vmail kernel:  ffff8100a6625ea8 0000000000000082 
00002aaaaacbd000 0000000000001000
May  2 21:47:53 vmail kernel:  ffffff9ca6854280 0000000000000009 
ffff8100a65f8040 ffff8100581f20c0
May  2 21:47:54 vmail kernel:  000017f236120882 0000000000000465 
ffff8100a65f8228 0000000000000000
May  2 21:47:54 vmail kernel: Call Trace:
May  2 21:47:55 vmail kernel:  [<ffffffff80063c63>] 
__mutex_lock_slowpath+0x60/0x9b
May  2 21:47:55 vmail kernel:  [<ffffffff80063cad>] 
.text.lock.mutex+0xf/0x14
May  2 21:47:55 vmail kernel:  [<ffffffff80013ed6>] 
generic_file_llseek+0x2a/0x8b
May  2 21:47:55 vmail kernel:  [<ffffffff8002563b>] sys_lseek+0x40/0x60
May  2 21:47:58 vmail kernel:  [<ffffffff8005d229>] tracesys+0x71/0xe0
May  2 21:47:58 vmail kernel:  [<ffffffff8005d28d>] tracesys+0xd5/0xe0
May  2 21:47:58 vmail kernel:
May  2 22:42:53 vmail nrpe[29199]: Could not read request from client, 
bailing out...
May  2 22:42:53 vmail nrpe[29199]: INFO: SSL Socket Shutdown.
May  2 22:42:53 vmail nrpe[29201]: Could not read request from client, 
bailing out...
May  2 22:42:53 vmail nrpe[29201]: INFO: SSL Socket Shutdown.
May  2 22:42:53 vmail nrpe[29203]: Could not read request from client, 
bailing out...
May  2 22:42:53 vmail nrpe[29203]: INFO: SSL Socket Shutdown.
May  2 22:48:14 vmail kernel: INFO: task kjournald:382 blocked for more 
than 120 seconds.
May  2 22:48:20 vmail kernel: "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
May  2 22:48:20 vmail kernel: kjournald     D ffff8100b78fc0c0     0   
382     31           407   371 (L-TLB)
May  2 22:48:20 vmail kernel:  ffff8100b8dbbcf0 0000000000000046 
ffff8100bff965c0 ffff8100a732c940
May  2 22:48:20 vmail kernel:  ffff810002d941a0 000000000000000a 
ffff810037f5f820 ffff8100b78fc0c0
May  2 22:48:20 vmail kernel:  00001b85cd09baa4 0000000000001325 
ffff810037f5fa08 000000008006e665
May  2 22:48:20 vmail kernel: Call Trace:
May  2 22:48:20 vmail kernel:  [<ffffffff8006ecd9>] 
do_gettimeofday+0x40/0x90
May  2 22:48:20 vmail kernel:  [<ffffffff8005a412>] getnstimeofday+0x10/0x29
May  2 22:48:20 vmail kernel:  [<ffffffff8001558e>] sync_buffer+0x0/0x3f
May  2 22:48:20 vmail kernel:  [<ffffffff800637de>] io_schedule+0x3f/0x67
May  2 22:48:20 vmail kernel:  [<ffffffff800155c9>] sync_buffer+0x3b/0x3f
May  2 22:48:20 vmail kernel:  [<ffffffff80063a0a>] __wait_on_bit+0x40/0x6e
May  2 22:48:20 vmail kernel:  [<ffffffff8001558e>] sync_buffer+0x0/0x3f
May  2 22:48:20 vmail kernel:  [<ffffffff80063aa4>] 
out_of_line_wait_on_bit+0x6c/0x78
May  2 22:48:20 vmail kernel:  [<ffffffff800a34d5>] 
wake_bit_function+0x0/0x23
May  2 22:48:20 vmail kernel:  [<ffffffff88033a47>] 
:jbd:journal_commit_transaction+0x559/0x10c2
May  2 22:48:20 vmail kernel:  [<ffffffff8003d7be>] 
lock_timer_base+0x1b/0x3c
May  2 22:48:20 vmail kernel:  [<ffffffff8004ad59>] 
try_to_del_timer_sync+0x7f/0x88
May  2 22:48:20 vmail kernel:  [<ffffffff880376a3>] 
:jbd:kjournald+0xc1/0x213
May  2 22:48:20 vmail kernel:  [<ffffffff800a34a7>] 
autoremove_wake_function+0x0/0x2e
May  2 22:48:20 vmail kernel:  [<ffffffff800a328f>] 
keventd_create_kthread+0x0/0xc4
May  2 22:48:20 vmail kernel:  [<ffffffff880375e2>] :jbd:kjournald+0x0/0x213
May  2 22:48:20 vmail kernel:  [<ffffffff800a328f>] 
keventd_create_kthread+0x0/0xc4
May  2 22:48:20 vmail kernel:  [<ffffffff80032632>] kthread+0xfe/0x132
May  2 22:48:20 vmail kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
May  2 22:48:20 vmail kernel:  [<ffffffff800a328f>] 
keventd_create_kthread+0x0/0xc4
May  2 22:48:20 vmail kernel:  [<ffffffff80032534>] kthread+0x0/0x132
May  2 22:48:20 vmail kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
May  2 22:48:20 vmail kernel:
May  2 22:48:20 vmail kernel: INFO: task httpd:2775 blocked for more 
than 120 seconds.
May  2 22:48:20 vmail kernel: "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
May  2 22:48:20 vmail kernel: httpd         D ffff8100a9323040     0  
2775   2541          2776  2774 (NOTLB)
May  2 22:48:20 vmail kernel:  ffff8100a4b19aa8 0000000000000082 
ffff8100b999e9d0 ffff8100a4b19b38
May  2 22:48:20 vmail kernel:  7ffffffffffffcd8 000000000000000a 
ffff8100a65837a0 ffff8100a9323040
May  2 22:48:20 vmail kernel:  00001b85cd33ccf5 00000000000354bc 
ffff8100a6583988 000000008006e665
May  2 22:48:20 vmail kernel: Call Trace:
May  2 22:48:20 vmail kernel:  [<ffffffff8006ecd9>] 
do_gettimeofday+0x40/0x90
May  2 22:48:20 vmail kernel:  [<ffffffff80062ffd>] thread_return+0x5d/0xfe
May  2 22:48:20 vmail kernel:  [<ffffffff8005a412>] getnstimeofday+0x10/0x29
May  2 22:48:20 vmail kernel:  [<ffffffff8001558e>] sync_buffer+0x0/0x3f
May  2 22:48:20 vmail kernel:  [<ffffffff800637de>] io_schedule+0x3f/0x67
May  2 22:48:20 vmail kernel:  [<ffffffff800155c9>] sync_buffer+0x3b/0x3f
May  2 22:48:20 vmail kernel:  [<ffffffff80063922>] 
__wait_on_bit_lock+0x36/0x66
May  2 22:48:20 vmail kernel:  [<ffffffff8001558e>] sync_buffer+0x0/0x3f
May  2 22:48:20 vmail kernel:  [<ffffffff800639be>] 
out_of_line_wait_on_bit_lock+0x6c/0x78
May  2 22:48:20 vmail kernel:  [<ffffffff800a34d5>] 
wake_bit_function+0x0/0x23
May  2 22:48:20 vmail kernel:  [<ffffffff8803181e>] 
:jbd:do_get_write_access+0x54/0x522
May  2 22:48:20 vmail kernel:  [<ffffffff80019d4f>] __getblk+0x25/0x22c
May  2 22:48:20 vmail kernel:  [<ffffffff88031d0e>] 
:jbd:journal_get_write_access+0x22/0x33
May  2 22:48:20 vmail kernel:  [<ffffffff8804dd3b>] 
:ext3:ext3_reserve_inode_write+0x38/0x90
May  2 22:48:20 vmail kernel:  [<ffffffff8804ddb4>] 
:ext3:ext3_mark_inode_dirty+0x21/0x3c
May  2 22:48:20 vmail kernel:  [<ffffffff88050d39>] 
:ext3:ext3_dirty_inode+0x63/0x7b
May  2 22:48:20 vmail kernel:  [<ffffffff80013d53>] 
__mark_inode_dirty+0x29/0x16e
May  2 22:48:20 vmail kernel:  [<ffffffff80016642>] 
__generic_file_aio_write_nolock+0x28a/0x3b6
May  2 22:48:20 vmail kernel:  [<ffffffff80021872>] 
generic_file_aio_write+0x67/0xc3
May  2 22:48:20 vmail kernel:  [<ffffffff8804c1c2>] 
:ext3:ext3_file_write+0x16/0x91
May  2 22:48:20 vmail kernel:  [<ffffffff800183ac>] do_sync_write+0xc7/0x104
May  2 22:48:20 vmail kernel:  [<ffffffff800e403c>] 
do_readv_writev+0x26e/0x291
May  2 22:48:20 vmail kernel:  [<ffffffff800a34a7>] 
autoremove_wake_function+0x0/0x2e
May  2 22:48:20 vmail kernel:  [<ffffffff80033902>] do_setitimer+0x5b4/0x689
May  2 22:48:20 vmail kernel:  [<ffffffff80016b29>] vfs_write+0xce/0x174
May  2 22:48:20 vmail kernel:  [<ffffffff800173f2>] sys_write+0x45/0x6e
May  2 22:48:20 vmail kernel:  [<ffffffff8005d28d>] tracesys+0xd5/0xe0
May  2 22:48:20 vmail kernel:
May  2 22:51:10 vmail nrpe[29461]: Error: Could not complete SSL 
handshake. 5
May  2 23:46:10 vmail nrpe[31317]: Error: Could not complete SSL 
handshake. 5
May  3 00:08:50 vmail nrpe[32073]: Error: Could not complete SSL 
handshake. 5
May  3 00:08:50 vmail nrpe[32069]: Error: Could not complete SSL 
handshake. 5

=============================================================