Paul (Crunch) wrote:
> On 04/04/2012 12:31 PM, Nataraj wrote:
>> On 04/04/2012 09:16 AM, Jonathan Alstead wrote:
>>> Hello,
>>>
>>> Recently our dell sc1425 server has been locking up with kernel freezes
>>> and required a hard reboot on each occasion. I've looked on the centos
>>> forums with limited success - each problem seems slightly different
>>> (some failure on high load, some not). Our kernel is
>>> 2.6.18-274.17.1.el5
>>> and /var/log/messages show the following errors:
>>>
>>> Apr 3 12:41:25 sp2 kernel: INFO: task mysqld:15345 blocked for more
>>> than 120 seconds.
<snip>
>>> Apr 3 12:41:25 sp2 kernel: Call Trace:
>>> Apr 3 12:41:25 sp2 kernel: [<c0622f16>]
>>> rwsem_down_write_failed+0x126/0x141
>>> Apr 3 12:41:25 sp2 kernel: [<c0439989>] .text.lock.rwsem+0x2b/0x3a
>>> Apr 3 12:41:25 sp2 kernel: [<c046aa6a>] sys_mprotect+0xbd/0x1eb
>>>
>>> Apr 3 12:41:25 sp2 kernel: [<c0404f4b>] syscall_call+0x7/0xb
>>>
>>> Apr 3 12:41:25 sp2 kernel: =======================
>>> Apr 3 12:41:25 sp2 kernel: INFO: task clamd:15721 blocked for more
>>> than
>>> 120 seconds.
<snip>
>>> Apr 3 12:41:26 sp2 kernel: Call Trace:
>>> Apr 3 12:41:26 sp2 kernel: [<c041f863>] default_wake_function+0x0/0xc
>>> Apr 3 12:41:26 sp2 kernel: [<c048e994>] destroy_inode+0x38/0x47
>>> Apr 3 12:41:26 sp2 kernel: [<c0622f16>]
>>> rwsem_down_write_failed+0x126/0x141
>>> Apr 3 12:41:26 sp2 kernel: [<c0439989>] .text.lock.rwsem+0x2b/0x3a
>>> Apr 3 12:41:26 sp2 kernel: [<c046a32b>] sys_munmap+0x24/0x41
>>>
>>> Apr 3 12:41:26 sp2 kernel: [<c0404f4b>] syscall_call+0x7/0xb
<snip>
Looking at the stack traces, and that two completely separate processes
are being blocked at the same time, I have to suggest another possibility:
the drive that /var is on may be having problems... and if it can't be
written to, then it can't log errors, either.
mark