[CentOS] Cause for kernel panic

[Ross Walker]

On Dec 14, 2011, at 10:49 PM, Nicolas Ross <rossnick-lists at cybercat.ca> wrote:

> Hi ! On an 8-node cluster, one of the node did a kernel panic.
> 
> The only bit of information I have is on a ssh console I had open, which 
> said :
> 
> 
> Message from syslogd at node108 at Dec 14 19:00:15 ...
>  kernel:------------[ cut here ]------------
> 
> Message from syslogd at node108 at Dec 14 19:00:15 ...
>  kernel:invalid opcode: 0000 [#1] SMP
> 
> Message from syslogd at node108 at Dec 14 19:00:15 ...
>  kernel:last sysfs file: 
> /sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map
> 
> Message from syslogd at node108 at Dec 14 19:00:15 ...
>  kernel:Stack:
> 
> Message from syslogd at node108 at Dec 14 19:00:15 ...
>  kernel:Call Trace:
> 
> Message from syslogd at node108 at Dec 14 19:00:15 ...
>  kernel:Code: 01 00 00 e8 26 8a cd e0 85 c0 0f 85 0e ff ff ff 48 89 df 
> e8 76 f8 ff ff e9 01 ff ff ff 31 d2 eb d4 48 89 de 31 ff e8 c3 e3 ff ff 
> <0f> 0b eb fe 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48
> 
> Message from syslogd at node108 at Dec 14 19:00:15 ...
>  kernel:Kernel panic - not syncing: Fatal exception
> 
> 
> From this, is there a way to determine the cause ? kdump is not 
> confirgured nor used, since the fencing of the node renders kdump useless.
> 
> This is the second time in a few weeks it happens.

Setup netconsole to log kernel messages to the node on the "left". Then you can get the the oops messages if any node crashes.

-Ross