Howdy,

I've got a CentOS 7 VM that occasionally becomes unresponsive. There's a "list_add corruption" entry in /var/log/messages, included below.

This is on VMware. We have lots of other VMs which are running just fine. Only a few are CentOS 7 VMs, though, so I can't rule out some kind of environment issue.

It's up-to-date on OS patches, and running kernel 3.10.0-514.26.2.el7.x86_64. When it locks up, there isn't any indication on its console, other than it looks like it's waiting at a login prompt. I did some Googling, which suggested similar issues have been around a while, but the results I found tended to be for CentOS 6 and older kernels.

Can anyone help me understand what this means?

Jul 31 16:58:15 den-nagios kernel: ------------[ cut here ]------------ Jul 31 16:58:15 den-nagios kernel: WARNING: at lib/list_debug.c:33 __list_add+0xac/0xc0() Jul 31 16:58:15 den-nagios kernel: list_add corruption. prev->next should be next (ffff880174795358), but was 64756f6c6373646e. (prev=ffff880806a44458). Jul 31 16:58:15 den-nagios kernel: Modules linked in: dell_rbu dcdbas ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_ mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock vfat fat intel_power clamp coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon sg pcspkr vmw_vmci shpchp i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_gen eric Jul 31 16:58:15 den-nagios kernel: vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul crct10dif_common ttm crc32c_intel ata_piix drm serio_raw libata vmxnet3 vmw_pvscsi i2c_core fjes dm_mirror dm_region_hash dm_log dm_mod Jul 31 16:58:15 den-nagios kernel: CPU: 6 PID: 127439 Comm: ssh Not tainted 3.10.0-514.26.2.el7.x86_64 #1 Jul 31 16:58:15 den-nagios kernel: Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.0.B64.1506250318 06/25/2015 Jul 31 16:58:15 den-nagios kernel: ffff880634f83c00 00000000ac63a691 ffff880634f83bb8 ffffffff81687133 Jul 31 16:58:15 den-nagios kernel: ffff880634f83bf0 ffffffff81085cb0 ffff880782e74210 ffff880174795358 Jul 31 16:58:15 den-nagios kernel: ffff880806a44458 ffff880174794000 0000000000000000 ffff880634f83c58 Jul 31 16:58:15 den-nagios kernel: Call Trace: Jul 31 16:58:15 den-nagios kernel: [<ffffffff81687133>] dump_stack+0x19/0x1b Jul 31 16:58:15 den-nagios kernel: [<ffffffff81085cb0>] warn_slowpath_common+0x70/0xb0 Jul 31 16:58:15 den-nagios kernel: [<ffffffff81085d4c>] warn_slowpath_fmt+0x5c/0x80 Jul 31 16:58:15 den-nagios kernel: [<ffffffff815b6410>] ? ip_fragment.constprop.54+0x90/0x90 Jul 31 16:58:15 den-nagios kernel: [<ffffffff8133399c>] __list_add+0xac/0xc0 Jul 31 16:58:15 den-nagios kernel: [<ffffffff81096c6b>] __internal_add_timer+0xab/0x130 Jul 31 16:58:15 den-nagios kernel: [<ffffffff81096d22>] internal_add_timer+0x32/0x70 Jul 31 16:58:15 den-nagios kernel: [<ffffffff810992ed>] mod_timer+0x13d/0x220 Jul 31 16:58:15 den-nagios kernel: [<ffffffff815591a8>] sk_reset_timer+0x18/0x30 Jul 31 16:58:15 den-nagios kernel: [<ffffffff815d0cae>] tcp_connect+0x74e/0x9f0 Jul 31 16:58:15 den-nagios kernel: [<ffffffff810eb045>] ? ktime_get_real+0x25/0x70 Jul 31 16:58:15 den-nagios kernel: [<ffffffff81568f99>] ? secure_tcp_sequence_number+0x69/0x90 Jul 31 16:58:15 den-nagios kernel: [<ffffffff815d5886>] tcp_v4_connect+0x376/0x4e0 Jul 31 16:58:15 den-nagios kernel: [<ffffffff815ec685>] __inet_stream_connect+0xb5/0x330 Jul 31 16:58:15 den-nagios kernel: [<ffffffff811b1461>] ? handle_mm_fault+0x6b1/0x1000 Jul 31 16:58:15 den-nagios kernel: [<ffffffff815ec938>] inet_stream_connect+0x38/0x50 Jul 31 16:58:15 den-nagios kernel: [<ffffffff81555f07>] SYSC_connect+0xe7/0x120 Jul 31 16:58:15 den-nagios kernel: [<ffffffff8121d72d>] ? set_close_on_exec+0x4d/0x70 Jul 31 16:58:15 den-nagios kernel: [<ffffffff81556d0e>] SyS_connect+0xe/0x10 Jul 31 16:58:15 den-nagios kernel: [<ffffffff81697809>] system_call_fastpath+0x16/0x1b Jul 31 16:58:15 den-nagios kernel: ---[ end trace 64494d91939a3eb0 ]---

Thanks, -Chris