Hello CentOS Xen Heroes, Yesterday, I have installed testing kernel 4.9.23-26.el6.x86_64 from virt-xen-testing repo. It crashed today morning. Hardware is a pretty ancient, testing machine (CO6 PV guests only), but had no problems yet. It was stable on 4.9*, including testing 4.9.15-22.el6.x86_64 Console output: [59826.069427] general protection fault: 0000 [#1] SMP [59826.069463] Modules linked in: xt_physdev br_netfilter xt_mac ebt_arp xen_pciback xen_gntalloc ebtable_filter ebtables bridge 8021q mrp garp stp llc xt_CT xt_addrtype iptable_raw nf_log_ ipv4 nf_log_common xt_LOG xt_limit xt_owner iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_sta te nf_conntrack ip6table_filter ip6_tables blktap xen_netback xen_blkback xen_gntdev xen_evtchn xenfs xen_privcmd ppdev parport_pc parport fjes 3c59x asus_atk0110 pcspkr serio_raw k8temp vi a_rhine mii i2c_viapro shpchp raid1 pata_via pata_acpi ata_generic sata_via [59826.069842] CPU: 0 PID: 32443 Comm: sshd Not tainted 4.9.23-26.el6.x86_64 #1 [59826.069860] Hardware name: System manufacturer System Product Name/K8V-VM, BIOS 0201 06/13/2006 [59826.069882] task: ffff880014c8c900 task.stack: ffffc90043820000 [59826.069896] RIP: e030:[<ffffffff812405d6>] [<ffffffff812405d6>] kmem_cache_alloc_node+0xa6/0x1c0 [59826.070019] RSP: e02b:ffffc90043823b08 EFLAGS: 00010246 [59826.070019] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000ec741e [59826.070019] RDX: 0000000000ec741d RSI: 00000000024000c0 RDI: ffff0000ffff0000 [59826.070019] RBP: ffffc90043823b68 R08: 000000000001cd00 R09: 0000000000000000 [59826.070019] R10: 0000000000007ff0 R11: ffffc900407ebe88 R12: ffff88001a8013c0 [59826.070019] R13: ffff88001a8013c0 R14: 00000000ffffffff R15: 00000000024000c0 [59826.070019] FS: 00007f652071e7c0(0000) GS:ffff88001f800000(0000) knlGS:0000000000000000 [59826.070019] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 [59826.070019] CR2: 00007fff0870ad60 CR3: 000000000f940000 CR4: 0000000000000660 [59826.070019] Stack: [59826.070019] ffffc90043823b38 ffffffff818d5e36 ffff880016711758 ffffffff81792687 [59826.070019] ffff880015d1ec10 ffff880015d1ec00 ffffc90043823c38 ffff880015d1ec00 [59826.070019] 00000000024000c0 00000000ffffffff ffff88001a8013c0 0000000000000024 [59826.070019] Call Trace: [59826.070019] [<ffffffff818d5e36>] ? mutex_lock+0x16/0x40 [59826.070019] [<ffffffff81792687>] ? __alloc_skb+0x57/0x200 [59826.070019] [<ffffffff81792687>] __alloc_skb+0x57/0x200 [59826.070019] [<ffffffff817e2af0>] ? netlink_attachskb+0x1b0/0x1b0 [59826.070019] [<ffffffff817df900>] ? netlink_overrun+0x50/0x50 [59826.070019] [<ffffffff817e319c>] netlink_ack+0x6c/0x170 [59826.070019] [<ffffffff8115526b>] audit_receive+0x8b/0xa0 [59826.070019] [<ffffffff817e30aa>] netlink_unicast+0x19a/0x220 [59826.070019] [<ffffffff817e3ed2>] netlink_sendmsg+0x302/0x380 [59826.070019] [<ffffffff81787ba9>] sock_sendmsg+0x49/0x60 [59826.070019] [<ffffffff8178a006>] SYSC_sendto+0x116/0x170 [59826.070019] [<ffffffff81287efc>] ? __alloc_fd+0x4c/0x1b0 [59826.070019] [<ffffffff810033c1>] ? syscall_trace_enter+0x201/0x2c0 [59826.070019] [<ffffffff8115b199>] ? __audit_syscall_exit+0x229/0x2b0 [59826.070019] [<ffffffff8178a06e>] SyS_sendto+0xe/0x10 [59826.070019] [<ffffffff81003e7a>] do_syscall_64+0x7a/0x240 [59826.070019] [<ffffffff8106f1a7>] ? do_page_fault+0x37/0x90 [59826.070019] [<ffffffff818d83eb>] entry_SYSCALL64_slow_path+0x25/0x25 [59826.070019] Code: e8 36 41 39 c6 74 17 48 8b 4d b8 44 89 f2 44 89 fe 4c 89 e7 e8 2c f7 ff ff 48 89 c3 eb 2c 49 63 44 24 20 4d 8b 04 24 48 8d 4a 01 <48> 8b 1c 07 48 89 f8 49 8d 30 e8 eb b 4 1e 00 3c 01 75 95 49 63 [59826.070019] RIP [<ffffffff812405d6>] kmem_cache_alloc_node+0xa6/0x1c0 [59826.070019] RSP <ffffc90043823b08> [59826.076823] general protection fault: 0000 [#2] SMP [59826.077009] Modules linked in: xt_physdev br_netfilter xt_mac ebt_arp xen_pciback xen_gntalloc ebtable_filter ebtables bridge 8021q mrp garp stp llc xt_CT xt_addrtype iptable_raw nf_log_ ipv4 nf_log_common xt_LOG xt_limit xt_owner iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_sta te nf_conntrack ip6table_filter ip6_tables blktap xen_netback xen_blkback xen_gntdev xen_evtchn xenfs xen_privcmd ppdev parport_pc parport fjes 3c59x asus_atk0110 pcspkr serio_raw k8temp vi a_rhine mii i2c_viapro shpchp raid1 pata_via pata_acpi ata_generic sata_via [59826.077733] CPU: 0 PID: 32443 Comm: sshd Tainted: G D 4.9.23-26.el6.x86_64 #1 [59826.077733] Hardware name: System manufacturer System Product Name/K8V-VM, BIOS 0201 06/13/2006 [59826.077733] task: ffff880014c8c900 task.stack: ffffc90043820000 [59826.077733] RIP: e030:[<ffffffff81241ad5>] [<ffffffff81241ad5>] kmem_cache_alloc+0x85/0x1b0 [59826.077733] RSP: e02b:ffff88001f803ba8 EFLAGS: 00010286 [59826.077733] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000ec741e [59826.077733] RDX: 0000000000ec741d RSI: 0000000002080020 RDI: 000000000001cd00 [59826.077733] RBP: ffff88001f803bf8 R08: ffff88001f81cd00 R09: ffff0000ffff0000 [59826.077733] R10: 0000000000000003 R11: 0000000000000000 R12: ffff88001a8013c0 [59826.077733] R13: ffff88001a8013c0 R14: ffffffff8179240e R15: 0000000002080020 [59826.077733] FS: 00007f652071e7c0(0000) GS:ffff88001f800000(0000) knlGS:0000000000000000 [59826.077733] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 [59826.077733] CR2: 00007fff0870ad60 CR3: 000000000f940000 CR4: 0000000000000660 [59826.077733] Stack: [59826.077733] ffff88001f803bb8 ffffffff81025105 ffff88001f803bc8 ffff88001f816760 [59826.077733] 0000000000000780 0000000000000780 ffff8800144af880 0000000000000000 [59826.077733] 0000000000000200 ffff8800144af880 ffff88001f803c38 ffffffff8179240e [59826.077733] Call Trace: [59826.077733] <IRQ> [59826.077733] [<ffffffff81025105>] ? xen_clocksource_read+0x15/0x20 [59826.077733] [<ffffffff8179240e>] __build_skb+0x2e/0xe0 [59826.077733] [<ffffffff817929fa>] __netdev_alloc_skb+0xda/0x120 [59826.077733] [<ffffffff810d0a5a>] ? activate_task+0x7a/0x90 [59826.077733] [<ffffffffc0027368>] rhine_rx+0x2b8/0x530 [via_rhine] [59826.077733] [<ffffffffc00276be>] rhine_napipoll+0xde/0x190 [via_rhine] [59826.077733] [<ffffffff817abfdb>] napi_poll+0xdb/0x1f0 [59826.077733] [<ffffffff81119aae>] ? process_timeout+0xe/0x10 [59826.077733] [<ffffffff81119732>] ? call_timer_fn+0x42/0x140 [59826.077733] [<ffffffff817ac1b5>] net_rx_action+0xc5/0x240 [59826.077733] [<ffffffff8111b1d6>] ? run_timer_softirq+0x276/0x2c0 [59826.077733] [<ffffffff810d833c>] ? sched_clock_local+0x1c/0x90 [59826.077733] [<ffffffff810d844a>] ? sched_clock_cpu+0x9a/0xb0 [59826.077733] [<ffffffff818db0ce>] __do_softirq+0xce/0x2bf [59826.077733] [<ffffffff810d844a>] ? sched_clock_cpu+0x9a/0xb0 [59826.077733] [<ffffffff810ab1d5>] irq_exit+0x105/0x110 [59826.077733] [<ffffffff8150e3f9>] xen_evtchn_do_upcall+0x39/0x50 [59826.077733] [<ffffffff818d93de>] xen_do_hypervisor_callback+0x1e/0x40 [59826.077733] <EOI> [59826.077733] [<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20 [59826.077733] [<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20 [59826.077733] [<ffffffff81103fff>] ? console_unlock+0x2ff/0x3e0 [59826.077733] [<ffffffff81024c5d>] ? xen_force_evtchn_callback+0xd/0x10 [59826.077733] [<ffffffff810253b6>] ? check_events+0x16/0x30 [59826.077733] [<ffffffff81025393>] ? xen_restore_fl_direct_reloc+0x4/0x4 [59826.077733] [<ffffffff81032b3d>] ? oops_end+0x7d/0xe0 [59826.077733] [<ffffffff81032cab>] ? die+0x5b/0x90 [59826.077733] [<ffffffff81030084>] ? do_general_protection+0xe4/0x170 [59826.077733] [<ffffffff818d97a8>] ? general_protection+0x28/0x30 [59826.077733] [<ffffffff812405d6>] ? kmem_cache_alloc_node+0xa6/0x1c0 [59826.077733] [<ffffffff812406c5>] ? kmem_cache_alloc_node+0x195/0x1c0 [59826.077733] [<ffffffff818d5e36>] ? mutex_lock+0x16/0x40 [59826.077733] [<ffffffff81792687>] ? __alloc_skb+0x57/0x200 [59826.077733] [<ffffffff81792687>] ? __alloc_skb+0x57/0x200 [59826.077733] [<ffffffff817e2af0>] ? netlink_attachskb+0x1b0/0x1b0 [59826.077733] [<ffffffff817df900>] ? netlink_overrun+0x50/0x50 [59826.077733] [<ffffffff817e319c>] ? netlink_ack+0x6c/0x170 [59826.077733] [<ffffffff8115526b>] ? audit_receive+0x8b/0xa0 [59826.077733] [<ffffffff817e30aa>] ? netlink_unicast+0x19a/0x220 [59826.077733] [<ffffffff817e3ed2>] ? netlink_sendmsg+0x302/0x380 [59826.077733] [<ffffffff81787ba9>] ? sock_sendmsg+0x49/0x60 [59826.077733] [<ffffffff8178a006>] ? SYSC_sendto+0x116/0x170 [59826.077733] [<ffffffff81287efc>] ? __alloc_fd+0x4c/0x1b0 [59826.077733] [<ffffffff810033c1>] ? syscall_trace_enter+0x201/0x2c0 [59826.077733] [<ffffffff8115b199>] ? __audit_syscall_exit+0x229/0x2b0 [59826.077733] [<ffffffff8178a06e>] ? SyS_sendto+0xe/0x10 [59826.077733] [<ffffffff81003e7a>] ? do_syscall_64+0x7a/0x240 [59826.077733] [<ffffffff8106f1a7>] ? do_page_fault+0x37/0x90 [59826.077733] [<ffffffff818d83eb>] ? entry_SYSCALL64_slow_path+0x25/0x25 [59826.077733] Code: 05 91 86 dc 7e 4d 8b 08 49 8b 40 10 4d 85 c9 0f 84 24 01 00 00 48 85 c0 0f 84 1b 01 00 00 49 63 44 24 20 49 8b 3c 24 48 8d 4a 01 <49> 8b 1c 01 4c 89 c8 48 8d 37 e8 ec 9 f 1e 00 3c 01 75 b6 49 63 [59826.077733] RIP [<ffffffff81241ad5>] kmem_cache_alloc+0x85/0x1b0 [59826.077733] RSP <ffff88001f803ba8> [59826.100347] ---[ end trace 58d7c3379dbe9d6a ]--- [59826.100533] Kernel panic - not syncing: Fatal exception in interrupt [59826.100721] Kernel Offset: disabled (XEN) Hardware Dom0 crashed: rebooting machine in 5 seconds. Regards, -- Piotr Gackiewicz Intertele S.A. - operator systemów ITL.PL i DOMENY.ITL.PL al. T. Rejtana 10, 35-310 Rzeszów TEL: +48 17 8507580, FAX: +48 17 8520275 http://www.itl.pl - niezawodne usługi hostingowe http://domeny.itl.pl - tanie domeny internetowe http://www.intertele.pl