On Thu, 20 Jul 2017, Kevin Stange wrote: > On 07/20/2017 05:31 AM, Piotr Gackiewicz wrote: >> On Wed, 19 Jul 2017, Johnny Hughes wrote: >> >>> On 07/19/2017 09:23 AM, Johnny Hughes wrote: >>>> On 07/19/2017 04:27 AM, Piotr Gackiewicz wrote: >>>>> On Mon, 17 Jul 2017, Johnny Hughes wrote: >>>>> >>>>>> Are the testing kernels (kernel-4.9.37-29.el7 and >>>>>> kernel-4.9.37-29.el6, >>>>>> with the one config file change) working for everyone: >>>>>> >>>>>> (turn off: CONFIG_IO_STRICT_DEVMEM) >>>>> >>>>> Hello. >>>>> Maybe it's not the most appropriate thread or time, but I have been >>>>> signalling it before: >>>>> >>>>> 4.9.* kernels do not work well for me any more (and for other people >>>>> neither, as I know). Last stable kernel was 4.9.13-22. >> >> I think I have nailed down the faulty combo. >> My tests showed, that SLUB allocator does not work well in Xen Dom0, on >> top of Xen Hypervisor. >> Id does not work at least on one of my testing servers (old AMD K8 (1 proc, >> 1 core), only 1 paravirt guest). >> If kernel with SLUB booted as main (w/o Xen hypervisor), it works well. >> If booted as Xen hypervisor module - it almost instantly gets page >> allocation failure. >> >> >> SLAB=>SLUB was changed in kernel config, starting from 4.9.25. Then >> problems >> started to explode in my production environment, and on testing server >> mentioned >> above. >> >> After recompiling recent 4.9.34 with SLAB - everything works well on >> that testing machine. >> A will try to test 4.9.38 with the same config on my production servers. > > I was having page allocation failures on 4.9.25 with SLUB, but these > problems seem to be gone with 4.9.34 (still with SLUB). Have you > checked this build? It was moved to the stable repo on July 4th. Yes, 4.9.34 was failing too. And this was actually the worst case, with I/O error on guest: Jul 16 06:01:03 dom0 kernel: [452360.743312] CPU: 0 PID: 28450 Comm: 12.xvda3-0 Tainted: G O 4.9.34-29.el6.x86_64 #1 Jul 16 06:01:03 guest kernel: end_request: I/O error, dev xvda3, sector 9200640 Jul 16 06:01:03 dom0 kernel: [452360.758931] SLUB: Unable to allocate memory on node -1, gfp=0x2000000(GFP_NOWAIT) Jul 16 06:01:03 guest kernel: Buffer I/O error on device xvda3, logical block 1150080 Jul 16 06:01:03 guest kernel: lost page write due to I/O error on xvda3 Jul 16 06:01:03 guest kernel: Buffer I/O error on device xvda3, logical block 1150081 Jul 16 06:01:03 guest kernel: lost page write due to I/O error on xvda3 Jul 16 06:01:03 guest kernel: Buffer I/O error on device xvda3, logical block 1150082 Jul 16 06:01:03 guest kernel: lost page write due to I/O error on xvda3 Jul 16 06:01:03 guest kernel: Buffer I/O error on device xvda3, logical block 1150083 Jul 16 06:01:03 guest kernel: lost page write due to I/O error on xvda3 Jul 16 06:01:03 guest kernel: Buffer I/O error on device xvda3, logical block 1150084 Jul 16 06:01:03 guest kernel: lost page write due to I/O error on xvda3 Jul 16 06:01:03 dom0 kernel: [452361.449389] 12.xvda3-0: page allocation failure: order:0, mode:0x2200000(GFP_NOWAIT|__GFP_NOTRACK) Jul 16 06:01:03 dom0 kernel: [452361.449685] CPU: 1 PID: 28450 Comm: 12.xvda3-0 Tainted: G O 4.9.34-29.el6.x86_64 #1 Jul 16 06:01:03 dom0 kernel: [452361.449934] Hardware name: Supermicro X8SIL/X8SIL, BIOS 1.0c 02/25/2010 Jul 16 06:01:03 guest kernel: end_request: I/O error, dev xvda3, sector 6102784 Jul 16 06:01:03 dom0 kernel: [452361.462103] SLUB: Unable to allocate memory on node -1, gfp=0x2000000(GFP_NOWAIT) Jul 16 06:01:03 dom0 kernel: [452361.676257] 12.xvda3-0: page allocation failure: order:0, mode:0x2200000(GFP_NOWAIT|__GFP_NOTRACK) Jul 16 06:01:03 dom0 kernel: [452361.676531] CPU: 0 PID: 28450 Comm: 12.xvda3-0 Tainted: G O 4.9.34-29.el6.x86_64 #1 Jul 16 06:01:03 guest kernel: end_request: I/O error, dev xvda3, sector 6127872 Jul 16 06:01:03 dom0 kernel: [452361.692171] SLUB: Unable to allocate memory on node -1, gfp=0x2000000(GFP_NOWAIT) Jul 16 06:01:07 dom0 kernel: [452365.438565] 12.xvda3-0: page allocation failure: order:0, mode:0x2200000(GFP_NOWAIT|__GFP_NOTRACK) Jul 16 06:01:07 dom0 kernel: [452365.438870] CPU: 0 PID: 28450 Comm: 12.xvda3-0 Tainted: G O 4.9.34-29.el6.x86_64 #1 Jul 16 06:01:07 dom0 kernel: [452365.454213] SLUB: Unable to allocate memory on node -1, gfp=0x2000000(GFP_NOWAIT) Jul 16 06:01:07 guest kernel: end_request: I/O error, dev xvda3, sector 6477112 Jul 16 06:01:09 dom0 kernel: [452366.732994] 12.xvda3-0: page allocation failure: order:0, mode:0x2200000(GFP_NOWAIT|__GFP_NOTRACK) Jul 16 06:01:09 dom0 kernel: [452366.733306] CPU: 0 PID: 28450 Comm: 12.xvda3-0 Tainted: G O 4.9.34-29.el6.x86_64 #1 Jul 16 06:01:09 dom0 kernel: [452366.746362] SLUB: Unable to allocate memory on node -1, gfp=0x2000000(GFP_NOWAIT) Jul 16 06:01:09 guest kernel: end_request: I/O error, dev xvda3, sector 6546488 Jul 16 06:01:09 guest kernel: Buffer I/O error on device xvda3, logical block 818311 Jul 16 06:01:09 guest kernel: lost page write due to I/O error on xvda3 Jul 16 06:01:09 guest kernel: Buffer I/O error on device xvda3, logical block 818312 Jul 16 06:01:09 guest kernel: lost page write due to I/O error on xvda3 Jul 16 06:01:09 guest kernel: Buffer I/O error on device xvda3, logical block 818313 Jul 16 06:01:09 guest kernel: lost page write due to I/O error on xvda3 Jul 16 06:01:09 guest kernel: Buffer I/O error on device xvda3, logical block 818314 Jul 16 06:01:09 guest kernel: lost page write due to I/O error on xvda3 Jul 16 06:01:09 guest kernel: Buffer I/O error on device xvda3, logical block 818315 Jul 16 06:01:09 dom0 kernel: [452366.913734] 12.xvda3-0: page allocation failure: order:0, mode:0x2200000(GFP_NOWAIT|__GFP_NOTRACK) Jul 16 06:01:09 dom0 kernel: [452366.914002] CPU: 1 PID: 28450 Comm: 12.xvda3-0 Tainted: G O 4.9.34-29.el6.x86_64 #1 Jul 16 06:01:09 guest kernel: end_request: I/O error, dev xvda3, sector 6366208 Jul 16 06:01:09 dom0 kernel: [452366.929809] SLUB: Unable to allocate memory on node -1, gfp=0x2000000(GFP_NOWAIT) Jul 16 06:01:09 dom0 kernel: [452367.288193] 12.xvda3-0: page allocation failure: order:0, mode:0x2200000(GFP_NOWAIT|__GFP_NOTRACK) Jul 16 06:01:09 dom0 kernel: [452367.288455] CPU: 1 PID: 28450 Comm: 12.xvda3-0 Tainted: G O 4.9.34-29.el6.x86_64 #1 Jul 16 06:01:09 dom0 kernel: [452367.301690] SLUB: Unable to allocate memory on node -1, gfp=0x2000000(GFP_NOWAIT) Jul 16 06:01:09 guest kernel: end_request: I/O error, dev xvda3, sector 6630656 Jul 16 06:01:10 dom0 kernel: [452368.253435] 12.xvda3-0: page allocation failure: order:0, mode:0x2200000(GFP_NOWAIT|__GFP_NOTRACK) Jul 16 06:01:10 dom0 kernel: [452368.253701] CPU: 0 PID: 28450 Comm: 12.xvda3-0 Tainted: G O 4.9.34-29.el6.x86_64 #1 Jul 16 06:01:10 guest kernel: end_request: I/O error, dev xvda3, sector 6708224 Regards, -- Piotr Gackiewicz Intertele S.A. - operator systemów ITL.PL i DOMENY.ITL.PL al. T. Rejtana 10, 35-310 Rzeszów TEL: +48 17 8507580, FAX: +48 17 8520275 http://www.itl.pl - niezawodne usługi hostingowe http://domeny.itl.pl - tanie domeny internetowe http://www.intertele.pl