[CentOS-virt] kernel-4.9.37-29.el7 (and el6)

Thu Jul 20 20:14:13 UTC 2017
Piotr Gackiewicz <p.gackiewicz at intertele.pl>

On Thu, 20 Jul 2017, Kevin Stange wrote:

> On 07/20/2017 05:31 AM, Piotr Gackiewicz wrote:
>> On Wed, 19 Jul 2017, Johnny Hughes wrote:
>>
>>> On 07/19/2017 09:23 AM, Johnny Hughes wrote:
>>>> On 07/19/2017 04:27 AM, Piotr Gackiewicz wrote:
>>>>> On Mon, 17 Jul 2017, Johnny Hughes wrote:
>>>>>
>>>>>> Are the testing kernels (kernel-4.9.37-29.el7 and
>>>>>> kernel-4.9.37-29.el6,
>>>>>> with the one config file change) working for everyone:
>>>>>>
>>>>>> (turn off: CONFIG_IO_STRICT_DEVMEM)
>>>>>
>>>>> Hello.
>>>>> Maybe it's not the most appropriate thread or time, but I have been
>>>>> signalling it before:
>>>>>
>>>>> 4.9.* kernels do not work well for me any more (and for other people
>>>>> neither, as I know). Last stable kernel was 4.9.13-22.
>>
>> I think I have nailed down the faulty combo.
>> My tests showed, that SLUB allocator does not work well in Xen Dom0, on
>> top of Xen Hypervisor.
>> Id does not work at least on one of my testing servers (old AMD K8 (1 proc,
>> 1 core), only 1 paravirt guest).
>> If kernel with SLUB booted as main (w/o Xen hypervisor), it works well.
>> If booted as Xen hypervisor module - it almost instantly gets page
>> allocation failure.
>>
>>
>> SLAB=>SLUB was changed in kernel config, starting from 4.9.25. Then
>> problems
>> started to explode in my production environment, and on testing server
>> mentioned
>> above.
>>
>> After recompiling recent 4.9.34 with SLAB - everything works well on
>> that testing machine.
>> A will try to test 4.9.38 with the same config on my production servers.
>
> I was having page allocation failures on 4.9.25 with SLUB, but these
> problems seem to be gone with 4.9.34 (still with SLUB).   Have you
> checked this build?  It was moved to the stable repo on July 4th.

Yes, 4.9.34 was failing too. And this was actually the worst case, with I/O error on guest:

Jul 16 06:01:03 dom0 kernel: [452360.743312] CPU: 0 PID: 28450 Comm: 12.xvda3-0 Tainted: G           O    4.9.34-29.el6.x86_64 #1
Jul 16 06:01:03 guest kernel: end_request: I/O error, dev xvda3, sector 9200640
Jul 16 06:01:03 dom0 kernel: [452360.758931] SLUB: Unable to allocate memory on node -1, gfp=0x2000000(GFP_NOWAIT)
Jul 16 06:01:03 guest kernel: Buffer I/O error on device xvda3, logical block 1150080
Jul 16 06:01:03 guest kernel: lost page write due to I/O error on xvda3
Jul 16 06:01:03 guest kernel: Buffer I/O error on device xvda3, logical block 1150081
Jul 16 06:01:03 guest kernel: lost page write due to I/O error on xvda3
Jul 16 06:01:03 guest kernel: Buffer I/O error on device xvda3, logical block 1150082
Jul 16 06:01:03 guest kernel: lost page write due to I/O error on xvda3
Jul 16 06:01:03 guest kernel: Buffer I/O error on device xvda3, logical block 1150083
Jul 16 06:01:03 guest kernel: lost page write due to I/O error on xvda3
Jul 16 06:01:03 guest kernel: Buffer I/O error on device xvda3, logical block 1150084
Jul 16 06:01:03 guest kernel: lost page write due to I/O error on xvda3
Jul 16 06:01:03 dom0 kernel: [452361.449389] 12.xvda3-0: page allocation failure: order:0, mode:0x2200000(GFP_NOWAIT|__GFP_NOTRACK)
Jul 16 06:01:03 dom0 kernel: [452361.449685] CPU: 1 PID: 28450 Comm: 12.xvda3-0 Tainted: G           O    4.9.34-29.el6.x86_64 #1
Jul 16 06:01:03 dom0 kernel: [452361.449934] Hardware name: Supermicro X8SIL/X8SIL, BIOS 1.0c 02/25/2010
Jul 16 06:01:03 guest kernel: end_request: I/O error, dev xvda3, sector 6102784
Jul 16 06:01:03 dom0 kernel: [452361.462103] SLUB: Unable to allocate memory on node -1, gfp=0x2000000(GFP_NOWAIT)
Jul 16 06:01:03 dom0 kernel: [452361.676257] 12.xvda3-0: page allocation failure: order:0, mode:0x2200000(GFP_NOWAIT|__GFP_NOTRACK)
Jul 16 06:01:03 dom0 kernel: [452361.676531] CPU: 0 PID: 28450 Comm: 12.xvda3-0 Tainted: G           O    4.9.34-29.el6.x86_64 #1
Jul 16 06:01:03 guest kernel: end_request: I/O error, dev xvda3, sector 6127872
Jul 16 06:01:03 dom0 kernel: [452361.692171] SLUB: Unable to allocate memory on node -1, gfp=0x2000000(GFP_NOWAIT)
Jul 16 06:01:07 dom0 kernel: [452365.438565] 12.xvda3-0: page allocation failure: order:0, mode:0x2200000(GFP_NOWAIT|__GFP_NOTRACK)
Jul 16 06:01:07 dom0 kernel: [452365.438870] CPU: 0 PID: 28450 Comm: 12.xvda3-0 Tainted: G           O    4.9.34-29.el6.x86_64 #1
Jul 16 06:01:07 dom0 kernel: [452365.454213] SLUB: Unable to allocate memory on node -1, gfp=0x2000000(GFP_NOWAIT)
Jul 16 06:01:07 guest kernel: end_request: I/O error, dev xvda3, sector 6477112
Jul 16 06:01:09 dom0 kernel: [452366.732994] 12.xvda3-0: page allocation failure: order:0, mode:0x2200000(GFP_NOWAIT|__GFP_NOTRACK)
Jul 16 06:01:09 dom0 kernel: [452366.733306] CPU: 0 PID: 28450 Comm: 12.xvda3-0 Tainted: G           O    4.9.34-29.el6.x86_64 #1
Jul 16 06:01:09 dom0 kernel: [452366.746362] SLUB: Unable to allocate memory on node -1, gfp=0x2000000(GFP_NOWAIT)
Jul 16 06:01:09 guest kernel: end_request: I/O error, dev xvda3, sector 6546488
Jul 16 06:01:09 guest kernel: Buffer I/O error on device xvda3, logical block 818311
Jul 16 06:01:09 guest kernel: lost page write due to I/O error on xvda3
Jul 16 06:01:09 guest kernel: Buffer I/O error on device xvda3, logical block 818312
Jul 16 06:01:09 guest kernel: lost page write due to I/O error on xvda3
Jul 16 06:01:09 guest kernel: Buffer I/O error on device xvda3, logical block 818313
Jul 16 06:01:09 guest kernel: lost page write due to I/O error on xvda3
Jul 16 06:01:09 guest kernel: Buffer I/O error on device xvda3, logical block 818314
Jul 16 06:01:09 guest kernel: lost page write due to I/O error on xvda3
Jul 16 06:01:09 guest kernel: Buffer I/O error on device xvda3, logical block 818315
Jul 16 06:01:09 dom0 kernel: [452366.913734] 12.xvda3-0: page allocation failure: order:0, mode:0x2200000(GFP_NOWAIT|__GFP_NOTRACK)
Jul 16 06:01:09 dom0 kernel: [452366.914002] CPU: 1 PID: 28450 Comm: 12.xvda3-0 Tainted: G           O    4.9.34-29.el6.x86_64 #1
Jul 16 06:01:09 guest kernel: end_request: I/O error, dev xvda3, sector 6366208
Jul 16 06:01:09 dom0 kernel: [452366.929809] SLUB: Unable to allocate memory on node -1, gfp=0x2000000(GFP_NOWAIT)
Jul 16 06:01:09 dom0 kernel: [452367.288193] 12.xvda3-0: page allocation failure: order:0, mode:0x2200000(GFP_NOWAIT|__GFP_NOTRACK)
Jul 16 06:01:09 dom0 kernel: [452367.288455] CPU: 1 PID: 28450 Comm: 12.xvda3-0 Tainted: G           O    4.9.34-29.el6.x86_64 #1
Jul 16 06:01:09 dom0 kernel: [452367.301690] SLUB: Unable to allocate memory on node -1, gfp=0x2000000(GFP_NOWAIT)
Jul 16 06:01:09 guest kernel: end_request: I/O error, dev xvda3, sector 6630656
Jul 16 06:01:10 dom0 kernel: [452368.253435] 12.xvda3-0: page allocation failure: order:0, mode:0x2200000(GFP_NOWAIT|__GFP_NOTRACK)
Jul 16 06:01:10 dom0 kernel: [452368.253701] CPU: 0 PID: 28450 Comm: 12.xvda3-0 Tainted: G           O    4.9.34-29.el6.x86_64 #1
Jul 16 06:01:10 guest kernel: end_request: I/O error, dev xvda3, sector 6708224

Regards,

-- 
Piotr Gackiewicz
Intertele S.A. - operator systemów ITL.PL i DOMENY.ITL.PL
al. T. Rejtana 10, 35-310 Rzeszów
TEL: +48 17 8507580, FAX: +48 17 8520275

http://www.itl.pl       - niezawodne usługi hostingowe
http://domeny.itl.pl    - tanie domeny internetowe
http://www.intertele.pl