[CentOS] kernel memory accounting

Sat Mar 11 14:30:18 UTC 2017
Wensheng Deng <wd35 at nyu.edu>

Thank you! With 'cp' the job was killed. There was error message:
Mar 10 09:50:04  kernel: SLUB: Unable to allocate memory on node -1
(gfp=0x8020)
Mar 10 09:50:04  kernel:  cache: kmalloc-64(5:step_0), object size: 64,
buffer size: 64, default order: 0, min order: 0
Mar 10 09:50:04  kernel:  node 0: slabs: 4, objs: 256, free: 0
Mar 10 09:50:04  kernel:  node 1: slabs: 0, objs: 0, free: 0

When I replaced the cp command with 'dd bs=4M iflag=direct oflag=direct
...', the file copying ran happily to completion.


On Sat, Mar 11, 2017 at 8:48 AM, Jonathan Billings <billings at negate.org>
wrote:

> On Mar 10, 2017, at 3:51 PM, Wensheng Deng <wd35 at nyu.edu> wrote:
> >
> > I have 3.10 kernel. I am running some data processing job, need to first
> > copy big (>5 GB) input files. The jobs were killed, because the system
> > thought I used 5 GB memory from the file copying.
>
> If you’re using ‘cp’ you probably aren’t using 5G of RAM.  That’s not how
> ‘cp’ works.  Actual errors might be helpful here.
>
> If you are running batch processing and don’t want the OOM Killer to ever
> get involved, the cgroup memory accounting features actually let you turn
> it off with memory.oom_control=1 in the cgroup.  You can also turn off the
> heuristic overcommit memory manager (see https://www.kernel.org/doc/
> Documentation/vm/overcommit-accounting for details) but I suggest
> figuring out your problem with copying files first.
>
> --
> Jonathan Billings <billings at negate.org>
>
>
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos
>