Thank you! With 'cp' the job was killed. There was error message: Mar 10 09:50:04 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x8020) Mar 10 09:50:04 kernel: cache: kmalloc-64(5:step_0), object size: 64, buffer size: 64, default order: 0, min order: 0 Mar 10 09:50:04 kernel: node 0: slabs: 4, objs: 256, free: 0 Mar 10 09:50:04 kernel: node 1: slabs: 0, objs: 0, free: 0
When I replaced the cp command with 'dd bs=4M iflag=direct oflag=direct ...', the file copying ran happily to completion.
On Sat, Mar 11, 2017 at 8:48 AM, Jonathan Billings billings@negate.org wrote:
On Mar 10, 2017, at 3:51 PM, Wensheng Deng wd35@nyu.edu wrote:
I have 3.10 kernel. I am running some data processing job, need to first copy big (>5 GB) input files. The jobs were killed, because the system thought I used 5 GB memory from the file copying.
If you’re using ‘cp’ you probably aren’t using 5G of RAM. That’s not how ‘cp’ works. Actual errors might be helpful here.
If you are running batch processing and don’t want the OOM Killer to ever get involved, the cgroup memory accounting features actually let you turn it off with memory.oom_control=1 in the cgroup. You can also turn off the heuristic overcommit memory manager (see https://www.kernel.org/doc/ Documentation/vm/overcommit-accounting for details) but I suggest figuring out your problem with copying files first.
-- Jonathan Billings billings@negate.org
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos