We were previously running 5.1 x86_64 and recently updated to 5.2 using yum. Under 5.1 we were having problems when running jobs using torque and the solution had been to add the following items to the files noted
"* soft memlock unlimited" in /etc/security/limits.conf "session required pam_limits.so" in /etc/pam.d/{rsh,sshd}
This changed the max locked memory setting in ulimit as follows:
Before the change rsh nodeX ulimit -a still gives us max locked memory (kbytes, -l) 32
After the change rsh nodeX ulimit -a max locked memory (kbytes, -l) 16505400
The nodes have 16gb of memory.
Now after the 5.2 updates those files are all the same and on most of the nodes we haven't yet rebooted them due to log running processes but a few nodes have been restarted and now that jobs are starting to be put on them we are back to max locked memory of 32k rather than 16gb.
The error we are receiving on those jobs is :
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes. This will severely limit memory registrations. libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes. This will severely limit memory registrations. Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(306).......: Initialization failed MPID_Init(113)..............: channel initialization failed MPIDI_CH3_Init(167).........: MPIDI_CH3I_RDMA_init(138)...: rdma_setup_startup_ring(333): cannot create cq Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(306).......: Initialization failed MPID_Init(113)..............: channel initialization failed MPIDI_CH3_Init(167).........: MPIDI_CH3I_RDMA_init(138)...: rdma_setup_startup_ring(333): cannot create cq rank 45 in job 1 nodeX_35175 caused collective abort of all ranks exit status of rank 45: return code 1 rank 44 in job 1 nodeX_35175 caused collective abort of all ranks exit status of rank 44: return code 1
The full output of :
rsh nodeX ulimit -a
connect to address x.x.x.x port 544: Connection refused Trying krb4 rsh... connect to address x.x.x.x port 544: Connection refused trying normal rsh (/usr/bin/rsh) core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 135168 max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 135168 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
Any ideas, suggestions or items I could roll back would be appreciated. I looked through the list of packages that were updated and the only one that I could see that was related was pam. ssh and rsh were not updated.
Thank you, Rob