Hello,
I have several systems which I recently updated with
yum -y update
to all the latest packages. These systems use yum-priorities and use
the CentOS (priority 1) EPEL (priority 5) and rpmforge (priority 10)
repositories. After the updates, dhcpd stopped working with a SIGPIPE
error which occurs shortly after it attempts to fork into the
background. I worked around that problem by building a new server with
no additional repos, only CentOS and dhcpd works fine on that system.
Since then I have found the problem, or similar problems with a few
more applications. Here is what the tail of an strace of pbs_mom as it
attempts to fork into the background:
listen(5, 512) = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 6
setsockopt(6, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
bind(6, {sa_family=AF_INET, sin_port=htons(15003),
sin_addr=inet_addr("0.0.0.0")}, 16) = 0
listen(6, 512) = 0
fcntl(4, F_SETLK, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0
clone(Process 23938 attached (waiting for parent)
Process 23938 resumed (parent 23937 ready)
child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0x2aaaaad30db0) = 23938
[pid 23937] exit_group(0) = ?
getsockname(3, 0x7fff6b7728a0, [128]) = -1 ENOTSOCK (Socket
operation on non-socket)
fcntl(3, F_GETFD) = 0
dup(3) = 7
fcntl(7, F_SETFD, 0) = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 8
close(3) = 0
fcntl(8, F_GETFD) = 0
dup2(8, 3) = 3
fcntl(3, F_SETFD, 0) = 0
close(8) = 0
write(3, "\25\3\1\0\22\334\362\36\233\253\205\2633\323\322q\4\3T\rxK\210",
23) = -1 EPIPE (Broken pipe)
--- SIGPIPE (Broken pipe) @ 0 (0) ---
Process 23938 detached
This is pretty much the same thing that happened to dhcpd. In both
cases they applications work fine in debug mode when they don't
attempt to fork, but quietly die when ran normally. A third set of
apps, wrappers for the client part of torque (pbs_mom) do this:
stat("/usr/local/sbin/pbs_iff", {st_mode=S_IFREG|S_ISUID|0755,
st_size=21412, ...}) = 0
pipe([5, 6]) = 0
clone(Process 24068 attached (waiting for parent)
Process 24068 resumed (parent 24067 ready)
child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0x2aaaaad31ce0) = 24068
[pid 24067] close(6) = 0
[pid 24067] fcntl(5, F_GETFL) = 0 (flags O_RDONLY)
[pid 24067] read(5, <unfinished ...>
[pid 24068] getsockname(3, {sa_family=AF_INET, sin_port=htons(41855),
sin_addr=inet_addr("129.123.148.49")}, [1164321820984213520]) = 0
[pid 24068] getpeername(3, {sa_family=AF_INET, sin_port=htons(636),
sin_addr=inet_addr("129.123.20.92")}, [68719476752]) = 0
[pid 24068] fcntl(3, F_GETFD) = 0x1 (flags FD_CLOEXEC)
[pid 24068] dup(3) = 7
[pid 24068] fcntl(7, F_SETFD, FD_CLOEXEC) = 0
[pid 24068] socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 8
[pid 24068] close(3) = 0
[pid 24068] fcntl(8, F_GETFD) = 0
[pid 24068] dup2(8, 3) = 3
[pid 24068] fcntl(3, F_SETFD, 0) = 0
[pid 24068] close(8) = 0
[pid 24068] write(3,
"\25\3\1\0\22\346h\357n\r\17x\374B\312\217\374x\276\311\217\342%", 23)
= -1 EPIPE (Broken pipe)
[pid 24068] --- SIGPIPE (Broken pipe) @ 0 (0) ---
Process 24068 detached
<... read resumed> "", 4) = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
close(5) = 0
wait4(24068, [{WIFSIGNALED(s) && WTERMSIG(s) == SIGPIPE}], 0, NULL) = 24068
close(4) = 0
write(2, "No Permission.\n", 15No Permission.
) = 15
write(2, "qstat: cannot connect to server "..., 63qstat: cannot
connect to server moab.hpc.usu.edu (errno=15007)
) = 63
exit_group(-1) = ?
Once again, the app dies after it attempts to fork into the
background. There are other things running on these systems that can
successfully fork and I have been unable to figure out any pattern,
other than if I don't use additional repos then it doesn't seem to
break. That may be coincidental though, I haven't repeated it enough
yet to be certain.
Any hints or suggestions would be appreciated. Unfortunately I noticed
this after deciding it was "safe" to update *all* my machines and so
I'm suffering through a lot of rebuilds/restores because of this.
Thanks,
jbh