Attempting to lookup why rsyslogd is listening on the high port UDP/51427. Have not succeeded in what this port is used for and what directive controls what interface it binds to.
[root@bedrock ~]# netstat --listen --inet --program --numeric | grep syslog udp 0 0 0.0.0.0:51427 0.0.0.0:* 66655/rsyslogd
On Thu, 6 Sep 2018 at 14:11, Adam Tauno Williams awilliam@whitemice.org wrote:
Attempting to lookup why rsyslogd is listening on the high port UDP/51427. Have not succeeded in what this port is used for and what directive controls what interface it binds to.
[root@bedrock ~]# netstat --listen --inet --program --numeric | grep syslog udp 0 0 0.0.0.0:51427 0.0.0.0:* 66655/rsyslogd
I don't think that is really rsyslogd running.. first rsyslogd usually is running with a much lower pid as it starts early on. All the boxes I saw had it running under 1000. If the system is set up to actually listen to the internet it will look like
[root@log01 ~][PROD]# netstat --list --inet --program --numeric | grep rsys tcp 0 0 0.0.0.0:514 0.0.0.0:* LISTEN 1078/rsyslogd tcp 0 0 0.0.0.0:5000 0.0.0.0:* LISTEN 1078/rsyslogd udp 0 0 0.0.0.0:514 0.0.0.0:* 1078/rsyslogd
and those are set in /etc/rsyslogd.conf
[root@log01 ~][PROD]# egrep '514|5000' /etc/rsyslog.conf $UDPServerRun 514 $InputTCPServerRun 514 $InputTCPServerRun 5000
I would do a ps auxww | grep 66655 and see what is running and then check to see if that binary is what it is supposed to be.
-- Adam Tauno Williams mailto:awilliam@whitemice.org GPG D95ED383 OpenGroupware Developer http://www.opengroupware.us/ _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
On 2018-09-06 14:06, Adam Tauno Williams wrote:
Attempting to lookup why rsyslogd is listening on the high port UDP/51427. Have not succeeded in what this port is used for and what directive controls what interface it binds to.
[root@bedrock ~]# netstat --listen --inet --program --numeric | grep syslog udp 0 0 0.0.0.0:51427 0.0.0.0:* 66655/rsyslogd
Adam,
You might want to try running:
lsof -i -P | grep LISTEN | grep :51427
to determine what process is actually listening to that port.
On Thu, 2018-09-06 at 15:10 -0400, Mike Burger wrote:
On 2018-09-06 14:06, Adam Tauno Williams wrote:
Attempting to lookup why rsyslogd is listening on the high port UDP/51427. Have not succeeded in what this port is used for and what directive controls what interface it binds to. [root@bedrock ~]# netstat --listen --inet --program --numeric | grep syslog udp 0 0 0.0.0.0:51427 0.0.0.0:* 66655/rsyslogd
You might want to try running: lsof -i -P | grep LISTEN | grep :51427 to determine what process is actually listening to that port.
That is what is strange; lsof does *not* see the port as listening, but it is visible if I search by inode [and it is clearly rsyslogd - but I cannot find any documentation indicating what this is].
[root@bedrock ~]# netstat --inet --program --numeric --listen | grep syslog udp 0 0 0.0.0.0:51427 0.0.0.0:* 66655/rsyslogd [root@bedrock ~]# lsof -i -P | grep LISTEN rpcbind 810 rpc 8u IPv4 19806 0t0 TCP *:111 (LISTEN) rpcbind 810 rpc 11u IPv6 19809 0t0 TCP *:111 (LISTEN) sshd 1156 root 3u IPv4 23045 0t0 TCP *:22 (LISTEN) sshd 1156 root 4u IPv6 23047 0t0 TCP *:22 (LISTEN) snmpd 1158 root 8u IPv4 26937 0t0 TCP localhost:199 (LISTEN) rpc.statd 1196 rpcuser 9u IPv4 24024 0t0 TCP *:662 (LISTEN) rpc.statd 1196 rpcuser 11u IPv6 24030 0t0 TCP *:662 (LISTEN) avagent.b 1431 root 4u IPv6 26892 0t0 TCP *:28002 (LISTEN) avagent.b 1431 root 6u IPv6 28867 0t0 TCP localhost:38061 (LISTEN) master 1535 root 13u IPv4 26579 0t0 TCP localhost:25 (LISTEN) master 1535 root 14u IPv6 26580 0t0 TCP localhost:25 (LISTEN) smbd 1663 root 35u IPv6 28676 0t0 TCP *:445 (LISTEN) smbd 1663 root 36u IPv4 28677 0t0 TCP *:445 (LISTEN) [root@bedrock ~]# netstat -e --inet --program --numeric --listen | grep syslog udp 0 0 0.0.0.0:51427 0.0.0.0:* 0 5032773 66655/rsyslogd [root@bedrock ~]# lsof | awk 'NR==1 || /5032773/' COMMAND PID TID USER FD TYPE DEVICE SIZE/OFF NODE NAME rsyslogd 66655 root 4u IPv4 5032773 0t0 UDP *:51427 in:imjour 66655 66657 root 4u IPv4 5032773 0t0 UDP *:51427 rs:main 66655 66658 root 4u IPv4 5032773 0t0 UDP *:51427
The port is closed on the firewall; but I file a report which includes every listening port and its justification.
NOTE: rsyslog exhibits this behavior on every CentOS6 & CentOS7 host. And all our host log via rsyslogd via UDP/514 to a central NMS with a syslog receiver.
On 07.09.2018 12:32, Adam Tauno Williams wrote:
On Thu, 2018-09-06 at 15:10 -0400, Mike Burger wrote:
lsof -i -P | grep LISTEN | grep :51427 to determine what process is actually listening to that port.
That is what is strange; lsof does *not* see the port as listening,
because there is no LISTEN in the lsof output for UDP ports.
just execute
lsof -i :51427 -P
should show you the process.
best regards Ulf
On Thu, Sep 06, 2018 at 02:06:37PM -0400, Adam Tauno Williams wrote:
Attempting to lookup why rsyslogd is listening on the high port UDP/51427. Have not succeeded in what this port is used for and what directive controls what interface it binds to.
[root@bedrock ~]# netstat --listen --inet --program --numeric | grep syslog udp 0 0 0.0.0.0:51427 0.0.0.0:* 66655/rsyslogd
The 51427 is the ephemeral port on the client side of the UDP session. You can verify this by running tcpdump to capture traffic when a syslog message is passed.
I can report that I also see this netstat (and similar with ss) state for systems with rsyslog set up to send to a remote log server, where ss reports that the process has UNCONN state on high UDP ports.
I suspect it's part of the UDP handshake that rsyslog uses for sending syslogs, but I'm not familiar enough with how it works to say definitively. Since it's UDP, it's a sessionless protocol, so it's not strictly LISTENing, but with ss you can see it's UNCONN, which other daemons that *are* listening for UDP traffic also report.
It is quite interesting to me, and if anyone knows why this works this way, I'd be happy to hear. I did some tests with 'nc -u' and I couldn't get similar results.
On Fri, 2018-09-07 at 09:20 -0400, Jonathan Billings wrote:
[root@bedrock ~]# netstat --listen --inet --program --numeric | grep syslog udp 0 0 0.0.0.0:51427 0.0.0.0:* 66655/rsyslogd
The 51427 is the ephemeral port on the client side of the UDP session. You can verify this by running tcpdump to capture traffic when a syslog message is passed.
Thanks, I was suspecting something like this as the only way to make that port disappear was to disable remote logging.
I suspect it's part of the UDP handshake that rsyslog uses for sending syslogs, but I'm not familiar enough with how it works to say definitively.
It was puzzling because I can't find any reference to this behavior in any documentation.
Since it's UDP, it's a sessionless protocol, so it's not strictly LISTENing, but with ss you can see it's UNCONN, which other daemons that *are* listening for UDP traffic also report.
Right, distinguishing between the listening and open in UDP is always somewhere between tedious and impossible. Perhaps I should investigate logging over TCP! :)
Anyway, I have something to write in the report now.
On Fri, 7 Sep 2018 at 09:20, Jonathan Billings billings@negate.org wrote:
On Thu, Sep 06, 2018 at 02:06:37PM -0400, Adam Tauno Williams wrote:
Attempting to lookup why rsyslogd is listening on the high port UDP/51427. Have not succeeded in what this port is used for and what directive controls what interface it binds to.
[root@bedrock ~]# netstat --listen --inet --program --numeric | grep syslog udp 0 0 0.0.0.0:51427 0.0.0.0:* 66655/rsyslogd
The 51427 is the ephemeral port on the client side of the UDP session. You can verify this by running tcpdump to capture traffic when a syslog message is passed.
That makes perfect sense. I had checked several boxes on our side and had not found any with that port being shown but we use TCP for logging so I expect that is why. My apologies to Mr Billings for going for the worst possible scenario.
I can report that I also see this netstat (and similar with ss) state for systems with rsyslog set up to send to a remote log server, where ss reports that the process has UNCONN state on high UDP ports.
I suspect it's part of the UDP handshake that rsyslog uses for sending syslogs, but I'm not familiar enough with how it works to say definitively. Since it's UDP, it's a sessionless protocol, so it's not strictly LISTENing, but with ss you can see it's UNCONN, which other daemons that *are* listening for UDP traffic also report.
It is quite interesting to me, and if anyone knows why this works this way, I'd be happy to hear. I did some tests with 'nc -u' and I couldn't get similar results.
-- Jonathan Billings billings@negate.org _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
-
Adam Tauno Williams -
Jonathan Billings -
Mike Burger -
Stephen John Smoogen -
Ulf Volmer