I have a CentOS release 5.8 that has snmp traps being sent to it. I've been trying to forward the snmp traps to another system. I've tried forwarding with snmpd/snmptrapd, iptables, and some forwarding programs. I can see snmp traps getting delivered to the system with tcpdump and wireshark, but no matter what app I run, the traps do not appear to be reaching the application or port 162. It seems like the packets are possibly being dropped right away.
iptables is wide open:
# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination
Chain FORWARD (policy ACCEPT) target prot opt source destination
Chain OUTPUT (policy ACCEPT) target prot opt source destination
If I run the apps I can see port 162 open and closed depending on what I have running, so I'm sure there's not a specific app running already on that port.
Anyone have any ideas on what could be happening to these packets and why they might not be reaching port 162 on this host?
Thanks, James
On 10/4/2012 9:40 AM, James Pifer wrote:
I have a CentOS release 5.8 that has snmp traps being sent to it. I've been trying to forward the snmp traps to another system. I've tried forwarding with snmpd/snmptrapd, iptables, and some forwarding programs. I can see snmp traps getting delivered to the system with tcpdump and wireshark, but no matter what app I run, the traps do not appear to be reaching the application or port 162. It seems like the packets are possibly being dropped right away.
iptables is wide open:
# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination
Chain FORWARD (policy ACCEPT) target prot opt source destination
Chain OUTPUT (policy ACCEPT) target prot opt source destination
If I run the apps I can see port 162 open and closed depending on what I have running, so I'm sure there's not a specific app running already on that port.
Anyone have any ideas on what could be happening to these packets and why they might not be reaching port 162 on this host?
Just a follow up. I ran tcpdump for port 162 for a little while and when I stopped I see this at the end:
737 packets captured 737 packets received by filter 0 packets dropped by kernel
So I guess the kernel is not dropping them. Still can't explain why applications are not picking them up.
Any help is appreciated.
Thanks, James
On Thu, Oct 4, 2012 at 12:17 PM, James Pifer jep@obrien-pifer.com wrote:
On 10/4/2012 9:40 AM, James Pifer wrote:
I have a CentOS release 5.8 that has snmp traps being sent to it. I've been trying to forward the snmp traps to another system. I've tried forwarding with snmpd/snmptrapd, iptables, and some forwarding programs. I can see snmp traps getting delivered to the system with tcpdump and wireshark, but no matter what app I run, the traps do not appear to be reaching the application or port 162. It seems like the packets are possibly being dropped right away.
iptables is wide open:
# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination
Chain FORWARD (policy ACCEPT) target prot opt source destination
Chain OUTPUT (policy ACCEPT) target prot opt source destination
If I run the apps I can see port 162 open and closed depending on what I have running, so I'm sure there's not a specific app running already on that port.
Anyone have any ideas on what could be happening to these packets and why they might not be reaching port 162 on this host?
Just a follow up. I ran tcpdump for port 162 for a little while and when I stopped I see this at the end:
737 packets captured 737 packets received by filter 0 packets dropped by kernel
So I guess the kernel is not dropping them. Still can't explain why applications are not picking them up.
Any help is appreciated.
I'd try strace'ing the app that is supposed to be receiving them to see if the socket opens are working and what happens with a packet arrives on the port.
I'd try strace'ing the app that is supposed to be receiving them to see if the socket opens are working and what happens with a packet arrives on the port.
No idea what this means. snmptrapd keeps running (strace snmptrapd -f -Le -c /etc/snmp/snmptrapd.conf), but I see this over and over after the initial start:
gettimeofday({1349372532, 120897}, NULL) = 0 gettimeofday({1349372532, 120917}, NULL) = 0 gettimeofday({1349372532, 120934}, NULL) = 0 gettimeofday({1349372532, 120950}, NULL) = 0 select(9, [3 5 7 8], [], [], {5, 0}) = 0 (Timeout) gettimeofday({1349372537, 120615}, NULL) = 0 gettimeofday({1349372537, 120637}, NULL) = 0 gettimeofday({1349372537, 120655}, NULL) = 0 gettimeofday({1349372537, 120670}, NULL) = 0 gettimeofday({1349372537, 120686}, NULL) = 0 gettimeofday({1349372537, 120703}, NULL) = 0 gettimeofday({1349372537, 120721}, NULL) = 0 gettimeofday({1349372537, 120737}, NULL) = 0 select(9, [3 5 7 8], [], [], {5, 0}) = 0 (Timeout) gettimeofday({1349372542, 119701}, NULL) = 0 gettimeofday({1349372542, 119726}, NULL) = 0 gettimeofday({1349372542, 119744}, NULL) = 0 gettimeofday({1349372542, 119760}, NULL) = 0 gettimeofday({1349372542, 119776}, NULL) = 0 gettimeofday({1349372542, 119794}, NULL) = 0 gettimeofday({1349372542, 119813}, NULL) = 0 gettimeofday({1349372542, 119829}, NULL) = 0 select(9, [3 5 7 8], [], [], {5, 0}) = 0 (Timeout) gettimeofday({1349372547, 118753}, NULL) = 0 gettimeofday({1349372547, 118777}, NULL) = 0 gettimeofday({1349372547, 118794}, NULL) = 0 gettimeofday({1349372547, 118811}, NULL) = 0 gettimeofday({1349372547, 118827}, NULL) = 0 gettimeofday({1349372547, 118844}, NULL) = 0 gettimeofday({1349372547, 118862}, NULL) = 0 gettimeofday({1349372547, 118878}, NULL) = 0 select(9, [3 5 7 8], [], [], {0, 1760}) = 0 (Timeout) gettimeofday({1349372547, 120727}, NULL) = 0 gettimeofday({1349372547, 120745}, NULL) = 0 gettimeofday({1349372547, 120761}, NULL) = 0 gettimeofday({1349372547, 120777}, NULL) = 0 gettimeofday({1349372547, 120793}, NULL) = 0 gettimeofday({1349372547, 120809}, NULL) = 0 send(7, "\1\r\0\0\r\0\0\0\0\0\0\0\307\203\225!\0\0\0\0", 20, 0) = 20 gettimeofday({1349372547, 120884}, NULL) = 0 gettimeofday({1349372547, 120908}, NULL) = 0 gettimeofday({1349372547, 120929}, NULL) = 0 select(9, [3 5 7 8], NULL, NULL, {0, 1}) = 1 (in [7], left {0, 1}) getsockname(7, {sa_family=AF_FILE, path=@""}, [2]) = 0 recv(7, "\1\22\0\0\r\0\0\0\0\0\0\0\307\203\225!\10\0\0\0K9\0\0\0\0\0\0", 65536, 0) = 28
James Pifer wrote:
I'd try strace'ing the app that is supposed to be receiving them to see if the socket opens are working and what happens with a packet arrives on the port.
No idea what this means. snmptrapd keeps running (strace snmptrapd -f -Le -c /etc/snmp/snmptrapd.conf), but I see this over and over after the initial start:
gettimeofday({1349372532, 120897}, NULL) = 0 gettimeofday({1349372532, 120917}, NULL) = 0 gettimeofday({1349372532, 120934}, NULL) = 0 gettimeofday({1349372532, 120950}, NULL) = 0 select(9, [3 5 7 8], [], [], {5, 0}) = 0 (Timeout)
<snip> Do you have ntp running on all the servers?
mark
On 10/4/2012 1:56 PM, m.roth@5-cent.us wrote:
James Pifer wrote:
I'd try strace'ing the app that is supposed to be receiving them to see if the socket opens are working and what happens with a packet arrives on the port.
No idea what this means. snmptrapd keeps running (strace snmptrapd -f -Le -c /etc/snmp/snmptrapd.conf), but I see this over and over after the initial start:
gettimeofday({1349372532, 120897}, NULL) = 0 gettimeofday({1349372532, 120917}, NULL) = 0 gettimeofday({1349372532, 120934}, NULL) = 0 gettimeofday({1349372532, 120950}, NULL) = 0 select(9, [3 5 7 8], [], [], {5, 0}) = 0 (Timeout)
<snip> Do you have ntp running on all the servers?
mark
Not necessarily. SNMP traps are coming from all different kinds of devices. I can't imagine wrong times would mess up snmptrpd. Are you thinking that's what it's having a problem with?
Even if I try to just to a udp forward, with socat, iptables, or a couple specific forwarding apps I tried, nothing seems to get to the apps.
I might just try restarting this server during the night. Maybe something is just hosed.
Any other ideas?
Thanks, James
On Thu, Oct 4, 2012 at 12:45 PM, James Pifer jep@obrien-pifer.com wrote:
No idea what this means. snmptrapd keeps running (strace snmptrapd -f -Le -c /etc/snmp/snmptrapd.conf), but I see this over and over after the initial start:
recv(7, "\1\22\0\0\r\0\0\0\0\0\0\0\307\203\225!\10\0\0\0K9\0\0\0\0\0\0", 65536, 0) = 28
Do you know what it was receiving here?
On 10/4/2012 2:52 PM, Les Mikesell wrote:
On Thu, Oct 4, 2012 at 12:45 PM, James Pifer jep@obrien-pifer.com wrote:
No idea what this means. snmptrapd keeps running (strace snmptrapd -f -Le -c /etc/snmp/snmptrapd.conf), but I see this over and over after the initial start:
recv(7, "\1\22\0\0\r\0\0\0\0\0\0\0\307\203\225!\10\0\0\0K9\0\0\0\0\0\0", 65536, 0) = 28
Do you know what it was receiving here?
Difficult tell as a lot of snmp traps are received.
Reboot didn't help, but modifying my snmpd.conf and adding "master agentx" did the trick. Apparantly snmpd was quietly denying snmptrapd from connecting. Just happened to come across the suggestion.
Now I need to figure out how to have the traps forwarded but retain the real source of the trap.
Thanks for the help.
James
On 10/05/2012 11:53 AM, James Pifer wrote:
Now I need to figure out how to have the traps forwarded but retain the real source of the trap.
If you want to forward the traps without modifying the source address on the UDP packet, you'll need to use iptables. Add a DNAT rule to the PREROUTING chain in the nat table.
On 10/7/2012 4:03 AM, Gordon Messmer wrote:
On 10/05/2012 11:53 AM, James Pifer wrote:
Now I need to figure out how to have the traps forwarded but retain the real source of the trap.
If you want to forward the traps without modifying the source address on the UDP packet, you'll need to use iptables. Add a DNAT rule to the PREROUTING chain in the nat table. _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
I was able to add the following to snmptrad.conf and the source address stays:
format1 '%B [%b]: Trap %#v\n format2 '%B [%b]: Trap %#v\n
Thanks, James
-
Gordon Messmer -
James Pifer -
Les Mikesell -
m.roth＠5-cent.us