We have an issue with some customers who refuse to accept ICMP traffic to their mail servers. It seems that they have put Mordac, preventer of information services in charge of their firewall policy (http://en.wikipedia.org/wiki/List_of_minor_characters_in_Dilbert#Mordac).
My mail logs are showing that customers who specifically disallow ICMP traffic have many "Connection Reset" entries in our logs:
Oct 14 08:00:50 mailsrv sendmail[2024]: m9ED0Yf5002021: to=customername@customer.org, delay=00:00:16, xdelay=00:00:16, mailer=esmtp, pri=42476, relay=mail.customer.org. [XX.XX.XX.XX], dsn=4.0.0, stat=Deferred: Connection reset by mail.customer.org.
I have disabled pmtu discovery on our routers as well as on all our outbound mail servers. Is there anything else I can do on our side to help the situation?
Sean Carolan wrote:
We have an issue with some customers who refuse to accept ICMP traffic to their mail servers. It seems that they have put Mordac, preventer of information services in charge of their firewall policy (http://en.wikipedia.org/wiki/List_of_minor_characters_in_Dilbert#Mordac).
BUT ICMP IS BAD!!!!!¡¡¡¡¡
My mail logs are showing that customers who specifically disallow ICMP traffic have many "Connection Reset" entries in our logs:
Oct 14 08:00:50 mailsrv sendmail[2024]: m9ED0Yf5002021: to=customername@customer.org, delay=00:00:16, xdelay=00:00:16, mailer=esmtp, pri=42476, relay=mail.customer.org. [XX.XX.XX.XX], dsn=4.0.0, stat=Deferred: Connection reset by mail.customer.org.
I have disabled pmtu discovery on our routers as well as on all our outbound mail servers. Is there anything else I can do on our side to help the situation?
So you basically broke your internet connection because of stupid customers? No, there isn't anything you can do on your side - especially if you don't know how large their MTU is set (which you cannot discover, as they forbid you to do so). So you can only hope that you get exactly the same MTU as they have (and that there is nothing inbetween which has a lower MTU).
It is their problem. If they don't want to play by the rules, they should have to sit out the problems they themselves created.
Ralph
on 10-14-2008 6:24 AM Ralph Angenendt spake the following:
Sean Carolan wrote:
We have an issue with some customers who refuse to accept ICMP traffic to their mail servers. It seems that they have put Mordac, preventer of information services in charge of their firewall policy (http://en.wikipedia.org/wiki/List_of_minor_characters_in_Dilbert#Mordac).
BUT ICMP IS BAD!!!!!¡¡¡¡¡
My mail logs are showing that customers who specifically disallow ICMP traffic have many "Connection Reset" entries in our logs:
Oct 14 08:00:50 mailsrv sendmail[2024]: m9ED0Yf5002021: to=customername@customer.org, delay=00:00:16, xdelay=00:00:16, mailer=esmtp, pri=42476, relay=mail.customer.org. [XX.XX.XX.XX], dsn=4.0.0, stat=Deferred: Connection reset by mail.customer.org.
I have disabled pmtu discovery on our routers as well as on all our outbound mail servers. Is there anything else I can do on our side to help the situation?
So you basically broke your internet connection because of stupid customers? No, there isn't anything you can do on your side - especially if you don't know how large their MTU is set (which you cannot discover, as they forbid you to do so). So you can only hope that you get exactly the same MTU as they have (and that there is nothing inbetween which has a lower MTU).
It is their problem. If they don't want to play by the rules, they should have to sit out the problems they themselves created.
Sometimes you can't be so hard headed when you are dealing with customers. You usually are trying to get them to give money to YOU, not your competitor.
If I told my customers that "It is your problem", I would no longer have customers to worry about!
Scott Silva wrote:
on 10-14-2008 6:24 AM Ralph Angenendt spake the following:
So you basically broke your internet connection because of stupid customers? No, there isn't anything you can do on your side - especially if you don't know how large their MTU is set (which you cannot discover, as they forbid you to do so). So you can only hope that you get exactly the same MTU as they have (and that there is nothing inbetween which has a lower MTU).
It is their problem. If they don't want to play by the rules, they should have to sit out the problems they themselves created.
Sometimes you can't be so hard headed when you are dealing with customers. You usually are trying to get them to give money to YOU, not your competitor.
If I told my customers that "It is your problem", I would no longer have customers to worry about!
But your competitor wouldn't be able to send them mails either >:)
As said, they deliberately broke their internet connection, so there isn't much you can do except setting your MTU to an extremely low value and hope that there's nothing in between which has an even lower MTU.
So your best choice would be to do some consulting and give them some advice on what they did wrong and how they can selectively block ICMP types (for example redirect and such).
Cheers,
Ralph
Thanks for the information. If I understand this correctly, the client would have to convince the owner of each and every router hop along the way to disable PMTU discovery if he insists on dropping all ICMP packets?
And Scott hit the nail on the head with this comment:
Sometimes you can't be so hard headed when you are dealing with customers. You usually are trying to get them to give money to YOU, not your competitor.
If I told my customers that "It is your problem", I would no longer have customers to worry about!
If you've ever dealt with with one of these paranoid Mordac-type security managers you know exactly what I'm talking about. In our case the path of least resistance was to disable pmtu discovery, and tell the customer that we've done all we possibly can to alleviate the issue on our end. Hopefully they come to their senses and allow ICMP packets like every major ISP and mail provider on the Internet.
On Oct 14, 2008, at 1:59 PM, Sean Carolan wrote:
If you've ever dealt with with one of these paranoid Mordac-type security managers you know exactly what I'm talking about. In our case the path of least resistance was to disable pmtu discovery, and tell the customer that we've done all we possibly can to alleviate the issue on our end. Hopefully they come to their senses and allow ICMP packets like every major ISP and mail provider on the Internet.
Yes, but then you have broken your equipment, and possibly lost the ability to communicate with many more customers.
Yes, I've dealt with these people. If they turn off all ICMP, they often drop fragments as well, making the problem even worse. You can sometimes get them to listen by asking them if their Internet access seems a little "weird" in that some sites work sometimes or downloads are slow or they can't get some email :-)
They'll usually say yes and then you might be able to get them to listen, and hopefully send them a bill.
--Chris
Yes, I've dealt with these people. If they turn off all ICMP, they often drop fragments as well, making the problem even worse. You can sometimes get them to listen by asking them if their Internet access seems a little "weird" in that some sites work sometimes or downloads are slow or they can't get some email :-)
What do you do when the receiving server is at Microsoft? (mx4.hotmail.com)
Oct 28 11:18:25 exmx2 sendmail[19190]: m9SGHXL4019186: to=some.user@hotmail.com, delay=00:00:52, xdelay=00:00:52, mailer=esmtp, pri=43523, relay=mx4.hotmail.com. [65.54.244.232], dsn=4.0.0, stat=Deferred: Connection reset by mx4.hotmail.com.
I'm not surprised that Microsoft would make such a boneheaded decision as to block all ICMP traffic to their hotmail servers, however would this not cause a ton of customer email to simply be lost? Do they just not care? I guess at this point my only choice may be to lower my MTU size.
Ralph Angenendt wrote:
As said, they deliberately broke their internet connection, so there isn't much you can do except setting your MTU to an extremely low value and hope that there's nothing in between which has an even lower MTU.
It doesn't have to be extremely low, it just has to be low enough. The usual reason for needing to be less than the 1500 bytes permitted by ethernet would be using some sort of tunnel protocol for PPOE or a VPN. 1460 might keep everybody happy.
Les Mikesell wrote:
Ralph Angenendt wrote:
As said, they deliberately broke their internet connection, so there isn't much you can do except setting your MTU to an extremely low value and hope that there's nothing in between which has an even lower MTU.
It doesn't have to be extremely low, it just has to be low enough. The usual reason for needing to be less than the 1500 bytes permitted by ethernet would be using some sort of tunnel protocol for PPOE or a VPN. 1460 might keep everybody happy.
Might being the operative word here, yes.
Ralph
Sean Carolan wrote on Tue, 14 Oct 2008 08:13:34 -0500:
My mail logs are showing that customers who specifically disallow ICMP traffic have many "Connection Reset" entries in our logs:
Could somebody explain why ICMP might play a role in mail delivery?
Kai
Kai Schaetzl wrote:
Sean Carolan wrote on Tue, 14 Oct 2008 08:13:34 -0500:
My mail logs are showing that customers who specifically disallow ICMP traffic have many "Connection Reset" entries in our logs:
Could somebody explain why ICMP might play a role in mail delivery?
It doesn't really. If the OP had PMTU discovery turned on it would affect most all communications not just email. I can't ever remember having it on for external networks, there's never been a need in my case.
It's just likely that the only communications between the OP's systems and the other side was email.
nate
On Tue, October 14, 2008 09:31, Kai Schaetzl wrote:
Sean Carolan wrote on Tue, 14 Oct 2008 08:13:34 -0500:
My mail logs are showing that customers who specifically disallow ICMP traffic have many "Connection Reset" entries in our logs:
Could somebody explain why ICMP might play a role in mail delivery?
ICMP is involved in IP routing, including MTU discovery, announcing failed connections, and so forth. Email is delivered over IP. QED.
On 2008-10-14 16:31, Kai Schaetzl wrote:
Sean Carolan wrote on Tue, 14 Oct 2008 08:13:34 -0500:
My mail logs are showing that customers who specifically disallow ICMP traffic have many "Connection Reset" entries in our logs:
Could somebody explain why ICMP might play a role in mail delivery?
Any host may reply to a IP-datagram (tcp included) with e.g. ICMP type 3, code 4 "datagram too large" and indicating the maximum size in the ICMP reply.
Disallowing these ICMP packets can result in a TCP handshake that succeeds, but hangs when the next packets with real data are blocked.
http://en.wikipedia.org/wiki/PMTUD
Kai Schaetzl wrote:
My mail logs are showing that customers who specifically disallow ICMP traffic have many "Connection Reset" entries in our logs:
Could somebody explain why ICMP might play a role in mail delivery?
It is required for any TCP conversation, unless as a matter of luck you happen to have the same MTU capability across the whole path - or the end points restrict their MTUs arbitrarily to a size that everything can handle.
Kai Schaetzl wrote:
Sean Carolan wrote on Tue, 14 Oct 2008 08:13:34 -0500:
My mail logs are showing that customers who specifically disallow ICMP traffic have many "Connection Reset" entries in our logs:
Could somebody explain why ICMP might play a role in mail delivery?
If you don't know the smallest MTU on the path to the mail server, you might not be able to send packets over that path, especially if DF is set.
Ralph
Ralph Angenendt wrote on Tue, 14 Oct 2008 17:24:08 +0200:
If you don't know the smallest MTU on the path to the mail server, you might not be able to send packets over that path, especially if DF is set.
But if it's not set? Shouldn't most devices have it not set?
Kai
Kai Schaetzl wrote:
Ralph Angenendt wrote on Tue, 14 Oct 2008 17:24:08 +0200:
If you don't know the smallest MTU on the path to the mail server, you might not be able to send packets over that path, especially if DF is set.
But if it's not set? Shouldn't most devices have it not set?
Routers should fragment as needed and the receiving stack will reassemble. Windows tends to set DF on a lot of packets unnecessarily.
On Tue, October 14, 2008 12:31, Kai Schaetzl wrote:
Ralph Angenendt wrote on Tue, 14 Oct 2008 17:24:08 +0200:
If you don't know the smallest MTU on the path to the mail server, you might not be able to send packets over that path, especially if DF is set.
But if it's not set? Shouldn't most devices have it not set?
Yes, most devices should fragment if necessary (DF not set).
Most devices should also pass/accept ICMP messages relating to their connections. Deliberately configuring them not to is asking for trouble; those messages are part of the protocol for a reason.
(Fragmentation introduces more work and effectively many more lost packets in most setups, so the flow will be jumpy and less efficient even if it mostly works.)
Kai Schaetzl wrote:
Ralph Angenendt wrote on Tue, 14 Oct 2008 17:24:08 +0200:
If you don't know the smallest MTU on the path to the mail server, you might not be able to send packets over that path, especially if DF is set.
But if it's not set? Shouldn't most devices have it not set?
Fragmentation is bad. That's why you do PMTUD - to see which is the lowest MTU in the path. You then set your packet sizes accordingly and set the DF bit. If the lowest MTU in the path changes to an even lower one you get an error and can continue with smaller packet sizes.
If you disallow PMTUD - well, you're asking for trouble >:)
http://www.znep.com/~marcs/mtu/ has a rather good discussion about that.
Ralph
Kai Schaetzl a écrit :
Ralph Angenendt wrote on Tue, 14 Oct 2008 17:24:08 +0200:
If you don't know the smallest MTU on the path to the mail server, you might not be able to send packets over that path, especially if DF is set.
But if it's not set? Shouldn't most devices have it not set?
It's not per device. It's a method to improve performances. http://www.znep.com/~marcs/mtu/
Sean Carolan a écrit :
We have an issue with some customers who refuse to accept ICMP traffic to their mail servers. It seems that they have put Mordac, preventer of information services in charge of their firewall policy (http://en.wikipedia.org/wiki/List_of_minor_characters_in_Dilbert#Mordac).
My mail logs are showing that customers who specifically disallow ICMP traffic have many "Connection Reset" entries in our logs:
Oct 14 08:00:50 mailsrv sendmail[2024]: m9ED0Yf5002021: to=customername@customer.org, delay=00:00:16, xdelay=00:00:16, mailer=esmtp, pri=42476, relay=mail.customer.org. [XX.XX.XX.XX], dsn=4.0.0, stat=Deferred: Connection reset by mail.customer.org.
I have disabled pmtu discovery on our routers as well as on all our outbound mail servers. Is there anything else I can do on our side to help the situation?
Consider setting a small MTU (or MSS, ....) for the borked networks instead of changing your setup globally. something like
ip route add 192.0.2.0/24 via 10.0.0.1 mtu 1000
-
Chris Boyd -
David Dyer-Bennet -
Kai Schaetzl -
Les Mikesell -
mouss -
nate -
Paul Bijnens -
Ralph Angenendt -
Scott Silva -
Sean Carolan