Ww just had our switch replaced with a pair of 3750G's, old and new all have 48 ports, so we now have some open ports.... Anyway, my manager was looking at issues yesterday, and discovered that for a while, off and on, from several systems on the new switches, he could see traffic between *other* servers and systems elsewhere in the building... which, of course, shouldn't be possible with a switch.
He tells me that some switches, if they were overwhelmed with traffic, would give up and go into hub mode, but he's under the impression that was written out of the firmware years ago, while these are new switches.
Anyone run into this?
mark
On Wed, Feb 6, 2013 at 10:01 AM, m.roth@5-cent.us wrote:
Ww just had our switch replaced with a pair of 3750G's, old and new all have 48 ports, so we now have some open ports.... Anyway, my manager was looking at issues yesterday, and discovered that for a while, off and on, from several systems on the new switches, he could see traffic between *other* servers and systems elsewhere in the building... which, of course, shouldn't be possible with a switch.
He tells me that some switches, if they were overwhelmed with traffic, would give up and go into hub mode, but he's under the impression that was written out of the firmware years ago, while these are new switches.
Anyone run into this?
A switch will forward to all ports until it learns the mac address (from return traffic) of the correct destination port. So a little bit of traffic leaking to the wrong place within a broadcast domain is fairly normal. A lot means you have a broken switch or one that can't handle the size of the MAC address table it needs. Or you have some strange traffic (udp w/no return packets) or firewalling that keeps the switch from ever seeing the target MAC and restricting the destination to the associated port. Or someone is spoofing the MAC to confuse the switch so they can sniff more than otherwise.
On Wed, Feb 6, 2013 at 11:01 AM, m.roth@5-cent.us wrote:
Ww just had our switch replaced with a pair of 3750G's, old and new all have 48 ports, so we now have some open ports.... Anyway, my manager was looking at issues yesterday, and discovered that for a while, off and on, from several systems on the new switches, he could see traffic between *other* servers and systems elsewhere in the building... which, of course, shouldn't be possible with a switch.
He tells me that some switches, if they were overwhelmed with traffic, would give up and go into hub mode, but he's under the impression that was written out of the firmware years ago, while these are new switches.
I suppose anything is possible, but I've never heard or seen that happen first hand.
If one of your hosts intermittently loses connectivity, the switch will broadcast that traffic to all ports because it can't find the host's MAC address.
(And what Les said about the switch broadcasting traffic until it learns MAC addresses.)
Last week we had a co-located customer participating in a DOS attack and we made the mistake of shutting off the port he was on. Wouldn't you know it that the inbound traffic was broadcast out all operational ports on that switch because the switch couldn't locate the host. That didn't seem to cause any problems, but it did make the switch port graphs pretty. ;)
Anyone run into this?
mark
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Wed, Feb 6, 2013 at 11:19 AM, SilverTip257 silvertip257@gmail.com wrote:
If one of your hosts intermittently loses connectivity, the switch will broadcast that traffic to all ports because it can't find the host's MAC address.
(And what Les said about the switch broadcasting traffic until it learns MAC addresses.)
Some spanning tree events will force the switch to re-learn MACs too.
Les Mikesell wrote:
On Wed, Feb 6, 2013 at 11:19 AM, SilverTip257 silvertip257@gmail.com wrote:
If one of your hosts intermittently loses connectivity, the switch will broadcast that traffic to all ports because it can't find the host's MAC address.
(And what Les said about the switch broadcasting traffic until it learns MAC addresses.)
Some spanning tree events will force the switch to re-learn MACs too.
I should have mentioned this switch is *only* in use on our subnet, though of course we go through it to go Out There, there are gov't firewalls outside of it. All the traffic is only on our subnet, in this case, and the weirdness was intermittent.
At the time, there were two heavy users (me, doing an offline backup, from one room to another, the latter with the server being hit by me in it, and at that switch, and another user doing heave scientific computing). That is, of course, in addition to all the other normal traffic from dozens of other servers.
Btw, he's not seeing it today, but I'm not running any more backups just now....
mark "gigabytes and gigabytes... of backup"
On 06/02/2013 18:34, m.roth@5-cent.us wrote:
Les Mikesell wrote:
On Wed, Feb 6, 2013 at 11:19 AM, SilverTip257 silvertip257@gmail.com wrote:
If one of your hosts intermittently loses connectivity, the switch will broadcast that traffic to all ports because it can't find the host's MAC address.
(And what Les said about the switch broadcasting traffic until it learns MAC addresses.)
Some spanning tree events will force the switch to re-learn MACs too.
I should have mentioned this switch is *only* in use on our subnet, though of course we go through it to go Out There, there are gov't firewalls outside of it. All the traffic is only on our subnet, in this case, and the weirdness was intermittent.
At the time, there were two heavy users (me, doing an offline backup, from one room to another, the latter with the server being hit by me in it, and at that switch, and another user doing heave scientific computing). That is, of course, in addition to all the other normal traffic from dozens of other servers.
Btw, he's not seeing it today, but I'm not running any more backups just now....
You may have some trunking issues if you use VLANs, inter-operate these switches with non-Cisco equipment and have left every port on the switch in the default VLAN1.
Have you actually configured the switch, or did you just plug it in and get running?
Giles Coochey wrote:
On 06/02/2013 18:34, m.roth@5-cent.us wrote:
Les Mikesell wrote:
On Wed, Feb 6, 2013 at 11:19 AM, SilverTip257 silvertip257@gmail.com wrote:
If one of your hosts intermittently loses connectivity, the switch will broadcast that traffic to all ports because it can't find the
host's
MAC address.
(And what Les said about the switch broadcasting traffic until it learns MAC addresses.)
Some spanning tree events will force the switch to re-learn MACs too.
I should have mentioned this switch is *only* in use on our subnet, though of course we go through it to go Out There, there are gov't
firewalls
outside of it. All the traffic is only on our subnet, in this case, and the weirdness was intermittent.
At the time, there were two heavy users (me, doing an offline backup, from one room to another, the latter with the server being hit by me in
it,
and at that switch, and another user doing heave scientific computing).
That
is, of course, in addition to all the other normal traffic from dozens of other servers.
Btw, he's not seeing it today, but I'm not running any more backups just now....
You may have some trunking issues if you use VLANs, inter-operate these switches with non-Cisco equipment and have left every port on the switch in the default VLAN1.
Have you actually configured the switch, or did you just plug it in and get running?
Unfortunately, *we* don't control these switches. They're from the networking division, which actually controls networking throughout the campus. We've had them in, they claim they looked at the switches remotely, and everything's wonderful.... We'll see what happens next week, when I do the next offline backups.
Btw, we are on our own VLAN. The switch is on it.
mark
On Thu, Feb 7, 2013 at 10:53 AM, m.roth@5-cent.us wrote:
Have you actually configured the switch, or did you just plug it in and get running?
Unfortunately, *we* don't control these switches. They're from the networking division, which actually controls networking throughout the campus. We've had them in, they claim they looked at the switches remotely, and everything's wonderful.... We'll see what happens next week, when I do the next offline backups.
Btw, we are on our own VLAN. The switch is on it.
Is the traffic in question to something directly connected to this switch and just appearing mirrored to the wrong port or perhaps broadcast to all of them? Or is the actual destination on some other switch where this one shouldn't even be in the path? If you want to track the problem down you need to look backwards from the target and figure out why the switches in between did not learn the correct path. It is not likely to be related to the traffic bandwidth unless some intermediate link is flooded to the point that nothing works.
Les Mikesell wrote:
On Thu, Feb 7, 2013 at 10:53 AM, m.roth@5-cent.us wrote:
Have you actually configured the switch, or did you just plug it in and get running?
Unfortunately, *we* don't control these switches. They're from the networking division, which actually controls networking throughout the campus. We've had them in, they claim they looked at the switches remotely, and everything's wonderful.... We'll see what happens next week, when I do the next offline backups.
Btw, we are on our own VLAN. The switch is on it.
Is the traffic in question to something directly connected to this switch and just appearing mirrored to the wrong port or perhaps broadcast to all of them? Or is the actual destination on some other switch where this one shouldn't even be in the path? If you want to track the problem down you need to look backwards from the target and figure out why the switches in between did not learn the correct path. It is not likely to be related to the traffic bandwidth unless some intermediate link is flooded to the point that nothing works.
Let's try ASCII art: (campus net)->[vlan]->[new switch in rm. 1]-> server 1 \ -> server 3 ->[switch in rm. 2]->server 2
And he was seeing traffic between 1 and 2 on 3. And he tried another server in rm. 1, and saw it.
Does that make it clearer?
mark
On Thu, Feb 7, 2013 at 1:52 PM, m.roth@5-cent.us wrote:
Btw, we are on our own VLAN. The switch is on it.
Is the traffic in question to something directly connected to this switch and just appearing mirrored to the wrong port or perhaps broadcast to all of them? Or is the actual destination on some other switch where this one shouldn't even be in the path? If you want to track the problem down you need to look backwards from the target and figure out why the switches in between did not learn the correct path. It is not likely to be related to the traffic bandwidth unless some intermediate link is flooded to the point that nothing works.
Let's try ASCII art: (campus net)->[vlan]->[new switch in rm. 1]-> server 1 \ -> server 3 ->[switch in rm. 2]->server 2
And he was seeing traffic between 1 and 2 on 3. And he tried another server in rm. 1, and saw it.
Does that make it clearer?
Do you have a huge number of machines on this network? The switches have to store the whole table of all MACs on each side for the ports and a 3750 should default to default to somewhere between 3K and 12K depending on the configuration. A 'show mac address-table count' on the switch should show the number of active entries and the available space. I've never had to fiddle with that, but there should be commands to tune the size and aging times.
Les Mikesell wrote:
On Thu, Feb 7, 2013 at 1:52 PM, m.roth@5-cent.us wrote:
Btw, we are on our own VLAN. The switch is on it.
Is the traffic in question to something directly connected to this switch and just appearing mirrored to the wrong port or perhaps broadcast to all of them? Or is the actual destination on some other switch where this one shouldn't even be in the path? If you want to track the problem down you need to look backwards from the target and figure out why the switches in between did not learn the correct path. It is not likely to be related to the traffic bandwidth unless some intermediate link is flooded to the point that nothing works.
Let's try ASCII art: (campus net)->[vlan]->[new switch in rm. 1]-> server 1 \ -> server 3 ->[switch in rm. 2]->server 2
And he was seeing traffic between 1 and 2 on 3. And he tried another server in rm. 1, and saw it.
Does that make it clearer?
Do you have a huge number of machines on this network? The switches have to store the whole table of all MACs on each side for the ports and a 3750 should default to default to somewhere between 3K and 12K depending on the configuration. A 'show mac address-table count' on the switch should show the number of active entries and the available space. I've never had to fiddle with that, but there should be commands to tune the size and aging times.
No, not huge numbers. The old switch they replaced was a 48 port, of which *maybe* 2-3 were empty. The new -they've got two of them cabled together (and there is much rejoicing). I don't believe *we* can get on their managed switch. *sigh*
mark
On Thu, Feb 7, 2013 at 2:45 PM, m.roth@5-cent.us wrote:
Let's try ASCII art: (campus net)->[vlan]->[new switch in rm. 1]-> server 1 \ -> server 3 ->[switch in rm. 2]->server 2
And he was seeing traffic between 1 and 2 on 3. And he tried another server in rm. 1, and saw it.
Does that make it clearer?
Do you have a huge number of machines on this network? The switches have to store the whole table of all MACs on each side for the ports and a 3750 should default to default to somewhere between 3K and 12K depending on the configuration. A 'show mac address-table count' on the switch should show the number of active entries and the available space. I've never had to fiddle with that, but there should be commands to tune the size and aging times.
No, not huge numbers. The old switch they replaced was a 48 port, of which *maybe* 2-3 were empty. The new -they've got two of them cabled together (and there is much rejoicing). I don't believe *we* can get on their managed switch. *sigh*
Not just on 'that' switch. It has to learn the MACs of all machines across all interconnected switches across all the VLANs trunked to/through it. They'll age out periodically making the switches broadcast to forgotten/unknown targets but that should get resolved early in the arp process before tcp connections send big packets.
-
Giles Coochey -
Les Mikesell -
m.roth＠5-cent.us -
SilverTip257