Nate:

Thanks for the suggestion. I am looking into it now.

Currently, I use Cacti to graph the data coming from my switches. Do you know if that will that work with any of these switches?

Thanks, Neil

-- Neil Aggarwal, (281)846-8957, www.JAMMConsulting.com Will your e-commerce site go offline if you have a DB server failure, fiber cut, flood, fire, or other disaster? If so, ask about our geographically redundant database system.

-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of nate Sent: Friday, October 23, 2009 8:50 AM To: centos@centos.org Subject: Re: [CentOS] Switch to measure traffic at IP level?

Neil Aggarwal wrote:

...

Hello all:

Is there a network switch that will give me traffic stats at an IP address level?

any of these can: http://www.sflow.org/products/network.php

Myself I'm biased towards Extreme networks having used them for almost 10 years now, very easy to use.

nate

---

CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos

On Fri, Oct 23, 2009 at 4:04 PM, nate centos@linuxpowered.net wrote:

Neil Aggarwal wrote:

...

Nate:

Thanks for the suggestion.  I am looking into it now.

Currently, I use Cacti to graph the data coming from my switches.  Do you know if that will that work with any of these switches?

Cacti will work for port based monitoring, it won't work for flow-based monitoring though.

http://www.sflow.org/products/collectors.php

ntop would probably be the main "free" flow based monitor, though there are some really really nice commercial products.

Inmon has a couple free tools as well sflowtrend, and another command line tool which can dump the contents of sflow data to STDOUT for parsing by a script.

nate

---

Can one setup a Linux server to offer sflow? If the Linux host can run sflow, then it's easy to capture the bandwidth usage on the host? I'm sitting with the same problem, and rely on snmpd on each VPS, but this isn't ideal - especially if clients disable snmpd

http://www.sflow.org/products/collectors.php

ntop would probably be the main "free" flow based monitor, though there are some really really nice commercial products.

I just need something that gives me a usage graph at daily, weekly, and monthly intervals with a 95% line.

It looks like ntop works with RRD so that seems like a good solution.

Thanks for your help!

Neil

-- Neil Aggarwal, (281)846-8957, www.JAMMConsulting.com Will your e-commerce site go offline if you have a DB server failure, fiber cut, flood, fire, or other disaster? If so, ask about our geographically redundant database system.

nate wrote:

Neil Aggarwal wrote:

...

I just need something that gives me a usage graph at daily, weekly, and monthly intervals with a 95% line.

Don't use RRD for billing 95% it will not be accurate as RRD averages values over time. I use rtg (not mrtg, but rtg) together with SNMP to get accurate 95% readings at the interface/port level.

sflow is also a sampling mechanism, but it is pretty accurate depending on the interval, you can see this site for accuracy on sflow:

http://www.sflow.org/packetSamplingBasics/index.htm

If your just using rrd for casual monitoring, no problem but if your using it for billing at 95% then be careful.

RRD stores a certain number of samples at their collected values, then as they age, averages them into larger and larger time values per sample, with the numbers of each set being configurable. If you keep the raw samples for the time span of the calculation you should get accurate values.

Neil Aggarwal wrote:

Hello everyone:

I was just reading an ntop guide and it mentioned many switches have port mirroring.

According to what I am reading, the Cisco I am using will copy all traffic to the mirror port. Then, I can monitor what is going on from there.

That seems like a good way to do this.

Are there any pitfalls with this approach?

yeah, a 1gig port can't handle all the traffic from N 1gig ports. heck, ti can't even handle all the traffic from a single full duplex connection

btw, someone mentioned NTOP... I played with this and found it can consume a LOT of cpu calculating statistics on the fly.

Neil Aggarwal wrote:

...

yeah, a 1gig port can't handle all the traffic from N 1gig ports. heck, ti can't even handle all the traffic from a single full duplex connection

That is a good point. My traffic is light right now so I might be able to use it until the traffic grows.

What kind of internet bandwidth do you have - that's going to be a limiting factor anyway. I've had some trouble keeping ntop running for long intervals but there are ways to database collected results so you could restart it without losing data. I'm not sure if it has a 95th percentile calculation, but it can summarize in a lot of other ways.

Neil Aggarwal wrote:

Are there any pitfalls with this approach?

Performance is the biggest one. Port mirroring often involves the CPU, and is really not built for scaling. If your traffic levels are very low it may work fine. Port mirroring is often a low priority task so if the switch is busy it will drop packets on the mirror to try to ensure availability on the normal ports.

If you have cisco gear they have NetFlow which is similar to sFlow but NetFlow is often a software service so has performance impact as well, depending on the precise equipment your using.

Would ntop be a good tool for it?

Looks like ntop has nProbe which can collect data from a mirrored port, put it in a NetFlow packet and send it to ntop or another collector device.

So it really depends on the scale your operating at, if it's only 1 server with say less than 1Gbit/s of throughput your probably OK. If it's more, sFlow is the only thing that can scale to very high data rates and still be cost effective as it's implemented in the hardware of the switches.

The Extreme X350 for example is a very budget minded gigabit switch, not much layer 3, or stacking, online pricing puts it in the $2000 range for 48 GbE, and has hardware sFlow - http://www.extremenetworks.com/products/summit-x350.aspx

Optional 10GbE (even 10GbaseT for 10GbE over CAT5/6/6a) as well.

Can go to the high end which is roughly triple the price though offers quite a bit more features.

nate

Alternately you could use something like Munin to monitor on the box itself.

I took a look and I think it requires software running on each guest to report the data back to the centralized system. Is that correct?

If so, I am looking for a solution that does not require any software on the guest machines.

Thanks, Neil

-- Neil Aggarwal, (281)846-8957, www.JAMMConsulting.com Will your e-commerce site go offline if you have a DB server failure, fiber cut, flood, fire, or other disaster? If so, ask about our geographically redundant database system.

Matt:

why not just use your current cacti setup and enable snmp on each of the hosts? That seems like the simplest and cheapest approach.

As I understand it, I would actually have to enable snmp on each of the guests, not the hosts.

Am I wrong?

Thanks, Neil

-- Neil Aggarwal, (281)846-8957, www.JAMMConsulting.com Will your e-commerce site go offline if you have a DB server failure, fiber cut, flood, fire, or other disaster? If so, ask about our geographically redundant database system.

Matt:

depending on what your using for virtualization, you might be able to pull

all

of those stats with snmp on the host.

I am using KVM on CentOS 5.4

Let me know if you think it is possible to gather everything I need at the host without requirining anything from the guests.

Thanks, Neil

-- Neil Aggarwal, (281)846-8957, www.JAMMConsulting.com Will your e-commerce site go offline if you have a DB server failure, fiber cut, flood, fire, or other disaster? If so, ask about our geographically redundant database system.

Couple of diff ways to go about this.

I use Cacti and monitor throughput via SNMP for the dom0.

I also enable SNMP on my switch and monitor throughput of individual ports.

However if you want granularity like monitoring at the network layer (IP or layer 3), then you can look at the netflow like plugin (flow- tools and flow-scan) for Cacti and monitor actual traffic patterns of individual hosts.

Or get a netflow or sflow collector in general depending on what your switch supports and enable netflow/sflow just on those ports of interest, like the one your dom0 is on.

On Oct 25, 2009, at 6:14 PM, Mathew S. McCarrell wrote:

On Fri, Oct 23, 2009 at 12:07 PM, Neil Aggarwal <neil@jammconsulting.com

...

wrote:

Matt:

...

depending on what your using for virtualization, you might be able

to pull all

...

of those stats with snmp on the host.

I am using KVM on CentOS 5.4

Let me know if you think it is possible to gather everything I need at the host without requirining anything from the guests.

You might be able to use "virsh domifstat" or something similar.

Matt

-- Mathew S. McCarrell Clarkson University '10

mccarrms@gmail.com mccarrms@clarkson.edu 1-518-314-9214

---

CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos

Neil Aggarwal wrote:

Matt:

...

why not just use your current cacti setup and enable snmp on each of the hosts? That seems like the simplest and cheapest approach.

As I understand it, I would actually have to enable snmp on each of the guests, not the hosts.

Am I wrong?

Yes - the host would see the total traffic. The only other way to separate it would be something upstream (switch/router, etc.) that knows how to do sflow plus a collector device. These are typically pretty expensive. For some small number of guests it might be cheaper to add NICs to your hosts and bridge the guests to individual NICs where you could monitor on either the host interface or the connected switch port. You might be able to simulate this with some clever use of vlans but I'm not sure how they interact with the virtual nic bridges.

I have a CentOS system that is hanging at boot. Sendmail takes forever (and a few other apps hang as well...mainly network apps). This has proven in the pas to be a NIC misconfiguration or a network issue. I think that is what it is on this one too. Is there a way when I see an app haning at boot to make the server stop trying to load the hung app and bring the OS up into the GI so that I get to fixing it? Thanks in advance.

Larry Kemp Network Engineer U.S. Metropolitan Telecom, LLC

On Fri, Oct 23, 2009 at 12:12 PM, Kemp, Larry Larry.Kemp@usmetrotel.comwrote:

I have a CentOS system that is hanging at boot. Sendmail takes forever (and a few other apps hang as well...mainly network apps). This has proven in the pas to be a NIC misconfiguration or a network issue. I think that is what it is on this one too. Is there a way when I see an app haning at boot to make the server stop trying to load the hung app and bring the OS up into the GI so that I get to fixing it? Thanks in advance.

Larry Kemp Network Engineer U.S. Metropolitan Telecom, LLC _______________________________________________

If your having network apps hang, I would take a look at your /etc/hosts file and make sure it is correct. I've had an issue in the past with sendmail hanging during boot and an incorrect /etc/hosts file was the cause.

Matt

-- Mathew S. McCarrell Clarkson University '10

mccarrms@gmail.com mccarrms@clarkson.edu 1-518-314-9214

On Sun, Oct 25, 2009 at 3:23 PM, aurfalien@gmail.com wrote:

During boot, you'll see (for a real brief moment), something to the effect "press I for interactive startup...". A few seconds after pressing it, you will be prompted to load services with a y/n. Once in Ubuntu, I entered rescue mode by entering grub startup options at the command prompt, namely single user mode but I can't recall exactly how I did this  I imagine it would apply to any Linux distro. For me, sendmail and other network services (not NFS though) took forever to load because of fubar'd network stuff.

On Oct 25, 2009, at 1:01 PM, Mathew S. McCarrell wrote:

On Fri, Oct 23, 2009 at 12:12 PM, Kemp, Larry Larry.Kemp@usmetrotel.com wrote:

...

I have a CentOS system that is hanging at boot. Sendmail takes forever (and a few other apps hang as well...mainly network apps). This has proven in the pas to be a NIC misconfiguration or a network issue. I think that is what it is on this one too. Is there a way when I see an app haning at boot to make the server stop trying to load the hung app and bring the OS up into the GI so that I get to fixing it? Thanks in advance.

Larry Kemp Network Engineer U.S. Metropolitan Telecom, LLC _______________________________________________

If your having network apps hang, I would take a look at your /etc/hosts file and make sure it is correct.  I've had an issue in the past with sendmail hanging during boot and an incorrect /etc/hosts file was the cause.

Matt

-- Mathew S. McCarrell Clarkson University '10

mccarrms@gmail.com mccarrms@clarkson.edu 1-518-314-9214 _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos

---

CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos

i seem to recall similar situation and the netplugd helped but in my case it was because the Cat5 cable was unplugged or the switch was powered off. i am not sure why it isn't on by default, maybe NetworkManager was supposed to take over the responsibilities of Netplugd, but clearly failed. ifconfig would say eth0 was UP even though it was not plugged-in. Since netplug daemon has been running, ifconfig hasn't lied again.

IIRC, all i did to turn it on and enable it was, but you may have to yum it down first: chkconfig netplug on

and as it comes back up, hit a key to get the boot prompt, pick the kernel you want to boot, hit 'e', then select the kernel line and hit 'e' to edit and add 'single' to the end of the line, and 'b' to boot it.

Ahh, yes this was it.

A handy thing to remember.