[CentOS] Who's eating our bandwidth?

Wed Nov 4 14:40:13 UTC 2009
Brian Mathis <brian.mathis at gmail.com>

On Wed, Nov 4, 2009 at 4:16 AM, Niki Kovacs <contact at kikinovak.net> wrote:
> Hi,
>
> I've recently setup a new server for our public libraries. For the last
> two years, this has been my first "big" job, since it involves
> networking eleven small to medium size public libraries.
>
> There was a hiccup some time ago when the administration hiring me
> wanted to do it on their own, but it took them less than two weeks to
> get the server hacked and lose everything. So they decided to hire me
> back :o)
>
> I've rented a little dedicated server at the french provider Ikoula.
> Really a small thing, a KVM amounting to 1/2 a processor core, 512 MB
> RAM and 25 GB of disk space. Usually there should be no more than like
> ten people working simultaneously on the library management software
> (running atop MySQL).
>
> For the last few days, users reported that the install was "terribly
> slow". I checked, and indeed, the application took quite some time to
> respond.
>
> First thing, I wonder if the configuration I chose is too modest for the
> setup.
>
> Then, I took a peek in /var/log/httpd and the *-access.log files show
> quite some activity. Some haphazard whois on various IP addresses show
> me that these are no library users from around here. Like: Bogota?!?
> Peking?!? And quite some search engines. Since I don't need search
> engines for our application, I'm going to have to find a way to banish
> these.
>
> The log files are not very handy to decipher, so I googled a bit, and I
> think today I'm going to check out AWStats, which seems to be the right
> thing to use in that case.
>
> I'm also wondering about activity on other ports, but here also I'm
> taking stabs in the dark. Probably SSH, but I don't know where eventual
> failed attempts get logged.
>
> I also googled a bit, and I think in this domain, fail2ban will be my
> next experiment.
>
> I have this strange feeling that the next step in the "wise" direction
> consists in describing my ignorance :o)
>
> Any suggestions?
>
> Cheers from the sunny south of France,
>
> Niki

It sounds to me like your server is more attractive to people than the
application you have running on it.  Your apache may be running as an
open proxy, or people might be attempting to use it as one, even if
it's not.

The apache logs files will show what files people are trying to
access.  They might not be easy to read, but as a sysadmin you need to
get used to that, as that is where the information is.  If there are
many files that do not exist on your server, they may be trying to
scan your server or use it as a proxy.

You should also monitor your bandwidth.  Your ISP should have a
control panel that allows you to see your usage.  You should be able
to tell right away if it's very high or not.  Large traffic could mean
someone is using your server to transfer files, or maybe you just have
a very popular service.

You should look at all the logs in /var/log.  /var/log/secure is where
SSH login attempts get sent.  You should also look at the maillog to
see if anyone is using your server to send spam email.