We're seeing on a few of our servers - and sometimes it's only occasionally on some of those - where fail2ban's running happily, AFAIK, but there's an attack (from China, Brazil, etc) on ssh, and they don't seem to be banned; I see many, many sorries for wrong username or password.
It *seems* to work again once restarted.
mark
Hello Mark,
On Mon, 2012-08-13 at 09:26 -0400, m.roth@5-cent.us wrote:
We're seeing on a few of our servers - and sometimes it's only occasionally on some of those - where fail2ban's running happily, AFAIK, but there's an attack (from China, Brazil, etc) on ssh, and they don't seem to be banned; I see many, many sorries for wrong username or password.
Known issue: https://bugzilla.redhat.com/show_bug.cgi?id=833056 . Presumably the default notifier pyInotify is bugged so fail2ban will *not* notice logs being rotated and read from the rotated log file (which doesn't receive updates anymore).
You need to set backend=gamin in jail.conf.
Regards, Leonard.
Leonard den Ottolander wrote:
Hello Mark,
On Mon, 2012-08-13 at 09:26 -0400, m.roth@5-cent.us wrote:
We're seeing on a few of our servers - and sometimes it's only occasionally on some of those - where fail2ban's running happily, AFAIK, but there's an attack (from China, Brazil, etc) on ssh, and they don't seem to be banned; I see many, many sorries for wrong username or password.
Known issue: https://bugzilla.redhat.com/show_bug.cgi?id=833056 . Presumably the default notifier pyInotify is bugged so fail2ban will *not* notice logs being rotated and read from the rotated log file (which doesn't receive updates anymore).
You need to set backend=gamin in jail.conf.
Remember reading about that, and on the server I happen to be looking at, it's been set that way since 18 May. Any other ideas?
Btw, I'm updating, even as I type, that server from 6.2 to 6.3.
mark
Hello Mark,
On Mon, 2012-08-13 at 10:48 -0400, m.roth@5-cent.us wrote:
Remember reading about that, and on the server I happen to be looking at, it's been set that way since 18 May. Any other ideas?
The first thing I can think of is you forgot to restart the service after making the configuration change, but then you mentioned restarting the service makes the issue disappear, so that's not it :) .
Perhaps you are seeing machines that only make a single attempt between log rotations? Or perhaps your machines get so many connects that you've hit a resource limit in fail2ban? Just guessing, I haven't had any issues since changing the backend to gamin.
Regards, Leonard.
Leonard den Ottolander wrote:
Hello Mark,
On Mon, 2012-08-13 at 10:48 -0400, m.roth@5-cent.us wrote:
Remember reading about that, and on the server I happen to be looking at, it's been set that way since 18 May. Any other ideas?
The first thing I can think of is you forgot to restart the service after making the configuration change, but then you mentioned restarting the service makes the issue disappear, so that's not it :) .
Perhaps you are seeing machines that only make a single attempt between log rotations? Or perhaps your machines get so many connects that you've hit a resource limit in fail2ban? Just guessing, I haven't had any issues since changing the backend to gamin.
Don't think so. I just dug up one that happened over the weekend: greping out the lines from secure, I've got 131 connection/disconnect pairs: Aug 10 17:44:56 <my server> sshd[12350]: Connection from 114.113.199.142 port 511 871 Aug 10 17:44:57 <my server> sshd[12341]: Received disconnect from 114.113.199.144 2: 11: Bye Bye
Yet in messages, all I see from this server (it happens to be our loghost) for the entire minute of 17:44 are ordinary DHCP messages.
mark
Hello Mark,
On Mon, 2012-08-13 at 11:30 -0400, m.roth@5-cent.us wrote:
Aug 10 17:44:56 <my server> sshd[12350]: Connection from 114.113.199.142 port 511 871 Aug 10 17:44:57 <my server> sshd[12341]: Received disconnect from 114.113.199.144 2: 11: Bye Bye
I'm confused, what exactly is the problem? Fail2ban only triggers on "Failed password from" entries (assuming you use the default filter, and I don't suppose you want it to trigger on disconnects because that would ban legitimate hosts that connect frequently).
The above looks like someone connecting then disconnecting without even attempting a login.
Regards, Leonard.
On Mon, Aug 13, 2012 at 9:01 AM, Leonard den Ottolander leonard@den.ottolander.nl wrote:
Hello Mark,
On Mon, 2012-08-13 at 11:30 -0400, m.roth@5-cent.us wrote:
Aug 10 17:44:56 <my server> sshd[12350]: Connection from 114.113.199.142 port 511 871 Aug 10 17:44:57 <my server> sshd[12341]: Received disconnect from 114.113.199.144 2: 11: Bye Bye
The above looks like someone connecting then disconnecting without even attempting a login.
Some attack programs are too stupid to give up even if they find that password and keyboard interactive authentication is turned off. One kept trying for weeks.
Gé Weijers wrote:
On Mon, Aug 13, 2012 at 9:01 AM, Leonard den Ottolander leonard@den.ottolander.nl wrote:
Hello Mark,
On Mon, 2012-08-13 at 11:30 -0400, m.roth@5-cent.us wrote:
Aug 10 17:44:56 <my server> sshd[12350]: Connection from 114.113.199.142 port 511 871 Aug 10 17:44:57 <my server> sshd[12341]: Received disconnect from 114.113.199.144 2: 11: Bye Bye
The above looks like someone connecting then disconnecting without even attempting a login.
Some attack programs are too stupid to give up even if they find that password and keyboard interactive authentication is turned off. One kept trying for weeks.
Then there are the ones who don't realize that, but are the low-flyers, that spend weeks trying to break in, but at the same time hit slowly, so that they won't be noticed by programs like fail2ban.
mark
Hello Gé,
On Mon, 2012-08-13 at 09:41 -0700, Gé Weijers wrote:
Some attack programs are too stupid to give up even if they find that password and keyboard interactive authentication is turned off. One kept trying for weeks.
Well I guess one *could* filter on the disconnect string to block such IPs, but one has to make sure that legitimate hosts are white listed, or that the findtime is sufficiently small to not let legitimate hosts trigger the rule. White listing is surely the safest approach.
Regards, Leonard.
Leonard den Ottolander wrote:
Hello Gé,
On Mon, 2012-08-13 at 09:41 -0700, Gé Weijers wrote:
Some attack programs are too stupid to give up even if they find that password and keyboard interactive authentication is turned off. One kept trying for weeks.
Well I guess one *could* filter on the disconnect string to block such IPs, but one has to make sure that legitimate hosts are white listed, or that the findtime is sufficiently small to not let legitimate hosts trigger the rule. White listing is surely the safest approach.
Sorry, can't do that with servers whose websites are open to the world, and when folks here have collaborators around the world.
mark
ObDisclaimer: I do not speak for my employer, or for the US federal government agency that I work in. No, it's neither the NSA, nor DOD.
On Mon, 2012-08-13 at 13:30 -0400, m.roth@5-cent.us wrote:
Sorry, can't do that with servers whose websites are open to the world, and when folks here have collaborators around the world.
Well if those people have to log in using SSH from all across the world white listing would not be feasible. But I wouldn't expect you to let those collaborators log in to your log host from anywhere? So white listing and blocking anyone else (on single or repeated connect or disconnect) should be an option there. Although in this scenario you could just as well block port 22 for every but the white listed hosts using iptables, unless you need the ban messages for statistical purposes.
Also, in the default configuration for the ssh-iptables jail the bans resulting from failed SSH logins will only block access to port 22. So even if someone would trigger a ban by attempting too many logins within the set findtime they would still be able to access other services.
Regards, Leonard.
-
Gé Weijers -
Leonard den Ottolander -
m.roth＠5-cent.us