Hi,
Need some help about this as it's gotten me really concerned.
I'm probably reading too much into this but for about two weeks now my daily log has increased by almost 10 times.
After running through a couple of days of logs with a script, it seems that I'm getting flooded on SMTP from this IP 219.64.114.52 which belongs to VSNL and appears to be statically assigned IP (219.64.114.52.chn.bb-static.vsnl.net). This IP address is apparently listed in the spamhous.org Policy Block List, eXploit Block List and Composite Block List, which basically indicates it's either an open proxy or a hijacked system.
I'm not sure what it's trying to do, but for exactly 10 hours a day which correspond to India 9:30am or so until 7pm or so, I will get massive amounts of SMTP connections from this host. It will attempt to masquerade as domains on my server while trying to send to non-existent accounts on these domains.
2008-08-06 13:32:58 H=(****.com) [219.64.114.52] F=lnyz@hush.com rejected RCPT <484f6f23.8020304@****.com>: 2008-08-06 13:32:58 H=(****.com) [219.64.114.52] incomplete transaction (connection lost) from djclg@hotmail.com 2008-08-06 13:32:58 unexpected disconnection while reading SMTP command from (****.com) [219.64.114.52] 2008-08-06 13:32:58 H=(****.com) [219.64.114.52] F=<48720243.8060909@****.com> rejected RCPT <285.8030501@****.com>: 2008-08-06 13:32:58 H=(****.com) [219.64.114.52] incomplete transaction (connection lost) from lnyz@hush.com 2008-08-06 13:32:58 unexpected disconnection while reading SMTP command from (****.com) [219.64.114.52]
At this point, I thought it was just a case of a dedicated spamming, until I decided I had enough of multi-megabytes daily logs flooding my mailbox, plus the fact it was probably contributing to an increase server load in the past weeks as the mail daemon had to handle the connections.
So I thought I could just block the IP using iptables.
I had a bad experience locking myself out by accident after editing the iptables file so for this time I decided to test from command line first using instructions from the Internet like this
/sbin/iptables -A RH-Firewall-1-INPUT -s 219.64.114.52 -j DROP
and I got an error that chain/command
/sbin/iptables -L produces "blank" output
[root@myserver confused]# /sbin/iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination
Chain FORWARD (policy ACCEPT) target prot opt source destination
Chain OUTPUT (policy ACCEPT) target prot opt source destination
which was of course a shock to me, since that seems to say that my server firewall is basically non-existent.
I did a /sbin/service iptables restart and iptables -L produced the expected output showing all the rules on file. I could then add the new rule from command line without any messages.
Minutes later, my tail -f on the exim log started spewing the smtp messages AGAIN.
iptables -L again shows NO RULES
Everytime I restart, iptables, for a short while, the rules are there. But minutes later, it's wiped. So I'm very concerned that the server had been compromised and something is wiping my iptables.
Or am I just badly mistaken about the way iptables -L is supposed to work?
If not, what should I do next to find and eliminate this problem? Thanks in advance for any advice!
On Wed, Aug 6, 2008 at 7:48 AM, Noob Centos Admin centos.admin@gmail.com wrote:
/sbin/iptables -A RH-Firewall-1-INPUT -s 219.64.114.52 -j DROP
I'd recommend you add the extra rules by editing /etc/sysconfig/iptables instead. At least that way you can be sure they'll survive restarts off iptables.
Next check that the output from '/sbin/chkconfig iptables --list'
looks like this: 'iptables 0:off 1:off 2:on 3:on 4:on 5:on 6:off'
otherwise, do '/sbin/chkconfig iptables on' so you're sure the service starts as intended.
If not, what should I do next to find and eliminate this problem? Thanks in advance for any advice!
Check the crontabs and follow up on the entries. Don't forget to also look in /var/spool/cron/ Are there any strange processes running? What does the logfiles say?
Wait an hour or so, and you'll see more (competent) advice coming in
BR Bent
Bent Terp wrote:
On Wed, Aug 6, 2008 at 7:48 AM, Noob Centos Admin centos.admin@gmail.com wrote:
/sbin/iptables -A RH-Firewall-1-INPUT -s 219.64.114.52 -j DROP
I'd recommend you add the extra rules by editing /etc/sysconfig/iptables instead. At least that way you can be sure they'll survive restarts off iptables.
I rather prefer to add rules using the command and then issuing service iptables save when I'm adding one or two simple rules.. If completely redesigning the firewall or adding in many complex rules, then I edit the iptables file.
More information, after noting the cyclical shutdown of the firewall, I looked into crontab and found a line that stops apf every 5 minutes and directs the output to null.
I cannot copy the exact line now because of my stupidity (good reason why I call myself a noob).
After noting this, which obviously is not a line I entered, which I suspect (wrongly) was injected by some hacker, I removed it. Then proceeded to check apf which was installed by a third party script.
As I noted the comments in the apf.conf, I realized that the autoshutdown of the firewall was due to development settings in the apf.conf file to prevent lockout due to bad firewall configurations. And just as I had the "OH SHIT" thought, my SSH got disconnected and I promptly found myself locked out of the server.
Since I followed some of the rules about SSH and used a non-standard port for SSH and disable SSHD listening on the default port 22, I've no way back into the server and all services on that server are now apparently dead to the way. :(
So I'm now prepping for a long ride to the IDC if a reboot doesn't help my stupidity.
On Wed, Aug 6, 2008 at 8:29 AM, Noob Centos Admin centos.admin@gmail.com wrote:
Since I followed some of the rules about SSH and used a non-standard port for SSH and disable SSHD listening on the default port 22, I've no way back
IMNSHO that's not particularly effective - much better to set up SSH keys and either set 'PermitRootLogin without-password' in /etc/ssh/sshd_config; or set 'PermitRootLogin no', and then su or sudo from your regular user - I know the latter IS more secure, but it's also more annoying to work with....
So I'm now prepping for a long ride to the IDC if a reboot doesn't help my stupidity.
Remember to reinstall from scratch if your server has been compromised - there are thousands of dark dusty corners for the bugs to hide, once they're inside, so don't expect to be able to flush them out.
Good luck! /Bent
On Wed, Aug 6, 2008 at 3:06 PM, Bent Terp bent@terp.se wrote:
On Wed, Aug 6, 2008 at 8:29 AM, Noob Centos Admin centos.admin@gmail.com wrote:
Since I followed some of the rules about SSH and used a non-standard port for SSH and disable SSHD listening on the default port 22, I've no way
back
IMNSHO that's not particularly effective - much better to set up SSH keys and either set 'PermitRootLogin without-password' in /etc/ssh/sshd_config; or set 'PermitRootLogin no', and then su or sudo from your regular user - I know the latter IS more secure, but it's also more annoying to work with....
I did that too, no root login and everytime I have to su from normal user. It is a pain to work with especially with having to use full pathnames for commands instead of say just doing a "service httpd restart". But I figured it was better safe than sorry and as well as I can do since I could not figure out how to properly create a self-sign SSL cert.
Remember to reinstall from scratch if your server has been compromised
- there are thousands of dark dusty corners for the bugs to hide, once
they're inside, so don't expect to be able to flush them out.
Well, the thing is I'm not sure if it's compromised since now it became obvious that the iptables is just being reset by the apf settings.. which is at the moment a good thing since on reboot, apf re-added the lines to disable the firewall every 5 minutes so I'm able to get back into the server.
Now I just have to figure out where exactly can I add the block for the offending VNSL IP address and have it work without choking up. However, I decided to try whatever it is on Saturday so clients won't be hopping mad why everything's dead.
On Wed, 2008-08-06 at 15:14 +0800, Noob Centos Admin wrote:
.. snip
I did that too, no root login and everytime I have to su from normal user. It is a pain to work with especially with having to use full pathnames for commands instead of say just doing a "service httpd restart".
If you use su only, you assume root privileges without the root environment. Rather do su - which gives you the full root environment, including path. The same holds for other users, i..e su - joe switches the user to the user joe with full environment.
Hi,
If you use
su only, you assume root privileges without the root environment. Rather do su - which gives you the full root environment, including path. The same holds for other users, i..e su - joe switches the user to the user joe with full environment.
Thanks a million for that! Going to save me a ton of time from issuing whereis command to find commands when I need to follow instructions off a website!
Noob Centos Admin wrote:
Hi,
If you use
su only, you assume root privileges without the root environment. Rather do su - which gives you the full root environment, including path. The same holds for other users, i..e su - joe switches the user to the user joe with full environment.
Thanks a million for that! Going to save me a ton of time from issuing whereis command to find commands when I need to follow instructions off a website!
Plus we have a Wiki page that explains in some detail here:
http://wiki.centos.org/TipsAndTricks/BecomingRoot
It's a common mistake/misunderstanding most people make to start with :)
If server is not compromised, just edit the smtp configs to deny acceptance from that ip block
Why doesn't the server have an ILO port or something to that effect?
- rh
Hi,
On Wed, Aug 6, 2008 at 3:07 PM, Robert - elists lists07@abbacomm.netwrote:
If server is not compromised, just edit the smtp configs to deny acceptance from that ip block
The EXIM configurations are even more nightmarish than iptables, which at least made some sort of sense. I've been plugging the ip address into the various bad_sender bad_host etc files in the exim configuration directory but it's still not ignoring it. The EXIM smpt/MTA will still accept the connection, then check and realize hey something's not quite right, then issue a reject before the VNSL machine terminates the connection. So the server's still wasting resources handling tens of thousands of such transaction and chewing up log space at the same time.
Hence I have to resort to just blocking from iptables.
Of course, it could very well be my own admitted incompetence that I'm doing something wrong here so Exim is not working the way I expect. I'm very very wary about messing any deeper with the mail settings because a server that's obviously dead to the world is much easier to notice than client emails mysteriously disappearing for days due to bad config before they realize it.
Why doesn't the server have an ILO port or something to that effect?
Well, my boss's a cheapskate and his clients are cheapskate so a couple of years back I was assigned the server administration job on top of my regular day role to setup the server with OTS parts. Hence the half baked setup based on a tight budget and whatever information I can glean from the internet and the good folks on forums and mailing lists.
So for the ILO? Well, only today did the term enter my mind. Although I did vaguely remember suggestions for a remote reboot button but it was beyond my know how to setup.
A possible remote reboot can be setup from a on that server obscure web page URL to a privileged script that is password protected
Inexpensive reset button
- rh
Thanks Steward and Robert for those suggestions, they make plenty of sense!.
About the two SSH terminal, if I activate a wrong firewall change that blocks the SSH port, would it not also terminate the existing terminals since new packets going in would be rejected, or does it not affect already established TCP connections?
Probably also going to make a script to shutdown the firewall as well as one for reboot. Since so far all 3 times my noobness involves firewalling myself out, although in a slightly different way each time!
From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Noob Centos Admin Sent: Wednesday, August 06, 2008 5:31 PM To: CentOS mailing list Subject: Re: [CentOS] Help: Server security compromised?
Thanks Steward and Robert for those suggestions, they make plenty of sense!.
About the two SSH terminal, if I activate a wrong firewall change that blocks the SSH port, would it not also terminate the existing terminals since new packets going in would be rejected, or does it not affect already established TCP connections?
Probably also going to make a script to shutdown the firewall as well as one for reboot. Since so far all 3 times my noobness involves firewalling myself out, although in a slightly different way each time!
Seen this?
http://www.askbjoernhansen.com/2007/09/18/safely_change_firewall_rules_remot... y.html
On Thu, Aug 7, 2008 at 1:54 AM, Sorin Srbu sorin.srbu@gmail.com wrote:
Seen this?
http://www.askbjoernhansen.com/2007/09/18/safely_change_firewall_rules_remot...
Unfortunately, only after you pointed it out :( But thankfully whoever wrote APF apparently knows this, hence it does insert an automatic reset of the firewall after 5 minutes.
_____
From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Noob Centos Admin Sent: Thursday, August 07, 2008 5:17 AM To: CentOS mailing list Subject: Re: [CentOS] Help: Server security compromised?
On Thu, Aug 7, 2008 at 1:54 AM, Sorin Srbu sorin.srbu@gmail.com wrote:
Seen this?
http://www.askbjoernhansen.com/2007/09/18/safely_change_firewall_rules_remot... y.html
Unfortunately, only after you pointed it out :( But thankfully whoever wrote APF apparently knows this, hence it does insert an automatic reset of the firewall after 5 minutes.
No worries, we've all been there at one time or another. Consider it a learning experience. 8-)
Noob Centos Admin wrote:
On Thu, Aug 7, 2008 at 1:54 AM, Sorin Srbu <sorin.srbu@gmail.com mailto:sorin.srbu@gmail.com> wrote:
Seen this? http://www.askbjoernhansen.com/2007/09/18/safely_change_firewall_rules_remotely.htmlUnfortunately, only after you pointed it out :( But thankfully whoever wrote APF apparently knows this, hence it does insert an automatic reset of the firewall after 5 minutes
Hi,
My US$0.02 on this.....I'm a fan of apf as a front-end to iptables...but it takes some reading to understand the switches and the entire RAB (reactive address blocking) configuration options. Sadly, RAB is poorly documented, but with a bit of tinkering, I've enjoyed this feature tremendously as it cuts down on the hammering I used to get to port 22 by the bots and script kiddies.
If you've a static IP at your workstation, add your IP address to the apf nicely formed 'allow_hosts.rules' file, usually located in /etc/apf. This is a simple IP address or IP block list (using slash notation, i.e. 192.168.1.0/24) to allow access to an IP or range of IPs. Further, the deny_hosts.rules list is the same format for hosts to always deny.
/usr/local/sbin/apf -a <ip address || ip block> will add to the allow list *and* flush and reload the iptables back-end so you don't have to restart apf
likewise /usr/local/sbin/apf -d <ip address || ip block> will add to the deny list *and* flush and reload the iptables back-end so you don't have to restart apf
Once the firewall is configured properly, set DEVEL to 0 in the conf.apf file, then restart apf. The authors rightly include DEVEL mode which crons a shutdown every 5 mins so you're not locked out for long. Trust me, I've been bitten by this (more than I care to admit)
There are other CLI switches, all well documented on the apf site (http://rfxnetworks.com/apf.php) http://rfxnetworks.com/appdocs/README.apf
HTH, -Ray
On Thu, Aug 7, 2008 at 11:53 PM, Ray Leventhal centos@swhi.net wrote:
My US$0.02 on this.....I'm a fan of apf as a front-end to iptables...but it takes some reading to understand the switches and the entire RAB (reactive address blocking) configuration options. Sadly, RAB is poorly documented, but with a bit of tinkering, I've enjoyed this feature tremendously as it cuts down on the hammering I used to get to port 22 by the bots and script kiddies.
Sad to say my usual tasks keep me sufficiently occupied that I hardly have the time to study what APF actually does. It came with ELS (Easy Linux Security) scripts with directadmin, sounds like A Good Idea (tm) so I just installed it. Personally I'm aghast at the manner in which I'm running the server but practically there is only that much time I can devote to being the server admin.
If you've a static IP at your workstation, add your IP address to the apf
nicely formed 'allow_hosts.rules' file, usually located in /etc/apf. This is a simple IP address or IP block list (using slash notation, i.e. 192.168.1.0/24) to allow access to an IP or range of IPs. Further, the deny_hosts.rules list is the same format for hosts to always deny.
I had considered this allowed only x.x.x.x ip strategy very early on since it appeared to be an obvious way to head off attacks/probes from external parties. Unfortunately, like most folks, I'm on dynamic IP. My primary role also requires me to run around very often, necessitating urgent administration from a variety of potential sub-networks from whichever ISP happens to be providing access at the location. So I figured it would be quite impractical to attempt to limit access to only certain IP addresses.
Although thinking about it now, extending the concept from a previous suggestion, I suppose it is theoretically possible to write a privileged script accessible from one of the server hosted domains to activate an allow-host rule addition to the firewall and a cronjob that routinely activates another script to removed added hosts after 1 hour or something. So anytime access is needed, I would hit the website to activate the script to open up SSH access to the IP I am using at the moment and then SSH in.
But of course, easier said than done since I barely know shell scripting and allowing exec in PHP had always been met with a big frown personally. :D
About the two SSH terminal, if I activate a wrong firewall change that blocks the SSH port, would it not also terminate the existing terminals since new packets going in would be rejected, or does it not affect already established TCP connections?
It depends upon what you are doing and in which order you do it. Unfortunately, I'm not an expert in iptables - I refuse to spend time learning more than the basics on it, since I don't like it. IMO the structure and rules are byzantine and unnecessarily flexible/complex, so when fiddling about with the firewall, usually its just simple commands to open/close ports or do connection limiting/throttling, and I don't ever touch port 22.
FWIW, when doing a complex task, instead of typing in commands in a terminal, I begin writing a script to do those commands. At the very same time I develop a 'rollback' script to undo those commands in case of error. Experimenting on a Centos 5.2 server which I have console access. Upon an error condition, I then modify the script, play the 'rollback' script, and reissue the script. So gradually the script and its 'antidote' are built to where I'm satisfied they work. Then and only then, do I use that script on production and remote servers which are also running CentOS 5.2 The only problem which my method is that getting these scripts to be 100% correct even in the face of malevolent conditions such as DNS timeouts and hardware errors makes them 2-3 times as long and yukky and hard to read.
Hi, the more completely you lock down a server, the harder it will be for you to do some useful work on it. These matters require a balance between security and ease-of-use for the admins. Its especially important not to cut your bridges when administering a remote server.
Despite many people advising to use keys and change ports etc etc, you really only need to do 3 things to stop dead any unauthorised SSH logins: 1. prevent direct root logins 2. create a user account (just for SSH logins) with an unusual name and give that account a very good password (20 character alphanumeric). Only allow that user to login via SSH. 3. give root a password of similar complexity.
Doing just these three will ensure that not only will no-one ever be likely to get in via SSH, but you will be able to SSH in from anywhere from any computer.
Furthermore, when doing any work with firewalls or ssh on a remote server, you must *always* have more than one SSH shell open. Don't close the last shell until you have tested your changes and are confident you won't lock yourself out.
-
Bent Terp -
D Steward -
Hendrik Strydom -
Morten Nilsen -
Ned Slider -
Noob Centos Admin -
Ray Leventhal -
Robert - elists -
Sorin Srbu -
Sorin＠Gmail