Our network consists of aaa.bbb.ccc.0/19.  That's CIDR notation for 8,192 addresses.

But what has that got to do with "www.yahoo.com moved into our /19" .... your comment is pretty unclear.

IMHO, fully updated purpose-built servers running 4.8 should have more or less the same vulnerablity profile as 5.5 IFF RH is doing a good job of backporting security fixes.

Why are you so sure it was a bind issue? What logs/research has come to that conclusion?

Would bind 9.7 really have helped you if you were hacked or was your vulnerability elsewhere - and if so where? Was this the same server that you posted where you had mangled the install with force reinstalling rpms from SL and/or oracle that you posted about before for instance?

I am supported in that statement by my mentor at FedEx but NOT by my mentor at Internet2.

Your mentor? What do you mean by that?

We installed openssl-1.0.0c Jan 23 20:30 27 minutes after filing the original post IIRC.

If you were so gung ho about security that you wanted bleeding edge bind even newer than current centos 5 why are you so out of date on your openssl libraries. Given that you are out of date on those as per your previous posts would the currently released bind on rhel5 iff it was already on c5 really have been installed? If you were that desperate you could have built the srpms yourself.... or taken 9.7 from c5-testing.

You have posted the same rubbish over and over without any substantiation with wild allegations.

Post details if you need help or just please stop ranting to no point.

James

Joe, Randy and James are my mentors of 15, 5 and 5 years, respectively, and all said the same thing, namely "nuke and repave, be sure to be current on BIND" since it is a purpose-built box (ns1).

Perhaps is it a difference in language and what you mean by mentor and where I would mean old colleague/peer who I have discussed this with.

They have stated their opinions and you can follow that - but then you would be diverging from the point of RHEL somewhat with a custom built BIND.

Remember that the version number you see on BIND is not always the equivalent of upstream due to backports. You should check the relevant RHEL errata, the package %changelog and CVE to get a better understanding of what exploits are known and what has been patched.

With 20/20 hindsight, it is clear that I shouldn't have posted the original post asking the list for help and hopefully informing other potential targets of the risk (read: there were no responses to the original post, therefore it was posted to the wrong audience).

Err... this isn't the whole story/truth.

I just searched your emails on this list. the first reference to bind was the 16th feb with the thread "Blasphemous" with complaints (and no substance) to Redhat not having current Bind - despite the fact 9.7 is in the then released 5.6... you suggested an alt repo "for critical internet functions." No where did you indicate you had a name server hacked/altered/poisoned... although you pointed out your credit card prcessing system was running Redhat linux 7.3 (Valhalla) and was nearing 10 years old.... this from someone complaining about teh 'age'' of BIND in RHEL/CentOS.

There was no time for forensics at the time of the discovery; just time to get advice and react.

Then you have no way of telling what happened. For future reference a better reaction is to isolate the server (whether physical or virtual) and put a new system in place to serve the need for it whilst you analyze what happened to the previous. Without that knowledge you cannot mitigate any issues or discover where the failure was, if any.

What follows is from a few moments ago.

===details=== ===box was last nuked and repaved Jul 28  2008 ===much unnecessary software removed Jul 28 2008, 57 tasks active per 'ps auxw | wc -l'

This is irrelevant to the point at hand.

===current nmap (same nmap results as on problem day) Starting Nmap 5.21 ( http://nmap.org ) at 2011-02-18 18:38 CST Note: Host seems down. If it is really up, but blocking our ping probes, try -PN Nmap done: 1 IP address (0 hosts up) scanned in 0.19 seconds vaden@turtlehill:/opt$ nmap -A -PN ns1.texoma.net Starting Nmap 5.21 ( http://nmap.org ) at 2011-02-18 18:38 CST Nmap scan report for ns1.texoma.net (209.151.96.2) Host is up (0.0012s latency). Not shown: 998 filtered ports PORT    STATE SERVICE VERSION 53/tcp  open  domain 987/tcp open  ssh     OpenSSH 3.9p1 (protocol 2.0) | ssh-hostkey: 1024 36:dc:c8:29:b1:d3:8a:b1:e6:cf:2b:4c:70:ed:c8:9a (DSA) |_1024 10:f9:a6:d2:32:68:15:3a:9f:04:3a:89:05:1e:b8:52 (RSA) Service detection performed. Please report any incorrect results at http://nmap.org/submit/ . Nmap done: 1 IP address (1 host up) scanned in 26.44 seconds

So you have SSH exposed and Domain requests exposed. Not surprising but irrelevant in and of itself.

vaden@turtlehill:/opt$ ===named.conf security in 2008 [root@ns1 data]# cat /var/named/chroot/etc/named.conf | more ### # #  attribution: By Rob Thomas, noc at cymru.com #               http://www.cymru.com/Documents/secure-bind-template.html #  -and- # http://www.knowplace.org/pages/howtos/split_view_with_bind_9_howto.php # #  at the behest of #  Dr. Joe Redacted (redacted1.edu)

#  Dr. Randall Redacted (redacted2.edu)

Without adequate details such as whether IP requests were limited to your allotted IP addresses and other config details this doesn't help.

ssh port not on 22

This is fundamentally irrelevant. This is a very visible server given it is a primary nameserver for you. A simple nmap as you showed above presents any potential hacker with the correct port for SSH given a targeted attack.

distro's standard iptables save ssh port

Perhaps here you made a security mistake and should have configured it differently - for example limiting connection attempts, set up fail2ban, limit inbound SSH from known IPs for management purposes from your corporate network, not had SSH publically visable, etc. Without more detail it is impossible to say what went wrong and how the system could be potentially secured.

If you have a specific point of vulnerability you have encountered - whether a known CVE or not - I would urge you to open a bugzilla ticket with reproducible steps.

If you got hacked through poor configuration and monitoring then it's your own fault quite frankly and perhaps for something you see as such a key service you should hire a proper admin and pay for a Redhat license so you have an SLA to full back on for bugs. That is what it is there for.

Regardless of the above I urge you to stop posting irrelevant nonsense and pushing the signal to noise ratio of the list to such intolerable levels.

James

On Saturday, February 19, 2011 12:57:40 am Larry Vaden wrote:

Through this experience, starting with a hacked or poisoned name server, or, quite frankly, the perception of one, I have learned what people really see.

Having a server hacked is one of the worst things that can happen in IT; not of course as bad as a real heart attack, for sure.

Having a server hacked puts you in a wierd mindset, most certainly.

If your server was really hacked, I'd start from scratch, and set the new one up more defensively.

On Sat, Feb 19, 2011 at 04:37:57PM -0600, Larry Vaden wrote:

THANKS for your input; there exists a consensus, so that's what will be done (replace 4.8 with 5.x). Troy says Fermi (a great target for the miscreants/actors) runs stock BIND as it appears in RHEL/CentOS/SL, so, to be redundant, what is good enough for the national labs is good enough for us. Joe won't be happy with us, but we need to be pragmatic.

Just out of curiosity, have you ever thought about hiring a competent admin?

John

On Fri, Feb 18, 2011 at 7:39 PM, James Hogarth james.hogarth@gmail.com wrote:

...

Joe, Randy and James are my mentors of 15, 5 and 5 years, respectively, and all said the same thing, namely "nuke and repave, be sure to be current on BIND" since it is a purpose-built box (ns1).

Perhaps is it a difference in language and what you mean by mentor and where I would mean old colleague/peer who I have discussed this with.

Wikipedia says "This is the source of the modern use of the word mentor: a trusted friend, counselor or teacher, usually a more experienced person." I am not their peer; they are my mentors. They have been invaluable over the 25 combined years of mentorship to this rural ISP.

Remember that the version number you see on BIND is not always the equivalent of upstream due to backports. You should check the relevant RHEL errata, the package %changelog and CVE to get a better understanding of what exploits are known and what has been patched.

Johnny has remarked on the importance of trust.

My trust in RedHat went down when I learned they are not shipping all the SRPMs. Some say it is due to human error. If that is the case, why should I think they are better at backporting security fixes than at making sure a manifest of SRPMs is complete and correct?

On Sat, Feb 19, 2011 at 6:58 AM, James Hogarth james.hogarth@gmail.com wrote:

For the record an archive of all mails sent to the mailing list appears here... in this case in date order.

http://lists.centos.org/pipermail/centos/2011-January/date.html

To give you the benefit of the doubt I search there for your mail by subject and by the datestamp you allege.

It is not there - ergo it never arrived at the list.

OR it was silently dropped ... only the list manager and others in the know would know if mail from former subscribers is silently dropped.

On Saturday, February 19, 2011 01:51:55 am Larry Vaden wrote:

My trust in RedHat went down when I learned they are not shipping all the SRPMs. Some say it is due to human error. If that is the case, why should I think they are better at backporting security fixes than at making sure a manifest of SRPMs is complete and correct?

To be fair to Red Hat, it might be different people doing the backporting than are responsible for the packaging. Might not, but might be.

And for their purposes a missing build requirement package isn't really a bug, since it builds fine for them, and they get the patched package out to their customers. And their customers won't typically be rebuilding from source RPM. So, like in any other job, the less important tasks and issues go to the bottom of the list, while the more important 'get the deliverable to the customer' takes top spot.

They have finite resources; they're going to use those finite resources frugally, and thus stay in business (which everybody using CentOS should want them to do).

On Fri, 2011-02-18 at 18:32 -0500, Lamar Owen wrote:

On Friday, February 18, 2011 04:15:28 pm Always Learning wrote:

...

Don't understand what you mean by 'within our /19'.

I think I do; he's an ISP, and apparently someone inside his address block ... has hacked in some way the zone file(s) or the cache for his nameserver so that his customers, who would ordinarily use his DNS server as their recursive resolver, now see www.yahoo.com (among who knows what others) as pointing to a different address ....

Thank you for explaining Larry had his DNS servers hacked or poisoned.

.... to prevent such things I would recommend to Larry that he use the great iptables tools that CentOS provides ... ... to restrict the addresses that can actually ssh into his server, and only allow port 53 UDP and TCP traffic into and out of his DNS servers to his customers.

Agreed. IPtables is a very useful tool to block unauthorised accesses in and (heaven forbid) out of one's servers. Every server is screwed down to the barest minimum and every port that can be changed from its default is. No servers share the same non-standard port numbers. SSH access is limited to 3 static IP addresses. Aggressive blocking with IPtables can prevent a lot of time wasting aggro.

I also ban some Chinese blocks and even more Taiwan blocks from port 80 to reduce web hacking and lots of Taiwanese blocks from port 25.