There is a CentOS 5.2 machine that is sometimes found to be offline. It runs a few websites but nothing very high traffic. I happened to notice a few days ago that before it went down, one of the sites written in PHP was throwing errors that it could not connect to the MySQL backend. Two hours later, the whole server was down and wasn't even responding to SSH.
It's not my box, but I may have opportunity to look at it. After going through dmesg and messages, if I don't find anything obvious, what should I start looking for? What are the likely, common culprits and how to identify them? Is there a page of the fine manual that addresses issues like this?
Thanks.
Dotan Cohen wrote on 01/23/2012 08:39 AM:
There is a CentOS 5.2 machine ...
I'd have a look at why an apparently Internet-facing server is 5 point releases, plus a lot of subsequent errata, behind the current 5.7 release level; and what resultant vulnerabilities might have been exploited.
Phil
On Mon, Jan 23, 2012 at 16:23, Phil Schaffner Philip.R.Schaffner@nasa.gov wrote:
I'd have a look at why an apparently Internet-facing server is 5 point releases, plus a lot of subsequent errata, behind the current 5.7 release level; and what resultant vulnerabilities might have been exploited.
Thanks. There are a lot of very specific software on that server that precludes it from being updated. I believe that 5.2 still is seeing security updates, no?
In any case, a complete reinstall with either 5.2 or a latter version is pretty much out of the question for now, though I will try to see what needs to be done in that direction. In the meantime, where should I concentrate my efforts?
Thanks.
On 2012-01-23 15:13, Dotan Cohen wrote:
On Mon, Jan 23, 2012 at 16:23, Phil Schaffner Philip.R.Schaffner@nasa.gov wrote:
I'd have a look at why an apparently Internet-facing server is 5 point releases, plus a lot of subsequent errata, behind the current 5.7 release level; and what resultant vulnerabilities might have been exploited.
Thanks. There are a lot of very specific software on that server that precludes it from being updated. I believe that 5.2 still is seeing security updates, no?
In any case, a complete reinstall with either 5.2 or a latter version is pretty much out of the question for now, though I will try to see what needs to be done in that direction. In the meantime, where should I concentrate my efforts?
I think it has been intimated to you that the reason the system has been acting slowly is because it has already been compromised. A system acting in an unresponsive manner is a symptom that it has been compromised.
You may not want to take the system offline, but you cannot trust your system to tell you anything while it is online in a compromised state.
You could take a packet capture of what is going through it's network port (using a SPAN port on the switch), and analyse that for strange port activity.
Otherwise, I would shut it down, make a complete copy of the hard disk having booted off a live or rescue CD and analyse the copy (you can bring the system back up while you analyse the copy, but of course you may put your other systems at risk by doing so).
Thanks, all. I suppose that you all are right, considering that 5.2 is no longer supported. I was under the impression that this is an older but up-to-date install. This server sits in a datacenter hundreds or thousands of kilometers from anyone related to it, so I will back it all up via rsync. Do I risk my home Debian or Fedora boxes by downloading the server's files to them? Of course I won't deliberately execute any files that I download, and I won't be root, but I'd like to know if I need to take any extra precautions.
Thanks!
Hi Dotan,
Le 23/01/2012 17:49, Dotan Cohen a écrit :
Thanks, all. I suppose that you all are right, considering that 5.2 is no longer supported. I was under the impression that this is an older but up-to-date install. This server sits in a datacenter hundreds or thousands of kilometers from anyone related to it, so I will back it all up via rsync. Do I risk my home Debian or Fedora boxes by downloading the server's files to them? Of course I won't deliberately execute any files that I download, and I won't be root, but I'd like to know if I need to take any extra precautions.
Are you really sure it is CentOS 5.2 ? I am very surprised of that, as any 'yum update' would have update to 5.7. And for a public web server, I am surprised that no update at all have been done.
Could you send to us the result of : # cat /etc/redhat-release
For example, it is what I get from a new installed machine (CentOS 6) : $ cat /etc/redhat-release CentOS release 6.2 (Final)
There could be other reasons why a machine becomes irresponsive (sleeping states for example)...
Alain
Dotan Cohen wrote:
On Mon, Jan 23, 2012 at 16:23, Phil Schaffner Philip.R.Schaffner@nasa.gov wrote:
I'd have a look at why an apparently Internet-facing server is 5 point releases, plus a lot of subsequent errata, behind the current 5.7 release level; and what resultant vulnerabilities might have been exploited.
Thanks. There are a lot of very specific software on that server that precludes it from being updated. I believe that 5.2 still is seeing security updates, no?
No.
Dotan Cohen wrote:
On Mon, Jan 23, 2012 at 16:23, Phil Schaffner Philip.R.Schaffner@nasa.gov wrote:
I'd have a look at why an apparently Internet-facing server is 5 point releases, plus a lot of subsequent errata, behind the current 5.7 release level; and what resultant vulnerabilities might have been exploited.
Thanks. There are a lot of very specific software on that server that precludes it from being updated. I believe that 5.2 still is seeing security updates, no?
In any case, a complete reinstall with either 5.2 or a latter version is pretty much out of the question for now, though I will try to see what needs to be done in that direction. In the meantime, where should I concentrate my efforts?
Thanks.
Hi Dotan,
I think that you are mistaken in your belief that 5.2 is still receiving security updates. CentOS 5 is still receiving updates but to benefit from them you have to be at the latest point release, at the moment 5.7.
Regards
ChrisG
On 23-01-12 16:13, Dotan Cohen wrote:
Thanks. There are a lot of very specific software on that server that precludes it from being updated. I believe that 5.2 still is seeing security updates, no?
5.2 does not get security updates. My guess is your box has been compromised. Boot the box with a live CD/DVD and get an image of the harddisk(s) so you can analyze what happened to it.
In any case, a complete reinstall with either 5.2 or a latter version is pretty much out of the question for now, though I will try to see what needs to be done in that direction. In the meantime, where should I concentrate my efforts?
There is no other option than to reinstall the box with 5.7 (or whatever the latest is) and *always* update the box. I would also throw out that "specific software". Vendors who force you to stay with a version of an OS that no longer gets security updates should be avoided at all cost.
Regards, Patrick
On Mon, Jan 23, 2012 at 9:13 AM, Dotan Cohen dotancohen@gmail.com wrote:
On Mon, Jan 23, 2012 at 16:23, Phil Schaffner Philip.R.Schaffner@nasa.gov wrote:
I'd have a look at why an apparently Internet-facing server is 5 point releases, plus a lot of subsequent errata, behind the current 5.7 release level; and what resultant vulnerabilities might have been exploited.
Thanks. There are a lot of very specific software on that server that precludes it from being updated. I believe that 5.2 still is seeing security updates, no?
No, if you were doing updates, you would be at 5.7 now. It you aren't doing updates there are well known exploits against anything earlier than 5.4 or so.
In any case, a complete reinstall with either 5.2 or a latter version is pretty much out of the question for now, though I will try to see what needs to be done in that direction. In the meantime, where should I concentrate my efforts?
First you have to make sure that the tools you are going to use for diagnosis haven't been compromised. An rpm -Va is a first cut at finding files that are changed from the copies distributed. Also, if you have a known-good backup or offline system, run md5sum on netstat, top, ps, lsof, ssh and sshd and compare to the versions on this system. If it is just a software bug, it may be a program not closing files or leaking memory. Netstat or lsof should show open files and connections - if they keep going up, look for the process causing it. Top will show what is using memory. Ps will show the running processes - look for anything you don't expect to be running. If you have mysql running, try 'mysqladmin status' and see if you have many 'slow queries'.
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Dotan Cohen Sent: Monday, January 23, 2012 10:14 To: CentOS mailing list Subject: Re: [CentOS] Machine becoming irresponsive
On Mon, Jan 23, 2012 at 16:23, Phil Schaffner Philip.R.Schaffner@nasa.gov wrote:
I'd have a look at why an apparently Internet-facing server is 5
point
releases, plus a lot of subsequent errata, behind the current 5.7 release level; and what resultant vulnerabilities might have been
exploited.
Thanks. There are a lot of very specific software on that server that precludes it from being updated. I believe that 5.2 still is seeing security updates, no?
No. The minor release 5.2 (exactly) is no longer getting updates, but the major release it is in (5) is up to minor release 5.7. If you are using the CentOS update mechanisms in the standard configured way, then you may already be up to 5.7, but simply do not understand the fact.
"What is the versioning/release scheme of CentOS and how does it compare to the upstream vendor?" https://www.centos.org/modules/smartfaq/faq.php?faqid=34
"How do I get updates for CentOS?" https://www.centos.org/modules/smartfaq/faq.php?faqid=8
The upstream vendor has some information that may help you understand EL OS minor revisions (or point releases): "Red Hat Enterprise Linux Life Cycle" https://access.redhat.com/support/policy/updates/errata/?cid=332371
'"Red Hat Enterprise Linux Compatibility Policies"' https://access.redhat.com/kb/docs/DOC-5155
Too bad I can't find a centos FAQ pointing to these or explaining it at a CentOS level, as I am beginning to think "My ancient version of CentOS is still getting updates, right?" or "I installed X.y of CentOS is it possible to upgrade to X.(y+1) of CentOS without a full reinstall?" are becoming FAQ but keep being stated in slightly different ways.
In any case, a complete reinstall with either 5.2 or a latter version is pretty much out of the question for now, though I will try to see what needs to be done in that direction. In the meantime, where should I concentrate my efforts?
I would suggest investigating the advice of those who are suggesting you look at the possibility of the box already being compromised. (as it is better to _KNOW_ for sure.)
Also understanding the actual version of your system may help you. These three commands may help (if the machine is not compromised, and perhaps even if it is): rpm -qa *release* rpm -qa --last |head uname -r
Studying for the Linux+ exam (or similar) may point you in many directions you have not yet known existed.
On Mon, Jan 23, 2012 at 7:39 AM, Dotan Cohen dotancohen@gmail.com wrote:
It's not my box, but I may have opportunity to look at it. After going through dmesg and messages, if I don't find anything obvious, what should I start looking for?
Forwarding on behalf of Mark whose emails are being rejected:
Patrick Lists wrote:
On 23-01-12 16:13, Dotan Cohen wrote:
<snip>
There is no other option than to reinstall the box with 5.7 (or whatever
the latest is) and *always* update the box. I would also throw out that "specific software". Vendors who force you to stay with a version of an OS that no longer gets security updates should be avoided at all cost.
And, for that matter, if it's in-house software, you managers are going to *have* to bite the bullet and spend the money to have your in-house people fix the software so that it is compatible with current releases.
Beyond that, how do you *know* that it's not compatible with 5.7? Have you tried it on a 5.7 box? At least bring up a 5.7 VM and try it. If you don't have one, you really, really need to have a test system - if your managers have had development and test done on a production box, and they don't understand why this is bad, then I might recommend looking for another job, where they're not amateurs, in the worst sense of the word.
mark
-
Alain Péan -
Chris Geldenhuis -
Denniston, Todd A CIV NAVSURFWARCENDIV Crane -
Dotan Cohen -
Giles Coochey -
John R Pierce -
Les Mikesell -
Nicolas Thierry-Mieg -
Patrick Lists -
Phil Schaffner