Philip Amadeo Saeli wrote:
Thank you all for the helpful and informative replies. However, I have some additional questions (interspersed below).
For some background, the organization I'm doing this for is a significantly resource constrained, very small company, so I have been having to take carefully measured steps in upgrading their systems and bringing them into conformance. In all cases, the systems were set up by others prior to my time with them.
In particular, it would better, given their constraints, if I could get their CentOS 4 system up to standards prior to migrating to a CentOS 5 system (which I'd already proposed to them). Even migrating to CentOS 5 doesn't, by itself, solve my problem (see comments below).
- Bob McConnell rmcconne@lightlink.com [2010-11-07 07:50:42 -0500]:
- Once you finish tweaking the configuration, test all of your
software, web pages, etc.
- Do a complete acceptance test on the production server. (We actually
use a second Internet facing server for acceptance tests before committing changes to the production server.)
How does one set up a test [web] server which has a number of sites, all secured via SSL certs which are bound to the domain, and hence the IP address, of the sites on the server? They do have a developmental server which uses one, company-issued SSL cert to secure all of the test sites. However, the Apache config for this is substantially different than that for the production server.
It appears that I'd have to set up an additional, special testing DNS space with new IP addressess, or to enter them into the hosts file(s) of the web client test systems. Also, I would not be able to simply copy the httpd config file(s) from the production system to the test system due to having to have different IP addresses for each site. Or is there some other way to do this? I'm really stumped over this one.
It's probably not necessary to use the same certificates, as long as those you use are the same type and format. When we need to test to that level, we actually set up our own CA to generate test certificates for internal use only. We're testing for the functionality, not the specific certificates. Having said that, most of our in-house testing is done without SSL. Even servers that are normally part of a VPN are only tested within a LAN rather than on the VPN. Most of that functionality comes from third party FLOSS applications, so we are rather confident that any problems will be fixed before we run across them.
- Use YUM to update your test server at least once a week.
- As soon as you finish testing all of the updates each week, use YUM
to install them on the production server. (But don't ever do this on Friday. If you missed something, you don't want to have to work on the weekend.) 9. Subscribe to announcements and several security mailing lists to get advanced warning of any known issues that need to be patched immediately. 10. Start tracking RedHat/CentOS 6 release candidates ASAP.
Officially, by PCI rules we have 30 days after release of an OS update to get it installed on Internet facing systems. So the auditors will give us one pass on their monthly validation cycle before they start to complain. This does give us some time to test for problems and correct them before updating the production servers. But this requires a test server that is configured exactly like the production server so we can make sure the updates won't break any of our applications before we will install them in production.
We have one developer from each product team, one QA manager, one Support tech and an IT tech that track these issues and make sure our servers are up to date. As one of the developers in that group, I monitor CentOS announcements and two security lists, forwarding relevant messages to the entire group. There is a similar but larger group tracking Microsoft updates.
In addition to CentOS and Apache, we also track updates to PHP, PostgreSQL and a couple dozen supporting packages and maintenance tools.
Bob McConnell N2SPP
Bob, your thoughtful, insightful, informative, and detailed reply is very helpful. Thanks! My biggest hangup WRT the above is exactly how to set up a test server that very closely mirrors the production server without needing to have to maintain significant configuration changes, esp WRT an SSL-secured web server.
How close the mirror has to be is a question only you can answer. But you have to be practical. What are you actually testing? If you have your own software that is providing SSL and other system level capabilities, then yes, you need to include them on your test system. If you are looking at a mix of releases that are not in sync with each other, yes you may need to test a little more thoroughly. But in most cases you will likely be using software that is part of the OS distribution, or what other developers have targeted for that OS. If you can't trust them and their community to provide you with reliable tools, why are you using them in the first place?
On a related tack:
- Robert Heller heller@deepsoft.com [2010-11-07 07:13:27 -0500]:
Red Hat backports security updates (from newer versions). So long as you have been applying the standard O/S updates (eg 'yum update') regularly, your http is up-to-date WRT security updates.
Even with CentOS 5, the available versions of, especially PHP and MySQL, are not up-to-date enough feature-wise for the web site code on the system, so later versions have to be used. The httpd version is not so critical in this case. How can one cope with this situation, i.e., the general situation of needing features not supported under a current version of an OS so the vendor supplied updates are no longer applicable.
In some cases I've been able to find 3rd party repos which have had the RPMs I've needed (e.g., rpmforge, dag, epel, remi) but the ones needed are not always available (as is the case now). I have set up repos and built my own RPMs in the past, but there is no budget for that in this case. Also, it is a lot of work, especially for a very few systems.
--Phil [former N2IWR]
Bob McConnell N2SPP