Can any one clarify this, is auto updating at all production servers recommended or not? need to know your opinion, how do you manage the update?
-mu
madunix wrote:
Can any one clarify this, is auto updating at all production servers recommended or not? need to know your opinion, how do you manage the update?
I guess that depends on your situation. For me, if it's a package that I know isn't going to mess with users being logged on, or something of the like, then I don't see it as a problem.
But if it's a package update like samba, or httpd, or something similar, you might want to hold off on auto updates until you can do them manually and restart any services that are needed with it.
I know this doesn't answer your question directly, but I think it's dependent on the packages being updated. Personally I like to restart immediately any services associated with the updates, then you know any and all files that are changed with the update are applied.
Regards, Max
madunix wrote:
Can any one clarify this, is auto updating at all production servers recommended or not? need to know your opinion, how do you manage the update?
I don't, but I watch the centos-announce list to see when important updates are released. Our operations dept schedules all changes on a shared calendar so no one is surprised, although I can't recall a centos update ever causing a problem. After doing a minor update on a non-production machine, I might just 'ssh servername yum -y update' to a list of production servers. For larger sets or point releases, I would first have done a 'yum install yum-downloadonly' on each server so I can 'ssh servername yum -y --downloadonly update' which can run unattended ahead of time, then when it is convenient to babysit the box, complete the update and reboot. For point releases, be sure to read the release notes - there may be special instructions like the need to update glibc* separately for 5.3.
Can any one clarify this, is auto updating at all production servers recommended or not? need to know your opinion, how do you manage the update?
I've worked on projects where backend configuration files changed in syntax or architecture between releases, which were released as updates... so I know for a fact updates can break a running system.
I always disable auto-updates on production systems, and run the updates during a schedule maintenance window.
-geoff
--------------------------------- Geoff Galitz Blankenheim NRW, Germany http://www.galitz.org/ http://german-way.com/blog/
Geoff Galitz wrote:
Can any one clarify this, is auto updating at all production servers recommended or not? need to know your opinion, how do you manage the update?
I've worked on projects where backend configuration files changed in syntax or architecture between releases, which were released as updates... so I know for a fact updates can break a running system.
On CentOS? Fedora does that all the time but _not_ having behavior-changing updates in the long life of a major release is most of the point of 'enterprise' distributions. It's probably not perfect - and I wouldn't do auto-updates on production servers either but it should at least be very unusual for a CentOS update to break anything.
On Thu, 2009-08-13 at 08:53 -0500, Les Mikesell wrote:
Geoff Galitz wrote:
Can any one clarify this, is auto updating at all production servers recommended or not? need to know your opinion, how do you manage the update?
I've worked on projects where backend configuration files changed in syntax or architecture between releases, which were released as updates... so I know for a fact updates can break a running system.
On CentOS? Fedora does that all the time but _not_ having behavior-changing updates in the long life of a major release is most of the point of 'enterprise' distributions. It's probably not perfect - and I wouldn't do auto-updates on production servers either but it should at least be very unusual for a CentOS update to break anything.
I agree. However, my experience with CentOS requires me to state that updates modifying config files is rare, but NOT unknown. Precautions such as testing on non-production systems, should be taken, if only to deal with Murphy's Law.
On Thu, 13 Aug 2009, Les Mikesell wrote:
Geoff Galitz wrote:
...so I know for a fact updates can break a running system.
On CentOS? Fedora does that all the time but _not_ having behavior-changing updates in the long life of a major release is most of the point of 'enterprise' distributions. It's probably not perfect - and I wouldn't do auto-updates on production servers either but it should at least be very unusual for a CentOS update to break anything.
You mean like every time new Apache2 updates come along they _shouldn't_ break my failover cluster???
I've gotten into the habit of doing this each time I see an "httpd" update come down 'cuz a reboot will kill the cluster again:
If I did a reboot I need to edit haresources and restart heartbeat to get my DRBD drive back, then add links back in:
*) Remove "httpd" from the end of /etc/ha.d/haresources *) Restart heartbeat so that it'll mount my DRBD drive. *) Fix up the symlinks in /replicated/etc/httpd/:
logs -> ../../var/log/httpd modules -> /usr/lib64/httpd/modules run -> /var/run
*) Add "httpd" back into the haresources file. *) Restart heartbeat to check that the cluster comes up ok.
"httpd" updates remove the symlinks (that maybe "drbdlinks" creates? Can't recall) which point into my DRBD drive.
If I didn't do a reboot yet then I can fix things up with a subset of the above 'cuz the DRBD drive will still be mounted.
The above is probably necessary because I'm doing something wrong... Do I need to break the cluster before the update so that Apache2 doesn't touch the DRBD drive/drbdlinks stuff, then restart the cluster? That seems likely.
Hi,
On Thu, Aug 13, 2009 at 12:32, Curt Millshacker@fluke.com wrote:
You mean like every time new Apache2 updates come along they _shouldn't_ break my failover cluster???
If you are using a cluster, and configuration files in a shared directory, I believe you should configure your daemon not to use the config files in the system directories such as /etc/httpd and to get them from another path.
In the case of Apache, I think you could fix your issue by removing all the symbolic links in /etc/httpd and leaving it as installed by the RPM, and then editing /etc/sysconfig/httpd and adding something like OPTIONS="-d /replicated/etc/httpd" or OPTIONS="-f /replicated/etc/httpd/conf/httpd.conf" or even both.
That way, RPM upgrades should never touch the config files and symbolic links under /replicated/etc/httpd, and even if they change things under /etc/httpd you wouldn't care about it since you are not using those...
The only think is that a naive sysadmin could try to edit the config files in /etc/httpd but would not see any effects from it, since Apache is not using those at all... But in any case, if you have a cluster you have special procedures to restart and reload Apache anyway, so any sysadmin should know what he's doing before messing with something in your environment... I suggest you work around that by creating some README.txt files in those directories explaining that the "live" configuration files are in /replicated/etc/httpd instead.
This is certainly not a complete procedure on how to configure things so that upgrades don't break your cluster, but I believe the ideas outlined above could lead you there if you set up a test environment and experiment a little bit with it.
HTH, Filipe
On Thu, 13 Aug 2009, Filipe Brandenburger wrote:
This is certainly not a complete procedure on how to configure things so that upgrades don't break your cluster, but I believe the ideas outlined above could lead you there if you set up a test environment and experiment a little bit with it.
Thanks. It makes sense and is easy to implement.
Thanks.
-mu
On Thu, Aug 13, 2009 at 8:39 PM, Curt Millshacker@fluke.com wrote:
On Thu, 13 Aug 2009, Filipe Brandenburger wrote:
This is certainly not a complete procedure on how to configure things so that upgrades don't break your cluster, but I believe the ideas outlined above could lead you there if you set up a test environment and experiment a little bit with it.
Thanks. It makes sense and is easy to implement.
-- Curt Mills, WE7U hacker at fluke dot com Senior Methods Engineer/SysAdmin "Lotto: A tax on people who are bad at math!" "Windows: Microsoft's tax on computer illiterates!" -- WE7U "The world DOES revolve around me: I picked the coordinate system!"
Please be advised that this email may contain confidential information. If you are not the intended recipient, please do not read, copy or re-transmit this email. If you have received this email in error, please notify us by email by replying to the sender and by telephone (call us collect at +1 202-828-0850) and delete this message and any attachments. Thank you in advance for your cooperation and assistance.
In addition, Danaher and its subsidiaries disclaim that the content of this email constitutes an offer to enter into, or the acceptance of, any contract or agreement or any amendment thereto; provided that the foregoing disclaimer does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment to this email. _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Curt Mills wrote:
On Thu, 13 Aug 2009, Filipe Brandenburger wrote:
This is certainly not a complete procedure on how to configure things so that upgrades don't break your cluster, but I believe the ideas outlined above could lead you there if you set up a test environment and experiment a little bit with it.
Thanks. It makes sense and is easy to implement.
Basically, if you edit any config file supplied by an RPM, subsequent updates have to decide whether to keep your old file, possibly missing any changes that the new version should have or replace it, probably breaking whatever you changed. In general, local settings are moved into files under /etc/sysconfig that are merged with the stock config files at runtime when possible. Likewise, httpd configs can usually be done in snippets in /etc/httpd/conf.d to avoid conflict with the main file. Clusters introduce a new problem if you have drbd syncing copies of things that need to be different when one side is updated, though.
madunix ha scritto:
Can any one clarify this, is auto updating at all production servers recommended or not? need to know your opinion, how do you manage the update?
-mu
I'm a very lazy sysadmin, and, although I know that is better to have full control over updates... I let yum-cron do the updates for me.
I administer very few server, I always have a good backup handy, and wen I'm on vacation I disable the auto-updates...
For now (about 3 years of doing this) I never had real issues. The worse thing that happened was the updates not working for some dependency issue, which needed to be sorted manually.
Again, I guess it really depends on what you are administering and if/how you can handle a server outage, so you have to carefully make your own decision based on your environment.
On 8/13/09, madunix madunix@gmail.com wrote:
Can any one clarify this, is auto updating at all production servers recommended or not? need to know your opinion, how do you manage the update?
The NSA Guide to the Secure Configuration of RHEL 5 indicates this is OK, but not with updatesd which they believe is not mature enough for an enterprise environment and may introduce unnecessary overhead. They suggest a cron job that calls yum to do this.
Lanny Marcus wrote:
On 8/13/09, madunix madunix@gmail.com wrote:
Can any one clarify this, is auto updating at all production servers recommended or not? need to know your opinion, how do you manage the update?
The NSA Guide to the Secure Configuration of RHEL 5 indicates this is OK, but not with updatesd which they believe is not mature enough for an enterprise environment and may introduce unnecessary overhead. They suggest a cron job that calls yum to do this.
I build a bunch of these updates and test most of the ones I don't build before we release them ... and I STILL don't auto update servers in production. I may be a bit cautious, but I can't imagine I would ever set any production server, in any OS, to every update automatically.
If I was ever going to do it, CentOS would be the OS ... but with my job on the line I'll just do the updates by hand and watch the output :D
We do auto update all our CentOS Infrastructure servers via cron and they hardly ever have issues.
On Thu, Aug 13, 2009 at 7:59 AM, madunixmadunix@gmail.com wrote:
Can any one clarify this, is auto updating at all production servers recommended or not? need to know your opinion, how do you manage the update?
For a production server, I don't auto-update. There are too many variables involved that may negatively affect the server's functions.
When implementing updates, I prefer to plan ahead, test on a non-critical server, create a change plan that includes a roll-back plan, and then schedule an outage window that allows enough time to revert to the original version if necessary.
If it's a minor update that should not in any way negatively affect a production system, I might skip scheduling an outage window, but I'll still have a roll-back plan.
This may be an overly cautious approach, but it's the one that I use for production systems.
HTH,
M
-
Curt Mills -
Filipe Brandenburger -
Geoff Galitz -
Johnny Hughes -
Lanny Marcus -
Les Mikesell -
Lorenzo Quatrini -
madunix -
Max Hetrick -
Meenoo Shivdasani -
Ron Loftin