Does anyone know of any sort of Linux utility that does something like what Solaris' Live Upgrade (http://docs.oracle.com/cd/E19455-01/806-7933/index.html) does?
In my past life as a Solaris sys-admin, I found this an extremely useful tool for upgrading and patching running systems, as well as for maintaining redundant boot environments on separate system disks for disaster situations.
----- Original Message ----- | Does anyone know of any sort of Linux utility that does something | like | what Solaris' Live Upgrade | (http://docs.oracle.com/cd/E19455-01/806-7933/index.html) does? | | In my past life as a Solaris sys-admin, I found this an extremely | useful | tool for upgrading and patching running systems, as well as for | maintaining redundant boot environments on separate system disks for | disaster situations. | -- | Tim Evans | 5 Chestnut Court | Linux/UNIX Consulting | Owings Mills, MD 21117 | http://www.tkevans.com/ | 443-394-3864 | http://www.come-here.com/News/ | tkevans@tkevans.com
Nothing really until BTRFS comes of age. I suppose you could snapshot your LVM volumes before performing the upgrade but to my knowledge there is nothing similar to Live Upgrade for CentOS
On 01/28/13 21:27, James A. Peltier wrote:
----- Original Message ----- | Does anyone know of any sort of Linux utility that does something | like | what Solaris' Live Upgrade | (http://docs.oracle.com/cd/E19455-01/806-7933/index.html) does? | | In my past life as a Solaris sys-admin, I found this an extremely | useful | tool for upgrading and patching running systems, as well as for | maintaining redundant boot environments on separate system disks for | disaster situations.
Nothing really until BTRFS comes of age. I suppose you could snapshot your LVM volumes before performing the upgrade but to my knowledge there is nothing similar to Live Upgrade for CentOS
It does sound like you can do the roughly the same with LVM snapshots. Reading the introduction of the solaris document you linked; it seems as if the solaris upgrade is applied on say a snapshot; and then the system is rebooted into the upgraded environment; and if it works, great, if not you need a reboot back into the original state.
Wheras with CentOS 6; you take a snapshot of the root partition (easy as "lvcreate --snapshot --name RootSnapshot --size 2G /dev/VolGroup/Root"), and then do an upgrade with a reboot. If it works; you're set, if not, just revert back to the snapshot (lvconvert --merge VolGroup/RootSnapshot) and reboot; you'd be back to the state before the upgrade.
-xrx
On 01/28/2013 01:05 PM, xrx wrote:
On 01/28/13 21:27, James A. Peltier wrote:
----- Original Message ----- | Does anyone know of any sort of Linux utility that does something | like | what Solaris' Live Upgrade | (http://docs.oracle.com/cd/E19455-01/806-7933/index.html) does? | | In my past life as a Solaris sys-admin, I found this an extremely | useful | tool for upgrading and patching running systems, as well as for | maintaining redundant boot environments on separate system disks for | disaster situations.
Nothing really until BTRFS comes of age. I suppose you could snapshot your LVM volumes before performing the upgrade but to my knowledge there is nothing similar to Live Upgrade for CentOS
It does sound like you can do the roughly the same with LVM snapshots. Reading the introduction of the solaris document you linked; it seems as if the solaris upgrade is applied on say a snapshot; and then the system is rebooted into the upgraded environment; and if it works, great, if not you need a reboot back into the original state.
Wheras with CentOS 6; you take a snapshot of the root partition (easy as "lvcreate --snapshot --name RootSnapshot --size 2G /dev/VolGroup/Root"), and then do an upgrade with a reboot. If it works; you're set, if not, just revert back to the snapshot (lvconvert --merge VolGroup/RootSnapshot) and reboot; you'd be back to the state before the upgrade.
Thanks. You also need to manage the grub and fstab configurations to allow the second boot environment to be visible, bootable, and mountable.
On 01/28/13 22:14, Tim Evans wrote:
On 01/28/2013 01:05 PM, xrx wrote:
On 01/28/13 21:27, James A. Peltier wrote:
----- Original Message ----- | Does anyone know of any sort of Linux utility that does something | like | what Solaris' Live Upgrade | (http://docs.oracle.com/cd/E19455-01/806-7933/index.html) does? | | In my past life as a Solaris sys-admin, I found this an extremely | useful | tool for upgrading and patching running systems, as well as for | maintaining redundant boot environments on separate system disks for | disaster situations.
Nothing really until BTRFS comes of age. I suppose you could snapshot your LVM volumes before performing the upgrade but to my knowledge there is nothing similar to Live Upgrade for CentOS
It does sound like you can do the roughly the same with LVM snapshots. Reading the introduction of the solaris document you linked; it seems as if the solaris upgrade is applied on say a snapshot; and then the system is rebooted into the upgraded environment; and if it works, great, if not you need a reboot back into the original state.
Wheras with CentOS 6; you take a snapshot of the root partition (easy as "lvcreate --snapshot --name RootSnapshot --size 2G /dev/VolGroup/Root"), and then do an upgrade with a reboot. If it works; you're set, if not, just revert back to the snapshot (lvconvert --merge VolGroup/RootSnapshot) and reboot; you'd be back to the state before the upgrade.
Thanks. You also need to manage the grub and fstab configurations to allow the second boot environment to be visible, bootable, and mountable.
Are you talking about CentOS? There is no need to change the fstab or grub; the upgrade gets applied on the main volume (where the OS can be upgraded on the fly without a reboot if it works out; or optionally with a reboot if you want to be extra sure). The snapshot is only there if the update goes bad; in which case you'd run the merge command to revert back to the original state.
-xrx
On 01/28/2013 01:20 PM, xrx wrote:
On 01/28/13 22:14, Tim Evans wrote:
On 01/28/2013 01:05 PM, xrx wrote:
On 01/28/13 21:27, James A. Peltier wrote:
----- Original Message ----- | Does anyone know of any sort of Linux utility that does something | like | what Solaris' Live Upgrade | (http://docs.oracle.com/cd/E19455-01/806-7933/index.html) does? | | In my past life as a Solaris sys-admin, I found this an extremely | useful | tool for upgrading and patching running systems, as well as for | maintaining redundant boot environments on separate system disks for | disaster situations.
Nothing really until BTRFS comes of age. I suppose you could snapshot your LVM volumes before performing the upgrade but to my knowledge there is nothing similar to Live Upgrade for CentOS
It does sound like you can do the roughly the same with LVM snapshots. Reading the introduction of the solaris document you linked; it seems as if the solaris upgrade is applied on say a snapshot; and then the system is rebooted into the upgraded environment; and if it works, great, if not you need a reboot back into the original state.
Wheras with CentOS 6; you take a snapshot of the root partition (easy as "lvcreate --snapshot --name RootSnapshot --size 2G /dev/VolGroup/Root"), and then do an upgrade with a reboot. If it works; you're set, if not, just revert back to the snapshot (lvconvert --merge VolGroup/RootSnapshot) and reboot; you'd be back to the state before the upgrade.
Thanks. You also need to manage the grub and fstab configurations to allow the second boot environment to be visible, bootable, and mountable.
Are you talking about CentOS? There is no need to change the fstab or grub; the upgrade gets applied on the main volume (where the OS can be upgraded on the fly without a reboot if it works out; or optionally with a reboot if you want to be extra sure). The snapshot is only there if the update goes bad; in which case you'd run the merge command to revert back to the original state.
Thanks, again. What you've described is sort of the bass-ackward to what LU does.
It creates one or more "alternate boot environment(s)," then newfs's it, mounts it, copies the running system to it, then applies upgrades/patches to it. It does not touch the running environment (except to create a housekeeping database for the multiple boot environments). The target boot environment is made bootable, and the grub configuration is updated to include the second bootable environment, and the /etc/fstab file on the second environment is modified to reflect the disks/LV's it uses.
The whole idea is not to touch the running system, apply changes to the alternate system, then boot to the alternate when change is done. Once the reboot to the new environment is done, you can further use LU to replace the un-upgraded/un-patched original environment with a copy of the new one.
On 01/28/13 22:54, Tim Evans wrote:
On 01/28/2013 01:20 PM, xrx wrote:
On 01/28/13 22:14, Tim Evans wrote:
On 01/28/2013 01:05 PM, xrx wrote:
On 01/28/13 21:27, James A. Peltier wrote:
----- Original Message ----- | Does anyone know of any sort of Linux utility that does something | like | what Solaris' Live Upgrade | (http://docs.oracle.com/cd/E19455-01/806-7933/index.html) does? | | In my past life as a Solaris sys-admin, I found this an extremely | useful | tool for upgrading and patching running systems, as well as for | maintaining redundant boot environments on separate system disks for | disaster situations.
Nothing really until BTRFS comes of age. I suppose you could snapshot your LVM volumes before performing the upgrade but to my knowledge there is nothing similar to Live Upgrade for CentOS
It does sound like you can do the roughly the same with LVM snapshots. Reading the introduction of the solaris document you linked; it seems as if the solaris upgrade is applied on say a snapshot; and then the system is rebooted into the upgraded environment; and if it works, great, if not you need a reboot back into the original state.
Wheras with CentOS 6; you take a snapshot of the root partition (easy as "lvcreate --snapshot --name RootSnapshot --size 2G /dev/VolGroup/Root"), and then do an upgrade with a reboot. If it works; you're set, if not, just revert back to the snapshot (lvconvert --merge VolGroup/RootSnapshot) and reboot; you'd be back to the state before the upgrade.
Thanks. You also need to manage the grub and fstab configurations to allow the second boot environment to be visible, bootable, and mountable.
Are you talking about CentOS? There is no need to change the fstab or grub; the upgrade gets applied on the main volume (where the OS can be upgraded on the fly without a reboot if it works out; or optionally with a reboot if you want to be extra sure). The snapshot is only there if the update goes bad; in which case you'd run the merge command to revert back to the original state.
Thanks, again. What you've described is sort of the bass-ackward to what LU does.
It creates one or more "alternate boot environment(s)," then newfs's it, mounts it, copies the running system to it, then applies upgrades/patches to it. It does not touch the running environment (except to create a housekeeping database for the multiple boot environments). The target boot environment is made bootable, and the grub configuration is updated to include the second bootable environment, and the /etc/fstab file on the second environment is modified to reflect the disks/LV's it uses.
The whole idea is not to touch the running system, apply changes to the alternate system, then boot to the alternate when change is done. Once the reboot to the new environment is done, you can further use LU to replace the un-upgraded/un-patched original environment with a copy of the new one.
Right, but if the end result is being able to recover from a bad upgrade; they both do the job. In fact, I'd argue that what you described sounds a lot worse (in being complicated, & always requiring a reboot) than what CentOS 6 does for the same use case. If you're going to have downtime in the time required to reboot solaris, try out the upgrade, and reboot again if the upgrade was borked; then not touching the running environment isn't a huge advantage since you'd need to arrange a lot of downtime anyway. With CentOS, instead of the downtime being on a reboot; it will (or rather may) be during/after the update (yum does not take very long). If the update goes bad; then you can go back to the untouched environment with a merge & reboot. So the CentOS method could be a lot faster & easier overall.
Is there any particular reason you prefer LU over snapshots if both can give you the non-updated environment, if required, at the end?
-xrx
On 01/28/2013 06:54 PM, Tim Evans wrote:
It creates one or more "alternate boot environment(s)," then newfs's it,
This is redundant on CentOS
mounts it, copies the running system to it, then applies upgrades/patches to it. It does not touch the running environment
not true, you have severe performance loss and no reliable way to measure the impact of what that changeset is going to bring in. Patching the running system, with a failback option is far better. Unless you dont trust the platform at all. Lack of a reasonable manageable and predictable packaging system can cause that lack of trust.
(except to create a housekeeping database for the multiple boot environments). The target boot environment is made bootable, and the
the idea of bootable on Linux is limited to the kernel and the initrd ( which even has a shell ). Unless you installed a kernel in the transactino set, there is no need to update grub, since your existing running boot components are still intact ( if not, you can measure and handle it ). If you do update the kernel, the setup will retain multiple copies and your last running kernel is always retained as an option to boot from next boot cycle
grub configuration is updated to include the second bootable environment, and the /etc/fstab file on the second environment is modified to reflect the disks/LV's it uses.
Take a look at the yum lvm plugin, it already does most of this - with some assumptions on what its going to snapshot and how you want to handle that.
The whole idea is not to touch the running system, apply changes to the alternate system, then boot to the alternate when change is done. Once the reboot to the new environment is done, you can further use LU to replace the un-upgraded/un-patched original environment with a copy of the new one.
This kind of highlights the issue here, you are attempting to look at this with too far a solaris process mindset. Most CentOS best-practises are driven towards keeping a running system up, by not breaking it or needing to do downtime efforts. Some packages due to their nature will enforce this, like glibc; but on the whole, its considered acceptable to patch live and keep going, with the understanding that there is a failover option / failback opton, rather than assume a running system must be taken down for a patch to be applied.
There are exceptions, but most of them are niche implementation driven ones, on the whole CentOS is built to be patched in place. Take a look at the yum lvm plugin on how you can fairly-easily implement the extra step you mentioned ( i.e tweak fstab to boot from a diff lv / uuid ).
- KB
On 01/28/2013 07:55 PM, Karanbir Singh wrote:
On 01/28/2013 06:54 PM, Tim Evans wrote:
It creates one or more "alternate boot environment(s)," then newfs's it,
This is redundant on CentOS
mounts it, copies the running system to it, then applies upgrades/patches to it. It does not touch the running environment
not true, you have severe performance loss and no reliable way to measure the impact of what that changeset is going to bring in. Patching the running system, with a failback option is far better. Unless you dont trust the platform at all. Lack of a reasonable manageable and predictable packaging system can cause that lack of trust.
(except to create a housekeeping database for the multiple boot environments). The target boot environment is made bootable, and the
the idea of bootable on Linux is limited to the kernel and the initrd ( which even has a shell ). Unless you installed a kernel in the transactino set, there is no need to update grub, since your existing running boot components are still intact ( if not, you can measure and handle it ). If you do update the kernel, the setup will retain multiple copies and your last running kernel is always retained as an option to boot from next boot cycle
grub configuration is updated to include the second bootable environment, and the /etc/fstab file on the second environment is modified to reflect the disks/LV's it uses.
Take a look at the yum lvm plugin, it already does most of this - with some assumptions on what its going to snapshot and how you want to handle that.
The whole idea is not to touch the running system, apply changes to the alternate system, then boot to the alternate when change is done. Once the reboot to the new environment is done, you can further use LU to replace the un-upgraded/un-patched original environment with a copy of the new one.
This kind of highlights the issue here, you are attempting to look at this with too far a solaris process mindset. Most CentOS best-practises are driven towards keeping a running system up, by not breaking it or needing to do downtime efforts. Some packages due to their nature will enforce this, like glibc; but on the whole, its considered acceptable to patch live and keep going, with the understanding that there is a failover option / failback opton, rather than assume a running system must be taken down for a patch to be applied.
There are exceptions, but most of them are niche implementation driven ones, on the whole CentOS is built to be patched in place. Take a look at the yum lvm plugin on how you can fairly-easily implement the extra step you mentioned ( i.e tweak fstab to boot from a diff lv / uuid ).
Thanks to everyone for their replies. I suppose it's not possible in this forum to ask such a question and not get into religion. Kinda like the U.S. Congress.
No one has yet shown how a byte-for-byte, fully redundant, bootable disk(set) can be created and kept up to date that will allow immediate recovery from a catastrophic failure of the primary disk(set) with nothing more than a reboot.
FWIW, Solaris' problems are not technical. Rather, they're Oracle's licensing and support policies that have essentially fired all its small system customers.
Am 29.01.2013 um 20:03 schrieb Tim Evans tkevans@tkevans.com:
Thanks to everyone for their replies. I suppose it's not possible in this forum to ask such a question and not get into religion. Kinda like the U.S. Congress.
No one has yet shown how a byte-for-byte, fully redundant, bootable disk(set) can be created and kept up to date that will allow immediate recovery from a catastrophic failure of the primary disk(set) with nothing more than a reboot.
FWIW, Solaris' problems are not technical. Rather, they're Oracle's licensing and support policies that have essentially fired all its small system customers.
I have to say, in my experience the LU-process works (reasonably) well for systems without zones. Once you have zones, it gets much, much more complicated.
It's a nice idea, though, and to me it makes sense (you don't want the system to be in a state where only half the patches are installed, however minuscule the effects of that may be).
On 1/29/2013 12:03, Tim Evans wrote:
I suppose it's not possible in this forum to ask such a question and not get into religion. Kinda like the U.S. Congress.
Um. Yes. <shakes head to clear the loony>
I know you think you're asking for an additional freedom that you feel CentOS doesn't provide, but what you're really looking for is a reason to go back and shackle yourself to a proprietary technology.
Wise man say: "Free your mind...and your ass will follow." Wise man was on acid when he said that[1] but the point...the POINT I say...is that you're being trapped by a material desire which if satisfied costs more than it's worth when you add in *all* the costs. Kinda like a jelly doughnut. A really big, expensive jelly doughnut.
In addition to the problems with proprietary technology, there's at least one more problem with Solaris' boot environment snapshots: they take space, and space costs money. Oh, you say, it "only" costs a few tens of gigabytes for each system version upgrade. What's the problem in today's terabyte world?
Here's the problem: You want to set up a Solaris VM for development and testing, and read that Solaris needs just 7 GB or so to install[2] everything, so you generously give it 20 GB on your laptop.[3] Life is fine: you install some additional software, you do your dev work, you iterate. Then six months later an OS upgrade comes out, you attempt to install it, it runs you out of disk space creating the snapshot, and then fails to boot. (Yes, actual genuine war story. Made in USA.)
But wait, weren't snapshots supposed to save me from unbootable systems? Sure, as long as you have prodigious free disk space. I don't know about you, but several gigs of free disk on a VM still feels like a lot to me.
You get bit other ways by this feature, too. It requires ZFS, which by most accounts requires 64-bit CPUs and a gig of RAM per terabyte of disk just to manage it. This means that if I give up on my reasonable wish for a 20 GB VM and give it a dedicated HDD or three to play with, now I have to go and set aside beaucoup RAM for it, too. Or maybe a whole system.
I'm not saying that all of this is out of reach. Yes, I get the fact that I'm bitching about under $1000 of hardware. My point is that features have a cost, and different OSes "charge" you different amounts through their designer's choices about which features to include and which to skip for now.
In the end, we are in fact talking rationality here, not religion. How much do you want to pay for those features? TANSTAAFL.
FWIW, Solaris' problems are not technical. Rather, they're Oracle's licensing and support policies that have essentially fired all its small system customers.
That pool is deeper than you realize.
Maybe above you were rolling your eyes at my wish for a usable 20 GB OpenSolaris VM.
Consider this: I have five or six flavors of Linux on that laptop, plus all the BSDs, Windows 8, and the native OS X. When it comes time for me to test a new version of my software, guess which OS it isn't going to get tested on, purely because I can't afford to dedicate a whole system to it? Or if I *am* fortunate enough to be able to afford a dedicated OpenSolaris box, it's a good chance it isn't where I am now, so it might as well not exist.
Do you expect developers like me to multiply the cost by 2, 3, 4x just so we can have a dedicated test box at every development location? Or maybe punch holes in firewalls so we can remotely use a remote box on the back-end somewhere; that isn't "free" either, once you tally the security risks it buys.
Here's why you care about my development problems: if my software doesn't work on your platform, what does it cost you to do the fixes to it to make it work, or seek out an alternative that does work?
How many more are there like me who also might like to test on Solaris occasionally if it were easy, but don't because it isn't? How many packages are there with Solaris portability problems as a result? I'd guess there are hundreds of such packages that you personally would like to use, but either can't, or could if you wanted to put in the time to port them yourself.
All this because Solaris chased the high end at the expense of the smaller systems.
What's happening today to Solaris is simply what happened to all the lesser proprietary Unixes before it. The market's been dissolving from the bottom, and now the mountaintop is dissolving, too.
It's the old UnixWare vs Linux arguments, recapitulated. You remember how that story ended, don't you?
Yes, I know you're switching to Linux now. What I'm trying to get across is that you should take Linux on its own terms, not try to force it into the Solaris mold. It is the way it is for a pile of good, rational reasons.
[1] http://goo.gl/uSIUN [2] http://goo.gl/RnXpW [3] Generous = 10-50% of the free space on a typical laptop SSD.
On 01/29/2013 02:03 PM, Tim Evans wrote:
Thanks to everyone for their replies.
Excellent discussion, Tim. Thanks for bringing this up and coming back with such good arguments.
Some of the objections I've seen on this thread:
1. LiveUpgrade takes too much CPU and disk space. Not if your system is sized appropriately. Simple recovery procedures are worth considerable effort and expense, which is why these features exist in Solaris. Redhat and Fedora aren't quite there yet. But at least people are talking about it here. 2. YUM and various combinations of LVM are faster? The goal isn't to make the overall implementation process faster, although it's been reasonable on most major upgrades to Solaris. The important thing is to make the reboot and resumption of services faster, including back-out. A major upgrade will require a reboot in any case. If you're just doing a minor patch not requiring changes to running processes, you can always use LiveUpgrade to test it out without impacting your running system. 3. There's a Solaris mindset that makes people prefer LiveUpgrade? Not so. AIX and some versions of SVR4 have had ABE support (Teradata comes to mind). I think AIX lets you upgrade running processes in some cases. TMOS with F5 has it. Networking gear often has the ability to upgrade a target volume and rapidly boot off it with the ability to boot back. LiveUpgrade in Solaris is just an implementation of offline upgrading/maintenance.
I see several techniques on this thread with potential to make offline maintenance possible in the near future, or obtain a useful alternative (pardon the pun).
It seems to me that if YUM, RHN, and RPM were designed to work in an alternate root, we'd be 90% done with the project of implementing offline maintenance for Redhat/Fedora. If yum upgrade --root=/a would work, GRUB and /etc/fstab changes could readily be automated with existing tools.
I hope this discussion stays active in Linux circles.
Steve
Tim Evans wrote:
On 01/28/2013 01:05 PM, xrx wrote:
On 01/28/13 21:27, James A. Peltier wrote:
| Does anyone know of any sort of Linux utility that does something | like what Solaris' Live Upgrade | (http://docs.oracle.com/cd/E19455-01/806-7933/index.html) does?
<snip>
Nothing really until BTRFS comes of age. I suppose you could snapshot your LVM volumes before performing the upgrade but to my knowledge there is nothing similar to Live Upgrade for CentOS
It does sound like you can do the roughly the same with LVM snapshots.
<snip>
Wheras with CentOS 6; you take a snapshot of the root partition (easy as "lvcreate --snapshot --name RootSnapshot --size 2G /dev/VolGroup/Root"), and then do an upgrade with a reboot. If it works; you're set, if not, just revert back to the snapshot (lvconvert --merge VolGroup/RootSnapshot) and reboot; you'd be back to the state before the upgrade.
Thanks. You also need to manage the grub and fstab configurations to allow the second boot environment to be visible, bootable, and mountable.
Actually, what we do, such as the upgrade from 5 to 6, is to create /boot/new and /new, rsync from an updated machine, then copy over /etc/sysconfig/network-scripts/ifcfg-eth* to the new, the /etc/ssh to the new, /etc/fstab, and, oh, yes, /etc/grub.d/rules/70-persistant-net.rules; then using zsh's files/modules(?) load that shell into memory, them create /boot/old and /old, move * old, mv lost+found, and /root back, then mv new/* ., Check /etc/grub.conf, and reboot.
If we need to go back - and we did on several, when we found out the speed issues to NFS-mounted home directories - mv * new, mv old/* ., sync, and reboot.
mark mark
-
James A. Peltier -
Karanbir Singh -
m.roth＠5-cent.us -
Rainer Duffner -
Stephen Wynne -
Tim Evans -
Warren Young -
xrx