Hey all,
This mail has been a long time in coming, but with the upcoming expiration of security support for Xen 4.8, it's time to start thinking about what our update policy will be for the Xen packages in general.
Citrix is committed to officially supporting one Xen version at a time through the CentOS Virt SIG. (Others in the community are welcome to support others.) But we'd like input as to which version the community would like to be supported at any one time.
Please express your opinion on each option by replying as follows: -2: This is a bad idea, and I would argue against this -1: I'm not happy with this, but I wouldn't argue against this. 0: No opinion. 1: I'm happy with this, but I wouldn't argue for it. 2: This is a great idea, and I'd argue for it.
There are several possible options:
1. Always support the newest option. This means we get all the newest features from Xen in the Virt SIG by default; but also means we get all the newest bugs.
1a. Always support the newest option once it has at least one point release. This balances the newness with a bit of extra testing.
1b. Always support the second-to-newest version (e.g., when 4.13 comes out, switch to 4.12.x)
2. Always support the oldest security-supported version. This means we get the most stable version of Xen; but it does mean it is several years behind as far as features go. It also means that further bugfixes do not happen automatically, and further bugs found will need to be
3. Always support the oldest fully-supported version. Reasonably stable, reasonably old, still gets bugfixes.
4. Support a version until it's out of security support, then jump to the newest version. This minimizes the number of upgrades required (although may make each upgrade more painful).
4a. Support a version until it's out of full support, then jump to the newest version.
Any other options?
For my part, I think 1a, 1b, and 3 are all reasonable options.
-George
On Thu, Nov 28, 2019 at 1:12 PM George Dunlap dunlapg@umich.edu wrote:
Hey all,
This mail has been a long time in coming, but with the upcoming expiration of security support for Xen 4.8, it's time to start thinking about what our update policy will be for the Xen packages in general.
Citrix is committed to officially supporting one Xen version at a time through the CentOS Virt SIG. (Others in the community are welcome to support others.) But we'd like input as to which version the community would like to be supported at any one time.
Please express your opinion on each option by replying as follows: -2: This is a bad idea, and I would argue against this -1: I'm not happy with this, but I wouldn't argue against this. 0: No opinion. 1: I'm happy with this, but I wouldn't argue for it. 2: This is a great idea, and I'd argue for it.
There are several possible options:
- Always support the newest option. This means we get all the newest
features from Xen in the Virt SIG by default; but also means we get all the newest bugs.
1a. Always support the newest option once it has at least one point release. This balances the newness with a bit of extra testing.
1b. Always support the second-to-newest version (e.g., when 4.13 comes out, switch to 4.12.x)
- Always support the oldest security-supported version. This means we
get the most stable version of Xen; but it does mean it is several years behind as far as features go. It also means that further bugfixes do not happen automatically, and further bugs found will need to be
- Always support the oldest fully-supported version. Reasonably
stable, reasonably old, still gets bugfixes.
- Support a version until it's out of security support, then jump to
the newest version. This minimizes the number of upgrades required (although may make each upgrade more painful).
4a. Support a version until it's out of full support, then jump to the newest version.
Any other options?
For my part, I think 1a, 1b, and 3 are all reasonable options.
-George
1a and 1b are good options for me. I currently run 4.12.
Karl
On 11/28/19 12:12 PM, George Dunlap wrote:
Hey all,
This mail has been a long time in coming, but with the upcoming expiration of security support for Xen 4.8, it's time to start thinking about what our update policy will be for the Xen packages in general.
Citrix is committed to officially supporting one Xen version at a time through the CentOS Virt SIG. (Others in the community are welcome to support others.) But we'd like input as to which version the community would like to be supported at any one time.
Please express your opinion on each option by replying as follows: -2: This is a bad idea, and I would argue against this -1: I'm not happy with this, but I wouldn't argue against this. 0: No opinion. 1: I'm happy with this, but I wouldn't argue for it. 2: This is a great idea, and I'd argue for it.
There are several possible options:
- Always support the newest option. This means we get all the newest
features from Xen in the Virt SIG by default; but also means we get all the newest bugs.
1a. Always support the newest option once it has at least one point release. This balances the newness with a bit of extra testing.
1b. Always support the second-to-newest version (e.g., when 4.13 comes out, switch to 4.12.x)
- Always support the oldest security-supported version. This means we
get the most stable version of Xen; but it does mean it is several years behind as far as features go. It also means that further bugfixes do not happen automatically, and further bugs found will need to be
- Always support the oldest fully-supported version. Reasonably
stable, reasonably old, still gets bugfixes.
- Support a version until it's out of security support, then jump to
the newest version. This minimizes the number of upgrades required (although may make each upgrade more painful).
4a. Support a version until it's out of full support, then jump to the newest version.
Any other options?
For my part, I think 1a, 1b, and 3 are all reasonable options.
By supporting only even numbered releases as is the case now, it has not been possible to do hot migration based upgrades which means that we have to do full reboots of our entire environment every so often. Right now we're running on Xen 4.8 and transitioning to 4.12 directly. We skipped 4.10 because we felt that 4.12 has been out and stable for long enough. Ideally if every major build of Xen were provided we would transition by hot migrations up to the next release periodically and stay on a security supported release each time one is going toward EOL.
Personally I would love to see at the very least transitional packages for each Xen version available to allow for easier hot migrations to the latest release, under the assumption that such migrations are considered "supported" upstream. I believe you said this was to be expected in a previous conversation we had on IRC.
I don't really think we should drop a release before its security support ends, unless we have *really clear* communication to repo users as to the life cycles of these builds in advance.
I get why providing updates for 5 major releases concurrently is prohibitive for the entire security support period, though if it were more automated, maybe it would be easier to manage.
I think the keys are making sure that the lifecycles are clearly communicated in advance and that there's a fairly reliable path for people to move up to the latest version that is suitable for production use. So I wouldn't say no to a 1 + 1a + 1b configuration, with the idea that 1 is effectively "testing" to become stable at 1a, then simultaneously always provide 1b as well. That would, by my interpretation mean there are always 2 or 3 supported versions. Right now, 4.12 "stable" and 4.11 "legacy" would be supported. When 4.13 comes out, 4.13 would be "testing" but would be fully maintained with security and point release updates. When 4.13.1 is released it would become "stable," 4.11 would be deprecated and 4.12 would become "legacy."
However, during the transitional period maybe we need to commit to supporting 4.10 until its security support ends.
On 12/2/19 11:08 AM, Kevin Stange wrote:
On 11/28/19 12:12 PM, George Dunlap wrote:
Hey all,
This mail has been a long time in coming, but with the upcoming expiration of security support for Xen 4.8, it's time to start thinking about what our update policy will be for the Xen packages in general.
Citrix is committed to officially supporting one Xen version at a time through the CentOS Virt SIG. (Others in the community are welcome to support others.) But we'd like input as to which version the community would like to be supported at any one time.
Please express your opinion on each option by replying as follows: -2: This is a bad idea, and I would argue against this -1: I'm not happy with this, but I wouldn't argue against this. 0: No opinion. 1: I'm happy with this, but I wouldn't argue for it. 2: This is a great idea, and I'd argue for it.
There are several possible options:
- Always support the newest option. This means we get all the newest
features from Xen in the Virt SIG by default; but also means we get all the newest bugs.
1a. Always support the newest option once it has at least one point release. This balances the newness with a bit of extra testing.
1b. Always support the second-to-newest version (e.g., when 4.13 comes out, switch to 4.12.x)
- Always support the oldest security-supported version. This means we
get the most stable version of Xen; but it does mean it is several years behind as far as features go. It also means that further bugfixes do not happen automatically, and further bugs found will need to be
- Always support the oldest fully-supported version. Reasonably
stable, reasonably old, still gets bugfixes.
- Support a version until it's out of security support, then jump to
the newest version. This minimizes the number of upgrades required (although may make each upgrade more painful).
4a. Support a version until it's out of full support, then jump to the newest version.
Any other options?
For my part, I think 1a, 1b, and 3 are all reasonable options.
By supporting only even numbered releases as is the case now, it has not been possible to do hot migration based upgrades which means that we have to do full reboots of our entire environment every so often. Right now we're running on Xen 4.8 and transitioning to 4.12 directly. We skipped 4.10 because we felt that 4.12 has been out and stable for long enough. Ideally if every major build of Xen were provided we would transition by hot migrations up to the next release periodically and stay on a security supported release each time one is going toward EOL.
Personally I would love to see at the very least transitional packages for each Xen version available to allow for easier hot migrations to the latest release, under the assumption that such migrations are considered "supported" upstream. I believe you said this was to be expected in a previous conversation we had on IRC.
I don't really think we should drop a release before its security support ends, unless we have *really clear* communication to repo users as to the life cycles of these builds in advance.
I get why providing updates for 5 major releases concurrently is prohibitive for the entire security support period, though if it were more automated, maybe it would be easier to manage.
I think the keys are making sure that the lifecycles are clearly communicated in advance and that there's a fairly reliable path for people to move up to the latest version that is suitable for production use. So I wouldn't say no to a 1 + 1a + 1b configuration, with the idea that 1 is effectively "testing" to become stable at 1a, then simultaneously always provide 1b as well. That would, by my interpretation mean there are always 2 or 3 supported versions. Right now, 4.12 "stable" and 4.11 "legacy" would be supported. When 4.13 comes out, 4.13 would be "testing" but would be fully maintained with security and point release updates. When 4.13.1 is released it would become "stable," 4.11 would be deprecated and 4.12 would become "legacy."
However, during the transitional period maybe we need to commit to supporting 4.10 until its security support ends.
I realized I didn't respond in any way to rate the options as requested. I don't really support any configuration that doesn't provide overlapping support for at least two versions simultaneously, so I've added my own options. Sorry!
1: -1 1a: -1 1b: -1 1 + 1a + 1b: +2 2: -1 3: -1 2 + 3: +1 4: -1 4a: -1
On Mon, Dec 2, 2019 at 5:08 PM Kevin Stange kevin@steadfast.net wrote:
By supporting only even numbered releases as is the case now, it has not been possible to do hot migration based upgrades which means that we have to do full reboots of our entire environment every so often. Right now we're running on Xen 4.8 and transitioning to 4.12 directly. We skipped 4.10 because we felt that 4.12 has been out and stable for long enough. Ideally if every major build of Xen were provided we would transition by hot migrations up to the next release periodically and stay on a security supported release each time one is going toward EOL.
Personally I would love to see at the very least transitional packages for each Xen version available to allow for easier hot migrations to the latest release, under the assumption that such migrations are considered "supported" upstream. I believe you said this was to be expected in a previous conversation we had on IRC.
The original purpose of only doing every other release was to make sure that maintenance of the packages wouldn't take too much of our time; but I think at the current state of things, updating ever release should be fine. (Particularly now that we've spread out releases again.)
I don't really think we should drop a release before its security support ends, unless we have *really clear* communication to repo users as to the life cycles of these builds in advance.
Indeed, the purpose of this email is in fact to make such a clear communication. Citrix (i.e., Anthony & I) have committed to providing up-to-date packages for one version at a time; this is meant to give people input into which version that is. The Virt SIG cannot as a whole commit to supporting releases until security support ends unless others step up and make commitments to do so.
I get why providing updates for 5 major releases concurrently is prohibitive for the entire security support period, though if it were more automated, maybe it would be easier to manage.
Indeed; I think the main barrier to having the whole thing automated from xen.git/staging-VV to mirror.centos.org is testing. If we had at least a reasonable subset of automated testing between buildlogs and mirror, I'd feel comfortable with an automated push. Alternately, if people were comfortable with "the community has 1 week to test and object before an automated push happens", we could do that too.
-George
On 12/12/19 8:25 AM, George Dunlap wrote:
On Mon, Dec 2, 2019 at 5:08 PM Kevin Stange kevin@steadfast.net wrote:
I don't really think we should drop a release before its security support ends, unless we have *really clear* communication to repo users as to the life cycles of these builds in advance.
Indeed, the purpose of this email is in fact to make such a clear communication. Citrix (i.e., Anthony & I) have committed to providing up-to-date packages for one version at a time; this is meant to give people input into which version that is. The Virt SIG cannot as a whole commit to supporting releases until security support ends unless others step up and make commitments to do so.
We should probably provide a matrix of which Xen versions are offered by the SIG, who is maintaining them, and when they will last be supported (roughly if it's not 100% clear due to upstream scheduling).
There's a bunch of legacy Xen4CentOS and other confusing Xen docs in the CentOS documentation that need to be cleaned up, removed, or unified.
I'm happy to continue working on and testing releases for whichever branch I'm currently on (which is 4.8 right now, but I'm moving on since upstream is done with security support). However I can't make commitments to support or test versions I am not actively running in production, nor provide specific life cycle guarantees. I made that mistake with 4.4, though meltdown was an extenuating circumstance; I simply couldn't handle that kind of backport myself.
That said, I *may* offer continued support and testing for 4.12 when 4.14's release pushes it out of maintenance by Virt SIG. That's something I'm going to have to play by ear, I guess.
I don't want to burden Steven Haigh any, but I wonder if there's a way we could combine some of our efforts to make both "Xen made easy!" and the Virt SIG Xen easier to manage.
On 2019-12-13 04:54, Kevin Stange wrote:
I don't want to burden Steven Haigh any, but I wonder if there's a way we could combine some of our efforts to make both "Xen made easy!" and the Virt SIG Xen easier to manage.
I've been thinking about this for a while. The problem has been the workflows between myself and the SIG are so far apart, its hard to look at how to merge them.
I have full CI between my own git and packages to the mirrors - which is good, but has its down sides. I also don't have the restrictions of the CentOS build system to deal with - which is great for my workflow ;)
I'm prepping packages for CentOS 8 now - but its exposing quite a number of problems around the main toolsets for Xen - and instead of just adding my own patch and moving on (ala Fedora's Xen packages), I'm trying to get fixes in upstream where possible.
My todo list currently includes: 1) Replace all #! that include env python to a specific python version/binary. ie /usr/bin/python or /usr/bin/python3. The configure portions of this are complete, but the scripts need to be altered to have the detected python version populated.
2) There's no brctl in CentOS 8. The network scripts need to be re-written to use 'ip' instead. To maintain compatibility, the network scripts need to support both brctl and ip commands and use whichever is present on the system.
3) UEFI support needs to be tested. The preferred UEFI boot method for Dom0 is to use grub to then boot Xen - but this needs to be tested.
So while me moving to try and get these fixed upstream is great - it should make everyone's job easier in the long term. Although I have the same problem as the SIG - there's just not enough people to test / patch stuff. The more we deep dive into 4.13, the further away I can see the release happening :(
hi
if you need someone to test the centos8 pkgs, Im interested :)
--- ------ Greetz
Am 13.12.2019 01:31, schrieb Steven Haigh:
On 2019-12-13 04:54, Kevin Stange wrote:
I don't want to burden Steven Haigh any, but I wonder if there's a way we could combine some of our efforts to make both "Xen made easy!" and the Virt SIG Xen easier to manage.
I've been thinking about this for a while. The problem has been the workflows between myself and the SIG are so far apart, its hard to look at how to merge them.
I have full CI between my own git and packages to the mirrors - which is good, but has its down sides. I also don't have the restrictions of the CentOS build system to deal with - which is great for my workflow ;)
I'm prepping packages for CentOS 8 now - but its exposing quite a number of problems around the main toolsets for Xen - and instead of just adding my own patch and moving on (ala Fedora's Xen packages), I'm trying to get fixes in upstream where possible.
My todo list currently includes:
- Replace all #! that include env python to a specific python
version/binary. ie /usr/bin/python or /usr/bin/python3. The configure portions of this are complete, but the scripts need to be altered to have the detected python version populated.
- There's no brctl in CentOS 8. The network scripts need to be
re-written to use 'ip' instead. To maintain compatibility, the network scripts need to support both brctl and ip commands and use whichever is present on the system.
- UEFI support needs to be tested. The preferred UEFI boot method for
Dom0 is to use grub to then boot Xen - but this needs to be tested.
So while me moving to try and get these fixed upstream is great - it should make everyone's job easier in the long term. Although I have the same problem as the SIG - there's just not enough people to test / patch stuff. The more we deep dive into 4.13, the further away I can see the release happening :(
Hi,
On 13/12/2019 01.31, Steven Haigh wrote:
- There's no brctl in CentOS 8. The network scripts need to be
re-written to use 'ip' instead. To maintain compatibility, the network scripts need to support both brctl and ip commands and use whichever is present on the system.
I think the default is to use Network Manager / nmcli? I'm not sure if NM is going to do things with interfaces you created through ip.
regards Nils
On Fri, Dec 13, 2019 at 1:06 PM Nils Meyer nils@nm.cx wrote:
On 13/12/2019 01.31, Steven Haigh wrote:
- There's no brctl in CentOS 8. The network scripts need to be
re-written to use 'ip' instead. To maintain compatibility, the network scripts need to support both brctl and ip commands and use whichever is present on the system.
I think the default is to use Network Manager / nmcli? I'm not sure if NM is going to do things with interfaces you created through ip.
For VMs, these interfaces are created by Xen's toolstack when the VM is started and destroyed by Xen's toolstack when the VM is destroyed; generally speaking all that's needed is for them to be plumbed into the bridge with the right mac address. The guest then does all of its own DHCP &c. As long as that can be done via `ip`, there's nothing NM really needs to do.
-George
On Fri, Dec 13, 2019 at 12:32 AM Steven Haigh netwiz@crc.id.au wrote:
On 2019-12-13 04:54, Kevin Stange wrote:
I don't want to burden Steven Haigh any, but I wonder if there's a way we could combine some of our efforts to make both "Xen made easy!" and the Virt SIG Xen easier to manage.
I've been thinking about this for a while. The problem has been the workflows between myself and the SIG are so far apart, its hard to look at how to merge them.
I have full CI between my own git and packages to the mirrors - which is good, but has its down sides. I also don't have the restrictions of the CentOS build system to deal with - which is great for my workflow ;)
AIUI Anthony's personal build script kicks off a CBS build, runs the result in osstest (the upstream XenProject's CI system), then pushes them to testing.
XenProject's CI system is more focused on interoperability with all the other projects we interact with than on depth of coverage; but that's just about right for the CentOS packages -- only the packaging itself should really need to be tested. Everything else should already have been tested by upstream partners before the release.
IIRC from the last discussion we had, probably one of the more annoyingly sticky issues is dealing with different configurations; in particular different dom0 kernel configurations. Any major changes risks having one of our downstreams suddenly stop working.
-George
On Thu, Nov 28, 2019 at 6:12 PM George Dunlap dunlapg@umich.edu wrote: newest version.
Any other options?
Thanks to everyone who has responded so far. I plan to collect responses on 12 December (2 weeks from when I sent the initial email) and try to make a decision.
-George
On Thu, Nov 28, 2019 at 6:12 PM George Dunlap dunlapg@umich.edu wrote:
Hey all,
This mail has been a long time in coming, but with the upcoming expiration of security support for Xen 4.8, it's time to start thinking about what our update policy will be for the Xen packages in general.
Citrix is committed to officially supporting one Xen version at a time through the CentOS Virt SIG. (Others in the community are welcome to support others.) But we'd like input as to which version the community would like to be supported at any one time.
Please express your opinion on each option by replying as follows: -2: This is a bad idea, and I would argue against this -1: I'm not happy with this, but I wouldn't argue against this. 0: No opinion. 1: I'm happy with this, but I wouldn't argue for it. 2: This is a great idea, and I'd argue for it.
There are several possible options:
- Always support the newest option. This means we get all the newest
features from Xen in the Virt SIG by default; but also means we get all the newest bugs.
1a. Always support the newest option once it has at least one point release. This balances the newness with a bit of extra testing.
1b. Always support the second-to-newest version (e.g., when 4.13 comes out, switch to 4.12.x)
- Always support the oldest security-supported version. This means we
get the most stable version of Xen; but it does mean it is several years behind as far as features go. It also means that further bugfixes do not happen automatically, and further bugs found will need to be
- Always support the oldest fully-supported version. Reasonably
stable, reasonably old, still gets bugfixes.
- Support a version until it's out of security support, then jump to
the newest version. This minimizes the number of upgrades required (although may make each upgrade more painful).
4a. Support a version until it's out of full support, then jump to the newest version.
So the voting results look sort of like this:
1: 0, -2 1a: 1, -1 1b: 1, 2 -> 1 or 1a or 1b: +2
2: 0, -2 3: 0, 2 4: 0, -1, -1 4a: 0, -2, -1
Meaning 1b, "Always support the second-to-newest version" seems to be the best fit.
Since a 4.13 release is imminent, I think we'll probably switch to 4.12 as the default / main supported release once that's out, and then update every release.
-George
-
Christoph -
George Dunlap -
Karl Johnson -
Kevin Stange -
Nils Meyer -
Steven Haigh