Hi,
This email is the amalgamation of many conversations that have been running in various venues for the last couple of months. I am posting this here for comments, as we move towards implementing the process ( which has been validated with builds and releases already, the glue components are what we are working on now ).
If you are seeing this somewhere other than CentOS-Devel mailing list, I highly encourage you to join that list for comments and to track progress of this effort.
------
At this point were looking at building two sets of atomic machine instances:
a) the 4 weekly cycle that fits into the 2/4 model from upstream where Fedora runs a 2 weekly cycle, CentOS Atomic SIG runs the 4weekly downstream from there. The actual process by which CentOS Atomic SIG maps to every alternate Fedora build is a TBH point, but there needs to be some specific map there in the short term.
b) the downstream build, where we consume the content delivered via the RHEL Atomic Host content, and deliver a similar functional story. Due to the nature of how the atomic builds work, we wont have an identical build, but we should get to a close enough feature compatibility.
any atomic platform is backed by an ostree repo, which is composed of the rpms and any other content needed for the atomic platform to run on. it would be to a atomic host what a yum repo is to a rpm based CentOS Linux install.
For (a) above, I propose we run an ostree repo, managed by a centos-release-projectatomic rpm file; the ostree repo for this should be delivered from buildlogs.centos.org, and updated in sync with the rest of the content in the CentOS Linux repos, as well as the atomic sig run repos in the community build service. In order to keep things simple, it might be good enough to run this ostree update on a nightly basis.
I propose this nightly build be run from an automated harness at ci.centos.org, which would pull down the last ostree repo, do its updates, commit the changeset, and have a way to push back into the buildlogs.centos.org host the new repo. As an additional downstream trigger, we should then build nightly the images derived from this ostree repo ( this would be the GenericCloud, Vagrant, ISO files etc for the atomic build that maps to the latest commit in that ostree repo). For numbering, I propose we use datestamp with a build id eg: 20150715_01, this maps well to user expectations and also sets suitable confidence in the build.
Key requirement to applying updates is that the atomic host needs a reboot to consume the latest ostree propagated changes. So while the updated tree would be available nightly, uses might chose to still apply it nightly, they can control to a fairly good granularity the point in time they wish to run.
The atomic sig can then decide which point in time build they wish to promote as their 4wk 'release' image. The ostree repo + images + documentation for that build can then be promoted to a release location. This would then be announced via projectatomic.io
I'm also proposing that this build be called the project atomic build.
for (b) above, I'm proposing we use the same process as for (a), except for the following major changes : - content in (a) is signed, so while we can build things nightly, we cant promote that content as is. When the Atomic SIG decides on a good tree to promote, the entire buildrun will then be done, against the SIG's accepted metadata, inside the CentOS Build Services ( reimzul ). Resulting in a testable, signed atomic content sit ( ostree repo + updates + images, all signed ). The SIG can then chose to redo, rework, release as needed.
- content for (b) will be released to mirror.centos.org (ostree repos), and cloud.centos.org ( backing instance images ). the centos-release-atomic will point at mirror.centos.org's ostree repo.
- the update cycle for the ostree repo should be undecided, so we can promote a new update if there are security issues that merit this work, and also when there is a new version of a core component that we might want to push out out of band ( i.e not wait on the next upstream build )
- numbering for (b) should be date driven as well, to look similar as YYYYMMDD_BuildId
- The release cycle for this should be documented in an 'Atomic Host' section on wiki.centos.org/Download, and announced via the generic centos announcement channels.
- This release would be called the CentOS Atomic Host.
regards,
ref: * https://lists.fedoraproject.org/pipermail/cloud/2015-March/004976.html
* http://www.projectatomic.io/blog/2015/06/centos-atomic-host-rebuild-released...
Thanks for posting this!
On Wed, Jul 15, 2015, at 06:18 AM, Karanbir Singh wrote:
For (a) above, I propose we run an ostree repo, managed by a centos-release-projectatomic rpm file; the ostree repo for this should be delivered from buildlogs.centos.org, and updated in sync with the rest of the content in the CentOS Linux repos, as well as the atomic sig run repos in the community build service. In order to keep things simple, it might be good enough to run this ostree update on a nightly basis.
With current rpm-ostree it is very inexpensive to just run it constantly (e.g every 5 minutes), or better, edge triggered from push notification from one of the rpm-md input repos being regenerated, and it will quickly do nothing if there's nothing to do.
See: https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2015-June...
I propose this nightly build be run from an automated harness at ci.centos.org, which would pull down the last ostree repo, do its updates, commit the changeset, and have a way to push back into the buildlogs.centos.org host the new repo.
Transitioning it from CI to CD then. There's lots of interesting aspects to this...
As an additional downstream trigger, we should then build nightly the images derived from this
Not sure I understand the rationale of this nightly thing here either. Why not just trigger all images when the ostree commit changes? That can be implemented by a few lines of shell to compare the previous/new commit IDs, and/or rpm-ostree compose tree --touch-if-changed.
ostree repo ( this would be the GenericCloud, Vagrant, ISO files etc for the atomic build that maps to the latest commit in that ostree repo). For numbering, I propose we use datestamp with a build id eg: 20150715_01, this maps well to user expectations and also sets suitable confidence in the build.
That versioning seems fine to me.
Key requirement to applying updates is that the atomic host needs a reboot to consume the latest ostree propagated changes. So while the updated tree would be available nightly, uses might chose to still apply it nightly, they can control to a fairly good granularity the point in time they wish to run.
That sounds like a different branch name?
The atomic sig can then decide which point in time build they wish to promote as their 4wk 'release' image. The ostree repo + images + documentation for that build can then be promoted to a release location. This would then be announced via projectatomic.io
I'm also proposing that this build be called the project atomic build.
for (b) above, I'm proposing we use the same process as for (a), except for the following major changes :
- content in (a) is signed,
Is *not* signed? Or is? Note that OSTree actually supports multiple GPG signatures per commit, intended for "binary promotion" workflows. But:
so while we can build things nightly, we cant promote that content as is.
Right, and the SIG build may have things that aren't in the rebuild, so it wouldn't make sense to binary-promote.
- the update cycle for the ostree repo should be undecided, so we can
promote a new update if there are security issues that merit this work, and also when there is a new version of a core component that we might want to push out out of band ( i.e not wait on the next upstream build )
I doubt anyone would object if we unconditionally always included the Z stream errata and released the tree commit immediately after the RPM rebuild.
Why not continuously build the ostree based on package changes to Docker, Kub, etcd, flannel, etc. This way the triggering happens when packages are updated. It would be great if Centos had something like fedmsg or our downstream CI message bus. Thoughts? -== @ri ==-
-------- Original message -------- From: Colin Walters walters@verbum.org Date: 07/15/2015 5:40 PM (GMT-05:00) To: centos-devel@centos.org Subject: Re: [CentOS-devel] numbering and building the CentOS Atomic story
Thanks for posting this!
On Wed, Jul 15, 2015, at 06:18 AM, Karanbir Singh wrote:
For (a) above, I propose we run an ostree repo, managed by a centos-release-projectatomic rpm file; the ostree repo for this should be delivered from buildlogs.centos.org, and updated in sync with the rest of the content in the CentOS Linux repos, as well as the atomic sig run repos in the community build service. In order to keep things simple, it might be good enough to run this ostree update on a nightly basis.
With current rpm-ostree it is very inexpensive to just run it constantly (e.g every 5 minutes), or better, edge triggered from push notification from one of the rpm-md input repos being regenerated, and it will quickly do nothing if there's nothing to do.
See: https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2015-June...
I propose this nightly build be run from an automated harness at ci.centos.org, which would pull down the last ostree repo, do its updates, commit the changeset, and have a way to push back into the buildlogs.centos.org host the new repo.
Transitioning it from CI to CD then. There's lots of interesting aspects to this...
As an additional downstream trigger, we should then build nightly the images derived from this
Not sure I understand the rationale of this nightly thing here either. Why not just trigger all images when the ostree commit changes? That can be implemented by a few lines of shell to compare the previous/new commit IDs, and/or rpm-ostree compose tree --touch-if-changed.
ostree repo ( this would be the GenericCloud, Vagrant, ISO files etc for the atomic build that maps to the latest commit in that ostree repo). For numbering, I propose we use datestamp with a build id eg: 20150715_01, this maps well to user expectations and also sets suitable confidence in the build.
That versioning seems fine to me.
Key requirement to applying updates is that the atomic host needs a reboot to consume the latest ostree propagated changes. So while the updated tree would be available nightly, uses might chose to still apply it nightly, they can control to a fairly good granularity the point in time they wish to run.
That sounds like a different branch name?
The atomic sig can then decide which point in time build they wish to promote as their 4wk 'release' image. The ostree repo + images + documentation for that build can then be promoted to a release location. This would then be announced via projectatomic.io
I'm also proposing that this build be called the project atomic build.
for (b) above, I'm proposing we use the same process as for (a), except for the following major changes :
- content in (a) is signed,
Is *not* signed? Or is? Note that OSTree actually supports multiple GPG signatures per commit, intended for "binary promotion" workflows. But:
so while we can build things nightly, we cant promote that content as is.
Right, and the SIG build may have things that aren't in the rebuild, so it wouldn't make sense to binary-promote.
- the update cycle for the ostree repo should be undecided, so we can
promote a new update if there are security issues that merit this work, and also when there is a new version of a core component that we might want to push out out of band ( i.e not wait on the next upstream build )
I doubt anyone would object if we unconditionally always included the Z stream errata and released the tree commit immediately after the RPM rebuild. _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org http://lists.centos.org/mailman/listinfo/centos-devel
On 16/07/15 00:28, Ari LiVigni wrote:
Why not continuously build the ostree based on package changes to Docker, Kub, etcd, flannel, etc. This way the triggering happens when packages are updated. It would be great if Centos had something like fedmsg or our downstream CI message bus.
this is what Colin mentioned as well, with the repo md changing we can trigger things off. I believe the RDO / Cloud SIG already run a packstack validation on every rpm build.
the only tricky thing we need to solve here is getting the 'last' ostree repo onto the ci worker node, and then pushing it back out at something where it can then be fetched for the next build.
regards
Thoughts?
-== @ri ==-
-------- Original message -------- From: Colin Walters walters@verbum.org Date: 07/15/2015 5:40 PM (GMT-05:00) To: centos-devel@centos.org Subject: Re: [CentOS-devel] numbering and building the CentOS Atomic story
Thanks for posting this!
On Wed, Jul 15, 2015, at 06:18 AM, Karanbir Singh wrote:
For (a) above, I propose we run an ostree repo, managed by a centos-release-projectatomic rpm file; the ostree repo for this should be delivered from buildlogs.centos.org, and updated in sync with the rest of the content in the CentOS Linux repos, as well as the atomic sig run repos in the community build service. In order to keep things simple, it might be good enough to run this ostree update on a nightly basis.
With current rpm-ostree it is very inexpensive to just run it constantly (e.g every 5 minutes), or better, edge triggered from push notification from one of the rpm-md input repos being regenerated, and it will quickly do nothing if there's nothing to do.
See: https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2015-June...
I propose this nightly build be run from an automated harness at ci.centos.org, which would pull down the last ostree repo, do its updates, commit the changeset, and have a way to push back into the buildlogs.centos.org host the new repo.
Transitioning it from CI to CD then. There's lots of interesting aspects to this...
As an additional downstream trigger, we should then build nightly the images derived from this
Not sure I understand the rationale of this nightly thing here either. Why not just trigger all images when the ostree commit changes? That can be implemented by a few lines of shell to compare the previous/new commit IDs, and/or rpm-ostree compose tree --touch-if-changed.
ostree repo ( this would be the GenericCloud, Vagrant, ISO files etc for the atomic build that maps to the latest commit in that ostree repo). For numbering, I propose we use datestamp with a build id eg: 20150715_01, this maps well to user expectations and also sets suitable confidence in the build.
That versioning seems fine to me.
Key requirement to applying updates is that the atomic host needs a reboot to consume the latest ostree propagated changes. So while the updated tree would be available nightly, uses might chose to still apply it nightly, they can control to a fairly good granularity the point in time they wish to run.
That sounds like a different branch name?
The atomic sig can then decide which point in time build they wish to promote as their 4wk 'release' image. The ostree repo + images + documentation for that build can then be promoted to a release location. This would then be announced via projectatomic.io
I'm also proposing that this build be called the project atomic build.
for (b) above, I'm proposing we use the same process as for (a), except for the following major changes :
- content in (a) is signed,
Is *not* signed? Or is? Note that OSTree actually supports multiple GPG signatures per commit, intended for "binary promotion" workflows. But:
so while we can build things nightly, we cant promote that content as is.
Right, and the SIG build may have things that aren't in the rebuild, so it wouldn't make sense to binary-promote.
- the update cycle for the ostree repo should be undecided, so we can
promote a new update if there are security issues that merit this work, and also when there is a new version of a core component that we might want to push out out of band ( i.e not wait on the next upstream build )
I doubt anyone would object if we unconditionally always included the Z stream errata and released the tree commit immediately after the RPM rebuild. _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org http://lists.centos.org/mailman/listinfo/centos-devel
CentOS-devel mailing list CentOS-devel@centos.org http://lists.centos.org/mailman/listinfo/centos-devel
On 07/16/2015 10:33 AM, Karanbir Singh wrote:
On 16/07/15 00:28, Ari LiVigni wrote:
Why not continuously build the ostree based on package changes to Docker, Kub, etcd, flannel, etc. This way the triggering happens when packages are updated. It would be great if Centos had something like fedmsg or our downstream CI message bus.
this is what Colin mentioned as well, with the repo md changing we can trigger things off.
IIRC, it is not possible to detect which build has been updated. If it was kubernetes, etcd or docker? As the md file is archived, URLTrigger can not be used (with condition using xpath expresion). But this is a problem of all repo md files.
I believe the RDO / Cloud SIG already run a packstack validation on every rpm build.
the only tricky thing we need to solve here is getting the 'last' ostree repo onto the ci worker node, and then pushing it back out at something where it can then be fetched for the next build.
regards
Thoughts?
-== @ri ==-
-------- Original message -------- From: Colin Walters walters@verbum.org Date: 07/15/2015 5:40 PM (GMT-05:00) To: centos-devel@centos.org Subject: Re: [CentOS-devel] numbering and building the CentOS Atomic story
Thanks for posting this!
On Wed, Jul 15, 2015, at 06:18 AM, Karanbir Singh wrote:
For (a) above, I propose we run an ostree repo, managed by a centos-release-projectatomic rpm file; the ostree repo for this should be delivered from buildlogs.centos.org, and updated in sync with the rest of the content in the CentOS Linux repos, as well as the atomic sig run repos in the community build service. In order to keep things simple, it might be good enough to run this ostree update on a nightly basis.
With current rpm-ostree it is very inexpensive to just run it constantly (e.g every 5 minutes), or better, edge triggered from push notification from one of the rpm-md input repos being regenerated, and it will quickly do nothing if there's nothing to do.
See: https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2015-June...
I propose this nightly build be run from an automated harness at ci.centos.org, which would pull down the last ostree repo, do its updates, commit the changeset, and have a way to push back into the buildlogs.centos.org host the new repo.
Transitioning it from CI to CD then. There's lots of interesting aspects to this...
As an additional downstream trigger, we should then build nightly the images derived from this
Not sure I understand the rationale of this nightly thing here either. Why not just trigger all images when the ostree commit changes? That can be implemented by a few lines of shell to compare the previous/new commit IDs, and/or rpm-ostree compose tree --touch-if-changed.
ostree repo ( this would be the GenericCloud, Vagrant, ISO files etc for the atomic build that maps to the latest commit in that ostree repo). For numbering, I propose we use datestamp with a build id eg: 20150715_01, this maps well to user expectations and also sets suitable confidence in the build.
That versioning seems fine to me.
Key requirement to applying updates is that the atomic host needs a reboot to consume the latest ostree propagated changes. So while the updated tree would be available nightly, uses might chose to still apply it nightly, they can control to a fairly good granularity the point in time they wish to run.
That sounds like a different branch name?
The atomic sig can then decide which point in time build they wish to promote as their 4wk 'release' image. The ostree repo + images + documentation for that build can then be promoted to a release location. This would then be announced via projectatomic.io
I'm also proposing that this build be called the project atomic build.
for (b) above, I'm proposing we use the same process as for (a), except for the following major changes :
- content in (a) is signed,
Is *not* signed? Or is? Note that OSTree actually supports multiple GPG signatures per commit, intended for "binary promotion" workflows. But:
so while we can build things nightly, we cant promote that content as is.
Right, and the SIG build may have things that aren't in the rebuild, so it wouldn't make sense to binary-promote.
- the update cycle for the ostree repo should be undecided, so we can
promote a new update if there are security issues that merit this work, and also when there is a new version of a core component that we might want to push out out of band ( i.e not wait on the next upstream build )
I doubt anyone would object if we unconditionally always included the Z stream errata and released the tree commit immediately after the RPM rebuild. _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org http://lists.centos.org/mailman/listinfo/centos-devel
CentOS-devel mailing list CentOS-devel@centos.org http://lists.centos.org/mailman/listinfo/centos-devel
On 17/07/15 16:52, Jan Chaloupka wrote:
On 07/16/2015 10:33 AM, Karanbir Singh wrote:
On 16/07/15 00:28, Ari LiVigni wrote:
Why not continuously build the ostree based on package changes to Docker, Kub, etcd, flannel, etc. This way the triggering happens when packages are updated. It would be great if Centos had something like fedmsg or our downstream CI message bus.
this is what Colin mentioned as well, with the repo md changing we can trigger things off.
IIRC, it is not possible to detect which build has been updated. If it was kubernetes, etcd or docker? As the md file is archived, URLTrigger can not be used (with condition using xpath expresion). But this is a problem of all repo md files.
sure, and this is a good thing - a founding concept of CI is that you want to run all tests, even when the core component you care about does not change :: the idea being you validate your code still works within a changing codebase ( in this case the repo itself ).
- KB
On Fri, 2015-07-17 at 23:56 +0100, Karanbir Singh wrote:
On 17/07/15 16:52, Jan Chaloupka wrote:
On 07/16/2015 10:33 AM, Karanbir Singh wrote:
On 16/07/15 00:28, Ari LiVigni wrote:
Why not continuously build the ostree based on package changes to Docker, Kub, etcd, flannel, etc. This way the triggering happens when packages are updated. It would be great if Centos had something like fedmsg or our downstream CI message bus.
this is what Colin mentioned as well, with the repo md changing we can trigger things off.
IIRC, it is not possible to detect which build has been updated. If it was kubernetes, etcd or docker? As the md file is archived, URLTrigger can not be used (with condition using xpath expresion). But this is a problem of all repo md files.
sure, and this is a good thing - a founding concept of CI is that you want to run all tests, even when the core component you care about does not change :: the idea being you validate your code still works within a changing codebase ( in this case the repo itself ).
I agree testing within the repo should be done as well for unit testing and light integration testing, but I also think doing CI of real packaged components in RPM form or in ostree is better for true CI and eventually CD IMHO.
It can be a downstream set of jobs after the repo tests are validated.
- KB
-
Ari LiVigni -
Colin Walters -
Jan Chaloupka -
Karanbir Singh