Hi, all,
I'd like to propose a new Special Interest Group focused on integration efforts around CentOS Stream.
----
# Introduction
Integration is verifying that products and services built on top of RHEL or CentOS Stream will continue to work on CentOS Stream and the next release of RHEL and will not break on package updates.
As RHEL content becomes available only after the release, RHEL-based services traditionally use a catching-up integration pattern: people have to adjust their products and services to work on new RHEL *after* the update is shipped. Adjusting the services takes time, eating into the supported RHEL lifecycle period. It also reduces the options for how we can deal with breaking changes.
CentOS Stream provides a way to enable forward-looking integration: you can do the integration early during the development before the change is shipped to the CentOS Stream or RHEL repositories. This allows us to prevent or at least prepare better for any breaking changes, which might be shipped via CentOS Stream or RHEL updates.
# Purpose of the SIG
Provide a shared space to develop and maintain tooling and knowledge base on collaborative gating and testing of CentOS Stream updates before they are published to CentOS mirrors. This includes both - package-level and compose-level integration.
# Goals
* Document existing integration workflows used by other SIGs. * Identify common issues. * Manage, develop and promote the Third-Party CI for CentOS Stream. - Document events and triggers. - Document standard pipelines. * Develop the Integration toolkit.
# Deliverables
* CentOS Stream Integration Guide
The main focus of the SIG at least at the beginning would be to collect and maintain the knowledge base for integration patterns and tools used by other SIGs and community members.
* CentOS Third-Party CI service
Eventually SIG may take the ownership of the third-party CI service (for example based on CentOS Stream Zuul CI) - this is to be discussed further.
# First work items
* Develop and document solution to the following question: “I have a test case/task I want to run on every change in CentOS and get notified about the failure. How do I do it?”
* Interview other SIGs for their integration workflows to identify common patterns, solutions and issues.
# Logistics (to be adjusted later by the SIG members)
* Matrix channel - No IRC * Discourse tag - No mailing list * Docs in some GitLab repo - No wiki * Bi–weekly text-only office hours on Matrix channel * (?) Occasional recorded video meeting
# Questions
1) Is the scope limited to integrations done by CentOS SIGs?
No. There are many ways one can integrate with the CentOS Stream. CentOS Project provides the option to do it via a CentOS SIG, but it is just one of the many possibilities. One can run integration workflows on their own infrastructure (both in the open or in private). We welcome everyone and would like to share the knowledge and tooling (and work items :) ) as much as possible.
2) Is the scope limited to integration with other FOSS projects?
No. While all the contributions (tooling, configurations, docs) of the SIG should be open and licensed under the licenses approved by the CentOS Project, you can participate in the SIG even when your primary project/product or service is not public. For example you can use integration tooling in your private labs and optionally contribute test results without sharing the access to your test systems.
----
This is awesome. Thank you Aleksandra!
My interest in integration is because I have been working on a CI for the company I work for that tests the updates against third-party packages and internal tools we use, before making the updates available in our internal mirrors. Our internal CI uses a little bit of ansible and a lot of tmt. I have been interested in increasing test coverage in general since we use cs9 in important scenarios. When the packages are from the official repos, I have brought the contributions to t_functional. That project has made it quite simple to contribute to CentOS CI.
I've learned to some extent the ins and outs of the CI by browsing docs, conference videos, source code, and the help of some community members. I actually learned about tmt along the way. But I still have lots of questions. I can't wait for the documentation that comes out of this SIG. It will be of great help to contributors.
Best Regards, Carlos.
On 8/31/23 05:52, Aleksandra Fedorova wrote:
Hi, all,
I'd like to propose a new Special Interest Group focused on integration efforts around CentOS Stream.
# Introduction
Integration is verifying that products and services built on top of RHEL or CentOS Stream will continue to work on CentOS Stream and the next release of RHEL and will not break on package updates.
As RHEL content becomes available only after the release, RHEL-based services traditionally use a catching-up integration pattern: people have to adjust their products and services to work on new RHEL *after* the update is shipped. Adjusting the services takes time, eating into the supported RHEL lifecycle period. It also reduces the options for how we can deal with breaking changes.
CentOS Stream provides a way to enable forward-looking integration: you can do the integration early during the development before the change is shipped to the CentOS Stream or RHEL repositories. This allows us to prevent or at least prepare better for any breaking changes, which might be shipped via CentOS Stream or RHEL updates.
# Purpose of the SIG
Provide a shared space to develop and maintain tooling and knowledge base on collaborative gating and testing of CentOS Stream updates before they are published to CentOS mirrors. This includes both - package-level and compose-level integration.
# Goals
- Document existing integration workflows used by other SIGs.
- Identify common issues.
- Manage, develop and promote the Third-Party CI for CentOS Stream.
- Document events and triggers. - Document standard pipelines.
- Develop the Integration toolkit.
# Deliverables
- CentOS Stream Integration Guide
The main focus of the SIG at least at the beginning would be to collect and maintain the knowledge base for integration patterns and tools used by other SIGs and community members.
- CentOS Third-Party CI service
Eventually SIG may take the ownership of the third-party CI service (for example based on CentOS Stream Zuul CI) - this is to be discussed further.
# First work items
- Develop and document solution to the following question:
“I have a test case/task I want to run on every change in CentOS and get notified about the failure. How do I do it?”
- Interview other SIGs for their integration workflows to identify
common patterns, solutions and issues.
# Logistics (to be adjusted later by the SIG members)
- Matrix channel - No IRC
- Discourse tag - No mailing list
- Docs in some GitLab repo - No wiki
- Bi–weekly text-only office hours on Matrix channel
- (?) Occasional recorded video meeting
# Questions
- Is the scope limited to integrations done by CentOS SIGs?
No. There are many ways one can integrate with the CentOS Stream. CentOS Project provides the option to do it via a CentOS SIG, but it is just one of the many possibilities. One can run integration workflows on their own infrastructure (both in the open or in private). We welcome everyone and would like to share the knowledge and tooling (and work items :) ) as much as possible.
- Is the scope limited to integration with other FOSS projects?
No. While all the contributions (tooling, configurations, docs) of the SIG should be open and licensed under the licenses approved by the CentOS Project, you can participate in the SIG even when your primary project/product or service is not public. For example you can use integration tooling in your private labs and optionally contribute test results without sharing the access to your test systems.
Aleksandra,
I have put this on the agenda for the next board meeting.
Thanks,
Amy
*Amy Marrich*
She/Her/Hers
Principal Technical Marketing Manager - Cloud Platforms
Red Hat, Inc https://www.redhat.com/
amy@redhat.com
Mobile: 954-818-0514
Slack: amarrich
IRC: spotz https://www.redhat.com/
On Thu, Aug 31, 2023 at 7:53 AM Aleksandra Fedorova alpha@bookwar.info wrote:
Hi, all,
I'd like to propose a new Special Interest Group focused on integration efforts around CentOS Stream.
# Introduction
Integration is verifying that products and services built on top of RHEL or CentOS Stream will continue to work on CentOS Stream and the next release of RHEL and will not break on package updates.
As RHEL content becomes available only after the release, RHEL-based services traditionally use a catching-up integration pattern: people have to adjust their products and services to work on new RHEL *after* the update is shipped. Adjusting the services takes time, eating into the supported RHEL lifecycle period. It also reduces the options for how we can deal with breaking changes.
CentOS Stream provides a way to enable forward-looking integration: you can do the integration early during the development before the change is shipped to the CentOS Stream or RHEL repositories. This allows us to prevent or at least prepare better for any breaking changes, which might be shipped via CentOS Stream or RHEL updates.
# Purpose of the SIG
Provide a shared space to develop and maintain tooling and knowledge base on collaborative gating and testing of CentOS Stream updates before they are published to CentOS mirrors. This includes both - package-level and compose-level integration.
# Goals
- Document existing integration workflows used by other SIGs.
- Identify common issues.
- Manage, develop and promote the Third-Party CI for CentOS Stream.
- Document events and triggers.
- Document standard pipelines.
- Develop the Integration toolkit.
# Deliverables
CentOS Stream Integration Guide
The main focus of the SIG at least at the beginning would be to
collect and maintain the knowledge base for integration patterns and tools used by other SIGs and community members.
CentOS Third-Party CI service
Eventually SIG may take the ownership of the third-party CI service
(for example based on CentOS Stream Zuul CI) - this is to be discussed further.
# First work items
- Develop and document solution to the following question: “I have a test case/task I want to run on every change in CentOS and
get notified about the failure. How do I do it?”
- Interview other SIGs for their integration workflows to identify
common patterns, solutions and issues.
# Logistics (to be adjusted later by the SIG members)
- Matrix channel - No IRC
- Discourse tag - No mailing list
- Docs in some GitLab repo - No wiki
- Bi–weekly text-only office hours on Matrix channel
- (?) Occasional recorded video meeting
# Questions
- Is the scope limited to integrations done by CentOS SIGs?
No. There are many ways one can integrate with the CentOS Stream. CentOS Project provides the option to do it via a CentOS SIG, but it is just one of the many possibilities. One can run integration workflows on their own infrastructure (both in the open or in private). We welcome everyone and would like to share the knowledge and tooling (and work items :) ) as much as possible.
- Is the scope limited to integration with other FOSS projects?
No. While all the contributions (tooling, configurations, docs) of the SIG should be open and licensed under the licenses approved by the CentOS Project, you can participate in the SIG even when your primary project/product or service is not public. For example you can use integration tooling in your private labs and optionally contribute test results without sharing the access to your test systems.
-- Aleksandra Fedorova Matrix: @bookwar:fedora.im Fediverse: @bookwar@fosstodon.org _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 2023-09-01 21:51, Amy Marrich wrote:
Aleksandra,
I have put this on the agenda for the next board meeting.
This was discussed and approved in yesterday's board meeting (see https://git.centos.org/centos/board/issue/120). I will work with Aleksandra to get the necessary resources spun up.
Cheers Davide
On 9/14/23 06:58, Davide Cavalca wrote:
On 2023-09-01 21:51, Amy Marrich wrote:
Aleksandra,
I have put this on the agenda for the next board meeting.
This was discussed and approved in yesterday's board meeting (see https://git.centos.org/centos/board/issue/120). I will work with Aleksandra to get the necessary resources spun up.
Today we have created following resources:
* Matrix channel #centos-integration:fedora.im * issue tracker on GitLab: https://gitlab.com/CentOS/Integration/general/-/issues * forum thread at Discourse: https://discussion.fedoraproject.org/t/centos-integration-sig-everyone-is-we...
Everyone is of course invited to join.
Thanks for this Aleksandra!!
As Cloud and NFV SIGs member I'm really interested in contributing validation jobs to validate pre-release CentOS Stream content and find out issues as early as possible in the workflow.
Let us know if we can help somehow in this new SIG!
Best regards,
Alfredo
On Thu, Aug 31, 2023 at 2:53 PM Aleksandra Fedorova alpha@bookwar.info wrote:
Hi, all,
I'd like to propose a new Special Interest Group focused on integration efforts around CentOS Stream.
# Introduction
Integration is verifying that products and services built on top of RHEL or CentOS Stream will continue to work on CentOS Stream and the next release of RHEL and will not break on package updates.
As RHEL content becomes available only after the release, RHEL-based services traditionally use a catching-up integration pattern: people have to adjust their products and services to work on new RHEL *after* the update is shipped. Adjusting the services takes time, eating into the supported RHEL lifecycle period. It also reduces the options for how we can deal with breaking changes.
CentOS Stream provides a way to enable forward-looking integration: you can do the integration early during the development before the change is shipped to the CentOS Stream or RHEL repositories. This allows us to prevent or at least prepare better for any breaking changes, which might be shipped via CentOS Stream or RHEL updates.
# Purpose of the SIG
Provide a shared space to develop and maintain tooling and knowledge base on collaborative gating and testing of CentOS Stream updates before they are published to CentOS mirrors. This includes both - package-level and compose-level integration.
# Goals
- Document existing integration workflows used by other SIGs.
- Identify common issues.
- Manage, develop and promote the Third-Party CI for CentOS Stream.
- Document events and triggers.
- Document standard pipelines.
- Develop the Integration toolkit.
# Deliverables
CentOS Stream Integration Guide
The main focus of the SIG at least at the beginning would be to
collect and maintain the knowledge base for integration patterns and tools used by other SIGs and community members.
CentOS Third-Party CI service
Eventually SIG may take the ownership of the third-party CI service
(for example based on CentOS Stream Zuul CI) - this is to be discussed further.
# First work items
- Develop and document solution to the following question: “I have a test case/task I want to run on every change in CentOS and
get notified about the failure. How do I do it?”
- Interview other SIGs for their integration workflows to identify
common patterns, solutions and issues.
# Logistics (to be adjusted later by the SIG members)
- Matrix channel - No IRC
- Discourse tag - No mailing list
- Docs in some GitLab repo - No wiki
- Bi–weekly text-only office hours on Matrix channel
- (?) Occasional recorded video meeting
# Questions
- Is the scope limited to integrations done by CentOS SIGs?
No. There are many ways one can integrate with the CentOS Stream. CentOS Project provides the option to do it via a CentOS SIG, but it is just one of the many possibilities. One can run integration workflows on their own infrastructure (both in the open or in private). We welcome everyone and would like to share the knowledge and tooling (and work items :) ) as much as possible.
- Is the scope limited to integration with other FOSS projects?
No. While all the contributions (tooling, configurations, docs) of the SIG should be open and licensed under the licenses approved by the CentOS Project, you can participate in the SIG even when your primary project/product or service is not public. For example you can use integration tooling in your private labs and optionally contribute test results without sharing the access to your test systems.
-- Aleksandra Fedorova Matrix: @bookwar:fedora.im Fediverse: @bookwar@fosstodon.org _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
Hi, How will the SIG work with ci tests for EPEL packages? I was wondering if those tests must be in one side (EPEL), or will be on both (EPEL&Int. SIG 3rd Party services), or if there will be some point of integration (e.g., some unified place for published results, etc...). I see EPEL builds in bodhi but I'm not sure what the state of automated CI in EPEL is in respect to CentOS Stream.
Regards, Carlos.
On 8/31/23 05:52, Aleksandra Fedorova wrote:
Hi, all,
I'd like to propose a new Special Interest Group focused on integration efforts around CentOS Stream.
# Introduction
Integration is verifying that products and services built on top of RHEL or CentOS Stream will continue to work on CentOS Stream and the next release of RHEL and will not break on package updates.
As RHEL content becomes available only after the release, RHEL-based services traditionally use a catching-up integration pattern: people have to adjust their products and services to work on new RHEL *after* the update is shipped. Adjusting the services takes time, eating into the supported RHEL lifecycle period. It also reduces the options for how we can deal with breaking changes.
CentOS Stream provides a way to enable forward-looking integration: you can do the integration early during the development before the change is shipped to the CentOS Stream or RHEL repositories. This allows us to prevent or at least prepare better for any breaking changes, which might be shipped via CentOS Stream or RHEL updates.
# Purpose of the SIG
Provide a shared space to develop and maintain tooling and knowledge base on collaborative gating and testing of CentOS Stream updates before they are published to CentOS mirrors. This includes both - package-level and compose-level integration.
# Goals
- Document existing integration workflows used by other SIGs.
- Identify common issues.
- Manage, develop and promote the Third-Party CI for CentOS Stream.
- Document events and triggers. - Document standard pipelines.
- Develop the Integration toolkit.
# Deliverables
- CentOS Stream Integration Guide
The main focus of the SIG at least at the beginning would be to collect and maintain the knowledge base for integration patterns and tools used by other SIGs and community members.
- CentOS Third-Party CI service
Eventually SIG may take the ownership of the third-party CI service (for example based on CentOS Stream Zuul CI) - this is to be discussed further.
# First work items
- Develop and document solution to the following question:
“I have a test case/task I want to run on every change in CentOS and get notified about the failure. How do I do it?”
- Interview other SIGs for their integration workflows to identify
common patterns, solutions and issues.
# Logistics (to be adjusted later by the SIG members)
- Matrix channel - No IRC
- Discourse tag - No mailing list
- Docs in some GitLab repo - No wiki
- Bi–weekly text-only office hours on Matrix channel
- (?) Occasional recorded video meeting
# Questions
- Is the scope limited to integrations done by CentOS SIGs?
No. There are many ways one can integrate with the CentOS Stream. CentOS Project provides the option to do it via a CentOS SIG, but it is just one of the many possibilities. One can run integration workflows on their own infrastructure (both in the open or in private). We welcome everyone and would like to share the knowledge and tooling (and work items :) ) as much as possible.
- Is the scope limited to integration with other FOSS projects?
No. While all the contributions (tooling, configurations, docs) of the SIG should be open and licensed under the licenses approved by the CentOS Project, you can participate in the SIG even when your primary project/product or service is not public. For example you can use integration tooling in your private labs and optionally contribute test results without sharing the access to your test systems.
On 9/13/23 23:37, Carlos Rodriguez-Fernandez wrote:
Hi, How will the SIG work with ci tests for EPEL packages? I was wondering if those tests must be in one side (EPEL), or will be on both (EPEL&Int. SIG 3rd Party services), or if there will be some point of integration (e.g., some unified place for published results, etc...). I see EPEL builds in bodhi but I'm not sure what the state of automated CI in EPEL is in respect to CentOS Stream.
Good question.
It touches so many interesting topics at once :)
For example, what does "being on the side" mean?
When we run a test, we have multiple things:
1. content, the test scenario written by someone; 2. event, which triggers the test; 3. worker, which runs the test; 4. results, which are stored somewhere; 5. feedback, which is shown to someone; 6. decision, which is made based on test results.
So who does what where?
Let's consider the task:
run a package build and a tmt test for it from a dist-git of an EPEL9 package minetest, on every update of the glibc package in the CentOS Stream 9.
One possible implementation I have in mind would work as follows:
----
1. content
Content is stored in src.fedoraproject.org/rpms/<package name>:<epel9> and managed by the package maintainer of the EPEL branch
2. event
Event is created by the CentOS Stream infrastructure (message or GitLab webhook)
3. worker
Can be CentOS Stream Zuul CI.
4. results
The long-term storage is probably not needed here, so the standard storage assigned by the worker with retention about a week will be enough.
5. feedback
One part of the feedback should be provided to the EPEL maintainer of minetest - how? Another to the CentOS package maintainer of glibc.
6. decision
Here the change happens on the CentOS Stream side, so decision to land or not to land a change is done by the CentOS package maintainer (Red Hat employee)
----
In this example the EPEL Project doesn't even have to participate in this work as a project. All of this can be implemented between the EPEL package maintainer and CentOS SIG.
Yet as EPEL represents thousands of packages we probably don't want to handle each of them separately. Handling ownership changes alone would be a burden. Also ideally for every test we run on change in CentOS Stream we should run same test on change in EPEL.
So we need to create some kind of relationship between EPEL and CentOS Stream Integration efforts so that EPEL CI and CentOS Stream Third-Party CI are working together.
It may also be beneficial for EPEL to provide full control over CI workers to EPEL maintainers, therefore to not use the CentOS Stream CI for it, but rather introduce a different setup.
I am not sure yet how exactly it will work in the end. But I see the purpose of the CentOS Integration SIG is to be the place where we can have this conversation.
And I'd like to invite EPEL folks to it.
Thank you so much Aleksandra for the detailed response.
On 9/20/23 06:31, Aleksandra Fedorova wrote:
On 9/13/23 23:37, Carlos Rodriguez-Fernandez wrote:
Hi, How will the SIG work with ci tests for EPEL packages? I was wondering if those tests must be in one side (EPEL), or will be on both (EPEL&Int. SIG 3rd Party services), or if there will be some point of integration (e.g., some unified place for published results, etc...). I see EPEL builds in bodhi but I'm not sure what the state of automated CI in EPEL is in respect to CentOS Stream.
Good question.
It touches so many interesting topics at once :)
For example, what does "being on the side" mean?
When we run a test, we have multiple things:
- content, the test scenario written by someone;
- event, which triggers the test;
- worker, which runs the test;
- results, which are stored somewhere;
- feedback, which is shown to someone;
- decision, which is made based on test results.
So who does what where?
Let's consider the task:
run a package build and a tmt test for it from a dist-git of an EPEL9 package minetest, on every update of the glibc package in the CentOS Stream 9.
One possible implementation I have in mind would work as follows:
- content
Content is stored in src.fedoraproject.org/rpms/<package name>:<epel9> and managed by the package maintainer of the EPEL branch
- event
Event is created by the CentOS Stream infrastructure (message or GitLab webhook)
- worker
Can be CentOS Stream Zuul CI.
- results
The long-term storage is probably not needed here, so the standard storage assigned by the worker with retention about a week will be enough.
- feedback
One part of the feedback should be provided to the EPEL maintainer of minetest - how? Another to the CentOS package maintainer of glibc.
- decision
Here the change happens on the CentOS Stream side, so decision to land or not to land a change is done by the CentOS package maintainer (Red Hat employee)
In this example the EPEL Project doesn't even have to participate in this work as a project. All of this can be implemented between the EPEL package maintainer and CentOS SIG.
Yet as EPEL represents thousands of packages we probably don't want to handle each of them separately. Handling ownership changes alone would be a burden. Also ideally for every test we run on change in CentOS Stream we should run same test on change in EPEL.
So we need to create some kind of relationship between EPEL and CentOS Stream Integration efforts so that EPEL CI and CentOS Stream Third-Party CI are working together.
It may also be beneficial for EPEL to provide full control over CI workers to EPEL maintainers, therefore to not use the CentOS Stream CI for it, but rather introduce a different setup.
I am not sure yet how exactly it will work in the end. But I see the purpose of the CentOS Integration SIG is to be the place where we can have this conversation.
And I'd like to invite EPEL folks to it.
-
Aleksandra Fedorova -
Alfredo Moralejo Alonso -
Amy Marrich -
Carlos Rodriguez-Fernandez -
Davide Cavalca