# Goals
The purpose of this SIG is to quantify the potential benefits of applying existing compiler technology to distribution packages, targeting more recent CPUs, and evaluating different options for how these optimizations can be maintained in a scalable way, and delivered to end users.
Today, there is a significant delay between the release of new x86-64 ISAs and the adoption across the entire CentOS distribution. For example, CentOS 9 Stream released with the x86-64-v2 ISA baseline, nine to thirteen years after such CPUs became commercially available. The concern is that this discrepancy leaves more recent CPU features unused, and end users do not see the best possible performance for the CPUs they use.
Parts of the distribution are built with run-time selected optimizations, but this only applies to specialized functional areas, such as string manipulation, certain mathematical operations, and cryptography, but not (for example) to language interpreters that are part of CentOS.
# Status
Proposed: RFC and looking for a sponsoring Governing Board member
# What’s in scope
* Explore the immediate runtime performance impact of package-specific ISA-related work on key workloads e.g. language interpreters.
* Evaluate the performance benefit at run-time of building CentOS with auto-vectorization and the very-cheap cost model and different x86-64 micro-architecture levels.
* Explore ways how these performance-enhancing builds can be maintained at the distribution level and made available to end users for installation.
* Assess the image/container size impact of those additional optimized builds (this may not be applicable to some delivery mechanisms).
* Explore ways to alert users that they do not get optimal performance because of misconfigured hypervisors.
* It is possible, but not likely, that work across the entire distribution may be needed to enable shadow stacks. (Userspace enablement happened as part of CentOS 8, but the kernel parts are not upstream and have changed significantly, so there is some uncertainty.) If large-scale work affecting many packages is needed, the SIG would be a natural place to prototype changes before integration into the distribution.
# What's not in scope
* Enablement for future x86 CPUs and chipsets, assemblers, linkers, and compiler support for new ISAs including regular performance work on individual packages (e.g., identified upstream backports). Hardware enablement must follow existing processes.
* Any 32-bit work (such as 64-bit time_t, or a 32-bit kernel) is explicitly excluded.
* The focus will be on userspace changes.
* Rebuilds for potential observability improvements are out of scope (although shadow stacks are expected to help in this a
* Adding additional hosts to the CentOS infrastructure in support of this SIG needs to be negotiated with the CentOS infrastructure team. The CPU microarchitecture level of the existing infrastructure is already sufficient.
# Roadmap
March 2023: Start the SIG and coordinate with partners April 2023: Adopt existing CentOS builder resources to deliver the required builds. May 2023: Initial performance analysis results.
Rest TBD based on results of exploration.
Resources
They SIG could benefit from spare x86-64 Koji builder capacity.
On Mon, Feb 27, 2023 at 12:23 PM Florian Weimer fweimer@redhat.com wrote:
# Goals
The purpose of this SIG is to quantify the potential benefits of applying existing compiler technology to distribution packages, targeting more recent CPUs, and evaluating different options for how these optimizations can be maintained in a scalable way, and delivered to end users.
Today, there is a significant delay between the release of new x86-64 ISAs and the adoption across the entire CentOS distribution. For example, CentOS 9 Stream released with the x86-64-v2 ISA baseline, nine to thirteen years after such CPUs became commercially available. The concern is that this discrepancy leaves more recent CPU features unused, and end users do not see the best possible performance for the CPUs they use.
Parts of the distribution are built with run-time selected optimizations, but this only applies to specialized functional areas, such as string manipulation, certain mathematical operations, and cryptography, but not (for example) to language interpreters that are part of CentOS.
# Status
Proposed: RFC and looking for a sponsoring Governing Board member
# What’s in scope
Explore the immediate runtime performance impact of package-specific ISA-related work on key workloads e.g. language interpreters.
Evaluate the performance benefit at run-time of building CentOS with auto-vectorization and the very-cheap cost model and different x86-64 micro-architecture levels.
Explore ways how these performance-enhancing builds can be maintained at the distribution level and made available to end users for installation.
Assess the image/container size impact of those additional optimized builds (this may not be applicable to some delivery mechanisms).
Explore ways to alert users that they do not get optimal performance because of misconfigured hypervisors.
It is possible, but not likely, that work across the entire distribution may be needed to enable shadow stacks. (Userspace enablement happened as part of CentOS 8, but the kernel parts are not upstream and have changed significantly, so there is some uncertainty.) If large-scale work affecting many packages is needed, the SIG would be a natural place to prototype changes before integration into the distribution.
# What's not in scope
Enablement for future x86 CPUs and chipsets, assemblers, linkers, and compiler support for new ISAs including regular performance work on individual packages (e.g., identified upstream backports). Hardware enablement must follow existing processes.
Any 32-bit work (such as 64-bit time_t, or a 32-bit kernel) is explicitly excluded.
The focus will be on userspace changes.
Rebuilds for potential observability improvements are out of scope (although shadow stacks are expected to help in this a
Adding additional hosts to the CentOS infrastructure in support of this SIG needs to be negotiated with the CentOS infrastructure team. The CPU microarchitecture level of the existing infrastructure is already sufficient.
# Roadmap
March 2023: Start the SIG and coordinate with partners April 2023: Adopt existing CentOS builder resources to deliver the required builds. May 2023: Initial performance analysis results.
Rest TBD based on results of exploration.
Resources
They SIG could benefit from spare x86-64 Koji builder capacity.
Note that SIGs do not have access to distribution Koji. This would be hugely problematic to do in CBS given all the potential collisions in NVRs that it could create. I would suggest if this SIG is greenlit that a dedicated Koji instance would be spun up for this.
I also would not suggest calling this the x86 SIG, since that is misleading. Consider expanding to compiler flag analysis and tuning across all CentOS architectures rather than just narrowly looking at x86_64 micro-architecture levels. That ensures this SIG has some reasonable longevity rather than being a weird short-term experiment thing. It is worth exploring improvements for all architectures, since research on that front doesn't happen enough as it is. You could still start with x86, but giving opportunities to grow beyond that would be better for the community overall if this is done.
Another aspect, I'm not sure what value this provides because existing CentOS/RHEL releases can't change, so wouldn't this make more sense to do as alternative Koji roots for Fedora ELN, which is closer to CentOS Stream 10 than anything in the CentOS Project has?
Additionally, what is your plan for community engagement? What kind of contribution model do you plan to have? Finally, do you have a group of folks in mind you're interested in reaching out to participate in CentOS for this?
-- 真実はいつも一つ！/ Always, there's only one truth!
On Mon, Feb 27, 2023 at 4:46 PM Neal Gompa ngompa13@gmail.com wrote:
On Mon, Feb 27, 2023 at 12:23 PM Florian Weimer fweimer@redhat.com wrote:
# Goals
The purpose of this SIG is to quantify the potential benefits of applying existing compiler technology to distribution packages, targeting more recent CPUs, and evaluating different options for how these optimizations can be maintained in a scalable way, and delivered to end users.
Today, there is a significant delay between the release of new x86-64 ISAs and the adoption across the entire CentOS distribution. For example, CentOS 9 Stream released with the x86-64-v2 ISA baseline, nine to thirteen years after such CPUs became commercially available. The concern is that this discrepancy leaves more recent CPU features unused, and end users do not see the best possible performance for the CPUs they use.
Parts of the distribution are built with run-time selected optimizations, but this only applies to specialized functional areas, such as string manipulation, certain mathematical operations, and cryptography, but not (for example) to language interpreters that are part of CentOS.
# Status
Proposed: RFC and looking for a sponsoring Governing Board member
# What’s in scope
Explore the immediate runtime performance impact of package-specific ISA-related work on key workloads e.g. language interpreters.
Evaluate the performance benefit at run-time of building CentOS with auto-vectorization and the very-cheap cost model and different x86-64 micro-architecture levels.
Explore ways how these performance-enhancing builds can be maintained at the distribution level and made available to end users for installation.
Assess the image/container size impact of those additional optimized builds (this may not be applicable to some delivery mechanisms).
Explore ways to alert users that they do not get optimal performance because of misconfigured hypervisors.
It is possible, but not likely, that work across the entire distribution may be needed to enable shadow stacks. (Userspace enablement happened as part of CentOS 8, but the kernel parts are not upstream and have changed significantly, so there is some uncertainty.) If large-scale work affecting many packages is needed, the SIG would be a natural place to prototype changes before integration into the distribution.
# What's not in scope
Enablement for future x86 CPUs and chipsets, assemblers, linkers, and compiler support for new ISAs including regular performance work on individual packages (e.g., identified upstream backports). Hardware enablement must follow existing processes.
Any 32-bit work (such as 64-bit time_t, or a 32-bit kernel) is explicitly excluded.
The focus will be on userspace changes.
Rebuilds for potential observability improvements are out of scope (although shadow stacks are expected to help in this a
Adding additional hosts to the CentOS infrastructure in support of this SIG needs to be negotiated with the CentOS infrastructure team. The CPU microarchitecture level of the existing infrastructure is already sufficient.
# Roadmap
March 2023: Start the SIG and coordinate with partners April 2023: Adopt existing CentOS builder resources to deliver the required builds. May 2023: Initial performance analysis results.
Rest TBD based on results of exploration.
Resources
They SIG could benefit from spare x86-64 Koji builder capacity.
Note that SIGs do not have access to distribution Koji. This would be hugely problematic to do in CBS given all the potential collisions in NVRs that it could create. I would suggest if this SIG is greenlit that a dedicated Koji instance would be spun up for this.
I also would not suggest calling this the x86 SIG, since that is misleading. Consider expanding to compiler flag analysis and tuning
I don't think it's misleading at all. The proposal is pretty clear that they're looking at x86_64 and that's it. It might not be what you would recommend, but I don't think they are misleading anyone with the proposal as written.
across all CentOS architectures rather than just narrowly looking at x86_64 micro-architecture levels. That ensures this SIG has some reasonable longevity rather than being a weird short-term experiment thing. It is worth exploring improvements for all architectures, since research on that front doesn't happen enough as it is. You could still start with x86, but giving opportunities to grow beyond that would be better for the community overall if this is done.
I disagree. What you suggest is certainly a direction that could happen, but it's big, grand, and unwieldy. By retaining a scope on a specific architecture, the SIG can focus and make concrete, actionable suggestions and improvements.
I certainly appreciate larger and more comprehensive efforts, but I think we collectively bite off more than we can chew sometimes. I find a SIG with a specific goal refreshing.
Another aspect, I'm not sure what value this provides because existing CentOS/RHEL releases can't change, so wouldn't this make more sense to do as alternative Koji roots for Fedora ELN, which is closer to CentOS Stream 10 than anything in the CentOS Project has?
I think there's certainly an aspect of this that ELN plays a part in, but I don't see it as a good basis for the comparison aspects without doing more throwaway work.
josh
Additionally, what is your plan for community engagement? What kind of contribution model do you plan to have? Finally, do you have a group of folks in mind you're interested in reaching out to participate in CentOS for this?
-- 真実はいつも一つ！/ Always, there's only one truth! _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
W dniu 27.02.2023 o 22:54, Josh Boyer pisze:
I also would not suggest calling this the x86 SIG, since that is misleading. Consider expanding to compiler flag analysis and tuning
I don't think it's misleading at all.
My first thought was "Oh, no! Someone wants to revive 32-bit x86 back from grave!" so yes, it is misleading.
"Modernize x86-64", "Test architecture optimizations" are direction which I would go with name.
* Marcin Juszkiewicz:
W dniu 27.02.2023 o 22:54, Josh Boyer pisze:
I also would not suggest calling this the x86 SIG, since that is misleading. Consider expanding to compiler flag analysis and tuning
I don't think it's misleading at all.
My first thought was "Oh, no! Someone wants to revive 32-bit x86 back from grave!" so yes, it is misleading.
I really thought the meaning has largely shifted to 64 bits now, like it previously shifted from 16 bits to 32 bits.
If - will be allowed in all the required places (except the RPM architecture name itself), we can certainly call it the x86-64 SIG.
Thanks, Florian
On Mon, Feb 27, 2023 at 6:27 PM Marcin Juszkiewicz marcin.juszkiewicz@linaro.org wrote:
W dniu 27.02.2023 o 22:54, Josh Boyer pisze:
I also would not suggest calling this the x86 SIG, since that is misleading. Consider expanding to compiler flag analysis and tuning
I don't think it's misleading at all.
My first thought was "Oh, no! Someone wants to revive 32-bit x86 back from grave!" so yes, it is misleading.
It's not. The only way someone would be misled in that manner is if they didn't actually read the proposal and just read the 'Subject:' line of the email. You clearly didn't just read the subject :)
"Modernize x86-64", "Test architecture optimizations" are direction which I would go with name.
Those are good descriptions of purpose, but I wouldn't necessarily use them for names.
josh
On Mon, Feb 27, 2023, at 15:45, Neal Gompa wrote:
On Mon, Feb 27, 2023 at 12:23 PM Florian Weimer fweimer@redhat.com wrote:
# Goals
The purpose of this SIG is to quantify the potential benefits of applying existing compiler technology to distribution packages, targeting more recent CPUs, and evaluating different options for how these optimizations can be maintained in a scalable way, and delivered to end users.
Today, there is a significant delay between the release of new x86-64 ISAs and the adoption across the entire CentOS distribution. For example, CentOS 9 Stream released with the x86-64-v2 ISA baseline, nine to thirteen years after such CPUs became commercially available. The concern is that this discrepancy leaves more recent CPU features unused, and end users do not see the best possible performance for the CPUs they use.
Parts of the distribution are built with run-time selected optimizations, but this only applies to specialized functional areas, such as string manipulation, certain mathematical operations, and cryptography, but not (for example) to language interpreters that are part of CentOS.
# Status
Proposed: RFC and looking for a sponsoring Governing Board member
# What’s in scope
Explore the immediate runtime performance impact of package-specific ISA-related work on key workloads e.g. language interpreters.
Evaluate the performance benefit at run-time of building CentOS with auto-vectorization and the very-cheap cost model and different x86-64 micro-architecture levels.
Explore ways how these performance-enhancing builds can be maintained at the distribution level and made available to end users for installation.
Assess the image/container size impact of those additional optimized builds (this may not be applicable to some delivery mechanisms).
Explore ways to alert users that they do not get optimal performance because of misconfigured hypervisors.
It is possible, but not likely, that work across the entire distribution may be needed to enable shadow stacks. (Userspace enablement happened as part of CentOS 8, but the kernel parts are not upstream and have changed significantly, so there is some uncertainty.) If large-scale work affecting many packages is needed, the SIG would be a natural place to prototype changes before integration into the distribution.
# What's not in scope
Enablement for future x86 CPUs and chipsets, assemblers, linkers, and compiler support for new ISAs including regular performance work on individual packages (e.g., identified upstream backports). Hardware enablement must follow existing processes.
Any 32-bit work (such as 64-bit time_t, or a 32-bit kernel) is explicitly excluded.
The focus will be on userspace changes.
Rebuilds for potential observability improvements are out of scope (although shadow stacks are expected to help in this a
Adding additional hosts to the CentOS infrastructure in support of this SIG needs to be negotiated with the CentOS infrastructure team. The CPU microarchitecture level of the existing infrastructure is already sufficient.
# Roadmap
March 2023: Start the SIG and coordinate with partners April 2023: Adopt existing CentOS builder resources to deliver the required builds. May 2023: Initial performance analysis results.
Rest TBD based on results of exploration.
Resources
They SIG could benefit from spare x86-64 Koji builder capacity.
Note that SIGs do not have access to distribution Koji. This would be hugely problematic to do in CBS given all the potential collisions in NVRs that it could create. I would suggest if this SIG is greenlit that a dedicated Koji instance would be spun up for this.
We can provide separate disttags to this group. I think that given the mission they'll be incentivized to pay close attention to the NVRs between the distro and SIG content.
I also would not suggest calling this the x86 SIG, since that is misleading. Consider expanding to compiler flag analysis and tuning across all CentOS architectures rather than just narrowly looking at x86_64 micro-architecture levels. That ensures this SIG has some reasonable longevity rather than being a weird short-term experiment thing. It is worth exploring improvements for all architectures, since research on that front doesn't happen enough as it is. You could still start with x86, but giving opportunities to grow beyond that would be better for the community overall if this is done.
Another aspect, I'm not sure what value this provides because existing CentOS/RHEL releases can't change, so wouldn't this make more sense to do as alternative Koji roots for Fedora ELN, which is closer to CentOS Stream 10 than anything in the CentOS Project has?
Additionally, what is your plan for community engagement? What kind of contribution model do you plan to have? Finally, do you have a group of folks in mind you're interested in reaching out to participate in CentOS for this?
-- 真実はいつも一つ！/ Always, there's only one truth! _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
--Brian
On Mon, 2023-02-27 at 18:23 +0100, Florian Weimer wrote:
# Goals
The purpose of this SIG is to quantify the potential benefits of applying existing compiler technology to distribution packages, targeting more recent CPUs, and evaluating different options for how these optimizations can be maintained in a scalable way, and delivered to end users.
Would this work fit under the charter of the existing (and largely dormant) Alternative Architectures SIG?
Today, there is a significant delay between the release of new x86-64 ISAs and the adoption across the entire CentOS distribution. For example, CentOS 9 Stream released with the x86-64-v2 ISA baseline, nine to thirteen years after such CPUs became commercially available. The concern is that this discrepancy leaves more recent CPU features unused, and end users do not see the best possible performance for the CPUs they use.
Parts of the distribution are built with run-time selected optimizations, but this only applies to specialized functional areas, such as string manipulation, certain mathematical operations, and cryptography, but not (for example) to language interpreters that are part of CentOS.
# Status
Proposed: RFC and looking for a sponsoring Governing Board member
# What’s in scope
- Explore the immediate runtime performance impact of package-
specific ISA-related work on key workloads e.g. language interpreters.
- Evaluate the performance benefit at run-time of building CentOS
with auto-vectorization and the very-cheap cost model and different x86- 64 micro-architecture levels.
- Explore ways how these performance-enhancing builds can be
maintained at the distribution level and made available to end users for installation.
- Assess the image/container size impact of those additional
optimized builds (this may not be applicable to some delivery mechanisms).
- Explore ways to alert users that they do not get optimal
performance because of misconfigured hypervisors.
- It is possible, but not likely, that work across the entire
distribution may be needed to enable shadow stacks. (Userspace enablement happened as part of CentOS 8, but the kernel parts are not upstream and have changed significantly, so there is some uncertainty.) If large-scale work affecting many packages is needed, the SIG would be a natural place to prototype changes before integration into the distribution. # What's not in scope
- Enablement for future x86 CPUs and chipsets, assemblers, linkers,
and compiler support for new ISAs including regular performance work on individual packages (e.g., identified upstream backports). Hardware enablement must follow existing processes.
- Any 32-bit work (such as 64-bit time_t, or a 32-bit kernel) is
explicitly excluded.
The focus will be on userspace changes.
Rebuilds for potential observability improvements are out of scope
(although shadow stacks are expected to help in this a
- Adding additional hosts to the CentOS infrastructure in support of
this SIG needs to be negotiated with the CentOS infrastructure team. The CPU microarchitecture level of the existing infrastructure is already sufficient. # Roadmap
March 2023: Start the SIG and coordinate with partners April 2023: Adopt existing CentOS builder resources to deliver the required builds. May 2023: Initial performance analysis results.
Rest TBD based on results of exploration.
Resources
They SIG could benefit from spare x86-64 Koji builder capacity.
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 28/02/2023 01:04, Shaun McCance wrote:
On Mon, 2023-02-27 at 18:23 +0100, Florian Weimer wrote:
# Goals
The purpose of this SIG is to quantify the potential benefits of applying existing compiler technology to distribution packages, targeting more recent CPUs, and evaluating different options for how these optimizations can be maintained in a scalable way, and delivered to end users.
Would this work fit under the charter of the existing (and largely dormant) Alternative Architectures SIG?
<snip> The "AltArch SIG" was just created initially to bootstrap other architectures for CentOS 7, in comparison with i{3,6}86, x86_64 that we were only building/providing in the past. Once it was bootstrapped, it was back to Core SIG rebuilding in parallel packages for all these architectures but the altarch sig per se doesn't exist anymore ? (imho)
On 2/28/23 00:40, Fabian Arrotin wrote:
On 28/02/2023 01:04, Shaun McCance wrote:
On Mon, 2023-02-27 at 18:23 +0100, Florian Weimer wrote:
# Goals
The purpose of this SIG is to quantify the potential benefits of applying existing compiler technology to distribution packages, targeting more recent CPUs, and evaluating different options for how these optimizations can be maintained in a scalable way, and delivered to end users.
Would this work fit under the charter of the existing (and largely dormant) Alternative Architectures SIG?
<snip> The "AltArch SIG" was just created initially to bootstrap other architectures for CentOS 7, in comparison with i{3,6}86, x86_64 that we were only building/providing in the past. Once it was bootstrapped, it was back to Core SIG rebuilding in parallel packages for all these architectures but the altarch sig per se doesn't exist anymore ? (imho)
Other than for CentOS-7, where I still build and release packages for teh alternative arches, I would agree.
On 27/02/2023 18:23, Florian Weimer wrote:
# Goals
<snip>
Resources
They SIG could benefit from spare x86-64 Koji builder capacity.
Can you elaborate on that ? You mean just using existing x86_64 builders (matching x86-64-v2 specs but nothing like v3), right ? Just trying to capture things before it's even just accepted as a SIG .. and not after, realizing that we wouldn't have what you're searching for :)
* Fabian Arrotin:
On 27/02/2023 18:23, Florian Weimer wrote:
# Goals
<snip> > Resources > They SIG could benefit from spare x86-64 Koji builder capacity. >
Can you elaborate on that ? You mean just using existing x86_64 builders (matching x86-64-v2 specs but nothing like v3), right ? Just trying to capture things before it's even just accepted as a SIG .. and not after, realizing that we wouldn't have what you're searching for :)
I think the CentOS builders are already compatible with x86-64-v4, so no configuration changes are needed. I verified that for x86-02.stream.rdu2.redhat.com: it reports all the necessary CPU features.
Thanks, Florian
On 02/03/2023 06:20, Florian Weimer wrote:
- Fabian Arrotin:
On 27/02/2023 18:23, Florian Weimer wrote:
# Goals
<snip> > Resources > They SIG could benefit from spare x86-64 Koji builder capacity. >
Can you elaborate on that ? You mean just using existing x86_64 builders (matching x86-64-v2 specs but nothing like v3), right ? Just trying to capture things before it's even just accepted as a SIG .. and not after, realizing that we wouldn't have what you're searching for :)
I think the CentOS builders are already compatible with x86-64-v4, so no configuration changes are needed. I verified that for x86-02.stream.rdu2.redhat.com: it reports all the necessary CPU features.
Well, yes, but that's not the infra that SIGs are using either : Stream is itself built elsewhere (different koji/infra), reason why I was asking if you need something on the infra *where* SIGs are building packages (older than the Stream infra)
Here are the capabilities from one of the builders in CBS:
This program interpreter self-identifies as: /lib64/ld-linux-x86-64.so.2
Shared library search path: (libraries located via /etc/ld.so.cache) /lib64 (system search path) /usr/lib64 (system search path)
Subdirectories of glibc-hwcaps directories, in priority order: x86-64-v4 x86-64-v3 (supported, searched) x86-64-v2 (supported, searched)
Legacy HWCAP subdirectories under library search path directories: haswell (AT_PLATFORM; supported, searched) tls (supported, searched) avx512_1 x86_64 (supported, searched)
* Fabian Arrotin:
On 02/03/2023 06:20, Florian Weimer wrote:
- Fabian Arrotin:
On 27/02/2023 18:23, Florian Weimer wrote:
# Goals
<snip> > Resources > They SIG could benefit from spare x86-64 Koji builder capacity. >
Can you elaborate on that ? You mean just using existing x86_64 builders (matching x86-64-v2 specs but nothing like v3), right ? Just trying to capture things before it's even just accepted as a SIG .. and not after, realizing that we wouldn't have what you're searching for :)
I think the CentOS builders are already compatible with x86-64-v4, so no configuration changes are needed. I verified that for x86-02.stream.rdu2.redhat.com: it reports all the necessary CPU features.
Well, yes, but that's not the infra that SIGs are using either : Stream is itself built elsewhere (different koji/infra), reason why I was asking if you need something on the infra *where* SIGs are building packages (older than the Stream infra)
I had not realized this. For the x86-64-v3 builds, we can use the SIG infrastructure from a build machine perspective (thanks Brian for confirming). Depending on what we discover, this may not be the right way to deliver the most valuable optimizations to end users, though.
Thanks, Florian
We discussed this at the CentOS Board meeting last week and generally agreed with the overall idea of the SIG. There were two main questions that came up:
1) What is the contribution model the SIG is envisioning? How will people be able to participate in the SIG?
Could you add this to the proposal?
2) There are existing Intel people working in the Hyperscaler SIG on some optimized libraries. How would the Hyperscaler SIG coordinate with the X86 SIG in this regard? There is a desire to share work and not duplicate effort.
This seems like something we can work out once the SIG is approved.
I am happy to act as the Board sponsor for this SIG and I believe we should work through any missing pieces or logistics before next month's Board meeting.
josh
On Mon, Feb 27, 2023 at 12:23 PM Florian Weimer fweimer@redhat.com wrote:
# Goals
The purpose of this SIG is to quantify the potential benefits of applying existing compiler technology to distribution packages, targeting more recent CPUs, and evaluating different options for how these optimizations can be maintained in a scalable way, and delivered to end users.
Today, there is a significant delay between the release of new x86-64 ISAs and the adoption across the entire CentOS distribution. For example, CentOS 9 Stream released with the x86-64-v2 ISA baseline, nine to thirteen years after such CPUs became commercially available. The concern is that this discrepancy leaves more recent CPU features unused, and end users do not see the best possible performance for the CPUs they use.
Parts of the distribution are built with run-time selected optimizations, but this only applies to specialized functional areas, such as string manipulation, certain mathematical operations, and cryptography, but not (for example) to language interpreters that are part of CentOS.
# Status
Proposed: RFC and looking for a sponsoring Governing Board member
# What’s in scope
Explore the immediate runtime performance impact of package-specific ISA-related work on key workloads e.g. language interpreters.
Evaluate the performance benefit at run-time of building CentOS with auto-vectorization and the very-cheap cost model and different x86-64 micro-architecture levels.
Explore ways how these performance-enhancing builds can be maintained at the distribution level and made available to end users for installation.
Assess the image/container size impact of those additional optimized builds (this may not be applicable to some delivery mechanisms).
Explore ways to alert users that they do not get optimal performance because of misconfigured hypervisors.
It is possible, but not likely, that work across the entire distribution may be needed to enable shadow stacks. (Userspace enablement happened as part of CentOS 8, but the kernel parts are not upstream and have changed significantly, so there is some uncertainty.) If large-scale work affecting many packages is needed, the SIG would be a natural place to prototype changes before integration into the distribution.
# What's not in scope
Enablement for future x86 CPUs and chipsets, assemblers, linkers, and compiler support for new ISAs including regular performance work on individual packages (e.g., identified upstream backports). Hardware enablement must follow existing processes.
Any 32-bit work (such as 64-bit time_t, or a 32-bit kernel) is explicitly excluded.
The focus will be on userspace changes.
Rebuilds for potential observability improvements are out of scope (although shadow stacks are expected to help in this a
Adding additional hosts to the CentOS infrastructure in support of this SIG needs to be negotiated with the CentOS infrastructure team. The CPU microarchitecture level of the existing infrastructure is already sufficient.
# Roadmap
March 2023: Start the SIG and coordinate with partners April 2023: Adopt existing CentOS builder resources to deliver the required builds. May 2023: Initial performance analysis results.
Rest TBD based on results of exploration.
Resources
They SIG could benefit from spare x86-64 Koji builder capacity.
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
* Josh Boyer:
We discussed this at the CentOS Board meeting last week and generally agreed with the overall idea of the SIG. There were two main questions that came up:
- What is the contribution model the SIG is envisioning? How will
people be able to participate in the SIG?
Could you add this to the proposal?
I don't have a live document for the proposal right now.
The general CentOS collaboration mechanism is a bit unclear to me, to be honest. It doesn't seem to be like Fedora, where anyone can create an account and try to get a package into the distribution. Not that this is something I expect as part of this effort; the focus should be on building existing stuff in new and exciting ways.
Anyway, here are some concrete items for collaboration:
I hope that performance experts will be able to share benchmarking results based on the optimized builds the SIG produces, and investigate glitches we encounter. Any kind of testing will help.
Some packages will fail to build with -march=x86-64-v3. Triaging these failures, reporting them upstream etc., will be required, and that part does not require any special privileges (as long as the build logs and buildroots are public, which I expect them to be).
We would benefit from expert guidance on GCC parameter defaults, like preferred vector sizes for certain operations.
Once we explore delivery mechanisms, exploring how builds can be consumed by end users in custom CI systems etc., and feedback on that will be helpful.
Do you think this provides some clarification?
- There are existing Intel people working in the Hyperscaler SIG on
some optimized libraries. How would the Hyperscaler SIG coordinate with the X86 SIG in this regard? There is a desire to share work and not duplicate effort.
Is it more than just zlib?
https://cbs.centos.org/koji/packages?tagID=2620
zlib upstream doesn't take architecture optimization patches, and maintaining the downstream patches (some of them are in CentOS proper already) is an ongoing hassle.
I've looked at
https://sigs.centos.org/hyperscale/content/repositories/main/
That seems to be focus on CentoS 8 Stream. The package versions in 9 are actually higher than what's mentioned there.
I am happy to act as the Board sponsor for this SIG and I believe we should work through any missing pieces or logistics before next month's Board meeting.
Thanks, Florian
On Wed, Mar 15, 2023 at 1:37 PM Florian Weimer fweimer@redhat.com wrote:
- Josh Boyer:
We discussed this at the CentOS Board meeting last week and generally agreed with the overall idea of the SIG. There were two main questions that came up:
- What is the contribution model the SIG is envisioning? How will
people be able to participate in the SIG?
Could you add this to the proposal?
I don't have a live document for the proposal right now.
The general CentOS collaboration mechanism is a bit unclear to me, to be honest. It doesn't seem to be like Fedora, where anyone can create an account and try to get a package into the distribution. Not that this is something I expect as part of this effort; the focus should be on building existing stuff in new and exciting ways.
Anyway, here are some concrete items for collaboration:
I hope that performance experts will be able to share benchmarking results based on the optimized builds the SIG produces, and investigate glitches we encounter. Any kind of testing will help.
Some packages will fail to build with -march=x86-64-v3. Triaging these failures, reporting them upstream etc., will be required, and that part does not require any special privileges (as long as the build logs and buildroots are public, which I expect them to be).
We would benefit from expert guidance on GCC parameter defaults, like preferred vector sizes for certain operations.
Once we explore delivery mechanisms, exploring how builds can be consumed by end users in custom CI systems etc., and feedback on that will be helpful.
Do you think this provides some clarification?
- There are existing Intel people working in the Hyperscaler SIG on
some optimized libraries. How would the Hyperscaler SIG coordinate with the X86 SIG in this regard? There is a desire to share work and not duplicate effort.
Is it more than just zlib?
https://cbs.centos.org/koji/packages?tagID=2620
zlib upstream doesn't take architecture optimization patches, and maintaining the downstream patches (some of them are in CentOS proper already) is an ongoing hassle.
zlib-ng does, and the guy working on the zlib package in Hyperscale is interested in proposing switching Fedora's zlib implementation to zlib-ng partly for that reason. Another alternative is using zstd as a zlib replacement, which we're also considering for a proposal to Fedora.
I've looked at
https://sigs.centos.org/hyperscale/content/repositories/main/
That seems to be focus on CentoS 8 Stream. The package versions in 9 are actually higher than what's mentioned there.
We definitely have stuff in CentOS Stream 9, that page needs to be updated to reflect that it's in CentOS Stream 9 too. But a lot of the backport work has been pushed into CentOS Stream 9 itself rather than forking packages, in large part to the less awful contribution process.
We just released a bunch of updated content for CentOS Stream 9 in the past couple of weeks even.
-- 真実はいつも一つ！/ Always, there's only one truth!
On Wed, 15 Mar 2023 14:46:31 -0400 Neal Gompa ngompa13@gmail.com wrote:
On Wed, Mar 15, 2023 at 1:37 PM Florian Weimer fweimer@redhat.com wrote:
- Josh Boyer:
We discussed this at the CentOS Board meeting last week and generally agreed with the overall idea of the SIG. There were two main questions that came up:
- What is the contribution model the SIG is envisioning? How will
people be able to participate in the SIG?
Could you add this to the proposal?
I don't have a live document for the proposal right now.
The general CentOS collaboration mechanism is a bit unclear to me, to be honest. It doesn't seem to be like Fedora, where anyone can create an account and try to get a package into the distribution. Not that this is something I expect as part of this effort; the focus should be on building existing stuff in new and exciting ways.
Anyway, here are some concrete items for collaboration:
I hope that performance experts will be able to share benchmarking results based on the optimized builds the SIG produces, and investigate glitches we encounter. Any kind of testing will help.
Some packages will fail to build with -march=x86-64-v3. Triaging these failures, reporting them upstream etc., will be required, and that part does not require any special privileges (as long as the build logs and buildroots are public, which I expect them to be).
We would benefit from expert guidance on GCC parameter defaults, like preferred vector sizes for certain operations.
Once we explore delivery mechanisms, exploring how builds can be consumed by end users in custom CI systems etc., and feedback on that will be helpful.
Do you think this provides some clarification?
- There are existing Intel people working in the Hyperscaler SIG on
some optimized libraries. How would the Hyperscaler SIG coordinate with the X86 SIG in this regard? There is a desire to share work and not duplicate effort.
Is it more than just zlib?
https://cbs.centos.org/koji/packages?tagID=2620
zlib upstream doesn't take architecture optimization patches, and maintaining the downstream patches (some of them are in CentOS proper already) is an ongoing hassle.
zlib-ng does, and the guy working on the zlib package in Hyperscale is interested in proposing switching Fedora's zlib implementation to zlib-ng partly for that reason. Another alternative is using zstd as a zlib replacement, which we're also considering for a proposal to Fedora.
is there a place where the plans can be tracked? I am sure many HW/CPU vendors are or would be interested in zlib-ng replacing zlib.
Dan
-
Brian Stinson -
Dan Horák -
Fabian Arrotin -
Florian Weimer -
Johnny Hughes -
Josh Boyer -
Marcin Juszkiewicz -
Neal Gompa -
Shaun McCance