To the Linux Community at Large:
I reported to this list back in January, 2010 that the standard x86_64 kernel, when built from the src.rpm and modified for AMD K8 / K10 Extensions would not build. I reported this here and to Red Hat via Bugzilla ID number 558367. RH AS / Centos 5.3 worked fine. That was Centos 5.4 / Red Hat Enterprise AS 5 Update 4. Today we tried to optimize Red Hat Enterprise AS 5 Update 5. Same problem. At last check all kernels from 2.6.18-10 to 2.6.18-194 won't build with AMD specific optimizations.
For 9 years now we have used and supported Centos and before that white box. We have sold, installed and provided consulting on Red Hat Linux and RH Enterprise Linux, Centos, Scientific Linux, OpenSUSE and SUSE Linux Enterprise.
In April this year, the bug was still marked as new and we had to recommend to a customer to use SUSE Linux Enterprise instead of Red Hat Enterprise, as Novell's product supports optimizations in its kernel for AMD Opterons. 12 SLES Enterprise copies were sold to the customer. I'm sure that would have easily paid for the fix required in Redhat's kernel.
I am beginning to wonder if Red Hat is getting too big? Or that it just does not care. Other ideas less pleasant come to mind.... Today, the old bug was still marked as new (6+ months and counting). I entered a new bug report for RH 5.5 for the same issue. Is there no way, unless you are a huge customer, to get your bug listed as anything except LOW PRIORITY??
Now we are looking at the AMD G34 CPU's and are building some demo units. I think its time to benchmark these systems with the working - non optimized Red Hat / Centos Linux versus the optimized Opensuse / SLES Linux for standard server functions and publish them.
Has anyone else had this NON response from Red Hat for similar issues? I'd like to hear from them.
Thanks for reading....
On Thu, 8 Jul 2010 at 3:39pm, Seth Bardash wrote
I am beginning to wonder if Red Hat is getting too big? Or that it just does not care. Other ideas less pleasant come to mind.... Today, the old bug was still marked as new (6+ months and counting). I entered a new bug report for RH 5.5 for the same issue. Is there no way, unless you are a huge customer, to get your bug listed as anything except LOW PRIORITY??
It has been stated many times and on many fora that Red Hat's bugzilla is not a mechanism for support. They are under no obligation to address issues raised there. Is it nice when they do? Absolutely. Should you expect (nay, demand) it? Nope. The proper way to get Red Hat to address an issue is to open a ticket via your support contract with them.
Now we are looking at the AMD G34 CPU's and are building some demo units. I think its time to benchmark these systems with the working - non optimized Red Hat / Centos Linux versus the optimized Opensuse / SLES Linux for standard server functions and publish them.
While that may be interesting to compare distributions, I think it would do little to evaluate the benefit of the kernel CPU optimizations. There are just too many other variables. I would be very interested to see numbers comparing the exact same Red Hat distribution benchmarked with and without the kernel optimizations (you said 5.3 worked just fine). Do you have previous numbers on that showing a marked benefit?
On Thu, Jul 08, 2010 at 06:35:47PM -0400, Joshua Baker-LePain wrote:
It has been stated many times and on many fora that Red Hat's bugzilla is not a mechanism for support. They are under no obligation to address issues raised there. Is it nice when they do? Absolutely.
There are two issues you're conflating here. The first, paramount one is: Is Red Hat taking responsibility for bugs people have taken the effort to accurately report to them? This is a measure of any software project, totally separate from the issue of whether and for what the project leads provide paid support. In particular, if they are marketing this software to anyone - even if the person kind enough to report the bug is not a paying customer - they have a responsibility _to their paying customers_ to resolve all serious bugs in a timely manner - or at least to indicate in their bugzilla why they are rejecting fixing them.
This is an implicit contract that runs across all open source software projects. We can't pretend that Red Hat is ignorant of it. If they're chosing to ignore it this a violation of our community's ethics. Not that the rest of us are Boy Scouts either. Still it's worthy of discussion and complaint - the tribute vice owes to virtue and all that.
Besides, the poster here is making a serious point about Red Hat losing sales. Smart companies pay attention to that sort of detail. It's not being unkind to point out to them when they're missing out on a profit opportunity. They owe it not just to their customers, but to their shareholders, to stay awake on that.
Best, Whit
On Thu, 8 Jul 2010, Whit Blauvelt wrote:
This is an implicit contract that runs across all open source software projects.
"implicit contract" ?
You know 'Whit' -- you deserve congratulations
You were just promoted into the 'hot air windbag' 'internet lawyer' 'I am entitled to it, just because I breathe' category
How about celebrating your promotion by leaving this mailing list?
-- Russ herrold
On Thu, 8 Jul 2010 at 8:16pm, Whit Blauvelt wrote
On Thu, Jul 08, 2010 at 06:35:47PM -0400, Joshua Baker-LePain wrote:
It has been stated many times and on many fora that Red Hat's bugzilla is not a mechanism for support. They are under no obligation to address issues raised there. Is it nice when they do? Absolutely.
There are two issues you're conflating here. The first, paramount one is: Is Red Hat taking responsibility for bugs people have taken the effort to accurately report to them? This is a measure of any software project, totally separate from the issue of whether and for what the project leads provide paid support. In particular, if they are marketing this software to anyone - even if the person kind enough to report the bug is not a paying customer - they have a responsibility _to their paying customers_ to resolve all serious bugs in a timely manner - or at least to indicate in their bugzilla why they are rejecting fixing them.
To be clear here, the "bug" in question is not present in any binaries that Red Hat ships. None of their paying customers will ever experience this bug while running in a supported configuration. It's a case of "you broke it, you get to keep the pieces".
On Friday 09 July 2010, Joshua Baker-LePain wrote:
On Thu, 8 Jul 2010 at 8:16pm, Whit Blauvelt wrote
On Thu, Jul 08, 2010 at 06:35:47PM -0400, Joshua Baker-LePain wrote:
It has been stated many times and on many fora that Red Hat's bugzilla is not a mechanism for support. They are under no obligation to address issues raised there. Is it nice when they do? Absolutely.
There are two issues you're conflating here. The first, paramount one is: Is Red Hat taking responsibility for bugs people have taken the effort to accurately report to them? This is a measure of any software project, totally separate from the issue of whether and for what the project leads provide paid support. In particular, if they are marketing this software to anyone - even if the person kind enough to report the bug is not a paying customer - they have a responsibility _to their paying customers_ to resolve all serious bugs in a timely manner - or at least to indicate in their bugzilla why they are rejecting fixing them.
To be clear here, the "bug" in question is not present in any binaries that Red Hat ships.
To be fair, this is only true if you're refering to the fact that he could not recompile the kernel in a different way. If you consider the main bug to be that redhat doesn't provide the optimized kernel in the first place...
/Peter
None of their paying customers will ever experience this bug while running in a supported configuration. It's a case of "you broke it, you get to keep the pieces".
On Thu, Jul 08, 2010 at 08:16:05PM -0400, Whit Blauvelt wrote:
Besides, the poster here is making a serious point about Red Hat losing sales.
If the OP is making a serious point about RH losing sales, perhaps he should tell RedHat about it, instead of posting trollbait to an unrelated mailing list (and addressing it to "the Linux community at large", which the CentOS mailing list is surely not).
You mention some sort of implicit open source contract: the other side of that is the user, who needs to report problems properly and expect an appropriate response. Would you complain to RedHat support about a problem with packages in the centosplus repository?
#!perl use "what everyone already else said about unsupported configurations";
--keith
On 07/08/10 2:39 PM, Seth Bardash wrote:
To the Linux Community at Large:
I reported to this list back in January, 2010 that the standard x86_64 kernel, when built from the src.rpm and modified for ...
I don't understand how you think that this is a bug. you modified something and it broke. the provided redhat binary kernels work fine as advertised.
On 7/8/2010 6:25 PM, John R Pierce wrote:
On 07/08/10 2:39 PM, Seth Bardash wrote:
To the Linux Community at Large:
I reported to this list back in January, 2010 that the standard x86_64 kernel, when built from the src.rpm and modified for ...
I don't understand how you think that this is a bug. you modified something and it broke. the provided redhat binary kernels work fine as advertised.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
OK, There seems to be some confusion about my motives / objectives / point of view. Please let me clear those up and I apologize if my approach was askew or this answer is too long winded.
Lets start with why I posted:
I posted my observations ( and reported the bugs) so that anyone else running RH AS > 5.3 or Centos that was trying to optimize their kernel for AMD CPU's would know there was a problem. A problem that Red Hat caused and has chosen not to address. That was the only reason I posted! To inform.
Red Hat takes a standard kernel and modifies it, extensively. The standard kernel.org kernel works fine and supports this function. What is standard and supported in a kernel is decided by kernel.org. What is standard and supported in a Red Hat kernel is decided by Red Hat. The standard binary is built with the Generic X84_64 option selected. Red Hat's source code also includes an AMD Opteron option and an Intel EM64T option. Since the changes made to Red Hat's kernel by Red Hat since kernel 2.6.18-9 have broken one of these options but not the other two options I thought it might be useful to Red Hat and the Centos community to know that the AMD option is broken. The Generic x86_64 option and the Intel option work and compile fine. I tested these also. This is a bug, not something I broke. If I broke it, I would feel an obligation to fix it. Red Hat made the changes to the kernel source that broke this option.
Next, I have only posted to this list over the past 5 years when I could report a problem I thought might be of interest or add, help fix or supply code for a work around, on an issue with RH Linux or Centos. So I think the comment about troll bait is out of line. Just my opinion, I could be wrong. :)
I don't expect RH to do anything except what is in Red Hat's interests.
This can be influenced (very slightly) by non-paying users, fully supported paying users and to some extent Systems Integrators (like my company) if they are listening and if, again, it is in Red Hat's interests.
As a business owner, I would want to know about a problem with my product, especially if it costs me customers or profit or future sales. That is why we supplied some of the early drivers for 3ware cards for the Centos 3 OS. Our customers needed them, we supplied them free of charge. We have done this many times for many different linux based systems.
I have received responses off this list that suggest that there are other people and organizations that indeed use optimized kernels in conjunction with RH AS and Centos. Some use a modified RH kernel, some use a newer standard kernel from kernel.org with options changed, some use the real-time extensions to the standard kernel. All of them have found that it provides better performance for THEIR applications. We have done similar work for many customers.
When we changed the kernel and break it, we fix it.
My intent was to inform and hear from people that had similar issues and to learn what they might have done to work around them. Not to cause a debate on business practices, criticize Red Hat or inflame the Centos community.
Thanks for reading.........
On Fri, 9 Jul 2010 at 8:32am, Seth Bardash wrote
My intent was to inform and hear from people that had similar issues and to learn what they might have done to work around them. Not to cause a debate on business practices, criticize Red Hat or inflame the Centos community.
I appreciate your clarification. I think what got folks up in arms was an impression from your original email (from, e.g., the all caps bits and implications of them having gotten too big) that you were indeed criticizing RH.
And I'd still be interested (as in, genuinely curious, not skeptical) to hear what sorts of applications benefit from optimized kernels (HPC? I/O intense?) and what kind of performance increases one can get.
And I'd still be interested (as in, genuinely curious, not skeptical) to hear what sorts of applications benefit from optimized kernels (HPC? I/O intense?) and what kind of performance increases one can get.
I'd be interested as well. I can see HPC applications potentially benefiting but what about "real world" applications? Web, Email, Databases, App Servers, etc
I'm specifically curious about it as I spent some time in Gentoo Land and optimized everything seemed to be the mantra. In my own experience the *apparent* performance difference between a system compiled as "AMD optimal" vs "Standard x86-64" was minimal to the point of being unnoticeable.
On Thursday 08 July 2010, Seth Bardash wrote:
To the Linux Community at Large:
I reported to this list back in January, 2010 that the standard x86_64 kernel, when built from the src.rpm and modified for AMD K8 / K10 Extensions would not build. I reported this here and to Red Hat via Bugzilla ID number 558367. RH AS / Centos 5.3 worked fine. That was Centos 5.4 / Red Hat Enterprise AS 5 Update 4. Today we tried to optimize Red Hat Enterprise AS 5 Update 5. Same problem. At last check all kernels from 2.6.18-10 to 2.6.18-194 won't build with AMD specific optimizations.
Do you have any data to back up this position? (that optimizing the kernel for a specific processor is of any significant real world use)
...
Now we are looking at the AMD G34 CPU's and are building some demo units. I think its time to benchmark these systems with the working - non optimized Red Hat / Centos Linux versus the optimized Opensuse / SLES Linux for standard server functions and publish them.
Too many (other) variables change you'd not be able to say that it had to do with the processor specific optimization of the kernel.
For this you'd need to benchmark the same dist but with optimized and non-optimized kernel.
Also, to impress me, the benchmark in question would have to be a relevant high level benchmark, not kernel micro benchmarks.
/Peter
-
Drew -
John R Pierce -
Joshua Baker-LePain -
Keith Keller -
Peter Kjellstrom -
R P Herrold -
Seth Bardash -
Whit Blauvelt