Hi,
i would like to add some more to the discussion started at http://lists.centos.org/pipermail/centos-devel/2015-April/013163.html
1. On the plot attached to http://bugs.centos.org/view.php?id=8447 one can see that since the CentOS 7 release the number of unresolved issues on bugs.centos.org has increased, and the number is currently more than 50 unresolved issues per month. Many issues do not obtain any attention (nothing in the notes). This continues for several months, and is an unprecedented situation.
For me it shows that the CentOS community has not enough resources to deal with the reported issues. From this point of view it would be better to have CentOS issues integrated into RHEL's bugzilla, but the decision should also take into account Red Hat's long time plans for CentOS, unknown to me.
2. A single example I would like to bring up is the fate of http://bugs.centos.org/view.php?id=8249 The reporter made a substantial effort to collect usability issues encountered during an installation of CentOS, got asked to report the issues at bugzilla what he did, and there this got (politely) closed https://bugzilla.redhat.com/show_bug.cgi?id=1197377 This seems to be his only bug at bugzilla.redhat.com.
Maybe if CentOS was at bugzilla then CentOS developers could focus more on the "open-source" way of dealing with people's reports, which will counteract a bit Red Hat's enforcement of compliance with it's strategies.
3. One more point, and it has to do with the way Fedora/EPEL package updates are handled. When I update an RPM package fixing a bug for Fedora/EPEL the update almost never gets any reviews. The update is sitting for some fixed amount of time (2 weeks for EPEL) and after that I push it to stable (still without any review). I'll bring the famous case here what the result of such releases could potentially be: https://bugzilla.redhat.com/show_bug.cgi?id=1054350 (actually I don't know if the offending release was reviewed or not). Or another case which affected me: https://bugzilla.redhat.com/show_bug.cgi?id=1063493 Red Hat changed major version of software (mpich2 -> mpich3) which resulted in a couple months of empty running reviews (2 weeks each) at EPEL in order to fix all dependencies. I'm not familiar with the role CentOS could have in the process of preparation of new RHEL updates, but if there is anything that could be done to improve the RPM package update process, it should be considered as an important factor in case of merging CentOS issues to bugzilla.
Best regards,
Marcin
On 04/14/2015 03:17 PM, Marcin Dulak wrote:
Hi,
i would like to add some more to the discussion started at http://lists.centos.org/pipermail/centos-devel/2015-April/013163.html
On the plot attached to http://bugs.centos.org/view.php?id=8447 one can see that since the CentOS 7 release the number of unresolved issues on bugs.centos.org has increased, and the number is currently more than 50 unresolved issues per month. Many issues do not obtain any attention (nothing in the notes). This continues for several months, and is an unprecedented situation.
How is it unprecedented?
For me it shows that the CentOS community has not enough resources to deal with the reported issues.
You're right. We could absolutely use more community members willing to step up and help triage/fix/duplicate the bugs.
From this point of view it would be better to have CentOS issues integrated into RHEL's bugzilla,
Unsure what you mean here. Putting our bugs in bugzilla would simply mean we move from not responding to them on bugs.centos to not responding to them in bugzilla. We don't get any additional resources simply by using bugzilla.
A single example I would like to bring up is the fate of http://bugs.centos.org/view.php?id=8249 The reporter made a substantial effort to collect usability issues encountered during an installation of CentOS, got asked to report the issues at bugzilla what he did, and there this got (politely) closed https://bugzilla.redhat.com/show_bug.cgi?id=1197377 This seems to be his only bug at bugzilla.redhat.com.
Maybe if CentOS was at bugzilla then CentOS developers could focus more on the "open-source" way of dealing with people's reports, which will counteract a bit Red Hat's enforcement of compliance with it's strategies.
Elaborate here please? The core system does not change. That's been a distro fundamental from the beginning. If RH closes a bug, we inherit their change (or lack thereof). SIGs are the way for groups of interested people to work together and make those changes themselves.
One more point, and it has to do with the way Fedora/EPEL package updates are handled. When I update an RPM package fixing a bug for Fedora/EPEL the update almost never gets any reviews.
This is a Fedora/EPEL issue, and is something we as a distro don't have any control over. We've had discussions a few times with the Fedora/EPEL folks about this but there is no simple answer or immediate fix.
The update is sitting for some fixed amount of time (2 weeks for EPEL) and after that I push it to stable (still without any review). I'll bring the famous case here what the result of such releases could potentially be: https://bugzilla.redhat.com/show_bug.cgi?id=1054350 (actually I don't know if the offending release was reviewed or not). Or another case which affected me: https://bugzilla.redhat.com/show_bug.cgi?id=1063493 Red Hat changed major version of software (mpich2 -> mpich3) which resulted in a couple months of empty running reviews (2 weeks each) at EPEL in order to fix all dependencies.
So step in. Contribute feedback, jump on the EPEL-devel mailing list and request feedback for packages. Join the relevant irc channels and request/give feedback.
I'm not familiar with the role CentOS could have in the process of preparation of new RHEL updates,
Effectively 0. We see the updates when they land in git, the same as everyone else.
but if there is anything that could be done to improve the RPM package update process, it should be considered as an important factor in case of merging CentOS issues to bugzilla.
RHEL and EPEL are quite separate, so I don't really follow what you mean here.
In my eyes, there are two benefits from using rh's bugzilla vs our own tracker.
1. It's one less thing to manage. 2. Bugs that have upstream relevance could (in theory) be more easily tagged/cloned without asking the user to duplicate as we currently do. This is still a hypothetical, as we've not talked with the bugzilla folks yet to see how any of this would work, or what would be possible.
On Tue, Apr 14, 2015 at 4:51 PM, Jim Perrin jperrin@centos.org wrote:
So step in. Contribute feedback, jump on the EPEL-devel mailing list and request feedback for packages. Join the relevant irc channels and request/give feedback.
Some of us try. There is a serious learning curve for Fedora and EPEL to get involved in publishing patches to their code: I've tried on at least 3 distinct occassions, and gotten bogged down every time in the "koji" setups. "Look it up on the web" doesn't help, and IRC is not documentation. The variety of bugzillas and credentials needed for the multiple systems, CentOS, RHEL, Fedora, EPEL, etc. all get confusing.
I'm not familiar with the role CentOS could have in the process of preparation of new RHEL updates,
Effectively 0. We see the updates when they land in git, the same as everyone else.
I'm going to be very confused if you do not, individually, have RHEL licenses for early RPM and SRPM review. Are you saying that the git repo updates occur simultaneously, or before, RPM and SRPM publication for RHEL customers? I can imagine "clean room" reasons to avoid access for CentOS core developers, but as a DevOps guy, I'll be surprised.
but if there is anything that could be done to improve the RPM package update process, it should be considered as an important factor in case of merging CentOS issues to bugzilla.
RHEL and EPEL are quite separate, so I don't really follow what you mean here.
I agree. I personally find RHEL useless without EPEL these days, though. There are consistently too many perl and python modules and useful components backported from Fedora that I need to do even modest work. This especially includes 'mock', for cleanly building patched RHEL or CentOS packages.
In my eyes, there are two benefits from using rh's bugzilla vs our own tracker.
- It's one less thing to manage.
- Bugs that have upstream relevance could (in theory) be more easily
tagged/cloned without asking the user to duplicate as we currently do. This is still a hypothetical, as we've not talked with the bugzilla folks yet to see how any of this would work, or what would be possible.
If it's feasible, I'd appreciate it. Bugzilla is very flexible and sculptable to many different workflows, and I tend to submit patches and workarounds that would be good for both CentOS, RHEL, and Scientific Linux users to all see.
On 04/14/2015 05:46 PM, Nico Kadel-Garcia wrote:
On Tue, Apr 14, 2015 at 4:51 PM, Jim Perrin jperrin@centos.org wrote:
So step in. Contribute feedback, jump on the EPEL-devel mailing list and request feedback for packages. Join the relevant irc channels and request/give feedback.
Some of us try. There is a serious learning curve for Fedora and EPEL to get involved in publishing patches to their code: I've tried on at least 3 distinct occassions, and gotten bogged down every time in the "koji" setups. "Look it up on the web" doesn't help, and IRC is not documentation. The variety of bugzillas and credentials needed for the multiple systems, CentOS, RHEL, Fedora, EPEL, etc. all get confusing.
I'm not familiar with the role CentOS could have in the process of preparation of new RHEL updates,
Effectively 0. We see the updates when they land in git, the same as everyone else.
I'm going to be very confused if you do not, individually, have RHEL licenses for early RPM and SRPM review. Are you saying that the git repo updates occur simultaneously, or before, RPM and SRPM publication for RHEL customers? I can imagine "clean room" reasons to avoid access for CentOS core developers, but as a DevOps guy, I'll be surprised.
Stand by to be surprised ... the first time I see any code from Red Hat that goes into CentOS is when it lands in git.centos.org (or for older distros, ftp.redhat.com).
Community members of the QA channel can verify that we send information into that channel when updates are found on ftp or git. I then build and push the updates.
I do not have a RHEL subscription or access to RHEL SRPMS from inside RHN and I never have.
I build almost every SRPM that gets released into every CentOS version, and my access to these things is unchanged from what it was 1, 2, 5, or 10 years ago.
<snip>
Thanks, Johnny Hughes
On Tue, Apr 14, 2015 at 7:17 PM, Johnny Hughes johnny@centos.org wrote:
On 04/14/2015 05:46 PM, Nico Kadel-Garcia wrote:
On Tue, Apr 14, 2015 at 4:51 PM, Jim Perrin jperrin@centos.org wrote:
Effectively 0. We see the updates when they land in git, the same as everyone else.
I'm going to be very confused if you do not, individually, have RHEL licenses for early RPM and SRPM review. Are you saying that the git repo updates occur simultaneously, or before, RPM and SRPM publication for RHEL customers? I can imagine "clean room" reasons to avoid access for CentOS core developers, but as a DevOps guy, I'll be surprised.
Stand by to be surprised ... the first time I see any code from Red Hat that goes into CentOS is when it lands in git.centos.org (or for older distros, ftp.redhat.com).
Thank you for clarifying that. Since some CentOS key developers are now Red Hat employees, the workflow is not completely clear.
Community members of the QA channel can verify that we send information into that channel when updates are found on ftp or git. I then build and push the updates.
I do not have a RHEL subscription or access to RHEL SRPMS from inside RHN and I never have.
Lord, I have, precisely for development and debugging for both communities.
I build almost every SRPM that gets released into every CentOS version, and my access to these things is unchanged from what it was 1, 2, 5, or 10 years ago.
<snip>
Thanks, Johnny Hughes
Thanks for explaining that. I remain surprised: I actually publish tools for using 'reposync' to pull an internal mirror of RHEL repositories for customers with licenses for RHEL, partly for patching and building modified packages with 'mock' and publishing updates back to RHEL or, as appropriate, CentOS. It's also handy for internal updating against the frequently RHN yum repositories. The easy access to CentOS repositories, and more graceful and efficient rsync mirroring of that content, is actually a reason that some of my clients have used CentOS instead of RHEL.
Nico Kadel-Garcia
On Tue, Apr 14, 2015 at 10:51 PM, Jim Perrin jperrin@centos.org wrote:
On 04/14/2015 03:17 PM, Marcin Dulak wrote:
Hi,
i would like to add some more to the discussion started at http://lists.centos.org/pipermail/centos-devel/2015-April/013163.html
On the plot attached to http://bugs.centos.org/view.php?id=8447 one can see that since the CentOS 7 release the number of unresolved issues on bugs.centos.org has increased, and the number is currently more than 50 unresolved issues per month. Many issues do not obtain any attention (nothing in the notes). This continues for several months, and is an unprecedented situation.
How is it unprecedented?
it looks unprecedented to me on the plot. There has never been a time on bugs.centos.org with that many bugs left open per month for such a long period of time.
Best regards,
Marcin
For me it shows that the CentOS community has not enough resources to deal with the reported issues.
You're right. We could absolutely use more community members willing to step up and help triage/fix/duplicate the bugs.
From this point of view it would be better to have CentOS issues integrated into RHEL's bugzilla,
Unsure what you mean here. Putting our bugs in bugzilla would simply mean we move from not responding to them on bugs.centos to not responding to them in bugzilla. We don't get any additional resources simply by using bugzilla.
A single example I would like to bring up is the fate of http://bugs.centos.org/view.php?id=8249 The reporter made a substantial effort to collect usability issues encountered during an installation of CentOS, got asked to report the issues at bugzilla what he did, and there this got (politely) closed
https://bugzilla.redhat.com/show_bug.cgi?id=1197377
This seems to be his only bug at bugzilla.redhat.com.
Maybe if CentOS was at bugzilla then CentOS developers could focus more on the "open-source" way of dealing with people's reports, which will counteract a bit Red Hat's enforcement of compliance with it's strategies.
Elaborate here please? The core system does not change. That's been a distro fundamental from the beginning. If RH closes a bug, we inherit their change (or lack thereof). SIGs are the way for groups of interested people to work together and make those changes themselves.
One more point, and it has to do with the way Fedora/EPEL package updates are handled. When I update an RPM package fixing a bug for Fedora/EPEL the update almost never gets any reviews.
This is a Fedora/EPEL issue, and is something we as a distro don't have any control over. We've had discussions a few times with the Fedora/EPEL folks about this but there is no simple answer or immediate fix.
The update is sitting for some fixed amount of time (2 weeks for EPEL) and after that I push it to stable (still without any review). I'll bring the famous case here what the result of such releases could potentially be: https://bugzilla.redhat.com/show_bug.cgi?id=1054350 (actually I don't know if the offending release was reviewed or not). Or another case which affected me: https://bugzilla.redhat.com/show_bug.cgi?id=1063493 Red Hat changed major version of software (mpich2 -> mpich3) which resulted in a couple months of empty running reviews (2 weeks each) at EPEL in order to fix all dependencies.
So step in. Contribute feedback, jump on the EPEL-devel mailing list and request feedback for packages. Join the relevant irc channels and request/give feedback.
I'm not familiar with the role CentOS could have in the process of preparation of new RHEL updates,
Effectively 0. We see the updates when they land in git, the same as everyone else.
but if there is anything that could be done to improve the RPM package update process, it should be considered as an important factor in case of merging CentOS issues to bugzilla.
RHEL and EPEL are quite separate, so I don't really follow what you mean here.
In my eyes, there are two benefits from using rh's bugzilla vs our own tracker.
- It's one less thing to manage.
- Bugs that have upstream relevance could (in theory) be more easily
tagged/cloned without asking the user to duplicate as we currently do. This is still a hypothetical, as we've not talked with the bugzilla folks yet to see how any of this would work, or what would be possible.
-- Jim Perrin The CentOS Project | http://www.centos.org twitter: @BitIntegrity | GPG Key: FA09AD77 _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org http://lists.centos.org/mailman/listinfo/centos-devel
On 04/15/2015 06:09 AM, Marcin Dulak wrote:
On Tue, Apr 14, 2015 at 10:51 PM, Jim Perrin <jperrin@centos.org mailto:jperrin@centos.org> wrote:
On 04/14/2015 03:17 PM, Marcin Dulak wrote: > Hi, > > i would like to add some more to the discussion started at > http://lists.centos.org/pipermail/centos-devel/2015-April/013163.html > > 1. > On the plot attached to http://bugs.centos.org/view.php?id=8447 > one can see that since the CentOS 7 release the number of unresolved > issues on bugs.centos.org <http://bugs.centos.org> has increased, > and the number is currently more than 50 unresolved issues per month. > Many issues do not obtain any attention (nothing in the notes). > This continues for several months, and is an unprecedented situation. How is it unprecedented?it looks unprecedented to me on the plot. There has never been a time on bugs.centos.org http://bugs.centos.org with that many bugs left open per month for such a long period of time.
You do net seem to understand .. CentOS does NOT fix or close technical bugs that exist in both RHEL and CentOS. We only fix bugs that we create in the few packages we modify that are not in RHEL source code, if we introduce them.
CentOS rebuilds RHEL source code .. if there is a bug in the RHEL source code, CentOS fixes it when it is fixed in the RHEL source code and released.
Bugs.centos.org is a tool for the community to help each other fix, then report to Red Hat (if it is a bug in RHEL code). It is NOT a place to get support for CentOS.
CentOS does not now, nor have we ever provides support via bugs.centos.org. If there is a bug that effects you .. fix it and report what you did to fix it. Any and all support (if you want to use that term) for CentOS does now, and always has, come from the community.
<snip>
Thanks, Johnny Hughes
On 14/04/15 21:17, Marcin Dulak wrote:
Hi,
i would like to add some more to the discussion started at http://lists.centos.org/pipermail/centos-devel/2015-April/013163.html
On the plot attached to http://bugs.centos.org/view.php?id=8447 one can see that since the CentOS 7 release the number of unresolved issues on bugs.centos.org has increased, and the number is currently more than 50 unresolved issues per month. Many issues do not obtain any attention (nothing in the notes). This continues for several months, and is an unprecedented situation.
Many of the bugs that are raised on bugs.centos.org are reporting real errors in the packages and since CentOS only repackages what Redhat provides, this is really not the right place to report the problems. Those bugs that are reported say against the kernel resulting in a panic are not usually ones that CentOS will ever fix - the real solution is to report the bug on bugzilla.redhat.com or via Redhat Support channels. When a new point release comes out then many of those bugs could probably be closed ... well if the bug has been fixed upstream! The bugs that should be on bugs.centos.org should be of the "this works in RHEL but not in CentOS" and "this package is broken because it doesn't recreate the way that it's pacakged in RHEL" variety.
A single example I would like to bring up is the fate of http://bugs.centos.org/view.php?id=8249 The reporter made a substantial effort to collect usability issues encountered during an installation of CentOS, got asked to report the issues at bugzilla what he did, and there this got (politely) closed https://bugzilla.redhat.com/show_bug.cgi?id=1197377 This seems to be his only bug at bugzilla.redhat.com.
The reporter has probably voiced the thoughts that many of us have had when confronted with the "new improved" installer in RHEL/CentOS 7. But to bundle all those complaints up into one bug report is not the right way to do it, either in CentOS or in Redhat. One bug, one bug report. This is also a CentOS bug report that's a prime example of the above: CentOS don't write the installer, Redhat do. If there are bugs to be fixed then they need to be fixed upstream so it's really rather pointless reporting it on bugs.centos.org at all. All those complaints in that one bugzilla should have been split out into many bugzilla reports and then they might have been fixed individually.
Maybe if CentOS was at bugzilla then CentOS developers could focus more on the "open-source" way of dealing with people's reports, which will counteract a bit Red Hat's enforcement of compliance with it's strategies.
I'm not entirely sure what the open-source" way of dealing with people's reports is! If my experience on various freenode IRC channels is anything to go by it's just to say, "yeah, that's how it works and I can't be bothered to fix it but if you want to send me a patch then I'll consider it". Either that or outright rudeness and ridicule :-( ... no, actually the usual response is "compile it from git and if you can recreate it on that then maybe I'll look at it".
My own concern about using Redhat's bugzilla is that it's not known for being a speed demon. There are times when I've tried to use it where I've seen response times that feel like minutes but are probably not as long as that. How it would perform with the added load of handling bugs.centos.org traffic as well is anyone's guess.
In addition, I'm not sure that bugzilla is exactly user friendly. This doesn't mean that I'm a fan-boi of Mantis, I'm not sure that I'm very keen on that either and it has some quite significant strangenesses.
Trevor
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Hi,
as an (most times) just end user of EL6/7 I'd like to say that I would prefer bugzilla.redhat.com for centos bugs as well.
It makes shifting bugs between RHEL and CentOS very much simpler.
The reason many bugs get reported first vs CentOS ist people do not really know the relationship between RHEL and CentOS.
And the people who know the relationship have often no license to test if it's a CentOS or RHEL Bug (you would need a RHEL license to try to reproduce).
When you have some experience you can of course tell with a high probability if it's CentOS or upstream related.
TBH: Most (upstream)bugs tend to get closed anyway if someone finds out your not using RHEL. (This get's masked via sentences like "Please contact your support channel if you want to raise the priority of this problem")
I wish RedHat would improve this behaviour, but you can not really argue if you get it for free.
Back to the Bugzilla Question:
I really think bugzilla has way better features and usability than mantis, but that's just me, and of course you have the ability to shift a bug to epel/fedora/rhel if it turns out you reported it against the wrong product.
I think this might be increasingly important when you look at all the sig products which incorporate many projects already using bugzilla.redhat.com like ovirt, glusterfs and so on.
kind regards
Sven
On Wednesday, April 15, 2015, Sven Kieske svenkieske@gmail.com wrote:
The reason many bugs get reported first vs CentOS ist people do not really know the relationship between RHEL and CentOS.
Agreed. No matter how much the CentOS devs say this, a large number of users don't know this.
And the people who know the relationship have often no license to test if it's a CentOS or RHEL Bug (you would need a RHEL license to try to reproduce).
Note that most of the CentOS devs don't either, including those employed by Red Hat. They can request an employee sub, but don't have one by default, and that helps keep things clean.
TBH: Most (upstream)bugs tend to get closed anyway if someone finds out your not using RHEL. (This get's masked via sentences like "Please contact your support channel if you want to raise the priority of this problem")
If it's a legitimate bug (defect) then that shouldn't happen. If it's a RFE then, yes, that's proper - support and input into future direction (big or small) is a significant part of the RHEL value proposition (sales-y enough for you?). If you have an example of the former then let me know. That's a standing offer.
My take on the overall question is that if the CentOS team can get the flexibility needed for their own purposes then it seems like a big win. The project has always been tied to Red Hat, so worrying about the bug tracker being owned by RH seems rather silly. The ability to properly transfer bugs to upstream (be it RH or Fedora) and making the relationship more obvious and transparent to reporters cannot be overstated.
FYI, the usual reasons for RHEL bugs to be marked private is either confidential customer info or security. And it annoys employees too. I would hope that CentOS sourced, non-security bugs wouldn't fall into this, but really cannot say with any certainty. That is something that would need to be discussed with the RH Bz team and the CentOS devs.
On 04/15/2015 12:45 PM, Sven Kieske wrote:
TBH: Most (upstream)bugs tend to get closed anyway if someone finds out your not using RHEL. (This get's masked via sentences like "Please contact your support channel if you want to raise the priority of this problem")
I wish RedHat would improve this behaviour, but you can not really argue if you get it for free.
While I have seen this, generally I would disagree, although I have seen on numerous occasions that a fix would not be pushed until the next "dot" release (i.e. not a candidate for the bug fix stream for the current "dot" release), and in those cases I was asked that if I could open a RH issue with under my account then maybe it would happen, which is completely fair.
FWIW, I'm for coagulating the bug trackers.
Thanks, David
-
David Mansfield -
Jim Perrin -
Johnny Hughes -
Marcin Dulak -
Nico Kadel-Garcia -
Sven Kieske -
Tom Sorensen -
Trevor Hemsley