hi guys,
we need a way to build embargo'd content like reimzul's priv-build, that does not make stuff public till a specific date ( the actual push to public can be manual );
how would we achieve this on cbs ? I guess the git stuff is easier since it can just be a scratch build from srpm and people need not push the git content back to git.centos.org till embargo expiry.
- KB
Currently koji doesn't have any read access controls, or at least not any related to builds. Most folks would probably either use a private koji, or just run mock against koji's yum repos.
Are we really only talking about a scratch build type scenario, or do folks want a real build that is hidden? Difference being that in the former case, you'd end up running a non-scratch rebuild after the embargo lifts, while in the latter you'd flip a switch and have the build appear.
The latter case is much harder (and also begs for embargo lockdown in source control), but neither is trivial to implement.
On Thu, Sep 25, 2014 at 12:25 PM, Karanbir Singh mail-lists@karan.org wrote:
hi guys,
we need a way to build embargo'd content like reimzul's priv-build, that does not make stuff public till a specific date ( the actual push to public can be manual );
how would we achieve this on cbs ? I guess the git stuff is easier since it can just be a scratch build from srpm and people need not push the git content back to git.centos.org till embargo expiry.
- KB
-- Karanbir Singh +44-207-0999389 | http://www.karan.org/ | twitter.com/kbsingh GnuPG Key : http://www.karan.org/publickey.asc _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org http://lists.centos.org/mailman/listinfo/centos-devel
On 09/25/2014 08:50 PM, Mike McLean wrote:
Currently koji doesn't have any read access controls, or at least not any related to builds. Most folks would probably either use a private koji, or just run mock against koji's yum repos.
Are we really only talking about a scratch build type scenario, or do folks want a real build that is hidden? Difference being that in the former case, you'd end up running a non-scratch rebuild after the embargo lifts, while in the latter you'd flip a switch and have the build appear.
The latter case is much harder (and also begs for embargo lockdown in source control), but neither is trivial to implement.
What we really need is a way for $person to build 'stuff' without it going public. If that means the sources cant hit git.centos.org then thats ok, we just need to make sure folks understand that no-commit-to-git.c.o till public.
If that then imples that koji's targets can only be used once stuff is public, that too might be fine ( as long as the person has had the change to test builds against the same target in the past ).
Our exposure in this case would then be limited to the time it takes to build something, and since the scratch builds will use the same repos - we can be fairly confident (!) about result; we'd still test it, but we can go into the testing with a higher level of confidence.
On 09/26/2014 02:33 PM, Karanbir Singh wrote:
Currently koji doesn't have any read access controls, or at least not any related to builds. Most folks would probably either use a private koji, or just run mock against koji's yum repos.
Are we really only talking about a scratch build type scenario, or do folks want a real build that is hidden? Difference being that in the former case, you'd end up running a non-scratch rebuild after the embargo lifts, while in the latter you'd flip a switch and have the build appear.
The latter case is much harder (and also begs for embargo lockdown in source control), but neither is trivial to implement.
What we really need is a way for $person to build 'stuff' without it going public. If that means the sources cant hit git.centos.org then thats ok, we just need to make sure folks understand that no-commit-to-git.c.o till public.
If that then imples that koji's targets can only be used once stuff is public, that too might be fine ( as long as the person has had the change to test builds against the same target in the past ).
Our exposure in this case would then be limited to the time it takes to build something, and since the scratch builds will use the same repos - we can be fairly confident (!) about result; we'd still test it, but we can go into the testing with a higher level of confidence.
reviving an old thread - we're again in a situation where we need SIGs to be able to build security updates and do some testing around content that is under embargo.
Depending on how much work this is, do we go down the route of replacing one of the koji builders with a reimzul like instance that runs private builds ? reimzul can generate per-build htaccess entries for basic-auth, that way ( apart from folks who have shell access to the machine ), there is some level of privacy as well.
If we do go down that route, we'd need a way for koji to import those builds at a later stage, when they need to be signed and released.
On Mar 05 11:41, Karanbir Singh wrote:
On 09/26/2014 02:33 PM, Karanbir Singh wrote:
Currently koji doesn't have any read access controls, or at least not any related to builds. Most folks would probably either use a private koji, or just run mock against koji's yum repos.
Are we really only talking about a scratch build type scenario, or do folks want a real build that is hidden? Difference being that in the former case, you'd end up running a non-scratch rebuild after the embargo lifts, while in the latter you'd flip a switch and have the build appear.
The latter case is much harder (and also begs for embargo lockdown in source control), but neither is trivial to implement.
What we really need is a way for $person to build 'stuff' without it going public. If that means the sources cant hit git.centos.org then thats ok, we just need to make sure folks understand that no-commit-to-git.c.o till public.
If that then imples that koji's targets can only be used once stuff is public, that too might be fine ( as long as the person has had the change to test builds against the same target in the past ).
Our exposure in this case would then be limited to the time it takes to build something, and since the scratch builds will use the same repos - we can be fairly confident (!) about result; we'd still test it, but we can go into the testing with a higher level of confidence.
reviving an old thread - we're again in a situation where we need SIGs to be able to build security updates and do some testing around content that is under embargo.
Depending on how much work this is, do we go down the route of replacing one of the koji builders with a reimzul like instance that runs private builds ? reimzul can generate per-build htaccess entries for basic-auth, that way ( apart from folks who have shell access to the machine ), there is some level of privacy as well.
If we do go down that route, we'd need a way for koji to import those builds at a later stage, when they need to be signed and released.
When we brought this up at the CBS meeting I remember talking about having the SIGs do local mockbuilds (perhaps on devcloud?) and then commit to their branches after the embargo is lifted. Are there some requirements that aren't covered by that workflow?
Brian
-- Brian Stinson bstinson@ksu.edu | IRC: bstinson | Bitbucket/Twitter: bstinsonmhk
On 05/03/15 15:02, Brian Stinson wrote:
On Mar 05 11:41, Karanbir Singh wrote:
On 09/26/2014 02:33 PM, Karanbir Singh wrote:
Currently koji doesn't have any read access controls, or at least not any related to builds. Most folks would probably either use a private koji, or just run mock against koji's yum repos.
Are we really only talking about a scratch build type scenario, or do folks want a real build that is hidden? Difference being that in the former case, you'd end up running a non-scratch rebuild after the embargo lifts, while in the latter you'd flip a switch and have the build appear.
The latter case is much harder (and also begs for embargo lockdown in source control), but neither is trivial to implement.
What we really need is a way for $person to build 'stuff' without it going public. If that means the sources cant hit git.centos.org then thats ok, we just need to make sure folks understand that no-commit-to-git.c.o till public.
If that then imples that koji's targets can only be used once stuff is public, that too might be fine ( as long as the person has had the change to test builds against the same target in the past ).
Our exposure in this case would then be limited to the time it takes to build something, and since the scratch builds will use the same repos - we can be fairly confident (!) about result; we'd still test it, but we can go into the testing with a higher level of confidence.
reviving an old thread - we're again in a situation where we need SIGs to be able to build security updates and do some testing around content that is under embargo.
Depending on how much work this is, do we go down the route of replacing one of the koji builders with a reimzul like instance that runs private builds ? reimzul can generate per-build htaccess entries for basic-auth, that way ( apart from folks who have shell access to the machine ), there is some level of privacy as well.
If we do go down that route, we'd need a way for koji to import those builds at a later stage, when they need to be signed and released.
When we brought this up at the CBS meeting I remember talking about having the SIGs do local mockbuilds (perhaps on devcloud?) and then commit to their branches after the embargo is lifted. Are there some requirements that aren't covered by that workflow?
there is no way to prep-ahead and test-ahead using this, also it means folks need to be around and standing by as embargo is lifted. in many cases, thats not possible.
I'm assuming that you meant commit to their branch, then trigger a build, and then do the testing.
On Mar 05 15:09, Karanbir Singh wrote:
On 05/03/15 15:02, Brian Stinson wrote:
On Mar 05 11:41, Karanbir Singh wrote:
On 09/26/2014 02:33 PM, Karanbir Singh wrote:
Currently koji doesn't have any read access controls, or at least not any related to builds. Most folks would probably either use a private koji, or just run mock against koji's yum repos.
Are we really only talking about a scratch build type scenario, or do folks want a real build that is hidden? Difference being that in the former case, you'd end up running a non-scratch rebuild after the embargo lifts, while in the latter you'd flip a switch and have the build appear.
The latter case is much harder (and also begs for embargo lockdown in source control), but neither is trivial to implement.
What we really need is a way for $person to build 'stuff' without it going public. If that means the sources cant hit git.centos.org then thats ok, we just need to make sure folks understand that no-commit-to-git.c.o till public.
If that then imples that koji's targets can only be used once stuff is public, that too might be fine ( as long as the person has had the change to test builds against the same target in the past ).
Our exposure in this case would then be limited to the time it takes to build something, and since the scratch builds will use the same repos - we can be fairly confident (!) about result; we'd still test it, but we can go into the testing with a higher level of confidence.
reviving an old thread - we're again in a situation where we need SIGs to be able to build security updates and do some testing around content that is under embargo.
Depending on how much work this is, do we go down the route of replacing one of the koji builders with a reimzul like instance that runs private builds ? reimzul can generate per-build htaccess entries for basic-auth, that way ( apart from folks who have shell access to the machine ), there is some level of privacy as well.
If we do go down that route, we'd need a way for koji to import those builds at a later stage, when they need to be signed and released.
When we brought this up at the CBS meeting I remember talking about having the SIGs do local mockbuilds (perhaps on devcloud?) and then commit to their branches after the embargo is lifted. Are there some requirements that aren't covered by that workflow?
there is no way to prep-ahead and test-ahead using this, also it means folks need to be around and standing by as embargo is lifted. in many cases, thats not possible.
I'm assuming that you meant commit to their branch, then trigger a build, and then do the testing.
Something like:
1) Create VM on devcloud 2) Pull package from git.c.o 3) Patch with embargoed content 4) `centpkg mockbuild` 5) Local tests - Should be valid since they would be building against koji repos 6) (Once the Embargo is lifted) Commit, push, and build in koji 7) CI Tests? 8) Tag/Sign/Release
Looking back, that was our "temporary" workflow we talked about in our meeting in September[1].
For the permanent solution (whether reimzul or something else) we may want to look at some convenience methods in the client tools.
Brian
[1]http://www.centos.org/minutes/2014/september/centos-devel.2014-09-29-13.01.l...
On Thu, Mar 5, 2015 at 4:11 PM, Brian Stinson bstinson@ksu.edu wrote:
When we brought this up at the CBS meeting I remember talking about having the SIGs do local mockbuilds (perhaps on devcloud?) and then commit to their branches after the embargo is lifted. Are there some requirements that aren't covered by that workflow?
there is no way to prep-ahead and test-ahead using this, also it means folks need to be around and standing by as embargo is lifted. in many cases, thats not possible.
I'm assuming that you meant commit to their branch, then trigger a build, and then do the testing.
Something like:
- Create VM on devcloud
- Pull package from git.c.o
- Patch with embargoed content
- `centpkg mockbuild`
- Local tests
- Should be valid since they would be building against koji repos
- (Once the Embargo is lifted) Commit, push, and build in koji
- CI Tests?
- Tag/Sign/Release
One of the main issues with this is how long it takes from the time the public embargo is lifted until there are signed packages ready for users to download. And as KB said, it also requires a dev to be able to actually available to start / babysit / &c the whole process at the specific time the embargo is lifted, which may not be very practical.
Ideally we'd like to be able to have binaries in the public repos as soon as the embargo is lifted. Furthermore, in the case of Xen at least, we are (in general) allowed to share the binary packages with other members of the pre-disclosure list. (There are a number of public cloud providers who use CentOS, for example.) It would be nice to be able to provide them with the same (signed) binary we intend to release when the embargo period is lifted.
-George
how would we achieve this on cbs ? I guess the git stuff is easier since it can just be a scratch build from srpm and people need not push the git content back to git.centos.org till embargo expiry.
Scratch builds are public in Koji, you can't use them for embargoed content, random example http://cbs.centos.org/koji/taskinfo?taskID=1943
Cheers, Alan
On 09/25/2014 11:25 AM, Karanbir Singh wrote:
hi guys,
we need a way to build embargo'd content like reimzul's priv-build, that does not make stuff public till a specific date ( the actual push to public can be manual );
how would we achieve this on cbs ? I guess the git stuff is easier since it can just be a scratch build from srpm and people need not push the git content back to git.centos.org till embargo expiry.
Devil's advocate here,
How much of an issue is this? As far as I know, fedora doesn't deal with embargoed code at all. Is this something we need to care about, or can we simply adopt the fedora approach and state: Use the embargo time to come up with a plan of action, and build/push once public.
On 03/11/2015 12:01 PM, Jim Perrin wrote:
On 09/25/2014 11:25 AM, Karanbir Singh wrote:
hi guys,
we need a way to build embargo'd content like reimzul's priv-build, that does not make stuff public till a specific date ( the actual push to public can be manual );
how would we achieve this on cbs ? I guess the git stuff is easier since it can just be a scratch build from srpm and people need not push the git content back to git.centos.org till embargo expiry.
Devil's advocate here,
How much of an issue is this? As far as I know, fedora doesn't deal with embargoed code at all. Is this something we need to care about, or can we simply adopt the fedora approach and state: Use the embargo time to come up with a plan of action, and build/push once public.
Its a pretty big deal I feel - its one thing that will likely keep the actual CentOS buildservices from ever being opened up.
We cant use the fedora angle at all - they have a very different problem space than we do.
On Wed, Mar 11, 2015 at 12:07 PM, Karanbir Singh mail-lists@karan.org wrote:
On 03/11/2015 12:01 PM, Jim Perrin wrote:
On 09/25/2014 11:25 AM, Karanbir Singh wrote:
hi guys,
we need a way to build embargo'd content like reimzul's priv-build, that does not make stuff public till a specific date ( the actual push to public can be manual );
how would we achieve this on cbs ? I guess the git stuff is easier since it can just be a scratch build from srpm and people need not push the git content back to git.centos.org till embargo expiry.
Devil's advocate here,
How much of an issue is this? As far as I know, fedora doesn't deal with embargoed code at all. Is this something we need to care about, or can we simply adopt the fedora approach and state: Use the embargo time to come up with a plan of action, and build/push once public.
Its a pretty big deal I feel - its one thing that will likely keep the actual CentOS buildservices from ever being opened up.
We cant use the fedora angle at all - they have a very different problem space than we do.
What does CentOS do about security patches on the core? Does it strive to have updates built and tested as soon as the embargo is lifted?
It seems like at least in the Xen4CentOS case, being able to provide pre-embargo binaries to cloud providers will be a big win, and having updated packages quickly would be pretty critical. I don't want to say it *must* be done, but it certainly seems desirable to me.
-George
On 03/11/2015 04:03 PM, George Dunlap wrote:
What does CentOS do about security patches on the core? Does it strive to have updates built and tested as soon as the embargo is lifted?
depends on the code- in some cases, where upstreams are going to work with us we do pre-patch and stage content that we can control code for - this tends to mostly be content outside of the distro.
For the distro itself, we rely on the source drops from upstream - but we dont really test or need to own too much of the code lifecycle there.
Building is just one part of this. We also have to think about source control and test infrastructure.
For read access, a git repo is all or nothing, so I presume we'll need a system where we handle embargoed code in a separate, locked-down git repo (not just a branch) and merge that in after the embargo lifts. Can we make this work with gitblit? Can we make such a workflow tolerable in centpkg?
Similarly, with test infra we'll need mechanisms to test the embargoed builds without making them visible.
The access controls for all of this will have to line up.
Speaking of access controls. How fine grained do we need this to be? One big "access to embargo" group? Per-sig? Per-package? Finer?
As far as the build system goes, I see a few options: 1) separate (small) koji instance for embargoed builds. This would only really work for a coarse grained access plan 2) a reimzul instance as kb suggests 3) modify koji to support read access controls (highly nontrivial and invasive, but maybe we could do it anyway)
On Wed, Mar 11, 2015 at 12:54 PM, Karanbir Singh mail-lists@karan.org wrote:
On 03/11/2015 04:03 PM, George Dunlap wrote:
What does CentOS do about security patches on the core? Does it strive to have updates built and tested as soon as the embargo is lifted?
depends on the code- in some cases, where upstreams are going to work with us we do pre-patch and stage content that we can control code for - this tends to mostly be content outside of the distro.
For the distro itself, we rely on the source drops from upstream - but we dont really test or need to own too much of the code lifecycle there.
-- Karanbir Singh +44-207-0999389 | http://www.karan.org/ | twitter.com/kbsingh GnuPG Key : http://www.karan.org/publickey.asc _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org http://lists.centos.org/mailman/listinfo/centos-devel
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 03/12/2015 06:48 AM, Mike McLean wrote:
Building is just one part of this. We also have to think about source control and test infrastructure.
For read access, a git repo is all or nothing, so I presume we'll need a system where we handle embargoed code in a separate, locked-down git repo (not just a branch) and merge that in after the embargo lifts. Can we make this work with gitblit? Can we make such a workflow tolerable in centpkg?
Similarly, with test infra we'll need mechanisms to test the embargoed builds without making them visible.
The access controls for all of this will have to line up.
Presuming we won't need this test infra up all the time, only when there is something to test, can we keep a VM &/or container process on ice, only bringing it up when something needs to be tested?
This is around making do with what we have currently. If we need dedicated hardware, that's something to think through. E.g., I wonder if there are any potential sponsors who see value in this process?
Speaking of access controls. How fine grained do we need this to be? One big "access to embargo" group? Per-sig? Per-package? Finer?
As far as the build system goes, I see a few options: 1) separate (small) koji instance for embargoed builds. This would only really work for a coarse grained access plan 2) a reimzul instance as kb suggests 3) modify koji to support read access controls (highly nontrivial and invasive, but maybe we could do it anyway)
We could start with a reimzul instance, or even multiple ones, kept on ice until needed. Then move to a Koji instance done similarly, if needed. Long term, I'd definitely like to see us improving Koji in these ways so we can eventually retire reimzul and stand-alone mini-Kojis.
- - Karsten
On Wed, Mar 11, 2015 at 12:54 PM, Karanbir Singh mail-lists@karan.org wrote:
On 03/11/2015 04:03 PM, George Dunlap wrote:
What does CentOS do about security patches on the core? Does it strive to have updates built and tested as soon as the embargo is lifted?
depends on the code- in some cases, where upstreams are going to work with us we do pre-patch and stage content that we can control code for - this tends to mostly be content outside of the distro.
For the distro itself, we rely on the source drops from upstream
- but we dont really test or need to own too much of the code
lifecycle there.
-- Karanbir Singh +44-207-0999389 | http://www.karan.org/ | twitter.com/kbsingh GnuPG Key : http://www.karan.org/publickey.asc _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org http://lists.centos.org/mailman/listinfo/centos-devel
_______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org http://lists.centos.org/mailman/listinfo/centos-devel
- -- Karsten 'quaid' Wade .^\ CentOS Doer of Stuff http://TheOpenSourceWay.org \ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41
-
Alan Pevec -
Brian Stinson -
George Dunlap -
Jim Perrin -
Karanbir Singh -
Karsten Wade -
Mike McLean