Morning, folks. Happy July 4th.
I'm hopping back onto centos-devel to raise some concerns about git.centos.org components. Overall it's working. There are things I don't like, but as things stand that's *my* problem. These issues, however, affect others, and I don't see a good bugzilla mentioned at git.centos.org itself.
1) Many repositories are being listed with excessive "/" in their names. It messes up alphabetization and can get quite confusing.
rpms/docker, for example, is listed as "https://git.centos.org/summary/?r=////rpms/docker.git"
2) The "show_possible_srpms.sh" script relies on checking for the word "import" in the git logs to determine SRPM versions, embedded in the git logs themselves. This raises the risk of *any* commit that uses that word for other reasons to report an invalid SRPM version number. As much as I dislike relying on a git log rather than a signed tag to do a build, please, refine this script to at least grep more carefully for 'import' as the leading word, and ideally sanity check the rest of the import line for the SRPM number.
Obligatory XKCD comic about sanitizing inputs: http://xkcd.com/327/
3) Anyone who attempts to replicate any of the git repositories for improved local access is at risk of corruption or the embedding of trojans in the local repository, due to the lack of GPG signed tags or similar verification of the contents. I realize that the use of the "package.medata" information provides git commit hash tags, which help verification, but that data *itself* can be reset in a trojaned local git repository.
Please consider the use of signed GPG tags for actual SRPM updates, rather than merely relying on '[package].metadata, to help assure provenance for people who may test or rebuild security components.
Thanks for the attention: I'll stick around for a while, to try and pay back the support for others working with this.
Nico Kadel-Garcia nkadel@gmail.com
On 2014-07-04 15:46, Nico Kadel-Garcia wrote:
- Many repositories are being listed with excessive "/" in their
names. It messes up alphabetization and can get quite confusing.
rpms/docker, for example, is listed as
It's not entirely clear what list you're talking about, but there's something severely wrong with the activity page. Just about everything is listed with and without a trailing ".git" and various duplicate slashes. There's currently 13 variants of anaconda (in the "active repositories" listing), two of which have a trailing slash, breaking the highlighting of the package name. The output of `centos.git.repolist.py -b c7` only has four anacondas listed...
On 07/04/2014 03:13 PM, Elias Persson wrote:
On 2014-07-04 15:46, Nico Kadel-Garcia wrote:
- Many repositories are being listed with excessive "/" in their
names. It messes up alphabetization and can get quite confusing.
rpms/docker, for example, is listed asIt's not entirely clear what list you're talking about, but there's something severely wrong with the activity page. Just about everything is listed with and without a trailing ".git" and various duplicate slashes. There's currently 13 variants of anaconda (in the "active repositories" listing), two of which have a trailing slash, breaking the highlighting of the package name. The output of `centos.git.repolist.py -b c7` only has four anacondas listed...
local caching issue - and people hitting git.centos.org with crawlers that dont understand urls cleanly.
the fix is ready and staged, I just dont want to upgrade that infra at this point. Its safe to ignore the extras /'s in the mean time.
- KB
On 07/04/2014 10:12 AM, Karanbir Singh wrote:
On 07/04/2014 03:13 PM, Elias Persson wrote:
On 2014-07-04 15:46, Nico Kadel-Garcia wrote:
- Many repositories are being listed with excessive "/" in their
names. It messes up alphabetization and can get quite confusing.
rpms/docker, for example, is listed asIt's not entirely clear what list you're talking about, but there's something severely wrong with the activity page. Just about everything is listed with and without a trailing ".git" and various duplicate slashes. There's currently 13 variants of anaconda (in the "active repositories" listing), two of which have a trailing slash, breaking the highlighting of the package name. The output of `centos.git.repolist.py -b c7` only has four anacondas listed...
local caching issue - and people hitting git.centos.org with crawlers that dont understand urls cleanly.
the fix is ready and staged, I just dont want to upgrade that infra at this point. Its safe to ignore the extras /'s in the mean time.
- KB
We worked with the gitblit team (via their bugs page) to get this fixed upstream, once we finish with 7 release, we will roll it out to git.centos.org. We know it is an issue, as we are using that to build CentOS 7 right now.
On 07/04/2014 08:46 AM, Nico Kadel-Garcia wrote:
Morning, folks. Happy July 4th.
I'm hopping back onto centos-devel to raise some concerns about git.centos.org components. Overall it's working. There are things I don't like, but as things stand that's *my* problem. These issues, however, affect others, and I don't see a good bugzilla mentioned at git.centos.org itself.
- Many repositories are being listed with excessive "/" in their
names. It messes up alphabetization and can get quite confusing.
rpms/docker, for example, is listed as
This one is a known issue caused by bad client requests being cached. Currently the resolution is to clear the server-side cache periodically. Once 7 is out the door, we'll have a bit more free time to look at resolving this properly.
- The "show_possible_srpms.sh" script relies on checking for the word
"import" in the git logs to determine SRPM versions, embedded in the git logs themselves. This raises the risk of *any* commit that uses that word for other reasons to report an invalid SRPM version number. As much as I dislike relying on a git log rather than a signed tag to do a build, please, refine this script to at least grep more carefully for 'import' as the leading word, and ideally sanity check the rest of the import line for the SRPM number.
Obligatory XKCD comic about sanitizing inputs:
Patches accepted. Please submit them to the list, and if/when the pass sanity/peer review (other list members) we'll apply them.
On 07/04/2014 02:46 PM, Nico Kadel-Garcia wrote:
Please consider the use of signed GPG tags for actualSRPM updates, rather than merely relying on '[package].metadata, to help assure provenance for people who may test or rebuild security components.
the content you get is pushed over https, the implementation on git.centos.org seems fairly secure. the content into the machine is via ssh, over a guranteed ( in as much as network can be guranteed ) link.
we are also preventing anyone else from being able to commit with the source importer username/email and or using the word 'import' as the first chat in the commit.
some of this is convention, but as the source that we consume, we are fairly sure of what is going through. If there are any specific concerns about code, do point them out - and if its security related, then email security@centos.org instead of a public list.
regards
On Fri, Jul 4, 2014 at 11:15 AM, Karanbir Singh mail-lists@karan.org wrote:
On 07/04/2014 02:46 PM, Nico Kadel-Garcia wrote:
Please consider the use of signed GPG tags for actualSRPM updates, rather than merely relying on '[package].metadata, to help assure provenance for people who may test or rebuild security components.
the content you get is pushed over https, the implementation on git.centos.org seems fairly secure. the content into the machine is via ssh, over a guranteed ( in as much as network can be guranteed ) link.
we are also preventing anyone else from being able to commit with the source importer username/email and or using the word 'import' as the first chat in the commit.
Thanks. But Karanbir, "commit" is not the problem I'm referring to. It's the ability to substitute a trojaned, fake repository in between you and the client, to commit a "man-in-the-m8iddle" attack Valid SSL certificates, and a clean repoository at git.centos.org: the ability to verify a particulr "tag" with a GPG tag, particularly a cloned local working copy with the "tags" from upstream for reference, is invaluable. Red Hat's, and CentOS's, SRPM repositories avoided this by having GPG signed SRPM's for reference. The GPG signature chain is more safely managed and verified, in many ways, than the SSL has been.
some of this is convention, but as the source that we consume, we are fairly sure of what is going through. If there are any specific concerns about code, do point them out - and if its security related, then email security@centos.org instead of a public list.
I'll take this issue over there.
regards
-- Karanbir Singh +44-207-0999389 | http://www.karan.org/ | twitter.com/kbsingh GnuPG Key : http://www.karan.org/publickey.asc _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org http://lists.centos.org/mailman/listinfo/centos-devel
On 07/05/2014 05:08 AM, Nico Kadel-Garcia wrote:
On Fri, Jul 4, 2014 at 11:15 AM, Karanbir Singh mail-lists@karan.org wrote:
On 07/04/2014 02:46 PM, Nico Kadel-Garcia wrote:
Please consider the use of signed GPG tags for actualSRPM updates, rather than merely relying on '[package].metadata, to help assure provenance for people who may test or rebuild security components.
the content you get is pushed over https, the implementation on git.centos.org seems fairly secure. the content into the machine is via ssh, over a guranteed ( in as much as network can be guranteed ) link.
we are also preventing anyone else from being able to commit with the source importer username/email and or using the word 'import' as the first chat in the commit.
Thanks. But Karanbir, "commit" is not the problem I'm referring to. It's the ability to substitute a trojaned, fake repository in between you and the client, to commit a "man-in-the-m8iddle" attack Valid SSL
what about git.centos.org's ssl cert looks invalid ? Also, if you are doubting SSL as a transport, we've all got bigger problems.
- KB
On Sat, Jul 5, 2014 at 8:23 AM, Karanbir Singh mail-lists@karan.org wrote:
On 07/05/2014 05:08 AM, Nico Kadel-Garcia wrote:
On Fri, Jul 4, 2014 at 11:15 AM, Karanbir Singh mail-lists@karan.org wrote:
On 07/04/2014 02:46 PM, Nico Kadel-Garcia wrote:
Please consider the use of signed GPG tags for actualSRPM updates, rather than merely relying on '[package].metadata, to help assure provenance for people who may test or rebuild security components.
the content you get is pushed over https, the implementation on git.centos.org seems fairly secure. the content into the machine is via ssh, over a guranteed ( in as much as network can be guranteed ) link.
we are also preventing anyone else from being able to commit with the source importer username/email and or using the word 'import' as the first chat in the commit.
Thanks. But Karanbir, "commit" is not the problem I'm referring to. It's the ability to substitute a trojaned, fake repository in between you and the client, to commit a "man-in-the-m8iddle" attack Valid SSL
what about git.centos.org's ssl cert looks invalid ? Also, if you are doubting SSL as a transport, we've all got bigger problems.
Today? *Nothing*. It's not git.centos.org, itself, that I'm worried about. The proper use of private/public key pairs, such as SSH and SSL private keys, can help protect the central repository from improper access, and I assume that you folks are also doing code review before pulls are accepted into the primary repositories.
It's man-in-the-middle attacks for remote software downloaders. They'e vulnerable to trojaned repositories. placed *in between* their local client and git.centos.org. And SSL itself is vulnerable to forged keys using stolen signature authorities, whether local to a remote environment or such as the stolen "GlobalSign" keys. There are even environments that put proxies in front of *incoming* traffic, locally, to slap a new internal and internally signed SSL key on it, and *deliberately* do man-in-the-middle monitoring at the proxy. There are others that block most external HTTP/HTTPS and route only a few allowed external websites through an internal DNS address at the proxy: they've effectively insisted on being a man-in-the-middle for their own internal monitoring and traffic control reasons, and they can wind up using internally signed keys. They certainly can't use *your* keys for that!!!!
There's also verification of the content in local working repositories. If I make a fork of https://github.com/rpms/kernel over to "https://github.com/nkadel/centos-kernel-test-usb/", and someone else works with me on it and makes their own clone, they have to have a chain of trust between their work, your work, and my work. I would find it much safer, and more useful, have something like a GPG signed git tag in the repo that cleanly defines your release. It helps profoundly to ensure that the tag, which can be tied to a specific SRPM number quite safely and easily, also ensures the full provenance of the code as being what you folks over at CentOS actually published. Otherwise, we enter a potentially pretty painful world of having to manage and verify git commit checksums, and that way lies unstable hackery scripting madness.
The GPG keys on the SRPM's have been one of the very useful checks of veracity and provenance on varous projects, I'll save commentary on how much more useful I think consistent tag naming conventions are than doing 'grep' on string logs to deduce commits. But the security benefits of
I realize that it's work, but at least in a technology sense, I do think that it could be integrated into the git.centos.org repository. Frankly, I was a more than a bit surprised to see that tags, especially GPG signed tags, weren't already in use. for defining the SRPM numbers. I do think that it would be much cleaner than trying to derive build numbers from "git log" analysis.
I might be misunderstanding this thread... but if package signing and/or message digest are not being applied to SRPM, I also agree this is a problem.
On Sat, Jul 5, 2014 at 10:58 AM, Nico Kadel-Garcia nkadel@gmail.com wrote:
On Sat, Jul 5, 2014 at 8:23 AM, Karanbir Singh mail-lists@karan.org wrote:
On 07/05/2014 05:08 AM, Nico Kadel-Garcia wrote:
Thanks. But Karanbir, "commit" is not the problem I'm referring to. It's the ability to substitute a trojaned, fake repository in between you and the client, to commit a "man-in-the-m8iddle" attack Valid SSL
what about git.centos.org's ssl cert looks invalid ? Also, if you are doubting SSL as a transport, we've all got bigger problems.
Today? *Nothing*. It's not git.centos.org, itself, that I'm worried about. The proper use of private/public key pairs, such as SSH and SSL private keys, can help protect the central repository from improper access, and I assume that you folks are also doing code review before pulls are accepted into the primary repositories.
Yes. SSL and SSH *can* provide privacy in transit and authentication of the source and destination (often only the destination). But it doesn't protect the content at the source or the target or at transit points, and authentication of the destination host isn't as useful or as well scrutinized as many people believe.
In the case of SSL... the recent "Heartbleed" exploit clearly demonstrates that SSL is not as invulnerable as people have been lead to believe. But, SSL has long had known issues and been circumvented. Edward Snowden changed our suspicions to beliefs that the US government was dabbling in this practice including potentially intercepting the SSL traffic directed towards large companies that I won't name here.
In the case of SSH... how many of you actually look up the SSH public key through a secondary channel (not SSH to the server!), before you accept the SSH public key as valid into your ~/.ssh/known_hosts file? I know I'm as guilty as probably all of you, and I generally do not.
It's man-in-the-middle attacks for remote software downloaders. They'e vulnerable to trojaned repositories. placed *in between* their local client and git.centos.org. And SSL itself is vulnerable to forged keys using stolen signature authorities, whether local to a remote environment or such as the stolen "GlobalSign" keys. There are even environments that put proxies in front of *incoming* traffic, locally, to slap a new internal and internally signed SSL key on it, and *deliberately* do man-in-the-middle monitoring at the proxy. There are others that block most external HTTP/HTTPS and route only a few allowed external websites through an internal DNS address at the proxy: they've effectively insisted on being a man-in-the-middle for their own internal monitoring and traffic control reasons, and they can wind up using internally signed keys. They certainly can't use *your* keys for that!!!!
Our company is looking into exactly these technologies. Both HTTPS and SSH transparent proxies. Once deployed, I won't be able to SSH from work out to a machine on the Internet without my session being "in the clear" and under the complete control of the proxy machine. The intent of this is supposedly to prevent data exfiltration but what it amounts to is that the company wants the ability to monitor all traffic in and out, and to be able to do a sort of "deep packet inspection" or even manipulation that would allow them to better filter the content and in the case of SSH, the ability to disable the use of port forwarding over SSH as a tunnel. They are real. Yes, a person worried about such a thing could check the signature of the server and the signature would probably not match... but for SSH, do you check the signature? And for SSL, what happens if they also intercept or manipulate the certificate lookups?
The point here is that HTTPS and SSH provide an amount of privacy and security. It does not make *any* guarantee about the authenticity of the content. It is sort of like having a rare painting be transported in a security truck, and presuming that because the painting arrives in a security truck, the painting must still be the original. The only way to determine if the painting is the original is to actually check the painting itself. How it arrives is irrelevant.
In a less extreme case, it just needs somebody to "hack into" one of the CentOS distribution points and replace the source and the .metadata with something of their own. Or it could just be a disgruntled or financially-rewarded volunteer who quietly replaces a server package. Without signatures, it can be difficult to detect this sort of thing.
Now, Git itself provides some of these protections... I'm only learning about CentOS 7 and git.centos.org, but I suspect a capability availability here is to use the Git SHA-1 commit identifiers as a means of authenticating content... but this wouldn't be good enough for files produced or distributed outside of Git...
Anyways... I just wanted to add support to the argument that signatures of some sort are desirable and that SSH and SSL are not on their own sufficient.
On 07/05/2014 01:11 PM, Mark Mielke wrote:
I might be misunderstanding this thread... but if package signing and/or message digest are not being applied to SRPM, I also agree this is a problem.
We will sign all RPMs and SRPMs that we release ... this is about the git tree.
<snip>
On 07/05/2014 08:21 PM, Johnny Hughes wrote:
On 07/05/2014 01:11 PM, Mark Mielke wrote:
I might be misunderstanding this thread... but if package signing and/or message digest are not being applied to SRPM, I also agree this is a problem.
We will sign all RPMs and SRPMs that we release ... this is about the git tree.
<snip>
Mark, CentOS devs suggested that it is better to build (your own packages like Scientific Linux people for example) from git then RPMS, to better reproduce entire building process so Nico is arguing/pointing possible caveats.
Considering that, according to news articles, NSA (or other agency with access to internet infrastructure between git.centos.org and client in question) can intercept SSL request posing as some kind of SSL proxy, pretending they are (for example) git.centos.org, create secure connection to the client (which thinks it is talking to original git.centos.org), then create secure SSL connection between agencies SSL proxy and git.centos.org.
Once those connections is created, it is easy to inject/replace code from git.centos.org so that back-door is introduced. Client/user building that code (rebuilding CentOS to feel safer?) might not realize that every single system that uses his packages is now compromised and vulnerable to attack.
On Sat, Jul 5, 2014 at 2:11 PM, Mark Mielke mark.mielke@gmail.com wrote:
I might be misunderstanding this thread... but if package signing and/or message digest are not being applied to SRPM, I also agree this is a problem.
You got most of it. I'm strongly urging the sue of GPG signed 'git tags' for particular SRPM bundles in the git.centos.org repositories, rather than relying on analyzing "git log" to mark particular SRPM's.
On Sat, Jul 5, 2014 at 10:58 AM, Nico Kadel-Garcia nkadel@gmail.com wrote:
Yes. SSL and SSH *can* provide privacy in transit and authentication of the source and destination (often only the destination). But it doesn't protect the content at the source or the target or at transit points, and authentication of the destination host isn't as useful or as well scrutinized as many people believe.
Amen. There are long threads about this, including some analyses by Bruce Schneier. at https://www.schneier.com/blog/archives/2011/10/the_security_of_4.html, he writes:
The most interesting entry in that table is the "CA compromise" one, because those are incidents that could affect any or every secure web or email server on the Internet. In at least 248 cases, a CA chose to indicate that it had been compromised as a reason for revoking a cert. Such statements have been issued by 15 distinct CA organizations.
That was 2011. There's no reason to think it's gotten better. Coupled with the interposition of deliberately compromised proxies, either as a matter of internal policy or due to cracking, and SSL cannot be considered sufficient for software repository authentication.
Our company is looking into exactly these technologies. Both HTTPS and SSH transparent proxies. Once deployed, I won't be able to SSH from work out to
Ahh. Are you using 'NetIntercept'? Nice hardware, very powerful, does both man-in-the-middle monitoring and recording it do local disk for meta-analysis. Scary, scary, scary stuff..
Now, Git itself provides some of these protections... I'm only learning about CentOS 7 and git.centos.org, but I suspect a capability availability here is to use the Git SHA-1 commit identifiers as a means of authenticating content... but this wouldn't be good enough for files produced or distributed outside of Git...
Some of that is in the '[packagename].medatada' file. Problem is, it's inside the repository itself. The more common approach, built into git directly for *exactly* this sort of use, is to use GPG signed tags. It's possible to remove and replace a tag, but the GPG signature helps assure that if that occurs, at least it was *intentional* by the owner of the GPG tag.
Anyways... I just wanted to add support to the argument that signatures of some sort are desirable and that SSH and SSL are not on their own sufficient.
Thanks! I do think it will help assuage some concerns.
On 07/06/2014 05:25 AM, Nico Kadel-Garcia wrote:
Some of that is in the '[packagename].medatada' file. Problem is, it's inside the repository itself. The more common approach, built into git directly for *exactly* this sort of use, is to use GPG signed tags. It's possible to remove and replace a tag, but the GPG signature helps assure that if that occurs, at least it was *intentional* by the owner of the GPG tag.
if you can MITM the content, nothing signed or otherwise is assured to be in any sate.
On Sun, Jul 6, 2014 at 5:29 AM, Karanbir Singh mail-lists@karan.org wrote:
On 07/06/2014 05:25 AM, Nico Kadel-Garcia wrote:
Some of that is in the '[packagename].medatada' file. Problem is, it's inside the repository itself. The more common approach, built into git directly for *exactly* this sort of use, is to use GPG signed tags. It's possible to remove and replace a tag, but the GPG signature helps assure that if that occurs, at least it was *intentional* by the owner of the GPG tag.
if you can MITM the content, nothing signed or otherwise is assured to be in any sate.
First note: in many environments, MITM is how things work normally. In others, the routers and proxies and DNS are *not* secured or easily re-implemented to be secure, for many, many different reasons. No matter how careful you are with git.centos.org itself, it's going to be vulnerable to trojaned intermediate repositories being substituted because no one at CentOS or Red Hat has enough control over the Internet to fix these.
For folks unfamiliar with the technology, one can *reduce* the risk, and provide a much more robust content verification, by GPG signing tags. The GPG signature is applied to the commit checksums embedded in the tag, and normally have multiple, very separate means of propagation, They're been effectively used for RPM and SRPM verificiation for many years, and in my opinion form a much more verifiable chain of trust. They're also distinct from the transmission protocol, so they still apply if the site is HTTP accessed or SSH accessed or even local filesystem accessed or even if someone walks a tarball of the git repo from one desk to another. They don't solve everything, but Lord, they help verify that the tag in the *local* git clone has content signed by the upstream repository owners. I certainly know of, and have built, local high security build systems that have a carefully built internal mirror of CentOS or RHEL or Scientific Linux, locked down content, and no Internet access: not having to verify that content against the centralized https://git.centos.org/, and being able to apply GPG verification to the SRPM's from upsteam, has been useful.
If CentOS is going to GPG sign the SRPM's when published, great, that's helpful. But if you expect developers to be going straight to the git repo, instead of the SRPM's for their development environments, it has to be considered an independent channel and its security concerns considered separately.
Is there any place where a trojaned site can cause real damage? Absolutely. Any site that does serious development with locally modified kernels or applications on top of CentOS (*cough* Cisco *cough*) could have the source for its customized components corrupted by such an attack. It's *difficult* to code review entire repositories of upsteam software. The result can be widespread commercial software and hardware with embedded vulnerabilities. (*cough* Cisco ASA firewall *cough*).
Is it happening right now? Good question. I don't know. Who would *admit* that it had happened, or even notice at first?
On 07/06/2014 11:11 AM, Nico Kadel-Garcia wrote:
if you can MITM the content, nothing signed or otherwise is assured to be in any sate.
First note: in many environments, MITM is how things work normally. In others, the routers and proxies and DNS are *not* secured or easily re-implemented to be secure, for many, many different reasons. No matter how careful you are with git.centos.org itself, it's going to be vulnerable to trojaned intermediate repositories being substituted because no one at CentOS or Red Hat has enough control over the Internet to fix these.
You are repeatedly saying this, but failing to actually quantify it.
your gpg key is pretty much null routed once someone can MITM the content anyway, and i also assume you realise that a gpg sign is still with the same sort of code/key as an ssl connection is.
the bottom line is that unless you can get an authoritative manner of initial keyexchange, that you can absolutely trust, nothing else down the line is going to be any more secure than the initial handover. I'm happy to setup a keysign and keyexchange event at every dojo we run, but even that is likely to only reach a small fraction of the entire userbase.
So the only way to get the originating gpg key ( that you'd verify against ) is over ssl on the internet, which also implies that having git.centos.org behind the same leave lof trust puts us no worse off.
regards
On Sun, Jul 6, 2014 at 10:06 AM, Karanbir Singh mail-lists@karan.org wrote:
On 07/06/2014 11:11 AM, Nico Kadel-Garcia wrote:
First note: in many environments, MITM is how things work normally. In others, the routers and proxies and DNS are *not* secured or easily re-implemented to be secure, for many, many different reasons. No matter how careful you are with git.centos.org itself, it's going to be vulnerable to trojaned intermediate repositories being substituted because no one at CentOS or Red Hat has enough control over the Internet to fix these.
You are repeatedly saying this, but failing to actually quantify it.
your gpg key is pretty much null routed once someone can MITM the content anyway, and i also assume you realise that a gpg sign is still with the same sort of code/key as an ssl connection is.
It's may be the same classification of cryptographic functions... but it is not the same. With SSL/SSH you are, at least in theory, authenticating the origin host. This makes no promise about the origin content.
A PGP-signed tag makes a promise about the content but does *not* make a promise about the origin host.
So, they are not the same.
In the case of git.centos.org, somebody who manipulates the repositories could likely fake the repository content being consistent, but would have a more difficult time faking a signature where the public key is stored elsewhere that git.centos.org (such as one of the PGP key servers). The SSL/SSH for git.centos.org would continue to pass, but the signature would not.
The other case already mentioned was the case that the content has an intermediate resting point. That is, I clone from git.centos.org, and then I publish to somebody else. How do they know which content is authentic... mine or git.centos.org? Being able to point to the signed tag shows cryptographically who signed the content and this could be an important differentiator in determining which source is the "real" source.
Now:
the bottom line is that unless you can get an authoritative manner of initial keyexchange, that you can absolutely trust, nothing else down the line is going to be any more secure than the initial handover. I'm happy to setup a keysign and keyexchange event at every dojo we run, but even that is likely to only reach a small fraction of the entire userbase.
So the only way to get the originating gpg key ( that you'd verify against ) is over ssl on the internet, which also implies that having git.centos.org behind the same leave lof trust puts us no worse off.
I think you may be dismissing the two scenarios... 1) git.centos.org gets compromised, 2) intermediate resting point. Both of these require the *content* to be signed. You are only providing authentication of the original source which does not address either of these points effectively.
So, I don't think it is correct that the "bottom line" is as you state above. The real "bottom line" is that SSL/SSH only authenticates the source host. It provides *no* guarantee for the source content.
If you authenticated the content, you wouldn't actually need to authenticate the source host. This is how Bittorrent works in general. It doesn't matter how the bytes get from point A to point B, as long as the final result has the correct *content*. SSL/SSH do not guarantee the correct *content* so it is rather ineffective as a means of authenticating the content. It's saying "trust me... I'm git.centos.org... what could go wrong?"
On Sun, Jul 6, 2014 at 5:23 PM, Mark Mielke mark.mielke@gmail.com wrote:
A PGP-signed tag makes a promise about the content but does *not* make a promise about the origin host.
So, they are not the same.
In the case of git.centos.org, somebody who manipulates the repositories could likely fake the repository content being consistent, but would have a more difficult time faking a signature where the public key is stored elsewhere that git.centos.org (such as one of the PGP key servers). The SSL/SSH for git.centos.org would continue to pass, but the signature would not.
The other case already mentioned was the case that the content has an intermediate resting point. That is, I clone from git.centos.org, and then I publish to somebody else. How do they know which content is authentic... mine or git.centos.org? Being able to point to the signed tag shows cryptographically who signed the content and this could be an important differentiator in determining which source is the "real" source.
In theory, someone could publish a faked GPG tag to go with it. But GPG tags are *much* more portable and applicable to arbitrary content. They seem to be fundamental to the way Red Hat has been publishing RPM and SRPM's for years, and the way CentOS does, to allow reasonably safe third party mirrors to carry the software.
Now:
the bottom line is that unless you can get an authoritative manner of initial keyexchange, that you can absolutely trust, nothing else down the line is going to be any more secure than the initial handover. I'm happy to setup a keysign and keyexchange event at every dojo we run, but even that is likely to only reach a small fraction of the entire userbase.
There are too many SSL environments that have *no* reliable key exchange for the upstream site, due to mandataory or mishandled proxies or local caching of the software repository. I've pointed out a few.
Worse, once a git.centos.org repository is cloned locally, upstream SSL is no longer necessarily used. It is certainly possible to clone a repository with all commits *up to* the commit transaction for the final SRPM build and manipulation of '[package].metadata', then inject a local commit after that change, along with arbitrary trojan content.
The link to the upstream SSL repository would no longer necessarily be used. One could do a comparison against the upstream repository at git.centos.org, but that's an extra and unexpected step.
So the only way to get the originating gpg key ( that you'd verify against ) is over ssl on the internet, which also implies that having git.centos.org behind the same leave lof trust puts us no worse off.
Nonsense. It can come from any of dozens if not hundreds of mirror sites worldwide. And it can be verified in a way that SSL cannot, by reviewing the signature chain personally rather than relying on centralized certification authorities that have repeatedly proven themselves incompetent, or that are deliberately suborned as a matter of local policy.
I think you may be dismissing the two scenarios... 1) git.centos.org gets compromised, 2) intermediate resting point. Both of these require the *content* to be signed. You are only providing authentication of the original source which does not address either of these points effectively.
I'm less worried about git.centos.org getting p0wned. I am worried about what you refer to as "intermediate resting points". I'm also concerned that if *I* make a fork of, say, the OpenSSH repository, write patches, and submit them or use them, that I have a reliable and consistent "benchmark" version of the software. If I get p0wned, GPG tags on the buld versions of upstream software would help me review and verify, locally, that the template from upstream is good.
So, I don't think it is correct that the "bottom line" is as you state above. The real "bottom line" is that SSL/SSH only authenticates the source host. It provides *no* guarantee for the source content.
You've captured the substance of my concern.
If you authenticated the content, you wouldn't actually need to authenticate the source host. This is how Bittorrent works in general. It doesn't matter how the bytes get from point A to point B, as long as the final result has the correct *content*. SSL/SSH do not guarantee the correct *content* so it is rather ineffective as a means of authenticating the content. It's saying "trust me... I'm git.centos.org... what could go wrong?"
And there are a stack of environments where the SSL is inaccessible, such as isolated build hosts with no external network access. (I work with some of thiose.)
There's been a lot of discussion on this, which is surprising given that this is the wrong place for it. Maybe next we can discuss how to improve AutoYAST or IIS.
Some people seem to have forgotten that CentOS, despite the recent change in employment status of several of its core members, is the *downstream* rebuilder. They get their sources via git.centos.org from a prominent North American Linux reseller, just like the rest of us.
Read that again: CentOS is not special. They are *consumers* of the sources, not the producers.
No, really.
So if CentOS pushed a zillion signed tags to git.centos.org, that'd only mean that CentOS trusts those sources. If, as Nico suggested, git.centos.org was pwned, then CentOS just certified bogus sources. IOW, a signed tag from Jim, or Johnny, or KB, or any of the other CentOS devs means precisely fuck-all, and not a bit more, because they get the sources through the EXACT SAME distribution channel that the rest of us do. CentOS is not special, they just look that way. Really, if this is the assurance you want, just add your own signed tags to your local repo -- it's just as meaningful.
What do we want? FUCK-ALL! When do we want it? NOW!
If you want *actual* cryptographic assurance that the sources you're grabbing from git.centos.org are the same sources pushed there by the *upstream* vendor, maybe, just maybe, you should ask that upstream vendor. Otherwise it's just garbage in, garbage out, and they're only certifying that the sources you download match the sources someone else downloaded. I guess misery loves company, but that sure doesn't seem helpful.
I'll warn you, though, since the specter of the all-powerful NSA was raised: they already have Red Hat's signing keys. And yours, too.
On Sun, Jul 6, 2014 at 5:54 PM, Nico Kadel-Garcia nkadel@gmail.com wrote:
On Sun, Jul 6, 2014 at 5:23 PM, Mark Mielke mark.mielke@gmail.com wrote:
A PGP-signed tag makes a promise about the content but does *not* make a promise about the origin host.
So, they are not the same.
In the case of git.centos.org, somebody who manipulates the repositories could likely fake the repository content being consistent, but would
have a
more difficult time faking a signature where the public key is stored elsewhere that git.centos.org (such as one of the PGP key servers). The SSL/SSH for git.centos.org would continue to pass, but the signature
would
not.
The other case already mentioned was the case that the content has an intermediate resting point. That is, I clone from git.centos.org, and
then I
publish to somebody else. How do they know which content is authentic... mine or git.centos.org? Being able to point to the signed tag shows cryptographically who signed the content and this could be an important differentiator in determining which source is the "real" source.
In theory, someone could publish a faked GPG tag to go with it. But GPG tags are *much* more portable and applicable to arbitrary content. They seem to be fundamental to the way Red Hat has been publishing RPM and SRPM's for years, and the way CentOS does, to allow reasonably safe third party mirrors to carry the software.
Now:
the bottom line is that unless you can get an authoritative manner of initial keyexchange, that you can absolutely trust, nothing else down the line is going to be any more secure than the initial handover. I'm happy to setup a keysign and keyexchange event at every dojo we run, but even that is likely to only reach a small fraction of the entire
userbase.
There are too many SSL environments that have *no* reliable key exchange for the upstream site, due to mandataory or mishandled proxies or local caching of the software repository. I've pointed out a few.
Worse, once a git.centos.org repository is cloned locally, upstream SSL is no longer necessarily used. It is certainly possible to clone a repository with all commits *up to* the commit transaction for the final SRPM build and manipulation of '[package].metadata', then inject a local commit after that change, along with arbitrary trojan content.
The link to the upstream SSL repository would no longer necessarily be used. One could do a comparison against the upstream repository at git.centos.org, but that's an extra and unexpected step.
So the only way to get the originating gpg key ( that you'd verify against ) is over ssl on the internet, which also implies that having git.centos.org behind the same leave lof trust puts us no worse off.
Nonsense. It can come from any of dozens if not hundreds of mirror sites worldwide. And it can be verified in a way that SSL cannot, by reviewing the signature chain personally rather than relying on centralized certification authorities that have repeatedly proven themselves incompetent, or that are deliberately suborned as a matter of local policy.
I think you may be dismissing the two scenarios... 1) git.centos.org
gets
compromised, 2) intermediate resting point. Both of these require the *content* to be signed. You are only providing authentication of the original source which does not address either of these points
effectively.
I'm less worried about git.centos.org getting p0wned. I am worried about what you refer to as "intermediate resting points". I'm also concerned that if *I* make a fork of, say, the OpenSSH repository, write patches, and submit them or use them, that I have a reliable and consistent "benchmark" version of the software. If I get p0wned, GPG tags on the buld versions of upstream software would help me review and verify, locally, that the template from upstream is good.
So, I don't think it is correct that the "bottom line" is as you state above. The real "bottom line" is that SSL/SSH only authenticates the
source
host. It provides *no* guarantee for the source content.
You've captured the substance of my concern.
If you authenticated the content, you wouldn't actually need to
authenticate
the source host. This is how Bittorrent works in general. It doesn't
matter
how the bytes get from point A to point B, as long as the final result
has
the correct *content*. SSL/SSH do not guarantee the correct *content* so
it
is rather ineffective as a means of authenticating the content. It's
saying
"trust me... I'm git.centos.org... what could go wrong?"
And there are a stack of environments where the SSL is inaccessible, such as isolated build hosts with no external network access. (I work with some of thiose.) _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org http://lists.centos.org/mailman/listinfo/centos-devel
On Sun, Jul 6, 2014 at 9:04 PM, Chris St. Pierre < chris.a.st.pierre@gmail.com> wrote:
There's been a lot of discussion on this, which is surprising given that this is the wrong place for it. Maybe next we can discuss how to improve AutoYAST or IIS.
Some people seem to have forgotten that CentOS, despite the recent change in employment status of several of its core members, is the *downstream* rebuilder. They get their sources via git.centos.org from a prominent North American Linux reseller, just like the rest of us.
Read that again: CentOS is not special. They are *consumers* of the sources, not the producers.
You have some interesting points... but, this discussion is just as important for consumers as for producers, and CentOS is both a consumer *and* a producer, as are many other downstream distros based upon upstream distros.
If you don't believe security is possible... that's fine. Because perfect security is impossible. But, that doesn't mean people shouldn't try. CentOS *does* sign SRPM, do they not? Why do they do this? Obviously, somebody believes this aspect is important?
It seems like some people just want to do what they're already doing (for better or for worse) and it doesn't really matter what the request is, or the merit of the request. Which is fine... but just please admit to it.
So if CentOS pushed a zillion signed tags to git.centos.org, that'd only mean that CentOS trusts those sources. If, as Nico suggested, git.centos.org was pwned, then CentOS just certified bogus sources. IOW, a signed tag from Jim, or Johnny, or KB, or any of the other CentOS devs means precisely fuck-all, and not a bit more, because they get the sources through the EXACT SAME distribution channel that the rest of us do. CentOS is not special, they just look that way. Really, if this is the assurance you want, just add your own signed tags to your local repo -- it's just as meaningful.
What do we want? FUCK-ALL! When do we want it? NOW!
You are right about "just add your own signed tags". Actually, everybody who derives should sign what their derived works. That makes it possible to track back if or when something bad does happen, and we can see where the problem was introduced. So RedHat should sign their tags, and CentOS should sign any tags that they create. It's not "FUCK-ALL"... It's evidence that a particular process was followed that was approved by a particular person. It's a paper trail that is more difficult to forge. It's not different from a signature on a form authorizing a change. Yes they can be forged... but that doesn't mean that "no signature" is better than "signature that can theoretically be forged".
If you want *actual* cryptographic assurance that the sources you're grabbing from git.centos.org are the same sources pushed there by the *upstream* vendor, maybe, just maybe, you should ask that upstream vendor. Otherwise it's just garbage in, garbage out, and they're only certifying that the sources you download match the sources someone else downloaded. I guess misery loves company, but that sure doesn't seem helpful.
Yes, the upstream vendor should be asked. That doesn't really add or remove merit to CentOS signing any tags that CentOS creates.
I'll warn you, though, since the specter of the all-powerful NSA was raised: they already have Red Hat's signing keys. And yours, too.
They might... but it really sounds like you are saying that because it is possible for NSA to get past any security, therefore no security makes perfect sense. It sounds like extending your thinking would conclude that signing the SRPM is also useless. And for this... if you really do think this... I think you are quite wrong.
On Sun, Jul 6, 2014 at 9:42 PM, Mark Mielke mark.mielke@gmail.com wrote:
If you don't believe security is possible... that's fine. Because perfect security is impossible. But, that doesn't mean people shouldn't try. CentOS *does* sign SRPM, do they not? Why do they do this? Obviously, somebody believes this aspect is important?
CentOS *produces* the SRPMs. They should sign them -- it verifies that this is the SRPM CentOS built, not something masquerading as such. It makes no guarantee as to the content or provenance of the sources, though, beyond the degree to which we already trust CentOS.
Signing the sources is an entirely different matter, since CentOS did not populate them and has no way to verify them independently of the upstream producer. We want a signed tag on the git repo in order to guarantee that these are the sources that upstream provided, not something masquerading as such. A signed tag from CentOS only certifies that these are the sources CentOS *thinks* upstream provided, which really truly is worth fuck-all because the chain of trust was broken by *upstream*.
It seems like some people just want to do what they're already doing (for
better or for worse) and it doesn't really matter what the request is, or the merit of the request. Which is fine... but just please admit to it.
To be clear, I'm not doing anything. I just like mailing lists with good SNR. If I was a CentOS core dev, I already would have written a script to push a cryptographically signed tag to every repo, which would be completely useless because, again, the chain of trust was broken by *upstream*.
Yes, the upstream vendor should be asked. That doesn't really add or remove merit to CentOS signing any tags that CentOS creates.
Actually, it does. If CentOS signs a known delta to an unknown (or at least unproven) base, that isn't actually valuable. I.e., only if upstream can be convinced to sign their tags would it be useful for CentOS to do the same. Until then, a signed tag from CentOS just tells us that someone trusted made a change to something untrusted, and the net result is still untrusted because -- say it with me this time -- the chain of trust was broken by *upstream*.
I'll warn you, though, since the specter of the all-powerful NSA was
raised: they already have Red Hat's signing keys. And yours, too.
They might... but it really sounds like you are saying that because it is possible for NSA to get past any security, therefore no security makes perfect sense. It sounds like extending your thinking would conclude that signing the SRPM is also useless. And for this... if you really do think this... I think you are quite wrong.
My only beef is with any security model that has to resort to magic (or the NSA, they're the same in this context) to explain the threat. Especially when that model proposes to close the barn door after the horse is already gone.
On 2014-07-07 04:52, Chris St. Pierre wrote:
On Sun, Jul 6, 2014 at 9:42 PM, Mark Mielke <mark.mielke@gmail.com mailto:mark.mielke@gmail.com> wrote:
If you don't believe security is possible... that's fine. Because perfect security is impossible. But, that doesn't mean people shouldn't try. CentOS *does* sign SRPM, do they not? Why do they do this? Obviously, somebody believes this aspect is important?CentOS *produces* the SRPMs. They should sign them -- it verifies that this is the SRPM CentOS built, not something masquerading as such. It makes no guarantee as to the content or provenance of the sources, though, beyond the degree to which we already trust CentOS. Signing the sources is an entirely different matter, since CentOS did not populate them and has no way to verify them independently of the upstream producer. We want a signed tag on the git repo in order to guarantee that these are the sources that upstream provided, not something masquerading as such. A signed tag from CentOS only certifies that these are the sources CentOS *thinks* upstream provided, which really truly is worth fuck-all because the chain of trust was broken by *upstream*.
A signed tag from CentOS would say "this is the content from which we built our SRPM". It wouldn't be "signing the sources" any more than the signing of SRPMs would be. Why would that be bad?
[...] Until then, a signed tag from CentOS just tells us that someone trusted made a change to something untrusted, and the net result is still untrusted because -- say it with me this time -- the chain of trust was broken by *upstream*.
And this would be different for signed (S)RPMs how, exactly?
On 07/07/2014 09:22 AM, Elias Persson wrote:
A signed tag from CentOS would say "this is the content from which we built our SRPM". It wouldn't be "signing the sources" any more than the signing of SRPMs would be. Why would that be bad?
but an srpm has more than whats in git, and we cant know what we are going to release, till the code is built, qa'd and passed release testing etc. so signing a commit will always be an after thought.
also, putting distro keys on developer laptops and circulating them around town isnt a nice thing, expand that with increasing contributor base who are able to branch and build their own content into SIG's etc, and you dramatically expand that problem base.
[...] Until then, a signed tag from CentOS just tells us that someone trusted made a change to something untrusted, and the net result is still untrusted because -- say it with me this time -- the chain of trust was broken by *upstream*.
And this would be different for signed (S)RPMs how, exactly?
an SRPM is a reproduceable build source, comprehensive and includes build time metadata. this will match back to delivered buildlogs result that can be mapped, using the timestamp, to the exact environment it was built in.
A git tag would mostly just indicate what spec was used, and would come after the event, with no validation on what content changed under and around the hood by the time this code was released.
Two very different things.
On Mon, Jul 7, 2014 at 4:22 AM, Elias Persson delreich@takeit.se wrote:
A signed tag from CentOS would say "this is the content from which we built our SRPM". It wouldn't be "signing the sources" any more than the signing of SRPMs would be. Why would that be bad?
It's not bad as such, just useless. If I, the end user, am concerned about the sources having been illicitly tampered with, all this tells me is that I've got the same (untrusted, possibly trojaned) sources as CentOS. Big whoop. If upstream signed the content they push, though, then I'd be able to put my trust in upstream -- i.e., in the people who are actually curating and creating the vast majority of the content.
Until that happens, the tiny bit of value that might be derived from having a cryptographically secure means of knowing that I've been pwned in exactly the same was as CentOS is just not worth the work that would go into it. It might even be a little misleading, since there would be an appearance of trust and security where none actually existed. This really seriously needs to start with upstream, or it's all for naught.
[...] Until then, a signed tag from CentOS just tells us that someone trusted made a change to something untrusted, and the net result is still untrusted because -- say it with me this time -- the chain of trust was broken by *upstream*.
And this would be different for signed (S)RPMs how, exactly?
It's not much different, really. A signed RPM tells me that CentOS did, in fact, build that RPM. It eliminates one possible point of contamination, but it does not ensure an unbroken chain of trust. Back in the olden days, it would have told me that CentOS built it using opaque processes from a signed upstream source -- the SRPMs at ftp.redhat.com -- so there was still a break in the chain of trust, since CentOS's processes were insufficiently public. They're now increasingly public, so a signed RPM tells me that CentOS built the RPM, using well-known processes, from which point I could follow back the build logs and discover the unsigned, untrusted sources it was built from. Whereupon the chain of trust again disappears.
Do you trust CentOS to properly vet all of the sources they pull from git.centos.org? I don't. You saw how quickly they moved from RHEL 7.0 GA to CentOS 7.0 RC -- they weren't doing code audits. Upstream's security and audit processes are completely opaque, but I at least have some confidence that they exist. Without them signing the sources, though, all of the cryptographic assurance people have been talking about in this thread disappears completely.
On Sun, Jul 6, 2014 at 9:04 PM, Chris St. Pierre chris.a.st.pierre@gmail.com wrote:
There's been a lot of discussion on this, which is surprising given that this is the wrong place for it. Maybe next we can discuss how to improve AutoYAST or IIS.
Some people seem to have forgotten that CentOS, despite the recent change in employment status of several of its core members, is the *downstream* rebuilder. They get their sources via git.centos.org from a prominent North American Linux reseller, just like the rest of us.
Read that again: CentOS is not special. They are *consumers* of the sources, not the producers.
They are special. Various members of the CentOS team are now Red Hat employees (http://seven.centos.org/2014/06/congratulations-to-red-hat-for-rhel7/)
Do keep up with curent events.
On Sun, Jul 6, 2014 at 10:06 PM, Nico Kadel-Garcia nkadel@gmail.com wrote:
They are special. Various members of the CentOS team are now Red Hat employees ( http://seven.centos.org/2014/06/congratulations-to-red-hat-for-rhel7/)
Do keep up with curent events.
And yet, amazingly, they're *still* downstream. And not even very privileged downstream. They get their sources the same exact way you do. Do try to understand how very little things have actually changed. It's important -- in fact, it's the difference between talking to someone who has the ability to implement what propose, and shouting into the void.
On 07/07/2014 03:06 AM, Nico Kadel-Garcia wrote:
Read that again: CentOS is not special. They are *consumers* of the sources, not the producers.
They are special. Various members of the CentOS team are now Red Hat employees (http://seven.centos.org/2014/06/congratulations-to-red-hat-for-rhel7/)
Do keep up with curent events.
That doesnt change what Chris said though.
We trust that source ( and the origin of that source ), which is why we build and deliver the distro derived from there ( we do patch a few ), and if someone needs the gpg signed content to setup cascading trust, do so from the srpms.
- KB
On Mon, Jul 7, 2014 at 4:59 AM, Karanbir Singh mail-lists@karan.org wrote:
On 07/07/2014 03:06 AM, Nico Kadel-Garcia wrote:
Read that again: CentOS is not special. They are *consumers* of the sources, not the producers.
They are special. Various members of the CentOS team are now Red Hat employees (http://seven.centos.org/2014/06/congratulations-to-red-hat-for-rhel7/)
Do keep up with curent events.
That doesnt change what Chris said though.
We trust that source ( and the origin of that source ), which is why we build and deliver the distro derived from there ( we do patch a few ), and if someone needs the gpg signed content to setup cascading trust, do so from the srpms.
- KB
And you folks at git.centos.org and the CentOS core developer group, I have some confidence in. Certainly Red Hat does, they hired a bunch of you. The result is that I assume you have good access from which you are building your imported git.centos.org sources, either direct git exports from Red Hat's internal git repos or a full feed Red Hat subscription to work the SRPM's from. I'm actually quite curious which you use, I don't see anything at git.centos.org to indicate. Either way, though, I have some confidence in *your* access to upstream resources.
I'm qhappy to see git.centos.org. I really appreciate the visibility. But as soon as I start pulling its contents from some of the weird intervening proxy setups I've seen in the wild, I'm not talking directly to your repo anymore. Moreover, once I start doing any work in my own clone, or if I share that clone with other developers, I lose verifiability of who wrote what parts. That's what the git tags would provide, and potentially much more consistently than analyzing the 'git log' content for imports. In fact, that practice of encoding SRPM names in the git log 'import' lines is basically re-inventing tags, but without the GPG signature.
Thinking about it, those 'import' labeled commits might be just the right ones to use to create actual tags. Name the tag after what is in the git log 'import' notated SRPM version, sign those, and you've got a much better verifiable source for remote clones to use. And much more legible and easier for developer to review, who can do a ''git checkout kernel-3.1.0-0.0.1' and be working in a sacrosanct local tag checkout, or do 'git diff kernel-3.1.0-0.0.1 kernel-3.1.0-0.0.2' to compare the tags much more easily than they can by deriving and manipulating commit tags read from the logs.
Also, one of the more useful tricks I've been doing with git is to do a "rebase", which is much easier with tags than by deriving the 'git log' import information. It's certainly more *legible*. But consistently labeled 'git tag' use let's me refer to those tags directly, and when I check one out locally it's in "read only" mode
It's extra work, I admit. I don't know if you're signing the RPM's and SRPM's manually, or with an automated build system. But if we're going to develop, and want to be able to rely on the code with concerns about the provenance, well, git.centos.org is a separate channel with different security needs. It's not technologically difficult to bring it up to a similar standard as the RPM and SRPM files.
On Mon, Jul 7, 2014 at 6:22 AM, Nico Kadel-Garcia nkadel@gmail.com wrote:
And you folks at git.centos.org and the CentOS core developer group, I have some confidence in. Certainly Red Hat does, they hired a bunch of you. The result is that I assume you have good access from which you are building your imported git.centos.org sources, either direct git exports from Red Hat's internal git repos or a full feed Red Hat subscription to work the SRPM's from. I'm actually quite curious which you use, I don't see anything at git.centos.org to indicate. Either way, though, I have some confidence in *your* access to upstream resources.
This is where I've been trying to tell you you're wrong. They've made it clear that they use git.centos.org, just like the rest of us. The name on their paycheck doesn't make them special in this regard. The domain in their email address doesn't make them special in this regard. They are subject to the same limitations that we are, which is why asking *them* to certify that sources they *only consume* is pure folly.
Four CentOS core members work for upstream. That does *not* make CentOS upstream.
On 07/07/2014 07:18 AM, Chris St. Pierre wrote:
On Mon, Jul 7, 2014 at 6:22 AM, Nico Kadel-Garcia <nkadel@gmail.com mailto:nkadel@gmail.com> wrote:
And you folks at git.centos.org <http://git.centos.org> and the CentOS core developer group, I have some confidence in. Certainly Red Hat does, they hired a bunch of you. The result is that I assume you have good access from which you are building your imported git.centos.org <http://git.centos.org> sources, either direct git exports from Red Hat's internal git repos or a full feed Red Hat subscription to work the SRPM's from. I'm actually quite curious which you use, I don't see anything at git.centos.org <http://git.centos.org> to indicate. Either way, though, I have some confidence in *your* access to upstream resources.This is where I've been trying to tell you you're wrong. They've made it clear that they use git.centos.org http://git.centos.org, just like the rest of us. The name on their paycheck doesn't make them special in this regard. The domain in their email address doesn't make them special in this regard. They are subject to the same limitations that we are, which is why asking *them* to certify that sources they *only consume* is pure folly.
This is correct in one sense, we do indeed consume git.centos.org, like everyone else. We do NOT have access to the machine(s) where the commits are made from (where the SRPMs become a git tree), that is done by upstream. There is a purposeful isolation between the resources of the CentOS team and the RHEL team.
We do know that things that show up on git.centos.org came from a specific ip address, used a specific key/user combination from that ipaddress to deliver the content on git.centos.org .. so we know where it originated from. (If they are imported by the upstream user), so we have confidence that content is authentic (as in; it came from upstream).
So, the authenticity of the code is not in question .. it is provided by upstream or by the person listed in the git log. (for content that is not from upstream, ie the scripts in centos-git.common).
Four CentOS core members work for upstream. That does *not* make CentOS upstream.
This is absolutely true ... we (the 4 people hired by Red Hat that produce CentOS) do not work from anywhere we did not work before we were hired by Red Hat (we all work from home on CentOS, then and now). Nor do we have any access to any resources or information that the RHEL team at Red Hat does not make public. We also do not have access to any code going into CentOS before it shows up on git.centos.org.
The process on our end has not changed, other than where we get the code from.
All of this stolen CA's to make an SSL certificate that looks valid to a user who uses git.centos.org, then targeting that user's DNS to make them download bad code from git.centos.org so that they can then inject code, while theoretically possible, is silly.
Besides, Red Hat provides source code to their Customers via RHN in SRPM form. They provide the expanded Code to the community via git.centos.org.
If you want to validate the code is the same, then subscribe to RHEL and do your own comparison.
RHEL is RHEL and CentOS is CentOS .. use CentOS if you want CentOS .. Use RHEL if you want RHEL ... and if you are another rebuilder, and if you don't want to use git.centos.org, then talk to and sign an agreement with Red Hat to do something else. All the rest of this is just silly.
On 07/06/2014 09:06 PM, Nico Kadel-Garcia wrote:
On Sun, Jul 6, 2014 at 9:04 PM, Chris St. Pierre chris.a.st.pierre@gmail.com wrote:
There's been a lot of discussion on this, which is surprising given that this is the wrong place for it. Maybe next we can discuss how to improve AutoYAST or IIS.
Some people seem to have forgotten that CentOS, despite the recent change in employment status of several of its core members, is the *downstream* rebuilder. They get their sources via git.centos.org from a prominent North American Linux reseller, just like the rest of us.
Read that again: CentOS is not special. They are *consumers* of the sources, not the producers.
They are special. Various members of the CentOS team are now Red Hat employees (http://seven.centos.org/2014/06/congratulations-to-red-hat-for-rhel7/)
Do keep up with curent events.
Yes. They pay the salary for some of us. We're STILL not special, and we STILL don't have the ability to reach in and influence the business units. We've said that time and again as well. http://www.zdnet.com/centos-7-is-on-its-way-7000030443/
Do try to keep up when you're telling others to keep up.
Now, the point of saying that was to say this: Lets at least pretend to be civil on this list. You've come storming into the list, with a condescending attitude, talking down to several. Do you honestly feel that being an asshole on the list helps your cause? If the merits of your argument are sound, it'll speak for itself. Being an asshole actually HURTS your cause and weakens your argument.
Please be civil on the list.
On Mon, Jul 7, 2014 at 6:40 AM, Jim Perrin jperrin@centos.org wrote:
On 07/06/2014 09:06 PM, Nico Kadel-Garcia wrote:
On Sun, Jul 6, 2014 at 9:04 PM, Chris St. Pierre chris.a.st.pierre@gmail.com wrote:
There's been a lot of discussion on this, which is surprising given that this is the wrong place for it. Maybe next we can discuss how to improve AutoYAST or IIS.
Some people seem to have forgotten that CentOS, despite the recent change in employment status of several of its core members, is the *downstream* rebuilder. They get their sources via git.centos.org from a prominent North American Linux reseller, just like the rest of us.
Read that again: CentOS is not special. They are *consumers* of the sources, not the producers.
They are special. Various members of the CentOS team are now Red Hat employees (http://seven.centos.org/2014/06/congratulations-to-red-hat-for-rhel7/)
Do keep up with curent events.
Yes. They pay the salary for some of us. We're STILL not special, and we STILL don't have the ability to reach in and influence the business units. We've said that time and again as well. http://www.zdnet.com/centos-7-is-on-its-way-7000030443/
Do try to keep up when you're telling others to keep up.
I read the article, and others at the time. When someone pays your salary and funds your build hardware, that's pretty "special" compared to other open source and freeware teams managing various software forks. It's not unheard of, and I've no moral or ethical issue with it. (I'm thinking of Zmanda and AMANDA, and the various Subversion related companies, with which I've also dealt in the past.) But I'm afraid the funding relationship does, indeed, make it "special" compared to other developers or rebuilders.
It seems a healthy and reasonable relationship. Red Hat has been an excellent participant in free software and open source for many years, and support of CentOS is a good sign of that ongoing support. That doesn't worry me, at least. And gods forbid, if some business unit did try to improperly influence you, CentOS and Red Hat have shown good historical resistance to administrative abuse and I'd expect you to handle it safely.
Now, the point of saying that was to say this: Lets at least pretend to be civil on this list. You've come storming into the list, with a condescending attitude, talking down to several. Do you honestly feel that being an asshole on the list helps your cause? If the merits of your argument are sound, it'll speak for itself. Being an asshole actually HURTS your cause and weakens your argument.
Sorry if it seems that way. I'm trying to raise real issues of software provenance and repository security and workflow. I'll admit that I was startled to have to try to explain the usefulness of GPG signed git tags, and I do seem to have stepped on some toes with related issues.
I'm not worried about https://git.centos.org security itself. (At least, no more than any well supported open source website!). The CentOS developers and maintainers have earned some confidence in their work. It's the intermediate offsite staging, whether in a local working repository or possibly a trojaned intermediate repository, that I'm concerned about. That's outside of the direct control, but a certain level of verification of local repositories, especially with well labeled tags, could help improve confidence for anyone working from your git repo in local repositories, rather than working from raw SRPM's.
Please be civil on the list.
Trying to. I do wind up openly contradicting someone when I think they're mistaken, as I just tried to contradict you about the lack of any "special" relationship of CentOS to Red Hat. But I did try to do so civilly with you, and to be careful to provide evidence and reasoning to support the conclusion.
That's about the best I can do.
-
Chris St. Pierre -
Elias Persson -
Jim Perrin -
Johnny Hughes -
Karanbir Singh -
Ljubomir Ljubojevic -
Mark Mielke -
Nico Kadel-Garcia