Your Wkii page here:
https://wiki.centos.org/FAQ/CentOSStream#Where_is_the_source_code.3F
After discussion in which it was confirmed that TLS *could* be implemented "but traditionally we have not done so", was just updated by Manuel Wolfshant with the following lie:-
*Note: downloads are hosted on a mirror network, where we cannot mandate that every mirror node runs SSL/TLS, hence using regular http and not enforcing https*
False statements are disgusting to begin with, but ones that attempt to excuse the lazy decision to put all CentOS customers at risk are totally unacceptable. LE is free and easy to use and setup - it's a no-brainer to fix this problem, assuming someone isn't getting a kickback from some 3-letter-agency to leave this exploitable security hole open ?
Greetings,
----- Original Message -----
Your Wkii page here:
https://wiki.centos.org/FAQ/CentOSStream#Where_is_the_source_code.3F
After discussion in which it was confirmed that TLS *could* be implemented "but traditionally we have not done so", was just updated by Manuel Wolfshant with the following lie:-
Note: downloads are hosted on a mirror network, where we cannot mandate that every mirror node runs SSL/TLS, hence using regular http and not enforcing https
False statements are disgusting to begin with, but ones that attempt to excuse the lazy decision to put all CentOS customers at risk are totally unacceptable. LE is free and easy to use and setup - it's a no-brainer to fix this problem, assuming someone isn't getting a kickback from some 3-letter-agency to leave this exploitable security hole open ? _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
I was saddened to see your email to the mailing list contained a lie.. in that you spelled wiki incorrectly. Such mistakes are disgusting and lazy.
TYL,
On Wed, 2021-02-10 at 06:48 +1000, Chris Drake wrote:
Your Wkii page here:
https://wiki.centos.org/FAQ/CentOSStream#Where_is_the_source_code.3F
After discussion in which it was confirmed that TLS *could* be implemented "but traditionally we have not done so", was just updated by Manuel Wolfshant with the following lie:-
*Note: downloads are hosted on a mirror network, where we cannot mandate that every mirror node runs SSL/TLS, hence using regular http and not enforcing https*
False statements are disgusting to begin with, but ones that attempt to excuse the lazy decision to put all CentOS customers at risk are totally unacceptable. LE is free and easy to use and setup - it's a no- brainer to fix this problem, assuming someone isn't getting a kickback from some 3-letter-agency to leave this exploitable security hole open ? _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
Well..
*Technically* CentOS users are not customers - at all in fact - unless they also happen to also own a paid RHEL subscription.
Now onto the issue at hand. While the info should be accurate, I don't think it's a big deal.
TLS is certainly preferable for the mirror network, it isn't entirely required from a security point of view.
Realistically TLS shines most when you're transporting customer (user data) or are dealing with some kind of sensitive information, trying to stop prying eyes etc.
From a mirror perspective it's not overly important because the only
protection TLS can add in this case is to prevent RPM tampering. But even if someone intercepted your connection and successfully switched the RPM while it was downloading the risk is minimal.
This is because your local machine has the GPG key identity required for the packages. All CentOS (and most RPM distros) sign their packages with a GPG key, which package managers then use to verify the RPM has not been tampered with.
That's why if you want to install a non-GPG signed package from a repo you need to specifically tell yum/dnf to ignore GPG signing.
So, TLS or not, if your package has been swapped with a fake, your package manager should notice this and refuse to install that package.
The biggest problem from a security point of view would probably be a rogue mirror that serves up modified packages.. a rogue mirror could also carry TLS.
That's why you sign the RPM for security.
If you're worried that someone sees the package your about to install, just remember two things:
1) Mirrors are public, everyone can easily know what packages and their versions exist.
2) If there's an exploitable package just waiting to be installed for an easy exploit.. whoever has that exploit will probably be randomly probing systems for it - they won't need to monitor a mirrors downloads..
This of course is all my quick and short personal opinion, I don't work for Red Net. I reserve the right to be dead wrong.
Just my 3c.
- Jake
On Tuesday, February 9, 2021 7:41 PM, Jake Shipton listmail@crazylinuxnerd.net wrote:
On Wed, 2021-02-10 at 06:48 +1000, Chris Drake wrote:
Your Wkii page here: https://wiki.centos.org/FAQ/CentOSStream#Where_is_the_source_code.3F After discussion in which it was confirmed that TLS could be implemented "but traditionally we have not done so", was just updated by Manuel Wolfshant with the following lie:- Note: downloads are hosted on a mirror network, where we cannot mandate that every mirror node runs SSL/TLS, hence using regular http and not enforcing httpsFalse statements are disgusting to begin with, but ones that attempt to excuse the lazy decision to put all CentOS customers at risk are totally unacceptable. LE is free and easy to use and setup - it's a no- brainer to fix this problem, assuming someone isn't getting a kickback from some 3-letter-agency to leave this exploitable security hole open ?
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
Well..
Technically CentOS users are not customers - at all in fact - unless they also happen to also own a paid RHEL subscription.
Now onto the issue at hand. While the info should be accurate, I don't think it's a big deal.
TLS is certainly preferable for the mirror network, it isn't entirely required from a security point of view.
Realistically TLS shines most when you're transporting customer (user data) or are dealing with some kind of sensitive information, trying to stop prying eyes etc.
From a mirror perspective it's not overly important because the only protection TLS can add in this case is to prevent RPM tampering. But even if someone intercepted your connection and successfully switched the RPM while it was downloading the risk is minimal.
This is because your local machine has the GPG key identity required for the packages. All CentOS (and most RPM distros) sign their packages with a GPG key, which package managers then use to verify the RPM has not been tampered with.
That's why if you want to install a non-GPG signed package from a repo you need to specifically tell yum/dnf to ignore GPG signing.
So, TLS or not, if your package has been swapped with a fake, your package manager should notice this and refuse to install that package.
The biggest problem from a security point of view would probably be a rogue mirror that serves up modified packages.. a rogue mirror could also carry TLS.
That's why you sign the RPM for security.
If you're worried that someone sees the package your about to install, just remember two things:
Mirrors are public, everyone can easily know what packages and their versions exist.
If there's an exploitable package just waiting to be installed for an easy exploit.. whoever has that exploit will probably be randomly probing systems for it - they won't need to monitor a mirrors downloads..
This of course is all my quick and short personal opinion, I don't work for Red Net. I reserve the right to be dead wrong.
Just my 3c.
I agree that TLS is being oversold as being a security benefit in this situation.
As long as we are being pedantic about repository security, my person observation is the best point of attack is the repo XML files. These are not signed. If a rogue mirror or a man in the middle attack did take place, this seems like the best target. From what I can tell, DNF (and libxml2) typically are parsing these files while running as root. A zero-day against libxml2 would be gold.
I would not go as far as to indicate this proves anyone has been lazy. Red Hat has been responsive to patching libxml2 as soon as they become aware of problems. But it would be nice if this was handled better.
What excites me about Stream is the potential to push security enhancements forward sooner than just contributing into Fedora does. We may be able to backport important changes into Stream that the RHEL team may not have taken the time to in the past.
Maybe someday we will see a pull request to createrepo to support adding a signature element towards the top of the XML file. And then another pull request for dnf to perform signature checking before xml parsing, preferably running in an unpriviledged/sandboxed way when possible.
Normally even after the pull requests have proven themselves in Fedora for a while we would have to wait for the next RHEL cycle to see the changes in RHEL itself. But I think Stream may give hope of getting something like "gpgcheckxml" for dnf into a Technology Preview for Stream/RHEL 8.
Right now it is still unclear to what degree we will just be mere consumers of Stream or how much we will be drivers of the roadmap. It seems like in the short term we might have to purpose a dnf-NG SIG group instead of being able to contribute back directly. But even that would be an improvement over how things worked in the past.
On Wed, Feb 10, 2021 at 02:42:35AM +0000, redbaronbrowser via CentOS-devel wrote:
On Tuesday, February 9, 2021 7:41 PM, Jake Shipton listmail@crazylinuxnerd.net wrote:
As long as we are being pedantic about repository security, my person observation is the best point of attack is the repo XML files. These are not signed. If a rogue mirror or a man in the middle attack did take place, this seems like the best target. From what I can tell, DNF (and libxml2) typically are parsing these files while running as root. A zero-day against libxml2 would be gold.
Repo metadata is signed.
John
On Tuesday, February 9, 2021 8:58 PM, John R. Dennison jrd@gerdesas.com wrote:
On Wed, Feb 10, 2021 at 02:42:35AM +0000, redbaronbrowser via CentOS-devel wrote:
As long as we are being pedantic about repository security, my person observation is the best point of attack is the repo XML files. These are not signed. If a rogue mirror or a man in the middle attack did take place, this seems like the best target. From what I can tell, DNF (and libxml2) typically are parsing these files while running as root. A zero-day against libxml2 would be gold.
Repo metadata is signed.
From what I can tell, the repo metadata is hashed by sha256 but that is not the same a cryptographically signed.
What are you finding is performing a verification of the repomd.xml against the CentOS public key before parsing it with libxml2?
From what I can tell, the repo metadata is hashed by sha256 but that is not the same a cryptographically signed.
What are you finding is performing a verification of the repomd.xml against the CentOS public key before parsing it with libxml2?
There are two kinds of signatures:
gpg signatures on rpms (option gpgcheck) == verify source and integrity of the RPM
gpg signatures on yum metadata (option repo_gpgcheck) == verifies the metadata and thus what you see in the repository.
This verifies that nobody tampered with what has been published and thus the list of packages you get is what was published.
while the first is enabled (https://git.centos.org/rpms/centos-release/blob/c8s/f/SOURCES/CentOS-Base.re... or https://git.centos.org/rpms/centos-repos/blob/c8/f/SOURCES/CentOS-Linux-Base...) the second is not.
It would also be great to do the second by default, especially since the mirrorlist keeps being pure http.
More on background: https://blog.packagecloud.io/eng/2014/11/24/howto-gpg-sign-verify-rpm-packag...
What is important with http as a pure transport for mirrors is what Julien pointed out:
Even if you have gpgcheck and repo_gpgcheck enabled and there is a 0day in libxml2 an active attacker can keep serving you the outdated repository information and you won't get any updates. But all seems fine, since you verify signature of both. If you don't verify metadata (as it is now) an active attacker can even serve you updated content for all the other packages except for libxml2 and while keeping you thinking that you have your system up2date (since it runs and installs updates) your system can be kept vulnerable.
If the source from where you are fetching the content is using https and you are validating the certificate it is much harder (or nearly impossible) to deliver you an outdated (or tampered) view on the repository.
Now what is the state of how you get your content using dnf:
you fetch the mirrorlist
curl "http://mirrorlist.centos.org/?release=8-stream&arch=x86_64&repo=AppS..."
=> you get a list of mirrors with their own hostname
and you fetch the current metadata from one of these mirrors.
Since you are redirected to mirrors using their own hostname, they can likely use LE (or whatever they want) to get a certificate signed by a well known CA.
And it looks like nearly all of the mirrors I get back do that:
tldr; one does not answer on 443 and one has a certificate not signed by a well known CA.
# curl -s "http://mirrorlist.centos.org/?release=8-stream&arch=x86_64&repo=AppS..." | while read -r repo; do echo -n "${repo} "; curl -s -o /dev/null --connect-timeout 2 -I $(echo $repo | sed "s@http://@https://@") && echo OK || echo FAILED; done http://pkg.adfinis.com/centos/8-stream/AppStream/x86_64/os/ OK http://mirror.init7.net/centos/8-stream/AppStream/x86_64/os/ OK http://linuxsoft.cern.ch/centos/8-stream/AppStream/x86_64/os/ OK http://centos.mirror.net-d-sign.de/8-stream/AppStream/x86_64/os/ FAILED http://artfiles.org/centos.org/8-stream/AppStream/x86_64/os/ OK http://mirror.infonline.de/centos/8-stream/AppStream/x86_64/os/ OK http://mirror.netzwerge.de/centos/8-stream/AppStream/x86_64/os/ OK http://ftp.plusline.net/centos/8-stream/AppStream/x86_64/os/ OK http://ftp.rz.uni-frankfurt.de/pub/mirrors/centos/8-stream/AppStream/x86_64/... FAILED http://mirror.imt-systems.com/centos/8-stream/AppStream/x86_64/os/ OK
So the hard thing is to get mirrorlist behind https and nagging the few mirrors not yet arrived in 2021 to enable proper https.
Now if we look at what Fedora does (e.g. for EPEL):
curl -s "https://mirrors.fedoraproject.org/metalink?repo=epel-source-8&arch=x86_6..." | grep protocol= | sed 's/.*protocol="(.*)" type=.*/\1/' | sort | uniq -c 64 http 36 https 50 rsync
=> there is a reasonable amount of https mirrors out there, but still quite a lot http - don't know how dnf selects them...
Same picture for Fedora repos, since it's the same infra.
Anyway how it could make work, so that folks could hop into a more secure delivery is:
1. serve mirrorlist under http (default as by now) & https (new!) 2. server mirrorlist as xml with more metadata per mirror (protocol & type) as for epel (new!) 3. make it possible that dnf can prefer/enable/disable protocols (new!)
then with CentOS 9 Stream one could switch to
1. have mirrorlist by default over https (http variant still available) 2. enable repo_gpgcheck by default 3. let https be the preferred method for dnf (can be downgraded to http for folks behind ssl terminating proxies without local mirrors)
However, I guess since things are intervened with Fedora and Fedora also has:
1. repo_gpgcheck not enabled by default :( 2. a mixed list of http, https and rsync mirrors 3. no way on dnf (afaik) to prefer https
it's probably a good starting point over there.
~pete
On Wed, Feb 10, 2021 at 10:20:19AM +0100, Peter Meier wrote:
Now if we look at what Fedora does (e.g. for EPEL):
curl -s "https://mirrors.fedoraproject.org/metalink?repo=epel-source-8&arch=x86_6..." | grep protocol= | sed 's/.*protocol="(.*)" type=.*/\1/' | sort | uniq -c 64 http 36 https 50 rsync
If you add '&protocol=https' you will only get https mirrors from that Fedora link.
Adrian
Am 10.02.21 um 10:20 schrieb Peter Meier:
However, I guess since things are intervened with Fedora and Fedora also has:
- repo_gpgcheck not enabled by default :(
I had asked this before. JFYI:
https://bugzilla.redhat.com/show_bug.cgi?id=1851242
- a mixed list of http, https and rsync mirrors
- no way on dnf (afaik) to prefer https
it's probably a good starting point over there.
-- Leon
On Wednesday, February 10, 2021 7:32 AM, Leon Fauster via CentOS-devel centos-devel@centos.org wrote:
Am 10.02.21 um 10:20 schrieb Peter Meier:
However, I guess since things are intervened with Fedora and Fedora also has:
- repo_gpgcheck not enabled by default :(
I had asked this before. JFYI:
https://bugzilla.redhat.com/show_bug.cgi?id=1851242
- a mixed list of http, https and rsync mirrors
- no way on dnf (afaik) to prefer https
it's probably a good starting point over there.
That bugzilla thread has an interesting point, EPEL is using a metalink delivered over https to provide the size and hashsums of the repomd.xml.
CentOS differs from EPEL in the following ways:
1. CentOS delivers mirrorlist instead of metalink
2. CentOS delivers mirrorlist over http by default instead of https
While EPEL mirroring has some of the same challages as CentOS, they seem to have handled it in a cleaner way.
It is true that not all mirrors are TLS enabled.
But "mirrorlist.centos.org" is under CentOS direct control. Also, the centos-linux-repos is authored by CentOS.
It should be possible for CentOS to respond to this request by updating the centos-linux-repos to use https for the mirrorlist and enable repo_gpgcheck by default.
It would also be nice if migration to using metalink instead of mirrorlist was a goal of the CentOS infrastructure team.
Am 10.02.21 um 16:36 schrieb redbaronbrowser:
But "mirrorlist.centos.org" is under CentOS direct control. Also, the centos-linux-repos is authored by CentOS.
I am not part of the CentOS team therefore "non-authoritative answer".
I was also subject to this assumption, but if I have understood it correctly "mirrorlist.centos.org" is glued together with sponsored/community donated machines. So, not in the realm of CentOS.
Just do:
$ dig +short -t A mirrorlist.centos.org
-- Leon
On Wednesday, February 10, 2021 9:51 AM, Leon Fauster via CentOS-devel centos-devel@centos.org wrote:
Am 10.02.21 um 16:36 schrieb redbaronbrowser:
But "mirrorlist.centos.org" is under CentOS direct control. Also, the centos-linux-repos is authored by CentOS.
I am not part of the CentOS team therefore "non-authoritative answer".
I was also subject to this assumption, but if I have understood it correctly "mirrorlist.centos.org" is glued together with sponsored/community donated machines. So, not in the realm of CentOS.
Just do:
$ dig +short -t A mirrorlist.centos.org
Maybe it is time to establish a metalink.centos.org that is behind a CDN such as cloudflare.
Migrating to repo_gpgcheck for the centos-linux-repos package would be nice to.
However, I guess since things are intervened with Fedora and Fedora also has:
- repo_gpgcheck not enabled by default :(
I had asked this before. JFYI:
So two things I learned out of that thread and related links (thank you all for them!):
1. EPEL + Fedora use metalink that a) is served over https and b) contains checksums of the current + last 2 valid previous once
2. you can tell metalink by adding protocol=https to only return https mirrors.
And that gives you a good enough chain of trust and you will fetch the content over https. Which also protects you from a passive attacker learning about what services and software you have on your boxes.
Now on the CentOS side, you still have to enable repo_gpgcheck, since the main repositories are being served using a simple mirrorlist over http pointing to http mirrors without any checksums like metalink.
Meaning:
For Stream 8:
dnf config-manager --save --setopt=*.repo_gpgcheck=1 appstream baseos \ extras powertools
And CentOS 7:
yum-config-manager --save --setopt=*.repo_gpgcheck=1 base updates \ extras centosplus cr centos-sclo-rh fasttrack centos-sclo-sclo
Gives you a way to validate the served repodata.
Now, this still allows an active attacker to keep you getting an outdated view on the repository to lock you out of updates.
Thus it would still be beneficial to either make the mirrorlist available over https containing https only servers. OR also using metalink over https to redirect to correct mirrors and thus including checksums for the current repodata.
~pete
Am 10.02.21 um 02:41 schrieb Jake Shipton:
On Wed, 2021-02-10 at 06:48 +1000, Chris Drake wrote:
Your Wkii page here:
https://wiki.centos.org/FAQ/CentOSStream#Where_is_the_source_code.3F
After discussion in which it was confirmed that TLS *could* be implemented "but traditionally we have not done so", was just updated by Manuel Wolfshant with the following lie:-
*Note: downloads are hosted on a mirror network, where we cannot mandate that every mirror node runs SSL/TLS, hence using regular http and not enforcing https*
False statements are disgusting to begin with, but ones that attempt to excuse the lazy decision to put all CentOS customers at risk are totally unacceptable. LE is free and easy to use and setup - it's a no- brainer to fix this problem, assuming someone isn't getting a kickback from some 3-letter-agency to leave this exploitable security hole open ? _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
Well..
*Technically* CentOS users are not customers - at all in fact - unless they also happen to also own a paid RHEL subscription.
Now onto the issue at hand. While the info should be accurate, I don't think it's a big deal.
TLS is certainly preferable for the mirror network, it isn't entirely required from a security point of view.
Realistically TLS shines most when you're transporting customer (user data) or are dealing with some kind of sensitive information, trying to stop prying eyes etc.
From a mirror perspective it's not overly important because the only protection TLS can add in this case is to prevent RPM tampering. But even if someone intercepted your connection and successfully switched the RPM while it was downloading the risk is minimal.
This is because your local machine has the GPG key identity required for the packages. All CentOS (and most RPM distros) sign their packages with a GPG key, which package managers then use to verify the RPM has not been tampered with.
That's why if you want to install a non-GPG signed package from a repo you need to specifically tell yum/dnf to ignore GPG signing.
So, TLS or not, if your package has been swapped with a fake, your package manager should notice this and refuse to install that package.
The biggest problem from a security point of view would probably be a rogue mirror that serves up modified packages.. a rogue mirror could also carry TLS.
That's why you sign the RPM for security.
dnf not checking gpg signature sounds scary:
https://github.com/ansible/ansible/blob/v2.9.13/changelogs/CHANGELOG-v2.9.rs...
-- Leon
On 2/9/21 3:48 PM, Chris Drake wrote:
Your Wkii page here:
https://wiki.centos.org/FAQ/CentOSStream#Where_is_the_source_code.3F https://wiki.centos.org/FAQ/CentOSStream#Where_is_the_source_code.3F
After discussion in which it was confirmed that TLS *could* be implemented "but traditionally we have not done so", was just updated by Manuel Wolfshant with the following lie:-
/Note: downloads are hosted on a mirror network, where we cannot mandate that every mirror node runs SSL/TLS, hence using regular http and not enforcing https/
False statements are disgusting to begin with, but ones that attempt to excuse the lazy decision to put all CentOS customers at risk are totally unacceptable. LE is free and easy to use and setup - it's a no-brainer to fix this problem, assuming someone isn't getting a kickback from some 3-letter-agency to leave this exploitable security hole open ?
This kind of personal attack on our trusted, respected community members is simply not acceptable on this list.
Perhaps you would like to reach out to each of our mirror hosts and help them get LE set up on each of their servers?