1. Your info page here:
https://wiki.centos.org/FAQ/CentOSStream#Where_is_the_source_code.3F
links to an insecure download resource: http://mirror.centos.org/centos/8-stream/
2. You are not running a secure server:
https://mirror.centos.org/centos/8-stream/ => connection times out
*. Hopefully you understand the implications of the above - if not, run a build and take a look at the number of warnings related to unsigned code that your systems ignore. Better still - fix your systems so they always hard-fails on everything unsigned it encounters. It only takes one single unsigned mistake in any of your packages to expose all users to compromise when you're not using secure servers. Insecure servers in 2021 are completely unnecessary.
3. Source code is still missing. The folder structure exists, but none of the files are in there.
Some new examples
https://git.centos.org/rpms/sendmail/tree (no source)
https://git.centos.org/rpms/sendmail/archive/imports/c8s/sendmail-8.15.2-34.... (linked from git - 404)
https://vault.centos.org/centos/8-stream/AppStream/Source/SPackages/ (empty)
https://composes.centos.org/CentOS-Stream-8-20210108.n.2/compose/BaseOS/sour... (incomplete)
# yumdownloader --source sendmail Last metadata expiration check: 2:09:27 ago on Mon 08 Feb 2021 09:45:31 PM GMT. No package sendmail-8.15.2-34.el8.src available. Exiting due to strict setting. Error: No package sendmail-8.15.2-34.el8.src available.
Might I suggest you ask someone in the build team to fix or write whatever script is needed to make "yumdownloader" work? Obviously, since they're building stuff, *they* know where the source code **really** is - so it would only take 5 or 10 minutes to glue your existing tools (like yumdownloader) into whatever new location someone seems to have dreamed up for the actual source.
Spending the few minutes to fix what every administrator already knows around source packaging/distro systems is a far better idea than making them all learn entirely new things (which will probably change a few more times before everyone's happy anyhow)
All the above carry security implications - we really need to know what source was used to build our products, and we really need to be able to download binaries from properly secure locations (preferably all with working signatures, but that's a whole other problem, so TLS distro endpoints is at least an interim mitigation).
# yumdownloader --source sendmail Last metadata expiration check: 2:09:27 ago on Mon 08 Feb 2021 09:45:31 PM GMT. No package sendmail-8.15.2-34.el8.src available. Exiting due to strict setting. Error: No package sendmail-8.15.2-34.el8.src available.
Might I suggest you ask someone in the build team to fix or write whatever script is needed to make "yumdownloader" work? Obviously, since they're building stuff, *they* know where the source code **really** is - so it would only take 5 or 10 minutes to glue your existing tools (like yumdownloader) into whatever new location someone seems to have dreamed up for the actual source.
It has been pointed out multiple times (also during the Dojo), that the team is working on delivering the sources as SRPMs for CentOS Stream in the repositories as they are for CentOS Linux. So stay tuned.
Meanwhile all the sources used to build CentOS Stream content has always been available through https://git.centos.org/ and there are the following tools to consume dist-git content easily:
https://git.centos.org/centos-git-common
Meanwhile: Keep in mind - and this was always communicated this way - the shift of direction is by the end of 2021 and the announcement was done early to give everybody a clear heads up and also gather feedback on what is important. BUT this also means that not everything is in place yet as you know it from CentOS Linux. Nevertheless, the team now works on CentOS Linux 7 & 8 + making stream ready to replace 8 + making sure Stream 9 is able to start.
And yes it would be nice if in 2021 all connections are done through TLS.
~pete
Hi Peter,
"working on delivering" is nice, but it's a GPL legal requirement that this be done, so getting it completed should be priority.
"Meanwhile all the sources used to build CentOS Stream content has always been available through https://git.centos.org/ "
Did you follow my link? I found at least one source that is missing - so it looks like whoever is doing the build is not in fact using that repo to do it from.
It blows my mind how insecure this all is - security news is packed with daily exploits being discovered, yet everyone still seems happy to run sketchy code downloaded from insecure web sites for which none of the source that was used really exists when you go looking for it, and where the entire build and installation process is programmed to ignore missing and invalid digital signatures...
On Tue, Feb 9, 2021 at 8:08 PM Peter Meier peter.meier@immerda.ch wrote:
# yumdownloader --source sendmail Last metadata expiration check: 2:09:27 ago on Mon 08 Feb 2021 09:45:31 PM GMT. No package sendmail-8.15.2-34.el8.src available. Exiting due to strict setting. Error: No package sendmail-8.15.2-34.el8.src available.
Might I suggest you ask someone in the build team to fix or write whatever script is needed to make "yumdownloader" work? Obviously, since they're building stuff, *they* know where the source code **really** is - so it would only take 5 or 10 minutes to glue your existing tools (like yumdownloader) into whatever new location someone seems to have dreamed up for the actual source.
It has been pointed out multiple times (also during the Dojo), that the team is working on delivering the sources as SRPMs for CentOS Stream in the repositories as they are for CentOS Linux. So stay tuned.
Meanwhile all the sources used to build CentOS Stream content has always been available through https://git.centos.org/ and there are the following tools to consume dist-git content easily:
https://git.centos.org/centos-git-common
Meanwhile: Keep in mind - and this was always communicated this way - the shift of direction is by the end of 2021 and the announcement was done early to give everybody a clear heads up and also gather feedback on what is important. BUT this also means that not everything is in place yet as you know it from CentOS Linux. Nevertheless, the team now works on CentOS Linux 7 & 8 + making stream ready to replace 8 + making sure Stream 9 is able to start.
And yes it would be nice if in 2021 all connections are done through TLS.
~pete
Am 09.02.21 um 21:57 schrieb Chris Drake:
Hi Peter,
"working on delivering" is nice, but it's a GPL legal requirement that this be done, so getting it completed should be priority.
"Meanwhile all the sources used to build CentOS Stream content has always been available through https://git.centos.org/ https://git.centos.org/ "
Did you follow my link? I found at least one source that is missing - so it looks like whoever is doing the build is not in fact using that repo to do it from.
It blows my mind how insecure this all is - security news is packed with daily exploits being discovered, yet everyone still seems happy to run sketchy code downloaded from insecure web sites for which none of the source that was used really exists when you go looking for it, and where the entire build and installation process is programmed to ignore missing and invalid digital signatures...
Chris, please take a step back and take a look at some details in a elaborated way. For instance as Fabian already answered, git sources are not in any master branch, they are in sub branches. Additional bin blobs are in a look-aside space outside of git. This is all explained in the wiki. About signed packages, could you please explain your POV called "ignore missing and invalid digital signature".
-- Thanks Leon
On 09/02/2021 07:09, Chris Drake wrote:
- Your info page here:
*. Hopefully you understand the implications of the above - if not, run a build and take a look at the number of warnings related to unsigned code that your systems ignore. Better still - fix your systems so they always hard-fails on everything unsigned it encounters. It only takes one single unsigned mistake in any of your packages to expose all users to compromise when you're not using secure servers. Insecure servers in 2021 are completely unnecessary.
rpm packages are signed and can be verified on your side (depending on your yum config). That's the gpgcheck parameter there.
The transport then does not matter that much; anyhow, I agree, also having a the option to pull rpms down over a secured link would give another layer of trust.
Matthias
On 09 Feb 11:50, Matthias Runge wrote:
On 09/02/2021 07:09, Chris Drake wrote:
- Your info page here:
*. Hopefully you understand the implications of the above - if not, run a build and take a look at the number of warnings related to unsigned code that your systems ignore. Better still - fix your systems so they always hard-fails on everything unsigned it encounters. It only takes one single unsigned mistake in any of your packages to expose all users to compromise when you're not using secure servers. Insecure servers in 2021 are completely unnecessary.
rpm packages are signed and can be verified on your side (depending on your yum config). That's the gpgcheck parameter there.
The transport then does not matter that much; anyhow, I agree, also having a the option to pull rpms down over a secured link would give another layer of trust.
Hello Matthias,
The issue is that someone doing a man in the middle attack over http could serve an old version of the mirrors and have properly signed versions of everything with known vulnerabilities.
Regards,
Matthias _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
The issue is that someone doing a man in the middle attack over http could serve an old version of the mirrors and have properly signed versions of everything with known vulnerabilities.
Exactly, this is the main (and valid!) concern for serving things over plain http. Thus should be addressed.
But as we learned through that thread, none of that actually attributes to the other claims initially made, since they all have been debunked to be wrong.
~pete
On 2/9/21 1:09 AM, Chris Drake wrote:
- Your info page here:
https://wiki.centos.org/FAQ/CentOSStream#Where_is_the_source_code.3F https://wiki.centos.org/FAQ/CentOSStream#Where_is_the_source_code.3F
links to an insecure download resource: http://mirror.centos.org/centos/8-stream/ http://mirror.centos.org/centos/8-stream/
As a question that gets asked several times a year, it would be great if someone could update that entry on the wiki (or perhaps link to somewhere that it's been addressed) to reflect *why* this is http and https?
In short, it's because downloads are hosted on a mirror network, where we cannot mandate that every mirror node run SSL/TLS. Well, I suppose we *could*, but traditionally we have not done so, as the additional requirement is likely to reduce the number of willing participants in that mirror network.
On 2/9/21 4:10 PM, Rich Bowen wrote:
On 2/9/21 1:09 AM, Chris Drake wrote:
- Your info page here:
https://wiki.centos.org/FAQ/CentOSStream#Where_is_the_source_code.3F https://wiki.centos.org/FAQ/CentOSStream#Where_is_the_source_code.3F
links to an insecure download resource: http://mirror.centos.org/centos/8-stream/ http://mirror.centos.org/centos/8-stream/
As a question that gets asked several times a year, it would be great if someone could update that entry on the wiki (or perhaps link to somewhere that it's been addressed) to reflect *why* this is http and https?
Done
In short, it's because downloads are hosted on a mirror network, where we cannot mandate that every mirror node run SSL/TLS. Well, I suppose we *could*, but traditionally we have not done so, as the additional requirement is likely to reduce the number of willing participants in that mirror network.
The irrational suggestion that maybe some participants might be less willing to mirror secure resources is absurd - if anything, it will be the opposite - no security-conscious service is going to want to be associated with distributing insecure binaries.
Please stop making this worse - if you can't or don't want to fix it, go away and assign this to someone who cares about our security.
Like I said in my report - CentOS is not secure during installation or build, because missing and mismatched signatures exist and are ignored. Distributing files from insecure servers is a vector that makes those oversights exploitable.
On Wed, Feb 10, 2021 at 12:19 AM Manuel Wolfshant wolfy@nobugconsulting.ro wrote:
On 2/9/21 4:10 PM, Rich Bowen wrote:
On 2/9/21 1:09 AM, Chris Drake wrote:
- Your info page here:
https://wiki.centos.org/FAQ/CentOSStream#Where_is_the_source_code.3F https://wiki.centos.org/FAQ/CentOSStream#Where_is_the_source_code.3F
links to an insecure download resource: http://mirror.centos.org/centos/8-stream/ http://mirror.centos.org/centos/8-stream/
As a question that gets asked several times a year, it would be great if someone could update that entry on the wiki (or perhaps link to somewhere that it's been addressed) to reflect *why* this is http and https?
Done
In short, it's because downloads are hosted on a mirror network, where we cannot mandate that every mirror node run SSL/TLS. Well, I suppose we *could*, but traditionally we have not done so, as the additional requirement is likely to reduce the number of willing participants in that mirror network.
On 2/9/21 12:42, Chris Drake wrote:
The irrational suggestion that maybe some participants might be less willing to mirror secure resources is absurd - if anything, it will be the opposite - no security-conscious service is going to want to be associated with distributing insecure binaries.
Please stop making this worse - if you can't or don't want to fix it, go away and assign this to someone who cares about our security.
This sort of response is not acceptable, and will not be tolerated on this list.
Am 09.02.21 um 15:10 schrieb Rich Bowen:
On 2/9/21 1:09 AM, Chris Drake wrote:
- Your info page here:
https://wiki.centos.org/FAQ/CentOSStream#Where_is_the_source_code.3F https://wiki.centos.org/FAQ/CentOSStream#Where_is_the_source_code.3F
links to an insecure download resource: http://mirror.centos.org/centos/8-stream/ http://mirror.centos.org/centos/8-stream/
As a question that gets asked several times a year, it would be great if someone could update that entry on the wiki (or perhaps link to somewhere that it's been addressed) to reflect *why* this is http and https?
In short, it's because downloads are hosted on a mirror network, where we cannot mandate that every mirror node run SSL/TLS. Well, I suppose we *could*, but traditionally we have not done so, as the additional requirement is likely to reduce the number of willing participants in that mirror network.
Just curious - mirror.centos.org can still provide the content via TLS-only or not?
Just imagine working on a fedora workstation building manually via mock and I want to verify a rpm. Should I download the key via
http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-Official ?
(I known they exist other ways)
If a 3rd party mirror "serves" only over http: then this a different issue.
-- Leon
In short, it's because downloads are hosted on a mirror network, where we cannot mandate that every mirror node run SSL/TLS. Well, I suppose we *could*, but traditionally we have not done so, as the additional requirement is likely to reduce the number of willing participants in that mirror network.
Somehow Fedora made it work, would be nice to have it as well for CentOS Stream.
I know now maybe someone comes out and points me to the differences between how Fedora manages their mirror network and how it works for CentOS. BUT it's 2021 and browsers are starting to make https mandatory!
~pete
On 09/02/2021 17:41, Peter Meier wrote:
In short, it's because downloads are hosted on a mirror network, where we cannot mandate that every mirror node run SSL/TLS. Well, I suppose we *could*, but traditionally we have not done so, as the additional requirement is likely to reduce the number of willing participants in that mirror network.
Somehow Fedora made it work, would be nice to have it as well for CentOS Stream.
I know now maybe someone comes out and points me to the differences between how Fedora manages their mirror network and how it works for CentOS. BUT it's 2021 and browsers are starting to make https mandatory!
~pete
Just my two cents on initial request so let's recap
initial request was about securely retrieveing sources :
- *all* sources used to build centos 7/8/8-stream are hosted on https : https://git.centos.org/ - instructions to rebuild a src.rpm from git *are* on https enabled wiki : https://wiki.centos.org/Sources#Usage
Now for people not willing to use git, and waiting for src.rpm to land on vault, it's also enforced with HSTS/TLS : https://vault.centos.org
And , as some people mentioned, mirror.centos.org is built from sponsored/community donated machine,s so due to the private key laying around, we always decided to not enforce https on mirror.centos.org. Why ? because, as said by some people already, the transport doesn't really matter *as* all rpm packages are already gpg signed *and* people aren't supposed to point to mirror.centos.org but rather point to mirrorlist, itself redirecting to external validated mirrors, on which we can't enforce to use https either (and again packages are gpg signed already)
One doesn't have to validate gpg keys through mirror.centos.org, as we also centralized *all* gpg key on main website, itself using HSTS/https : https://www.centos.org/keys/
So to recap : - you want to rebuild a src.rpm from git ? all happening over https - you don't want to rebuild it but rather consume it directly ? all happening over https too (vault) - you can to validate that key used to sign pkgs on http mirrors is the correct one ? happening over https through website where we list the gpg keys (including for SIGs)
Does that mean that we'll never find another way to have https without any tls cert/key on filesystems from these mirror.centos.org donated nodes ? we can and I thought about it already but clearly my day job/focus is on other priorities for the moment :)
Thanks for the detailed answer:
Am 09.02.21 um 18:00 schrieb Fabian Arrotin: <snip>
And , as some people mentioned, mirror.centos.org is built from sponsored/community donated machine,s so due to the private key laying around, we always decided to not enforce https on mirror.centos.org.
<snip>
Does that mean that we'll never find another way to have https without any tls cert/key on filesystems from these mirror.centos.org donated nodes ? we can and I thought about it already but clearly my day job/focus is on other priorities for the moment :)
To be honest, I had a different assumption in place and the two above statements proved that I was at a wrong starting point.
I thought that mirror.centos.org as a SOA for the mirror-network is in the realm for the CentOS project. This seems not to be true and explains everything else.
About the initial request: Independent from the valid statements about the insurance of the assets integrity (by gpg signing). It would be good practice to have more then one defense line.
Already in place, repo_gpgcheck. The main repos have support for repo_gpgcheck (thanks for that):
This can be enabled with (CentOS Linux 8):
# dnf config-manager --save --setopt=*.repo_gpgcheck=1 appstream baseos cr devel extras fasttrack plus powertools
Unfortunately the "CentOS-Linux-Sources" repos under vault do not provide a signed repomd.xml.
Just some thoughts.
-- Leon
On 09/02/2021 07:09, Chris Drake wrote:
- Your info page here:
<snip>
- Source code is still missing. The folder structure exists, but none
of the files are in there.
Some new examples
https://git.centos.org/rpms/sendmail/tree https://git.centos.org/rpms/sendmail/tree (no source)
What about checking in the correct branch ? For 8-stream, it's all in the c8s branch : https://git.centos.org/rpms/sendmail/tree/c8s
On 2/9/21 6:02 PM, Fabian Arrotin wrote:
On 09/02/2021 07:09, Chris Drake wrote:
- Your info page here:
<snip> > > 3. Source code is still missing. The folder structure exists, but none > of the files are in there. > > Some new examples > > https://git.centos.org/rpms/sendmail/tree > <https://git.centos.org/rpms/sendmail/tree> (no source)
What about checking in the correct branch ? For 8-stream, it's all in the c8s branch : https://git.centos.org/rpms/sendmail/tree/c8s
Since it was pointed out that the source is missing (404) and folks keep insisting on it, this is how you get it:
$ cd /tmp/ $ git clone https://git.centos.org/rpms/sendmail [...] $ git clone https://git.centos.org/centos-git-common [...] $ cd sendmail/ $ git checkout -b c8s origin/c8s Branch 'c8s' set up to track remote branch 'c8s' from 'origin'. Switched to a new branch 'c8s' $ git log commit 3aa4661d2c59bce27e291feec0d76e94d9afaa17 (HEAD -> c8s, tag: imports/c8s/sendmail-8.15.2-34.el8, origin/c8s) Author: CentOS Sources bugs@centos.org [...] $ ../centos-git-common/get_sources.sh Retrieving https://git.centos.org/sources/sendmail/c8s/5801d4b06f4e38ef228a5954a44d1763... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2155k 100 2155k 0 0 1335k 0 0:00:01 0:00:01 --:--:-- 1336k $ rpmbuild -bs SPECS/sendmail.spec setting SOURCE_DATE_EPOCH=1606780800 Wrote: /tmp/sendmail/SRPMS/sendmail-8.15.2-34.fc33.src.rpm
And as Fabian explained in the other email in this thread, all of that over https and thus far from being "not secure".
~pete
On 2/8/21 10:09 PM, Chris Drake wrote:
- Your info page here:
links to an insecure download resource: http://mirror.centos.org/centos/8-stream/ ... take a look at the number of warnings related to unsigned code ... we really need to be able to download binaries ... preferably all with working signatures, but that's a whole other problem, so TLS distro endpoints is at least an interim mitigation
Generally speaking, TLS provides three things: privacy, authentication, and integrity. For package installation, privacy isn't an essential component. There's nothing secret in the package contents. They're all publicly available. TLS can authenticate the server, but it would fail to authenticate the origin of the software itself. In short, it would authenticate the wrong thing. You'd get integrity. Nothing wrong with that, per se.
The package distribution network as it is does not provide privacy, as it is not an essential requirement. It does, however, provide authentication and integrity by signing the package files themselves. This has the advantage that the origin (CentOS) is authenticated properly, as well as remaining friendly to caching proxy servers.
Running the package distribution network over https would lose the advantage of proxy friendliness, increase the costs placed on the mirror sites (you cited LetsEncrypt certificates, but are not considering the overhead of encrypting gigabit or 10-gigabit data channels), and duplicate characteristics already provided by signing the packages.
Additionally, I feel like insisting on HTTPS for the channel overlooks one of the most important considerations in the overall design of the package distribution network, and that is that the mirrors aren't actually trusted systems. CentOS does not control them, and doesn't audit them for correctness. If a mirror is compromised, and a bad package is served from one, HTTPS would do absolutely nothing to protect end users. HTTPS would imply trust where none should be inferred. The system must work over untrusted channels, which makes HTTPS an unnecessary and costly overhead.
I'm a little confused at your last point about working signatures. You seem to understand that TLS wouldn't be adequate to the purpose of software signing, and it sounds like you're suggesting that the software isn't signed now.
-
Chris Drake -
Fabian Arrotin -
Gordon Messmer -
Jim Perrin -
Julien Pivotto -
Leon Fauster -
Manuel Wolfshant -
Matthias Runge -
Peter Meier -
plageat＠tut.by -
Rich Bowen