I just had a merge request[0] pulled into the osinfo-db project that adds CentOS Stream 9 details. This is used by tools like GNOME Boxes to discover and download ISOs as well as configure generic VM information. The MR stems from a user reporting Boxes wasn't listing Stream 9 as an available option.
As part of the process I had to provide direct URLs to the Stream 9 ISOs. I would have preferred to have used the MirrorManager links, such as:
https://mirrors.centos.org/mirrorlist?path=/9-stream/BaseOS/x86_64/iso/CentO...
But these URLs don't redirect properly with tools like wget or curl, instead returning a list of available mirror links to download from. It works just fine in a browser, maybe I don't know the proper flags to pass to make this work as expected. Currently the base URL used is from the mirror.stream.centos.org site.
Is there a way to make the MM links work, or is pointing to the CentOS mirror alright for this?
[0] https://gitlab.com/libosinfo/osinfo-db/-/merge_requests/366
-- Mike Rochefort
On 11/10/21 18:40, Mike Rochefort wrote:
I just had a merge request[0] pulled into the osinfo-db project that adds CentOS Stream 9 details. This is used by tools like GNOME Boxes to discover and download ISOs as well as configure generic VM information. The MR stems from a user reporting Boxes wasn't listing Stream 9 as an available option.
As part of the process I had to provide direct URLs to the Stream 9 ISOs. I would have preferred to have used the MirrorManager links, such as:
https://mirrors.centos.org/mirrorlist?path=/9-stream/BaseOS/x86_64/iso/CentO...
But these URLs don't redirect properly with tools like wget or curl, instead returning a list of available mirror links to download from. It works just fine in a browser, maybe I don't know the proper flags to pass to make this work as expected. Currently the base URL used is from the mirror.stream.centos.org site.
I guess using the `-L` option for curl should do the trick:
``` $ curl -JLO "https://mirrors.centos.org/mirrorlist?path=/9-stream/BaseOS/x86_64/iso/CentO..." ```
as for wget, it follows redirects automagically, but you can add `--content-disposition` into the mix to get the "correct" filename:
``` $ wget --content-disposition "https://mirrors.centos.org/mirrorlist?path=/9-stream/BaseOS/x86_64/iso/CentO..." --2021-11-10 18:57:03-- https://mirrors.centos.org/mirrorlist?path=/9-stream/BaseOS/x86_64/iso/CentO... Resolving mirrors.centos.org (mirrors.centos.org)... 18.185.136.17, 18.159.254.57, 18.133.140.134, ... Connecting to mirrors.centos.org (mirrors.centos.org)|18.185.136.17|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://mirror.netsite.dk/centos-stream/9-stream/BaseOS/x86_64/iso/CentOS-St... [following] --2021-11-10 18:57:03-- https://mirror.netsite.dk/centos-stream/9-stream/BaseOS/x86_64/iso/CentOS-St... Resolving mirror.netsite.dk (mirror.netsite.dk)... 185.224.16.132, 2a0c:cc00:0:1604::1 Connecting to mirror.netsite.dk (mirror.netsite.dk)|185.224.16.132|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 8299479040 (7.7G) [application/octet-stream] Saving to: ‘CentOS-Stream-9-latest-x86_64-dvd1.iso’
CentOS-Stream-9-latest-x86_64-dvd1.iso 0%[ ...
```
Is there a way to make the MM links work, or is pointing to the CentOS mirror alright for this?
[0] https://gitlab.com/libosinfo/osinfo-db/-/merge_requests/366
-- Mike Rochefort
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
It would seem the missing aspect is actually that I'm not wrapping the URL in quotes.
I can present this to the osinfo group, not sure if they'll accept it as is, though.
-- Mike Rochefort
On Wed, 10 Nov 2021 at 15:32, Mike Rochefort mroche@redhat.com wrote:
It would seem the missing aspect is actually that I'm not wrapping the URL in quotes.
I can present this to the osinfo group, not sure if they'll accept it as is, though.
How does it work with the mirrormanager for Fedora Project Linux?
On Wed, Nov 10, 2021 at 3:53 PM Stephen John Smoogen smooge@gmail.com wrote:
How does it work with the mirrormanager for Fedora Project Linux?
The correlating Fedora files[0] use the following link: https://download.fedoraproject.org/pub/fedora/linux/releases/35/Server/x86_6...
[0] https://gitlab.com/libosinfo/osinfo-db/-/blob/master/data/os/fedoraproject.o...
Doesn't seem like "curl -J" wants to drop the link parameters: "CentOS-Stream-9-latest-x86_64-boot.iso&redirect=1&protocol=https"
On Wed, Nov 10, 2021 at 4:24 PM Mike Rochefort mroche@redhat.com wrote:
Doesn't seem like "curl -J" wants to drop the link parameters: "CentOS-Stream-9-latest-x86_64-boot.iso&redirect=1&protocol=https"
Reordering params solves this.
On Wed, Nov 10, 2021 at 04:24:04PM -0500, Mike Rochefort wrote:
On Wed, Nov 10, 2021 at 3:53 PM Stephen John Smoogen smooge@gmail.com wrote:
How does it work with the mirrormanager for Fedora Project Linux?
The correlating Fedora files[0] use the following link: https://download.fedoraproject.org/pub/fedora/linux/releases/35/Server/x86_6...
Which, btw, returns a 302 redirect to a specific mirror.
On Thu, Nov 18, 2021 at 6:13 PM Matthew Miller mattdm@mattdm.org wrote:
Which, btw, returns a 302 redirect to a specific mirror.
More info was added to the ticket post-merge, but I provided a reordered param MM link (redirect..protocol..path) for the Stream objects and according to another user it worked in their testing. I still haven't verified it myself with GNOME Boxes, but once I do I'll submit another PR to correct this in osinfo-db. Should provide a better overall experience for anyone using the database for ISO retrieval.
One thing I'm not sure of is which protocol to use, though. I'd personally choose HTTPS.
On Thu, Nov 18, 2021 at 06:25:59PM -0500, Mike Rochefort wrote:
On Thu, Nov 18, 2021 at 6:13 PM Matthew Miller mattdm@mattdm.org wrote:
Which, btw, returns a 302 redirect to a specific mirror.
More info was added to the ticket post-merge, but I provided a reordered param MM link (redirect..protocol..path) for the Stream objects and according to another user it worked in their testing. I
I was curious about the reordering part and the reason reordering helps is that curl tries to guess the local file name and if path is at the end curl guesses it right. The reordering does not change the content you get from the server in any way.
still haven't verified it myself with GNOME Boxes, but once I do I'll submit another PR to correct this in osinfo-db. Should provide a better overall experience for anyone using the database for ISO retrieval.
One thing I'm not sure of is which protocol to use, though. I'd personally choose HTTPS.
You will get http and https mirrors if you do not specify the protocol option. Using protocol you can limit it to a certain protocol.
Adrian
On Fri, Nov 19, 2021 at 2:49 AM Adrian Reber adrian@lisas.de wrote:
I was curious about the reordering part and the reason reordering helps is that curl tries to guess the local file name and if path is at the end curl guesses it right. The reordering does not change the content you get from the server in any way.
Yes, I would rather (from a UX point of view) have things match with a clean output vs having a mangled filename. GNOME Boxes itself appends a ~ character to its own downloads, so it's not the biggest issue.
On Fri, Nov 19, 2021 at 2:49 AM Adrian Reber adrian@lisas.de wrote:
You will get http and https mirrors if you do not specify the protocol option. Using protocol you can limit it to a certain protocol.
Yes. The question was more about what the Project believes should be the default protocol here for use in osinfo-db. At the moment I have it configured to serve the mirrorlist over HTTPS and select only HTTPS mirrors. However, I know HTTPS can be an issue in some environments, such as with corporate firewalls that like to MITM traffic (a usual suspect for things like repos/registries in tooling). Whether that's an impact in a download like this, I'm not sure as I will not claim to be a networking or security guru. So from a layman perspective I'm trying to figure out whether this should cover the broadest use case or the most secure (again, this will define things for any project using osinfo-db).
# Using 'both' to signify the absence of param or combined use of http,https values. http{,s}://mirrors.centos.org/mirrorlist?protocol={both,http,https}
https://gitlab.com/libosinfo/osinfo-db/-/merge_requests/376
-
Adrian Reber -
František Šumšal -
Matthew Miller -
Mike Rochefort -
Stephen John Smoogen