[CentOS-devel] New lookaside structure available

Wed Apr 27 13:08:32 UTC 2022
Pierre-Yves Chibon <pingou at pingoured.fr>

A while ago, we spoke about the structure of the lookaside cache used by Red Hat
and SIGs [1].
We have made progress on this, there is now two ways SIGs can upload their
tarball to the lookaside cache:
- Using the current endpoint at: https://git.centos.org/sources/upload.cgi
  This works the way it always has, you can use `lookaside_upload` from [2] to
  interact with it, as before. In other words: nothing changes there.
  For the record, this uses the structure: `baseurl/pkgname/branch/hash`.
  Example:
  https://git.centos.org/sources/kernel/c8s/0c4e10577cfd4b4f8e3d83c0406da8ab05eb775f

- Using the new endpoint at: https://git.centos.org/sources/sig_upload.cgi
  This allows any SIG member to upload to the lookaside cache using the same
  structure as the one used in CentOS Stream and Fedora:
  `baseurl/pkgname/tarball/hashtype/hash/tarball`.
  Example:
  https://sources.stream.centos.org/sources/rpms/kernel/linux-5.14.0-62.el9.tar.xz/sha512/f7aeac0fe5bf594933cd35b7ecc94ea8ddcbfedc04fa769c4da298e7bf105df116375d44711d944c748c85f61f96f6149be34c76eb37f28aa1f16359a9122abf/linux-5.14.0-62.el9.tar.xz
  You can use the `lookaside_upload_sig` script from [3] to interact with it.

So we can now either:
- not change anything
or
- use the new endpoint to use the new lookaside structure

What are the benefits of the new structure?
- It allows to use a `sources` file similar to the one used in CentOS Stream and
  Fedora, which combined with the changes to support flat dist-git layout should
  make backporting from CentOS Stream or Fedora easier.
- If two SIGs are using the same tarball, only one will be uploaded to the
  lookaside cache (so, it allows a little disk space saving on the server side).
- It no longer ties sources uploaded to a specific git branch, which if you are
  using gitlab, means you no longer have to follow the branching structure that
  git.centos.org needs due to its dual purpose.


Finally, the recent changes made to the infrastructure are all documented at:
https://sigs.centos.org/guide/

More specifically, the changes for the lookaside and git repository structures
can be found at:
https://sigs.centos.org/guide/git/
The documentation on using gitlab is at:
https://sigs.centos.org/guide/gitlab/


Happy hacking!

Pierre

[1] https://lists.centos.org/pipermail/centos-devel/2022-February/120226.html
[2] https://git.centos.org/centos-git-common/blob/master/f/lookaside_upload
[3] https://git.centos.org/centos-git-common/blob/master/f/lookaside_upload_sig