Hi,
Is it on purpose that under AppStream, BaseOS, PowerTools (but not under extras, fasttrack and centosplusplus) there is an os and a kickstart directory for each architecture? On a quick glance the directory layout and contents seem rather identical.
Asking as "we" do a local mirror of releases and started wondering about the amount of disk space 8 takes.
Kaj
On 26/09/2019 10:06, Kaj Niemi wrote:
Hi,
Is it on purpose that under AppStream, BaseOS, PowerTools (but not under extras, fasttrack and centosplusplus) there is an os and a kickstart directory for each architecture? On a quick glance the directory layout and contents seem rather identical.
Asking as “we” do a local mirror of releases and started wondering about the amount of disk space 8 takes.
Hi,
kickstart is a snapshot of os at GA/release time. So that permits people to deploy with exactly same content without having a moving target, as BaseOS/AppStream will be including updates as they land.
WRT disk space, hopefully, as explained on how to mirror correctly (https://wiki.centos.org/HowTos/CreatePublicMirrors) , you use -H for rsync, as we heavily use hardlinks in our trees so that it reduces space and also used bandwidth to sync mirror content
On 26/09/2019 10:06, Kaj Niemi wrote:
Hi,
Is it on purpose that under AppStream, BaseOS, PowerTools (but not under extras, fasttrack and centosplusplus) there is an os and a kickstart directory for each architecture? On a quick glance the directory layout and contents seem rather identical.
Asking as “we” do a local mirror of releases and started wondering about the amount of disk space 8 takes.
Hi,
kickstart is a snapshot of os at GA/release time. So that permits people to deploy with exactly same content without having a moving target, as BaseOS/AppStream will be including updates as they land.
WRT disk space, hopefully, as explained on how to mirror correctly (https://wiki.centos.org/HowTos/CreatePublicMirrors) , you use -H for rsync, as we heavily use hardlinks in our trees so that it reduces space and also used bandwidth to sync mirror content
Hi,
I was just wondering why it wasn't implemented with symlinks? They have the advantage that they can be recognized easily IMHO. You could then go a step further and even create an updates/ directory with only symlinks to the corresponding files. It would allow to still have an updates/ directory which some of us are missing now.
Regards, Simon
On Thu, 26 Sep 2019 at 05:29, Simon Matter via CentOS-devel centos-devel@centos.org wrote:
On 26/09/2019 10:06, Kaj Niemi wrote:
Hi,
Is it on purpose that under AppStream, BaseOS, PowerTools (but not under extras, fasttrack and centosplusplus) there is an os and a kickstart directory for each architecture? On a quick glance the directory layout and contents seem rather identical.
Asking as “we” do a local mirror of releases and started wondering about the amount of disk space 8 takes.
Hi,
kickstart is a snapshot of os at GA/release time. So that permits people to deploy with exactly same content without having a moving target, as BaseOS/AppStream will be including updates as they land.
WRT disk space, hopefully, as explained on how to mirror correctly (https://wiki.centos.org/HowTos/CreatePublicMirrors) , you use -H for rsync, as we heavily use hardlinks in our trees so that it reduces space and also used bandwidth to sync mirror content
Hi,
I was just wondering why it wasn't implemented with symlinks? They have the advantage that they can be recognized easily IMHO. You could then go a step further and even create an updates/ directory with only symlinks to the corresponding files. It would allow to still have an updates/ directory which some of us are missing now.
From dealing with mirroring issues a long time, large number of
symlinks in a directory have broken various filesystems that various systems use. You end up with sites with missing symlinks, broken symlinks or other anomalies. If you hardlink the mirror might download the file twice but it will be there. If you symlink hundreds of files, you are debugging client issues to a mirror you don't control and you have to make your check script even more complicated to confirm that things are there versus just look to be there.
It isn't that it can't be done, it just means extra work in multiple places to make it work consistently.
Regards, Simon
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On Thu, 26 Sep 2019 at 05:29, Simon Matter via CentOS-devel centos-devel@centos.org wrote:
On 26/09/2019 10:06, Kaj Niemi wrote:
Hi,
Is it on purpose that under AppStream, BaseOS, PowerTools (but not
under
extras, fasttrack and centosplusplus) there is an os and a kickstart directory for each architecture? On a quick glance the directory
layout
and contents seem rather identical.
Asking as “we” do a local mirror of releases and started wondering
about
the amount of disk space 8 takes.
Hi,
kickstart is a snapshot of os at GA/release time. So that permits people to deploy with exactly same content without having a moving target, as BaseOS/AppStream will be including updates
as
they land.
WRT disk space, hopefully, as explained on how to mirror correctly (https://wiki.centos.org/HowTos/CreatePublicMirrors) , you use -H for rsync, as we heavily use hardlinks in our trees so that it reduces
space
and also used bandwidth to sync mirror content
Hi,
I was just wondering why it wasn't implemented with symlinks? They have the advantage that they can be recognized easily IMHO. You could then go a step further and even create an updates/ directory with only symlinks to the corresponding files. It would allow to still have an updates/ directory which some of us are missing now.
From dealing with mirroring issues a long time, large number of symlinks in a directory have broken various filesystems that various systems use. You end up with sites with missing symlinks, broken symlinks or other anomalies. If you hardlink the mirror might download the file twice but it will be there. If you symlink hundreds of files, you are debugging client issues to a mirror you don't control and you have to make your check script even more complicated to confirm that things are there versus just look to be there.
Interesting, I never saw issues with symlinks. Was it a problem with filesystems or a problem with the sync tool?
A bit OT here but: I do have problems syncing large filesystems which contain large number of hardlinks using rsync. I get the following error and don't know how to go on. That's with rsync -aH:
building file list ... ERROR: out of memory in hashtable_node [sender] rsync error: error allocating core memory buffers (code 22) at util.c(117) [sender=3.0.6]
I've built latest rsync to try but it fails the same way. Is the only way to fix this to add more memory to the systems?
Thanks, Simon
Hi Fabian,
Thanks for the quick response. Turns out that we had never used "-H" when rsyncing previous releases either, works now... :>
Just to be clear, with the structure currently being:
8.0.1905/AppStream/[arch]/kickstart/ 8.0.1905/AppStream/[arch]/os/ 8.0.1905/BaseOS/[arch]/kickstart/ 8.0.1905/BaseOS/[arch]/os/ [ etc .. ]
- kickstart/ will never be updated while os/ is constantly updated? Thus, os contains whatever updates within the current release? - So is it fair to say that the old updates directory will not exist at all in CentOS 8 like it did in previous versions? - Is there a particular reason for the fasttrack directory, then?
What will happen when there is a new RHEL8 release and a CentOS release is created (I invented the year/month in the example) - will there be a new top-level directory like below:
8.1.1912/AppStream/[arch]/kickstart/ 8.1.1912/AppStream/[arch]/os/ 8.1.1912/BaseOS/[arch]/kickstart/ 8.1.1912/BaseOS/[arch]/os/ [ etc .. ]
Kaj
-----Original Message----- From: CentOS-devel centos-devel-bounces@centos.org On Behalf Of Fabian Arrotin Sent: 26 September, 2019 12:08 To: centos-devel@centos.org Subject: Re: [CentOS-devel] os/ and kickstart/ seem identical on Centos 8?
On 26/09/2019 10:06, Kaj Niemi wrote:
Hi,
Is it on purpose that under AppStream, BaseOS, PowerTools (but not under extras, fasttrack and centosplusplus) there is an os and a kickstart directory for each architecture? On a quick glance the directory layout and contents seem rather identical.
Asking as "we" do a local mirror of releases and started wondering about the amount of disk space 8 takes.
Hi,
kickstart is a snapshot of os at GA/release time. So that permits people to deploy with exactly same content without having a moving target, as BaseOS/AppStream will be including updates as they land.
WRT disk space, hopefully, as explained on how to mirror correctly (https://wiki.centos.org/HowTos/CreatePublicMirrors) , you use -H for rsync, as we heavily use hardlinks in our trees so that it reduces space and also used bandwidth to sync mirror content
Use rsync with the hardlink option to sync the content from upstream and take less disk space,
Sent from my iPhone
On Sep 26, 2019, at 4:06 AM, Kaj Niemi kajtzu@a51.org wrote:
Hi,
Is it on purpose that under AppStream, BaseOS, PowerTools (but not under extras, fasttrack and centosplusplus) there is an os and a kickstart directory for each architecture? On a quick glance the directory layout and contents seem rather identical.
Asking as “we” do a local mirror of releases and started wondering about the amount of disk space 8 takes.
Kaj
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
-
Fabian Arrotin -
Kaj Niemi -
Nico Kadel-Garcia -
Simon Matter -
Stephen John Smoogen