Hello, My environment is using CentOS 4.2 on all workstations and I would like to setup a local mirror just for the updates. I only x86, x86_64 and ia64 architectures. How can I setup a rsync mirror of the updates folder for only these architectures? I am very new to rsync, so please apologize if this is a very ignorant question.
Thank you, Vijay Avarachen
-- "Knowledge is the only wealth that grows as you spend it, and diminishes as you save it." -- ancient Sanskrit saying
Stephen, Thank you very much for your help. It works pretty good :-) but has a few glitches. I read the rsync man pages and according to it the exclusion list contains files and folder name patterns to be excluded. The exclusion works but it still creates symlinks :-( $ du -s -h * 0 2 0 3 0 3.1 0 3.3 0 3.4 0 3.5 0 4 439M 4.2 0 RPM-GPG-KEY-CentOS-2 4.0K RPM-GPG-KEY-centos4 0 RPM-GPG-KEY-CentOS-4 4.0K TIME 4.0K timestamp.txt
Is there a way to stop it from creating these symlinks that dont' point to anything. Here is my exclusion list: /2.1/ /2/ /3.1/ /3.3/ /3.4/ /3.5/ /3.6/ /3/ /4/ /4.0/ /4.1/ /4.3beta/ /HEADER.images/ /RPM-GPG-KEY-CentOS-2/ /RPM-GPG-KEY-CentOS-4/ /build/ /graphics/ /HEADER.html/ /RPM-GPG-KEY-CentOS-3/ /RPM-GPG-KEY-CentOS-4/
#4.2 excludes /4.2/Readme.txt /4.2/addons/ /4.2/apt/ /4.2/centosplus/ /4.2/contrib/ /4.2/csgfs/ /4.2/docs/ /4.2/extras/ /4.2/isos/ /4.2/os/ /4.2/testing/ #Only get i386, ia64, x86_64 /4.2/updates/SRPMS/ /4.2/updates/alpha/ /4.2/updates/ppc/ /4.2/updates/s390/ /4.2/updates/s390x/
Thanks, Vijay Avarachen
On 1/5/06, Stephen Weyland stephen.weyland@mercuryblue.com.au wrote:
Vijay Avarachen wrote:
Hello, My environment is using CentOS 4.2 on all workstations and I would like to setup a local mirror just for the updates. I only x86, x86_64 and ia64 architectures. How can I setup a rsync mirror of the updates folder for only these architectures? I am very new to rsync, so please apologize if this is a very ignorant question.
I just finished figuring this out and documenting, so here you go. You can probably fine tune it as this is my first attempt.
### How to use rsync to create a local mirror of the insallation files for CentOS, of course this could be used for anything else with modifications. ### We are going to create 2 files and edit a 3rd ### 1- Create a directory to store some files in. ### 2- A script to run rsync --- "update.sh" ### 3- An exclusion list so we don't download a bunch of stuff we don't want --- "rsync-exclude.list" ### 4- We need to schedule the update to run by modifying --- /etc/crontab, (crontab -e would be better)
1- Create a directory to store the files "update.sh" and " rsync-exclude.list" mkdir /opt/mirror
2- Create "update.sh" script to run rsync # See a list of mirrors at http://www.centos.org/modules/tinycontent/index.php?id=13 # --delete means delete files locally that no longer exist on the mirror vi /opt/mirror/update.sh rsync -aqzH --exclude-from=/opt/mirror/rsync-exclude.list --delete rsync.planetmirror.com::centos /var/ftp/pub/centos/ # make it executable chmod 700 /opt/mirror/update.sh
3- Create an exclude list vi /opt/mirror/rsync-exclude.list # add a list of the files you want to be excluded in the rsync process # files with a slash at the beginning are referenced to the root of the rsync directory you connected to. # in this case rsync.planetmirror.com/centos (the ::centos above is the /centos here) # files with a trailing slash indicate any directory with that name anywhere. e.g "apt/" will skip any directory in any sub-tree named apt
/2.1/ /2/ /3.1/ /3.3/ /3.4/ /3.5/ /3.6/ /3/ # The following need to be excluded from /4/ apt/ docs/ isos/ # The following need to be excluded from all directories they are present in, like /4/os/ and /4/updates/ ... We only want the i386 directory. SRPMS/ alpha/ ia64/ ppc/ s390/ s390x/ x86_64/
4- Add a line to /etc/crontab to run the job once a day or once a week or whenever, the following shows run on the 1st minute, 1st hour, Sunday
1 1 * * sun root /opt/mirror/update.sh
Thank you, Vijay Avarachen
-- "Knowledge is the only wealth that grows as you spend it, and diminishes as you save it." -- ancient Sanskrit saying
CentOS-mirror mailing list CentOS-mirror@centos.orghttp://lists.centos.org/mailman/listinfo/centos-mirror
-- Stephen Weyland
CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
-- "Knowledge is the only wealth that grows as you spend it, and diminishes as you save it." -- ancient Sanskrit saying
Ah! Thank God there is a inclusion option in exclusion list :-) http://gentoo-wiki.com/TIP_Exclude_categories_from_emerge_sync
#INCLUSIONS #Only get i386, ia64, x86_64 + updates/i386** + updates/ia64** + updates/x86_64**
#EXCLUSIONS Readme.txt addons/ apt/ centosplus/ contrib/ csgfs/ docs/ extras/ isos/ os/ testing/ SRPMS/ alpha/ ppc/ s390/ s390x/
Thanks, Vijay Avarachen
On 1/5/06, Vijay Avarachen vavarachen@gmail.com wrote:
Stephen, Thank you very much for your help. It works pretty good :-) but has a few glitches. I read the rsync man pages and according to it the exclusion list contains files and folder name patterns to be excluded. The exclusion works but it still creates symlinks :-( $ du -s -h * 0 2 0 3 0 3.1 0 3.3 0 3.4 0 3.5 0 4 439M 4.2 0 RPM-GPG-KEY-CentOS-2 4.0K RPM-GPG-KEY-centos4 0 RPM-GPG-KEY-CentOS-4 4.0K TIME 4.0K timestamp.txt
Is there a way to stop it from creating these symlinks that dont' point to anything. Here is my exclusion list: /2.1/ /2/ /3.1/ /3.3/ /3.4/ /3.5/ /3.6/ /3/ /4/ /4.0/ /4.1/ /4.3beta/ /HEADER.images/ /RPM-GPG-KEY-CentOS-2/ /RPM-GPG-KEY-CentOS-4/ /build/ /graphics/ /HEADER.html/ /RPM-GPG-KEY-CentOS-3/ /RPM-GPG-KEY-CentOS-4/
#4.2 excludes /4.2/Readme.txt /4.2/addons/ /4.2/apt/ /4.2/centosplus/ /4.2/contrib/ /4.2/csgfs/ /4.2/docs/ /4.2/extras/ /4.2/isos/ /4.2/os/ /4.2/testing/ #Only get i386, ia64, x86_64 /4.2/updates/SRPMS/ /4.2/updates/alpha/ /4.2/updates/ppc/ /4.2/updates/s390/ /4.2/updates/s390x/
Thanks, Vijay Avarachen
On 1/5/06, Stephen Weyland stephen.weyland@mercuryblue.com.au wrote:
Vijay Avarachen wrote:
Hello, My environment is using CentOS 4.2 on all workstations and I would like to setup a local mirror just for the updates. I only x86, x86_64 and ia64 architectures. How can I setup a rsync mirror of the updates folder for only these architectures? I am very new to rsync, so please apologize if this is a very ignorant question.
I just finished figuring this out and documenting, so here you go. You can probably fine tune it as this is my first attempt.
### How to use rsync to create a local mirror of the insallation files for CentOS, of course this could be used for anything else with modifications. ### We are going to create 2 files and edit a 3rd ### 1- Create a directory to store some files in. ### 2- A script to run rsync --- "update.sh" ### 3- An exclusion list so we don't download a bunch of stuff we don't want --- "rsync-exclude.list" ### 4- We need to schedule the update to run by modifying --- /etc/crontab, (crontab -e would be better)
1- Create a directory to store the files "update.sh" and " rsync-exclude.list" mkdir /opt/mirror
2- Create "update.sh" script to run rsync # See a list of mirrors at http://www.centos.org/modules/tinycontent/index.php?id=13 # --delete means delete files locally that no longer exist on the mirror vi /opt/mirror/update.sh rsync -aqzH --exclude-from=/opt/mirror/rsync-exclude.list --delete rsync.planetmirror.com::centos /var/ftp/pub/centos/ # make it executable chmod 700 /opt/mirror/update.sh
3- Create an exclude list vi /opt/mirror/rsync-exclude.list # add a list of the files you want to be excluded in the rsync process # files with a slash at the beginning are referenced to the root of the rsync directory you connected to. # in this case rsync.planetmirror.com/centos (the ::centos above is the /centos here) # files with a trailing slash indicate any directory with that name anywhere. e.g "apt/" will skip any directory in any sub-tree named apt
/2.1/ /2/ /3.1/ /3.3/ /3.4/ /3.5/ /3.6/ /3/ # The following need to be excluded from /4/ apt/ docs/ isos/ # The following need to be excluded from all directories they are present in, like /4/os/ and /4/updates/ ... We only want the i386 directory. SRPMS/ alpha/ ia64/ ppc/ s390/ s390x/ x86_64/
4- Add a line to /etc/crontab to run the job once a day or once a week or whenever, the following shows run on the 1st minute, 1st hour, Sunday
1 1 * * sun root /opt/mirror/update.sh
Thank you, Vijay Avarachen
-- "Knowledge is the only wealth that grows as you spend it, and diminishes as you save it." -- ancient Sanskrit saying
CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
-- Stephen Weyland
CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
-- "Knowledge is the only wealth that grows as you spend it, and diminishes as you save it." -- ancient Sanskrit saying
-- "Knowledge is the only wealth that grows as you spend it, and diminishes as you save it." -- ancient Sanskrit saying
Stephen, Actually I didn't have to do any of that...it was matter of simply changing the rsync root. Here is my script:
#!/bin/sh EXCLUSION="/etc/rsync-exclude.list" RSYNC_HOST="rsync.planetmirror.com" RSYNC_ROOT="centos/4.2/" DESTINATION="/home/ftp/pub/Linux/CentOS/4.2/" /usr/bin/rsync -aqzH --exclude-from=$EXCLUSION --delete $RSYNC_HOST::$RSYNC_ROOT $DESTINATION /bin/chown -R vijay.avarachen:spduslisadmins $DESTINATION
Cheers! Vijay Avarachen
On 1/5/06, Stephen Weyland stephen.weyland@mercuryblue.com.au wrote:
Vijay Avarachen wrote:
Stephen, Thank you very much for your help. It works pretty good :-) but has a few glitches. I read the rsync man pages and according to it the exclusion list contains files and folder name patterns to be excluded. The exclusion works but it still creates symlinks :-( $ du -s -h * 0 2 0 3 0 3.1 0 3.3 0 3.4 0 3.5 0 4 439M 4.2 0 RPM-GPG-KEY-CentOS-2 4.0K RPM-GPG-KEY-centos4 0 RPM-GPG-KEY-CentOS-4 4.0K TIME 4.0K timestamp.txt
Is there a way to stop it from creating these symlinks that dont' point to anything. Here is my exclusion list:
you might consider the following "For normal users, you should use /4/ and not /4.0/ in your path. Please see this FAQ concerning the CentOS release scheme:" http://www.centos.org/modules/smartfaq/faq.php?faqid=34 symlinks need to be included for this reason, i.e /4/ points to /4.2/ . The other dead symlinks are not hurting anything.
/2.1/ /2/ /3.1/ /3.3/ /3.4/ /3.5/ /3.6/ /3/ /4/
as above on why /4/ should not be excluded.
/4.0/ /4.1/ /4.3beta/ /HEADER.images/ /RPM-GPG-KEY-CentOS-2/ /RPM-GPG-KEY-CentOS-4/ /build/ /graphics/ /HEADER.html/ /RPM-GPG-KEY-CentOS-3/ /RPM-GPG-KEY-CentOS-4/
#4.2 excludes /4.2/Readme.txt /4.2/addons/ /4.2/apt/ /4.2/centosplus/ /4.2/contrib/ /4.2/csgfs/ /4.2/docs/ /4.2/extras/ /4.2/isos/ /4.2/os/ /4.2/testing/ #Only get i386, ia64, x86_64 /4.2/updates/SRPMS/ /4.2/updates/alpha/ /4.2/updates/ppc/ /4.2/updates/s390/ /4.2/updates/s390x/
i.e. /4/ is a symlink to the latest release, in this case 4.2/ which is why mine was configured as it was
also if you use the following method in your exclusion list addons/ apt/ etc, you will exclude that dir from all locations in the tree
Thanks, Vijay Avarachen
On 1/5/06, Stephen Weyland stephen.weyland@mercuryblue.com.au wrote:
Vijay Avarachen wrote:
Hello, My environment is using CentOS 4.2 on all workstations and I would like to setup a local mirror just for the updates. I only x86, x86_64 and ia64 architectures. How can I setup a rsync mirror of the updates folder for only these architectures? I am very new to rsync, so please apologize if this is a very ignorant question.
I just finished figuring this out and documenting, so here you go. You can probably fine tune it as this is my first attempt.
### How to use rsync to create a local mirror of the insallation files for CentOS, of course this could be used for anything else with modifications. ### We are going to create 2 files and edit a 3rd ### 1- Create a directory to store some files in. ### 2- A script to run rsync --- "update.sh" ### 3- An exclusion list so we don't download a bunch of stuff we don't want --- "rsync-exclude.list" ### 4- We need to schedule the update to run by modifying --- /etc/crontab, (crontab -e would be better)
1- Create a directory to store the files "update.sh" and " rsync-exclude.list" mkdir /opt/mirror
2- Create "update.sh" script to run rsync # See a list of mirrors at http://www.centos.org/modules/tinycontent/index.php?id=13 # --delete means delete files locally that no longer exist on the mirror vi /opt/mirror/update.sh rsync -aqzH --exclude-from=/opt/mirror/rsync-exclude.list --delete rsync.planetmirror.com::centos /var/ftp/pub/centos/ # make it executable chmod 700 /opt/mirror/update.sh
3- Create an exclude list vi /opt/mirror/rsync-exclude.list # add a list of the files you want to be excluded in the rsync process # files with a slash at the beginning are referenced to the root of the rsync directory you connected to. # in this case rsync.planetmirror.com/centos (the ::centos above is the /centos here) # files with a trailing slash indicate any directory with that name anywhere. e.g "apt/" will skip any directory in any sub-tree named apt
/2.1/ /2/ /3.1/ /3.3/ /3.4/ /3.5/ /3.6/ /3/ # The following need to be excluded from /4/ apt/ docs/ isos/ # The following need to be excluded from all directories they are present in, like /4/os/ and /4/updates/ ... We only want the i386 directory. SRPMS/ alpha/ ia64/ ppc/ s390/ s390x/ x86_64/
4- Add a line to /etc/crontab to run the job once a day or once a week or whenever, the following shows run on the 1st minute, 1st hour, Sunday
1 1 * * sun root /opt/mirror/update.sh
Thank you, Vijay Avarachen
-- "Knowledge is the only wealth that grows as you spend it, and diminishes as you save it." -- ancient Sanskrit saying
CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
-- Stephen Weyland
CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
-- "Knowledge is the only wealth that grows as you spend it, and diminishes as you save it." -- ancient Sanskrit saying
CentOS-mirror mailing list CentOS-mirror@centos.orghttp://lists.centos.org/mailman/listinfo/centos-mirror
-- Thanks
Stephen Weyland Mercury Blue 8398-1210
CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
-- "Knowledge is the only wealth that grows as you spend it, and diminishes as you save it." -- ancient Sanskrit saying
Vijay Avarachen wrote:
Stephen, Actually I didn't have to do any of that...it was matter of simply changing the rsync root. Here is my script:
#!/bin/sh EXCLUSION="/etc/rsync-exclude.list" RSYNC_HOST="rsync.planetmirror.com http://rsync.planetmirror.com" RSYNC_ROOT="centos/4.2/" DESTINATION="/home/ftp/pub/Linux/CentOS/4.2/" /usr/bin/rsync -aqzH --exclude-from=$EXCLUSION --delete $RSYNC_HOST::$RSYNC_ROOT $DESTINATION /bin/chown -R vijay.avarachen:spduslisadmins $DESTINATION
The point that Stephen was trying to make is : with your rsync_root set to centos/4.2/ - when the next update is released, you will end up with a blank mirror.
by setting it to centos/4/ you stay with the most recently released pkgs.
Ah! Thanks Karnbir. I will make the necessary changes to my script. Also now it explains why all those symlinks existed.
On 1/6/06, Karanbir Singh mail-lists@karan.org wrote:
Vijay Avarachen wrote:
Stephen, Actually I didn't have to do any of that...it was matter of simply changing the rsync root. Here is my script:
#!/bin/sh EXCLUSION="/etc/rsync-exclude.list" RSYNC_HOST="rsync.planetmirror.com http://rsync.planetmirror.com" RSYNC_ROOT="centos/4.2/" DESTINATION="/home/ftp/pub/Linux/CentOS/4.2/" /usr/bin/rsync -aqzH --exclude-from=$EXCLUSION --delete $RSYNC_HOST::$RSYNC_ROOT $DESTINATION /bin/chown -R vijay.avarachen:spduslisadmins $DESTINATION
The point that Stephen was trying to make is : with your rsync_root set to centos/4.2/ - when the next update is released, you will end up with a blank mirror.
by setting it to centos/4/ you stay with the most recently released pkgs.
-- Karanbir Singh : http://www.karan.org/ : 2522219@icq _______________________________________________ CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
-- "Knowledge is the only wealth that grows as you spend it, and diminishes as you save it." -- ancient Sanskrit saying
-
Karanbir Singh -
Stephen Weyland -
Vijay Avarachen