Re: [CentOS-devel] Interested in IA64 build.

On Thursday, October 04, 2012 11:45:14 AM Lamar Owen wrote:

Ok, well, I've learned a lot since last Friday. Especially as it pertains to circular dependencies and how to properly (hopefully) step from one release up to another. ...

And after I wrote that, I realized that I hadn't documented how I did that. So, in this admittedly longish e-mail, I'll attempt to do that.

Also, I failed to note that the precise build chain that I ended up following is likely not the same as the chain used building the supported CentOS builds, since I started with a different set of packages and did not build intervening updates. And my chain was compiled semi-automatically using iterations of invocations of smock.pl with possibly a different mix of repos each time, and thus the build I have is likely not completely reproducible from the c5-wip tree Karanbir posted a while back.

And, well, honestly, I'm not as concerned with reproducibility as I am with getting a self-hosting 5.8 build as an end result, from which I will then build the 5.8 updates and/or the 5.9 tree once it is released; if 5.9 comes out before I get to the 5.8 updates then 5.9 comes afterwards, and I go on to 5.9 updates. The intermediate steps will be different each time through, depending upon the base from where you start, and the order in which you hand-resolve each and every one of the several hidden dependencies; knowing that going into this process I specifically did not document every step in exact detail, as it's not important, at least in my opinion. I'll give enough info below for anyone who can figure out how to get mock working and building things, along with the use of createrepe, to follow.

Here's how I'm stepping from CentOS 5.5 to 5.6. To duplicate you'll need a working binary repo, one that is complete enough to satisfy the dependencies of buildsys-build (available on dev.centos.org, and its spec is available in the mock package itself). If you are building a complete tree using smock.pl, smock.pl will figure out the buildrequires that are documented in the SRPM header and will sequence the builds accordingly. It's the undocumented, hidden, or conditional buildrequires that will bite you, and you need binaries for those packages in your repos for the buildroots. I say buildroots (plural), and not buildroot (singular), since each mock invocation that smock.pl makes will start with the base buildroot and install the various buildrequires needed into it for the specific package that particular mock invocation is building.

First, get and install mock 1.0.28 plus all dependencies from EPEL. Also get some example source RPM's and make sure rpmbuild as a normal user works. If you want an experience closer to what the CentOS build is like, then forgo mock 1.0.28 from EPEL and use the patchedf mock-0.6.13 from CentOS-Extras; my configuration won't work for you, but you'll have a closer-to-CentOS-build experience, if that's what you want. I'm more concerned about getting up to date first, and 'correct' second. So I may downgrade mock to 0.6.13 and use the CentOS-provided configs (and other info that's been posted to the mailing list over the years) to do the final 'self-hosting' build at 5.8 (or 5.9). But anyway, back to the documenting....

Add yourself as a member of the 'mock' group.

For performance, mount /var/lib/mock as a tmpfs of several GB, if you have the RAM to spare (my ia64 builder, winterstar, has 54GB, so I gave it a 26GB tmpfs on /var/lib/mock). My experience was that this trimmed a pretty substantial amount of time off of some of the lengthier builds, like the kernel.

Now, obtain smock.pl. ( part of fedora-mingw-devel) See: http://git.annexia.org/?p=fedora-mingw.git;a=tree;f=smock

Read the README, and read the script to see what it's going to do. You will need to create initial repos in the target directory, and you'll need to do the proper configuration for smock.pl to 'find itself'. I cheated and symlinked ~/public_html to my build tree in /opt/build/. Once you understand what smock.pl will, and won't, do, put it somewhere in your path (like /usr/local/bin ) and make it executable.

And, yeah, you need to understand what's going on; this is not a recipe with quantities of ingredients, exact cooking temperatures, and exact cooking times here. I can, for instance, tell you how to cook sausage gravy; but I can't give you a recipe, since the amount of grease rendered from the sausage will vary, and thus I need to say things like 'put in enough flour to barely absorb all the grease and to make a moist, not dry or wet, but moist, putty (not paste and not dough)' and ' pour in enough milk to quickly boil and 'batterize' the flour-grease mixture' or 'pour in enough milk to make the gravy look just almost too thin' : you need to understand what you're doing to follow these directions. The reason is simple: you're not going to be building things in exactly the same order as upstream, CentOS, or I did, and thus you're probably going to get stopped in your build attempt in different places than we did. I know I experienced at least one issue in my ia64 build than wasn't experienced in the i386 or x86_64 builds.

Now, the mock configuration file is the piece that drives everything. Here's the one that I'm using right now to churn the 5.6 updates for ia64. Watch for line wrap. I'm going to notate each section; my notations will be prefixed with a ----> :

++++++++++++++++++++++++ [lowen@winterstar ~]$ cat /etc/mock/centos-56-ia64.cfg #!/usr/bin/python -tt

----> The above line marks this configuration file as being a python script, ----> with all the syntax rules etc of a python script.

import os config_opts['root'] = 'centos-56-ia64' config_opts['target_arch'] = 'ia64' config_opts['clean'] = True

----> These configuration options tell mock the buildroot chroot to use, ----> the architecture for which to build, and whether to clean the chroot

config_opts['chroot_setup_cmd'] = 'install buildsys-build zip zlib-devel autoconf pkgconfig libsmbclient-devel gstreamer-plugins-good-devel dhclient'

----> This previous line, which is one line, is where hidden buildrequires: ----> injection occurs in my buildsys. CentOS does things differently; see ----> the top three entries in the changelog for the mock in CentOS-Extras ----> for the patch CentOS uses to mock 0.6.13. I'm looking at eventually ----> updating the HintsAndMacros patch for mock 1.0.28, but that's after ----> I get things built.

config_opts['plugin_conf']['ccache_enable'] = False

----> This above line is critical to make the system work properly.

----> Note the triple double quotes below. This is a multiline string, being ----> assigned to config_opts['yum.conf'] and anything inside the paired ----> triple double quotes is a yum configuration file and not a python script.

config_opts['yum.conf'] = """ [main] cachedir=/var/cache/yum debuglevel=1 logfile=/var/log/yum.log reposdir=/dev/null retries=20 obsoletes=1 gpgcheck=0 assumeyes=1

# repos

----> The first repo stanza below was the bootstrapping SLC 5.4 binaries, which ----> I don't need anymore for building the 5.6 updates.

#[slc-64bit-local-repo] #name=slc-64-local.repo #baseurl=file:///opt/build/repos/slc54 #enabled=1 #gpgcheck=0 #

----> Comment below says it all. Well, almost all. It's as built under ----> SLC 54 plus an accumulation of 5.5 packages over at least four ----> full iterations.

# CentOS 5.5 as built under SLC 5.4 [c5-os] name=c5-local.repo baseurl=file:///opt/build/repos/centos/5.5/os/ia64/ enabled=1 gpgcheck=0

----> This section below was only for testing, but I left it in place as ----> reminder to myself....

#[c5-32-bit] #name=c5-32-bit.repo #baseurl=file:///opt/build/repos/centos/5.5/os/i386/ #enabled=1 #gpgcheck=0

#newer m4 for buildsys

----> This is documented by SL CERN on their SLC5 build page.

[m4-c5] name=m4-c5.repo baseurl=file:///opt/build/buildsys/m4-c5 enabled=1 gpgcheck=0

----> The repo below is the most dynamic piece of the whole ----> puzzle, and has changed with the multiple iterations. ----> This holds the built-from-source libunwind, among others. ----> As the contents of this repo changed over time, and with ----> each iteration, the exact contents isn't important, but I ----> will say that valgrind-devel i386 lives here, even for an ia64 ----> build. I can try to reconstruct the order of packages in and ----> out of this repo, but, really, I just put the packages in here ----> that the error logs told me the build needed. Once the ----> 5.5 package of the same name built, out of this repo it ----> went.... and, of course, I ran createrepo on it for every ----> modification.....

# other things the buildsys needs, pulled from downrev [buildroot-reqs] name=buildroot-reqs.repo baseurl=file:///opt/build/buildsys/reqs enabled=1 gpgcheck=0

----> The repo below is the buildsys-build rpm, pulled from the centos ----> dev server, and basically verbatim from the mock rpm (do an ----> rpm -ql mock to find the spec file.....)

[groups] name=groups #baseurl=http://dev.centos.org/centos/buildsys/5/ baseurl=file:///opt/build/Factory-Install/groups/5/

# Change USERNAME and if necessary the distribution name and # architecture, and INSERT this file to the relevant distributions # in /etc/mock (eg. to /etc/mock/fedora-9-i386.cfg etc.)

----> The below repo is the dynamic repo that smock maintains. ----> in my instructions below I will try to consistently call it the [smock] ----> repo.

[smock] name=smock #baseurl=http://127.0.0.1/USERNAME-smock/yum/fedora-9/$basearch # If you do not want to use a http server, you need to specify a direct path to the # repository. Then comment out the other baseurl line. baseurl=file:///opt/build/public_html/smock/yum/centos-56/ia64 enabled=1 keepcache=0

""" ----> Note the closing triple double quotes; we're not in Kansas anymore

----> The below syntax is for mock 1.0.28. This syntax will not work for ----> mock 0.6.13 as distributed in CentOS-Extras. See the mock config ----> Johnny posted several months ago for the syntax for 0.6.13

config_opts['macros'] ['%_topdir'] = "/builddir/build" config_opts['macros'] ['%_rpmfilename'] = "%%{NAME}-%%{VERSION}-%%{RELEASE}.%%{ARCH}.rpm"

# Change the next two lines to reflect yourself.

config_opts['macros'] ['%packager'] = "Lamar Owen lowen@pari.edu" config_opts['macros'] ['%vendor'] = "PARI" config_opts['macros'] ['%distribution'] = "PARI Internal Altix EL5 Rebuild"

----> I initially followed the advice below, changing %dist to first be ---->"pari.lro" (which mangles the release) and then ".pari.lro" but then ----> I found that %dist must be .el5 for certain packages to ----> properly build. (gdb, for one, at least on ia64). A grep through all ----> spec files to find conditional buildrequires would be necessary to ----> find all of the packages that this macro impacts. ----> And, of course, I now have 'pari.lro' and '.pari.lro' tagged packages ----> in my 5.5 repos that will eventually be replaced when I mass ----> rebuild 5.8/9.

# please change this to reflect the Distro Tree and Repo hosting packages! config_opts['macros'] ['%dist'] = ".el5" config_opts['macros'] ['%centos_ver'] = "5"

#need this to get el5 builddeps from some packages. config_opts['macros'] ['%el5'] = "1" config_opts['macros'] ['%rhel'] = "5" config_opts['macros'] ['%_smp_mflags'] = "-j30"

----> The above line is for my 30 CPU Altix; change this to match your number ----> of CPU's and thus the desired number of parallel compiles make will do.

[lowen@winterstar ~]$ +++++++++++++++++++++++++++

Ok, now we need to test mock, and we do this by initializing the chroot:

mock -r centos-56-ia64 --init

If this fails, you have other problems. Look at the output, it will tell you what your problem is. This has to work, or nothing else will. And I can't give you a recipe that is guaranteed to work, every time; you need to understand yum and how to populate a repository properly in order to fix issues with the chroot init. I figured it out; so can you, it just takes some thought and some work, you're capable of doing that.

Now, generate a list of packages to build, and put them in a single line (xargs -n 10000 works well for this), and put it in, say, ~/smock-packages.txt.

The first two packages to build are critical, and to build them you need to temporarily comment out the [smock] repo in the yum.conf section of the mock config. Change dir to the SRPMS directory containing the new centos-release package, and issue (I'm on ia64, so I use that here): smock.pl --arch=ia64 --distro=centos-56 centos-release*src.rpm

This gets the centos-release package and the centos-release-notes package, and you need both, and they need each other, and if you put one in the [smock] repo without the other, and the [smock] repo is enabled for chroot populating, it will break the chrooted yum installs and nothing will build. So leave [smock] disabled for building these two.

Once those two build, uncomment the [smock] repo in the mock configuration. I know, I could probably just set enable=0 instead of commenting it out, but I chose to comment it out, and uncomment it back in instead.

For the build itself, I like to first see what smock.pl is going to do with a: smock.pl --dryrun --arch=ia64 --distro=centos-56 `cat ~/smock-packages.txt`

Smock will send to stderr any dependency loops prefixed with a 'tsort:' and will send the package names in build order out stdout. If you have the downrev packages that will satisfy the deploops, you don't need to worry about them. If you don't have those packages, you'll need to build or find at least one package in each deploop and put it in the [buildroot-reqs] (which is part of the reason I put it there), being sure to run createrepo every time you hand modify any repo.

Save the output somewhere, if nothing else so you can see where the build is by looking at stdout later.

I use nohup; you can use screen or dtach or whatever. My actual building command line is: nohup time smock.pl --keepgoing --arch=ia64 --distro=centos-56 \ `cat ~/smock-packages.txt` &

And I then tail -f nohup.out to watch the build.

Now, smock.pl has a buglet of sorts, and as the number of packages accumulates this buglet makes the build get progressively slower. Smock is maintaining a repository of its own; in the build loop it does a createrepo both before and after each and every package build. When a repo gets more than a few hundred packages in it, the createrepo can take longer than the package build does.

I made a modified 'smock2' that removed the automatic createrepo for the initial pass at the 5.5 packages, and any other time I knew the package set was going to be large, to save some on time, or when the buildroot seed repo contained a vast majority of the build dependencies. Or if I disabled the [smock] repo; if the [smock] repo isn't being used by mock, then why spend the time doing a createrepo on it?

Now, my 5.6 update build isn't complete, so I really don't know how well this 'stepwise' staging is going to work in practice. I should know tomorrow morning; the buid is at:

[lowen@winterstar ~]$ buildstatus56 No IA64 target: 4 No Package Found: 0 Scratch total: 8 Built binaries: 286 Built SRPMS: 86 [lowen@winterstar ~]$

Hmm, there are 8 builds in scratch; this is where failed build build, root, and state logs are kept, as well as where the currently running build, root, and state logs are put until a good build, when they get moved to the logs tree under the binary RPMS dir. So, that means 7 failed builds; 4 of those are due to having not IA64 target for the build. So 3 builds, so far, that I'll need to further investigate.

Tomorrow.

I hope this post is helpful to someone; if nothing else I'm putting it in my own archive for my own reference..... :-)