On 05/21/2010 11:05 AM, James A. Peltier wrote:

So what is it that you need? Just for someone like me to actually run it? What would be required if I wanted to maintain my own build and keep it up to date as an un-official copy?

Yes, when I did the bootstrap - I had interest from 3 people, 1 of whom then changed jobs and his workplace dropped the ia64 cluster and moved to x86_64, the second guy had 1 ia64 machine which developed a problem and they didnt think its worth fixing it. I've not heard from the third chap since. So not sure what is going on.

I'm interested in getting involved and contributing back. Of course I'm also not sure how much work is involved just yet.

There isn't really that much work involved - not nearly as much as powerpc/powerpc64 effort. If you are going to put in the effort, it might be worth doing something semi official - as long as we can retain the general build and distro policy of / for CentOS and we can get to a state were we track build packages from i386 / x86-64.

There is a build machine hosted in the UK ( its a dell 2950 with 2 ia64 1.4Ghz - couple of gigs of ram and is on a fairly good network ). So we would need to get together atleast another machine for testing purposes. I'm guessing you already have some kit in place, which might be usable ?

How much of time are you really going to put into this ? I dont mean to sound patronising, but it would to be good to work with someone who has some level of commitment. Given that I have no personal interest in ia64, and no hardware of my own that is ia64, and that rhel6 isn't going to have a ia64 tree[1]; We really dont want to be in a situation wherein we put something out, and then struggle to keep things moving along.[2]

- KB

[1]: does'nt mean we cant do one

[2]: within reason. No one expects anyone to commit on contract, 10 hrs a week for 5 years. But it should not be a case of only putting this effort in through the summer holidays, or a 'quiet phase at work' etc.

PS: try not top posting

Hi,

On 05/21/2010 12:02 PM, James A. Peltier wrote:

Alright, I'm willing to commit some time, but before diving in, as a general rough overview, what type of timeframe are we looking at to maintain a port? 5-10 hours a day, week, month?

Sounds good, Let me now go away and see where are with ia64, whats needed and what sort of time scales we can work against. I know that Russ has also done some work on ia64, so maybe once we have a ToDo list, a realtime syncup on irc would be a good way forward.

- KB

Hi,

On 05/21/2010 06:31 PM, R P Herrold wrote:

I have a couple IA64 units powered off which can move to a datacenter and go live

That will come in handy for testing.

The question I really have is -- it is unlikely that this, ppc, ppc64, and sparc [and the s390x build I have been working on privately] have general appeal that should 'clutter' the mirrors in general. An 'optional' approach on these items seems 'fairer' as to disk real estate for them

yes, I dont think we will have nearly as much traffic on these arch's to merit putting them on the main mirror system.

Probably the CentOS infrastructure team need to discuss this as to where reference 'side architecture' CentOS builds will 'live', outside the general mirroring

good point, I've added it to the list of things.

Once I have a bootstrap into CentOS 5, and then 6 as time indicates, it may well be possible to 'wire' a ground up rebuild on the current CentOS builder and testing process, and any future builds into that current process

Sounds good! There is some work that Ray has been doing as well, and I think we should have install media for centos-5 out from there in the next few days. Let me collect packages and put them on dev.c.o. But you raise an interesting point w.r.t 5 and 6 - at this stage, is it worth considering 6 to be a more productive target than 5 for arch's that will be supported through to 6 ( ie, not ia64 ) ?

The unit also 'tweets' its progress at http://twitter.com/buildmonbot

nice!

- KB

Hi,

Ok, so the IA64 build is actually quite straight forward but needs a large disk. We just need to start the builders and make sure it can churn through all packages done since 5.0 - if all is well, then things *should* just build, since they did so upstream! and the i386/x86_64 packages for CentOS built out of the same code base. Once we can get to the stage that 5.5 is at now, we can look at doing an iso set and build tree.

No idea how long this is going to take. I am presently moving the 5.0 + updates tree's into place ( the src.rpms ), once that is done we can see what sort of a buildrate we get per day and do some numbers from there on.

I don't want to overload dev.c.o with the output, so let me have a chat with Ralph and Tru and we can workout what is a good place to post this content.

Ideally, it would be good to have a Failed: report that lists the issues encountered during the build be posted online right away. So people can look at it and see if anything specific needs doing. For things that do build, we can setup a six hourly 'sweep' that moves packages into place.

More details in a day or so.

- KB

On 05/26/2010 04:08 PM, James A. Peltier wrote:

Thanks for your work on this Karanbir and team. ;)

You are welcome!

First the good news - the builds just started.

then the other news - you just became the ia64 team :)

Lets let the build run for a few hours and then we can workout what needs extracted, how and when.

- KB

Any follow up to this? Are the builds still running? Just wanting to know what needs to be done. :)

----- "Karanbir Singh" mail-lists@karan.org wrote:

| On 05/26/2010 04:08 PM, James A. Peltier wrote: | > Thanks for your work on this Karanbir and team. ;) | > | | You are welcome! | | First the good news - the builds just started. | | then the other news - you just became the ia64 team :) | | Lets let the build run for a few hours and then we can workout what | needs extracted, how and when. | | - KB | _______________________________________________ | CentOS-devel mailing list | CentOS-devel@centos.org | http://lists.centos.org/mailman/listinfo/centos-devel

That's 2PM in Vancouver, which is fine for me. I've scheduled it for then for 1 hour. I can work with that limit though. :)

----- "Karanbir Singh" mail-lists@karan.org wrote:

| On 06/05/2010 07:18 PM, James A. Peltier wrote: | > Any follow up to this? Are the builds still running? Just wanting | to know what needs to be done. :) | > | | I was looking at this a short while back, the builds are running. I | think an irc catchup is called for. I'm going to propose Wed 9th at | 2100 | UTC n #centos-devel | | - KB | _______________________________________________ | CentOS-devel mailing list | CentOS-devel@centos.org | http://lists.centos.org/mailman/listinfo/centos-devel

On Saturday, June 05, 2010 03:19:40 PM Karanbir Singh wrote:

On 06/05/2010 07:18 PM, James A. Peltier wrote:

...

Any follow up to this? Are the builds still running? Just wanting to know what needs to be done. :)

I was looking at this a short while back, the builds are running. I think an irc catchup is called for. I'm going to propose Wed 9th at 2100 UTC n #centos-devel

Ok, bringing up a two-year-old thread....

We finally got the electrical up (Mitsubishi 9900B 500KVA UPS woo hoo....) in our research building's data center, and have a 20 CPU Altix 3700 up, running, and available to do this. I know about and am pulling the 'c5-wip' tree Karanbir built a while back, and have allocated 1.4TB of disk on our Clariions here to the effort. Now, I know the last time this was brought up it didn't go very far; but I'm still interested in doing this, even if for private use here on our three Altix systems.

James, did you get very far? Karanbir, what would be the chances of a 'refresh' of the ia64 c5-wip tree to 5.8?

I can give more details via private e-mail.

On 08/28/2012 04:32 PM, Lamar Owen wrote:

We finally got the electrical up (Mitsubishi 9900B 500KVA UPS woo hoo....) in our research building's data center, and have a 20 CPU Altix 3700 up, running, and available to do this. I know about and am pulling the 'c5-wip' tree Karanbir built a while back, and have allocated 1.4TB of disk on our Clariions here to the effort. Now, I know the last time this was brought up it didn't go very far; but I'm still interested in doing this, even if for private use here on our three Altix systems.

*fear*

James, did you get very far? Karanbir, what would be the chances of a 'refresh' of the ia64 c5-wip tree to 5.8?

it should be fairly straightforward - the yum ( mx actually ) issues that i ran into way back when, are also resolved now. But, is there any interest ?

On Tuesday, August 28, 2012 06:24:02 PM Karanbir Singh wrote:

On 08/28/2012 04:32 PM, Lamar Owen wrote:

...

We finally got the electrical up (Mitsubishi 9900B 500KVA UPS woo hoo....) in our research building's data center, and have a 20 CPU Altix 3700 up, running, and available to do this. I know about and am pulling the 'c5-wip' tree Karanbir built a while back, and have allocated 1.4TB of disk on our Clariions here to the effort. Now, I know the last time this was brought up it didn't go very far; but I'm still interested in doing this, even if for private use here on our three Altix systems.

*fear*

Fear? I'm afraid you lost me there, Karanbir. In any case, I mention 'private' simply that, even if there isn't any other interest, I'm still likely to do this myself, regardless, just not as a 'signed and sealed' CentOS distribution, just a private rebuild since I want to fully utilize this donated box for science, and I don't have a cluster of x86_64 boxes available. If there's other interest, then, well that's a different story and would need a 'signed and sealed' CentOS. But my needs are more modest. I know when I e-mail the CERN SLC-ia64 maintainer a while back he said that they had stopped supporting SLC5.x on ia64 (the last DVD ISO I found of SLC5 wouldn't boot, unfortunately).

...

James, did you get very far? Karanbir, what would be the chances of a 'refresh' of the ia64 c5-wip tree to 5.8?

it should be fairly straightforward - the yum ( mx actually ) issues that i ran into way back when, are also resolved now. But, is there any interest ?

Probably just from me. :-) IA64 has never been a very popular arch, but it's what I have to work with at the moment.

Just need to get bootstrapped to do the builds if there isn't any other interest; the ideal, at least from my point of view, is a CentOS5.x ISO to at least get a minimal install up for building, and initializing the build environment so that I can pull the source RPM's and do the rebuild for our use (again, assuming no other interest from the wider CentOS community). I'm willing to work from a 'less than ideal' point; having the 5.8 binaries at the moment would be a fantastic second place, and working through the pointers to the build environment that have been posted in the past is my fallback.

The machine came with some older SLES and RHEL DVD's along with SGI's ProPack of fairly recent vintage (last version that supported RHEL5, in fact), so I can get the box booted (in fact it is booted and running now.....). There is one EL5 rebuild for IA64 for which an older version is actually freely downloadable and could be used to bootstrap the process. (I'm trying to not use the 'O' word....).

I'm perfectly fine just pulling the source RPMs and working through what I know will not be a trivial process building my own once I get the thing bootstrapped up to a CentOS 5.8 equivalent level and a buildsystem set up, assuming that there's no wider interest in the CentOS community. And I might even look at going to EL6, but as painful as that build was for everyone, it would probably take a while, and I honestly don't know if I could get that done. At least an EL6 on IA64 wouldn't have to pass upstream binary compatibility testing for the IA64 side, just the ia32el side for the few things that would need it. EL5 is sufficient for what I want that box to do. And Debian 6 is too unstable on that hardware, and everything else on site is CentOS anyway, so Debian isn't really a solid option.

Anyway, thanks for the reply, it is appreciated. I'm not going to kid myself into thinking that this will be easy if I roll my own; since you already have the buildsystem set up it would be easier for you, but if there's no other interest I'll just have to slog through it myself. And I expect that it may take some time to get it just to rebuild. And I'm not even going to try to do a 'binary compatibility' test; I'll be rebuilding what we need from source anyway, so just having a reasonably close base is sufficient.

On 08/29/2012 08:44 PM, Lamar Owen wrote:

On Tuesday, August 28, 2012 06:24:02 PM Karanbir Singh wrote:

...

[rebuidling c5 on ia64] should be fairly straightforward - the yum ( mx actually ) issues that i ran into way back when, are also resolved now. But, is there any interest ?

FWIW, I now have a working mock setup on the Altix box

Have a look at mockchain. Quite helpful when building a large amount of SRPMs:

https://skvidal.wordpress.com/2012/04/20/mockchain-use-cases-and-examples/

http://fedorapeople.org/cgit/skvidal/public_git/scripts.git/tree/mock

Regards, Patrick

On 08/31/2012 08:09 PM, Lamar Owen wrote:

I may try to build the set of updates first, and then try to respin C5.8 after that. It will probably take some time....

happy to help..

On Wednesday, September 05, 2012 11:48:41 AM Lamar Owen wrote:

I'll shoot you an e-mail a little later today or tomorrow about what I've found thus far, and a couple of questions; that is, once I get caught up from a long weekend.... :-)

Thanks for the offer, it is most appreciated...

An update for the -devel list.... I've been keeping Karanbir updated, but this might have a little wider interest, now that I'm actually getting somewhere.

After having a difficult time getting a build started on our 20 CPU Altix 3700, I punted and tried doing the same builds on our smaller 2 CPU Altix 3200 (same basic machine as the 3700, just smaller). Using the last Scientific Linux CERN 5.4 IA64 build as my base, once installed I was able to successfully build the kernel, which I had not been able to do on the larger box. This pointed to either an SMP/NUMA issue with IA64 Linux, or an issue with the 20 CPU's box's hardware.

Since we have another large box (30 CPU Altix 350) that was successfully running tests on Debian 6, and since I'd really prefer not to use Debian (but I can go back to it if I have to) I put SLC 5.4 on it, which was a little more of an adventure due to the partitioning.... after getting the build environment set up, I was indeed able to successfully build the CentOS 5.8 updated kernel, which I had not been able to do on the 20 CPU box, but had done, much more slowly, on the 2 CPU box (many hours on the 2 CPU box; less than 2 hours on the 30 CPU box, and most of that time was disk I/O writing out the binary RPMs). Hmm, something is wrong, hardware-wise, with the 20 CPU box, it seems.

After a few false starts, I got mock up and running, version 1.0.28 from EPEL (it's a noarch RPM; installing the RPM out of EPEL's x86_64 repo worked fine, just had to rebuild a couple of its dependencies). I then rolled smock.pl over to it; mockchain, while it looks like a good piece of software, apparently is too new for mock 1.0.28, so it's smock for now.

Now, the 5.8 glibc doesn't want to build using the SLC 5.4 binary RPMs as 'seed' for the buildroot; so after careful thought, and understanding how long this might take, I mirrored the SRPMS for CentOS 5.5, 5.6, and 5.7 (already have 5.8 down). Test builds of both the 5.5 kernel and glibc were successful, and so I set the box to building the full 'os' repo of the CentOS 5.5 SRPMS, using the SLC 5.4 binaries to 'seed' the buildroot. Once the 5.5 set is built, I'll re-seed the build root with the 5.5 binaries, and either rebuild 5.5 or 5.6, and then step up one rev at a time until I get 5.8 to rebuild. At that point, I plan to do an actual iso spin of 5.8 (internal use only at the moment, unless there is wider interest), and try to install it on the 2 CPU Altix 3200. Maybe 5.9 will be out by that time (I figure it will take at least a month to build things stepwise).

In any case, the local rebuild of CentOS 5.5 using the SLC5.4 binaries as 'seed' started Saturday evening; as of 8:20A today it has successfully rebuilt 443 source RPMS, producing 1243 binary RPMS, and have seen 35 packages fail thus far. I think most of those failures are due to my forgetting that m4 version 1.4.8 is required, and m4 1.4.5 is provided in the SLC54 repos, so once this first pass completes I'll retry the failed packages (very easy to do with smock).

The biggest help was having SLC 5.4 available in an installable form, even though I had to do it as a network install; the install DVD didn't boot on the Altix systems, but the small boot.iso did, and serving up the packages on a webserver here an HTTP install was quick and painless. The CERN folk, especially Jaroslaw Polok, did the biggest part of the grunt work there, and I thank the CERN team.

Oh, and if you're interested in this sort of thing, pics of both the 30 CPU and the 2 CPU boxen (they occupy the same rack) can be seen a little way down the page at: http://forums.nekochan.net/viewtopic.php?f=14&t=16725868

On Monday, September 24, 2012 10:00:39 AM Matthew Patton wrote:

In the interest of time why not build just the 250 or so packages that comprise a minimal system plus compiler support?

In the interest of completeness, I'd like to build everything.

Further, this is secondarily serving as the commissioning tests for this machine.

So while I understand the sentiment, it will work out better for us to build everything.

And then I'm going to rebuild some third-part repo SRPMs.

On Monday, September 24, 2012 08:36:38 AM Lamar Owen wrote: ...

In any case, the local rebuild of CentOS 5.5 using the SLC5.4 binaries as 'seed' started Saturday evening; as of 8:20A today it has successfully rebuilt 443 source RPMS, producing 1243 binary RPMS, and have seen 35 packages fail thus far.

...

And an update. Following an iterative process of building the failed packages as a set over and over against the 5.4 buildroot seed binaries, I was able to get it down to 118 source packages that failed to build, and 3,245 binaries successfully built. Two of those packages blocked me from uprevving my buildroot to 5.5: chkconfig, and lvm2. Since neither of those packages actually does any building, I seeded a 'reqs' repo with those two, plus the 32-bit valgraind-devel package, along with m4 1.4.8, and uprevved the buildroot (this is done by pointing the repos in the mock config to the newer binaries; mock configs are essentially modified yum configs, and work pretty much the same way, with quirks). This brought the number of successfully rebuilt packages up to 3,389 binaries, with 81 failed builds. Fortunately, none of the failed packages would prevent a yum update-style cross-grade from SLC 5.4 to CentOS 5.5. (the really key package that is still failing is anaconda; but that's not a big problem right now since I'm not going to try to spin media until I get up to 5.8, I just need to be careful about backups and archives of the built repos!).

If you want the gory details, I'll be glad to share, but the short and sweet of it is that I now have one Altix box running my own local rebuild of CentOS 5.5 on ia64.

I'm now dong a 'self-hosting' build of 5.5 using the previously built 5.5 binaries as the buildroot, and once that's done I'll update/cross-grade the buildhost to the self-hosted 5.5 and start on uprevving to 5.6. The 5.5-> 5.6 upgrade wasn't the easiest one we've had, and I'm kindof dreading it in a way, but the process thus far has been very educational and enlightening when it comes to what kind of mechanics really go in to doing something like a rebuild of the upstream sources.

And many thanks to Karanbir and Johnny for the encouragement and assistance with the thought processes going into this. While I'm not currently using the same mock version as CentOS is, nor am I using the same process and hooks, the insights they've provided have been key to getting this far. And, of course, Russ Herrold's archived notes and correspondence were instrumental in me thinking I could even do such as thing as this in the first place.... thanks guys!

On Thu, 4 Oct 2012, Lamar Owen wrote:

But I want to note that I have done what I have done thus far using only publicly available information about the CentOS build process and my own knowledge of RPM building from my five years of maintaining the packages for PostgreSQL; it takes thought and iterative legwork to make this stuff rebuild. And I'm not using the centos-used version of mock, nor am I using any 'secret' configurations from the centos developers; just pointers to the right directions in general, which have been most helpful. And reading build logs and trying to ferret out why that package didn't build (was it the gcc version, or is there a missing dep, or do I need to turn off parallel builds, or....?).

Go, Lamar, go

If you need 'blank' artwork template files to amend the branding, please let me know out of band

-- Russ herrold

On Thursday, October 04, 2012 12:10:16 PM Karanbir Singh wrote:

On 10/04/2012 04:57 PM, R P Herrold wrote:

...

If you need 'blank' artwork template files to amend the branding, please let me know out of band

why would we not use the centos artwork here ?

If I'm not able to get binary compatibility, I don't want users thinking they're running CentOS, with the full binary compatibility that implies. Rebranding in this case wouldn't be so that it would be 'my' build, but rebranding would be so that my users won't get the wrong impression of CentOS when they find binary incompatibilities.

And when I say 'users' I'm talking about my own internal users; if we can get binary compatibility then I'm fine with this becoming the base for 'real' CentOS ia64 and being distributed as such, should that be desired and enough interest, for the bandwidth it will require, is expressed.

On 10/04/2012 05:18 PM, Lamar Owen wrote:

No magic there, just basic sysadmin know-how from experience with Unix systems since 1987.... :-P

The key piece of my build setup is smock.pl. I'll be posting some time this afternoon more details on my setup, inlcuding a rough outline of the process I have thus far used (once I am satisfied that my process looks like it might actually work, that is, since I haven't gotten up to 5.6 yet.....).

And it *is* a manual process; there are steps that I have used that would be difficult to automate in any reasonable manner.

I think its worth doing the 'find srpms that exclude ia64' and work out their dep chain - I suspect you have quite a few more RPMS than are needed to be upstream compatible.

Also worth keeping in mind is that while its a good idea to build the BuildArch: noarch ones' you should not block on those failing - I remember a bunch were not happy on ia64's arch type.

And while it might seem like I could sidestep up to 5.8, I want to iterate through the whole process, using the full tree, for my own reasons.

Hello Lamar Owen,

I think there is value in a self-hosted (can fully recompile itself) build of CentOS.

I sometimes use a recompile with koji as some test-load for kvm guest systems. A sample recompile can be seen on http://jur-linux.org/koji/

Some rpms failing a recompile (not all will be relevant to IA64) are: gpm, geronimo-specs, rhythmbox, evolution-connector, gstreamer-plugins-base, gaim, totem

best regards,

Florian La Roche

On Saturday, October 06, 2012 08:35:36 AM Lamar Owen wrote: ...

A bigger problem in going from 5.5 to 5.6 is the exact build time for a package relative to the rebased gettext 0.17 that was introduced in 5.6. See https://bugzilla.redhat.com/show_bug.cgi?id=523713 for info. The short form: if a package is failing due to MKINSTALLDIRS being undefined, then you need to build it with the older gettext...

A separate 5.6 updates rebuild status report later in the morning, once I get a few more of the 11 failing packages (that are relevant to ia64) to rebuild.

Three more down.... e2fsprogs, util-linux, and w3m all needed the older gettext. I'll leave it as an exercise for the reader to research if the current versions in 5.8 will build with the gettext in 5.8.... :-)

On Saturday, October 06, 2012 09:51:37 AM Lamar Owen wrote:

On Saturday, October 06, 2012 09:11:16 AM Lamar Owen wrote:

...

Three more down.... e2fsprogs, util-linux, and w3m all needed the older gettext. I'll leave it as an exercise for the reader to research if the current versions in 5.8 will build with the gettext in 5.8.... :-)

Ok, and for the status record:

I'm working through the 5.5 and 5.6 build failures, but have started the 5.7 package build in the background, using the same two-step I documented in a previous message (that is, disable the [smock] automatic repo, build centos-release and centos-release-notes, and, avahi and buildsys-macros this time, then re-enable the [smock] repo and build the rest).

The first four packages are building now; avahi from 5.5 didn't want to upgrade cleanly (a %postun failure) and that caused every build to fail after avahi ( since the failure occurred during the chroot yum update, about halfway through the build) built. And buildsys-macros is new for 5.7; so to be safe I'm building it separately. And those four packages built successfully, so now on the the other 204, and leave it running until Monday.....

Anyway, the build failures I've found, and that I've been working through (and I've googled and searched bugzilla.r.c, but not bugzilla.c.o, for these):

Failed on 5.5:

aspell-es, aspell-no, aspell-pt: upstream bug reports have been filed, but not fixed, quite a while back. I'm going to use the SLC5.4 versions for now.

convmv (fails tests in mock chroot with a 'smartness test failed' error; builds fine with rpmbuild as normal user, will research later)

cyrus-imapd (note: will come back to this one later, since it requires i386 packages)

gnbd-kmod (requires an older kernel-devel version than 5.5's, will need to rebuild the supporting kernel to have its -devel available in the buildroot)

jsch (segfaults compiler; researching. Won't build with rpmbuild or mock, using the 5.5 gcj.)

libsvstreams (won't build in mock or in rpmbuild; throws error in wvvector.h)

lsof (causes an arcane, insane, and lying error: error: Package already exists: %package debuginfo

For which you can find Russ's solution at: http://www.oldrpm.org/hintskinks/debuginfo/

(hint: the very last comment in the Changelog of lsof is the culprit) This one posed a quandary; the fix is to edit the Changelog in the spec, changing the bogus '%install' to '%_install' instead, and I'm still researching how to fix this the 'proper rebuilder' way. ) )

metacity (there is a syntax error of some kind some where; the buildrequires listed by rpm -qip --requires on the src.rpm is funky, but it does rebuild with rpmbuild, just not with mock; using the rpmbuild handbuilt package for now)

notification-daemon (dbus-binding-tool throws an error in both mock and rpmbuild; using SLC 5.4 version for now while I investigate)

perl-XML-Parser/perl-XML-Simple (documented bugreport already on bugzilla.r.c about these two; has to do with perl-XML-SAX and a Parser order; in any case, won't build in mock or rpmbuild and requires further research. Using the SLC 5.4 versions for now)

(yelp fails unless SLC 5.4 perl-XML-Parser is in place, then it builds)

python-docs ( find-debuginfo.sh throws the error: find: /var/tmp/python-docs-2.4.3-root: No such file or directory when building in mock and in rpmbuild; researching how to fix this one too, but low priority)

Failures in 5.6 (updated from 5.5 packages only):

perl-XLM-Parser, perl-XML-Simple, and yelp, for the same reasons as 5.5.

metacity and kudzu needed handbuilding on a C5.5 host.

anaconda built once kudzu was built.

e2fsprogs, util-linux, w3m, and pam built under older gettext.

esc (has multiple failures, even when injecting nss-devel into the buildroot by hand. Builds fine with rpmbuild as a normal user; placed in my handbuilds repo for now).

And 5.7 is churning now, along with a normal-user rpmbuild of the 92 kernel for gnbd.....as my buildstatus script reports:

[lowen@winterstar ~]$ buildstatus57 No IA64 target: 1 No Package Found: 0 Scratch total: 2 Built binaries: 159 Built SRPMS: 54 [lowen@winterstar ~]$

(it'll slow down when it hits glibc, gcc, sblim, kernel, xulrunner, and a few other largish packages.....and I may have to recreate the chroot cache every so often; it's almost a big enough inconvenience that I'm tempted to disable the root cache altogether)

And I've successfully updated my test host (the 2 CPU Altix) to 5.6 (once I populated the local repositories properly.....): CentOS release 5.6 (Final) Kernel 2.6.18-238.el5 on an ia64 ...

I've not upgraded the main buildhost from SLC 5.4 yet; until I find a failure that is due to the buildhost being that far downrev from the build chroot I'm going to leave it in a 'building' condition, since it is successfully building, and a botched upgrade would be painful right now.... :-)

But I want to reiterate, and I can't emphasize this enough, that what I am doing is nowhere near as difficult as what the CentOS build team did when doing this for i386 and x86_64, since I'm not yet doing thorough QA nor am I yet targeting strict binary compatibility; I can see doing that could easily make the builds take two to ten times longer.

On Monday, October 08, 2012 09:28:29 AM Lamar Owen wrote:

And the 5.7 new/changed packages that are relevant to ia64 are built, sans three: kudzu, NetworkManager, and mod_nss. I'm going to look into those three failures, but in the meantime I'll put the builder to work getting the 5.8 changed/new packages built.

And NetworkManager and kudzu handbuild fine. Kudzu does not; I need to rebuild pciutils to not include zlib-devel, and am doing that now....

Since 5.8 includes new pciutils, doing that one instead of the downrev, and rebuilding kudzu as part of 5.8 instead of going downrev. That may come back to bite me later, but, doing it anyway. :-)

On Oct 8, 2012, at 9:28 AM, Lamar Owen wrote:

...but in the meantime I'll put the builder to work getting the 5.8 changed/new packages built.

Again, my first order of business is to get up to a 5.8, even if that 5.8 is not fully upstream binary compatible.

And, well, after 34 hours, 9 minutes, and 53 seconds (with an average of 213% CPU usage; having multiple CPUs does help some), the 5.8 GA tree is built, with no build failures that weren't expected (final buildstatus run:

[lowen@winterstar ~]$ buildstatus58 No IA64 target: 5 No Package Found: 4 Scratch total: 9 Built binaries: 991 Built SRPMS: 237 [lowen@winterstar ~]$

and all the No Package Found errors are due to requirements that aren't available in IA64 form...) Cool, the first build run with no real build failures. Nice.

And so, my first goal is met: I have a yummable 5.8 tree (through steps; I have a reasonably full 5.5 tree, and the 5.6, 5.7, and 5.8 incrementals). The test upgrade on my Altix 3200 demonstrated yet again how my misstep with a 'dotless' %dist tag is going to continue to bite me, and how that I need to do a full rebuild at some point. (yum reinstall * doesn't help here, since the dist tag belongs to the release, and reinstall only replaces identical NEVRA with identical NEVRA, so the older disttag doesn't get replaced, and the older disttag is leftover from the interim bootstrap tree..... argh).

But, after correcting the one issue, I then did the standard two-step (yum upgrade kernel glibc rpm yum; yum upgrade) after editing the repo files (and renaming CentOS-Vault.repo to CentOS-Vault.repo.rpmsave, since there are no ia64 files in the Vault....), and some waiting ( 19:34.69, to be exact), and a reboot later, I have:

CentOS release 5.8 (Final) Kernel 2.6.18-308.el5 on an ia64

...

The next step is step building the 5.8 updates, probably in %{BUILDTIME} order as the first attempt, and getting a fully updated system. given some of the substantial changes since 5.8 GA (like Firefox 10, for one) the updates will likely take some time, and I'm honestly not sure if I should just try to rebuild the latest (although that's essentially what I've done up until now, for each point release)). I haven't yet looked at the full contents of the 5.8/updates/SRPMS dir, since I've been 'blinders on get 5.8 GA built' for the last two and a half weeks (of spare time work; the buildbox has churned essentially 'round the clock building, but my time has been in relatively short spurts getting things ready to kick off). So that's a task for tomorrow.

Then I'll upgrade my SLC 5.4 buildhost, now that the bulk of the initial grunt work is done. And I'll make sure to backup all the repos, logs, and other items prior to the upgrade.....

A full rebuild, using the automated QA stuff for QA (neat logging to the #centos-devel IRC channel!), and binary compatibility checking is in my long-term plans; short term there are the straggler packages from 5.5 and 5.6, some third party packages, and some third party whole repos to work on for packages I need on these boxes, like xrdp, and some manual QA of packages as I use the boxes, since they are still in the 'commissioning' phase. And I have a 20 CPU box to do a scratch install on, once I fix its hardware issues, so actually getting an installable tree and making an installable ISO would be a nice thing, and another skillset to learn a little about.....

But the first major milestone is met, and that's cool.

Hi Lamar,

Thanks for the update.

On 01/21/2013 04:57 PM, Lamar Owen wrote:

up the space heater ^H^H^H buildhost (again, photos of the buildhost can be seen at http://forums.nekochan.net/viewtopic.php?f=14&t=16725868 ).....

Wonder what the electric bill for that is :)

Almost all of the 5.9 GA updated/new packages are built (just a couple of packages are being stubborn, and those have a history of being stubborn, at least on IA64 and on this buildsystem; I'm using smock),

Have you looked at mockchain ?

and I have test-updated the second Altix box to 5.9: CentOS release 5.9 (Final) Kernel 2.6.18-348.el5 on an ia64

cool

Third-party repos are also on the plate, since there are several packages we need that are in either EPEL or RepoForge. Those are probably higher on the list than making the build fully self-hosted, really.

One thing that you might run into is that the packagers on those repo's would almost certainly not have been testing builds on IA64, so consider some sort of a feedback loop : maybe even just a http interface to the logs and status ( let apache do the listing with options indexes' turned on ).

But if more were interested in the IA64 build, the priorities could definitely change. Given enough interest, install media would be a logical step, and while that is on my list of things to do, it's not

Happy to promote this build effort a bit and try to drum up support. It would essentially boil down to : if we start it, if we promote it - are we then in a place where we can keep the builds going over a reasonable period of time. If the answer there is, YES, then lets start making the noise.

w.r.t the system updates and tracking that - the buildsystem at this end which does the i686 / x86_64 / powerpc builds is capable of triggers via http call back. If that helps, lets set that up.

Regards,

On Thursday, October 04, 2012 11:45:14 AM Lamar Owen wrote:

Ok, well, I've learned a lot since last Friday. Especially as it pertains to circular dependencies and how to properly (hopefully) step from one release up to another. ...

And after I wrote that, I realized that I hadn't documented how I did that. So, in this admittedly longish e-mail, I'll attempt to do that.

Also, I failed to note that the precise build chain that I ended up following is likely not the same as the chain used building the supported CentOS builds, since I started with a different set of packages and did not build intervening updates. And my chain was compiled semi-automatically using iterations of invocations of smock.pl with possibly a different mix of repos each time, and thus the build I have is likely not completely reproducible from the c5-wip tree Karanbir posted a while back.

And, well, honestly, I'm not as concerned with reproducibility as I am with getting a self-hosting 5.8 build as an end result, from which I will then build the 5.8 updates and/or the 5.9 tree once it is released; if 5.9 comes out before I get to the 5.8 updates then 5.9 comes afterwards, and I go on to 5.9 updates. The intermediate steps will be different each time through, depending upon the base from where you start, and the order in which you hand-resolve each and every one of the several hidden dependencies; knowing that going into this process I specifically did not document every step in exact detail, as it's not important, at least in my opinion. I'll give enough info below for anyone who can figure out how to get mock working and building things, along with the use of createrepe, to follow.

Here's how I'm stepping from CentOS 5.5 to 5.6. To duplicate you'll need a working binary repo, one that is complete enough to satisfy the dependencies of buildsys-build (available on dev.centos.org, and its spec is available in the mock package itself). If you are building a complete tree using smock.pl, smock.pl will figure out the buildrequires that are documented in the SRPM header and will sequence the builds accordingly. It's the undocumented, hidden, or conditional buildrequires that will bite you, and you need binaries for those packages in your repos for the buildroots. I say buildroots (plural), and not buildroot (singular), since each mock invocation that smock.pl makes will start with the base buildroot and install the various buildrequires needed into it for the specific package that particular mock invocation is building.

First, get and install mock 1.0.28 plus all dependencies from EPEL. Also get some example source RPM's and make sure rpmbuild as a normal user works. If you want an experience closer to what the CentOS build is like, then forgo mock 1.0.28 from EPEL and use the patchedf mock-0.6.13 from CentOS-Extras; my configuration won't work for you, but you'll have a closer-to-CentOS-build experience, if that's what you want. I'm more concerned about getting up to date first, and 'correct' second. So I may downgrade mock to 0.6.13 and use the CentOS-provided configs (and other info that's been posted to the mailing list over the years) to do the final 'self-hosting' build at 5.8 (or 5.9). But anyway, back to the documenting....

Add yourself as a member of the 'mock' group.

For performance, mount /var/lib/mock as a tmpfs of several GB, if you have the RAM to spare (my ia64 builder, winterstar, has 54GB, so I gave it a 26GB tmpfs on /var/lib/mock). My experience was that this trimmed a pretty substantial amount of time off of some of the lengthier builds, like the kernel.

Now, obtain smock.pl. ( part of fedora-mingw-devel) See: http://git.annexia.org/?p=fedora-mingw.git;a=tree;f=smock

Read the README, and read the script to see what it's going to do. You will need to create initial repos in the target directory, and you'll need to do the proper configuration for smock.pl to 'find itself'. I cheated and symlinked ~/public_html to my build tree in /opt/build/. Once you understand what smock.pl will, and won't, do, put it somewhere in your path (like /usr/local/bin ) and make it executable.

And, yeah, you need to understand what's going on; this is not a recipe with quantities of ingredients, exact cooking temperatures, and exact cooking times here. I can, for instance, tell you how to cook sausage gravy; but I can't give you a recipe, since the amount of grease rendered from the sausage will vary, and thus I need to say things like 'put in enough flour to barely absorb all the grease and to make a moist, not dry or wet, but moist, putty (not paste and not dough)' and ' pour in enough milk to quickly boil and 'batterize' the flour-grease mixture' or 'pour in enough milk to make the gravy look just almost too thin' : you need to understand what you're doing to follow these directions. The reason is simple: you're not going to be building things in exactly the same order as upstream, CentOS, or I did, and thus you're probably going to get stopped in your build attempt in different places than we did. I know I experienced at least one issue in my ia64 build than wasn't experienced in the i386 or x86_64 builds.

Now, the mock configuration file is the piece that drives everything. Here's the one that I'm using right now to churn the 5.6 updates for ia64. Watch for line wrap. I'm going to notate each section; my notations will be prefixed with a ----> :

++++++++++++++++++++++++ [lowen@winterstar ~]$ cat /etc/mock/centos-56-ia64.cfg #!/usr/bin/python -tt

----> The above line marks this configuration file as being a python script, ----> with all the syntax rules etc of a python script.

import os config_opts['root'] = 'centos-56-ia64' config_opts['target_arch'] = 'ia64' config_opts['clean'] = True

----> These configuration options tell mock the buildroot chroot to use, ----> the architecture for which to build, and whether to clean the chroot

config_opts['chroot_setup_cmd'] = 'install buildsys-build zip zlib-devel autoconf pkgconfig libsmbclient-devel gstreamer-plugins-good-devel dhclient'

----> This previous line, which is one line, is where hidden buildrequires: ----> injection occurs in my buildsys. CentOS does things differently; see ----> the top three entries in the changelog for the mock in CentOS-Extras ----> for the patch CentOS uses to mock 0.6.13. I'm looking at eventually ----> updating the HintsAndMacros patch for mock 1.0.28, but that's after ----> I get things built.

config_opts['plugin_conf']['ccache_enable'] = False

----> This above line is critical to make the system work properly.

----> Note the triple double quotes below. This is a multiline string, being ----> assigned to config_opts['yum.conf'] and anything inside the paired ----> triple double quotes is a yum configuration file and not a python script.

config_opts['yum.conf'] = """ [main] cachedir=/var/cache/yum debuglevel=1 logfile=/var/log/yum.log reposdir=/dev/null retries=20 obsoletes=1 gpgcheck=0 assumeyes=1

# repos

----> The first repo stanza below was the bootstrapping SLC 5.4 binaries, which ----> I don't need anymore for building the 5.6 updates.

#[slc-64bit-local-repo] #name=slc-64-local.repo #baseurl=file:///opt/build/repos/slc54 #enabled=1 #gpgcheck=0 #

----> Comment below says it all. Well, almost all. It's as built under ----> SLC 54 plus an accumulation of 5.5 packages over at least four ----> full iterations.

# CentOS 5.5 as built under SLC 5.4 [c5-os] name=c5-local.repo baseurl=file:///opt/build/repos/centos/5.5/os/ia64/ enabled=1 gpgcheck=0

----> This section below was only for testing, but I left it in place as ----> reminder to myself....

#[c5-32-bit] #name=c5-32-bit.repo #baseurl=file:///opt/build/repos/centos/5.5/os/i386/ #enabled=1 #gpgcheck=0

#newer m4 for buildsys

----> This is documented by SL CERN on their SLC5 build page.

[m4-c5] name=m4-c5.repo baseurl=file:///opt/build/buildsys/m4-c5 enabled=1 gpgcheck=0

----> The repo below is the most dynamic piece of the whole ----> puzzle, and has changed with the multiple iterations. ----> This holds the built-from-source libunwind, among others. ----> As the contents of this repo changed over time, and with ----> each iteration, the exact contents isn't important, but I ----> will say that valgrind-devel i386 lives here, even for an ia64 ----> build. I can try to reconstruct the order of packages in and ----> out of this repo, but, really, I just put the packages in here ----> that the error logs told me the build needed. Once the ----> 5.5 package of the same name built, out of this repo it ----> went.... and, of course, I ran createrepo on it for every ----> modification.....

# other things the buildsys needs, pulled from downrev [buildroot-reqs] name=buildroot-reqs.repo baseurl=file:///opt/build/buildsys/reqs enabled=1 gpgcheck=0

----> The repo below is the buildsys-build rpm, pulled from the centos ----> dev server, and basically verbatim from the mock rpm (do an ----> rpm -ql mock to find the spec file.....)

[groups] name=groups #baseurl=http://dev.centos.org/centos/buildsys/5/ baseurl=file:///opt/build/Factory-Install/groups/5/

# Change USERNAME and if necessary the distribution name and # architecture, and INSERT this file to the relevant distributions # in /etc/mock (eg. to /etc/mock/fedora-9-i386.cfg etc.)

----> The below repo is the dynamic repo that smock maintains. ----> in my instructions below I will try to consistently call it the [smock] ----> repo.

[smock] name=smock #baseurl=http://127.0.0.1/USERNAME-smock/yum/fedora-9/$basearch # If you do not want to use a http server, you need to specify a direct path to the # repository. Then comment out the other baseurl line. baseurl=file:///opt/build/public_html/smock/yum/centos-56/ia64 enabled=1 keepcache=0

""" ----> Note the closing triple double quotes; we're not in Kansas anymore

----> The below syntax is for mock 1.0.28. This syntax will not work for ----> mock 0.6.13 as distributed in CentOS-Extras. See the mock config ----> Johnny posted several months ago for the syntax for 0.6.13

config_opts['macros'] ['%_topdir'] = "/builddir/build" config_opts['macros'] ['%_rpmfilename'] = "%%{NAME}-%%{VERSION}-%%{RELEASE}.%%{ARCH}.rpm"

# Change the next two lines to reflect yourself.

config_opts['macros'] ['%packager'] = "Lamar Owen lowen@pari.edu" config_opts['macros'] ['%vendor'] = "PARI" config_opts['macros'] ['%distribution'] = "PARI Internal Altix EL5 Rebuild"

----> I initially followed the advice below, changing %dist to first be ---->"pari.lro" (which mangles the release) and then ".pari.lro" but then ----> I found that %dist must be .el5 for certain packages to ----> properly build. (gdb, for one, at least on ia64). A grep through all ----> spec files to find conditional buildrequires would be necessary to ----> find all of the packages that this macro impacts. ----> And, of course, I now have 'pari.lro' and '.pari.lro' tagged packages ----> in my 5.5 repos that will eventually be replaced when I mass ----> rebuild 5.8/9.

# please change this to reflect the Distro Tree and Repo hosting packages! config_opts['macros'] ['%dist'] = ".el5" config_opts['macros'] ['%centos_ver'] = "5"

#need this to get el5 builddeps from some packages. config_opts['macros'] ['%el5'] = "1" config_opts['macros'] ['%rhel'] = "5" config_opts['macros'] ['%_smp_mflags'] = "-j30"

----> The above line is for my 30 CPU Altix; change this to match your number ----> of CPU's and thus the desired number of parallel compiles make will do.

[lowen@winterstar ~]$ +++++++++++++++++++++++++++

Ok, now we need to test mock, and we do this by initializing the chroot:

mock -r centos-56-ia64 --init

If this fails, you have other problems. Look at the output, it will tell you what your problem is. This has to work, or nothing else will. And I can't give you a recipe that is guaranteed to work, every time; you need to understand yum and how to populate a repository properly in order to fix issues with the chroot init. I figured it out; so can you, it just takes some thought and some work, you're capable of doing that.

Now, generate a list of packages to build, and put them in a single line (xargs -n 10000 works well for this), and put it in, say, ~/smock-packages.txt.

The first two packages to build are critical, and to build them you need to temporarily comment out the [smock] repo in the yum.conf section of the mock config. Change dir to the SRPMS directory containing the new centos-release package, and issue (I'm on ia64, so I use that here): smock.pl --arch=ia64 --distro=centos-56 centos-release*src.rpm

This gets the centos-release package and the centos-release-notes package, and you need both, and they need each other, and if you put one in the [smock] repo without the other, and the [smock] repo is enabled for chroot populating, it will break the chrooted yum installs and nothing will build. So leave [smock] disabled for building these two.

Once those two build, uncomment the [smock] repo in the mock configuration. I know, I could probably just set enable=0 instead of commenting it out, but I chose to comment it out, and uncomment it back in instead.

For the build itself, I like to first see what smock.pl is going to do with a: smock.pl --dryrun --arch=ia64 --distro=centos-56 `cat ~/smock-packages.txt`

Smock will send to stderr any dependency loops prefixed with a 'tsort:' and will send the package names in build order out stdout. If you have the downrev packages that will satisfy the deploops, you don't need to worry about them. If you don't have those packages, you'll need to build or find at least one package in each deploop and put it in the [buildroot-reqs] (which is part of the reason I put it there), being sure to run createrepo every time you hand modify any repo.

Save the output somewhere, if nothing else so you can see where the build is by looking at stdout later.

I use nohup; you can use screen or dtach or whatever. My actual building command line is: nohup time smock.pl --keepgoing --arch=ia64 --distro=centos-56 \ `cat ~/smock-packages.txt` &

And I then tail -f nohup.out to watch the build.

Now, smock.pl has a buglet of sorts, and as the number of packages accumulates this buglet makes the build get progressively slower. Smock is maintaining a repository of its own; in the build loop it does a createrepo both before and after each and every package build. When a repo gets more than a few hundred packages in it, the createrepo can take longer than the package build does.

I made a modified 'smock2' that removed the automatic createrepo for the initial pass at the 5.5 packages, and any other time I knew the package set was going to be large, to save some on time, or when the buildroot seed repo contained a vast majority of the build dependencies. Or if I disabled the [smock] repo; if the [smock] repo isn't being used by mock, then why spend the time doing a createrepo on it?

Now, my 5.6 update build isn't complete, so I really don't know how well this 'stepwise' staging is going to work in practice. I should know tomorrow morning; the buid is at:

[lowen@winterstar ~]$ buildstatus56 No IA64 target: 4 No Package Found: 0 Scratch total: 8 Built binaries: 286 Built SRPMS: 86 [lowen@winterstar ~]$

Hmm, there are 8 builds in scratch; this is where failed build build, root, and state logs are kept, as well as where the currently running build, root, and state logs are put until a good build, when they get moved to the logs tree under the binary RPMS dir. So, that means 7 failed builds; 4 of those are due to having not IA64 target for the build. So 3 builds, so far, that I'll need to further investigate.

Tomorrow.

I hope this post is helpful to someone; if nothing else I'm putting it in my own archive for my own reference..... :-)

Yes, when I did the bootstrap - I had interest from 3 people, [...]

So I'll add myself to that "interest" too; we have three boxes here (two rx2600's and one rx4640 with 4 CPUs and 32 GB RAM). I could also invest "some" time, where "some" can be a few hours a week to longer, depending on my current workload.