[CentOS] IBM BladeCenter HS20 blades
Brendon Colby
bren at lh.net
Tue Apr 26 20:40:27 UTC 2005
We have purchased a product with our SAN to take a snapshot of one LUN
so we can image that to another LUN for super fast installs. Therefore,
I wanted to avoid having to set up xCat.
The problem with booting the CentOS 4 DVD was that the CDROM was not
being detected. Since the CDROM in the media tray is USB, I switched
over to another blade, and back, and the install then found the CDROM,
and the rest went perfectly.
Thanks for your help on this.
Brendon Colby
John Burk wrote:
> This problem also exhibits in RHEL3-U4, so CentOS 3.4 inherited it.
>
> CentOS 3.4 is treated as rhas3 by xcat. The first time you run 'nodeset
> <host> install' for a particular OS, the /tftpboot/xcat/rhas3/<arch>
> tree gets populated with files, including install.gz, which is a
> compressed disk image which in turn contains all the kernel modules,
> including tg3.
>
> 1.) First thing you need to do is delete (or rename) the
> /tftpboot/xcat/rhas3/<arch>/ directory.
>
> 2.) Next is to fix a minor bug in xcat's 'mkinstall' located in
> $XCATROOT/install/rhas3/ARCH/bin/. I posted the fix to the xcat user's
> list, and have included the hacked mkinstall as an attachment. Please
> note that this mkinstall is for 'x86', but it should be pretty
> straightforward to see how to make the change for x86_64 if that's your
> arch.
>
> << cut&paste'd from my post in the xcat user's list >>
> I went the second route and dug around inside
> $XCATROOT/install/rhas3/ARCH/bin/mkinstall. Turns out that there is a
> fix already inside this script and a module already built, but that it
> only will be applied to RHEL3-U4 and not CentOS 3.4
>
> It's because of the file test at line 375 of mkinstall: the version
> string in the kernel rpm that ships on the RHEL3-U4 cd's is 2.4.21-27,
> but the CentOS-3.4 kernel version string is 2.4.21-27.0.1. This version
> mismatch is echo'd at lines 383 and 386 also.
> << eo snip >>
>
> 3.) 'cd' to the $XCATROOT/install/rhas3/ARCH/ directory. The fix only
> gets applied if you're in this directory, as the path the to bcm5700.o
> is relative - Egan Ford probably knows why he did it this way, I'm just
> hacking on his script and wanted to muck with it as little as possible.
>
> 4.) run 'nodeset <some_CentOS3.4_host> install'. Because there is not
> an install.gz in the /tftpboot/xcat/rhas3/<arch> directory, it will be
> built, and because you're in the $XCATROOT/install/rhas3/ARCH/
> directory, the pre-built bcm5700.o module which lives in
> $XCATROOT/install/rhas3/ARCH/data/27.EL will get put into the install.gz
> as tg3.o.
>
> 5.) 'rreset' your blades and watch them properly install CentOS 3.4 via
> nfs...
>
> John Burk
>
> Sr. Technical Director
> Mainframe Entertainment
>
> 604.628.1019
>
>
>
>
> Niswonger Charles M Contr AFRL/MNGG wrote:
>
>> John,
>>
>> I'm running 14 older HS20's right now and have 14 of the new
>> 8843 3.6 GHz EM64T blades coming in. Right now we're running SuSE 9.2
>> and xcat, but we've seen the same tg3 bug that you mentioned. I'm
>> curious as to the workaround that you used for CentOS. With the
>> multi-level load needed to install the bootable image onto the blade
>> we still don't have a satisfactory workaround...
>>
>> Mike Niswonger, Systems Engineer
>> charles.niswonger at eglin.af.mil
>> Air Force Research Lab / MNGG (KHILS),
>> SYColeman Corp.
>> 101 W Eglin Blvd, Bldg. 13, Rm. 115
>> Eglin AFB, FL 32542-5427
>> (850)882-4446, x1217
>> (850)882-4451, fax
>>
>> -----Original Message-----
>> From: centos-bounces at centos.org [mailto:centos-bounces at centos.org] On
>> Behalf Of John Burk
>> Sent: Friday, April 15, 2005 1:42 PM
>> To: CentOS mailing list
>> Subject: Re: [CentOS] IBM BladeCenter HS20 blades
>>
>> I have 60 HS20's running CentOS 3.4, but I do the install via the xcat
>> toolset over nfs.
>>
>> Have you looked at xcat yet to manage the os on your blades? Well
>> worth the time to setup, written primarily by IBM, and free.
>>
>> http://www.xcat.org/
>>
>> BTW, what processors are in your blades? I'm running the 3.2GHz
>> Nocona's, if your are similar you'll probably need to use these kernel
>> parameters in your grub.conf:
>>
>> acpi=noirq noexec=off noapic
>>
>> And if you do go with nfs installs via xcat, the tg3 driver in 3.4 is
>> broken. E-mail me and I'll send you the work-around. It's already in
>> xcat but it takes a bit of finesse to apply the fix.
>>
>> John Burk
>>
>> Sr. Technical Director
>> Mainframe Entertainment
>>
>> 604.628.1019
>>
>>
>>
>>
>> Brendon Colby wrote:
>>
>> > Greetings,
>> >
>> > We have purchased an IBM BladeCenter and I am in the process of
>> > testing Linux installation on these things (boot off SAN i.e. qla2300
>> > driver, not using internal drives). My distro of choice is Debian,
>> > however, since I'm really not interested in trying to hand compile all
>> > the drivers, I decided to try CentOS (which I'm so far very impressed
>> > with).
>> >
>> > On boot, as with the Debian Sarge disk, the CDROM disappears. The
>> > installation media complains that it cannot find the CD. I'm testing
>> > using the DVD ISO of CentOS 4. Has anyone been successful in getting
>> > CentOS 3/4 installed on a BladeCenter server? I noticed back in 11/04
>> > there were a few posts on it.
>> >
>> > Thanks.
>> >
>> > Brendon Colby
>> > _______________________________________________
>> > CentOS mailing list
>> > CentOS at centos.org
>> > http://lists.centos.org/mailman/listinfo/centos
>> >
>> >
>> _______________________________________________
>> CentOS mailing list
>> CentOS at centos.org
>> http://lists.centos.org/mailman/listinfo/centos
>>
>
> ------------------------------------------------------------------------
>
> #!/bin/ksh
> #egan at us.ibm.com
> #See $XCATROOT/README for license.
> #
>
> if [ -z "$XCATROOT" ]
> then
> if [ -r /etc/sysconfig/xcat ]
> then
> . /etc/sysconfig/xcat
> else
> if [ -r /etc/rc.config ]
> then
> . /etc/rc.config
> fi
> fi
> fi
>
> if [ -z "$XCATROOT" ]
> then
> if [ -r "$(dirname $0)/../lib/xcatroot" ]
> then
> . $(dirname $0)/../lib/xcatroot
> fi
> fi
>
> if [ -z "$XCATROOT" ]
> then
> echo "$(basename $0): env XCATROOT not defined!" >&2
> exit 1
> fi
>
> if [ ! -d "$XCATROOT" ]
> then
> echo "$(basename $0): XCATROOT $XCATROOT does not exist!" >&2
> exit 1
> fi
>
> . $XCATROOT/lib/functions
>
> function usage {
> echo "\nUsage: ${MYNAME} [singlenode]\n"
> }
>
> needhelp $* && usage >&2 && exit 0
>
> if [ "$#" != "1" ]
> then
> usage >&2
> exit 1
> fi
>
> NODE=$1
> NODENAME=$1
>
> if ! isanode $1 2>/dev/null
> then
> echo "${MYNAME}: $1 is not a node" >&2
> exit 1
> fi
>
> if NODETYPE=$(tabdb $NODETYPETAB $NODE 3)
> then
> :
> else
> echo "${MYNAME}: $NODE nodetype image not defined in $NODETYPETAB" >&2
> exit 1
> fi
>
> if OSVER=$(tabdb $NODETYPETAB $NODE 1)
> then
> :
> else
> echo "${MYNAME}: $NODE nodetype osver not defined in $NODETYPETAB" >&2
> exit 1
> fi
>
> if ARCH=$(tabdb $NODETYPETAB $NODE 2)
> then
> :
> else
> echo "${MYNAME}: $NODE nodetype arch not defined in $NODETYPETAB" >&2
> exit 1
> fi
>
> if NODERES=$(whatismyres2 $NODE)
> then
> RESNAME=$NODERES
> else
> echo "${MYNAME}: $NODE noderes not defined in $NODERESTAB" >&2
> exit 1
> fi
>
> MYOSVER=rhas3
> MYARCH=x86
>
> if [ "$OSVER" != "$MYOSVER" -o "$ARCH" != "$MYARCH" ]
> then
> echo "${MYNAME}: Install directory does not match $NODE in $NODETYPETAB"
> exit 1
> fi
>
> if INSTALLDIR=$(tabdb $SITETAB installdir 1)
> then
> :
> else
> echo "$MYNAME: installdir not defined in $SITETAB"
> exit 1
> fi
>
> DISTDIR=$OSVER/$ARCH
> CFDIR=scripts/$OSVER/$ARCH
> PXET=install.pxe
> GRUBT=install.grub
> ELILOT=elilo.conf
> KERNEL=installz
> INITRD=install.gz
>
> FILES="$INSTALLDIR/$DISTDIR/images/pxeboot/initrd.img $INSTALLDIR/$DISTDIR/images/pxeboot/vmlinuz /usr/share/terminfo/x/xterm $TFTPDIR/$TFTPXCATROOT $TFTPDIR"
>
> if [ -r $TFTPDIR/$TFTPXCATROOT/$OSVER/$ARCH/$KERNEL -a -r $TFTPDIR/$TFTPXCATROOT/$OSVER/$ARCH/$INITRD ]
> then
> DOIMAGE=0
> else
> DOIMAGE=1
> fi
>
> EXIT=0
> for i in $FILES
> do
> if [ ! -r $i ]
> then
> echo "${MYNAME}: $i missing"
> EXIT=1
> fi
> if [ "$EXIT" = "1" ]
> then
> exit 1
> fi
> done
>
> if [ -r $XCATPREFIX/tftp/pxelinux.0 ]
> then
> mkdir $TFTPDIR >/dev/null 2>&1
> cp -f $XCATPREFIX/tftp/pxelinux.0 $TFTPDIR
> else
> echo "$MYNAME: $XCATPREFIX/tftp/pxelinux.0 missing"
> exit 1
> fi
>
> if [ -r $XCATPREFIX/tftp/default ]
> then
> mkdir $TFTPDIR/pxelinux.cfg >/dev/null 2>&1
> cp -f $XCATPREFIX/tftp/default $TFTPDIR/pxelinux.cfg/default
> else
> echo "$MYNAME: $XCATPREFIX/tftp/default missing"
> exit 1
> fi
>
> MASTER=$(tabdb $SITETAB master 1) \
> || echo "${MYNAME}: master not defined in $SITETAB"
> MASTER_IP=$(nametoip $MASTER)
>
> #MASTER_IPS=$(
> # IPS=""
> # for i in $(dnsnr "@$MASTER(-eth[0-9])?")
> # do
> # if IP=$(nametoip $i)
> # then
> # IPS="$IPS $IP"
> # fi
> # done
> # print $IPS
> #)
>
> MASTER_IPS=$(
> IPS=""
> for i in $(ifconfig | egrep '^eth[0-9]' | awk '{print $1}')
> do
> IP=$(ifconfig $i | perl -pi -e 's/ +/:/g' | grep inet:addr: | awk -F: '{print $4}')
> IPS="$IPS $IP"
> done
> print $IPS
> )
>
> if CONFIG_SERVER=$(tabdb $NODERESTAB $RESNAME $noderes_config_server)
> then
> CONFIG_IP=$(nametoip $CONFIG_SERVER)
> else
> CONFIG_IP=$MASTER_IP
> CONFIG_SERVER=$MASTER
> fi
>
> RSHC=$(tabdb $SITETAB rsh 1) \
> || echo "${MYNAME}: rsh not defined in $SITETAB"
>
> if [ "$(basename $RSHC)" = "rsh" ]
> then
> RSH="Y"
> else
> RSH="N"
> fi
>
> SERIAL=$(tabdb $NODERESTAB $RESNAME $noderes_serial) \
> || echo "${MYNAME}: serial defined in $NODERESTAB"
>
> INSTALL_NFS=$(tabdb $NODERESTAB $RESNAME $noderes_nfs_install) \
> || echo "${MYNAME}: nfs not defined in $NODERESTAB"
> INSTALL_NFS_IP=$(nametoip $INSTALL_NFS)
>
> if [ "$INSTALL_NFS_IP" = "noip" ]
> then
> echo "Invalid Install NFS for $NODENAME"
> fi
>
> INSTALL_NIC=$(tabdb $NODERESTAB $RESNAME $noderes_netdevice) \
> || echo "${MYNAME}: install_nic not defined in $NODERESTAB"
>
> INSTALL_DIR=$(tabdb $NODERESTAB $RESNAME $noderes_install_dir) \
> || echo "${MYNAME}: nfs dir not defined in $NODERESTAB"
>
> INSTALL_SRC_DIR=$INSTALL_DIR/$DISTDIR
> INSTALL_CF_DIR=$INSTALL_DIR/$CFDIR
>
> mkdir -p $INSTALL_CF_DIR
>
> FILE=$INSTALLDIR/$CFDIR/$NODETYPE-$NODERES
> INSTALL_CF_FILE=$NODETYPE-$NODERES
>
> if [ ! -r $NODETYPE.tmpl ]
> then
> if [ ! -r base/$NODETYPE.tmpl ]
> then
> echo "${MYNAME}: cannot read template $NODETYPE.tmpl"
> exit 1
> else
> TMPLDIR="base"
> fi
> else
> TMPLDIR="."
> fi
>
> export MASTER
> export MASTER_IP
> export MASTER_IPS
> export CONFIG_IP CONFIG_SERVER
> export RSH
> export OSVER
> export ARCH
> export NODE
> export NODERES
> export NODENAME
> export NODETYPE
> export INSTALL_NFS
> export INSTALL_NFS_IP
> export INSTALL_SRC_DIR
> export INSTALL_CF_DIR
> export INSTALL_CF_FILE
> export KERNEL
> export INITRD
>
> perl -pi -e 's/\\\n$//' <$TMPLDIR/$NODETYPE.tmpl | LANG=C subvars.pl >$FILE
>
> mkdir -p $TFTPDIR/$TFTPXCATROOT/$OSVER/$ARCH
>
> if [ -r "$PXET" ]
> then
> PXEF=$TFTPDIR/$TFTPXCATROOT/$OSVER/$ARCH/$NODETYPE-$NODERES.pxe
> perl -pi -e 's/\\\n$//' <$PXET | LANG=C subvars.pl >$PXEF
> if [ "$SERIAL" = "0" -o "$SERIAL" = "1" ]
> then
> :
> else
> perl -pi -e "s/^SERIAL/#SERIAL/g" $PXEF
> # perl -pi -e "s/(console[^ ]* )//" $PXEF
> perl -pi -e "s/(console[^ ]* ){2}//" $PXEF
> fi
> fi
>
> if [ -r "$GRUBT" ]
> then
> GRUBF=$TFTPDIR/$TFTPXCATROOT/$OSVER/$ARCH/$NODETYPE-$NODERES.grub
> perl -pi -e 's/\\\n$//' <$GRUBT | LANG=C subvars.pl >$GRUBF
> if [ "$SERIAL" = "0" -o "$SERIAL" = "1" ]
> then
> :
> else
> perl -pi -e "s/^serial/#serial/g" $GRUBF
> perl -pi -e "s/^terminal/#terminal/g" $GRUBF
> # perl -pi -e "s/(console[^ ]* )//" $GRUBF
> perl -pi -e "s/(console[^ ]* ){2}//" $GRUBF
> fi
> fi
>
> if [ -r "$ELILOT" ]
> then
> ELILOC=$TFTPDIR/$TFTPXCATROOT/$OSVER/$ARCH/$NODETYPE-$NODERES.elilo
> perl -pi -e 's/\\\n$//' <$ELILOT | LANG=C subvars.pl >$ELILOC
> if [ "$SERIAL" = "0" -o "$SERIAL" = "1" ]
> then
> :
> else
> perl -pi -e "s/^SERIAL/#SERIAL/g" $ELILOC
> # perl -pi -e "s/console[^ ]* //" $ELILOC
> perl -pi -e "s/(console[^ ]* ){2}//" $ELILOC
> fi
> fi
>
> if [ "$DOIMAGE" = "0" ]
> then
> exit 0
> fi
>
> cp -f $INSTALLDIR/$DISTDIR/images/pxeboot/vmlinuz $TFTPDIR/$TFTPXCATROOT/$OSVER/$ARCH/$KERNEL
> cp -f $INSTALLDIR/$DISTDIR/images/pxeboot/initrd.img $TFTPDIR/$TFTPXCATROOT/$OSVER/$ARCH/$INITRD
>
> TMPDIR=/tmp/.mkinstall-$OSVER-$ARCH
>
> if getlock mkinstall-$OSVER-$ARCH 60
> then
> :
> else
> echo "${MYNAME} Failed to get $TMPDIR lock"
> exit 1
> fi
>
> rm -rf $TMPDIR
> mkdir -p $TMPDIR
>
> gunzip -c $TFTPDIR/$TFTPXCATROOT/$OSVER/$ARCH/$INITRD >$TMPDIR/initrd 2>/dev/null
> mkdir -p $TMPDIR/root
> mount -o loop $TMPDIR/initrd $TMPDIR/root
> #mkdir -p $TMPDIR/root/etc/terminfo/x
> #mkdir -p $TMPDIR/root/etc/terminfo/v
> #mkdir -p $TMPDIR/root/etc/terminfo/b
> #cp -f /usr/share/terminfo/x/xterm $TMPDIR/root/etc/terminfo/v/vt100
> #cp -f /usr/share/terminfo/x/xterm $TMPDIR/root/etc/terminfo/b/bterm
> #cp -f /usr/share/terminfo/x/xterm $TMPDIR/root/etc/terminfo/x/xterm
>
> ###
> #rm -f $TMPDIR/root/etc/terminfo/v/vt100-nav
>
> perl -pi -e 's/^.*qla2.00.*\n$//' $TMPDIR/root/modules/pcitable
>
> #MYDIR=$(pwd)
> #cd $TMPDIR
>
> #gunzip -c root/modules/modules.cgz | cpio -id 2>/dev/null
>
> #if [ -r $MYDIR/data/e100.o ]
> #then
> # rm -f 2.4.21-3.ELBOOT/eepro100.o
> # cp -f $MYDIR/data/e100.o 2.4.21-3.ELBOOT/eepro100.o
> # cat $MYDIR/data/e100.pcitable >> root/modules/pcitable
> #fi
>
> #if [ -r $MYDIR/data/bcm5700.o ]
> #then
> # cp -f $MYDIR/data/bcm5700.o 2.4.21-3.ELBOOT/bcm5700.o
> # cat $MYDIR/data/bcm5700.pcitable >> root/modules/pcitable
> # cat $MYDIR/data/bcm5700.module-info >> root/modules/module-info
> #fi
>
> #if [ -r $MYDIR/data/e1000.o ]
> #then
> # cp -f $MYDIR/data/e1000.o 2.4.21-3.ELBOOT/e1000.o
> # cat $MYDIR/data/e1000.pcitable >> root/modules/pcitable
> # cat $MYDIR/data/e1000.module-info >> root/modules/module-info
> #fi
>
> #find 2.4.21-3.ELBOOT -type f -print | cpio -o -H crc 2>/dev/null | gzip -c -9 >root/modules/modules.cgz
>
> #cd $MYDIR
>
>
> # default kernel version that ships with RHEL3-U4
> KERNELVER=2.4.21-27
> #
> # try and detect if we're actually installing CentOS instead of RHEL
> # ( tests cut&paste'd from XCATROOT/sbin/copycds )
> #
> if [ -f $INSTALLDIR/$DISTDIR/.discinfo ]
> then
> if egrep "1104951135.38" $INSTALLDIR/$DISTDIR/.discinfo >/dev/null 2>&1
> then
> if grep "i386" $INSTALLDIR/$DISTDIR/.discinfo >/dev/null 2>&1
> then
> # kernel version in the CentOS-3.4 distro
> KERNELVER=2.4.21-27.0.1
> fi
>
> fi
> fi
>
> #
> # fix the broken tg3 driver in RHEL3-U4/CentOS-3.4
> #
> if [ -f $INSTALLDIR/$DISTDIR/RedHat/RPMS/kernel-${KERNELVER}.EL.i686.rpm ]
> then
> MYDIR=$(pwd)
> cd $TMPDIR
> gunzip -c root/modules/modules.cgz | cpio -id 2>/dev/null
>
> if [ -r $MYDIR/data/27.EL/bcm5700.o ]
> then
> cp -f $MYDIR/data/27.EL/bcm5700.o ${KERNELVER}.ELBOOT/i386/tg3.o
> fi
>
> find ${KERNELVER}.ELBOOT -type f -print | cpio -o -H crc 2>/dev/null | gzip -c -9 >root/modules/modules.cgz
>
> cd $MYDIR
> fi
>
> umount $TMPDIR/root
> gzip -9 -f $TMPDIR/initrd
> cp -f $TMPDIR/initrd.gz $TFTPDIR/$TFTPXCATROOT/$OSVER/$ARCH/$INITRD
>
> rm -rf $TMPDIR
>
> freelock mkinstall-$OSVER-$ARCH 0
>
> cd $TFTPDIR/$TFTPXCATROOT/$OSVER/$ARCH
>
> chown -f root.root *.gz *.pxe *.nbi *.eb
> chmod -f 444 *.gz *.pxe *.nbi *.eb
>
> exit 0
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
More information about the CentOS
mailing list