Thursday, October 16, 2008

Building Lustre 1.6.5.1 against the latest Redhat Kernel

I was at a customer site this week and had the need to build Lustre 1.6.5.1 against the latest kernel from Redhat, 2.6.18-92.1.13. Being this process has multiple steps, I thought I would document it so that others do not have to reinvent the wheel.

1) Prep a build environment
- cd ~
- yum install rpm-build redhat-rpm-config unifdef
- mkdir -p rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
- echo '%_topdir %(echo $HOME)/rpmbuild' > .rpmmacros
- rpm -i http://mirror.centos.org/centos/5/updates/SRPMS/kernel-2.6.18-92.1.13.el5.src.rpm
- cd ~/rpmbuild/SPECS
- rpmbuild -bp --target=`uname -m` kernel-2.6.spec 2> prep-err.log | tee prep-out.log

2) Download and install quilt (quilt is used for applying kernel patches from a series file)
- cd ~
- wget http://download.savannah.gnu.org/releases/quilt/quilt-0.47.tar.gz
- gunzip quilt-0.47.tar.gz
- tar -xvf quilt-0.47.tar
- cd quilt-0.47
- ./configure
- make
- make install

3) Prepare the Lustre source code
- Download from http://www.sun.com/software/products/lustre/get.jsp
- mv lustre-1.6.5.1.tar.gz /usr/src
- gunzip lustre-1.6.5.1.tar.gz
- tar -xvf lustre-1.6.5.1.tar

4) Apply the Lustre kernel-space patches to the kernel source tree
- cd /root/rpmbuild/BUILD/kernel-2.6.18/linux-2.6.18.x86_64/
- ln -s /usr/src/lustre-1.6.5.1/lustre/kernel_patches/series/2.6-rhel5.series series (there are several diffrent series files in the series dir, choose the one closest to your environment)
- ln -s /usr/src/lustre-1.6.5.1/lustre/kernel_patches/patches patches
- quilt push -av

5) Compile a new kernel from source
- make distclean
- make oldconfig dep bzImage modules
- cp /boot/config-`uname -r` .config
- make oldconfig || make menuconfig
- make include/asm
- make include/linux/version.h
- make SUBDIRS=scripts
- make rpm
- rpm -ivh ~/rpmbuild/RPMS/kernel-2.6.18prep-1.x86_64.rpm
- mkinitrd /boot/initrd-2.6.18-prep.img 2.6.18-prep
- Update /etc/grub.conf with new kernel boot information

6) Reboot system with new, patched kernel

7) Compile Lustre with the new kernel running
- cd /usr/src/lustre-1.6.5.1
- ./configure --with-linux=/root/rpmbuild/BUILD/kernel-2.6.18/linux-2.6.18.x86_64
- make rpms (Build RPMs will be in ~/rpmbuild/RPMS)

8) Install the appropriate RPMs for your environment

3 comments:

k3di said...

thanks for your article. There are not more documentation on this subject.

I want to ask some questions. I want to use o2ib for lustre networking. How should I do this? I've tried lots of things. But they did not work.

I have rhel 5.1 kernel (2.6.18-53.el5), OFED 1.3.1, Lustre 1.6.5.1 and perfctr patch for getting performance counters from kernel.

The first thing I did is to apply the perfctr patch (from the package PAPI). Then applied the lustre patches. There were no errors regarding to patch process.

After building the kernel and rebooting the host with this newly built kernel successfuly, I've build OFED 1.3.1 with kernel source that I've just built.

And then tried to compile and install the rpms of lustre. But the resulting rpms not worked as expected. Lots of unknown symbol outputs came out while try to mount the lsutre file system.

There must be some problems. Do I have done something wrong? Or what could be wrong with the process I proceed? If you could tell something about that, I'll be very happy. Thanks in advance.

Regards,

Ender

thuannvn said...

Thank you for your useful article

Mag said...

thanks