Lustre 1.6.6 with MX 1.2.7

Below is the process for installing Lustre 1.6.6 while using MX (Myricom) as the transport.

1) Compile and install Lustre Kernel
- yum install rpm-build redhat-rpm-config
- mkdir -p rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
- echo '%_topdir %(echo $HOME)/rpmbuild' > .rpmmacros
- rpm -ivh kernel-lustre-source-2.6.18-92.1.10.el5_lustre.1.6.6.x86_64.rpm (can be obtained from http://www.sun.com/software/products/lustre/get.jsp)
- make distclean
- make oldconfig dep bzImage modules
- cp /boot/config-`uname -r` .config
- make oldconfig || make menuconfig
- make include/asm
- make include/linux/version.h
- make SUBDIRS=scripts
- make rpm
- rpm -ivh ~/rpmbuild/kernel-lustre-2.6.18-92.1.10.el5_lustre.1.6.6.x86_64.rpm
- mkinitrd /boot/2.6.18-92.1.10.el5_lustre.1.6.6
- Update /etc/grub.conf with new kernel boot information
- /sbin/shutdown 0 -r

2) Compile and install MX Stack
- cd /usr/src/
- gunzip mx_1.2.7.tar.gz (can be obtained from www.myri.com/scs/)
- tar -xvf mx_1.2.7.tar
- cd mx-1.2.7
- ln -s common include
- ./configure --with-kernel-lib
- make
- make install

3) Compile and install Lustre
- cd /usr/src/
- gunzip lustre-1.6.6.tar.gz (can be obtained from http://www.sun.com/software/products/lustre/get.jsp)
- tar -xvf lustre-1.6.6.tar
- cd lustre-1.6.6
- ./configure --with-linux=/usr/src/linux --with-mx=/usr/src/mx-1.2.7
- make
- make rpms (at the bottom of the output it will show location of the generated RPMs)
- rpm -ivh lustre-1.6.6-2.6.18_92.1.10.el5_lustre.1.6.6smp.x86_64.rpm
lustre-modules-1.6.6-2.6.18_92.1.10.el5_lustre.1.6.6smp.x86_64.rpm
lustre-ldiskfs-3.0.6-2.6.18_92.1.10.el5_lustre.1.6.6smp.x86_64.rpm

4) Add the following lines to /etc/modprobe.conf
options kmxlnd hosts=/etc/hosts.mxlnd
options lnet networks=mx0(myri0),tcp0(eth0)

5) Populate myri0 Configuration with proper IP addresses
- vim /etc/sysconfig/network-scripts/myri0

6) Populate /etc/hosts.mxlnd with the following information
# IP HOST BOARD EP_ID

7) Start Lustre by mounting the disks that contain the MGS, MDT and OSS data stores