Lustre Install Notes

From Wiki
Jump to: navigation, search

Create a mirror

cd /pub/lustre
wget -r -np -nH https://downloads.hpdd.intel.com/public/lustre/lustre-2.9.0

Cluster Setup

Note: SELinux must be *disabled* on server nodes in order for Lustre to work
sed -i -e "s/enabled/disabled/" /etc/sysconfig/selinux && reboot

Install packages

CentOS 7.4

cd /etc/yum.repos.d
curl -O https://www.watters.ws/rpms/lustre/lustre-2.10.1/server/e2fsprogs.repo
curl -O https://www.watters.ws/rpms/lustre/lustre-2.10.1/server/lustre.repo
yum update && yum -y install lustre sg3_utils openmpi

For benchmarks install the test package:

yum -y install lustre-tests 

Reboot into the lustre kernel:

reboot

Create File System

on management server:

mkfs.lustre --fsname=lustre --mgs /dev/sdb1

on metadata server:

mgs=10.201.67.2
mkfs.lustre --fsname=lustre --mdt --mgsnode=${mgs}@tcp1 --reformat /dev/sdb1

on object server:

mgs=10.201.67.2
mkfs.lustre --ost --fsname=lustre --mgsnode=${mgs}@tcp1 --index=$n --reformat /dev/sdb

mount FS:

mkdir -p /var/mnt/ostnum
mount -t lustre /dev/sdb /var/mnt/ostnum

Install Lustre Client

cd /etc/yum.repos.d
curl -O https://www.watters.ws/rpms/lustre/lustre-2.10.1/client/lustre-client.repo
yum -y install lustre-client

Packages can also be built from the SRPM.

cd /usr/src
wget https://downloads.hpdd.intel.com/public/lustre/lustre-2.9.0/el7.3.1611/client/SRPMS/lustre-2.9.0-1.src.rpm
wget https://downloads.hpdd.intel.com/public/lustre/lustre-2.9.0/el7.3.1611/client/SRPMS/lustre-client-dkms-2.9.0-1.el7.src.rpm
yum install openssl-devel
rpmbuild --rebuild --without servers lustre-2.9.0-1.src.rpm
yum localinstall lustre-client-2.9.0-1.el7.centos.x86_64.rpm kmod-lustre-client-2.9.0-1.el7.centos.x86_64.rpm
echo "options lnet networks=tcp1(eth0)" > /etc/modprobe.d/lnet.conf
modprobe lnet

Setting up the client is very straightforward. All clients mount the same file system identified by the MDS. Use the following commands, specifying the IP address of the MDS server:

mkdir -p /var/mnt/lustre
mount -t lustre <mgs ip>@<nid_name>:/<fs name> /var/mnt/lustre

For example:

mount -t lustre 10.201.67.2@tcp1:/lustre /var/mnt/lustre/

Once the mount has been completed, the Lustre file system is ready to use.

Lustre Testing

Install packages.

yum -y install perl-Readonly perl-List-MoreUtils
rpm -ivh /pub/util/iozone-3.414-1.el7.rf.x86_64.rpm

Command line used:

cd /var/mnt/lustre && iozone -l 32 -O -i 0 -i 1 -i 2 -e -+n -r 4K -s 4G

Build rpms

cd /usr/src
git clone git://git.whamcloud.com/fs/lustre-release.git
cd lustre-release
git checkout -b v2.8 v2_8_0
sh ./autogen.sh 
./configure --disable-server --enable-client --with-linux=/lib/modules/`uname -r`/build
make && make rpms

You may need to edit lustre.spec.in to fix missing file errors.

Troubleshooting

Identify disk for UUID:

ls -lh /dev/disk/by-label | grep UUID