Lustre Install Notes

From Wiki
Jump to: navigation, search

Create a mirror

cd /pub/lustre
wget -r -np -nH

Cluster Setup

Note: SELinux must be *disabled* on server nodes in order for Lustre to work
sed -i -e "s/enabled/disabled/" /etc/sysconfig/selinux && reboot

Install packages

CentOS 7.4

cd /etc/yum.repos.d
curl -O
curl -O
yum update && yum -y install lustre sg3_utils openmpi

For benchmarks install the test package:

yum -y install lustre-tests 

Reboot into the lustre kernel:


Create File System

on management server:

mkfs.lustre --fsname=lustre --mgs /dev/sdb1

on metadata server:

mkfs.lustre --fsname=lustre --mdt --mgsnode=${mgs}@tcp1 --reformat /dev/sdb1

on object server:

mkfs.lustre --ost --fsname=lustre --mgsnode=${mgs}@tcp1 --index=$n --reformat /dev/sdb

mount FS:

mkdir -p /var/mnt/ostnum
mount -t lustre /dev/sdb /var/mnt/ostnum

Install Lustre Client

cd /etc/yum.repos.d
curl -O
yum -y install lustre-client

Packages can also be built from the SRPM.

cd /usr/src
yum install openssl-devel
rpmbuild --rebuild --without servers lustre-2.9.0-1.src.rpm
yum localinstall lustre-client-2.9.0-1.el7.centos.x86_64.rpm kmod-lustre-client-2.9.0-1.el7.centos.x86_64.rpm
echo "options lnet networks=tcp1(eth0)" > /etc/modprobe.d/lnet.conf
modprobe lnet

Setting up the client is very straightforward. All clients mount the same file system identified by the MDS. Use the following commands, specifying the IP address of the MDS server:

mkdir -p /var/mnt/lustre
mount -t lustre <mgs ip>@<nid_name>:/<fs name> /var/mnt/lustre

For example:

mount -t lustre /var/mnt/lustre/

Once the mount has been completed, the Lustre file system is ready to use.

Lustre Testing

Install packages.

yum -y install perl-Readonly perl-List-MoreUtils
rpm -ivh /pub/util/iozone-3.414-1.el7.rf.x86_64.rpm

Command line used:

cd /var/mnt/lustre && iozone -l 32 -O -i 0 -i 1 -i 2 -e -+n -r 4K -s 4G

Build rpms

cd /usr/src
git clone git://
cd lustre-release
git checkout -b v2.8 v2_8_0
sh ./ 
./configure --disable-server --enable-client --with-linux=/lib/modules/`uname -r`/build
make && make rpms

You may need to edit to fix missing file errors.


Identify disk for UUID:

ls -lh /dev/disk/by-label | grep UUID