Infiniband is a switched fabric communication link used in HPC environments and data centers. It features high throughput and low latency and currently popular data rate is 40 Gbps (unidirectional). So it must cost lots of money to build anything with Infiniband. That is not exactly true if you can use used or refurbished equipment.

I have chosen Openfiler as it features almost all SAN/NAS functions with simple management interface (and it features built in IETD for iSCSI which is the best performing iSCSI target implementation under Linux). Secondly it supports IB out of the box (and so does CentOS/RHEL 6.x) :)

All you need are:

  • two Infiniband cards – available on ebay for about 60 Euros piece (4x DDR which is about 16 Gbps) – I bought two Mellanox ConnectX 4xDDR cards dual ports for 75 Euros each
  • CX4 (SFF8470 connection on both ends) cable – also available on ebay also for about 60 Euros
  • two standard boxes equipped with at least one 8x PCI-e slot or 16x

I will not cover the installation of Openfiler which is pretty much strait forward. The most important thing is IB setup after installation – this has to be done through CLI. So lets get going.

Firstly check if the cards are recognized by the OS. You can do it with lspci -k, which will also tell you which module is used by the card. My mellanox cards were using  mlx4_core driver. You also need to load:

mlx4_ib
ib_umad
ib_ipoib
ib_uverbs

You can put it at the end of /etc/rc.local file:

/sbin/modprobe mlx4_ib
/sbin/modprobe ib_umad
/sbin/modprobe ib_ipoib
/sbin/modprobe ib_uverbs

The second thing is the Subnet Manager, which takes care of the Infiniband network (as the name suggests :) ). But for Openfiler (and CentOS/RHEL) the built-in driver creates the devices in /dev instead of /dev/infiniband . We will need to add the proper rules to /etc/udev/rules.d/99-infiniband.rules

KERNEL=="umad*", NAME="infiniband/%k"
KERNEL=="issm*", NAME="infiniband/%k"
KERNEL=="ucm*", NAME="infiniband/%k", MODE="0666"
KERNEL=="uverbs*", NAME="infiniband/%k", MODE="0666"
KERNEL=="ucma", NAME="infiniband/%k", MODE="0666"
KERNEL=="rdma_cm", NAME="infiniband/%k", MODE="0666"

Now you can reboot both machines. After they are back online, run ibstat command – it should print all information about your IB card. Now edit /etc/sysconfig/opensm file and put just one line inside:

GUIDS="all"

Now you can run service opensmd start . If you issue ibstat command you will notice that the status of your card should be Link: up. So you are ready to get the ib intefaces up and running with ifconfig ib0 192.168.1.1/24 (and ifconfig ib0 192.168.1.2/24 on the other machine). A quick ping test should let you know if everything is ok. To test performance of your setup run conary install iperf  and then: iperf -s (on one the 192.168.1.1 box) iperf -c 192.168.1.1 (on 192.168.1.2 box) This will test native IP performance. Mine was as below:

infiniband

Not 16 Gbps if you ask me. We can do some tuning because out-of-the-box MTU for IB cards is about 2KB. To bump it to 64K do the following:

echo connected > /sys/class/net/ib0/mode
ifconfig ib0 mtu 65520

This should bump the performance up by 15%:

infiniband-64mtu

 

And that is just for one connection. If you run iperf -c 192.168.1.1 -P 8 you will get about 12Gbps aggregated throughput. So to sum up: if you are looking to build high performance SAN/NAS for about 200 Euros then try used Infiniband equipment. It is cheaper then Fibre Channel and has better performance :)