Linux Host – OFED Driver 설치 무작정 따라하기
Test Model: DL380 Gen10
1. Set WP w/ UEFI boot mode
a. Restore Set Default
b. Set WP to HPC.
2. Install RHEL 8.7 - Server with GUI
3. prerequisite
# mkdir -p /media/odd
# mount -t iso9660 -o loop /tmp/RHEL-8.7.0-20221013.1-x86_64-dvd1.iso /media/odd
or # mount -o loop /dev/cdrom /media/odd
# vim /etc/yum.repos.d/odd.repo
[RHEL8DVD_BaseOS]
name=RHEL8DVD_BaseOS
baseurl=file:///media/odd/BaseOS/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
[RHEL8DVD_AppStream]
name=RHEL8DVD_AppStream
baseurl=file:///media/odd/AppStream/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
# dnf -y groupinstall "Development Tools"
# dnf -y install kernel-devel-$(uname -r) kernel-headers-$(uname -r)
# dnf -y install createrepo python36-devel kernel-rpm-macros gcc-gfortran tk kernel-modules-extra tcl mstflint
방화벽 비활성화:
# systemctl status firewalld
# systemctl stop firewalld
4. Install OFED
https://docs.nvidia.com/networking/display/MLNXOFEDv461000/Installing+Mellanox+OFED
# tar -xvzf MLNX_OFED_LINUX-23.04-1.1.3.0-rhel8.7-x86_64.tgz
# cd MLNX_OFED_LINUX-23.04-1.1.3.0-rhel8.7-x86_64
For unattended installation, use the --force installation option while running the MLNX_OFED installation script:
# ./mlnxofedinstall --add-kernel-support --force
# dracut -f
# reboot
5. IB 장치 동작 확인
# lspci |grep mellanox -i
86:00.0 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]
86:00.1 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]
# /etc/init.d/openibd status
HCA driver loaded
Configured IPoIB devices:
ib0 ib1
Currently active IPoIB devices:
ib0
ib1
Configured Mellanox EN devices:
Currently active Mellanox devices:
ib0
ib1
The following OFED modules are loaded:
rdma_ucm
rdma_cm
ib_ipoib
mlx5_core
mlx5_ib
ib_uverbs
ib_umad
ib_cm
ib_core
mlxfw
# ip s a
or # ifconfig -a
# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 94:40:c9:ee:93:c0 brd ff:ff:ff:ff:ff:ff
altname enp2s0f0
inet 192.168.0.21/21 brd 10.55.175.255 scope global noprefixroute eno1
valid_lft forever preferred_lft forever
inet6 fe80::9640:c9ff:feee:93c0/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: eno2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether 94:40:c9:ee:93:c1 brd ff:ff:ff:ff:ff:ff
altname enp2s0f1
4: eno3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether 94:40:c9:ee:93:c2 brd ff:ff:ff:ff:ff:ff
altname enp2s0f2
5: eno4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether 94:40:c9:ee:93:c3 brd ff:ff:ff:ff:ff:ff
altname enp2s0f3
6: enp1s0f4u4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
link/ether 36:54:80:71:93:d8 brd ff:ff:ff:ff:ff:ff
7: ib0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN group default qlen 256
link/infiniband 00:00:11:07:fe:80:00:00:00:00:00:00:94:40:c9:ff:ff:a5:75:a0 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
8: ib1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN group default qlen 256
link/infiniband 00:00:19:07:fe:80:00:00:00:00:00:00:94:40:c9:ff:ff:a5:75:a1 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
9: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 52:54:00:c9:a6:17 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
or # nmcli
# ibdev2netdev
mlx5_0 port 1 ==> ib0 (Up)
mlx5_1 port 1 ==> ib1 (Down)
# ibhosts
# iblinkinfo
# ibstat
# sminfo
# ibnetdiscover
#
