NVMe Disk를 시스템에서 추가 또는 제거하기
- ProLiant의 경우 Hot Add (Online 상에서 신규 Disk 추가)는 현 기준 지원하지 않으니 주의
- 기존 Disk를 교체하거나, 제거하는 작업에 참조
Test 환경
- DL380 Gen9 (RHEL 7.5)
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 558.7G 0 disk
├─sda1 8:1 0 200M 0 part /boot/efi
├─sda2 8:2 0 1G 0 part /boot
└─sda3 8:3 0 557.5G 0 part
├─rhel-root 253:0 0 50G 0 lvm /
└─rhel-home 253:2 0 3.4T 0 lvm /home
sr0 11:0 1 4.3G 0 rom
nvme0n1 259:0 0 1.5T 0 disk
└─nvme0n1p1 259:1 0 1.5T 0 part
├─rhel-swap 253:1 0 15.7G 0 lvm [SWAP]
└─rhel-home 253:2 0 3.4T 0 lvm /home
nvme1n1 259:2 0 1.5T 0 disk
└─nvme1n1p1 259:3 0 1.5T 0 part
└─rhel-home 253:2 0 3.4T 0 lvm /home
2개의 NVMe를 장착한 상태에서, RHEL을 설치한 상태
NVMe의 전원 제어를 위해 OS의 커널 파라메터를 추가
# vim /etc/default/grub
add below line's tail - "pci=pcie_bus_perf"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet pci=pcie_bus_perf"
# grub2-mkconfig -o /boot/grub2/grub.cfg
# grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg
변경된 GRUB을 적용 후 리부팅.
시스템의 NVMe를 검색
# find /sys/devices |egrep 'nvme[0-9][0-9]?$'
e.g.)
/sys/devices/pci0000:00/0000:00:03.0/0000:08:00.0/0000:09:09.0/0000:0b:00.0/nvme/nvme0
/sys/devices/pci0000:00/0000:00:03.0/0000:08:00.0/0000:09:0a.0/0000:0c:00.0/nvme/nvme1
전원 제어를 위한 bus 정보 확인
# egrep -H '.*' /sys/bus/pci/slots/*/address
/sys/bus/pci/slots/0-1/address:0000:0c:00
/sys/bus/pci/slots/0-2/address:0000:0d:00
/sys/bus/pci/slots/0-3/address:0000:0e:00
/sys/bus/pci/slots/0-4/address:0000:0f:00
/sys/bus/pci/slots/0-5/address:0000:10:00
/sys/bus/pci/slots/0/address:0000:0b:00
/sys/bus/pci/slots/1/address:0000:08:00
/sys/bus/pci/slots/2/address:0000:05:00
# grep '0b:00' /sys/bus/pci/slots/*/address
/sys/bus/pci/slots/0/address:0000:0b:00
# grep '0c:00' /sys/bus/pci/slots/*/address
/sys/bus/pci/slots/0-1/address:0000:0c:00
IO 중인 자료를 저장
# blockdev --flushbufs /dev/nvme1n1
NVMe 전원을 종료
# echo 0 > /sys/bus/pci/slots/0-1/power
# lspci | grep -i "non-vol"
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 558.7G 0 disk
├─sda1 8:1 0 200M 0 part /boot/efi
├─sda2 8:2 0 1G 0 part /boot
└─sda3 8:3 0 557.5G 0 part
├─rhel-root 253:0 0 50G 0 lvm /
└─rhel-home 253:2 0 3.4T 0 lvm /home
sr0 11:0 1 4.3G 0 rom
nvme0n1 259:0 0 1.5T 0 disk
└─nvme0n1p1 259:1 0 1.5T 0 part
├─rhel-swap 253:1 0 15.7G 0 lvm [SWAP]
└─rhel-home 253:2 0 3.4T 0 lvm /home
NVMe 전원을 인입
# echo 1 > /sys/bus/pci/slots/0-1/power
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 558.7G 0 disk
├─sda1 8:1 0 200M 0 part /boot/efi
├─sda2 8:2 0 1G 0 part /boot
└─sda3 8:3 0 557.5G 0 part
├─rhel-root 253:0 0 50G 0 lvm /
└─rhel-home 253:2 0 3.4T 0 lvm /home
sr0 11:0 1 4.3G 0 rom
nvme0n1 259:0 0 1.5T 0 disk
└─nvme0n1p1 259:1 0 1.5T 0 part
├─rhel-swap 253:1 0 15.7G 0 lvm [SWAP]
└─rhel-home 253:2 0 3.4T 0 lvm /home
nvme2n1 259:4 0 1.5T 0 disk
└─nvme2n1p1 259:5 0 1.5T 0 part
└─rhel-home 253:2 0 3.4T 0 lvm /home
추가 참조 작업 - Disk 추가 후 사용을 위해 ...
기존 구성에 SW RAID 사용 환경이라면 NVMe를 제거하고, 다시 추가하는 명령
---------------------------------
# cat /proc/mdstat
# mdadm --manage /dev/md0 --fail /dev/nvme1n1
# mdadm --manage /dev/md0 --add /dev/nvme1n1
---------------------------------
신규 Disk를 단일 볼륨(JBOD) 등으로 장착하는 경우,
New Disk ------------------------
# fdisk /dev/nvme1n1
a) 'n' new partition
b) 'p' partition
c) '1' 1st
d) 'w' save
# mkfs -t ext4 /dev/nvme1n1p1
or
# mkfs.xfs /dev/nvme1n1p1
# mount /dev/nvme1n1p1 /mnt/
---------------------------------