정보와 지식의 올바른 공유 - http://infoages.com

ProLiant G5 LFF 시스템에 12Gb/s SAS Disk 장착 시 비정상 동작

증상
ProLiant G5 시스템의 Disk를 교체 시, HPE에서 제공하는 자동 대체 되는 부품(12Gb/s)을 사용하는 경우,
- 교체 후, 신규 Disk가 정상 인식되지 않고 장애 상태로 지속 표시됨 (동일 슬롯 또는 다른 슬롯으로 이동해도 동일)
 
원인
G5 세대 Disk backplane board에서 12Gb/s Disk와 호환 문제로 Disk를 정상적으로 동작시키지 못함

환경
• ProLiant G5 System에서 아래 용량의 Disk를 사용 중, Disk 교체가 필요한 경우,
Affected Servers: (이 외, G5 이하 System에서 Disk 교체 후, 장애 상태가 지속되는 경우, 적용 될 수 있음)
HPE ProLiant ML350 G5 / ML310 G5p / ML150 G3

솔루션
G5 System에 한하여,
기존 사용 부품의 용량 대비, 일반 대체되는 부품(73757x-001, 12Gb/s)의 용량을 기준으로,

아래 파트(8682xx-001, 6Gb/s)로 대체하여 사용 한다.

New ProLiant G5-only spare parts:
•868219-001 SPS-DRV HD G5 300GB 6G 15K 3.5 SAS DP HP / 300GB 이하
•868221-001 SPS-DRV HD G5 450GB 6G 15K 3.5 SAS DP HP / 450GB
•868222-001 SPS-DRV HD G5 600GB 6G 15K 3.5 SAS DP HP / 600GB

iLO 4의 fw 2.5x를 사용하는 Gen8/Gen9 ProLiant Server에 2 port HBA의 한 개 포트만 연결한 경우 장애 상태로 표시될 수 있음

증상

1. 2 포트 HBA의 한 개 포트만 연결된 경우,

 iLO의 System Information – Health Summary Page에서 네트워크 항목이 “Degraded”로 표시됨

2. 동일 조건에서, 두 개 포트가 모두 연결된 경우, 정상 상태로 표시됨

3. HBA 상세 Page에서 한 개 포트만 연결된 HBA에 “Degraded”로 표시됨

4. 동일 조건에서 두 개 포트가 모두 연결된 HBA에 정상으로 표시됨

환경

·    Server : ProLiant Gen8/Gen9

·    iLO4 fw : 2.5x

Gen10/SN1600Q HBA fw를 RHEL 6.9에서 관리하기

Test System
DL380 Gen10 + RHEL 6.9
- SN1600Q 1p x 2ea
- SN1600Q 2p x 1ea

1. 설치 후 상태 확인
--------------------------------------------------------------------------------------
# dmesg |egrep 'SN1600Q'
qla2xxx [0000:11:00.0]-00fb:0: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:12:00.0]-00fb:1: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:61:00.0]-00fb:2: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
qla2xxx [0000:61:00.1]-00fb:3: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
--------------------------------------------------------------------------------------
# dmesg |egrep 'qla2xxx'
qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.07.00.26.06.8-k.

qla2xxx [0000:11:00.0]-00fb:0: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:11:00.0]-00fc:0: ISP2261: PCIe (8.0GT/s x8) @ 0000:11:00.0 hdma+ host#=0 fw=8.07.18 (d0d5).

qla2xxx [0000:12:00.0]-00fb:1: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:12:00.0]-00fc:1: ISP2261: PCIe (8.0GT/s x8) @ 0000:12:00.0 hdma+ host#=1 fw=8.07.18 (d0d5).

qla2xxx [0000:61:00.0]-00fb:2: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
qla2xxx [0000:61:00.0]-00fc:2: ISP2261: PCIe (8.0GT/s x8) @ 0000:61:00.0 hdma+ host#=2 fw=8.03.10 (d0d5).
qla2xxx [0000:61:00.1]-00fb:3: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
qla2xxx [0000:61:00.1]-00fc:3: ISP2261: PCIe (8.0GT/s x8) @ 0000:61:00.1 hdma+ host#=3 fw=8.03.10 (d0d5).
--------------------------------------------------------------------------------------

2. SPP2018.03을 통해 fw downgrade, offline interactive mode 를 이용.

--------------------------------------------------------------------------------------
# dmesg |egrep 'qla2xxx'
qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.07.00.26.06.8-k.

qla2xxx [0000:11:00.0]-00fb:0: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:11:00.0]-00fc:0: ISP2261: PCIe (8.0GT/s x8) @ 0000:11:00.0 hdma+ host#=0 fw=8.05.65 (d0d5).

qla2xxx [0000:12:00.0]-00fb:1: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:12:00.0]-00fc:1: ISP2261: PCIe (8.0GT/s x8) @ 0000:12:00.0 hdma+ host#=1 fw=8.07.18 (d0d5).

qla2xxx [0000:61:00.0]-00fb:2: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
qla2xxx [0000:61:00.0]-00fc:2: ISP2261: PCIe (8.0GT/s x8) @ 0000:61:00.0 hdma+ host#=2 fw=8.05.65 (d0d5).
qla2xxx [0000:61:00.1]-00fb:3: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
qla2xxx [0000:61:00.1]-00fc:3: ISP2261: PCIe (8.0GT/s x8) @ 0000:61:00.1 hdma+ host#=3 fw=8.05.65 (d0d5).
--------------------------------------------------------------------------------------

!_! 한 개 포트가 fw가 정상 적용되지 않음을 확인

SPP를 통해 확인해 보니 적용은 된 것처럼 보이나 로드(Activate)가 정상적으로 안된 상태로 판단
SPP2018.03을 통해 fw 를 offline interactive mode 를 이용해서 다시한 번 재적용 후 모두 정상 적용된 것을 확인

--------------------------------------------------------------------------------------
# dmesg |egrep 'qla2xxx'
qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.07.00.26.06.8-k.

qla2xxx [0000:11:00.0]-00fb:0: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:11:00.0]-00fc:0: ISP2261: PCIe (8.0GT/s x8) @ 0000:11:00.0 hdma+ host#=0 fw=8.05.65 (d0d5).

qla2xxx [0000:12:00.0]-00fb:1: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:12:00.0]-00fc:1: ISP2261: PCIe (8.0GT/s x8) @ 0000:12:00.0 hdma+ host#=1 fw=8.05.65 (d0d5).

qla2xxx [0000:61:00.0]-00fb:2: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
qla2xxx [0000:61:00.0]-00fc:2: ISP2261: PCIe (8.0GT/s x8) @ 0000:61:00.0 hdma+ host#=2 fw=8.05.65 (d0d5).
qla2xxx [0000:61:00.1]-00fb:3: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
qla2xxx [0000:61:00.1]-00fc:3: ISP2261: PCIe (8.0GT/s x8) @ 0000:61:00.1 hdma+ host#=3 fw=8.05.65 (d0d5).
--------------------------------------------------------------------------------------

4. Driver Upgrade - smart component "kmod-qlgc-qla2xxx-8.08.00.08.06.0_k1-1.rhel6u9.x86_64.rpm" 이용

--------------------------------------------------------------------------------------
# dmesg |egrep 'qla2xxx'
qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.08.00.08.06.0-k1.
--------------------------------------------------------------------------------------

5. fw Upgrade - smart component
--------------------------------------------------------------------------------------
# rpm -ivh firmware-fc-qlogic-2018.09.01-1.1.x86_64.rpm
warning: firmware-fc-qlogic-2018.09.01-1.1.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 26c2b797: NOKEY
Preparing...                ########################################### [100%]
   1:firmware-fc-qlogic     ########################################### [100%]
# cd /usr/lib/x86_64-linux-gnu/firmware-fc-qlogic-2018.09.01-1.1/
# ls
adapters.properties  CP036356.xml  hpqlafwupdate             hpqlafwupdate.txt  payload.json  reference.xml  x103Cx1939  x1590x00FA
bk017085.bin         hld60159.bin  hpqlafwupdate.properties  hpsetup            Q25A8377.BIN  setup          x1590x00F9
# ./setup
---------------------------------------------------
Loading references...done.
Performing discovery...
WARNING! : No supported adapters found.
./.setup
---------------------------------------------------
Loading references...done.
Performing discovery...
WARNING! : No supported adapters found.

!_! 설치 가이드를 수행해도 정상적으로 적용이 되지 않는다. (전혀 안되는 거 같은데 ㅡ_ㅡ;)

# ./hpqlafwupdate
Validating Flash Image File... Success
Updating Flash on HBA port(s) -  SN1600Q. Please wait...
Updating Adapter FC Boot Code... Success
Updating Adapter Extended MPI Fw ... Success
Updating Adapter MPI Fw ... Success
Updating Adapter PEP Fw... Success
Updating Adapter FC RISC Fw... Success
Updating Adapter Primary Image Status... Success
Updating Adapter Secondary Image Status... Success
Updating Adapter Flash Memo Block... Success
Updating Adapter FC VPD 0... Success
Updating Adapter FC VPD 1... Success
Flash update complete. Changes have been saved to all ports of this HBA.
You must reboot in order for the changes to become effective.
Info: HBA 3 - SN1600Q does not need to be upgraded.
Validating Flash Image File... Success
Updating Flash on HBA instance 0 -  SN1600Q. Please wait...
Updating Adapter FC Boot Code... Success
Updating Adapter Extended MPI Fw ... Success
Updating Adapter MPI Fw ... Success
Updating Adapter PEP Fw... Success
Updating Adapter FC RISC Fw... Success
Updating Adapter Primary Image Status... Success
Updating Adapter Secondary Image Status... Success
Updating Adapter Flash Memo Block... Success
Updating Adapter FC VPD 0... Success
Updating Adapter FC VPD 1... Success
Flash update complete. Changes have been saved to all ports of this HBA.
You must reboot in order for the changes to become effective.
Validating Flash Image File... Success
Updating Flash on HBA instance 1 -  SN1600Q. Please wait...
Updating Adapter FC Boot Code... Success
Updating Adapter Extended MPI Fw ... Success
Updating Adapter MPI Fw ... Success
Updating Adapter PEP Fw... Success
Updating Adapter FC RISC Fw... Success
Updating Adapter Secondary Image Status... Success
Updating Adapter Primary Image Status... Success
Updating Adapter Flash Memo Block... Success
Updating Adapter FC VPD 0... Success
Updating Adapter FC VPD 1... Success
Flash update complete. Changes have been saved to all ports of this HBA.
You must reboot in order for the changes to become effective.
Info: Updates completed successfully (3 out of 3).
--------------------------------------------------------------------------------------

설치가 수행되고, 적용이 되었음을 확인
--------------------------------------------------------------------------------------
# dmesg |egrep 'qla2xxx'
qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.08.00.08.06.0-k1.
qla2xxx 0000:11:00.0: PCI INT A -> GSI 32 (level, low) -> IRQ 32
qla2xxx [0000:11:00.0]-00fb:0: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:11:00.0]-00fc:0: ISP2261: PCIe (8.0GT/s x8) @ 0000:11:00.0 hdma+ host#=0 fw=8.07.18 (d0d5).

qla2xxx [0000:12:00.0]-00fb:1: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:12:00.0]-00fc:1: ISP2261: PCIe (8.0GT/s x8) @ 0000:12:00.0 hdma+ host#=1 fw=8.07.18 (d0d5).

qla2xxx [0000:61:00.0]-00fb:2: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
qla2xxx [0000:61:00.0]-00fc:2: ISP2261: PCIe (8.0GT/s x4) @ 0000:61:00.0 hdma+ host#=2 fw=8.07.18 (d0d5).
qla2xxx [0000:61:00.1]-00fb:3: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
qla2xxx [0000:61:00.1]-00fc:3: ISP2261: PCIe (8.0GT/s x4) @ 0000:61:00.1 hdma+ host#=3 fw=8.07.18 (d0d5).
--------------------------------------------------------------------------------------

6. fw downgrade - smart component
--------------------------------------------------------------------------------------
# rpm -ivh firmware-fc-qlogic-2018.02.01-1.14.x86_64.rpm
warning: firmware-fc-qlogic-2018.02.01-1.14.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 26c2b797: NOKEY
Preparing...                ########################################### [100%]
   1:firmware-fc-qlogic     ########################################### [100%]
# cd /usr/lib/x86_64-linux-gnu/firmware-fc-qlogic-2018.02.01-1.14/
# ls
adapters.properties  CP032879.xml  hpqlafwupdate             hpqlafwupdate.txt  payload.json  reference.xml  x103Cx1939  x1590x00FA
bk017066.bin         hld60150.bin  hpqlafwupdate.properties  hpsetup            Q25A3377.BIN  setup          x1590x00F9
# ./setup
---------------------------------------------------
Loading references...done.
Performing discovery...
WARNING! : No supported adapters found.
# ./hpsetup
---------------------------------------------------
Loading references...done.
Performing discovery...
WARNING! : No supported adapters found.
# ./hpqlafwupdate
Info: HBA 2 - SN1600Q does not need to be upgraded.
Info: HBA 3 - SN1600Q does not need to be upgraded.
Info: HBA 0 - SN1600Q does not need to be upgraded.
Info: HBA 1 - SN1600Q does not need to be upgraded.
Info: No updates required.
# ./hpqlafwupdate -f
Validating Flash Image File... Success
Updating Flash on HBA port(s) -  SN1600Q. Please wait...
Updating Adapter FC Boot Code... Success
Updating Adapter Extended MPI Fw ... Success
Updating Adapter MPI Fw ... Success
Updating Adapter PEP Fw... Success
Updating Adapter FC RISC Fw... Success
Updating Adapter Secondary Image Status... Success
Updating Adapter Primary Image Status... Success
Updating Adapter Flash Memo Block... Success
Updating Adapter FC VPD 0... Success
Updating Adapter FC VPD 1... Success
Flash update complete. Changes have been saved to all ports of this HBA.
You must reboot in order for the changes to become effective.
Info: HBA instance 3 - SN1600Q does not need to be upgraded.
Validating Flash Image File... Success
Updating Flash on HBA instance 0 -  SN1600Q. Please wait...
Updating Adapter FC Boot Code... Success
Updating Adapter Extended MPI Fw ... Success
Updating Adapter MPI Fw ... Success
Updating Adapter PEP Fw... Success
Updating Adapter FC RISC Fw... Success
Updating Adapter Secondary Image Status... Success
Updating Adapter Primary Image Status... Success
Updating Adapter Flash Memo Block... Success
Updating Adapter FC VPD 0... Success
Updating Adapter FC VPD 1... Success
Flash update complete. Changes have been saved to all ports of this HBA.
You must reboot in order for the changes to become effective.
Validating Flash Image File... Success
Updating Flash on HBA instance 1 -  SN1600Q. Please wait...
Updating Adapter FC Boot Code... Success
Updating Adapter Extended MPI Fw ... Success
Updating Adapter MPI Fw ... Success
Updating Adapter PEP Fw... Success
Updating Adapter FC RISC Fw... Success
Updating Adapter Primary Image Status... Success
Updating Adapter Secondary Image Status... Success
Updating Adapter Flash Memo Block... Success
Updating Adapter FC VPD 0... Success
Updating Adapter FC VPD 1... Success
Flash update complete. Changes have been saved to all ports of this HBA.
You must reboot in order for the changes to become effective.
Info: Updates completed successfully (4 out of 4).
--------------------------------------------------------------------------------------

--------------------------------------------------------------------------------------
# dmesg |egrep 'qla2xxx'
qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.08.00.08.06.0-k1.

qla2xxx [0000:11:00.0]-00fb:0: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:11:00.0]-00fc:0: ISP2261: PCIe (8.0GT/s x8) @ 0000:11:00.0 hdma+ host#=0 fw=8.05.65 (d0d5).

qla2xxx [0000:12:00.0]-00fb:1: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:12:00.0]-00fc:1: ISP2261: PCIe (8.0GT/s x8) @ 0000:12:00.0 hdma+ host#=1 fw=8.05.65 (d0d5).

qla2xxx [0000:61:00.0]-00fb:2: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
qla2xxx [0000:61:00.0]-00fc:2: ISP2261: PCIe (8.0GT/s x4) @ 0000:61:00.0 hdma+ host#=2 fw=8.05.65 (d0d5).
qla2xxx [0000:61:00.1]-00fb:3: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
qla2xxx [0000:61:00.1]-00fc:3: ISP2261: PCIe (8.0GT/s x4) @ 0000:61:00.1 hdma+ host#=3 fw=8.05.65 (d0d5).
--------------------------------------------------------------------------------------

7. Install fc fw via SPP2018.09 offline interactive mode

--------------------------------------------------------------------------------------
[root@DL380G10C42U11 ~]# dmesg |egrep 'qla2xxx'
qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.08.00.08.06.0-k1.

qla2xxx [0000:11:00.0]-00fb:0: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:11:00.0]-00fc:0: ISP2261: PCIe (8.0GT/s x8) @ 0000:11:00.0 hdma+ host#=0 fw=8.07.18 (d0d5).

qla2xxx [0000:12:00.0]-00fb:1: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:12:00.0]-00fc:1: ISP2261: PCIe (8.0GT/s x8) @ 0000:12:00.0 hdma+ host#=1 fw=8.07.18 (d0d5).

qla2xxx [0000:61:00.0]-00fb:2: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
qla2xxx [0000:61:00.0]-00fc:2: ISP2261: PCIe (8.0GT/s x4) @ 0000:61:00.0 hdma+ host#=2 fw=8.07.18 (d0d5).
qla2xxx [0000:61:00.1]-00fb:3: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
qla2xxx [0000:61:00.1]-00fc:3: ISP2261: PCIe (8.0GT/s x4) @ 0000:61:00.1 hdma+ host#=3 fw=8.07.18 (d0d5).
--------------------------------------------------------------------------------------

Intel Xeon E5 v4(Broadwell)시리즈 CPU를 사용하는 ProLiant Gen9 서버의 예기치 않은 시스템 재시작 또는 종료 발생 가능 이슈

증상

아래 중 한 개 이상의 정보가 확인됨

시나리오 1.

연관된 로그(footprint)정보가 남지 않고, 예기치 않은 시스템 재시작 또는 종료가 발생됨

시나리오 2.

예기치 않은 시스템 재시작 또는 종료가 발생된 후 리부팅 중, Smart Array 관련 lockup 0x12가 기록됨

시나리오 3.

예기치 않은 시스템 재시작 또는 종료가 발생된 후 아래 형태의 UMCE 코드가 IML상에 확인됨

Uncorrectable Machine Check Exception (Board 0, Processor 2, APIC ID 0x00000040, Bank 0x00000004, Status 0xBA000000'73000402, Address 0x00000000'00000000, Misc 0x00000000'00000000)

노란색 마크된 정보가 일치된 형태로 확인되며, 그 외 형태의 UMCE 이벤트가 함께 나타날 수 있음.

시나리오 2.와 함께 나타날 수 있음.

원인

CPU의 일시적인 내부 오동작으로, 시스템 내부에서 IO 요청(많은 경우 PCI 확장 장치의 요청)이 정상 처리되지 못해,

해당 요청이 비정상 종료(timeout)됨

이 현상은 치명적 오류로 시스템 리부팅을 유발하게 되는데, 리부팅 과정에서 Smart Array가 lockup(응답 지연으로 인한 IO 처리 실패)을 경험할 수 있음 (lockup 0x12 기록)

HW 로그상에 Smart Array의 오류로 기록되지만(파생된 이벤트), 이슈 촉발의 원인이 Smart Array의 장애가 아님

(Smart Array 교체 필요치 않음)

환경

·    Intel Xeon E5 v4 Series CPU를 사용하는 ProLiant Gen9 System

- E5-2600 v4 Series Intel Processor

- E5-4600 v4 Series Intel Processor

솔루션

Action Plan 1.

What: Upgrade BIOS to v2.64

Problems Fixed:

This revision of the System ROM includes the latest revision of the Intel microcode which addresses an issue where the system could experience an unexpected reset or shutdown with no errors logged to the Integrated Management Log (IML). This issue only impacts systems configured with Intel Xeon 2600 v4 series processors. This issue is not unique to HPE servers.

This revision of the System ROM includes the latest revision of the Intel microcode which addresses an issue where the system could experience an unexpected reset or shutdown with no errors logged to the Integrated Management Log (IML) when processor C-states are enabled. This issue does not exist when processor C-states are disabled. This issue is not unique to HPE servers.

참조 Advisory ------------------------------------------

Advisory: ProLiant Gen9 Series Servers - Unexpected Reset or Shutdown May Occur on ProLiant Gen9 Servers

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00060570en_us

-----------------------------------------------------------

NVMe Disk의 fw를 nvme-cli 툴을 통해 업그레이드 해보자.

사전작업:

- nvme-cli를 의존 패키지와 개별 설치하거나, yum을 통해 설치하고자 한다면, yum repository가 구성되어 있어야 한다.

- 설치하고자 하는 NVMe disk의 fw 파일을 HPE 웹사이트에서 다운로드 하여, 설치한다.

   (fw rpm package installed before. / rpm -ivh fw-name.rpm)

-- fw image는 패키지 설치 후 "/usr/lib/x86_64-linux-gnu/firmware-hdd-*/*.rel" 형태로 확인되게 됨

1. Install nvme-cli package.
# yum install nvme*
<snip>
===================================================================================================================================================================
Package                                     Arch                            Version                                      Repository                          Size
===================================================================================================================================================================
Installing:
nvme-cli                                    x86_64                          1.4-3.el7                                    RHEL7-DVD                          190 k
nvmetcli                                    noarch                          0.5-1.el7                                    RHEL7-DVD                           35 k
Installing for dependencies:
pyparsing                                   noarch                          1.5.6-9.el7                                  RHEL7-DVD                           94 k
python-configshell                          noarch                          1:1.1.fb23-3.el7                         RHEL7-DVD                           67 k
python-kmod                                 x86_64                          0.9-4.el7                                    RHEL7-DVD                           57 k
python-urwid                                x86_64                          1.1.1-3.el7                                  RHEL7-DVD                          654 k

요런 패키지들이 설치 됨.

2. Check NVMe device(s).
# nvme list
Node             SN                   Model                                    Namespace Usage                      Format           FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------

/dev/nvme0n1     <% NVMe disk SN %>   MO001600KWJSN               1           1.60  TB /   1.60  TB      4 KiB +  0 B   4ICDHPK0

3. fw download to NVMe drive
nvme fw-download <device> [--fw=<firmware-file> | -f <firmware-file>]

                                              [--xfer=<transfer-size> | -x <transfer-size>] [--offset=<offset> | -o <offset>]

# nvme fw-download /dev/nvme0n1 -f /usr/lib/x86_64-linux-gnu/firmware-hdd-1656c1b14a-HPK1-1.1/4ICDHPK1.rel
Firmware download success

4. activate fw to target device.
nvme fw-activate <device> [--slot=<slot> | -s <slot>] [--action=<action> | -a <action>]

# nvme fw-activate /dev/nvme0n1 -s 1 -a 1
Success activating firmware action:1 slot:1

5. reset for loading fw.
# nvme reset /dev/nvme0

6. confirm fw version.
# nvme list
Node             SN                   Model                                    Namespace Usage                      Format           FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1     <% NVMe disk SN %>   MO001600KWJSN               1           1.60  TB /   1.60  TB      4 KiB +  0 B   4ICDHPK1

#