Gen10/SN1600Q HBA fw를 RHEL 6.9에서 관리하기

 

Test System
DL380 Gen10 + RHEL 6.9
- SN1600Q 1p x 2ea
- SN1600Q 2p x 1ea

 

1. 설치 후 상태 확인
--------------------------------------------------------------------------------------
# dmesg |egrep 'SN1600Q'
qla2xxx [0000:11:00.0]-00fb:0: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:12:00.0]-00fb:1: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:61:00.0]-00fb:2: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
qla2xxx [0000:61:00.1]-00fb:3: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
--------------------------------------------------------------------------------------
# dmesg |egrep 'qla2xxx'
qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.07.00.26.06.8-k.

qla2xxx [0000:11:00.0]-00fb:0: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:11:00.0]-00fc:0: ISP2261: PCIe (8.0GT/s x8) @ 0000:11:00.0 hdma+ host#=0 fw=8.07.18 (d0d5).

qla2xxx [0000:12:00.0]-00fb:1: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:12:00.0]-00fc:1: ISP2261: PCIe (8.0GT/s x8) @ 0000:12:00.0 hdma+ host#=1 fw=8.07.18 (d0d5).

qla2xxx [0000:61:00.0]-00fb:2: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
qla2xxx [0000:61:00.0]-00fc:2: ISP2261: PCIe (8.0GT/s x8) @ 0000:61:00.0 hdma+ host#=2 fw=8.03.10 (d0d5).
qla2xxx [0000:61:00.1]-00fb:3: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
qla2xxx [0000:61:00.1]-00fc:3: ISP2261: PCIe (8.0GT/s x8) @ 0000:61:00.1 hdma+ host#=3 fw=8.03.10 (d0d5).
--------------------------------------------------------------------------------------

 

2. SPP2018.03을 통해 fw downgrade, offline interactive mode 를 이용.

--------------------------------------------------------------------------------------
# dmesg |egrep 'qla2xxx'
qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.07.00.26.06.8-k.

qla2xxx [0000:11:00.0]-00fb:0: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:11:00.0]-00fc:0: ISP2261: PCIe (8.0GT/s x8) @ 0000:11:00.0 hdma+ host#=0 fw=8.05.65 (d0d5).

qla2xxx [0000:12:00.0]-00fb:1: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:12:00.0]-00fc:1: ISP2261: PCIe (8.0GT/s x8) @ 0000:12:00.0 hdma+ host#=1 fw=8.07.18 (d0d5).

qla2xxx [0000:61:00.0]-00fb:2: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
qla2xxx [0000:61:00.0]-00fc:2: ISP2261: PCIe (8.0GT/s x8) @ 0000:61:00.0 hdma+ host#=2 fw=8.05.65 (d0d5).
qla2xxx [0000:61:00.1]-00fb:3: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
qla2xxx [0000:61:00.1]-00fc:3: ISP2261: PCIe (8.0GT/s x8) @ 0000:61:00.1 hdma+ host#=3 fw=8.05.65 (d0d5).
--------------------------------------------------------------------------------------

!_! 한 개 포트가 fw가 정상 적용되지 않음을 확인

 


SPP를 통해 확인해 보니 적용은 된 것처럼 보이나 로드(Activate)가 정상적으로 안된 상태로 판단
SPP2018.03을 통해 fw 를 offline interactive mode 를 이용해서 다시한 번 재적용 후 모두 정상 적용된 것을 확인

--------------------------------------------------------------------------------------
# dmesg |egrep 'qla2xxx'
qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.07.00.26.06.8-k.

qla2xxx [0000:11:00.0]-00fb:0: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:11:00.0]-00fc:0: ISP2261: PCIe (8.0GT/s x8) @ 0000:11:00.0 hdma+ host#=0 fw=8.05.65 (d0d5).

qla2xxx [0000:12:00.0]-00fb:1: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:12:00.0]-00fc:1: ISP2261: PCIe (8.0GT/s x8) @ 0000:12:00.0 hdma+ host#=1 fw=8.05.65 (d0d5).

qla2xxx [0000:61:00.0]-00fb:2: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
qla2xxx [0000:61:00.0]-00fc:2: ISP2261: PCIe (8.0GT/s x8) @ 0000:61:00.0 hdma+ host#=2 fw=8.05.65 (d0d5).
qla2xxx [0000:61:00.1]-00fb:3: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
qla2xxx [0000:61:00.1]-00fc:3: ISP2261: PCIe (8.0GT/s x8) @ 0000:61:00.1 hdma+ host#=3 fw=8.05.65 (d0d5).
--------------------------------------------------------------------------------------

 

4. Driver Upgrade - smart component "kmod-qlgc-qla2xxx-8.08.00.08.06.0_k1-1.rhel6u9.x86_64.rpm" 이용

--------------------------------------------------------------------------------------
# dmesg |egrep 'qla2xxx'
qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.08.00.08.06.0-k1.
--------------------------------------------------------------------------------------

 

5. fw Upgrade - smart component
--------------------------------------------------------------------------------------
# rpm -ivh firmware-fc-qlogic-2018.09.01-1.1.x86_64.rpm
warning: firmware-fc-qlogic-2018.09.01-1.1.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 26c2b797: NOKEY
Preparing...                ########################################### [100%]
   1:firmware-fc-qlogic     ########################################### [100%]
# cd /usr/lib/x86_64-linux-gnu/firmware-fc-qlogic-2018.09.01-1.1/
# ls
adapters.properties  CP036356.xml  hpqlafwupdate             hpqlafwupdate.txt  payload.json  reference.xml  x103Cx1939  x1590x00FA
bk017085.bin         hld60159.bin  hpqlafwupdate.properties  hpsetup            Q25A8377.BIN  setup          x1590x00F9
# ./setup
---------------------------------------------------
Loading references...done.
Performing discovery...
WARNING! : No supported adapters found.
./.setup
---------------------------------------------------
Loading references...done.
Performing discovery...
WARNING! : No supported adapters found.

 

!_! 설치 가이드를 수행해도 정상적으로 적용이 되지 않는다. (전혀 안되는 거 같은데 ㅡ_ㅡ;)

 

# ./hpqlafwupdate
Validating Flash Image File... Success
Updating Flash on HBA port(s) -  SN1600Q. Please wait...
Updating Adapter FC Boot Code... Success
Updating Adapter Extended MPI Fw ... Success
Updating Adapter MPI Fw ... Success
Updating Adapter PEP Fw... Success
Updating Adapter FC RISC Fw... Success
Updating Adapter Primary Image Status... Success
Updating Adapter Secondary Image Status... Success
Updating Adapter Flash Memo Block... Success
Updating Adapter FC VPD 0... Success
Updating Adapter FC VPD 1... Success
Flash update complete. Changes have been saved to all ports of this HBA.
You must reboot in order for the changes to become effective.
Info: HBA 3 - SN1600Q does not need to be upgraded.
Validating Flash Image File... Success
Updating Flash on HBA instance 0 -  SN1600Q. Please wait...
Updating Adapter FC Boot Code... Success
Updating Adapter Extended MPI Fw ... Success
Updating Adapter MPI Fw ... Success
Updating Adapter PEP Fw... Success
Updating Adapter FC RISC Fw... Success
Updating Adapter Primary Image Status... Success
Updating Adapter Secondary Image Status... Success
Updating Adapter Flash Memo Block... Success
Updating Adapter FC VPD 0... Success
Updating Adapter FC VPD 1... Success
Flash update complete. Changes have been saved to all ports of this HBA.
You must reboot in order for the changes to become effective.
Validating Flash Image File... Success
Updating Flash on HBA instance 1 -  SN1600Q. Please wait...
Updating Adapter FC Boot Code... Success
Updating Adapter Extended MPI Fw ... Success
Updating Adapter MPI Fw ... Success
Updating Adapter PEP Fw... Success
Updating Adapter FC RISC Fw... Success
Updating Adapter Secondary Image Status... Success
Updating Adapter Primary Image Status... Success
Updating Adapter Flash Memo Block... Success
Updating Adapter FC VPD 0... Success
Updating Adapter FC VPD 1... Success
Flash update complete. Changes have been saved to all ports of this HBA.
You must reboot in order for the changes to become effective.
Info: Updates completed successfully (3 out of 3).
--------------------------------------------------------------------------------------

설치가 수행되고, 적용이 되었음을 확인
--------------------------------------------------------------------------------------
# dmesg |egrep 'qla2xxx'
qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.08.00.08.06.0-k1.
qla2xxx 0000:11:00.0: PCI INT A -> GSI 32 (level, low) -> IRQ 32
qla2xxx [0000:11:00.0]-00fb:0: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:11:00.0]-00fc:0: ISP2261: PCIe (8.0GT/s x8) @ 0000:11:00.0 hdma+ host#=0 fw=8.07.18 (d0d5).

qla2xxx [0000:12:00.0]-00fb:1: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:12:00.0]-00fc:1: ISP2261: PCIe (8.0GT/s x8) @ 0000:12:00.0 hdma+ host#=1 fw=8.07.18 (d0d5).

qla2xxx [0000:61:00.0]-00fb:2: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
qla2xxx [0000:61:00.0]-00fc:2: ISP2261: PCIe (8.0GT/s x4) @ 0000:61:00.0 hdma+ host#=2 fw=8.07.18 (d0d5).
qla2xxx [0000:61:00.1]-00fb:3: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
qla2xxx [0000:61:00.1]-00fc:3: ISP2261: PCIe (8.0GT/s x4) @ 0000:61:00.1 hdma+ host#=3 fw=8.07.18 (d0d5).
--------------------------------------------------------------------------------------

 

6. fw downgrade - smart component
--------------------------------------------------------------------------------------
# rpm -ivh firmware-fc-qlogic-2018.02.01-1.14.x86_64.rpm
warning: firmware-fc-qlogic-2018.02.01-1.14.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 26c2b797: NOKEY
Preparing...                ########################################### [100%]
   1:firmware-fc-qlogic     ########################################### [100%]
# cd /usr/lib/x86_64-linux-gnu/firmware-fc-qlogic-2018.02.01-1.14/
# ls
adapters.properties  CP032879.xml  hpqlafwupdate             hpqlafwupdate.txt  payload.json  reference.xml  x103Cx1939  x1590x00FA
bk017066.bin         hld60150.bin  hpqlafwupdate.properties  hpsetup            Q25A3377.BIN  setup          x1590x00F9
# ./setup
---------------------------------------------------
Loading references...done.
Performing discovery...
WARNING! : No supported adapters found.
# ./hpsetup
---------------------------------------------------
Loading references...done.
Performing discovery...
WARNING! : No supported adapters found.
# ./hpqlafwupdate
Info: HBA 2 - SN1600Q does not need to be upgraded.
Info: HBA 3 - SN1600Q does not need to be upgraded.
Info: HBA 0 - SN1600Q does not need to be upgraded.
Info: HBA 1 - SN1600Q does not need to be upgraded.
Info: No updates required.
# ./hpqlafwupdate -f
Validating Flash Image File... Success
Updating Flash on HBA port(s) -  SN1600Q. Please wait...
Updating Adapter FC Boot Code... Success
Updating Adapter Extended MPI Fw ... Success
Updating Adapter MPI Fw ... Success
Updating Adapter PEP Fw... Success
Updating Adapter FC RISC Fw... Success
Updating Adapter Secondary Image Status... Success
Updating Adapter Primary Image Status... Success
Updating Adapter Flash Memo Block... Success
Updating Adapter FC VPD 0... Success
Updating Adapter FC VPD 1... Success
Flash update complete. Changes have been saved to all ports of this HBA.
You must reboot in order for the changes to become effective.
Info: HBA instance 3 - SN1600Q does not need to be upgraded.
Validating Flash Image File... Success
Updating Flash on HBA instance 0 -  SN1600Q. Please wait...
Updating Adapter FC Boot Code... Success
Updating Adapter Extended MPI Fw ... Success
Updating Adapter MPI Fw ... Success
Updating Adapter PEP Fw... Success
Updating Adapter FC RISC Fw... Success
Updating Adapter Secondary Image Status... Success
Updating Adapter Primary Image Status... Success
Updating Adapter Flash Memo Block... Success
Updating Adapter FC VPD 0... Success
Updating Adapter FC VPD 1... Success
Flash update complete. Changes have been saved to all ports of this HBA.
You must reboot in order for the changes to become effective.
Validating Flash Image File... Success
Updating Flash on HBA instance 1 -  SN1600Q. Please wait...
Updating Adapter FC Boot Code... Success
Updating Adapter Extended MPI Fw ... Success
Updating Adapter MPI Fw ... Success
Updating Adapter PEP Fw... Success
Updating Adapter FC RISC Fw... Success
Updating Adapter Primary Image Status... Success
Updating Adapter Secondary Image Status... Success
Updating Adapter Flash Memo Block... Success
Updating Adapter FC VPD 0... Success
Updating Adapter FC VPD 1... Success
Flash update complete. Changes have been saved to all ports of this HBA.
You must reboot in order for the changes to become effective.
Info: Updates completed successfully (4 out of 4).
--------------------------------------------------------------------------------------

--------------------------------------------------------------------------------------
# dmesg |egrep 'qla2xxx'
qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.08.00.08.06.0-k1.

qla2xxx [0000:11:00.0]-00fb:0: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:11:00.0]-00fc:0: ISP2261: PCIe (8.0GT/s x8) @ 0000:11:00.0 hdma+ host#=0 fw=8.05.65 (d0d5).

qla2xxx [0000:12:00.0]-00fb:1: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:12:00.0]-00fc:1: ISP2261: PCIe (8.0GT/s x8) @ 0000:12:00.0 hdma+ host#=1 fw=8.05.65 (d0d5).

qla2xxx [0000:61:00.0]-00fb:2: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
qla2xxx [0000:61:00.0]-00fc:2: ISP2261: PCIe (8.0GT/s x4) @ 0000:61:00.0 hdma+ host#=2 fw=8.05.65 (d0d5).
qla2xxx [0000:61:00.1]-00fb:3: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
qla2xxx [0000:61:00.1]-00fc:3: ISP2261: PCIe (8.0GT/s x4) @ 0000:61:00.1 hdma+ host#=3 fw=8.05.65 (d0d5).
--------------------------------------------------------------------------------------

 

7. Install fc fw via SPP2018.09 offline interactive mode

--------------------------------------------------------------------------------------
[root@DL380G10C42U11 ~]# dmesg |egrep 'qla2xxx'
qla2xxx [0000:00:00.0]-0005: QLogic Fibre Channel HBA Driver: 8.08.00.08.06.0-k1.

qla2xxx [0000:11:00.0]-00fb:0: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:11:00.0]-00fc:0: ISP2261: PCIe (8.0GT/s x8) @ 0000:11:00.0 hdma+ host#=0 fw=8.07.18 (d0d5).

qla2xxx [0000:12:00.0]-00fb:1: QLogic SN1600Q - HPE SN1600Q 32Gb 1p FC HBA.
qla2xxx [0000:12:00.0]-00fc:1: ISP2261: PCIe (8.0GT/s x8) @ 0000:12:00.0 hdma+ host#=1 fw=8.07.18 (d0d5).

qla2xxx [0000:61:00.0]-00fb:2: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
qla2xxx [0000:61:00.0]-00fc:2: ISP2261: PCIe (8.0GT/s x4) @ 0000:61:00.0 hdma+ host#=2 fw=8.07.18 (d0d5).
qla2xxx [0000:61:00.1]-00fb:3: QLogic SN1600Q - HPE SN1600Q 32Gb 2p FC HBA.
qla2xxx [0000:61:00.1]-00fc:3: ISP2261: PCIe (8.0GT/s x4) @ 0000:61:00.1 hdma+ host#=3 fw=8.07.18 (d0d5).
--------------------------------------------------------------------------------------

Posted by 스쳐가는인연

Intel Xeon E5 v4(Broadwell)시리즈 CPU를 사용하는 ProLiant Gen9 서버의 예기치 않은 시스템 재시작 또는 종료 발생 가능 이슈

 

증상

아래 중 한 개 이상의 정보가 확인됨

 

시나리오 1.

연관된 로그(footprint)정보가 남지 않고, 예기치 않은 시스템 재시작 또는 종료가 발생됨

 

시나리오 2.

예기치 않은 시스템 재시작 또는 종료가 발생된 후 리부팅 중, Smart Array 관련 lockup 0x12가 기록됨

 

시나리오 3.

예기치 않은 시스템 재시작 또는 종료가 발생된 후 아래 형태의 UMCE 코드가 IML상에 확인됨

Uncorrectable Machine Check Exception (Board 0, Processor 2, APIC ID 0x00000040, Bank 0x00000004, Status 0xBA000000'73000402, Address 0x00000000'00000000, Misc 0x00000000'00000000)

 

노란색 마크된 정보가 일치된 형태로 확인되며, 그 외 형태의 UMCE 이벤트가 함께 나타날 수 있음.

시나리오 2.와 함께 나타날 수 있음.

 

원인

CPU의 일시적인 내부 오동작으로, 시스템 내부에서 IO 요청(많은 경우 PCI 확장 장치의 요청)이 정상 처리되지 못해,

해당 요청이 비정상 종료(timeout)

 

이 현상은 치명적 오류로 시스템 리부팅을 유발하게 되는데, 리부팅 과정에서 Smart Array lockup(응답 지연으로 인한 IO 처리 실패)을 경험할 수 있음 (lockup 0x12 기록)

HW 로그상에 Smart Array의 오류로 기록되지만(파생된 이벤트), 이슈 촉발의 원인이 Smart Array의 장애가 아님

(Smart Array 교체 필요치 않음)

 

환경

·    Intel Xeon E5 v4 Series CPU를 사용하는 ProLiant Gen9 System

- E5-2600 v4 Series Intel Processor

- E5-4600 v4 Series Intel Processor

 

솔루션

Action Plan 1.

What: Upgrade BIOS to v2.64

 

Problems Fixed:

This revision of the System ROM includes the latest revision of the Intel microcode which addresses an issue where the system could experience an unexpected reset or shutdown with no errors logged to the Integrated Management Log (IML). This issue only impacts systems configured with Intel Xeon 2600 v4 series processors. This issue is not unique to HPE servers.

 

This revision of the System ROM includes the latest revision of the Intel microcode which addresses an issue where the system could experience an unexpected reset or shutdown with no errors logged to the Integrated Management Log (IML) when processor C-states are enabled. This issue does not exist when processor C-states are disabled. This issue is not unique to HPE servers.

 

참조 Advisory ------------------------------------------

Advisory: ProLiant Gen9 Series Servers - Unexpected Reset or Shutdown May Occur on ProLiant Gen9 Servers

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00060570en_us

-----------------------------------------------------------

Posted by 스쳐가는인연