정보와 지식의 올바른 공유 - http://infoages.com

증상
Linux OS를 설치한 Embedded SmartArray B140i Controller를 RAID mode로 사용하는 DL20 Gen9 시스템에서 FAN #3이 빠르게 동작하고, 04-HD-max 센서가 인식되지 않음

원인
OS inbox Driver로 인해 AHCI controller로 인식된 SmartArray B140i controller가 HPE Array Controller로 인식되지 않고일반 AHCI controller로 인식되어 발생

Inbox driver로 설치된 경우,
# dmesg |egrep ahci
[ 1.998337] ahci 0000:00:17.0: version 3.0
[ 1.998346] ahci 0000:00:17.0: enabling device (0406 -> 0407)
[ 1.998572] ahci 0000:00:17.0: irq 28 for MSI/MSI-X
[ 2.009134] ahci 0000:00:17.0: AHCI 0001.0301 32 slots 6 ports 6 Gbps 0x3f impl RAID mode
[ 2.009139] ahci 0000:00:17.0: flags: 64bit ncq sntf pm led clo only pio slum part deso sadm sds apst
[ 2.011261] scsi host1: ahci
[ 2.011346] scsi host2: ahci
[ 2.011412] scsi host3: ahci
[ 2.013162] scsi host4: ahci
[ 2.013836] scsi host5: ahci
[ 2.016348] scsi host6: ahci

SA B140i(hpdsa) driver로 설치된 경우,
# dmesg| grep ahci
[ 1.980262] hpdsa_ahci_request_irq: Setting msi interrupts
[ 1.980273] hpdsa_ahci_request_irq: h:ffff945231dcd000 Got msi irq 28 for board id 0xc01590 name=hpdsa0
[ 1.980275] hpdsa_ahci_pci_init: Using 64bit DMA for hpdsa0
[ 1.980276] hpdsa_ahci_init_one: Initialized ahci 8 port adapter
[ 1.980277] hpdsa_ahci_init_one: B140I found, allowing it to control ahci devices.

환경
• Embedded Smart Array B140i를 RAID mode로 사용하는 DL20 Gen9 시스템

솔루션
Action Item.
What: OS 설치 시 Driver Update Disk(DUD) 기능을 통해, SA B140i driver를 주 Controller Driver로 인식시켜 줌
When: 초기 설치 시,
NOTE: 참고로, CentOS도 같은 방식으로 설치 가능하나, HPE에서 드라이버를 공식 지원하는 OS가 아님
본 방식으로 설치는 선택 사항이며, HPE에서 추가 지원을 하지 않음.
To do. (e.g.) CentOS 7.x 설치)
1. CentOS 7.6 미디어로 부팅 후, 부팅 초기, 설치 메뉴(Install CentOS 7)에서 'e' Key를 눌러 편집
2. "linuxefi" line 끝에 "modprobe.blacklist=ahci inst.dd" 추가

3. +로 부팅 시작
4. 부팅 중 driver disk 추가 메뉴가 출력되면, 'r' 키로 장치 목록 확인
5. 인식된 장치 중 driver disk 선택 'n' (n=1~)
6. driver가 인식된 후 'c'키를 눌러 진행 선택.
(다시한 번 다른 driver disk 인식 여부에 대한 질의가 나오며, 'c'키를 한 번 더 누르면 설치 시작)

7. 이 후 진행은 일반 OS 설치와 동일.

HPE Dynamic Smart Array B140i SATA RAID Controller Driver for Red Hat Enterprise Linux 7 (64-bit)
1.2.10-139(A)(27 Sep 2018)
https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_439cda7afa7e41bd8124a7a5df&swEnvOid=4176#tab2
DUD의 경우, 설치 절차를 참조하여 USB Key를 생성하거나, .img(image 파일을 생성하여)을 iLO Virtual Media로 mount하여 이용할 수 있음.

참고 문서
Advisory: HPE ProLiant DL20 Gen9 Server - Fan May Frequently Run at a High Speed on Certain HPE ProLiant DL20 Gen9 Servers
https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00003578en_us

OS 설치 후 FAN 동작:

SA B-Series에서 구성한 Array 볼륨(Disk)을 SA P-Series 또는 Smart HBA H-Series로 이동

요약:
- 작업 전 반드시 백업이 필요
- B-Series > P-Series : O
- B-Series > H-Series (RAID mode) : O
- P/H Series > B-Series: X

참조 자료:
HPE Smart Storage Administrator User Guide
https://support.hpe.com/hpsc/doc/public/display?docId=c03909334
P. 6. Support for configuration tasks

HP Dynamic Smart Array RAID Controller User Guide
https://support.hpe.com/hpsc/doc/public/display?docId=c03326739

P.18 Moving hard drives and arrays
You can move drives from a B-series controller to a P-series controller
You can move drives from a B-series controller to a host bus adapter.

You cannot move drives from a P-series controller to a B-series controller.
You cannot move drives from a B120i controller to a B320i controller, or vice versa.

HPE Dynamic Smart Array B140i RAID Controller User Guide
https://support.hpe.com/hpsc/doc/public/display?docId=c04441385

The B140i controller has the following features:
RAID 0, 1, 10, 5

P.13 Moving hard drives and arrays
You can move drives from a B-series controller to a P-series controller.

You can move drives from a B-series controller to a host bus adapter. However, any volume configuration will not be migrated to the new HBA.
You cannot move drives from a P-series controller to a B-series controller

HPE Smart Array Controllers User Guide
https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c04441382

P.37 Moving drives and arrays
You can move drives to other ID positions on the same array controller.
You can also move a complete array from one controller to another, even if the controllers are on different servers.

HPE H240ar, H240, and H241 Smart HBA User Guide
https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c04441332

The HPE H240ar, H240, and H241 Smart HBAs contain the following basic features:
Supports RAID 0, 1, 10, and 5

테스트 환경:
Linux Host(SY480 Gen10) + 3PAR Storage

1. Configure BIOS
1) Set Default use BIOS Memu
2) Set HPC or Virtualization - Max Performance

3) VSP Configuration ----------------
BIOS/Platform Configuration (RBSU) > System Options > Serial Port Options >
- Embedded Serial Port > COM1
- Virtual Serial Port > COM2

- BIOS Serial Console and EMS
-> Select BIOS Serial Console Port -> "Auto" to "Virtual Serial Port"
-> Select BIOS Serial Console Emulation Mode -> VT100+ (default).
-> Select BIOS Serial Console Baud Rate -115200 (default).
-> EMS Console -> Disabled (default)
---------------------------------------

2. connects via ssh to iLO IP Address
hpiLO-> vsp

3. Install RHEL 7.5 (Server with GUI + Compatibility Libraries)
- enabled kdump - 384MB (set manually)

4. Configure Network
# systemctl stop NetworkManager
# systemctl disable NetworkManager

# vim /etc/sysconfig/network-scripts/ifcfg-ens3f0
cf. ifcfg-ens3f0 - dhcp or static assign IP Address (eth port)

5. Install SPP2019.03.1

6. Configure VSP from Host
# vim /etc/default/grub

add below line's tail - "console=tty0 console=ttyS1,115200"
GRUB_CMDLINE_LINUX="crashkernel=384M rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet console=tty0 console=ttyS1,115200"

skip this for next (duplicate) -----
# grub2-mkconfig -o /boot/grub2/grub.cfg
# grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

# reboot
------------------------------------

7. Configure Kdump
A. Mount DVD Media
# mkdir /media/odd
# mount -o loop /dev/cdrom /media/odd

B. Edit configuration
# vim /etc/yum.repos.d/RH7-DVD.repo

It may like below ----------------------
[RHEL7-DVD]
NAME=RHEL7-DVD
BASEURL=file:///media/odd
ENABLED=1
GPGCHECK=0
------------------------------------------

C. Update list Repo
# yum repolist all

D. Check rpm
# rpm -qa | grep kexec-tools

if not exsist
# yum install kexec-tools

E. Add Kernel Parameters
# vim /etc/default/grub
GRUB_CMDLINE_LINUX="crashkernel=384M rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet console=tty0 console=ttyS1,115200 nmi_watchdog=1"

# grub2-mkconfig -o /boot/grub2/grub.cfg
# grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

F. Add Kernel Parameters
# vim /etc/sysctl.conf
kernel.unknown_nmi_panic = 1
kernel.panic_on_unrecovered_nmi = 1
kernel.panic_on_io_nmi = 1
kernel.panic_on_oops = 1
kernel.panic = 1
kernel.sysrq = 1

G. Check Service
# systemctl status kdump.service

# reboot

8. Trigger crash
A. via Magic key.
# echo c > /proc/sysrq-trigger

B. via Virtual NMI Button from iLO

C. Check creation state of Crash Dump
# ll /var/crash

^^^ for Local volume
-----------------------------------------------------------
v v v for Remote volume (Boot from SAN)

1. Configure/Check BIOS for BFS
---------------------------------
Configure Network Boot
System Utilities > System Configuration > BIOS/Platform Configuration (RBSU) > Network Options > Network Boot Options > PCIe Slot Network Boot
set Network Boot from port(s)

Configure FC/FCoE Scan Policy
System Utilities > System Configuration > BIOS/Platform Configuration (RBSU) > Storage Options > Fibre Channel/FCoE Scan Policy
set Scan Configured Targets Only

Configure UEFI POST Discovery Mode
System Utilities > System Configuration > BIOS/Platform Configuration (RBSU) > System Options > Boot Time Optimizations > UEFI Post Discovery Mode
set Auto or Force Full Discovery

Check
Device Hardware Configuration
MBA Configuration > Legacy Boot Protocol
set FCoE
FCoe Boot Configuration > FCoE General Parameters > Boot to FCoE Target
set Enabled
FCoe Boot Configuration > FCoE General Parameters > HBA Boot Mode
check Enabled

FCoe Boot Configuration > FCoE Target Parameters
set Connect # Enabled and Boot LUN #
---------------------------------

1. Install RHEL 7.5 to BFS / remote volume (Server with GUI + Compatibility Libraries)
- Network & Hostname > ens3f2 (BFS port) > General > Automatically connect to this network when it is available

2. Check FCoE port
# cat /etc/sysconfig/network-scripts/ifcfg-fcoe0
TYPE=Ethernet
DEVICE=ens3f2
ONBOOT=yes

그 외 항목들은 Local volume 구성과 동일

RHEL 6.x를 운용하는 Gen10 System 에서, RESTful Interface Tool(ilorest)를 설치 후 약 하루 지난 뒤 구동 실패

증상
RESTful Interface Tool을 설치 후 약 하루 정도 시간이 지난 뒤부터, 구동 시, 오류와 함께 실행되지 않음
(ilorest 구문 관계 없음)

# ilorest -v
Cannot open self /usr/sbin/ilorest or archive /usr/sbin/ilorest.pkg

원인
RHEL 6의 prelink 라는 기능 때문에 발생됨.

prelink는 응용프로그램 수행 시간을 단축/가속 하기 위한 일종의 캐시로 이해할 수 있으며,
공통 라이브러리 등을 미리 메모리에 상주 시키고, 해당 라이브러리가 필요한 프로그램에 바로 연결해 주어,
특정 프로그램이 실행 시 라이브러리를 호출하고 메모리에 로드 하는 등의 시간을 줄이기 위한 기술.

개발 환경이 다변화 되어 기대하지 않은 오류(ilorest가 이 부분에 해당)가 확인되고, 시스템이 충분히(?) 빨라졌기 때문에, 더 이상 효용이 상대적으로 낮아짐에 최근에는 사용하지 않음. (레드햇 기준)

Redhat에서 가이드 하는 방법을 통해, 구동 실패 이슈를 해소할 수 있음.

Questions about Prelinking in Red Hat Enterprise Linux
https://access.redhat.com/solutions/61691
Prelink
http://people.redhat.com/jakub/prelink.pdf

prelink is supported in RHEL5 and RHEL6. In RHEL7, prelink is not supported. prelink was deprecated and disabled by default because it no longer offers any significant benefits and is not stable enough across supported architectures.

What is prelinking
Prelinking is the operation upon which prelink modifies binaries to dynamically link them to binaries

Why is this default in Red Hat Enterprise Linux?
Prelinking significantly improves application start up times (often on the order for 5x speedup). It shortens the time the OS spends dynamically linking a program (since it is done ahead of time).

How to prevent prelink from prelinking specific executables
To filter out a program from being optimized you can open /etc/prelink.conf and add
-b /usr/bin/program
if you wish to avoid prelink to work on /usr/bin/program.

환경
RHEL 6을 사용하는 HPE ProLiant Gen10 System (보고된 환경)

솔루션
Action Item.
What: Ilorest 구동에 prelink가 관여하지 않도록 변경 (ilorest를 blacklist 등록)
Why: ilorest 수행 장애를 해소하기 위해
What if/Next: TBD
To do.
1) prelink와 ilorest 의 연결을 제어(참조하지 않도록 수정)
# echo "-b /usr/sbin/ilorest" > /etc/prelink.conf.d/ilorest.conf
2) ilorest 삭제 후 재설치

HPE RESTful Interface Tool 2.4.x에서 추가된 serverinfo commands

Check System Health -----------------------
serverinfo --fans
serverinfo --thermals
serverinfo --power

serverinfo --processors --memory
-------------------------------------------
ilorest -d serverinfo --fans
ilorest -d serverinfo --thermals
ilorest -d serverinfo --power

ilorest -d serverinfo --processors --memory
-------------------------------------------

Linux
ilorest list | egrep 'SmartStorageBattery|Status|State|Health'

Windows
ilorest list | findstr "SmartStorageBattery Status State Health"