topic Re: Smart Array P420i Fatal Errors in ADU Logs in ProLiant Servers (ML,DL,SL)

Smart Array P420i Fatal Errors in ADU Logs

chrisr6 — Tue, 30 May 2017 09:24:48 GMT

Server hangs every day at random times. Motherboard (on board controller), HDD's and cables have been replaced. Logs:

[05/19 05:59:46] Fatal PR=0x9c04f0f0 D001 Op=03 PLErr=00 IopErr=22 S=00
[05/19 06:01:44] Fatal PR=0x9c052840 D000 Op=a3 PLErr=00 IopErr=22 S=00
[05/19 06:01:44] Fatal PR=0x9c052840 D000 Op=a3 PLErr=00 IopErr=22 S=00
[05/19 06:01:44] Fatal PR=0x9c052840 D000 Op=a3 PLErr=00 IopErr=22 S=00
[05/19 06:01:44] Fatal PR=0x9c052840 D000 Op=03 PLErr=00 IopErr=22 S=00
[05/19 06:01:44] Fatal PR=0x9c052840 D001 Op=a3 PLErr=00 IopErr=22 S=00
[05/19 06:01:44] Fatal PR=0x9c052840 D001 Op=a3 PLErr=00 IopErr=22 S=00
[05/19 06:01:44] Fatal PR=0x9c052840 D001 Op=a3 PLErr=00 IopErr=22 S=00
[05/19 06:01:44] Fatal PR=0x9c052840 D001 Op=03 PLErr=00 IopErr=22 S=00
[05/19 06:01:49] Fatal PR=0x9c04b5f0 D000 Op=a3 PLErr=00 IopErr=22 S=00
[05/19 06:01:49] Fatal PR=0x9c052bf0 D000 Op=a3 PLErr=00 IopErr=22 S=00
[05/19 06:01:49] Fatal PR=0x9c052490 D000 Op=a3 PLErr=00 IopErr=22 S=00
[05/19 06:01:49] Fatal PR=0x9c04ed40 D000 Op=03 PLErr=00 IopErr=22 S=00
[05/19 06:01:49] Fatal PR=0x9c04ed40 D001 Op=a3 PLErr=00 IopErr=22 S=00
[05/19 06:01:49] Fatal PR=0x9c04ed40 D001 Op=a3 PLErr=00 IopErr=22 S=00
[05/19 06:01:49] Fatal PR=0x9c04ed40 D001 Op=a3 PLErr=00 IopErr=22 S=00
[05/19 06:01:49] Fatal PR=0x9c04ed40 D001 Op=03 PLErr=00 IopErr=22 S=00
[05/19 06:03:44] Fatal PR=0x9c04fc00 D000 Op=a3 PLErr=00 IopErr=22 S=00
[05/19 06:03:44] Fatal PR=0x9c04fc00 D000 Op=a3 PLErr=00 IopErr=22 S=00
[05/19 06:03:44] Fatal PR=0x9c04fc00 D000 Op=a3 PLErr=00 IopErr=22 S=00
[05/19 06:03:44] Fatal PR=0x9c04fc00 D000 Op=03 PLErr=00 IopErr=22 S=00
[05/19 06:03:44] Fatal PR=0x9c04fc00 D001 Op=a3 PLErr=00 IopErr=22 S=00
[05/19 06:03:44] Fatal PR=0x9c04fc00 D001 Op=a3 PLErr=00 IopErr=22 S=00
[05/19 06:03:44] Fatal PR=0x9c04fc00 D001 Op=a3 PLErr=00 IopErr=22 S=00
[05/19 06:03:44] Fatal PR=0x9c04fc00 D001 Op=03 PLErr=00 IopErr=22 S=00
[05/19 06:03:48] Fatal PR=0x9c04e990 D000 Op=a3 PLErr=00 IopErr=22 S=00
[05/19 06:03:48] Fatal PR=0x9c04e990 D000 Op=a3 PLErr=00 IopErr=22 S=00
[05/19 06:03:48] Fatal PR=0x9c04e990 D000 Op=a3 PLErr=00 IopErr=22 S=00
[05/19 06:03:48] Fatal PR=0x9c04e990 D000 Op=03 PLErr=00 IopErr=22 S=00
[05/19 06:03:49] Fatal PR=0x9c04e990 D001 Op=a3 PLErr=00 IopErr=22 S=00
[05/19 06:03:49] Fatal PR=0x9c04e990 D001 Op=a3 PLErr=00 IopErr=22 S=00
[05/19 06:03:49] Fatal PR=0x9c04e990 D001 Op=a3 PLErr=00 IopErr=22 S=00
[05/19 06:03:49] Fatal PR=0x9c04e990 D001 Op=03 PLErr=00 IopErr=22 S=00
[05/19 06:05:44] Fatal PR=0x9c04d370 D000 Op=a3 PLErr=00 IopErr=22 S=00
[05/19 06:05:44] Fatal PR=0x9c04d370 D000 Op=a3 PLErr=00 IopErr=22 S=00
[05/19 06:05:44] Fatal PR=0x9c04d370 D000 Op=a3 PLErr=00 IopErr=22 S=00

Re: Smart Array P420i Fatal Errors in ADU Logs

peyrache — Tue, 30 May 2017 09:41:21 GMT

firmware version please ?

Re: Smart Array P420i Fatal Errors in ADU Logs

chrisr6 — Tue, 30 May 2017 09:50:41 GMT

5.22

Re: Smart Array P420i Fatal Errors in ADU Logs

peyrache — Tue, 30 May 2017 09:52:21 GMT

I assume no last firmware version ? shall you update please

Re: Smart Array P420i Fatal Errors in ADU Logs

chrisr6 — Tue, 30 May 2017 09:57:42 GMT

Model: Smart Array P420i Controller

Controller Status: OK

Firmware Version: 5.22

Serial Number: 001438027C2F120

Rebuild Priority: Low

Expand Priority: Unknown

Number Of Ports: 2

Internal SAS Ports: 2

External SAS Ports: 0

HBA Mode Support: Not Available

Sanitize Erase Support: Not Available

I will update to the latest version (Which one do you recommend, SBS 2011 SP1)

Re: Smart Array P420i Fatal Errors in ADU Logs

Torsten. — Tue, 30 May 2017 12:32:04 GMT

latest firmware version is 8.0

link and list of fixes:

http://h20564.www2.hpe.com/hpsc/swd/public/detail?sp4ts.oid=5295169&swItemId=MTX_d1dacc37a6764efcb644a0fc0a&swEnvOid=4064#tab-history

Re: Smart Array P420i Fatal Errors in ADU Logs

chrisr6 — Wed, 31 May 2017 08:11:41 GMT

Hi Torsten, that link is for Linux is it not? I upgraded to this version (6.68) via Windows: http://h20564.www2.hpe.com/hpsc/swd/public/detail?sp4ts.oid=5295169&swItemId=MTX_4c52b1d101ae413fa24a461400&swEnvOid=4064

After upgrading to this firmware, the HDD's to the latest firmware and the latest drivers the server has just gone down again this morning. Not having much luck with this one.

Re: Smart Array P420i Fatal Errors in ADU Logs

chrisr6 — Wed, 31 May 2017 08:27:52 GMT

OK so it turns out if you put Server 2008R2 as your O/S when looking for P420i drivers/firmware it lists 6.68 as the latest available update via windows. If you put server 2012 then it lists the 8.00 update. So I am updating to 8.00 and crossing all digits.

Re: Smart Array P420i Fatal Errors in ADU Logs

chrisr6 — Wed, 31 May 2017 09:07:55 GMT

Updated to 8.00 but errors are still there so I am counting down till I get a phone call that the server is down. Here is the ADU log since the update;

HAL Device Params:
asic_rev: 0x1
boot_block_rev: 0x30392e31
nvram_pns: 0x0
bootloader_rev: 0x0
sram_mem_start: 0x0
sram_mem_end: 0x0
cache_mem_start: 0x9c000000
cache_mem_end: 0x9c1fffff
uncached_mem_start: 0xbc000000
uncached_mem_end: 0xbc1fffff
mem_used_by_HAL: 0x30000

***** HP Smart Array Controller Firmware *****
Copyright 1990, 2016 Hewlett-Packard Development Company, L.P.
Firmware version          = 8.00
Build number              = 0 (standard build)
Subversion                = 13634
Bootblock version         = 1.90
HAL-PL version            = 01.90.011.000
Build date & time         = Aug 18 2016 13:50:06

PASSED: HAL API initialization
PASSED: Memory initialization
PASSED: PCIe Bridge initialization

Initializing I2C...
LocalNVRAM: SSID=0x3354103C, nv_flags=0x00/0x00000000, cfg_sig=0xA22F74D1
PASSED: I2C initialization
NVRAM Update: ILA returned 0x0
MPI current image:
    BoardID      =0x1B
    MPI Revision =0x08
    FormatRev    =0x03
    Bootstrap    =0x02
PASSED: Bootstrap NVRAM Checksum
Platform NVRAM auto-compare detected identical images
PASSED: Command I/O initialization
Board is P420i           d Rev=B; running on platform DL360 Gen8
Using Adaptive Voltage Scaling(AVS) : No
Using WWN to generate new board SN=001438027C2F120
PASSED: Drive parameter initialization

Diagnostic Pointers Initialized
PASSED: Task initialization

Initializing XOR Engine...
Starting DMA SelfTest...
PASSED: DMA SelfTest
PASSED: XOR Engine initialization
PASSED: PAL CO initialization
Start Up Complete
PASSED: DBG_TRACE buffers allocated
Starting protocol device 0 scan device...
PASSED: Protocol device 0 scan device
Adapter 0 Bus Reset
Drive SN:
CDB=0x000000000000
CC Sense Data--
00: 70 00 06 00 00 00 00 0A 00 00 00 00 29 02 02 00 00 00 00 00 00 00 00 00 00 00 00 00
1C: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Fatal PR=0x9c04b610 D000 Op=00 PLErr=02 IopErr=04 S=02 KCQ=6:29:02
Drive SN:
CDB=0x000000000000
CC Sense Data--
00: 70 00 06 00 00 00 00 28 00 00 00 00 29 02 00 00 00 00 00 00 11 05 01 00 00 00 00 00
1C: 00 00 00 00 00 00 00 22 23 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Fatal PR=0x9c04b610 D001 Op=00 PLErr=02 IopErr=04 S=02 KCQ=6:29:02
Total TUR time taken = 0ms
INFO: Resetting I2C bus 5
INFO: Resetting I2C bus 6
INFO: Smart Carrier Type 1 Image in ROM is version = 11

***** Configured logical drives (count=1) *****
ccf=0x00 mccf=0x00000000 mscf=0x00 mocf=0x00000000 dcf=0x00 xp=64 rp=64 std=150
Controller-based encryption is disabled.
Array A Unit U00: RAID 1
U00 from 2 drives: D000 D001
stripsize=512 (256 KiB) volstate=OK datadrives=1 paritygroups=1 cache=disabled SmartPath=disabled/disabled
offset=0x0 logical_blocks=0x45DC2FB0 (558 GiB)
uf=0x0 srf=0x1 dt=2 pdm=0 psf=4 bd=0x0 naz=0x0 nwz=0x0 bsf=512 muf=0x0

***** Discovered Devices *****
Device      [BoxIndex]Port:BoxOnPort:Bay
   Path|Paths            ,Type Vendor ,Product         ,Rev ,SerialNumber       [,misc]
D000 p0|0x1 [00]P1I:01:01,Disk HP      ,EG0600FCVBK     ,HPD9,S0M090M90000B342A830,10K,SCFW=11,SCTYPE=1
D001 p0|0x1 [00]P1I:01:02,Disk HP      ,EG0600JETKA     ,HPD4,5680A01QFUYB1618    ,10K,SCFW=11,SCTYPE=1
D010 p0|0x1 [??]   :00:00,SGPIO                         ,    ,
D011 (Controller)      :Protocol Adapter                ,    ,5001438027C2F120
Total Discovered   SAS Devices =    2
Total Discovered SATA Devices =    0
Total Discovered SAS Expanders =    0
Total Discovered   SEP Devices =    0
Total Discovered Other Devices =    1

***** Discovered Devices - Additional Information *****
Device                   ,WWN              ,WWN hash, Handle
D000 p0|0x1 [00]P1I:01:01,5000C50067DFC571, 0xF40D19,01020001
D001 p0|0x1 [00]P1I:01:02,5000039708181AD2, 0xF73E1E,00010000
D010 p0|0x1 [??]   :00:00,5001438027C2F120, 0xB9C29A,00000001
D011 (Controller)        ,5001438027C2F120, 0xD891FD

***** Controller and Expander Physical Link Rates *****
Device                Phy:0123456789 10         20         30         40         50         60         70         80
D011 (Controller)        :AA000000
Adapter supported link rates: Min=0x8, Max=0xA
Total        Future     phys =    0
Total       12 Gbps (B) phys =    0
Total        6 Gbps (A) phys =    2
Total        3 Gbps (9) phys =    0
Total      1.5 Gbps (8) phys =    0
Total      Disabled (1) phys =    0
Total       Unknown (0) phys =    6
Total        Vacant (V) phys =    0
Total         Other     phys =    0

PCI device rev=0x1 lanes=8 rate=8.0 GT/s (6400 MB/sec) de-emphasis=-3.5dB
pci_bandwidth_MBs = 6400
sas_bandwidth_MBs = 996
memory_bandwidth_MBs = 1688
Starting first command initialization...
*** Memory info ***
Pool        Size       Used      Avail Used%
----- ---------- ---------- ---------- -----
LREQ          44         12         32   27%
PREQ         155         28        127   18%
LPRAM     530432     477280      53152   89% fragments=38
SHRAM     256000      78784     177216   30% fragments=22
XRRAM     256000     256000          0 100% largest=         0 (bytes)
XRRAM        500        500          0 100% largest=         0 (sectors)
PASSED: ROM Flash Driver Initialization
ROM_Sanity_Check: actimage=0x01ROM_Auto_Flash updating ROM Image B...PASSED: Update to ROM Active Flag Checksum
Done!

ALERT: Invalid command, req_type=0x00000003, cmd=0x69, opcode=0x27
[05/31 09:47:53] Host: Time is 5/31/2017 9:47:53
[05/31 09:47:53] Host: PCI slot is 0
[05/31 09:47:53] ALERT: Invalid command, req_type=0x00000003, cmd=0x73, opcode=0x26
[05/31 09:47:53] ALERT: Invalid command, req_type=0x00000003, cmd=0xE5, opcode=0x26
[05/31 09:47:53] ALERT: Invalid command, req_type=0x00000003, cmd=0xE5, opcode=0x26
[05/31 09:47:53] iLO command polling interval=1.000 seconds, hostaddr=0x0BDF83020.
[05/31 09:47:53] Host: Time is 5/31/2017 9:47:53
[05/31 09:47:53] Host: PCI slot is 0
[05/31 09:47:53]
[05/31 09:48:05] CPLD Revision 0x0C   (0x0C0C)
[05/31 09:48:05]      Comms Version :   Revision 2
[05/31 09:48:05]      Programmable :   No
[05/31 09:48:05] Starting OCSD 2 0 0x00000000BDF7F000
[05/31 09:48:05] Monitoring sensor ASIC:smsc_emc_ext:0
[05/31 09:48:05] Start sensor monitoring index=0
[05/31 09:48:05] Temperature Monitoring Update Interval:5
[05/31 09:48:05] Host: Int13 BIOS unit 0x80 is CISS LUN 0x4000000000000000
[05/31 09:48:18] iLO command polling interval=0.090 seconds, hostaddr=0x0BDF83020.
[05/31 09:50:08] Host: Time is 5/31/2017 9:50:08
[05/31 09:51:10] Fatal PR=0x9c051240 D000 Op=a3 PLErr=00 IopErr=22 S=00
[05/31 09:51:10] Fatal PR=0x9c04cfe0 D000 Op=a3 PLErr=00 IopErr=22 S=00
[05/31 09:51:10] Fatal PR=0x9c04fc20 D000 Op=a3 PLErr=00 IopErr=22 S=00
[05/31 09:51:10] Fatal PR=0x9c04e250 D001 Op=a3 PLErr=00 IopErr=22 S=00
[05/31 09:53:14] Fatal PR=0x9c052860 D000 Op=a3 PLErr=00 IopErr=22 S=00
[05/31 09:53:14] Fatal PR=0x9c052860 D000 Op=a3 PLErr=00 IopErr=22 S=00
[05/31 09:53:14] Fatal PR=0x9c04f110 D000 Op=a3 PLErr=00 IopErr=22 S=00
[05/31 09:53:14] Fatal PR=0x9c052860 D001 Op=a3 PLErr=00 IopErr=22 S=00
[05/31 09:53:18] Fatal PR=0x9c04dea0 D000 Op=a3 PLErr=00 IopErr=22 S=00
[05/31 09:53:18] Fatal PR=0x9c04f110 D000 Op=a3 PLErr=00 IopErr=22 S=00
[05/31 09:53:18] Fatal PR=0x9c04dea0 D000 Op=a3 PLErr=00 IopErr=22 S=00

Re: Smart Array P420i Fatal Errors in ADU Logs

peyrache — Thu, 01 Jun 2017 16:02:14 GMT

What is server hardware ?
Shall you post adu.zip please

Re: Smart Array P420i Fatal Errors in ADU Logs

chrisr6 — Fri, 02 Jun 2017 12:16:01 GMT

Downlaod report: https://ufile.io/nz819

Thank you for helping

Re: Smart Array P420i Fatal Errors in ADU Logs

chrisr6 — Tue, 06 Jun 2017 10:37:56 GMT

HP ProLiant DL360p Gen8

Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz 2.00GHz, 1536KB Level 2 cache

32,768 MB RAM

Re: Smart Array P420i Fatal Errors in ADU Logs

peyrache — Tue, 06 Jun 2017 15:59:02 GMT

Could you post complete adu.zip file or drop on ftp site
Thanks

JY

Re: Smart Array P420i Fatal Errors in ADU Logs

havelocke — Wed, 14 Jun 2017 15:30:52 GMT

Any update to this as I have a P420i on a DL360P Gen8 doing exactly the same thing.....

Re: Smart Array P420i Fatal Errors in ADU Logs

peyrache — Fri, 16 Jun 2017 14:50:34 GMT

Hello
Shall you post ahs logs of servers please
Thanks
JY