Betreff: Query: ProLiant DL360 Gen9 - 440ar Raid 6 - disk latency - Prolaint server - DL36

[PLP-Lackner]

I have a ProLiant DL360 Gen9 with a 440ar Raid 6 and high disk latency.

I run a ESXI v7.0.3 on it with the latest drivers for the Raid.

After six months suddenly i have a very high disk latency. I Also tried a spare 440ar Raid controler and also changed
the calbles from the raid to the hard disks.
I have 4 Samsung SSD disk in use.

I tried du analyze with the esxtop ofer the esxi ssh connection and could see that i have DAVG/cmd rate, over 200 and even more.
Is it possible to analyze each hard disk if one hase a problem?'
I dont gent any erros in the iLO or the esxi logs.

[support_s]

System recommended content:

1. Compaq ProLiant DL360 Generation 2 Server Setup and Installation Guide

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[PLP-Lackner]

Please can you be more detailed what should me help in this manual to find out what my problem causes?

[TVVJ]

Hello,

You may download and install the "Smart Storage Administrator (SSA) CLI for VMware 7.0" and generate the SmartSSD Wear Gauge Report. You may refer to page 20, section "7.3.6 Generating a Diagnostic Report" of the HPE Smart Storage Administrator CLI User Guide for the syntax to generate the report. Click here for a video on generating the SmartSSD Wear Gauge Report.

To view the report, extract the generated zip file and open the HTML document. Review it for any disk issue, cache module issue or any controller issue.

Regards, 

[PLP-Lackner]

Hello @TVVJ ,

i make now a check with the "Smart Storage Administrator (SSA) CLI for VMware 7.0". It says everything is okay.
I have no idea what i should do now.

================================================================================

Controller:    Smart Array P440ar Slot:0

Controller Attributes

-----------------------------------------------

4k Raid Supported : False

Ace Enabled : True

Ace Supported : True

Adapter Type : Smart Array

All Raid Rpi Supported : False

Alternate Msa Cache Algorithm : False

Alternate Msa Preferred Path Algorithm : False

Alternate Msa Surface Delay Algorithm : False

Array Count : 1

Backup Hardware Error : No Error

Backup Power And Green Backup Status Supported : True

Backup Power Status : Uninitialized

Backup Power Type : Hpe Megacell

Battery Good Count : 1

Battery Pack Count : 1

Battery Status : Ok

Bit14 Miscflags Id Phys Drive Support : False

Bmic Sensor Supported : False

Boot Connector Supported : False

Boot Volume Primary : 00000000000000000000000000000000

Boot Volume Secondary : 00000000000000000000000000000000

Boot Volume Supported : True

Bus Type : Pci

Cached Data Logical Drive Count : 0

Cache Array Count : 0

Cache Board Present : True

Cache Disabled : False

Cache Has Bbwc : True

Cache Logical Drive Count : 0

Cache Memory Not Fully Battery Backed : False

Cache Module Memory Data Width : 64

Cache Module Memory Transfer Rate : 1866

Cache Module Temperature : 43

Cache Operations Supported : True

Cache Parity Read Errors : 0

Cache Parity Write Errors : 0

Cache Physical Drive Count : 0

Cache Ratio : 100 0

Cache Size Total : 3719168

Cache Size Total Bbbwc : 3719168

Cache Size Total Bbbwc Kb : 1859584

Cache Size Total Kb : 1859584

Cache Size Total Non Bbbwc : 0

Cache Size Total Non Bbbwc Kb : 0

Cache Status : Not Configured

Clear Controller Config Supported : True

Clear Nvram Supported : False

Connector Mode Runtime Change Supported : False

Controller Cache Serial Number : **confidential info erased**

Controller Cold Reboot Required : False

Controller Current Power Mw : 10238

Controller Disabled Config Status : Ok

Controller Firmware Family : Fishman

Controller Has Unsupported Sas Drives : False

Controller I2c Enabled : False

Controller I2c Get Supported : False

Controller I2c Set Supported : False

Controller Manufacturing Data Format : Default

Controller Mctp Enabled : False

Controller Mctp Get Supported : False

Controller Mctp Set Supported : False

Controller Memory Size : 4194304

Controller Memory Size Kb : 2097152

Controller Mode : Smartarray

Controller Name : P440ar

Controller Oem Id : Hpe

Controller Ofa Supported : False

Controller Port Rate Limit Supported : False

Controller Port Supports Hba : True

Controller Port Supports Mixed : False

Controller Port Supports Raid : True

Controller Product Id : P440ar          

Controller Reboot Status : Not Required

Controller Sanitize Supported : True

Controller Serial Number : **Confidential info erased**

Controller Supports Bp Discovery Protocol Config : False

Controller Supports Nvme : False

Controller Supports Reboot Reasons : False

Controller Temperature : 48

Controller Vendor Id : HP      

Controller Zmr Mode : False

Cpld Revision : 3

Cpu Revision : 40

Current Parallel Surface Scan Count : 1

Current Rebuild Mode : Automatic

Data Logical Drive Count : 1

Data Physical Drive Count : 4

Dcache Error Handling Supported : False

Ddff Version : Sas And Sata

Debug Token Supported : False

Default Cache Ratio : ATTR_VALUE_CACHE_RATIO_10_90

Degraded Performance Optimization : Disabled

Degraded Performance Optimization Supported : True

Device Id : 272380473

Device Id Hex : 103C3239

Diagnostic Mode Enabled : False

Driver Enables Hiwatt : False

Driver Name : nhpsa

Driver Supports Hiwatt : False

Driver Version : 70.0051.0.100-4vmw

Drives Per Port : 256

Drive And Carrier Authentication Support : True

Drive Erase Enabled : True

Drive Erase On Hba : False

Drive Erase On Mixed : False

Drive Erase On Raid : False

Drive Erase Supported : True

Drive Long Product Id Support : False

Drive Write Cache : Disabled

Drive Write Cache Policy Support : False

Dual Domain Enabled : True

Dual Domain Supported : True

Dwc Unchanged Supported : False

Elevator Sort : Enabled

Enable Disable Right Phys Supported : False

Encrypted Data Logical Drive Count : 0

Encryption Acoustics Mixing Supported : False

Encryption Boot Password Set : False

Encryption Boot Password Unlock Attempts : 0

Encryption Controller Locked : False

Encryption Controller Password Enabled : False

Encryption Crypto Officer Password Set : False

Encryption Csp Nvram Status : Passed

Encryption Dynamic Roles Supported : False

Encryption Enabled : False

Encryption Feature Breakout Supported : False

Encryption Feature Enabled Allow Plaintext Volumes : True

Encryption Feature Enabled Boot Password : True

Encryption Feature Enabled Boot Time Remote Auth : True

Encryption Feature Enabled Configurable User Roles : True

Encryption Feature Enabled Crypto Erase : True

Encryption Feature Enabled Data Rekey : True

Encryption Feature Enabled Destructive Encode : True

Encryption Feature Enabled Disable Encryption On Unit : True

Encryption Feature Enabled Drive Rekey : True

Encryption Feature Enabled Encode : True

Encryption Feature Enabled Encryption Disable : True

Encryption Feature Enabled Firmware Lock : True

Encryption Feature Enabled Import Foreign Keys : True

Encryption Feature Enabled Key Manager Conversion : True

Encryption Feature Enabled Local Express : True

Encryption Feature Enabled Local Key Cache : True

Encryption Feature Enabled Local Mode : True

Encryption Feature Enabled Master Key Rekey : True

Encryption Feature Enabled Password Recovery User Role : True

Encryption Feature Enabled Pause Resume Boot Password : True

Encryption Feature Enabled Remote Mode : True

Encryption Feature Enabled Return To Factory : True

Encryption Feature Enabled Users : True

Encryption Feature Enabled Volatile Keys : True

Encryption Fw Locked : False

Encryption Has Locked Volumes Missing Boot Password : False

Encryption Has Suspended Boot Password : False

Encryption Hiwatt Mixing Supported : True

Encryption Ikey Rekey In Progress : False

Encryption Keymanager Mode Conversion Supported : True

Encryption Key Manager Retrieval Status : Not Needed

Encryption Ldrives Disabled Bad Boot Password : False

Encryption Ldrives Disabled Dek Wont Decrypt : False

Encryption Ldrives Disabled Ikek Mismatch : False

Encryption Ldrives Disabled Key Missing From Eskm : False

Encryption Ldrives Disabled Key Retreival Connectivity Error : False

Encryption Local Kek Cache Retry Attempts : 0

Encryption Local Kek Cache Retry Interval : 0

Encryption Local Kek Set : False

Encryption Local Key Cache Enabled : False

Encryption Local Key Cache Persistent : False

Encryption Local Key Cache Supported : True

Encryption Lockout Seconds Remaining Crypto : 0

Encryption Lockout Seconds Remaining User : 0

Encryption Mixed Volumes Enabled : False

Encryption Mode : Bypass

Encryption One Button Mode : False

Encryption Password Unlock Attempts Remaining Crypto : 10

Encryption Password Unlock Attempts Remaining User : 10

Encryption Reason Rtf : Not Me

Encryption Recovery Params Set : False

Encryption Remote Kek Set : False

Encryption Remote Mode Key Mismatch : False

Encryption Rtc Sram Status : Passed

Encryption Selftest Status : Passed

Encryption Skip Controller Password Enabled : False

Encryption Skip Controller Password Supported : True

Encryption Standalone Mode Enabled : False

Encryption Supported : True

Encryption User Password Set : False

Encryption Volume Creation Enabled : False

Enhanced Rpi Supported : True

Expander Minimum Spinup Time Supported : False

Expand Priority : Medium

Extended Drive Data Supported : False

Fail Drives With Smart Errors : Disabled

Family Id Supported : False

Firmware Allows Recover Cache Module : False

Firmware Can Wait For Cache Room : True

Firmware Compile Stamp : May 17 2019 17:03:13

Firmware Part Number : 749796-001

Firmware Request Coalescing : True

Firmware Supports Configuring Raid1 Write Buffering : False

Firmware Supports Inconsistency Event Notification Control : True

Firmware Supports Nobattery Write Cache : True

Firmware Supports Raid1 Write Buffering : True

Firmware Supports Reorder Command : True

Firmware Version : 7.00

Firmware Version Pending : 7.00

Flashlight Supported : False

Flash Controller Once : False

Flexible Latency Scheduler Setting : Default

Flexible Latency Scheduler Supported : True

Gasgauge Supported : True

Greater Than 2tib Physical Drives Supported : True

Green Backup Preservation Enable : False

Green Backup Preservation Set : False

Green Backup Version Major : 0

Green Backup Version Minor : 0

Hardware Revision : B

Has Splitcache : False

Has Swapped Cable Flags : False

Hba Encryption Supported : False

Hba Mixed Mode Enabled : False

Hba Mixed Mode Supported : False

Hba Mode Changed : False

Hba Mode Enabled : False

Hba Mode Led Support : False

Hba Mode Physical Drive Count : 0

Hba Mode Supported : True

Host Serial Number : **Confidential info erased**

Hotplug Supported : True

Icache Error Handling Supported : False

Inactive Firmware Revision : 7.00

Inconsistency Repair Policy : Disabled

Inconsistency Repair Policy Supported : True

Io Accel Advanced Writes Supported : False

Io Bypass Supported : True

Is Adg Supported : True

Is Rollback Supported : True

Is Software Raid : False

Is Tofu Supported : False

Large Write Cache Bypass Threshold Supported : False

Last Backup Time : 0

Last Restore Time : 0

License Keys Supported : True

Logical Drive Count : 1

Logical Drive Erase Supported : False

Long Box Connector Name Supported : False

Long Connector Names Supported : False

Lun Profiling Enabled : True

Lun Profiling Supported : True

Marketing Name : Smart Array P440ar

Max Drives Per Array : 128

Max License Keys Reached : False

Max Logical Volumes : 64

Max Parallel Surface Scan Count : 16

Max Safe Full Stripe Size : 24576

Max Safe Full Stripe Size Kb : 12288

Minimum Memory Raid : False

Min Strip Size Supported : False

Missing Drive Count Supported : False

Mnp Delay : 60

Modify Rebuild Mode Supported : True

Move Lun Enabled : True

Move Lun Supported : True

Mpg Supported : True

Multi Path Supported : True

Ncq Supported : True

Notify Driver On Event Supported : False

No Battery Write Cache : Disabled

Nvram Checksum Status : Ok

Nway Mirroring Enabled : True

Nway Mirroring Supported : True

Online Split Mirror Enabled : True

Online Split Mirror Supported : True

Oob Messaging Interface Supported : False

Overprovisioning Optimization Supported : True

Parallel Surface Scan Supported : True

Parity Raid Supported : True

Pcie Gen : Gen3

Pcie Relaxed Ordering Supported : False

Pcie Width : 8

Pci Bus : 3

Pci Device : 0

Pci Domain : 0

Pci Function : 0

Pending Controller Mode : Smartarray

Persistent Event Log Supported : False

Physical Drive Count : 4

Pms Running : False

Ports : 2

Ports External : 0

Ports Internal : 2

Post Prompt Timeout : 15

Power Consumption : 10238

Power Mode Configured : ATTR_VALUE_POWER_MODE_PERFORMANT

Power Mode Default : ATTR_VALUE_POWER_MODE_PERFORMANT

Power Mode Low Automated Supported : True

Power Mode Low Supported : True

Power Mode Operational : ATTR_VALUE_POWER_MODE_PERFORMANT

Power Mode Performant Supported : True

Power Mode Supported : True

Power Mode Warning Changed Drive : False

Power Mode Warning Changed Mode : False

Power Mode Warning Reboot : Not Required

Power Mode Warning Temperature : False

Predictive Spare Rebuild : Disabled

Predictive Spare Rebuild Supported : True

Queue Depth : Automatic

Raid1 Write Buffering : False

Raid 01 Stripe Size : 65535

Raid 45 Stripe Size : 2048

Raid 50 Enabled : True

Raid 50 Supported : True

Raid 60 Enabled : True

Raid 60 Supported : True

Raid 6 Stripe Size : 2048

Rapid Parity Initialization Supported : True

Rapid Rebuild Supported : True

Rebuild Priority : Rapid High

Recovery Rom Active Image : A

Recovery Rom Supported : True

Redundant Paths To All Drives : False

Redundant Paths To Some Drives : False

Requery Command Supported : True

Roaming Spares Supported : True

Sanitize Lock Antifreeze Supported : False

Sanitize Lock Freeze Supported : False

Sanitize On Hba : False

Sanitize On Mixed : False

Sanitize On Raid : True

Sed Info Supported : False

Sed Refresh Security Status Supported : False

Sense Support Raid Migration : True

Sense Support Stripe Migration : True

Sense Support Volume Expansion : True

Sense Support Volume Extension : True

Signed Firmware Bypass : False

Signed Firmware Required : False

Slot : 0

Smart Hba Mode Supported : True

Smr Virtualization Supported : False

Spare Physical Drive Count : 0

Split Mirror Enabled : True

Split Mirror Supported : True

Ssd Caching 128kb Cache Line Supported : False

Ssd Caching 256kb Cache Line Supported : True

Ssd Caching 4k Cache Vols Supported : False

Ssd Caching 4k Data Vols Supported : False

Ssd Caching 64kb Cache Line Supported : True

Ssd Caching Algo0 Bytes Per Cache Line : 50

Ssd Caching Algo0 Datavols Per Cachevol : 1

Ssd Caching Algo0 Fixed Header Size : 17276

Ssd Caching Algo0 Maxcachevol Bits : 27

Ssd Caching Algo0 Maxdatavol Bits : 32

Ssd Caching Algo0 Type : 1

Ssd Caching Algo1 Bytes Per Cache Line : 58

Ssd Caching Algo1 Datavols Per Cachevol : 1

Ssd Caching Algo1 Fixed Header Size : 17348

Ssd Caching Algo1 Maxcachevol Bits : 27

Ssd Caching Algo1 Maxdatavol Bits : 32

Ssd Caching Algo1 Type : 2

Ssd Caching Config Cache Line Supported : True

Ssd Caching Enabled : False

Ssd Caching Max Strip Size Kb : 256

Ssd Caching Memory Available : 1701478400

Ssd Caching Memory Used : 0

Ssd Caching Min Strip Size Kb : 8

Ssd Caching Raid1 Raid10 Enabled : True

Ssd Caching Raid5 Raid50 Enabled : False

Ssd Caching Raid5 Writeback Enabled : True

Ssd Caching Raid6 Raid60 Enabled : False

Ssd Caching Supported : True

Ssd Caching Version : 2

Status : Ok

Subsystem Id : 272376256

Subsystem Id Hex : 103C21C0

Surface Analysis Delay : 3

Surface Analysis Delay Extended : 30

Surface Analysis Inconsistency Event Notification : False

Surface Analysis Mode Aggressive : False

Surface Analysis Requires Fw Update : False

Survival Mode Configurable : False

Survival Mode Enabled : True

Survival Mode Supported : True

Total Number Of Backups : 0

Transformation Cache Size : 262144

Transformation Cache Size Kb : 131072

Type : Array Controller

Unassigned Physical Drive Count : 0

Unique Id : AC:936208878

Vod Enabled : True

Vod Supported : True

Wait For Cache Room : False

Wwn Port : 51402EC001E1BE70

Zoning Enabled : False

***********************************************************

Prop Table

***********************************************************

PCI Vendor/Device ID   :103C3239

PCI SS Vendor/Device ID:103C21C0

PCI Device class       :

Device Base ID         :nhpsa0

Device Node            :0

Slot Info              :UNKNOWN

Serial Number          :

Controller Status      :OK

Driver Type            :

Driver ID              :nhpsa

Driver Name            :nhpsa

Driver Version         :70.0051.0.100-4vmw

Driver IO BypassSupport:NO

PCI Address (DBDF)     :0000030000

OS Description         :Adapter: Microchip Technology

Protocol Type          :7

Protocol Address       :CISS:0000000000000000

Device Type            :ARRAY_CONTROLLER

Parent Protocol Address:

IDE Channel            :

[Vinky_99]

Hello @PLP-Lackner

Good Day!

High disk latency on your ProLiant DL360 Gen9 server with a Smart Array P440ar RAID controller can be a frustrating issue, but there are several steps you can take to diagnose and potentially resolve the problem. Here's a guide to help you troubleshoot:

>> Confirm that all the physical components are in good condition, including cables and hard drives.
>> Ensure that the SSDs are functioning correctly. You can test them in another system or using SSD diagnostic tools if available.

>> Use the "Smart Storage Administrator (SSA) CLI for VMware" to verify the health of your RAID array. Make sure all drives are online, and there are no errors reported.

>> Confirm your RAID configuration. Ensure that the RAID level, stripe size, and other settings are optimal for your workload.
>> Check for any predictive failure notifications for drives in the RAID array.

>> Ensure that your server's firmware and drivers are up to date. Sometimes, updating the firmware and drivers can resolve performance issues.

>> Check ESXi logs for any errors or warnings related to the storage subsystem. Sometimes, there might be clues in these logs.

>> Use tools like esxtop to monitor disk performance metrics. You've mentioned DAVG/cmd rate. Pay attention to other metrics like KAVG and GAVG to identify potential issues.
>> High DAVG can indicate slow response time from the storage, which might be due to various factors, including a failing drive.

>> Some background tasks, such as RAID rebuilds, can cause temporary high disk latency. Ensure that no such tasks are running.

>> Ensure that the SSDs are not nearing their capacity limits, which can lead to performance degradation.
>> Evaluate the server's workload and make sure it hasn't increased significantly. Sometimes, high disk latency can be attributed to increased demand on the storage subsystem.

>> If you've ruled out other issues and suspect a failing disk, consider replacing individual SSDs one at a time to identify if one of them is causing the problem. Keep an eye on RAID rebuild status and system performance during this process.

>> Check the server's environment for factors that might contribute to overheating or other physical issues. Ensure proper ventilation and cooling.
External Factors:

>> Sometimes, issues on the network or with connected storage devices can cause latency. Check if there are any network-related issues or conflicts.
Contact Support:

>> If the problem persists and you're unable to identify the root cause, it's a good idea to contact HP/HPE support for further assistance.

Remember to back up your data before making significant changes to your RAID configuration or hardware components, and take caution when replacing disks to avoid data breakage

Hope this helps! Let me know