MSA 2050 SSD read cache on controller B only 4% allocated

[czobi]

I have an issue on HPE MSA 2050 SAS storage pool B read cache. It has a RAID5 disk group and an SSD read cache. The volume assigned to is a storage for hyper-v virtual machines. The cache has 182214 free, but only 7825 allocated pages.

Pool A has two disk RAID5 groups and a read cache with same size, it is used for also VMs and backups. The read cache is the same size and all pages are allocated.

I expect pool B read cache should also be allocated fully after a certain amount of time. Is it an error or feature?

[support_s]

System recommended content:

1. HPE MSA 2050 SAN Storage - Option Parts

2. HPE MSA 2050 SAN Storage - Product Information Reference | Drives

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[shiva_jr]

Hi @czobi ,
    If you have the Read cache, it only concerned with the read operation.  Please try the following-
1)  Your data activity migh have more read operation in Pool A data than in Pool B.
     Review the read performance by IOPS and throughput. If it has more read operation, it is common to see the read cache utilization.

2) Update the MSA firmware, it may help the controller to manage IOs.

3) You can refer this document (page no.136) to improve the performance.

4) You can flush/delete the read cache and recreate, it won't affect the volumes.

5) If you have concern in capacity sizing, you can refer this document.
Along with this, post1 and post2 may also help you.

I hope, my post help you, please give Kudos and mark my post as Accepted Solutions.

Regards,
Shiva_jr

[support_s]

Hello,

Let us know if you were able to resolve the issue.

If you are satisfied with the answers then kindly click the "Accept As Solution" button for the most helpful response so that it is beneficial to all community members.

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

[czobi]

Thanks for your reply! I more deeply examined the issue. I've already tried all in your checklist earlier.
The read cache issue is not the main one. The overall storage performance was very low. Let me describe my configuration.
It has two HPE DL380 Gen10 servers (HOST1 and HOST2), double CPU, 256GB RAM each. They connected to the MSA 2050 SAS. Both servers are running Windows 2022 Standard, with hyper-v failover clustering. Most VMs are Windows servers. This configuration has been working for more than 6 years without any major problem. The hosts were upgraded from WS 2016 about a year ago.
The main issue we have since several months the poor storage performance on VMs. As tried to find the source of the issue I set up a Zabbix appliance and monitor the hosts, VMs and the MSA. 

Finally, I might find the source of the problem: the MSA has relatively high CPU load (about 60%). The interesting thing is if I migrate all VMs to HOST2 the CPU load on MSA suddenly drops between 0 and 10% and the whole system performance become very good. And the read cache in Controller B is starting to allocate much more pages (reaching about 50%).. Very strange behaviour. Then, if I migrate one or several VMs to HOST1, the CPU load goes back to 60% and the overall performance drops. I attached several screen shots of zabbix graphs with interested time range. On 14th March a full system shutdown, poweroff and reastart was done. On 16h March all VMs were migrated to HOST2.

Controller A CPU load drop at VM migrationController B CPU load drop at VM migrationController A cache write utilization changeController A read cache utilizationController B read cache utilizationController B cache write utilization change

[JonPaul · Accepted solution]

@czobi 
what is the configuration of the Pools, Disk-groups, volumes on the MSA?
This looks like a configuration issue on HOST1. 
Are both hosts connected to both MSA controllers?
Some things to check on the MSA:
CLI>  show unwriteable-cache   ==> this would show if orphaned data is left in the cache, this happens when a volume is removed while the host is still actively using it.
CLI>  show controller-statistics   ==> look for forwarded commands,  this indicates a multipath issue where the host is using non-optimal paths to the LUNs.  The MSA will then forward the I/O internally between the controllers and causes some to significant delay and increase in CPU utilization.
You can look at the host connections with the CLI as well:
CLI> set cli-parameters json    ==> returns all data fields in JSON output, almost easy to read.  set cli-parameters console  returns the easy to read output
CLI> show initiators   ==> returns info on the initiators
  "host-port-bits-a":8,      ==>  bitmap of the connected ports from the A controller:  1 == port 1, 2 == port 2, ... 8 == port 4,  9 == port 1 and 4
  "host-port-bits-b":8     ==>  bitmap of the connected ports from the B controller:  1 == port 1, 2 == port 2, ... 8 == port 4,  9 == port 1 and 4

[czobi]

OK, here is the configuration. Both servers connected to both controllers.  See below the output of the commands  you recommend. I also attached the MPIO configuration capture for both servers. Hope these help.

---

System Information
------------------
System Name: XXXXX
System Contact: XXXXX
System Location: XXXXX
System Information: XXXXX
Midplane Serial Number: **Confidential info erased**
Vendor Name: HPE
Product ID: MSA 2050 SAS
Product Brand: MSA Storage
SCSI Vendor ID: HPE
SCSI Product ID: MSA 2050 SAS
Enclosure Count: 1
Health: OK
Health Reason:
Other MC Status: Operational
PFU Status: Idle
Supported Locales: English (English), Spanish (español), French (français), German (Deutsch), Italian (italiano), Japanese (日本語),                                                                 Korean (한국어), Dutch (Nederlands), Chinese-Simplified (简体中文), Chinese-Traditional (繁體中文)

Controllers
-----------
Controller ID: A
Serial Number: **Confidential info erased**
Hardware Version: 1.0
CPLD Version: 1.9
MAC Address: 00:C0:FF:43:69:86
WWNN: 500C0FF3CF80F000
IP Address: 192.168.168.6
IP Subnet Mask: 255.255.255.0
IP Gateway: 192.168.168.168
Disks: 20
Virtual Pools: 2
Disk Groups: 5
System Cache Memory (MB): 8192
Host Ports: 4
Disk Channels: 2
Disk Bus Type: SAS
Status: Operational
Failed Over to This Controller: No
Fail Over Reason: Not applicable
Multi-core: Disabled
Health: OK
Health Reason:
Health Recommendation:
Position: Top
Phy Isolation: Enabled
Controller Redundancy Mode: Active-Active ULP
Controller Redundancy Status: Redundant

Controllers
-----------
Controller ID: B
Serial Number: **Confidential info erased**
Hardware Version: 1.0
CPLD Version: 1.9                    MAC Address: 00:C0:FF:43:6A:6D
WWNN: 500C0FF3CF80F000
IP Address: 192.168.168.7
IP Subnet Mask: 255.255.255.0
IP Gateway: 192.168.168.168
Disks: 20
Virtual Pools: 2
Disk Groups: 5
System Cache Memory (MB): 8192
Host Ports: 4
Disk Channels: 2
Disk Bus Type: SAS
Status: Operational
Failed Over to This Controller: No
Fail Over Reason: Not applicable
Multi-core: Disabled
Health: OK
Health Reason:
Health Recommendation:
Position: Bottom
Phy Isolation: Enabled
Controller Redundancy Mode: Active-Active ULP
Controller Redundancy Status: Redundant

                                           Controller A Versions        Controller B Versions
                                           ---------------------        ---------------------
Storage Controller CPU Type:               Broadwell 2200MHz            Broadwell 2200MHz
Bundle Version:                            VL270P008                    VL270P008
Base Bundle Version:                       V270                         V270
Build Date:                                Wed Oct  5 22:50:59 MDT 2022 Wed Oct  5 22:50:59 MDT 2022
Storage Controller Code Version:           VLS270R01-18                 VLS270R01-18
Storage Controller Code Baselevel:         VLS270R01-18                 VLS270R01-18
FU Version:                                6.30.27744x                  6.30.27744x
Storage Controller Loader Code Version:    31.000                       31.000
CAPI Version:                              3.20                         3.20
Management Controller Code Version:        VXM270P003-24                VXM270P003-24
Management Controller Loader Code Version: 6.30.27744x                  6.30.27744x
Management Controller Base Version:        VXM270P003-24                VXM270P003-24
Expander Controller Code Version:          3207                         3207
CPLD Code Version:                         1.9                          1.9
Hardware Version:                          1.0                          1.0
Host Interface Module Version:             2                            2
Host Interface Module Model:               10                           10
Backplane Type:                            7                            7
Host Interface Hardware (Chip) Version:    1                            1
Disk Interface Hardware (Chip) Version:    3                            3
SC Boot Memory Reference Code Version:     0.3.2.11                     0.3.2.11
CTK Version:                               No CTK present               No CTK present
FRU Versions
------------
Enclosure ID: 1
  Midplane Versions
  -----------------
  VPD Format Version: N/A
  VPD CRC: N/A                         Config Mismatch Version: N/A
  CPLD Version: N/A
  FRU Descriptor: N/A
  Part Number: 876133-001
  Midplane Serial Number: **Confidential info erased**
  Name             Location                                 Ctlr  Firmware Version Firmware Version (full) VPD Format Version VPD CRC                                                               
  CFG Format Version CFG CRC Boot Loader Version CPLD Version
  ----------------------------------------------------------------------------------------------------------------------------------
  Root Expander-A  Enclosure 1, Iom A                       A     N/A              N/A                     N/A             N/A                                                                   
  N/A                N/A     N/A                 N/A
  Root Expander-B  Enclosure 1, Iom B                       B     N/A              N/A                     N/A                N/A                                                                   
  N/A                N/A     N/A                 N/A
Name:
Firmware Version:
PSU DSP version:
VPD Format Version:
VPD CRC:
FRU Descriptor:
Part Number:
Power Supply Serial Number:

Ports Media    Target ID         Status        Speed(A) Health     Reason
Action                                                                                                                                              ------------------------------------------------------------------------------------------------------------------------------------
A1    SAS      500c0ff3cf80f000  Up            12Gb     OK
   Topo(C) Lanes Expected Active Lanes   Disabled Lanes
   -----------------------------------------------------
   Direct  4              4              0
A2    SAS      500c0ff3cf80f100  Up            12Gb     OK
   Topo(C) Lanes Expected Active Lanes   Disabled Lanes
   -----------------------------------------------------
   Direct  4              4              0
A3    SAS      500c0ff3cf80f200  Disconnected  Auto     N/A        There is no active connection to this host port.
  - If this host port is intentionally unused, no action is required.
  - Otherwise, use an appropriate interface cable to connect this host port to a switch or host.
  - If a cable is connected, check the cable and the switch or host for problems.
   Topo(C) Lanes Expected Active Lanes   Disabled Lanes
   -----------------------------------------------------
   Direct  4              0              0
A4    SAS      500c0ff3cf80f300  Disconnected  Auto     N/A        There is no active connection to this host port.
  - If this host port is intentionally unused, no action is required.
  - Otherwise, use an appropriate interface cable to connect this host port to a switch or host.
  - If a cable is connected, check the cable and the switch or host for problems.
   Topo(C) Lanes Expected Active Lanes   Disabled Lanes
   -----------------------------------------------------
   Direct  4              0              0
------------------------------------------------------------------------------------------------------------------------------------
Ports Media    Target ID         Status        Speed(A) Health     Reason
Action                                                                                                                                              
------------------------------------------------------------------------------------------------------------------------------------
B1    SAS      500c0ff3cf80f400  Disconnected  Auto     N/A        There is no active connection to this host port.
  - If this host port is intentionally unused, no action is required.
  - Otherwise, use an appropriate interface cable to connect this host port to a switch or host.
  - If a cable is connected, check the cable and the switch or host for problems.
   Topo(C) Lanes Expected Active Lanes   Disabled Lanes
   -----------------------------------------------------
   Direct  4              0              0
B2    SAS      500c0ff3cf80f500  Disconnected  Auto     N/A        There is no active connection to this host port.
  - If this host port is intentionally unused, no action is required.
  - Otherwise, use an appropriate interface cable to connect this host port to a switch or host.
  - If a cable is connected, check the cable and the switch or host for problems.
   Topo(C) Lanes Expected Active Lanes   Disabled Lanes
   -----------------------------------------------------
   Direct  4              0              0
B3    SAS      500c0ff3cf80f600  Up            12Gb     OK
   Topo(C) Lanes Expected Active Lanes   Disabled Lanes
   -----------------------------------------------------
   Direct  4              4              0
B4    SAS      500c0ff3cf80f700  Up            12Gb     OK
   Topo(C) Lanes Expected Active Lanes   Disabled Lanes
   -----------------------------------------------------
   Direct  4              4              0

Name Size Free Pool Tier % of Pool Own RAID Disks Status Current Job Job% Sec Fmt Health Reason
Action
------------------------------------------------------------------------------------------------------------------------------------
DiskGroupA01 3594.5GB 0B A Standard 28 A RAID5 4 FTOL 512n OK
DiskGroupA02 8989.9GB 1053.1GB A Standard 71 A RAID5 6 FTOL 512e OK
dgB01 7191.3GB 2563.0GB B Standard 100 B RAID5 7 FTOL 512n OK
rcA1 797.0GB 0B A Read Cache 0 A NRAID 1 UP 512e OK
rcB1 797.0GB 99.7GB B Read Cache 0 B NRAID 1 UP 512e OK
------------------------------------------------------------------------------------------------------------------------------------
Name Serial Number Blocksize Total Size Avail Snap Size OverCommit Disk Groups Volumes Low Thresh
Mid Thresh High Thresh Sec Fmt Health Reason Action
---------------------------------------------------------------------------------------------------------------------------
A 00c0ff43698600005034046001000000 512 12.5TB 1053.1GB 0B Enabled 3 4 50.00 %
75.00 % 98.29 % Mixed OK
B 00c0ff436a6d0000b7beb76601000000 512 7191.3GB 188.1GB 0B Disabled 2 2 50.00 %
75.00 % 97.01 % Mixed OK
---------------------------------------------------------------------------------------------------------------------------
Encl Encl WWN Name Location Rack Pos Vendor Model EMP A CH:ID Rev
EMP B CH:ID Rev Midplane Type Health Reason Action
------------------------------------------------------------------------------------------------------------------------
1 500C0FF03CF80F3C 0 0 HP SPS-CHASSIS 00:127 3207
01:127 3207 2U24-6Gv2 OK
------------------------------------------------------------------------------------------------------------------------
SKU
---
Part Number: Q1J29A
Serial Number: **Confidential info erased**
Revision: D1
Enclosure ID: 1

FRU
---
Name: CHASSIS_MIDPLANE
Description: SPS-MSA ES SFF Chassis
Part Number: 876133-001
Serial Number: **Confidential info erased**
Revision: C1
Dash Level:
FRU Shortname: Midplane/Chassis
Manufacturing Date: 2018-09-10 06:57:43
Manufacturing Location: Tianjin,TEDA,CN Manufacturing Vendor ID: 0x017C
FRU Location: MID-PLANE SLOT
Configuration SN: **Confidential info erased**
FRU Status: OK
Enclosure ID: 1

FRU
---
Name: RAID_IOM
Description: SPS-MSA 2050 SAS Controller
Part Number: 876129-001
Serial Number: **Confidential info erased**
Revision: D1
Dash Level:
FRU Shortname: RAID IOM
Manufacturing Date: 2018-10-06 01:37:55
Manufacturing Location: Tianjin,TEDA,CN
Manufacturing Vendor ID: 0x017C
FRU Location: UPPER IOM SLOT
Configuration SN: **Confidential info erased**
FRU Status: OK
Enclosure ID: 1

FRU
---
Name: RAID_IOM
Description: SPS-MSA 2050 SAS Controller
Part Number: 876129-001
Serial Number: **Confidential info erased**
Revision: D1
Dash Level:
FRU Shortname: RAID IOM
Manufacturing Date: 2018-10-06 06:20:29
Manufacturing Location: Tianjin,TEDA,CN
Manufacturing Vendor ID: 0x017C
FRU Location: LOWER IOM SLOT
Configuration SN: **Confidential info erased**
FRU Status: OK
Enclosure ID: 1

FRU
---
Name: POWER_SUPPLY
Description: FRU,Pwr Sply,595W,AC,2U,LC,HP ES
Part Number: 814665-001
Serial Number: **Confidential info erased**
Revision: C1
Dash Level:
FRU Shortname: AC Power Supply
Manufacturing Date: 2018-08-30 17:37:07
Manufacturing Location: Zhongshan,Guangdong,CN
Manufacturing Vendor ID:
FRU Location: LEFT PSU SLOT
Configuration SN: **Confidential info erased**
FRU Status: OK
Original SN: **Confidential info erased**
Original PN: 7001540-J000 Original Rev: AN
Enclosure ID: 1

FRU
---
Name: POWER_SUPPLY
Description: FRU,Pwr Sply,595W,AC,2U,LC,HP ES
Part Number: 814665-001
Serial Number: **Confidential info erased**
Revision: C1
Dash Level:
FRU Shortname: AC Power Supply
Manufacturing Date: 2018-08-30 07:56:52
Manufacturing Location: Zhongshan,Guangdong,CN
Manufacturing Vendor ID:
FRU Location: RIGHT PSU SLOT
Configuration SN: **Confidential info erased**
FRU Status: OK
Original SN: **Confidential info erased**
Original PN: 7001540-J000
Original Rev: AN
Enclosure ID: 1

FRU
---
Name: MEMORY CARD
Description: SPS Memory Card
Part Number: 768079-001
Serial Number:
Revision:
Dash Level:
FRU Shortname: Memory Card
Manufacturing Date: N/A
Manufacturing Location:
Manufacturing Vendor ID:
FRU Location: UPPER IOM MEMORY CARD SLOT
Configuration SN: **Confidential info erased**
FRU Status: OK
Enclosure ID: 1

FRU
---
Name: MEMORY CARD
Description: SPS Memory Card
Part Number: 768079-001
Serial Number:
Revision:
Dash Level:
FRU Shortname: Memory Card
Manufacturing Date: N/A
Manufacturing Location:
Manufacturing Vendor ID:
FRU Location: LOWER IOM MEMORY CARD SLOT
Configuration SN: **Confidential info erased**
FRU Status: OK
Enclosure ID: 1
Info: * Rates may vary. This is normal behavior.

---

Here are the command outputs.

#show-unwritable-cache:
Unwritable System Cache
-----------------------
Percent of unwritable cache in controller A: 0
Percent of unwritable cache in controller B: 0
# show controller-statistics
Durable ID CPU Load Power On Time (Secs) Bps IOPS
Reads Writes Data Read Data Written Num Forwarded Cmds
Reset Time Total Power On Hours
------------------------------------------------------------------------------------------
controller_A 1 604521 0B 0
56638026 23631468 7318.9GB 2263.7GB 0
2025-03-24 10:42:56 53058.27
controller_B 5 604439 29.7MB 3117
273654313 114444140 5456.1GB 4363.7GB 0
2025-03-24 10:43:04 53034.53
------------------------------------------------------------------------------------------

# show controller-statistics
Durable ID CPU Load Power On Time (Secs) Bps IOPS
Reads Writes Data Read Data Written Num Forwarded Cmds
Reset Time Total Power On Hours
------------------------------------------------------------------------------------------
controller_A 1 604521 0B 0
56638026 23631468 7318.9GB 2263.7GB 0
2025-03-24 10:42:56 53058.27
controller_B 5 604439 29.7MB 3117
273654313 114444140 5456.1GB 4363.7GB 0
2025-03-24 10:43:04 53034.53
------------------------------------------------------------------------------------------

# show initiators
{

"initiator":[
{
"object-name":"initiator",
"meta":"/meta/initiator",
"durable-id":"I0",
"nickname":"Host1_2",
"discovered":"Yes",
"mapped":"No",
"profile":"Standard",
"profile-numeric":0,
"host-bus-type":"SAS",
"host-bus-type-numeric":8,
"id":"51402ec010f36ce0",
"url":"/initiators/51402ec010f36ce0",
"host-id":"00c0ff436a6d000076cd726601010000",
"host-key":"H1",
"host-port-bits-a":1,
"host-port-bits-b":0
},
{
"object-name":"initiator",
"meta":"/meta/initiator",
"durable-id":"I1",
"nickname":"Host2_2",
"discovered":"Yes",
"mapped":"No",
"profile":"Standard",
"profile-numeric":0,
"host-bus-type":"SAS",
"host-bus-type-numeric":8,
"id":"51402ec010f371c4",
"url":"/initiators/51402ec010f371c4",
"host-id":"00c0ff436a6d000098cd726601010000",
"host-key":"H2",
"host-port-bits-a":2,
"host-port-bits-b":0
},
{
"object-name":"initiator",
"meta":"/meta/initiator",
"durable-id":"I2",
"nickname":"Host1_1",
"discovered":"Yes",
"mapped":"No",
"profile":"Standard",
"profile-numeric":0,
"host-bus-type":"SAS",
"host-bus-type-numeric":8,
"id":"51402ec010f36ce4",
"url":"/initiators/51402ec010f36ce4",
"host-id":"00c0ff436a6d000076cd726601010000",
"host-key":"H1",
"host-port-bits-a":0, "host-port-bits-b":4
},
{
"object-name":"initiator",
"meta":"/meta/initiator",
"durable-id":"I3",
"nickname":"Host2_1",
"discovered":"Yes",
"mapped":"No",
"profile":"Standard",
"profile-numeric":0,
"host-bus-type":"SAS",
"host-bus-type-numeric":8,
"id":"51402ec010f371c0",
"url":"/initiators/51402ec010f371c0",
"host-id":"00c0ff436a6d000098cd726601010000",
"host-key":"H2",
"host-port-bits-a":0,
"host-port-bits-b":8
}
],
"status":[
{
"object-name":"status",
"meta":"/meta/status",
"response-type":"Success",
"response-type-numeric":0,
"response":"Command completed successfully. (2025-03-31 10:45:08)",
"return-code":0,
"component-id":"",
"time-stamp":"2025-03-31 10:45:08",
"time-stamp-numeric":1743417908
}
]
}

MPIO configuration on HOST1.

MPIO Storage Snapshot on Monday, 31 March 2025, at 22:39:22.574

Registered DSMs: 1
================
+--------------------------------|-------------------|----|----|----|---|-----+
|DSM Name | Version |PRP | RC | RI |PVP| PVE |
|--------------------------------|-------------------|----|----|----|---|-----|
|Microsoft DSM |010.0000.20348.0001|0020|0003|0001|030|False|
+--------------------------------|-------------------|----|----|----|---|-----+

Microsoft DSM
=============
MPIO Disk6: 01 Paths, Round Robin with Subset, ALUA Not Supported
SN: **Confidential info erased**
Supported Load Balance Policies: FOO RR RRWS LQD WP LB

Path ID State SCSI Address Weight
---------------------------------------------------------------------------
0000000077020001 Active/Optimized 002|000|001|000 0
Adapter: HPE Smart Array P408i-a SR Gen10... (B|D|F: 092|000|000)
Controller: 46616B65436F6E74726F6C6C6572 (State: Active)

MPIO Disk5: 02 Paths, Least Blocks, Implicit Only
SN: **Confidential info erased**
Supported Load Balance Policies: FOO RRWS LQD WP LB

Path ID State SCSI Address Weight
---------------------------------------------------------------------------
0000000077010103 Active/Optimized 001|001|003|006 0
* TPG_State: Active/Optimized , TPG_Id: 0, TP_Id: 1
Adapter: HPE Smart Array E208e-p SR Gen10... (B|D|F: 018|000|000)
Controller: 46616B65436F6E74726F6C6C6572 (State: Active)

0000000077010102 Active/Unoptimized 001|001|002|006 0
TPG_State: Active/Unoptimized, TPG_Id: 1, TP_Id: 7
Adapter: HPE Smart Array E208e-p SR Gen10... (B|D|F: 018|000|000)
Controller: 46616B65436F6E74726F6C6C6572 (State: Active)

MPIO Disk4: 02 Paths, Round Robin with Subset, Implicit Only
SN: **Confidential info erased**
Supported Load Balance Policies: FOO RRWS LQD WP LB

Path ID State SCSI Address Weight
---------------------------------------------------------------------------
0000000077010103 Active/Optimized 001|001|003|005 0
* TPG_State: Active/Optimized , TPG_Id: 0, TP_Id: 1
Adapter: HPE Smart Array E208e-p SR Gen10... (B|D|F: 018|000|000)
Controller: 46616B65436F6E74726F6C6C6572 (State: Active)

0000000077010102 Active/Unoptimized 001|001|002|005 0
TPG_State: Active/Unoptimized, TPG_Id: 1, TP_Id: 7
Adapter: HPE Smart Array E208e-p SR Gen10... (B|D|F: 018|000|000)
Controller: 46616B65436F6E74726F6C6C6572 (State: Active)

MPIO Disk3: 02 Paths, Round Robin with Subset, Implicit Only
SN: **Confidential info erased**
Supported Load Balance Policies: FOO RRWS LQD WP LB

Path ID State SCSI Address Weight
---------------------------------------------------------------------------
0000000077010103 Active/Optimized 001|001|003|004 0
* TPG_State: Active/Optimized , TPG_Id: 0, TP_Id: 1
Adapter: HPE Smart Array E208e-p SR Gen10... (B|D|F: 018|000|000)
Controller: 46616B65436F6E74726F6C6C6572 (State: Active)

0000000077010102 Active/Unoptimized 001|001|002|004 0
TPG_State: Active/Unoptimized, TPG_Id: 1, TP_Id: 7
Adapter: HPE Smart Array E208e-p SR Gen10... (B|D|F: 018|000|000)
Controller: 46616B65436F6E74726F6C6C6572 (State: Active)

MPIO Disk2: 02 Paths, Round Robin with Subset, Implicit Only
SN: **Confidential info erased**
Supported Load Balance Policies: FOO RRWS LQD WP LB

Path ID State SCSI Address Weight
---------------------------------------------------------------------------
0000000077010103 Active/Optimized 001|001|003|003 0
* TPG_State: Active/Optimized , TPG_Id: 0, TP_Id: 1
Adapter: HPE Smart Array E208e-p SR Gen10... (B|D|F: 018|000|000)
Controller: 46616B65436F6E74726F6C6C6572 (State: Active)

0000000077010102 Active/Unoptimized 001|001|002|003 0
TPG_State: Active/Unoptimized, TPG_Id: 1, TP_Id: 7
Adapter: HPE Smart Array E208e-p SR Gen10... (B|D|F: 018|000|000)
Controller: 46616B65436F6E74726F6C6C6572 (State: Active)

MPIO Disk1: 02 Paths, Round Robin with Subset, Implicit Only
SN: **Confidential info erased**
Supported Load Balance Policies: FOO RRWS LQD WP LB

Path ID State SCSI Address Weight
---------------------------------------------------------------------------
0000000077010103 Active/Unoptimized 001|001|003|002 0
TPG_State: Active/Unoptimized, TPG_Id: 0, TP_Id: 1
Adapter: HPE Smart Array E208e-p SR Gen10... (B|D|F: 018|000|000)
Controller: 46616B65436F6E74726F6C6C6572 (State: Active)

0000000077010102 Active/Optimized 001|001|002|002 0
* TPG_State: Active/Optimized , TPG_Id: 1, TP_Id: 7
Adapter: HPE Smart Array E208e-p SR Gen10... (B|D|F: 018|000|000)
Controller: 46616B65436F6E74726F6C6C6572 (State: Active)

MPIO Disk0: 02 Paths, Round Robin with Subset, Implicit Only
SN: **Confidential info erased**
Supported Load Balance Policies: FOO RRWS LQD WP LB

Path ID State SCSI Address Weight
---------------------------------------------------------------------------
0000000077010103 Active/Unoptimized 001|001|003|001 0
TPG_State: Active/Unoptimized, TPG_Id: 0, TP_Id: 1
Adapter: HPE Smart Array E208e-p SR Gen10... (B|D|F: 018|000|000)
Controller: 46616B65436F6E74726F6C6C6572 (State: Active)

0000000077010102 Active/Optimized 001|001|002|001 0
* TPG_State: Active/Optimized , TPG_Id: 1, TP_Id: 7
Adapter: HPE Smart Array E208e-p SR Gen10... (B|D|F: 018|000|000)
Controller: 46616B65436F6E74726F6C6C6572 (State: Active)

MSDSM-wide default load balance policy: N\A

No target-level default load balance policies have been set.

================================================================================

MPIO Configuration on HOST2.

MPIO Storage Snapshot on Monday, 31 March 2025, at 22:41:17.486

Registered DSMs: 1
================
+--------------------------------|-------------------|----|----|----|---|-----+
|DSM Name | Version |PRP | RC | RI |PVP| PVE |
|--------------------------------|-------------------|----|----|----|---|-----|
|Microsoft DSM |010.0000.20348.0001|0020|0003|0001|030|False|
+--------------------------------|-------------------|----|----|----|---|-----+

Microsoft DSM
=============
MPIO Disk6: 01 Paths, Least Queue Depth, ALUA Not Supported
SN: **Confidential info erased**
Supported Load Balance Policies: FOO RR RRWS LQD WP LB

Path ID State SCSI Address Weight
---------------------------------------------------------------------------
0000000077020001 Active/Optimized 002|000|001|000 0
Adapter: HPE Smart Array P408i-a SR Gen10... (B|D|F: 092|000|000)
Controller: 46616B65436F6E74726F6C6C6572 (State: Active)

MPIO Disk5: 02 Paths, Least Blocks, Implicit Only
SN: **Confidential info erased**
Supported Load Balance Policies: FOO RRWS LQD WP LB

Path ID State SCSI Address Weight
---------------------------------------------------------------------------
0000000077010103 Active/Unoptimized 001|001|003|006 0
TPG_State: Active/Unoptimized, TPG_Id: 1, TP_Id: 8
Adapter: HPE Smart Array E208e-p SR Gen10... (B|D|F: 018|000|000)
Controller: 46616B65436F6E74726F6C6C6572 (State: Active)

0000000077010102 Active/Optimized 001|001|002|006 0
* TPG_State: Active/Optimized , TPG_Id: 0, TP_Id: 2
Adapter: HPE Smart Array E208e-p SR Gen10... (B|D|F: 018|000|000)
Controller: 46616B65436F6E74726F6C6C6572 (State: Active)

MPIO Disk4: 02 Paths, Round Robin with Subset, Implicit Only
SN: **Confidential info erased**
Supported Load Balance Policies: FOO RRWS LQD WP LB

Path ID State SCSI Address Weight
---------------------------------------------------------------------------
0000000077010103 Active/Unoptimized 001|001|003|005 0
TPG_State: Active/Unoptimized, TPG_Id: 1, TP_Id: 8
Adapter: HPE Smart Array E208e-p SR Gen10... (B|D|F: 018|000|000)
Controller: 46616B65436F6E74726F6C6C6572 (State: Active)

0000000077010102 Active/Optimized 001|001|002|005 0
* TPG_State: Active/Optimized , TPG_Id: 0, TP_Id: 2
Adapter: HPE Smart Array E208e-p SR Gen10... (B|D|F: 018|000|000)
Controller: 46616B65436F6E74726F6C6C6572 (State: Active)

MPIO Disk3: 02 Paths, Round Robin with Subset, Implicit Only
SN: **Confidential info erased**
Supported Load Balance Policies: FOO RRWS LQD WP LB

Path ID State SCSI Address Weight
---------------------------------------------------------------------------
0000000077010103 Active/Unoptimized 001|001|003|004 0
TPG_State: Active/Unoptimized, TPG_Id: 1, TP_Id: 8
Adapter: HPE Smart Array E208e-p SR Gen10... (B|D|F: 018|000|000)
Controller: 46616B65436F6E74726F6C6C6572 (State: Active)

0000000077010102 Active/Optimized 001|001|002|004 0
* TPG_State: Active/Optimized , TPG_Id: 0, TP_Id: 2
Adapter: HPE Smart Array E208e-p SR Gen10... (B|D|F: 018|000|000)
Controller: 46616B65436F6E74726F6C6C6572 (State: Active)

MPIO Disk2: 02 Paths, Round Robin with Subset, Implicit Only
SN: **Confidential info erased**
Supported Load Balance Policies: FOO RRWS LQD WP LB

Path ID State SCSI Address Weight
---------------------------------------------------------------------------
0000000077010103 Active/Unoptimized 001|001|003|003 0
TPG_State: Active/Unoptimized, TPG_Id: 1, TP_Id: 8
Adapter: HPE Smart Array E208e-p SR Gen10... (B|D|F: 018|000|000)
Controller: 46616B65436F6E74726F6C6C6572 (State: Active)

0000000077010102 Active/Optimized 001|001|002|003 0
* TPG_State: Active/Optimized , TPG_Id: 0, TP_Id: 2
Adapter: HPE Smart Array E208e-p SR Gen10... (B|D|F: 018|000|000)
Controller: 46616B65436F6E74726F6C6C6572 (State: Active)

MPIO Disk1: 02 Paths, Round Robin with Subset, Implicit Only
SN: **Confidential info erased**
Supported Load Balance Policies: FOO RRWS LQD WP LB

Path ID State SCSI Address Weight
---------------------------------------------------------------------------
0000000077010103 Active/Optimized 001|001|003|002 0
* TPG_State: Active/Optimized , TPG_Id: 1, TP_Id: 8
Adapter: HPE Smart Array E208e-p SR Gen10... (B|D|F: 018|000|000)
Controller: 46616B65436F6E74726F6C6C6572 (State: Active)

0000000077010102 Active/Unoptimized 001|001|002|002 0
TPG_State: Active/Unoptimized, TPG_Id: 0, TP_Id: 2
Adapter: HPE Smart Array E208e-p SR Gen10... (B|D|F: 018|000|000)
Controller: 46616B65436F6E74726F6C6C6572 (State: Active)

MPIO Disk0: 02 Paths, Round Robin with Subset, Implicit Only
SN: **Confidential info erased**
Supported Load Balance Policies: FOO RRWS LQD WP LB

Path ID State SCSI Address Weight
---------------------------------------------------------------------------
0000000077010103 Active/Optimized 001|001|003|001 0
* TPG_State: Active/Optimized , TPG_Id: 1, TP_Id: 8
Adapter: HPE Smart Array E208e-p SR Gen10... (B|D|F: 018|000|000)
Controller: 46616B65436F6E74726F6C6C6572 (State: Active)

0000000077010102 Active/Unoptimized 001|001|002|001 0
TPG_State: Active/Unoptimized, TPG_Id: 0, TP_Id: 2
Adapter: HPE Smart Array E208e-p SR Gen10... (B|D|F: 018|000|000)
Controller: 46616B65436F6E74726F6C6C6572 (State: Active)

MSDSM-wide default load balance policy: N\A

No target-level default load balance policies have been set.

================================================================================

[JonPaul]

Thanks for the info,
Some of the things to review were static and some will need to be monitored in an on-going basis.
It does show that there is no unwriteable cache in the MSA controllers and that the MPIO looks to be setup correctly,  There is 1 disk which is showing 'Least Blocks' as a load balancer policy:
     MPIO Disk5: 02 Paths, Least Blocks, Implicit Only
     SN: **confidential info erased**

But this is on both hosts so is not different and unlikely to be the issue between the hosts.

[czobi]

Thanks for your help!

Today I reinstalled HOST1 completely and it seems the issue had gone. The CPU load slightly increases (~15-20%) on both controllers after several VMs live migrated but the overall performance does not drop. Read caches on both controllers behaves normally.

I will be watching for a while to be sure.  

[czobi]

My optimism was a little premature. The CPU load on both MSA controllers was slowly increasing during the night reaching about 50-60%. Performance also dropped significantly. I migrated back every VMs to HOST2 and all issues were gone.

What else should I check to find a solution for the issues?