HPE 3PAR 9450: slow drive - high write latency

[MarioE]

In the last 4 weeks we have had two drives with a high write latency and a high service time:

cli% statpd -d 15 -iter 1 -ni
13:06:36 05/31/2025 r/w I/O per second KBytes per sec Svt ms IOSz KB Idle %
ID Port Cur Avg Max Cur Avg Max Cur Avg Cur Avg Qlen Cur Avg
...
32 3:1:3 t 501 501 501 8202 8202 8202 520.249 520.249 16.4 16.4 256 0 0
...

The drives had such a high service time that the entire storage was almost at a standstill. Around 1000 virtual machines were very slow, in some cases barely accessible.

The drives had to be set to failed by HPE, which brought the write latency of the 3PAR back into the normal range.
Both drives are the DOPM3840S5xnNMRI model with firmware revision 3P03.

Support did not find the problem with the first drive. After 12 hours, our Technology Consultant was able to solve the problem and set the drive to failed.
With the second drive, we were able to tell support directly to set the drive to failed. But even after more than 3 hours, HPE Support still did not take any action, so our Technology Consultant had to set the disk to failed again.

Does anyone know this problem? How can I set the disk to failed myself, without the support.

[Cali]

Hi,

the save Process is:

setpd ldalloc off <DiskID> To stop writing new data to the drive
showpd -s To check the current status of the PD and make sure disks are in the Degraded state
movepdtospare -f -vacate -nowait <DiskID> Vacate all data from PD
showpdch -mov See the remaining chunklets to be moved, which are the PDID, and Y is the Chunklet number. Repeat till the disk is empty
showpdch -spr <DiskID> Shows the Spare chunklets on the PD
removespare PDID:a Removes the Spare chunklets from the PD
dismisspd <DiskID> Dismisses the PD from the 3PAR
tunesys

Cali

[MarioE]

@Cali 

Thanks for the information. Have you ever had this problem and have you ever gone through the process like this?

I have executed the command:

cli% setpd ldalloc off 32
cli% showpd -s 32
Id CagePos Type -State-- -------Detailed_State-------- -SedState--
32 7:6:0 SSD degraded not_available_for_allocations not_capable

But I still had the same high service time on the drive.

cli% statpd -d 15 -iter 1 -ni
13:34:17 05/31/2025 r/w I/O per second KBytes per sec Svt ms IOSz KB Idle %
ID Port Cur Avg Max Cur Avg Max Cur Avg Cur Avg Qlen Cur Avg
32 3:1:3 t 494 494 494 8229 8229 8229 521.537 521.537 16.7 16.7 258 0 0

I was told by HPE that only support can set the drive to failed with root privileges. setpd ldalloc off <ID> does not set the drive to failed and does not prevent the load on the drive.

[veeyarvi]

Hi MarioE,

1. You can remove both paths from the PD as below:

cli%showpd -i   >> Note down the wwn of the PD in question

cli%controlpd path rem both <PD wwn>

2. Another option may be to power recycle the slow drive using the below command (after ldalloc off) until support is engaged.

setpd ldalloc off <pd id>
cgcmd -tc tty -C cage2 -p a -c "poweroffdrive 0" (where the drive in Cage2 Slot 0 will be powered off)
wait for a minute
cgcmd -tc tty -C cageX -p a -c "powerondrive 0"

Regards,
Veeyaarvi

[MarioE]

Hi Veeyaarvi

Thanks for the information.
Are these the official commands that the customer can execute for this problem?
As a customer, can I really not set the drive to failed?
According to the “HPE 3PAR Command Line Interface reference guide”:

controlpd

Syntax

controlpd spinup|spindown [options] <WWN>...

Description

The controlpd command spins a PD up or down. This command is used when replacing a PD in a drive magazine.

Options -ovrd

Specifies that the operation is forced, even if the PD is in use.

Specifiers <WWN>...

Specifies the World Wide Name of the PD. This specifier can be repeated to identify multiple PDs.

Subcommands spinup

Specifies that the PD is to spin up. If this subcommand is not used, then the spindown subcommand must be used.

spindown

Specifies that the PD is to spin down. If this subcommand is not used, then the spinup subcommand must be used.

Authority

• Super, Service

• Any role granted the pd_control right

Usage

• Access to all domains is required to run this command.

• The spin down operation cannot be performed on a PD that is in use unless the -ovrd option is used.

• Issuing the controlpd command puts the specified disk drive in a not ready state. Further, if this command is issued with the spindown subcommand, data on the specified drive becomes inaccessible.

Examples

The following example instigates the spin up of a PD identified by its WWN of 2000000087002078:

cli% controlpd spinup 2000000087002078 

---

I cannot find the command "cgcmd":

cli% help cgcmd
No help for cgcmd

[veeyarvi]

Hi MarioE,

cgcmd is used as a service command - it communicates directly with the enclosure and can differ between array models. In fact, for 9000 arrays, this command may not work since it uses a different model of cages than 3PAR 7000/8000 models. Support may help with the right options if you need. Otherwise, you can use the 'spindown' option with controlpd command.

The best option would be the path removal from both loops (option #1) in my previous output.

Regards,
Veeyaarvi

[support_s]

Hello,

Let us know if you were able to resolve the issue.

If you are satisfied with the answers then kindly click the "Accept As Solution" button for the most helpful response so that it is beneficial to all community members.

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".