significant I/O on the NL virtual volumes with no host activity

[djryan]

We have a strange I/O issue on our RAID 6 NL volumes on 24 disk. With absolutely no I/O from the host we are seeing non-stop I/O on these disks. I've included output from the SAN measuring this I/O. Any help would be appreciated. This is causing major issues at the host when we are running our own I/O.

Disk 14-25 and 40-51.

Dave

TTP-STORAGE-8400 cli% checkhealth -detail
Checking alert
Checking ao
Checking cabling
Checking cage
Checking cert
Checking dar
Checking date
Checking file
Checking fs
Checking host
Checking ld
Checking license
Checking network
Checking node
Checking pd
Checking pdch
Checking port
Checking qos
Checking rc
Checking snmp
Checking task
Checking vlun
Checking vv
Checking sp
Component ---------------Summary Description--------------- Qty
Alert New alerts 2
PD Disks experiencing a high level of I/O per second 24
---------------------------------------------------------------
2 total 26

Component -Identifier- -----------------------Detailed Description-----------------------
Alert sw_os An Update is Available
Alert sw_sysmgr Total SSD raw space usage at 178153G (above 88% of total 200200G).
PD disk:14 Disk is experiencing a high level of I/O per second: 274.4
PD disk:15 Disk is experiencing a high level of I/O per second: 291.2
PD disk:16 Disk is experiencing a high level of I/O per second: 289.0
PD disk:17 Disk is experiencing a high level of I/O per second: 276.8
PD disk:18 Disk is experiencing a high level of I/O per second: 281.6
PD disk:19 Disk is experiencing a high level of I/O per second: 295.6
PD disk:20 Disk is experiencing a high level of I/O per second: 291.4
PD disk:21 Disk is experiencing a high level of I/O per second: 292.8
PD disk:22 Disk is experiencing a high level of I/O per second: 284.6
PD disk:23 Disk is experiencing a high level of I/O per second: 290.0
PD disk:24 Disk is experiencing a high level of I/O per second: 298.2
PD disk:25 Disk is experiencing a high level of I/O per second: 292.8
PD disk:40 Disk is experiencing a high level of I/O per second: 277.2
PD disk:41 Disk is experiencing a high level of I/O per second: 281.4
PD disk:42 Disk is experiencing a high level of I/O per second: 279.0
PD disk:43 Disk is experiencing a high level of I/O per second: 280.4
PD disk:44 Disk is experiencing a high level of I/O per second: 269.0
PD disk:45 Disk is experiencing a high level of I/O per second: 294.2
PD disk:46 Disk is experiencing a high level of I/O per second: 286.8
PD disk:47 Disk is experiencing a high level of I/O per second: 271.2
PD disk:48 Disk is experiencing a high level of I/O per second: 280.2
PD disk:49 Disk is experiencing a high level of I/O per second: 289.4
PD disk:50 Disk is experiencing a high level of I/O per second: 298.0
PD disk:51 Disk is experiencing a high level of I/O per second: 272.2
-----------------------------------------------------------------------------------------
26 total

TTP-STORAGE-8400 cli% statpd -d 5 -iter 1 -p -cg 1 -mg 0 -pn 0 -rw
12:12:01 05/31/2019 r/w I/O per second KBytes per sec Svt ms IOSz KB Idle %
ID Port Cur Avg Max Cur Avg Max Cur Avg Cur Avg Qlen Cur Avg
14 0:1:2 r 165 165 165 3732 3732 3732 67.33 67.33 22.6 22.6 - - -
14 0:1:2 w 162 162 162 3393 3393 3393 83.43 83.43 20.9 20.9 - - -
14 0:1:2 t 327 327 327 7125 7125 7125 75.30 75.30 21.8 21.8 25 0 0
14 1:1:2 r 0 0 0 0 0 0 0.00 0.00 0.0 0.0 - - -
14 1:1:2 w 0 0 0 0 0 0 0.00 0.00 0.0 0.0 - - -
14 1:1:2 t 0 0 0 0 0 0 0.00 0.00 0.0 0.0 0 100 100
----------------------------------------------------------------------------------------
2 r 165 165 3732 3732 67.33 67.33 22.6 22.6 - - -
2 w 162 162 3393 3393 83.43 83.43 20.9 20.9 - - -
2 t 327 327 7125 7125 75.30 75.30 21.8 21.8 25 50 50

TTP-STORAGE-8400 cli% statpd -d 5 -iter 1 -p -cg 1 -mg 0 -pn 0 -rw
12:13:20 05/31/2019 r/w I/O per second KBytes per sec Svt ms IOSz KB Idle %
ID Port Cur Avg Max Cur Avg Max Cur Avg Cur Avg Qlen Cur Avg
14 0:1:2 r 180 180 180 3947 3947 3947 69.66 69.66 21.9 21.9 - - -
14 0:1:2 w 179 179 179 3688 3688 3688 99.68 99.68 20.6 20.6 - - -
14 0:1:2 t 359 359 359 7635 7635 7635 84.62 84.62 21.2 21.2 29 0 0
14 1:1:2 r 0 0 0 0 0 0 0.00 0.00 0.0 0.0 - - -
14 1:1:2 w 0 0 0 0 0 0 0.00 0.00 0.0 0.0 - - -
14 1:1:2 t 0 0 0 0 0 0 0.00 0.00 0.0 0.0 0 100 100
----------------------------------------------------------------------------------------
2 r 180 180 3947 3947 69.66 69.66 21.9 21.9 - - -
2 w 179 179 3688 3688 99.68 99.68 20.6 20.6 - - -
2 t 359 359 7635 7635 84.62 84.62 21.2 21.2 29 50 50

[Gary_L]

I would start with reviewing showvvcpg, this will confirm the volumes created in each CPG

Then use SSMC to check exported volume performance on the volumes created in NL

Alternately you could use the CLI to grab some real time exported volume info and specifically review the V's identified in the first step:
statvlun -rw -vvsum

Or srstatvlun -btsecs -6h -hires (to go back 6 hours) and compare against the same timeframe for NL PD's

Gary

[djryan]

Thank you very much I greatly appreciate your recommendations, in fact I was able to diagnose my problem from your commands. Thank you so much.

Dave