8200 - FC Free Capacity Critical. - PEAK0072

[starlen]

Thin provisioned VV xxx unable to allocate SD space from CPG FC_r1.

need some help on how to free up some space, lock hold on VV 

any ideas on how 

showpd -c -p -devtype FC
-------- Normal Chunklets --------- ---- Spare Chunklets ----
-- Used -- -------- Unused -------- - Used - ---- Unused ----
Id CagePos Type State Total OK Fail Free Uninit Unavail Fail OK Fail Free Uninit Fail
0 0:0:0 FC normal 1675 1544 0 0 0 0 0 0 0 131 0 0
1 0:1:0 FC normal 1675 1543 0 1 0 0 0 0 0 131 0 0
2 0:2:0 FC normal 1675 1544 0 0 0 0 0 0 0 131 0 0
3 0:3:0 FC normal 1675 1543 0 1 0 0 0 0 0 131 0 0
4 0:4:0 FC normal 1675 1544 0 0 0 0 0 0 0 131 0 0
5 0:5:0 FC normal 1675 1542 0 2 0 0 0 0 0 131 0 0
6 0:6:0 FC normal 1675 1544 0 0 0 0 0 0 0 131 0 0
7 0:7:0 FC normal 1675 1544 0 0 0 0 0 0 0 131 0 0
8 0:8:0 FC normal 1675 1544 0 0 0 0 0 0 0 131 0 0
9 0:9:0 FC normal 1675 1543 0 1 0 0 0 0 0 131 0 0
10 0:10:0 FC normal 1675 1545 0 0 0 0 0 0 0 130 0 0
11 0:11:0 FC normal 1675 1541 0 4 0 0 0 0 0 130 0 0
12 0:12:0 FC normal 1675 1545 0 0 0 0 0 0 0 130 0 0
13 0:13:0 FC normal 1675 1544 0 1 0 0 0 0 0 130 0 0
14 0:14:0 FC normal 1675 1544 0 1 0 0 0 0 0 130 0 0
15 0:15:0 FC normal 1675 1545 0 0 0 0 0 0 0 130 0 0
16 0:16:0 FC normal 1675 1545 0 0 0 0 0 0 0 130 0 0
17 0:17:0 FC normal 1675 1543 0 2 0 0 0 0 0 130 0 0
------------------------------------------------------------------------------------------
18 total 30150 27787 0 13 0 0 0 0 0 2350 0 0
3PAR_01_ITHEALTH cli% showcpg -sdg
-----(MiB)------
Id Name Warn Limit Grow Args
0 FC_r1 - - 32768 -ssz 2 -ha mag -t r1 -p -devtype FC
1 FC_r5 - - 32768 -ssz 5 -ha mag -t r5 -p -devtype FC
2 FC_r5_COPYCPG - - 32768 -t r5 -ha mag -ssz 4 -ss 128 -ch first -p -devtype FC
3 FC_r6 - - 32768 -ssz 8 -ha mag -t r6 -p -devtype FC
3PAR_01_ITHEALTH cli% showvv
-Rsvd(MiB)-- -(MiB)--
Id Name Prov Compr Dedup Type CopyOf BsId Rd -Detailed_State- Snp Usr VSize
1 .srdata full NA NA base --- 1 RW normal 0 61440 61440
0 admin full NA NA base --- 0 RW normal 0 10240 10240
2 ITHEALTH_DATASTORE01 full NA NA base --- 2 RW normal 0 15433728 15433728
7 OHS tpvv No No base --- 7 RW normal 512 4398464 7680000
---------------------------------------------------------------------------------------------------
4 total 512 19903872 23185408

[sbhat09]

Hello @starlen,

In the 'showpd -c' output you shared, chunklets are almost completely used. Those chunklets in spare, can't be used for data. System requires them in case of drive failures.

If you are looking to free up the space used by the VV, consider to tune the VV (Select the VV -> Actions -> Tune) and move it to another CPG of RAID-5 or RAID-6. So that it occupies lesser diskspace than it's present RAID-1.
If holding key is the issue. You can follow the below:
[Please run this command to check if there are any migration key stuck to the volume.
showvv -allkeys
Then run this command to clear the migration key
setvv –clrkey <keyname> <vvname>]

To free up some space for the CPG, you can run 'compactcpg' command
[For dry run in advance: compactcpg -dr <CPG_name>
To initiate compactcpg: compactcpg <CPG_name>]

Regards,
Srinivas Bhat

If you feel this was helpful please click the KUDOS! thumb below!
Note: All of my comments are my own and are not any official representation of the company.

[starlen]

does this show which keys are stuck to the volume, 

3PAR_01_ITHEALTH cli% showvv -allkeys
Id Name Key Value Timestamp
1 .srdata sr_owner 1 Sat Feb 08 10:10:36 SAST 2020
0 admin defaultcpg-FC-r1 1 Mon May 29 11:11:23 SAST 2017
0 admin defaultcpg-FC-r5 1 Mon May 29 11:11:23 SAST 2017
0 admin defaultcpg-FC-r6 1 Sun Mar 24 17:09:25 SAST 2019
---------------------------------------------------------------

Initiating tune virtual volume

OHS VV 

Jun 8, 2022 9:44:37 AM SAST

< 1 minute ago

2FF70002AC01C33B : Error: Could not allocate enough space from CPG FC_r1 to complete tune operations. Attempted to allocate 4423680 MB of SD space

task detail 

Jun 8, 2022 9:44:37 AM SAST Created
Jun 8, 2022 9:44:37 AM SAST Start user operation
Jun 8, 2022 9:44:37 AM SAST Executing tunealdvv operation on the array
Jun 8, 2022 9:44:37 AM SAST Executing tunetpvv operation on the array
Jun 8, 2022 9:44:37 AM SAST 2FF70002AC01C33B : Error: Could not allocate enough space from CPG FC_r1 to complete tune operations. Attempted to allocate 4423680 MB of SD space
Jun 8, 2022 9:44:37 AM SAST Failed: 2FF70002AC01C33B : Error: Could not allocate enough space from CPG FC_r1 to complete tune operations. Attempted to allocate 4423680 MB of SD space

          

[sbhat09]

Hello @starlen,

showvv -allkeys output is not showing any keys stuck. My statement was to address any such possibilities as you mentioned 'lock hold on VV' in your original post.

While you are doing tunevv, from which CPG to which CPG are you trying to move the VV? I mean which is the target CPG for tunevv?

If you are moving the VV within the same dev type (FC), there must be some FC chunklets available. Otherwise, the operation fails. If you are moving the VV to some other dev type (NL or SSD), the space availability in that dev type must be more than the used space of the VV.

The present condition looks like there is a severe space crunch in the system or in FC drives.

Regards,
Srinivas Bhat

If you feel this was helpful please click the KUDOS! thumb below!
Note: All of my comments are my own and are not any official representation of the company.

[sbhat09]

Hello @starlen,

Could you get that issue resolved? Please let me know if you need further assistance.

Regards,
Srinivas Bhat

If you feel this was helpful please click the KUDOS! thumb below!
Note: All of my comments are my own and are not any official representation of the company.

[starlen]

the issue is still not resolved 

[SUBHAJIT KHANBARMAN_1]

@starlen 

Cancel the tune task whatever you have started.

Check command outout "showsys -param" and look for "SparingAlgorithm" value. If this is set as "Default" or "Maximum" then we can change this to "Minimal". This will release some chunklets from Spare reserved to data space for use. Once you add new drives then change back this policy to previous one which was set before.

1. Please set system parameter & do admithw.

$ setsys SparingAlgorithm Minimal

2. Check system parameter.

$ showsys –param

3. Release chunklet & check free chunks on FC drives

$ admithw spares
$ showpd -c

https://support.hpe.com/hpesc/public/docDisplay?docId=c04204279 (page no 405)

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

********************************************************************************