P2000 G3 fault tolerant but with down disk(s)

[SJO]

Hi Community

i've got no idea. My P2000 G3 (13 Disks with 2 Spares) ist Degraded while Hwalth Reason is: The virtual disk is fault tolerant but with down disk(s).

Can anybody help me to fix this? 

Thank You, SJO

[iamjawad]

Hello,

Looks like the Vdisk is on Raid 5?

Have you assigned the other two spare as global spare?

If not, please do

1. In the Configuration View panel, right-click the system and select Provisioning > Manage Global Spares. The main panel shows information about available disks in the system. Existing spares are labeled GLOBAL SP
2. In the Disk Sets table, the number of white slots in the Disks field shows how many spares you can add.
3. In the enclosure view or list, only existing global spares and suitable available disks are selectable.
4. Select spares to remove, disks to add as spares, or both.
5. Click Modify Spares. If the task succeeds, the panel is updated to show which disks are now global spares.

Once the rebuild is complete, replace the faulty drive.

Get back to me with Logs if you need further assistance.

Tools > Save logs

 -JD

[Torsten.]

I assume this is a RAID 6 with a failed disk, hence it is still fault tolerant.

Check the status of the disks and the event log for details.

[SJO]

Hello Torsten

u are right. This is a Raid 6 Storage. i checked the disks, but all diskstatus are ok. only 1 disk is written in the log (have a look at the pic) but this disk is in the other Storage in the same Encolsure. 

Thank you Torsten

@ jd: Thank you for your help, i've got no global sp. 

the second pic is from the log file.

@ all: Do you have futher helpings for me

[Torsten.]

There is a problem with esxstorage, not with esxstorage1.

Can you select the physical view (at bottom) and take a screenshot?

Same for esxstorage vdisk, please.

[SJO]

Hope this is what you want, or something else?

Thanks for Helping

[Torsten.]

It says 13 disks, but 12 are shown.

1 is missing. Which one?

Please compare to the other vdisk.

Please select also the last entry in the left column (physical view).

Connect to the array using telnet/ssh, then do

> show system

> show disks

> show controllers

> show versions

[SJO]

Hi Torsten

you are right. It says 13 Disk, but 12 are shown. I don't know where the rest is. I'm realy speachless. I logged in via console and set up yout commands and some other commands. 

Thank u 

This is the console output:

********************************

# ## # ## # ## # # ####
## # # ## # # ## # # # # #
# # # # # # # # # #### #
# # # # # # # # # # #
### ## # ### ## # ### ## # # #

System Version: TS200R021
MC Version: L200R012
10.10.10.47 login: manage
Password:

HP StorageWorks MSA Storage P2000 G3 SAS
System Name: srv-nas-04
System Location:Ittigen
Version:L200R012

# show system
System Information
------------------
System Name: srv-nas-04
System Contact: support@imageware.ch
System Location: Ittigen
System Information: ESX NAS
Vendor Name: HP StorageWorks
Product ID: P2000 G3 SAS
Product Brand: MSA Storage
SCSI Vendor ID: HP
Enclosure Count: 1
Health: Degraded
Supported Locales: English (English), Spanish (español), French (français),
German (Deutsch), Italian (italiano), Japanese (???), Dutch (Nederlands),
Chinese-Simplified (????), Chinese-Traditional (????), Korean (???)

Success: Command completed successfully.

# show disks
Location Serial Number Vendor Rev How Used Type Size
Rate*(Gb/s) SP Status
---------------------------------------------------------------------------
1.1 6SE47ENE0000B148NR99 HP HPDE VDISK SAS
300.0GB 6.0 Up
1.2 6SE44RC20000B146M3C7 HP HPDE VDISK SAS
300.0GB 6.0 Up
1.3 6SE4ALKN0000B148KGKS HP HPDE VDISK SAS
300.0GB 6.0 Up
1.4 6SE44R4R0000B146PGZG HP HPDE VDISK SAS
300.0GB 6.0 Up
1.5 6SE45RPY0000B148M9NM HP HPDE VDISK SAS
300.0GB 6.0 Up
1.6 6SE473CB0000B148JEJT HP HPDE VDISK SAS
300.0GB 6.0 Up
1.7 PQHD8XNB HP HPD6 VDISK SAS
300.0GB 3.0 Up
1.8 6SE4AMJW0000B148NQ33 HP HPDE VDISK SAS
300.0GB 3.0 Up
1.9 6SE44L980000B146MZ1M HP HPDE VDISK SAS
300.0GB 6.0 Up
1.10 6SE4AQ4X0000B148MGTN HP HPDE VDISK SAS
300.0GB 6.0 Up
1.11 6SE499PD0000B148HYSY HP HPDE VDISK SAS
300.0GB 6.0 Up
1.12 6SE2XRW80000B132MQXL HP HPDF VDISK SAS
300.0GB 6.0 Up
1.13 PWHVD0MD HP HPD9 VDISK SAS
300.0GB 6.0 Up
1.14 PWHVK3JF HP HPD9 VDISK SAS
300.0GB 6.0 Up
1.15 S0K2T7VC0000K502CUUK HP HPD5 VDISK SAS
300.0GB 6.0 Up
1.16 35G0A0LHFTM91512 HP HPD0 VDISK SAS
300.0GB 6.0 Up
1.17 PWHVDLJD HP HPD9 VDISK SAS
300.0GB 3.0 Up
1.18 PWHTJ3HF HP HPD9 VDISK SAS
300.0GB 3.0 Up
1.19 PWHVJE0D HP HPD9 VDISK SAS
300.0GB 6.0 Up
1.20 PWHVD57D HP HPD9 VDISK SAS
300.0GB 6.0 Up
1.21 PWHVDG4D HP HPD9 VDISK SAS
300.0GB 6.0 Up
1.22 PWHTHP6D HP HPD9 VDISK SAS
300.0GB 6.0 Up
1.23 PWHVD3KD HP HPD9 VDISK SAS
300.0GB 3.0 Up
1.24 PWHVDKYD HP HPD9 VDISK SAS
300.0GB 3.0 Up
---------------------------------------------------------------------------
Info: * Rates may vary. This is normal behavior.

Success: Command completed successfully.

# show disks
Location Serial Number Vendor Rev How Used Type Size
Rate*(Gb/s) SP Status
---------------------------------------------------------------------------
1.1 6SE47ENE0000B148NR99 HP HPDE VDISK SAS
300.0GB 6.0 Up
1.2 6SE44RC20000B146M3C7 HP HPDE VDISK SAS
300.0GB 6.0 Up
1.3 6SE4ALKN0000B148KGKS HP HPDE VDISK SAS
300.0GB 6.0 Up
1.4 6SE44R4R0000B146PGZG HP HPDE VDISK SAS
300.0GB 6.0 Up
1.5 6SE45RPY0000B148M9NM HP HPDE VDISK SAS
300.0GB 6.0 Up
1.6 6SE473CB0000B148JEJT HP HPDE VDISK SAS
300.0GB 6.0 Up
1.7 PQHD8XNB HP HPD6 VDISK SAS
300.0GB 3.0 Up
1.8 6SE4AMJW0000B148NQ33 HP HPDE VDISK SAS
300.0GB 3.0 Up
1.9 6SE44L980000B146MZ1M HP HPDE VDISK SAS
300.0GB 6.0 Up
1.10 6SE4AQ4X0000B148MGTN HP HPDE VDISK SAS
300.0GB 6.0 Up
1.11 6SE499PD0000B148HYSY HP HPDE VDISK SAS
300.0GB 6.0 Up
1.12 6SE2XRW80000B132MQXL HP HPDF VDISK SAS
300.0GB 6.0 Up
1.13 PWHVD0MD HP HPD9 VDISK SAS
300.0GB 6.0 Up
1.14 PWHVK3JF HP HPD9 VDISK SAS
300.0GB 6.0 Up
1.15 S0K2T7VC0000K502CUUK HP HPD5 VDISK SAS
300.0GB 6.0 Up
1.16 35G0A0LHFTM91512 HP HPD0 VDISK SAS
300.0GB 6.0 Up
1.17 PWHVDLJD HP HPD9 VDISK SAS
300.0GB 3.0 Up
1.18 PWHTJ3HF HP HPD9 VDISK SAS
300.0GB 3.0 Up
1.19 PWHVJE0D HP HPD9 VDISK SAS
300.0GB 6.0 Up
1.20 PWHVD57D HP HPD9 VDISK SAS
300.0GB 6.0 Up
1.21 PWHVDG4D HP HPD9 VDISK SAS
300.0GB 6.0 Up
1.22 PWHTHP6D HP HPD9 VDISK SAS
300.0GB 6.0 Up
1.23 PWHVD3KD HP HPD9 VDISK SAS
300.0GB 3.0 Up
1.24 PWHVDKYD HP HPD9 VDISK SAS
300.0GB 3.0 Up
---------------------------------------------------------------------------
Info: * Rates may vary. This is normal behavior.

Success: Command completed successfully.

# show controllers
Controllers
-----------
Controller ID: A
Serial Number: CN8122R452
Hardware Version: 53
CPLD Version: 13
MAC Address: 00:C0:FF:12:8C:2B
WWNN: 500c0ff12a8ff000
IP Address: 10.10.10.94
IP Subnet Mask: 255.255.255.0
IP Gateway: 10.10.10.2
Disks: 24
Vdisks: 1
Cache Memory Size (MB): 2048
Host Ports: 4
Disk Channels: 2
Disk Bus Type: SAS
Status: Operational
Failed Over: No
Fail Over Reason: Not applicable
Position: Top
Phy Isolation: Enabled

Controller ID: B
Serial Number: CN8122R526
Hardware Version: 53
CPLD Version: 13
MAC Address: 00:C0:FF:12:AE:22
WWNN: 500c0ff12a8ff000
IP Address: 10.10.10.47
IP Subnet Mask: 255.255.255.0
IP Gateway: 10.10.10.2
Disks: 24
Vdisks: 1
Cache Memory Size (MB): 2048
Host Ports: 4
Disk Channels: 2
Disk Bus Type: SAS
Status: Operational
Failed Over: No
Fail Over Reason: Not applicable
Position: Bottom
Phy Isolation: Enabled

Success: Command completed successfully.

# show versions
Controller A Versions
---------------------
Bundle Version: TS200R021
Build Date: Fri Jun 11 13:54:41 MDT 2010

Controller B Versions
---------------------
Bundle Version: TS200R021
Build Date: Fri Jun 11 13:54:41 MDT 2010

Success: Command completed successfully.

# show lun
Info: Retrieving data...
Volume View [Serial Number (00c0ff128c2b0000ae34734e01000000) Name (esx storage000) ] Mapping:
Ports LUN Access Host-Port-Identifier Nickname
Profile
-----------------------------------------------------------------
A1,A3,B1,B3 1 read-write 500605b003d2be10
srv-esx-04-PCI3 Standard
A1,A3,B1,B3 1 read-write 500605b003d2ba30
srv-esx.03-PCI3 Standard
A1,A3,B1,B3 1 read-write 500605b003d2b370
srv-esx-04-PCI2 Standard
A1,A3,B1,B3 1 read-write 500605b003d2d310
srv-esx-03-PCI2 Standard
A1,A3,B1,B3 1 read-write 500605b005210160
srv-esx-06-PCI3 Standard
A1,A2,A3,A4,B1,B2,B3,B4 1 read-write 500605b005210164
srv-esx-06-PCI2 Standard
A1,A2,A3,A4,B1,B2,B3,B4 1 read-write 500605b007a45fd1
srv-esx-08 Standard
A1,A2,A3,A4,B1,B2,B3,B4 1 read-write 500605b007a46101
srv-esx-07 Standard
A1,A2,A3,A4,B1,B2,B3,B4 1 read-write 500605b008c527e4
srv-esx-06-PCI1 Standard
A1,A3,B1,B3 1 read-write all other hosts
Standard

Volume View [Serial Number (00c0ff12ae2200002443015101000000) Name (esx storage001) ] Mapping:
Ports LUN Access Host-Port-Identifier Nickname
Profile
-----------------------------------------------------------------
A1,A3,B1,B3 2 read-write 500605b003d2be10
srv-esx-04-PCI3 Standard
A1,A3,B1,B3 2 read-write 500605b003d2ba30
srv-esx.03-PCI3 Standard
A1,A3,B1,B3 2 read-write 500605b003d2b370
srv-esx-04-PCI2 Standard
A1,A3,B1,B3 2 read-write 500605b003d2d310
srv-esx-03-PCI2 Standard
A1,A2,A3,A4,B1,B2,B3,B4 2 read-write 500605b005210160
srv-esx-06-PCI3 Standard
A1,A2,A3,A4,B1,B2,B3,B4 2 read-write 500605b005210164
srv-esx-06-PCI2 Standard
A1,A2,A3,A4,B1,B2,B3,B4 2 read-write 500605b007a45fd1
srv-esx-08 Standard
A1,A2,A3,A4,B1,B2,B3,B4 2 read-write 500605b007a46101
srv-esx-07 Standard
A1,A2,A3,A4,B1,B2,B3,B4 2 read-write 500605b008c527e4
srv-esx-06-PCI1 Standard
not-mapped all other hosts
Standard

Success: Command completed successfully.
# show volumes
Vdisk Name Size Serial Number WR Policy
Cache Opt Read Ahead Size Type Class Qualifier
Volume Description
------------------------------------------------------------------------
esxstorage esx storage000 3296.5GB 00c0ff128c2b0000ae34734e01000000
write-back standard Default standard standard
N/A
esxstorage1 esx storage001 2996.8GB 00c0ff12ae2200002443015101000000
write-back standard Default standard standard
N/A
------------------------------------------------------------------------
Success: Command completed successfully.

# show vdisk-statistics
Name Serial Number Bytes per second IOPS
Number of Reads Number of Writes Data Read Data Written

----------------------------------------------------------------------
esxstorage 00c0ff128c2b00009d11734e00000000 0B
0 172400804 691191585 34.1TB
18.9TB
esxstorage1 00c0ff12ae2200002aedff5000000000 0B
0 385195016 1499964787 78.8TB
53.5TB
----------------------------------------------------------------------
Success: Command completed successfully.

# show disk-statistics
Disk Statistics
---------------
Durable ID: disk_1.1
Serial Number: 6SE47ENE0000B148NR99
Bytes per second: 0B
IOPS: 0
Number of Reads: 2108822609
Number of Writes: 200666587
Data Read: 128.5TB
Data Written: 4134.6GB

Durable ID: disk_1.17
Serial Number: PWHVDLJD
Bytes per second: 0B
IOPS: 0
Number of Reads: 1004123846
Number of Writes: 481584469
Data Read: 29.4TB
Data Written: 11.2TB

Durable ID: disk_1.2
Serial Number: 6SE44RC20000B146M3C7
Bytes per second: 0B
IOPS: 0
Number of Reads: 2109226764
Number of Writes: 200701454
Data Read: 128.5TB
Data Written: 4168.9GB

Durable ID: disk_1.18
Serial Number: PWHTJ3HF
Bytes per second: 0B
IOPS: 0
Number of Reads: 1006115458
Number of Writes: 484144152
Data Read: 29.5TB
Data Written: 11.3TB

Durable ID: disk_1.3
Serial Number: 6SE4ALKN0000B148KGKS
Bytes per second: 0B
IOPS: 0
Number of Reads: 2107199560
Number of Writes: 198905430
Data Read: 128.5TB
Data Written: 4150.8GB

Durable ID: disk_1.19
Serial Number: PWHVJE0D
Bytes per second: 0B
IOPS: 0
Number of Reads: 1003434902
Number of Writes: 480689678
Data Read: 29.4TB
Data Written: 11.2TB

Durable ID: disk_1.4
Serial Number: 6SE44R4R0000B146PGZG
Bytes per second: 0B
IOPS: 0
Number of Reads: 2107204864
Number of Writes: 199257896
Data Read: 128.5TB
Data Written: 4158.6GB

Durable ID: disk_1.20
Serial Number: PWHVD57D
Bytes per second: 0B
IOPS: 0
Number of Reads: 1001841532
Number of Writes: 479028310
Data Read: 29.4TB
Data Written: 11.2TB

Durable ID: disk_1.5
Serial Number: 6SE45RPY0000B148M9NM
Bytes per second: 0B
IOPS: 0
Number of Reads: 2109506073
Number of Writes: 201328051
Data Read: 128.5TB
Data Written: 4165.9GB

Durable ID: disk_1.21
Serial Number: PWHVDG4D
Bytes per second: 0B
IOPS: 0
Number of Reads: 1002687296
Number of Writes: 479687281
Data Read: 29.4TB
Data Written: 11.2TB

Durable ID: disk_1.6
Serial Number: 6SE473CB0000B148JEJT
Bytes per second: 0B
IOPS: 0
Number of Reads: 2110910987
Number of Writes: 202783624
Data Read: 128.6TB
Data Written: 4183.1GB

Durable ID: disk_1.22
Serial Number: PWHTHP6D
Bytes per second: 0B
IOPS: 0
Number of Reads: 1004175852
Number of Writes: 481563432
Data Read: 29.4TB
Data Written: 11.2TB

Durable ID: disk_1.7
Serial Number: PQHD8XNB
Bytes per second: 0B
IOPS: 0
Number of Reads: 5498856
Number of Writes: 2943526
Data Read: 329.2GB
Data Written: 323.0GB

Durable ID: disk_1.23
Serial Number: PWHVD3KD
Bytes per second: 0B
IOPS: 0
Number of Reads: 1003701089
Number of Writes: 481197670
Data Read: 29.5TB
Data Written: 11.3TB

Durable ID: disk_1.8
Serial Number: 6SE4AMJW0000B148NQ33
Bytes per second: 0B
IOPS: 0
Number of Reads: 2101353703
Number of Writes: 192784076
Data Read: 128.5TB
Data Written: 4064.4GB

Durable ID: disk_1.24
Serial Number: PWHVDKYD
Bytes per second: 0B
IOPS: 0
Number of Reads: 999245672
Number of Writes: 476254412
Data Read: 29.4TB
Data Written: 11.2TB

Durable ID: disk_1.9
Serial Number: 6SE44L980000B146MZ1M
Bytes per second: 0B
IOPS: 0
Number of Reads: 2104907314
Number of Writes: 196066349
Data Read: 128.5TB
Data Written: 4065.3GB

Durable ID: disk_1.10
Serial Number: 6SE4AQ4X0000B148MGTN
Bytes per second: 0B
IOPS: 0
Number of Reads: 2110104548
Number of Writes: 201819528
Data Read: 128.6TB
Data Written: 4168.9GB

Durable ID: disk_1.11
Serial Number: 6SE499PD0000B148HYSY
Bytes per second: 0B
IOPS: 0
Number of Reads: 2107787897
Number of Writes: 199784761
Data Read: 128.5TB
Data Written: 4154.9GB

Durable ID: disk_1.12
Serial Number: 6SE2XRW80000B132MQXL
Bytes per second: 0B
IOPS: 0
Number of Reads: 2051536051
Number of Writes: 170172213
Data Read: 126.9TB
Data Written: 3863.5GB

Durable ID: disk_1.13
Serial Number: PWHVD0MD
Bytes per second: 0B
IOPS: 0
Number of Reads: 998335612
Number of Writes: 475786613
Data Read: 29.4TB
Data Written: 11.2TB

Durable ID: disk_1.14
Serial Number: PWHVK3JF
Bytes per second: 0B
IOPS: 0
Number of Reads: 996987944
Number of Writes: 478037293
Data Read: 29.2TB
Data Written: 11.2TB

Durable ID: disk_1.15
Serial Number: S0K2T7VC0000K502CUUK
Bytes per second: 0B
IOPS: 0
Number of Reads: 1666588317
Number of Writes: 163999443
Data Read: 101.6TB
Data Written: 3386.0GB

Durable ID: disk_1.16
Serial Number: 35G0A0LHFTM91512
Bytes per second: 0B
IOPS: 0
Number of Reads: 614678890
Number of Writes: 369708000
Data Read: 18.4TB
Data Written: 8917.3GB

Success: Command completed successfully.

#

[Torsten.]

Hmm, the array has 24 disks, the second group has 12 disks, the first has 13 disks with 12 listed - strange.

I noticed the firmware is very, very old - maybe we hit a (known and fixed) bug?

A problem after replacing a disk?

Current version is TS252P001 - you have TS200R021 from year 2010.

Since everything is still fault tolerant, I would consider a firmware upgrade.

http://h20566.www2.hpe.com/hpsc/swd/public/detail?sp4ts.oid=4208988&swItemId=MTX_474cd8362e0e4ffea414bd8d50&swEnvOid=4064#tab-history

* RECOMMENDED * Online ROM Flash Component for Windows - HP P2000 G3 MSA Arrays

Version:TS252P001 (1 Mar 2016)
Version:TS252R007 (3 Jun 2015)
Version:TS251P006-02 (9 Sep 2014)
Version:TS230P008 (B) (15 Aug 2014)
Version:TS250P003 (A) (15 Aug 2014)
Version:TS251P005 (17 Jul 2014)
Version:TS251P004-02 (18 Jun 2014)
Version:TS251P003-02 (11 Jun 2014)
Version:TS251R004 (17 Mar 2014)
Version:TS250P003 (2 Oct 2013)
Version:TS250P002 (12 Jul 2013)
Version:TS250R023 (5 Apr 2013)
Version:TS240P004-02 (2 Nov 2012)
Version:TS240P003 (13 Jul 2012)
Version:TS230P008 (A) (6 Jul 2012)
Version:TS240P001 (4 Jun 2012)
Version:TS230P008 (22 Nov 2011)
Version:TS201P007 (16 Feb 2011)
Version:TS201P004 (13 Jan 2011)
Version:TS201R015 (22 Nov 2010)
Version:TS201R014 (15 Sep 2010)
Version:TS200R021 (18 Jun 2010)

[SJO]

Hi Torsten

I will make a Firmaware Update, and let you what's happend. Thank you very much for your help

Regards SJO