Solved: Re: Adding a new disk - port 2I:box 1:bay 0

andy1221 · ‎09-16-2022

Hello.
I have Dl360 Gen 9 server with Smart Array P440ar Controller.
I use two Raid1 - 4 ssd-drives.
I need to add 2 new ssd-drives.
I inserted 2 disks into the server (5-bay and 6-bay) and I tried to view the status of the disks.

ssacli ctrl slot=0 pd all show
Smart Array P440ar in Slot 0 (Embedded)
Array A

physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SATA SSD, 960 GB, OK)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SATA SSD, 960 GB, OK)

Array B

physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SATA SSD, 960 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SATA SSD, 960 GB, OK)

Unassigned
physicaldrive 2I:1:0 (port 2I:box 1:bay 0, SATA SSD, 960 GB, OK)
physicaldrive 2I:1:0 (port 2I:box 1:bay 0, SATA SSD, 960 GB, OK)

I don't understand - what is "bay 0"?
Help me please.
Thanks.

pchops · ‎09-19-2022

From what I recall (which is highly suspect), "bay 0" is not a valid bay number, except for maybe some older systems with built in drive cage SAS expanders. Since the physical bays in question (5 and 6) are on a different path (P440ar port, SAS cable, and drive bay port) from bays 1-4, my guess is that it could be a problem with any of those items in said path, or perhaps just one (or more) of the new drives. Can you provide the output from?:

ssacli controller slot=0 show config detail

I work for HPE

andy1221 · ‎09-20-2022

Hello pchops.
Thank you for reply.

ssacli controller slot=0 show config detail

Smart Array P440ar in Slot 0 (Embedded)
Bus Interface: PCI
Slot: 0
Serial Number: ***removed***
Cache Serial Number: ***removed***
RAID 6 Status: Enabled
Controller Status: OK
Hardware Revision: B
Firmware Version: 7.00
Firmware Supports Online Firmware Activation: False
Rebuild Priority: High
Expand Priority: Medium
Surface Scan Delay: 3 secs
Surface Scan Mode: Idle
Parallel Surface Scan Supported: Yes
Current Parallel Surface Scan Count: 1
Max Parallel Surface Scan Count: 16
Queue Depth: Automatic
Monitor and Performance Delay: 60 min
Elevator Sort: Enabled
Degraded Performance Optimization: Disabled
Inconsistency Repair Policy: Disabled
Wait for Cache Room: Disabled
Surface Analysis Inconsistency Notification: Disabled
Post Prompt Timeout: 15 secs
Cache Board Present: True
Cache Status: Not Configured
Drive Write Cache: Disabled
Total Cache Size: 2.0
Total Cache Memory Available: 1.8
Battery Backed Cache Size: 1.8
No-Battery Write Cache: Disabled
SSD Caching RAID5 WriteBack Enabled: True
SSD Caching Version: 2
Cache Backup Power Source: Batteries
Battery/Capacitor Count: 1
Battery/Capacitor Status: OK
SATA NCQ Supported: True
Spare Activation Mode: Activate on physical drive failure (default)
Controller Temperature (C): 42
Cache Module Temperature (C): 34
Number of Ports: 2 Internal only
Encryption: Not Set
Express Local Encryption: False
Driver Name: hpsa
Driver Version: 3.4.20
Driver Supports SSD Smart Path: True
PCI Address (Domain:Bus:Device.Function): 0000:03:00.0
Negotiated PCIe Data Rate: PCIe 3.0 x8 (7880 MB/s)
Controller Mode: RAID
Pending Controller Mode: RAID
Port Max Phy Rate Limiting Supported: False
Latency Scheduler Setting: Disabled
Current Power Mode: MaxPerformance
Survival Mode: Enabled
Host Serial Number: ***removed***
Sanitize Erase Supported: True
Primary Boot Volume: None
Secondary Boot Volume: None

Internal Drive Cage at Port 1I, Box 1, OK

Drive Bays: 4
Port: 1I
Box: 1
Location: Internal

Physical Drives
physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SATA SSD, 960 GB, OK)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SATA SSD, 960 GB, OK)
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SATA SSD, 960 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SATA SSD, 960 GB, OK)
physicaldrive 2I:1:0 (port 2I:box 1:bay 0, SATA SSD, 960 GB, OK)
physicaldrive 2I:1:0 (port 2I:box 1:bay 0, SATA SSD, 960 GB, OK)

Port Name: 1I
Port ID: 0
Port Connection Number: 0
SAS Address: 50014380409D75E0
Port Location: Internal
Port Phy Count: 4

Port Name: 2I
Port ID: 1
Port Connection Number: 1
SAS Address: 50014380409D75E4
Port Location: Internal
Port Phy Count: 4

Array: A
Interface Type: Solid State SATA
Unused Space: 0 MB (0.00%)
Used Space: 1.75 TB (100.00%)
Status: OK
MultiDomain Status: OK
Array Type: Data
Smart Path: enable

Logical Drive: 1
Size: 894.22 GB
Fault Tolerance: 1
Heads: 255
Sectors Per Track: 32
Cylinders: 65535
Strip Size: 256 KB
Full Stripe Size: 256 KB
Status: OK
Unrecoverable Media Errors: None
MultiDomain Status: OK
Caching: Disabled
Unique Identifier: 600508B1001C534B0B74EE1392ACCDD6
Disk Name: /dev/sda
Mount Points: None
Logical Drive Label: 0248BAD9PDNLH0BRH5733BC1D4
Mirror Group 1:
physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SATA SSD, 960 GB, OK)
Mirror Group 2:
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SATA SSD, 960 GB, OK)
Drive Type: Data
LD Acceleration Method: Smart Path

physicaldrive 1I:1:1
Port: 1I
Box: 1
Bay: 1
Status: OK
Drive Type: Data Drive
Interface Type: Solid State SATA
Size: 960 GB
Drive exposed to OS: False
Logical/Physical Block Size: 512/512
Firmware Revision: SCEKJ2.7
Serial Number: ***removed***
WWID: 30014380409D75E0
Model: ATA KINGSTON SEDC500
SATA NCQ Capable: True
SATA NCQ Enabled: True
Current Temperature (C): 26
Maximum Temperature (C): 39
SSD Smart Trip Wearout: Not Supported
PHY Count: 1
PHY Transfer Rate: 6.0Gbps
PHY Physical Link Rate: Unknown
PHY Maximum Link Rate: Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6
Sanitize Erase Supported: False
Shingled Magnetic Recording Support: None

physicaldrive 1I:1:2
Port: 1I
Box: 1
Bay: 2
Status: OK
Drive Type: Data Drive
Interface Type: Solid State SATA
Size: 960 GB
Drive exposed to OS: False
Logical/Physical Block Size: 512/512
Firmware Revision: SCEKJ2.7
Serial Number: ***removed***
WWID: 30014380409D75E1
Model: ATA KINGSTON SEDC500
SATA NCQ Capable: True
SATA NCQ Enabled: True
Current Temperature (C): 27
Maximum Temperature (C): 40
SSD Smart Trip Wearout: Not Supported
PHY Count: 1
PHY Transfer Rate: 6.0Gbps
PHY Physical Link Rate: Unknown
PHY Maximum Link Rate: Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6
Sanitize Erase Supported: False
Shingled Magnetic Recording Support: None

Array: B
Interface Type: Solid State SATA
Unused Space: 0 MB (0.00%)
Used Space: 1.75 TB (100.00%)
Status: OK
MultiDomain Status: OK
Array Type: Data
Smart Path: enable

Logical Drive: 2
Size: 894.22 GB
Fault Tolerance: 1
Heads: 255
Sectors Per Track: 32
Cylinders: 65535
Strip Size: 256 KB
Full Stripe Size: 256 KB
Status: OK
Unrecoverable Media Errors: None
MultiDomain Status: OK
Caching: Disabled
Unique Identifier: 600508B1001C97E978DAB1B974A7A966
Disk Name: /dev/sdb
Mount Points: None
Logical Drive Label: 0648BB0CPDNLH0BRH5733B6177
Mirror Group 1:
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SATA SSD, 960 GB, OK)
Mirror Group 2:
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SATA SSD, 960 GB, OK)
Drive Type: Data
LD Acceleration Method: Smart Path

physicaldrive 1I:1:3
Port: 1I
Box: 1
Bay: 3
Status: OK
Drive Type: Data Drive
Interface Type: Solid State SATA
Size: 960 GB
Drive exposed to OS: False
Logical/Physical Block Size: 512/512
Firmware Revision: SCEKJ2.7
Serial Number: ***removed***
WWID: 30014380409D75E2
Model: ATA KINGSTON SEDC500
SATA NCQ Capable: True
SATA NCQ Enabled: True
Current Temperature (C): 25
Maximum Temperature (C): 39
SSD Smart Trip Wearout: Not Supported
PHY Count: 1
PHY Transfer Rate: 6.0Gbps
PHY Physical Link Rate: Unknown
PHY Maximum Link Rate: Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6
Sanitize Erase Supported: False
Shingled Magnetic Recording Support: None

physicaldrive 1I:1:4
Port: 1I
Box: 1
Bay: 4
Status: OK
Drive Type: Data Drive
Interface Type: Solid State SATA
Size: 960 GB
Drive exposed to OS: False
Logical/Physical Block Size: 512/512
Firmware Revision: SCEKJ2.7
Serial Number: ***removed***
WWID: 30014380409D75E3
Model: ATA KINGSTON SEDC500
SATA NCQ Capable: True
SATA NCQ Enabled: True
Current Temperature (C): 27
Maximum Temperature (C): 38
SSD Smart Trip Wearout: Not Supported
PHY Count: 1
PHY Transfer Rate: 6.0Gbps
PHY Physical Link Rate: Unknown
PHY Maximum Link Rate: Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6
Sanitize Erase Supported: False
Shingled Magnetic Recording Support: None

Unassigned

physicaldrive 2I:1:0
Port: 2I
Box: 1
Bay: 0
Status: OK
Drive Type: Unassigned Drive
Interface Type: Solid State SATA
Size: 960 GB
Drive exposed to OS: False
Logical/Physical Block Size: 512/512
Firmware Revision: SCEKJ2.8
Serial Number: ***removed***
WWID: 30014380409D75E8
Model: ATA KINGSTON SEDC500
SATA NCQ Capable: True
SATA NCQ Enabled: True
Current Temperature (C): 24
Maximum Temperature (C): 32
SSD Smart Trip Wearout: Not Supported
PHY Count: 1
PHY Transfer Rate: 6.0Gbps
PHY Physical Link Rate: Unknown
PHY Maximum Link Rate: Unknown
Drive Authentication Status: Not Applicable
Sanitize Erase Supported: False
Shingled Magnetic Recording Support: None

Moderator Edit: Removed the Serial numbers for Privacy.

pchops · ‎09-20-2022

Thanks for providing that detail. My guess is that there is some kind of addressing conflict, since both of the new drives are reporting the same serial numbers (or they were before the moderators removed the serial numbers, which is a good thing) and WWIDs:

Serial Number: ***removed***
WWID: 30014380409D75E8

If possible, try inserting one drive at a time and see if SSACLI reports the correct bay number and what serial number/WWID shows up.

I work for HPE

andy1221 · ‎09-20-2022

Hello.
It is a view from ILO-interface:

-Physical Drive in Port 2I Box 1 Bay 0
Status OK
Serial Number **removed**
Model KINGSTON
Media Type SSD
Capacity 960 GB
Location Port 2I Box 1 Bay 0
Firmware Version SCEKJ2.8
Drive Configuration Unconfigured
Encryption Status Not Encrypted

-Physical Drive in Port 2I Box 1 Bay 0

Status OK
Serial Number **removed**
Model KINGSTON
Media Type SSD
Capacity 960 GB
Location Port 2I Box 1 Bay 0
Firmware Version SCEKJ2.8
Drive Configuration Unconfigured
Encryption Status Not Encrypted

I have another HP DL360 G9 server.
Let's call it server2.
It is identical to server1 (server with problems).
When I insert two 960G disks into server 2, it shows on server 2:

sacli ctrl slot=0 pd all show
Smart Array P440ar in Slot 0 (Embedded)
Array A
physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SATA SSD, 960 GB, OK)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SATA SSD, 960 GB, OK)
Array B
physicaldrive 2I:1:5 (port 2I:box 1:bay 5, SATA SSD, 960 GB, OK)
Unassigned
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SATA SSD, 960 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SATA SSD, 960 GB, OK)

I have 2 more disks (120G).
When I insert them into server1, I see the following:
ssacli ctrl slot=0 pd all show
Smart Array P440ar in Slot 0 (Embedded)
Array A
physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SATA SSD, 960 GB, OK)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SATA SSD, 960 GB, OK)
Array B
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SATA SSD, 960 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SATA SSD, 960 GB, OK)
Unassigned
physicaldrive 2I:1:0 (port 2I:box 1:bay 0, SATA HDD, 120 GB, OK)
physicaldrive 2I:1:0 (port 2I:box 1:bay 0, SATA HDD, 120 GB, OK)

ssacli controller slot=0 show config detail

Unassigned

physicaldrive 2I:1:0
Port: 2I
Box: 1
Bay: 0
Status: OK
Drive Type: Unassigned Drive
Interface Type: SATA
Size: 120 GB
Drive exposed to OS: False
Logical/Physical Block Size: 512/512
Firmware Revision: 3.ALC
Serial Number: **removed**
WWID: 30014380409D75E8
Model: ATA ST9120822AS
SATA NCQ Capable: True
SATA NCQ Enabled: True
Current Temperature (C): 22
PHY Count: 1
PHY Transfer Rate: 1.5Gbps
PHY Physical Link Rate: Unknown
PHY Maximum Link Rate: Unknown
Drive Authentication Status: Not Applicable
Sanitize Erase Supported: False
Shingled Magnetic Recording Support: None

Moderator Edit: Removed the Serial numbers for Privacy.

pchops · ‎09-20-2022

Since even the 120GB disks behave the same way in server1, it would seem to be something in the "P440ar port 2 <-> SAS cable <-> bay 1, port 2" path (call me Captain Obvious :). If you can, check the SAS cable seating at both ends (and their mating connectors at the P440ar and drive bay ends), and the drive bay backplane PCB (both sides) for any damage. It could also be the P440ar itself -- you could try swapping the cable connections to it to see if the problem stays on Port 2 or not. Of course, be sure to have your data backed up before attempting any of the above.

I work for HPE

andy1221 · ‎09-20-2022

I got it.
I will try to check it.
I will be able to do this later, since server 1 is in production and it will take a lot of action to stop it.
When I do, I will definitely let you know.

Thank you.

andy1221 · ‎09-22-2022

Hello.
We decided to look inside the servers.
On server 2 (with problem), it seemed to us that the cable was badly inserted into port 2 of raid controller.
When our worker fixed it, the server crashed (at 12:21PM).
Even IPMI-remote console said "no video".
We have restarted the server(power reset).

After it is loaded, the list of disks is as follows:
ssacli ctrl slot=0 pd all show
Smart Array P440ar in Slot 0 (Embedded)
Array A
physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SATA SSD, 960 GB, OK)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SATA SSD, 960 GB, OK)
Array B
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SATA SSD, 960 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SATA SSD, 960 GB, OK)
Unassigned
physicaldrive 2I:1:5 (port 2I:box 1:bay 5, SATA HDD, 120 GB, OK)
physicaldrive 2I:1:6 (port 2I:box 1:bay 6, SATA HDD, 120 GB, OK)

Bingo!

in ILO-logs we see the following message:

12:21PM - Severity - Critical - PCI Bus Error (Slot 0, Bus 0, Device 2, Function 2).

I hope it's P440ar.

andy1221 · ‎10-04-2022

We bought 2 new SSD drives and installed them in the server.
Result:
ssacli ctrl slot=0 pd all show
Smart Array P440ar in Slot 0 (Embedded)
Array A
physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SATA SSD, 960 GB, OK)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SATA SSD, 960 GB, OK)
Array B
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SATA SSD, 960 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SATA SSD, 960 GB, OK)
Unassigned
physicaldrive 2I:1:5 (port 2I:box 1:bay 5, SATA SSD, 960 GB, OK)
physicaldrive 2I:1:6 (port 2I:box 1:bay 6, SATA SSD, 960 GB, OK)
The problem was a poorly connected cable.
Pchops, thanks again for your help!