StoreVirtual Storage
Showing results for 
Search instead for 
Did you mean: 

Problem with HP P4000 DSM for MPIO on Windows 2003

Occasional Advisor

Problem with HP P4000 DSM for MPIO on Windows 2003

Good day

We are having a problem with our development SQL box that connects to our HP P4000 SAN. The MS iSCSI initiator, MS DSM, HP P4000, are all installed by the book.
We keep getting iSCSI sessions dropping off and this error appears in the Windows System log:

"A Path Verification request to a device on \Device\MPIODisk5 that is controlled by HP P4000 DSM for MPIO has failed. This may indicate a Path Failure."

Anybody come across this problem before?
And what is a good solution?

Many thanks in advance
Trusted Contributor

Re: Problem with HP P4000 DSM for MPIO on Windows 2003

If you have regular connection issues.. it could very well be network related.

Please check the following Networking best practices

â ¢ Gigabit Ethernet support â Each storage node comes equipped with at least two copper Gigabit Ethernet ports (802.3ab). To take advantage of full-duplex 1GbE capabilities, the cabling infrastructure must be Cat5e or Cat6 cabling to the storage nodes. Server connections and switch interconnects can also be done via fiber cabling instead of Cat5e or Cat6 cabling. For 10Gigabit implementations, Cat-6a or Cat-7 cabling is usually required for use with distances over 55 meters.
â ¢ Fully subscribed non-blocking backplanes â In order to achieve maximum performance on the HP P4000 SAN, it is important to select a switch that has a fully subscribed backplane, which means that the backplane must be capable of supporting all ports in full-duplex mode. For instance, if the switch has 24 1 Gb ports, it will require a 48 Gb backplane to support full-duplex Gigabit communications.
â ¢ Adequate per-port buffer cache â For optimal switch performance, HP recommends that the switch have at least 512 KB of buffer cache per port. Consult your switch manufacturer specifications for the total buffer cache. For example, if the switch has 48 1 Gb ports, the recommendation is to have at least 24 MB of buffer cache dedicated to those ports. If the switch aggregates cache among a group of ports (for example, 1 MB of cache per 8 ports), space your storage modules and servers appropriately to avoid cache oversubscription.
â ¢ Enable Flow Control on network switches and adapters. Flow Control ensures a receiver can make the sender pace its speed and is important in avoiding data loss.
Flow control support â IP storage networks are unique in the amount of sustained bandwidth that is required to maintain adequate performance levels under heavy workloads. Gigabit Ethernet flow control (802.3x) technology should be enabled on the switch to eliminate receive and/or transmit buffer cache pressure. The storage nodes should also be set to have flow control enabled. Note: Some switch manufacturers do not recommend configuring flow control when using jumbo frames, or vice versa. Consult the switch manufacturerâ s documentation for guidance on this issue. For optimal performance, HP recommends implementing flow control over jumbo frames. Flow control is required when using DSM/MPIO.
â ¢ Ensure spanning tree algorithm for detecting loops is turned off; loop detection introduces a delay in making a port become usable for data transfer and may lead to application timeouts
If supported by the switch infrastructure, HP recommends implementing Rapid Spanning Tree for faster Spanning Tree convergence.
If configurable on the switch, consider disabling spanning tree on the storage node and server switch ports so that they do not participate in the Spanning Tree convergence protocol timing.
Enable Rapid Spanning Tree protocol convergence or PortFast (Cisco) on the storage node and server switch ports.
â ¢ VLAN support â HP recommends implementing a separate subnet or VLAN for the IP storage network. If you are implementing VLAN technology within the switch infrastructure, you will typically need to enable VLAN tagging (802.1q) and/or VLAN trunking (802.1q) or Cisco Inter-Switch Link (ISL)). Consult your switch manufacturerâ s configuration guidelines when enabling VLAN support.
â ¢ Basic IP routing â The storage nodes can access external services such as DNS, SMTP, SNMP, Syslog, and so on. In order to allow this traffic outside the IP storage network, an IP route is required from the IP storage network to the LAN environment. Also, if the storage nodes are to be managed from a remote network, an IP route must exist to the storage nodes from the management station. Finally, if remote copy functionality is going to be used, the remote copy traffic must be routable between the primary/remote sites.
â ¢ Disable unicast storm control on iSCSI ports. Most switches have unicast storm control disabled by default. If your switch has this enabled, you should disable this on the ports connected to iSCSI hosts and targets to avoid packet loss.
â ¢ Segregate SAN and LAN traffic. iSCSI SAN interfaces should be separated from other corporate network traffic (LAN). Servers should use dedicated NICs for SAN traffic. Deploying iSCSI disks on a separate network helps to minimize network congestion and latency. Additionally, iSCSI volumes are more secure whenâ ¦ Segregate SAN & LAN traffic can be separated using port based VLANs or physically separate networks.
â ¢ Configure additional Paths for High Availability; use either Microsoft MPIO or MCS (multiple connections per session) with additional NICs in the server to create additional connections to the iSCSI storage array through redundant Ethernet switch fabrics.
â ¢ Unbind File and Print Sharing from the iSCSI NIC â on the NICs which connect only to the iSCSI SAN, unbind File and Print Sharing.
â ¢ Use Gigabit Ethernet connections for high speed access to storage. Congested or lower speed networks can cause latency issues that disrupt access to iSCSI storage and applications running on iSCSI devices. In many cases, a properly designed IP-SAN can deliver better performance than internal disk drives. iSCSI is suitable for WAN and lower speed implementations including replication where latency and bandwidth are not a concern.
â ¢ Use Server class NICs. It is recommended to use NICs which are designed for enterprise networking and storage applications.
â ¢ Use Jumbo Frames if supported in your network infrastructure. Jumbo Frames can be used to allow more data to be transferred with each Ethernet transaction and reduce the number of frames. This larger frame size reduces the overhead on both your servers and iSCSI targets. For end to end support, each device in the network needs to support Jumbo frames including the NIC and Ethernet switches.
â ¢ High network latency can be the primary cause of slow I/O performance, or worse, iSCSI drive disconnects. It is important to keep network latency on your HP P4500 Multi-Site SAN subnet below two milliseconds. Many factors can contribute to increasing network latency, but two are most common:
o Distance between storage cluster modules
o Router hops between storage cluster modules

Hoping the above helps.
â ¢ Configuring the HP P4500 Multi-Site SAN on a single IP subnet with Layer 2 switching will help to lower the network latency between storage cluster modules.
How to assign points? Click the KUDOS! star!
Occasional Advisor

Re: Problem with HP P4000 DSM for MPIO on Windows 2003

Thank you very much for the conclusive reply Jitun.
Software Manager
Occasional Visitor

Re: Problem with HP P4000 DSM for MPIO on Windows 2003

I recently had a problem like this as well and I found that I had some persistent targets in the iSCSI initiator that were no longer valid. I had done some volume changes and had both old and new persistent targets in the initiator. I identified the obsolete targets, removed them and haven't had issues since. I also refreshed the bindings after deleting the obsolete targets.

My assumption is that the iSCSI initiator attempts to reconnect the targets for a certain amount of time and then decides it needs to reset all connections in an attempt to recover those that are no longer valid. Sort of like a SCSI bus reset I guess.
Occasional Advisor

Re: Problem with HP P4000 DSM for MPIO on Windows 2003

We have this issue, it got better by applying the iscsi Timeout Reg patch below which is recommended by HP;

Setting iSCSI Timeout
â ¢ Default Initiator timeout (60 seconds) is too short; can result in potential connectivity loss for pending I/O operations
â ¢ Edit Registry to change the MaxRequestHoldTime value:
â Browse to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet
â Type Ctrl-F to search for MaxRequestHoldTime
â Be sure to find the one under CurrentControlSet
â Change MaxRequestHoldTime value to 600 decimal
â Reboot system to ensure new value takes effect
â ¢ Microsoft DSM sets a timeout of 30 seconds; do not install, or uninstall if it is already installed
â ¢ After installing/upgrading iSCSI Initiator, always confirm that the MaxRequestHoldTime value is still 600

We still get SQL dropping out occasionally or even MS clusters failing over with items like yours in the event viewer however it is less frequent.