StoreVirtual Storage
1748236 Members
3744 Online
108759 Solutions
New Discussion юеВ

Re: Excessive packet loss after VSA upgrade from 12.0 to 12.6

 
Stor_Mort
HPE Pro

Re: Excessive packet loss after VSA upgrade from 12.0 to 12.6

Hi Wraith, 

The continuous ping from CMC may not show the issue for several reasons. The 12.6 retransmission monitor runs on the storage nodes and results are reported on the CMC. Pings from the CMC may not be traversing the path which is in trouble. It's also possible that the retransmissions are intense but brief if they happen at moments of peak load.

Check the NIC firmware and drivers on the hosts and SAN nodes and update if necessary. If the problem only affects one host, try swapping the cable and SFP with known good ones. We find a lot of 10 Gb packet loss issues are because of cables. Sometimes we find older or low end switches that do not have sufficient switching and buffering resources to handle the peak load.

I am an HPE employee - HPE SImpliVity Support

Accept or Kudo

dcampregher
Honored Contributor

Re: Excessive packet loss after VSA upgrade from 12.0 to 12.6

Hi,

 

I have an environment with the same issues. There are some clusters with P4500G2 running the 12.5 version and 01 cluster running the SV4730 10GbE 12.6 version.

Only the new cluster 12.6 show in a first node  the alarm with the Excessive pack loss, even with no load on it.

Any update ?

-----------
Please assign Kudos
Stor_Mort
HPE Pro

Re: Excessive packet loss after VSA upgrade from 12.0 to 12.6

You will not get these network alerts on storage nodes that run LHOS 12.5 or earlier. The network monitoring function was added in 12.6. (Note that P4x00 G2 systems cannot be upgraded past 12.5 and so will never report these packet loss alerts.)

If it is only one of your 4730 systems that reports packet loss alerts, you may have a bad cable, a bad SFP+ or bad NIC card at either end of the cable. It is also possible to see these alerts if you have congestion in the network. Be sure there are no 1Gb paths that try to handle the 10Gb traffic.

Here is a fault isolation tip. If you have two redundant paths, disable one to see if the remaining path has the problem. Then restore it and disable the other path to confirm the result. If only one path is affected, substitute known good parts to identify the failing component. If both paths are affected, the problem is probably somewhere else in the network that is common to both paths.

I am an HPE employee - HPE SImpliVity Support

Accept or Kudo

cenkozkan
New Member

Re: Excessive packet loss after VSA upgrade from 12.0 to 12.6

Hi,

Same situation here. Just upgraded to 12.6. I have no noticable performance problems and am currently getting these notifications. They have started when the backup started. The Packet Loss is around 1%. Is it possible to disable the notifications ? The system is working very well and is very fast. I see no reason to mess with equipment.

Thanks

Stor_Mort
HPE Pro

Re: Excessive packet loss after VSA upgrade from 12.0 to 12.6

We recommend that your network have less than 0.1% packet loss to carry iSCSI traffic efficiently. Fixing the network is the best way to shut off the alarms.

There is no easy way to change the alarm threshold. You can open a support case with HPE to have one of our engineers log in to each node remotely and change the threshold permanently.

I am an HPE employee - HPE SImpliVity Support

Accept or Kudo

ITGuru
Visitor

Re: Excessive packet loss after VSA upgrade from 12.0 to 12.6


@cenkozkan wrote:

Hi,

Same situation here. Just upgraded to 12.6. I have no noticable performance problems and am currently getting these notifications. They have started when the backup started. The Packet Loss is around 1%. Is it possible to disable the notifications ? The system is working very well and is very fast. I see no reason to mess with equipment.

Thanks


Did you ever find a solution?  We just did a server refresh and have CMC 12.6 and same thing, after my backups start, the packet loss error messages start with 0.50% - 1% packet loss and even after the backups are finished, there is no way to clear this message but to reboot the VSA that is generating the error messages.

dcampregher
Honored Contributor

Re: Excessive packet loss after VSA upgrade from 12.0 to 12.6

Hello,

In my casy the Alert was generated by Broadcast of vMotion Network of vSphere inside the same L2 Switch. After create a segmented L2 network ( VLAN ) in the switches and the vSphere the Alert was automatically cleared.

So, i think thatyou have another trafic inside the same L2 Network of the iSCSI.

 

-----------
Please assign Kudos
ITGuru
Visitor

Re: Excessive packet loss after VSA upgrade from 12.0 to 12.6


@dcampregher wrote:

Hello,

In my casy the Alert was generated by Broadcast of vMotion Network of vSphere inside the same L2 Switch. After create a segmented L2 network ( VLAN ) in the switches and the vSphere the Alert was automatically cleared.

So, i think thatyou have another trafic inside the same L2 Network of the iSCSI.

 


Thank you so much this actually helps as we currently have all networks (iSCSI, vMotion and LAN) on the same L2 broadcast domain.

Stor_Mort
HPE Pro

Re: Excessive packet loss after VSA upgrade from 12.0 to 12.6

That's a great case study. Our experience is that packet loss over a few tenths of a percent causes negative performance impact. That's why the check for packet loss was added to LeftHand OS 12.6.

Please take these alerts seriously. An iSCSI storage system only works as well as the network it's using.

If you got a new car and the Engine Check light came on, would you tape over it and keep driving, or would you find out the cause and get it fixed before it fails? Similarly, it's best to investigate and remediate alerts in your IT systems before the issue becomes critical.

I am an HPE employee - HPE SImpliVity Support

Accept or Kudo