10gig iSCSI latency on MS CSV volumes

[RVil]

Hi,

I have 2 questions regarding 10gig iSCSI latency.

1. Each time we upgrade our HF20 array, we are bumbed with email alerts from Veeam ONE reporting too high read latency from the volumes.

Alarm: Cluster shared volume 2012 read latency
Time: 18-03-2019 18:48:58
Details: "Disk/CSV2012: Read Latency" (730.0 Milliseconds) is above a defined threshold (80.0 Milliseconds)

This is from all volumes. The array is connected to a c3750e stach running 10gig and our Hyper V servers is also running 10gig using Intel X520 nics.
None of the interfaces is showing any packet drops during the upgrade process (only sometimes during nightly backup when traffic is hiting 6-7Gbit/s we see a few packet drops). I know the c3750e has very small packet buffers, but as packet drops is none during array firmware upgrade, it cannot be a switch buffer issue. We have verified Microsoft performance monitor is also reporting these high read latencies, so it is not a Veeam ONE issue. All the 2012r2 Hyper V guests are using vhdx disks, so only our hosts are connected to the array with iSCSI using the newest Nimble toolkit.

2. Sometimes during working hours, we also get alerts for high read latency 70-80ms, even when the array is doing nearly nothing and our servers are also very light loaded. We have a CSV iSCSI volume containing only software for OS install and sometimes we also get alerts for this volume even though, it is not being accessed at all.

Any good ideas why we get these periodicly high read latencies from CSV volumes when bandwidth is low, no rx or tx errors, only light usage of our servers and nothing to see in Windows event logs?

Regards
Robert

[Thomas_Lam_HK · Accepted solution]

Hey Robert,

Just engage with Nimble Support, they can help perform some diagnostic even out of the array box, e.g. networking or host side......

[BrianTKoon]

What was the solution in the end?

[RVil]

1. When there is a failover, Data service daemon ( service responsible for connections to the hosts) restarts, as the service needs to run from the new active controller. During DSD restart, recovery process re-pins lot of metadata in the cache including heatmap recovery. On a lower core systems such as this HF20, it may take longer to perform the heatmap recovery causing high latency due to high CPU usage.

Other than this we did not come closer to a real solution, but maybe it´s something do to with Hyper-V inter host communication and/or i/o redirection, but we could not prove this.

That´s why we are eager to try to new HPE InfoSight Collector Toolkit when version 5.1.4.200 is released for the HF20 array.

/Robert