AF7000 - sudden change in snapshotting time

[sjarczyk]

We've been running scheduled snapshots for our MSSQL data and log volumes for years, without any issues. Normally, snapshot took ~5 mins to complete. Now, since few days, it's running for an hour and a half! No changes has been made on either Nimble or servers side. Nothing in 'snap info' output. We're running on 5.2.1.1100-1027043-opt currently. I guess what I'm looking for is some way to get to the logs/activity audit, to see what's happening during those 90 mins exactly. Unfortunately, can't find a way to do that.

[Satish04]

Hi sjarczyk,

If you see that the time it takes to finish scheduled snapshots has significantly increased, you should troubleshoot and find the source of the delay.

Below are some steps to investigate the issue:

1. Make sure that no additional processes, such as backups, maintenance, or excessive I/O from other applications, are running on the SQL Server or the HPE Nimble array that could be contributing to contention.
2. Despite your statement that the info output contains nothing, it's still a good idea to check the output for any unexpected problems or cautions.
Use info' both before and after a snapshot to check for any changes or specific events that were noted.
3. When taking snapshots, keep monitoring storage performance using the HPE Nimble storage management interface or other tools. Keep monitoring for any performance abnormalities, such as higher IOPS or latency spikes.
4. System logs and audit trails are often kept up to date by HPE Nimble storage arrays. Verify whether a system log exists that documents events associated with snapshot processes. This could provide hints on any mistakes or odd behavior.
5. Check the SQL Server logs for any warnings or faults about the snapshot procedure. This can involve mistakes in the way SQL Server communicates with the storage.
6. During the snapshot process, record and examine system and storage metrics using monitoring tools, such as performance monitoring software or third-party solutions. This can provide important context for what's going on.
7. Verify that performance on the HPE Nimble array or SQL Server has not been compromised by any automatic firmware or software updates. Updates can occasionally bring about modifications that impact performance.

Make sure your HPE Nimble storage and SQL server has been updated to the latest firmware or software update.

If the issue persists, I t's recommend reaching out to HPE Nimble support to resolve the issue.

Hope this helps.

Regards,
Satish

[sjarczyk]

@Satish04 

Thank you for your reply. There's nothing in audit logs or verbose info output for snapshots. The only thing I've noticed is login session timeout for a snapshot user - but that'd mean, IMO, there's nothing happening for 30 mins (session timeout). And yes, we've opened the case with Nimble support, and we're currently testing Alletra 6xxx as and replacement, but it still bothers me. I'm wondering if it's something to do with SW versions - our Nimble is running 5.2.1.1100-1027043-opt, and we use v7.10 of storage toolkit. On the other hand, it was like this for months, and the problems started recently...

Thank you,

Serg