Number of snapshots grows without bound

[Mark Fenton]

Interesting issue or two.

To start:  P4500 4-nodes, SANiQ 9.0 with appropriate DSM, MPIO etc. Host machine is BL460 running Windows Server 2008 R2 Enterprise. In addition to the underlying server functions, it's also a Hyper-V host with four or five VMs. The problem I'm having is with the volume where the VMs live.

1) Using VSS on the host with local retention policy (keep seven snaps) -- only the last seven are visible to the host. So far so good.

2) The number of snaps on the SAN grows without bound, however, and to keep space reasonable, I have to manually go in and delete the "excess" snapshots.

3) Some of the snapshots appear to get stuck with hanging I/O to the server. These look like off line/foreign disks to the server, and the only fix for them is to go into the iSCSI initiator and disconnect them manually.

I could supply pretty pictures if needed...

[oikjn]

I think I've had similar issues with app aware scheduled snapshots created for backups not getting deleted after they are getting used to create the backup.  seems to be something getting stuck on one host or another.  I've noticed that whenever it does happen it always happens on one node which I can fix by simply restarting that server that is having a problem.... so...  have you tried simply restarting the problem server?

[Mark Fenton]

Sorry not to have replied earlier -- vacation schedule and the like.

The server has been rebooted, and this does ameliorate the issue for a while, but within a few days, the behavior returns.

Another issue I've been seeing is that some of these snapshots appear to get "stuck" -- IO still active to them, for days on end after they should have disconnected. On the windows server, these appear in disk management as missing disks. If I delete them here, then the IO, of course, goes away, but still, more will eventually crop up. Frustrating.

Maybe I should just bite the bullet and upgrade everything to 9.5.

[Mark Fenton]

Just had another thought. Perhaps I need to have my Windows admin change the way he snaps his volumes. The underlying host is where the snapshots are being created, but the offending volume holds VM images and their "local" storage. Perhaps there is an issue with Hyper-V not releasing IO while the snapshot is being created.

[Bryan McMullan]

Are you using off-host backups? If so, there's another process that will create snapshots and will not allow them to delete until the backup software is done with them. Perhaps your admin was testing it?