P4000 VSA in Production?

[5y53ng]

I just wanted to add to Ted's post about settings the number of IOPS per path. This setting is not persistant in ESX(i). You can add a script to rc.local to reapply the IOPS setting after rebooting your hosts. The process to do so is detailed in the comments section of this article...

Look for a post in the comments by "Matt"

http://virtualgeek.typepad.com/virtual_geek/2009/09/a-multivendor-post-on-using-iscsi-with-vmware-vsphere.html

I would also like to add that I have 40 VSAs in seven clusters at four sites hosting approximately 300TB of storage and I have never had any issue. The most common nuisance I encounter is when an admin unknowingly powers-off adjacent nodes in a cluster.

[Paul Hutchings]

Thanks, that's very useful to know - are yours a mix of usage or all low IO?

In principle I'd have no objection to using them for everything, but again I'm assuming there's some artificial bottleneck beyond which it doens't matter what hardware you throw at a VSA (else why pay $150k for an SSD P4x00 when you could DIY with consumer SSDs and use RAID/Network RAID to provide redundancy?).

[5y53ng]

The environment is a mix in terms of disk usage, but there are some greedy SQL servers in our systems. My reason for going with the VSA was due to size contraints. There simply was not enough room for multiple 2U (or bigger) storage systems. The VSA was a savior with that in mind. This allowed me to allocate all of my space to max out CPUs, memory, and connectivity.

If it makes you feel any better I have managed to push 20k+ IOPS with a six node cluster on 36 spindles. Don't ask about que depths, but it can do it...

[Paul Hutchings]

Blimey, that's pretty encouraging!  We don't push anything like that tbh.  We "only" have a 2 node P4500 for our VM's which has been sufficient.

I have this dream where I go out and buy a couple of Dell T620's or something similar, stuffed with 2.5" spindles and just stick a bunch of VSA's on them, just not sure if I'd be setting myself up for a fall, though I don't see how as there'd be so many layer of redundancy at hardware and node level.

[5y53ng]

The VSA works pretty good IMO, definitely not the fastest setup, but it gets the job done. Ours have been in production for three years now. They are very reliable, just make sure everyone understand who works with them understands managers and how replication works and you're good to go.

[ccavanna]

I have a pair of Dell R710's running a vsa cluster. They have LFF 600GB 15k SAS drives and have pushed 10k+ iops while doing storage vmotion's. We have 9 cluster's over 9 sites, this year we are implementing 15 more locations with vsa clusters. The only thing I need to figure out to move everything to esxi is how to get the storage to automatically rescan after a reboot or power outage so everything boots on its own and I don't have to manually intervene. 

[Paul Hutchings]

Thanks both.  Can I ask you whether your VMs run on the same physical hosts that the VSA's run on?

I can see a pair of servers with lots of spindles and a lot of spare CPU and RAM, but it sounds simpler/neater splitting out the roles so that the boxes that do storage only do storage (with the VSA's set to automatic start), and the boxes that run guests only running guests.

It seems too messy otherwise trying to get the "all-in-one" solution to boot up cleanly etc.

[5y53ng]

I agree getting the hosts to see the storage is difficult after a power outage. Since all of the storage nodes are powered off, none of the volumes are available.

If only HP and and VMware would team up to come up with a way to perform a delayed rescan... :)

Scripting the rescan wouldn't be difficult, but the timing is. I wonder if pinging the VIP would be a valid test for a script to continue and start a rescan, maybe then power up all the virtual machines? If the VIP were available I suppose you could retrieve a list of targests, but whether or not the targets are available is another story...

[ccavanna]

Well... We are running ESX 4.1 U2 on all of our VSA's and yes we are running vm's off the same hosts. We run a sql server and a domain controller split over the 2 hosts. One thing to watch out for is network loops. We have had that happen in the facilities from someone in the plant causing it and had to reboot the hosts and everything. That is the only real issue i've had over the past year with them and its only happened twice. I found a script on the old forums and have been using that with some tweaks of my own. It works great for power outage type senario's and truthfully we have had several power outages in our manufacturing facilities and no one has had to interviene yet everything has came back up cleanly and ran like a top. 

The reason why we haven't moved to ESXi is because HP and vMware haven't made it easy to do it. But i've heard its possible with the vMA and some pearl scripting. I just have not had any time to investigate that since I am currently working on replacing our current infrastructure Blades chassis, blades, and about 90TB worth of p4500 sas and MDL sas.

[Paul Hutchings]

Thanks both.  All very encouraging.  It also leans me towards dedicated VSA boxes, which isn't that big of a deal given how cheap and full you can stuff a 2U full of disks.