Reclaiming Space from a thin provisioned netword raid-5 volume

[oikjn]

yes you are right.  StoreVirtual not StoreServ.  Still getting used to the new names.  Glad to see you are here.  I wouldn't say 34 posts in ~1900 days since registering is perticualy active anywhere, but I do get it isn't your job.  IMO it is the job of the product manager to manage the product line and I'm shocked he/she appears to care nothing for this site.  Its not like this is hosted on a random 3rd party site, this is through HP and this is a cheap easy way to keep in contact with the primary users of the product...  assigning someone on the StoreVirtual team to monitor the forum for an hour a week would likely provide more value to the team in terms of understanding how their product is actually used and the issues and concerns real users actually have than any amount of coding that person might manage in that hour.

[Gediminas Vilutis]

v12, according to a few months old data, should be out in January, but only god knows, for how long it will be postponed this time. HP for sure wont tell us :) Unmap is promised to be there.

For all those, who wants to run away from LHands. So far, if your SAN is iscsi based, I would stay away from 3PAR as well. We have a mixed environment with LHs and 3PARs and performance of 3PARs over iSCSI is much worse, than LH. Over FC 3PAR works like a champ, so I guess, iscsi subsystem on 3PAR is not much cleaned/optimised/debuged after it was implemented. When i complained about that to HPs regional storage 'evangelist', he said 'well, 3par is FC-born array with iSCSI connectivity, when LH was doing iSCSI before iSCSI was standartized, so what do you want from it?" :(

[Patrick Neuner]

Thanks for sharing this information!

[TechMarketing]

T10 Unmap (Space Reclamation) will be included within the next release of the LeftHand OS. The next release is 12.0.  

[oikjn]

source please.

[CalvinZito]

---

@oikjn wrote:

source please.

---

Hi everyone - apologies that I've been delayed in coming back to you. With the HP shutdown and lots of things happening, I forgot to come back to you here.  A couple of general comments:

1. As I've read through this thread again, I see lots of people questioning HP's commitment to StoreVirtual.  I can assure you that HP is committed to StoreVirtual and there is a significant engineer team working on it.  If you're paying attention to the marketing of storage these days, I'm sure you see lots of discussions about software-defined storage (SDS). SDS is a strategic initiative in HP Storage and StoreVirtual is key to that. 
2. I've asked the StoreVirtual team to write a blog post about the enhancements we've had with StoreVirtual. Obviously T-10 UNMAP is a big deal and none of us are happy that its taken longer than we wanted, but I hope the blog post Brad Katz and team are working on helps you see the forest through the trees - there's been a lot happening with StoreVirtual over the last couple of years.
3. There was a lot of discussion here about why HP hasn't been talking publicly about our StoreVirtual roadmap. At HP, we don't publicly disclose details of roadmaps. At times we'll give general guidance but HP can't disclose HP Restricted information without a signed non-disclosure agreement. You can usually get more details through your HP representative.  That doesn't mean we'll show you a roadmap for the next 2 years, but you can get more by talking to your HP rep.

@oikjn asked about a source for T-10 UNMAP.  I can't confirm that its coming in version 12.0 because that is HP Restricted and I can't confirm the OS release but I'll leave it to you to "guess".  We did announce some of the future enhancements coming at VMworld in September of 2014.  I have a blog post with a video that talks about what was announced and what's coming and that includes UNMAP.  Here's a link to that blog.

My job is focused on social media though support isn't something I can do - I'm not technical enough anymore for that (my BS in Electronic Engineering is 31 years old and I've been in HP Storage marketing for 25 years now) but I'm ALWAYS happy to help.  The best way to reach me if you aren't getting what you need is email: hpstorageguy at hp dot com.

[Patrick Neuner]

Thanks for your information, I just want to add 2 things: 

"but you can get more by talking to your HP rep". 

My HP Rep told me about 2 1/2 years ago, it will come soon. 

Last year we got hold to one of the product managers by email, that stated something similar, it's on high priority on the roadmap and soon... 

So basically yes, I understand why you don't disclose this information, as HP doesn't seem to know it themselves. So on the other side, it means, we really have to wait and hope, and if that takes 3 years. For us, the wait was too long, too many promises done, that we just have to move away because a new solution becomes cheaper than extending the lefthand. (which once UNMAP is there, wouldn't even be close to necessary). If HP would have disclosed some information and definite timeframe, we could plan and might have decided different. 

Yes, I somehow understand your point, but still it made you loose a lefthand customer as we were forced to make decisions based on information (or not available information) we had. We are small and probably you don't loose a lot, as we stay with many other HP hardware, but not SANs or Storage anymore. 

Patrick

[CalvinZito]

@Patrick NeunerI am really sorry to hear the frustration that HP caused you in this situation.  It isn't a shinning moment for HP and I'm equally as sorry to hear you decided to move away from StoreVirtual but I certainly understand your frustration.

There's nothing I could have done to change the schedule but I'm always willing to be a point of contact to help get the information you need to make better decisions.  Either PM me here or drop me an email at hpstorageguy at hp dot com.

[David_Tocker]

Annnnnd - back at step one.

I was going to write a clever vitriol filled comment - but nah, whats the point?

We certainly feel a bit ripped off after being told the same story as everyone else, but they still work well enough for a test environment.

Moving to StorServ was a great experience, the product is not just a little bit better - its night and day and delivers performance that you would be dreaming of achieving on Storvirtual, with less drives and a similar (lower?) price point.

Even if version 12 has unmap support - it will be over two years since promised originally, so the units we have will be out of warranty anyway soon enough. Its probably unlikely to work on G2 units anyway....

[a_o]

@David_Tocker

Can you expand on your comments regarding StoreServ?

I'm about done with StoreVirtual -  with all the delays, I'm convinced it's the red-headed stepchild in HP storage's solutions.

It now seems that for about twice the cost of support for a single P4000, you can now get an all flash 3PAR 7200c.

With the new features such as Fle Persona, StoreOnce Recovery Manager Central integration and Adaptive Flash Cache , I'm looking to add a single (2 node) 7200 to my network. 

I know that StoreServ seems to be geared more towards FC.
As ours is pure ISCSI environment, how is the ISCSI performance?

Do you have the Flash Cache implemented? If so, are the performance benefits as advertized?
Lastly, are you able to peer-motion between StoreServ and StoreVirtual ( and vice-versa) as advertized?

Thanks.

[Patrick Neuner]

You should check the post from Gediminas Vilutis  just some post up. 
He wrote, that FC is really good, ISCSI not. I can't tell from my own experiences, as we never used it.

Patrick

[Bart_Heungens]

@a_o

where do you see the advertisement of peer motion between 3PAR and Lefthand (easier to type compared to StoreServ and StoreVirtual) ;-)

I know there is peer motion between all (old, new, physical, virtual) Lefthand nodes and there is peer motion between 3PAR SAN's (old, new, big, small)... It would be great that there would be also between 3PAR and Lefthand, however I never saw an official announcement on it?

Other remark: if everyone thinks that Lefthand is a stepchild, why would they put it in the new Gen9 servers? With those servers you can deploy in the box with Intelligent Provisioning the VSA on your server.

Further if it would be a stepchild why they would give it away for free (at least the 1TB version) with any server equiped with Intel v3 processors? If there would be no future for it don't promote it imho

And to answer (partially) your question on iSCSI, I am using my 3PAR in my lab/datacenter with FC direct attach and iSCSI, both work like I would expect no issues so far. I did not have the time however to do some performance testing so far...

[oikjn]

@bart,

     list the big feature changes since v9.5.  If you remove "improvements to..." or "enchanced..."  or "increase....", or "now supports XXX latest OS", there have been very few if any important NEW features since version 9.5 which was released in in2011!   v10-11 were released in 2012-13 and included AO support for the VSA, but that pretty much ended the new features since the rest are of marginal increases at best...the "Rest API"  and support for SCOM only and 50TB licenses are more improvelements over novel features such as Unmap, dedupe, compression... whatever.  

Updating the software to support the latest operating systems or putting the existing software onto new hardware is at best a minimum requirement for mantaining the software and not really an indication that it is getting actively developed.  In 2009 you could say without reservation that lefthand was an industry leader for VSAs, can you really say that now?  They haven't done anything since then to lead new features that others might emulate. 

[CalvinZito]

@oikjn  Would love to hear from you what StoreVirtual VSA doesn't do that in your mind means it has fallen behind.  And obviously I know you (and I) aren't happy with the delay in getting UNMAP support but I'd like to know what other features it's missing. 

Personally, I think AO is huge. I don't think there's another VSA type product out there that can do sub-LUN tiering.

Also, curious what others have done (as you said, remove "improvements to..." or "enchanced..."  or "increase....", or "now supports XXX latest OS") that make those other solutions so much more compelling than StoreVirtual VSA.

I'm asking this in all sincerity - I'd like to understand where you're coming from and feed that directly back to the product team.

[oikjn]

I don't know what's the next leading new feature that everybody is going to want to copy, but for sure others are already including things such as in-line dedupe and compression, ram-read cache, not to mention universal SMI-S support and UNMAP and features like VAAI, ODX, and VCAI are also becoming more and more main-stream.  In 2015, those are features to expect in a SAN not exotic bells and whistles that are left to the domain of the tier-1 systems.

Just spitballing, but a future idea I haven't seen, and maybe this really gets more into a file-level system and isn't appropriate at the san level, but some sort of smart integrated defrag for the common file systems to either eliminate or minimize long-term file fragmentation like an integration with something like a program from condusiv would be a nice added feature that would help keep performance on a LUN at optimal levels by reducing unneeded inefficient IO.  Another pie in the sky feature might be some sort of DSM compression for increasing on-the-wire throughput which would be helpful for both WAN links and those limited by 1Gb local links.

[a_o]

@Bart_Heungens,

I stand corrected. I went back and looked at my notes, and from what I wrote down in late 2012,  Peer-motion is between similar systems only. It's just that I assumed that it was also possible between LH and 3PAR, as they were announced around the same time frame. The name "Storage Federation" is also tricky, as it connotes something between dissimilar systems.
As you said, it would be something nice to have  - especially when migrating (upgrading) from LH to 3PAR.

As it is, I use Veeam, so this hould not be much of an issue.

@Storage_guy,

Since you asked, I'll give you  my own take. Personally, in 2011, I evaluated Lefthand, Starwind and Dell. The Finalists were LH and Starwind - VSAs obviously. I went with LH for a wide variety of reasons. I still think it's the better product.

But since 2011, Starwind has added or significantly enhanced the following features to (or in) their VSAN product.

1, Unmap/Trim

2, Automated Storage Tiering

3, In-line Deduplication (plus IO blender)

4, On-The-Fly and at rest Compression
5, SMI-S provider  - better SCVMM 2012 support.

6, Target based Auto Defrag.

And there are thngs that it seems LH will never have, or are difficult to do without rebuliding the VSA like,

a, Scale Out - i.e. add capacity to existing nodes, not just adding nodes.

b, Built-in VTL  - i.e. not just snapshot

[Bart_Heungens]

@a_o

on your last point scale-out: you can now already grow your VSA with new disks online without rebuilding or so... As long as your license covers the storage underneath, I did already online upgrades of storage on the VSA in production environments...

And I sell always a self-build 'converged system' to my customers including StoreVirtual VSA for SAN (on SAS disks) and StoreOnce VSA (on MDL SAS disks) for backup-target with dedupe. 2 VSA's are fine for me since it gives more flexibility, don't see the adfvantage (yet today) to combine...

Have many installations like that, with happy customers...

Bart

[a_o]

I also have gone mostly VSA . I started with NSMs (have 8 of them as a 4 cluster NR1 config), then went with VSA after I saw that they were more flexible and were more performant than my P4K G2s.  I've got 6 VSAs running as pure tier 0  (SAS SSDs/ BLC IO accelerators). This was before AO came into being. So, I know what you're talking about. This is in addition to SSD/SATA AO enabled VSAs.

I brought up the VTL feature because at 50% cluster space utilization, I had to turn off Snapshotting that I was using as the source for remote snapshotting/copying to another cluster. The cluster would be become over provisoned as a result of trying to 'back' it up. Now, if the LUNs were a de-duped and UNMAPed...I digress.

I wonder how some here handle some of these real-world short comings of SAN I/Q.

Am I missing something here? How do you expand a VSA's capacity without 'Re-Configuring RAID'?
I've done that a few times. Essentially, one puts the node in a faux 'maintainance' mode by temporary having it's RAID not intialized, add the storage, 'Re-Configuring RAID' and re-stripe the Network RAID, Rinse and repeat on the other nodes in the cluster.

[Bart_Heungens]

There is no need for a complete re-stripe/rebuild of the VSA node...

Or you put the server in 'maintenance mode' (where the other VSA + a FOM keeps the volume online), and during reboot you enter the SSA (Smart Storage Administrator or also known as the ACU Array Configuration Utility) where you can grow the logical drive on RAID level. After that in VMware itself you can grow the datastore...

What works also is accessing the SSA through CLI when ESXi is running... Of course you must have the HP version of ESXi installed and verify that you have SSA as well in it.

VMware is not that easy, on a Windows server I can grow volumes completely online through the GUI...

Regarding your dedupe issue: you put it on the same VSA's on a remote site with also SAS/SSD drives? Wouldn't it be better to have MDL SAS drives? Much cheaper per GB for backups...

[a_o]

I think we're on the same page, but are looking at it from two different POVs.

I'm not saying the VSA node needs to be re-installed per se.

Since I run mostly (90%) on Hyper V  (I have two ESXi hosts running LH VSAs), I'll delineate the steps I've previously used to expand an existing VSA. I might be missing a step.

1,  In Hyper-V Manager, I shut down the VSA VM and present additonal storage to the VM. This usually takes the form of adding one or more VHDx file(s), or I expand the original  'Data' VHDx file to use the additional underlying storage. The 'configuration' VHDx file remains untouched.

2,  Reboot the VM. Now, the new storage is recognized by the LH's underlying Linux OS.

3,  In CMC,  login and "Reconfigure RAID". This usually destroys the data, as it's essentially creating a new raid set in the LH's underlying Linux OS.

4,   Restripe occurs as necessary.

Ideally,  there should be a RMC menu option in CMC that says something like 'Extend Storage...' 

Regarding  the backup scenario, the  destination MG is running on a couple of BL460s, each with a couple of 400GB SSD drives in Raid1 as boot drives.  The blades are each connected to a D2700 with 12 x 3TB SATA drives in a Raid5 config.
Each VSA is AO configured as  200GB Tier 0 (SSD on the boot drive) and 10TB SATA Tier 1 on the D2700.

[David_Tocker]

Hey there.

We have a 7200 with 25 900gb 10k drives.

This replaced our 4 p4500 g2 nodes - and it absolutely wipes the floor performance wise.

We don't use iSCSI on the Storserv - I have been told its performance is not great, at least compared to FC.

The 8GB FC gear is pretty cheap these days, I would just bite the bullet and go that way. The performance will be better than even 10Gb iSCSI.

We havent found it necessary to look at flash caching etc, the 7200 has about 24GB of ram onboard and uses it for caching and seems to perform really well.

Have not needed to Peer-Motion between them - we have System Centre VMM setup and just setup a side by side environment and migrated everything across over a few weekends.

The storage integration into VMM actually works with the StorServ - its a little bit wierd (numbers aren't accurate) but it will provision a LUN and assign it to the cluster nodes automagically, so its better than the StorVirtual in which the integration does not actually let you do anything (High five StorVirtual/Lefthand/P4000/whatever you will be called next year Team!)

Stability is improved - using Hyper-V with the P4000s we had little stability problems popping up occasionally - the P4000s themselves never went down, but the iSCSI connections would drop once in a while, even over different switches, servers and operating systems. Performance was never really satisfactory on the P4000s either. Always 'felt' slow.

The StorServ never seems to do anything particularly fast, but also never does anything slowly either - It seems to be setup for hosting enviroments etc so nothing can ever take over the IO and effect other applications.

In summary: Highly recommended.

[a_o]

Thanks for the feedback.

A few questions though...

Are you running plain FC or is FCOE?

Are the 7200's  controllers able to run FCOE natively?

As we're running a mostly 2008R2 HyperV cluster setup,  ISCSI is great because CSVs can be presented to any Hyper V host on the network.  Our main DR server room  is about 1000ft away from the main datacenter - connected via TOR H3C switches with IRF.  As a side note, I've even deployed a stretch cluster by putting a VSA  node at a remote site (20 miles away) -  connected by a VPN without a problem with LHs multi-site functionality.

It would seem to me that this kind of topology would be problemmatic to replicate with 3PAR using just FC,  is it not ?

I've been seriously looking to moving to 2012 R2 Hyper V clusters because CSVs are no longer needed.
With the new 3PAR 7200c and their Converged controllers and File share 'personalilty' software, one would just put SMB shares anywhere on the network, and use them as shared storage for a Hyper-V cluster.

Lastly, I'm not sure if you know the answer, but can a 7200 be upgraded to a 7200c by just swapping the controllers?

[Bart_Heungens]

@a_o answer on your last question: yes you can simply replace the non-c controllers with the new c version...

For FCoE you need a duak-port 10Gb iSCSI/FCoE adapter in the PCIe slot of each controller.

Be aware that, if you want File Persona (NAS) functionality you need another network adapter in that same PCIe slot... So today you have to choose FCoE or NAS...

[CalvinZito]

---

@a_o wrote:

Are the 7200's  controllers able to run FCOE natively?

Lastly, I'm not sure if you know the answer, but can a 7200 be upgraded to a 7200c by just swapping the controllers?

---

The HP 3PAR StoreServ 7200 (in fact across the product line) doesn't support native FCoE - it's done via an adapter.

An existing 7200 system can be upgraded to 7200c via controller swap.  If it's something you're seriously thinking about, you should get contact your HP Rep.

[Gediminas Vilutis]

---

Bart_Heungens wrote: 

And to answer (partially) your question on iSCSI, I am using my 3PAR in my lab/datacenter with FC direct attach and iSCSI, both work like I would expect no issues so far. I did not have the time however to do some performance testing so far...

---

If you have 3PAR, connected to iSCSI, could you check flow control pause frame statistics on ports, where 3par controllers are connected? Some switches show it in standard 'show' or 'display interface' command output. If your switch does not show this statistics, then snmpwalk'ing oid 1.3.6.1.2.1.10.7.10.1.3 would do the trick.

Basically my problem with 3par is that write throughput to it via iscsi is terrible. Array starts to flood switches (ProCurve 6600) with pause frames when write speed from hosts reaches ~300 Mbps per interface and raises to something like 5000-6000 pause frames per second rate if I push traffic above 500Mbps (that is about 2 Gbps per array). Disk drives on backend are more or less iddling (~70-80 IOPS per 10krpm drive) during this. Problem now is reported to HP and is escalated to L3 support, no feedback yet.

So at the moment, until this is not fixed, I still can say - 'iscsi? no 3par.'. YMMV.