StoreVirtual Storage
cancel
Showing results for 
Search instead for 
Did you mean: 

P4500 Multiple CSV Volumes Poor Performance

SOLVED
Go to solution
mcc_will
Occasional Advisor

P4500 Multiple CSV Volumes Poor Performance

Hello,

 

We're running 3 DL380 G7s in a failover cluster.  They're running Windows Server 2008 R2 hosting Hyper-V virtual machines.

 

We have two CSV volumes on the P4500 for virtual machines.  The first CSV volume is flawless and works correctly. The second CSV is setup exactly the same.  However there are errors that occur in Failover Clustering and read/write access fails and performs poorly on the second volume.

 

The error I receive is...Cluster Shared Volume 'Volume2' ('NAME') is no longer available on this node because of 'STATUS_DEVICE_BUSY(80000011)'. All I/O will temporarily be queued until a path to the volume is reestablished

 

It happens periodically and I don't see anything in event logs that will help.

 

Thanks

Bill

 

 

 

P.S.This thread has been moved from Storage Area Networks (SAN) (Enterprise) to HP StoreVirtual / HP LeftHand Storage- HP Forums Moderator

13 REPLIES
Jan Soska
Honored Contributor

Re: P4500 Multiple CSV Volumes Poor Performance

Hello, I am afraid this is task for HP support.

 

Jan

oikjn
Honored Contributor

Re: P4500 Multiple CSV Volumes Poor Performance

first place I would look is in your iscsi config.  did you switch mpio from vendor to round robin?  are all connections active?  What does CMC show for DSM connections on your volumes, they should look just about the same.

 

Have you tried disconnecting the removing your targets on the windows servers and reconnecting them new making sure you are following the best practices?  There are a lot of clicks to select to setup each connection and even though I've done it a thousand times, I make sure to verify I set the targets and sources correctly and every once and a while I find I made a mistake and have to recreate it.

mcc_will
Occasional Advisor

Re: P4500 Multiple CSV Volumes Poor Performance

Hello, thank you for the reply.

 

MPIO is still Vendor and all connections are active. In CMC I only see the iSCSI sessions active, both CSV volumes have the same sessions to the servers.  Everything looks the same. I do see two of the nodes are not assigned to a site in the multi-site SAN.  I didn't build this one out originally so I have little to go on what was thought originally.

 

I did disconnect from the volume and reconnect with the same results, I made sure I did everything correctly.

 

I'd like to solve this problem for the future. If I could expand the existing CSV without any issues that would solve some of the problems.

 

Thanks,
BIll

mcc_will
Occasional Advisor

Re: P4500 Multiple CSV Volumes Poor Performance

Or did you mean they are supposed to be round robin? Not sure if you mean that is how it should or shouldn't be configured. Both CSV volumes are Vendor, should the both be round robin?
oikjn
Honored Contributor

Re: P4500 Multiple CSV Volumes Poor Performance

they should be set to round robin.  There is a BPA somewhere burried in the HP site that walks you through the configuration.    With MPIO and DSM installed and enabled, the HP DSM will manage the links and use them all w/ round robin and I believe vendor specific only does A/P... seems counter-intuitive that you don't use the vendor one, but you shouldn't and the manual says not to.  You should see a link from each server NIC to the coordinating memeber VSA and then another connection to each node from each server NIC that says "DSM".  If you look at the traffic on your server by each nic you should see traffic going across all of them.

mcc_will
Occasional Advisor

Re: P4500 Multiple CSV Volumes Poor Performance

Thanks, this makes sense. DSM is installed and I have the Round Robin option available. Not sure what it would change. I don't see anything marked as DSM in the CMC.  All the NICs are active (they're teamed in 3 teams). All output.  I only have one session and one path listed in iSCSI initiator to the device.  I don't like changing this option. Is it reversable or not otherwise matter if it'a Failover or Round Robin. I believe Failover and Vendor are the same, correct?

 

Thanks,
Bill

oikjn
Honored Contributor
Solution

Re: P4500 Multiple CSV Volumes Poor Performance

I think vendor and failover are the same.

 

do you have the HP DSM installed and have you installed MPIO.  you really should read and follow the HP guide for setting up your iscsi connections.  It walks you through every step.

 

I believe it says switching to round robin isn't reversable, but there is no reason not to have it round robin.  the fact that you don't see any iscsi connection listed as saying DSM indicates that something isnt' setup correctly.  The HP DSM will greatly improve your performance since it will allow load balancing across the nodes and it also make sure to request data from the node with the data on it instead of a random node which then has to request the data from the other node which is a waste.

 

switch to round robin... no reason not to.  I don't think it will even interrupt your connection.  if you want to be ultra paranoid, you can always turn on redirected access for those CSV disks before you do that so you aren't risking anything.

mcc_will
Occasional Advisor

Re: P4500 Multiple CSV Volumes Poor Performance

I'm not sure if I have the right guide or not. Nothing I have from this one seems wrong other than round robin. Also jumbo packets aren't on, and I think that'll help another issue we're experiencing.

 

I want to mark this as the answer but getting ok to make changes is slow. Hopefully I'll be able to do it soon.


Thanks,
Bill

mcc_will
Occasional Advisor

Re: P4500 Multiple CSV Volumes Poor Performance

Hello,

 

Thanks for your help.  This seems to be working in my testing. The DSM needed to be updated and then on my test volume it started showing up as DSM in the CMC.  I redid the connection on that server to try to alieviate any other issues.  It'll be a slow process to get to everything.  I now have 7 iSCSI sessions (6 DSM).  Someone must have messed up the install the first time.

 

One thing that was weird was that the VM I'm using lost the iSCSI sessions and access to the server the first time.

 

Thank you very much for your help.

 

Bill

oikjn
Honored Contributor

Re: P4500 Multiple CSV Volumes Poor Performance

glad to hear its working now.

 

You should probably add a 2nd NIC each of your Hyper-V servers for iSCSI.  I thought the hyper-v validation tests required it, but I guess not.  Adding the 2nd nic will give you more throughput for each server to the storage and also provide link redundancy so your LUNs don't go down even if something happens to one of the links.

 

For me, I have both nics in the same subnet and keep the gateway removed from the connection so only iscsi traffic flows.  I'm not doing multi-site to the hosts so there is no need for those connections to talk outside of the local targets on the same subnet.

mcc_will
Occasional Advisor

Re: P4500 Multiple CSV Volumes Poor Performance

Hey,

 

There's actually many nics in these the 4 internal NICs are teamed for management. Then 4 SFP+ 10Gb ports split across two cards for iSCSI and network access.  So, there's a 20Gb iSCSI connection to the SAN and 20Gb to the rest of the network.  As well as 20Gb between sites in the Multi-Site SAN.  Of course redundant so speed would be reduced to 10Gb if something fails.

 

Thanks,
Bill

oikjn
Honored Contributor

Re: P4500 Multiple CSV Volumes Poor Performance

hmm...  not sure how that one is handled.... I didn't think Microsoft supports iscsi teamed and instead suggest that you do not team and use MPIO to handle failover and throughput.  If you have two 10gb for network and two 10gb dedicated for iSCSI, you should remove that iSCSI team and then setup each NIC on its own following the HP manual.