StoreVirtual Storage
1747985 Members
4987 Online
108756 Solutions
New Discussion

Re: HP P4300 SAN cluster seems slow. Am I expecting too much?

 
Jay68
Occasional Advisor

Re: HP P4300 SAN cluster seems slow. Am I expecting too much?

Well, we had a power outage last night and and migrated everthing over to our DR site.

I took the oppertunity to rip MPIO/DSM off the server and remove the iSCSI configuration. After plumbing it all back in and changing the MPIO load balance policy from Vendor Specific to Round Robin, I now get this:

 

Sites-iSCSI

 

It looks like this has fixed the issue where NIC .15.103 was not connecting to the SAN. Also, the sites information explains why  HQ server will only talk to the local nodes (.15.1 & .15.2) and the DR server the DR nodes (.15.3 & .15.4) but, both servers' primary connections are all to HQ node .15.2. I guess this means that this node is acting as the gateway for the volumes and that chopping up the LUNS may increase the number of primary connections.

 

Is there any benefit to having these 4 nodes split into two different sites as they're connected by 10GB fibre? Both sites have separate power sources so I'd like to be able to make sure that non of the LUNs are lost if node .15.1 and .15.2 are lost or if it's .15.3 and .15.4 that go down. If I don't sepcify the sites, could a LUN end up on node .15.1 and Network RAIDed to .15.2?

 

I think I may have confused issues slightly on the filesystem sizes. The sector size in CMC is reported as 512bytes but the NTFS cluster size reported by diskpart is 4K. I've read another forums thread where a poster said their VMs seemed slow and that formatting the cluster size of the volumes to 64K did make things snappier.

Emilo
Trusted Contributor

Re: HP P4300 SAN cluster seems slow. Am I expecting too much?

It looks like this has fixed the issue where NIC .15.103 was not connecting to the SAN.

Enabling round robin is what fixed that issue. Here is the formula to make sure you have the correct number of connections.  Volumes * host * host nics * ( storage nodes +1) so in your case you should have 6 iSCSI connections per volume. Which it appears you do.

 

Also, the sites information explains why  HQ server will only talk to the local nodes (.15.1 & .15.2) and the DR server the DR nodes (.15.3 & .15.4) but, both servers' primary connections are all to HQ node .15.2. I guess this means that this node is acting as the gateway for the volumes and that chopping up the LUNS may increase the number of primary connections.

 

I see that you have multi-site enabled and you have the servers (sw iscsi)  setup at each site. That is how you can control which host will act as the gateway.  Make sure that the servers connect to the volumes that are closest to the users, and that 'site preference' is enabled.  With only tow nodes int the cluster you are not going to pickup much benifit ,but don't forget by enabling muliti-path gateway node is only used for administrative purposes (management stuff) you are getting direct connections from the DSM driver. With MS DSM/MIPO the driver has intimate knowledge of the  layout of the storage custer (normally performed by the  gateway) it can calculate the location of any block and allows the iSCSI driver to contact the storage system that owns the block directly, without using the standard gateway approach). Look at the algorythim below and see what you want the behaviour to be. I got this from the manual by the way.

 

Is there any benefit to having these 4 nodes split into two different sites as they're connected by 10GB fibre? Both sites have separate power sources so I'd like to be able to make sure that non of the LUNs are lost if node .15.1 and .15.2 are lost or if it's .15.3 and .15.4 that go down. If I don't sepcify the sites, could a LUN end up on node .15.1 and Network RAIDed to .15.2?

 

There is a specific algorythim that is built in to handle just how the fail-over will occur.  This all depends on if site preference is enabled or not.

 

If no site preference:

DSM sessions connect to all storage nodes.

 

If site preference is used in multi-site configuration:

DSM sessions only connect to preferred site, say Site A.

 

If site preference used, but preferred site is unavailable:

No DSM sessions connect.

Single iSCSI sessions connect to non-preferred site.

 

If site preference is used, but preferred is not in the cluster

DSM sessions for volumes not in that cluster will be established with the other site(s) that have the storage nodes for that cluster. (Will act as if you had no site preference.)

 

I hope this helps. If so dont forget to give kudos and that is solved.

 

 

oikjn
Honored Contributor

Re: HP P4300 SAN cluster seems slow. Am I expecting too much?

I must say I am jelous of that 10Gb fibre intersite connection :)

 

I would definitely keep your Nodes assigned to sites to ensure you keep your site availability for data in case of a site failure, but you could try simply locially moving one or two servers from their current site to the FOM site (not physically, just logically) or make the servers "unassigned" to any site.  That would have them connect to all nodes, but at the cost of potential latency.  Generally you want to keep the traffic on the local site only to minimize bandwidth and latency issues as wirting to the second site would require effectively four hops of data to move along for you to request something, pass to the  DR site, have it NR10ed back to the primary, confirmed back to the DR and then confirmed back to the server.... that delay in latency generally outweighs any potential bandwidth gains associated with connecting to the additional nodes.  If you have the link speed and latency (as you should with 10Gb fibre) you can certanly try keeping some servers assigned to the primary site and others unassigned and see which works better in your situation.

 

 

Looks like you fixed the problem with the iSCSI sessions and you now have it configured correctly.  generally I find it easy to forget about moving the DSM to round-robin for one or two NICs and it sounds like thats what happened w/ whomever set it up the first time. 

 

Don't get destracted by the non-dsm session... think of that session equivalent to what the FOM is to the management group.  It doesn't actually move real data across that connection, its only there as a start to help the DSM figure out and make its connections, once one DSM connection is up, that can handle everything the non-DSM connection can do as well if the non-dsm connection were to go down....  not that you would want to test that on your production system, but if you setup a VSA lab, you could test this failover by setting up a similar configuration that you have now and then simply reboot or shut off the VSA that is currently the "gateway" and as long as you have the 2nd DSM connected to each nic, you will not see any interruption in LUN access....  this is exactly why the system can do a non-disruptive system update even through they generally require nodes to reboot.

 

 

Jay68
Occasional Advisor

Re: HP P4300 SAN cluster seems slow. Am I expecting too much?

So would it be better to remove the multi-site configuration? The HQ and DR sites are two server rooms on the opposite side of a road. Originally the connection was a 1GB fibre link and site preference was set up to prevent this link being saturated.

 

The link speed has since been upgraded to 10GB so I'm wondering if there is any need for a multi-site set up? There should be no issues for the HQ and DR servers to connect to all four nodes from a network perspective. The only requirement we have, is that due to these server rooms being on separate power grids, I need to be able to make sure that the LUNs are available if the HQ site has a power outage (node 15.1 and 15.2 are lost), or if the DR site has a power outage (nodes .15.3 or .15.4 are lost). I need to be able to make sure that .3 and .4 are replicas of .1 and .2.

 

Am I correct in thinking that it is installation order that determines this? Just to make things extra complex we installed these in the order .1, .3, .2, .4 but this should mean the HQ nodes are replicated to DR.

 

I feel that there is a fair bit more performance to be had out of this thing, it's just knowing where to tweak it.

Jay68
Occasional Advisor

Re: HP P4300 SAN cluster seems slow. Am I expecting too much?

It sounds better than it is, HQ building is literally 1min walk from the DR building. We're just incredibly lucky that both buildings are on different power grids.

 

I get the feeling that this slowness is probably down to a latency issue rather than throughput so I'll look at chopping up my LUNs and formatting them with a 64k cluster size first of all. If I still have issues I'll then I'll try moving the servers to an unassigned group and see how things go then.

 

Big thanks to everyone for all your help! 

oikjn
Honored Contributor

Re: HP P4300 SAN cluster seems slow. Am I expecting too much?

I don't know if I would define anything you've said as "slow".  remember that throughput is a function of I/O rate and I/O size.  your disks can only provide so much I/O.  After taking into account any disk raid I/O penalty is your I/O really lower than expected?  Watch your LUN I/O stats, Node I/O states and cluster I/O stats as well as latency on CMC and then watch I/O on your hosts.  It could be that every requested I/O on the host generates 5+ I/O to the SAN because of cluster formatting size.  Otherwise, it could simply be that the two are close together, but your nodes are only capable of 2000 I/O and in order to increase throughput you would have to increase your I/O average size.  I think we can say now that your SAN connections to the nodes are all correct and the issue isn't your host to SAN connections, so now you should probably focust on your node I/O and host I/O to see if they are what you would expect.