Operating System - HP-UX
1748006 Members
4493 Online
108757 Solutions
New Discussion юеВ

Slow performance after SAN migration

 
Syed Mehmood Ul-Hasan
Occasional Advisor

Slow performance after SAN migration

We recently upgraded our SAP from HP XP1024 to XP12000. However, we are experiencing slow performance since the migration.
Looking at the KI trace and SAR numbers, HP confirmed the new SAN is healthy.
We traced waits for Oracle processes and it points to I/O bottleneck.
We are trying to figure out the root cause of this slow performance.
Any advise/suggestion is welcome.

Thanks
7 REPLIES 7
Steven E. Protter
Exalted Contributor

Re: Slow performance after SAN migration

Shalom,

I suggest you measure performance in some way.

Glance/gpm can do it, as can a script I developed based on a HP script.

http://www.hpux.ws/?p=6

You may think you know the answer, but you need to figure out which disk it is.

Most SAN's do have utilities to track performance of presented LUNS.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Syed Mehmood Ul-Hasan
Occasional Advisor

Re: Slow performance after SAN migration

Thanks Steven.
It is not one disk or few disks that we have performance issue with.
There are 2 servers we moved to this new SAN and both have performance issues since migration. Performance issue is not in any specific area, it is general i.e. all batch jobs are taking more time to complete.
Mel Burslan
Honored Contributor

Re: Slow performance after SAN migration

Syed,

If the only change was to migrate the storage from xp1024 to xp12k and you are sure absolutely nothing else has changed since then, makes it sound like your SAN fabric is not optimized and may have bad FO cables, modules, HBAs and what not.

SAN performance problems are very hard to determine unless one sits on the switches and analyzes the I/O streams and performance.

My guess is, your connectivity to the new array is less than optimal, and definitely worse off than your connection to old xp1024. Unless you are a SAN expert, I'd strongly suggest using the FO switch provider's consulting arm and employ their services for a few days to analyze the performance.

By the way we did the same migration and we have reached blazing speeds compared to running on 1024. So, it is not normal that you are having performance issues.

HTH
________________________________
UNIX because I majored in cryptology...
Syed Mehmood Ul-Hasan
Occasional Advisor

Re: Slow performance after SAN migration

We verified the SAN fabric and it seems optimized.
We have been told by storage experts that with XP1024 (old SAN) we were getting 5ms of I/O turnaround i.e. the server waits for 5 ms before it gets the requested I/O from SAN.
With XP12K (new SAN) we are getting 10-15 ms of I/O turnaround which for XP12K is normal.
The reason for 5ms of turnaound with XP1024 is that potentially we were getting it from the SAN cache.
With XP12k SAN we have more servers attached to it whereas XP1024 did not have many servers attached to it.
Therefore we are not able to cache per server as much with XP12K as we were able to with XP1024.
Does this make sense?

To resolve the performance issue we tried increasing the Oracle SGA and it did improve performance. Not as much as it was with XP1024. This performace increase is due to the increase in Oracle database buffer.

Any ideas/suggestions on the root cause of the performance degradation?
Thanks
Bill Hassell
Honored Contributor

Re: Slow performance after SAN migration

> The reason for 5ms of turnaound with XP1024 is that potentially we were getting it from the SAN cache.

I think you have hit on the biggest contribution to the delays.

> With XP12k SAN we have more servers attached to it whereas XP1024 did not have many servers attached to it.

Certainly that will impact every host connected to the storage.

> Therefore we are not able to cache per server as much with XP12K as we were able to with XP1024.

And that's a bad thing. As you have seen with a large Oracle SGA, there is no substitute for lots of memory. If you can add several hundred GB os RAM to the XP12k, you will see a lot better performance.

Although disks are betting larger (up to terabytes), average performance has not changed that much. It still take a few milliseconds to move to the next track, average access time is still in the 10-20 ms range and full track seeks are dozens of milliseconds. The only way to mask these access times is to arrange the disks in parallel (striping), have multiple access ports (HBAs) and 'remember' a *lot* of data by using a memory cache.

It is hard to expect similar performance when the new array is being hit hard with more hosts and less memory. You might see if there are premium performance configs you can use for heavy Oracle LUNs and migrate lower priority LUNs to archive status.


Bill Hassell, sysadmin
Olivier Masse
Honored Contributor

Re: Slow performance after SAN migration

I've been able to reduce performance bottlenecks on my arrays by adjusting the SCSI queue depth. You might try increasing it two fold or more, and see if it changes something.
Syed Mehmood Ul-Hasan
Occasional Advisor

Re: Slow performance after SAN migration

Thanks Bill and Olivier.

It seems to me the main issue is the SAN cache.
And the only solution is to add more cache to improve performance.