XP256 disk array: I/O performance degradation

[Fedon Kadifeli]

Our production database server (V2600 system connected through redundant FC interfaces to an XP256 disk system; no SAN) is suffering a disk I/O degradation. I/O performance has decreased by a factor of 1.5 in Oracle exports (db -> disk dump files), Oracle online backups (db -> LTO tape) and ordinary file backups (filesystem -> LTO tape). For example, a speed of 1.28 GB/min has dropped to 0.85 GB/min; and a speed of 1.8 GB/min has dropped to 1.2 GB/min.

The same problem occurred during September; it lasted about 10 days and later turned to normal without doing anything. At that time and this time again, I inspected system logs, perfview histories and XP256 PM graphs, but I was not able to locate the problem.

Please, note that I have checked the DWL (Duplex Write Line) setting of the XP256 (in Perfomance Manager); it is fixed to 50% and DWL
Scheduler is disabled (it has always been as this).

We have another machine (N4000) on the same MC/ServiceGuard cluster which is also connected to this disk array through redundant FC interfaces. I did some tests from this system and I saw the same performance degradation.

Are there any users which have encountered the same or similar problems before?

[Sameer_Nirmal]

Hi,

We had a problem of Disk I/O bottleneck with XP 256 connected to Superdome and N4000 servers. The Oracle based application used to slowdown often during normal working . Backup/export used to be slow as well. This perforamnce issue was only came out after around 4 years since installation.

We found the im-proper layout of DB on the array as a cause and capability of XP 256 was in question. Eventually ( knowing running cost and future specific growth/requirements )we replaced it with EVA 5000.

Since in your case , performace issue is not often and you have identified the cause of the problem as disk I/O. So there has to be a massive change in the behaviour of the I/O against the XP making the DISK I/O as bottleneck.
For example, while oracle online backups , there might be some batch jobs runnig etc.

Oracle has very complex and invariable I/O patterns while doing its activities. Moreover DB gets increased day by day making it more complex. So its all depends on how the setup is made for application & storage.

There are many aspects influence the performance of the XP array.
1. SCSI Queue depth at the host level
2. Bad configuration of XP FC ports , portmapping, primary and alternate paths configured
3. Cache size
4. Array groups/CU configured , RAID used , LUNs distribution etc
5. Striping at host or array level
6. Oracle DB size & layout

Bottom line is
Monitor the complete system at OS and H/w level. Do the tunning at host/storage level by knowing the I/O patterns of Oracle and application as whole consulting application/DBAs.
This should given you idea what makes this issue occuring after an interval.

It is always better to use " Performance Advisor" instead of PM.

[Alzhy]

Most XP / Hitachi Array performance degradation issues is largely due to introducing improper allocation and layout of data files (specially Oracle/RDBMS storage). Most likely subsequent to the initial layout and allocations, allocations have been made without regard to spreading out the components of the storage unit (LVM lvol or VxVM volume) via strict adherence to striping rules. If you think this has been observed throughout the years .. then check more closely system activity - if normal system operation is still there BUT your issue is just with the sudden drop of backup and export throughputs -- then look at your system more closely. Do you have jobs running alongside these exports and backups?

[Fedon Kadifeli]

One month ago I decided to power cycle the XP256 disk system (as a last resort).

Guess what?

The performance degradation problem is gone! The disk system started working as the days before September. I am monitoring the system for one month; no performance degradation observed during that period.

My opinion is that there is some bug in XP256 firmware (possibly a memory leak) that results in loosing some portion of its cache memory. Rebooting the disk system corrected this.

[Alzhy]

Rebooting the XP possibly meant all your connected Servers was rebooted/restarted/refreshed as well.

And guess what? It may also be possible something got "reset" on your servers.

All of my XP's and Lightinings and USPs have never been powered down/rebooted since they rolled in. Even during firmware updates.

So whatever it is that was cassing "I/O degradation" - is certainly not your storage array.

[Fedon Kadifeli]

Nelson,

I do NOT think so. The servers (which are very few) have been rebooted several times in that period (for other maintance reasons).

It is the reboot of the XP256 disk system that made the difference. In that problematic period we had upgraded the firmware of the disk system to the latest one (with no reboot on the disk system); but nothing changed.

I STRONGLY believe that the LUSE feature which we started using first time in September has something to do with this performance problem.

The first time I defined a LUSE volume (that was last September) the degradation started and lasted about 10 days (explained above).

The second time I defined more LUSE volumes the performance problem reocurred and never returned to normal until I rebooted the XP256.

[Alzhy]

Well if you've full support - I will strongly suggest you get HP involved. They should have all the diagnostics tools to establish the real issue.

If you've measureware, gather stats as well as sar.


Good luck.