EMC performance low

[firas_1]

EMC storage connected directly to N4000 HPUX 11.0. This connection is a ne one. Before it was jamica disks. After connected to EMC found that system performance degraded to be very slow. Ned your help.

[Stefan Farrelly]

Well thats a shock. Normally EMC are a lot faster than a JBOD like a Jamaica.

You need to provide a lot more info first;
1. ioscan -fknCdisk
2. bdf
3. sar -d 1 10
4. sar 1 10

and what makes you sure theyre slower ?

[firas_1]

Hereis: /etc/fstab and glance o/p. attached.
Still jamica and EMC storage connected to the system. there is no perfomance problem when running the application (BAAN/ORACLE) from the mount point of the Jamica. When start running it from EMC, In glanc o/p cpu utilization start increasing and reaches 100%. other resources like disks and memory no increase.

/dev/oraemc/lvol4 /ora4 vxfs delaylog,largefiles,rw,suid 0 2
/dev/oraemc/lvol5 /ora5 vxfs delaylog,largefiles,rw,suid 0 2
/dev/oraemc/lvol3 /ora3 vxfs delaylog,largefiles,rw,suid 0 2
/dev/oraemc/lvol1 /ora1 vxfs delaylog,largefiles,rw,suid 0 2
/dev/oraemc/lvol6 /arc1 vxfs delaylog,largefiles,rw,suid 0 2
/dev/oraemc/lvol7 /app vxfs delaylog,largefiles,rw,suid 0 2
/dev/oraemc/lvol8 /arc2 vxfs delaylog,largefiles,rw,suid 0 2
/dev/oraemc/lvol2 /ora2 vxfs delaylog,largefiles,rw,suid 0 2

[Stefan Farrelly]

Im not sure thats a problem.

When you run it from Jamaica the disk throughput is quite low, therefore your application may be i/o bound - which means it doest use much cpu because its being held back by the i/o speed.

When you run it from the EMC the disk throughput is so much faster now that your application is now not i/o bound, which enables it to run a lot faster - consuming more cpu. This is what it then does. Users should notice their jobs completing much faster now ?

What is the sar 1 10 output (and sar -d 1 10)when running it from EMC as opposed to Jamaica ? This should confirm it.

[Robert Thorneycroft · Accepted solution]

You should really consider logging this as a problem with EMC! I run a 1.5TB database on EMC (also with an N class as the DB server), the databse is also mirrored three ways and the performance is exceptional.

There is definitely a problem sonewhere if you are getting this level of performance, my own guess would be that it is likely to be with the connectivity between the EMC and the host especially as you say the connection is a ne (new?) one.

EMC should be able to dial into your box and diagnose the problem for you, often even when the problem is located on the host rather than the array.

Other things you might want ask is if the cache is working, and possibly the layout of data on the physical, rather than logical disks on your system (As several logical disks will reside on the same physical, and may result in a hotspot if they happen to contain maybe online and archive redo logs as well as some of your heaviest hit tables and rollback segments.)

Hope this helps.

Kind regards,

Robert Thorneycroft.

[firas_1]

The users feeling the system is very slow when running the application from EMC, so ther are not feeling its faster, the opposite. now the sar o/p will forward itto you once scheduled for downtime to run the apps from the EMC. but for time being, if you have more ideas wil be greatful

[Robert Thorneycroft]

Just out of interest what is the connection you have to the array?

You might well be getting a bottleneck at the adapters if you are just running a few SCSI connections.

My system has a total of 8 Fibre channel connections which seems to keep that data flowing pretty smoothly, also do you use powerpath to do load distribution across your adapters? It might be that all of the data is trying to passs down one connection whilst others are completely idle.

Regards,

Robert Thorneycroft

[firas_1]

By the way, copying huge data to the EMC was in an exception speed. if this will shed more light.

[Stefan Farrelly]

Thats good - as you would expect, the EMC is very fast. This backs up my earlier opinion that when using the EMC its allowing your applications to run faster thus using more cpu and giving the 'impression' to the users its performing worse (as your server is running at almost max cpu whereas it wasnt before).

I think you may have 2 types of queries. Normal users doing quick lookups, and those running large reports. As a result of your faster EMC those large reports will run a lot faster (as I/O is quicker and thus they consume more cpu now) but this will have the knock on effect that those users doing quick lookups feel performance is worse as with less cpu available their quick lookups take a little longer. This is a common problem, we just had one like it this week. In the end we had to get those running large reports to not run so many at once or change the way their reports run so it is less intensive on the system - this then freed up some cpu so others could get quick lookups done faster.

[Robert Thorneycroft]

I believe Stefan may well be right, looking again at your original glance output as you mentioned CPU is always very high.

What you need to do is look within the application to try to work out which processes are the ones eating all of your CPU.

I do not know BAAN as we are a SAP site but we have to carefully restrict batch processing during the day down to only a few concurrent processes, and prevent users from running reports online which will last for more than about a minute.

You might well be able to solve a lot of this problem by tuning your reports as well. A poorly written report can consume large amounts of resource for hours, this can often be reduced to neglegable levels with a re-write.

I would recommend asking your company if you can buy Spotlight on Oracle and SQLab from Quest Software. These are (Cheap) Oracle tools for people like me who are not Oracle DBAs that help you tune a system. Within a month of using these two tools I tuned about 40% of the logical reads and 50% of the physical reads out of the database, which essentially saved our company upgrading to a larger server.

Kind regards,

Robert Thorneycroft

[Tim D Fulford]

Comming in late

1 - MeasureWare, check out the historical performance of the disks, use the BYDSK
extract -xt -d ....
Check out the service times and the individual disk queues.
2 - EMC I do not deal with EMC, but from what I hear they are very good (well excellent) I'm sure your technical/account manager would take a look
3 - Sequential scans, are you sure cron is running the update ststs etc???
4 - Kernel params, I seem to remember a parm something like "max_scsi_qdepth" (again ask EMC)

You really neeed to supply the sar -d 20 1 output then others can clearly see what disks are doing what.

Regards

Tim