topic HP-UX Performance Issue in Operating System - HP-UX

HP-UX Performance Issue

Vinod_22 — Thu, 24 Jun 2010 09:21:37 GMT

Dear all,
I will brief here very strange and peculiar performance problem with two HP-Itanuim Servers
The Server details are
Rx2660 (Good Configuration but very bad performance)
CPU: Two dual core (1.42 GHz, 12MB 9120N) processor
Memory: 16GB
Disk: 146GB 10k SAS
OS version HP-UX 11.31 release Sep2009
N/W: 1gbps
Script runs time: 68 min

Rx4640 (Not that great Configuration as 2660 but very impressive and faster performance)
CPU: HP mx2 dual processor
Memory 8GB
Disk: 73GB
OS version HP-UX 11.31 release Sep2009
N/W: 1gbps
Script runs time: 20 Sec

When we run congnos script the script will get complete in 15sec on rx4640 vs. it takes more than hour (68min) on Rx2660.
When the congnos script runs it call DB which is running on other Server and choose the fields and performs insert to another DB. In this case both Servers ( 2660 & 4640) are connecting to same DB Server. Also the DB client is also same on both the Servers.

If I compare both Server CPU, memory, disk and N/W the Slower system (2660) has all good horsepower compare to rx4640 but, when performance comes it acts like 10year old box.

We have tried to bring both OS same version and all kernel parameters same but not able to understands why the performance differ with such huge unbelievable difference.

Really appreciate if we get faster response on this.

Re: HP-UX Performance Issue

Modris Bremze — Thu, 24 Jun 2010 10:11:29 GMT

Maybe it's not the servers themselves, but something in between them - how are both servers connected to that "other DB server" they both rely on to run the script? Have you looked at both server CPU utilization and system load while your script is running?

Re: HP-UX Performance Issue

S. Ney — Thu, 24 Jun 2010 12:40:16 GMT

Have you looked at glance on the 4640?
sar -L
iostat -L -t
Do you have tusc installed on the 4640?

Re: HP-UX Performance Issue

study unix — Thu, 24 Jun 2010 12:46:59 GMT

try to use top or glance ,but glance needs to be paid

Re: HP-UX Performance Issue

Vinod_22 — Thu, 24 Jun 2010 14:01:12 GMT

Please find below o/p for the both Servers
rx4640 (Great Perf)
rx2660 (Bad Perf)
#uname -a >>/tmp/perfor/hp1.txt
#bdf >>/tmp/perfor/hp1.txt
#dmesg >>/tmp/perfor/hp1.txt
#uptime >>/tmp/perfor/hp1.txt
#model >>/tmp/perfor/hp1.txt
#sar -u 2 10 >>/tmp/perfor/hp1.txt
#sar -d 2 10 >>/tmp/perfor/hp1.txt
#sar -b 2 10 >>/tmp/perfor/hp1.txt
#sar -v 2 10 >>/tmp/perfor/hp1.txt
#vmstat 2 10 >>/tmp/perfor/hp1.txt
#iostat 2 10 >>/tmp/perfor/hp1.txt
#swapinfo -tam >>/tmp/perfor/hp1.txt
#top -d2 -s10 -f /tmp/toperf.txt
#cat /tmp/toperf.txt
#ipcs â mob >>/tmp/perfor/hp1.txt
#exit

Re: HP-UX Performance Issue

Vinod_22 — Thu, 24 Jun 2010 14:02:42 GMT

top for rx4640

Re: HP-UX Performance Issue

Vinod_22 — Thu, 24 Jun 2010 14:03:56 GMT

top for rx2660

Re: HP-UX Performance Issue

Vinod_22 — Thu, 24 Jun 2010 14:05:41 GMT

rx2660 complete logs

These both Servers are connected to DB host thru L2 N/W switch.

Re: HP-UX Performance Issue

Modris Bremze — Thu, 24 Jun 2010 14:45:09 GMT

The lines

Synchronous Page I/O error occurred while paging to/from NFS server qiublrboanfs001
file system is /home

Look suspicious in rx2660 logs. How much are /home and NFS involved?

Re: HP-UX Performance Issue

rick jones — Thu, 24 Jun 2010 23:56:28 GMT

The stats you've collected seem to cover everything except the network. If we can assume that when these Cognos scripts are running they are the only things happening on the systems, you might:

netstat -s > before.netstat
lanadmin -g mibstats > before.lanadmin
- run the cognos script -
netstat -s > after.netstat
lanadmin -g mibstats > after.lanadmin

where you replace with the ppa of the lan in use - from your description I would guess '0' from lan0 but that is only a guess.

You can run the before and after files through "beforeafter" from ftp://ftp.cup.hp.com/dist/networking/tools

beforeafter before after > delta

and that will subtract all the numbers it finds in before from their counterparts in after - giving you the change over the interval.

Re: HP-UX Performance Issue

Vinod_22 — Fri, 25 Jun 2010 10:45:36 GMT

The path information for the installation is as follows:
Â Â
1. Cognos AW Runtime (Scriupts are being run from here) : /apps/sterling/sbi/cognos/ap/bin (nfs mount is /apps/sterling & NFS host:qiublrboanfs001)
2. Oracle Client : /home/oracle11g_client/oracle_homeÂ Â Â ( nfs mount is /home & NFS host:qiublrboanfs001 )

Here both the file systems are exported by single NFS Server qiublrboanfs001 and it is the same env for rx2600 and rx4640.

In below trace Between two Red marked line it has take near 7mins , where it has to happened in less then second.

Trace During Application
---------------------------------------------------------------------------------------------------

jobstream -- start run on qiublrboaapp011.blrqa-nis (25-Jun-2010 15:56:34)
[VARIABLEÂ Â - 15:56:36] initialLoadStartDate = ''
[VARIABLEÂ Â - 15:56:36] initialLoadEndDate = ''
[VARIABLEÂ Â - 15:56:36] DS_LOG_DIR = '/apps/sterling/sbi/cognos/ap/datamanager/log/20100625155634'
[VARIABLEÂ Â - 15:56:36] DS_UDA_FETCH_ROWS = '50'
[VARIABLEÂ Â - 15:56:36] DS_UDA_BULKWRITE_ROWS = '50'
[VARIABLEÂ Â - 15:56:36] DS_LOB_MAXMEM_SIZE = '65536'
[VARIABLEÂ Â - 15:56:36] DS_LOB_BUF_SIZE = '8000'
[VARIABLEÂ Â - 15:56:36] DS_DBMS_TRIM = 'SPACE'
[VARIABLEÂ Â - 15:56:36] DS_RUN_TIMESTAMP = 2010-06-25 15:56:35
[VARIABLEÂ Â - 15:56:36] DS_JOBSTREAM_NAME = 'Configuration'
[VARIABLEÂ Â - 15:56:36] DS_JOBSTREAM_FNAME = 'Configuration'
[VARIABLEÂ Â - 15:56:36] DS_AUDIT_ID = 1
[VARIABLEÂ Â - 15:56:36] DS_JOB_AUDIT_ID = 1
[VARIABLEÂ Â - 15:56:36] DS_RUN_ID = 1
[VARIABLEÂ Â - 15:56:36] DS_LOG_NAME = '/apps/sterling/sbi/cognos/ap/datamanager/log/20100625155634/Job_Configuration_0001.log'
[VARIABLEÂ Â - 15:56:36] DS_MAX_RECURSION = '100'
[VARIABLEÂ Â - 15:56:36] . TRACE_VALUES = 'PROGRESS,DETAIL,INTERNAL,SQL,EXECUTEDSQL,USER,VARIABLE'
[VARIABLEÂ Â - 15:56:36] . AUDIT_VALUES = 'TIMING,ALERT,USER'
[PROGRESSÂ Â - 15:56:36] JobStream 'Configuration'; starting
[INTERNALÂ Â - 15:56:36] Start Node 1 'Start'; Idle -> Succeeded
[INTERNALÂ Â - 15:56:36] Build Node 2 'YFS_CAT_DOMAIN_LOCALE'; Pending -> Executing
[INTERNALÂ Â - 15:56:36] Build Node 3 'YFS_REGION_SCHEMA'; Pending -> Executing
[INTERNALÂ Â - 15:56:36] Build Node 4 'YFS_REGION'; Pending -> Executing
[INTERNALÂ Â - 15:56:36] Build Node 5 'YFS_STATUS_MODIFICATION_TYPE'; Pending -> Executing
[INTERNALÂ Â - 15:56:36] Build Node 6 'YFS_COMMON_CODE'; Pending -> Executing
[INTERNALÂ Â - 15:56:36] Build Node 7 'YFS_REGION_USAGE'; Pending -> Executing
[INTERNALÂ Â - 15:56:36] Build Node 8 'YFS_DATA_SECURITY_GROUP'; Pending -> Executing
[INTERNALÂ Â - 15:56:36] Build Node 9 'YFS_REGION_BEST_MATCH'; Pending -> Executing
[INTERNALÂ Â - 15:56:36] Build Node 10 'YFS_ORGANIZATION'; Pending -> Executing
[INTERNALÂ Â - 15:56:36] Build Node 11 'YFS_CUSTOMER_GRADE'; Pending -> Executing
[INTERNALÂ Â - 15:56:36] Build Node 12 'YFS_CHARGE_CATEGORY'; Pending -> Executing
[INTERNALÂ Â - 15:56:36] Build Node 13 'YFS_PAYMENT_TYPE'; Pending -> Executing
[INTERNALÂ Â - 15:56:36] Build Node 14 'YFS_HOLD_TYPE'; Pending -> Executing
[INTERNALÂ Â - 15:56:36] Build Node 15 'YFS_DOCUMENT_PARAMS'; Pending -> Executing
[INTERNALÂ Â - 15:56:36] Build Node 16 'YFS_LOCALIZED_STRINGS'; Pending -> Executing
[INTERNALÂ Â - 15:56:36] Build Node 17 'YFS_CATEGORY_DOMAIN'; Pending -> Executing
[INTERNALÂ Â - 15:56:37] Build Node 15 'YFS_DOCUMENT_PARAMS'; initializing
[PROGRESSÂ Â - 15:56:38] Build Node 15 'YFS_DOCUMENT_PARAMS'; executing (pid 3689)
[DETAILÂ Â Â Â - 15:56:38] Build Node 15 'YFS_DOCUMENT_PARAMS'; component 'YFS_DOCUMENT_PARAMS', RunId 1, AuditId 2
[INTERNALÂ Â - 15:56:39] Build Node 15 'YFS_DOCUMENT_PARAMS'; Executing -> Succeeded
[PROGRESSÂ Â - 15:56:39] Build Node 15 'YFS_DOCUMENT_PARAMS'; succeeded
[VARIABLEÂ Â - 15:56:39] RESULT = TRUE
[VARIABLEÂ Â - 15:56:39] DM_COMPONENT_AUDIT_ID = 2
[INTERNALÂ Â - 16:03:38] Build Node 4 'YFS_REGION'; initializing
[INTERNALÂ Â - 16:03:38] Build Node 13 'YFS_PAYMENT_TYPE'; initializing
[PROGRESSÂ Â - 16:03:38] Build Node 4 'YFS_REGION'; executing (pid 3678)
[PROGRESSÂ Â - 16:03:38] Build Node 13 'YFS_PAYMENT_TYPE'; executing (pid 3687)
[DETAILÂ Â Â Â - 16:03:39] Build Node 4 'YFS_REGION'; component 'YFS_REGION', RunId 1, AuditId 3
[DETAILÂ Â Â Â - 16:03:39] Build Node 13 'YFS_PAYMENT_TYPE'; component 'YFS_PAYMENT_TYPE', RunId 1, AuditId 4
[INTERNALÂ Â - 16:03:39] Build Node 4 'YFS_REGION'; Executing -> Succeeded
[PROGRESSÂ Â - 16:03:39] Build Node 4 'YFS_REGION'; succeeded
[VARIABLEÂ Â - 16:03:39] RESULT = TRUE
[VARIABLEÂ Â - 16:03:39] DM_COMPONENT_AUDIT_ID = 3
[INTERNALÂ Â - 16:03:39] Build Node 13 'YFS_PAYMENT_TYPE'; Executing -> Succeeded
[PROGRESSÂ Â - 16:03:39] Build Node 13 'YFS_PAYMENT_TYPE'; succeeded
[VARIABLEÂ Â - 16:03:39] RESULT = TRUE
[VARIABLEÂ Â - 16:03:39] DM_COMPONENT_AUDIT_ID = 4

Re: HP-UX Performance Issue

Vinod_22 — Fri, 25 Jun 2010 10:54:56 GMT

Hi Rick and here are the all netstat and lanadmin logs.

beofre= before running cognos script
after1= while running cognos script sample 1
after2= while running cognos script sample 2
after3= while running cognos script sample 3

Re: HP-UX Performance Issue

Vinod_22 — Fri, 25 Jun 2010 10:58:34 GMT

Hi Rick and Bremze,

Please see some more lanshow output from slow porf Server i.e. rx2660.

Re: HP-UX Performance Issue

Vinod_22 — Fri, 25 Jun 2010 10:59:55 GMT

Hello Bremze,

I have exlpained the NFS setup in the attachment also some details about cognos script

Re: HP-UX Performance Issue

rick jones — Fri, 25 Jun 2010 17:13:17 GMT

Well, a cursory check of the network stats looks clean.

Re: HP-UX Performance Issue

chris huys_4 — Fri, 25 Jun 2010 18:58:24 GMT

Hi,

Longshot, but, the problem systems vmstat output shows 1 blocked process.

procs memory page faults cpu
r b w
1 1 0
1 1 0
1 1 0
1 1 0
1 1 0

If this blocked process "starts when"/"disappears after" the cognos job is done, most probably this process has something to do with why the job taking so long.

So then find the process and see where its blocked up on.

glance will give the wait reason for the process, the hp support crashinfo utility can give you, with crashinfo -c /stand/vmunix /dev/kmem, the wait reason/process stack of the process

And did you log a support call with cognos, ibm?, support ?

Greetz,
Chris

Re: HP-UX Performance Issue

Basheer_2 — Sat, 26 Jun 2010 06:44:27 GMT

Hello Vinod,

you are comparing an ENTRY level server with a Medium Level Server.

Perf will NOT be same on both the servers.

Re: HP-UX Performance Issue

Dave Olker — Sat, 26 Jun 2010 21:42:24 GMT

Since you have debug logging available at the application layer and you can see things like an unexplained 7-minute delay at certain points of the run, my suggestion would be to use tusc to collect system-call level logging of both the fast and slow runs. Comparing the two tusc outputs should show you which system calls are stalling for 7 minutes when they should be instantaneous.

Once we know what the application is doing during that 7 minute period we can better decide where to look at the slower system's configuration.

Regards,

Dave

Re: HP-UX Performance Issue

Shiva Bhaskar — Mon, 28 Jun 2010 10:18:07 GMT

hi Chris Huys ,

Pls find the attached crashinfo out put ,

where we have collected the info 5 times while cognos scripting.

Re: HP-UX Performance Issue

chris huys_4 — Mon, 28 Jun 2010 20:27:45 GMT

Hi,

My ie has problems with saving/opening the rar attachment. A text file as attachment should be better..

Greetz,
Chris