HP-UX Performance Issue

[Vinod_22]

The path information for the installation is as follows:
  
1. Cognos AW Runtime (Scriupts are being run from here) : /apps/sterling/sbi/cognos/ap/bin (nfs mount is /apps/sterling & NFS host:qiublrboanfs001)
2. Oracle Client : /home/oracle11g_client/oracle_home    ( nfs mount is /home & NFS host:qiublrboanfs001 )

Here both the file systems are exported by single NFS Server qiublrboanfs001 and it is the same env for rx2600 and rx4640.


In below trace Between two Red marked line it has take near 7mins , where it has to happened in less then second.

Trace During Application
---------------------------------------------------------------------------------------------------

jobstream -- start run on qiublrboaapp011.blrqa-nis (25-Jun-2010 15:56:34)
[VARIABLE   - 15:56:36] initialLoadStartDate = ''
[VARIABLE   - 15:56:36] initialLoadEndDate = ''
[VARIABLE   - 15:56:36] DS_LOG_DIR = '/apps/sterling/sbi/cognos/ap/datamanager/log/20100625155634'
[VARIABLE   - 15:56:36] DS_UDA_FETCH_ROWS = '50'
[VARIABLE   - 15:56:36] DS_UDA_BULKWRITE_ROWS = '50'
[VARIABLE   - 15:56:36] DS_LOB_MAXMEM_SIZE = '65536'
[VARIABLE   - 15:56:36] DS_LOB_BUF_SIZE = '8000'
[VARIABLE   - 15:56:36] DS_DBMS_TRIM = 'SPACE'
[VARIABLE   - 15:56:36] DS_RUN_TIMESTAMP = 2010-06-25 15:56:35
[VARIABLE   - 15:56:36] DS_JOBSTREAM_NAME = 'Configuration'
[VARIABLE   - 15:56:36] DS_JOBSTREAM_FNAME = 'Configuration'
[VARIABLE   - 15:56:36] DS_AUDIT_ID = 1
[VARIABLE   - 15:56:36] DS_JOB_AUDIT_ID = 1
[VARIABLE   - 15:56:36] DS_RUN_ID = 1
[VARIABLE   - 15:56:36] DS_LOG_NAME = '/apps/sterling/sbi/cognos/ap/datamanager/log/20100625155634/Job_Configuration_0001.log'
[VARIABLE   - 15:56:36] DS_MAX_RECURSION = '100'
[VARIABLE   - 15:56:36] . TRACE_VALUES = 'PROGRESS,DETAIL,INTERNAL,SQL,EXECUTEDSQL,USER,VARIABLE'
[VARIABLE   - 15:56:36] . AUDIT_VALUES = 'TIMING,ALERT,USER'
[PROGRESS   - 15:56:36] JobStream 'Configuration'; starting
[INTERNAL   - 15:56:36] Start Node 1 'Start'; Idle -> Succeeded
[INTERNAL   - 15:56:36] Build Node 2 'YFS_CAT_DOMAIN_LOCALE'; Pending -> Executing
[INTERNAL   - 15:56:36] Build Node 3 'YFS_REGION_SCHEMA'; Pending -> Executing
[INTERNAL   - 15:56:36] Build Node 4 'YFS_REGION'; Pending -> Executing
[INTERNAL   - 15:56:36] Build Node 5 'YFS_STATUS_MODIFICATION_TYPE'; Pending -> Executing
[INTERNAL   - 15:56:36] Build Node 6 'YFS_COMMON_CODE'; Pending -> Executing
[INTERNAL   - 15:56:36] Build Node 7 'YFS_REGION_USAGE'; Pending -> Executing
[INTERNAL   - 15:56:36] Build Node 8 'YFS_DATA_SECURITY_GROUP'; Pending -> Executing
[INTERNAL   - 15:56:36] Build Node 9 'YFS_REGION_BEST_MATCH'; Pending -> Executing
[INTERNAL   - 15:56:36] Build Node 10 'YFS_ORGANIZATION'; Pending -> Executing
[INTERNAL   - 15:56:36] Build Node 11 'YFS_CUSTOMER_GRADE'; Pending -> Executing
[INTERNAL   - 15:56:36] Build Node 12 'YFS_CHARGE_CATEGORY'; Pending -> Executing
[INTERNAL   - 15:56:36] Build Node 13 'YFS_PAYMENT_TYPE'; Pending -> Executing
[INTERNAL   - 15:56:36] Build Node 14 'YFS_HOLD_TYPE'; Pending -> Executing
[INTERNAL   - 15:56:36] Build Node 15 'YFS_DOCUMENT_PARAMS'; Pending -> Executing
[INTERNAL   - 15:56:36] Build Node 16 'YFS_LOCALIZED_STRINGS'; Pending -> Executing
[INTERNAL   - 15:56:36] Build Node 17 'YFS_CATEGORY_DOMAIN'; Pending -> Executing
[INTERNAL   - 15:56:37] Build Node 15 'YFS_DOCUMENT_PARAMS'; initializing
[PROGRESS   - 15:56:38] Build Node 15 'YFS_DOCUMENT_PARAMS'; executing (pid 3689)
[DETAIL     - 15:56:38] Build Node 15 'YFS_DOCUMENT_PARAMS'; component 'YFS_DOCUMENT_PARAMS', RunId 1, AuditId 2
[INTERNAL   - 15:56:39] Build Node 15 'YFS_DOCUMENT_PARAMS'; Executing -> Succeeded
[PROGRESS   - 15:56:39] Build Node 15 'YFS_DOCUMENT_PARAMS'; succeeded
[VARIABLE   - 15:56:39] RESULT = TRUE
[VARIABLE   - 15:56:39] DM_COMPONENT_AUDIT_ID = 2
[INTERNAL   - 16:03:38] Build Node 4 'YFS_REGION'; initializing
[INTERNAL   - 16:03:38] Build Node 13 'YFS_PAYMENT_TYPE'; initializing
[PROGRESS   - 16:03:38] Build Node 4 'YFS_REGION'; executing (pid 3678)
[PROGRESS   - 16:03:38] Build Node 13 'YFS_PAYMENT_TYPE'; executing (pid 3687)
[DETAIL     - 16:03:39] Build Node 4 'YFS_REGION'; component 'YFS_REGION', RunId 1, AuditId 3
[DETAIL     - 16:03:39] Build Node 13 'YFS_PAYMENT_TYPE'; component 'YFS_PAYMENT_TYPE', RunId 1, AuditId 4
[INTERNAL   - 16:03:39] Build Node 4 'YFS_REGION'; Executing -> Succeeded
[PROGRESS   - 16:03:39] Build Node 4 'YFS_REGION'; succeeded
[VARIABLE   - 16:03:39] RESULT = TRUE
[VARIABLE   - 16:03:39] DM_COMPONENT_AUDIT_ID = 3
[INTERNAL   - 16:03:39] Build Node 13 'YFS_PAYMENT_TYPE'; Executing -> Succeeded
[PROGRESS   - 16:03:39] Build Node 13 'YFS_PAYMENT_TYPE'; succeeded
[VARIABLE   - 16:03:39] RESULT = TRUE
[VARIABLE   - 16:03:39] DM_COMPONENT_AUDIT_ID = 4

[Vinod_22]

Hi Rick and here are the all netstat and lanadmin logs.

beofre= before running cognos script
after1= while running cognos script sample 1
after2= while running cognos script sample 2
after3= while running cognos script sample 3

[Vinod_22]

Hi Rick and Bremze,


Please see some more lanshow output from slow porf Server i.e. rx2660.

[Vinod_22]

Hello Bremze,

I have exlpained the NFS setup in the attachment also some details about cognos script

[rick jones]

Well, a cursory check of the network stats looks clean.

[chris huys_4]

Hi,

Longshot, but, the problem systems vmstat output shows 1 blocked process.

procs memory page faults cpu
r b w
1 1 0
1 1 0
1 1 0
1 1 0
1 1 0

If this blocked process "starts when"/"disappears after" the cognos job is done, most probably this process has something to do with why the job taking so long.

So then find the process and see where its blocked up on.

glance will give the wait reason for the process, the hp support crashinfo utility can give you, with crashinfo -c /stand/vmunix /dev/kmem, the wait reason/process stack of the process

And did you log a support call with cognos, ibm?, support ?

Greetz,
Chris

[Basheer_2]

Hello Vinod,

you are comparing an ENTRY level server with a Medium Level Server.

Perf will NOT be same on both the servers.

[Dave Olker]

Since you have debug logging available at the application layer and you can see things like an unexplained 7-minute delay at certain points of the run, my suggestion would be to use tusc to collect system-call level logging of both the fast and slow runs. Comparing the two tusc outputs should show you which system calls are stalling for 7 minutes when they should be instantaneous.

Once we know what the application is doing during that 7 minute period we can better decide where to look at the slower system's configuration.

Regards,

Dave

[Shiva Bhaskar]

hi Chris Huys ,

Pls find the attached crashinfo out put ,

where we have collected the info 5 times while cognos scripting.

[chris huys_4]

Hi,

My ie has problems with saving/opening the rar attachment. A text file as attachment should be better..

Greetz,
Chris