HP-UX Performance Issue

[Vinod_22]

Dear all,
I will brief here very strange and peculiar performance problem with two HP-Itanuim Servers
The Server details are
Rx2660 (Good Configuration but very bad performance)
CPU: Two dual core (1.42 GHz, 12MB 9120N) processor
Memory: 16GB
Disk: 146GB 10k SAS
OS version HP-UX 11.31 release Sep2009
N/W: 1gbps
Script runs time: 68 min


Rx4640 (Not that great Configuration as 2660 but very impressive and faster performance)
CPU: HP mx2 dual processor
Memory 8GB
Disk: 73GB
OS version HP-UX 11.31 release Sep2009
N/W: 1gbps
Script runs time: 20 Sec

When we run congnos script the script will get complete in 15sec on rx4640 vs. it takes more than hour (68min) on Rx2660.
When the congnos script runs it call DB which is running on other Server and choose the fields and performs insert to another DB. In this case both Servers ( 2660 & 4640) are connecting to same DB Server. Also the DB client is also same on both the Servers.

If I compare both Server CPU, memory, disk and N/W the Slower system (2660) has all good horsepower compare to rx4640 but, when performance comes it acts like 10year old box.

We have tried to bring both OS same version and all kernel parameters same but not able to understands why the performance differ with such huge unbelievable difference.

Really appreciate if we get faster response on this.

[Modris Bremze]

Maybe it's not the servers themselves, but something in between them - how are both servers connected to that "other DB server" they both rely on to run the script? Have you looked at both server CPU utilization and system load while your script is running?

[S. Ney]

Have you looked at glance on the 4640?
sar -L
iostat -L -t
Do you have tusc installed on the 4640?

[study unix]

try to use top or glance ,but glance needs to be paid

[Vinod_22]

Please find below o/p for the both Servers
rx4640 (Great Perf)
rx2660 (Bad Perf)
#uname -a >>/tmp/perfor/hp1.txt
#bdf >>/tmp/perfor/hp1.txt
#dmesg >>/tmp/perfor/hp1.txt
#uptime >>/tmp/perfor/hp1.txt
#model >>/tmp/perfor/hp1.txt
#sar -u 2 10 >>/tmp/perfor/hp1.txt
#sar -d 2 10 >>/tmp/perfor/hp1.txt
#sar -b 2 10 >>/tmp/perfor/hp1.txt
#sar -v 2 10 >>/tmp/perfor/hp1.txt
#vmstat 2 10 >>/tmp/perfor/hp1.txt
#iostat 2 10 >>/tmp/perfor/hp1.txt
#swapinfo -tam >>/tmp/perfor/hp1.txt
#top -d2 -s10 -f /tmp/toperf.txt
#cat /tmp/toperf.txt
#ipcs â mob >>/tmp/perfor/hp1.txt
#exit

[Vinod_22]

top for rx4640

[Vinod_22]

top for rx2660

[Vinod_22]

rx2660 complete logs

These both Servers are connected to DB host thru L2 N/W switch.

[Modris Bremze]

The lines

Synchronous Page I/O error occurred while paging to/from NFS server qiublrboanfs001
file system is /home

Look suspicious in rx2660 logs. How much are /home and NFS involved?

[rick jones]

The stats you've collected seem to cover everything except the network. If we can assume that when these Cognos scripts are running they are the only things happening on the systems, you might:

netstat -s > before.netstat
lanadmin -g mibstats > before.lanadmin
- run the cognos script -
netstat -s > after.netstat
lanadmin -g mibstats > after.lanadmin

where you replace with the ppa of the lan in use - from your description I would guess '0' from lan0 but that is only a guess.

You can run the before and after files through "beforeafter" from ftp://ftp.cup.hp.com/dist/networking/tools

beforeafter before after > delta

and that will subtract all the numbers it finds in before from their counterparts in after - giving you the change over the interval.