Performance problem with Informix IDS 9.40 and HPUX 11i

[Kevin Keller_2]

After installing patches PHKL_33258, PHKL_30992 I've been having some major Informix performance problem. I pulled the patches and the performance problem stayed. What's causing the problem and how can I fix it? Thank you.

[A. Clay Stephenson]

Since you removed the patches and the problems remain, I very much doubt the patches had anything to do with your performance problems. You need to ask what else changed at about the same time. Was any SQL changed or added? Have data volumes increased? You simply don't provide enough data to even begin the process. Very conservatively 95% of database performance problems lie within the database. You need to get your DBA's involved in locating the
bottlenecks. You should be doing the the same with perfomance tools like Glance to find out if your are disk bound, memory bound, or CPU bound. Are you seeing significant page-out rates because the DBA's grabbed too much shared memory? Have you "improved" the timeslice value recently? These are things you should be telling me rather than me having to ask.

[Kevin Keller_2]

Absolutely nothing has been changed other than the installation/removal of the two patches. This is the first time the kernel has been touched in a year. The Informix settings haven't been changed in two years. I am also noticing checkpoint reporting 20-40 seconds, it used to be 0-1 seconds. I did run glance and noticed the CPUs went from 30-40% (originally) to 70-90%. The system is an N4000, with 4 CPUs @500MHZ, 4GB RAM, the OS is HP11i. The raw partitions are running on a VA7100 disk array with two fibre channel pathways. The fibre channel drivers are dated Dec 2003.

[Steve Lewis]

Were there any changed parameters in the /stand/system file that had not been included in the previous kernel. I know its old, but it coul dbe that you forgot that you had entered something in there but didn't rebuild the kernel until these patches were installed.

If you really want to undo the patch install, then restore from your make_recovery tape. You did take one didn't you? If not, then try restoring your previous kernel to a new name, such as vmunix.kevin then reboot that kernel and try again.

The first patch is for vxfs. Because you said that your informix storage is raw LUNs, not on cooked files, then there won't be much i/o going through that patch code.
The second patch is a more likely candidate, but not for checkpoint times.

Check your fibrechannel stats using fcmsstat. Then check your sar disk stats during checkpoints. Is it a hot lun? Whats your scsi_max_queue_depth now and before? Have your luns changed controller paths now?

There are many possibilities still.

[Peter Godron]

Kevin,
as both patches require reboot to activate, did you reboot after uninstall?
Also, why did you want to install the patches?

[Kevin Keller_2]

I did restore from the ignite made before the patches and driver. It didn't help. The patches for an Ultra Density Optical drive. When the infomix problem developed I pulled the patches, drivers and the drive. The Informix problem still exists. Glance shows the disks on the VA7100 are not being hit that extensively. The cpus get occassional spikes, but that's normal. Swap space is never used. RAM is 70% used. Resmon's event log are empty. Anything not Informix related is running fine. It sounds like I woke up a problem within Informix.

[Kevin Keller_2]

I did restore from the ignite made before the patches and driver. It didn't help. The patches are for an Ultra Density Optical drive. When the infomix problem developed I pulled the patches, drivers and the drive. The Informix problem still exists. Glance shows the disks on the VA7100 are not being hit that extensively. The cpus get occassional spikes, but that's normal. Swap space is never used. RAM is 70% used. Resmon's event log are empty. Anything not Informix related is running fine. It sounds like I woke up a problem within Informix.