Performance problem after upgrade from 11.23 to 11.31

[Marijan Matic]

On an old machine  - model "9000/800/rp4440  " (4 PA-RISC 8800 processors (1000 MHz, 64 MB)) with 16GB we have upgraded HPUX from 11.23 to 11.31. Now, from time to time we have system with very high response time. Basically, all normal operations (login, directory listing, ... ) take very, very long time, even though the CPU usage during that period is very low. Also, memory system is occuped only 50%, pagging is not used at all, I/O  seems not to be overloaded too. Sometimes this situation is solved after some period of time without intervention, but during working hour machine has to be restarted, because we can't wait too much such situation during the day.

We used many performance monitor tools, except glance. As I already mentioned, we did not detect the cause of this situation and one thing we  haven't checked is the network system. Also we noticed that thi situation happend more often after NwtWorker backup utility is in use.

One of measureable issues we have detecetd is slow execution of lstat64 system call. We used TUSC to perform a 'll' command on a directory with 15k files.

For example:

06:31:46 [ls -l /test ]{2035012} (0.000054) lstat64("/test/FILE_00249395_20130301063957.Z", 0x77ff0468) = 0
                            st_dev: 64 0x000003
                            st_ino: 42172
                           st_mode: S_IFREG|0640
                          st_nlink: 1
                           st_rdev: 0 0x000000
                           st_size: 0
                        st_blksize: 8192
                         st_blocks: 0
                            st_uid: 0
                            st_gid: 3
                           st_aclv: 0
                            st_acl: 0
                         st_fstype: 10
                          st_atime: Thu Feb 26 15:29:17 2015
                          st_mtime: Thu Feb 26 15:29:17 2015
                          st_ctime: Thu Feb 26 15:29:17 2015
06:32:29 [ls -l /test ]{2035012} lstat64(0x40001888, 0x77ff0468) [running]
06:32:29 [ls -l /test ]{2035012} (42.357834) lstat64("/test/FILE_00249396_20130301064257.Z", 0x77ff0468) = 0
                            st_dev: 64 0x000003
                            st_ino: 42173
                           st_mode: S_IFREG|0640
                          st_nlink: 1
                           st_rdev: 0 0x000000
                           st_size: 0
                        st_blksize: 8192
                         st_blocks: 0
                            st_uid: 0
                            st_gid: 3
                           st_aclv: 0
                            st_acl: 0
                         st_fstype: 10
                          st_atime: Thu Feb 26 15:29:17 2015
                          st_mtime: Thu Feb 26 15:29:17 2015
                          st_ctime: Thu Feb 26 15:29:17 2015
06:32:29 [ls -l /test ]{2035012} (0.000063) lstat64("/test/FILE_00249397_20130301064557.Z", 0x77ff0468) = 0

As you can notice, there is a 42seconds of delay during the execution lstat64 system call. This is observed on a local drive (Veritas 4.1 is installed). The whole directory listing 'll' sometimes takes 15min, sometimes even 20min. When system is fresh reboted it takes few seconds.

Inode cache is used below the max value. Basically all kernel tunables are in use bellow the maximum values, except the disk cache. Still, the hit ratio is high, so there is no need to increase the cache. Since system is hosts Oracle 9i DB, veritas is configured as Oracle recommends (no mutiple read/write threads, deafult read ahead params, ...)

Just to note. System did not change in hardware or softvare after the upgrade. All issues staretd after the upgrade. I don't know if someone had similar issues after the upgrade to 11.31. If you can get a hint what should be checked I would appreaciate very much.

[sapoguheman]

Better to approach HP if you have any SW/HW contract :)

[chindi]

Hi ,

Check the filesystem version after upgrade .

Also can you create a new filesystem say /test and put those 15k files in it and observe the timings and let us know .

[Marijan Matic]

We still have this problem.

Well, we tested the nfs mount point with the same content, during the period when system slows down, and nfs works fine. Only vxfs partitions, local and remote, starts to have troubles after some period of time.