fuser uses too much CPU

[Fedon Kadifeli]

This is an rp7440 (HP-UX 11.31) system with 8 cores. It is not busy at the moment. However when I run fuser to find processes using files on a specific mount point, the fuser command takes too much CPU time to complete:

# model
9000/800/rp7440

# bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 4096000 1376656 2698296 34% /
/dev/vg00/lvol1 1998072 159360 1638904 9% /stand
/dev/vg00/lvol8 10256384 3965016 6242592 39% /var
/dev/vg00/lvola 20480000 22166 19179291 0% /var/adm/crash
/dev/vg00/lvol7 10256384 1958504 8233096 19% /usr
/dev/vg00/lvol6 1048576 53784 992960 5% /tmp
/dev/vg00/lvol5 10256384 4237384 5972072 42% /opt
/dev/vg00/lvol4 524288 56032 465016 11% /home

# swapinfo -mt
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 16000 411 15589 3% 0 - 1 /dev/vg00/lvol2
dev 16000 413 15587 3% 0 - 1 /dev/vg00/lvol9
reserve - 1238 -1238
memory 31142 8354 22788 27%
total 63142 10416 52726 16% - 0 -

# ps -ef | wc -l
208

# w
9:31am up 139 days, 17:40, 1 user, load average: 0.16, 0.14, 0.09
User tty login@ idle JCPU PCPU what
fedkad pts/0 9:23am w

# time fuser -cu /home
/home: 29334co(fedkad) 22801co(fedkad) 26421co(fedkad)


real 2:29.4
user 0.0
sys 2:28.9

[Fedon Kadifeli]

Please note: This problem is not specific to a certain mount point or file. Other examples:

# time fuser -cu /stand
/stand:

real 2:29.8
user 0.1
sys 2:29.1

# time fuser -u /var/adm/syslog/syslog.log
/var/adm/syslog/syslog.log: 1806o(root)


real 2:25.9
user 0.0
sys 2:25.2

[Johnson Punniyalingam]

can you check from Glance ? what's your CPU usage ?


# sar -Mu 5 50
# sar -Mq 5 50
-- What is the CPU load?

[Kapil Jha]

can you check if PHCO_31613 is installed.

Is this problem in only one box or all are suffering from the same fever :)

BR,
Kapil+

[Fedon Kadifeli]

Kapil,

This patch is for 11.23; and as far as I see there is not an equivalent patch for 11.31.

Our system is 11.31.

Thanks.

[Fedon Kadifeli]

CPU usage (kernel mode) is high as I specified in my original post.

# time fuser -u /var/adm/syslog/syslog.log
/var/adm/syslog/syslog.log: 1806o(root)


real 2:24.2
user 0.1
sys 2:24.0

----

# sar -Mu 5 50

HP-UX dbrsrv B.11.31 U 9000/800 03/24/10

10:15:43 cpu %usr %sys %wio %idle
10:15:48 0 0 2 77 21
1 0 4 0 96
2 0 1 99 0
3 0 1 99 0
4 1 1 0 98
5 0 76 22 2
6 0 0 0 99
7 1 24 0 74
system 1 14 37 49
10:15:53 0 0 0 0 100
1 1 4 0 95
2 0 0 100 0
3 1 1 98 0
4 0 1 0 99
5 0 1 98 0
6 0 53 0 47
7 0 47 0 53
system 0 13 37 49
10:15:58 0 5 3 15 76
1 1 4 10 84
2 1 2 96 0
3 1 2 80 17
4 1 2 0 96
5 1 1 89 9
6 0 100 0 0
7 0 0 0 99
system 1 15 36 48
[...]

----

# sar -Mq 5 50

HP-UX dbrsrv B.11.31 U 9000/800 03/24/10

10:15:42 cpu runq-sz %runocc swpq-sz %swpocc
10:15:47 0 0.0 0
1 0.0 0
2 0.0 0
3 0.0 0
4 0.0 0
5 1.0 40
6 0.0 0
7 0.0 0
system 1.0 5 0.0 0
10:15:52 0 0.0 0
1 0.0 0
2 0.0 0
3 0.0 0
4 0.0 0
5 0.0 0
6 0.0 0
7 0.0 0
system 0.0 0 0.0 0
10:15:57 0 0.0 0
1 0.0 0
2 0.0 0
3 0.0 0
4 0.0 0
5 0.0 0
6 0.0 0
7 0.0 0
system 0.0 0 0.0 0
10:16:02 0 0.0 0
1 0.0 0
2 0.0 0
3 0.0 0
4 0.0 0
5 1.0 20
6 1.0 20
7 0.0 0
system 1.0 5 0.0 0
[...]

[Fedon Kadifeli]

We have another box with almost (99.9%) the same configuration. It had the same problem, until I rebooted it yesterday. After the reboot, there is no problem currently on this box:

# uptime
10:24am up 23:52, 23 users, load average: 0.58, 0.68, 0.70

# time fuser -u /var/adm/syslog/syslog.log
/var/adm/syslog/syslog.log: 1350o(root) 3044o(root)


real 0.5
user 0.0
sys 0.3

[Johnson Punniyalingam]

looks like to me you have "CPU" Bottle neck

can we check "which process" using most of your cpu ..?

# top

# vmstat 5 5

if its related to application process,causing the "High CPU usage" will check with application team,

This may not be a (System)problem. It could be a lot of sleeping processes caused by bad application design.


or as you said you had similar problem, in another server .after reboot the server is OK,

on top of that also check

follow , below link collect info (performance report and post)

http://www.hpux.ws/system.perf.sh


any hardware I/O problem. Could be disk or cable or controller.
System patch level are up to date ?
Any errors in dmesg ?
/var/adm/syslog/syslog.log ?
/var/opt/resmon/log/event.log ?

[Fedon Kadifeli]

There is no CPU, memory, disk, or network bottleneck on this system, since I moved all the applications to the other system. The system is practically idle. I checked this also with Glance.

Please note that this system is up for 139 days. And please note that we had the same problem on the other system before rebooting it.

I repeat with another example:

# uptime
11:08am up 139 days, 19:17, 3 users, load average: 0.03, 0.03, 0.03

# ll -R /var/adm/crash
total 2
-rwxr-xr-x 1 root root 1 Aug 6 2008 bounds
drwxr-xr-x 2 root root 96 Aug 7 2008 lost+found

/var/adm/crash/lost+found:
total 0

# time fuser -u /var/adm/crash
/var/adm/crash:

real 2:30.5
user 0.1
sys 2:29.8

---

System: dbrsrv Wed Mar 24 11:10:51 2010
Load averages: 0.24, 0.11, 0.06
225 processes: 152 sleeping, 73 running
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.05 0.8% 0.0% 0.4% 98.8% 0.0% 0.0% 0.0% 0.0%
1 1.70 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0% 0.0%
2 0.02 0.0% 0.0% 1.2% 98.8% 0.0% 0.0% 0.0% 0.0%
3 0.02 0.2% 0.0% 1.2% 98.6% 0.0% 0.0% 0.0% 0.0%
4 0.02 1.8% 0.0% 1.0% 97.2% 0.0% 0.0% 0.0% 0.0%
5 0.07 0.0% 0.0% 4.0% 96.0% 0.0% 0.0% 0.0% 0.0%
6 0.02 0.2% 0.0% 1.0% 98.8% 0.0% 0.0% 0.0% 0.0%
7 0.02 0.2% 0.0% 1.0% 98.8% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.24 0.4% 0.0% 13.7% 85.9% 0.0% 0.0% 0.0% 0.0%

System Page Size: 4Kbytes
Memory: 832380K (649992K) real, 2332472K (1759412K) virtual, 22007832K free Page# 1/13

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
1 pts/2 25011 root 240 20 1964K 240K run 2:04 100.85 100.67 fuser
5 ? 3 root 152 20 36K 36K sleep 7805:11 3.70 3.69 statdaemon
0 ? 23169 root 148 20 7876K 1560K sleep 2:34 2.20 2.20 mma
6 ? 23056 root 148 20 7876K 1560K sleep 2:23 2.16 2.15 mma
7 ? 23007 root 148 20 7876K 1560K sleep 2:22 2.01 2.01 mma
7 ? 3017 root -16 20 90692K 66040K run 1719:21 0.73 0.72 midaemon
3 ? 3528 root 168 20 369M 7600K sleep 1523:48 0.68 0.68 utild
2 pts/1 25273 root 178 20 5176K 1012K run 0:00 0.72 0.65 top
5 ? 5 root 152 20 1188K 1188K run 0:03 0.65 0.64 signald
0 ? 25757 hpsmh 152 20 54164K 8320K run 0:42 0.51 0.51 httpd
3 ? 2609 cimsrvr 152 20 57000K 17964K run 973:22 0.49 0.49 cimservermain
6 ? 2055 root 152 20 64044K 7888K run 68:23 0.48 0.47 ovcd
4 ? 2614 root 152 20 589M 277M run 304:26 0.43 0.43 cimprovagt
4 ? 70 root 152 20 17676K 17676K run 499:27 0.33 0.33 vxfsd
5 ? 86 root 152 20 576K 576K run 47:59 0.31 0.31 pagezerod
2 ? 16296 root -27 20 34020K 29044K run 463:12 0.31 0.31 cmcld
dbrsrv:/#



Only the fuser process is in the "top" list.

[Dennis Handly]

What is the size of your file cache?
Are you up to date on patches?