topic Re: fuser uses too much CPU in Operating System - HP-UX

fuser uses too much CPU

Fedon Kadifeli — Wed, 24 Mar 2010 06:35:09 GMT

This is an rp7440 (HP-UX 11.31) system with 8 cores. It is not busy at the moment. However when I run fuser to find processes using files on a specific mount point, the fuser command takes too much CPU time to complete:

# model
9000/800/rp7440

# bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 4096000 1376656 2698296 34% /
/dev/vg00/lvol1 1998072 159360 1638904 9% /stand
/dev/vg00/lvol8 10256384 3965016 6242592 39% /var
/dev/vg00/lvola 20480000 22166 19179291 0% /var/adm/crash
/dev/vg00/lvol7 10256384 1958504 8233096 19% /usr
/dev/vg00/lvol6 1048576 53784 992960 5% /tmp
/dev/vg00/lvol5 10256384 4237384 5972072 42% /opt
/dev/vg00/lvol4 524288 56032 465016 11% /home

# swapinfo -mt
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 16000 411 15589 3% 0 - 1 /dev/vg00/lvol2
dev 16000 413 15587 3% 0 - 1 /dev/vg00/lvol9
reserve - 1238 -1238
memory 31142 8354 22788 27%
total 63142 10416 52726 16% - 0 -

# ps -ef | wc -l
208

# w
9:31am up 139 days, 17:40, 1 user, load average: 0.16, 0.14, 0.09
User tty login@ idle JCPU PCPU what
fedkad pts/0 9:23am w

# time fuser -cu /home
/home: 29334co(fedkad) 22801co(fedkad) 26421co(fedkad)

real 2:29.4
user 0.0
sys 2:28.9

Re: fuser uses too much CPU

Fedon Kadifeli — Wed, 24 Mar 2010 06:47:27 GMT

Please note: This problem is not specific to a certain mount point or file. Other examples:

# time fuser -cu /stand
/stand:

real 2:29.8
user 0.1
sys 2:29.1

# time fuser -u /var/adm/syslog/syslog.log
/var/adm/syslog/syslog.log: 1806o(root)

real 2:25.9
user 0.0
sys 2:25.2

Re: fuser uses too much CPU

Johnson Punniyalingam — Wed, 24 Mar 2010 06:56:12 GMT

can you check from Glance ? what's your CPU usage ?

# sar -Mu 5 50
# sar -Mq 5 50
-- What is the CPU load?

Re: fuser uses too much CPU

Kapil Jha — Wed, 24 Mar 2010 07:06:14 GMT

can you check if PHCO_31613 is installed.

Is this problem in only one box or all are suffering from the same fever :)

BR,
Kapil+

Re: fuser uses too much CPU

Fedon Kadifeli — Wed, 24 Mar 2010 07:15:07 GMT

Kapil,

This patch is for 11.23; and as far as I see there is not an equivalent patch for 11.31.

Our system is 11.31.

Thanks.

Re: fuser uses too much CPU

Fedon Kadifeli — Wed, 24 Mar 2010 07:21:46 GMT

CPU usage (kernel mode) is high as I specified in my original post.

# time fuser -u /var/adm/syslog/syslog.log
/var/adm/syslog/syslog.log: 1806o(root)

real 2:24.2
user 0.1
sys 2:24.0

----

# sar -Mu 5 50

HP-UX dbrsrv B.11.31 U 9000/800 03/24/10

10:15:43 cpu %usr %sys %wio %idle
10:15:48 0 0 2 77 21
1 0 4 0 96
2 0 1 99 0
3 0 1 99 0
4 1 1 0 98
5 0 76 22 2
6 0 0 0 99
7 1 24 0 74
system 1 14 37 49
10:15:53 0 0 0 0 100
1 1 4 0 95
2 0 0 100 0
3 1 1 98 0
4 0 1 0 99
5 0 1 98 0
6 0 53 0 47
7 0 47 0 53
system 0 13 37 49
10:15:58 0 5 3 15 76
1 1 4 10 84
2 1 2 96 0
3 1 2 80 17
4 1 2 0 96
5 1 1 89 9
6 0 100 0 0
7 0 0 0 99
system 1 15 36 48
[...]

----

# sar -Mq 5 50

HP-UX dbrsrv B.11.31 U 9000/800 03/24/10

10:15:42 cpu runq-sz %runocc swpq-sz %swpocc
10:15:47 0 0.0 0
1 0.0 0
2 0.0 0
3 0.0 0
4 0.0 0
5 1.0 40
6 0.0 0
7 0.0 0
system 1.0 5 0.0 0
10:15:52 0 0.0 0
1 0.0 0
2 0.0 0
3 0.0 0
4 0.0 0
5 0.0 0
6 0.0 0
7 0.0 0
system 0.0 0 0.0 0
10:15:57 0 0.0 0
1 0.0 0
2 0.0 0
3 0.0 0
4 0.0 0
5 0.0 0
6 0.0 0
7 0.0 0
system 0.0 0 0.0 0
10:16:02 0 0.0 0
1 0.0 0
2 0.0 0
3 0.0 0
4 0.0 0
5 1.0 20
6 1.0 20
7 0.0 0
system 1.0 5 0.0 0
[...]

Re: fuser uses too much CPU

Fedon Kadifeli — Wed, 24 Mar 2010 07:25:50 GMT

We have another box with almost (99.9%) the same configuration. It had the same problem, until I rebooted it yesterday. After the reboot, there is no problem currently on this box:

# uptime
10:24am up 23:52, 23 users, load average: 0.58, 0.68, 0.70

# time fuser -u /var/adm/syslog/syslog.log
/var/adm/syslog/syslog.log: 1350o(root) 3044o(root)

real 0.5
user 0.0
sys 0.3

Re: fuser uses too much CPU

Johnson Punniyalingam — Wed, 24 Mar 2010 08:03:04 GMT

looks like to me you have "CPU" Bottle neck

can we check "which process" using most of your cpu ..?

# top

# vmstat 5 5

if its related to application process,causing the "High CPU usage" will check with application team,

This may not be a (System)problem. It could be a lot of sleeping processes caused by bad application design.

or as you said you had similar problem, in another server .after reboot the server is OK,

on top of that also check

follow , below link collect info (performance report and post)

http://www.hpux.ws/system.perf.sh

any hardware I/O problem. Could be disk or cable or controller.
System patch level are up to date ?
Any errors in dmesg ?
/var/adm/syslog/syslog.log ?
/var/opt/resmon/log/event.log ?

Re: fuser uses too much CPU

Fedon Kadifeli — Wed, 24 Mar 2010 08:12:43 GMT

There is no CPU, memory, disk, or network bottleneck on this system, since I moved all the applications to the other system. The system is practically idle. I checked this also with Glance.

Please note that this system is up for 139 days. And please note that we had the same problem on the other system before rebooting it.

I repeat with another example:

# uptime
11:08am up 139 days, 19:17, 3 users, load average: 0.03, 0.03, 0.03

# ll -R /var/adm/crash
total 2
-rwxr-xr-x 1 root root 1 Aug 6 2008 bounds
drwxr-xr-x 2 root root 96 Aug 7 2008 lost+found

/var/adm/crash/lost+found:
total 0

# time fuser -u /var/adm/crash
/var/adm/crash:

real 2:30.5
user 0.1
sys 2:29.8

---

System: dbrsrv Wed Mar 24 11:10:51 2010
Load averages: 0.24, 0.11, 0.06
225 processes: 152 sleeping, 73 running
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.05 0.8% 0.0% 0.4% 98.8% 0.0% 0.0% 0.0% 0.0%
1 1.70 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0% 0.0%
2 0.02 0.0% 0.0% 1.2% 98.8% 0.0% 0.0% 0.0% 0.0%
3 0.02 0.2% 0.0% 1.2% 98.6% 0.0% 0.0% 0.0% 0.0%
4 0.02 1.8% 0.0% 1.0% 97.2% 0.0% 0.0% 0.0% 0.0%
5 0.07 0.0% 0.0% 4.0% 96.0% 0.0% 0.0% 0.0% 0.0%
6 0.02 0.2% 0.0% 1.0% 98.8% 0.0% 0.0% 0.0% 0.0%
7 0.02 0.2% 0.0% 1.0% 98.8% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.24 0.4% 0.0% 13.7% 85.9% 0.0% 0.0% 0.0% 0.0%

System Page Size: 4Kbytes
Memory: 832380K (649992K) real, 2332472K (1759412K) virtual, 22007832K free Page# 1/13

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
1 pts/2 25011 root 240 20 1964K 240K run 2:04 100.85 100.67 fuser
5 ? 3 root 152 20 36K 36K sleep 7805:11 3.70 3.69 statdaemon
0 ? 23169 root 148 20 7876K 1560K sleep 2:34 2.20 2.20 mma
6 ? 23056 root 148 20 7876K 1560K sleep 2:23 2.16 2.15 mma
7 ? 23007 root 148 20 7876K 1560K sleep 2:22 2.01 2.01 mma
7 ? 3017 root -16 20 90692K 66040K run 1719:21 0.73 0.72 midaemon
3 ? 3528 root 168 20 369M 7600K sleep 1523:48 0.68 0.68 utild
2 pts/1 25273 root 178 20 5176K 1012K run 0:00 0.72 0.65 top
5 ? 5 root 152 20 1188K 1188K run 0:03 0.65 0.64 signald
0 ? 25757 hpsmh 152 20 54164K 8320K run 0:42 0.51 0.51 httpd
3 ? 2609 cimsrvr 152 20 57000K 17964K run 973:22 0.49 0.49 cimservermain
6 ? 2055 root 152 20 64044K 7888K run 68:23 0.48 0.47 ovcd
4 ? 2614 root 152 20 589M 277M run 304:26 0.43 0.43 cimprovagt
4 ? 70 root 152 20 17676K 17676K run 499:27 0.33 0.33 vxfsd
5 ? 86 root 152 20 576K 576K run 47:59 0.31 0.31 pagezerod
2 ? 16296 root -27 20 34020K 29044K run 463:12 0.31 0.31 cmcld
dbrsrv:/#

Only the fuser process is in the "top" list.

Re: fuser uses too much CPU

Dennis Handly — Wed, 24 Mar 2010 08:19:09 GMT

What is the size of your file cache?
Are you up to date on patches?

Re: fuser uses too much CPU

Fedon Kadifeli — Wed, 24 Mar 2010 08:37:02 GMT

Dennis,

System has 32 GB memory and

# kctune | grep filecache
filecache_max 979635732 3% Imm (auto disabled)
filecache_min 326545244 1% Imm (auto disabled)

The system was last patched 5 months ago with HP recommended patch bundle.

Re: fuser uses too much CPU

Johnson Punniyalingam — Wed, 24 Mar 2010 09:27:06 GMT

Check below link , tips on filecache

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1287943

Checked with one of my (Dev Database server)

Phys Mem : 8.0gb

# kctune |grep filecache
filecache_max 651795824 8% Imm (auto disabled)
filecache_min 244423434 3% Imm (auto disabled)

Re: fuser uses too much CPU

MaSa_3 — Wed, 24 Mar 2010 11:34:04 GMT

Hello Fedon,
I am currently facing a similar problem - so I found your posting.

I have got 6 productive HP-UX boxes(11.11) almost the same application Software and in general very low system load at all.
Three of the boxes patched about 3 month ago with HP recommended Patches about 5 month old, running fuser very slow with high CPU load in system mode.

The tree other boxes patched more then one year ago and uptime is also more then one year, those systems do not show this issue.
You can run fuser is without impacting system performance - also an other test system where I can do what I like - but I can not reproduce the problem.

Rebooting one of the impacted boxes is not option in this state of the analysis.

A wild guess might be, a specific patch together with an uptime, longer about 100 days.
We noticed the issue the fist time in files-system clean up job about two weeks ago which uses fuser and never not use significant CPU resources before

What I noticed is that the fuser command is running "pstat" system calls in loop, changing just the last parameter until the function returns with return code "0".
And then continuing with an other "pstat" call(see tusc output below ).
On the boxes that are OK it took just a few iterations on the others it took sometimes several 10000 iterations.

I have just started the analysis and keep you informed on the progress.

#time /usr/sbin/fuser /etc/hosts
/etc/hosts:

real 0m51.06s
user 0m0.04s
sys 0m50.50s
#what /usr/sbin/fuser
/usr/sbin/fuser:
fuser.c $Date: 2004/03/27 22:08:43 $Revision: r11.11/2 PATCH_11.11 (PHCO_30075)
$Revision: @(#) all R11.11_BL2004_0422_1 PATCH_11.11 PHCO_30075
Thu Apr 22 05:57:31 PDT 2004 $

#tusc /usr/sbin/fuser /etc/hosts

....snip....
pstat(PSTAT_PROC_VM, 0x7eff3f80, 104, 24924, 27324) ....................... = 1
pstat(PSTAT_PROC_VM, 0x7eff3f80, 104, 24924, 27325) ....................... = 1
pstat(PSTAT_PROC_VM, 0x7eff3f80, 104, 24924, 27326) ....................... = 1
pstat(PSTAT_PROC_VM, 0x7eff3f80, 104, 24924, 27327) ....................... = 1
pstat(PSTAT_PROC_VM, 0x7eff3f80, 104, 24924, 27328) ....................... = 1
pstat(PSTAT_PROC_VM, 0x7eff3f80, 104, 24924, 27329) ....................... = 1
pstat(PSTAT_PROC_VM, 0x7eff3f80, 104, 24924, 27330) ....................... = 1
pstat(PSTAT_PROC_VM, 0x7eff3f80, 104, 24924, 27331) ....................... = 1
pstat(PSTAT_PROC_VM, 0x7eff3f80, 104, 24924, 27332) ....................... = 1
pstat(PSTAT_PROC_VM, 0x7eff3f80, 104, 24924, 27333) ....................... = 1

... snip.....
pstat(PSTAT_PROC_VM, 0x7eff3f80, 104, 11571, 83) .......................... = 1
pstat(PSTAT_PROC_VM, 0x7eff3f80, 104, 11571, 84) .......................... = 1
pstat(PSTAT_PROC_VM, 0x7eff3f80, 104, 11571, 85) .......................... = 1
pstat(PSTAT_PROC_VM, 0x7eff3f80, 104, 11571, 86) .......................... = 1
pstat(PSTAT_PROC_VM, 0x7eff3f80, 104, 11571, 87) .......................... = 1
pstat(PSTAT_PROC_VM, 0x7eff3f80, 104, 11571, 88) .......................... = 1
pstat(PSTAT_PROC_VM, 0x7eff3f80, 104, 11571, 89) .......................... = 1
pstat(PSTAT_PROC_VM, 0x7eff3f80, 104, 11571, 90) .......................... = 1
pstat(PSTAT_PROC_VM, 0x7eff3f80, 104, 11571, 91) .......................... = 1
pstat(PSTAT_PROC_VM, 0x7eff3f80, 104, 11571, 92) .......................... = 1
pstat(PSTAT_PROC_VM, 0x7eff3f80, 104, 11571, 93) .......................... = 1
pstat(PSTAT_PROC_VM, 0x7eff3f80, 104, 11571, 94) .......................... = 1
pstat(PSTAT_PROC_VM, 0x7eff3f80, 104, 11571, 95) .......................... = 0
pstat(PSTAT_PROC, 0x7eff0aa8, 1176, 10, 184) .............................. = 0
write(2, "\n", 1) ......................................................... = 1
sysconf(_SC_CPU_VERSION) .................................................. = 532
brk(0x400015f0) ........................................................... = 0
brk(0x400035d8) ........................................................... = 0
brk(0x40006000) ........................................................... = 0
ioctl(1, TCGETA, 0x7eff0cd8) .............................................. ERR#25 ENOTTY

write(1, "\n", 1) ......................................................... = 1
exit(0) ................................................................... WIFEXITED(0)
#

Re: fuser uses too much CPU

Fedon Kadifeli — Wed, 24 Mar 2010 11:44:57 GMT

Hi MaSa,

Although my system is 11.31 (not the same as yours), I think we are facing the same problem.

I have seen high number pstat system calls in Glance.

The problem may not be directly related to the fsuser executable itself, which has not changed for quite some time:

# what /usr/sbin/fuser
/usr/sbin/fuser:
$Revision: 92453-07 linker linker crt0.o B.11.16.01 030415 $
$Revision: B.11.31_LR

# ll /usr/sbin/fuser
-r-xr-xr-x 1 bin bin 32768 Feb 15 2007 /usr/sbin/fuser

Re: fuser uses too much CPU

MaSa_3 — Wed, 24 Mar 2010 13:06:24 GMT

Hi,
your are right, it is not the same HP-UX version, it is not the same fuser executable and it is not related to that executable.

I just noticed that the "losf" command also shows the same behavior.
It took minutes of CPU usage in system mode to run to completion.

lsof and fuser need the same data from the HP-UX Kernel, lsof even more.
And the lsof trace has also this pstat loops as fuser.

Do you see the same behavior?

I am not really sure what I should learn form this pstat calls.
Return Value "1" is usually "EPERM" -- insufficient permission/not super user.
Is it busy waiting on some locked resources/system tables becoming available?

What I try next is to shutdown some subsystems not related to end user businesses, and see if things change.

Ideas welcome! Thanks in advance!

Re: fuser uses too much CPU

MaSa_3 — Wed, 24 Mar 2010 14:19:32 GMT

Hi,
shutting down unneeded subsystems pushs me closer to a solution!

Stopping and restarting "utild" witch is started from "init" solves the problem.
After kicking out this daemon "fuser" and "lsof" behave as normal not consuming significant CPU resources anymore.
Restarting this daemon did not make the thing worse again.

Dose your system run utild also?

If your inittab looks like my one, you can just kill the running utild and it is restarted by init. (see commands below)

It works on all of my systems impacted.

The next step might be to look for patches or updates for that peace of software -- but not today!

So I recommend to give it a try!
Good Luck!

# uptime
1:18pm up 116 days, 3:45, 5 users, load average: 0.05, 0.07, 0.08
# time /usr/sbin/fuser /etc/hosts
/etc/hosts:
real 0m40.56s
user 0m0.04s
sys 0m40.37s <<<<<<<<<<<<< Bevor !!!!!

# grep utild /etc/inittab
util:23456:respawn:/usr/lbin/utild # Utilization Provider daemon

# ps -ef | grep utild
root 26952 1 0 Jan 12 ? 192:29 /usr/lbin/utild
# kill 26952
# ps -ef | grep utild
root 13647 1 0 16:00:27 ? 0:00 /usr/lbin/utild
# time /usr/sbin/fuser /etc/hosts
/etc/hosts:
real 0m0.04s
user 0m0.00s
sys 0m0.03s <<<<<<<<<<<<<<<<<< After !!!

#what /usr/lbin/utild
/usr/lbin/utild:
01.08.03.01
#swlist -l file | grep utild
utilProvider.UTIL-ENG-A-MAN: /usr/share/man/man1m.Z/utild.1m
utilProvider.UTIL-RUN: /usr/lbin/utild

Re: fuser uses too much CPU

Fedon Kadifeli — Wed, 24 Mar 2010 14:42:26 GMT

Yes MaSa!

You are right! utild seems to be the culprit.

# time lsof >/tmp/lo

real 2:31.9
user 1.6
sys 2:28.9

# grep utild /etc/inittab
util:23456:respawn:/usr/lbin/utild # Utilization Provider daemon

# ps -ef | grep utild
root 3528 1 0 Nov 4 ? 1526:55 /usr/lbin/utild
root 9965 10626 0 17:38:10 pts/2 0:00 grep utild

# kill 3528

# ps -ef | grep utild
root 9987 1 2 17:38:18 ? 0:00 /usr/lbin/utild
root 10002 10626 0 17:38:19 pts/2 0:00 grep utild

# time lsof >/tmp/lo

real 3.7
user 1.5
sys 2.1

# what /usr/lbin/utild
/usr/lbin/utild:
$Revision: 92453-07 linker linker crt0.o B.11.16.01 030415 $
HP aC++ B3910B X.03.37.01 C++ Standard Library (RogueWave Version 1.2.1)
HP aC++ B3910B X.03.37.01 Classic Iostream Library
HP aC++ B3910B X.03.37.01 Language Support Library
01.08.01.01

# swlist -l file | grep utild
utilProvider.UTIL-ENG-A-MAN: /usr/share/man/man1m.Z/utild.1m
utilProvider.UTIL-RUN: /usr/lbin/utild

# ll /usr/lbin/utild
-r-xr-xr-x 1 bin bin 806912 Nov 16 2008 /usr/lbin/utild

# time fuser -u /var/adm/crash
/var/adm/crash:

real 0.0
user 0.0
sys 0.0

Re: fuser uses too much CPU

Dennis Handly — Thu, 25 Mar 2010 06:11:14 GMT

>MaSa: What I noticed is that the fuser command is running "pstat" system calls in loop, changing just the last parameter until the function returns with return code "0".

Yes, that's how you can call pstat_getprocvm(2) and iterate over the segments for PID 24924. And it seems that this PID has way too many memory regions, > 27333!

>I am not really sure what I should learn form this pstat calls.

You can know exactly what it is doing. It is probably scanning for mmapped files?

>Return Value "1" is usually "EPERM"

No, it means it returned one memory region.

>Stopping and restarting "utild"

And that was PID 24924?

Re: fuser uses too much CPU

Fedon Kadifeli — Thu, 25 Mar 2010 06:44:28 GMT

Yes,

It seems that utild has many memory regions (MEMMAP). I checked from Glance.

Is this a general behavior for utild?

Re: fuser uses too much CPU

Dennis Handly — Thu, 25 Mar 2010 06:53:07 GMT

>Is this a general behavior for utild?

I'm not sure. It seems like an awful lot. You should contact the Response Center so you can mention this problem with utild(1m) and fuser/lsof.