cancel
Showing results for 
Search instead for 
Did you mean: 

Sar, Glance 11.31 03/2013

 
MR Dorobantu Dragos
Frequent Advisor

Sar, Glance 11.31 03/2013

Hello,

 

I have a problem with sar and glance after applying the latest bundle patch 11.31 03/2013, this are not starting at all.

For glance the process hung and for sar after a while I can kill the process.

 

Any idea or suggestion?

This are applied:

  FEATURE11i                            B.11.31.1303.391b Feature Enablement Patches for HP-UX 11i v3, March 2013
  HWEnable11i                           B.11.31.1303.391b Hardware Enablement Patches for HP-UX 11i v3, March 2013
  QPKAPPS                               B.11.31.1303.391 Applications Patches for HP-UX 11i v3, March 2013
  QPKBASE                               B.11.31.1303.391 Base Quality Pack Bundle for HP-UX 11i v3, March 2013

 

Thanks,

 

7 REPLIES 7
MR Dorobantu Dragos
Frequent Advisor

Re: Sar, Glance 11.31 03/2013

After I started sar, the prosses hangs.

 

vPAR01:/#ps -ef | grep sar
    root 16138 14534  0 10:55:04 pts/1     0:00 grep sar
    root 16025 15287  0 10:53:30 pts/3     0:00 sar -o fis 1 1
vPAR01:/#lsof -p 16025
COMMAND   PID USER   FD   TYPE             DEVICE SIZE/OFF    NODE NAME
sar     16025 root  cwd    DIR             64,0x3     8192       2 /
sar     16025 root  txt    REG             64,0x7   226452    2595 /usr/sbin/sar
sar     16025 root  mem    REG             64,0x7  5024328   29734 /usr/lib/hpux32/libc.so.1
sar     16025 root  mem    REG             64,0x7  6350152   27257 /usr/lib/hpux32/libm.so.1
sar     16025 root  mem    REG             64,0x7  1153840   29192 /usr/lib/hpux32/dld.so
sar     16025 root  mem    REG             64,0x7   190768   29195 /usr/lib/hpux32/uld.so
sar     16025 root  mem    REG             64,0x7    76456   29193 /usr/lib/hpux32/libdl.so.1
sar     16025 root    0u   STR            157,0x3   0t2997    1820 /dev/pts/3->ldterm->ptem->pts
sar     16025 root    1u   STR            157,0x3   0t2997    1820 /dev/pts/3->ldterm->ptem->pts
sar     16025 root    2u   STR            157,0x3   0t2997    1820 /dev/pts/3->ldterm->ptem->pts
sar     16025 root    3u   REG             64,0x3        0    2363 /fis
sar     16025 root    4r  FIFO 0xe00000072c568a88      0t0 4758678

 

 

I have applied the latest patch for SAR:

 

vPAR01:/#swlist -l patch PHCO_43002

# Initializing...

# Contacting target "vPAR01"...

#

# Target:  vPAR01:/ #

# PHCO_43002                    1.0            sar(1M) cumulative patch

# PHCO_43002.ADMN-ENG-A-MAN        1.0               OS-Core.ADMN-ENG-A-MAN    applied

# PHCO_43002.SYS-ADMIN             1.0               OS-Core.SYS-ADMIN    applied

# PHCO_43002.SYS2-ADMIN            1.0               OS-Core.SYS2-ADMIN    applied

 

 

 

 

Dennis Handly
Acclaimed Contributor

Re: sar, Glance 11.31 03/2013

Can you attach gdb to sar to get a stack trace?

Or better yet, use tusc to find the last hanging syscall?

MR Dorobantu Dragos
Frequent Advisor

Re: sar, Glance 11.31 03/2013

I attached the file for following command:

tusc -E -o sar.out sar -d 1 1

MR Dorobantu Dragos
Frequent Advisor

Re: sar, Glance 11.31 03/2013

close(4) .......................................................................................................................... [entry]
close(4) .......................................................................................................................... = 0
sigsetreturn(NULL, 0x6211988, 48640) .............................................................................................. [entry]
sigsetreturn(NULL, 0x6211988, 48640) .............................................................................................. = 0
mmap(NULL, 3352, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) .......................................................... [entry]
mmap(NULL, 3352, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) .......................................................... = 0x777f4000
sysconf(_SC_CPU_VERSION) .......................................................................................................... [entry]
sysconf(_SC_CPU_VERSION) .......................................................................................................... = 768
brk(0x400276d0) ................................................................................................................... [entry]
brk(0x400276d0) ................................................................................................................... = 0
brk(0x400296c0) ................................................................................................................... [entry]
brk(0x400296c0) ................................................................................................................... = 0
brk(0x4002a000) ................................................................................................................... [entry]
brk(0x4002a000) ................................................................................................................... = 0
brk(0x4002c000) ................................................................................................................... [entry]
brk(0x4002c000) ................................................................................................................... = 0
open(0x200000007fffcf70, O_RDONLY, 0) ............................................................................................. [entry]
open("/usr/lib/nls/loc/hpux32/locales.3/C", O_RDONLY, 0) .......................................................................... ERR#2 ENOENT
open(0x200000007fffe530, O_RDONLY, 0103030) ....................................................................................... [entry]
open("/usr/lib/nls/msg//sar.cat", O_RDONLY, 0103030) .............................................................................. ERR#2 ENOENT
open(0x200000007fffe530, O_RDONLY, 0103030) ....................................................................................... [entry]
open("/usr/lib/nls////sar.cat", O_RDONLY, 0103030) ................................................................................ ERR#2 ENOENT
sigprocmask(SIG_SETMASK, 0x200000007ffff130, 0x200000007ffff150) .................................................................. [entry]
sigprocmask(SIG_SETMASK, 0x7ffff130, 0x7ffff150) .................................................................................. = 0
sigprocmask(SIG_SETMASK, 0x200000007ffff150, 0x200000007ffff130) .................................................................. [entry]
sigprocmask(SIG_SETMASK, 0x7ffff150, 0x7ffff130) .................................................................................. = 0
sigsetreturn(NULL, 0x6211988, 48640) .............................................................................................. [entry]
sigsetreturn(NULL, 0x6211988, 48640) .............................................................................................. = 0
sigprocmask(SIG_SETMASK, 0x200000007ffff130, 0x200000007ffff150) .................................................................. [entry]
sigprocmask(SIG_SETMASK, 0x7ffff130, 0x7ffff150) .................................................................................. = 0
sigprocmask(SIG_SETMASK, 0x200000007ffff150, 0x200000007ffff130) .................................................................. [entry]
sigprocmask(SIG_SETMASK, 0x7ffff150, 0x7ffff130) .................................................................................. = 0
sigvector(SIGINT, 0x200000007ffff170, 0x200000007ffff180) ......................................................................... [entry]
sigvector(SIGINT, 0x7ffff170, 0x7ffff180) ......................................................................................... = 0
sigvector(SIGTERM, 0x200000007ffff170, 0x200000007ffff180) ........................................................................ [entry]
sigvector(SIGTERM, 0x7ffff170, 0x7ffff180) ........................................................................................ = 0
sigvector(SIGHUP, 0x200000007ffff170, 0x200000007ffff180) ......................................................................... [entry]
sigvector(SIGHUP, 0x7ffff170, 0x7ffff180) ......................................................................................... = 0
sigvector(SIGALRM, 0x200000007ffff170, 0x200000007ffff180) ........................................................................ [entry]
sigvector(SIGALRM, 0x7ffff170, 0x7ffff180) ........................................................................................ = 0
sigprocmask(SIG_UNBLOCK, 0x200000007ffff190, NULL) ................................................................................ [entry]
sigprocmask(SIG_UNBLOCK, 0x7ffff190, NULL) ........................................................................................ = 0
stat64(0x2000000040010300, 0x200000007ffff1b0) .................................................................................... [entry]
stat64("/usr/lib/hpux32/libsar.so.1", 0x7ffff1b0) ................................................................................. ERR#2 ENOENT
pipe() ............................................................................................................................ [entry]
pipe() ............................................................................................................................ = 4 (5)
fork() ............................................................................................................................ [entry]
fork() ............................................................................................................................ = 22541
close(5) .......................................................................................................................... [entry]
close(5) .......................................................................................................................... = 0
sigvector(SIGHUP, 0x200000007ffff170, 0x200000007ffff180) ......................................................................... [entry]
sigvector(SIGHUP, 0x7ffff170, 0x7ffff180) ......................................................................................... = 0
sigvector(SIGINT, 0x200000007ffff170, 0x200000007ffff180) ......................................................................... [entry]
sigvector(SIGINT, 0x7ffff170, 0x7ffff180) ......................................................................................... = 0
sigvector(SIGTERM, 0x200000007ffff170, 0x200000007ffff180) ........................................................................ [entry]
sigvector(SIGTERM, 0x7ffff170, 0x7ffff180) ........................................................................................ = 0
lseek64(4, 0, SEEK_SET) ........................................................................................................... [entry]
lseek64(4, 0, SEEK_SET) ........................................................................................................... ERR#29 ESPIPE
read(4, 0x200000004001d270, 20944) ................................................................................................ [entry]
read(4, 0x4001d270, 20944) ........................................................................................................ [sleeping]

At this point remains in sleeping state.

 

Dennis Handly
Acclaimed Contributor

Re: sar, Glance 11.31 03/2013

pipe() ............................................................................................................................ = 4 (5)
fork() ............................................................................................................................ = 22541
close(5) .......................................................................................................................... = 0
lseek64(4, 0, SEEK_SET) ........................................................................................................... ERR#29 ESPIPE
read(4, 0x4001d270, 20944) ........................................................................................................ [sleeping]

 

It appears that the child sar is not sending any data, it may be hung on something else.

You need to use the -fp tusc options to follow the fork.

MR Dorobantu Dragos
Frequent Advisor

Re: sar, Glance 11.31 03/2013

[2712] stat64("/usr/lib/hpux32/libsar.so.1", 0x7ffff1c0) .......................................................................... ERR#2 ENOENT
[2712] pipe() ..................................................................................................................... = 4 (5)
[2712] fork() ..................................................................................................................... = 2713
[2713] fork() ...................................................................................(returning as child ...) ......... = 2712
[2712] close(5) ................................................................................................................... = 0
[2712] sigvector(SIGHUP, 0x7ffff180, 0x7ffff190) .................................................................................. = 0
[2712] sigvector(SIGINT, 0x7ffff180, 0x7ffff190) .................................................................................. = 0
[2712] sigvector(SIGTERM, 0x7ffff180, 0x7ffff190) ................................................................................. = 0
[2713] close(1) ................................................................................................................... = 0
[2713] dup(5) ..................................................................................................................... = 1
[2712] lseek64(4, 0, SEEK_SET) .................................................................................................... ERR#29 ESPIPE
[2713] execve("/usr/lbin/sa/sadc", 0x7fffed40, 0x7ffff68c) ........................................................................ = 0 [32-bit]
[2713] mmap(NULL, 8192, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) ......................................... = 0x777fa000
[2713] open("/usr/lib/hpux32/dld.so", O_RDONLY, 0) ................................................................................ = 6
[2713] read(6, "7fE L F 0102010101\0\0\0\0\0\0\0".., 1024) ........................................................................ = 1024
[2713] mmap(NULL, 741808, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_FILE|MAP_SHLIB, 6, 0) ............................................... = 0xc0040000
[2713] sysconf(_SC_PAGE_SIZE) ..................................................................................................... = 4096
[2713] mmap(NULL, 7384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_SHLIB, -1, 0) ......................................... = 0x777f8000
[2713] mmap(0x777f5000, 12040, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FILE|MAP_SHLIB, 6, 786432) ................................... = 0x777f5000
[2713] close(6) ................................................................................................................... = 0
[2713] mmap(NULL, 8192, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) ......................................... = 0x777f2000
[2713] sysconf(_SC_PAGE_SIZE) ..................................................................................................... = 4096
[2713] stat("/usr/lib/hpux32/dpd", 0x7ffff1d0) .................................................................................... = 0
[2713] open("/usr/lib/hpux32/dpd", O_RDONLY, 0) ................................................................................... = 6
[2713] fcntl(6, F_SETFD, 0) ....................................................................................................... = 0
[2713] mmap(NULL, 16384, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) ........................................ = 0x777ec000
[2713] getdents(6, 0x777ec028, 8192) .............................................................................................. = 80
[2713] getdents(6, 0x777ec028, 8192) .............................................................................................. = 0
[2713] close(6) ................................................................................................................... = 0
[2713] getuid() ................................................................................................................... = 0 (0)
[2713] getgid() ................................................................................................................... = 3 (3)
[2713] open("/usr/lbin/sa/sadc", O_RDONLY, 0) ..................................................................................... = 6
[2713] pread(6, "\0\0\002\0\0\0, \0\0\0\aH P \0\0".., 60, 436) .................................................................... = 60
[2713] close(6) ................................................................................................................... = 0
[2713] utssys(0x7fffe620, 60, 0) .................................................................................................. = 0
[2713] procxsec(4, -1, 0x7fffe5f0, 40) ............................................................................................ = 0
[2713] open("/usr/lib/hpux32/libm.so.1", O_RDONLY, 0) ............................................................................. = 6
[2713] fstat(6, 0x7ffff110) ....................................................................................................... = 0
[2713] read(6, "7fE L F 0102010101\0\0\0\0\0\0\0".., 52) .......................................................................... = 52
[2713] pread(6, "7fE L F 0102010101\0\0\0\0\0\0\0".., 1024, 0) .................................................................... = 1024
[2713] mmap(NULL, 4946528, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_SHLIB, 6, 0) ....................................................... = 0xc0800000
[2713] mmap(NULL, 8940, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_SHLIB, 6, 4980736) .................................................. = 0x777e8000
[2713] close(6) ................................................................................................................... = 0
[2713] mmap(NULL, 16384, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) ........................................ = 0x777e4000
[2713] open("/usr/lib/hpux32/libc.so.1", O_RDONLY, 0) ............................................................................. = 6
[2713] fstat(6, 0x7ffff110) ....................................................................................................... = 0
[2713] read(6, "7fE L F 0102010101\0\0\0\0\0\0\0".., 52) .......................................................................... = 52
[2713] pread(6, "7fE L F 0102010101\0\0\0\0\0\0\0".., 1024, 0) .................................................................... = 1024
[2713] mmap(NULL, 3000592, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_SHLIB, 6, 0) ....................................................... = 0xc0300000
[2713] madvise(0xc0300000, 0x2dc910, MADV_NORMAL) ................................................................................. = 0
[2713] mmap(NULL, 47912, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_SHLIB, -1, 0) ........................................ = 0x777d8000
[2713] mmap(0x777d0000, 29908, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_SHLIB, 6, 3014656) ........................................... = 0x777d0000
[2713] close(6) ................................................................................................................... = 0
[2713] mmap(NULL, 16384, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) ........................................ = 0x777cc000
[2713] open("/usr/lib/hpux32/libdl.so.1", O_RDONLY, 0) ............................................................................ = 6
[2713] fstat(6, 0x7ffff110) ....................................................................................................... = 0
[2713] read(6, "7fE L F 0102010101\0\0\0\0\0\0\0".., 52) .......................................................................... = 52
[2713] pread(6, "7fE L F 0102010101\0\0\0\0\0\0\0".., 1024, 0) .................................................................... = 1024
[2713] mmap(NULL, 16080, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_SHLIB, 6, 0) ......................................................... = 0xc0004000
[2713] mmap(NULL, 224, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_SHLIB, 6, 65536) ..................................................... = 0x777fe000
[2713] close(6) ................................................................................................................... = 0
[2713] sigsetreturn(NULL, 0x6211988, 48640) ....................................................................................... = 0
[2713] mmap(NULL, 3352, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) ................................................... = 0x777f4000
[2713] sysconf(_SC_CPU_VERSION) ................................................................................................... = 768
[2713] brk(0x40010560) ............................................................................................................ = 0
[2713] brk(0x40012550) ............................................................................................................ = 0
[2713] brk(0x40013000) ............................................................................................................ = 0
[2713] brk(0x40015000) ............................................................................................................ = 0
[2713] open("/usr/lib/nls/loc/hpux32/locales.3/C", O_RDONLY, 0) ................................................................... ERR#2 ENOENT
[2713] open("/usr/lib/nls/msg//sadc.cat", O_RDONLY, 0103030) ...................................................................... ERR#2 ENOENT
[2713] open("/usr/lib/nls////sadc.cat", O_RDONLY, 0103030) ........................................................................ ERR#2 ENOENT
[2713] sysconf(0x4c2e) ............................................................................................................ = 1
[2713] time(NULL) ................................................................................................................. = 1373450294
[2713] stat64("/usr/lib/hpux32/libsar.so.1", 0x7fffcbd0) .......................................................................... ERR#2 ENOENT
[2713] rtprio(0, 90) .............................................................................................................. = 1001
[2713] sigblock(0x2002) ........................................................................................................... = 0
[2713] sigprocmask(SIG_SETMASK, 0x7fffcad0, 0x7fffcaf0) ........................................................................... = 0
[2713] sigprocmask(SIG_SETMASK, 0x7fffcaf0, 0x7fffcad0) ........................................................................... = 0
[2713] sigsetreturn(NULL, 0x6211988, 48640) ....................................................................................... = 0
[2713] sigprocmask(SIG_SETMASK, 0x7fffcad0, 0x7fffcaf0) ........................................................................... = 0
[2713] sigprocmask(SIG_SETMASK, 0x7fffcaf0, 0x7fffcad0) ........................................................................... = 0
[2713] sigvector(SIGALRM, 0x7fffcb20, 0x7fffcb30) ................................................................................. = 0
[2713] setitimer(ITIMER_REAL, 0x7fffcb40, NULL) ................................................................................... = 0
[2713] brk(0x40042000) ............................................................................................................ = 0
[2713] pstat(PSTAT_DISKINFO, 0x40019670, 928, 128, 0) ............................................................................. = 94
[2713] brk(0x4004c000) ............................................................................................................ = 0
[2713] pstat(PSTAT_DISKINFO, 0x4002eb30, 928, 128, 94) ............................................................................ = 0

 

Now i can see the child, but the process still remains "hung"

vPAR01:/#ps -ef | grep 2712
    root  2713  2712  0 12:58:14 pts/4     0:00 /usr/lbin/sa/sadc 1 2
    root  2712  2711  0 12:58:14 pts/4     0:00 sar -d 1 1
    root  3069 21697  0 13:04:49 pts/5     0:00 grep 2712

I have a lot of sadc processes on my system.

 

Dennis Handly
Acclaimed Contributor

Re: sar, Glance 11.31 03/2013

[2713] pstat(PSTAT_DISKINFO, 0x40019670, 928, 128, 0) ............................................................................. = 94
[2713] brk(0x4004c000) ............................................................................................................ = 0
[2713] pstat(PSTAT_DISKINFO, 0x4002eb30, 928, 128, 94) ............................................................................ = 0
>Now I can see the child, but the process still remains "hung"

 

Is this process taking up lots of CPU?  The last pstat(2) was successful, though it returned 0.

 

>I have a lot of sadc processes on my system.

 

Who is their parent?