Why calls to functions in libclntsh.so often come from *ROOT* in the Call Graph section?

[Dennis Handly]

>I mean there is no explanation what it means. Is it relevant to my situation?

It may be related to the inlining case I mentioned?  You can experiment with this be removing the Minimal Line Table by:

1) Make a copy of libclntsh

2) Make the copy writable

3) strip -l the copy

4)  Use LD_LIBRARY_PATH to point to the copy and rerun caliper.  (Make sure it is using the copy.)

Then see if *ROOT* goes away?

>I will upload HP wdb disassemble for libclntsh::kpufch().

I think the caller is more interesting.  And you haven't dumped the $cold_ part.

It could be that Caliper doesn't handle hot/cold code correctly?

And the disassembly shows that rp is saved at kpufch+0x42.

>libclntsh::kpufch() is called more often than libmy_lib.so::my_sub_function():

The issue is where the sample is taken and how much work Caliper does to unwind.  wdb is dumb and uses the industrial strength libunwind to unwind and that must get it right.  Caliper may take some shortcuts for speed.

>Moreover I see that lots of functions from libclntsh are called from *ROOT*

That has lots of $cold_ sections.

>Still the same picture.

Ideally if they couldn't fix it, they could explain it better.  :-)

>kpufch0.txt 87 KB

Shows it has a $cold_ section.

>test_oci.55.2.txt 177 KB

The Function Details show only samples in $cold_kpufch.  At least to the 1% cutoff.

>aCC: HP C/aC++ B3910B A.06.15 [May 16 2007]

This is pretty old.

[blackwater]

> It could be that Caliper doesn't handle hot/cold code correctly?
> That has lots of $cold_ sections.
> Shows it has a $cold_ section.

Sorry, does $cold_ section have any special meaning?

The call graph section has both $cold_kpufch0 and kpufch and they both according to Caliper are called from *ROOT*? Does $cold prefix have any specail meaning on IA64?

9.29 1 libclntsh.so.11.1::$cold_kpufch0 [15]
3.74 1 libclntsh.so.11.1::kpufch [26]

[Dennis Handly]

>does $cold_ section have any special meaning?

Functions compiled with PBO are split into hot and cold sections, the latter are in $cold_.

>Does $cold prefix have any special meaning on IA64?

Yes, for HP-UX's compiler toolchain.

[blackwater]

> It may be related to the inlining case I mentioned?  You can experiment with this be removing the Minimal Line Table by:

> 1) Make a copy of libclntsh
> 2) Make the copy writable
> 3) strip -l the copy
> 4)  Use LD_LIBRARY_PATH to point to the copy and rerun caliper.  (Make sure it is using the copy.)

> Then see if *ROOT* goes away?

It didn't help. I copied and stripped libclntsh.so.11.1 and ran HP caliper.
This is output:

ls -l /home/has_test/PERF_HAS_TESTS/HPUX/11.31/ia64/sk/test.oci.acc.1th.libclntsh/libclnt*
lrwxr-xr-x   1 has_test   users           19 Nov 22 15:28 /home/has_test/PERF_HAS_TESTS/HPUX/11.31/ia64/sk/test.oci.acc.1th.libclntsh/libclntsh.so -> ./libclntsh.so.11.1
-rwxr-xr-x   1 has_test   users      100883568 Nov 22 15:31 /home/has_test/PERF_HAS_TESTS/HPUX/11.31/ia64/sk/test.oci.acc.1th.libclntsh/libclntsh.so.11.1

ls -l /u01/app/oracle/product/11.2.0/cl11203/lib/libclntsh.so /u01/app/oracle/product/11.2.0/cl11203/lib/libclntsh.so.10.1 /u01/app/oracle/product/11.2.0/cl11203/lib/libclntsh.so.11.1 /u01/app/oracle/product/11.2.0/cl11203/lib/libclntsh_nopthread.so /u01/app/oracle/product/11.2.0/cl11203/lib/libclntst11.a
lrwxr-xr-x   1 oracle     oinstall        60 Sep 26 16:55 /u01/app/oracle/product/11.2.0/cl11203/lib/libclntsh.so -> /u01/app/oracle/product/11.2.0/cl11203/lib/libclntsh.so.11.1
lrwxr-xr-x   1 oracle     oinstall        12 Sep 26 16:54 /u01/app/oracle/product/11.2.0/cl11203/lib/libclntsh.so.10.1 -> libclntsh.so
-rwxr-xr-x   1 oracle     oinstall   120559312 Sep 26 16:53 /u01/app/oracle/product/11.2.0/cl11203/lib/libclntsh.so.11.1
-rwxr-xr-x   1 oracle     oinstall   120559312 Sep 26 16:54 /u01/app/oracle/product/11.2.0/cl11203/lib/libclntsh_nopthread.so
-rw-r--r--   1 oracle     oinstall         0 Oct 21  2011 /u01/app/oracle/product/11.2.0/cl11203/lib/libclntst11.a

pldd 4770
4770:   /home/has_test/PERF_HAS_TESTS/HPUX/11.31/ia64/sk/test.oci.acc.1th.libclntsh/test_oci
/home/has_test/PERF_HAS_TESTS/HPUX/11.31/ia64/sk/test.oci.acc.1th.libclntsh/test_oci
/usr/lib/hpux64/dld.so
/home/has_test/PERF_HAS_TESTS/HPUX/11.31/ia64/sk/test.oci.acc.1th.libclntsh/libclntsh.so.11.1
/home/has_test/PERF_HAS_TESTS/HPUX/11.31/ia64/sk/test.oci.acc.1th.libclntsh/libmy_lib.so
/usr/lib/hpux64/libstd_v2.so.1
/usr/lib/hpux64/libCsup.so.1
/usr/lib/hpux64/libm.so.1
/usr/lib/hpux64/libunwind.so.1
/usr/lib/hpux64/libc.so.1
/usr/lib/hpux64/libdl.so.1
/home/has_test/PERF_HAS_TESTS/HPUX/11.31/ia64/sk/test.oci.acc.1th.libclntsh/libnnz11.so
/usr/lib/hpux64/librt.so.1
/usr/lib/hpux64/libpthread.so.1
/usr/lib/hpux64/libnsl.so.1
/usr/lib/hpux64/libdiskown.so
/usr/lib/hpux64/libuca.so.1
/usr/lib/hpux64/libxti.so.1
/usr/lib/hpux64/libnss_files.so.1
/usr/lib/hpux64/libnss_dns.so.1


I will upload a caliper report. This is just a part of it

------------------------------------------------------------------------
                     100.00            1                     *ROOT* [1]
[26]      3.8      9.49                -                 libclntsh.so.11.1::kpufch [26]
                      83.01          323/465        69       libclntsh.so.11.1::kpufch0 [22]
                       7.50          344/1746       20       libclntsh.so.11.1::$cold_kpummTLSGET1 [56]
------------------------------------------------------------------------

[Dennis Handly]

>It didn't help. I copied and stripped libclntsh.so.11.1 and ran HP caliper.

Then is isn't the inlining problem.

It appeared you ran caliper on an existing process, 4770.  That will also have *ROOT* for functions that have a sample but no matching one for their caller.

[blackwater]

> It appeared you ran caliper on an existing process, 4770. That will also have *ROOT* for functions that have a sample but no matching one for their caller.

That is true. I run caliper for a already running process. But why is it a problem?
caliper scgprof -e 10 -o $1 `ps -u $USER | awk '$4~/test_oci/ {print $1}'`

[blackwater]

>   And you haven't dumped the $cold_ part.

Dump of assembler code for function OCIStmtFetch:
Dump of assembler code for function kpufch:
Dump of assembler code for function $cold_kpufch:

I have uploaded disassemle.txt.

[Dennis Handly]

>But why is it a problem?

Because for those functions, they won't have a parent.  The call counts don't match up because sampling started after the function was called.

But I would expect SOME of the calls to show up with non-*ROOT* if called again.

[blackwater]

> Ideally if they couldn't fix it, they could explain it better. :-)
Could you clarify, please. Do you mean Oracle or HP Caliper team?

[blackwater]

>Because for those functions, they won't have a parent.  The call counts don't match up because sampling started after the function was called.

> But I would expect SOME of the calls to show up with non-*ROOT* if called again.

So you mean that both functions still have to have normal call stack, not from  *ROOT*?

Is a problem of HP Caliper or libclntsh.so?

By the way, one question is not clear to me.

I have breakpoint on $cold_kpufch but it is never hit. Yet HP caliper shows that it is called quite often:

(gdb) info breakpoints
Num Type           Disp Enb Address    What
1   breakpoint     keep y   0x9fffffffbbbd88d0 in kpufch at kpuf.c:2079
        breakpoint already hit 88 times
4   breakpoint     keep n   0x9fffffffbb46abd0 <$cold_kpufch>

Do you know why?