Caliper says "Bad or invalid debug information"

[blackwater]

When I profile a small test built with gcc 4.3.1 I get "Bad or invalid debug information".

So I have written a simple test and compiled it on HP-UX ia64 11.31 with gcc 4.3.1.
This was a command to build the test:
g++ -g -O2 -pthread -mlp64 tz_test.cpp -o tz_test

Then I profiled it with caliper A.4.3.2:
caliper frof tz_test

When I profile tz_test caliper gives me this diagnostic message for the test:
Diagnostic Messages
----------------------------------------------------------------------
+ Bad or invalid debug information in tz_test
No source attribution possible.

and in the "Function details" there are lines like this:
19.32 [(No source information) tz_test::worker_thread::thread_func(), 0x40000000000211d0]

What could be done in order to fix this problem and to generate with gcc 4.3.1 correct debug information for caliper?

By the way there are some more things that might be relevant to my question:
1) wdb 6.1 correctly finds debug information in the executable built with gcc 4.3.1
2) gcc –O0 gives the same diagnostic message
3) when I compile the same test with aCC A.06.15 and then profile it I don't get this diagnostic message.
Everything is correct.
For aCC this was a command to build the test:
aCC -g0 -mt +W2461 +DD64 +O2 tz_test.cpp -o tz_test

[Dennis Handly · Accepted solution]

You are likely out of luck. Caliper doesn't like your foreign devil compiler's debug info.

And unless you are optimizing with aC++, you are likely wasting your time with caliper and g++.

You could try reporting it on the "Tech support" page off of:
http://www.hp.com/go/caliper

>Then I profiled it with caliper A.4.3.2: caliper frof tz_test

Was that fprof?
The latest Caliper is 5.2.

[blackwater]

> And unless you are optimizing with aC++, you are likely wasting your time with caliper and g++.
Why? What is wrong with caliper and gcc on HP-UX ia64?

> Was that fprof?
Yes, fprof.

> The latest Caliper is 5.2.
Asked to install and waiting for a response.

[Dennis Handly]

>What is wrong with caliper and gcc on HP-UX ia64?

gcc doesn't begin to give the performance that aC++ can give.

So what do you expect caliper to provide for gcc compiled code?

[blackwater]

> gcc doesn't begin to give the performance that aC++ can give.

Dennis,

I compile my tests and the main application with aCC A.06.15. I use this command:
aCC -g0 -mt +DD64 +O2 my_source.cpp.
As for this test I don't see that aCC builds code which is faster than gcc does.
As for the main application I have even seen some decrease in performance.
Could you please suggest some other aCC options to use to get better performance?

[Dennis Handly]

>Could you please suggest some other aCC options to use to get better performance?

+O3, -ipo, +Oprofile=collect (run instrumentation phase) then +Oprofile=use

Where does caliper say you are spending your time? scgprof

[blackwater]

> +O3, -ipo, +Oprofile=collect (run instrumentation phase) then +Oprofile=use

Dennis, thanks a lot.

I have got some more question that are related to caliper.

1) How can I be sure that a modification makes an application better or worse?

For example I have changed an algorithm in the function foo() and I want to check if it is now better or worse or there is no change. Let's assume that my modification doesn't result in a huge improvement so if I run a test for 1 minute the result might be 0,1 seconds better or 0,1 worse. Even though the improvement is not significant I simply want to know that there has been some improvement. By the way let's assume that the application is a daemon application.

After reading the caliper "User guide" I think that I should use the ecount measument and it will show me results of even small changes in my code. Of course without telling me where the improvement happened exactly which is OK since let's assume I have done only one modification.

Am I right that I have to use ecount in this situation?

2) And if I am right about ecount there is another question. What values in an ecount report tells me that there has been some improvement?

I run caliper in this way:
caliper ecount -e 60 -o test_result.txt

So first I measure in this way the old release of the application and then the modified release. However I am not sure what figures show me exactly that the modified version is better. Could you help me in this? Should I compare IA64_INST_RETIRED? Or should I compare cycles lost due to stalls? Or Effective CPI ? Or what?

3) How does current global CPU utilization of a HP-UX server affect "caliper ecount" and all other measurements?

My application is CPU bound. For example if I run "caliper ecount -e 60 " when the global CPU utilization is 15% and when it is 50% will it affect this measurement? As I said in my example my improvement is not significant. So if I get a better result according to "caliper ecount" might it be just because the global CPU utilization was lower at the moment of taking the measurement?

4) I have read here http://h20338.www2.hp.com/enterprise/downloads/Using%20compilers%20to%20get%20optimal%20performance-Final%20Transcript.pdf that "for memory-intensive applications you really want to consider using large pages".

My application normally allocates a few GBs for its data so I think should follow this advice. The question is that I am not sure what value I should set for the virtual memory page size that must be used for data in this situation? The paper recommends using 1M. However if I normally allocates a few GBs should I set 64M or even a bigger value?

Or is it useful in this situation to use "caliper ecount" just to measure the possible effect? I mean first run "caliper ecount" then run "chatr +pd 64M my_application", then run again "caliper ecount" and compare results? Or should I just follow the advice in the paper and simply set 1M?

[blackwater]

I would like to clarify the forth question. The application does not one allocate one buffer which is a few Gbs in size. Rather it reads data from a database, put them in different mostly small structures and classes and when it finishes loading data glance usually shows that VSS is a few GBs.

If a database doesn't have lots of records then after loading the records VSS is around 90 MB.

[Dennis Handly]

>1) How can I be sure that a modification makes an application better or worse?
>Am I right that I have to use ecount in this situation?

Typically that's not how you go about improving the whole application. You start with something like scgprof or fprof.

>2) What values in an ecount report tells me that there has been some improvement?

This is where you use Tech support on the caliper page.

> 4) I have read here that "for memory-intensive applications you really want to consider using large pages".

Yes. Do you get lots of DTLB misses?

>The paper recommends using 1M.

I heard that the kernel has problems over 1M.

>I mean first run caliper then run "chatr +pd 64M", then run again caliper and compare results?

If that shows fewer DTLB misses then that's good.

>put them in different mostly small structures and classes

Large pages still can help, provided the data isn't scattered all over too much.

[blackwater]

As for the first question the answer is that caliper might not understand debug information that gcc puts in an executable. It also has also been mentioned in the caliper documentation I recall.

As for the question about setting the virtual memory page size that should be used for data I should not use any sizes that greater than 1M.