topic Re: Why so many threads? in Operating System - Linux

Why so many threads?

Jose M. del Rio — Mon, 26 Mar 2007 15:23:03 GMT

Hi,
we are using a 3rd-party executable. After upgrading to its latest version, we notice it allocates now a lot of memory, just since the beginning.
By using tusc and glance we can see it is creating on startup as many threads as processors there are, each with a 8 Mb mmap() region. The previous version created only one.
According to its developers, they are not creating the threads expressly nor compiling the program with the +Oparallel switch but the matter is: setting the environment variable MP_NUMBER_OF_THREADS=1 does the trick and only one thread is created.
Questions:
1) How can this be? Is there another compiler option that can lead to the creation of parallel threads?
2) Given an executable, is there a way to know the options it was compiled with?
Thanks.
Regards.

Re: Why so many threads?

Steven E. Protter — Mon, 26 Mar 2007 15:45:19 GMT

Shalom,

You don't say what the third party executable is.

You updated the program and its performance characteristics changed.

You did not patch your HP-UX system or make changes to it.

The developers are not accurate, they changed something.

Most of the time when something like this occurs its a result of the last thing you did, which was to update the executable.

It is possible you did so without following system prepartion guidelines, but you should have been notified of that, certainly when you contacted the application vendor.

If you mentioned the name of the third party product, you might find someone who has experienced this issue can provide assistance. Right now you've given us very little to go by.

SEP

Re: Why so many threads?

Emil Velez — Mon, 26 Mar 2007 21:37:46 GMT

ON HPUX 11.0 and later the OS supported kernel threads. A programmer creates multiple threads to do parallel operations on different processors. Evidently the programmer put a capability to limit the number of threads which limits the programs capability to utilize a multiprocessor box. Threads are a valid way of modeling applications so a multithreaded application is not a problem. In fact some of the HP-UX OS applications like the midaemon and vxfsd and other processes have multiple threads. IF you want to see this if you have glance you can goto the process report and add the column "PROC_THREAD_COUNT" to the report and see what processes on HP-UX utilize multiple threads.

Re: Why so many threads?

Dennis Handly — Tue, 27 Mar 2007 00:51:48 GMT

>1) How can this be? Is there another compiler option that can lead to the creation of parallel threads?

+Oopenmp will also create threads.

>2) Given an executable, is there a way to know the options it was compiled with?

If you have a recent linker patch, use /usr/ccs/bin/footprints.

You can also use chatr(1) on the executable and look for libcps.

Re: Why so many threads?

Jose M. del Rio — Tue, 27 Mar 2007 07:04:56 GMT

Steven: the executable is 'gsrvr.shared'. It is included in ESRI's ArCSDE software.

Emil: in this case, the problem with having multiple threads is: a gsrvr is a dedicated server process, servicing a unique client connection. Our server has 16 CPUs so every gsrvr creates on startup +15 additional threads. Each thread has, among other regions, a 8 Mb MMAP region. That makes 15*8 Mb = 120 Mb per gsrvr. It is not uncommon for us to have 400 concurrent connections => 400 * 120 Mb = 48 Gb. Our swap space gets exhausted.

Dennis:
- chatr shows it was linked with libcps.2. Does this mean either +Oparallel or +Oopenmp were used?
- none of our servers have the footprints command. Do you know which product we should install?

Thanks to all of you.

Re: Why so many threads?

James R. Ferguson — Tue, 27 Mar 2007 07:12:26 GMT

Hi Jose:

For 11.23 PHSS_34860 would bring you the 'footprint' binary. It's contained in the lastest linker patch as Dennis noted.

For 11.11 use patch PHSS_35381.

If you are running 11.31 you should already have it. The online manpages for 11.31 offer insight into its output.

Regards!

...JRF...

Re: Why so many threads?

Dennis Handly — Tue, 27 Mar 2007 08:02:34 GMT

>- chatr shows it was linked with libcps.2. Does this mean either +Oparallel or +Oopenmp were used?

Yes.

Re: Why so many threads?

Jose M. del Rio — Tue, 27 Mar 2007 11:01:23 GMT

Thanks James & Dennis.
It's incredible how useful this forum can be :-)
Here is footprint's output.
According to the man page, I guess the fact that it can't find any compilation unit is because the program was not compiled with an HP compiler? Or maybe some compiler switch disabled footprint in the object file?
It seems elfdump can't find compilation units either.

ensnada6:/arcsde/sdeexe91/bin#/usr/ccs/bin/footprints -v gsrvr.shared
Scanning gsrvr.shared ...
Shared library: /usr/lib/pa20_64/libCsup.2 (system library - not scanned)
Shared library: /usr/lib/pa20_64/libstream.2 (system library - not scanned)
Shared library: /usr/lib/pa20_64/libcl.2 (system library - not scanned)
Shared library: /usr/lib/pa20_64/libnsl.1 (system library - not scanned)
Shared library: /usr/lib/pa20_64/libdl.1 (system library - not scanned)
Shared library: /usr/lib/pa20_64/libm.2 (system library - not scanned)
Shared library: /usr/lib/pa20_64/librt.2 (system library - not scanned)
Shared library: /usr/lib/pa20_64/libF90.1 (system library - not scanned)
Shared library: /usr/lib/pa20_64/libcps.2 (system library - not scanned)
Shared library: /usr/lib/pa20_64/libpthread.1 (system library - not scanned)
Shared library: /usr/lib/pa20_64/libnss_dns.1 (system library - not scanned)
Shared library: /usr/lib/pa20_64/libstd.2 (system library - not scanned)
Shared library: /usr/lib/pa20_64/libc.2 (system library - not scanned)

Compiler footprint summary

Number of compilation units: 0

Earliest compiler timestamp: N/A
Latest compiler timestamp: N/A
Linker timestamp: N/A

Total size: 520.9K (text) 19.3K (data)

ensnada6:/arcsde/sdeexe91/bin#elfdump -dc gsrvr.shared

gsrvr.shared:

Compilation Unit Dictionary from gsrvr.shared:

Language Name
Product_id ; Version_id
Compile time; Source time

[not present]

Regards.

Re: Why so many threads?

Dennis Handly — Tue, 27 Mar 2007 21:53:38 GMT

>I guess the fact that it can't find any compilation unit is because the program was not compiled with an HP compiler? Or maybe some compiler switch disabled footprint in the object file?

The executable has probably been stripped. Since you do have libCsup, it was compiled with aC++.

Re: Why so many threads?

Dennis Handly — Tue, 27 Mar 2007 21:56:44 GMT

>The executable has probably been stripped.

Hmm, my strip won't work on ELF files. While I've filed a bug report on that, I wasn't aware it was fixed yet.

Re: Why so many threads?

Jose M. del Rio — Wed, 28 Mar 2007 02:57:45 GMT

Thanks for your answer but I can't understand what 'strip' means in this context.
Regards.

Re: Why so many threads?

Dennis Handly — Wed, 28 Mar 2007 03:13:38 GMT

>but I can't understand what 'strip' means in this context.

Good question. For PA32, you can use strip(1) on an executable. This will remove the compunit/notes sections that footprints/odump/elfdump use.

If you use file(1) on gsrvr.shared, does it say it is stripped?

Re: Why so many threads?

Jose M. del Rio — Wed, 28 Mar 2007 03:35:27 GMT

Thanks for your clarification.
Here is the output from file:
ensnada6:/arcsde/sdeexe91/bin#file gsrvr.shared
gsrvr.shared: ELF-64 executable object file - PA-RISC 2.0 (LP64)
ensnada6:/arcsde/sdeexe91/bin#

Re: Why so many threads?

Dennis Handly — Wed, 28 Mar 2007 04:09:28 GMT

>gsrvr.shared: ELF-64 executable object file - PA-RISC 2.0 (LP64)

This isn't stripped so perhaps you were right about not being compiled with a HP compiler.

Unless they know about an ELF tool that can do selective stripping of sections?

It seem strange that it is using libCsup and libstd but that may be for the benefit of any shlibs they dynamically load?

Re: Why so many threads?

Jose M. del Rio — Wed, 28 Mar 2007 04:21:40 GMT

Don't worry.
It's up to them to clarify why they are including CPSlib.
Thank you very much for your help.
I've learnt a lot.