Out of memory?

[Steven Richards]

Hi guys,

I was wondering if someone can help me, I'm working on an 11.23 PA HP-UX (64 x 1ghz CPU, 164gb Memory) server and a customer has come to me with the following fault "An error is being thrown from the perl script. I think the perl process is is unable to hold 3M customers in the perl hash and is quitting with out of memory"

Is there any kernel param or such that would limit the amount of memory the perl process could have any help or ideas greatly appreciated?

I looked at ulimit for the user and memory was set to unlimited, and according to glance the servers physical memory wasn't a problem. Couldn't think of anywhere else to look.

Many thanks, Steve.

[Anshumali]

Not Sure....but you might want to look at:

maxssiz_64bit
maxtsiz_64bit
maxdsiz_64bit

[Steven Richards]

Thanks for the reply.

The parameteres you mention seem to make sense...

maxtsiz specifies the maximum shared-text segment size in bytes.

maxtsiz specifies the maximum shared-text segment size in bytes

maxdsiz_64bit specify the maximum data segment size, in bytes, for an executing process.

These are currently set on this system to:-


maxdsiz_64bit 2147483647 (2gb?)
maxssiz_64bit 1073741824 (1gb?)
maxtsiz_64bit 1073741824 (1gb?)

They are all dynamic, and I guess they would all need increasing at the same time?

Cheers, Steve.

[Dennis Handly · Accepted solution]

>These are currently set on this system to:
>maxdsiz_64bit 2147483647 (2gb?)
>I guess they would all need increasing at the same time?

Only maxdsiz_64bit needs to be increased.
Also is your perl a 64 bit application? If not, you need to also increase maxdsiz.

[Hein van den Heuvel]

I never studied this in depth, but you may want to check the details on your perl build using : perl -V
Check for: use64bitall
Is is defined or undef?

Also dealing with , 3 M records may warrant you to rethink the algoritme used and its details such as hash-key-size and amount-of-data-saved per key. How big are the keys and values on average? Round up to say 32 bytes boundary (just guessing) and multiply by 3,000,000. How much data are we talking about?

Can you alter the source to pause every 500,000 records loaded and check the memory used by the process?

You may want to use a work file, or hash on a file, or make the hash be lseek offsets back onto the original data intead of the records itself.

Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[Bill Hassell]

Generally, maxdsiz_64 is set to several GB such as 4GB or 8GB and maxdsiz is set to 2GB. These are not defaults so maxdsiz is likely a few hundred megs. And as mentioned, your Perl code would have to be 64bit to take advantage of the maxdsiz_64 limit.

[Steven Richards]

Hi Dennis /Hein / Bill,

Thanks for your replies.

I checked 'Perl -V' and use64bitall is undef, which I assume means that maxdsiz is the key kernel param now. This is currently set to 1073741824 (1gb if my maths is correct!).

Thanks for all the specific Perl stuff, unfortunately I have no experience of Perl at all, I will pass this on to the customer who has reported the issue and see if it is of any help.

Many thanks again. Steve.

[Dennis Handly]

>I checked 'Perl -V' and use64bitall is undef, which I assume means that maxdsiz is the key kernel param now.

Yes, you can increase it to 4 Gb but you probably can't use more than 3, if you use chatr +q3p enable +q4p enable.
To get the last 1 Gb you must link perl with -N.