1833824 Members
2218 Online
110063 Solutions
New Discussion

Re: Out of memory?

 
SOLVED
Go to solution
Steven Richards
Frequent Advisor

Out of memory?

Hi guys,

I was wondering if someone can help me, I'm working on an 11.23 PA HP-UX (64 x 1ghz CPU, 164gb Memory) server and a customer has come to me with the following fault "An error is being thrown from the perl script. I think the perl process is is unable to hold 3M customers in the perl hash and is quitting with out of memory"

Is there any kernel param or such that would limit the amount of memory the perl process could have any help or ideas greatly appreciated?

I looked at ulimit for the user and memory was set to unlimited, and according to glance the servers physical memory wasn't a problem. Couldn't think of anywhere else to look.

Many thanks, Steve.
A pint of vodka my good woman
7 REPLIES 7
Anshumali
Esteemed Contributor

Re: Out of memory?

Not Sure....but you might want to look at:

maxssiz_64bit
maxtsiz_64bit
maxdsiz_64bit
Dreams are not which you see while sleeping, Dreams are which doesnt allow you to sleep while you are chasing for them!!
Steven Richards
Frequent Advisor

Re: Out of memory?

Thanks for the reply.

The parameteres you mention seem to make sense...

maxtsiz specifies the maximum shared-text segment size in bytes.

maxtsiz specifies the maximum shared-text segment size in bytes

maxdsiz_64bit specify the maximum data segment size, in bytes, for an executing process.

These are currently set on this system to:-


maxdsiz_64bit 2147483647 (2gb?)
maxssiz_64bit 1073741824 (1gb?)
maxtsiz_64bit 1073741824 (1gb?)

They are all dynamic, and I guess they would all need increasing at the same time?

Cheers, Steve.
A pint of vodka my good woman
Dennis Handly
Acclaimed Contributor
Solution

Re: Out of memory?

>These are currently set on this system to:
>maxdsiz_64bit 2147483647 (2gb?)
>I guess they would all need increasing at the same time?

Only maxdsiz_64bit needs to be increased.
Also is your perl a 64 bit application? If not, you need to also increase maxdsiz.
Hein van den Heuvel
Honored Contributor

Re: Out of memory?

I never studied this in depth, but you may want to check the details on your perl build using : perl -V
Check for: use64bitall
Is is defined or undef?

Also dealing with , 3 M records may warrant you to rethink the algoritme used and its details such as hash-key-size and amount-of-data-saved per key. How big are the keys and values on average? Round up to say 32 bytes boundary (just guessing) and multiply by 3,000,000. How much data are we talking about?

Can you alter the source to pause every 500,000 records loaded and check the memory used by the process?

You may want to use a work file, or hash on a file, or make the hash be lseek offsets back onto the original data intead of the records itself.

Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting
Bill Hassell
Honored Contributor

Re: Out of memory?

Generally, maxdsiz_64 is set to several GB such as 4GB or 8GB and maxdsiz is set to 2GB. These are not defaults so maxdsiz is likely a few hundred megs. And as mentioned, your Perl code would have to be 64bit to take advantage of the maxdsiz_64 limit.


Bill Hassell, sysadmin
Steven Richards
Frequent Advisor

Re: Out of memory?

Hi Dennis /Hein / Bill,

Thanks for your replies.

I checked 'Perl -V' and use64bitall is undef, which I assume means that maxdsiz is the key kernel param now. This is currently set to 1073741824 (1gb if my maths is correct!).

Thanks for all the specific Perl stuff, unfortunately I have no experience of Perl at all, I will pass this on to the customer who has reported the issue and see if it is of any help.

Many thanks again. Steve.
A pint of vodka my good woman
Dennis Handly
Acclaimed Contributor

Re: Out of memory?

>I checked 'Perl -V' and use64bitall is undef, which I assume means that maxdsiz is the key kernel param now.

Yes, you can increase it to 4 Gb but you probably can't use more than 3, if you use chatr +q3p enable +q4p enable.
To get the last 1 Gb you must link perl with -N.