Huge memory allocation fails on HP 64 Bit Itanium.

[scoolj]

Hi!

I wrote a test program to allcoate memory more than 4GB. But from the results, I understand it cant allocate more than 4GB, due to current OS configuration.

Memory test results below.

bash-3.00$ kctune | grep 64
...
maxdsiz_64bit 4294967296 Default Immed
maxssiz_64bit 268435456 Default Immed
maxtsiz_64bit 1073741824 Default Immed
...

top before running the memtest
System: unknown Tue May 20 15:39:29 2008
Load averages: 0.00, 0.01, 0.01
169 processes: 145 sleeping, 22 running, 1 stopped, 1 zombie
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.01 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
1 0.00 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.00 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%

Memory: 105696K (65192K) real, 566584K (346876K) virtual, 1042560K free Page# 1/7 <------- Free memory
...

Use memtest to allocate 4100MB(4+ GB)
bash-3.00$ ./memtest 4100
4294967296 Bytes allocated - Cant allocate beyond <------- As expected, allocated only 4GB
De-allocating memory.... Hit enter to proceed...

top after memtest allocated the memory & before memtest deallocated the memory
System: unknown Tue May 20 15:40:44 2008
Load averages: 0.00, 0.01, 0.01
170 processes: 146 sleeping, 22 running, 1 stopped, 1 zombie
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.00 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
1 0.00 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.00 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%

Memory: 171940K (117552K) real, 4758500K (4462344K) virtual, 975044K free Page# 1/7 <------- Free memory decreased

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
1 ? 48 root 152 20 3616K 3616K run 26:36 0.78 0.78 vxfsd
1 pts/2 16931 jaltaru 154 20 4195M 65784K sleep 0:00 1.35 0.40 memtest <------- memtest already using 4GB memory.
0 ? 1524 root 152 20 57912K 5208K run 1:48 0.20 0.20 cimserver
...

top after terminating the memtest
System: unknown Tue May 20 15:41:24 2008
Load averages: 0.00, 0.01, 0.01
169 processes: 145 sleeping, 22 running, 1 stopped, 1 zombie
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.00 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
1 0.00 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.00 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%

Memory: 106672K (53832K) real, 564836K (278096K) virtual, 1040328K free Page# 1/7 <------- Free memory increased
...


I believe the SIZE column in the top output is actually showing the AMOUNT OF MEMORY CURRENTLY GOT ALLOCATED and in USE by the memory test program.

If I am correct, only way to allocate more than 4GB is to install extra memory and increase maxdsiz_64bit accordingly.

Pls validate and suggest if you have a different opinion.

Br,
Srini

[Don Morris_1]

Yes, that's what maxdsiz_64bit *does* for 64-bit processes (man 5 maxdsiz) assuming this is just malloc() [or sbrk() or whatnot.. data segment, anyway].

So logically enough -- if you must go above it, you have to raise it.

SIZE is [from man top -- don't "believe", there are man pages for this stuff.]:

"Total virtual size of the process in kilobytes. This includes virtual sizes of text, data, stack, mmap regions, shared memory regions and IO mapped regions. This may also include virtual memory regions shared with other processes."

So no -- it isn't exactly showing the amount of memory currently allocated and in use, it is showing *more* (because you have to include all the other stuff than just your data here).

On the final point -- this is *virtual* memory in use, not physical... unless you also reference the memory (write to the resulting pointer, read (garbage if you don't write first), etc.) then there's no backing RAM involved. And you very obviously don't because the RES field for the same process is only 65784Kb. (RES is the physical consumption of the entire process, SIZE is the virtual consumption).

So no, you don't need to install extra memory. Virtual memory requires _swap_ reservation. If "swapinfo -atm" shows that you have much more than the amount you want to snarf up [and you aren't telling us why you want to do this in the first place... but assuming you have a good reason] then raising maxdsiz_64bit will suffice. Being IPF, that means 11iv2 or v3, so this is a dynamic tunable... just raise it and run your test again... should be fine (until you run out of swap or hit the new limit).

Standard caveats [and this is in the man page] -- running the system out of swap as a test is not polite and is the reason the tunable exists to limit this type of memory pig. All new virtual objects (barring few special cases) are required to reserve swap.. so if you eat it all, new malloc() / fork() /etc. calls will be failed. Which can result in a system that's rather hard to use until your test program goes away.

Additional caveat -- if you try to use the equivalent amount of physical memory, you could push the system into real swapping and cause lousy performance. Don't do it unless this is your own box and you don't care.

[Bill Hassell]

And so you can see the current limits in an easier display, I have attached a simple program to allocate memory until the kernel says: no more. This program does not actually write the pages so it runs very fast. In the next posting from me, there is a program that asks you how much memory you would like to add and after getting the allocation, writes a simple pattern which forces the kernel to put the pages into real RAM.

This one program can be compiled 5 different ways including a 64bit program in order to demonstrate the limitations for 32 and 64 bit code.

See next attachment.

[Bill Hassell]

Oops, the previous attachment writes data into the allocated memory. This one just requests the RAM but leaves it untouched. This program will run very quickly and stop when malloc fails.

The previous one will use a *lot* of kernel and user CPU time as you add large chunks (500 megs or larger) and will create a siginificant impact on the entire system as you reach certain thresholds. The biggest impact will be as you exceed available memory and start real swapping. At that point, the system will begin to freeze and virtual memory is rearranged for new data. It is great program to demonstrate the impact of really large memory programs. Note that maxdsiz (maxdsiz_64 also) and maximum swap space will limit the 64 bit program. The enclosed programs compiled in 64 bit mode can allocate terabytes of memory -- if you are patient and set the limit high enough and have enough swap reservation space.

[scoolj]

Thanks.

Memory test program that I coded takes MB as parameter, but it actually allocates in
splits (1 MB each) in a loop. So I was not trying to allocate in one shot/chunk.

I did printf the total memory so far allocated by memtest program, whenever I request more than 4GB allocation, it just responds as...
"4294967296 Bytes allocated - Cant allocate beyond"

QUESTION: Ok, if SIZE is the total virtual size of a program, Is there any other way to list "dynamically allocated memory" of a running program.

Meanwhile, I will do some tests with the test programs attached to this log and also examine the swapinfo.

Br,
Srini

[Don Morris_1]

pstat_getprocvm() [man 2 pstat] will report on virtual objects within a running process, PS_DATA is the type for the dynamic data.

Sample code attached.

[Srimalik]

How did you compile your program???
please provide the command

[Dennis Handly]

>Srikrishan: How did you compile your program???

Are you asking about Don's? Probably +DD32 -D_PSTAT64 or +DD64 option.

[scoolj]

I used only +DD64 flag to compile the program. Anything else was required or missing ?

Br,
Srini

[Dennis Handly]

>I used only +DD64 flag to compile the program. Anything else was required or missing?

No, only +DD64 and a large enough maxdsiz_64bit are all you need.
If you are on IPF, you should use +w64bit to make sure you don't have any porting issues.