gawk error memory allocation error

[sirAmitSharma]

Hi,

i have following code

gawk -F, '{FS=","}
{
if (substr($1,3,1)=="-") array[$5","$1","$5","$6","substr($1,1,2)","$2","$7","$4]++
else arraytmp[$5","$1","$5","$6","$1","$2","$7","$4]++
}
END { for (counter in array) {print counter","array[counter] }
{ for (count in arraytmp) {print count","arraytmp[count]}} }
' Infile > OutFile

the Infile has more that 400 million records, by virute of this operation outfile will be reduce to 60million.

Error coming is:

gawk: cmd. line:4: (FILENAME=- FNR=8226772) fatal: more_blocks: freep: can't allocate 4000 bytes of memory (Not enough space)

Additional Information:

$ uname -a
HP-UX smsciuc0 B.11.31 U ia64 3651711977 unlimited-user license

$ swapinfo
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 67108864 0 67108864 0% 0 - 1 /dev/vg00/lvol2
reserve - 455428 -455428
memory 63799772 8378200 55421572 13%

$ kctune maxdsiz maxdsiz_64bit maxssiz maxssiz_64bit maxtsiz maxtsiz_64bit
Tunable Value Expression Changes
maxdsiz 1073741824 Default Immed
maxdsiz_64bit 4294967296 Default Immed
maxssiz 8388608 Default Immed
maxssiz_64bit 268435456 Default Immed
maxtsiz 100663296 Default Immed
maxtsiz_64bit 1073741824 Default Immed

PLZ HELP 

P.S. this thread has been moevd from General to HP-UX > languages. - Hp forum Moderator

[Bill Hassell]

I would bump maxdsiz_64bit from 4GB to 16GB.

This limit will allow programs to allocate large amounts of space.

[Dennis Handly]

>Infile has more than 400 million records, ... outfile will be reduced to 60 million.

Can you break up the file and then combine the results?

Is gawk a 32 or 64 program?  If the former you'll need to increase maxdsiz to 4 GB.

And you'll have to use chatr(1) to take advantage.

[sirAmitSharma]

Changed the same but still the error , reboot also done.... :(

[sirAmitSharma]

On using the kcusage found that the error comes when the usage is 1GB.
$ kcusage
Tunable Usage / Setting
=============================================
filecache_max 669466624 / 32665481216
maxdsiz 1054195712 / 4294963200
maxdsiz_64bit 46727168 / 17179869184

[Dennis Handly]

>Changed the same but still the error, reboot also done

Reboot not needed since kctune(1m) says "immediate".

As I mentioned above, chatr(1) or relinking with -N must be used to get > 1 GB in a 32 bit app.

What does this show: file /path-to-executable/gawk

These may help:

http://h30499.www3.hp.com/t5/White-Papers/32bits-virtual-addressing-and-Memory-Windows/m-p/2442383

http://h30499.www3.hp.com/t5/Languages-and-Scripting/A-simple-C-program-crashing-on-HP-machine/m-p/4919847#M32801

http://h30499.www3.hp.com/t5/Languages-and-Scripting/Out-of-memory/m-p/4935279#M33135

http://h30499.www3.hp.com/t5/Languages-and-Scripting/chatr-Quickie-How-to-Check-if-an-Executabale-is-chatr-d-to-use/m-p/3781049

http://h30499.www3.hp.com/t5/Languages-and-Scripting/chatr-Fails-to-requadrant-executable/m-p/3896400

http://h30499.www3.hp.com/t5/Languages-and-Scripting/Is-there-any-limitation-on-memory-a-child-process-can-use/m-p/4009315

http://h30499.www3.hp.com/t5/Languages-and-Scripting/chatr-and-C/m-p/3900953

The last one implies that if this is a Integrity executable, then you'll need to build and relink gawk with -N.