Problem in awk

[Mary Rice]

Hello,

I am trying to convert a large amount of data in flat ASCII files
for later insertion into an Informix database. I am trying to use awk to massage the data. I keep getting this error message:
awk: Input line .... cannot be longer than 3,000 bytes.
The source line number is 17.

I have looked at the man pages for awk and I don't see any argument that allows me to increase this limit. Am I missing something?

Please help, Mary

[A. Clay Stephenson · Accepted solution]

Hi Mary:

That question has come up before and the answer is to use the Gnu version of awk 'gawk' which removes those restrictions. You can download an executable version from any of the HP-UX Porting Centre's.

http://hpux.cs.utah.edu/hppd/hpux/Gnu/gawk-3.1.0/

You will find that gawk is compatible with your version of awk and is in fact a superset of awk. Download it and you should be all set.

Regards, Clay

[Sanjay_6]

Hi Mary,

Try this link,

http://us-support.external.hp.com/cki/bin/doc.pl/sid=68d111b104dddf4e8b/screen=ckiDisplayDocument?docId=200000049016754

This is what this doc says,


1). Set large_ncargs_enable to 1.
2). Rebuild the kernel.
3). Reboot the system.

Hope this helps.

Regds

[John Palmer]

Hi Mary,

awk has a line limit of 3000 bytes.

You could try installing Gnu awk which can be configured to have a larger buffer.

You can get gawk here:
http://hpux.cs.utah.edu/hppd/hpux/Gnu/gawk-3.1.0/

Regards,
John

[Mary Rice]

Hello,

Sanjay, I can't find the kernel parameter that you suggested. How do I increase it? I am running HP-UX 11.0 32-bit on an HP9000 D-380. Do I need a patch?

Thank you, Mary

[James R. Ferguson]

Hi Mary:

I disagree with Sanjay's solution. The 'ncargs_enable' applied to 10.x and dealt with allowing longer parameters or lists (argument space). This is intrinsically enabled in 11.x releases.

This does *not* relate to 'awks' internal limitation of 3000-bytes per line.

The suggestion to use a GNU version of 'awk' as provided by Clay and by John is the appropriate answer.

Regards!

...JRF...

[Sanjay_6]

Hi Mary,

Try this patch,

http://us-support.external.hp.com/wpsl/bin/doc.pl/screen=wpslDisplayPatch/sid=b6df6eb510848343a1?PATCH_PATH=/hp-ux_patches/s700_800/11.X/PHCO_23876&HW=s800&OS=11.00

Hope this helps.

Regds

[A. Clay Stephenson]

Hi Mary:

You can patch the kernel and SAM and you can try to play with ncargs till the cows come home and you are still going to be facing the INHERENT limit of awk. This problem has absolutely nothing to do with Sanjay's 'solution'. First of all, it does not apply to an 11x box. Secondly, his solution which he obviously didn't bother to examine applies to environment space specifically to exec() system calls. Sanjay has been the unfortunate victim of a search engine - it returned just what he asked. It returned quite a bit of data but with very little relevance. One might say that there is a signal/noise ratio problem here. I suppose that I am most disturbed that after JRF's posting, Sanjay went ahead and listed a link to patches. By all means, if you need current patches apply them but not to fix this problem.

Install gawk and I assure you that you will have no trouble processing your input.

Regards, Clay

[Sanjay_6]

Hi JRF and Clay,

Can you post your contact info. I think we should have a talk.

Regds
Sanjay

[Mary Rice]

Hello everyone and thank you. I was reluctant to install non-HP software but finally I tried Clay's and John's advice and installed the Gnu version of awk. It worked the first time!

Thank you, Mary

[Patrick Wallek]

What do you need contract info. for Sanjay?

I have a feeling you may be outnumbered here. I happen to agree with JRF and A. Clay 100%.

[Jim Turner]

Hi Sanjay,

No crown here, so take this FWIW . . .

I too have grown quite weary of seeing your knee-jerk, link-infested replies. Hell, I can teach my 8-year-old son to run a search engine and rack up forum points. But that's not the "point", is it?

If you know the answer, state it plainly so that the person who posted the question doesn't have to go on some demented wild goose chase from link to link to link.

Is your interest in helping fellow admins or just amassing points?

Regards,
Jim

[Mary Rice]

Hello again everyone. As part of this same project to insert voter data into a program written in Informix ESQL/C, I have been asked to add a 'Sound Index' to the scrolling index. The idea is that names like 'Tomas' and 'Thomas' will appear near each other. I've never heard of a 'Sound Index'. Is there such a thing?

Thanks for all your help so far.

Mary

[A. Clay Stephenson]

Hello again Mary,

Yes, there is such a thing (Soundex) which generates a four-character string (A100-Z656) off the top of my head from a list of name. It would proably be best if your start a new thread with this as this does not pertain to awk and others looking for similar solutions might be sent astray. The good news is that I have such a function and will post it if you submit a new thread.

Regards, Clay

[Mary Rice]

Hi Clay and thank you. I am new to the Forums and didn't know the proper procedure. I will repost my second request.

Mary

[James R. Ferguson]

Hi Mary:

Absolutely! We've use one for years, although it's written in Algol for a Unisys server.

The general idea is to reduce alphabetic strings into simple similar keys that can then be used in searching and matching.

For instance, my name, "Ferguson" has several common spelling variations which "sound-alike". On the telephone you might spell it "Ferguson", "Furguson", "Furgason", "Furgasen", etc.

The algorithm we use retains the first letter; drops all vowels (a,e,i,o,u); and maps the remaining letters to a single digit.

All of the above variations of "Ferguson" would return a "soundex" key of F625.

Does this help?

Regards!

...JRF...