topic Re: Problem in awk in Operating System - HP-UX

Problem in awk

Mary Rice — Fri, 11 Jan 2002 18:02:14 GMT

Hello,

I am trying to convert a large amount of data in flat ASCII files
for later insertion into an Informix database. I am trying to use awk to massage the data. I keep getting this error message:
awk: Input line .... cannot be longer than 3,000 bytes.
The source line number is 17.

I have looked at the man pages for awk and I don't see any argument that allows me to increase this limit. Am I missing something?

Please help, Mary

Re: Problem in awk

A. Clay Stephenson — Fri, 11 Jan 2002 18:07:15 GMT

Hi Mary:

That question has come up before and the answer is to use the Gnu version of awk 'gawk' which removes those restrictions. You can download an executable version from any of the HP-UX Porting Centre's.

http://hpux.cs.utah.edu/hppd/hpux/Gnu/gawk-3.1.0/

You will find that gawk is compatible with your version of awk and is in fact a superset of awk. Download it and you should be all set.

Regards, Clay

Re: Problem in awk

Sanjay_6 — Fri, 11 Jan 2002 18:12:14 GMT

Hi Mary,

Try this link,

http://us-support.external.hp.com/cki/bin/doc.pl/sid=68d111b104dddf4e8b/screen=ckiDisplayDocument?docId=200000049016754

This is what this doc says,

1). Set large_ncargs_enable to 1.
2). Rebuild the kernel.
3). Reboot the system.

Hope this helps.

Regds

Re: Problem in awk

John Palmer — Fri, 11 Jan 2002 18:12:27 GMT

Hi Mary,

awk has a line limit of 3000 bytes.

You could try installing Gnu awk which can be configured to have a larger buffer.

You can get gawk here:
http://hpux.cs.utah.edu/hppd/hpux/Gnu/gawk-3.1.0/

Regards,
John

Re: Problem in awk

Mary Rice — Fri, 11 Jan 2002 19:16:42 GMT

Hello,

Sanjay, I can't find the kernel parameter that you suggested. How do I increase it? I am running HP-UX 11.0 32-bit on an HP9000 D-380. Do I need a patch?

Thank you, Mary

Re: Problem in awk

James R. Ferguson — Fri, 11 Jan 2002 19:34:58 GMT

Hi Mary:

I disagree with Sanjay's solution. The 'ncargs_enable' applied to 10.x and dealt with allowing longer parameters or lists (argument space). This is intrinsically enabled in 11.x releases.

This does *not* relate to 'awks' internal limitation of 3000-bytes per line.

The suggestion to use a GNU version of 'awk' as provided by Clay and by John is the appropriate answer.

Regards!

...JRF...

Re: Problem in awk

Sanjay_6 — Fri, 11 Jan 2002 19:37:36 GMT

Hi Mary,

Try this patch,

http://us-support.external.hp.com/wpsl/bin/doc.pl/screen=wpslDisplayPatch/sid=b6df6eb510848343a1?PATCH_PATH=/hp-ux_patches/s700_800/11.X/PHCO_23876&HW=s800&OS=11.00

Hope this helps.

Regds

Re: Problem in awk

A. Clay Stephenson — Fri, 11 Jan 2002 21:17:30 GMT

Hi Mary:

You can patch the kernel and SAM and you can try to play with ncargs till the cows come home and you are still going to be facing the INHERENT limit of awk. This problem has absolutely nothing to do with Sanjay's 'solution'. First of all, it does not apply to an 11x box. Secondly, his solution which he obviously didn't bother to examine applies to environment space specifically to exec() system calls. Sanjay has been the unfortunate victim of a search engine - it returned just what he asked. It returned quite a bit of data but with very little relevance. One might say that there is a signal/noise ratio problem here. I suppose that I am most disturbed that after JRF's posting, Sanjay went ahead and listed a link to patches. By all means, if you need current patches apply them but not to fix this problem.

Install gawk and I assure you that you will have no trouble processing your input.

Regards, Clay

Re: Problem in awk

Sanjay_6 — Fri, 11 Jan 2002 21:55:31 GMT

Hi JRF and Clay,

Can you post your contact info. I think we should have a talk.

Regds
Sanjay

Re: Problem in awk

Mary Rice — Fri, 11 Jan 2002 21:59:56 GMT

Hello everyone and thank you. I was reluctant to install non-HP software but finally I tried Clay's and John's advice and installed the Gnu version of awk. It worked the first time!

Thank you, Mary

Re: Problem in awk

Patrick Wallek — Fri, 11 Jan 2002 22:02:11 GMT

What do you need contract info. for Sanjay?

I have a feeling you may be outnumbered here. I happen to agree with JRF and A. Clay 100%.

Re: Problem in awk

Jim Turner — Fri, 11 Jan 2002 22:10:57 GMT

Hi Sanjay,

No crown here, so take this FWIW . . .

I too have grown quite weary of seeing your knee-jerk, link-infested replies. Hell, I can teach my 8-year-old son to run a search engine and rack up forum points. But that's not the "point", is it?

If you know the answer, state it plainly so that the person who posted the question doesn't have to go on some demented wild goose chase from link to link to link.

Is your interest in helping fellow admins or just amassing points?

Regards,
Jim

Re: Problem in awk

Mary Rice — Mon, 14 Jan 2002 21:27:25 GMT

Hello again everyone. As part of this same project to insert voter data into a program written in Informix ESQL/C, I have been asked to add a 'Sound Index' to the scrolling index. The idea is that names like 'Tomas' and 'Thomas' will appear near each other. I've never heard of a 'Sound Index'. Is there such a thing?

Thanks for all your help so far.

Mary

Re: Problem in awk

A. Clay Stephenson — Mon, 14 Jan 2002 21:31:36 GMT

Hello again Mary,

Yes, there is such a thing (Soundex) which generates a four-character string (A100-Z656) off the top of my head from a list of name. It would proably be best if your start a new thread with this as this does not pertain to awk and others looking for similar solutions might be sent astray. The good news is that I have such a function and will post it if you submit a new thread.

Regards, Clay

Re: Problem in awk

Mary Rice — Mon, 14 Jan 2002 21:34:49 GMT

Hi Clay and thank you. I am new to the Forums and didn't know the proper procedure. I will repost my second request.

Mary

Re: Problem in awk

James R. Ferguson — Mon, 14 Jan 2002 21:50:27 GMT

Hi Mary:

Absolutely! We've use one for years, although it's written in Algol for a Unisys server.

The general idea is to reduce alphabetic strings into simple similar keys that can then be used in searching and matching.

For instance, my name, "Ferguson" has several common spelling variations which "sound-alike". On the telephone you might spell it "Ferguson", "Furguson", "Furgason", "Furgasen", etc.

The algorithm we use retains the first letter; drops all vowels (a,e,i,o,u); and maps the remaining letters to a single digit.

All of the above variations of "Ferguson" would return a "soundex" key of F625.

Does this help?

Regards!

...JRF...