Grep acting different between HPUX 10.20 and HPUX 11.11

[Tony Kuehn]

Ok, I have a application file that is coming from DOS and grep is treating it differently between HPUX 10.20 and HPUX 11.11. The DOS file is created from a third party application and I was told I could not modify the file!! (Believe me I have tried to get the app team to understand that would be the easiest solution!) The file ends without a newline character so if you cat the file both 10.20 and 11.i will print the output and then append the command prompt after that. This is the same format for the output with 10.20, however, with 11.i it inserts a newline for you so the output from the grep is separate from the command prompt. Here is what how the output looks.
########### HPUX 10.20 ##############
dev1:/tmp>cat me5
helloappdev1:/tmp>

dev1:/tmp>grep lo me5
helloappdev1:/tmp>

############### HPUX 11.i ############
dev2:/tmp>cat me5
hellodv1csite:/tmp>

dev2:/tmp>grep lo me5
hello
dv1csite:/tmp>

I was wondering if anyone knows how to make the grep functionality act consistant (does matter which way the output looks as long as 10.20 and 11.i are consistant). Like I said they will not let me modify the input file before the grep. I also suggested using a Perl one liner "perl -ne 'print if //' filename. The application team did not like that idea either.
Please advise, if you have any suggestions for regular expersion for the grep command or ways to remove any/all DOS (or other) formating charaters from a line. Or if you have similair problem and your solution (patch etc...)
P.S. If you want to try this on your systems create a text file on Windows/DOS with notepad and FTP to 10.20 and 11.i servers. Do not have a return at the end of the last line.

Thanks!!!!!!

[Steven E. Protter]

grep is pretty basic and was probably not intentionally modified by the OS.

I'd check for patches, both 10.20 and 11.11 and see if there is anything for grep

SEP

[A. Clay Stephenson · Accepted solution]

Well, the most straightforward way to do this would be to copy the 10.20 version of grep to /usr/bin/ogrep (or something similar) on the 11.11 box. The converse (11.11 to 10.20 cannot be done).

[John Poff]

Hi,

Have you tried using the 'dos2ux' command on the file? Even if they won't let you modify it maybe you could use it to create a temporary file to use for your grep?

Just an idea.

JP

[Bill Hassell]

grep performs no formatting at all. It prints a line of text exactly as the line appears in the document. The assumption is that each line is alpha-numeric, terminated by a newline character (LF) and there are no binary control characters in the file. Now the real problem is that you (and the app developers) are trying to mix two completely different operating systems together, expecting their file formats to be the same, a very bad assumption.

A PC-based text file is:

text CR LF
more text CR LF
last line CR LF
CTRL-Z

but Unix text files are:

text LF
more text LF
last line LF
NUL

So the first question is: how did the file get from a PC to Unix? By ftp (then use the ASCII option when transferring), or by filesystem sharing as in SAMBA? If the latter, you will *always* have formatting issues because the two systems (PC and Unix) use different tools for the creation of the files and representation of the data. There is no reason to expect them to be compatible at all. However, HP-UX recognized this problem a long time ago, and provides bi-directional converter tools (dos2ux and ux2dos) to handle the differences.

As for the inconsistent behavior, this is a typical corner case that is undefined between versions of grep and there is no special option to add a newline if the last line contains CTRL-Z and no CR+LF pair. So you'll have to translate the file before using grep (which is sort of a special option):

dos2ux pc_file_name | grep

Most likely this will standardize the file format and should produce the same result on both HP-UX systems.

If not, grep was never designed as a formatting tool and a regular expression does not perform formatting, it is used only for pattern matching. With enough perl or awk code, you could probably make it work but the real issue is expecting internal data formats to be the same with foreign systems. That's why there is an ASCII option in ftp.

[H.Merijn Brand (procura]

Are you sure there are no `strange' characters in the file, like escapes, returns or newlines?

# od -x me5

And are the versions of grep comparable? iow, are you using HP's version of grep on both sides, or is either version GNU grep, behaving differently

# grep -V

GNU grep will return the version, HP's grep will return a usage message

Enjoy, Have FUN! H.Merijn

[Tony Kuehn]

Sorry, I forgot to mention that I did try dos2ux and this does not seem to help the problem. (I thought that would be answer, too!!) The file systems are "shared" via ASU so the application team does not transfer the files via ftp/sfpt.
I was able to copy a 10.20 grep binary and put it it /usr/bin/ as 1020grep and then put in alias in the users profile for grep to ran 1020grep (as this is what they wanted). Thanks for all your help/suggestions!!!!!