Automating changes in text files

[unixnewbie]

Hi alll,

I am new to unix,partcularly to shell scripting.I have an appplication which generates data files which I have to edit (using vi) and search for a line starting with a particular pattern (numeric or alphanumeric eg: "003" or "abc" 0r "a13" etc....).If i find that particular line I have check whether the next line starts with another particular pattern and if it is NOT I need to combine both the lines in to one (ie, no CR or "enter") in between.I tried with sed and some scripting but still no progress. I will be extremely happy if I get some script in which I can pass on the file name and the patterns to be searched and have the file changed (as I have a lot ).Please help .Thanks in advance

[Dennis Handly]

It would probably be simpler to do it in awk where you can actually program.
awk '
/^003/ || /^abc/ || /^a13/ {
printf "%s", $0
getline
if ($0 ~ "^next" )
print "" # newline
}
{ print } ' file

If you want the patterns to be variables:
awk -v p1="^003" -v p2="^abc" -v p3="^a13" -v p4="^next" '
$0 ~ p1 || $0 ~ p2 || $0 ~ p3 {
printf "%s", $0
getline
if ($0 ~ p4)
print "" # newline
}
{ print } ' file

And since you are using EREs, you can combine the N patterns:
awk -v p1="^(003)|(abc)|(a13)" -v p4="^next" '
$0 ~ p1 {
printf "%s", $0
getline
if ($0 ~ p4)
print "" # newline
}
{ print } ' file

[unixnewbie]

Dear dennis,

Thanks for that. But please clarify the following (may be too silly..)
1) Can I just type in all this to file and run that as a script ?
2) will the same work on Linux also ??
3) Can u suggest changes so that I can use it as a script and I can run it just by typing

./script

eg: ./scr myfile 000 003

and all the lines in myfile starting with 000 will combine with the next line if it is NOT starting with 003.(other lines no changes)

Thanks again,

[john korterman]

Hi,

try running the attached as you have suggested; it may or may not work.

regards,
John K.

[Hein van den Heuvel]

Here is a perl one-liner that works for the problem as stated

$ perl -e '$a=shift; $b=shift; while (<>) {if (/^$a/){chomp;print;$_=<>;$_="\n".$_ if /^$b/} print}
'


It user "^" to anchor the match pattern to the beginning of lines as specified.
Questions to ask yourself.

- What to do if there are mutliple starts
- Glue together just one next line or everything up to the next end
- What to do if there is no end in sight
- What to do if there extra ends
- Can the start line ever be a last line in a file.

Here is the perl script in slow motion:

$ perl -e ' ' # start perl with scrip in text
$a=shift; # pick up start pattern from command line
$b=shift; # pick up stop
while (<>) { # loop through input file
if (/^$a/) { # looks like the trigger?
{chomp; # prepare by removing new-line
print; # pritn first chunk already
$_=<>; # read next chunk
$_="\n".$_ # pre-pend a new-line only...
if /^$b/ # if end pattern seen
}
print # print the current line
} # loop


Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[Sandman!]

Try the sed construct below:

# sed '/^000/N;/.*\n003/s/\n/ /p' file

~cheers

[Sandman!]

To execute sed construct from within a shell script do as shown:

# ./uxscript file 000 003

...contents of uxscript are listed below.

======================= uxscript =======================

#!/usr/bin/sh

sed "/^$2/N;/.*\n$3/!s/\n/ /p" $1

[Dennis Handly]

>1) Can I just type in all this to file and run that as a script?

Yes.

>2) will the same work on Linux also??

It should.

>3) Can you suggest changes so that I can use it as a script and I can run it just by typing
./script
eg: ./scr myfile 000 003

Note the current script just creates a new file, if you want to update it, you would need to rename it.

Ah, it is much easier than what I thought you needed.

#!/bin/sh
# Usage: $0 file pat1 pat2
awk -v p1="^$2" -v p2="^$3" '
$0 ~ p1 {
printf "%s", $0
getline
if ($0 ~ p2)
print "" # newline
}
{ print } ' $1

>Sandman: To execute sed construct from within a shell script do as shown:

Yes, I thought about using sed like that. I thought I had to look for N patterns at the same time.
Your script inserts a blank, mine doesn't.

[unixnewbie]

Sandman, The sed construct on execution shows the result but the file is not changing. If I redirect the o/p to another file ./sand myfile 000 003 > myfile1 I get what I want but the line replicates.ie, it identifies the lines, joines them perfect, but it will add one more line in myfile1 (a duplicate the same joined line).Any changes ?? ..Thanks..

Handly, I tried ur script in Linux (#!/bin/sh)(not yet on HP)
./denn newfile orgfile 000 003.
If I am not wrong the newfile should contain what I need.But it is still 0Kb.Please advise. Thank you

John and Hein,I will try your suggestions and get back. Thanks a lot....

[unixnewbie]

sandman sorry I forgot sed don't change the input file.

[Dennis Handly]

>but it will add one more line in myfile1 (a duplicate the same joined line). Any changes??

It works fine for me. Are you running it on Linux? Perhaps that useless "p" needs to be removed there?
sed "/^$2/N;/.*\n$3/!s/\n/ /" $1

>I tried your script in Linux (#!/bin/sh)
./denn newfile orgfile 000 003

Oops, you invoke it:
./denn orgfile 000 003 > newfile

That "Usage: $0" comment was an indication that it prints: ./denn file pat1 pat2

[unixnewbie]

Dear Dennis, It was on Linux and YES (p)that solves it.Your script (forgive me for the troubles.. ) also does same thing but the output file is a little smaller in size than the input and in the case of sed script both are equal in size.........???
Great help 4 me.. Thanks.......

[Dennis Handly · Accepted solution]

>but the output file is a little smaller in size than the input and in the case of sed script both are equal in size???

Yes, I mentioned that when I was commenting on Sandman's script:
Your script inserts a blank, mine doesn't.

If you want that blank, then use:
if ($0 ~ p2)
print "" # newline
else
printf " " # blank

If you don't want that blank:
sed "/^$2/N;/.*\n$3/!s/\n//" $1

[unixnewbie]

Yes dennis. I think that finishes it.If u don't mind can u suggest me link where I can find out myself what that little sed contruct does........Thanks..

Thank you sandman

[Dennis Handly]

>If you don't mind can you suggest me link where I can find out myself what that little sed construct does.

Well, you can look at sed(1):
http://docs.hp.com/en/B2355-60130/sed.1.html

Basically what it does is:
sed "/^$2/N;/.*\n$3/!s/\n//" $1
1) Finds a line that starts with $2.
2) N appends the next line into the existing line with an embedded newline.
3) Then finds a line where it doesn't (!) have a newline followed by $3. And if it doesn't, it substitutes (s) nothing, for that newline.
4) Then it writes those 2 lines out.
5) Other lines are just read and written.

HP-UX used to have a hard copy manual with more details about sed. I don't think it was ever moved to electronic form. google did find the Tru64 version:
http://h30097.www3.hp.com/docs/base_doc/DOCUMENTATION/HTML/AA-PS32D-TET1_html/pst4.html#sed

[James R. Ferguson]

Hi:

For a 'sed' tutorial, you might find the GNU manuals helpful:

http://www.gnu.org/software/sed/manual/sed.html

Regards!

...JRF...

[Hein van den Heuvel]

For completeness...

The perl solution as suggested gets a little ugly when adding a space:

$ perl -e '$a=shift; $b=shift; while (<>) {if (/^$a/){chomp; $n=<>; $_ .= (($n =~/^$b/)? "\n" : " "
) . $n} print}' 100 is x.txt' old > new

So a better perl solution would be:

$ perl -e '$a=shift; $b=shift; while (<>) {if (/^$a/){$_ .= <>; s/\n/ / unless /\n$b/} print}' 100
edis x.txt
old > new


The core of that in slow motion:

if (/^$a/){ # start pattern?
$_ .= <>; # add next line to current
s/\n/ / # replace new-line in middle...
unless /\n$b/} # unless next started with end-pattern

fwiw,
Hein.

[Sandman!]

Under Linux the sed(1) substitution cmd takes the optional "p" switch which works if the "-n" cmd line option is given otherwise it produces a duplicate line for every one that gets sucessfully substituted. This does not happen on HP-UX and Dennis' post has explained the sed construct very well. Also checkout the sed tutorials at the "seders grab bag" :)

http://sed.sourceforge.net/grabbag/tutorials/

~cheers

[unixnewbie]

Thanks everybody