topic Re: Automating changes in text files in Operating System - Linux

Automating changes in text files

unixnewbie — Thu, 06 Sep 2007 02:31:23 GMT

Hi alll,

I am new to unix,partcularly to shell scripting.I have an appplication which generates data files which I have to edit (using vi) and search for a line starting with a particular pattern (numeric or alphanumeric eg: "003" or "abc" 0r "a13" etc....).If i find that particular line I have check whether the next line starts with another particular pattern and if it is NOT I need to combine both the lines in to one (ie, no CR or "enter") in between.I tried with sed and some scripting but still no progress. I will be extremely happy if I get some script in which I can pass on the file name and the patterns to be searched and have the file changed (as I have a lot ).Please help .Thanks in advance

Re: Automating changes in text files

Dennis Handly — Thu, 06 Sep 2007 04:33:36 GMT

It would probably be simpler to do it in awk where you can actually program.
awk '
/^003/ || /^abc/ || /^a13/ {
printf "%s", $0
getline
if ($0 ~ "^next" )
print "" # newline
}
{ print } ' file

If you want the patterns to be variables:
awk -v p1="^003" -v p2="^abc" -v p3="^a13" -v p4="^next" '
$0 ~ p1 || $0 ~ p2 || $0 ~ p3 {
printf "%s", $0
getline
if ($0 ~ p4)
print "" # newline
}
{ print } ' file

And since you are using EREs, you can combine the N patterns:
awk -v p1="^(003)|(abc)|(a13)" -v p4="^next" '
$0 ~ p1 {
printf "%s", $0
getline
if ($0 ~ p4)
print "" # newline
}
{ print } ' file

Re: Automating changes in text files

unixnewbie — Thu, 06 Sep 2007 07:22:21 GMT

Dear dennis,

Thanks for that. But please clarify the following (may be too silly..)
1) Can I just type in all this to file and run that as a script ?
2) will the same work on Linux also ??
3) Can u suggest changes so that I can use it as a script and I can run it just by typing

./script

eg: ./scr myfile 000 003

and all the lines in myfile starting with 000 will combine with the next line if it is NOT starting with 003.(other lines no changes)

Thanks again,

Re: Automating changes in text files

john korterman — Thu, 06 Sep 2007 08:02:21 GMT

Hi,

try running the attached as you have suggested; it may or may not work.

regards,
John K.

Re: Automating changes in text files

Hein van den Heuvel — Thu, 06 Sep 2007 10:01:27 GMT

Here is a perl one-liner that works for the problem as stated

$ perl -e '$a=shift; $b=shift; while (<>) {if (/^$a/){chomp;print;$_=<>;$_="\n".$_ if /^$b/} print}
'

It user "^" to anchor the match pattern to the beginning of lines as specified.
Questions to ask yourself.

- What to do if there are mutliple starts
- Glue together just one next line or everything up to the next end
- What to do if there is no end in sight
- What to do if there extra ends
- Can the start line ever be a last line in a file.

Here is the perl script in slow motion:

$ perl -e ' ' # start perl with scrip in text
$a=shift; # pick up start pattern from command line
$b=shift; # pick up stop
while (<>) { # loop through input file
if (/^$a/) { # looks like the trigger?
{chomp; # prepare by removing new-line
print; # pritn first chunk already
$_=<>; # read next chunk
$_="\n".$_ # pre-pend a new-line only...
if /^$b/ # if end pattern seen
}
print # print the current line
} # loop

Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

Re: Automating changes in text files

Sandman! — Thu, 06 Sep 2007 14:04:38 GMT

Try the sed construct below:

# sed '/^000/N;/.*\n003/s/\n/ /p' file

~cheers

Re: Automating changes in text files

Sandman! — Thu, 06 Sep 2007 15:17:55 GMT

To execute sed construct from within a shell script do as shown:

# ./uxscript file 000 003

...contents of uxscript are listed below.

======================= uxscript =======================

#!/usr/bin/sh

sed "/^$2/N;/.*\n$3/!s/\n/ /p" $1

Re: Automating changes in text files

Dennis Handly — Thu, 06 Sep 2007 16:08:21 GMT

>1) Can I just type in all this to file and run that as a script?

Yes.

>2) will the same work on Linux also??

It should.

>3) Can you suggest changes so that I can use it as a script and I can run it just by typing
./script
eg: ./scr myfile 000 003

Note the current script just creates a new file, if you want to update it, you would need to rename it.

Ah, it is much easier than what I thought you needed.

#!/bin/sh
# Usage: $0 file pat1 pat2
awk -v p1="^$2" -v p2="^$3" '
$0 ~ p1 {
printf "%s", $0
getline
if ($0 ~ p2)
print "" # newline
}
{ print } ' $1

>Sandman: To execute sed construct from within a shell script do as shown:

Yes, I thought about using sed like that. I thought I had to look for N patterns at the same time.
Your script inserts a blank, mine doesn't.

Re: Automating changes in text files

unixnewbie — Fri, 07 Sep 2007 00:27:58 GMT

Sandman, The sed construct on execution shows the result but the file is not changing. If I redirect the o/p to another file ./sand myfile 000 003 > myfile1 I get what I want but the line replicates.ie, it identifies the lines, joines them perfect, but it will add one more line in myfile1 (a duplicate the same joined line).Any changes ?? ..Thanks..

Handly, I tried ur script in Linux (#!/bin/sh)(not yet on HP)
./denn newfile orgfile 000 003.
If I am not wrong the newfile should contain what I need.But it is still 0Kb.Please advise. Thank you

John and Hein,I will try your suggestions and get back. Thanks a lot....

Re: Automating changes in text files

unixnewbie — Fri, 07 Sep 2007 01:25:33 GMT

sandman sorry I forgot sed don't change the input file.

Re: Automating changes in text files

Dennis Handly — Fri, 07 Sep 2007 01:38:48 GMT

>but it will add one more line in myfile1 (a duplicate the same joined line). Any changes??

It works fine for me. Are you running it on Linux? Perhaps that useless "p" needs to be removed there?
sed "/^$2/N;/.*\n$3/!s/\n/ /" $1

>I tried your script in Linux (#!/bin/sh)
./denn newfile orgfile 000 003

Oops, you invoke it:
./denn orgfile 000 003 > newfile

That "Usage: $0" comment was an indication that it prints: ./denn file pat1 pat2

Re: Automating changes in text files

unixnewbie — Fri, 07 Sep 2007 02:09:18 GMT

Dear Dennis, It was on Linux and YES (p)that solves it.Your script (forgive me for the troubles.. ) also does same thing but the output file is a little smaller in size than the input and in the case of sed script both are equal in size.........???
Great help 4 me.. Thanks.......

Re: Automating changes in text files

Dennis Handly — Fri, 07 Sep 2007 02:24:59 GMT

>but the output file is a little smaller in size than the input and in the case of sed script both are equal in size???

Yes, I mentioned that when I was commenting on Sandman's script:
Your script inserts a blank, mine doesn't.

If you want that blank, then use:
if ($0 ~ p2)
print "" # newline
else
printf " " # blank

If you don't want that blank:
sed "/^$2/N;/.*\n$3/!s/\n//" $1

Re: Automating changes in text files

unixnewbie — Fri, 07 Sep 2007 02:35:59 GMT

Yes dennis. I think that finishes it.If u don't mind can u suggest me link where I can find out myself what that little sed contruct does........Thanks..

Thank you sandman

Re: Automating changes in text files

Dennis Handly — Fri, 07 Sep 2007 02:52:22 GMT

>If you don't mind can you suggest me link where I can find out myself what that little sed construct does.

Well, you can look at sed(1):
http://docs.hp.com/en/B2355-60130/sed.1.html

Basically what it does is:
sed "/^$2/N;/.*\n$3/!s/\n//" $1
1) Finds a line that starts with $2.
2) N appends the next line into the existing line with an embedded newline.
3) Then finds a line where it doesn't (!) have a newline followed by $3. And if it doesn't, it substitutes (s) nothing, for that newline.
4) Then it writes those 2 lines out.
5) Other lines are just read and written.

HP-UX used to have a hard copy manual with more details about sed. I don't think it was ever moved to electronic form. google did find the Tru64 version:
http://h30097.www3.hp.com/docs/base_doc/DOCUMENTATION/HTML/AA-PS32D-TET1_html/pst4.html#sed

Re: Automating changes in text files

James R. Ferguson — Fri, 07 Sep 2007 04:44:14 GMT

Hi:

For a 'sed' tutorial, you might find the GNU manuals helpful:

http://www.gnu.org/software/sed/manual/sed.html

Regards!

...JRF...

Re: Automating changes in text files

Hein van den Heuvel — Fri, 07 Sep 2007 06:51:04 GMT

For completeness...

The perl solution as suggested gets a little ugly when adding a space:

$ perl -e '$a=shift; $b=shift; while (<>) {if (/^$a/){chomp; $n=<>; $_ .= (($n =~/^$b/)? "\n" : " "
) . $n} print}' 100 is x.txt'

old > new

So a better perl solution would be:

$ perl -e '$a=shift; $b=shift; while (<>) {if (/^$a/){$_ .= <>; s/\n/ / unless /\n$b/} print}' 100
edis x.txt

old > new

The core of that in slow motion:

if (/^$a/){ # start pattern?
$_ .= <>; # add next line to current
s/\n/ / # replace new-line in middle...
unless /\n$b/} # unless next started with end-pattern

fwiw,
Hein.

Re: Automating changes in text files

Sandman! — Fri, 07 Sep 2007 12:28:20 GMT

Under Linux the sed(1) substitution cmd takes the optional "p" switch which works if the "-n" cmd line option is given otherwise it produces a duplicate line for every one that gets sucessfully substituted. This does not happen on HP-UX and Dennis' post has explained the sed construct very well. Also checkout the sed tutorials at the "seders grab bag" :)

http://sed.sourceforge.net/grabbag/tutorials/

~cheers

Re: Automating changes in text files

unixnewbie — Thu, 13 Sep 2007 12:19:14 GMT

Thanks everybody