topic Re: Shell script. in Operating System - HP-UX

Shell script.

LEJARRE Patrick — Thu, 27 Jul 2000 11:23:50 GMT

I would like to extract from a input text file the lines that would match the following specifications:
The extracted lines should contain a given string
AND
The line before the first occurence in the input file should also be extracted
AND
All the extracted lines should be deleted from the input file.
The result should be placed ino a output text file.

I think that an "awk" guru could help me best.

Re: Shell script.

Andreas Voss — Thu, 27 Jul 2000 11:34:27 GMT

Hi,

here you are:

awk '{
if($0 ~ "pattern")
{
print $0;
print prev;
}
prev=$0;
}'

You have to modify "pattern" for your requirements.

Cheers

Andrew

Re: Shell script.

Andreas Voss — Thu, 27 Jul 2000 11:35:49 GMT

Sorry title mistake in the output order:
Instead:
print $0
print prev
use:
print prev
print $0

Re: Shell script.

LEJARRE Patrick — Thu, 27 Jul 2000 13:23:13 GMT

hi
your answer helped me a lot but the lines containing the search string are printed twice:

awk '{if ($0 ~ "pattern") {print prev;print $0;} prev=$0;}' inputfile > outputfile

I solved the problem with the following additional commands:

head -1 outputfile > outputfile1
line=`wc -l outputfile | awk '{print $1}'`
line=`expr $line - 1`
tail -$line outputfile | sort | uniq > outputfile2
cat outputfile1 outputfile2 > outputfile

I don't know wether it is possible to do it within the awk command, but it works perfectly.

Thanks.

Re: Shell script.

LEJARRE Patrick — Thu, 27 Jul 2000 13:29:34 GMT

The initial goals aren't fullfilled yet.
The extracted lines should be deleted from the input file.

Re: Shell script.

LEJARRE Patrick — Thu, 27 Jul 2000 15:36:16 GMT

Hi Andrew !
Below the solution.

Take a look at the -v option of awk to assign a parameter to be passed to the command.

Pat

---------------------------------------------
Shell script
---------------------------------------------

#!/sbin/sh

# Initialisation
search_string=$1
inputfile=$2
outputfile=$3
outputfile1=${outputfile}_tmp1
outputfile2=${outputfile}_tmp2

# Main
awk -v search_string=$search_string '{if ($0 ~ search_string) {print prev;print $0;} prev=$0;}' $inputfile > $outputfile

head -1 ${outputfile} > $outputfile1
nb_lines=`wc -l ${outputfile} | awk '{print $1}'`
nb_lines=`expr $nb_lines - 1`

tail -$nb_lines ${outputfile} | sort | uniq > $outputfile2
cat $outputfile1 $outputfile2 > ${outputfile}

sed -e 's/ /?/g' ${outputfile} > ${outputfile}$$
mv ${outputfile}$$ ${outputfile}
sed -e 's/"/@/g' ${outputfile} > ${outputfile}$$
mv ${outputfile}$$ ${outputfile}

sed -e 's/ /?/g' ${inputfile} > ${inputfile}$$
mv ${inputfile}$$ ${inputfile}
sed -e 's/"/@/g' ${inputfile} > ${inputfile}$$
mv ${inputfile}$$ ${inputfile}

ligne=`head -1 ${outputfile}`

grep -v "$ligne" ${inputfile} > ${inputfile}$$
mv ${inputfile}$$ ${inputfile}

grep -v "$search_string" ${inputfile} > ${inputfile}$$
mv ${inputfile}$$ ${inputfile}

sed -e 's/?/ /g' ${outputfile} > ${outputfile}$$
mv ${outputfile}$$ ${outputfile}
sed -e 's/@/"/g' ${outputfile} > ${outputfile}$$
mv ${outputfile}$$ ${outputfile}

sed -e 's/?/ /g' ${inputfile} > ${inputfile}$$
mv ${inputfile}$$ ${inputfile}
sed -e 's/@/"/g' ${inputfile} > ${inputfile}$$
mv ${inputfile}$$ ${inputfile}

Re: Shell script.

James R. Ferguson — Thu, 27 Jul 2000 16:37:15 GMT

Pat:

Let's use Andreas' suggestion amended as follows:

awk '{
if ($0 ~ /pattern/)
{
print $0 >> "/tmp/file.1";
print prev >> "/tmp/file.1";
}
else
{
print $0 >> "/tmp/file.2";
prev=$0;
}
}' /tmp/file

I think you will find that /tmp/file.2 is the input file stripped according to your mandate.

...JRF...

Re: Shell script.

Alan Riggs — Thu, 27 Jul 2000 17:10:07 GMT

How about:

INFILE=xxxx
OUTFILE=yyyyy

awk '{if($0 ~ "AND")
{
print prev
print $0
}
prev=$0
}' $INFILE > $OUTFILE
while read LINE
do
grep -v "$LINE" $INFILE > tmpfile
mv tmpfile $INFILE
done < $OUTFILE

Re: Shell script.

curt larson — Thu, 27 Jul 2000 18:25:07 GMT

#!/usr/bin/ksh

mfile=file_before_deletions

file1=mfile_after_deletions
file2=deletions_from_mfile

pattern=your_pattern

count=$(grep -c $pattern $mfile)

#if count = 0 no lines to delete
#
if (( $count == 0 )) ;then
cp $mfile $file1
touch $file2
else
#get line number of first occurance
#
line_num=$(grep -n $pattern $mfile | awk -F: '{print $1;exit;}')

#if the first occurance is the first line
#you can't delete the previous line
#so just delete the lines with the pattern
#
if (( $line_num == 1 )) ;then
grep -v $pattern $mfile > $file1
grep $pattern $mfile > $file2
else
#
# get the line number of the line
# before the first occurance
#
(( line_num = $line_num - 1 ))

# use sed to delete line then grep for all rest
#
sed -e "${line_num}d" $mfile | grep -v $pattern > $file1

#for deletions file use sed to print the line
#before the first occurance then grep the rest
#
sed -ne "${line_num}p" $mfile > $file2
grep $pattern $mfile >> $file2
fi
fi

Re: Shell script.

curt larson — Thu, 27 Jul 2000 20:05:30 GMT

and if you know there is at least one occurance and it isn't the first line, then this will work also:

ex -s +"/$pattern/ | .- d | 1,\$ global /$pattern/d | w > $file_with_deletions | q!" your_file

Re: Shell script.

LEJARRE Patrick — Fri, 28 Jul 2000 05:45:25 GMT

Hi Curt !
I read your script closely. I find it a little more complicated than the Andreas and Alan's scripts. I'm afraid your script would be more time and cpu consuming than the Andreas and Alan's ones.

Thank you very much for your contribution!
Pat.

Re: Shell script.

curt larson — Fri, 28 Jul 2000 09:43:51 GMT

Well, my script is more complicated for a couple of reasons:

1) I extract only the line before the first occurance of the pattern as specified, not the line before every occurance of the pattern as alan's does

2) my script handles the situation if the pattern occurs in the first line. Alan's will output an empty line being prev will have no value at that time and will later remove all the blank lines from the file. I don't think this was your intention.

As far as time and cpu, my script would only do 3 greps and an awk on the file worst case (double that if you want to count creating the file of the deleted lines which could be removed if it isn't necessary), where if there is only 2 matches in the file alan's script will do 4 greps and an awk on the file. And with more matches alan's would do even more greps increasing by 2 greps for every line with a pattern match.

But, if you'd like to do it even faster:

#!/usr/bin/ksh

line_num=$(grep -n $pattern $mfile | awk -F: '{print $1;exit;}')

if [[ -n $line_num ]] ;then
if (( $line_num > 1 )) ;then

(( line_num = $line_num - 1 ))
sed -e '$line_num d'
-e '/$pattern/d' $mfile > $tmpfile

else
sed -e '/$pattern/d' $mfile > $tmpfile
fi
mv $tmpfile $mfile
fi

sorry, but I have a different opinion then you

Re: Shell script.

curt larson — Fri, 28 Jul 2000 11:30:33 GMT

and if you really want to spend the time maintaining an awk script:

#!/usr/bin/ksh

cat infile | awk '

BEGIN { First = "no";
Match = "no";
getline;
if ( $0 ~ /pattern/ ) {
First = "yes";
Match = "yes";
}
prev = $0;
}
{
if ( $0 ~ /pattern/ ) {
if ( First == "no" ) {
First = "yes";
}
else {
if ( Match == "no" ) {
print prev; }
}
Match = "yes";
}
else {
if ( Match == "no" ) {
print prev; }
Match = "no";
}
prev = $0
}
END
{ if ( Match == "no" ) print prev; }
' > outfile

mv outfile infile

myself the greps, seds, and eds are simplier to understand.