topic Re: script to delete macthing lines in Operating System - HP-UX

script to delete macthing lines

John McDen — Fri, 12 Apr 2002 15:46:13 GMT

I have a huge file with few duplicate entries. I want to delete all duplicate entries from the file.

original file

xyz123abc
ghi567zxc
ghi567zxc
abc345000
...
...

output file

xyz123abc
abc345000
...
...

The file is sorted do anybody have any script to do that?

Thanks in advacne

Re: script to delete macthing lines

hpuxrox — Fri, 12 Apr 2002 15:47:01 GMT

cat filename | sort -u

Re: script to delete macthing lines

John McDen — Fri, 12 Apr 2002 15:51:03 GMT

no i want to delete both the lines .. sort -u is for unique

Thanks anyway

Re: script to delete macthing lines

John Carr_2 — Fri, 12 Apr 2002 15:54:09 GMT

Hi

cat original | sort -u > tempfile

diff original tempfile > output_file

john.

Re: script to delete macthing lines

A. Clay Stephenson — Fri, 12 Apr 2002 15:54:47 GMT

In that case,
cat filename | uniq -u (but filename must already be sorted)

Re: script to delete macthing lines

SHABU KHAN — Fri, 12 Apr 2002 15:55:33 GMT

John,

Here is a simple shell script to strip duplicate lines in a file ...

-------------------
#!/bin/sh

# dup - Will search for duplicate lines in a file

if [ -z "$1" ]; then
echo
echo "syntax: dup [filename]"
echo
exit
fi

file=$1

echo -n "Searching Dups..."
cat /dev/null > $file.n
while read line
do
found=`grep "$line" $file.n`
if [ -n "$found" ]; then
echo -n "."
else
echo "$line" >> $file.n
fi
done < $file
old=`wc -l $file | awk '{ print $1 }'`
new=`wc -l $file.n | awk '{ print $1 }'`

echo "`expr $old - $new` dups found"
echo "Newfile: $file.n"
--------------
Syntax is:

prompt>scriptname

Thanks,
Shabu