script to delete macthing lines

[John McDen]

I have a huge file with few duplicate entries. I want to delete all duplicate entries from the file.

original file

xyz123abc
ghi567zxc
ghi567zxc
abc345000
...
...

output file

xyz123abc
abc345000
...
...

The file is sorted do anybody have any script to do that?

Thanks in advacne

[hpuxrox]

cat filename | sort -u

[John McDen]

no i want to delete both the lines .. sort -u is for unique

Thanks anyway

[John Carr_2]

Hi

cat original | sort -u > tempfile

diff original tempfile > output_file

john.

[A. Clay Stephenson · Accepted solution]

In that case,
cat filename | uniq -u (but filename must already be sorted)

[SHABU KHAN]

John,

Here is a simple shell script to strip duplicate lines in a file ...

-------------------
#!/bin/sh

# dup - Will search for duplicate lines in a file

if [ -z "$1" ]; then
echo
echo "syntax: dup [filename]"
echo
exit
fi

file=$1

echo -n "Searching Dups..."
cat /dev/null > $file.n
while read line
do
found=`grep "$line" $file.n`
if [ -n "$found" ]; then
echo -n "."
else
echo "$line" >> $file.n
fi
done < $file
old=`wc -l $file | awk '{ print $1 }'`
new=`wc -l $file.n | awk '{ print $1 }'`

echo "`expr $old - $new` dups found"
echo "Newfile: $file.n"
--------------
Syntax is:

prompt>scriptname

Thanks,
Shabu