topic Re: Help on removing duplicate in 2 files in Operating System - HP-UX

Help on removing duplicate in 2 files

Mike_305 — Wed, 13 May 2009 21:03:43 GMT

Hello,

I have 2 files and trying to remove duplicates from these two files.

How can I do this?

Appreciate your help and thanks in advance.

Thanks.

Re: Help on removing duplicate in 2 files

James R. Ferguson — Wed, 13 May 2009 21:21:10 GMT

Hi:

And what do you want to do with the duplicates --- retain them in one file or a new file or discard them from both?

What constitutes a duplicate --- a whole matching line or just a field within a line?

A bit better definition of the environment and the objective would be helpful.

Regards!

...JRF...

Re: Help on removing duplicate in 2 files

Mike_305 — Wed, 13 May 2009 21:46:25 GMT

Hello James,

Absolutely, my bad. I am trying to compare two files that has the list disk like /dev/dsk/cXtYdZ.

Basically, I donâ t want to use the same disk that I have in LVMTAB and list I created.

Thanks in advance.

Re: Help on removing duplicate in 2 files

Mike_305 — Wed, 13 May 2009 22:07:09 GMT

Hello,

If I use "grep" by it self I can see the duplicate and give me the output that is in both files but I need to figure out which is not duplicate disk and not in use by other VG's on the systems.

I am doing this and not helping.

for X in $(cat disk.list)
do
grep ${X} new.disk.list
done

An "-v" option give me somethong different.

Appreciate your help.

Thanks....

Re: Help on removing duplicate in 2 files

James R. Ferguson — Wed, 13 May 2009 22:13:38 GMT

HI (again):

You can use 'grep' and a file of the devices you want to sift:

Given that '/etc/lvmtab' is a binary file, we need to use 'strings'. You could do:

# strings /tmp/lvmtab|grep -v vg|grep -f /tmp/mydev

...and:

# strings /tmp/lvmtab|grep -v vg|grep -v -f /tmp/mydev

...where the /tmp/mydev file lists the devices one per line.

Regards!

...JRF...

Re: Help on removing duplicate in 2 files

Mike_305 — Wed, 13 May 2009 23:15:46 GMT

Hello James,

I got the file ready by filtering the output from powermt command but now I am trying to insert /dev/dsk in front of it using "sed".

Once, I have that done I think I will be in place.

Thanks for your help.

Re: Help on removing duplicate in 2 files

Dennis Handly — Thu, 14 May 2009 01:30:04 GMT

>I am doing this and not helping.
>for X in $(cat disk.list)

You need to use vector methods:
fgrep -v -f disk.list new.disk.list

This excludes all of the lines in disk.list from new.disk.list. (Other options are needed of some lines are substrings of other longer ones, add -x.)

>insert /dev/dsk in front of it using "sed".

sed 's:^:/dev/dsk:' file

Re: Help on removing duplicate in 2 files

Mike_305 — Fri, 15 May 2009 20:53:48 GMT

Thanks for everyones help. Appreciate that very much.