System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

Help on removing duplicate in 2 files

SOLVED
Go to solution
Mike_305
Super Advisor

Help on removing duplicate in 2 files

Hello,

I have 2 files and trying to remove duplicates from these two files.

How can I do this?

Appreciate your help and thanks in advance.

Thanks.
If there is problem then don't think as problem, think as opportunity.
7 REPLIES
James R. Ferguson
Acclaimed Contributor

Re: Help on removing duplicate in 2 files

Hi:

And what do you want to do with the duplicates --- retain them in one file or a new file or discard them from both?

What constitutes a duplicate --- a whole matching line or just a field within a line?

A bit better definition of the environment and the objective would be helpful.

Regards!

...JRF...

Mike_305
Super Advisor

Re: Help on removing duplicate in 2 files

Hello James,

Absolutely, my bad. I am trying to compare two files that has the list disk like /dev/dsk/cXtYdZ.

Basically, I donâ t want to use the same disk that I have in LVMTAB and list I created.

Thanks in advance.


If there is problem then don't think as problem, think as opportunity.
Mike_305
Super Advisor

Re: Help on removing duplicate in 2 files

Hello,

If I use "grep" by it self I can see the duplicate and give me the output that is in both files but I need to figure out which is not duplicate disk and not in use by other VG's on the systems.

I am doing this and not helping.

for X in $(cat disk.list)
do
grep ${X} new.disk.list
done

An "-v" option give me somethong different.


Appreciate your help.

Thanks....
If there is problem then don't think as problem, think as opportunity.
James R. Ferguson
Acclaimed Contributor

Re: Help on removing duplicate in 2 files

HI (again):

You can use 'grep' and a file of the devices you want to sift:

Given that '/etc/lvmtab' is a binary file, we need to use 'strings'. You could do:

# strings /tmp/lvmtab|grep -v vg|grep -f /tmp/mydev

...and:

# strings /tmp/lvmtab|grep -v vg|grep -v -f /tmp/mydev

...where the /tmp/mydev file lists the devices one per line.

Regards!

...JRF...
Mike_305
Super Advisor

Re: Help on removing duplicate in 2 files

Hello James,

I got the file ready by filtering the output from powermt command but now I am trying to insert /dev/dsk in front of it using "sed".

Once, I have that done I think I will be in place.

Thanks for your help.
If there is problem then don't think as problem, think as opportunity.
Dennis Handly
Acclaimed Contributor
Solution

Re: Help on removing duplicate in 2 files

>I am doing this and not helping.
>for X in $(cat disk.list)

You need to use vector methods:
fgrep -v -f disk.list new.disk.list

This excludes all of the lines in disk.list from new.disk.list. (Other options are needed of some lines are substrings of other longer ones, add -x.)

>insert /dev/dsk in front of it using "sed".

sed 's:^:/dev/dsk:' file
Mike_305
Super Advisor

Re: Help on removing duplicate in 2 files

Thanks for everyones help. Appreciate that very much.

If there is problem then don't think as problem, think as opportunity.