- Community Home
- >
- Servers and Operating Systems
- >
- Operating Systems
- >
- Operating System - Linux
- >
- Eliminating Dups
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Forums
Discussions
Discussions
Discussions
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-16-2007 04:40 AM
10-16-2007 04:40 AM
HPUX 11x PARISC system.
I have a file that looks like this
John,Doe
John,Doe
John,Doe
Mary,Poppin
Mary,Poppin
Mary,Poppin
I'm writing a script in perl and I would like to know if anyone knows a method to get rid of duplicate lines.
thanks in advance for any help/suggestions.
Solved! Go to Solution.
- Tags:
- uniq
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-16-2007 04:55 AM
10-16-2007 04:55 AM
Re: Eliminating Dups
uniq myfile should do what you want.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-16-2007 04:57 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-16-2007 05:05 AM
10-16-2007 05:05 AM
Re: Eliminating Dups
In Perl, use a hash to collect the unique items.
#!/usr/bin/perl
use strict;
use warnings;
my %things;
while (<>) {
chomp;
$things{$_}++;
}
for my $key (sort keys %things) {
print "$key\n";
}
1;
Regards!
...JRF...
- Tags:
- Perl
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-16-2007 05:05 AM
10-16-2007 05:05 AM
Re: Eliminating Dups
sort -u
grep ^$ (to get rid of blank lines.
SEP
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
- Tags:
- Sort
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-16-2007 05:13 AM
10-16-2007 05:13 AM
Re: Eliminating Dups
One possible way:
map {chomp;$seen{$_}++ unless exists $seen{$_}} ;
@singles = keys %seen;
__DATA__
John,Doe
John,Doe
John,Doe
Mary,Poppin
Mary,Poppin
Mary,Poppin
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-16-2007 05:14 AM
10-16-2007 05:14 AM
Re: Eliminating Dups
generally its easy when your input is sorted.
The command
uniq filename
will output only different lines.
In perl, remember the last valid line and skip identical ones:
...
my $last;
while (<>) {
if ($last eq $_) { next; }
$last = $_;
print $_;
}
...
Change the print statement by your code.
mfG Peter
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-16-2007 05:15 AM
10-16-2007 05:15 AM
Re: Eliminating Dups
Use James' solution...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-16-2007 05:24 AM
10-16-2007 05:24 AM
Re: Eliminating Dups
And printing them as they come, skipping dups could look like:
$ perl -ne 'print unless $x{$_}++'
Hein.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-16-2007 05:28 AM
10-16-2007 05:28 AM
Re: Eliminating Dups
------------------------------------
#!/usr/bin/perl -w
use strict;
use English;
use constant TRUE => 1;
my %exists = ();
my @uniqs = ();
my $stat = 0;
my $fname = "myfile";
my $cc = open(FH,$fname);
if (defined($cc))
{
my $s = '';
while (defined($s =
{
chomp($s);
unless ($exists{$s})
{
$exists{$s} = TRUE;
push(@uniqs,$s);
}
}
close(FH);
my $i = 0;
while ($i <= $#uniqs)
{
print $uniqs[$i],"\n";
++$i;
}
}
else
{
$stat = $ERRNO;
printf ("Can't open %s status %d\n",$fname,$stat);
}
exit($stat);
---------------------------------------
as the previous examples do, it uses a hash to keep up with the data that has already been read and only if the entry is the first encounter does it add to the output array.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-16-2007 05:35 AM
10-16-2007 05:35 AM
Re: Eliminating Dups
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-16-2007 05:36 AM
10-16-2007 05:36 AM