topic combine records in file with awk? in Operating System - Linux

combine records in file with awk?

Stuart Abramson — Tue, 30 Aug 2005 07:47:21 GMT

I have a file that looks like this:

270095000 1527 REG 8838720 vgeb_rdp0_q c100t5d2
270095000 1527 REG 8838720 vgeb_rdp0_q c91t5d2
270098000 1527 M(8) 70709760 Avail c100t5d5
270098000 1527 M(8) 70709760 Avail c91t5d5
2700A5000 1527 M(8) 70709760 Avail c100t2d2
2700A5000 1527 M(8) 70709760 Avail c91t2d2
2700A7000 1527 M(8) 70709760 Avail c100t1d6
2700A7000 1527 M(8) 70709760 Avail c91t1d6
2700C5000 1527 REG 8838720 vgeb_rdp0_q c101t10d7
2700C5000 1527 REG 8838720 vgeb_rdp0_q c92t10d7

We combine a daily "syminq" with "/etc/lvmtab" to figure out which PVs/HVs are in what volume groups.

I want to combine records where field 1 is the same, in such a way that I get the complete record 1, and the LAST FIELD from subsequent records with the same field 1:

270095000 1527 REG 8838720 vgeb_rdp0_q c100t5d2 c91t5d2

Can you do that with awk? (We already do it with "ksh" but it's clumsy.)

Re: combine records in file with awk?

Jean-Luc Oudart — Tue, 30 Aug 2005 08:30:25 GMT

Hi,

sort | awk '
BEGIN{fld1=$1;printf("%s",$0); getline;}
{if($1==fld1) { printf("%s",$6); }
else { fld1=$1; printf("\n%s",$0); }
}
END{printf("\n")}'

Regards
Jean-Luc

Re: combine records in file with awk?

Tim Nelson — Tue, 30 Aug 2005 09:15:17 GMT

Check this script out.
Gathers all disk information, reports vg, lv, mounts, size, and free extents

Output in both formatted text and csv

Re: combine records in file with awk?

Hein van den Heuvel — Tue, 30 Aug 2005 11:07:26 GMT

With your data lines in a file 'x', the following seems to work:

awk '{if ($1==old){line = line $NF} else { if (old){print line}; line=$0; old=$1 }}END{print line}' x

270095000 1527 REG 8838720 vgeb_rdp0_q c100t5d2 c91t5d2
270098000 1527 M(8) 70709760 Avail c100t5d5 c91t5d5
2700A5000 1527 M(8) 70709760 Avail c100t2d2 c91t2d2
2700A7000 1527 M(8) 70709760 Avail c100t1d6 c91t1d6
2700C5000 1527 REG 8838720 vgeb_rdp0_q c101t10d7 c92t10d7

Build up a print line in variable 'line'
If the current input starts out the same, then add the current last field to 'line'.
If not the same, print line (unless first), and reset to current input.
At end print print the final line assembled.

Hein.

Re: combine records in file with awk?

Volker Borowski — Tue, 30 Aug 2005 11:11:48 GMT

Hi,

did you consider the join command ?
Should do exactly what you like to do.

Volker

Re: combine records in file with awk?

James R. Ferguson — Tue, 30 Aug 2005 11:48:41 GMT

Hi Stuart:

You can easily do this in perl along these lines:

#!/usr/bin/perl -an
if ($F[0] eq $prev) {
push( @list, $F[5] )
} else {
print "$prev @list\n" if (@list);
$prev = $F[0];
@list = ();
push( @list, $F[5] );
}
END { print "$prev @list\n" }

Regards!

...JRF...

Re: combine records in file with awk?

Sandman! — Tue, 30 Aug 2005 14:16:23 GMT

I second Hein's awk solution...quite elegantly and artfully done.

regards!

Re: combine records in file with awk?

Stuart Abramson — Wed, 31 Aug 2005 07:51:51 GMT

All good answers. Thanks very much.

I forgot something. Look at this case:

2700A7000 1527 M(8) 70709760 Avail c100t1d6
2700A7000 1527 M(8) 70709760 Avail c91t1d6
2700C5000 1527 REG 8838720 Avail c101t10d7
2700C5000 1527 REG 8838720 vgeb_rd c92t10d7
2700C5000 1527 REG 8838720 Avail c95t10d7

Volume 00C5 is presented 3 times with three device file entries, but only one of them is added to the VG. So, I want the combined record to look like this:

2700C5000 1527 REG 8838720 vgeb_rd c92t10d7 c101t10d7 c95t10d7

I need some logic that says "if any of the X volumes with the same Serial Number are in a Volume Group, use that VG name, not the Avail name.."

Re: combine records in file with awk?

Hein van den Heuvel — Wed, 31 Aug 2005 08:28:51 GMT

easy... in my example, just try replace 'Avail' with the current $5 field.
The will only do something once.
In an awk script:

{if ($1==old) {
line = line $NF
sub (/Avail/,$5,line);
} else {
if (old) {
print line;
}
line=$0;
old=$1;
}
}
END { print line }

Hein.

Re: combine records in file with awk?

Sandman! — Wed, 31 Aug 2005 14:44:00 GMT

Stuart,

Here's an awk construct that outputs what you're looking for. It concatenates multiple instances of the first field in the input file with a single stream in the output file.

# awk '
> {
> if(NR==1) {
> line=$0
> prev=$1
> }
> else
> if($1==prev)
> line=line" "$NF
> else {
> print line
> line=$0
> prev=$1
> }
> }
>
> END {print line}
> ' input_file

Re: combine records in file with awk?

Stuart Abramson — Wed, 31 Aug 2005 15:48:39 GMT

Thanks all, especially Hein.

Stuart