from horizon text format into vertical text format

[Matthew_50]

set 1,
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
12 0 0 0 0 0 0 18k 0 0 0 0 0 0 0 0


16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
0 22m 0 0 0 0 0 0 0 0 0 0 17k 0 0 0


Total
23m

set 2.
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1m 0 0 0 0 0 0 800k 0 0 0 0 0 0 0 0


16 17 18 19 20 21 22 23 24 Total
0 0 200k 0 0 0 0 0 0 2m

each set, number of columns will be dynamicly output from other scripts, but will end with "Total" column, for example, how do i convert set 1 into,
0:12
1:0
.
.
.
7:18k
.
.
.
17:22m
.
28:17k
.
Total:23m

and set 2 into

0:1m
.
.
7:800k
.
.
18:200k
.
.
Total:2m


shell script is prefer, thank you.

[James R. Ferguson]

Hi:

This appears to meet your needs:

# cat ./reformat#!/usr/bin/awk -f
/^Total/ {print;next;print;next}
/^[ ]*$/ {next}
{
LINE1=$0
n=split(LINE1,a," ")
getline LINE2
m=split(LINE2,b," ")
for (i=1; i}

...

Run as:

# ./reformat file

...where "file" contains a set of data as shown.

Note that in the regular expression:

^[ ]*$/

the whitespace consists of a blank character followed by a TAB character. This skips blank lines in the input.

Regards!

...JRF...

[Matthew_50]

what if data looks like following ?

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
=============================================
12 0 0 0 0 0 0 18k 0 0 0 0 0 0 0 0


16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
=============================================
0 22m 0 0 0 0 0 0 0 0 0 0 17k 0 0 0

Total
=========
40m

same output. how do i skip "=", must using grep -v "=" ? Thank you.

[Steven E. Protter]

Shalom,

grep -v excludes the search parameter.

Only use grep -v for excludes. Use other options for grep to give you equals.

SEP

[James R. Ferguson · Accepted solution]

Hi (again):

> same output. how do i skip "=", must using grep -v "=" ?

No, 'awk' has all the power of 'grep' and more!

# cat ./reformat
#!/usr/bin/awk -f
/^Total/ {print;next;print;next}
/^[ ]*$/ {next}
{
LINE1=$0
n=split(LINE1,a," ")
getline LINE2
if (LINE2~/======/) {next;getline LINE2}
m=split(LINE2,b," ")
for (i=1; i}

...run as:

# ./reformat file

Regards!

...JRF...

[James R. Ferguson]

Hi (again):

> SEP: Only use grep -v for excludes

Yes, and that's exactly what he wants to do. My point in _not_ using 'grep' was why have an extra process when 'awk' can do it all?

However, if Matthew wanted to use my original solution he could have done:

# grep -v "======" file | ./reformat

Regards!

...JRF...

[Steven E. Protter]

Yea,

JRF reads my mind and interprets what I meant to say. He's da man.

SEP

[Matthew_50]

what is source like this.

16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Total
==============================================================================================
slot 9: 144m 0 212m 67m 123 0 789 0 0 0 0 0 0 0 0 0 423m

and reformat into

slot 9/16: 144m
slot 9/17: 0
slot 9/18: 212m
slot 9/19: 67m
.
.
.
slot 9/Total: 423m

possible ? Thanks.

[Matthew_50]

attach input sample

[James R. Ferguson]

Hi Matthew:

Given your posted input, you could use the attached Perl script.

Run as:

# ./reformat file

Regards!

...JRF...

[Hein van den Heuvel]

This perl seems to do what you desired....

If it sees a line starting with just numbers it adds those as entries into an array @tag

If it sees a line starting with "sl...:" then it tkaes the first part as a name, and add the rest as values to an other array @val.

If the last entry in the @tag array is 'Total' then report and clear arrays for a next go around.

Hein.

-------------------- reformat.pl ------
while (<>) {
push @tag,split if /^\s*\d+\s+/;
if (/^(sl.*):(.*)/) {
$nam=$1;
$_ = $2;
push @val,split;
print " -- $#tag -- @tag[-1] --", join '*',@tag, "\n";
if (@tag[-1] eq 'Total') {
for ( $i=0;$i<=$#val;$i++) {
print "$nam/$tag[$i]:$val[$i]\n";
}
@tag = @val = ();
}
}
}
------------

[Matthew_50]

Hello JRF, Hein

I have to say thank you from the bottom of my heart. You are really expert :-)

[Matthew_50]

hmm, need to reformat using perl from input as pervious one

slot 1/01: 4.3k
slot 1/02: 120
slot 1/03: 68.7k
.
slot 9/16: 144m
slot 9/17: 0
slot 9/18: 212m
slot 9/19: 67m

into

slot 1/01: 4300
slot 1/02: 120
slot 1/03: 68700
.
slot 9/16: 144000000
slot 9/17: 0
slot 9/18: 212000000
slot 9/19: 67000000

[James R. Ferguson]

Hi (again) Matthew:

> hmm, need to reformat using perl from input as pervious one

OK, then this should satisfy your requirement:

# cat ./reformat
#!/usr/bin/perl
use strict;
use warnings;
my ( @first, @second, $slot, $n, $value );
while (<>) {
next if /^===/;
if (m{^\s*\d}) {
@first = split;
next;
}
if (m{^slot}) {
@second = split;
$slot = shift @second;
( $slot = shift @second ) =~ s/://;
}
if ( m{^\s*$} or eof ) {
for ( $n = 0; $n < @first; $n++ ) {
( $value = $second[$n] ) =~ s/(\d+)k/$1*1000/e;
( $value = $second[$n] ) =~ s/(\d+)m/$1*1000*1000/e;
print 'slot', $slot, '/', $first[$n], ':', $value, "\n";
}
}
}
1;

Regards!

...JRF...

[James R. Ferguson]

Hi Matthew:

Oops, I meant to add that while I used multipliers of 1000 and 1000*1000 for 'K' and 'M' respectivly, it would be better to use 1024 and 1024*1024 :

( $value = $second[$n] ) =~ s/(\d+)k/$1*1000/e;
( $value = $second[$n] ) =~ s/(\d+)m/$1*1000*1000/e;

Regards!

...JRF...

[Matthew_50]

How can I combine following into perl as standard input to the while you provide ?

$count=0;
while (<>)
{
$count++;
if ( $count >= 17 && $count <= 57 )
{
print;
}
}
1;

Thanks

[James R. Ferguson]

Hi:

The <> operator is a read of STDIN.

# cat ./reader
#!/usr/bin/perl
use strict;
use warnings;
my $count=0;
while (<>) {
$count++;
if ( $count >= 17 && $count <= 57 ) {
print;
}
}

...execute as:

# ./reader file

OR:

# cat file | ./reader

...where 'cat' is any process that produces STDOUT.

All this said, you don't need to count lines yourself. Perl has a built-in line-number:

# cat ./reformat2
#!/usr/bin/perl
use strict;
use warnings;
while (<>) {
if ( $. >= 17 && $. <= 57 ) {
print;
}
}

...executed just like before.

Regards!

...JRF...

[Hein van den Heuvel]

Matthew wrote >>

>> How can I combine following into perl as standard input to the while you provide ?

Can you re-phrase that? What are you trying to accomplish here? What is file being read? The earlier provided input, or the output provided so far ?

>> if ( $count >= 17 && $count <= 57 )

Sounds to me like you are trying to skip the first 16 entries generated. Why not skip generating them instead? What is the 'rule' here?

JRF wrote >> ( $value = $second[$n] ) =~ s/(\d+)k/$1*1000/e;

Nice use of the 'e' in the search and replace to tell it to "Evaluate the right side as an expression."
And that's what needed to do the * 1024.
For use in my example:
$_ = $val[$i];
s/(\d+)k/$1*1024/e;
s/(\d+)m/$1*1024*1024/e;
print "$nam/$tag[$i]:$_\n";

But if * 1000 is requested then a simple string substitute will do fine:
s/k/000/;
s/m/000000/;

Cheers,
Hein.

[Matthew_50]

Hello JRF,

the pattern seems not working with

slot4/25:4.3k
slot4/26:0
slot4/27:4.3k
slot4/28:4.3k
slot4/29:4.3k
slot4/30:0
slot4/31:0
slot4/Total:17k

output as input

[Matthew_50]

Hello Hein,

The field contain the "k" might include the . dot, same does "m" field.

[Matthew_50]

Let me re-summarize,

input as attach sample file.

1. extract line 17 to line 57
2. reformat & convert into following output

slot1/0:0
slot1/1:0
.
.
slot1/total:0
slot4/0:0
.
.
slot4/25:4300
.
slot4/27:4300
.
slot4/total:17000
.
.
slot9/16:144000000
.
.
slot9/19:67000000
slot9/total:423000000

[Hein van den Heuvel]

Ok... so there is no need for filter for line 17 - 57.
That's just a crazy restriction to get yo into trouble later.
What you really want is to recognize the report pattern, ending in 'total' like my example already did.
It simply ignores the noise before ( $. < 17) and after ( $. > 57 )

I failed to see the decimal value requirement, and so it seems did Jim.
The \d+ use looks for charactes 0 thru 9, but here we need to use [0-9.]+ to add the .

One possible solution then becomes:

------------------ reformat.pl ----
while (<>) {
push @tag,split if /^\s*\d+\s+/;
if (/^(sl.*):(.*)/) {
$nam=$1;
$_ = $2;
push @val,split;
#debug print " -- $#tag -- @tag[-1] --", join '*',@tag, "\n";
if (@tag[-1] eq 'Total') {
for $i ( 0 .. $#val) {
$_ = $val[$i];
s/([0-9.]+)k/$1*1000/e;
s/([0-9.]+)m/$1*1000*1000/e;
print "$nam/$tag[$i]:$_\n";
}
@tag = @val = ();
}
}
}



Run as: perl reformat.pl < data.file

Or embed in a shell script with the shebang: #!/usr/bin/env perl ...

Btw... if you want to surpress all lines with zeros, change the print to:

$_ and print "$nam/$tag[$i]:$_\n";

Add back in all 'total' lines with:

($_ or $tag[$i] eq 'Total') and print "$nam/$tag[$i]:$_\n";


Hein.

[Matthew_50]

hmm, after some try & error.

see attach for the result.

[Matthew_50]

Thank you JRF, Hein being with me so late,
almost time for me to start work. @@~

[James R. Ferguson]

Hi Matthew:

I had to abandon you yesterday for family-time. Hein was correct, I failed to see the decimal requirement let alone the last code I posted wasn't quite what we needed.

That said, here's a version that handles integer or decimal values with or without a suffix of "k" or "m". The suffix is evaluated case-insensitively. The output is expressed in integer values, rounded up to the nearest integer. Once again I have used the multiplicative factor of 1024 in lieu of 1000. In all:

# cat ./reformat
#!/usr/bin/perl
use strict;
use warnings;
my ( @first, @second, $slot, $n, $value );
while (<>) {
next if /^===/;
if (m{^\s*\d}) {
@first = split;
next;
}
if (m{^slot}) {
@second = split;
$slot = shift @second;
( $slot = shift @second ) =~ s/://;
}
if ( m{^\s*$} or eof ) {
for ( $n = 0; $n < @first; $n++ ) {
$value = $second[$n];
if ( $value =~ m/k$/i ) {
$value =~ s/((\d+)(\.\d+)?)k/$1*1024/ie;
}
elsif ( $value =~ m/m$/i ) {
$value =~ s/((\d+)(\.\d+)?)m/$1*1024*1024/ie;
}
printf "slot %s/%s:%-8.0f\n", $slot, $first[$n], $value;
}
}
}
1;

Regards!

...JRF...