Calculate rolling minimum value across 3 rows in an input file - perl challenge?

[Duncan Edmonstone]

Hi folks, have a scripting challenge for those out there... I have an input file that looks like the following:

12/15/2010 09:10:00 myhost 0.02
12/15/2010 09:15:00 myhost 0.02
12/15/2010 09:20:00 myhost 0.02
12/15/2010 09:25:00 myhost 0.02
12/15/2010 09:30:00 myhost 1.01
12/15/2010 09:35:00 myhost 1.01
12/15/2010 09:40:00 myhost 1.01
12/15/2010 09:45:00 myhost 1.01
12/15/2010 09:50:00 myhost 1.01
12/15/2010 09:55:00 myhost 1.00
12/15/2010 10:00:00 myhost 1.00
12/15/2010 10:05:00 myhost 1.01
12/15/2010 10:10:00 myhost 0.01

What I need to do is add an extra column that finds the minimum value in each rolling 15 minute period from this data and then find the maximum value for that last column... so the output would look like


12/15/2010 09:10:00 myhost 0.02
12/15/2010 09:15:00 myhost 0.02
12/15/2010 09:20:00 myhost 0.02 0.02
12/15/2010 09:25:00 myhost 0.02 0.02
12/15/2010 09:30:00 myhost 1.01 0.02
12/15/2010 09:35:00 myhost 1.01 0.02
12/15/2010 09:40:00 myhost 1.01 1.01
12/15/2010 09:45:00 myhost 1.01 1.01
12/15/2010 09:50:00 myhost 1.01 1.01
12/15/2010 09:55:00 myhost 1.00 1.00
12/15/2010 10:00:00 myhost 1.00 1.00
12/15/2010 10:05:00 myhost 1.01 1.00
12/15/2010 10:10:00 myhost 0.01 0.01

Max Value: 1.01

I'm guessing perl would be the best tool to tackle this, but I haven't touched perl in nearly 8 years... any pointers or skeleton scripts I can build on would be great...

Regards,

Duncan

[James R. Ferguson]

Hi Duncan:

See if this meets your needs:

# cat ./myview
#!/usr/bin/perl
use strict;
use warnings;
use List::Util qw(min max);
my ( $mymin, $mymax, @vals );
while (<>) {
chomp;
push @vals, (split)[3];
shift @vals if @vals > 3;
$mymin = min(1E6, @vals);
$mymax = max(0, @vals);
if ($. > 2) {
print "$_ $mymin\n";
}
else {
print "$_\n";
}
}
print "\nMax Value: $mymax\n";
1;

...run as:

# ./myview file

Regards!

...JRF...

[Duncan Edmonstone]

Jim,

Close, but when I have >25 rows in my file the "Max Value" becomes incorrect - why would that be?

Duncan

[James R. Ferguson · Accepted solution]

HI (again) Duncan:

ENOCOFFEE

...I originally computed the maximum value only for the running groups therefore throwing away the truly highest value seen.

Try this version:

# cat ./myview
#!/usr/bin/perl
use strict;
use warnings;
use List::Util qw(min max);
my @vals;
my ( $mymin, $mymax ) = ( 1_000_000, 0 );
while (<>) {
chomp;
push @vals, (split)[3];
shift @vals if @vals > 3;
$mymin = min(1_000_000, @vals);
$mymax = max(0, $mymax, @vals);
if ($. >= 3) {
print "$_ $mymin\n";
}
else {
print "$_\n";
}
}
print "\nMax Value: $mymax\n";
1;

Regards!

...JRF...

[James R. Ferguson]

Hi Duncan:

This version cleans things up and improves performance:

# cat ./myview
#!/usr/bin/perl
use strict;
use warnings;
use List::Util qw(min max);
my @vals;
my ( $mymin, $mymax ) = ( 0, 0 );
while (<>) {
chomp;
my $element = (split)[3];
push @vals, $element;
shift @vals if @vals > 3;
$mymin = min(@vals);
$mymax = max($mymax, $element);
if ($. >= 3) {
print "$_ $mymin\n";
}
else {
print "$_\n";
}
}
print "\nMax Value: $mymax\n";
1;

Regards!

...JRF...

[Duncan Edmonstone]

Jim,

Picked up another little nit... I think cos I didn't describe the problem very well.

The "maximum" figure we display at the end needs to be for the "15 minute peak" column (i.e. column 4, not column 3), I think the nature of my test data meant I didn't pick this up before, but if I put a nice bit of rogue data in like this:

12/15/2010 09:10:00 myhost 0.02
12/15/2010 09:15:00 myhost 0.02
12/15/2010 09:20:00 myhost 0.02
12/15/2010 09:25:00 myhost 0.02
12/15/2010 09:30:00 myhost 1.01
12/15/2010 09:30:00 myhost 9.99
12/15/2010 09:35:00 myhost 1.01
12/15/2010 09:40:00 myhost 1.01
12/15/2010 09:45:00 myhost 1.01
12/15/2010 09:50:00 myhost 1.01
12/15/2010 09:55:00 myhost 1.00
12/15/2010 10:00:00 myhost 1.00
12/15/2010 10:05:00 myhost 1.01
12/15/2010 10:10:00 myhost 0.01


Then the output is:

12/15/2010 09:10:00 myhost 0.02
12/15/2010 09:15:00 myhost 0.02
12/15/2010 09:20:00 myhost 0.02 0.02
12/15/2010 09:25:00 myhost 0.02 0.02
12/15/2010 09:30:00 myhost 1.01 0.02
12/15/2010 09:30:00 myhost 9.99 0.02
12/15/2010 09:35:00 myhost 1.01 1.01
12/15/2010 09:40:00 myhost 1.01 1.01
12/15/2010 09:45:00 myhost 1.01 1.01
12/15/2010 09:50:00 myhost 1.01 1.01
12/15/2010 09:55:00 myhost 1.00 1.00
12/15/2010 10:00:00 myhost 1.00 1.00
12/15/2010 10:05:00 myhost 1.01 1.00
12/15/2010 10:10:00 myhost 0.01 0.01



Max Value: 9.99


when it should be 1.01 (the max value in column 4)

Cheers,

Duncan

[James R. Ferguson]

Hi Duncan!

> The "maximum" figure we display at the end needs to be for the "15 minute peak" column (i.e. column 4, not column 3),

That's easy, we'll just take our maximum from the "15 minute peak"s.

# cat ./myview
#!/usr/bin/perl
use strict;
use warnings;
use List::Util qw(min max);
my @vals;
my ( $mymin, $mymax ) = ( 0, 0 );
while (<>) {
chomp;
push @vals, (split)[3];
shift @vals if @vals > 3;
$mymin = min(@vals);
$mymax = max(@vals);
if ($. >= 3) {
print "$_ $mymin\n";
}
else {
print "$_\n";
}
}
print "\nMax Value: $mymax\n";
1;

Regards (and Happy New Year!)

...JRF...

[Duncan Edmonstone]

Thanks Jim!

And now you know how badly my perl skills have fallen away that I couldn't even spot that!

Duncan

(Happy New Year!)

[Duncan Edmonstone]

...closed

[James R. Ferguson]

Hi (again) Duncan:

This last version is flawed. The corrected version should be:

# cat ./myview
#!/usr/bin/perl
use strict;
use warnings;
use List::Util qw(min max);
my @vals;
my ( $mymin, $mymax ) = ( 0, 0 );
while (<>) {
chomp;
push @vals, (split)[3];
shift @vals if @vals > 3;
$mymin = min(@vals);
$mymax = max($mymax, $mymin);
if ($. >= 3) {
print "$_ $mymin\n";
}
else {
print "$_\n";
}
}
print "\nMax Value: $mymax\n";
1;

No points for this please.

Regards!

...JRF...