T4 EVA CSV problem

[Art Wiens]

I'm getting an error when using TLVIZ (v1.6-9) to open the CSV's pertaining to the GGA files (see attached). Not sure how "data row # 153" relates to the actual file layout.

Can these files be salvaged?

Cheers,
Art

[Art Wiens]

BTW, I can open them with Excel.

[Hein van den Heuvel]

Hello Art,

The error seems clear and explicit.
Maybe a configuration change on the EVA?

Maybe the data was:
A1, A2, A3, B1, B2, B3, C1, C2, C2
Now target B goes away and the sources output values: a1, a2, a3, c1, c2, c2
But those are just numbers. (no XML here... fortunately!).
So is this really: a1, a2, a3, c1, c2, c2, 0, 0, 0 ?
or: a1, a2, a3, 0, 0, 0, c1, c2, c2 ?

You can probably tell. But TLviz van not.

I would still be sure to try with TLviz V2 but expect that it will call out the same issue. Still: http://trendsthatmatter.com/nbu/t4-tools/TLViz-2.0-1.exe

When you open with Excel, does the data around row (data) 153 ( + header offsets) allow you recognize what has changed?
For similar problems I have written (perl) scripts to add zero-columns to fill out no longer available data.

Other than the data provider injecting a new header line, what can the visualizer do? It can not magically decide what is good, what is bad right?

It will likely fix itself on restart, or like I said, you can script around.

hth,
Hein

[Art Wiens]

If I compare it (in Excel) to a file that "works" from a few months ago, I don't see an obvious difference. Same column headers that start and stop on the same line numbers, data points (times) are the same, same number of lines in both files. I have attached a ZIP of the two - 11-Jan is ok, 14-Mar fails. I was not aware of a V2 TLVIZ ... I'll give that a go.

Thanks,
Art

[Art Wiens]

Actually I do see the difference now ... at line 375 it seems to go off the rails.

[Hoff]

That Excel error message stinks.

Try OpenOffice.org suite?

Try iWork Numbers?

Try jTLViz?

Try counting and appending six commas onto row 153?

Convert and salvage the files into (more strictly-constructed) CSV or into XML or such using perl or python or php or DCL?

I have libcsv ported to VMS and details posted, if you're interested.

[Art Wiens]

Hoff : "That Excel error message stinks."

That's actually a TLVIZ error message ... Excel can open the CSV just fine. No need for a wholesale office uplift yet ;-)

If you look at the second attachment, at the 14-Mar file, you can see where the problem occured. We did have "some sort" of EVA related problem last night (which is why I was trying to look at these files), perhaps it (the EVA's) provided the EVA data collector on VMS "crap". I haven't heard an explanation from the storage folks yet. I got involved because a disk to disk backup that normally takes 15 minutes took an hour and 15 minutes!

Cheers,
Art

ps. the v2.0-1 TLVIZ reports the same error in the exact same way.

[Hoff]

The. Error. Message. (Still.) Stinks.

[Hein van den Heuvel]

What stinks about the TLviz message?
Seems clear and clean. What more could it possibly report?


Now Excel on the other hand does it's usual crap on the OpenVMS date formatting.
Even when selecting "text" Format, for which the description reads "The text is displayed exactly as entered" it munges the text irreversibly to numbers. :-(

Anyway, the file contains junk. Records. Just look at the date's. The following perl reports each day found and its count:

# perl -ne "$x{$1}++ if /^\s*(\d+-...-\d\d\d\d)/;}{ print qq($_ $x{$_}\n) for (keys %x) " art.csv
15-Mar-2010 60
07-Feb-2106 33
14-Mar-2010 1346

Now fix those non-2010 dates by bluntly replacing them with the last seen 2010 date:

# perl -pe "next unless /^(\d\d-...-.*?,)/; $x=$1; if (!/-2010/) {s/$x/$g/} else {$g=$x}" art.csv >
x.csv

re-count:
15-Mar-2010 60
14-Mar-2010 1379

Now TLviz will 'eat' the file without chocking and pretty much just ignore the bad lines.

It's still not 'right', but it gives you something to look at.

For example the data for "PD Write RPS" looks ok-ish.

You can get rid of the error message, but possible create confusion by blindly adding zero columns to the line:

# perl -pe "next unless /2010 /; chomp; $x=split /,/; $c=$x unless $c; $_ .= q(,0)x($c-$x).qq(\n)" x.
csv > y.csv

for non-perl folks, the 'x' operator multiplies/replicates strings, for any positive multiplier. That q(text) is the same as a single quote (non-interpolated) string but easier for DCL usage. Similar for qq(text) but that is double-quoted and interpolated making \n into a new-line.
For example:

# perl -le "print qq(aap noot mies\n) x 3"
aap noot mies
aap noot mies
aap noot mies


More/Better recovery is likely possible but would be 'real work'.

Cheers,
Hein

[Hoff]

Ok... Prior to getting a technical writer involved, I'd look at something akin to this...

--
TLViz Vx.y-z has detected a file corruption at record [line] in file [name].

An incorrect number of columns are present within the record; [columns] were expected and [columns] found.

Processing will attempt to continue at the next valid record located in the file.
--

Or better, create a cache of these diagnostics to the side, and offer to display a listing of all errors after the processing has completed, or after TLViz has given up on trying to load the file. "Do you want to review formatting errors detected within this file?" or some such dialog.

--
Hyperlinks to the documentation and to where you can check for updates would probably be viewed as overkill with a Windows product, but it tends to be the norm on some other platforms.