topic Re: awk, external command and quotes in Operating System - OpenVMS

awk, external command and quotes

labadie_1 — Mon, 28 May 2007 01:55:19 GMT

I am struggling with awk and an external command.

I want to do, in a awk script, an external call to a Vms command

If I do
$ gawk :== $ sys$common:[syshlp.examples.tcpip.snmp]gawk

Then if I create a simple file a.awk
BEGIN{}
{ system(" wr sys$output f$time() ") }
Then the command
$ gawk/input=a.awk sys$input
followed by another RETURN
displays correctly the current time.

Now, if I want to issue a command with quotes, like
wr sys$output f$getjpi("20400416","PRCNAM")

I get various errors.

I have tried various tripling of the quotes, to no avail.

Thanks for any hint.

Re: awk, external command and quotes

labadie_1 — Mon, 28 May 2007 02:18:11 GMT

Of course, putting the command in a file and doing in the awk @file
works, but does not solve my problem, as I will have a pid varying

:-)

Re: awk, external command and quotes

John Gillings — Mon, 28 May 2007 03:06:02 GMT

Gerard,

Have you tried using PIPE? Perhaps something like:

$ PIPE WRITE SYS$OUTPUT "something" | -
GAWK/input=a.awk SYS$PIPE:

I find this is often a simple way of generating variable input without writing a temporary file. If the input is complex, you can use a command procedure to avoid having multiple continuation lines in the PIPE command, like this:

$ self=F$PARSE(";",F$ENVIRONMENT("PROCEDURE"))
$ IF p1.EQS."INPUT" THEN GOTO Input
$ IF p1.EQS."OUTPUT" THEN GOTO Output
$ PIPE @'self' "INPUT" | @'self' "OUTPUT"
$ EXIT
$
$ Input:
$ ! Generate your input stream here
$ ! using WRITE SYS$OUTPUT
$ EXIT
$
$ Output:
$ ! Execute command to consume the output
$ EXIT

(in your case you probably don't need the output branch, use your AWK command instead).

Re: awk, external command and quotes

labadie_1 — Mon, 28 May 2007 03:18:04 GMT

Thanks John for the general method.

I realize I have not correctly explained my problem.

I am still working on the procedure I posted in the thread on "monitor network traffic"
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1120946
(By the way I have fixed a bug)
It displays some interesting data, like

BG270 ,Stream , 705,49153, 127.0.0.1, 127.0.0.1, 280, 0, 15, 0
BG778 ,Stream , 705,49185, 127.0.0.1, 127.0.0.1, 18748, 0, 1853, 0
BG281 ,Stream ,49154, 705, 127.0.0.1, 127.0.0.1, 25704, 0, 1837, 0
BG30531,Stream ,57678, 1800, 193.248.1.155, 10.196.10.30, 62464, 0, 245, 0
BG260 ,Dgram , 161, 0, , , 200310, 0, 5372, 0
BG604 ,Dgram , 3181, 0, 0.0.0.0, , 364228, 0, 35016, 0
BG32505,Stream , 1190,32801, 193.248.1.156, 10.196.11.80, 556708, 0, 59625, 0
BG24149,Stream ,50031, 1501, 193.248.1.155, 10.196.138.225, 2365544, 0, 253419, 0
BG671 ,Stream , 1190,42865, 193.248.1.156, 10.196.138.225, 15110388, 0, 260926, 0
BG710 ,Stream ,49182, 1616, 193.248.1.155, 193.248.120.121, 23229596, 0, 2488443, 0
BG711 ,Stream ,49183, 1515, 193.248.1.155, 193.248.120.120, 195537460, 0, 20934757, 0
BG749 ,Stream , 1190,60648, 193.248.1.156, 193.248.120.121, 826776008, 0, 3051698, 0
BG1201 ,Stream , 1190,64562, 193.248.1.156, 193.248.120.120, 880459944, 0, 22705543, 0
Device | type | ploc | prem | host local | host remote | bytes rec | bytes sent | I/O rec | I/O sen |
____________________________________________________________________________________________________________________

And I thought it could be good to add the pid and processname owning the Bg device.

So it is while I am inside awk that I want to get some info from Vms, typically

f$getdvi("bg123","pid")
and
f$getjpi(f$getdvi("bg123","pid"),"prcnam")

Re: awk, external command and quotes

labadie_1 — Mon, 28 May 2007 04:06:04 GMT

In fact, I think the problem is in the syntax: in the awk file, I must have
{ system(" vmscommand ")}
So as long as I have
$ sh system
$ sh time
$ write sys$output f$time()
$ wr sys$output 1

It works fine.

But when I want to issue a command such as
$ wr sys$output "abcdef"
or any other with the character "
the parser wrongly takes the first " of the vms command as the end of the vms command

In fact, if you put as vms command
write sys$output "abcdef"
or
write sys$output

you get the same result
_Expression

:-(

Re: awk, external command and quotes

Steven Schweda — Mon, 28 May 2007 07:58:56 GMT

You're not thinking UNIX.

alp $ type a.awk
BEGIN{}
{ system(" write sys$output f$time() ") }

alp $ gawk /inp = a.awk sys$input

28-MAY-2007 07:55:48.97

And:

alp $ type b.awk
BEGIN{}
{ system(" write sys$output f$edit( f$time(), "COLLAPSE") ") }

alp $ type b.awk
BEGIN{}
{ system(" write sys$output f$edit( f$time(), "COLLAPSE") ") }
alp $ gawk /inp = b.awk sys$input

%DCL-W-ARGREQ, missing argument - supply all required arguments

But:

alp $ type c.awk
BEGIN{}
{ system(" write sys$output f$edit( f$time(), \"COLLAPSE\") ") }

alp $ gawk /inp = c.awk sys$input

28-MAY-200707:56:54.23

Re: awk, external command and quotes

Hein van den Heuvel — Mon, 28 May 2007 10:55:31 GMT

Free advice...

If you are going to do those getjpi prcnam for several sessions, then you may want to just pre-read some "show system" and stick the results into an array:

---- awk script ----
BEGIN { while ("show system" | getline) {
getjpi_prcnam[$1] = substr($0,10,15)
}
}
---------------------

Now you can just use getjpi_prcnam[pid] and know you are not spawning the system to a crawl.

But was the original input not from "show dev bg"? So you could just pick up process name and pic fron shwo dev/bg/full:

-----------------
BEGIN { while ("show device bg/full" | getline) {
if (/^Device/ ) {split ($2,x,","); bg = x[1];}
if (/UIC/) { split ($0,x,"\""); prcnam[bg] = x[2] }
if (/Owner process ID/) { pid[bg] = $4; print bg, x[2], $4; bg="X" }
}

for (bg in prcnam) {
print bg, pid[bg], prcnam[bg];
}
}

Or in original order:
--------------------------------

BEGIN { while ("show device bg/full" | getline) {
if (/^Device/ ) {units++; split ($2,x,","); bg[units] = x[1];}
if (/UIC/) { split ($0,x,"\""); prcnam[units] = x[2] }
if (/Owner process ID/) { pid[units] = $4; getjpi_prcnam[$4]=x[2] }
}
while (i++ < units) {
print bg[i], pid[i], prcnam[i];
}
}

I suspect the PERL is more suitable for this job though.

Enjoy,
Hein.

Re: awk, external command and quotes

labadie_1 — Mon, 28 May 2007 14:14:11 GMT

Steven

Thanks for the tip, I should have known better, as I had struggled with a Linux command on several other Linux nodes with rsh and echo $HOSTNAME
:-)

Hein
Thanks for the valuable info, but as I do
$ ana/sys
tcpip sh dev/fu bg

I do not have the pid and processname owning the bg edvice.

$ sh dev /fu bgxxx
gives it, but not the counters, remote and local port, remote and local host...

And
$ tcpip sh dev/fu bgxxx
gives data with a "bad format", not good for searching

Re: awk, external command and quotes

Hein van den Heuvel — Mon, 28 May 2007 14:43:25 GMT

>> $ ana/sys .... tcpip sh dev/fu bg
>> I do not have the pid and processname owning the bg edvice.

I only see a PID in that output if there is an active IRP.

So I'm suggesting that if you do the regular DCL $SHOW DEV BG/FULL early, in the BEGIN section of the awk, then you can establish a known mapping from BG device name to PID and/or PRCNAM, through a simple bunch of arrays.
Then as you process the ANAL/SYS output, you can latch on to the BG name and look it up in those arrays for pid and/or prcnam.

>>. $ tcpip sh dev/fu bgxxx
gives data with a "bad format", not good for searching

yeah, sort of. It creates a long (815) byte record with embedded CR-LF.

That's what breaks a simple $PIPE tcpip sh dev/fu bgxxx | TYPE/PAGE/

It's not all that hard to process and specificall PERL automatically breaks it down into lines.

If you stick the output into a file, and then $convert/fdl="record; format stream" then the output will show as individual lines as well.

fwiw,
Hein.

Re: awk, external command and quotes

Sebastian Bazley — Tue, 29 May 2007 09:04:10 GMT

You could surely just pass the pid into the script:

$ gawk -v pid=val -f script.awk

Re: awk, external command and quotes

Sheldon Smith — Tue, 29 May 2007 09:07:37 GMT

FYI: With judicious use of quotes, you can do useful things directly from DCL *without* a secondary script file. The trick is understanding when to 'escape' a quotation mark, and when to double it. Or both. 8^)
Here are a couple 'one-liners' from my collection:

Initialize any disks that have *never* been initialized:
------
pipe sho dev dg | gawk "BEGIN {print ""$set noon""}; /Online/ {print ""$mou /over=id /noassi"", $1; print ""$if f$mess( $status, \""ident\"") .eqs. \""%NOHOMEBLK\"" -""; print "" then init/sys /limit"", $1, ""UNUSED""}" sys$pipe: | @sys$pipe:

Show the size of all locally mounted disks (in MB):
------
pipe show dev d/mou | GAWK "/Mounted/ && !/(remote mount)/ {print ""$ write sys$output \"""" $1 ""\t\"", f$getdvi( \"""" $1 ""\"", \""maxblock\""), \"" bl.\t\"", f$getdvi( \"""" $1 ""\"", \""maxblock\"")/2048, \"" MB\""""}" sys$pipe: | @sys$pipe:

Re: awk, external command and quotes

labadie_1 — Tue, 29 May 2007 09:23:47 GMT

Sebastian

I am not sure I understand: I have a variable number of BG devices, say 50, so passing a pid will not help

Sheldon

Thanks a bunch for the procedures, I will have a close look

Re: awk, external command and quotes

Sebastian Bazley — Tue, 29 May 2007 09:32:33 GMT

Sorry, missed that part of the thread.

However you could join the pids (e.g. with a comma) if you wanted, and then split them again...

Or of course, put the raw pid details in a file that is read by the awk script.

Re: awk, external command and quotes

labadie_1 — Fri, 01 Jun 2007 03:00:30 GMT

I have attached the last version of my awk script.

It defines some symbols
- statbyr
sorts the BG Devices by Bytes received
- statbys
sorts the BG Devices by Bytes sent
- statior
sorts the Bg devices by I/O received
- statios
sorts the Bg devices by I/O sent

It is highly inefficient, as I do
for (bg in prcnam)
if (bg == toupper(dev))
printf ...

So if I have 200 bg devices, I will be 200x200 times in this loop

I have been surprised to discover that without the line
for (bg in prcnam), I do not find any array member.

Some with a higher knowledge of awk may explain it.

Re: awk, external command and quotes

Hein van den Heuvel — Fri, 01 Jun 2007 05:07:02 GMT

Wow, that's just horrible.

1)
What's the tolower doing in "if (/^Device/) {tolower" ?
Did you mean:
if (/^Device/) {split (substr($2,1,(length($2)-2)),x,","); bg=tolower(x[1]) ;}

And is that substr stuff just to get of the ":"?
Then just split on that!
if (/^Device/) {split $2,x,":"); bg=tolower(x[1]) ;}

2)
... dev=toupper($2)
:
for (bg in prcnam)
if (bg == toupper(dev))

Is that second 'toupper' 'just in case the first one did not work?'

I think all you need is: bg = dev .. but at a place where all the other data is collected. I don't have a system with 'ucx shwo dev/full' available right not.
But let's assume 'Bytes transferred' is the last line.
In that case the solution show be:

/Bytes transferred/ {byt = 0 ; byt=$3 ; byr= 0 ; byr=$4 ;
bg = dev;
printf(...)
}

3) Should the printting of the header line (print("device... ) not be part of the BEGIN {} block?

4) Why 'awk' 4 times for 4 sort orders?
The data could change?! awk once, sort 4 times! Append to header lines after each sort.

Cheers,
Hein.

Re: awk, external command and quotes

labadie_1 — Fri, 01 Jun 2007 05:20:14 GMT

Hein wrote
>>>Wow, that's just horrible.
Yes, I concur
:-)

>>>1)
What's the tolower doing in "if (/^Device/) {tolower" ?

the bg device in the first list (with the pid) and the second list are one in uppercase and the other in lowercase, so the test failed

>>>And is that substr stuff just to get of the ":"?
yes, because comparing bg227 and bg227: will fail :-)

Then just split on that!
if (/^Device/) {split $2,x,":"); bg=tolower(x[1]) ;}

2)
... dev=toupper($2)
:
for (bg in prcnam)
if (bg == toupper(dev))

>>>Is that second 'toupper' 'just in case the first one did not work?'
I forgot this one.

>>>I think all you need is: bg = dev .. but at a place where all the other data is collected. I don't have a system with 'ucx shwo dev/full' available right not.
But let's assume 'Bytes transferred' is the last line.

Yes it is
In that case the solution show be:

/Bytes transferred/ {byt = 0 ; byt=$3 ; byr= 0 ; byr=$4 ;
bg = dev;
printf(...)
}

3) Should the printting of the header line (print("device... ) not be part of the BEGIN {} block?
May be, but anyway I want it at the back of the display, because I sort with the higher values at the end.

4) Why 'awk' 4 times for 4 sort orders?
The data could change?! awk once, sort 4 times! Append to header lines after each sort.

Yes of course

Thanks for your inputs.

Re: awk, external command and quotes

labadie_1 — Fri, 01 Jun 2007 05:45:44 GMT

I have done as you said, as Bytes transferred is the last motif searched

>>>But let's assume 'Bytes transferred' is the last line.
In that case the solution show be:

/Bytes transferred/ {byt = 0 ; byt=$3 ; byr= 0 ; byr=$4 ;
bg = dev;
printf(...)
}

And I no longer have the pid and prcnam.

There is something I have not understood in the flow of data in awk...

:-(

Re: awk, external command and quotes

Hein van den Heuvel — Fri, 01 Jun 2007 06:24:42 GMT

I suspect there is still a slight punctuation error on the device name.

Print them out for a debug run, may on 1 device?

Email me a sample SHOW DEV BG/FULL as well as a sample UCX SHOW DEV/FULL and I'll see if I can find time to cehck it.

Flow in AWK is a surprise for first time users.
The main flow is 'all code' for all record in the input.
The /something/ {block} construct is really an 'if ($0 look like something) then {block}'

Most awk programs have simple {} block with no conditional, making that run for every line.

You can add a BEGIN {} and END {} which run jsut once typically to setup, print header fin BEGIN, and to print a total in END.

Cheers,
Hein.

Re: awk, external command and quotes

labadie_1 — Fri, 01 Jun 2007 06:48:19 GMT

Hein

I have emailed you the various files.
Thanks for your time.

I am trying to debug.

Re: awk, external command and quotes

labadie_1 — Thu, 14 Jun 2007 03:03:13 GMT

Thanks to Hein, the procedure now displays the pid and processname for each bg, without looping too much :-)

It is available at
http://dcl.openvms.org/stories.php?story=07/06/04/4359262