topic Re: Pipe output to shell script in Operating System - HP-UX

Pipe output to shell script

Johan Barelds — Tue, 02 Dec 2003 15:25:05 GMT

Hi all,

I want to pipe the output from a command to a shell script to process it.
e.g. cat file1.txt|shellscript.sh
How can i process the data from within the shellscript.sh ?
I tried $1 but that's only for parameters.
Any suggestions?

Thanks!
Bo

Re: Pipe output to shell script

Patrick Wallek — Tue, 02 Dec 2003 15:28:06 GMT

You would be better off passing the file name to the script as an argument and then using cat, awk, sed, etc. within your script to process the data feil the file.

# ./shellscript.sh /tmp/file1.txt

# cat shellscript.sh

#!/usr/bin/sh

FILE=$1

awk 'some awk stuff' $FILE

Re: Pipe output to shell script

Steven E. Protter — Tue, 02 Dec 2003 15:29:48 GMT

I would add this kind of processing loop to the shellscript.

while read -r aa
do
var=$aa
# do some processing
done < $1

You can input the filename to be processed as $1

shellscript.sh file1.txt

SEP

Re: Pipe output to shell script

A. Clay Stephenson — Tue, 02 Dec 2003 15:29:50 GMT

You need to use the "read" or "line" commands.

For example:

while read A B C
do
echo "A=${A} B=${B} C=${C}"
done

That will read stdin until EOF. If there are more than three variables to parse anything beynd the 2 argument will be combined in ${C}.

man sh-posix for details.

Re: Pipe output to shell script

Johan Barelds — Tue, 02 Dec 2003 15:33:34 GMT

Thanks all!

The "while read xx" loop was what i was looking for!

Grz. Bo

Re: Pipe output to shell script

rmueller58 — Wed, 03 Dec 2003 09:52:41 GMT

to use a file as a input it depends on what you are trying to accomplish..

I define variables from information in the file, then use the variable within the script..

If you need to read information from the file in a and perform a task based on multiple variables using a for-loop or do-while is very useful...

For example:

export PEIDBNAME
echo $PEIDBNAME
export DIST
echo $DIST
awk '{print $1,$2,$3,$4,$5,$6,$7,$8}' $DIST$BUILD|pr -c 2 -t
echo "Enter Build # supported by this printer:\c"
read bldgnum
echo $bldgnum
touch /tmp/btemp
chmod 666 /tmp/btemp
export BN=`grep "$bldgnum" $DIST$BUILD`
echo $BN
grep "$BN" $DIST$BUILD >/tmp/btemp
export BNAME=`awk '{print tolower(substr($2,1,3))}' /tmp/btemp`
echo $BNAME
export BNAME2=`awk '{print substr($2,1,3)}' /tmp/btemp`
echo $BNAME2
#clear

#### SEE $DIST$BUILD EXAMPLE BELOW:
# more conbuild
001 ADMINISTRATION BUILDING
002 HIGH SCHOOL
003 MIDDLE SCHOOL
004 ELEMENTARY SCHOOL
005 SPECIAL ED SCHOOL

ANOTHER EXAMPLE
#####
search for a string in a file and create another file...

####

echo "Enter the School District ID to exclude from the passwd file:"
read $dist
grep -v -h $dist passwd.org > passwd.restricted

#####
ANOTHER EXAMPLE

which your shell script for input information you can use a for loop or do while ..

### CREATING A TEMP FILE ####

echo "Enter the District to Alert:\c"
read dst
echo "Please Enter the message you want to send:\c"
read mssg
echo "Please Enter your Name:\c"
read name

usrmsg |grep "$dst"|awk '{print $2, $3}'> /tmp/usralert
for fgls in `awk '{print $1}' /tmp/usralert`
do
export FGLSERVER=$fgls
export LOGNAME=`grep "$fgls" /tmp/usralert |awk '{print $2}'`
echo $FGLSERVER $LOGNAME $dst $mssg $name
fglrun message.42r "$mssg" "$name"

Just a couple of thoughts and ways I handle it..

the beauty of Un*x is there are a hundred ways to do one operation..

Re: Pipe output to shell script

Alex Ostapenko — Thu, 04 Dec 2003 08:01:02 GMT

Hi Bo --

Someone mentioned the "while read" method which you seemed to like. That's a nice quick'n'dirty technique, but for large input files (>1000 lines?), it can have severe performance problems.

Once you get the script loop working, you might want to convert the "while read" guts into awk, then simply put into your ksh script:

cat - | awk 'your program'

then pipe into your script just like you did with the "while read" version of the script:

cat your-data | your-script.ksh

In a real-life example, processing data that took 30 minutes with "while read" took only 90 seconds with "awk" (I don't remember how big the file was). It has something to do with the shell buffering only 1 char at a time versus awk buffering 1K chars at a time.

Good luck!

=:-) Alex

Re: Pipe output to shell script

Elmar P. Kolkman — Thu, 04 Dec 2003 08:15:29 GMT

The difference with awk and the shell is that the shell cannot buffer, because processes inside the while loop could also need the same stdin... awk is one process and can buffer because no other commands inside the 'loop' will need the stdin without using the awk itself.

The speed advantage also depends on what you want to do inside the loop. If you only do some pattern matching and re-formatting, awk is the way to go. But if you need to run difficult unix commands from it, the shell could be a better solution.