Pipe output to shell script

[Johan Barelds]

Hi all,

I want to pipe the output from a command to a shell script to process it.
e.g. cat file1.txt|shellscript.sh
How can i process the data from within the shellscript.sh ?
I tried $1 but that's only for parameters.
Any suggestions?

Thanks!
Bo

[Patrick Wallek]

You would be better off passing the file name to the script as an argument and then using cat, awk, sed, etc. within your script to process the data feil the file.

# ./shellscript.sh /tmp/file1.txt

# cat shellscript.sh

#!/usr/bin/sh

FILE=$1

awk 'some awk stuff' $FILE

[Steven E. Protter · Accepted solution]

I would add this kind of processing loop to the shellscript.

while read -r aa
do
var=$aa
# do some processing
done < $1

You can input the filename to be processed as $1

shellscript.sh file1.txt

SEP

[A. Clay Stephenson]

You need to use the "read" or "line" commands.

For example:

while read A B C
do
echo "A=${A} B=${B} C=${C}"
done

That will read stdin until EOF. If there are more than three variables to parse anything beynd the 2 argument will be combined in ${C}.

man sh-posix for details.

[Johan Barelds]

Thanks all!

The "while read xx" loop was what i was looking for!

Grz. Bo

[rmueller58]

to use a file as a input it depends on what you are trying to accomplish..

I define variables from information in the file, then use the variable within the script..

If you need to read information from the file in a and perform a task based on multiple variables using a for-loop or do-while is very useful...



For example:

export PEIDBNAME
echo $PEIDBNAME
export DIST
echo $DIST
awk '{print $1,$2,$3,$4,$5,$6,$7,$8}' $DIST$BUILD|pr -c 2 -t
echo "Enter Build # supported by this printer:\c"
read bldgnum
echo $bldgnum
touch /tmp/btemp
chmod 666 /tmp/btemp
export BN=`grep "$bldgnum" $DIST$BUILD`
echo $BN
grep "$BN" $DIST$BUILD >/tmp/btemp
export BNAME=`awk '{print tolower(substr($2,1,3))}' /tmp/btemp`
echo $BNAME
export BNAME2=`awk '{print substr($2,1,3)}' /tmp/btemp`
echo $BNAME2
#clear

#### SEE $DIST$BUILD EXAMPLE BELOW:
# more conbuild
001 ADMINISTRATION BUILDING
002 HIGH SCHOOL
003 MIDDLE SCHOOL
004 ELEMENTARY SCHOOL
005 SPECIAL ED SCHOOL





ANOTHER EXAMPLE
#####
search for a string in a file and create another file...

####

echo "Enter the School District ID to exclude from the passwd file:"
read $dist
grep -v -h $dist passwd.org > passwd.restricted


#####
ANOTHER EXAMPLE


which your shell script for input information you can use a for loop or do while ..

### CREATING A TEMP FILE ####

echo "Enter the District to Alert:\c"
read dst
echo "Please Enter the message you want to send:\c"
read mssg
echo "Please Enter your Name:\c"
read name

usrmsg |grep "$dst"|awk '{print $2, $3}'> /tmp/usralert
for fgls in `awk '{print $1}' /tmp/usralert`
do
export FGLSERVER=$fgls
export LOGNAME=`grep "$fgls" /tmp/usralert |awk '{print $2}'`
echo $FGLSERVER $LOGNAME $dst $mssg $name
fglrun message.42r "$mssg" "$name"



Just a couple of thoughts and ways I handle it..

the beauty of Un*x is there are a hundred ways to do one operation..

[Alex Ostapenko]

Hi Bo --

Someone mentioned the "while read" method which you seemed to like. That's a nice quick'n'dirty technique, but for large input files (>1000 lines?), it can have severe performance problems.

Once you get the script loop working, you might want to convert the "while read" guts into awk, then simply put into your ksh script:

cat - | awk 'your program'

then pipe into your script just like you did with the "while read" version of the script:

cat your-data | your-script.ksh

In a real-life example, processing data that took 30 minutes with "while read" took only 90 seconds with "awk" (I don't remember how big the file was). It has something to do with the shell buffering only 1 char at a time versus awk buffering 1K chars at a time.

Good luck!

=:-) Alex

[Elmar P. Kolkman]

The difference with awk and the shell is that the shell cannot buffer, because processes inside the while loop could also need the same stdin... awk is one process and can buffer because no other commands inside the 'loop' will need the stdin without using the awk itself.

The speed advantage also depends on what you want to do inside the loop. If you only do some pattern matching and re-formatting, awk is the way to go. But if you need to run difficult unix commands from it, the shell could be a better solution.