for loop problem

[Darrell Campbell]

I am trying to write a ksh script that has a for loop inside of it. I read a file that compares a date by using awk and for every line in the file that has that date, I then awk that output again and get the first field out of that line and populate a temp file. This temp file is what I try to use as a variable list in the for loop. To my knowledge, the first time through the loop, the for statement assigns the value of the first element in the list to the variable. The next time through the list, the for statement assigns the value of the second element in the list to the variable. What is actually happening is the the first time through the loop, the for statement is assigning all of the elements in the list to the variable. The second time through the loop, it assigns the second element as it should and it proceeds correctly from there. Does anybody have any idea why this would happen or any other suggestions?

[James R. Ferguson]

Hi Darrell:

Posting the actual code you have would help us help you.

Regards!

...JRF...

[AwadheshPandey]

Agreed with JRF, Pls post the code

[Darrell Campbell]

OK. Attached is the code that I have written

[A. Clay Stephenson]

I suspect that you have a quoting problem but w/o seeing your code and data it's difficult to do.

Try this to get some ideas:

create a text file, x.txt that looks something like this:

aaa bbb
ccc
ddd eee
fff gg

Now execute the following script and see if the results are what you expect:

#!/usr/bin/ksh

FILES="aa bb cc dd ee ff ggg"

echo "Test 1"
for F in ${FILES}
do
echo "${F}"
done

echo "Test 2"
for F in "${FILES}"
do
echo "${F}"
done

echo "Test 3"
for F in $(cat x.txt)
do
echo "${F}"
done

echo "Test 4"
for F in "$(cat x.txt)"
do
echo "${F}"
done


I suspect that one of these outputs will ring a bell.

[Patrick Wallek]

The first thing I would do is change this:

# Loop through and set temp files for users
user1=$(cat userqty_temp )
for user in $user1


to this:

for user in $(cat userqty_temp)

I don't know if it will make a difference, but it is possible.

Second, I would change the use of the back-ticks ( `statement` ) for executing stuff and instead use the $(statement) syntax. It drastically improves readability and prevents confusion of ' for `.

[James R. Ferguson]

Hi (again) Darrell:

Each time through the 'for' loop you do something like:

# cat accounts_to_process_temp |grep $user |awk '{print $3}' > userc_$user

...By using ">" you clobber the output file with each pass leaving one (the last) entry.

Regards!

...JRF...

[James R. Ferguson]

Hi (again) Darrell:

Each time through the 'for' loop you do something like:

# cat accounts_to_process_temp |grep $user |awk '{print $3}' > userc_$user

...By using ">" you clobber the output file with each pass leaving one (the last) entry.

I think you mean to use ">>" to collect multiple entries.

Regards!

...JRF...

[James R. Ferguson]

Hi Darrell:

As an aside:

Instead of spawning multiple processes in a pipeline to read a file (with 'cat'); discard lines (with 'grep') and then create a separate process ('awk') to print fields from the stream, you can do:

# awk -v user=Darrell '{if ($0 ~ user) {print $1}}' yourfile

...instead of:

# cat yourfile | grep Darrell | awk '{print $1}'

You will find this much more efficient and kinder on your system.

Regards!

...JRF...

[Josiah Henline]

Try changing the "for" line and add a "print" statement to see what the variable is. You can also try running the script with "ksh -x" for debugging.

head -1 userqty_tmp #Check the first line of the file.

for user in $(cat userqty_temp ) #This is the recommended way of doing the loop.
do
echo $user #This will tell you what value the variable has been assigned.
cat accounts_to_process_temp |grep $user |awk '{print $3}' > userc_$user
cat accounts_to_process_temp |grep $user |awk '{print $4}' > userpv_$user
cat accounts_to_process_temp |grep $user |awk '{print $5, $6}' > username_$user
cat accounts_to_process_temp |grep $user |awk '{print $7, $8}' > usersuper_$user
cat accounts_to_process_temp |grep $user |awk '{print $9}' > usersupermail_$user
# Find information from file
userCID=`cat /var/vmi_bin/security/account_security/userc_$user`
userPVID=`cat /var/vmi_bin/security/account_security/userpv_$user`
username=`cat /var/vmi_bin/security/account_security/username_$user`
usersuper=`cat /var/vmi_bin/security/account_security/usersuper_$user`
usersupermail=`cat /var/vmi_bin/security/account_security/usersupermail_$user`
print "user = $user"
print "userCID = $userCID"
print "userPVID = $userPVID"
print "username = $username"
print "usersuper = $usersuper"
print "usersupermail = $usersupermail"
done

[A. Clay Stephenson]

cat accounts_to_process_temp |grep $user |awk '{print $3}' > userc_$user

Using grep to match your users also has potential for failure (or perhaps I should say for working too well).

Consider the case where $user = "bob". That would match "bob","bobby","bubbabob","bob2" .... You can improve that using grep -E with anchored patterns but reallly awk is more than capable (and is a better choice) for doing the matching and the extraction.

[Bill Hassell · Accepted solution]

Another technique for reading file contents is to use cat/while read as in:

cat userqty_temp | while read user
do
echo $USER
...
done

but reading a file produces all the lines

One advantage to using read is that you can extract multiple items from each line into separate variables. Suppose you are reading a file with the user, the date, a home directory and size:

cat userqty_temp | while read USER DATE HOME SIZE
do
echo $USER $DATE $HOME $SIZE
...
done

I like to use UPPERCASE for variable names for readability.

[Darrell Campbell]

I agree with what you all are saying but regardless of what I have done, the first iteration puts thte entire list into the variables. Look at the following and see if the output makse sense. Look at user1 items and compare with the items under user2. See how all are listed in user1 but only the second in user2.

user = 1
userCID = istest
dc
userPVID = istest
campbelldc
username = Calvin Hartless
Joe Blow
usersuper = Alan Campbell
Darrell Campbell
usersupermail = campbelldc@vmi.edu
campbelldc@vmi.edu
user = 2
userCID = dc
userPVID = campbelldc
username = Joe Blow
usersuper = Darrell Campbell
usersupermail = campbelldc@vmi.edu

[Patrick Wallek]

What does your original data file look like? Can you post the first 3 or 4 lines?

[Darrell Campbell]

The original data file that I am extracting data from looks like this:
number|date|ColleagueuserID|PostViewuserID|username|supervisorname|superemail
1 051606 istest istest Calvin Hartless Alan Campbell campbelldc@vmi.edu
2 051606 dc campbelldc Joe Blow Darrell Campbell campbelldc@vmi.edu
3 051706 not_applicable webuser Big Boy Vern Beitzel campbelldc@vmi.edu

In an essense, I compare the date to tdays date and see how many people from this file should be processed.

[Bill Hassell]

OK, what was missing from the example was that you had more than just one space-separated item on the line. The cat command would then create a for loop that looks like this:

for user in 1 051606 istest istest Calvin Hartless Alan Campbell campbelldc@vmi.edu

which means that the loop will start with 1, then 051606, then istest, etc in each loop iteration. That's where the cat/read form will work much better -- and the advantage that you have immediate access to each element in the line:

cat userqty_temp | while read LINE DATE USER GROUP RESTOFLINE
do
echo $DATE
...
done

The RESTOFLINE is ambiguous because the user and supervisor names might not always be two words.

[Josiah Henline]

Darrell,

Bill is correct. The file cannot be parsed with the cat command. If each line is numbered, you can use grep to capture each line in a while loop using an array.

LINEN=$(wc -l file_name)
TMPVAR=1
while (TMPVAR < LINEN)
do
set -A FILELINE $(grep "^${TMPVAR} " file_name)
#run a command to extract ${FILELINE[0]}
#run a command to extract ${FILELINE[1]}
#run a command to extract ${FILELINE[2]}
#run a command to extract ${FILELINE[3]}
#run a command to extract ${FILELINE[4]}
#run a command to extract ${FILELINE[5]}
(( TMPVAR += 1 ))
done

This assumes that all names consist of the same number of words. ie. John Smith, Fred Jones. Note they all have two words.

[Peter Nikitka]

Hi,

I think, you can preprocess your file before doing your real data processing.
Assuming a fixed datestring 051606 to compare with

awk -v dstr=051606 'NF>1 && ($2==dstr)' yourfile

will preselect all entries of the given datestring.
You can then do more processing in awk or use Bills 'while'-solution
awk .. | while read v1 v2 v3 restofline
do
...
done

mfG Peter

[Sandman!]

IMHO change the grep pattern inside the for loop:

From...
cat accounts_to_process_temp|grep $user |awk '{print $3}' > userc_$user
To...
cat accounts_to_process_temp |grep "^$user " |awk '{print $3}' > userc_$user

repeat for each grep pattern within your for loop, so it matches exactly one instance, thereby eliminating the discrepancy below...
> the first iteration puts thte entire list into the variables <

cheers!

[Darrell Campbell]

You guys are great and some of the responses were very helpful. I tried/tested all of the different suggestions on items to look at, but I ended up using Bill's read suggestion for the program. I still think it should have worked with the "for" loop as I was using and still do not really understand why it would apply all to first iteration and then progress as desired. Regradless I appreciate the responses!!!!