Simple Pattern Matching Logic In Korn Shell

[pareshan]

I am trying to implement a logic which is not working,

I have two lists, market_list and file_system_list

cat market_list
ABC BCA CDA ERF RKT PFR NTO KDR KIT GGO SPT KOT VOT KDR WWL CLW OPE REG BVG YYU

-----market_list can be in horizontal or vertical form whatever works--------

cat file_system_list

/usr/local/g1
/imrga/geotax
/imrga/code1
/imrga/vertex
/usr/local/matro/mea82
/usr/local/STW/
/tlgop
/tlglogs
/imrga
/tglrav
/toplogs
/usr/local/STW/STW88
/topfbf
/toplogsmae


I have list of the servers which i haven't included here and i have to go all the servers and check for those filesystems for the percentage used.

Its ok for regular filesystems but the problem is with the filesystem which has market name on it

in my file_system_list which has in the list means it can be any one of those market which i have to try with one market at a time it may exist and it may not exist depending on the server.

how can I replace that
in the file_system_list with the name of the market from the market_list but one at a time and run the bdf command.
for example---this works but only for regular file systems not for those where I have to remove by market name one at a time and test its presence in all the servers

#!/bin/ksh
work_path=`pwd`
tmp1=$work_path/tmp1
for server in `cat server_list`
do
for file_system in `cat file_system_list`
do
ssh -o 'BatchMode yes' $server "id"
if [ $? = 0 ] ; then
echo "Checking File Systems Usage For $server" >> $tmp1
echo "---> ---> --> --> --> --> On $file_system" >> $tmp1
ssh -n $server "bdf $file_system" >> $tmp1
if [ $? != 0 ] ; then
echo "no $file_system in $server" >> $tmp1
fi
else
echo $server >> server_with_problems
fi
done
echo " All File System Check Completed For $server " >> $tmp1

done



Let me know if anything is not clear,

my problem is I am not able to get the logic to replace the by the name of the market from my market_list file one at a time and check it

thank you in advance
i will appreciate any help

[Dennis Handly · Accepted solution]

>I am not able to get the logic to replace the by the name of the market from my market_list file one at a time and check it

You just need another loop:
...
if [[ "$file_system" = *\* ]]; then
for fs in $(< market_list); do
xx=$(echo $file_system | sed -e "s::$fs:")
echo "---> ---> --> --> --> --> On $xx" >> $tmp1
echo ssh -n $server "bdf $xx" >> $tmp1
if [ $? -ne 0 ]; then
echo "no $xx in $server" >> $tmp1
fi
done
else # existing code
echo "---> ---> --> --> --> --> On $file_system" >> $tmp1
echo ssh -n $server "bdf $file_system" >> $tmp1
if [ $? -ne 0 ] ; then
echo "no $file_system in $server" >> $tmp1
fi
fi

Note testing the ssh result will return the bdf exit status. This isn't the case for remsh:
Note that the return value of remsh bears no
relation to the return value of the remote command.

[Peter Nikitka]

Hi,

I would loop just over the server list to get a complete bdf-output of each single server (much less ssh traffic) and then would parse that output.
- I would initialize an array with the real filsystem names and check against the bdf data.
- I assume the filesystem is the last field of the bdf-output, which I remove form this array
- the remaining array elements (if any) are the missing entries.

awk -v mkt="$(tr '\012' ' ' while ((getline ") for(i=1;i<=mk;i++) {fs2chk[++f]=$0; sub("",mktlst[i],fs2chk[++f])} else fs2chk[++f]=$0}
# now create an associative array
for(n=1;n<=f;n++) myfs2ck[fs2chk[n]]=fs2chk[n]}
NF>3 {delete myfs2ck[$NF]}
END { for (fsmis in myfs2ck) print "missing",fsmis}' bdf-outputfile

mfG Peter

PS. This code is not really tested ...

[pareshan]

Thanks Danny,

Its working but in else instead of

---
else
echo $server >> server_with_problems

it just should be
else
$server >> server_with_problems
i guess.

Nikita,

I am trying to understand the one you provided.
sounds good as can reduce the traffice and may be looks efficient too but not working in mine.

Could you explain me better
thanks

[Peter Nikitka]

Hi,

would you please tell me, WHAT is not working for you? Could you send me the output for the run of one server?

mfG Peter

[pareshan]

I dint get a whole idea of the script
I mean how to run where to start where does it ends

when i run it in my terminal as it is provided by you
it says


niki.ksh
niki.ksh[2]: /tmp/market_lst: Cannot find or open the file.
MK=0
awk: Cannot find or open file bdf-outputfile.
The source line number is 1.

[Peter Nikitka]

Hi,

I expected, you could puzzle the parts together by yourself - ok. Of course, you'll have to provide YOUR filenames, not mine.
I use plain filenames now - please modify them, if needed!

Modify your script this way:

#!/bin/ksh
work_path=`pwd`
tmp1=$work_path/tmp1
for server in `cat server_list`
do
echo "Checking File Systems Usage For $server"
ssh -n $server "bdf" > $tmp1
if [ $? != 0 ] ; then
awk -v mkt="$(tr '\012' ' ' while ((getline ") for(i=1;i<=mk;i++) {fs2chk[++f]=$0; sub("",mktlst[i],fs2chk[++f])} else fs2chk[++f]=$0}
# now create an associative array
for(n=1;n<=f;n++) myfs2ck[fs2chk[n]]=fs2chk[n]}
NF>3 {delete myfs2ck[$NF]}
END { for (fsmis in myfs2ck) print "missing",fsmis}' $tmp1
fi
echo " All File System Check Completed For $server "

done

[Dennis Handly]

>It's working but in else instead of ...
>it just should be
$server >> server_with_problems

I'm not sure why you would want to leave out the echo? I left out everything except the innermost for loop.

You can also take that loop out and expand the files in file_system_list once, then just have the two for loops.

[pareshan]

Nikita,
Thanks alot it did run but I am not sure whether I am not understanding you or you dint get my problem.

I need to go to all the servers Listed in server_list
and check for those file systems and their usages (bdf)
listed in
file_system_list

And in those file systems some of them have attached at the end and for thos I have to repalce that with the markets name from market_list one at a time and check existance of that file system, if exist it should give me a value of bdf otherwise it should print in the file itself or in other file there is no file system by that name in that particular server.


As far as I know what you are doing you are doing bdf first and with the name of the file systems there you are trying to match with the filesystem I have in the list?

But there should be somewhere to print all the result

see this is a complete script for the whole thing which is working 100 percent fine just its taking alot of time and I thought your one can solve my problem..

script
______________


#!/bin/ksh
#set -xv
work_path=`pwd`
echo $work_path
tmp1=$work_path/tmp1

for server in `cat server_list`; do
for file_system in `cat file_system_list`; do
ssh -o 'BatchMode yes' $server "id"
if [ $? = 0 ] ; then
echo "Checking File Systems Usage For $server" >> $tmp1
echo $file_system
if [[ "$file_system" = *\* ]]; then
for fs in $(< market_list); do
xx=$(echo $file_system | sed -e "s::$fs:")
echo "---> ---> --> --> --> --> On $xx" >> $tmp1
## echo ssh -n $server "bdf $xx" >> $tmp1
ssh -n $server "bdf $xx" >> $tmp1
if [ $? -ne 0 ]; then
echo "no $xx in $server" >> $tmp1
fi
done
else
##if [ echo $file_system | grep mkt|wc -l = 0 ]; then
echo "---> ---> --> --> --> --> On $file_system" >> $tmp1
ssh -n $server "bdf $file_system" >> $tmp1
if [ $? != 0 ] ; then
echo "no $file_system in $server" >> $tmp1
fi
fi
else
echo $server >> server_with_problems
fi
done
echo " All File System Check Completed For $server " >> $tmp1
done

---------------

Danny,
I have one questoin here, everything is fine but its taking long time (really long like hours) and the output is so huge because of so many servers and I am trying to make it more efficient.

I have like 200 servers where I need to run the script so I have divided all the servers by types
and I made the file called
server_type
which contains the type like

$ cat server_type
NG1
NG2
TopDB
TopDBHA
CUSTOMER
BATCH1
BATCH1bu
MAFGGDB
MAFGDB
REPORT
TUXEDO1
TUXEDO2
TUXEDO3
TUXEDO4
USAGEDB
USAGE1
USAGE2
USAGE3
USAGE4


and I also have a file name by those types which has the list of servers under those types

like
$ cat REPORT
bhpcrpt
bhperpt
bhpind3
bhpnbi3
dhpbos3
dhpcrpt
olyeller
uhpcrpt

$ cat USAGE1
ind2
bi2a
rtg10
rtg12
rtg8
ckett
alh2
bos2
dhpm2
dha2a
frod
gucho
maattn
moear
pecdfr
riin10


and like that.

Now I am thinking to change that code into like this

-----------------
$ cat file_sys_report.ksh
#!/bin/ksh
#set -xv
work_path=`pwd`
echo $work_path
tmp1=$work_path/tmp1

for st in `cat server_type`; do
##for server in `cat server_list`; do
for server in `$st`; do

for file_system in `cat file_system_list`; do
ssh -o 'BatchMode yes' $server "id"
if [ $? = 0 ] ; then
echo "Checking File Systems Usage For $server" >> $tmp1_$st
echo $file_system
if [[ "$file_system" = *\* ]]; then
for fs in $(< market_list); do
xx=$(echo $file_system | sed -e "s::$fs:")
echo "---> ---> --> --> --> --> On $xx" >> $tmp1
## echo ssh -n $server "bdf $xx" >> $tmp1_$st
ssh -n $server "bdf $xx" >> $tmp1_$st
if [ $? -ne 0 ]; then
echo "no $xx in $server" >> $tmp1_$st
fi
done
else
##if [ echo $file_system | grep mkt|wc -l = 0 ]; then
echo "---> ---> --> --> --> --> On $file_system" >> $tmp1_$st
ssh -n $server "bdf $file_system" >> $tmp1_$st
if [ $? != 0 ] ; then
echo "no $file_system in $server" >> $tmp1_$st
fi
fi
else
echo $server >> server_with_problems
fi
done
echo " All File System Check Completed For $server " >> $tmp1_$st
done
done
-------------------

but its not working I guess, it says those servers not found
this is a part of the error which is too long

CUSTOMER: bowie: not found
CUSTOMER[2]: dhpalh1: not found
CUSTOMER[3]: ferguson: not found
CUSTOMER[4]: larry: not found
CUSTOMER[5]: rubble: not found
CUSTOMER[6]: thaspen: not found
BATCH1: bhptlg66: not found
BATCH1[2]: bhptlg72: not found
BATCH1[3]: bhptlga4: not found
BATCH1[4]: bhptlga7: not found
BATCH1[5]: dhptg45a: not found
BATCH1[6]: dhptg46a: not found
BATCH1[7]: dhptg47a: not found
BATCH1[8]: dhptg48a: not found
BATCH1[9]: uhptlg18: not found
BATCH1bu: bhptlg67: not found
BATCH1bu[2]: bhptlg70: not found
BATCH1bu[3]: dhptg45b: not found
BATCH1bu[4]: dhptg46b: not found
BATCH1bu[5]: dhptg47b: not found
BATCH1bu[6]: dhptg48b: not found
BATCH1bu[7]: uhptlg21: not found


So I am wondering what is wrong there and how other way to make it efficient??

Thanks Alot
Appreciate Any Help

[James R. Ferguson]

Hi:

I just wanted to point out that it's "Dennis" and NOT "Danny" to whom you refer.

Secondly, I'm honored that you named a server after me! viz.

CUSTOMER[3]: ferguson:

:-}}

Regards!

...JRF...

[Dennis Handly]

This version computes the filesystems once:
work_path=$PWD
FS_LIST=/var/tmp/file_system_list2
tmp1=$work_path/tmp1
# compute filesystems
for file_system in $(< file_system_list); do
if [[ "$file_system" = *\* ]]; then
for fs in $(< market_list); do
echo $file_system | sed -e "s::$fs:"
done
else
echo "$file_system"
fi
done > $FS_LIST

for server in $(< server_list); do
ssh -o "BatchMode yes" $server "id"
if [ $? -eq 0 ]; then
echo "Checking File Systems Usage For $server" >> $tmp1
for file_system in $(< $FS_LIST); do
echo "---> ---> --> --> --> --> On $file_system" >> $tmp1
ssh -n $server "bdf $file_system" >> $tmp1
if [ $? -ne 0 ] ; then
echo "no $file_system in $server" >> $tmp1
fi
done
else
echo $server >> server_with_problems
fi
echo " All File System Check Completed For $server " >> $tmp1
done

rm -f $FS_LIST

I pulled the first ssh out of the inner loop.

>it's taking long time and the output is so huge because of so many servers and I am trying to make it more efficient.

If you need that huge output, you need to take a long time.

Does each server really have all of those filesystems? Or are some over NFS? If they are, there is no need to run bdf on them more than once. Ideally do it only on the server, otherwise on one client.

>I have like 200 servers where I need to run the script so I have divided all the servers by types and I made the file called
server_type

I'm not sure how that helps, unless you are only doing some at a time? Or you are doing them in parallel?

>it says those servers not found this is a part of the error which is too long
CUSTOMER: bowie: not found

Your second for loop syntax is wrong:
for st in $(< server_type); do
for server in $(< $st); do

You should also not use `` but use the easier to read $().

Note: This will not speed things up, nor shorten the output. You'll still be doing 200 servers.

[Peter Nikitka]

Hi Pareshan,

now it is clear, that
- you don't want to check, if there are filsystem missing
- not all possible filesystem names constructed by substituting '' with all pattern of the list must exist
- you want to match filesystem names against some pattern
- report their diskusage

Now try:
#!/bin/ksh
work_path=`pwd`
tmp1=$work_path/tmp1
tmp2=$work_path/dftmp
for server in `cat server_list`
do
echo "Checking File Systems Usage For $server"
ssh -n $server "bdf" > $tmp2
if [ $? != 0 ] ; then
awk -v mkt="$(tr '\012' ' ' while ((getline ") for(i=1;i<=mk;i++) {fs2chk[++f]=$0; sub("",mktlst[i],fs2chk[++f])} else fs2chk[++f]=$0}
# now create an associative array
for(n=1;n<=f;n++) myfs2ck[fs2chk[n]]=fs2chk[n]}
NF>3 {if(myfs2ck[$NF]) reportfs[$NF]=$0}
END { for (fs in reportfs) print reportfs[fs]}' $tmp2
fi
echo " All File System Check Completed For $server "
done | tee $tmp1

mfG Peter
PS: drop the 'tee' command, when there is no need to watch what happens during the runtime of this script; use
...
done >$tmp
instead then.

[Peter Nikitka]

Hi pareshan,

one follow up:
If all filesystems in play are local, use
bdf -l
as command - this can limit the output and speed up things.

mfG PETER