Simpler Navigation for Servers and Operating Systems
Completed: a much simpler Servers and Operating Systems section of the Community. We combined many of the older boards, so you won't have to click through so many levels to get at the information you need. Check the consolidated boards here as many sub-forums are now single boards.
General
cancel
Showing results for 
Search instead for 
Did you mean: 

Help with an indexing shell script.

SOLVED
Go to solution
Luke Morgan
Frequent Advisor

Help with an indexing shell script.

Hi,

I have a shell script that takes a directory of item files, and cuts out two values from each. A year/week value and a quantity. It then matches the year/week value (eg 200322) against an index file, and replaces it.

The index file contains year/week values in order and its second column contains an incremental value. Thus, after this script is run, each item file contains quantities with a number corresponding to its year/week.
After this, I have a perl script that pivots the item file into a row, with the indexed number indicating which column the quantity value is placed into.

The problem is, this indexing script (attached) takes forever and a day to complete as it is performing a number of embedded loops.
Can anyone think of a more elegant and fast way of performing the same action?

Thanks for your time

Luke
14 REPLIES
Claudio Cilloni
Honored Contributor

Re: Help with an indexing shell script.

at a first look, it seems to me that the first two lines ('idxyearweek=...' and 'index=...') of the innermost loop (for x...) generate always the same results every time the loop is executed.
Maybe generating there results only 1 time and then re-read them could help. And even better could be keep them into memory (if they didn't need plenty of ram space). I don't know how to do this in shell, I'm not so expert about it.
Personally, I'll re-write this script in Perl or (my favorite) Python; it should be easy and surely run faster.

hth
Claudio
Luke Morgan
Frequent Advisor

Re: Help with an indexing shell script.

Thanks for that Claudio.
If you know how to re-write the script easily in perl, could you help me?
My perl skills are worse than terrible! :)

Luke
Claudio Cilloni
Honored Contributor

Re: Help with an indexing shell script.

Since I started using Python (1 year ago) I immediately stopped using Perl, so my skills aren't so good now, too... (There's someone in this forum who hate me now :-)

I think I could rewrite it in 5-10 mins with the snake, but going to dig in my Perl's memories could take me much more time. Maybe somebody out there could come and help us!

Ciao
CC
Luke Morgan
Frequent Advisor

Re: Help with an indexing shell script.

If python is the way to go, I am open to using that.
Especially if it would take you 5mins to solve my problem! ;)
The script is running on RedHat 7.2
I think python is bundled with it by default?
It would appear that it is from a quick file search...
I have to confess to knowing very little about python. Can I just call it from a shell script?
If so, it seems that could be the ideal solution....

Luke
Claudio Cilloni
Honored Contributor

Re: Help with an indexing shell script.

I made the python script (it is attached).
you can run it as it was a shell script (look at the first like with the name of the interpreter; it must point to the path of your python executable. I'm using redhat 7.3, maybe /usr/bin/python2 is right).

you can give it the execute permission just like any other script, chmod +x . Even the .py extension isn't necessary.
Or you can run it with the interpreter directly:
$ python2

It is a little bit longer that your shell script, 'cause shell is more file-manipulation oriented (but python is great when manipulating strings).

Hope it works, I haven't tested it because i haven't your data.

Tell me if it runs.

Claudio

Luke Morgan
Frequent Advisor

Re: Help with an indexing shell script.

Thank you so much for that!
I named the script python.py and when I ran it, I got the following...

# /usr/bin/python python.py
Traceback (innermost last):
File "python.py", line 8, in ?
f = file(datadir + 'arrayfile', 'r')
NameError: file
Claudio Cilloni
Honored Contributor
Solution

Re: Help with an indexing shell script.

this error is caused by the old version of the interpreter, python 1.5. It is installed because RH7.2 needs it for its redhat-specific administrative tools.

You can do two things:

1. change the function name 'file' to 'open' (this is its name in python 1.5, but now it is deprecated),
2. use the python 2 interpreter. It should be installed:

$ rpm -qa | grep python2

should show a line like 'python2-xxx.xxx-xxx'
if it is, with the command

$ which python2

you can find the full path of the interpreter.

Hi
Claudio
Luke Morgan
Frequent Advisor

Re: Help with an indexing shell script.

Thank you Claudio,
your help has been superb!
The python script is excellent! Far quicker.
Only one small problem.
I can best illustrate by comparing the results from the original shell script, to the python result.

Shell script:
116|106
124|96
125|101
127|2
130|50

Python script:
116|106
124|96
125|101
127|2
130|50
116|106

124|96

125|101

127|2

130|50


Luke
Claudio Cilloni
Honored Contributor

Re: Help with an indexing shell script.

I cannot figure out what's the difference. could you post here a little subset of your data (with the directory structure) and the right results? so I can run my script and do a comparison.

Hi
Claudio
Claudio Cilloni
Honored Contributor

Re: Help with an indexing shell script.

you can put all in a .zip or .tar.gz.
Luke Morgan
Frequent Advisor

Re: Help with an indexing shell script.

Thanks again Claudio.
I have attached a small set of data as you asked.

Luke
Luke Morgan
Frequent Advisor

Re: Help with an indexing shell script.

I have re-run the script again this morning.

I am sorry, I made a mistake.
The only issue with the results, is that
there is a blank line between each value set.

I can write another script to remove the blanks but if the python could be altered slightly to remove them that would obviously be preferable.
Is that easy ?

Luke
Claudio Cilloni
Honored Contributor

Re: Help with an indexing shell script.

I corrected the script. It's attached.

This time should be good.

Have fun,
Claudio
Luke Morgan
Frequent Advisor

Re: Help with an indexing shell script.

Excellent!

Thank you very much for all your help.
You have saved me lots of time.
Much appreciated.

Luke