Help with an indexing shell script.

[Luke Morgan]

Hi,

I have a shell script that takes a directory of item files, and cuts out two values from each. A year/week value and a quantity. It then matches the year/week value (eg 200322) against an index file, and replaces it.

The index file contains year/week values in order and its second column contains an incremental value. Thus, after this script is run, each item file contains quantities with a number corresponding to its year/week.
After this, I have a perl script that pivots the item file into a row, with the indexed number indicating which column the quantity value is placed into.

The problem is, this indexing script (attached) takes forever and a day to complete as it is performing a number of embedded loops.
Can anyone think of a more elegant and fast way of performing the same action?

Thanks for your time

Luke

[Claudio Cilloni]

at a first look, it seems to me that the first two lines ('idxyearweek=...' and 'index=...') of the innermost loop (for x...) generate always the same results every time the loop is executed.
Maybe generating there results only 1 time and then re-read them could help. And even better could be keep them into memory (if they didn't need plenty of ram space). I don't know how to do this in shell, I'm not so expert about it.
Personally, I'll re-write this script in Perl or (my favorite) Python; it should be easy and surely run faster.

hth
Claudio

[Luke Morgan]

Thanks for that Claudio.
If you know how to re-write the script easily in perl, could you help me?
My perl skills are worse than terrible! :)

Luke

[Claudio Cilloni]

Since I started using Python (1 year ago) I immediately stopped using Perl, so my skills aren't so good now, too... (There's someone in this forum who hate me now :-)

I think I could rewrite it in 5-10 mins with the snake, but going to dig in my Perl's memories could take me much more time. Maybe somebody out there could come and help us!

Ciao
CC

[Luke Morgan]

If python is the way to go, I am open to using that.
Especially if it would take you 5mins to solve my problem! ;)
The script is running on RedHat 7.2
I think python is bundled with it by default?
It would appear that it is from a quick file search...
I have to confess to knowing very little about python. Can I just call it from a shell script?
If so, it seems that could be the ideal solution....

Luke

[Claudio Cilloni]

I made the python script (it is attached).
you can run it as it was a shell script (look at the first like with the name of the interpreter; it must point to the path of your python executable. I'm using redhat 7.3, maybe /usr/bin/python2 is right).

you can give it the execute permission just like any other script, chmod +x . Even the .py extension isn't necessary.
Or you can run it with the interpreter directly:
$ python2

It is a little bit longer that your shell script, 'cause shell is more file-manipulation oriented (but python is great when manipulating strings).

Hope it works, I haven't tested it because i haven't your data.

Tell me if it runs.

Claudio

[Luke Morgan]

Thank you so much for that!
I named the script python.py and when I ran it, I got the following...

# /usr/bin/python python.py
Traceback (innermost last):
File "python.py", line 8, in ?
f = file(datadir + 'arrayfile', 'r')
NameError: file

[Claudio Cilloni · Accepted solution]

this error is caused by the old version of the interpreter, python 1.5. It is installed because RH7.2 needs it for its redhat-specific administrative tools.

You can do two things:

1. change the function name 'file' to 'open' (this is its name in python 1.5, but now it is deprecated),
2. use the python 2 interpreter. It should be installed:

$ rpm -qa | grep python2

should show a line like 'python2-xxx.xxx-xxx'
if it is, with the command

$ which python2

you can find the full path of the interpreter.

Hi
Claudio

[Luke Morgan]

Thank you Claudio,
your help has been superb!
The python script is excellent! Far quicker.
Only one small problem.
I can best illustrate by comparing the results from the original shell script, to the python result.

Shell script:
116|106
124|96
125|101
127|2
130|50

Python script:
116|106
124|96
125|101
127|2
130|50
116|106

124|96

125|101

127|2

130|50


Luke

[Claudio Cilloni]

I cannot figure out what's the difference. could you post here a little subset of your data (with the directory structure) and the right results? so I can run my script and do a comparison.

Hi
Claudio