topic Re: Help with an indexing shell script. in Operating System - Linux

Help with an indexing shell script.

Luke Morgan — Wed, 02 Jul 2003 12:02:24 GMT

Hi,

I have a shell script that takes a directory of item files, and cuts out two values from each. A year/week value and a quantity. It then matches the year/week value (eg 200322) against an index file, and replaces it.

The index file contains year/week values in order and its second column contains an incremental value. Thus, after this script is run, each item file contains quantities with a number corresponding to its year/week.
After this, I have a perl script that pivots the item file into a row, with the indexed number indicating which column the quantity value is placed into.

The problem is, this indexing script (attached) takes forever and a day to complete as it is performing a number of embedded loops.
Can anyone think of a more elegant and fast way of performing the same action?

Thanks for your time

Luke

Re: Help with an indexing shell script.

Claudio Cilloni — Wed, 02 Jul 2003 12:36:25 GMT

at a first look, it seems to me that the first two lines ('idxyearweek=...' and 'index=...') of the innermost loop (for x...) generate always the same results every time the loop is executed.
Maybe generating there results only 1 time and then re-read them could help. And even better could be keep them into memory (if they didn't need plenty of ram space). I don't know how to do this in shell, I'm not so expert about it.
Personally, I'll re-write this script in Perl or (my favorite) Python; it should be easy and surely run faster.

hth
Claudio

Re: Help with an indexing shell script.

Luke Morgan — Wed, 02 Jul 2003 12:45:25 GMT

Thanks for that Claudio.
If you know how to re-write the script easily in perl, could you help me?
My perl skills are worse than terrible! :)

Luke

Re: Help with an indexing shell script.

Claudio Cilloni — Wed, 02 Jul 2003 13:08:30 GMT

Since I started using Python (1 year ago) I immediately stopped using Perl, so my skills aren't so good now, too... (There's someone in this forum who hate me now :-)

I think I could rewrite it in 5-10 mins with the snake, but going to dig in my Perl's memories could take me much more time. Maybe somebody out there could come and help us!

Ciao
CC

Re: Help with an indexing shell script.

Luke Morgan — Wed, 02 Jul 2003 13:14:13 GMT

If python is the way to go, I am open to using that.
Especially if it would take you 5mins to solve my problem! ;)
The script is running on RedHat 7.2
I think python is bundled with it by default?
It would appear that it is from a quick file search...
I have to confess to knowing very little about python. Can I just call it from a shell script?
If so, it seems that could be the ideal solution....

Luke

Re: Help with an indexing shell script.

Claudio Cilloni — Wed, 02 Jul 2003 13:43:41 GMT

I made the python script (it is attached).
you can run it as it was a shell script (look at the first like with the name of the interpreter; it must point to the path of your python executable. I'm using redhat 7.3, maybe /usr/bin/python2 is right).

you can give it the execute permission just like any other script, chmod +x . Even the .py extension isn't necessary.
Or you can run it with the interpreter directly:
$ python2

It is a little bit longer that your shell script, 'cause shell is more file-manipulation oriented (but python is great when manipulating strings).

Hope it works, I haven't tested it because i haven't your data.

Tell me if it runs.

Claudio

Re: Help with an indexing shell script.

Luke Morgan — Wed, 02 Jul 2003 14:02:28 GMT

Thank you so much for that!
I named the script python.py and when I ran it, I got the following...

# /usr/bin/python python.py
Traceback (innermost last):
File "python.py", line 8, in ?
f = file(datadir + 'arrayfile', 'r')
NameError: file

Re: Help with an indexing shell script.

Claudio Cilloni — Wed, 02 Jul 2003 14:15:57 GMT

this error is caused by the old version of the interpreter, python 1.5. It is installed because RH7.2 needs it for its redhat-specific administrative tools.

You can do two things:

1. change the function name 'file' to 'open' (this is its name in python 1.5, but now it is deprecated),
2. use the python 2 interpreter. It should be installed:

$ rpm -qa | grep python2

should show a line like 'python2-xxx.xxx-xxx'
if it is, with the command

$ which python2

you can find the full path of the interpreter.

Hi
Claudio

Re: Help with an indexing shell script.

Luke Morgan — Wed, 02 Jul 2003 14:34:44 GMT

Thank you Claudio,
your help has been superb!
The python script is excellent! Far quicker.
Only one small problem.
I can best illustrate by comparing the results from the original shell script, to the python result.

Shell script:
116|106
124|96
125|101
127|2
130|50

Python script:
116|106
124|96
125|101
127|2
130|50
116|106

124|96

125|101

127|2

130|50

Luke

Re: Help with an indexing shell script.

Claudio Cilloni — Thu, 03 Jul 2003 05:15:57 GMT

I cannot figure out what's the difference. could you post here a little subset of your data (with the directory structure) and the right results? so I can run my script and do a comparison.

Hi
Claudio

Re: Help with an indexing shell script.

Claudio Cilloni — Thu, 03 Jul 2003 06:40:21 GMT

you can put all in a .zip or .tar.gz.

Re: Help with an indexing shell script.

Luke Morgan — Thu, 03 Jul 2003 07:03:13 GMT

Thanks again Claudio.
I have attached a small set of data as you asked.

Luke

Re: Help with an indexing shell script.

Luke Morgan — Thu, 03 Jul 2003 07:13:22 GMT

I have re-run the script again this morning.

I am sorry, I made a mistake.
The only issue with the results, is that
there is a blank line between each value set.

I can write another script to remove the blanks but if the python could be altered slightly to remove them that would obviously be preferable.
Is that easy ?

Luke

Re: Help with an indexing shell script.

Claudio Cilloni — Thu, 03 Jul 2003 09:09:04 GMT

I corrected the script. It's attached.

This time should be good.

Have fun,
Claudio

Re: Help with an indexing shell script.

Luke Morgan — Thu, 03 Jul 2003 09:13:58 GMT

Excellent!

Thank you very much for all your help.
You have saved me lots of time.
Much appreciated.

Luke