Operating System - HP-UX
1829743 Members
1467 Online
109992 Solutions
New Discussion

Re: Faster method than caljd.sh to calculate dates

 
SOLVED
Go to solution

Faster method than caljd.sh to calculate dates

Hi experts:

I have a report which needs to determine the date 30 days from a given date unless that date falls on a weekend. In that case, I need the following Monday. A. Clay's caljd.sh script works perfectly but it is slow because I have to do thousands of these date calculations for each report run.
The dates are in mm/dd/yyyy form.

Using one of A.Clay's examples, I am doing this:

DTPLUS30=$(caljd.sh -S "/" $(caljd.sh -S "/" -c -n 30 -x 0 -x 6 $DT))

For an input date of 01/03/2000, $DTPLUS30 is 02/02/2000. All of my calculations are perfect. They are just SO SLOW!!! Sorry, A. Clay.

Does anybody know of a faster way to do this?

Thanks in advance,
Steve
17 REPLIES 17
Pete Randall
Outstanding Contributor

Re: Faster method than caljd.sh to calculate dates

Steve,

Have you tried the perl version: caljd.pl?


Pete


Pete
Mark Grant
Honored Contributor

Re: Faster method than caljd.sh to calculate dates

I haven't seen this script but thousands of these calculations are never going to be fast in a shell script. I would consider porting this script to C or possibly perl.

The shell isn't particularly good at math. You could try something weird like the following.

Get the output of "cal" into an array find the index of the element that represents the date you have and simply add thirty to it. That will work for the day of the month (which is the most annoying part). Take the resulting date. Hey, that's just so weird I might even try it myself. Counting days instead of calculating them!

Otherwise, some benefit might be had from preprocessing the report to make it more friendly to the script.

Never preceed any demonstration with anything more predictive than "watch this"
H.Merijn Brand (procura
Honored Contributor

Re: Faster method than caljd.sh to calculate dates

both caldj.sh and caldj.pl are available from my site in the contribution area. I hope they are still reasonable up to date.

Clay, if you're chimed in, could you please check, and if they are outdated, can you provide me with some more recent versions?

In the FAQ section, there's also a little perl example of using modules

http://www.cmve.net/~merijn or https://www.beepz.com/personal/merijn

Enjoy, have FUN! H.Merijn
Enjoy, Have FUN! H.Merijn
Steven E. Protter
Exalted Contributor

Re: Faster method than caljd.sh to calculate dates

The perl version is pretty fast in my D320 testing.

I never bothered with shell script version, though you remind me I need to do so on my Linux servers. Thanks.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
A. Clay Stephenson
Acclaimed Contributor
Solution

Re: Faster method than caljd.sh to calculate dates

Hi Steve:

The Perl version will be a bit faster but about a month ago I worked on a MUCH faster shell version - that I haven't posted yet; typical calculations are about 4-5x faster because almost all of the awk has been replaced with shell arithmatic and no awk temp files are needed. This means that you can easily yank the functions out and put them right in your script BUT you now have me thinking
that I can do better still in your case by allowing file input/output so that your shell script could call caljd.sh as a co-process. My guess is that it will be at least 10X faster. I'll try to post this by sometime this PM.

Clay
If it ain't broke, I can fix that.
Steven E. Protter
Exalted Contributor

Re: Faster method than caljd.sh to calculate dates

Just implemented caljd.sh in my month end processes. It sets the month and year for the name of web statistics and other log files that I'm archiving off into user directories and other places.

Thank you for a great script Mr. A. Clay Stevenson

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
A. Clay Stephenson
Acclaimed Contributor

Re: Faster method than caljd.sh to calculate dates

Hi Steve:

This version should fix you; in my little test it was about 20x faster doing repeated calculations. You still take a hit if skipping holidays (as defibed in /etc/acct/holidays) or if inputting/outputting month or week day names because those still rely upon awk but for the usual applications this version is much faster.

Again, you can simply yank out the cal_jdate, jdate_cal, and wkday functions and use them inside your own scripts.

Use it like this:

#!/usr/bin/sh

INFILE="./mydates"

caljd.sh -S "/" -n 30 -x 0 -x 6 -c -f | caljd.sh -S "/" -f |&
# The above is all one line

cat ${INFILE} | while read DT
do
print -p ${DT}
read -p S
echo "${DT} --> ${S}"
done

Note the new "-f" arguments. That instructs caljd.sh to do read stdin and write on stdout.
The command line args are parsed only once.

Pay careful attention to the "|&" invocation. That starts caljd.sh as a co-process. The -p arguments for the read and print statements direct the i/o from/to the co-process.

Now, I've got to do the same thing to caljd.pl but it should be much easier.


If it ain't broke, I can fix that.
A. Clay Stephenson
Acclaimed Contributor

Re: Faster method than caljd.sh to calculate dates

Stupid, stupid, stupid, ...

I forgot the attachment. Here is caljd.sh Version 2.2.

No points, please.

If it ain't broke, I can fix that.
A. Clay Stephenson
Acclaimed Contributor

Re: Faster method than caljd.sh to calculate dates

Oops, I just found a typo ('wriiten' instead of 'written') in the usage section; it works but here is the official version 2.2 of caljd.sh.


If it ain't broke, I can fix that.
A. Clay Stephenson
Acclaimed Contributor

Re: Faster method than caljd.sh to calculate dates

Okay Steve, here is the Perl version, caljd.pl 2.2 p. On repeated calculations using file input/output it's about another 5x faster than the newest caljd.sh or about 100x faster than the original (2.1) version. This was based upon an input of 100 calculations so the numbers only get better with higher numbers of calculations. Simply substitute caljd.pl for caljd.sh in my earlier small example. Invoke as caljd.pl -u for full usage.

If it ain't broke, I can fix that.
Steven E. Protter
Exalted Contributor

Re: Faster method than caljd.sh to calculate dates

A. Clay,

Please make sure Merijn gets copies of these guys, if you have a role in that. I have a hard time working with them off itrc, I found it easier to download them from the beepz site.

Thanks for all the hard work on these scripts.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
A. Clay Stephenson
Acclaimed Contributor

Re: Faster method than caljd.sh to calculate dates

Final thought Steve:

I suspect that there is a very high probability that you are doing repeated calculations for the same dates. This implies that a caching scheme would be a great enhancement for your report. The idea is that you first check the cache and if found use it otherwise call caljd.sh (or caljd.pl) to calculate the date and add it to the cache. Table lookups using awk's associative arrays (the only kind it really has) or Perl's hashes would be much faster than doing calculations. You could think of it like using "09/05/2003" as the index of an array rather than
the more typical numerical indices.

If it ain't broke, I can fix that.
H.Merijn Brand (procura
Honored Contributor

Re: Faster method than caljd.sh to calculate dates

Clay, are the URl available? Then I can replace the downloads with direct links to your versions. Always up to date then.

Otherwise please send by mail, so I will be sure they are correct

mail address in bottom of my site

Enjoy, have FUN! H.Merijn
Enjoy, Have FUN! H.Merijn
Steven E. Protter
Exalted Contributor

Re: Faster method than caljd.sh to calculate dates

I had to make a change to caljd.sh for Linux because there is not posix shell in there.

Its a one line change, but I was wondering if we wanted to post up that version. I've tested it and it gives the same answers on all of A. Clay's examples as HP-UX.

I know I haven't created the Merjin mirror yet. Been busy. Blaster is putting bucks in my pocket, half dozen friends are queuing up for the $99 clean it and protect it special.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
H.Merijn Brand (procura
Honored Contributor

Re: Faster method than caljd.sh to calculate dates

Still hard for many to write my name correct, but the two attachments were blindly copied to https://www.beepz.com/personal/merijn/ or http://www.cmve.net/~merijn/

Enjoy, have FUN! H.Merijn
Enjoy, Have FUN! H.Merijn

Re: Faster method than caljd.sh to calculate dates

Hi Clay:

I just tried your new caljd.pl and changed my report script to use the new -f option for file input and output. The report which took over 5 minutes to run on Friday now runs in about 4 seconds!!! Your idea of using a table lookup for calculations that have already been done is a good one but the report is more than fast enough now.

May I make one suggestion? You might think about including some examples about how to use the -f option in a script that actually uses a co-process. The ksh man pages don't really make it too clear about using "|&".

Thanks for all your help.

Regards,
Steve
H.Merijn Brand (procura
Honored Contributor

Re: Faster method than caljd.sh to calculate dates

I've given the perl version a quick look and when you do not run it in a separate process on every calculation, you can cache the results of the functions using Memoize, a standard perl caching module.

The only thing I saw right away to be slow and inefficient is the help and usage message(s), but they are not used in `real' production work.

Enjoy, have FUN! H.Merijn
Enjoy, Have FUN! H.Merijn