Finding your postings and unassigned responses

[Steven Sim Kok Leong]

Hi,

As a small contribution to the forum, I have setup a search engine that attempts to complement the ITRC search engine and ITRC forum user profile via the following features:

1) Allows any member to track all his past and present postings from Jan 2001 (not his responses).

2) Allows any member to track responses to his postings, of which he has yet assign points to.

3) Allows anyone to track a member's points participation as a percentage based on the number of responses assigned points over the total number of responses. Thus, a -100% for someone who never awards points and a 100% for someone who never fails to award points.

4) Allows anyone to identify the number of a member's postings that have no points awarded to any of the responses.

The search engine currently resides at:

https://www.beepz.com/cgi-bin/itrc.cgi

Caveats to note:

1) Please click-through the security warning because the website is relying on a dummy SSL certificate.

2) To minimize CPU and network resources, the program is configured to be single-threaded. As such, it takes up a substantial amount of time to complete processing. Thus, it is run at only daily intervals and at worst at an interval of 2 days. As a result, statistics may not be most up-to-date and are delayed at worst by 48 hrs.

3) Only postings in the HP-UX forum from Jan 2001 onwards are processed. Exact full usernames must be used. It is case sensitive.

Hope this helps and please feel free to feedback any problems. Regards.

Steven Sim Kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

[Steven Gillard_2]

Steven,

Good work, definitely something that should be included in the forum itself.
Here's a couple of suggestions for extra features if you have the time:

- number of responses a user has contributed, and the number of those that were assigned points

- % of those responses that were rated 8-10 (only including those that were actually assigned points)

Happy new year,
Steve

[Darrell Allen]

Niiiiceee!

Thanks Steven! I like it!

N/A for this reply please. Wish I could give YOU points!

Darrell

[John Bolene]

Hey Steve, interesting, I love it.

How about posting that cgi code if you don't mind? If you do mind, I understand also.

Maybe you need to submit it to the HP ITRC guys also.

Thinking about what the program must do, it will be a cpu drain not only on your server, but the ITRC. It would best be done on the ITRC server itself.

My 2 coppers.

[fg_1]

great search engine, found alot that i had not assigned points to. Cleanup is a good thing, everyone should do it.

great job.

[Thierry Poels_1]

hi,

I scored 100%, can I get a certificate for that whoehahahaha ;-)

pretty nice gadget

happy newyear!
Thierry.

[Steven Sim Kok Leong]

Hi everyone,

Thanks for your feedback. I am glad that the search engine is of some use to you. :-)

John, thanks for your feedback. I fully agree with you that such functionality is best performed locally by ITRC on its own database of posts. I am waiting for that to happen.

While waiting, I created this which is made up of two separate scripts:
1) a collector perl script that retrieves the raw data for processing and stores processed data locally
2) a retrieval perl CGI script that filters and massages displayed data from processed data.

The collector script runs once a day lynx -source every posting since Jan 2001 sequentially so that at any instance, there is only a single http connection to the forum.

In my humble opinion, this is less intensive than browsing a forum webpage in a graphical browser, firstly because the html code is not parsed and interpreted at all by the browser and secondly because embedded graphics are not loaded.

Also in comparison, the google web-crawler is much more intensive because it browses everything in its path on forums.itrc.hp.com, not just simply on the HP-UX postings since Jan 2001 from the forum's HP-UX section.

If google and other web-crawlers had significantly impacted the forum performance, I believe ITRC would have added robots.txt to block these regular crawls from the numerous search engine web-crawlers.

Based on this line of reasoning, I concluded that my collector's limited and primitively sequential web-crawl is safe.

The 5-item tuple of poster, subject, date, no. of assigned responses and no. of unassigned responses is computed and stored locally on my system, taking up minimal CPU cycles and a meagre 8.8 MB (current size) of uncompressed flat file storage space.

The retrieval script searches for matching tuples from this 8.8 MB of flat data locally and massages the filtered data to be displayed accordingly on your browser.

I seriously don't think ITRC would want my small scripts which are not going to be applicable if data is retrieved and parsed direct from the database.

Please feedback any problems and Happy New Year to one and all. :-)

Steven Sim kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

[John Bolene]

Cool design, you are to be commended.

I had thought from the first post that a person could only search the ITRC once a day, maybe you were keeping track of who used your program and that it searched the ITRC for each request. I did not know you had built a local database.

Since you have your database built, would it be better to just add messages from each day, going forward?

How long did it take to get all the posts?

Just thoughts.

[Sachin Patel]

Hi Steven,
Great!!! I have no word for it.

I have found my 7 unassigned reply and assignee them a points. It is absolutely great.

Sachin

[D. Jackson_1]

You rock dude!! This thing works great. You should at least get some bonus points for this..

DJ