Finding your postings and unassigned responses

[Steven Sim Kok Leong]

Hi,

As a small contribution to the forum, I have setup a search engine that attempts to complement the ITRC search engine and ITRC forum user profile via the following features:

1) Allows any member to track all his past and present postings from Jan 2001 (not his responses).

2) Allows any member to track responses to his postings, of which he has yet assign points to.

3) Allows anyone to track a member's points participation as a percentage based on the number of responses assigned points over the total number of responses. Thus, a -100% for someone who never awards points and a 100% for someone who never fails to award points.

4) Allows anyone to identify the number of a member's postings that have no points awarded to any of the responses.

The search engine currently resides at:

https://www.beepz.com/cgi-bin/itrc.cgi

Caveats to note:

1) Please click-through the security warning because the website is relying on a dummy SSL certificate.

2) To minimize CPU and network resources, the program is configured to be single-threaded. As such, it takes up a substantial amount of time to complete processing. Thus, it is run at only daily intervals and at worst at an interval of 2 days. As a result, statistics may not be most up-to-date and are delayed at worst by 48 hrs.

3) Only postings in the HP-UX forum from Jan 2001 onwards are processed. Exact full usernames must be used. It is case sensitive.

Hope this helps and please feel free to feedback any problems. Regards.

Steven Sim Kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

[Steven Gillard_2]

Steven,

Good work, definitely something that should be included in the forum itself.
Here's a couple of suggestions for extra features if you have the time:

- number of responses a user has contributed, and the number of those that were assigned points

- % of those responses that were rated 8-10 (only including those that were actually assigned points)

Happy new year,
Steve

[Darrell Allen]

Niiiiceee!

Thanks Steven! I like it!

N/A for this reply please. Wish I could give YOU points!

Darrell

[John Bolene]

Hey Steve, interesting, I love it.

How about posting that cgi code if you don't mind? If you do mind, I understand also.

Maybe you need to submit it to the HP ITRC guys also.

Thinking about what the program must do, it will be a cpu drain not only on your server, but the ITRC. It would best be done on the ITRC server itself.

My 2 coppers.

[fg_1]

great search engine, found alot that i had not assigned points to. Cleanup is a good thing, everyone should do it.

great job.

[Thierry Poels_1]

hi,

I scored 100%, can I get a certificate for that whoehahahaha ;-)

pretty nice gadget

happy newyear!
Thierry.

[Steven Sim Kok Leong]

Hi everyone,

Thanks for your feedback. I am glad that the search engine is of some use to you. :-)

John, thanks for your feedback. I fully agree with you that such functionality is best performed locally by ITRC on its own database of posts. I am waiting for that to happen.

While waiting, I created this which is made up of two separate scripts:
1) a collector perl script that retrieves the raw data for processing and stores processed data locally
2) a retrieval perl CGI script that filters and massages displayed data from processed data.

The collector script runs once a day lynx -source every posting since Jan 2001 sequentially so that at any instance, there is only a single http connection to the forum.

In my humble opinion, this is less intensive than browsing a forum webpage in a graphical browser, firstly because the html code is not parsed and interpreted at all by the browser and secondly because embedded graphics are not loaded.

Also in comparison, the google web-crawler is much more intensive because it browses everything in its path on forums.itrc.hp.com, not just simply on the HP-UX postings since Jan 2001 from the forum's HP-UX section.

If google and other web-crawlers had significantly impacted the forum performance, I believe ITRC would have added robots.txt to block these regular crawls from the numerous search engine web-crawlers.

Based on this line of reasoning, I concluded that my collector's limited and primitively sequential web-crawl is safe.

The 5-item tuple of poster, subject, date, no. of assigned responses and no. of unassigned responses is computed and stored locally on my system, taking up minimal CPU cycles and a meagre 8.8 MB (current size) of uncompressed flat file storage space.

The retrieval script searches for matching tuples from this 8.8 MB of flat data locally and massages the filtered data to be displayed accordingly on your browser.

I seriously don't think ITRC would want my small scripts which are not going to be applicable if data is retrieved and parsed direct from the database.

Please feedback any problems and Happy New Year to one and all. :-)

Steven Sim kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

[John Bolene]

Cool design, you are to be commended.

I had thought from the first post that a person could only search the ITRC once a day, maybe you were keeping track of who used your program and that it searched the ITRC for each request. I did not know you had built a local database.

Since you have your database built, would it be better to just add messages from each day, going forward?

How long did it take to get all the posts?

Just thoughts.

[Sachin Patel]

Hi Steven,
Great!!! I have no word for it.

I have found my 7 unassigned reply and assignee them a points. It is absolutely great.

Sachin

[D. Jackson_1]

You rock dude!! This thing works great. You should at least get some bonus points for this..

DJ

[Steven Sim Kok Leong]

Hi John,

Thanks for your suggestion.

I cannot just append new postings for each new day to the database because an old posting could have its old responses newly updated with points. Thus, a full browse is necessary.

ITRC can avoid that if they implement it locally since the point assignment function could be made to trigger an update to their user and points database.

Unlike ITRC, I have no way of getting a new point assignment to trigger off an update in my own local database.

Thus, a browse through all postings since Jan 2001 to retrieve updated point statuses is inevitable.

That collector takes a few hours to run at this point. Because it is single-threaded with no involved parallelism, the load is spread thinly and insignificantly across the hours.

Hope this helps. Regards.

Steven Sim Kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

[Steven Sim Kok Leong]

Hi Steve,

Thanks for your suggestions.

>> - number of responses a user has contributed, and the number of those that were assigned points

This is already available on your ITRC user profile.

>> - % of those responses that were rated 8-10 (only including those that were actually assigned points)

One question. How is this percentage going to be useful to the community?

Hope this helps. Regards.

Steven Sim Kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

[Steven Sim Kok Leong]

Hi,

I need to make one correction.

The points participation percentage, which is the number of assigned responses over the total number of responses, is between 0% to 100% inclusive, not -100% to 100% as originally stated.

0% indicates that points have not been awarded to any response
100% indicates that points have been awarded to all responses

Hope this helps. Regards.

Steven Sim Kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

[James R. Ferguson]

Hi Steven:

This is an *outstanding* contribution! In fact, it's something that desperately needs to be an intrinsic feature of the ITRC Forums.

I'd respectfully suggest that sorting the output by descending date order of the original post's creation whould be a valuable enhancement.

With my warmest regards, and thanks, Jim.

...JRF...

[Steven Sim Kok Leong]

Hi Jim,

Thanks for your suggestion.

You are right. It is currently sorted by date of last response rather than date of original posting. Will work out the descending sort by date of posting.

Happy New Year :) Regards.

Steven Sim Kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

[Steven Gillard_2]

Hi Steven,

I was suggesting some statistics on my responses to others rather than on the questions I have asked. As far as I can tell, all ITRC gives me is links to my 5 most recent responses, and then a list of threads for which one of my responses was ranked 8-10. It doesn't tell me how many responses I have posted or how effective in general these responses have been.

Now I think about it some more, maybe it's not such a great idea to have stats like this. I was just curious to see what the numbers would look like, but I can see a negative side to them.

Cheers,
Steve

[Patrick Wallek]

Outstanding piece of work Steven! It has been needed for a long time.

Have you given any thought to going back to 2000 or 1999 to get all the posts in the HP-UX forum?

I realize that would dramatically increase your processing time, but I was just curious.

[Craig Rants]

Well done Steve, you have created something that will definately make info searchs a little easier. You probably have peaked the creative juices of a few others as well. Keep up the good work.

Craig

[Krishna Prasad]

Great Work.....

Is there a way to view all of the responses you have made and the points assigned to them.

[Frank Slootweg]

Great tool, made by a user for the benefit of the other users!

Just a question: Do you intentionally exclude HP employees, i.e. people with an "hp" logo next to their name? I realize that most HP employees do not post, but only respond, but there are some which do post and which I could not find (examples: Dan Gazzaniga and Alex Glennie). Or am I missing something, doing something wrong, etc.?

[George_Dodds]

Excellent, this is what we have needed for a long time.

Allthough i am finding that there are 3 replies that i havent assigned points to, but i still cannot find them.

Maybe they were N/A's.

Great work and very quick!

Cheers

George

[Paula J Frazer-Campbell]

Well Done Steven,

A very very nice to have tool.


Paula

[Steven Sim Kok Leong]

Hi,

James,

Have added posting date. Tuples still sorted by last response for time being.

Steve and Ron,

Thanks for the clarifications. Will implement the nice-to-have features when I have the time.

Patrick,

2000 and beyond uses a different set of HTML formatting. Haven't check out the regex for extracting them yet. Will check that out when time permits.

Frank,

I checked on Dan's and Alex's postings. Their postings appear correctly. Did you by any chance added a space after their names? My puny script considers only exact names, word for word ie. it should be entered as "Dan Gazzaniga" and not "Dan Gazzaniga " with the space.

George,

My script takes into consideration N/A's as a valid responses. I believe your postings are either too recent (a few hours) or too old (2000 or older).

All,

Thanks for all the encouraging comments as well as the wonderful feedback and suggestions. :)

Hope I have not missed anyone out above. I apologise that I have been slow in implementing your suggestions because I was pretty tied up by work. More hackers on the prowl.

Hope this helps. Regards.

Steven Sim Kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

[James R. Ferguson]

Hi Steven:

"slow in implementing"? I think NOT!!! You put the HP developers to shame. A big thank you for some very nice new features!

With warm regards!

...JRF...

[Frank Slootweg]

Thanks Steven!

Yes, the trailing space after the name was the culprit.

*Because* I wanted to *prevent* typos, especially with Dan's name, I looked up a posting of the desired posters, clicked on their name (which brings you to their profile) and then copied-and-pasted their name.

However the to-be-copied line says, for example, "Alex Glennie - forum profile", but when you copy it, the browser (at least Internet Explorer) always includes the space between the lastname and the minus sign! I.e. when you 'move' over the name to mark it, you move a *'word'* at a time, not a character at a time.

Sigh! Oh well, another 'mystery' solved. :-)