topic Re: Finding your postings and unassigned responses in Operating System - HP-UX

Finding your postings and unassigned responses

Steven Sim Kok Leong — Fri, 28 Dec 2001 10:57:09 GMT

Hi,

As a small contribution to the forum, I have setup a search engine that attempts to complement the ITRC search engine and ITRC forum user profile via the following features:

1) Allows any member to track all his past and present postings from Jan 2001 (not his responses).

2) Allows any member to track responses to his postings, of which he has yet assign points to.

3) Allows anyone to track a member's points participation as a percentage based on the number of responses assigned points over the total number of responses. Thus, a -100% for someone who never awards points and a 100% for someone who never fails to award points.

4) Allows anyone to identify the number of a member's postings that have no points awarded to any of the responses.

The search engine currently resides at:

https://www.beepz.com/cgi-bin/itrc.cgi

Caveats to note:

1) Please click-through the security warning because the website is relying on a dummy SSL certificate.

2) To minimize CPU and network resources, the program is configured to be single-threaded. As such, it takes up a substantial amount of time to complete processing. Thus, it is run at only daily intervals and at worst at an interval of 2 days. As a result, statistics may not be most up-to-date and are delayed at worst by 48 hrs.

3) Only postings in the HP-UX forum from Jan 2001 onwards are processed. Exact full usernames must be used. It is case sensitive.

Hope this helps and please feel free to feedback any problems. Regards.

Steven Sim Kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

Re: Finding your postings and unassigned responses

Steven Gillard_2 — Fri, 28 Dec 2001 12:47:48 GMT

Steven,

Good work, definitely something that should be included in the forum itself.
Here's a couple of suggestions for extra features if you have the time:

- number of responses a user has contributed, and the number of those that were assigned points

- % of those responses that were rated 8-10 (only including those that were actually assigned points)

Happy new year,
Steve

Re: Finding your postings and unassigned responses

Darrell Allen — Fri, 28 Dec 2001 13:33:26 GMT

Niiiiceee!

Thanks Steven! I like it!

N/A for this reply please. Wish I could give YOU points!

Darrell

Re: Finding your postings and unassigned responses

John Bolene — Fri, 28 Dec 2001 13:41:00 GMT

Hey Steve, interesting, I love it.

How about posting that cgi code if you don't mind? If you do mind, I understand also.

Maybe you need to submit it to the HP ITRC guys also.

Thinking about what the program must do, it will be a cpu drain not only on your server, but the ITRC. It would best be done on the ITRC server itself.

My 2 coppers.

Re: Finding your postings and unassigned responses

fg_1 — Fri, 28 Dec 2001 13:44:05 GMT

great search engine, found alot that i had not assigned points to. Cleanup is a good thing, everyone should do it.

great job.

Re: Finding your postings and unassigned responses

Thierry Poels_1 — Fri, 28 Dec 2001 13:55:35 GMT

hi,

I scored 100%, can I get a certificate for that whoehahahaha ;-)

pretty nice gadget

happy newyear!
Thierry.

Re: Finding your postings and unassigned responses

Steven Sim Kok Leong — Fri, 28 Dec 2001 14:51:07 GMT

Hi everyone,

Thanks for your feedback. I am glad that the search engine is of some use to you. :-)

John, thanks for your feedback. I fully agree with you that such functionality is best performed locally by ITRC on its own database of posts. I am waiting for that to happen.

While waiting, I created this which is made up of two separate scripts:
1) a collector perl script that retrieves the raw data for processing and stores processed data locally
2) a retrieval perl CGI script that filters and massages displayed data from processed data.

The collector script runs once a day lynx -source every posting since Jan 2001 sequentially so that at any instance, there is only a single http connection to the forum.

In my humble opinion, this is less intensive than browsing a forum webpage in a graphical browser, firstly because the html code is not parsed and interpreted at all by the browser and secondly because embedded graphics are not loaded.

Also in comparison, the google web-crawler is much more intensive because it browses everything in its path on forums.itrc.hp.com, not just simply on the HP-UX postings since Jan 2001 from the forum's HP-UX section.

If google and other web-crawlers had significantly impacted the forum performance, I believe ITRC would have added robots.txt to block these regular crawls from the numerous search engine web-crawlers.

Based on this line of reasoning, I concluded that my collector's limited and primitively sequential web-crawl is safe.

The 5-item tuple of poster, subject, date, no. of assigned responses and no. of unassigned responses is computed and stored locally on my system, taking up minimal CPU cycles and a meagre 8.8 MB (current size) of uncompressed flat file storage space.

The retrieval script searches for matching tuples from this 8.8 MB of flat data locally and massages the filtered data to be displayed accordingly on your browser.

I seriously don't think ITRC would want my small scripts which are not going to be applicable if data is retrieved and parsed direct from the database.

Please feedback any problems and Happy New Year to one and all. :-)

Steven Sim kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

Re: Finding your postings and unassigned responses

John Bolene — Fri, 28 Dec 2001 15:47:07 GMT

Cool design, you are to be commended.

I had thought from the first post that a person could only search the ITRC once a day, maybe you were keeping track of who used your program and that it searched the ITRC for each request. I did not know you had built a local database.

Since you have your database built, would it be better to just add messages from each day, going forward?

How long did it take to get all the posts?

Just thoughts.

Re: Finding your postings and unassigned responses

Sachin Patel — Fri, 28 Dec 2001 16:26:44 GMT

Hi Steven,
Great!!! I have no word for it.

I have found my 7 unassigned reply and assignee them a points. It is absolutely great.

Sachin

Re: Finding your postings and unassigned responses

D. Jackson_1 — Fri, 28 Dec 2001 16:57:16 GMT

You rock dude!! This thing works great. You should at least get some bonus points for this..

DJ

Re: Finding your postings and unassigned responses

Steven Sim Kok Leong — Sat, 29 Dec 2001 00:52:24 GMT

Hi John,

Thanks for your suggestion.

I cannot just append new postings for each new day to the database because an old posting could have its old responses newly updated with points. Thus, a full browse is necessary.

ITRC can avoid that if they implement it locally since the point assignment function could be made to trigger an update to their user and points database.

Unlike ITRC, I have no way of getting a new point assignment to trigger off an update in my own local database.

Thus, a browse through all postings since Jan 2001 to retrieve updated point statuses is inevitable.

That collector takes a few hours to run at this point. Because it is single-threaded with no involved parallelism, the load is spread thinly and insignificantly across the hours.

Hope this helps. Regards.

Steven Sim Kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

Re: Finding your postings and unassigned responses

Steven Sim Kok Leong — Sat, 29 Dec 2001 01:00:22 GMT

Hi Steve,

Thanks for your suggestions.

>> - number of responses a user has contributed, and the number of those that were assigned points

This is already available on your ITRC user profile.

>> - % of those responses that were rated 8-10 (only including those that were actually assigned points)

One question. How is this percentage going to be useful to the community?

Hope this helps. Regards.

Steven Sim Kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

Re: Finding your postings and unassigned responses

Steven Sim Kok Leong — Sat, 29 Dec 2001 01:11:05 GMT

Hi,

I need to make one correction.

The points participation percentage, which is the number of assigned responses over the total number of responses, is between 0% to 100% inclusive, not -100% to 100% as originally stated.

0% indicates that points have not been awarded to any response
100% indicates that points have been awarded to all responses

Hope this helps. Regards.

Steven Sim Kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

Re: Finding your postings and unassigned responses

James R. Ferguson — Tue, 01 Jan 2002 23:45:10 GMT

Hi Steven:

This is an *outstanding* contribution! In fact, it's something that desperately needs to be an intrinsic feature of the ITRC Forums.

I'd respectfully suggest that sorting the output by descending date order of the original post's creation whould be a valuable enhancement.

With my warmest regards, and thanks, Jim.

...JRF...

Re: Finding your postings and unassigned responses

Steven Sim Kok Leong — Wed, 02 Jan 2002 09:35:00 GMT

Hi Jim,

Thanks for your suggestion.

You are right. It is currently sorted by date of last response rather than date of original posting. Will work out the descending sort by date of posting.

Happy New Year :) Regards.

Steven Sim Kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

Re: Finding your postings and unassigned responses

Steven Gillard_2 — Wed, 02 Jan 2002 10:51:29 GMT

Hi Steven,

I was suggesting some statistics on my responses to others rather than on the questions I have asked. As far as I can tell, all ITRC gives me is links to my 5 most recent responses, and then a list of threads for which one of my responses was ranked 8-10. It doesn't tell me how many responses I have posted or how effective in general these responses have been.

Now I think about it some more, maybe it's not such a great idea to have stats like this. I was just curious to see what the numbers would look like, but I can see a negative side to them.

Cheers,
Steve

Re: Finding your postings and unassigned responses

Patrick Wallek — Wed, 02 Jan 2002 14:54:37 GMT

Outstanding piece of work Steven! It has been needed for a long time.

Have you given any thought to going back to 2000 or 1999 to get all the posts in the HP-UX forum?

I realize that would dramatically increase your processing time, but I was just curious.

Re: Finding your postings and unassigned responses

Craig Rants — Wed, 02 Jan 2002 15:05:47 GMT

Well done Steve, you have created something that will definately make info searchs a little easier. You probably have peaked the creative juices of a few others as well. Keep up the good work.

Craig

Re: Finding your postings and unassigned responses

Krishna Prasad — Wed, 02 Jan 2002 16:22:10 GMT

Great Work.....

Is there a way to view all of the responses you have made and the points assigned to them.

Re: Finding your postings and unassigned responses

Frank Slootweg — Thu, 03 Jan 2002 11:28:32 GMT

Great tool, made by a user for the benefit of the other users!

Just a question: Do you intentionally exclude HP employees, i.e. people with an "hp" logo next to their name? I realize that most HP employees do not post, but only respond, but there are some which do post and which I could not find (examples: Dan Gazzaniga and Alex Glennie). Or am I missing something, doing something wrong, etc.?