Looking for ideas on how to do system reporting/monitoring.

[MAD_2]

I have been charged with a project to post reporting and a tool for easy monitoring of our systems by someone from management. Currently I am the administrator of two HPs running HP-UX 11.0, one Solaris 5.x, one SCO Unix 5.0.x, and one Unixware 7.x

I know I many only get help and ideas for HP here, but maybe someone else can advise on the other OS. What the request pretty much is about is the following:

- Monitor accounts and when users log in/out of the system.
- Monitor accounts and what creations/updates/deletions they may perform on the system.

Provide an automated report that can be posted on a daily bases or pulled on request (like posting it on a web page or something like that). I have not turned on auditing yet (we are on a "trusted" system) but I know turning it on will be necessary, I would like to get advise on that too and where to go to get a good guide/manual for setting it up so that I can calculate overhead of having it turned on.

Please share all your ideas and if aware of any excellent tools.

[harry d brown jr]

- openview (from hp) - we use it for almost everything (sun, hp, ibm, and M$ crap)
- network node manager - we use it in conjunction with openview
- glance/measureware/perfview (from hp) we use it for our Hp's & suns

Other "cheap" options:
- http://www.bb4.com/

live free or die
harry

[MAD_2]

Excellent Harry, thanks, I did not give you a 8-10 because I'd like to keep my message open since I expect more answers and ideas... Also, I can say the following about the systems I have:

1. One of the HP-UX 11.0 does have a license for HP-UX GlancePlus/UX Pak for s800 11.0 (Product B3701AA Revision C.03.50.00) -- Can this tool be used to generate reports to be posted on a website??? This is only for system performance, correct? How can I integrate user related information. Where can I find the manual that will provide me with the details (I got these manuals: HP OpenView GlancePlus for HP-UX 10.20 and 11.x Installation guide, HP OpenView MeasuereWare Agent - Data Source Integration Guide, and HP OpenView MeasureWare Agent User's Manual)... Can you give me a one or two liner of what specifically this will do for me since you seem to be very familiar with it.

2. The other HP-UX 11.0 does not have it installed and I extremely doubt we'll purchase it.

Thanks, and sorry for so much bothering. And yes, we're looking for cheap to free solutions, you know the story... Budget constraints.

[Rob_132]

"Free/cheap" <> "elegant"
(in other words, here's my non-elegant free input)

From a purely HPUX perspective:

1. To "Monitor accounts and when users log in/out of the system.":

last|egrep "still|`date +%a`" egrep "still|`date +%a`" egrep "still|`date +%a`"

2. To "Monitor accounts and what creations/updates they may perform on the system.":

for a in user1 user2 root
do
echo $a:
echo
echo "Modified:"
find / -xdev -user $a -mtime 1
echo
echo "Accessed:"
find / -xdev -user $a -atime 1
echo
echo "**********"
echo
done

3. To "Monitor accounts and what deletions they may perform on the system." :

alias

(as root):
cd /
mkdir junk
cp /usr/bin/rm /usr/bin/rm.ORIG_HPUX
vi /usr/bin/rm
/usr/bin/mv $1 /junk/$$_$1

************

Now, just set it up in a script, run once daily as a cron job, and mail the output to management and yourself. Set another cron to empty the /junk directory. Also, make sure where ever you make the /junk directory has sufficient space...the example of / is probably not the best - but was less typing for me!

Thanks for asking - you prompted me to do something one of my customers needed me to do anyways! Win/win.

By the way - never mistake this forum editor for 'vi' - escape simply erases your post!

[Rob_132]

sorry - I put " \" after the end of each of the lines in the "last" script, but they disappeared when posted. And a typo on my part was discovered.....

So, the command should have been:

last|egrep "still|`date +%a`"|
egrep "still|`date +%b`"|
egrep "still|`date +%e`"


....joined onto ONE line, or delimited by
" \" (space-backslash) at the end of each line.

[Michael Tully]

You could always have a look at 'sar'. It is really cheap (zero) and can run on all Unix systems. There are stacks of script examples. search for 'sar + scripts'. As suggested by Harry you can use glance. It is quite good really, we push it through scripts and then it gets published into pdf files for general viewing by management/us.

User related information can be used in different ways. You could use the uptime command to get the number as well as glance, but these won't give you any detail as to what they are actually doing. You would have to turn on system accounting/auditing to get anywhere. What are trying to actually capture ... other than deletions etc. You have to ask yourself, why do they have command line access anyway ...

Manuals for all openview products can be found here:
http://ovweb.external.hp.com/lpe/doc_serv/

Cheers
Michael

[Hemanth Gurunath Basrur]

Hi,

I have heard about WBEM, which performs system monitoring of resources on HP-UX, Linux or any OS. To know more about WBEM, please refer to http://docs.hp.com.

Check this link on WBEM

http://docs.hp.com/cgi-bin/fsearch/framedisplay?top=/hpux/onlinedocs/B8465-90001/B8465-90001_top.html&con=/hpux/onlinedocs/B8465-90001/00/00/4-con.html&toc=/hpux/onlinedocs/B8465-90001/00/00/4-toc.html&searchterms=WBEM&queryid=20030809-221853

As far as suditing of trusted systems is concerned, I would recommend that you read the following documents:

* Administering your trusted systems manual
http://docs.hp.com/cgi-bin/fsearch/framedisplay?top=/hpux/onlinedocs/B2355-90121/B2355-90121_top.html&con=/hpux/onlinedocs/B2355-90121/00/00/18-con.html&toc=/hpux/onlinedocs/B2355-90121/00/00/18-toc.html&searchterms=accounts%7cwhen%7cuser%7clogin%7cMonitoring&queryid=20030809-222932

* Managing Systems & Workgroups manual - Auditing a Trusted System section

http://docs.hp.com/cgi-bin/fsearch/framedisplay?top=/hpux/onlinedocs/B2355-90742/B2355-90742_top.html&con=/hpux/onlinedocs/B2355-90742/00/00/67-con.html&toc=/hpux/onlinedocs/B2355-90742/00/00/67-toc.html&searchterms=trusted%7csystems%7cAuditing&queryid=20030809-223146

Hope this helps.

Regards,
Hemanth

[MAD_2]

Thanks everyone one for your responses. I have not posted a 10 yet because it will take some time for me to analyze your input and see what tools of any of those mentioned here, would do the job (or what combination of tools; first I have to learn how to exploit Glance and the OpenView MeasureWare Agent). I will get back to this forum message later to post information about my implementations and check further responses as well as to post information that I may find somewhere else.

[Chris Wilshaw]

A fairly simple option for log in/out details

Add a couple of lines to /etc/profile (and to /etc/csh.login if any users are in the c-shell).

touch $HOME/.lastlogin
trap "touch $HOME/.lastlogout" 0

This will create (or update the timestamp) on a .lastlogin file in the users home directory.

The trap command will update the timestamp on the .lastlogout file when the shell exits.

[Steve Steel]

Hi


http://www.introcomp.co.uk/examples/index.html

A lot of good stuff.

Also

Go to www.docs.hp.com and search on auditing


Steve Steel

[MAD_2]

Harry, are all of these three products purchased separately or are they part of the same package?
- glance/measureware/perfview (from hp) we use it for our Hp's & suns

As I mentioned in one of my replies, we do have one of our systems with Glance Plus (One of the HP-UX 11.0 does have a license for HP-UX GlancePlus/UX Pak for s800 11.0 (Product B3701AA Revision C.03.50.00) -- Can this tool be used to generate reports to be posted on a website? This is only for system performance, correct?)

Thanks, oh goodness so much reading and so much to set up... The beauty of being an administrator.

[Mark Landin]

If you are looking for a good sytem performance monitoring tool for HP-UX and Solaris, check out the SOS product from http://www.lund.com.

Sorry, can't help you with the user auditing portion. At least not yet. :)

[kenny chia]

hi
From your original posts, it seems that you are tasked to monitor users and not the server itself

To monitor who has login, at what time and from where, you may want to look at the "last" command. With some formatting, you can create a trend chart of something similiar. You can try GNU gnuplot for graph plotting

for file monifications, there is always a command history file in the users home page (for bash, it is .bash_history). From that file, you could examine all the rm, cp or mv commands that the user has executed

[sysadm_1]

Hi Adam,

Why cant you use TCB (Trusted COmputing Base),and enable auditing for users.
It will give you deetails of user login and activities

Shaji

[MAD_2]

Kenny, in fact you are correct. My main priority at this time is the monitoring of users, here is the scenario:

I administer a few servers, all of them in a call center.
Most of the users (agents) only use one login (this login is shared by the application they use, and monitoring of what they do is rather based on input they make in the database, so the Unix login is just the gateway to their application; auditing of their input is better obtained through database queries).

The agent's login is not what I am interested on monitoring, I am mostly interested on the auditing of all other user accounts and what is done (file modifications -- to include ownerships and access rights, moves, deletions, etc). This is priority No. 1, however system and network performance are also on the top of the priorities, and I do know that we may be talking about one, two, or even three different tools here. So far I have received quite a few postings to do research, but more input is welcome, thanks.

[RolandH]

Read the manpage from the command script.

But be aware that the outputfiles could grow very fast.


HTH
Roland

[Tim Sanko]

I can't seem not to hit the escape key.

One last time:

The one thing I would do is to cycle the history files and email them to yourself. after passing them through an "grep |awk program " sieve. I restart mine at midnight.

I then get to peruse activity, however most users don't use their history like an admin will.

Then I recommend you kick SAR off hourly

here is a cron entry we use.
00 * * * * /root/SCRIPTS/performance.sh > /dev/null 2>&1


here is performance.sh

Month=`/usr/bin/date +%m`
Day=`/usr/bin/date +%d`
Year=`/usr/bin/date +%y`
Hour=`/usr/bin/date +%H`
Minute=`/usr/bin/date +%M`
LOGFILE=/root/LOG/SAR/$Year$Month$Day-$Hour$Minute.sar

/usr/sbin/sar -A -o $LOGFILE 60 60 > /dev/nul &

I then clean the directory which is NOT mounted under root with a cron entry

00 5 * * 1-6 find /root/LOG/SAR -name '*.txt*' -mtime +4 -exec rm -f {} \; #remove old SAR performance data

I daily run a series of reports off that I export to a spreadsheet and post the output on the network.

Once you have the sar data, you really have most of what you need.

Tim

[W.C. Epperson]

The Big Brother System and Network Monitor (http://bb4.com) is free, easily installed and configured, and can mesmerize managers for hours on end with its self-updating Web page showing a matrix of servers and routers and the states of various issues on them. ;)

User monitoring is not one of its default functions, but could it would be easy to roll in in as one of the "client side" scripts like the one that produces the "procs" cell on the default matrix. We use BB to monitor our routers for several issues, and our servers for ping response, cpu usage, disk usage, network connectivity of DNS, FTP, HTTP, POP, SMTP, etc., and for local processes like syslog. Saves us getting blind-sided a lot besides entertaining the brass.

[MAD_2]

Great W.C.
Yeah, I was just checking the bb4 (Big Brother) website. I guess I would have to download the Server application for HP-UX and client for Windows 2000, right? I did not see manuals so I wonder how it works, will you explain briefly.

Thanks!

[W.C. Epperson]

To view the monitor displays with a Windows machine, you just use a browser. You'd only want the Windows 2000 "client" for a Windows 2000 server that was going to be a client (monitored) node to a Big Brother server.

There are three components: a network monitor which performs checks against nodes on the network, a display monitor (which is usually run on the same box as the network monitor), and a client module which runs on the monitored systems performing local checks like disk usage, cpu, etc., and sends information to the network monitor. There are client monitor scripts preconfigured for a number of operating systems. The display monitor takes info from the network monitor and formats it into Web pages. So you need Apache or something to serve them up, but it's part of the default hp-ux distribution now, so that's easy.

It's really quite easy to install and get working on a default level. I'd suggest setting up one box as network monitor, display monitor, and a client to itself, and just poke around and get a feel for it. Once you do, it's very easy to propagate to other network nodes.

The doc page is:
http://bb4.com/bb/help/bb-help.html

Hope this helps.

[MAD_2]

WC, maybe I was not very clear with my question. Actually the scenario is this:

1. The system to be monitored is the HP-UX 11.0 (An RP5470)
2. What will be used to see the reports is a Win 2000 with Windows Explorer.
3. No, we do not want to use Apache on the application server that is to be monitored, maybe FTP the pages generated to a Win 2000 server where the pages can be published, but we do not want to run a webserver in the HP-UX.

Please explain which should be loaded based on that, thanks.

Also, folks, even if I turn on auditing on the HP, how can I make the took generate reports for management to view in a webpage (tools or ideas please). Only the "last -R" and "lastb -R" and the history files are not enough, output from the monitored events would be best.

Thanks for all of the responses given so far.

[MAD_2]

I have a few concerns regarding what I've read so far about auditing on a trusted system:

1. I keep readint the following statement: "Auditing must be turned on for normal system operation on a trusted system. If you need to turn auditing off to perform system administrative functions, bring the system into single user mode, turn auditing off, the perform the desired operations." I can't afford to do that, is it really necessary?

2. Another one I read is the following: "It is of the utmost importance that you monitor the audit files and archive them daily". How about if I can't do that, we run daily database coldbackups at night and once a week we also run a system backup (we only have two drives available on our RP-5470). So how do I clean up the audit log files and how do I bak them up?

Another thing I read that concerns me is the following: "... when either the audit file or the file system is getting full, and automatically switches to the auxiliary audit file if one is available. If no backup is located, audomon requests appropriate action so you can react to the conditions that could cause the system to shut down."

These all came out of the "Auditing Trusted Systems" portion of "Administering your HP-UX Trusted System" Document B2355-90121.

Please provide any wise advise.

[W.C. Epperson]

There's a Big Brother port for NT4, but I don't know if it will run on W2K server. I wouldn't touch W2K server with a pool cue, so I can't help you with that.

But if you don't want to run Apache on the HP, you should just run the BB client there and put the network and display monitors on the Web server machine, then you won't have to FTP the pages around (they update every minute or so). I'd dig an old 486 out of the closet and put Linux and BB on it before I'd fool with W2K server.