checklist for system amdinistration

[Shivkumar]

I need to check in the morning on the systems so that we can do preventive failure of production servers.

I would appreciate if someone shares the best practices on hpux so that failure of production servers can be minimized.

Thanks and regards,
Shiv

[morganelan · Accepted solution]

Hi, see this attached documentation.

[saju_2]

Hi shiv

The answer is not a straight forward one..It depends on the hardware you use and the different application/DB's running on it..

I always goes inside the datacentre and checks the LED's on the front panel of the server,storage,library,sanswitches etc.. This I feel is a very good practice as LED indication can give quite a lot of information about erros and failures.
Monitoing the log files is another good practice
Log into GSP and check event logs (SL option)
var/adm/syslog/syslog is one main file which can give indication to the h/w erros or OS errors .
/var/adm/cron/log
/var/adm/sudolog/sudo.log
/var/adm/sulog
/var/mail/root

And mainly the commands top,glance and sar is another simple but highly efficent monitoring tools.

Regards
CS

[Sudeesh]

As far as my experience goes, for H/W issues EMS will do the magic for you. have a look at below link.

http://docs.hp.com/en/B5735-90001/index.html

For OS part you can write a small shell script to check process, disk space etc. For a production I would suggest a monitoring tool like HP OpenView or Unicenter.

Sudeesh

[Joseph Loo]

hi shiv,

do u have backup done daily (depending on critically of the system)? do u have mirroring or raid on your system? do u monitor the network? do u secure or have u harden your system?

these r some of the questions u should address.

regards.

[Devender Khatana]

Hi Shiv,

The EMS notifications are always helpfule and gets you notified regarding the forthcoming problems in advence, giving you time to rectify the same. Also the other general procedure we follow for our routine checkups includes following.

1. Checking /var/adm/syslog/syslog.log
2. Checking dmesg outputs.
3. Checking multiple paths to storage (If you have)
4. Checking all lan cards for proper functioing or error messages.
5. Checking file system utilizations.
6. Taking ignite backups.

Collecting nickel outputs will have all these in one step except ignite backup.

HTH,
Devender

[Devender Khatana]

Hi,

The latest nickel is available here.

ftp://ftp.hp.com/pub/catia/Utils/nickel.shar

And also check MP/GSP of your system if it supports that for any error messages.

HTH,
Devender

[Sandman!]

Shiv,

Another great and immensely helpful website is:
http://www.auditnet.org/docs/HPUX.doc

regards!

[Michael Selvesteen_2]

Hi,

Some others to the list

1. Check audit log for user login/logout events and other events if you use auditing.

2. Check for adequate disk space.

3. Check login daemons like sshd, telnetd are properly working.

4. Use SAM to check the performance of the system

5. Check the network performance

[Sandman!]

If you're interested in securing your servers and applications check out the URL below. I'ave found it be invaluable for my day-to-day sysadmin tasks.

http://www.security-gurus.de/papers/hpux_checklist.pdf#search='hpux%20checklist'

cheers!

[Mahesh Kumar Malik]

Hi Shiv

Following can be checked daily:

1. bdf output to check file system sizes
2. /var/adm/wtmp file for growth, if required null this file
3. /var/adm/btmp file for growth, null it if required
4. /var/adm/syslog/syslog.log file for errors if any
5. /var/adm/syslog/mail.log file for errors
6. swapinfo
7. storage path health by say fcmsutil
8. top command to view sleeping and unwanted processes

Regards
Mahesh

[Ranjith_5]

Hi Shiva,

Attached the daily check list for servers includes main points and a script to check syslog errors. You can customise this excel sheet according to your need if reqired like Backup details etc.If you need more details on how to get automated mails on warnings or other notifications pls reply back.

Regards,
Syam

[Borislav Perkov]

Hi,

You can find it here:
http://www.unix.ittoolbox.com/browse.asp?c=UNIXPeerPublishing&r=http%3A%2F%2Fwww%2Ewashington%2Eedu%2FR870%2F
or in general
http://www.unix.ittoolbox.com/nav/t.asp?t=395&p=395&h1=395

Regards,
Borislav

[Cem Tugrul]

Shiv,

Please have a look of the link;

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=732447

Good Luck,

[Kent Ostby]

Shiv --

If you are running ServiceGuard on your box, checking "cmviewcl" output will give you the health of the cluster (on both systems).

Another thing that I tend to check is simply "uptime" so I can get a feel for the pressure on the system .

Best regards,

Oz