Procedures and good practices

[Simon Abbott]

Hello everyone,

Does anyone know of a good resource that details some tried and tested best practices for looking after a system. I have read various books about system and network admin but they seem to be mostly of a technical nature - what commands are available and what devices systems have to do clever things.

Unfortunatley, and probably like many others, my colleages and myself found our way into the industry by accident and we have no reference point and no higher levels of management to pass wisdom down to us.

The sort of things we are looking for are to do with are (like an earlier message) what log files to check and how often, what is generally acceptable to automate and what should be run manually and so on. I realise that this will probably depend a great deal on what the system is used for (and how important it is!) but some kind of guidelines and general rules would be great.

If anyone knows of any web sites or, better still, books (or other things) I would be greatful.

Many thanks,

Simon.

[harry d brown jr]

well, you have taken the first step by just being here in this forum. I don't think you could find a better forum for assistance.

Managing a system(s), is an everlasting job. As applications and resources change, you will find your job changing also. The best thing is to eliminate the mundane tasks by automating them, freeing you up to do the real tasks, which is system management. Capacity monitoring and planning should be your biggest responsibility, not user account set up and password resetting. The best way to accomplish this is by giving your business units the ability to set up their own accounts and password resets. The business units should also be responsible for their own disk space management, especially in the /home directory. Your job should be one more of a cop, slapping people on the wrist when they step out of line.

[Santosh Nair_1]

A few things off the top off my head...

Some of the more common logs to check routinely include /var/adm/syslog/syslog.log and /var/adm/sulog. Should probably check /var/adm/syslog/mail.log if this system handles lots of mail.

Also, set up sar. This will help you in tracking trends on the system, i.e. cpu/memory/disk usage going up over time, etc. Comes in very handy.

As for automating, cleaning up /tmp and /var/tmp is a good start. These filesystem tend to fill up frequently. Removing old core file from the system should also be automated. I also run some scripts to document the system.

I'm sure there are quite a few more things that could be done...this is my short list.

-Santosh

[James R. Ferguson · Accepted solution]

Hi Simon:

Here's a quick list of some thoughts and some links to get you started:

1. There are lots of excellent "white papers" at http://docs.hp.com . If you drill down, for instance, there is a wealth of information in documents with titles containing words like "designing", "white paper", "choosing", and "managing". Take a look at the "high availability section as an example:

http://docs.hp.com/hpux/ha/index.html

2. If you simply click on the left while here, in the ITRC; check the boxes "manuals", "community forums", "training", "planning" AND "general news"; and supply the boolean search "best and practices"; then you will be rewarded with well over 100 references to check.

3. "Best practices" for the thinks like kernel and database parameters are highly dependent upon your unique environment, but the HP-UX forum is a great point to start your query of "what-works?". Keyword searches of the Forum alone will yield great treasures of information.

A standard reference to kernel tuning is this often-cited link:

http://docs.hp.com/hpux/onlinedocs/os/KCparams.OverviewAll.html

4. With reference to patch managment practices:

I think you will find this thread provides a good, general, bipolar view:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,11866,0x899c0559ff7cd4118fef0090279cd0f9,00.html
For some specific suggestions of "best-practice", see this thread:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,11866,0x79f4f841489fd4118fef0090279cd0f9,00.html

My own comments/philosophies appear in both of the above. In the second post, Cheryl Griffin sums it all up quite well: "What you do depends on your patch philosophy and the risk that you are willing to take."

5. If you are developing software, a great site is this one:

http://devresource.hp.com

Happy reading!

...JRF...

[A. Clay Stephenson]

Hi Simon:

I think I will answer your question in a little bit different way - let the boxes themselves do it. My answer to this problem is not cheap but it is the method to consider if you are charged with keeping a herd of critical systems up. I would look into Openview
IT/O (New Name VantagePoint/Operations).
http://www.openview.com

The advantage of this method is that you can monitor many, many systems from 1 location. Various flavors of UNIX boxes, NT, Novell, ... can all be monitored. VP/O comes with a fairly good set of 'templates' to do the generic monitoring for each platform. These probably get up 90% of the way there. I warn you the last 10% is tough and you can spend a long time
killing the last few gnats.

Just as an example of what you can do:

Suppose that an Oracle server in another state is starting to run low on tablespace. VP/O can detect this and if you like can not only alert you but take the appropriate action automatically and then tell you it did it.

It takes a LONG time and a lot of work to get to this level of response but it can be done.
Most of the everyday, routine stuff is already done for you in VP/O (size of logs, inetd running?, lp up? NFS ok?, filesystem filling up?, etc.)

VP/O also includes Network Node Manager which monitors the health of your LAN/WAN. Again, it ain't cheap but I would seriously consider it.

Clay

[Rita C Workman]

Think about what is on your system.
LOGS:
Oracle - look for logs. Omniback - look for logs. Getting the hint. Logs build and must be checked and trimmed regularly. The problem is not all log files can be found by:
find / -name *.log -print
Many log files do not end in *.log. So run your find with *.log and log* and *log*...it will be painful and take time, but may help you.
Next:
There are system 'logs' that have names like wtmp. This little beauty and btmp are in /var/adm. They record all the login/logout info...and they will continue to grow until addressed. So watch out for these.
Final Big Tip:
Automate/Script. Remember that commands you key at the command line can be 'scripted'. So once you master a couple...start scripting them and then set them to run automatically so you don't have to keep doing it manually. Yep, just write and 'cron' those little tasks to run when you want them to automatically.
===============
So start looking for scripts...this Forum has an abundance of scripts...do a SEARCH. Folks put them here to be copied and used. Take advantage of these valuable freebies.
And organize your system...don't put scripts everywhere. Create a location for them and run them from there..it's easier to create and modify later.
LAST:
Read your syslogs...watch for things that say warn or fail or the all too important 'lbolt'. Make sure you know what the problem is..don't ignore a message if you don't know it...make sure about it first. !!

And keep in touch...this Forum is a great to learn and later return the favor to the next person.

Regards,
Rita

[Bill McNAMARA_1]

Quite simple...
Backup everything regularly!

I like to use ignite every friday before I go home!

Later,
Bill

[paul courry]

Send me your email address ( paul@courry.com ) and I'll ship you 4 papers from HPWorld on the best practices of applying patches to your system. (.pdf format)

[James Beamish-White]

FYI, the way to find all logs in one statement is:

find / -name *.log -o -name *log* -o -name log*

The only problem I have is when I'm a webserver where a huge amount of JSP login files come back... but you can always grep these out or add a " -a ! name *.jsp " to the find.

James

[Edward McCouch]

I am one of those people who also kind of stumbled into the hpux admin gig too (wanted to be an oracle dba; this is more fun though!) and let me tell you... this forum is the best place to figure anything out or get answers when you need them! Here are some of the things that I learned over the last year....

1. Become skilled in searching for answers among the posts. Sometimes you have to take a strange approach to get the answer you need, but it generally is in the forums somewhere. Chances are very good that someone somewhere had the problem before you did and was given an answer.

2. Don't be afraid to post. People really love helping out newbies and welcome you with open arms. You never have to worry about anyone being anything less than completely professional.

3. Have a test system somewhere in your environment. Skills and commands get lost quickly if you don't practice. Make sure you keep some kind of notebook around so that you can document processes and steps so you can back out of the holes you find yourself in.

4. Learn the "man" command. It will save you lots of time.

5. Try not to be overwhelmed by the myriad of details that make up this wonderful world called HP-UX. That is the hardest thing by far....


Welcome to HP-UX!