DO and DON'T for sys admin

[Steven Sim Kok Leong]

Hi,

DO (for security):

Follow the excellent CIS HP-UX Benchmark v1.0.3 from:

http://www.cisecurity.org/bench_HPUX.html

DON'T (for security):

Avoid a single point of security failure by distributing critical services onto separate dedicated servers i.e. a DNS server should only run DNS functions and should not double as webserver or database server. A critical server should be dedicated in a singular dedicated function. In short, don't put all your eggs in one basket.

Hope this helps. Regards.

Steven sim Kok Leong

[Sebastian Galeski_1]

Never do many changes at once, especially when You tune kernel. If You change something make some kind of notes for You and your partners.

[Paula J Frazer-Campbell]

HI

Set up and use a Change Control system

1. Identify change required.
2. Document it and check previous change controls done.
3. Identify ALL systems / sub systems / users affected.
4. Speak to users.
5. Plan procedure and document each stage.
6. Impliment on test server.
7. Test on test server.
8. Confirm with all affected that results as satisfactory.
9. Revise plan - include revert.
10. Get final clearance.
11. Impliment plan - document all the way.
12. Monitor and test.
13. File in Change control.


The more of the procedure that is documented the less chance of a failure.

BAD :- Change lan card.
1. Down server.
2. Fit new card.
3. UP serve and test.

Good :- Change lan card.
1. Identicy lan by part nO and ensure that correct card is available.
2. Inform business of problem / priority and estimated down time.
3. Get window for card change.
4. Backup server.
5. Ignite backup server.
6. ETC
7. ETC
8. Shutdown / shutdown -h now / OK yes/no


I am sure you see the difference.

I also document each command - more so when dealing with disks.


HTH

Paula

[Steve Lewis]

Don't:
put . in root's path;
test a user's script as root, because you may unwittingly install a trojan horse;
let your test system windows/prompts look similar to your live/production system, in case you run something on the wrong system by accident.

Do:
be paranoid about security;
build in security from the start, its a lot easier to get application developers to work around it from the beginning than to change a system later and suddenly find that things stop working;
install your security patches asap (e.g. snmp, ssh etc)

[James R. Ferguson]

Hi:

DO : Read the 'man' pages! There is an incredible wealth of information to be gained.

DON'T : Panic in a crisis. Rather remain methodical in your problem-solving. Don't be hasty just because the clock is ticking and the system isn't up. A bad situation can be made worse by a poorly planned attack.

Regards!

...JRF...

[Pete Randall]

Do:
Document
Plan
Communicate
Backup

Don't:
Assume anything


Pete

[Mark Vollmers]

Hi

Do try to simplify easy or often used commands with scripts to save time for bigger items. Do test these before letting them run free and check outputs occasionaly.

Don't write a ton of scripts automating everything just so you can sit around all day. automation is good but not everything should be automatic, and you're on your way to scripting yourself out of a job :)

Mark

[John Bolene]

Do

keep the system running smooth
this may mean not installing patches, terminating errant jobs and print files, or it may mean installing all the patches you can find

allow others to fail at times, as trial and error is still the best teacher. allow the failures to happen on the test machines, not in production, if at all possible

have test machines that you can play as much as you want, but can also restore back to a stable environment when you rm -rf at the wrong place


don't

keep your knowledge to yourself, you may want a vacation at times and if you are the only one who can do things, you may not get to go

[PIYUSH D. PATEL]

Hi,

Keep a hardcopy ( printout ) of all the important files and keep updating them whenever you make any changes.

Before issuing a command do read the man pages.

DONT PANIC.......HPUX FORUMS ARE THERE TO HELP YOU OUT !!!!!

Keep your backup media at another site also, incase of a fire you can still have your data.

Piyush

[Victor_5]

Do

1. Always make backup, and keep backup tape in a safe place
2. See two more seconds when you run some critical commands on production servers

Here is an example. One day, I need modify crontab, since it was a urgent case and I typed it very fast, I typed crontab -r instead of crontab -e because letters "r" and "e" are sitting together on the keyboard.
3. Inform every related person before you do some changes.
4. Always keep log file and good documentation.

Don't
1. Donot set up password longer than 8 characters in trusted system.
2. Donot make assumption in any case, especially when you trying to show something to your manager or to a meeting, you must provide something to support your conclusion.

[Helen French]

Hi there:

DO:
1) Love HP-UX
2) Partcipate in forums
3) Learn anything new everyday !

DONT:
1) Love M$ OS
2) Panic at crisis
3) Ignore patches

my $0.02
Shiju

[Martin Burnett_2]

Hello,

DO
1. Rename files, do not delete them before you are sure you will never need it again.
2. Assign points to hard working forum members who take the time to share and answer your questions.
3. Mentor a "newbie" sysadmin.
4. Document, document, document.

DON'T
1. Keep that important piece of infromation to yourself.
2. Harass your end-users, remember they are why you are here!
3. Try to do it all yourself.

Thanks for participating in the forums,

Martin
Chaos reigns within. Reflect, repent, and reboot. Order shall return.

[Mark Greene_1]

DO:
- label your tape drives if you have more than one of the same type
- change your root password every 3-4 months
- use mixed case and non-alpha characters in *all* your passwords
- review the syslog and root mail every day
- test your ability to restore from a backup at least once every 6 months

DON'T:
- assume that everyone knows which tape drive is 0 and which is 1
- leave the console logged-in as root if you are not in the room
- use the same root password on all of your systems
- blindly delete the syslog or root mail without reading it
- assume that all your back-ups completed without error, actually contain data, or are transactionally correct without having tested them to verify that this is so

HTH
mark

[Jon Mattatall]

Do -

When making changes, write it down. If interrupted while making changes, write it down. Keep a record of the changes.

Monitor your systems. Nothing looks as good as having solved a problem before the client can get through to you.

Be nice to your DBA's. One day you'll need the sys password.

Create a /root home dir -
mkdir /root
cp .profile .sh_history /root
chmod 700 /root
vipw (change / to /root)
A good place for ssh keys and such.

vi /etc/profile and add
export PS1=`id -un`@`hostname`':$PWD:$?'
because it's easy to lose track of where you are and what just happened.

Dont -

Hand out uid 0 accounts - use sudo/restricted SAM

Assume your backups are fine - check the tapes occasionally

Use wildcards unless you're SURE of what will happen.

Jon

[Richard Darling]

ABCD
"A"lways:
"B"ackup, show "C"ourtesy to all and "D"ocument...
Richard Darling

[Fabi]

DO's

1.- Make Manuals and MOP's (Method of Procedure) for all your special procedures
2.- Keep your UNIX and sysAdmin docs near

DON'T 's

1.- Never hide tech information
2.- Keep that ugly word "impossible" away from your dictionary, specially for hackers and crackers, like "its impossible for anyone to break in the system" never think this way

[MAD_2]

DO'S
- Keep registration numbers, system handles, serials, software keys, and other software/hardware documents in a place where you and the systems' manager can find it! You never know when you will need it again.
- Document software configuration parameters
- Train someone to act as a backup who can perform some of the basics when you are not around (vacations, sickness, etc.)
- Be patient!
- Learn to prioritize
- Use labels to identify devices, systems, etc.
- Change root passwords when anyone from your team who has access to it leaves the company
- Assign points to those who are trying to help you in the forums.

DON'T
- Drink anything when you are working next to your server (a $0.50 drink may cost your company thousands, and your job).
- Be afraid to ask, nobody knows everything!
- ASSUME
- Keep all the knowledge to yourself.

[Kurt Beyers.]

DON'T

-do a kernel rebuild via a modem connection to a server
-reset your NIC from a telnet session

[Balaji_3]

It is better to change any entry in /etc/passwd by the command /usr/sbin/vipw.

[Chris Wilshaw]

If possible, don't let your users have access to the command line.

They may not like being given a menu to start with, but as long as you can make it flexible enough to meet their needs, they can grow to love it.

[James Brelsford]

Working in my company's training centre where everyone including the cleaners seems to have the root password, here is my list.

DO
make backups the second the new software is installed,

if you get a chance make your own copies as well as system backups,

ALWAYS keep the last working software close at hand in case you have to fall back, software installation should not be a one way street,

get good at ignite, you will be doing it alot,

if you can automate something do it, 'cause if you have to do it once you'll have to do it lots.

check the forums OFTEN,

ask dumb questions, you GOTTA learn sometime better to ask a dumb question here than say I didn't know to the users and your boss,

have good humour, you'll need it!!!!!


DON'Ts

be a jerk,

you are not Borg (TM), write things down if not for yourself for the guy who follows you after you are fired.

let them get you down,

[Donald Kok]

DO make sure you know what a system is about when it is not working properly. I do this by mailing settings and configfiles to another machine from cron:

ioscan -fn >> /tmp/listfile
vgdisplay -v >> /tmp/listfile
lvlnboot -v >> /tmp/listfile
sysdef >> /tmp/listfile
lanscan >> /tmp/listfile
ifconfig lan0 >> /tmp/listfile
hostname >> /tmp/listfile
setboot >> /tmp/listfile
lsdev >> /tmp/listfile
lpstat -a -s >> /tmp/listfile
bdf >> /tmp/listfile
cat /etc/fstab >> /tmp/listfile
cat /etc/hosts >> /tmp/listfile
cat /etc/nsswitch.conf >> /tmp/listfile
cat /etc/rc.config.d/* >> /tmp/listfile
cat /etc/resolv.conf >> /tmp/listfile
cat /etc/inittab >> /tmp/listfile
cat /etc/inetd.conf >> /tmp/listfile
strings /etc/lvmtab >> /tmp/listfile
cat /stand/system >> /tmp/listfile
cat /var/adm/sbtab >> /tmp/listfile
lvdisplay /dev/vg*/lvol* >> /tmp/listfile
mailx -s settings donald@another_machine < /tmp/listfile

[Roger Baptiste]

DO:

Keep a handy document containing essential details of your system. Just the essential details! (name, IP, handle, Backup times, drives connected, applications runnings, DB running, contact user, contact dba).

Take ignite backup on a regular basis (especially if it is a Prod box).

Make sure the daily backups are running correctly.

Make copies of file before you change them.
eg: /etc/fstab ..

Implement monitoring measures for tracking error messages (syslog, dmesg ..)

Enjoy!

DON'T:
Lose the thirst to learn and improve.

HTh
raj

[Wodisch]

Hi Printaporn,

very nice thread, indeed!

My $0.02:

* DO:
- Actually test your Ignite-recovery-tapes and procedures
- Actually test your data-recovery-tapes and procedures
- document everything on at least two different places/forms of media (e.g.: disk & paper)
- use "ssh" and "scp" (or "rsync" over "ssh") instead of "telnet" and "ftp"
- use "usermod" instead of "vipw"
- setup, document, and test SOPs (Standard Operating Procedures), for those cases you are NOT around
- setup a process to regularly oder/buy new tapes and replacement of the old ones
- keep ALL your passwords in a sealed/closed envelope in a safe where your company could reach them in the worst case (you're not available - or dead)
- record and document all the incidents/problems in a place available to the rest of your team
- apply "chmod +t /tmp /var/tmp" to your systems
- start "pfs_mountd" and "pfsd" only when needed, and "kill -15 $(UNIX95=. ps -C pfsd -o pid='') $(UNIX95=. ps -C pfs_mountd -o pid='')" them after the "pfs_umount"
- subscribe to hp's security mailing list
- do check wether the answers you get here really work for you/your systems (but check on a test system!)
- keep a pre-configured "Emergency-License" fax (as file AND on paper) for all your OpenView products at hand
- use trip-wire like tools to identify changed config-files (in addition to versioning tools a'la SCCS/RCS/CVS)
- ask here
- check the "mail me" box when you post a new question (and do it FIRST, before entering even the subject)
- append your solution/conclusion/experience at the end of your thread

* DON'T:
- use aliases with the names of the original commands (like "alias rm=??rm -i??"), instead use names NOT used in any UN*X systems (e.g. "alias rmi='rm -i'") - then you get errors on a foreign system, at least
- reuse old tapes (keep them in a safe for months, then *destroy* - physically! You may use Alexander's beetle for that ;-)
- drink and root :-)
- use paperclips to open CD-ROM drives (or let anybody else do this)
- use "umount" on "pfs_mount"ed CD-ROMs
- use any KSH/POSIX-SH interactive enhancements outside of "if [ -t 0]; then" and "fi # [ -t 0 ]" (your scripts may not like them otherwise)!
- use unsupported *tricks* (except your KNOW what you do)
- do something without test (like modifying "init*ora" without restarting the instance afterwards)

Regards,
Wodisch

[Trond Haugen]

DO:
Know how long it takes to boot your system[s].
Know the time it takes to install a patch bundle.
Plan you upgrades and other changes.
Know your maintenance window.
Know the time it takes to reinstall your Ignite backup.

That way if anything goes wrong you will know when you have to start the Ignite-remote procedure if anything goes wrong.

Don't trust luck, there is always Murphy.

Regards,
Trond