topic Re: Suggested Troubleshooting Training Exercises in Operating System - HP-UX

Suggested Troubleshooting Training Exercises

Bill McNAMARA_1 — Wed, 20 Jun 2001 07:28:07 GMT

Hi, in a training environment can anyone suggest some interesting troubleshooting exercises..
I currently do the following:
1 delete /etc/lvmtab
2 Destroy the label file.
3 corrupt ISL
4 delete the kernel
5 move the boot disk to another HW path
6 create a file called -rf
7 resize /var
8 fsadm and single user
and my favourite:
9 rename /dev to /Dev

Any other popular final test type exercises
??

Thanks,
Bill

Re: Suggested Troubleshooting Training Exercises

Victor BERRIDGE — Wed, 20 Jun 2001 07:51:06 GMT

Hi Bill,

Create can happen cases like:
files that fill / => /dev/rmt/ (when someone misspells the correct device)
corrupt /etc/inittab (I like this one, you can spend quite some time on it if you never seen the case...)
ll>/stand/vmunix (I had this happen once on a AIX box...)

I have a meeting - got to leave

All the best
Victor

Re: Suggested Troubleshooting Training Exercises

eran maor — Wed, 20 Jun 2001 07:54:06 GMT

Hi Bill
i will try to answer all the answer .

1. if you delete the lvmtab you can oper the vgscan to detect all the vg that you had .
sometime when i have a corrupt lvmtab i m changing the name and doing vgscan again

2. dont know any other way then pvcreate but will wait for other answer

3. boot from the support cd .
booot and choose the Interact with IPL

boot the system and dd if=/usr/lib/uxbootlf of=/dev/rdsk/cXdXsX bs=512k

4. you can boot from the old kernel and you can also boot from the recovery disk and put another kernel .

5. you can seatch for the boot device in after interact with the system and change the hardware path of the boot system .

6. dont have a answer

7. you can boot in a single user mode and and change the size of the /var but if you have advent vxfs it will be much easier .

9. i has few days ago not the same problem but the recovery for the case is the same , one of my customer decide to clean the / and move the /etc to a diff. vg and after that the system didnt boot ... ha ha ha

i had 2 option

9.1 you can boot from the recovery disk , create the vg00 and the file system .
do a vgimport to the disk and after that change the file name of the dir .

9.2 create on a diff. disk hp-ux 11.00 and boot from there and then create the vg and filesystem do a vgimport and change the name of the directory .

i know that i m in the level of the Wizard or kings but this case is what of the best cases in the forum .

bye bill

Re: Suggested Troubleshooting Training Exercises

Eberardo Pipani_1 — Wed, 20 Jun 2001 08:17:43 GMT

Hi
I will suggest to you:
1- forget or delete the root passwd
2- boot the hpux with the quorum override option: hpux -lq
3- the disk with swap partition is scrach

Eberardo

Re: Suggested Troubleshooting Training Exercises

Charles Harris — Wed, 20 Jun 2001 09:19:22 GMT

Some of my favourites:-

Network

1. Kill inetd
2. Re-arrange the ports for polpular services
3. Add useless static routes
4. Down the ethernet adapters

General:

1 .Alias popular commands with annoying options
2. Create duplicate user id's manually
3 .Jumble the /sbin/init.d start order
4. Corrupt the groups file

Hardware:

1. Disconnect scsi terminaters
2. Use loopback cables for ethernet connection.
3. Disconnect peripheral power leads

Have fun!

-ChaZ-

Re: Suggested Troubleshooting Training Exercises

Vincenzo Restuccia — Wed, 20 Jun 2001 09:34:54 GMT

mv /stand/vmunix /tmp

Re: Suggested Troubleshooting Training Exercises

Paula J Frazer-Campbell — Wed, 20 Jun 2001 09:41:58 GMT

Hi Bill

Just at test commences "kill -9 1"

and then give them a simple fault to trace whilst the sever dies.

Paula

Re: Suggested Troubleshooting Training Exercises

Vincent Stedema — Wed, 20 Jun 2001 09:56:21 GMT

Change the root shell to /usr/bin/ksh and then reboot the system.

Re: Suggested Troubleshooting Training Exercises

Bill McNAMARA_1 — Wed, 20 Jun 2001 10:01:10 GMT

Good answers, I'm going to try some this afternoon! (after I duplicate scsi/fc ids!)

I should tell you briefly the method to fix this stuff...

1 delete /etc/lvmtab
hpux -lm
vgscan
(but vgscan will not create it ideally)
(the lvmtab needs reworking)

2 Destroy the label file.
lvlnboot and restore

3 corrupt ISL
Boot from CD support and restore.
lifcp (man) to replace the way you want.

4 delete the kernel
Boot recovery CD and restore kernel
from backup.

5 move the boot disk to another HW path
Device file changes... thus need to update rootconf and lvm.. -lm vgexport vgimport vg00

6 create a file called -rf
Delete by inode number (usually they delete
all inodes in all filesystems with the same number! ... but
find . -xdev -inode # -exec rm -i {} \;
The -i is very important (ls -i to get #)
Thanks to the forums for that! I user to use
\ to whack out the specialness... I usually
create the file with loads of leading spaces too which makes the \ a little difficult)

7 resize /var
lvextend is kind of confusing in that the extend -L size is the new size not the extension. lvchange sometimes needs to be done if the lv is strict on contiguous allocation policy.
I use fsadm, to reduce (must defrag)
And extendfs in single user to extend... although fsadm can extend online.

8 fsadm and single user
and my favourite:
9 rename /dev to /Dev
You must boot off the CD, mount your boot disk then mv the directory back.. they never get it!
You really need to be lucky to see the /dev/ on the ls...
They usually reignite the server!

Re: Suggested Troubleshooting Training Exercises

Alex Glennie — Wed, 20 Jun 2001 10:14:22 GMT

ask them to get XDMCP indirect working an HP xterminal so it can list both hosts on the local subnet and
those on a separate subnet via the chooser.

I've been giving myself a nose bleed trying to figure this out the last 2 days ....it's not helped by about 50 different docs each suggesting a different sysntax / method.

knowing your previous life Bill any info you have on this would be greatly appreciated and I'm sure I'll retrn the favour some day ;)

Re: Suggested Troubleshooting Training Exercises

Bill McNAMARA_1 — Wed, 20 Jun 2001 10:21:45 GMT

Did you try to get dt to do the broadcast for you.. the Xterm xdmcps to the host who will broadcast for the xterm!
ie don't do it via the xterm cos I think he'll only xdmcp indirect on his own subnet... like the protocol specifies.

Are you on 9.11? It's been a long time!
I'll have a look around..

Bill

Re: Suggested Troubleshooting Training Exercises

Alex Glennie — Wed, 20 Jun 2001 10:27:46 GMT

Bill,

Cheers for the reply .... yep I'm trying dt to do the BC and yep I'm using 9.11 latest patches etc ....

I've got one last thing to try .... after lunch, so hang back till then .... if I still can't crack it I'll start a new post so I can dish out the pts !

thanks

Alex

Re: Suggested Troubleshooting Training Exercises

A. Clay Stephenson — Wed, 20 Jun 2001 11:53:19 GMT

Hi Bill,

Intentionally corrupt or mv some of the libraries in /usr/lib so that even simple commands like cp no longer work but the /sbin commands (mount mv etc.) still work and they can fix the problem.

If you want to be particularly devious change the subnet mask so that it is off by 1 or 2 bits; your victim can then get to some addresses by not others. Try to lead them down the path by having them convert the ip addresses to binary.

I would also change the duplex network setting on one end of a connection so that the link still functions at a diminished level. Pings and telnets still work but ftp transfers are dogs.

Have fun, Clay

Re: Suggested Troubleshooting Training Exercises

John Poff — Wed, 20 Jun 2001 12:13:07 GMT

Bill,

Here are a few things that I have tripped over before:

1. Change the permissions on /etc/passwd so that it is not world readable. This will cause the 'whoami' command to return "Intruder Alert".

2. Change the 'lost+found' directory to 'found+lost' and have them try some filesystem work (extend the filesystem, etc.). The fsadm command will complain if 'lost+found' is missing, but I have seen high-priced consultants trip over it and wonder what was wrong.

3. To be really nasty, you can create a little script that keeps forking itself so that it eventually uses up all the processes (maxes out nproc). Probably won't happen too often in real life, but it will be fun watching them scramble to try to figure it out.

4. Create a file or a directory named '*'. I worked with a young programming wizard once who thought it was funny to create subdirectories named that way. "Sorry, son, but you're in the big leagues now. We don't do that here."

5. Have them delete a large file that is held open by process and see if they can figure out why the space doesn't free up. I wish I had a dollar for every time someone has come to me with this one!

Have fun!

JP

Re: Suggested Troubleshooting Training Exercises

Volker Borowski — Wed, 20 Jun 2001 13:09:05 GMT

Hi Bill,

- filesystem mounted over non empty directory with a GB file in it (Where is the space ?)

- NFS:
local FS mounted on /top/below
/etc/exports contains /top
mount localhost:/top /check
touch /check/below/where_is_it
ls -l /top/below/where_is_it

And my favorite:
- Delete Additional User with UID 0
(key is NOT to delete all files belonging to this user :)

Volker

Re: Suggested Troubleshooting Training Exercises

Patrick Wallek — Wed, 20 Jun 2001 13:13:57 GMT

Bill,

Some things that HP does in their HP-UX troubleshooting class:

Login Problems:
1. screw up users .profile so that logging in shows really strange behavior. HP put dtterm as last line of the shell, I think. This caused some interesting behavior when logging into CDE.
2. put a space at the end of a users entry in the passwd file after the shell.

Boot Problem:

1. Create a file in /etc/rc.config.d called 'x' that lists all variables in other files in /etc/rc.config.d but does nothing with them. This has the effect of nulling out all variables which causes some really strange symptoms when the machine boots.

Re: Suggested Troubleshooting Training Exercises

Victor BERRIDGE — Wed, 20 Jun 2001 13:18:01 GMT

Desynchronizing mirrors can be fun also, when you show them stale disks...
also degrading performance of a system which has dual path (alternate links) by crossing the connections...

Re: Suggested Troubleshooting Training Exercises

Victor BERRIDGE — Wed, 20 Jun 2001 13:32:04 GMT

Changing perms on files also like /etc/hosts
Or if you are using DNS:
remove or rename nsswitch.conf AND be sure resolv.conf is owned by root with perm 400

More vicious is to change a box of network having forgotten to change its address (old net) using DNS (updated adress)...

Re: Suggested Troubleshooting Training Exercises

Magdi KAMAL — Wed, 20 Jun 2001 14:52:35 GMT

Hi,

What's about putting the following command :

shutdown -ry 0

at the end of /etc/bcheckrc file, just before activating all interrupts !

This will reboot and shutdown the system in a infinite loop !

Re: Suggested Troubleshooting Training Exercises

Alan Riggs — Wed, 20 Jun 2001 16:23:39 GMT

Things that reproduce symptoms I have encountered on "live" systems:

Remove/change the root entry in /etc/shutdown.allow. WHy can't you bounce the box?

Place a "garbage" file in /etc/rc.config.d to crash the startup sequence.

Go from LVM maintenance mode to init 3 directly. How do you clear the /dev/root value in mnttab/bdf?

Have quorum enforced on mirrored root VG. Disable one of the disks. How to boot?