rebuilding dev environment

[maria paschali]

hi,

I have been having problems with my development environment (A400 running HPUX 11.0). System Panics, system files changing their permissions, applications not working and more. I don't know how or why this has come about. I have checked the /var/adm/* logs that reside in that directory but they don't tell me much. Apart from that directory and those logs, I am not too sure where to look. I am the only one who has access to the root password and I haven't changed anything (mainly because Im too scared I would break it :P).

The development environment contained 3 databases and 3 applications.

I would like to find out what is going on, but I am not too sure where to go from here.

Secondly, my boss has requested that I rebuild the server, however, I am not so sure it is a good idea if i don't know what the problem is. With the rebuild of the server I am going to remove all the application directories and all the /u0x/* directories. At this stage I don't know if it is better to re-install the operating system to ensure that all the system files have the correct permissions and it is a clean build or whether I should just run the production make_recovery tape on the development server. And copy the contents from the backup tapes onto the dev server.

If anyone can give me a heads up on where, how, why or whatever I should do, I would be very grateful.

Thanks

Maria

[Animesh Chakraborty]

Hi,
Are you sure there is no hardware problem?Sometimes the system panics may occur as a result of hardware failure.

After HP-UX experiences a system panic, the system:

May display an HPMC tombstone on the console if panic was caused by an HPMC. A tombstone is a list of register values used for troubleshooting.
May attempt to save a core file (an image of physical memory) to the dump device (by default this is the primary swap device).

Attempts to reboot.
Usually displays a panic message on the console. A panic message consists of several lines of text starting with the heading System Panic.

May attempt to copy the core file to the file system

I wud suggest to diagonose the problem first otherwise all your effort of reloading OS may go in vein.

[maria paschali]

The tombstone file has nothing in it. And it hasn't produced a core dump or anything like. I asked HP to check the GSP messages that I got, and so far they have told me that the version of GSP maybe old. I am still waiting to hear back from them. Unfortunately, I don't have the luxury of time. Hence why the request for the rebuild. But I still want to find out what went wrong.

Maria

[Steven Sim Kok Leong · Accepted solution]

Hi,

Some additional checks:

1) Check your cron jobs in /var/spool/cron/atjobs and /var/spool/cron/crontabs for suspicious entries.

2) Check your /etc/passwd file for any suspicious entries (normal user) other than root that contains uid 0 ie. 3rd field is 0(superuser) or gid 3 ie. 4th field is 3(sys).

3) Check your umask settings by typing umask. Check for suspicious world-writable files.

4) Look into the history file of root and other user accounts on the system if you have history enabled ie. /.sh_history and /home/*/.sh_history for suspicious commands being executed.

5) For files that permissions were changed, check their last modified date/time. Try to tally with any user logins during that period.

Hope this helps. Regards.

Steven Sim Kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

[Duncan Edmonstone]

Are you *sure* none of your developers know how to get in to the root account? Every company has a developer who sayas he can't do his job without root access!! ;o) Two quick checks:

grep "\-root$" /var/adm/sulog

shows all users who have attempted to su to root, where '+' = success and '-' = failure ...

last | grep root

All the times root was logged in recently... can you account for all these times?

Of course these files may have been doctored...

HTH

Duncan