topic Re: Unknown rebooting on C3000 in Operating System - HP-UX

Unknown rebooting on C3000

Phil Danter — Tue, 08 Feb 2005 04:34:10 GMT

We have a c3000 HP-UX 10.20 machine that has been rebooting by itself every now and again, not having a great deal of core Unix experience, could someone help me find any logs /info of what is happening?

/etc/shutdown.log just has:

15:12 Fri Jan 21, 2005. Reboot: (by Machine!root)
13:46 Wed Jan 26, 2005. Reboot: (by Machine!root)
11:54 Mon Feb 7, 2005. Reboot: (by Machine!root)

The root password was changed in Dec last year , to see if anyone was reboot on purpose, but it is still happening.

Any ideas/pointers would be greatful.

Re: Unknown rebooting on C3000

melvyn burnard — Tue, 08 Feb 2005 04:37:55 GMT

well someone has the authority to shutdown this machine by issuing a reboot command.

I woul dstart looking for a cron job that regularly does this, also try to work out when this started happening and see if you can tie into it any changes/updates/ peopleleaving etc.

Re: Unknown rebooting on C3000

Peter Godron — Tue, 08 Feb 2005 04:42:43 GMT

Phil,
is there a pattern to these shutdowns, such a every Wednesday or a time?
If yes look at crontab
How long is it between shutdowns?
Can you leave the machine in single user mode and wait for the shutdown?
This would check external command or internal failure.
If you think its an external command start the machine to full mode and leave network disconnected.
Regards

Re: Unknown rebooting on C3000

Phil Danter — Tue, 08 Feb 2005 04:58:42 GMT

Melvyn/Peter,

There are no crontab jobs for any user on this machine, we also thought that it could be an old employee, hence the changing of the root password.

There doesn't seem to be any pattern to the time/dates of the reboots.A few times it has rebooted, then all come back up fine, then about 1 hour or so later rebooted again? And times vary from 11am to 6/7pm.

Unfortunately, the purpose of this machines needs it to be networked and booted up 24/7, so we are unable to un-network the machine and see if it makes any difference, much as i liked to.

Re: Unknown rebooting on C3000

Peter Godron — Tue, 08 Feb 2005 05:07:19 GMT

Phil,
what is better ?:
To take the downtime/network unavailablity until the problem is investigated and fixed or a unreliable machine that may shut down at any time.
I would fix the problem once on for all, rather than live with the machine crashing.
Can you move the application onto another machine until the problem is fixed?

Regards

Re: Unknown rebooting on C3000

Phil Danter — Tue, 08 Feb 2005 05:14:55 GMT

Peter,

We are making some head way into moving the application to a temp machine for the short term, but can you suggest how to identify the problem?

I have tried log files etc and all they say is it is rebooting by root, but as the root password has changed, and only the trusted 2 have it, what can I do to see what is effecting/causing the reboots?

Re: Unknown rebooting on C3000

Henk Geurts — Tue, 08 Feb 2005 05:29:32 GMT

hi Phil

is there an OVO/ITO Agent installed on this server?
The client can be invaded and rebooted (trough the itoagent) from the OVO Server side with few traces....
Who manages the OVO Server?

Also check root's .sh-history....

regards.

Re: Unknown rebooting on C3000

Gordon Morrison — Tue, 08 Feb 2005 06:00:43 GMT

Hi Phil,
You say you've already checked for offending cron jobs, so:
First, I would check to see if there are any UID 0 accounts other than root, (and get rid of them!)
Second, see if there is a file called /etc/shutdown.allow
if so, any user listed in there could be the culprit. Prevent anyone other than root from using the shutdown command like so:
# > /etc/shutdown.allow
# chmod 400 /etc/shutdown.allow
Third, is there a /.rhosts file on this system? If possible, get rid of it (at least temporarily until you find the cuplrit)

Also check the history files of all users, looking for shutdown and/or su
and check /var/adm/sulog and syslog to see who might be switching to root just before these shutdowns occur.

There is also a chance that someone has left a SUID script or program lying around, perhaps as a back door.

The following script comes courtesy of the Centre for Internet Security, and will find any SUID & SGID executables on the system:

for part in \
`awk '($3 ~ /^(hfs|vxfs)$/) { print $2 }' /etc/fstab`
do find "$part" $ -perm -04000 -o -perm -02000 $ \
-type f -xdev -print
done | grep -v "^/var/adm/sw/"

Re: Unknown rebooting on C3000

Phil Danter — Tue, 08 Feb 2005 06:57:03 GMT

Hi Gordon,

Yes all crontab listings are empty, so tried as per your message:

There are no other users with uid 0

There was a /etc/shutdown.allow file but it was empty, and read only to root, removed and created again to make sure.

There isn't a /.rhosts file.

None of the users historys contain any shutdown/reboot or su with 6months of the "reboots".

Sulog only identifes what i have been doing today. Syslog has line coinsiding with the last "reboot" which is:

Feb 7 11:54:48 Machine syslogd: going down on signal 15

I tried the small script and the follow error occurs:
find: bad option -04000

Any ideas?

Is there anyway to see if any hardware could be causing a reboot, rather than reboot/shutdown commands?

Hi Henk, I don't believe any ITO/OVO agents are installed on this machine.

No .sh-history found.

P.S. Many thanks so far, for everyones help

Re: Unknown rebooting on C3000

Peter Godron — Tue, 08 Feb 2005 07:12:39 GMT

Phil,
having had a word around another possible cause could be a faulty capacitor.
Not sure that this could cause a tidy reboot, I would expect a panic and/or crash.
Scan the motherboard for anything that looks like a like barrel and is bulged at the top.They should normally be flat and shiny tops. As you havn't got a spare machine, you will not be able to swap a motherboard for testing. So perhaps it may be time to place a HP call?!
Regards

Re: Unknown rebooting on C3000

Kent Ostby — Tue, 08 Feb 2005 07:44:41 GMT

Also check the /var/spool/cron/atjobs directory to see if a former user or someone has placed a bunch of reboot commands out here.

Make sure that you don't have an empty /var/adm/cron/at.deny file either since an empty at.deny file could allow users to put in at jobs.

Change your root password. Have ONE person with the root password and have it written down inside a sealed envelope in case someone other than the one person needs it (its not a secret if more the one person knows it).

Finally, make sure your console is physically secure.

Re: Unknown rebooting on C3000

Gordon Morrison — Tue, 08 Feb 2005 08:19:23 GMT

Hi Phil,
I guess 10.20 can't handle that syntax :o/
try something like this:

find / -type f -exec ll {} \; |grep ^...s

Also, use "last" command to find out who was logged in when these reboots occurred. That should at least narrow your list of suspects.

As a last resort (I've never done this myself) you might want to consider temporarily moving /usr/sbin/shutdown and /usr/sbin/reboot to somewhere else and replacing them with scripts that will log the attempt (even send you an email, so they can't doctor the logs)

Re: Unknown rebooting on C3000

Phil Danter — Tue, 08 Feb 2005 09:03:41 GMT

Another machine will now be available for swapping while problem is investigated.

Although there are only 2 people who has the root password, 1 being me the other involved in trying to sort the problem with me, I will try the single person/envelope method.

../cron/atjobs does not exist
../at.deny does not exist

Still trying to find any Suid/Sgid files.

May have to try moving shutdown/reboot commands and logging/mailing any info.

Many thanks to all.

Re: Unknown rebooting on C3000

Robert-Jan Goossens — Tue, 08 Feb 2005 09:32:29 GMT

Hi Phil,

Try this

# find / $ -perm -4000 -o -perm -2000 $ -type f -exec ll {} \;

Regards,
Robert-Jan

Re: Unknown rebooting on C3000

B. Hulst — Tue, 08 Feb 2005 12:04:23 GMT

Hi,

Also correlate with the output of the last commmand (and the lastb).

I have had this same thing in the past.

You will see login attempts...
and reboots recorded there too!

And if root is on the console then check the serverroom door security records. :)

If it is a powerfailure then you only see the startup and not the reboot.

(And I assume you have the c3000 as part of a cluster.)

Regards,
Bob

Re: Unknown rebooting on C3000

Mark Ellzey — Wed, 09 Feb 2005 12:46:18 GMT

Phil,

Don't discount the possiblity that it's a hardware error. I have an old 715 that regularly reboots itself, at random intervals.

The box is a license server for some software that is so rarely used that I don't worry about fixing it. I just make sure the box is up if someone complains.

Regards,
Mark

Re: Unknown rebooting on C3000

Stephen Keane — Thu, 10 Feb 2005 11:47:49 GMT

A shot in the dark as well, but do you have ups connected to the server, or the ups_mond daemon running?

You could also rename the reboot/shutdown commands (temporarily) to see if the culprit shows up.

Re: Unknown rebooting on C3000

Eric Crosby_1 — Tue, 22 Feb 2005 13:35:01 GMT

I think most things have been covered, but here's something that may help...

move reboot and shutdown to reboot.orig and shutdown.orig

change perms on reboot.orig and shutdown.orig to 400

create wrapper scripts that records some info, then kicks off the cmd:
w
ps -ef
lsof (list of open files)

If this doesn't give you the info on what is rebooting, then I would suggest tunring auditing on.

Re: Unknown rebooting on C3000

MarkSyder — Wed, 23 Feb 2005 03:53:17 GMT

Possibly a long shot, but do you have any users with sudo access? If so, make sure they don't have the right to switch user to root.

Mark Syder (like the drink but spelt different)