topic Re: How to prove that the server is rebooted manually in Operating System - HP-UX

How to prove that the server is rebooted manually

Robert Peregrin — Sun, 03 Oct 2010 06:59:30 GMT

Hi,

I'm investigating the reason for the reboot of our rp2405 server. There was a down time for another server in the same time that the rp2405 server was rebooted. But the people who did the down time does not want to admit that they rebooted the server.

I have reason to believe that they rebooted the server based on the following:

Reboot time from shutdownlog:
02:40 Sat Oct 2, 2010. Reboot: (by s7cs!root)

Users logged in at the time of reboot:

root pts/1 Sat Oct 2 02:52 - 06:06 (03:14)
giza pts/0 Sat Oct 2 02:47 - 06:06 (03:18)
reboot system boot Sat Oct 2 02:45 still logged in
root pts/3 Sat Oct 2 02:35 - 02:39 (00:03)
root pts/2 Sat Oct 2 02:29 - 02:40 (00:10)
giza pts/1 Sat Oct 2 02:28 - 02:40 (00:12)

User activities at the time of reboot:

1 pts/1 25212 8 0000 0000 1285969471 Oct 2 00:44:31 2010
giza 1 pts/1 5021 7 0000 0000 1285975705 Oct 2 02:28:25 2010 157.234.229.16 157.234.229.16
LOGIN 2 pts/2 5103 6 0000 0000 1285975794 Oct 2 02:29:54 2010 172.24.30.40 s7s101
root 2 pts/2 5103 7 0000 0003 1285975794 Oct 2 02:29:54 2010 172.24.30.40 s7s101
LOGIN 3 pts/3 6564 6 0000 0000 1285976151 Oct 2 02:35:51 2010 172.20.238.156 s7sdb3
root 3 pts/3 6564 7 0000 0003 1285976151 Oct 2 02:35:51 2010 172.20.238.156 s7sdb3
root 3 pts/3 6564 8 0000 0000 1285976362 Oct 2 02:39:22 2010
root td pts/td 495 8 0000 0000 1285976435 Oct 2 02:40:35 2010
a7hcHttp a7fh 2088 8 0000 0000 1285976435 Oct 2 02:40:35 2010
krsd krsd 2081 8 0000 0000 1285976435 Oct 2 02:40:35 2010
1 pts/1 5021 8 0000 0000 1285976435 Oct 2 02:40:35 2010
5 pts/5 21889 8 0000 0000 1285976435 Oct 2 02:40:35 2010
a7hcHttp a7hc 2084 8 0000 0000 1285976435 Oct 2 02:40:35 2010
LOGIN cons console 2080 8 0000 0000 1285976435 Oct 2 02:40:35 2010
7 pts/7 1271 8 0000 0000 1285976435 Oct 2 02:40:35 2010
root 2 pts/2 5103 8 0000 0000 1285976435 Oct 2 02:40:35 2010
errlogdW errd 23679 8 0000 0000 1285976435 Oct 2 02:40:35 2010
sfd sfd 2082 8 0000 0000 1285976438 Oct 2 02:40:38 2010
root 0 pts/0 10031 8 0000 0000 1285976438 Oct 2 02:40:38 2010
root p3 ttyp3 21830 8 0000 0000 1285976439 Oct 2 02:40:39 2010
root p2 ttyp2 21831 8 0000 0000 1285976439 Oct 2 02:40:39 2010
root p3 ttyp3 21830 8 0000 0000 1285976440 Oct 2 02:40:40 2010
root p2 ttyp2 21831 8 0000 0000 1285976440 Oct 2 02:40:40 2010
system boot 0 2 0000 0000 1285976723 Oct 2 02:45:23 2010
run-level 3 0 1 0063 0123 1285976723 Oct 2 02:45:23 2010
vxenable vxen 61 5 0000 0000 1285976723 Oct 2 02:45:23 2010
vxenable vxen 61 8 0000 0000 1285976723 Oct 2 02:45:23 2010
bcheckrc brc1 62 5 0000 0000 1285976723 Oct 2 02:45:23 2010
bcheckrc brc1 62 8 0000 0000 1285976724 Oct 2 02:45:24 2010
cat cprt 102 5 0000 0000 1285976724 Oct 2 02:45:24 2010
cat cprt 102 8 0000 0000 1285976725 Oct 2 02:45:25 2010
giza 0 pts/0 1750 7 0000 0000 1285976879 Oct 2 02:47:59 2010 157.234.229.16 157.234.229.16
rc sqnc 107 8 0000 0000 1285976935 Oct 2 02:48:55 2010
getty cons 1999 5 0000 0000 1285976935 Oct 2 02:48:55 2010
krsd krsd 2000 5 0000 0000 1285976935 Oct 2 02:48:55 2010
sfd sfd 2001 5 0000 0000 1285976935 Oct 2 02:48:55 2010
errlogdW errd 2002 5 0000 0000 1285976935 Oct 2 02:48:55 2010
a7hcHttp a7hc 2003 5 0000 0000 1285976935 Oct 2 02:48:55 2010
a7hcHttp a7fh 2004 5 0000 0000 1285976935 Oct 2 02:48:55 2010
LOGIN cons console 1999 6 0000 0000 1285976935 Oct 2 02:48:55 2010
LOGIN 1 pts/1 4243 6 0000 0000 1285977128 Oct 2 02:52:08 2010 172.24.30.40 s7s101
root 1 pts/1 4243 7 0000 0003 1285977129 Oct 2 02:52:09 2010 172.24.30.40 s7s101
0 pts/0 1750 8 0000 0000 1285988810 Oct 2 06:06:50 2010
root 1 pts/1 4243 8 0000 0000 1285988810 Oct 2 06:06:50 2010
LOGIN 0 pts/0 373 6 0000 0000 1285998318 Oct 2 08:45:18 2010 10.32.99.98 10.32.99.98
giza 0 pts

History of user activities on pts/2:

rcp /etc/group s7sdb3:/etc/group
rcp /etc/passwd s7sdb3:/etc/passwd
exit

rlogin s7sdb3
rlogin s7sdb3
reboot -r

rlogin as2

Only giza user account is logged in at the time of Central Server reboot. But the user who owns this account is denying it.

I noticed that syslog went down on signal 15. Signal 15 is only issued manually by root user right?

Oct 2 02:40:35 s7cs syslogd: going down on signal 15

There was no new files under /var/adm/crash.

I was trying to login to the console to look for possible power problems but the terminal is always giving the message to use Ecf but it doesn't work when I'm pressing ctrl Ecf.

Your help will be appreciated.

Regards,

Robert Peregrin

Re: How to prove that the server is rebooted manually

SUDHAKAR_18 — Sun, 03 Oct 2010 08:20:02 GMT

[Read-only - use ^Ecf to attach to console.]

Presss control+E together and cf

Re: How to prove that the server is rebooted manually

TTr — Sun, 03 Oct 2010 13:02:59 GMT

So you are saying that this account rebooted the wrong server?
Does this account have root access or is it in the shutdown.allow file?

Is the shell command history on? Look in the history file of the account to see if you find anything javascript&colon;postAnswerSubmit('submit');

Re: How to prove that the server is rebooted manually

Dennis Handly — Mon, 04 Oct 2010 02:14:33 GMT

>Reboot time from shutdownlog:
>02:40 Sat Oct 2, 2010. Reboot: (by s7cs!root)

This pretty much says the system was rebooted by root.

>User activities at the time of reboot:

What produced this output?

>History of user activities on pts/2:
>reboot -r

What produced this? This points to the reboot.

>Only giza user account is logged in at the time of Central Server reboot. But the user who owns this account is denying it.

Well, there were two root logins at the same time.

Re: How to prove that the server is rebooted manually

johnsonpk — Mon, 04 Oct 2010 03:57:17 GMT

Hi Robert,

Reboot was initiated by root user , check root user login time and source IP by "last -R "

regards
Johnson

Re: How to prove that the server is rebooted manually

Robert Peregrin — Mon, 04 Oct 2010 08:31:06 GMT

Hi Guys,

Here are the answers to your questions:

It is possible that the giza account accidentally rebooted s7cs server or it could really have been done. This is why I need to verify if the shutdown log indicate that the reboot initiated by root was manual or automatic?

The giza account was now disabled by the Admin because of the incident so I'm unable to verify the command history for that account at this time.

User activities details were from the wtmp file.

History of user activities came from pts/2 file under .sh_history folder.

Regards,

Robert Peregrin

Re: How to prove that the server is rebooted manually

Robert Peregrin — Mon, 04 Oct 2010 09:10:36 GMT

Hi Guys,

Here is the output of last -R.

root pts/1 s7s101 Sat Oct 2 02:52 - 06:06 (03:14)
giza pts/0 157.234.229.16 Sat Oct 2 02:47 - 06:06 (03:18)
reboot system boot Sat Oct 2 02:45 still logged in
root pts/3 s7sdb3 Sat Oct 2 02:35 - 02:39 (00:03)
root pts/2 s7s101 Sat Oct 2 02:29 - 02:40 (00:10)
giza pts/1 157.234.229.16 Sat Oct 2 02:28 - 02:40 (00:12)

Regards,

Robert Peregrin

Re: How to prove that the server is rebooted manually

johnsonpk — Mon, 04 Oct 2010 09:36:53 GMT

>Only giza user account is logged in at the time of Central Server reboot. But the user who owns this account is denying it.

well there was two sessions initiated from IP 157.234.229.16, trace that IP address

>History of user activities on pts/2:
>reboot -r

>root pts/2 s7s101 Sat Oct 2 02:29 - 02:40 (00:10)

some one logged in from "s7s101 " and executed a reboot , so you may need to go through the user's command history and wtmp on that s7s101 as well to find out the source IP/hostname from the user logged in

Re: How to prove that the server is rebooted manually

Dennis Handly — Mon, 04 Oct 2010 11:24:47 GMT

>This is why I need to verify if the shutdown log indicate that the reboot initiated by root was manual or automatic?

There is no difference. All we know is a reboot was done. (What do you mean by automatic, a cron job?)

>The giza account was now disabled by the Admin because of the incident so I'm unable to verify the command history for that account at this time.

It seems Admin should disable the Admin account since root did it. :-) Unless you have shutdown.allow.

Why can't you see the history for that account? You are the admin aren't you?

Re: How to prove that the server is rebooted manually

Robert Peregrin — Tue, 05 Oct 2010 09:18:40 GMT

Hi,

I'm not the admin but I used to have almost the same privileges as the admin but the Admin has now given me only restricted access.

Note that the user giza can rlogin to s101 as root and from there, this account can rlogin to s7cs as root.

Regards,

Robert Peregrin

Re: How to prove that the server is rebooted manually

Robert Peregrin — Tue, 05 Oct 2010 10:16:42 GMT

There is no way I can prove the manual reboot as my access privillege has been restricted.