topic Re: Changed Run Levels in Operating System - HP-UX

Changed Run Levels

David Waters_3 — Thu, 27 Mar 2003 15:07:33 GMT

I had 5 servers go from run level 4 to 1 during daily processing. Has anyone ever experienced something of this sort. I have no explanation as to why it happened but all 5 servers servers changed run levels around the same time.

Servers are running as virtual partions on rp7400 machines. All were on different boxes.

Re: Changed Run Levels

Ross Zubritski — Thu, 27 Mar 2003 15:09:53 GMT

Dave,

This is highly irregular. I would assume the init 1 command was issued on these boxes.

Regards,

RZ

Re: Changed Run Levels

Pete Randall — Thu, 27 Mar 2003 15:09:54 GMT

How many people have the root password?

Pete

Re: Changed Run Levels

Patrick Wallek — Thu, 27 Mar 2003 15:10:46 GMT

Check your /etc/shutdownlog and /var/adm/syslog/syslog.log

It sounds like someone did either a 'shutdown' (with no args. it will take you down to single-user mode) or an 'init 1'.

Re: Changed Run Levels

John Dvorchak — Thu, 27 Mar 2003 15:10:55 GMT

Only if someone type init 1 on all of the systems. You didn't by chance change root's shell from /sbin/sh did you?

Re: Changed Run Levels

Stefan Farrelly — Thu, 27 Mar 2003 15:12:19 GMT

Thats unheard of - 5 servers all changing run levels by themselves ? The only this can happen is the init command. I would look at some script or program someone or something ran which erroneously did an init command not knowing it changes the run level!

Re: Changed Run Levels

John Dvorchak — Thu, 27 Mar 2003 15:13:12 GMT

Did you recently change anything in root's cron that could possibly have the word shutdown in it? Like a missnamed home grown script?

Re: Changed Run Levels

Ross Zubritski — Thu, 27 Mar 2003 15:15:45 GMT

Another interesting twist. grep for "init" on the .sh_history file(s)

Regards,

RZ

Re: Changed Run Levels

Cheryl Griffin — Thu, 27 Mar 2003 15:16:28 GMT

I would suspect a faulty application performing a kill of some sorts that does this.
See if anything has been logged:
# who -a
. system boot Mar 21 15:55
. run-level 4 Mar 21 15:55 4 0 S <-- look for the time that the run level was changed. There may be a record like this one.

Tracking the time at which the run level changed and what was running at that time, may help pinpoint the problem.

Re: Changed Run Levels

David Waters_3 — Thu, 27 Mar 2003 15:16:29 GMT

Additional Comment from Dave Waters:

There is no indication from the servers that someone ran a init 1 or shutdown. I've checked everything. I was no in the office that afternoon it happened so I never got a chance to view the syslog.

Re: Changed Run Levels

Cheryl Griffin — Thu, 27 Mar 2003 15:19:53 GMT

Here's a better command:
# who -a /etc/wtmp |grep run-level

It will produce a report like:
. run-level S Feb 28 14:49 S 4 1
. run-level 1 Feb 28 14:49 1 3 S
. run-level 4 Feb 28 14:56 4 0 S
. run-level 4 Mar 3 15:53 4 0 S
. run-level 4 Mar 3 16:22 4 0 S
. run-level 4 Mar 6 11:07 4 0 S
Cheryl

Re: Changed Run Levels

Patrick Wallek — Thu, 27 Mar 2003 15:20:43 GMT

There is nothing in HP-UX that I know of that would cause this behavior other than someone or something doing an 'init 1' or 'shutdown'.

Check the /var/adm/syslog/OLDsyslog.log if the machines haven't been reboot since this occurred. Also check the /var/adm/cron/log and see if there is an 'init' or 'shutdown' somewhere. Check all the scripts in root's crontab.

Re: Changed Run Levels

Steven E. Protter — Thu, 27 Mar 2003 18:31:54 GMT

You have had a security/stupidity breach.

I tend to lean toward the latter, but someone probably inadertantly issues the init command, or maybe a root cron script running on all the systems.

If you are confident that those who have the root password would not do something like that, then change the root passwords anyway, maybe someone is sharing.

If someone did hack your root password, this would be a great way to have some "fun" but not hurt anything.

I'd check the sulog, strings /var/adm/syslog/wtmp
strings /var/adm/syslog/btmp

I would do a thorough look at security, if there isn't anyone with root access stupid enough to do this. This could mean there is someone smart enough and dangerous enough to do it.

If you really want to be paranoid, is there anyone at the organization smart enough to sniff the network for passwords. If root has ftp access, authentication is clear text. Same thing for telnet.

You can go with secure shell to stop this kind of breach.

Secure Shell: a replacement for rcp ftp and telnet that encrypts passwords

http://www.software.hp.com/cgi-bin/swdepot_parser.cgi/cgi/displayProductInfo.pl?productNumber=T1471AA

Link and the Chris Vail cookbook attached.

SEP

SEP