Changed Run Levels

[David Waters_3]

I had 5 servers go from run level 4 to 1 during daily processing. Has anyone ever experienced something of this sort. I have no explanation as to why it happened but all 5 servers servers changed run levels around the same time.

Servers are running as virtual partions on rp7400 machines. All were on different boxes.

[Ross Zubritski]

Dave,

This is highly irregular. I would assume the init 1 command was issued on these boxes.

Regards,

RZ

[Pete Randall]

How many people have the root password?


Pete

[Patrick Wallek]

Check your /etc/shutdownlog and /var/adm/syslog/syslog.log

It sounds like someone did either a 'shutdown' (with no args. it will take you down to single-user mode) or an 'init 1'.

[John Dvorchak]

Only if someone type init 1 on all of the systems. You didn't by chance change root's shell from /sbin/sh did you?

[Stefan Farrelly · Accepted solution]

Thats unheard of - 5 servers all changing run levels by themselves ? The only this can happen is the init command. I would look at some script or program someone or something ran which erroneously did an init command not knowing it changes the run level!

[John Dvorchak]

Did you recently change anything in root's cron that could possibly have the word shutdown in it? Like a missnamed home grown script?

[Ross Zubritski]

Another interesting twist. grep for "init" on the .sh_history file(s)

Regards,

RZ

[Cheryl Griffin]

I would suspect a faulty application performing a kill of some sorts that does this.
See if anything has been logged:
# who -a
. system boot Mar 21 15:55
. run-level 4 Mar 21 15:55 4 0 S <-- look for the time that the run level was changed. There may be a record like this one.

Tracking the time at which the run level changed and what was running at that time, may help pinpoint the problem.

[David Waters_3]

Additional Comment from Dave Waters:

There is no indication from the servers that someone ran a init 1 or shutdown. I've checked everything. I was no in the office that afternoon it happened so I never got a chance to view the syslog.

[Cheryl Griffin]

Here's a better command:
# who -a /etc/wtmp |grep run-level

It will produce a report like:
. run-level S Feb 28 14:49 S 4 1
. run-level 1 Feb 28 14:49 1 3 S
. run-level 4 Feb 28 14:56 4 0 S
. run-level 4 Mar 3 15:53 4 0 S
. run-level 4 Mar 3 16:22 4 0 S
. run-level 4 Mar 6 11:07 4 0 S
Cheryl

[Patrick Wallek]

There is nothing in HP-UX that I know of that would cause this behavior other than someone or something doing an 'init 1' or 'shutdown'.

Check the /var/adm/syslog/OLDsyslog.log if the machines haven't been reboot since this occurred. Also check the /var/adm/cron/log and see if there is an 'init' or 'shutdown' somewhere. Check all the scripts in root's crontab.

[Steven E. Protter]

You have had a security/stupidity breach.

I tend to lean toward the latter, but someone probably inadertantly issues the init command, or maybe a root cron script running on all the systems.

If you are confident that those who have the root password would not do something like that, then change the root passwords anyway, maybe someone is sharing.

If someone did hack your root password, this would be a great way to have some "fun" but not hurt anything.

I'd check the sulog, strings /var/adm/syslog/wtmp
strings /var/adm/syslog/btmp

I would do a thorough look at security, if there isn't anyone with root access stupid enough to do this. This could mean there is someone smart enough and dangerous enough to do it.

If you really want to be paranoid, is there anyone at the organization smart enough to sniff the network for passwords. If root has ftp access, authentication is clear text. Same thing for telnet.

You can go with secure shell to stop this kind of breach.

Secure Shell: a replacement for rcp ftp and telnet that encrypts passwords

http://www.software.hp.com/cgi-bin/swdepot_parser.cgi/cgi/displayProductInfo.pl?productNumber=T1471AA


Link and the Chris Vail cookbook attached.

SEP

SEP