topic Re: Linux server running on VMware is rebooting automatically in Operating System - Linux

Linux server running on VMware is rebooting automatically

senthil_kumar_1 — Wed, 02 Jun 2010 18:29:22 GMT

Hi All,

We are running linux servers (SLES and RHEL) on VMWARE.

The server are rebooting automatically.

I have found following details from server log (/var/log/messages)

Jun 2 10:07:23 emdweb01 shutdown[26879]: shutting down for system reboot
Jun 2 10:07:23 emdweb01 init: Switching to runlevel: 6
Jun 2 10:07:25 emdweb01 auditd[2774]: The audit daemon is exiting.
Jun 2 10:07:25 emdweb01 kernel: audit(1275491245.818:7): audit_pid=0 old=2774 by auid=4294967295
Jun 2 10:07:28 emdweb01 sshd[3139]: Received signal 15; terminating.
Jun 2 10:07:29 emdweb01 zmd: ShutdownManager (WARN): Preparing to shut down...
Jun 2 10:07:29 emdweb01 zmd: ShutdownManager (WARN): Shutting down daemon...
Jun 2 10:07:30 emdweb01 kernel: VMware memory control driver unloaded
Jun 2 10:07:30 emdweb01 kernel: Kernel logging (proc) stopped.
Jun 2 10:07:30 emdweb01 kernel: Kernel log daemon terminating.
Jun 2 10:07:32 emdweb01 syslog-ng[1913]: syslog-ng version 1.6.8 going down
Jun 2 10:08:23 emdweb01 syslog-ng[1947]: syslog-ng version 1.6.8 starting
Jun 2 10:08:27 emdweb01 ifup: lo
Jun 2 10:08:27 emdweb01 ifup: lo
Jun 2 10:08:27 emdweb01 ifup: IP address: 127.0.0.1/8
Jun 2 10:08:27 emdweb01 ifup:
Jun 2 10:08:27 emdweb01 ifup: eth1 device: Intel Corporation 82545EM Gigabit Ethernet Controller (Cop
per) (rev 01)
Jun 2 10:08:27 emdweb01 ifup: eth1 configuration: eth-id-00:50:56:a0:68:bf
Jun 2 10:08:27 emdweb01 ifup: eth1
Jun 2 10:08:27 emdweb01 ifup: IP address: 10.179.9.13/16
Jun 2 10:08:27 emdweb01 ifup:
Jun 2 10:08:27 emdweb01 syslog-ng[1947]: Changing permissions on special file /dev/xconsole
Jun 2 10:08:27 emdweb01 syslog-ng[1947]: Changing permissions on special file /dev/tty10
Jun 2 10:08:27 emdweb01 syslog-ng[1947]: Cannot open file /dev/tty10 for writing (Permission denied)
Jun 2 10:08:27 emdweb01 ifup-route: Warning: Could not set up default route via interface
Jun 2 10:08:27 emdweb01 ifup-route: Command ip route replace to default via 10.178.9.1 returned:
Jun 2 10:08:27 emdweb01 ifup-route: . RTNETLINK answers: Network is unreachable
Jun 2 10:08:27 emdweb01 ifup-route: Configuration line: default 10.178.9.1 - -
Jun 2 10:08:27 emdweb01 ifup-route: This needs NOT to be AN ERROR if you set up multiple interfaces.
Jun 2 10:08:27 emdweb01 ifup-route: See man 5 routes how to avoid this warning.
Jun 2 10:08:27 emdweb01 ifup-route:
Jun 2 10:08:28 emdweb01 ifup: eth2 device: Intel Corporation 82545EM Gigabit Ethernet Controller (Cop
per) (rev 01)
Jun 2 10:08:28 emdweb01 ifup: eth2 configuration: eth-id-00:50:56:a0:7d:e9
Jun 2 10:08:28 emdweb01 ifup: eth2
Jun 2 10:08:28 emdweb01 ifup: IP address: 10.178.9.13/16
Jun 2 10:08:28 emdweb01 ifup:
Jun 2 10:08:29 emdweb01 auditd[2775]: Init complete, auditd 1.2.9 listening for events
Jun 2 10:08:29 emdweb01 ifup: sit0
Jun 2 10:08:29 emdweb01 ifup: No configuration found for sit0
Jun 2 10:08:29 emdweb01 kernel: klogd 1.4.1, log source = /proc/kmsg started.
Jun 2 10:08:29 emdweb01 kernel: AppArmor: AppArmor initialized
Jun 2 10:08:29 emdweb01 kernel: audit(1275491301.671:2): info="AppArmor initialized" pid=1767
Jun 2 10:08:29 emdweb01 kernel: ACPI: AC Adapter [ACAD] (on-line)
Jun 2 10:08:29 emdweb01 kernel: ACPI: Power Button (FF) [PWRF]
Jun 2 10:08:29 emdweb01 kernel: No dock devices found.
Jun 2 10:08:29 emdweb01 kernel: vmmemctl: module not supported by Novell, setting U taint flag.
Jun 2 10:08:29 emdweb01 kernel: VMware memory control driver initialized
Jun 2 10:08:29 emdweb01 kernel: vmmemctl: started kernel thread pid=2247
Jun 2 10:08:29 emdweb01 kernel: e1000: eth1: e1000_set_tso: TSO is Enabled
Jun 2 10:08:29 emdweb01 kernel: e1000: eth2: e1000_set_tso: TSO is Enabled
Jun 2 10:08:29 emdweb01 kernel: e1000: eth1: e1000_watchdog_task: NIC Link is Up 1000 Mbps Full Duplex, Flow C
ontrol: None
Jun 2 10:08:29 emdweb01 kernel: audit(1275491307.960:3): operation="inode_permission" requested_mask="rw" deni
ed_mask="r" name="/dev/tty10" pid=1947 profile="/sbin/syslog-ng"
Jun 2 10:08:29 emdweb01 kernel: e1000: eth2: e1000_watchdog_task: NIC Link is Up 1000 Mbps Full Duplex, Flow C
ontrol: None
Jun 2 10:08:29 emdweb01 kernel: audit(1275491308.956:4): audit_pid=2775 old=0 by auid=4294967295
Jun 2 10:08:29 emdweb01 kernel: NET: Registered protocol family 10
Jun 2 10:08:29 emdweb01 kernel: lo: Disabled Privacy Extensions
Jun 2 10:08:29 emdweb01 kernel: IPv6 over IPv4 tunneling driver
Jun 2 10:08:30 emdweb01 kernel: IA-32 Microcode Update Driver: v1.14
Jun 2 10:08:30 emdweb01 kernel: IA-32 Microcode Update Driver v1.14 unregistered
Jun 2 10:08:30 emdweb01 /usr/sbin/cron[3051]: (CRON) STARTUP (V5.0)
Jun 2 10:08:31 emdweb01 rcpowersaved: CPU frequency scaling is not supported by your processor.
Jun 2 10:08:31 emdweb01 rcpowersaved: enter 'CPUFREQ_ENABLED=no' in /etc/powersave/cpufreq to avoid this warni
ng.
Jun 2 10:08:31 emdweb01 rcpowersaved: Cannot load cpufreq governors - No cpufreq driver available
Jun 2 10:08:31 emdweb01 rcpowersaved: s2ram does not know your machine. See 's2ram -i' for details. (127)
Jun 2 10:08:31 emdweb01 rcpowersaved: Use SUSPEND2RAM_FORCE=yes to override this detection.
Jun 2 10:08:31 emdweb01 sshd[3151]: Server listening on :: port 22.
Jun 2 10:08:31 emdweb01 zmd: NetworkManagerModule (WARN): Failed to connect to NetworkManager
Jun 2 10:08:39 emdweb01 kernel: eth2: no IPv6 routers present
Jun 2 10:08:39 emdweb01 kernel: eth1: no IPv6 routers present
Jun 2 10:09:26 emdweb01 kernel: hda: lost interrupt

This is happening for second time after may 28th 2010.

My Questions:

1)Why the issue is happening?

2)Where is the problem?

Re: Linux server running on VMware is rebooting automatically

Tim Nelson — Wed, 02 Jun 2010 18:53:00 GMT

looks like a nice shutdown vs a crash..

either user is initiating, something in cron, or initiated from VM Console ?

Re: Linux server running on VMware is rebooting automatically

TwoProc — Wed, 02 Jun 2010 20:23:38 GMT

Vmware has "automatic" facilities to update your servers, patch them, etc. Your output looks like a clean ordered shutdown. If you can't find the reason in the virtual machines themselves, and it is happening across two different types of Linux installations - then it may be the host. Review the settings for the vmware host and see if some tools for automating patching, etc, are turned on. Look for a change since the 28th, of course.

Re: Linux server running on VMware is rebooting automatically

senthil_kumar_1 — Thu, 03 Jun 2010 12:30:58 GMT

Hi All,

I have found that the process "dialog" is utilizing more percentage of CPU.

Ex:

#top

top - 08:29:18 up 22:21, 1 user, load average: 1.41, 1.34, 1.30
Tasks: 67 total, 3 running, 64 sleeping, 0 stopped, 0 zombie
Cpu(s): 72.6%us, 26.4%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 1.0%si, 0.0%st
Mem: 16114344k total, 903528k used, 15210816k free, 132024k buffers
Swap: 2104504k total, 0k used, 2104504k free, 479052k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3188 root 25 0 9712 1836 1156 R 96.9 0.0 1156:44 dialog
1 root 16 0 796 312 256 S 0.0 0.0 0:00.28 init
2 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0

I have find the full path of the dialog file.

Ex:

# ps -ef | grep -i dialog
root 3187 3171 0 Jun02 ? 00:00:00 /opt/gnome/lib64/gdm/gdmopen -l /bin/sh -c /usr/bin/dialog --msgbox 'The GDM user '\''gdm'\'' does not exist. Please correct GDM configuration and restart GDM.' 16 70
root 3188 3187 86 Jun02 ? 19:17:15 /usr/bin/dialog --msgbox The GDM user 'gdm' does not exist. Please correct GDM configuration and restart GDM. 16 70

My Questions:

1)Why the process dialog is utilizing more percentage of CPU?

2)How to resolve the issue?

Re: Linux server running on VMware is rebooting automatically

TwoProc — Thu, 03 Jun 2010 13:47:20 GMT

Because the user "gdm" is not in /etc/passwd.
Did you or anyone else "clean" the /etc/passwd file?

Anyway, just create a user named "gdm" with a user id of 114 and a groupid of 120.

Re: Linux server running on VMware is rebooting automatically

senthil_kumar_1 — Fri, 04 Jun 2010 15:34:21 GMT

Hi John,

I have created the user ID in /etc/passwd like below.

# grep -i gdm /etc/passwd
gdm:x:50:15:Gnome Display Manager daemon:/var/lib/gdm:/bin/bash

I have created this with the reference of another Suse linux server.

Still the issue remains same.

Do we need to restart the server?

Re: Linux server running on VMware is rebooting automatically

TwoProc — Fri, 04 Jun 2010 16:36:25 GMT

Yes, I would think so,

Or you can just init to level 3, and then come back to level 5 from a command line shell.

If you're not in a gui (and I don't think you are because it's broken), and you're logged on into the console port, do the following:

As root:

init 3

hit return

init 5

Otherwise, yeah, then just reboot.

Re: Linux server running on VMware is rebooting automatically

senthil_kumar_1 — Mon, 07 Jun 2010 12:33:43 GMT

Hi John Joubert,

Shall I kill that process?

Re: Linux server running on VMware is rebooting automatically

TwoProc — Mon, 07 Jun 2010 23:24:26 GMT

When you run "init 3", it's going to kill the gdm for you, via a kill script in /etc/rc5.d.
Like wise when you re-enter init level 5, it will restart gdm via start script in /etc/rc5.d.