Server Management - Systems Insight Manager
cancel
Showing results for 
Search instead for 
Did you mean: 

Help identifying VMM task occurring under SIM 6.0 w/ ICE

Brian Hahne
Regular Advisor

Help identifying VMM task occurring under SIM 6.0 w/ ICE

We have a full installation of SIM 6.0, with ICE licensed on our servers. This is an upgrade from our install of 5.3. We were able to monitor virtual machines under 5.3, and it was "hooked" into our Vcenter server for management. We never did management or turning VM's on and off this way, but the hooks and connections were there.

So, the other day, I re-register the VM's and everything seemed ok, we could see all the VM's... so I let it ride overnight. In the morning, 2 of our 3 large VMWare servers (VMW 3.5 Enterprise) were dropped from the IO resource pool. Ok so this isn't a big deal to me, we weren't using IO yet anyway. I assumed it just added them to IO because it could. Looking at the vm's on 2 of the systems though, everything from name to status, ip, etc.. were "?" marks. So.. I re-registered the communication with SIM and the vm's came back.

Not 20 minutes later... the Vcenter server lost it's marbles. TONS of tasks showed up, within Vcenter, that were trying to "Reload Virtual Machine". Not sure what it was doing, the vm's weren't going up or down. But something was going on. The initiating user was the domain account we use to communicate amongst systems. So.. we knew SIM was doing it.
It would try and reload, say the VM couldn't talk to the VM host, and then try again.. over and over...

Does anyone know what this may have been doing?
I'd like to have VMM running, but I'm thinking I may try again and just disable the Vcenter integration til I know what the heck was going on.

I found no tasks running in SIM that would generate it.. So I'm not sure what it was.

Anyone have this happen or know what was going on?

Oh, if it matters, the VMWare hosts were registered with ICE, and had the performance monitoring turned on too... but this didn't seem to be related.

9 REPLIES
ozlace
Occasional Advisor

Re: Help identifying VMM task occurring under SIM 6.0 w/ ICE

Brian,
I actually had a client complaining about this and have asked them to open a support call. I had not seen it before until today! I just loaded up 6.1 here in my load and after about 12 hours suddenly the 'reload virtual machine' messages started. I am going to do some digging but let me know if anyone replies.
Brian Hahne
Regular Advisor

Re: Help identifying VMM task occurring under SIM 6.0 w/ ICE

I still haven't found a cause. I got bit by the SIM 6.x bug, so I worked with support on fixing that...

VMM and my continually crashing VCRM are my next 2 calls to support.. .plus the ISEE/WEBES not phoning home...

If you figure this one out first I look forward to your post.. I will post my calls to support too, but that won't be til Friday.
ozlace
Occasional Advisor

Re: Help identifying VMM task occurring under SIM 6.0 w/ ICE

I have had 2 problems with remote support pack both easily fixed. The first is the RSCC component not installing, search itrc using "smh rscc" first hit is the fix. Also phone home won't work properly if on system properties you are missing product number.
ozlace
Occasional Advisor

Re: Help identifying VMM task occurring under SIM 6.0 w/ ICE

Take alook at C:\Program Files\HP\Insight Control virtual machine management\log\hpvmmsvc.txt my reloads seem to be associated with the messages below. Looks like a java problem:

2010/07/22 02:38:20.516 | ERROR | pool-4-thread-4 | VM Failed to refresh performance data! ; nested exception is:
2010/07/22 02:38:20.517 | java.net.SocketTimeoutException: Read timed out
2010/07/22 02:38:21.017 | ERROR | pool-4-thread-7 | VM Failed to refresh performance data! ; nested exception is:
2010/07/22 02:38:21.017 | java.net.SocketTimeoutException: Read timed out
2010/07/22 02:38:21.317 | ERROR | pool-4-thread-14 | VM Failed to refresh performance data! ; nested exception is:
2010/07/22 02:38:21.317 | java.net.SocketTimeoutException: Read timed out
2010/07/22 02:38:21.317 | ERROR | pool-4-thread-6 | VM Failed to refresh performance data! ; nested exception is:
2010/07/22 02:38:21.317 | java.net.SocketTimeoutException: Read timed out
2010/07/22 02:38:21.317 | ERROR | pool-4-thread-19 | VM Failed to refresh performance data! ; nested exception is:
2010/07/22 02:38:21.317 | java.net.SocketTimeoutException: Read timed out
2010/07/22 02:38:22.117 | ERROR | pool-4-thread-18 | VM Failed to refresh performance data! ; nested exception is:
2010/07/22 02:38:22.117 | java.net.SocketTimeoutException: Read timed out
2010/07/22 02:38:24.116 | ERROR | pool-4-thread-13 | VM Failed to refresh performance data! ; nested exception is:
2010/07/22 02:38:24.116 | java.net.SocketTimeoutException: Read timed out
2010/07/22 02:38:32.215 | ERROR | pool-4-thread-8 | VM Failed to refresh performance data! ; nested exception is:
2010/07/22 02:38:32.215 | java.net.SocketTimeoutException: Read timed out
2010/07/22 02:38:41.613 | INFO | Timer-2 | Property update info of VirtualMachine vm-910: guest.disk (assign)
2010/07/22 02:39:09.907 | WARN | pool-2-thread-8 | VM Host 172.17.10.170 details refresh failed! Reason:null
2010/07/22 02:39:09.907 | WARN | pool-2-thread-8 | Update thread 28 failed for host :172.17.10.170Failure:java.lang.NullPointerException
2010/07/22 02:40:09.897 | WARN | pool-2-thread-1 | VM Host 172.17.10.170 details refresh failed! Reason:null
2010/07/22 02:40:09.897 | WARN | pool-2-thread-1 | Update thread 28 failed for host :172.17.10.170Failure:java.lang.NullPointerException
2010/07/22 02:41:09.903 | WARN | pool-2-thread-3 | VM Host 172.17.10.170 details refresh failed! Reason:null
2010/07/22 02:41:09.903 | WARN | pool-2-thread-3 | Update thread 28 failed for host :172.17.10.170Failure:java.lang.NullPointerException
2010/07/22 02:42:09.909 | WARN | pool-2-thread-5 | VM Host 172.17.10.170 details refresh failed! Reason:null
2010/07/22 02:42:09.909 | WARN | pool-2-thread-5 | Update thread 28 failed for host :172.17.10.170Failure:java.lang.NullPointerException
2010/07/22 02:43:01.714 | INFO | Timer-2 | Property update info of VirtualMachine vm-910: guest.disk (assign)
2010/07/22 02:43:09.915 | WARN | pool-2-thread-7 | VM Host 172.17.10.170 details refresh failed! Reason:null
2010/07/22 02:43:09.915 | WARN | pool-2-thread-7 | Update thread 28 failed for host :172.17.10.170Failure:java.lang.NullPointerException
2010/07/22 02:43:50.420 | ERROR | pool-4-thread-9 | VM Failed to refresh performance data! ; nested exception is:
2010/07/22 02:43:50.420 | java.net.SocketTimeoutException: Read timed out
2010/07/22 02:44:09.922 | WARN | pool-2-thread-8 | VM Host 172.17.10.170 details refresh failed! Reason:null
2010/07/22 02:44:09.922 | WARN | pool-2-thread-8 | Update thread 28 failed for host :172.17.10.170Failure:java.lang.NullPointerException
2010/07/22 02:45:09.928 | WARN | pool-2-thread-1 | VM Host 172.17.10.170 details refresh failed! Reason:null
2010/07/22 02:45:09.928 | WARN | pool-2-thread-1 | Update thread 28 failed for host :172.17.10.170Failure:java.lang.NullPointerException
2010/07/22 02:45:21.631 | INFO | Timer-2 | Property update info of VirtualMachine vm-910: guest.disk (assign)
2010/07/22 02:46:09.840 | WARN | pool-2-thread-3 | VM Host 172.17.10.170 details refresh failed! Reason:null
2010/07/22 02:46:09.840 | WARN | pool-2-thread-3 | Update thread 28 failed for host :172.17.10.170Failure:java.lang.NullPointerException
2010/07/22 02:46:15.141 | INFO | pool-3-thread-1 | Updating priority based on the vCenter check for 739D2442-3EA9-397A-0567-8FA5C6CFE89B
2010/07/22 02:46:15.141 | INFO | pool-3-thread-1 | UNSCH updateHostStatus vcGuid=739D2442-3EA9-397A-0567-8FA5C6CFE89B e= null connected = true
2010/07/22 02:47:01.651 | INFO | Timer-2 | Property update info of VirtualMachine vm-910: guest.disk (assign)
2010/07/22 02:47:09.852 | WARN | pool-2-thread-5 | VM Host 172.17.10.170 details refresh failed! Reason:null
2010/07/22 02:47:09.852 | WARN | pool-2-thread-5 | Update thread 28 failed for host :172.17.10.170Failure:java.lang.NullPointerException
2010/07/22 02:48:09.864 | WARN | pool-2-thread-7 | VM Host 172.17.10.170 details refresh failed! Reason:null
2010/07/22 02:48:09.864 | WARN | pool-2-thread-7 | Update thread 28 failed for host :172.17.10.170Failure:java.lang.NullPointerException
2010/07/22 02:49:09.876 | WARN | pool-2-thread-8 | VM Host 172.17.10.170 details refresh failed! Reason:null
2010/07/22 02:49:09.876 | WARN | pool-2-thread-8 | Update thread 28 failed for host :172.17.10.170Failure:java.lang.NullPointerException
2010/07/22 02:49:21.679 | INFO | Timer-2 | Property update info of VirtualMachine vm-910: guest.disk (assign)
2010/07/22 02:49:33.281 | ERROR | pool-4-thread-1 | VM Failed to refresh performance data! ; nested exception is:
2010/07/22 02:49:33.281 | java.net.SocketTimeoutException: Read timed out
2010/07/22 02:49:35.481 | ERROR | pool-4-thread-10 | VM Failed to refresh performance data! ; nested exception is:
Brian Hahne
Regular Advisor

Re: Help identifying VMM task occurring under SIM 6.0 w/ ICE

I have some similar errors, but I also have this one a lot:
2010/07/02 11:07:10.659 | ERROR | pool-3-thread-1 | Failed to get the latest event log! null

and this one:
167.171.85.64 | Request from SIM for hosts for VE 167.171.70.10 , Runtime Exception occured. The vCenter server specified is not managed by VMM


Not sure about the event log issue.
Does the vCenter server need to be registered with VMM? I put the ICE pack on it, I would have thought that was sufficient, given the ESX hosts were at the time registered as VM hosts. I had to unregister them to get the events to stop being created.


Here's a portion of the log:
2010/07/02 11:08:26.777 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Unable to find AgentInfo with UUID. Trying Serial number and ip...
2010/07/02 11:08:26.777 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Request from SIM for hosts for VE 167.171.80.70 , Runtime Exception occured. The vCenter server specified is not managed by VMM
2010/07/02 11:08:30.578 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Unable to find AgentInfo with UUID. Trying Serial number and ip...
2010/07/02 11:08:30.578 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Request from SIM for hosts for VE 167.171.80.96 , Runtime Exception occured. The vCenter server specified is not managed by VMM
2010/07/02 11:08:31.579 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Unable to find AgentInfo with UUID. Trying Serial number and ip...
2010/07/02 11:08:31.579 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Request from SIM for hosts for VE 167.171.80.94 , Runtime Exception occured. The vCenter server specified is not managed by VMM
2010/07/02 11:08:45.981 | INFO | pool-2-thread-6 | Connection state received:notResponding
2010/07/02 11:08:45.981 | INFO | pool-2-thread-6 | ESXi API connection lost...stopping all monitors
2010/07/02 11:08:46.082 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Unable to find AgentInfo with UUID. Trying Serial number and ip...
2010/07/02 11:08:46.082 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Request from SIM for hosts for VE 167.171.80.105 , Runtime Exception occured. The vCenter server specified is not managed by VMM
2010/07/02 11:08:48.582 | WARN | Timer-2 | An error occured during communication wih vCenter [Cause = null FaultString = (503)Service Unavailable]
2010/07/02 11:08:54.284 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Unable to find AgentInfo with UUID. Trying Serial number and ip...
2010/07/02 11:08:54.284 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Request from SIM for hosts for VE 167.171.80.120 , Runtime Exception occured. The vCenter server specified is not managed by VMM
2010/07/02 11:08:54.685 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Unable to find AgentInfo with UUID. Trying Serial number and ip...
2010/07/02 11:08:54.685 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Request from SIM for hosts for VE 167.171.80.118 , Runtime Exception occured. The vCenter server specified is not managed by VMM
2010/07/02 11:08:55.685 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Unable to find AgentInfo with UUID. Trying Serial number and ip...
2010/07/02 11:08:55.685 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Request from SIM for hosts for VE 167.171.80.121 , Runtime Exception occured. The vCenter server specified is not managed by VMM
2010/07/02 11:08:55.885 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Unable to find AgentInfo with UUID. Trying Serial number and ip...
2010/07/02 11:08:55.885 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Request from SIM for hosts for VE 167.171.80.125 , Runtime Exception occured. The vCenter server specified is not managed by VMM
2010/07/02 11:08:56.685 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Unable to find AgentInfo with UUID. Trying Serial number and ip...
2010/07/02 11:08:56.685 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Request from SIM for hosts for VE 167.171.80.137 , Runtime Exception occured. The vCenter server specified is not managed by VMM
2010/07/02 11:09:01.885 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Unable to find AgentInfo with UUID. Trying Serial number and ip...
2010/07/02 11:09:01.886 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Request from SIM for hosts for VE 167.171.80.143 , Runtime Exception occured. The vCenter server specified is not managed by VMM
2010/07/02 11:09:04.387 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Unable to find AgentInfo with UUID. Trying Serial number and ip...
2010/07/02 11:09:04.388 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Request from SIM for hosts for VE 167.171.80.149 , Runtime Exception occured. The vCenter server specified is not managed by VMM
2010/07/02 11:09:07.488 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Unable to find AgentInfo with UUID. Trying Serial number and ip...
2010/07/02 11:09:07.488 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Request from SIM for hosts for VE 167.171.80.151 , Runtime Exception occured. The vCenter server specified is not managed by VMM
2010/07/02 11:09:08.789 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Unable to find AgentInfo with UUID. Trying Serial number and ip...
2010/07/02 11:09:08.789 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Request from SIM for hosts for VE 167.171.80.150 , Runtime Exception occured. The vCenter server specified is not managed by VMM
2010/07/02 11:09:08.889 | ERROR | RMI TCP Connection(15919)-167.171.85.64 | Failed to re-scan HBAs on host vmesx2.palmettohealth.org Reason: ; nested exception is:
2010/07/02 11:09:08.889 | java.net.SocketTimeoutException: Read timed out
2010/07/02 11:09:09.389 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Unable to find AgentInfo with UUID. Trying Serial number and ip...
2010/07/02 11:09:09.389 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Request from SIM for hosts for VE 167.171.80.153 , Runtime Exception occured. The vCenter server specified is not managed by VMM
2010/07/02 11:09:09.789 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Unable to find AgentInfo with UUID. Trying Serial number and ip...
2010/07/02 11:09:09.789 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Request from SIM for hosts for VE 167.171.80.154 , Runtime Exception occured. The vCenter server specified is not managed by VMM
2010/07/02 11:09:10.389 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Unable to find AgentInfo with UUID. Trying Serial number and ip...
2010/07/02 11:09:10.389 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Request from SIM for hosts for VE 167.171.80.155 , Runtime Exception occured. The vCenter server specified is not managed by VMM
2010/07/02 11:09:10.689 | ERROR | pool-3-thread-1 | Failed to get the latest event log! null
2010/07/02 11:09:10.889 | ERROR | pool-3-thread-1 | Failed to get the latest event log! null
2010/07/02 11:09:15.790 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Unable to find AgentInfo with UUID. Trying Serial number and ip...
2010/07/02 11:09:15.791 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Request from SIM for hosts for VE 167.171.80.156 , Runtime Exception occured. The vCenter server specified is not managed by VMM
2010/07/02 11:09:17.591 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Unable to find AgentInfo with UUID. Trying Serial number and ip...
2010/07/02 11:09:17.592 | WARN | RMI TCP Connection(15971)-167.171.85.64 | Request from SIM for hosts for VE 167.171.80.157 , Runtime Exception occured. The vCenter server specified is not managed by VMM

2010/07/02 11:16:45.872 | WARN | pool-2-thread-1 | ****** AxisFault trying to connect to API (503)Service Unavailable ******
2010/07/02 11:16:45.872 | INFO | pool-2-thread-1 | Connection state received:notResponding
2010/07/02 11:16:45.872 | INFO | pool-2-thread-1 | ESXi API connection lost...stopping all monitors
2010/07/02 11:17:09.575 | WARN | Timer-2 | An error occured during communication wih vCenter [Cause = null FaultString = (503)Service Unavailable]
2010/07/02 11:17:10.476 | ERROR | pool-3-thread-1 | Failed to get the latest event log! null
2010/07/02 11:17:10.976 | ERROR | pool-3-thread-1 | Failed to get the latest event log! null
2010/07/02 11:17:22.177 | ERROR | pool-4-thread-14 | Host perfData : Failed to refresh performance data! (503)Service Unavailable
2010/07/02 11:17:45.981 | INFO | pool-2-thread-4 | Connection state received:notResponding
2010/07/02 11:17:45.981 | INFO | pool-2-thread-4 | ESXi API connection lost...stopping all monitors
2010/07/02 11:18:10.683 | ERROR | pool-3-thread-1 | Failed to get the latest event log! null
2010/07/02 11:18:10.884 | ERROR | pool-3-thread-1 | Failed to get the latest event log! null


Any thoughts?
We are running 64 bit.. I have the latest Java for 32 and 64 bit loaded..


ozlace
Occasional Advisor

Re: Help identifying VMM task occurring under SIM 6.0 w/ ICE

I think the the not managed by VMM is actually an after effect of the other errors. I have almost all the same messages.
ozlace
Occasional Advisor

Re: Help identifying VMM task occurring under SIM 6.0 w/ ICE

I had to disable VMM because virtual center was crashing so much HOWEVER another person responsible for the vcenter noticed that the vm's that were reloading were also on the same server and not the others in the cluster. Just wondering if you noticed that too?
Brian Hahne
Regular Advisor

Re: Help identifying VMM task occurring under SIM 6.0 w/ ICE

Well,
my configuration is that I have 3 VMWare ESX 3.5 servers. There is a vCenter server...

I registered the vCenter server by supplying the credentials in SIM, then registered the VMWare boxes as VM hosts, after applying the ICE packs to them.

After that, the vCenter server showed those restarts occurring on all 3 VM host boxes.

I am about to roll out 7 more VM hosts, DL 380 G6's.. I'd like to put them on ICE also but I want this fixed first.
ozlace
Occasional Advisor

Re: Help identifying VMM task occurring under SIM 6.0 w/ ICE

Mine is now fixed. We removed the errant VM from the cluster, rebuilt the ESXi image added it back into the cluster and then unregistered and re-registered the host in SIM. I now do not have the problem.

For you I would start with removing the host from the cluster and unregistering it and adding it back in and re-register it to see if it helps.