Agent failed to communicate

[Niranpsk]

Hi Experts,

Agent installation is successful however it is not communicating to CMP. I’m seeing following error from the VM.

java.net.SocketException: Connection reset
at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(Unknown Source)
at java.base/sun.nio.ch.SocketChannelImpl.read(Unknown Source)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:255)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:356)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
[nioEventLoopGroup-2-2] ERROR com.morpheus.agent.websocket.StompClient - Failed to connect websocket: Connection reset
[nioEventLoopGroup-2-3] INFO com.morpheus.agent.websocket.WebSocketClientHandler - WebSocket Client disconnected!

Any advise?

Thanks.
BR

[rbmorph · Accepted solution]

Do you have enable websocket set on the netscaler for the connection? This is required for agent communication. Looks like you should be able to set this on the netscaler profile by checking a box for this connection profile.

[Niranpsk]

@rboyd Thanks for the comments. We requested end-user to have a look.

[Niranpsk]

@Ollie_Phillips Thanks for the advise and help us to narrow down the issue. we will work with Load balancer support to check further on this.

Thank you again.

BR

[Niranpsk]

@Ollie_Phillips Thanks for suggestion. This is 3 Node HA. We did not enable 443 port for Nodes. we allowed 443 only for Load Balancer.

error seems changed to following.
ERROR com.morpheus.agent.websocket.StompClient - Failed to connect websocket: connection timed out after 30000 ms: /x.x.x.x:443

We will test it once port allowed. can we add all 3 nodes to VM agent config file? Otherwise, if configured node goes down, will the agent stop communicate with CMP?

Any specific configuration need to be enabled on netscaler load balancer to resolve this issue?

Thank you very much for your cmts

BR.

[Ollie-Phillips]

Hi, my suggestion to target a node was for testing only. AFAIK can not use multiple nodes in the Agent rb file. The LB should be made to proxy the connection in a HA solution, fall back is single node for connection but you’ve spotted the risks with that already.

[Ollie-Phillips]

Meant to add, i’ve no specific knowledge re Netscaler LB config I can share.

[Niranpsk]

Just an update, web socket enabled in the LB and that solved the issues. Thanks helping to narrow down the issue

[Ollie-Phillips]

Is this a HA load balanced solution? Agent wants a websocket connection and some load balancers can’t proxy/support it.

You can test by changing cloud settings appliance URL to a node in the appliance and the updating the agent on the VM you expect to connect.

It will bypass the LB