Potential resolution for Morpheus App Node CPU frequently shooting up to 100% for minutes at a time

[cdtaylor]

Hello,

Recently I was investigating a Morpheus application node CPU utilisation issue on a customer site. The customer had used a Morpheus Support knowledge article (Morpheus Support) to determine that every few minutes the grails.rabbit.queue.* threads were intermittently utilising 100% CPU for 150 seconds or more on the virtual machines hosting the morpheus-ui service. This was causing sluggish behaviour and slow response times.

The root cause of this issue was the customer site had an incorrectly configured NTP time server. When the NTP server was polled it periodically placed the application node virtual machines on a different time (in the past). Consequently a Hibernate (https://hibernate.org/) implemented While loop executed for a very long time as it had to catch up with the previous tracked timestamp it was using for transaction management, and this caused the CPU usage spike and thread locks.

With the above impact in mind, and to avoid a myriad of other potential problems, please ensure that there are correctly configured and synced system clocks on the infrastructure used to host the Morpheus App stack. Please also ensure that the clocks are in sync between all machines used to host the Morpheus App stack.

Cheers!

Chris