Morpheus with a Double Proxy

[#1]

At it’s heart, Morpheus is a tomcat web application. It runs nginx as a proxy to send incoming web traffic to the tomcat server bound to localhost. Normally, this functionality doesn’t matter and is very transparent. At least until nginx 1.23.0. As of Morpheus 6.2.2-1, the nginx version is 1.25.1.

I just wanted to get the benefits of MySQL 8 and other services with 6.2.2, so Morpheus is all I upgraded on my admittedly old CentOS 7 VM. Nothing else in my environment had changed. After the upgrade, the Morpheus logs were spammed with lots of errors that look like this:

ERROR c.m.StompHandshakeHandler - "Handshake failed due to invalid Upgrade header: websocket, websocket"

This was in my lab, and I started with an older Morpheus version so I figured it was an odd error that might only apply to special environments. I was half-right. I operate a lot of docker containers across a few hosts and I use a reverse proxy to make things easier. For those curious, it’s SWAG by Linuxserver.io. They don’t have a default proxy configuration for Morpheus so I had to build one myself. Here’s the relevant bit:

location / {
    include /config/nginx/proxy.conf;
    resolver 127.0.0.11 valid=30s;
    set $upstream_morpheus 192.168.x.x;
    proxy_pass https://$upstream_morpheus:443;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
}

Basically, this says to upgrade websocket connections using the header websocket. Looking good so far. The error above is stating that the header being received is websocket, websocket So what we have is a duplicate header. I’m not 100% sure what the behaviour was prior to 1.25.1, but a change made by nginx as of 1.23.0 changed the way they deal with duplicate headers. They are now combining the headers values with a comma as per RFC 2616 Section 4.2. IMHO, certain headers should not have this behaviour, like Upgrade.

I resolved the issue by commenting out the last two lines of the location config on my first proxy and everything went back to normal. Websockets from the agents are coming in normally and being upgraded to websockets at Morpheus’ embedded nginx proxy itself, rather than on by both of them.