X11 connection rejected because of wrong authentication

[Ashour]

Hi all,

I'm hitting this issue for the last 5 months, and there'is no known workaround for it.

When running Oracle OUI installer (runInstaller) I always get the following error message

X11 connection rejected because of wrong authentication.
X connection to localhost:10.0 broken (explicit kill or server shutdown).

Even when running the Oracle OUI is silent mode (no GUI); it exits with the same message.

However; xclock, firefox, swinstall, and other GUI tools are running fine with no issues, it's just the Oracle products such as OUI, netca, dbca, dbua, etc, are always showing the above message.

I also get this message when using putty or mobaXterm, I tried from different Windows clients (disabled anti virus and firewall), even got the error when connecting from a linux client.

Oracle support were totally useless, and always directed me that it's an OS issue.

I've cheked every other thread and still couldn't reach any fix.

HPUX 11.31 on Itanium 2 (integrity Blade).

Your help is much appreciated.

[Ashour]

-rw-------   1 oracle     oinstall       189 Nov  2 09:56 /home/oracle/.Xauthority

- There's enough disk space 

- sshd_config has (X11Forwarding yes)

- ssh_config has (Host *)  and (ForwardX11 yes)

- Tried ssh -X -Y oracle@server 

- export DISPLAY=localhost:10.0

- export XAUTHORITY=/home/oracle/.Xauthority

- Couldn't find any "rc" or "sshrc" files.

- Tried xauth add .... ... ...  (even recreated the ~/.Xauthority file)

- Tried xhost + 

[Matti_Kurkela]

After establishing a SSH connection to the server, are you using "su", "sudo" or similar commands to switch to another user account?

When a X11-forwarding SSH connection is established, two things are set up for the user the connection was logged in as:

- the DISPLAY variable is set to localhost:X.0, where X is 10 for the first X11-forwarding SSH session, and increasing for subsequent simultaneous X11-forwarding sessions.

- the ~/.Xauthority file is created if it does not exist, and is populated with a session cookie for the forwarded session.

If you switch users after connecting, you must ensure that the user account you're switching to has both of these things or their exact equivalents.

For example, if you're connecting as "ashour" and then su'ing to root, you must make sure the assigned DISPLAY value is copied exactly to the root's environment. Then, since root can read any file, you can simply export the XAUTHORITY variable to point to /home/ashour/.Xauthority. At that point, any root-owned process will be able to output its windows to your screen over the X11-forwarding SSH session.

But if you are switching to some other account, setting the DISPLAY and XAUTHORITY variables will not be enough: a non-root user won't be able to read any other user's .Xauthority file, since the file will be accessible by its owner only. Instead, before switching users, you must copy your .Xauthority file to some location that is accessible to the target user, and set the permissions of the copy so that the target user (and ideally nobody else) can read the file. Then you switch users, export the DISPLAY variable as usual, and export the XAUTHORITY variable to point to the copy of your .Xauthority file you've just made.

Make sure that the .profile or similar login scripts for the oracle user don't have any hard-coded DISPLAY variable settings: if the DISPLAY is set to any value other than the one generated by sshd for the X11-forwarding session, it won't work.

[Ashour]

Hi Matti,

I tried both, to login directly as oracle user or login as root and then "su - oracle", but usually I login diretly as oracle user.

I made sure that root has all the cookies in Oracle's xauth list, but still giving the same error.

Oracle user doesn't set DISPLAY variable in its login profile.

I also tried setting DISPLAY to different values (10,11,12,13,14,15, etc) and still not working

My point here is that xclock runs ok with Oracle user; i.e. the xauth and display settings are valid, it just doesn't work with Oracle products

[Matti_Kurkela]

Then it really seems like it's an Oracle problem, but proving it may be tricky, as you already have experienced.

You might have to dump the network traffic between the Oracle X11 client(s) and the ssh daemon: when DISPLAY is set to e.g. localhost:10.0, it means the X11 clients are supposed to connect to localhost, TCP port 6010 (i.e. the TCP port number is the display number + 6000). And you might want to do the same at your local workstation, between the SSH client and port 6000 of the local X11 server, at the same time. Comparing the simultaneous dumps from both those points might shed some light on the issue.

I think there might be one other possibility, though: the Oracle products and the JVM running the Oracle Universal Installer are IPv6-aware. They might be using the IPv6 API for that localhost connection, and if you have taken steps to block IPv6 on your system, that might cause the problem. Try using 127.0.0.1 in place of "localhost" in the DISPLAY variable; that should force IPv4 to be used.

[Ashour · Accepted solution]

Thanks Matti,

IPv6 isn't configured.

Anyway; Oracle (internal) support came back with a fix that worked 

set X11UseLocalhost variable to 'no' in the file /etc/opt/ssh/sshd_config 

Probably SSHD daemon needs to be restarted.

All good now.

[Matti_Kurkela]

From the sshd_config manpage:

X11UseLocalhost
Specifies whether sshd(8) should bind the X11 forwarding server
to the loopback address or to the wildcard address. By default,
sshd binds the forwarding server to the loopback address and sets
the hostname part of the DISPLAY environment variable to
localhost. This prevents remote hosts from connecting to the
proxy display. However, some older X11 clients may not function
with this configuration. X11UseLocalhost may be set to no to
specify that the forwarding server should be bound to the wild‐
card address. The argument must be yes or no. The default is
yes.

Looks like the Oracle installer and other those other Oracle binaries still qualify as "older X11 clients", then.

Good to know, thanks for sharing the solution.