topic Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13) in Operating System - OpenVMS

Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

BenAArmstrong — Mon, 29 Apr 2013 11:04:17 GMT

I am trying to start an Elasticsearch (lucene-based) server. With set proc/priv=all, it starts successfully. Without that, it fails with this stack dump (ip# obscured):

2013-04-25 14:19:31,467][WARN ][cluster.service          ] [Impulse] failed to reconnect to node [Impulse][efAKtSJzSt2BjEhFLG8_vg][inet[/1.1.1.1:9300]]
org.elasticsearch.transport.ConnectTransportException: [Impulse][inet[/1.1.1.1:9300]] General node connection failure
        at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:526)
        at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:483)
        at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:128)
        at org.elasticsearch.cluster.service.InternalClusterService$ReconnectToNodes.run(InternalClusterService.java:377)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: org.elasticsearch.common.netty.channel.ChannelException: java.net.SocketException: permission denied (errno:13)
        at org.elasticsearch.common.netty.channel.socket.DefaultSocketChannelConfig.setReceiveBufferSize(DefaultSocketChannelConfig.java:142)
        at org.elasticsearch.common.netty.channel.socket.DefaultSocketChannelConfig.setOption(DefaultSocketChannelConfig.java:50)
        at org.elasticsearch.common.netty.channel.socket.nio.DefaultNioSocketChannelConfig.setOption(DefaultNioSocketChannelConfig.java:71)
        at org.elasticsearch.common.netty.channel.DefaultChannelConfig.setOptions(DefaultChannelConfig.java:43)
        at org.elasticsearch.common.netty.channel.socket.nio.DefaultNioSocketChannelConfig.setOptions(DefaultNioSocketChannelConfig.java:54)
        at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:216)
        at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:183)
        at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:550)
        at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:505)
        ... 6 more
Caused by: java.net.SocketException: permission denied (errno:13)
        at sun.nio.ch.Net.setIntOption0(Native Method)
        at sun.nio.ch.Net.setIntOption(Net.java:157)
        at sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:431)
        at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
        at sun.nio.ch.SocketOptsImpl.receiveBufferSize(SocketOptsImpl.java:130)
        at sun.nio.ch.SocketOptsImpl$IP$TCP.receiveBufferSize(SocketOptsImpl.java:268)
        at sun.nio.ch.OptionAdaptor.setReceiveBufferSize(OptionAdaptor.java:122)
        at sun.nio.ch.SocketAdaptor.setReceiveBufferSize(SocketAdaptor.java:355)
        at org.elasticsearch.common.netty.channel.socket.DefaultSocketChannelConfig.setReceiveBufferSize(DefaultSocketChannelConfig.java:140)
        ... 14 more

I tried treating this as a java.policy issue, but even granting the broadest possible permissions does not help:

grant {
        permission java.security.AllPermission;
};

What am I missing? I compared this with a Linux system and the default java.policy seems the same on both systems, so I wonder if this isn't a Java permissions problem at all, but some OpenVMS permission issue, especially given that "set proc/priv=all" solves the problem (but in that case, I don't know which specific permission it wants, and the Java documentation doesn't say, or at least I haven't been able to find it yet).

The other odd thing is that we could swear this worked sometime in the past, perhaps at Java 1.6.0-2 or so. We're currently at 1.6.0-4 (and it failed in the same way with 1.6.0-3.p1).

Thanks,

Ben

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

Richard Brodie_1 — Mon, 29 Apr 2013 18:05:02 GMT

I would guess, since it's called from setReceiveBufferSize, that it's using SO_RCVBUF, for which the manual says:

'Sets the receive buffer size, in bytes. Requires an integer parameter and SYSPRV, BYPASS, or OPER privileges.'

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

BenAArmstrong — Tue, 30 Apr 2013 09:11:47 GMT

Ah, I see. Java does indeed set SO_RCVBUF:

http://docs.oracle.com/javase/6/docs/api/java/net/ServerSocket.html#setReceiveBufferSize(int)

Any suggestions as to how I could work around the issue without granting privileges? Here's an excerpt from elasticsearch:

ben@bgpc:~/elasticsearch/src/main/java/org/elasticsearch/discovery/zen/ping/multicast$ grep -rnC5  setReceiveBufferSize
MulticastZenPing.java-151-            // set the send interface
MulticastZenPing.java-152-            multicastInterface = networkService.resolvePublishHostAddress(address);
MulticastZenPing.java-153-            multicastSocket.setInterface(multicastInterface);
MulticastZenPing.java-154-            multicastSocket.joinGroup(InetAddress.getByName(group));
MulticastZenPing.java-155-
MulticastZenPing.java:156:            multicastSocket.setReceiveBufferSize(bufferSize);
MulticastZenPing.java-157-            multicastSocket.setSendBufferSize(bufferSize);
MulticastZenPing.java-158-            multicastSocket.setSoTimeout(60000);
MulticastZenPing.java-159-
MulticastZenPing.java-160-            this.multicastSocket = multicastSocket;

Oh, I found also that buffer size here defaults to 2048:

        this.address = componentSettings.get("address");
        this.port = componentSettings.getAsInt("port", 54328);
        this.group = componentSettings.get("group", "224.2.2.4");
        this.bufferSize = componentSettings.getAsInt("buffer_size", 2048);

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

abrsvc — Tue, 30 Apr 2013 12:00:12 GMT

I am not familiar with the particulars here, but if you can INSTALL the appropriate image with the privs, this should work. This will grant the privs to the image itself rather than the process allowing the function to work.

Dan

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

Richard Brodie_1 — Tue, 30 Apr 2013 13:10:31 GMT

That's problematic with an interpreted (at least for the purposes of discussing least privilege) language like Java. I can't think of anything better than starting the server process with OPER.

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

Steven Schweda — Tue, 30 Apr 2013 13:20:25 GMT

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

H.Becker — Tue, 30 Apr 2013 17:06:12 GMT

>>> what I have installed on my main Alpha system seems to be version 1.5, not 1.6, but ...

$ java -version

will tell you (you may need to run a java$*setup.com, where the wildcard needs to be replaced by the version number string, yeah, looks recursive, but a directory command for sys$startup: should get you out off the loop).

And yes,1.6 will very likely never make it to Alpha.

The JAVA$STARTUP.COM is ancient. It was from the time before they used the version number in the middle of the file name: Java 1.1.8. Also, the command procedure I found only installs /OPEN/SHARE/HEADER and checks whether the process has the privs to install that way.

Anyway, installing JAVA.EXE with privs seems worth a test. But then everybody running Java can make use of the privileg. So it seems better to run the server process with the required privileg.

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

BenAArmstrong — Tue, 30 Apr 2013 17:54:20 GMT

I would tend to agree that granting the process OPER is the most sensible solution. It just chafes me that a call to *reduce* the buffer size (looks like TCP/IP default for this system would be about 64K) is considered a privileged operation. I don't see the logic in it.

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

H.Becker — Tue, 30 Apr 2013 21:21:27 GMT

>>> It just chafes me that a call to *reduce* the buffer size (...) is considered a privileged operation.

It seems the call just sets the size and the code has no check whether the size is to be increased or reduced. The documentation doesn't distinguish, either:

Sets the receive buffer size, in bytes. Takes an integer parameter and requires a system UIC or SYSPRV, BYPASS, or OPER privilege.

I tried this little test on OpenVMS/Alpha V8.3 and TCPIP V5.6-9:

import java.net.*;

public class SetReceiveBufferSize {

public static void main(String[] args) throws Exception {

ServerSocket ss = new ServerSocket(9876);

System.out.println("RCVBUF option for this ServerSocket is: " + ss.getReceiveBufferSize());

ss.setReceiveBufferSize(234);

System.out.println("RCVBUF option for this ServerSocket is: " + ss.getReceiveBufferSize());

}

Without the required privilege it doesn't throw an exception, it just doesn't reduce the buffer size. (On Eisner, with Multinet and without privileges it reduces the size.) It looks like something changed in java$net_shr.exe, the native code, which calls setsockopt, or in the Java code which calls out into the native image.

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

BenAArmstrong — Wed, 01 May 2013 08:31:43 GMT

On which version of Java did you run that test?

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

BenAArmstrong — Wed, 01 May 2013 09:15:12 GMT

Test results on our development system:

VAV:BG> java "SetReceiveBufferSize"
RCVBUF option for this ServerSocket is: 61440
Exception in thread "main" java.net.SocketException: permission denied (errno:13)
        at java.net.PlainSocketImpl.socketSetOption(Native Method)
        at java.net.PlainSocketImpl.setOption(PlainSocketImpl.java:282)
        at java.net.ServerSocket.setReceiveBufferSize(ServerSocket.java:753)
        at SetReceiveBufferSize.main(SetReceiveBufferSize.java:6)
VAV:BG> set proc/priv=oper
VAV:BG> java "SetReceiveBufferSize"
RCVBUF option for this ServerSocket is: 61440
RCVBUF option for this ServerSocket is: 234
VAV:BG> java -version
java version "1.6.0"
Java(TM) SE Runtime Environment "1.6.0-4"
Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)
VAV:BG> tcpip show version

  HP TCP/IP Services for OpenVMS Industry Standard 64 Version V5.7 - ECO 4
  on an HP rx2600  (1.50GHz/6.0MB) running OpenVMS V8.4

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

BenAArmstrong — Wed, 01 May 2013 17:49:23 GMT

Thanks for that test case, btw. I will use it in a bug we will file on Java >= 6.0-3. I tried it on 6.0-2.p1 and that version does not have the problem. From my reading of the Java doc at Oracle, the request to change the buffer size doesn't need to be honored and should not throw an error. That will be at the heart of my argument this is a bug.

For the sake of completness of this thread, here's the output from 6.0-2.p1, first without privs:

$ java "SetReceiveBufferSize"
RCVBUF option for this ServerSocket is: 61440
RCVBUF option for this ServerSocket is: 61440

Then with SET PROC/PRIV=OPER (as before with 6.0-4):

$ java "SetReceiveBufferSize"
RCVBUF option for this ServerSocket is: 61440
RCVBUF option for this ServerSocket is: 234

We've filed the bug and have been assigned a case #. Will report back later when there's something to report.

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

H.Becker — Wed, 01 May 2013 20:34:14 GMT

>>> On which version of Java did you run that test?

$ java -version

java version "1.5.0"

Java(TM) 2 Runtime Environment, Standard Edition

Fast VM (build 1.5.0-8, build J2SDK.v.1.5.0:02/08/2012-17:12, native threads, jit_150)