Operating System - OpenVMS
1748142 Members
3535 Online
108758 Solutions
New Discussion

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

 
BenAArmstrong
Frequent Advisor

Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

I am trying to start an Elasticsearch (lucene-based) server. With set proc/priv=all, it starts successfully. Without that, it fails with this stack dump (ip# obscured):

 

 

2013-04-25 14:19:31,467][WARN ][cluster.service          ] [Impulse] failed to reconnect to node [Impulse][efAKtSJzSt2BjEhFLG8_vg][inet[/1.1.1.1:9300]]
org.elasticsearch.transport.ConnectTransportException: [Impulse][inet[/1.1.1.1:9300]] General node connection failure
        at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:526)
        at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:483)
        at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:128)
        at org.elasticsearch.cluster.service.InternalClusterService$ReconnectToNodes.run(InternalClusterService.java:377)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: org.elasticsearch.common.netty.channel.ChannelException: java.net.SocketException: permission denied (errno:13)
        at org.elasticsearch.common.netty.channel.socket.DefaultSocketChannelConfig.setReceiveBufferSize(DefaultSocketChannelConfig.java:142)
        at org.elasticsearch.common.netty.channel.socket.DefaultSocketChannelConfig.setOption(DefaultSocketChannelConfig.java:50)
        at org.elasticsearch.common.netty.channel.socket.nio.DefaultNioSocketChannelConfig.setOption(DefaultNioSocketChannelConfig.java:71)
        at org.elasticsearch.common.netty.channel.DefaultChannelConfig.setOptions(DefaultChannelConfig.java:43)
        at org.elasticsearch.common.netty.channel.socket.nio.DefaultNioSocketChannelConfig.setOptions(DefaultNioSocketChannelConfig.java:54)
        at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:216)
        at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:183)
        at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:550)
        at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:505)
        ... 6 more
Caused by: java.net.SocketException: permission denied (errno:13)
        at sun.nio.ch.Net.setIntOption0(Native Method)
        at sun.nio.ch.Net.setIntOption(Net.java:157)
        at sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:431)
        at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
        at sun.nio.ch.SocketOptsImpl.receiveBufferSize(SocketOptsImpl.java:130)
        at sun.nio.ch.SocketOptsImpl$IP$TCP.receiveBufferSize(SocketOptsImpl.java:268)
        at sun.nio.ch.OptionAdaptor.setReceiveBufferSize(OptionAdaptor.java:122)
        at sun.nio.ch.SocketAdaptor.setReceiveBufferSize(SocketAdaptor.java:355)
        at org.elasticsearch.common.netty.channel.socket.DefaultSocketChannelConfig.setReceiveBufferSize(DefaultSocketChannelConfig.java:140)
        ... 14 more

 I tried treating this as a java.policy issue, but even granting the broadest possible permissions does not help:

 

grant {
        permission java.security.AllPermission;
};

What am I missing? I compared this with a Linux system and the default java.policy seems the same on both systems, so I wonder if this isn't a Java permissions problem at all, but some OpenVMS permission issue, especially given that "set proc/priv=all" solves the problem (but in that case, I don't know which specific permission it wants, and the Java documentation doesn't say, or at least I haven't been able to find it yet).

 

The other odd thing is that we could swear this worked sometime in the past, perhaps at Java 1.6.0-2 or so. We're currently at 1.6.0-4 (and it failed in the same way with 1.6.0-3.p1).

 

Thanks,

Ben

 

12 REPLIES 12
Richard Brodie_1
Honored Contributor

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

I would guess, since it's called from setReceiveBufferSize, that it's using SO_RCVBUF, for which the manual says:

'Sets the receive buffer size, in bytes. Requires an integer parameter and SYSPRV, BYPASS, or OPER privileges.'

 

 

BenAArmstrong
Frequent Advisor

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

Ah, I see. Java does indeed set SO_RCVBUF:

 

http://docs.oracle.com/javase/6/docs/api/java/net/ServerSocket.html#setReceiveBufferSize(int)

 

Any suggestions as to how I could work around the issue without granting privileges? Here's an excerpt from elasticsearch:

 

ben@bgpc:~/elasticsearch/src/main/java/org/elasticsearch/discovery/zen/ping/multicast$ grep -rnC5  setReceiveBufferSize
MulticastZenPing.java-151-            // set the send interface
MulticastZenPing.java-152-            multicastInterface = networkService.resolvePublishHostAddress(address);
MulticastZenPing.java-153-            multicastSocket.setInterface(multicastInterface);
MulticastZenPing.java-154-            multicastSocket.joinGroup(InetAddress.getByName(group));
MulticastZenPing.java-155-
MulticastZenPing.java:156:            multicastSocket.setReceiveBufferSize(bufferSize);
MulticastZenPing.java-157-            multicastSocket.setSendBufferSize(bufferSize);
MulticastZenPing.java-158-            multicastSocket.setSoTimeout(60000);
MulticastZenPing.java-159-
MulticastZenPing.java-160-            this.multicastSocket = multicastSocket;

 

Oh, I found also that buffer size here defaults to 2048:

 

        this.address = componentSettings.get("address");
        this.port = componentSettings.getAsInt("port", 54328);
        this.group = componentSettings.get("group", "224.2.2.4");
        this.bufferSize = componentSettings.getAsInt("buffer_size", 2048);

 

abrsvc
Respected Contributor

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

I am not familiar with the particulars here, but if you can INSTALL the appropriate image with the privs, this should work. This will grant the privs to the image itself rather than the process allowing the function to work.

Dan
Richard Brodie_1
Honored Contributor

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

That's problematic with an interpreted (at least for the purposes of discussing least privilege) language like Java. I can't think of anything better than starting the server process with OPER.

Steven Schweda
Honored Contributor

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

 
H.Becker
Honored Contributor

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

>>> what I have installed on my main Alpha system seems to be version 1.5, not 1.6, but ...
 
$ java -version
will tell you (you may need to run a java$*setup.com, where the wildcard needs to be replaced by the version number string, yeah, looks recursive, but a directory command for sys$startup: should get you out off the loop).
 
And yes,1.6 will very likely never make it to Alpha.
 
The JAVA$STARTUP.COM is ancient. It was from the time before they used the version number in the middle of the file name: Java 1.1.8. Also, the command procedure I found only installs /OPEN/SHARE/HEADER and checks whether the process has the privs to install that way. 
 
Anyway, installing JAVA.EXE with privs seems worth a test. But then everybody running Java can make use of the privileg. So it seems better to run the server process with the required privileg.
 
BenAArmstrong
Frequent Advisor

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

I would tend to agree that granting the process OPER is the most sensible solution. It just chafes me that a call to *reduce* the buffer size (looks like TCP/IP default for this system would be about 64K) is considered a privileged operation. I don't see the logic in it.

H.Becker
Honored Contributor

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

>>> It just chafes me that a call to *reduce* the buffer size (...) is considered a privileged operation.
 

It seems the call just sets the size and the code has no check whether the size is to be increased or reduced. The documentation doesn't distinguish, either:

Sets the receive buffer size, in bytes. Takes an integer parameter and requires a system UIC or SYSPRV, BYPASS, or OPER privilege.
 
I tried this little test on OpenVMS/Alpha V8.3 and TCPIP V5.6-9:
 
import java.net.*;
public class SetReceiveBufferSize {
    public static void main(String[] args) throws Exception {
        ServerSocket ss = new ServerSocket(9876);
        System.out.println("RCVBUF option for this ServerSocket is: " + ss.getReceiveBufferSize());
        ss.setReceiveBufferSize(234);
        System.out.println("RCVBUF option for this ServerSocket is: " + ss.getReceiveBufferSize());
    }
}
 
Without the required privilege it doesn't throw an exception, it just doesn't reduce the buffer size. (On Eisner, with Multinet and without privileges it reduces the size.) It looks like something changed in java$net_shr.exe, the native code, which calls setsockopt, or in the Java code which calls out into the native image.
BenAArmstrong
Frequent Advisor

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

On which version of Java did you run that test?