topic Re: Data Access Gateway Direct memory in HPE Ezmeral Software platform

Data Access Gateway Direct memory

Harshkohli — Tue, 22 Jun 2021 19:07:29 GMT

I am not sure about where to fix or modify the direct memory for Data Access gateway.

Assuming below errors are there in logs which I am not sure.

ERROR [MapR GRPC Service Executor -2815] GrpcThreadPoolExecutor - io.grpc.internal.SerializingExecutor.run() leaked an exception. io.grpc.netty.shaded.io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 3003121951, max: 3014656000)

Do we need to update the heap memory here which is under warden.data-access-gateway.conf in data-access-gateway conf directory?

Re: Data Access Gateway Direct memory

tdunning — Tue, 22 Jun 2021 20:47:11 GMT

This typically happens when a streaming argument in grpc has too much data passed too quickly.

Others more expert in the ways of the gateway need to comment.

Re: Data Access Gateway Direct memory

Harshkohli — Fri, 25 Jun 2021 02:10:55 GMT

Thanks Ted. I finally got to the solution. OJAI query generally gets allocated some direct memory via Jetty/library and in case of extensive query this memory saturates which goes beyond the heap memory of the Data Access Gateway process.

Two ways to handle is either increase the heap of the DAG to accumulate more results or limit the number of results via grpc.service.ojai.query.result-limit parameter under properties.cfg.

ANy one of these should take care of the same. For now there is no way like Drill where we can set direct memoery for DAG.

Let me know if someone thinks otherwise or has some more pointers.