Need Help with Data Access Gateway Setup and random PAM authentication failure issue

[PrativaNayak]

Hi

We have done a 3 node Ezmeral cluster setup. Facing some random issues with Data Access Gateway. 

Out of 3 nodes, by default DAG was installed in 2 nodes and API server in 2 nodes. We explicitly installed DAG on the 3rd node. No change to API server.

While trying to access JSON DB via API, randomly on original 2 nodes sometime output comes and sometime 401 Unauthorized Error is thrown. In node-3 (where DAG installation had happened at the end), always Unauthorized error is thrown. API server log was showing PAM Authentication error.

Now we are not able to login to MCS with mapr user and in the log similar PAM Authentication error is thrown.

We haven't done any explicit PAM authentication setting, but looks like this perhaps is the root cause of the issue.

Ezmeral 7.8 version is installed.

Would appreciate any help/direction in this regard.

Thanks...

[okalinin]

Hi,

We typically see such issues being non-EDF related but failling into the following two categories:
- PAM/LDAP configuration issues
- OS PAM issues
- Password issues

Below is general guideline/sample on troubleshooting PAM failure issues:

(1) Install 'pamtester' package, e.g. for RHEL it is available in epel repository:

yum install pamtester

(2) Once pamtester is installed, test PAM authentication with the following commands for relevant PAM modules on non-working node:

pamtester sudo <username> authenticate
pamtester sshd <username> authenticate

(3) Compare files in /etc/pam.d/ in working vs non-working nodes, e.g.

md5sum /etc/pam.d/*

Run the command on both working / non-working nodes and check if there are any differences. Check if any files related to PAM and authentication (e.g. LDAP configuration) have been modified recently. Usually such comparative analysis involving pamtester and configuration review reveals the root cause of the issue.

[PrativaNayak]

Thanks. For the node in which authorization not working, I think it is because of non presence of API server in that node. Initial PAM configurations looks same in all the nodes. Howeve inconsistent behavior of other 2 nodes where sometime authorization is going through and sometime failing is more worrysome.

[okalinin]

You mentioned:

<quote>
Now we are not able to login to MCS with mapr user and in the log similar PAM Authentication error is thrown
</quote>

From that it sounds that the authentication issue is present on MCS (API server) nodes. In any case, my earlier comment applies to any node affected by PAM authentication failures, regardless of installed services. Would you be able to use the recommended approach with pamtester in your environment?

[PrativaNayak]

@okalinin Installed PAMTESTER in all 3 nodes and below is the successful authentication response from all 3 nodes.

Before executing this we had done some changes as instructed in the below page and restarted ezmeral. Post that MCS authentication is happening, in node-1 &2 behaviour continues to be random and node-3 not connecting. 

https://docs.ezmeral.hpe.com/datafabric-customer-managed/78/SecurityGuide/MapRTicketsAndPAM.html?hl=pam%2Cuser

And post above change, for node 1 & 2 PAM Authentication Error is not observed in api-server.log but randomly seen in data-access-gateway.log. I am suspecting it is perhaps some internal process of ezmeral which is blocking connection not really a OS level PAM error. Not sure..

[okalinin]

Could you please pick one node where the issue occurs and share exact log messages for both apiserver and DAG (data access gateway) authentication errors?
Also, you've ran pamtester tests for mapr user. Do authentication errors occur for mapr user or some other user?

[support_s]

Hello,

Let us know if you were able to resolve the issue.

If you are satisfied with the answers then kindly click the "Accept As Solution" button for the most helpful response so that it is beneficial to all community members.

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".