Login Error, "Permission Denied" in HP-UX 11.31 Itanium through root and other user

[Vedang]

I have four production servers and all of them can be accessed using VPN from another location using PuttY. I recently made certain changes to run HP-UX Apache Web Server on the HP-UX 11.31 Itanium Server. For that purpose I had changed permissions of /usr directory for Root, Group and Others to rwx-rwx-rwx. After all the configuring the HP-UX Apache Web Server, I changed the permission to /usr directory to its default setting as rwx-r-x-r-x, that is revoked write permissions to Group and Others to the /usr directory. After this I was able to login into the system using the root and other user (say jr-admin) and execute commands. I logged out and terminated the session and reconnected using PuttY again to this server and it shows Permission Denied while logging in. This error is shown for both root user and jr-admin user. The passwords and user named entered by me are correct, I have double checked them. I have no access to the system using PuttY through both the users and am not able to figure out what went wrong. I checked various solutions to check the /etc/securetty file or sshd_config files but I cannot do that since I have no access to the system. I also cannot restart the system through command line interface and physically restarting the system will require certain permissions and long procedures since this being a production server. Any help will be appreciated. Thank you !

Edit:1 -- I saw the logs in PuttY and it shows the following error:

Server Refused Keyboard  Interactive Auhentication 

Password Sent

Authentication Denied

[Steven Schweda]

> [...] I had changed permissions of /usr directory [...]

   _That_ sounds like a bad idea,

> [...] using PuttY [...]

   PuTTY can do many things.

> Server Refused Keyboard Interactive Auhentication

   So, that would be SSH using PuTTY?  Have you tried Telnet?
RSH/REMSH?

> This error is shown for both root user and jr-admin user.

   How about other users?

[ranganath ramachandra]

have you tried console login through iLO?

[Patrick Wallek]

Why did you change permissions on /usr?  That was a terrible thing to do.  When installing and configuring software you should NOT have to change permissions anywhere, especially if you are working as root.

Did you do a 'chmod /usr' or a 'chmod -R /usr'?  If you used the recursive option (chmod -R) then that changed ALL files AND directories under /usr.  If you did that both times (chmod -R 777 /usr and chmod -R 755 /usr) then you likely just hosed your system.

There are a lot of files in various direcotries under /usr that need permissions much different than 755.  There are quite a few that have either the setuid, setgid or both set.  If those get disabled then things don't work.

Some ways I can think of to fix this, assuming you can ever get logged in:

1) Run the 'swverify' command on your server.  This will check all filesets and bundles on the server and make sure they are configured appropriately.  Part of those checks are file permission checks.

2) Look at another system and check each and every file and directory under /usr and fix the problems.

If you can't get logged in, and you have an Ignite/UX make_tape_recovery or make_net_recovery backup, then it might be time to recover the system from that backup.

[Vedang]

@Steven Schweda There are only two users, one root and other jr-admin, and yes I tried using telnet but it shows the same error. Also previously while connecting using Putty it just asked a prompt as below: 

login as:

<new line to enter password but no text displayed>

and now it is showing as -

login as : <host-username>

host-username@host-ip's password:

and then shows the error message of Permission denied.

@ranganath ramachandra I don't think Console login is possible as we only use Putty for accessing the system

@Patrick Wallek  -- Yes I recursively changed all the permissions of the /usr directory. Is there any other way to connect the system through another HP-UX system on the same network so that I can revert all the changes ? Also is there any command that can be used to reset the permissions to the /usr directory to default ?

[Patrick Wallek]

>>Yes I recursively changed all the permissions of the /usr directory

That was not a good idea.  In fact, a very bad idea.

As has been mentioned previously, your only other option to log into the server may be via the MP / console.  If that does not work, then you may be out of luck.

If you do get logged into the console, the 2 things I mentioned previously are likely your best bets.

[Vedang]

@Patrick Wallek Okay ! I shall see trying your solutions and other workarounds (if any) as well. Thank you. 

[Torsten.]

IMHO its time to restore your ignite backup ...

[Steven Schweda]

> IMHO its time to restore your ignite backup ...

   Optimist.

    One other possibility would be to move the disk to a comparable,
working system, mount the damaged "/usr" file system somewhere, and
start comparing file permissions between the mess you've made and an
undamaged "/usr" tree.

   Presumably, "find -exec" with a suitable compare+fix script could go
through the whole damaged "/usr" tree, compare the permissions of every
file it finds with the corresponding file in the undamaged "/usr" tree
(if it exists there, of course), and, if a difference is found, use
chmod to repair the damage.  Sounds (to me) like a good exercise for the
reader.

[Dennis Handly]

>Run the swverify command on your server.

There is a "fix" option (-F) that should correct things.

>Presumably, "find -exec" with a suitable compare+fix script could go through the whole damaged "/usr" tree, compare the permissions of every

I think I posted two such scripts years ago.

[Vedang]

@Patrick Wallek I have two other HPUX Systems in the Production Environment and they are connected in the same network that is presumably they are a part of a vertical or horizontal server rack and none of them can be accessed via a console or a traditional monitor. But since they are on the same network I think they still can be used to connect and access each other. Assuming this to be an option, can you guide me exactly how should I access the frozen system with the damaged /usr from other HPUX system via ssh or some other option. If ever I could just login into the system even through a command line then I could think of changing each and every permission to its default by comparing it with those from a finely running system.

[Steven Schweda]

> [...] part of a vertical or horizontal server rack [...]

   The length of the wire is not the problem.

> [...] can you guide me exactly how should I access the frozen system
> [...]

> So, that would be SSH using PuTTY?  Have you tried Telnet?
> RSH/REMSH?

   Still wondering.

   You could try using NFS to access the damaged disk from a working
system, but I'd expect this to fail, too.

> [...] If ever I could just login into the system even through a
> command line then [...]

   True, but you may have made this impossible, hence my suggestion of
transplanting the damaged disk to a working system.

[Vedang]

@Steven Schweda Have tried rlogin and telnet but in both the cases the system is giving a "Permission Denied" error.