Max allowable local CLI processes of 24 exceeded

[L1nklight]

So ever since I installed the SSMC 2.0 product, I've been seeing this alert pop up quite a bit. It's popped up enough that 3PAR global support has contacted me several times to attempt to remediate the issue. Their troubleshooting up to this point has been to log in VIA HP Rooms and kill a bunch of open existing CLI connections using root access from the SPOCC. 

Is anyone else getting this warning? Here is what my session info looks like:

[Dennis Handly]

For your StoreServ model, the max number of connections is 96.  With the number of SSH (local) sessions 24.

Currently you only have the one SSH session.

>Their troubleshooting up to this point has been to log in VIA HP Rooms and kill a bunch

So what did they say?

>Is anyone else getting this warning?

You can get this warning if you're going crazy with the number of sessions.  It would be good to know what type and how many sessions when this happens.  Do you have many SSMC connections?

You can also hit this limit if each session is hung waiting for sysmgr, if real busy.

Since you are hitting a local session limit, if you have a remote CLI client installed, you may be able to do showuserconn to see the source of all those SSH sessions.  Also the event logs will show login/logoff so you can track this historically.

[L1nklight]

They basically said they don't know where the extra processes are comming from and they really don't have any idea of the reprocussions. That might not be for everyone at HP, but the the two individuals I spoke to, that was their response both times.

What I have noticed is that there seems to be some correlation betweeen performing certain activities in Veeam and getting that message. Because of the way I have setup Veeam, when the wsapi is disabled, Veeam fails to work properly. I have also voiced a concern with Veeam but their argument is that they are using the API as advertised by HP. They did do some digging and that was their verdict. 

My total number of connections to Inserv varies. The SSMC at any time can have up to 3 connections. System Reporter also seems to maintain one open connection. The other concern I have is the assumption that a "connection" correlates directly with a cli process. The error says too many "cli processes" but does that mean there are too many CLI logins? HP was unable to answer that question. 

For the most part I only get the alert on rare occasions. It's not very consistent and like I mentioned above certain operations in Veeam seem to have the ability to cause the error message to occur, but not on a frequent or reliable enough basis to troubleshoot. Sometimes simply adding a VM can do it. I have also seen the error occasionally when running a backup. Again though, it's not every time which makes it difficult to troubleshoot accurately. 

[Dennis Handly]

>They basically said they don't know where the extra processes are coming from and they really don't have any idea of the repercussions.

Hmm, the origin of the tpdtcl process should be obvious.  That's what showuserconn shows.

Also it should be in the event log.

Unless that's the problem, there are processes but no connections yet?

I think I have a bunch like that due to SSMC.

>there seems to be some correlation between performing certain activities in Veeam and getting that message.

Does anything clear up by itself?

>The other concern I have is the assumption that a "connection" correlates directly with a cli process.

No, the message should really say "tpdtcl server process".  These serve CLI, IMC and SSMC clients.

 

>For the most part I only get the alert on rare occasions.

If they don't go away, then there is a problem with hung connections.  But it it stops, then there are just too many.

[L1nklight]

I haven't seen the issue in a week or so. I am almost certain that when I do a showuserconn there are rarely situations when there are over 4 user connections listed. Next time I see the issue I will respond immediately with a screen cap and get your opinion of it. 

> Unless that's the problem, there are processes but no connections yet?

Sort of... there are either no further connections, but processes still going on, or what you suggested that there are processes but no connections. 

> Does anything clear up by itself?

I am not getting the alert/alarm right now. It infrequently comes up. If I do a showuserconn currently, I see the following:

• 3 Connections from user 3paradm from the SSMC server. Client: remote. ClientName: ssmc
• 1 Connection from user 3paradm from the SSMC server. Client:remote. ClientName:CLI
• 1 Connection from user 3paradm from the SSMC server. Client:remote. ClientName:--
• 1 Connection from me from my workstations. Client:local. ClientName:SSH

This is pretty much all I see ever from the showuserconn even when the alarm/alert triggers.

[Dennis Handly]

>I am almost certain that when I do a showuserconn there are rarely situations when there are over 4 user connections

I've found three arrays where SSMC is creating "dark" connections that aren't listed and are hung.  They don't show up with showuserconn but slowly take up connections.

You might be able to free them by stopping the SSMC Services on a Windows box.

>1 Connection from user 3paradm from the SSMC server. Client:remote. ClientName:--

I've seen this.  This should never happen, a client should advise their name.

>all I see ever from the showuserconn even when the alarm/alert triggers.

Yes.  A few you see and the rest are "dark".