topic Re: IMC not able to SSH into Cisco ASA's in IMC

IMC not able to SSH into Cisco ASA's

pattap — Mon, 27 Apr 2020 13:38:26 GMT

I've been having problems with getting IMC working with Cisco ASA's for some time. One of the issues being SSH test not working correctly, I have created SSH template specifically for the ASA's and that is user, password + super password (I hope this is equivalent to enable in Cisco World).

I am able however to SSH to the same ASA's from the actual server itself - putty

From the ASA's I can't see IMC getting to the firewalls ok but it seems that IMC resets the connection

ASA's are 5516x's and 5506's

Re: IMC not able to SSH into Cisco ASA's

jmpk — Tue, 28 Apr 2020 04:32:35 GMT

Hello ,

Can you please let me know the IMC version also let me know IMC able to identify ASA model and other details properly ?

1. Refresh and Re-sync the ASA in IMC

2. Share the screenshot of the error if you seen any in IMC

3. Failure logs from IMC server

Re: IMC not able to SSH into Cisco ASA's

pattap — Tue, 28 Apr 2020 08:05:40 GMT

Hi jmpk

- version iMC PLAT 7.3 (E0703)

1. done as the first step,

to comment on tha above, snmp works, device can be pinged, credentials verified, ASA allows IMC to ssh to itself - can ssh to it from the server with putty as a POC, from ASA perspective I can see IMC trying to connect but TCP RST is sent from the server at some point.

3. no idea which logs contain failed ssh test attempts, can you point me to the right direction?

Re: IMC not able to SSH into Cisco ASA's

jguse — Tue, 28 Apr 2020 12:36:49 GMT

Hello,

It sounds like it could it be the 'super' (equivalent to enable password on Cisco) password that is having issues. I'd suggest testing without the super password configured on both iMC SSH Settings and on the device, and seeing if that works.

On the IMC side, you could set 'imcnetresdm' to DEBUG (via System Configuration > Log Configuration) and then run an SSH Parameters test. Check the iMC\server\conf\log\imcnetresdm<date>.txt file afterwards and see if there is any relevant error.

In the case of a successful SSH parameter test on a Comware switch, I could see the following lines when it completes:

2020-04-28 15:28:04.970 [DEBUG (0)] [THREAD(13160)] prompt is: <5900AF>
2020-04-28 15:28:04.970 [DEBUG (0)] [THREAD(13160)] [CDevParamVerifyTask::verifySshParam] Connect success.

Try searching through the file for verifySshParam and see what result it shows.

Another option for debugging it would be to run a configuration backup after setting 'imccfgbakdm' to DEBUG. The backup log tends to be more verbose and provides the full text output from the device login and backup process when it fails. The script output is shown after an entry like this:

<Timestamp> [INFO (0)] [THREAD(12752)] [imcscriptttol] log: =============================== Begin=============================

Re: IMC not able to SSH into Cisco ASA's

pattap — Wed, 29 Apr 2020 08:03:59 GMT

Justin

Using user/password combo doesn't work either.

I checked the logs after testing ssh from IMC UI, the below is the result, fails to open ssh_v1 script but ssh2 is used?

2020-04-29 08:31:45.963 [DEBUG (0)] [THREAD(17336)] [CDevParamVerifyTask::verifySshParam()] dev_id = 7137
2020-04-29 08:31:45.963 [DEBUG (0)] [THREAD(17336)] [CTemplateMgrTask::generateNewSSHKeyFile] SSH is password authentication.
2020-04-29 08:31:45.965 [ERROR (-1)] [THREAD(17336)] [CSSHExecutor::login()] File :D:\Program Files\iMC\server\bin\..\..\server\conf\ssh_v1_devices.cfg can't open.
2020-04-29 08:31:45.965 [DEBUG (0)] [THREAD(17336)] [CSSHService::connect] tryconnect(): ip = 10.x.x.x,user = adm, secretkeyfile = , keypharse = , timeout = 10, port = 22,sshversion = SSHV2
2020-04-29 08:31:45.981 [ERROR (1)] [THREAD(17336)] [CDevParamVerifyTask::verifySshParam] Connect unknown error

After trying to backup the same device, it looks that IMC managed to login this time

2020-04-29 08:38:04.414 [INFO (0)] [THREAD(10888)] [imcscriptttol] tcl log after timeout: =============================== Begin=============================
2020-04-29 08:38:04.414 [INFO (0)] [THREAD(10888)]
adm@10.x.x.x's password:
User secadm logged in to asa-01
Logins over the last 91 days: 435. Last login: 07:27:50 UTC Apr 29 2020 from 10.x.x.x
Failed logins since the last login: 0. Last failed login: 09:21:27 UTC Apr 28 2020 from 10.x.x.x
Type help or '?' for a list of available commands.

asa-01/pri/act>
2020-04-29 08:38:04.414 [INFO (0)] [THREAD(10888)] [imcscriptttol] tcl log after timeout: =============================== End===============================
2020-04-29 08:38:04.414 [ERROR (-1)] [THREAD(10888)] [CScriptProcessor::processCfgLog] File :D:\Program Files\iMC\server\bin\..\..\server/tmp/scripttool_17516_2996553893_output.cfg can not open.
2020-04-29 08:38:05.420 [INFO (0)] [THREAD(10888)] [CScriptProcessor::processLog] File :D:\Program Files\iMC\server\bin\..\..\server\conf\log\imcscripttool_ICC_10.x.x.x.2020-04-29.txt is deleted

Re: IMC not able to SSH into Cisco ASA's

jguse — Thu, 30 Apr 2020 10:22:19 GMT

Hello,

IMC will use SSHv2 to connect to devices by default - ssh_v1_devices.cfg file is an optional file you could create only if you really need the software to use SSHv1 to access your device instead when it doesn't support SSHv2. I don't think you need this here.

From the imccfgbakdm output is looks like the script ends at your device's prompt: asa-01/pri/act>

I wonder if this could be due to the / characters in the prompt and/or that it shows /pri/act (Active-Standby ASA configuration?) after what I assume is the device hostname - IMC looking for the hostname followed by > and not finding it.

Could you please try disabling the /pri/act (I think the command is 'no prompt hostname priority state'), synchronize the device in iMC and test again to see if that makes a difference?

Re: IMC not able to SSH into Cisco ASA's

pattap — Thu, 30 Apr 2020 20:07:46 GMT

I have disabled the prompt, command was prompt hostname, there are other options but to keep it basic that did the trick.

You were right asa-01 is the hostname

Still no luck.

Logs from imcnetresdm.

2020-04-30 19:44:55.146 [INFO (2)] [THREAD(7328)] [CDevMgr::getDevSoftInfoForCisco] call iGetNextVbValue fail when access dev[10.x.x.x]
2020-04-30 19:44:55.168 [INFO (2)] [THREAD(7328)] [CDevMgr::getDevBridgeMacAddress()] call iGetNextVbValue fail when access dev[10.x.x.x]
2020-04-30 19:44:55.209 [ERROR (-1)] [THREAD(7328)] [CSSHExecutor::login()] File :D:\Program Files\iMC\server\bin\..\..\server\conf\ssh_v1_devices.cfg can't open.
2020-04-30 19:44:55.224 [ERROR (1)] [THREAD(7328)] [CSSHExecutor::login] fail to call CSSHService::connect(), ssh connect close.
2020-04-30 19:44:55.224 [ERROR (0)] [THREAD(7328)] [CConfigProcessor::login()] Failed to login: 4793

logs from imccfgbakdm, after failed attempt of backup. As previously IMC get to user exec mode ok,

2020-04-30 20:05:57.755 [INFO (0)] [THREAD(18160)] [imcscriptttol] tcl log after timeout: =============================== Begin=============================
2020-04-30 20:05:57.755 [INFO (0)] [THREAD(18160)]
admin@10.x.x.x's password:
User admin logged in to asa-01
Logins over the last 91 days: 226. Last login: 18:55:44 UTC Apr 30 2020 from 10.i.i.i
Failed logins since the last login: 0. Last failed login: 18:44:03 UTC Apr 30 2020 from 10.x.x.x
Type help or '?' for a list of available commands.

asa-01>
2020-04-30 20:05:57.755 [INFO (0)] [THREAD(18160)] [imcscriptttol] tcl log after timeout: =============================== End===============================

........................ some output omitted

I can't see an indication of successful login to privileged exec(enable) mode, not sure what to expect, it looks that IMC tries to run some scripts to achieve it

020-04-30 20:30:10.843 [INFO (0)] [THREAD(12688)] [CCLIScriptProcessor::gotoMode()] Begin, ip: 10.x.x.x, current mode: exec, new mode: enable
2020-04-30 20:30:10.843 [INFO (0)] [THREAD(12688)] [CTclExecutor::exec_impl()] Begin to exec: D:/Program Files/iMC/server/bin/../../server/conf/adapters/ICC/Cisco/CiscoASA5500 X Gen/enter_enable.tcl
2020-04-30 20:30:11.876 [INFO (0)] [THREAD(12688)] [CTclExecutor::exec_impl()] Finished.

Re: IMC not able to SSH into Cisco ASA's

jguse — Mon, 04 May 2020 07:41:12 GMT

Hello,

I see you have a custom adapter created for your ASA model. Custom adapters are generally not supported as we don't have the adapter scripts ourselves to investigate, but will try to give you some pointers here and hope you can fix this. Are the scripts in the adapter you created a copy of the Cisco\CiscoASA adapter folder?

The enter_enable.tcl script in the output you shared is the one that runs after enter_exec.tcl to run the "enable" command to get to privileged mode on the device, and it should find a # prompt after running enable to continue. I don't see this script running enable command at all in the output you shared, so it's more likely the issue is on enter_exec.tcl not recognizing that the device is providing the user exec mode prompt.

Comparing CiscoASA and CiscoIOSGeneric adapter, the IOS adapter matches the prompts more specifically with the device name (hostname shown in iMC). Using this method can help iMC scripts recognize the device prompt properly.

Specifically the initialize.tcl script which declares the variables used by the other scripts contains the following for ASA:
set exec_prompt >
set enable_prompt #

While for CiscoIOSGeneric it is the following:

set exec_prompt >
set exec_prompt $DevName$exec_prompt
set enable_prompt #
set enable_prompt $DevName$enable_prompt

I'm thinking you could test adding the two missing lines from CiscoIOSGeneric to your custom adapter's initialize.tcl and see if that helps your scripts to proceed. Restart iMC and synchronize your device after the change.

Just note that when you use $DevName variable like above, it gets replaced with the hostname of the device as shown in iMC - so in your case this should be "asa-01" and not include a domain/FQDN, otherwise it will not be able to match the device prompt.

Hope that helps. If you still run into issues afterwards please share the new tcl log.

Re: IMC not able to SSH into Cisco ASA's

DarkWillow — Tue, 05 May 2020 06:51:54 GMT

@jguse wrote:
Hello, mcdvoice survey
I see you have a custom adapter created for your ASA model. Custom adapters are generally not supported as we don't have the adapter scripts ourselves to investigate, but will try to give you some pointers here and hope you can fix this. Are the scripts in the adapter you created a copy of the Cisco\CiscoASA adapter folder?
The enter_enable.tcl script in the output you shared is the one that runs after enter_exec.tcl to run the "enable" command to get to privileged mode on the device, and it should find a # prompt after running enable to continue. I don't see this script running enable command at all in the output you shared, so it's more likely the issue is on enter_exec.tcl not recognizing that the device is providing the user exec mode prompt.
Comparing CiscoASA and CiscoIOSGeneric adapter, the IOS adapter matches the prompts more specifically with the device name (hostname shown in iMC). Using this method can help iMC scripts recognize the device prompt properly.
Specifically the initialize.tcl script which declares the variables used by the other scripts contains the following for ASA:
set exec_prompt >
set enable_prompt #
While for CiscoIOSGeneric it is the following:
set exec_prompt >
set exec_prompt $DevName$exec_prompt
set enable_prompt #
set enable_prompt $DevName$enable_prompt
I'm thinking you could test adding the two missing lines from CiscoIOSGeneric to your custom adapter's initialize.tcl and see if that helps your scripts to proceed. Restart iMC and synchronize your device after the change.
Just note that when you use $DevName variable like above, it gets replaced with the hostname of the device as shown in iMC - so in your case this should be "asa-01" and not include a domain/FQDN, otherwise it will not be able to match the device prompt.
Hope that helps. If you still run into issues afterwards please share the new tcl log.

Happy to hear that! Your comment made mine.