Swicthes stopped updating PCM2.0+

[Jonathan Axford]

Hi all,

I have been running PCM2.0+ for a while now and it has been working fine.
All of a sudden, Some of the switches have stopped responding and i can no longer update the software on them from the management server.

There are 3 5308 switches, 1 4108 and 2 9308 switches that are experiencing this problem.

You can scan for configurations without any problems, As soon as you start the software update wizard it times out on the first part.

Anyone come across this before or have any ideas???

Cheers

Jonathan

[Preston Gallwas]

Jonathan, I'd be interested to know what the CPU usage and processes are on your machine (and specs) when it times out.

I have a long standing case (since Sept-Oct) with HP open about extremely high CPU Usage from Mysqld-max-nt.exe in PCM 2.0 (and 2.1)+ when performing some operations. Unfortunately, no resolution has come yet.

[Devon Dawson]

Jonathan,

While I dig into this can you let me know if the switches in question successfully pass the "Test Communication Parameters" tests available via the right-click menu items and/or the toolbar for a specific device.

I'm wondering if perhaps the devices have too many telnet sessions left open or perhaps the communication parameters have changed (within PCM, the switch or both).

If you could run this test and let us know the results that would be very helpfull.

[Jonathan Axford]

Hi,

The process usagee is very low on the box, It stays at around 4% and occasionally spikes to 30% so i don't think that can be the problem.

I have about 48 switches currently being managed by the server, All of these will allow you to schedule a software update.

I have checked and double checked the Communication parameters in PCM. The comms parameter test is successful on all accounts.

I have upgraded to version 2.1 to see if this has any effect.

I hadn't thought of open telnet sessions etc. I will check this now...

[Les Ligetfalvy]

When I first started playing with Procurve switches and PCM, I did ot realize that improperly exiting a telnet session would not release it at the switch and prevent PCM from updating.

I have since learned to always do a proper logoff from a telnet session and to not linger in a session too long.

[Sergej Gurenko]

Customers reports that there is a problems with Q-trunks after upgrade. Recent HP firmwares support CDP in passive mode only. In some configurations half automatically discovered LACP/PaGP/1dotQ stops working after CDP disable.

[Devon Dawson · Accepted solution]

Jonathan,

The engineers responsible for the software update tool in PCM have come back to me with a few clarifying questions:

1)When you say that the devices "stop responding" I assume you mean the are not responding to the first step of the software update wizard. Is that accurate and if not in what other ways do the switches stop responding?

2)Are all of your devices of the types you mentioned (5308,4108,9308) failing or are the ones you mentioned a subset of what's on your network. Just wondering if this is a class-wide problem (that is all 5300's you have deployed) or device specific (that is, 3 of your N 5300's).

3) Have you tried increasing the timeout value used by PCM for CLI and SNMP access? You can configure this by selecting the device and launching the "Comm. Parameters in PCM" tool. If the first step of the Software Update tool fails it is usually due to a timeout or mismatched communication parameters.

4) If possible, can you try rebooting one of the devices in question and letting us know if that changes anything. This will clear out any lingering telnet sessions but from your report that the communication test was successful I don't think this is likely to be the problem.

4)After restarting your management server can you please try the following steps:
- Rediscover one of the devices that is failing
- Test the communication parameters again to make sure everything is successful
- Try scanning the configuration on the device using the Configuration Manager "Scan" tool.
- Lastly, try running the software update wizard once again on the same device.

We are very interested in what you experience in each of these steps as something does seem to be amiss here. Once you have tried this can you let us know how it went and attach the file:

PNM\server\logs\FWU-server.log

This should give us a view into what is going wrong.

Sorry you're running into this.

[Jonathan Axford]

Hi, Thanks for the reply, Here are the answers to your q's....

1)That is correct, they timeout on the first stage of the Software update wizard.
I can Still successfully scan for new configurations.

2)The devices that are failing are just a subset of what we have here.
4 out of 5 4108gl switches fail the update wizard.
4 out of 11 5300XL switches fail the update wizard
2 out of 4 9300 Switches fail.

3) I have tried, After you suggested it, increasing the timeout values configured in PCM. This has had no effect. I have checked and rechecked the communication settings and they are definately right.

4) I will try restarting one of the switches as soon as i can. Can't at the moment because all hell will break loose!

4)I have deleted one of the problem devices, Restarted the server and manually rediscovered the device. I reset the comms parameters and still no joy.

I have attached the requested log file.

Thanks for your help,

[Mohamed Hamedi]

OK, I think I found a way for you to eliminate the Telnet release issue that might be causing the Wizard to timeout. This should stop SWU from doing a Telnet into the device.

We can try this with the 5300 you have.
PNM\server\config\devconfig\sw5300.oid; open this file and change the variable that indicated something like the number of OS Image versions.
To be honest I am trying to do this from memory because I donâ t have access to one these files. It should have the number 2, change that to 1. You'll need to restart the PCM server.

See if the Wizard now moves to the next step.
If it doesn't timeout then you can bet that it was some sort of Telnet issue.

Again I must stress that you are doing this on your own risk, you should never mess with application files blah blah blah.

[Mohamed Hamedi]

Hi, just wanted to ask, have you checked that these devices have their default Telnet session set to CLI and not Menu?

Looking at the log, you can see that the password for telnet is invalid, can you try and use the CLI wizard and see if that works? My guess it should also timeout.