Switches, Hubs, and Modems
cancel
Showing results for 
Search instead for 
Did you mean: 

PCM+ Traffic Problems

SOLVED
Go to solution
Jason Scott
Regular Advisor

PCM+ Traffic Problems

On our old PCM installation we started to encounter a problem with traffic monitoring. When a port was selected the utilisation or top talkers graphs would say Awaiting Data and never refresh. After about 2 minutes the whole client would crash with the error 'The connection with the Procurve Network Manager service has been broken'.

I put this down to the older server hardware PCM was installed upon. Recently we've purchased a brand new high performance server with quad core processors and 4Gb RAM. For the first two days PCM ran fine but today I'm encountering the same errors with traffic monitoring.

The PCM installation is completely new with no migration of old data. Has anyone encountered this before or is aware of a fix? As it stands the traffic monitoring is a key feature for us. PCM monitors around 260 switches varying from 2600s, 53/5400s and 8200s.
27 REPLIES
cenk sasmaztin
Honored Contributor

Re: PCM+ Traffic Problems

hi Jason
what have you PCM+ licence

you have 260 switch on network
is this true..?
in that case you have must be
pcm+ 2.3 unlimited device licence

or

pcm+ 2.3 100 device licence x3deal
cenk

Jason Scott
Regular Advisor

Re: PCM+ Traffic Problems

Hi, we have the unlimited device license, however I've been advised by HP support to wait 1 week following the recent re-installation before reregistering our licenses in case we encountered problems necessitating another reinstall.
Renico
Occasional Advisor

Re: PCM+ Traffic Problems

Hi Jason,

Did you try to install the latest update?

ProCurve Manager Plus 2.322(Auto Update 7)

ftp://ftp.hp.com/pub/networking/software/pcm_2_3_322_update_7.zip
Jason Scott
Regular Advisor

Re: PCM+ Traffic Problems

We let the system patch itself to update 6 - it doesn't recognise update 7 via the GUI. I'll manually install this update, however I do not see any fix in the release notes relating to this particular problem.

In the server logs you can clearly see warnings from the trafficd.exe process failing and terminating. I blame Java! Any software I've come across written in Java is buggy.
Jason Scott
Regular Advisor

Re: PCM+ Traffic Problems

I've applied update 7 and initially traffic graphs and top talkers were working, but after around 5 minutes they have stopped again.

I'll raise this as a fault with our supplier.
Renico
Occasional Advisor

Re: PCM+ Traffic Problems

Hi Jason,

Just to be on the save side and eliminate possible config issues, could you please try the following command on a 5400 or 8200 switch:

snmp-server host 1.1.1.1 Public

with 1.1.1.1 representing your PCM server IP, and see if you can successfully gather stats afterwards?
cenk sasmaztin
Honored Contributor

Re: PCM+ Traffic Problems

hi jason
check pcm server log files

\Program Files\Hawlett-Packard\PNM\server\logs\CS-err.log

and

\Program Files\Hawlett-Packard\PNM\server\logs\SvrJCoreMain0.log

cenk

cenk sasmaztin
Honored Contributor

Re: PCM+ Traffic Problems

and send me log
cenk

Jason Scott
Regular Advisor

Re: PCM+ Traffic Problems

Thank you.

CS-err Log attached.
Jason Scott
Regular Advisor

Re: PCM+ Traffic Problems

And SWvJCore Log.
Jason Scott
Regular Advisor

Re: PCM+ Traffic Problems

Regarding the configuration, I'm sure its correct as we were seeing traffic details coming through and I believe they're still being recorded in the server database. Its the act of retrieving or viewing them which causes the problem. Just to be sure I've tried the manual configuration suggested and experienced the same results.

Also this happens if I use a local client on my machine and if I RDP onto the server and run the client from there.
cenk sasmaztin
Honored Contributor

Re: PCM+ Traffic Problems

2008 Oct 31, 09:32:09:426 java.lang.OutOfMemoryError: Java heap space

what is use operation system under PCM+

cenk

cenk sasmaztin
Honored Contributor

Re: PCM+ Traffic Problems

please check

*to receive statistic data make sure there are no other processes bound to port 162including the windows trap receiver service

*in the PCM+ Prefereance tool under the events node uncheck(unknow events to ignore)to see if the PCM managemet server can receive traps

*for SFlow PCM+ must be allowed to receive traffic on port 6343

check event browser window for erors related to the traffic manager

use the windows service tool to verify that procurve lunch service is running.

use the microsoft task manager to check that the Trafficd.exe and TLS.exe processes are running on the PCM managemet server

Restart PCM client
Restart there PCM managemet service
cenk

cenk sasmaztin
Honored Contributor

Re: PCM+ Traffic Problems

and you make all java update on your windows server from java.com
cenk

Jason Scott
Regular Advisor

Re: PCM+ Traffic Problems

Total 3405672
Available 1587076
System Cache 1647316

Commit Charge 1700M/5221M

The system has 4Gb memory installed, however its not running a 64bit OS so only 3GB available.
Peak memory seen is 2275652. The javaw.exe process always runs around 1,107,916k memory usage, which I think is atrocious and represents extremely bad programming.
Jason Scott
Regular Advisor

Re: PCM+ Traffic Problems

Will do, thanks for your time Cenk. I'll post back with the results.
Jason Scott
Regular Advisor

Re: PCM+ Traffic Problems

Trafficd.exe running, 94,660k
TLS.exe running, 2,516k

Procurve Network Manager Server service started
Traffic Launch service started
Datastore service started

The Windows SNMP service is also started - perhaps that conflicts?
The SNMP Trap Service however is not running - set to manual startup type.

I'm not seeing any errors in the computer event log from any PCM service.

If I look in the traffic data files I can see information coming in. On initially starting the client I can see the pie chart updating with utilisation etc. The 'awaiting data' problem occurs when clicking a port to see the utilisation graphs or top talkers. Thats when it all hangs. It was working for the first day or two, which makes me suspect it cannot handle the number of switches we have in the network.

cenk sasmaztin
Honored Contributor
Solution

Re: PCM+ Traffic Problems

Jason Scott
Regular Advisor

Re: PCM+ Traffic Problems

Cenk, you are a genious. I've restarted processes and all the traffic graphs and top talkers were working fine but the javaw.exe process kept increasing in memory size all the way up to 1108mb. Now my graphs and top talkers have all stopped working.

Do you think an upgrade to the very latest Java will fix this?
cenk sasmaztin
Honored Contributor

Re: PCM+ Traffic Problems

Do you think an upgrade to the very latest Java will fix this?

may be Jason go to java.com you make full update after retest.
cenk

Jason Scott
Regular Advisor

Re: PCM+ Traffic Problems

Thanks, I'll try and post results.

I just re-ran the test and all graphs worked until javaw.exe memory hit 1109mb. Clearly the cause.
cenk sasmaztin
Honored Contributor

Re: PCM+ Traffic Problems

I happy for your solve problem
you are welcome jason
cenk

Steve Britt
Respected Contributor

Re: PCM+ Traffic Problems

> ... the javaw.exe process kept increasing in memory size all the way up to 1108mb. Now my graphs and top talkers have all stopped working.

Jason,

The client process might be running out of memory. You could try expanding the amount of memory that it can use by modifying the script file client/sbin/riptide.ksh, used to launch the client, in your PCM install directory.

Replace the value "1024" in the string "-Xmx1024m" with something larger like maybe "1280" or something. This will increase the number of megabytes that the client JVM is allowed to use and may solve your problem, because it sounds like you're suspiciously close in size to that max of 1GB (1024MB). If it doesn't work at first keep trying it and increasing the value by, say, 128 each time until you succeed or hit about 2GB. I wouldn't bother going much beyond 2GB because you'll start to exceed the addressable memory space of the JVM some time around there.

Hopefully this will help ...

Regards,

SVB
André Beck
Honored Contributor

Re: PCM+ Traffic Problems

Jason,

I don't expect installing another JRE will change anything, PCM+ comes bundled with its own version of a JRE and uses that exclusively AFAIK, both on the server and UI client sides.

But welcome to the wonderful world of Java apps, where the coders are so remote from the actual machine they don't even recognize how broken their spheres of residence are. The mantras of "the JVM isn't a memory hog, it just grabs most of the memory that's there - should it stay unused?" and "But garbage collection is so much better than explicit allocation management!" are nice little ivory towers, they just start to break badly in the real world when the number of JVMs trying to run on the actual machine and OS exceeds *one*. The ultimate horror can be witnessed on Cisco LMS (Cw2k), where I think I counted no less than 40 concurrent javaw.exe running at the same time, trashing like hell even with just a handful of switches managed and more than two users logged in. Atrocity, as you say it. In that regard, PCM+ is among the better code still.

I've also faced that situation of traffic display never being updated in the UI and general slowness and resource overconsumption by trafficd in earlier versions of PCM+. It was a significantly smaller network (some 50 switches), but I remember that traffic monitoring was activated for each and every port. I reinstalled PCM+ on a new server in latest versions around July this year and started traffic management in auto mode with just ISL monitoring, and that seems to be stable now.

I think the hint to increase the top memory allocation limit for the trafficd JVM might be what you're looking for, even though just inflating to the configured limit doesn't need to be a sign of memory scarceness within the JVM, it may just be "well, the memory is there, so what" Java coder "thinking" (reminds of similar solitaire attitudes regarding the physical resources of our planet). I'd try to tinker with it anyway.

Andre.