topic Re: Non-paged dynamic memory in Operating System - OpenVMS

Non-paged dynamic memory

Wim Van den Wyngaert — Fri, 05 Nov 2004 03:07:24 GMT

I have about 60 AlphaStations with the same hardware and software. Only the name and address are different (VMS 7.2).

58 stations consume 2.5 MB np dyn mem. 2 consume 6.5 MB. This while using the same programs.

I checked with sda show pool/sum.

I found following big differences.
1) ORB normally takes 25 K but takes about 1 MB on the 2 stations
2) UCB normally takes 85 K but takes almost 2 MB on the 2 stations
3) MISC normally takes 100 K but takes 1.5 MB on the 2 stations

What does it all means and how can I find the process causing it ?

A virtual Duvel for he who solves it.

Wim

Re: Non-paged dynamic memory

Wim Van den Wyngaert — Fri, 05 Nov 2004 03:43:21 GMT

The Duvel is for me !

I found that BYTLM had been eaten too for certain processes. Then I found that 85% of available channels of that process were taken. Network problems caused the application to re-connect but without releasing the network devices.

Wim

Re: Non-paged dynamic memory

Volker Halle — Fri, 05 Nov 2004 03:47:45 GMT

Wim,

let's start looking at the UCBs (these should be Unit Control Blocks for devices).

SDA> SHOW POOL/HEAD/TYPE=UCB

type return until the list ends. It should tell you the UCB packet count and the bytes used for all those UCBs.

Then look at some of the UCB addresses found with SDA> SHOW DEV/ADDR= - take some addresses from the end of the UCB packet list.

Which kind of devices are these ?

Then do the same with /TYPE=ORB. ORBs are Object's Rights Blocks. What's the no. of packets ? You can format some of the ORBs with

SDA> READ SYSDEF
SDA> FORMAT

The associated device name is pointed to by
ORB$L_NAME_POINTER, so after you formatted an ORB, try:

SDA> EXAM @(.+orb$l_name_pointer);8

If the count of UCBs, ORBs and MISCs is about the same, you'll known that these packets somehow all are related to each other.

Volker.

PS: My guess would be: something from TCPIP, but you'll find out soon...

Re: Non-paged dynamic memory

Wim Van den Wyngaert — Fri, 05 Nov 2004 04:05:47 GMT

Volker,

Show dev net/fu shows a lot of devices allocated by a few processes. But in TCPIP there is no device and in decnet no connection. What could be the problem ?

The application people don't know why.

Wim
(half the Duvel is for you)

Re: Non-paged dynamic memory

Volker Halle — Fri, 05 Nov 2004 04:08:42 GMT

Wim,

so the UCBs/ORBs/MISCs are for which devices - TCPIP or DECnet ?

Volker.

Re: Non-paged dynamic memory

Wim Van den Wyngaert — Fri, 05 Nov 2004 04:44:54 GMT

The evidence is already gone (had to reset the stations). But it was decnet+. The proceses connect to processes on server node. If that fails, they retry in a loop.

But my main question is : why is the net device not deallocated ? Bug in decnet (7.3 eco 3) ?

Wim

Re: Non-paged dynamic memory

Volker Halle — Fri, 05 Nov 2004 04:52:18 GMT

Wim,

how are you going to ever document, analyze and find errors, if you 'delete the evidence' ? A forced crash would have provided the same results (after the reboot), but would have captured all the info needed...

If we assume those were NETn: devices, may there is a $DASSGN missing ?

Volker.

Re: Non-paged dynamic memory

Ian Miller. — Fri, 05 Nov 2004 05:14:24 GMT

The MISC could be buffers for the buffered I/O requests associated with the decnet+ devices.
But without a crash dump to enjoy its hard to say.

Re: Non-paged dynamic memory

Wim Van den Wyngaert — Fri, 05 Nov 2004 05:32:45 GMT

Sorry but too late for the crash dump. I'll take one next time.

I took the source of the application and it seems Volker is right. There is a dassgn missing (or my C knowledge is too bad and I misunderstood the coding).

Wim

Re: Non-paged dynamic memory

Wim Van den Wyngaert — Mon, 30 May 2005 03:46:02 GMT

Volker and others,

Same problem again on a different node. This time with a crash dump.

This time all non paged dynamic memory is gone (and node went into hang). Show pool/sum shows 52% is going to type unknown. Any suggestions ?

Wim

Re: Non-paged dynamic memory

Volker Halle — Mon, 30 May 2005 04:02:44 GMT

Wim,

it may not be trivial to find the underlying problem, but let's start like this:

SDA> SET OUT x.x
SDA> SHOW POOL/NONP/HEADER/TYPE=UNKNOWN
SDA> EXIT

$ EDIT/READ x.x

Go to the bottom of the file and look at the headers (first 4 longwords) of those unknown packets. Looking at the bottom of the file is based on the assumption, that 'something happened' and then some code started allocating those packets in some kind of loop. Can you spot any particular pattern (size, header contents) ?

Another possibilty would be to show all packets from non-paged pool (without /TYPE=UNKNOWN) to a file and apply the same kind of 'looking around'. You may see a repeating pattern of packet types filling up pool.

Consider to compare pool usage (packet types) with the running system to determine, if the high percentage of Unknown packets is the real problem.

If you've spotted some patterns, consider to include part of the SDA output in an attachment, but not a full pool listing ;-)

Volker.

Re: Non-paged dynamic memory

Wim Van den Wyngaert — Mon, 30 May 2005 05:09:28 GMT

The output of the first command seems to contain some Sybase server related info.
But why/what ? I'll post the output.

Wim

Re: Non-paged dynamic memory

Volker Halle — Mon, 30 May 2005 05:21:48 GMT

Wim,

the summary of your SDA> SHOW POOL/NONP/TYPE=UNKOWN output seems to indicate Unknown packets only consuming 7.4 from 55 million bytes of nonpaged pool.

Does CLUE MEM/STAT show expansion/allocation failures ?

Could you post SHOW POOL/NONP/SUMM ?

Don't expect a fast resolution (at least not with no access to the dump).

Volker.

Re: Non-paged dynamic memory

Wim Van den Wyngaert — Mon, 30 May 2005 05:37:48 GMT

Volker,

I was confused. I saw 52% and I thought it was 52% of the whole NP dyn mem. I checked on the running system and it also takes about 50% unknown.

I also posted the output of show mem, the prove that the memory was gone.

VPA also proved that it was gone since the moment VPA started during boot.

Wim

Re: Non-paged dynamic memory

Volker Halle — Mon, 30 May 2005 05:43:23 GMT

Wim,

your attachment looks like an SEA errlog analysis. Could you please post the SDA> SHOW MEM/POOL/SUMM output ?

If you're running VPA, you could tell, when the problem started. Did pool decrease slowly over time or did it just decrease suddenly ?

Volker.

Re: Non-paged dynamic memory

Wim Van den Wyngaert — Mon, 30 May 2005 05:49:36 GMT

Sorry Volker.

VPA didn't see a decrease. When it started less than 1 MB was free.

Wim

Re: Non-paged dynamic memory

Volker Halle — Mon, 30 May 2005 05:50:06 GMT

Wim,

correction:

SDA> SHOW POOL/NONP/SUMM

About some of the Unknown packets:

TCPIP does not use a standard OpenVMS packet header, but you can identify the TCPIP packets, which show up as Unknown:

If the pool packet size and the 4th longword of the packet header (count from right to left) have the SAME value and the VMS packet size word (low word in 3rd longword) is ZERO, it's most probably a TCPIP packet.

Volker.

Re: Non-paged dynamic memory

Volker Halle — Mon, 30 May 2005 05:59:02 GMT

Wim,

let's try to re-assess the problem description:

- are we talking about unexplained static nonpaged pool consumption on 2 of your 60 Alphastations ?
- if so, is it still ORBs, UCBs, and MISCs ?

- by the time VPA starts during startup, most of nonpaged pool is already consumed.

- and today one of those 2 stations hung and you forced a crash.

Anything else ?

Could you please attach the SDA> SHOW POOL/NONP/SUMM output ?

Volker.

Re: Non-paged dynamic memory

Wim Van den Wyngaert — Mon, 30 May 2005 06:07:53 GMT

Volker,

This is not the same problem. This time it concerns a GS160 node of an interbuilding FDDI cluster. I found it in hang and forced a crash. And after 2 minutes of boot, the memory seems to be gone.

Sorry for the mistakes but I have to use different tools to get it on my pc.

Wim

Re: Non-paged dynamic memory

Wim Van den Wyngaert — Mon, 30 May 2005 06:08:51 GMT

Now HP nailed me ...