SYMBIONT_xxx

[Volker Halle]

Art,

which symbionts are we talking about, i.e. what main image is running in all those (4-5) SYMBIONT_xxx processes you've been looking into with SDA ?

What is it you didn't find: 02E80103 ?

Did you check any working SYMBIONT ? SHOW SYS/PROC=SYMBIONT* should show increasing IOs for the 'working' ones. If you find 02E80103 in a working symbiont, we know that we are on the right track.

STOP/QUE/RESET could probably leave bad symbiont processes around. If the symbiont did not really finish it's initialization, it may also hang around after a /RESET...

Did you check OPERATOR.LOG for any symbiont/queue_manager related messages ?

Volker.

[Art Wiens]

Volker: as an example, attached is the results from trying to find out which queue SYMBIONT_695 belongs to using your steps.

Is this a "deceased" DCPS symbiont?

Art

[David Jones_21]

Did you try using command "$ tcpip show device BGxxx" to see if it is bound to a socket? If so, the remote address should tell you which printer it was trying to connect to.

[Art Wiens]

It's Process Software's TCPware not Digital's TCPIP ie. there is no TCPIP SHOW DEVICE command!

A DCL SHOW DEVICE gives what's in the attachment.

A NETCU SHOW CONNECTION doesn't show any BGxx: devices, only INETxxxx: .

Art

[Volker Halle]

Art,

if these are DCPS$SMB symbionts, you may want to check, if the DCPS$queuename_PID logical exists. If not, this may already be an indication of a 'bad' symbiont process.

Please check OPERATOR.LOG, DCPS is usually very good at sending error messages to OPCOM (consider to enable your terminal as an OPERATOR terminal with REPLY/ENABLE when troubleshooting DCPS printer problems).

Please verify a working DCPS queue symbiont, whether you can find the SCB with SDA, so we know, that this method really works.

Note that you can easily find out with SDA, when a process has been started:

SDA> SET PROC/IND=
SDA> EXA/time CTL$GQ_LOGIN

I noted that the IO operations count on the BG15: device is quite high, also the accumulated CPU time for SYMBIONT_695. For a non-working symbiont, why would it consume CPU and IOs ?

Volker.

[Art Wiens]

"Please verify a working DCPS queue symbiont, whether you can find the SCB with SDA, so we know, that this method really works."

Doesn't seem to. I used the steps to look at an existing DCPS$qname_PID and didn't find the SCB. The image list is the same as I attached earlier for the other PID. As well, the P0 address is the same.(?)

"CTL$GQ_LOGIN"

The login time for the pid in the attachment matches when I rebooted the system a few weeks back. I would think it has to be an orphaned process if that pid doesn't match one of the current DCPS$*PID logicals, no?

"OPERATOR.LOG, DCPS is usually very good at sending error messages to OPCOM "

The pertinent DCPS messages in the log are one of three:

%DCPS-W-NOT_READY, Printer is not ready

%DCPS-I-PRINTERSTALLED, Printer "IP_RawTCP/aa.bb.cc.dd:9100" is stalled

%DCPS-F-CONTERMINATED, Connection abnormally terminated

The site has 8 of these printers (HP8000's and HP9050's). They print an awful lot and run the queues themselves. Corelating problems and log entries will be a "challenge".

In case some of you haven't looked at my forum profile, my Personal Quote stands ! ;-)

Cheers,
Art

[David Jones_21]

You could brute force it, copy SYS$SYSTEM:DCPS$SMB.EXE multiple times to DCPS$SMB_n.EXE and re-init each queue with a unique /processor image. I suspect you'll find that no particular queue is causing the zombies.

[Antoniov.]

Art,
I carefully reread the full thread and I'm convinced there is no direct solution.
I can't find any property to link queue with its executor.

Antonio Vigliotti

[Volker Halle]

Art,

if your 'problem symbionts' are all running DCPS$SMB.EXE and they do NOT have an appropriate DCPS$queuename_PID logical, I would consider them to be 'bad'.

DCPS may be using the SMB routines directly and not the PSM routines, which explains, why you can't find the SCB data structure in a working DCPS symbiont.

Finding your 'wedged' symbionts could be as follows:

look at all SYMBIONT_nnn processes
for of each process, check:
1) running image DCPS$SMB.EXE ?
2) is there a logical DCPS$*_
3) obtain CPU and IO count, wait some seconds and re-check CPU and IO. No change ?

If all above checks return true, use STOP/ID to get rid of symbiont. If you stop a SYMBIONT process, there will be 2 OPCOM messages from QMAN - but they won't tell you the affected Queue Name ;-( The queue - if it was really being handled by this symbiont - will go to STOPPED state. So if you do a SHO QUE/DEV/OUT=x.x before and after stopping the symbiont, you could do a DIFF and find out, which queue state has changed.

Volker.

[Antoniov.]

Finding your 'wedged' symbionts could be as follows:

look at all SYMBIONT_nnn processes
for of each process, check:
1) running image DCPS$SMB.EXE ?
2) is there a logical DCPS$*_
3) obtain CPU and IO count, wait some seconds and re-check CPU and IO. No change ?


The 3.th point is dangerous. If printer queue is idle for some seconds, you can kill active processor.
On other hand, I guess DCPS*_ exists even process is stalled.

Antonio Vigliotti

[Jan van den Ende]

Art (and others)

I sent a request to Guy Peleg to shine his light on this issue.

I am curious what he has to say.

Proost.

Have one on me.

jpe

[Guy Peleg]

How about:

pipe show queue/full | search sys$input "Server queue ","/PROCESSOR"


We'll consider adding

show queue/processor=name and show
queue/processor_ pid=xxxxxxxx into
a future release of the O/S

Guy

[Art Wiens]

Thanks Guy, but I'm not sure how that helps me in this instance.

There has been much discussion here, but the original question to the thread has not been answered (yet ;-).

I have not been able to find any link between a symbiont pid/process and which queue it belongs to - other than the obvious functioning DCPS queues.

Cheers,
Art

[Volker Halle · Accepted solution]

Art,

here we go...

The information is hidden in the QUEUE MANAGER database file (assuming the default queue manager name):

$ DUMP/REC/OUT=X.X SYS$SYSTEM:SYS$QUEUE_MANAGER.QMAN$QUEUES
$ SEARCH X.X /WIND=(5,12)

You'll find the extended PID in column 1 of the SDA> SHOW SUMM output or the SHOW PROC output /1st line).

The first line reported by SEARCH will show the queue name. The SYMBIONT PID is stored at offset 0x00B0 in each QUEUE record, which has a symbiont process associated.

It HAD to be in the file, because otherwise a QUEUE_MANAGER failover could not have worked !

Hope this is not again considered a 'brute force' solution ;-)

Volker.

[Volker Halle]

Art,

please find attached a little DCL procedure to report the queues and their associated symbiont extended PIDs ;-)

Tested on OpenVMS V8.2 with a small amount of queues. No guarantees included.

Volker.

[Antoniov.]

Hi,
Volker's DCL procedure works on V7.3-2.

Antonio Vigliotti

[Martin Vorlaender]

Hi,

Volker's procedure works on V7.2-2.

cu,
Martin

[Volker Halle]

Hi,

and it also works on OpenVMS VAX V6.2 - so this should cover a large range of VMS versions...

Volker.

[Antoniov.]

Yes,
it works also on V6.2

Antonio Vigliotti

[Art Wiens]

Volker ... you da' man!!!

10 points for perserverance
10 points for "ease of use"
and
10 points for "the answer"!!

No I don't think it's so much brute force, but rather a good bit of hacking!

Thanks so much, that's exactly the correct answer to the question I posted.

Cheers,
Art

[Art Wiens]

Look what I just found:

3.17 Avoid STOP /QUEUE /RESET Usage for PrintServer Printer Which Is Rejecting Connections

If you issue a STOP /QUEUE /RESET command for a queue to a DIGITAL PrintServer printer while there is a job in the "Starting" state and while the printer is rejecting connections (because, for example, the PrintServer is powered off or is booting), the queue will stop. ***>>> Occasionally the symbiont process will not terminate.<<<*** Avoid issuing this command until the PrintServer printer becomes available. If the job is in the "Starting" state and also in the PrintServer printer's job queue, a STOP /QUEUE /RESET will execute correctly.

Not quite the correct environment (HP9000 not PrintServer), but at least there is aknowledgment that a DCPS symbiont can get left behind if "something bad happens".

Art

[Ing. Ferry Bolhar]

As suggested, I tried to look in SYS$SYSTEM:SYS$QUEUE_MANAGER.QMAN$QUEUES. However, it seems no all queues are stored in this file. On a server with over 900 server queues (no generic or batch), I can see 140 queues only when running the attached DCL procedure.

Does someone know why the other queue are not visible?

[Wim Van den Wyngaert]

Just created 1500 batch queues and they are all visible. What kind of queues are not visible ?

Wim

[David Jones_21]

Neat procedure, but the filespec it should be using is QMAN$MASTER:SYS$QUEUE_MANAGER.QMAN$QUEUES to cover sites (like mine) that have moved the queue database off the system disk. The guy with lots of queues is probably running several queue managers so the data is spread over several .qman$queues files.

[Ing. Ferry Bolhar]

Actually, this was the solution. There were 2 queue managers running on that node. I changed the DCL procedure now to support configurations like this (if someone is interested, I can post it). Many thanks for your help!

Kind greetings, Ferry