lpsched dumps with "Insufficient Memory"

[Massimo Bianchi]

Hi all,
we are on hpux 11.23 on a ia64 architecture.

our lpsched is dumping with a repetitive message:

lpsched: Insufficient memory to execute printer interface program.
lpsched: CHILD: Oct 6 23:41 ZPAGAM01 143 8600
lpsched: FIFO: Oct 6 23:41 m ZPAGAM01 r 24762

Again, and Again, and again.

ZPAGAM01 is a selfmade script, that converts PDF to PS (already working on another system).

lpsched gets into this thigh loop, and never ends.

Any suggestion ?
Regards,
Massimo

[OFC_EDM]

It may end up not being memory but to start how much memory do you have available on the system?

Output of top/sar or something equivalent would be a start.

Cheers

[Massimo Bianchi]

swapinfo -mt
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 8192 0 8188 0% 0 - 1 /dev/vg00/lvol2
dev 40000 0 40000 0% 0 - 2 /dev/vg00/lvol9
reserve - 15682 -15682
memory 24546 5373 19173 22%
total 72738 21055 51679 29% - 0 -

System: nrisap1 Tue Oct 7 15:04:52 2008
Load averages: 0.28, 0.15, 0.22
411 processes: 369 sleeping, 42 running
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.30 16.8% 0.0% 15.8% 67.3% 0.0% 0.0% 0.0% 0.0%
1 0.27 10.8% 0.0% 5.9% 83.3% 0.0% 0.0% 0.0% 0.0%
2 0.27 23.5% 0.0% 12.7% 63.7% 0.0% 0.0% 0.0% 0.0%
3 0.27 8.8% 0.0% 15.7% 75.5% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.28 15.5% 0.0% 12.6% 71.8% 0.0% 0.0% 0.0% 0.0%

Memory: 13038196K (7037008K) real, 16318252K (7722324K) virtual, 6567792K free
Page# 1/46

[OFC_EDM]

Well it's not memory.

Now the question everyone hates :)

How recently has your system been patched?

What's the current patch level?

[Massimo Bianchi]

Hello,
in attach the patch level.

This system is a fresh start.

We have the original system on a N-Class, and we are going to migrate to IA in the next few weeks.

There will be SAP & Oracle.

We are struggling with the sppoling subsystem.
It works with dummy printers (printing to network printer, copying to file...), but not with this script).

Regards,
Massimo

[Steven E. Protter]

Shalom,

The loop is consuming memory. At least I think it is.

http://www.hpux.ws/?p=8
Confirm with the tool above.

Also, are the systems patched the same way. lpshed needs to be current to avoid problems.

SEP

[Massimo Bianchi]

Hi SEP,
nice to see you back. I was lacking for some time...

Thanks for the script, but the memory used by lpsched is so small that is not traced correctly by the script (SAP & Oracle are allocating much much more).

lpsched is at patch latest-1, but latest is not addressing officially this problem.

The interface script is as easy as this:

#! /usr/bin/sh
export DISPLAY=localhost:0.0

#/IFR/scripts/rotazione_file /IFR/stampe/nonbollati/pagamenti/01/pagamenti_rina.
ps

FILENAME=$6
eval cat $FILENAME | /opt/Acrobat7/bin/acroread -toPostScript >> /IFR/stampe/non
bollati/pagamenti/01/pagamenti_rina.ps

Question is:
- why is it looping
- why is it consuming memory, since should be a one-shot command
- why is lp a 32 bit application ..

root:nrisap1:/etc/lp/interface #what $(which lpsched)
/usr/sbin/lpsched:
lpsched.c $Date: 2006/02/20 21:19:41 $Revision: r11.23/7 PATCH_11.23 (PHCO_34477)
outlist.c $Date: 2005/10/24 05:22:27 $Revision: r11.23/1 PATCH_11.23 (PHCO_34021)
lpio.c $Date: 2005/04/19 02:40:14 $Revision: r11.23/2 PATCH_11.23 (PHCO_32770)
pstatus.c $Date: 2005/04/19 02:41:58 $Revision: r11.23/1 PATCH_11.23 (PHCO_32770)
outputq.c $Date: 2007/06/28 13:33:13 $Revision: r11.23/5 PATCH_11.23 (PHCO_36440)
genfuns.c $Date: 2005/04/19 02:38:38 $Revision: r11.23/2 PATCH_11.23 (PHCO_32770)
sendmail.c $Date: 2004/11/24 01:35:26 $Revision: r11.23/1 PATCH_11.23 (PHCO_32278)
qstatus.c $Date: 2005/04/19 02:42:15 $Revision: r11.23/1 PATCH_11.23 (PHCO_32770)
$Revision: @(#) lp R11.23_BL2007_0630_1 PATCH_11.23 PHCO_36440

chatr $(which lpsched)
/usr/sbin/lpsched:
32-bit ELF executable
shared library dynamic path search:
LD_LIBRARY_PATH disabled first
SHLIB_PATH disabled second
embedded path enabled third Not Defined
shared library list:
libsec.so.1
libc.so.1
shared library binding:
deferred
global hash table disabled
global hash table size 1103
shared library mapped private disabled
mem_check value disabled
shared library segment merging disabled
shared vtable support disabled
explicit unloading disabled
linkage table protection disabled
segments:
index type address flags size
7 text 04000000 z---c- D (default)
8 data 40000000 ---m-- D (default)
executable from stack: D (default)
kernel assisted branch prediction enabled
lazy swap allocation for dynamic segments disabled
nulptr dereferences trap disabled
address space model: default
caliper dynamic instrumentation disabled
root:nrisap1:/etc/lp/interface #


Regards,
Massimo

[OldSchool]

well...

I think I'd start with verifying that acroread works properly (at the command line).

or replace

eval cat $FILENAME | /opt/Acrobat7/bin/acroread -toPostScript >> /IFR/stampe/non
bollati/pagamenti/01/pagamenti_rina.ps

in the script with something simple, like "cat" or "ll".

does the scheduler function properly with just a plain file, ascii or PS for example (something that doesn't process through this interface script)?

the message says its the interface script that runs out of memory, but I suspect its dumping for some reason.

even though it works on another machine doesn't mean its right on this particular installation, especially if you are switching platforms.

If you've already done so, what results did you get?

[Massimo Bianchi]

Hi OldSchool,
thanks for your tips, but I already tried to have a simple script that copies input -> output, and it works.

I know that if it is working on the other server, it doesn't mean anything... but:
- HP states that pa-risc executable can work, on ia64, through aries, even if with performance penalties
- the customer wants its SAP PDF converted to PS :(

I do not understand why the lpsched is running out of memory, looks like it is a recurrent something. But have no clue how to do a deeper analisys.

Regards,
Massimo

[OldSchool]

"thanks for your tips, but I already tried to have a simple script that copies input -> output, and it works."

So, lpsched appears *not* to be the issue.

"I know that if it is working on the other server, it doesn't mean anything... but:
- HP states that pa-risc executable can work, on ia64, through aries, even if with performance penalties"

Right...but at this point you don't *know* that this particular executable functions properly by itself..so it could still be an issue. What happens if you try to convert a PDF to PS at the command line? I know it *should* work, but I suspect it doesn't for some reason, like missing libraries, or shared libraries not in the shlib_path or some other, correctable, reason. since lpsched works when it isn't running the script, the prime suspect IS the script at this point, and the only *unusual* thing about the script is acroread (at least that I see).

If works from the command prompt, I would then suspect an issue with the environment the script is running in. At that point, I believe I'd dump the users environment to a file with "env > somefile", do the same it the interface script and compare the two.

I suspect that you would find a difference in the shared library paths of somesuch.

If acroread dumps at the command line, resolve that issue and apply the same fix in the interface script. Are you sure all of the prerequisites are installed, like CDE and GTK+ toolkit? See the following for more information.

http://howto.gp.mines.edu/local-apps/acroread-7.0.8/ReadMe.htm
http://howto.gp.mines.edu/local-apps/acroread-7.0.8/ReadMe.htm

[Dennis Handly]

>lpsched: Insufficient memory to execute printer interface program.

I'm not sure if this message is valid.
It could be any error related to fork?
Too many processes, not enough VM, etc.

>but the memory used by lpsched is so small

What does top show? Is it near maxdsiz?

>But have no clue how to do a deeper analysis.

Can you put an echo in your script to see if it even gets executed?
You might want to use tusc on lpsched to see what's really going on.

>- why is lp a 32 bit application

Why do you think it needs to be 64 bit?

>OldSchool: the message says its the interface script that runs out of memory, but I suspect it's dumping for some reason.

Right.

[Massimo Bianchi]

@OldScholl: i re-checked dependencies... from command line it works perfectly.

I even copied the ENV to the interface script, but it goes bunnying

@Dennis: VM and forks should not be a problem. The server is configured as an SAP ORACLE instance, plenty of room for that.

top is far from maxdsiz.. I can post the kernel parameter, but I don't think they will be usefull.

BTW i liked to think that if an OS is 64bit, all of its process are on 64bits ... just to avoid any inconvenience on low memory addresses :)


I do agree that it is dumping for some reason... but which ?

I also installed latest lp patch, just to be on safe side, but no change.

Regards,
Massimo

[Massimo Bianchi]

I tried to tusc the lpsched.

Not very much usefull, the only "strange" behaviour to me is a ENOTTY.. but in a queued job I do expect it.

help :)

Massimo

[Massimo Bianchi]

Sorry, wrong file attached in previous reply.

[Dennis Handly]

>I tried to tusc the lpsched.

Oops, you need more options to follow forks:
tusc -fp -o tusc.out lpsched-pid

I don't see any fork errors.

Looking at the code to lpsched, it prints that error if the exit status of your interface returns: exit-status & 2

How are you mapping your acroread exit status to your interface status?

[Massimo Bianchi]

Hi Dennis, thanks for the tusc input.

I did another try, in attach the printout.

For the mapping: nothi special.

As from the script, the exit code of the acrobat is trasnferred to the exiting shell.

but manually the shell works.

I even tried to logon as "lp" and execute it.

Interesting mistery...

[Massimo Bianchi]

FYI,
I also tried to change the script, issuing a remsh to execute the Acrobad in a remote terminal, on the same host.

Manually it works..... from lpsched NO.

Bha...

[Peter Nikitka · Accepted solution]

Hi,

from the tusc-output it is clear, that the 'real' acroreader program crashes.
I assume, that /opt/Acrobat7/bin/acroread is a front end shell script.
Use a 'sh -x /opt/Acrobat7/bin/acroread ... 2>/tmp/acro.err' in your interface script and check, what environment changing stuff is done there.
The search order for shared libraries seems to get modified:

...
stat("/opt/Acrobat7/Reader/hppahpux/lib/libdld.2", 0x7b03dfe8) ERR#2 ENOENT
[15665] stat("/opt/Acrobat7/Reader/hppahpux/sidecars/libdld.2", 0x7b03dfe8) ERR#2 ENOENT
[15665] stat("/opt/graphics/OpenGL/lib/libdld.2", 0x7b03dfe8) ERR#2 ENOENT
...

This may have side effects for the library
libgdk_pixbuf_xlib-2.0.sl

which seems no to be found.

mfG Peter

[Massimo Bianchi]

Great input.

FOr some strange reason, the lpsched calling acrobat delete all the environment variables.

I forced them INTO the acrobat wrapper, and it worked...

THank you very much.

Regards,
Massimo

[Dennis Handly]

>For the mapping: nothing special. As from the script, the exit code of the acrobat is transferred to the exiting shell.

You can't do that. You must map the exit status to what lpsched wants. Otherwise you get this bogus "Insufficient Memory" error.

>Peter: The search order for shared libraries seems to get modified:
stat("/opt/Acrobat7/Reader/hppahpux/lib/libdld.2", 0x7b03dfe8) ERR#2 ENOENT

Ignore this, that's how dld works with SHLIB_PATH.

>This may have side effects for the library libgdk_pixbuf_xlib-2.0.sl

Yes, dld reports that.

>For some strange reason, the lpsched calling acrobat delete all the environment variables.

Where are the variables defined? Before the user calls lp? But lpsched gets started by root in init.

You can add "-e -a" to tusc to see the exec parms and environment.

[Massimo Bianchi]

Hi Dennis,
the env variables where defined in hte interface script itself.

But somehow they were not passed to the acroread wrapper script.

Now seems to works.

Thanks,
Massimo

[Massimo Bianchi]

Up to now, solution found... maybe tomorrow will change ?

[Dennis Handly]

>the env variables where defined in the interface script itself. But somehow they were not passed to the acroread wrapper script.

I assume you exported them?
What does tusc "-fp -ea" show?