lpd "resource temporarily unavailable"

[Anthony_141]

We are running Redhat Advanced Server 4 (kernel 2.4.9-e.62) on a ML350 G4p.

We have several G4p's loaded the same and one today stopped printing.

It had only been up about 7 days.

When you did "lpstat -o" to check the print queues we saw "Server load too high". There were no hung print jobs in the /var/spool/lpd queues.

We tried bouncing lpd, but got "dfork failed- Resource temporarily unavailable".

Top, SAR, etc showed fine. Everything else was running fine.

We ended up booting the server and lpd started normally and the users could print.

Googling did not show any useful information.

Any suggestions on what caused this and how to fix it (without booting) in the future would be appreciated.

[Kodjo Agbenu]

Hello Anthony,

What priting system are u using ? LPRng or CUPS ?

Did you try restarting CUPS ?
Did you try to purge the printing queue ?

Try :

/etc/init.d/cupsd stop
rm -f /var/spool/cups/c00* /var/spool/cups/tmp/*
/etc/init.d/cupsd start

Good lcuk.
Kodjo

[Anthony_141]

We use lpd, not cupsd (hence the lpd references in my original note). But, thanks for the tip anyway!

[Vitaly Karasik_1]

- what is your lpd.conf?
- lpq output?
- something interesting into /var/log/messages?

Rgds,
Vitaly

[Anthony_141]

We never touched lpd.conf (took the defaults).

lpq and message out put was noted in original messages (Server load to high, dfork failed).

[Ivan Ferreira]

"Server load too high", dome daemons, like lpd and sendmail, won't run if the server load is high as indicated in the message, what was the server load (uptime) at the time of the problem? Some runaway process?

[Anthony_141]

"Server load too high", dome daemons, like lpd and sendmail, won't run if the server load is high as indicated in the message, what was the server load (uptime) at the time of the problem? Some runaway process?

We ran Top, SAR, etc. and there was a normal load on the server (which was very, very low - we don't hammer the systems by any means).

Our 2 databases were running fine,and everything else appeared to be normal.

Uptime was only 8 days.

We can't rule out a "runaway process", but idle time was 95% so it's not like the system was being hogged by anything.

[Anthony_141]

Did just see something in the SAR logs:

Before the server was booted, %memused was at 99%. After the reboot it went down to 40%-60% used. Also it looks like we were using most of our Swap pre-reboot.

We ran TOP and it looked "OK" at the time - anyone have any other suggestions on how to trace the memory hog if this happens again?

[Ivan Ferreira]

99% memory usage reported by sar is normal. Linux will always use all memory, if not by programs, by buffers/cache. As you rebooted the system, the buffers/cache was empty, but later will the memory. This is good.

By uptime I mean, the load average reported, not the system uptime itself, for example:

load average: 27.34, 27.20, 27.06

[Anthony_141]

Some load average data from right before we booted:

07:00:00 8 4137 0.01 0.10 0.11
07:10:00 5 4146 0.98 0.46 0.22
07:20:01 2 4159 0.13 0.21 0.17
07:30:00 3 4168 0.07 0.11 0.14
07:40:01 3 4176 0.02 0.07 0.09

07:40:01 runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15
07:50:00 1 4181 0.88 0.49 0.24
08:00:00 5 4201 0.05 0.22 0.24
08:10:00 5 4208 0.09 0.17 0.20
08:20:01 2 4213 0.46 0.35 0.25
Average: 5 3985 0.20 0.19 0.17

[Anthony_141]

Note that it should matter, but this is Redhat AS 2.1 Update 2(Redhat AS 4.0).

[Vitaly Karasik_1]

Anthony,

So is this RHEL2.1 (according to kernel version 2.4.9) or RHEL4?

Can you please send your lpd.conf & printcap files?

Try to add debug option to lpd.

[Anthony_141]

This is Redhat Advanced Server 2.1 (kernel is kernel 2.4.9-e.62)

We never touched lpd.conf (everything is commented out). Attached is our printcap file.

Also, I believe this issue started on Monday the 6th. SAR logs show swap going up to 25% used at that time and never went back down until the reboot. Gradually Swap usage grew from the 25% used to 100% used the morning of the 9th (when we started having lpd issues). Checking simlar systems normally swap is around 2-3% used.

[Vitaly Karasik_1]

Probably your server was really overloaded?
I suggest you to do two things:
- increase debug level for lpd ("-D" with some options)
- in addition to sar you can run "top" command via crontab in order to monitor cpu/memory utilization by each process

[Anthony_141]

Top and Sar looked "OK" before we rebooted (did sar -u 5 5 and we were 90% idle). Top showed nothing strange at the time.

[Vitaly Karasik_1]

but you spoke about unusual high memory consuming, right?

[Anthony_141]

SAR logs showed we were at 100% Swap usage, but running SAR from the command line showed "OK" at the time (a few minutes before we chose to reboot).

Again, we only did "sar -u 5 5". We didn't look at swap specifically from command line.

We tried bouncing lpd, ran TOP, ran SAR, checked a few other things but probably should have ran more SAR commands to get more "live" detail. We looked at the SAR logs after the reboot, which is when we saw that Swap usage started creeping up on Monday the 6th and maxed out the morning of the 9th.

I'm thinking lpd dying and not starting was a symptom, it probably wasn't specifically an lpd issue (but then again may have been). Something on the 6th started making the system use much more Swap than normal and it never went back down to a normal level until after the reboot.

[Vitaly Karasik_1]

>I'm thinking lpd dying and not starting >was a symptom, it probably wasn't >specifically an lpd issue

Agree!

[Anthony_141]

problem has not reoccurred