Communications with HP print server disrupted

[Michael Sillers]

We have many printers running from HP-UX with Jetdirect print servers. Until last Wednesday to the best I can determine, all was fine. Since then we've had a lot of trouble with repeated print jobs and delays. Many of our printers produced serial labels so the repeated print jobs can really be troublesome. Sometimes labels or pages of print outs are missed if we turn of job recovery. We've switched many to windows LPD and fed the print jobs through there which prevents the repeated print jobs but not the delay.

I've had out net administrator look for unusual traffic or anything else fishy and he claims all looks fine.

Users are getting upset. Does anyone have any ideas?

Thanks.

[Brad_Edwards]

I have a very similar problem occurring right now. We have several HP-UX application servers with standard UNIX/LP remote queues sending to a single virtual HP-UX UNIX/LP print server. The problem we are seeing seems to occur during the transfer from one machine to another. When a queue on an application server has multiple jobs pending the jobs take forever to print. Once the first one goes, usually 2-4 go with it, then the jobs stop again. If I disable, enable, reject, and accept the queue on the application server, sometimes I can get another 2 or 3 to go through before it hangs again.

I've tried restarting the inetd daemon on the virtual print server which seems to have the same effect as disable/reject/accept/enable. A few jobs will go through and then the process stops again.

If I leave the queue alone, the jobs will eventually go through, but the delays are being reported of 30 minutes or more. It's as if the application servers are waiting for an acknowledgement from the print server which never comes. Then the connection times out and another job or two goes through and then the application server's scheduler waits again for another acknowledgement which never comes. When we reset inetd or do the disable/reject/enable/accept trick, we're re-initializing that communication, but it's only a temporary fix.

Our network team cannot find anything that would be blocking traffic, so I'm left believing it's something blocking the acknowledgements on the server itself. The UNIX/LP application (scheduler) hasn't received an update in I don't know how many years, but this is a relatively new problem. Any help you can provide would be GREATLY appreciated!

[Michael Sillers]

Sounds like you've got it even worse than we do, Brad. Can you trace the start point to any particular day or date?

I did check the patches and we already have the latest which was released - I think - 2009.

[Brad_Edwards]

We may have two separate issues, but I have not been able to identify anything that was changed in our environment when this started happening for us (first indication showed up for us on 5/24/2011).

Our issue is very bizarre. It does not appear to be a network issue directly. When print jobs transfer, they all transfer quickly. I know it isn't related to the sending servers, because they are sending print just fine to other print servers in the environment without issue. The problem can be isolated to the point between when the LP Scheduler kicks off the /usr/sbin/rlp command and when the rlpdaemon on the print server is triggered. The log files on the sending server show the delay (1-3 minutes) between the FIFO entries and the CHILD entries, but none of the logs on the print server show any delay -- when print jobs hit the server, they're printed right out.

When there is a delay, the print jobs don't even show up on the print server itself. They just wait in the queue on the sending server.

There's something else, too. I've had multiple command windows open to one of our sending servers while this problem has been occurring. When it starts, I try running various commands such as "lpstat -oprinter" and "rlpstat -dprinter". All of these commands will hang. However, when the jobs are finally "released", ALL the commands respond simultaneously. Jobs will continue to flow for a while, then there's another pause.

I have had to redirect print from the print server having a problem to other print servers in the enterprise, which is sufficient as a workaround. This allowed me to have the virtual server rebooted. This had no effect, so I tried rebooting the entire virtual host. This also has not had any effect. I don't even know what to check at this point.

[Michael Mike Reaser]

What version(s) of HP-UX are you each running, and have you ensured that you've got all of the current lp-related patches installed? A Google of "lpstat hangs hp ux" brought back 3,000 results, most of which pointed to "install patches, watch the problems go away".

[Brad_Edwards]

Both servers are HP-UX ia64 virtual server running B.11.23 which didn't have the latest patches until today. Installed the most recent patch (the only one missing) and the problem remains. All sending servers are having the same problem, though, and they're various patch levels.

[Wilfred Chau_1]

Do you have any local print server residing close to the printers?

If you do, try
1) move all print queues to a fast disks on your hpux server
2) send all print jobs to the local print server

this is how we fixed our printing issue with delays and repeated print labels.

[Brad_Edwards]

Unfortunately, this print server already is in the area closest to the printers, but I don't even thing the problem has anything to do with the printers themselves. I'm working with the HPRC on this case, and here's how I described the problem for them:

---

First, this problem is happening with all printers on the print server, regardless of the printer status. In fact, when the delay occurs the scheduler on the print server hasnâ t even received the files from the sending system yet â nothing appears in the LP log file. At the point of the delay, the spool files remain in the remote queue on the sending system. The problem is very specifically between the point that files transfer between the sending server and the print server. After files arrive on the print server, they are printed immediately without issue.

Not only do the spool files wait to deliver to the print server during these delays, but even commands like â lpstat â oprinterâ and â rlpstat â dprinterâ also hang. After the delay passes (about 2 or 3 minutes) then, all of a sudden, several jobs in the queue will be sent to the print server, and any of the commands that were also hanging (like lpstat or rlpstat) will simultaneously respond. The logs on the sending server show the delay between when the files are submitted to the scheduler (the FIFO entries), and when they actually leave the system (the CHILD entries), but the logs on the print server do not show anything until the files start moving after the delay â this includes even the syslog.log message indicating the connection made from the sending server.

The delay can also occur when no spool files are in the queue. We can detect this by simply running â lpstat â oprinterâ or â rlpstat â dprinterâ on the sending server, and the command will hang when the delay is present. When the delay passes, the commands return as normal. When there is no delay present, these commands return immediately.

From the information above it would seem the there is a network issue between the sending server and the print server, but during one of these delays I was able to telnet to port 515 on the print server with no problem, so the network appears to be clear. I think we need to focus on the handshake between the /usr/sbin/rlp binary on the sending server and the inetd daemon on the print server. Any information or assistance you can provide would be greatly appreciated. Thank you!
-----

[Michael Sillers]

In my case, I'm running 11.11 and have already checked that I have the latest patches. lpstat has always taken a long time because our users have a tendancy to cut power to the print servers - the few that can't be local anyway - I've moved as many as possible locally.

The print jobs seem to hang in the queue but I can't be entirely sure as getting users to call when they are going to print is problematic. Anytime I try to dupliacte the problem, there seems to be no delay.