HP BiDi Channel Components crashing spoolersv.exe on 2008 Terminal Server

[Michael Chung - Haas]

We are using latest version drivers, indicated as Server 2008 compatible, for the following printers:

LaserJet 9050DN, PCL6
LaserJet 4250, PCL5e
Color LaserJet 9500, PS

Printers are hosted on a print server running Server 2003 R2.

Terminal Server is running Windows Server 2008 (x86). The HP BiDi Channel Components are causing the print spooler to suddenly terminate on the 2008 Terminal Server. This does not happen on our 2003 Terminal Servers.

To make the issue more complicated, crashes are intermittent.

I've identified two HP DLL's are the culprits (there are application errors logged for these DLL's at the same time the print spooler crashes):

hpzpnp.dll
hpcpn5r1.dll

Here are the descriptions from the event log for each of the following DLL's above:

Faulting application spoolsv.exe, version 6.0.6001.18000, time stamp 0x4791956c, faulting module hpcpn5r1.dll, version 0.3.1537.48, time stamp 0x47f649ae, exception code 0xc0000005, fault offset 0x00010a65, process id 0x1b7c, application start time 0x01c8fc055030d3fa.

Faulting application spoolsv.exe, version 6.0.6001.18000, time stamp 0x4791956c, faulting module hpzpnp.dll, version 0.3.1537.18, time stamp 0x478e03b2, exception code 0xc0000005, fault offset 0x00006026, process id 0xac8, application start time 0x01c8fc0b9bf57f0e.

Has anyone seen this problem before? I have done an exhaustive search on the web and not found anything about it.

Thanks,
Michael Chung

[Jeremy Shoemaker]

I have the same problem.

Have you found a solution?

I am still searching.

Thanks,
Jeremy Shoemaker

[Jeremy Shoemaker]

I am testing a possible solution from here:

http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/Server/Windows_Server_2008/Q_23680409.html

"we had a problem with this in 2003, what we had to do is disable SNMP on all the printer. Go to each printer properties, click on ports, edit and uncheck "SNMP Status Enabled". I very well could be a bad driver causing the problem. "

Jeremy L. Shoemaker

[Michael Chung - Haas]

Unfortunately, I'm still trying to find a solution as well.

I've done the following so far with no success (spooler still crashes)
- Disabled Interactive Services Detection service
- Disabled Net Driver HPZ12 service (installed w/ HP printer drivers)
- Disabled Pml Driver HPZ12 service (installed w/ HP printer drivers)
- Turned off UAC
- Tried previous version of printer driver
- Tried different version (PCL5,PCL6,PS) of driver

HP BiDi seems to be a component of all HP print drivers. I'm at a loss. However I will give the SNMP a shot and see if that helps.

Thanks,
Michael

[Michael Chung - Haas]

No luck with SNMP and LPR Byte Counting; SNMP was already off on all of my printers, and changing the printer protocol to RAW disabled LPR byte counting.

[Roger Faucher]

Have you considered removing the printers and adding them back in one at a time to try to isolate it to a particular printer/driver?

Make a great day!

Roger

[SchwartzM]

We are seeing the same thing. Multiple printers, primarily Laserjet 3600N, but all are using the hpzpnp.dll driver.

Other than the specific printer models we are having the same problem -- spooler on Windows 2008 Enterprise x86 terminal servers (2 separate ones) are crashing. Print server is Windows 2003 Standard.

Looking for help.

[SchwartzM]

We found that the Universal printer driver that HP offers would solve these issues for every printer except the HP 3600N because the 3600 doesn't support PCL5,6, or PS.

[Roger Faucher]

Possibility?:

http://tinyurl.com/5p9qpn

Make a great day!

Roger

[Michael Chung - Haas]

We're switched to the HP Universal Print Driver 4.5 in Traditional Mode, however we're still getting spooler crashes. The guilty DLL is different now (due to UPD being used): HPCPN081.DLL.

However this DLL still points to the HP Bi-Di component. Furthermore, Microsoft Installer events (description: "Product: 32 Bit HP CIO Components Installer -- Configuration completed successfully") still occur on every user login.

Here are details from the application error indicating spoolsv.exe crash & HP DLL:

Faulting application spoolsv.exe, version 6.0.6001.18000, time stamp 0x4791956c, faulting module hpcpn081.dll, version 0.3.1537.65, time stamp 0x47d5a86e, exception code 0xc0000005, fault offset 0x00010ce5, process id 0x704, application start time 0x01c923b51d0be503.

HP has asked us to host printers from a Windows 2008 server using UPD. Will report back on results from this test shortly.

Unfortunately using 3rd party solutions (as suggested in one of the replies) is not an option at this time.

Thanks to all who've responded thus far!

[James Peluso]

Wow This is pretty much the same problem I'm having with a Terminal Server. My problem is the printers are installed locally on XPe thin clients. When they go through RDP the printer follows. We don't have the option of hosting the printers on 2008 server.

:(

[Michael Chung - Haas]

Latest update:

We've installed UPD 4.5 on our print server for several HP printers, and the number of spooler crashes has decreased significantly. We are slowly moving all of our printers to UPD.

James--I'm not sure I understand the problem you are having. You have printers installed locally on each Thin Client, but you do not want them mapped to the Terminal Server via RDP? Or do you need them mapped to Terminal Server via RDP, but spooler crashes on Terminal Server?

[James Peluso]

I have a TS farm of 2008 servers.
I also have about 70 thin clients running XPe. The thin clients have several printers associated to each of them based on location. The printers were installed using printui and the UPD PCL6 and UPD PCL5 drivers.

When the XPe stations rdp to the Terminal server they redirect their printers. Which is exactly what I'm looking for. The problem is each of the Terminal Servers spike to 100% utilization once the XPe stations map the printers. The service that's maxed out is the MSIexec service.

I can kill the MSIexec service and immediately the spoolsv service spikes the processor utilization to 100%. I kill the spoolsv service and everything calms down for a bit. I'm going to try to correlate the problems with logins and see if it's during the RDP printer redirection that this problem is occurring.

In the Event Log it shows: Windows Installer reconfigured the product. Product Name: 32 Bit HP CIO Components Installer. Product Version: 2.1.4. Product Language: 1033. Reconfiguration success or error status: 0.

During this problem the spooler service spikes as well causing the processor utilization to choke out at 100% and fatline my Vmware server. Each Terminal server currently has 4ghz and two cores allocated.

Oddly enough in the event log right after the msiexec service is logged this error is logged:
A device or program has requested attention. Device or application: C:\Windows\System32\spoolsv.exe. Message title: [printer name] (redirected 3) Properties.

After researching this Microsoft states that the Interactive services message is occurring because of an incompatible driver. The only drivers that are installed on my system are UPD PCL 6 and UPD PCL 5 both revision 4.5. Both of these drivers are supposed to be compatible with server 2008.

[Michael Chung - Haas]

Hi James,

Yes I'm experiencing the same problem as well--I didn't mention it in my original post because the spooler crashes were more pressing at the time.

I've got an open case with HP and Microsoft Premier support on this issue. Running processexplorer on the msiexec.exe events determined:

The HP MSI installer creates ~1.6 MILLION registry API calls as a result of over 4000 subkeys created in the registry for every printer mapped. This is what kills your system performance.
While there is no official fix (supposed to be fixed in UPD 4.7.2 (from what HP told me it is due December08).

However here is the work around for the MSINSTALLER service running every single time a printer is connected:

Stop the print spooler.

You need to 'zero-byte' the MSI files that keep running on both the Terminal Server and your thin clients (might not be necessary on thin clients since you are using RDP Printer mapping and not MS Point and Print, but better to do it anyways).

On XPe thin clients and Terminal Servers, go to the following directory:
c:\windows\system32\spool\drivers\w32x86\3\

find the following MSI files:
cioum32.msi
HPZBDI32.msi

FYI: if it is a x64 system, you will need to do this to the x64 printer driver folder. You will find corresponding 64-bit MSI installers for the CIO/BIDO components there.

Next step is to create a zero-byte file with the exact same name as the HP MSI's. You can create a zero byte TXT file and simply rename the file (make sure to rename the three letter extension as well).

Replace the original HP MSI's with the zerobyte MSI's you've 'created'.

Next step is to delete all the registry keys that were created by the HP MSI's.

On your thin clients, you need go in the the local registry and delete the following keys:
HKU\ .DEFAULT\Software\Hewlett-Packard

On your terminal servers you need to go into the local registry and delete the following TWO keys:

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Terminal Server\Install\Software\Hewlett-Packard
HKU\ .DEFAULT\Software\Hewlett-Packard

I *highly* recommend configuring GPO Client Side preferences to continuously delete these keys. These registry entries will pollute your the user profiles of customers logging into that terminal server; they can easily increase profile size by 30mb.

Restart the print spooler service.

Now, when a user logs into the terminal server, the HP driver will still attempt to run the MSI, however instead it will now find the zerobyte version you've put in. MSINSTALL will simply open file and process nothing.

Not the prettiest solution, but it seems to be working for us.

Let me know if that works for you as well.

[Michael Chung - Haas]

James--one more thing:

You might not find the HPZBDI32.MSI file--that's okay. This is the equivalent of the cioum32.msi file if you are running NON-UPD drivers.

The non-UPD drivers have the same problem in Terminal Services 2008, just a different MSI file.

[James Peluso]

Michael,

Wow! Thanks! I'm going to try and perform this after hours while everyone is not connected to the terminal server. Thank you for the suggestion. Please keep us posted on your progress with MS and HP. I'll keep my fingers crossed for those new UPD drivers that resolve this issue.

I appreciate your responses,
cheers,

James

[robert johansson]

From HP

"Citrix environment- processor spike in MSIEXEC results in slow performance: This issue will be resolved in an update to UPD v4.7 targeted for early 2009. Temporary workaround is to create a zero byte cioum32.msi file. Renaming the \Windows\system32\spool\drivers\x32x86\3\cioum32.msi file to cioum32.msi.bak. Open Notepad, go to FILE-SAVE AS, and save file \Windows\system32\spool\drivers\x32x86\3\cioum32.msi."

/R

[Threonine_1]

Thanks Michael for the best workaround on this issue! Thanks for the update Robert. If possible, please provide a source reference for that update from HP. Is that from an advisory or a support ticket or something else?

[Michael Chung - Haas]

Hey Threonine,

You can reference my case number with HP if you like: 3603286094

To my knowledge, there is no official public advisory from HP on this issue. These recommendations have only been given after support tickets were opened with HP.

I have another update as well...Thru HP support, I've received updated DLL files from HP for both discrete and UPD drivers. These DLL's fix the problems encountered with Micrsoft Installer processes when mapping a network printer. I've since:

1) removed the MSI-workaround that I've described in my previous posts above

2) implemented the updated DLL files from HP

I have not experienced any spooler crashes, nor have I seen any cases of the Microsoft Installer service running when it should not.

Here are the new versions that HP gave me:
UPD
cioum.dll (x86) version 1.1.2.21
cioum.dll (x64) version 1.2.2.21

Discrete
hpzbdi.dll (x86) version 1.1.2.51
hpzbdi.dll (x64) version 1.2.2.51

Now before anyone goes asking me for these DLL's...HP specifically requested that I not hand these out, these are not official updates so you need to get them directly from HP support.

This is what I was told by HP Team who gave me access to DLL's:

"I think the fastest way [to get access to DLL's] would be for a customer to call the support line (800-hp-invent), explain the issue as well as explain that they know that there are new cio and hpzbdi DLLs that address the issues they are having, and could they just have their case escalated. Thatâ s about the only way I know how to do it quickly. Once the support folks escalate it, it will then be escalated again by the advanced support engineers, and eventually make it to my small team."

Hopefully this information will help more of you out there

[robert johansson]

Contact HP support! there is a beta driver for 4.7 UPD. I've tested it in Citrix StressPrinters tools. It's kinda slow, but no more processor spike in MSIEXEC.

/R

[Michael Chung - Haas]

Thanks for the information Robert.

I think I will wait for the official release. At this point the updated DLL files have solved my problems and I'm hesitant to put any additional 'beta' components into production.

[Wim Van den Borre]

Michael,

Can you send me these files somehow?

I've already contacted HP 4 times and don't seem to get an answer.

Kind regards,

Wim

[Michael Chung - Haas]

Wim,

As I stated in my post, HP has specifically asked me *not* to distribute these DLL files. You need to go thru support. Do you have a case number? I'm willing to pass along your information to the support team if you can give me a case number.

In the meantime, use the MSI zero-byte workaround I described in my post above. This seemed to resolve all of the problems we were having as well (prior to installing the updated DLL files).

Thanks,
Michael

[Don Broyles]

Your workaround to zero out the msi files is working well on our Windows 2003 Terminal server. Thanks.

I have one question and I apologize if the answer should be obvious. How are you using gpo to delete the keys?

Don Broyles

[Michael Chung - Haas]

Don,

I used GPO Preference Client Side Extensions which were introduced with Vista/Server 2008. You can download updates for XP and Server 2003 as well to support the new features in GPO.

See this KB article:
http://support.microsoft.com/?kbid=943729

I've attached a screenshot as well to show you what I've done. It will delete the keys everytime policy is refreshed. I haven't had any problems with printing after implementing these preferences.