HP BiDi Channel Components crashing spoolersv.exe on 2008 Terminal Server

[James Peluso]

Wow This is pretty much the same problem I'm having with a Terminal Server. My problem is the printers are installed locally on XPe thin clients. When they go through RDP the printer follows. We don't have the option of hosting the printers on 2008 server.

:(

[Michael Chung - Haas]

Latest update:

We've installed UPD 4.5 on our print server for several HP printers, and the number of spooler crashes has decreased significantly. We are slowly moving all of our printers to UPD.

James--I'm not sure I understand the problem you are having. You have printers installed locally on each Thin Client, but you do not want them mapped to the Terminal Server via RDP? Or do you need them mapped to Terminal Server via RDP, but spooler crashes on Terminal Server?

[James Peluso]

I have a TS farm of 2008 servers.
I also have about 70 thin clients running XPe. The thin clients have several printers associated to each of them based on location. The printers were installed using printui and the UPD PCL6 and UPD PCL5 drivers.

When the XPe stations rdp to the Terminal server they redirect their printers. Which is exactly what I'm looking for. The problem is each of the Terminal Servers spike to 100% utilization once the XPe stations map the printers. The service that's maxed out is the MSIexec service.

I can kill the MSIexec service and immediately the spoolsv service spikes the processor utilization to 100%. I kill the spoolsv service and everything calms down for a bit. I'm going to try to correlate the problems with logins and see if it's during the RDP printer redirection that this problem is occurring.

In the Event Log it shows: Windows Installer reconfigured the product. Product Name: 32 Bit HP CIO Components Installer. Product Version: 2.1.4. Product Language: 1033. Reconfiguration success or error status: 0.

During this problem the spooler service spikes as well causing the processor utilization to choke out at 100% and fatline my Vmware server. Each Terminal server currently has 4ghz and two cores allocated.

Oddly enough in the event log right after the msiexec service is logged this error is logged:
A device or program has requested attention. Device or application: C:\Windows\System32\spoolsv.exe. Message title: [printer name] (redirected 3) Properties.

After researching this Microsoft states that the Interactive services message is occurring because of an incompatible driver. The only drivers that are installed on my system are UPD PCL 6 and UPD PCL 5 both revision 4.5. Both of these drivers are supposed to be compatible with server 2008.

[Michael Chung - Haas]

Hi James,

Yes I'm experiencing the same problem as well--I didn't mention it in my original post because the spooler crashes were more pressing at the time.

I've got an open case with HP and Microsoft Premier support on this issue. Running processexplorer on the msiexec.exe events determined:

The HP MSI installer creates ~1.6 MILLION registry API calls as a result of over 4000 subkeys created in the registry for every printer mapped. This is what kills your system performance.
While there is no official fix (supposed to be fixed in UPD 4.7.2 (from what HP told me it is due December08).

However here is the work around for the MSINSTALLER service running every single time a printer is connected:

Stop the print spooler.

You need to 'zero-byte' the MSI files that keep running on both the Terminal Server and your thin clients (might not be necessary on thin clients since you are using RDP Printer mapping and not MS Point and Print, but better to do it anyways).

On XPe thin clients and Terminal Servers, go to the following directory:
c:\windows\system32\spool\drivers\w32x86\3\

find the following MSI files:
cioum32.msi
HPZBDI32.msi

FYI: if it is a x64 system, you will need to do this to the x64 printer driver folder. You will find corresponding 64-bit MSI installers for the CIO/BIDO components there.

Next step is to create a zero-byte file with the exact same name as the HP MSI's. You can create a zero byte TXT file and simply rename the file (make sure to rename the three letter extension as well).

Replace the original HP MSI's with the zerobyte MSI's you've 'created'.

Next step is to delete all the registry keys that were created by the HP MSI's.

On your thin clients, you need go in the the local registry and delete the following keys:
HKU\ .DEFAULT\Software\Hewlett-Packard

On your terminal servers you need to go into the local registry and delete the following TWO keys:

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Terminal Server\Install\Software\Hewlett-Packard
HKU\ .DEFAULT\Software\Hewlett-Packard

I *highly* recommend configuring GPO Client Side preferences to continuously delete these keys. These registry entries will pollute your the user profiles of customers logging into that terminal server; they can easily increase profile size by 30mb.

Restart the print spooler service.

Now, when a user logs into the terminal server, the HP driver will still attempt to run the MSI, however instead it will now find the zerobyte version you've put in. MSINSTALL will simply open file and process nothing.

Not the prettiest solution, but it seems to be working for us.

Let me know if that works for you as well.

[Michael Chung - Haas]

James--one more thing:

You might not find the HPZBDI32.MSI file--that's okay. This is the equivalent of the cioum32.msi file if you are running NON-UPD drivers.

The non-UPD drivers have the same problem in Terminal Services 2008, just a different MSI file.

[James Peluso]

Michael,

Wow! Thanks! I'm going to try and perform this after hours while everyone is not connected to the terminal server. Thank you for the suggestion. Please keep us posted on your progress with MS and HP. I'll keep my fingers crossed for those new UPD drivers that resolve this issue.

I appreciate your responses,
cheers,

James

[robert johansson]

From HP

"Citrix environment- processor spike in MSIEXEC results in slow performance: This issue will be resolved in an update to UPD v4.7 targeted for early 2009. Temporary workaround is to create a zero byte cioum32.msi file. Renaming the \Windows\system32\spool\drivers\x32x86\3\cioum32.msi file to cioum32.msi.bak. Open Notepad, go to FILE-SAVE AS, and save file \Windows\system32\spool\drivers\x32x86\3\cioum32.msi."

/R

[Threonine_1]

Thanks Michael for the best workaround on this issue! Thanks for the update Robert. If possible, please provide a source reference for that update from HP. Is that from an advisory or a support ticket or something else?

[Michael Chung - Haas]

Hey Threonine,

You can reference my case number with HP if you like: 3603286094

To my knowledge, there is no official public advisory from HP on this issue. These recommendations have only been given after support tickets were opened with HP.

I have another update as well...Thru HP support, I've received updated DLL files from HP for both discrete and UPD drivers. These DLL's fix the problems encountered with Micrsoft Installer processes when mapping a network printer. I've since:

1) removed the MSI-workaround that I've described in my previous posts above

2) implemented the updated DLL files from HP

I have not experienced any spooler crashes, nor have I seen any cases of the Microsoft Installer service running when it should not.

Here are the new versions that HP gave me:
UPD
cioum.dll (x86) version 1.1.2.21
cioum.dll (x64) version 1.2.2.21

Discrete
hpzbdi.dll (x86) version 1.1.2.51
hpzbdi.dll (x64) version 1.2.2.51

Now before anyone goes asking me for these DLL's...HP specifically requested that I not hand these out, these are not official updates so you need to get them directly from HP support.

This is what I was told by HP Team who gave me access to DLL's:

"I think the fastest way [to get access to DLL's] would be for a customer to call the support line (800-hp-invent), explain the issue as well as explain that they know that there are new cio and hpzbdi DLLs that address the issues they are having, and could they just have their case escalated. Thatâ s about the only way I know how to do it quickly. Once the support folks escalate it, it will then be escalated again by the advanced support engineers, and eventually make it to my small team."

Hopefully this information will help more of you out there

[robert johansson]

Contact HP support! there is a beta driver for 4.7 UPD. I've tested it in Citrix StressPrinters tools. It's kinda slow, but no more processor spike in MSIEXEC.

/R