ML350 G5 PCI Bus Error

[ncas]

I have 2 ML350 g5's and 1 of them has restarted 2 times in the last 2 weeks. The error that is registered is PCI Bus Error (Slot 3, Bus 0, Device 0, Function 0). There is nothing in slot 3, never was. no BSOD. both are in production and the 2nd server has been just fine. both are configured the same and built with the same software CD's. OS is windows 2003 enterprise with 6GB RAM. dual processors.

the only microsoft stuff is the OS and it has all of the latest patches and fixes. they went into production in June and have run great till now. we did restart the problem server 4 days ago for updates and the second restart happened yesterday.

[juan quesada]

known issue.

Update the Network Drivers before replacing any components.

http://h20000.www2.hp.com/bizsupport/TechSupport/DriverDownload.jsp?lang=en&cc=us&prodNameId=3279711&taskId=135&prodTypeId=15351&prodSeriesId=1121586〈=en&cc=us

[ncas]

thanks - we updated the drivers and will wait and see if it runs for more than 2 weeks.

[ncas]

server just restarted again since the update of the network card.

the restart happens at about the same time of day 11:04,11:05,11:03.

i do see a message that talks about the processors being mismatched but everything i have found so far shows them matching and the same message is on the other server.

i'm out of ideas of what to look for next.

[KarloChacon]

hi ncas

can you run a survey in your server, you have to use HP insight diagnostics - survey tab

select Advance - All in drop menus
to get all device information specially which component is this one "Slot 3, Bus 0, Device 0, Function 0"

I hope you run all drivers updates using Proliant Support pack 8.1 not only NIC drivers...???

also try this
use Firmware CD 8.2(FW CD) and update all once you do that update after System ROM since the latest version for system ROM is not included in that 8.2 CD
so do both not only the CD OK

link for FW CD 8.2
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=15351&prodSeriesId=1121586&prodNameId=3279711&swEnvOID=1005&swLang=8&mode=2&taskId=135&swItem=MTX-3d1aaa0b48c04b628789e598d3


link for the latest system ROM
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=15351&prodSeriesId=1121586&prodNameId=3279711&swEnvOID=1005&swLang=8&mode=2&taskId=135&swItem=MTX-ef86b18284b64e3b96b70c48b0

try and let us know
regards

[ncas]

We ran the firmware cd, software pack, ROM updates, and windows updates while we were at it. We turned off ASR to see if we got a different result.

Everything ran good until this afternoon.

The system just stopped responding. The display output was nothing and we plugged in a USB mouse and the server didnâ t act like it saw it. All of the lights on the front of the server were green and the hard drive lights were blinking.

The message we see when the system restarted was PCI Bus Error (Slot 0, Bus 0, Device 0, Function 0). This is different than the original messages we saw. We also see a message about a processor registry entry that has an error that it canâ t read. I found the location and exported that key but I donâ t know how to read what it contains.

This system can not have any downtime it is a server used for dictation at a large facility so we are under a lot of pressure to make sure this does not happen again.

[KarloChacon]

hi

so now message is this "PCI Bus Error (Slot 0, Bus 0, Device 0, Function 0"

it changed? right?

let me ask you have you called HP?

[ncas]

yes it changed (was always slot 3 before)

we are calling hp today.

[juan quesada]

new bios was released with this fix

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=15351&prodSeriesId=1121586&prodNameId=3279711&swEnvOID=1005&swLang=8&mode=2&taskId=135&swItem=MTX-ef86b18284b64e3b96b70c48b0

[ncas]

we already applied the bios update

we are opening a ticket with HP later today

[Jim_214]

NCAS,

I am having a similar issue with a DL380G5 that is running Windows 2003 SP2 and Cisco UNITY. Did you ever get a resolution to this issue with your call to HP?

[ncas]

Jim,

we are still having a problem. HP replaced the system board on 9/23 and it ran for 20 days, with a planned restart oct 1 and now its happened 4 times this week. last being 3 hours ago. we think we have traced it to a system load issue, but its had more work other days and never failed. we ran all the updates on every piece harware and software we could find. still have not seen a dmp file created by microsoft. we turned ASR off again.

look like replacement is our next option.

[Jim_214]

Thank you for the quick response, NCAS. This server has had this error a couple of times, including once during 'burnin' before the OS, etc. was loaded so I think we may have a 'lemon' that is going to get replaced.

[r14edge]

Hello NCAS,

I'm a bit in the same situation as you. I have a ML350 G5 in production that gives me PCI bus errors and headache. I've updated my system firmware and drivers, and I still got this issue. After calling HP, they suggested 2 possibilities.

1- My server might run a third party software that makes the server crash. To take actions, I move the only third party software that might cause crashs and its been 4 days now without any reboot. That will make sense if the load theory you describe is real.

2- The second point HP suggest me was the OS itself. I'm also running a W2K3 enterprise server with 9 GB of RAM and a single XEON processor. They suggest running a check on the system files to replace corrumped ones. So far, I havent done nothing on that side.

Just a quick question, did HP told you they will replace your system or it's you who will replace it? If its not too impolite, can you tell me when did you bought your system?


Thank you for your help,

[ncas]

System replacement will be on our own.

We have been running at a reduced capacity and have slowly adding more load to try and find a failure point. We did add a dual port network card since the slot number changed from 3 to 0 after we flashed the onboard network card. I guess out of desperation more than anything else.

We were told the 3rd party software possibility also but removing what we have installed would make the server useless to the customer.

We are operating on 4GB of memory, disabled DEP and PAE. We had 6GB at the start.

Our goal is to have it run for 30 days without a restart and we are on day 6.

Purchased the servers in April of this year.

[r14edge]

Thanks for your reply NCAS. The third party solution didn't not work for us. In a desperate move, we had HP changed our system board today althought I doubt it will work. The other possibility they mention was the Windows itself. How to debug a OS when all you have is that PCI bus error message?

So it's back to day 1 in our case. And we will investigate the OS option. If we found a solution to this, I will let you know.

Thanks

[Mike Jakemer]

I have the same issue. ML350 G5, exact error you are getting. (PCI BUS ERROR (Slot 3, Bus 0, Device 30, Function 0)) Have been having issue sporadically for about a year. Happens many times then goes away for months then comes back again. HP has replaced Mainboard, told us to update BIOS, FIRMWARE, DRIVERS...to no avail. Been on phone with them today for over 2 hours so far. This server could not be any more simple. Windows 2003 R2 32 Bit, 3 GB, RAID 5 3x72's, Exchange server 2003, nothing fancy. All generalized error messages point back to Hardware Malfunction...even BSOD indicates Hardware but HP. Any luck finding a solution?

[Dana Racine 1]

I was getting this error as well. It is a software error in the chipset drives, NOT a hardware error. I had called HP, and had to really bust them because they kept telling me to run diags, but I told them that this was for a device that didn't show up in Insight Manager "PCI Devices". Then I had the guy look in their internal "Red" database of internal docs, and type in the PCI device info, and he sent me a copy of an article saying that the Intel 5000P chipset drivers need to be updated. Here is the link they sent (this is for 2003, you will have to search for 2008):

. Click here to go to the Intel Download Center Intel Chipset Software Installation Utility webpage (http://downloadcenter.intel.com/filter_results.aspx?strTypes=all&ProductID=816&OSFullName=Windows+Server*+2003+Standard+Edition〈=eng&strOSs=94&submit=Go%21)

[ghassen]

hi
update bios ,if you have the same issue replace the mother board.
REGs

[stacy bolton]

Are there any updates to this post? I am receiving the same error - PCI Bus Error (Slot 0, Bus 0, Device 0, Function 0) - on 2 separate ML350 G5's

[Doof]

Hi,
I was getting (Slot 0, Bus 0, Device 0, Function 0) with random ASRs about once a day or two. Updated all the FW to the most current. After hearing that replacing the MB didn't work for some, I replaced about the only thing you can on a Blade, the backplane and the Cache module. Long story short, it was the cache module. If you have a spare blade, try swapping the module.