Strange problem on New DL580G2...

[MARTIN_67]

Hi

I am facing a strange problem on all our DL580G2

Here is the situation:
We are running SAP ( DB and CI for those who know SAP) on a DL760 (4Gb and 7 procs)with windows NT

For the users we have 4 Applications Servers which are networked with the database (on the DL760)

So these Application Servers (DL580G2 5Gb and 3 Procs) are installed with W2kAdServer US .

The problem we have is that suddenly one of the AS stops running SAP (but still responds to ping ) and when we connect (via ILO) the server is still UP but we cannot do anything , the screen is there seems normal the mouse works but we cannot start anything....)

The only solution is to reboot and restart SAP

This append on every servers at least one time !!!

The 4 AS were functionning well for 10 days and then suddenly one "freeze" sunday and another one "freeze" today !!!!

We have no logs on the event viewer when this appends !!
and moreover we see no explanation of this problem !

we try to put one of this server on NT 4 to see if this is related on the OS ...

So if someone has any idea about this behaviour (hardware , software..)
please help us

Thanks

[Stuart Robinson]

We've experienced the same problem. Check out http://www.intel.com/support/platform/ht/os.htm. Intel say HyperThreading is not supported by them. We're disabling our Hyperthreading and waiting to see what happens.

[Stuart Robinson]

Disabling hyperthreading has had no joy. Our box is still unstable. We are now looking at a potential video driver issue.

In the SSD only ATI Rage IIC PCI drivers are supplied, but on DL580-G2 server the video card is an ATI Rage XL PCI. I have spoken to other contacts and ATI write a specific fix for the issue.

Current driver version is 5.0.2184.1 and the version my contacts have is 5.0.2195.4015.

[Stuart Robinson · Accepted solution]

Check out http://wwss1pro.compaq.com/support/reference_library/viewdocument.asp?source=EL020923_CW02.xml&dt=3&pid=-1 as well

[MARTIN_67]

Hi

I was out of the office the last two days... we also had disabled the HT on two of our servers to see what happend , as for now they are still ok.

But the last thing you say is very interesting, the serial number of our servers "says" that they have been produced the week 37 !! so they could possibly have the first criteria for the VRM problem , we will look in the data center to see if they have the second one.

Thanks for your help

More later

[Jeff Mathews]

I just wanted to note that the article by Intel reference above is for the Hyperthreading of P4 chips, and not the Xeon model.

Jeff

[Jakub Nowakowski]

We have similar problem with DL580. We have 3 x 1.6 Xeon / 6GB RAM / Win2K Advanced Server. After the console hangs we can ping the server. This server was produced in 46th week. We suspected HBA card/drivers, but it was a wrong trace. We found newer ATI drivers for the video card 5.00.2179.1 and 5.0.2195.5012. Can somebody comment this or recommend other solution. TIA.

[Jakub Nowakowski]

Ha!

It seems we solved our problem! There is a SQL Sever Enterprise installed on this server. For the testing purposes it is configured to use fixed memory size. The fixed size is pointing to the maximum possible size! Some additional applications (we used pinball to test the video adapter driver) are causing the problem. We changed for a while the SQL memory parameter to smaller fixed size. And we started dozens of applications. Everything worked O.K. Now we are testing this server with VA 7100, 2 HBAs and AutoPath installed. Nothing freezes!!!

[Stuart Robinson]

We have changed out servers to boot using VGA mode (/basevideo). Since doing this, the boxes have been stable.

Out ATI Rage XL driver is that which ships with MS Windows 2000 server, version 5.0.2184.1 dated 18/11/1999.

We have been advised to apply the Support Paq after any Windows service packs, but if the video driver is the issue, it will never get updated as it's not included in any Support Paq.

We have raised this issue, but have been advised that the updated driver isn't shipped in the Support Paq as the Microsoft driver is the correct one.

We have no plans to implement newer drivers from ATI though.

[Stuart Robinson]

I have discussed the SQL fixed memory issue with our DBAs and they have explained the following: -

By configuring the fixed memory to the maximum, the operating system cannot get memory back from SQL to run other applications, hence the server "hanging" when trying to run additional applications. They recommend that if you fix the memory, always leave yourself about 20% for the operating system to perform additional processes, and where possible don't have any other applications running on the server.

[MARTIN_67]

Hi Stuart,



I would like to thank you for your idea and your research....


You were right !!

Compaq has changed the VRM on all 4 AS we have. All those VRM's were version 3 ! and as it was noticed on the cutomer notice you gave me it has to be at least 4!!

So we change them for a version 5.


It is quite funny to notice that on version 3 there is 2 industrial capacitors and one which seems added from hand !!!

On version 5 there are 4 "industrial" capacitors !!

So i think these servers were certainly released without hard testing !! (but it seems like it will be more and more the case !)

The problem is that Compaq has not contacted his customers who buy these servers to change these VRM from his own and moreover wheh i tell them to check these VRM (telling them to check this notice they don't find!) they spend 2 weeks asking me for survey and so on!!!

One more time we had to raise the pressure to have the VRM changed for a problem they don't solve and that were not aware of!! (i speak for the french support!!)

nevertheless , i would like tothank you again

[Stuart Robinson]

Unfortunately, even after swapping the VRMs and the system board twice we are still experiencing issues. We are now getting a replacement server and sending our, obviously, faulty model back for extensive testing/diagnostics.

Ironically all our other G2 servers are working fine, maybe we just got the runt of the litter.

[MARTIN_67]

Hi,


i was thinking that this problem was solved but on monday one of the four server has freezed another time !!!

We also are about to change the I/O board and the procs on this server ...

If this does not help we will also asked for a replacement ...


Some servers of these series are really bugged !!

[Lars Eik]

We also seem to have a DL580 with ASR booting the server, or rather tries to boot. It ends in a pingable state where no other services, or the console is reachable. Just the blue screen (not bsod) that hangs like when you take a normal shutdown and the desktop and all is exited. The server seems to be produced very recently according to serialnumber info. Has hung three times in two days. Help.

[Wilson Sinclair]

We experienced similar problems with our DL580 G2 servers earlier in the year. In the end we had to have one of them swapped out and sent back to engineering for root analysis.

The latest response we hear is that HP engineering believe that the CA EL021206_CW01 actually does reference G2 as well as G1 servers.

This is despite the advisory stating "Note: This issue only affects the first generation of the ProLiant DL580 server; it does not affect ProLiant DL580 G2 servers.
"
Since moving to Smart Array 5300 2.92 firmware we have not seen the issue. Might be worth checking this out.

[Lance Carley]

We are also seeing this problem on 2 out of 4 DL580G2s we have.

They are DL580G2, 4x1.6Ghz and each 6GB RAM and we use the Integrated 5i controller.

We run 2 Lotus Domino partitions on each DL580G2 server, and from time to time (minimum of 24hrs) the servers hang. This leaves them pingable, and at console winlogon is sometimes still there and the mouse responds. The only resolution is a reboot. Unfortunately, we can't get a memory dump either as the machines have >2GB.

We are running on with the /basevideo boot.ini setting and are waiting to see what happens.


We are persuing this problem with HP in the UK and France and are thus far having no joy from them.

Lance

lance.carley@bnpparibas.com

[olivier_17]

My contrbution tu your problems...

I was confronted to boot problem a few week ago, on a 580G2 from a customer. After spending at least 2h30 on phone w/HP helpdesk (gsm...) and having 3 different people i received the info that there is problem with some memory board that cause erratic comportement on these server (boot or blue screen ...), near 1 lever there is a 4digit number (must be 0244) and there is a small picture near that digit ... ask helpdesk to send u that info or send me a mail if they cannot provide u the information
rem: i m in Europe, and i don t know if there is difference between our servers and other non-european ...

blairono@systemat.be

[James Grant_2]

Hi all,

Has anyone found a resolution to their problems?

We are experiencing the same symptoms but so far we are looking at the problem from a software angle.

We are running SAP (Central Instance) on our DL580 G2 clustered (Win2k Advanced). In the past we have found that the box will freeze up and only respond to pings -requiring a reboot. Microsoft and HP are looking at the issue and for us its as a result of Pool Paged Bytes running out. Once this occurs the box freezes up.

Since we have become aware of this we have instigated preventative reboots and monitored the pool paged bytes (memory object) value in perfmon. When it reaches around 190Mb we know the box will freeze up.

Very interested to hear how people have gone!

[James Grant_2]

Anyone got an update on their problems?

We had an HP engineer come out and take digital photos of our clustered node's memory boards as there have been issues with the board causing the machine to randomly lock up or ASR. This is not an official HP product advisory but there are still reported issues.

However, from the photos taken HP have concluded that the memory board in our SAP node does not have the markings of those that cause these problems.

Thanks.

[David C_3]

Unfortunately, there is not a lot of detail about your server configuration or the components installed in your server, so there is not much information to go on.

Have you made certain that all support software and firmwares are up to date?

First, I would suggest checking that all firmwares are up to date. This would include:

1) DL580 G2 system ROM (currently dated 2/4/2003)

2) Smart Array controller f/w (SA-530x should be at 2.92 minimum, SA-5i should be at 2.38 minimum)

3) Hard drive firmwares (various f/w revs depending on drive model)

Next, check to see that the latest support software for Windows 2000 is installed. The latest Proliant Support Software for Windows 2000 is version 6.30A.

If this does not help, can you provide a list any PCI cards or controllers that are installed in your server? What array controller and what slot, what NIC and what slot, etc?

Have you opened a call with HP support? Any success?

[Dinis Teixeira]

This is for shure a problem of equipment "QUALITY", we are facing. We are runnig Suse Linux 7.3, (that is a excyse for Compaq/HP, is not supported??). But this problem is found with Windows NT/2K, and RedHat Linux.
We must join, against HP/Compaq to force a solution, send link??s of this foruns to them, they must stop with the excuses!

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x815b35a43b46d71190080090279cd0f9,00.html

Dinis

[Victor Pendlebury]

Hi

I know that the latest Bios resolves an issue with the system freezing with large file transfert

http://h18004.www1.hp.com/support/files/server/us/download/18458.html

Hope it helps

Vic