High load

[Consty]

In addition,
-I checked /vmunix and link date, they are all from october 2004
-There is traffic in the interconnect because the filesystems are mounted on node B and the programs (oracle processes) are running on node A.
-The "sysconfigtab" was restored at the version before last software installation.

Thanks

[Consty]

Hein van den Heuve,

I have attached the content of /var/adm/messages to help me analyse what is inside.

Regarding interconnect, what I have just been told is that when you start nodeA first then nodeB, the system halts, we should start nodeB first then nodeA (not normal), in addition as nodeB cannot support all the load the filesystems are held by nodeB and the applications are in nodeA. This seems not good for Oracle if SqlNet is not set. I do not know how TruCluster manage it.

Taking into account all this can you help me explain what is going on ?

Thanks
Consty

[Hein van den Heuvel]

The last boot entry for nodea shows:

Nov 21 10:39:27
emx0: Using console topology setting of : Loop
***** HARDWARE ERROR *****
status 0x80000000, 0xA8=0x000025bc, 0xAC=0x00000000: HW ERR:EBUS Parity Error
EMX DRIVER ERR: emx0: emx_log_adap_err - status 0x80000000, 0xA8=0x000025bc, 0xAC=0x00000000: HW ERR:EBUS Parity Error


Seems to me that node A lost connection to the SAN. The disk will be transparently served through node B... at a price though.

It woudl explain you have to boot B first, and considering that Oracle is running on A, and is likely to do the bulk of the disk IO this is bad.

You can explore further with cfsmgr and drd (Distributed Raw Device) tools, but I'd go after the hardware asap, calling in support as needed.

In the mean time, you may want to consider to swap the node assigments. Boot what always was node B as node A and visa versa, bringing the storage close to its main usage.

hth,
Hein.

[Consty]

Thanks Hein, Thanks all of you,
I'll check everything and let you know.
Regards
Consty

[Rob Leadbeater]

Hi Consty,

You would also be advised to check the various firmware revisions. If you do get HP Support involved, one of the first things they'll ask you to do is to get everything up to supported revisions.

From a quick flick through the messages files, you would need to update the firmware on you fibre HBAs, and the MSA1000.

The firmware flashing of the FCA2354 *may* solve the hardware error that Hein pointed out, although if you go back through that messages file, you'll see that this error has been happening for a *long* time - at least a couple of years if not longer. Swap the HBA.

The MSA1000 is also running a rather old firmware 4.32. You should probably upgrade this to at least 4.48, if not 5.20 which should give some performance boosts.

Cheers,

Rob

P.S. Don't forget to assign points

[Rob Leadbeater]

The firmware on the ES45s is also very old: 6.6. The latest is 7.3.

You can get this from here:

http://ftp.digital.com/pub/DEC/Alpha/firmware/index.html

Cheers,

Rob

[Consty]

Hi all,

What happened is that HP representative removed 4Gb memory from nodeA and put it in nodeB, so nodeB rebooted normaly, users worked the whole day without problem. The problem is not yet solved but people can work,we have to continue with investigation to bring the system to its previous state.
Just to remind that the system was working properly for many years since the day when the administrator installed some softwares in it, the kernel_idle process grew and it became difficult to work as usual. The first thing to do is to explain what happenened ...

Thanks again
Consty

[Consty]

Hi all,
Please help me analyse the output here attached.
Thanks
Consty

[Rob Leadbeater]

Hi,

Are you still having similar problems, or is this a new question ?

For more accurate diagnosis, you should install the latest version of sys_check. You're currently running V133, latest is V144, available here:

http://h30097.www3.hp.com/sys_check/

Cheers,

Rob

[Consty]

Hi,
The problem was not solved, as I said in a
previous message the RAM has been increased to allow them to work, but the problem is the same. /var is filled by mqueue and oracle's mail.
Thanks for help
Consty

[Rob Leadbeater]

> but the problem is the same. /var is
> filled by mqueue and oracle's mail.

I don't remember that as being the original problem.

Maybe you could let us know whether the problem with the HBA has been sorted, before moving on to something else...

Cheers,

Rob

[Consty]

Hi,
We stopped Node A which apparently had a hardware problem, we took it memory and increased Node B's to allow people to work. Users are working but there still exist the original problem that was slow response time, "kernel_idle" high, in addition /var is now filling quickly.
Thanks
Consty

[Michael Schulte zur Sur]

Hi Consty,

from what I see from the sys_check file, your system is ill-equipped with swap space. It is less than 50% of the main memory, resulting in bad usage of the main memory (more that 4GB free) and the UBC and virtual memory fighting for the resources. Add more swap space and check the requirements and suggestions, Oracle has for system parameters.

greetings,

Michael

[Michael Schulte zur Sur]

Strange problem, though the post arrived it did not change the last update.