HP 9000 and HP e3000 Servers
cancel
Showing results for 
Search instead for 
Did you mean: 

V-2600 CPU 0 maxed out

Bryan D. Quinn
Respected Contributor

V-2600 CPU 0 maxed out

We have a V-2600 running SAP/Oracle. On Monday morning the OS hung, meaning I could put in my user id but it would not come back and ask for my password. It was just frozen. I could log into SAP and it seemed to be running fine. The OS was just frozen. My first thought was that something ate up the memory in the box (we have 28GB).

This scenario is not a new one, in the seven years I have been here this exact scenario has happened about five times. Always just after midnight on Sunday night/Monday morning, with the exception of one time when it happened at 8am on Monday morning. We reboot all our UNIX servers every Sunday night at 7:30pm, so this is happening just a few hours after a reboot.

Ok, this time when it happened we did some snooping from inside of SAP. Looking at some OS information and noticed that the following process had CPU 0 pegged out.

/opt/java/bin/../bin/PA_RISC/native_threads/jre -lazyswap -n

It seems to me that one of the previous times that this happened we saw another process that was eating a CPU, whether it was CPU 0 I do not recall.

So, my question is for all you that might have more knowledge of the V's and it's architecture, would it be possible that if you pegged out CPU 0 it would hang the box? Is there some magic about CPU 0 that I am not aware of?

Of course this is a long shot, that process might not have anything to do with the system hanging. So, based on that if anybody sees anything that I might be missing in this scenario, please let me know.

Any help would be greatly GREATLY appreciated.

Thanks,
-Bryan
12 REPLIES
Torsten.
Acclaimed Contributor

Re: V-2600 CPU 0 maxed out

The very first idea I have if I read about java in combination with an old server is: patches. IMHO you should check (e.g. a patch session on ITRC or via SWA - software assistant) if your system needs patches.

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Bryan D. Quinn
Respected Contributor

Re: V-2600 CPU 0 maxed out

Hello Torsten,

I think we are up to date there...or should I say as up to date as we can get on an old box. I will definitely check that out.

I will respond with what I found.

thanks,
-Bryan
Bill Hassell
Honored Contributor

Re: V-2600 CPU 0 maxed out

Each CPU in HP-UX is equal except that CPU 0 is usually the Monarch processor and becomes the control center for other CPUs when they run kernel code. There is nothing magic about using 100% of any CPU -- that will not hang the system. But as mentioned, patches should be up to date *and* Java should also be updated. Also check on swap space (swapinfo -tam) to make sure you are not reaching the limits. 28GB is a lot of space but Java and SAP and Oracle can eat this space very rapidly if not constrained. Also check your syslog for any problems.


Bill Hassell, sysadmin
Bryan D. Quinn
Respected Contributor

Re: V-2600 CPU 0 maxed out

Torsten and Bill I appreciate your replies.

Once again, this morning we had the same problem. And once again the process eating up the CPU was:

/opt/java/bin/../bin/PA_RISC/native_threads/jre -lazyswap -n

I am not sure what Java is used for, other than the fact that we had to install it to do the Oracle install.

Does anyone know what purpose this process would serve on an Oracle/SAP box, other than to do an install or upgrade?

Thanks,
-Bryan
Torsten.
Acclaimed Contributor

Re: V-2600 CPU 0 maxed out

No idea what it is used for, but I still think you should check for patches:

http://h18012.www1.hp.com/java/patches/index.html

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Bryan D. Quinn
Respected Contributor

Re: V-2600 CPU 0 maxed out

Thanks Torsten

I will check the patches again. Make sure I did not miss something.

Thanks,

-Bryan
Bryan D. Quinn
Respected Contributor

Re: V-2600 CPU 0 maxed out

Further investigating this problem, I found that the parent process for the above mentioned process is:

/bin/ksh ./emShell.ksh EM /opt/SiteNetML em.cfg

Which is our Liebert monitoring utility. Which I know nothing about because I did not install it, it was installed before my time.
Chris Vail
Honored Contributor

Re: V-2600 CPU 0 maxed out

I also ran an 2600 for a couple of years and experienced the same problem. However, I left a 'back door' in it for just such an emergency. The back door in this case was a .rhosts file. I found that even a thoroughly hung system would frequently respond to an RPC request.

When the system hung, I could login to another system and force a reboot:
remsh $HOST "shutdown -ry 0"

Please note that at one point I typed
remsh $HOST|shutdown -ry 0

This was one of the biggest fat-finger events I'd ever done.
Bryan D. Quinn
Respected Contributor

Re: V-2600 CPU 0 maxed out

Thanks Chris for the information.

Did you ever resolve the issue?

Did you figure out what caused the process to peg out like that?

Thanks,
-Bryan
Chris Vail
Honored Contributor

Re: V-2600 CPU 0 maxed out

Yes I do know what caused the system to max out, and it is a situation that no Sysadmin should ever have to deal with again....one that belongs in the 'BAD ARCHITECTURE IDEAS HALL OF SHAME".

We hired Anderson Consulting (those lovely people who brought you ENRON) now known as Accenture to write our transaction clearing house application. They chose software from a company they owned that was written in Visual Basic. They wrote a Unix wrapper for VB and put it on the 2600. Microsoft software is written for single-processor, low memory environments. When a resource becomes available in a M$ environment, the M$ OS grabs that resource and uses it. IOW: the product does not play well with others. In this case, the application grabbed every CPU, every buffer, every kilobyte of RAM, every IO channel, every controller, every ethernet card. It used so many system resources for this one application that there were no clock cycles left over to do the relatively unimportant job of operating the computer.

Every time they ran this application, the 2600 rolled over and played dead. No, it wasn't 'playing dead', it WAS dead. VB killed it. This is when I learned to put in the .rhosts back door. Of course, VB had seized every IO channel, and then complained that there was no disks! The back door let me reboot the computer and restart the database. Once they started the application, it would die again. This happened every day for a month before somebody in management finally got hit with the clue bat.

We then moved the application back to where it belonged...on a desktop system, and just ran Oracle on the 2600. All was good and happy in the world.

The moral of the story is simple: 1) NO M$ APPLICATIONS ON UNIX SYSTEMS, and 2) WHY ARE YOU PUTTING AN APPLICATION AND ITS DATABASE ON THE SAME SERVER TO BEGIN WITH?

That was in 2001 and I'm still mad about it.


Chris
Bryan D. Quinn
Respected Contributor

Re: V-2600 CPU 0 maxed out

Oh man, that sucks.

I have sent a ticket to Liebert, since I know that Site Net Multi Link is spawning the java process that is suspicious. I am not sure what is going on. Maybe they will have some insight to possible issues with their software. Our box is pretty stagnate and has been for many years. We are a very conservative shop and just keep the box humming. So nothing has changed to cause this problem.

I have definitely documented your note about remsh, I will try this the next we have a hang.

Thanks for your help.

Hopefully I will get to the bottom of this soon.

-Bryan
Torsten.
Acclaimed Contributor

Re: V-2600 CPU 0 maxed out

Chris, this is exactly the reason why you need such people - nobody else can have such ideas - develop something like a framework to simulate something like a PC just to run something like a script ... ;-)

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!