program dead loop

[Son dam bi]

My server was crashed yesterday , I found that the reason is because a program ( developed by our programmer ) have problem , due to the data is incorrect , the program is looping then used up all system memory , it make the system crashed , can advise is there any way that I prevent it happens again ?


ps. as there are too many programs is ruuning , I can't ask them to check all programs.

[Suraj K Sankari]

HI,
>>the program is looping then used up all system memory , it make the system crashed , can advise is there any way that I prevent it happens again ?

Better you can ask your programmer to monitor there program when they test....
and before implementing they should test there program 2 3 times....
other thing you can extend your memory...

Suraj

[Hein van den Heuvel · Accepted solution]

Son dam bi>> My server was crashed yesterday , I found that the reason is because a program ( developed by our programmer ) have problem

I beg to differ. If a normal, but incorrect, program can crash a server, than that server is poorly managed. It suggests that all the knobs that are there to protect you were set to 'eleven' for that extra push over the cliff (Spinal Tap. http://www.youtube.com/watch?v=EbVKWCpNFhY )

I'm not arguing that the program is not broken, and should not be fixed. But QA testing should have stopped it production system settings may have been overly permissive without the resources to back that up.

Check out 'maxdsiz' and friends, in context of the processes allowed and swap space allocated.

You may want to check the system when it is running normally and record the various memory usages (RSS/VSS from Glance or UNIX95= ps -ef... )

>> rogram is looping then used up all system memory , it make the system crashed

If that is really true then the guilty party will be visible in the crash dump.

>> ps. as there are too many programs is ruuning , I can't ask them to check all programs.

There are likely to have been early signs and even console warning. Keep an eye on tools like TOP and Glance to see whether a process gets carried away using SYSTEM time and memory.


Suraj>> thing you can extend your memory...

au contraire, mon amie.
That's just silly. Unless you'll mean you'll have a little more time for the hurting to happen before postponed but inevitable crash.

Cheers,
Hein.

[Venkatesh BL]

> the program is looping then used up all system memory

How did you confirm this? Can you show the output? I am not sure that analysis is correct.

[Dennis Handly]

As Hein said, a 64 bit process can quickly use up all of swap if maxdsiz_64bit is set too high. A 32 bit process would have to work harder, unless it has a fork bomb.

>Venkatesh: How did you confirm this? Can you show the output?

Well, if there are messages about lack of swap space, that would be a clue.
Monitoring "swapinfo -tam" would indicate the problem but not the program.