Can performance hogged system silently reap daemons such as cron?

[Ralph Grothe]

I have a heavily loaded db server (N4000) which presumably was so much swapping (vmstat reported >30 page outs on avg.) that even any login attempt would fail.
When I finally could login I realized that some important daemons weren't running.
Most notatbly cron, which I had to restart after removing many pending atjobs from the queue.
Also the diagmond wasn't running, which I realized when I wanted to have a glance at HW logs.
I don't even know yet which other vital daemons may have passed away.
Can it be that they were reaped owe to system overload?
What can one do to prevent from such unwanted daemon deceasure?

Regards
Ralph

[harry d brown jr]

Ralph,

Post your kernel parameters and swapinfo, I suspect that you are having memory and process issues.

live free or die
harry

[H.Merijn Brand (procura]

a (perl) script started from /etc/inittab using

reap:2345:respawn:...

[Ralph Grothe]

Thanks Harry for your offer.

The box has 8 GB physical RAM.

Attached you may find my tunables, and swapinfo

[A. Clay Stephenson]

Hi Ralph:

Before you do anything else, set timeslice to 10; you used that reallu dumb tuned parameter set; didn't you?

Timeslice set 10 to 1 can cause all sorts of problems including excessive context switches.

[Ralph Grothe]

Hi Clay,

yes, you're right in your assumption that I used the dumb OLTP parameter template for a db server, obviously not knowing better.
You think that the timeslice of 1 (what are the units, is it in seconds?) induces too many context switches?

[harry d brown jr]

Ralph,

You can usually reduce your dbc_max_pct to 10.

Most importantly, YOUR TIMESLICE is 1 !!!!!!!!!

Change it to 10!!!!!

Jack up your file limits and tty parameters. And look at your maxdsiz (& _64bit) and shmmax.

Here's a list of what I have set:

dbc_max_pct 10
dbc_min_pct 5
maxdsiz 0x40000000
maxdsiz_64bit 0x40000000
maxfiles 512
maxfiles_lim 2048
npty 200
nstrpty 150
shmmax 0x40000000
shmmni 2560
shmseg 120
timeslice 10


live free or die
harry

[harry d brown jr]

BAD RAPLH!!! As soon as I saw that 1 for timeslice it looked like you used the "template" garbage. Have fun :-))

live free or die
harry

[A. Clay Stephenson]

Hi Ralph:

Timeslice is allocated in units of 10 milliseconds ticks so that a setting of 10 means 100 msec. Stephen Ciullo (the best kernel guy I know) defines it something like this: "It's the maximum number of ticks that you ain't gonna get.". Your process is switched before it reaches this maximum. If set to 1, I'll give you one guess as to how many ticks you get. Your system is the perfect example of the effect of timeslice set to 1; as the system begins to get loaded it does context switching and very little else. You will also see bizarre semaphore problems and really strange things like your terminal arrow keys don't work very well. In that case, the process minght see the ESCAPE but not the following '[' character before a timeout so that it thinks it has seen two separate keypresses. The truly amazing thing about this parameter set is that it has persisted for so long as well as the terrible max_dbc_pct=50 default value. I keep hearing 'real soon now' but I've heard that for years.

Regards, Clay

[Ralph Grothe]

Clay, Harry,

many thanks for your valuable hints.
I will be trying to persuade our customer to built a new kernel during next system downtime.
Please be clement for not having assigned full points to your replies, but I am interested in hopefully further suggestions from others as well whom I don't want to distract by the magic rabbit ;-)
Just post some dummy reply so that I can give you full points later.