Can performance hogged system silently reap daemons such as cron?

[Ralph Grothe]

I have a heavily loaded db server (N4000) which presumably was so much swapping (vmstat reported >30 page outs on avg.) that even any login attempt would fail.
When I finally could login I realized that some important daemons weren't running.
Most notatbly cron, which I had to restart after removing many pending atjobs from the queue.
Also the diagmond wasn't running, which I realized when I wanted to have a glance at HW logs.
I don't even know yet which other vital daemons may have passed away.
Can it be that they were reaped owe to system overload?
What can one do to prevent from such unwanted daemon deceasure?

Regards
Ralph

[harry d brown jr]

Ralph,

Post your kernel parameters and swapinfo, I suspect that you are having memory and process issues.

live free or die
harry

[H.Merijn Brand (procura]

a (perl) script started from /etc/inittab using

reap:2345:respawn:...

[Ralph Grothe]

Thanks Harry for your offer.

The box has 8 GB physical RAM.

Attached you may find my tunables, and swapinfo

[A. Clay Stephenson]

Hi Ralph:

Before you do anything else, set timeslice to 10; you used that reallu dumb tuned parameter set; didn't you?

Timeslice set 10 to 1 can cause all sorts of problems including excessive context switches.

[Ralph Grothe]

Hi Clay,

yes, you're right in your assumption that I used the dumb OLTP parameter template for a db server, obviously not knowing better.
You think that the timeslice of 1 (what are the units, is it in seconds?) induces too many context switches?

[harry d brown jr]

Ralph,

You can usually reduce your dbc_max_pct to 10.

Most importantly, YOUR TIMESLICE is 1 !!!!!!!!!

Change it to 10!!!!!

Jack up your file limits and tty parameters. And look at your maxdsiz (& _64bit) and shmmax.

Here's a list of what I have set:

dbc_max_pct 10
dbc_min_pct 5
maxdsiz 0x40000000
maxdsiz_64bit 0x40000000
maxfiles 512
maxfiles_lim 2048
npty 200
nstrpty 150
shmmax 0x40000000
shmmni 2560
shmseg 120
timeslice 10


live free or die
harry

[harry d brown jr]

BAD RAPLH!!! As soon as I saw that 1 for timeslice it looked like you used the "template" garbage. Have fun :-))

live free or die
harry

[A. Clay Stephenson]

Hi Ralph:

Timeslice is allocated in units of 10 milliseconds ticks so that a setting of 10 means 100 msec. Stephen Ciullo (the best kernel guy I know) defines it something like this: "It's the maximum number of ticks that you ain't gonna get.". Your process is switched before it reaches this maximum. If set to 1, I'll give you one guess as to how many ticks you get. Your system is the perfect example of the effect of timeslice set to 1; as the system begins to get loaded it does context switching and very little else. You will also see bizarre semaphore problems and really strange things like your terminal arrow keys don't work very well. In that case, the process minght see the ESCAPE but not the following '[' character before a timeout so that it thinks it has seen two separate keypresses. The truly amazing thing about this parameter set is that it has persisted for so long as well as the terrible max_dbc_pct=50 default value. I keep hearing 'real soon now' but I've heard that for years.

Regards, Clay

[Ralph Grothe]

Clay, Harry,

many thanks for your valuable hints.
I will be trying to persuade our customer to built a new kernel during next system downtime.
Please be clement for not having assigned full points to your replies, but I am interested in hopefully further suggestions from others as well whom I don't want to distract by the magic rabbit ;-)
Just post some dummy reply so that I can give you full points later.

[John Payne_2]

With a 'heavily loaded' system, the timeslice at 1 will hurt more than anything else you have to fix, most likely. Clay and Harry are right. You should change it as soon as possible...

Also, 800MB is the absolute max that you buffer cache should be. This means that dbc_max_pct should be at most 10. (Maybe lower.) This is the reason: The buffer cache holds the most recently read files from disk onto memory. The theory is that it is faster to read from memory than disk. When your buffer cache is too large, the system has to look through it for whatever file it wants before it goes to disk. By the time that you look through a large cache (larger than 800MB), you could have read fom the disk already! I try to pick a percentage that puts my machines at about 300-400MB. While a read form disk 'costs' more overhead than a read form memory, when there is tons to look for in memory, and then you don't find the file and have go to disk... (I hope you see what I mean.)

Anyway, you should problibly take their advice as soon as possible. (Tell your users they do not have a choice...)

John

[John Payne_2]

Oh yeah, one other thing. Once you get memory pressure under control, you won't actually need 8GB of swap space... You may consider changing swapmemon from 0 to 1 so you can reclaim some of your disk space from swap later on. (Do a search on seudoswap here in the forum to get a whole bunch of threads about what it is and why it is good.)

Hope it helps

John