Kernel tuning.

[Kenny Chau]

Hi all experts,

I had 2 servers, L2000 and N4000, both running HP-UX 11.0. I tried to run some jobs on both the servers but got different run time. The N4000 used about 6 hours but the L2000 took only 3.5 hours. Both using the same databases and same programs. Why the time differnece quite large?

I had checked the kernel parameters and only find out that 2 kernel parameters in L2000 are larger than N4000 and the others are less or same. These 2 kernel parameters are "nkthread" and "semmns". Will this the main reason why the jobs that running on the L2000 is faster than N4000? If so, how to tune these 2 parameters? Any formulas?

ps. the databases are about 6GB in total and 17 jobs running parallel, only 1 job takes longer time to finish.

Thanks.
Kenny.

[Steven Sim Kok Leong]

Hi,

Also consider the hardware configuration of your system ie.
1) no. of CPUs, CPU speed
2) size of memory and swap (if in use)
3) disks (raid? mirror? stripe?)
4) controllers?
5) patch level of OSes
6) any other applications running on the system, etc

Try using glance to monitor the performance of your jobs.

Hope this helps. Regards.

Steven Sim Kok Leong

[Scott Van Kalken]

This attachment should help - I tend to use it a lot when changing parameters.

[steven Burgess_2]

Kenny

Check out the following doc

http://docs.hp.com/hpux/onlinedocs/os/KCparams.OverviewAll.html

Hope this helps

Steve

[Kenny Chau]

Hi all,

My L2000 has 1 CPU (440Hz), 1GB RAM, 3 HDs (internal, non-mirror, same HD while running jobs). While I was doing the test, there are many applications running at the same time. My N4000 had 4 CPUs (440Hz each), 1GB RAM, 2 SC10 Disk array(HD using for that application is 12HD, mirrored, different HD while running jobs). While I am running the jobs, no other application is running in this server.

I know that mirror disk cause slower performance and I already make the application using different HDs (multi volume for Databases).

I had used glance to capture the performance for the whole night but still have no clue why the N4000 takes longer time to run jobs. So I think it is the kernel parameters tuning problem. Is that true?

Thanks.
Kenny.

[Jakes Louw_1]

Are you saying that the single CPU server processed FASTER than the 4 CPU server?
If that is the case, then almost certainly your problem is I/O. Glance should show which disk/s are being hammered, and you should be able to drill down to lvol/filesystem level to see where the contention is.

[oiram]

Hi,

Have you checked if the slow system is swapping? If the buffer cache is too big this can happen.One way to increase the performance is set the fs_async parameter to 1(if the problem is in the I/O subsystem). Altough this is dangerous and can corrupt the HFS file systems if happens a powerdown or a panic. Why don??t post your kernel parameters here?

Best regards.

[Kenny Chau]

Thanks Jakes and Oiram.

Jakes, you are right, the 1 cpu server run jobs faster than the 4 cpu server. But I can tell you that the 1 cpu server is using 3 internal HD, 1 HD store the 6GB DB. The 4 cpu server is using the SC10 disk array and the 6GB DB is distributed into 3 HDs and from the DB manual, it said the 6GB DB distributed into different HDs will faster the Disk I/O. But when I run the same jobs in these 2 servers, the 1 cpu server is running faster than the 4 cpu server. But the main different is the 1 cpu server did not have mirror disk and the 4 cpu server had. I already ignore this because I know that mirror disk will make jobs run longer time. So do you think it is the kernel parameter setting problem?

Oiram, here are part of the kernel setting in these 2 servers:

1 CPU server:
max_thread_proc 64 64 Static N/A
nkthread 3635 3635 Static N/A
nproc 2068 2068 Static N/A
semmni 128 128 Static N/A
semmns 3072 3072 Static N/A


4 cpu server:

max_thread_proc 256 256 Static N/A
nkthread 2048 2048 Static N/A
nproc 3220 3220 Static N/A
semmni 256 256 Static N/A semmns 2048 2048 Static N/A

Do you think I should need to increase the nkthread to improve the performance of the 4 cpu server?

Thanks for your help.
Kenny.

[Michael Tully]

Hi Kenny,

I suggest you look hard at your kernel parameters on the N class. I've two and they run like theres no tomorrow. There are two things I would like to point out if not already.

Buffer cache. Make sure that this is no more than 300Mb. Also you may need to increase your shared memory and semaphore areas. Have a look at these:

semmap 15002
semmni 8192
semmns 16384
semmnu 4096
semume 128
semvmx 65535
shmmax 1073741824
shmmni 1024
shmseg 1024

Without knowing what application your running it is a little difficult to give direct recommendations.

-Michael

[Kenny Chau]

Thanks Michael, my company is running some Progress DB. Quite an uncommon database. So I can not get any information from here except consult my HP consultant. Your suggestion is quite strong to my server. I think it will solve the problem but not so sure my server can support this setting. Maybe I will try to setup a bit lower than your settings to see if it can solve my problem.

Thanks very much.
Kenny.

[Steven Gillard_2]

Also compare the patch levels of the two systems - is it possible that the L class is more recently patched?

The max_thread_proc and nkthread parameters are unlikely to be the cause of the performance problems, but it is an issue that you have nproc > nkthread on the N. I recommend you correct this. As others have suggested, also check the buffer cache and IPC related parameters (shm* / sem*).

The other area to focus on is the database - is it configured identically on each system? I would go over this very carefully.

Finally - measure. Check the measureware data, and collect glance / sar / vmstat data while the jobs are running. This will help you determine where the bottleneck might be.

Regards,
Steve

[Todd Larchuk]

I am assuming that you have a multithreaded application or a multiprocess application. If your application is single threaded and running in a single process, then that application can only make use of 1 CPU. The extra 3 CPUs in the N-class server will not be of any help. So, you may actually be comparing the performance of 1 CPU on the L with 1 CPU on the N.

[Jason VanDerMark]

Interesting issue. I would be curious to see what the current utilized values are for your ninodes, nflocks, shared memory tables, etc. Could you go into glance on each system and press 't' and then put the contents into a text file for viewing. Check to see if any of these parameters are being maxed out. Thanks for your time.

Regards,
Jason V.

[Sandip Ghosh]

Have you got any message in dmesg or syslog.log regarding the kthread:table is full.
Anyway you can increase both the parameters and see what is the result. Those parameters doesn't have any diverse effect on the system if it increased.

Sandip

[pap]

Hi,
1. It depends on what kind of cron jobs are runing.
2.The other thing to consider is that if both the machines are running on same network, becasue if they are on different network then network jobs will take the different time due to different network traffic conditions.

3. What are the SCSI controllers used in both types of machines. That also matters Disk I/O rates if the cron job has to do something with disk operations.

4. The swap momory size in both machines are same? If not that also matters the different in time while executing th same job.

Hope this poits will bring you something might to look at while anylising your constraint.

-pap