Disk I/O thresholds being exceeded

[Ciaran Byrne]

Hi,
I am having an issue with sporadic parts of the application crashes i.e. 1 or 2 processes per day of a 30-40 process application. It looks like one of the I/O related kernel parameters is the culprit. Here is what the measureware report shows for this process for PROC_INTEREST
| K D |

Also, the PROC_DISK_PHYS_IO jumps above 10000. Can anyone point to what kernel parameters could be responsible for the process crash. Because of the number of processes and because it is sporadic it makes it difficult to point truss/tusc at.

Any help would be greatly appreciated.

Regards.
Ciaran

[Michael Tully]

Hi,

Ususally if a kernel parameter has actually reached a threshold limit, it would be recorded in either /var/adm/syslog/syslog.log file or even 'dmesg'.
In Glance, does any particular disk show any symptoms of IO thrashing. If this particular problem has only just surfaced, you could even have a bad disk. Information on bad disks, will also be reported in the syslog.log file.

Regards
Michael

[Ciaran Byrne]

Hi Michael,
thanks for your response. I am having the same issue all 4 application servers that are being used all of which have local I/O. I am thinking thats the problem is application related. It has just been released so this related to either the new application version / our custom code or if the kernel parameters are not tuned properly. With all the I/O I am wondering is the application hitting something like maxfiles (1548) or some other less known kernel parameter.

Thanks,
Ciaran

[A. Clay Stephenson]

One of the first things that you should do is instruct your users to issue an echo $? immediately after a program crash and note that value. If the application was coded with any degree of discipline, $? should be the value of errno set by the offending system call. That will be a very valueable clue.

You can also narrow this down by examining the core file with a debugger; the stack trace will provide a trail of breadcrumbs leading from the crash.

[Bill Hassell]

If you have true application crashes, there will be a core file which can be examined using the file command:

# file core

This will tell you the name of the application and the reason it core dumped. Now this won't be a useful answer since the application made a serious mistake internally and the operating system terminated the program. Pure and simple, it is an application defect.

Even if the application is failing because of some kernel parameter, the program is defective because it neither reported the problem nor did it properly test that a resource was actually assigned before attempting to use it.

Send the core files to the manufacturer of the application and ask that the problems be analyzed and the defects corrected. Kernel parameters such as nfile and maxdsiz are commonly implicated in program crashes but there is no excuse for a program to create a file and ignore the operating system's response that there is no room, or to request additional memory and start using it when the kernel said memory was not available.