Why does ioscan run on its own

[Michael Leis]

Occasionally, ioscan seems to run on my K420. This system is already a very busy system. When ioscan kicks off it takes a lot of cpu resources (top showed at one moment %WCPU 37.79 %CPU 6.85). Along with everything else, the CPU reached 91% busy. Why does ioscan run on it's own, and is there anything I can do to control how often / when it runs?

[Jeff Schussele · Accepted solution]

Hi Michael,

Several things can kick off ioscans - SAM, EMS, stm suite - all I believe can run ioscans.

Rgds,
Jeff

[Sridhar Bhaskarla]

Hi,

Do you have things like sysinfo etc running on the box through cron periodically or any other monitoring software like VPO, Tivoli etc?

Even HAO (High Availability Observatory) would run ioscan depending on the interval set.

Check root's crontab first.

-Sri

[Chris Wilshaw]

Assuming that other users are definitely not running it,
it's most likely running as part of the diagnostics software on the system, as part of a monitoring package that you have installed, or it's being called from a cron job.

If it's diag/monitoring software, the tuning would depend on the software versions being used.

Check for cron jobs using

crontab -l (will display for your current user)

or check the files in /usr/spool/cron/crontabs

[Uday_S_Ankolekar]

Check if you are having any cron jobs to run ioscan , If you are a root execute crontab -l and see.

-USA..

[Helen French]

Several things to check up:
1) cron jobs (crontab -l)
2) EMS and other hardware monitors setup?
3) SAM
4) Any program/utilities on the cron job

[Bill McNAMARA_1]

do you have autoraid or fc60s?

you should run ioscan with the -k option instead if it ends up being some kind of cron task... it'll be much nicer to your system for more or less the same effect.

[Colin Topliss]

It's most likely to be STM. It will run periodic scans of the hardware. The data log resides in /var/stm/logs/sys/scan_hw_log.

The time interval can be configured in /var/stm/config/sys/diagmond.cfg - look for CHK_IOSCAN_TIME

To see if stm is running, look for diagmond:

sys=> ps -ef |grep diagmond
root 12986 10943 0 16:59:34 pts/13 0:00 grep diagmond
root 11041 1 0 16:53:57 ? 0:03 /usr/sbin/stm/uut/bin/sys/diagmond

[Khalid A. Al-Tayaran]

Hi,

Periodic runs of cfg2html and nickel can run ioscan.

Check your crontab.

[Steven E. Protter]

Lets assume crontab is clean.

Lets say there is a disk or other hardware device that's not wired correctly or is going on or off line.

The system might "think" its getting new hardware and scan it.

Do this:

dmesg

Check the file /var/adm/syslog/syslog.log for unsuual events coinciding with your ioscan runs.

You might have a disk getting ready to go bad. It could be anything, even a built in lan card going a little nuts.

Good Luck.

SEP

[Michael Leis]

Looks like EMS was the culprit. My crontab was clean, and most of the other apps/commands suggested were not be used. Thanks to all of you for the help!

A question for Colin Topliss: I am curious about one suggestion you made (and gave detail to) regarding diagmond. I checked the /var/stm/config/sys/diagmond.cfg file and everything (and I do mean everything) is commented out. If it's all commented out does that mean that it doesnt run at all OR does it still run based on some default?

[Helen French]

For the last question: All lines commented out on a diagmond.cfg file is normal. You don't need to worry about it unless you want to change some configurations or parameters. It will work with the default values. You should be very careful while changing that file.

[Andrew Merritt_2]

Diagmond does not periodically run ioscan. It does periodically check to see if an ioscan has been run, updating the results held in the kernel, and will update its files accordingly.

The diagnostics programs do not initiate a new ioscan, they only access the in-kernel data (the equivalent of 'ioscan -k').

None of the hardware monitors should be running 'ioscan', though there is a problem with two of them (armmon and fc60mon) doing exactly this at start-up time in the recent releases of OnlineDiag (A.34.00 and A.35.00, HWE0209 and HWE0212 respectively). This is covered by CR JAGae50172, which will be fixed in patches to be released in the near future.

If it is the diagnostics that are causing the ioscans to run (or even if it isn't), you should be able to tell from the ps listing what the parent process that started the ioscan process is. When you say ioscan is running on its own, what do you mean exactly?

[Michael Leis]

Andrew -
Periodically, I have a crontab that runs that simply does a "top" command and stores the top 10 processes in the list to a file. This is done for the time being so we can try to spot any unusual process that was running when we find out performance took a hit. It's worked well, we've been able to make a number of adjustments to the timing of things and resolved a lot of our problems. However, on more than one occasion a CPU has been hit hard and I've found that ioscan was the top process on that particular CPU. Based on the times it's happened, it is extremely unlikely (like 1 in 1 million) that anyone issued ioscan as a command. I've reviewed the other crontabs and there is nothing there that calls for it either. Nothing special running then, and it doesn't happen at the same time every time.

[Andrew Merritt_2]

Michael, I'd suggest modifying the script temporarily to capture the parent of any ioscan that is running (perhaps by doing a full 'ps -ef'). I think that's going to be the best way to identify what is actually starting the ioscan. Once you know what it is, we can then work out why it is happening.

[Michael Leis]

Good plan -- I did so and have found that it is running around every two hours. It is being initiated by /usr/sbin/stm/uut/bin/tools/monitor/ha_disk_array.

[Andrew Merritt_2]

Michael,
ha_disk_array does run 'ioscan', but it should only be with the '-k' option, which shouldn't be causing a problem (though it might use a lot of CPU for a short time). Does the listing you got show if '-k' was given?

How long does the ioscan run for? I suspect that there isn't a real problem, and that you are catching the ioscan when it is being run, and it just happens to be CPU intensive for a short while, however you may have data that shows otherwise.

ha_disk_array polls the devices it monitors every 15 minutes, and runs ioscan -k to convert the LUN address to the device path, so it can test raw i/o (it runs ioscan -k for each LUN, which is perhaps less than optimal :-( ). You can change the polling frequency of ha_disk_array by modifying the /var/stm/config/tools/monitor/ha_disk_array.cfg file. Don't set it to too long, as that would mean it could be longer before a problem was detected.

[Michael Leis]

The -k was used in the command. You are right, it is CPU intensive for a short period of time. Now that I understand better what is causing it and why, I'll let things ride for now. So far the users havent complained, so that further indicates to me that it's a very short hit. Thanks for your help. I really appreciate it.