Internal disk so busy it locks the rx2880-i2 hpux11.3 system

[Steve Post]

rx2880-i2 hpux11.3

I have a program that reads in text to Sybase.  The program use java.  When it runs on the old pa-risc boxes, it works fine.   When it runs on the my itanium box it hangs the box for 2 minutes. 

I see the queue on the internal changes from 0, to 43,000.  I see the average wait to process data to the disk changes from 0 microsecs to 23,569.85 seconds.  I see the internal disks are 100% busy, and 100% of the system is stuck waiting on those darned internal disks. 

I see a lot of paging in.  I see 30 blocked processes (vs none).  I see hardly any memory is in use at all.

I run saconfig  /dev/ciss0 and see no cache on the internal disks. I see the "some cache card thingie" is not installed.  It is not installed because I don't have it.  It does not actually EXIST on the box.  I know the internal disks are mirrored raid 1/0.  I do not want to mess with the internal disk layout because this is where hpux resides.  I'm using those.   I can't shutdown the box. 

One way I can greatly help performance is to move the work area of this job to disks on an EMC Vnx Disk array.  The array disks seem to handle the stuff with no problem at all.  But I would like to understand why a more modern system can actually be more junky-er.

Why do the internal disks run so lousy? 

Is there a parameter I can tweek on /dev/ciss0 to help without rebooting or destroying anything? 

Maybe a kernel parameter?

Maybe a parameter in saconfig or sautil?

one more thing... I am paging out when the system is locking up.  pi=379, po=674, All of the swap logical volumes are on the internal disks.   I ran  ipcs -mob and saw no change in memory usage.  I did NOT run swapinfo -tam (guess I should have).   So I bet SOME more help would be to move some of that swap to the EMC Vnx Disk array.

[Duncan Edmonstone]

Steve,

Have a read of the following thread:

http://h30499.www3.hp.com/t5/General/IO-performance-puzzle/td-p/3977326

I can't explain why you didn't see this problem on a PA-RISC box - are you completely sure that was working off just a single internal boot disk?

[Torsten.]

I think the optional 512MB cache modul for the P410i will speed up the disks.

[Steve Post]

it will take some time to go through the referenced thread.

I am not in a position to buy anything, or reboot anything.  I trying to understand before I affect the use of the system.  Well...Affect it worse than I have.

[Steve Post]

I looked through the thread.  there was a lot of talk.  But the kernel parameter mentioned does not exist.

I will look at the queue for the internal disk array /dev/dsk/ciss0.  I have a GUESS.....

the internal disks are busy, so they tell HPUX to put they tasks on the side...I guess.

and HPUX does that.

But the swapspace logical volumes are also ALL on the same internal disk.  So it would seem to run in an infinite loop.  Just saying it sounds like an infinite loop.

Perhaps my imagination is not too good.  Here is what I am imagining:

process:  the disk is busy.  put me in the swap space.

hpux: ok.

process in the swapspace: the disk is busy. put me in the swap space.

hpux: ok

process in the swapspace going into the swapspace:  the disk is busy. put me in the swap space.

hpux: sure....why not?

process in the swapspace going into the swapspace that is going into the swapspace:  hey... can't I get in here. too?  that darned disk is busy.  why not put me in swap space.

hpux:  sigh.... you too?  FINE.....

etc, etc, etc, etc,

[Dennis Handly]

>pi=379, po=674,    I ran ipcs -mob and saw no change in memory usage.  I did NOT run swapinfo -tam

It would help to show use the swapinfo output.  I'm not sure what good ipcs would show you?

You can also get page in and out for disk I/O (mapped files).

>But the swapspace logical volumes are also ALL on the same internal disk.

But are you paging data to swap or just doing lots of I/O?

[Duncan Edmonstone]

>I looked through the thread.  there was a lot of talk.  But the kernel parameter mentioned does not exist.

Gee, funny I must have dreamt it then when I wrote that post...

Hang on...

# uname -rs
HP-UX B.11.31
# kctune -v disksort_seconds
Tunable             disksort_seconds
Description         Maximum time period for grouping incoming I/O requests (secs; 0=no limit)
Module              fs_util
Current Value       0 [Default]
Value at Next Boot  0 [Default]
Value at Last Boot  0
Default Value       0
Constraints         disksort_seconds >= 0
                    disksort_seconds <= 256
Can Change          Immediately or at Next Boot

As I think I said in my post in that thread - it isn't documented,as you are discouraged from using it, but the point in those posts about not doing alrge amounts of sequential IO to system disks holds true...

[chris huys_4]

Hi,

log a support call with HP.

and ask them to ask you to provide a kitrace output, when the issue appears. 

Best Regards,

Chris

[Steve Post]

Dennis:  I think I am just sending a lot of io.  I will need to rerun my test while running swapinfo -tam.  I was running ipcs -mab just to get more info while this error happens.  

Dear Mr Edmonstone: LOL!  I jump to conclusions too fast.  But I got a kick out of your response.  I will leave that kernel parameter alone.

Chris Huys:  I am not calling HP yet.  I don't like wasting their time with a question that is too vague.

at this point I might be running  full bore the wrong way but? Well? 

My error is very repeatable.  But it might also be just plain nonsense.  I have a job that read/writes data to a database that is not on internal disks.  But it uses java, that is on /opt.  I run one or two at once?  No problem.  A couple of them hang on for a while because they run notoriously slow.  But others run easily under a second.   Now if I add more of the jobs (the easy/fast ones), until I have 17 jobs at once?  The sucker just hangs everything for 5-10 minutes.  I can get the problem to go away as I move more and more program files to an array disk and link it back.  But that is a bandaid.  

I found another forum entry about this.  The guy tried increasing the max_q_depth.  via      

scsimgr [set_attr|save_attr] -D /dev/rdisk/disk2 -a max_q_depth=32

His system ran a bit slower.  But the big hang up in the system went away.

The current Q depth is 8 for that internal disk.  I think I might be quickly running for the solution without identifying the cause.  But I know the avwait of the disk is 27,754.  And the avserv is 15.29.  This seems wrong.

So my NEW test:  run my stuff with scapinfo -tam on.  Change the queue from 8 to 16.  Repeat the test. 

My best information that I can interpret has been from sar -d.  I ran it every minute.

I'll let you know how it goes.....  then maybe call HP.

Steve

 oops.  forgot the forum link.

http://h30499.www3.hp.com/t5/System-Administration/Disk-Utilization-at-100/td-p/5280090

update2:

I ran the repeatable job that hangs.  It hangs up the system for 2-10 minutes. 

waittime on the disk: 12,251.35.  server time: 14.13. 

Total time for the test about 15 minutes.

I change max_q_length to 32 via scsimgr.

hangtime: 0 

waittime on the disk: 0.1   server time:  4.39

Total time:  40 seconds.

It seems that I have the solution to my test.  Increasing the queue sort of helped.   

Well I am not sure.  I undid it.  I made the queue 8 again.  And it still runs fast.

Maybe something else changed.  So jury is still out.    Once again.  I've driven in a circle.

[Torsten.]

Well, if you read all the data from external disks, the next question would be about the SAN ... switches, arrays, paths, etc.