SOLVED
Let me try and explain why your system is becoming sluggish and what you *could* do and what you *should* do to resolve it.
HP-UX is most definitely a Server OS (and certainly from 11.23 onwards the OS doesn't even run on workstations), and for a long time the developers have been tuning the OS to provide better server performance.
Some of the assumptions we make about Servers is that we have *separate disks for the OS* and don't use those disks for anything much apart from the OS, and that we *break out differing workloads onto separate disks*. Various algorithms in the kernel make these assumptions when endeavouring to provide balanced performance.
So for instance, when HP-UX looks at queues to disks, it sorts those queues based on 2 parameters (in order):
1) The priority of the requesting process
2) The block number of the requested IO in ascending order
The first always has to be the case in a timesharing system, the second is designed to reduce disk head retractions and therefore increase overall IO efficiency (by ensuring that for IOs of equal priority in the queue the disk heads scan across the disk just once). This generally works great when we separate out IO loads onto separate spindles as we would on a well architected server.
However one of the issues with this is that when we do a large amount of sequential IO to a disk (e.g.. a restore!), any random IO to the same disk is going to suffer as its requests will keep getting pushed down the IO queue by the sorting algorithm preferring the sequential IO. When that disk is the OS disk (which is pretty much always doing small amounts of random IO as OS commands are run and libraries are loaded) the upshot can be a very sluggish system.
Luckily HP know that there are situations where we don't have any choice but to do this, and so provide the kernel parameter disksort_seconds. You won't find much info on this as we'd really rather people did the right thing and didn't do large sequential IOs to the system disk, but here's what it does.
When disksort_seconds is changed from its default value of zero, we change the sorting algorithm as follows:
In order:
1) The priority of the requesting process
2) The time the IO has been on the queue relative to disksort_seconds
3) The block number of the requested IO in ascending order
So now, for any given IO of the same priority I know that my IO will get processed before any IOs that came in 'disksort_seconds' later.
This is fairer on the random IOs, but ultimately delivers less efficient IO.
You don't tend to see this on OSs like Windows and Linux, which ultimately come from a PC lineage where with one (usually slow - 7200rpm) disk in a system, these sorts of algorithms wouldn't be efficient, so they usually have something similar to disksort_seconds enabled by default (IIRC Linux calls it the deadline IO scheduler)
So you *could*:
Alter disksort_seconds to 0, 1, 2, 4 , 8 , 16, 32, 64, 128, or 256 seconds. The lower values should certainly have the effect of making the system less sluggish, but will also no doubt slow down your restore to some degree.
But you *should*
Stop doing large sequential IOs to your system disk.
One other thing you could try might be to nice the data protector restore agent, as priority is still what is sorted on first in the sorting algorithm. I can't say what Data Protector would think of that though - it could cause other issues.
HTH
Duncan
I am an HPE Employee
OK - you're strecthing my knowledge of this subject considerably now, but let me try and answer your questions as best I can:
Hein's questions first:
Q1) I'm surprised to see the unit for disksort_seconds is seconds. Well, not very surprised given a name like that, but a single second already seems an awful long time, let alone settings higher than that!
If an application, after getting a disksort boost, needs a further IO, it could be outside the current 'sweep' and have to wait for a long time again. Yikes.
*A1) No because any random IOs even on the current sweep will be satisfied before those sequential IOs that came in disksort_seconds later, but you are right to point out it doesn't help a great deal, Thats why its largely undocumented,
Q2) This is a great algortime for simple, local, disks. But when applied to SAN controllers could it not spell disaster?
*A2) How so? I think the real question here is "without understanding the geometry of a SAN LUN can we come up with a better algorithm", to which I think the answer is no we can't - we need to know too much about how the LUN is physically made up to come up with any sensible logic - and we just can't do that right now.
Q2A) With 4 or 8 or maybe 50 disks behind a LUN don't we want at least as many IOs outstanding the the LUN as to increase the chances to make all disk busy and contribute?
*A2A) Yes we do - and thats what we use queue_depth for with scsictl(1m). DOn't confuse IOs in the disks IO queue (that which disksort_seconds effects) with concurrent outstanding IOs which queue_depth controls. queue_depth lets us control how many outstanding IOs (IOs that have been pushed out the pipe to disk) we have at any one time.
Q2B) The EVA will actually postpone allocating actual storage chunks (1MB? 4MB?) untill first written. Thus if you have two LV's on a PV which are filled up gradually the physical blocks on the underlying disks will be interspersed, not ordered by LV.
Q2C) If you have a striping (raid-0, raid-5) controller with a large chunk size, then issueing the IOs strictly in order will focus the IOs on 1 or a few members underlying to the LUN thus letting IO power go to waste?
*A2B & A2c) WHilst this could be the case, how can the OS know about these specific geometries? Designing for one case could caise issues with another. Given that large amounts of IO to SAN LUNs are typically handled more efficiently by the disk array than a single SCSI disk could hope to provide, why worry about those situations?
Q3) Could it also explain lots of SYSTEM time as sorting 1000+ IO elements, or at the very least keeping them in order, takes some CPU cycles alright!
Can we readily recognize this particular CPU consuptions with a profiling tool?
*3C) I've never profiled CPU utilisation during these types of events so can't comment.
Now Alex's
Q1) Does the disk IO queuing algorithm apply to SAN disks as well as local disk?
*A1) IT applies to all LUNs - the OS knows no difference between a local disk and a SAN LUN.
Q2) It seems that prioritizing IOs based on block number does not ensure in-order writes. Is that correct? Are the O_DSYNC/O_SYNC flags for open(2) the only way to ensure in-order writes?
*A2) No - don't worry about that - I think the explanation given may simplify things a little, but it ceratinly can't lead to corruptions caused by failing to follow write order rules. Certainly for a filesystem the filesystem code takes care of all this anyway and I came across the following on the man page for disk:
"Buffering is done in such a way that concurrent access through multiple opens and mounting the same physical device is correctly handled to avoid operation sequencing errors."
Q3) I'm noticing an "sblksched" process running with a priority of 100. Is that the IO request scheduling process?
*A3) No sblksched are the streams schedulers. See kernel parm NSTRBLKSCHED.
Q4) What about the queue length? Doing scsictl on the local disk in question shows me: queue_depth = 8. What is glance showing me with Qlen?
*A4) See answer to Hein's question A2A above. queue_depth is the number of concurrent outstanding SCSI IOs to the LUN from the OS, not the total number of outstanding IOs for the disk.
Not sure I've helped much there but happy to try anyway.
HTH
Duncan
I am an HPE Employee