Very Busy Mirrored Root Disks

[Ty Roberts]

This may be a common normal occurance, but I wanted to as the forum to get some feedback. We have a new rp8400 that is fibre attached to an XP512. We have two internal disks that are our mirrord root disks. It seems that these disks are always busy (Glance Usually reports 85% utilization), but to me this seems to be abnormally high. Is there some setting I could have wrong?? Does this seem to be the norm or does anyone else find this to be a bit high? I know that processes are using the standard libraries, but these disks are always getting hit much harder than any other disk..


Here is the output from the lvnlboot command on this machine..
# lvlnboot -v 2>/dev/null | more
Boot Definitions for Volume Group /dev/vg00:
Physical Volumes belonging in Root Volume Group:
/dev/dsk/c0t6d0 (0/0/0/2/0.6.0) -- Boot Disk
/dev/dsk/c6t6d0 (1/0/0/2/0.6.0) -- Boot Disk
Boot: lvol1 on: /dev/dsk/c0t6d0
/dev/dsk/c6t6d0
Root: lvol3 on: /dev/dsk/c0t6d0
/dev/dsk/c6t6d0
Swap: lvol2 on: /dev/dsk/c0t6d0
/dev/dsk/c6t6d0
Dump: lvol2 on: /dev/dsk/c0t6d0, 0

Does anyone have any input for me???

Thanks,
Ty

[A. Clay Stephenson]

Without knowing more, it's difficult to say but my experience has been that on 'normal' (whatever that means) well tuned boxes, the boot drives get hit down in the low single percent digits 'most' of the time.

The first thing that occurs to me is that you are paging. I would look at vmstat and swapinfo to see if you are swapping. That would certainly cause this behavior. The other thing is that you may have a very active STM configured and you are doing excessive disk checks. Of course, the other obvious thing is that you are spawning a lot of new processes. We just need more data but instinctively something sounds wrong.

[harry d brown jr]

Ty,

Are you running oracle, with the logs on vg00??

How does your swap space look?

live free or die
harry

[Jeff Schussele · Accepted solution]

Hi Ty,

Run
swapinfo
to see how much swap you're using. Sounds like you may be swapping a lot.

Then run
sar -d 5 10

This will report, ten times - 5 seconds apart, on disk activity. If the /dev/dsk/c0t6d0 dev is the heavy user & you're using quite a bit of swap - then run

sar -w 10 10

To see if you're paging out/in which would confirm that you swapping processes out to disk & back in to memory.

Then you have to investigate just *what* is using up all your memory OR reserving all the swap space.

HTH,
Jeff

[Tim D Fulford]

I can't realy add to the above as far as diagnostics go. The only time I've seen the root disk go through the roof is when we were out of memory & it was paging out etc

To give you an idea of what is on my root disk
o HP-UX 11.00 standard stuff namely
/, /stand, /opt, /usr, /tmp, /var, /export(or /home) & swap+dump.
o Database binaries (Informix) & configs
o Application binaries & config files
o Application & datbase log files (like syslog.log NOT redo or roll back segments)

I find that the above runs at 5%ish. If the system is not swapping and if you have anything that is not a binary or simple log files look carefully into how it hits the disk.

just my 0.02???

Tim

[Martin Johnson]

Try using top or glance to see which processes are using CPU. You could have a looping process, however it is more likely that you are swaping/paging (use the commands mention in previous replies to check swaping).


Marty

[Jeff Schussele]

Hi (again) Ty,

One thing that could be using up a sizable amount of memory could be the
dbc_max_pct
kernel parameter. This defaults to 50 & if left there could be using up to 50% of your memory for buffer cache. Which very few systems need. If still at 50, I'd bring it down to the 15-25 range & get that memory back for application use. You sometimes can go even lower, espicially when running Oracle & letting Oracle do it's own caching.
Note you'll need to relink the kernel & reboot for the change to take effect.

HTH,
Jeff

[Victor BERRIDGE]

Hi,
As my predecessors have mentionned with the infos you gave - what to say?
What are you running? how much RAM? /proc...
How much swap and utilisation?
Remember all that is being logged finishes somewhere on the root disk (/var... /tmp ..) and where is your swap?
What do you get from swapinfo?
Have you thought of creating (far better performance since you have an XP512) a primary swap on the XP ?

Good luck

Victor

[Bill Hassell]

With no additional information, all you can do is to conclude that all is normal. With a *lot* of work using sar options, you might get an idea of which disks are busy, but none of the classic Unix tools can tie the disk activity to a specific process.

There's lots of reasons for disk activity, some are very wasteful and some are required. Load a copy of Glance onto your system and print a copy of the manuals from the /opt/perf/docs directory. Even the simple bargraph in Glance will tell you whether the disk activity is swap activity or file access.

[Ty Roberts]

Thanks for all of your suggestions / questions. The thing that makes me think it is not so much a swap issue is that we have 25.7 GB of Pseudo Swap. I do not think it is even starting to touch the Disk Swap we hava allocated on vg00.

Attached you will find some info and stats I have collected. In terms of the sar -w command, do the pswch/s numbers seem high?? I have not found anything to lead me in eihter direction...

The System has 16 CPU's
32GB of Physical Memory
41.7 GB of Swap
16GB of DIsk Swap
25.7 GB of Pseudo Swap
It runs 2 Instances of Oracle and a J.D. Edwards Installation

If I use GLANCE and look at IO by file system it seems as though /opt, /usr/, and /var are the ones getting the most hit. We do not have any applications logging or writing to anything in vg00, they write to thier own logs directories on disk withtin the XP.

Maybe this will spawn some new ideas or suggestions...

Thanks!
Ty

[Tim D Fulford]

Ty...refering to Jeffs reply

Jeff.. I know extra free memory is a nice thing but..

I thought dbc_max_pct (Dynamic Buffer Cache, max limit %) was "dynamic", if it was requred for anything else it would automatically be reduced down, (limited by dbc_min_pct)? Or is sawp etc at a lower priority so it does not de-allocate it buffer cache?

Anyway, just my addition to this thread..

Tim

[Pete Randall]

Ty,

Just a thought - we run Informix and during one particular process an error message is generated repeatedly. This error message causes a lookup in /opt/informix to resolve the NLS language (i.e. en-us). Consequently the I/O rates on vg00 go right through the roof. I'm trying to remember how we tracked that down. Glance would give us the file system or lvol - maybe lsof??

In any case, is there any chance that you might have a similar situation?

Pete

[Stefan Farrelly]

Your stats look very interesting. Yes, there is no paging, which is good. But yes, during the day your boot disk (c0t6d0) is being hit abnormally hard, but its mirror (c6t6d0) is not. That is very wierd! Normally if all of vg00 is mirrored then if your boot disk is being hit your mirror should be too!

Do you have any lvols in vg00 which are NOT mirrored. Please check them all. If you can find one this is our chief suspect.

I know your apps are all on an XP but they will still use stuff in /var/tmp or /tmp for temporary files - which are on your internal disks, but again, both the boot and mirror disk should be equally hit. My guess is you have a non mirrored vg00 lvol which is being used. I eagerly await your checking!

[Stefan Farrelly]

Petes earlier reply has hit it on the head. You wont see activity on the mirror if only the boot disk is being READ (not written to) so its likely some file which is being read constantly. You should use gpm, display the process list, sort by disk i/o and this should id the chief suspect processes, then drill down on each to see which files they have open (using gpm or lsof for that bit), that should id which fil is being hit. Once you find it you could mv it to an XP filesystem and leave a symbolic link on vg00 pointing to the XP filesystem. This should speed it up a lot.

[Ty Roberts]

I just checked and all lvol's are mirrored and there are no stale extents. Could this maybe have something to do with the mirror write cache do you think???

[Peter Kloetgen]

Hi Ty,

did you set the kernel parameter:

swapmem_on = 1 ?

This makes swapping more effectively. Hope this helps.

Allways stay on the bright side of life!

Peter

[Ty Roberts]

In response to the "Heavy Read" posting I did some watching and researching in Glance and it seems as though when some of our Oracle Processes hit a high Phy IO RT they are opening files in /usr/lib and /var/. What popped into my head was maybe the socket connection that was open in /var like the one listed below (line from lsof) would be the one causing the problem.. any thoughts on this???

oracle 18439 oracle 8u unix 64,0x7 0t0 17119 /var/spool/sockets/pwgr/client18439 (0xe2365680)

[Ty Roberts]

Yes
swapmem_on is set to 1

[Tim D Fulford]

Your service times are appaulling... 25ms (I would expect 4-8ms) you also have disk queues. None of these are good...

I think your disks are busy because
1 - Lots of info is going their way (No sh*t Sherlock!!) AND they are thrashing (head is constantly moving from inside to outside of disk)..
2 - They are having hardware problems

I would start to look at how much info they are dealing with. I personally use Measure Ware & plot stuff against time...to use glance I would do...

# glance -u #** this checks IO by disk
use "S + " to select the disk

# glance -v #** IO by Logical Volume
use "S + " to see more detail

# glance -i #** IO by Filesystem
use "S + " to see more detail

The only other thing I have seen that is similar to this was with fc60 and ScsiMaxQueueDepth (or similar?) was set too low (8) and it OCCASIONALLY caused the disks to go beserk. I must stress this is a BIG stab in the dark.

Tim

[Tim D Fulford]

I'm pritty slow at these postings

pwgr (Password & Grooup caching deamon/thingy)

This has caused problems on our system but with pwgr running away & eating up CPU. We solved it by turning it off
(set PWGR=0 /etc/rc.config.d/pwgr & /sbin/init.d/pwgr stop)

If you have a large password file I suspect this process will be of assistance, so stop & restart it, if not shut it down

Tim

[Stefan Farrelly]

Ty,

These libraries under /lib etc. are usually kept in memory most of the time so these shouldnt be causing your high reads on your book disk. Just to check you do have some buffer cache configured ? sysdef | bufpages
will show what its set to.

I think the most likely culprit are socket files in /var/spool/sockets/pwgr (used by Oracle and oracle users). cd to this dir and ll -tr it, how many files are in there and are there hundreds from today ? this causes a high hit on the boot disk. You could move this directory to an XP filesystem and leave a symbolic link there under /var/sockets but you would need all your users off and applications shutdown before you could try.

[Ty Roberts]

I think we may have a winner... I just went into the directory and seems that there are close to 800 sockets in that directory from today..

wcores1a [/var/spool/sockets/pwgr]
# ls -ltr | grep -c "Jun 18"
761

Would everyone agree that this is a large number and it should be moved to an XP disk???

[Ty Roberts]

I checked the bufpages and it is set to 419020. We have not changed this, I guess it is the system default. Is this a proper setting???

[Pete Randall]

I don't know, Ty - I show 783 on my puny little N4000 with 6 processors and 228 active Informix sessions.

Pete

[Stefan Farrelly]

You do have too many files in /var/spool/sockets/pwgr. I would definitely try moving the pwgr directory to an XP and see if that fixes your problem.

Your buffer cache is set to 1.7GB !! This far too high. HP recommend no more than about 300MB otherwise it slows down performance. You should reset to about 74000 and build a new kernel and reboot (or if bufpages not set in the system file then set dbc_mac_pct to only 2).