Operating System - HP-UX
1748112 Members
3508 Online
108758 Solutions
New Discussion юеВ

HP-UX 11.11, wc -l on a BIG ASCII File is slow and is in "Cache Wait"

 
Alzhy
Honored Contributor

HP-UX 11.11, wc -l on a BIG ASCII File is slow and is in "Cache Wait"

Anyone encountered this issue before?

Today, on one of our few remaining HP-UX servers - we experienced (and continue to) weirdness with doing file reads specifically "wc -l" on "large files".

For instance, a 1GB File with ~12 million lines completes a "wc -l" process in over 30 minutes! Whilst runing - Glance shows the process is waiting on Cache.

sar -b (read stats shows varying 90 to 100 % stats)

Server in question is a SD vPar with 48GB of RAM, dbc_max of 10% (4.8GB Buffer Cache).

This only started happening today. And the size of the files (1 to 20 GB) being processed by a sort and wc -l process is no different than the previous years this system has been in existence.

Is there a known and rare cache coherency bug in 11.11? Is it time to allocate more Buf Cache? TLB issues perhaps since this is a vPar? Our other 11.11 Systems that are at the same rev of OS are not suffering the same issue and have even signifcantly smaller Caches.


The only change the server had undergone in recent days were new vPars having been carved. This particular server has been up for 10 days. New vPars were created a few days ago.

We've a case with HP and we've yet to receive any clues from them.

Wanted to try a "sync" but...

TIA!

Hakuna Matata.
9 REPLIES 9
Dennis Handly
Acclaimed Contributor

Re: HP-UX 11.11, wc -l on a BIG ASCII File is slow and is in "Cache Wait"

>And the size of the files (1 to 20 GB) being processed by a sort and wc -l process is no different than the previous years this system has been in existence.

Using the buffer cache for wc(1) on huge files isn't a particularly good idea, unless you want to do this several times in a row. And if you did, you should cache the previous size and lines and only read the new data at the end.

As to sort, it may be more helpful.
Unfortunately you don't have much user control.
Alzhy
Honored Contributor

Re: HP-UX 11.11, wc -l on a BIG ASCII File is slow and is in "Cache Wait"

The system has been in place and issue-free for several years -- so eveyone is dumbfounded.

I know -- shell utilities are cache centric / cache crazy and these files are new files so virgin reads. I've suggested to them in the past to use Perl or even C code to open the file cacheless or force direct io but change just is difficult to effect in a long stable system.

Still awaiting word from HP to see if there are other measures apart from a platform reboot.

Any other ideas?
Hakuna Matata.
James R. Ferguson
Acclaimed Contributor

Re: HP-UX 11.11, wc -l on a BIG ASCII File is slow and is in "Cache Wait"

Hi:

Could you consider copying the file(s) in question to a mountpoint where you can disable the buffer cache (i.e mounting with 'mincahce=direct,convosync=direct')?

Regards!

...JRF...
Alzhy
Honored Contributor

Re: HP-UX 11.11, wc -l on a BIG ASCII File is slow and is in "Cache Wait"

Already did sir, first knee-jerk reaction actually.

Lot better but nowhere near the normal filesystem.
Hakuna Matata.
Dennis Handly
Acclaimed Contributor

Re: HP-UX 11.11, wc -l on a BIG ASCII File is slow and is in "Cache Wait"

So the problem is only on the wc and not sort?

You may have run into a kind of ultraviolet cachestrophe (catastrophe). :-)
Alzhy
Honored Contributor

Re: HP-UX 11.11, wc -l on a BIG ASCII File is slow and is in "Cache Wait"

I think file reads are affected as the process involves sort (syncsort) and wc -l in whatever weired process our warehouse regularly performs.

This based on some file read routines (that uses the cache) I've done so far.

Hakuna Matata.
Alzhy
Honored Contributor

Re: HP-UX 11.11, wc -l on a BIG ASCII File is slow and is in "Cache Wait"

HP has been clueless as well but suggests increasing buf cache and memory. They however insist some system changes occured.

One possibility I am looking at is to do a platform reboot. The SUperdome is one big nPar with many vPars. And the only recent change done was a couple of vPars was carved whilst this particular vPar was running.
I am suspecting Memory Locality or Fragmentation occured and could be alleviated by a platform reboot so all vPars wil be freshen'ed up..

Far out Crazy?

I've no other ideas 'cept to wait for HP to come up with other findings and thoeries.


Hakuna Matata.
James R. Ferguson
Acclaimed Contributor

Re: HP-UX 11.11, wc -l on a BIG ASCII File is slow and is in "Cache Wait"

Hi (again) Alzhy:

> The only change the server had undergone in recent days were new vPars having been carved. This particular server has been up for 10 days. New vPars were created a few days ago.

Then, as you suggested, if you can reboot, it seems like it might be informative. Perhaps before hand, HP Support might want to have a forced dump?

Regards!

...JRF...
Alzhy
Honored Contributor

Re: HP-UX 11.11, wc -l on a BIG ASCII File is slow and is in "Cache Wait"

K.. server/vPar was rebooted and guess what happened? It dragged down all the other vPars in the server (a SuperDome)! SO I guess my theory seem to be true and HP is right in insisting a "change" happened. The vPar gyrations on the background (new vPars getting established) was the ONLY change that had happened.

The reboot of this affected server was originally meant to increase buffer cache allocation to address the cache wait issue (but nary an evidence exist in sar -b stats) as suggested and concurred by HP. After the cache addition the issue was actually worse. We went from ~ 5GB of cache to 20GB - as the hope is our 3 to 10GB files will have "ample" cache. BUt it turns out it was more of an issue as the OS is now agressive at caching everything and I/O simply went through the roof!

We've since brought it back and things appear to have returned to normal -- although another change was actually discovered (and that is thee was a erduction in I.O paths to the disks making up the lvol/.filesystem where these large files being sorted/wc'd reside)...


Hakuna Matata.