System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

Strange performance issue

 
Elmar P. Kolkman
Honored Contributor

Strange performance issue

Hi gurus,
we are trying to implement a backup with nearline storage and ran into a strange issue. When we backup we have a speed of about 30Mb/s to an NFS filesystem. But when the file reaches the 2Gb, the speed drops down to about 3Mb/s. So we are looking at the backup software. But now I've done some more testing, and the problem is not limited to the backup software.

If I run a cp of a file of 10 Gb (from a local to disk to a NFS mounted disk) and monitor the growth on the copy, it grows at the same rate, about 40Mb/s. But if I copy the file using 'cat > ' the speed drops down from about 37Mb/s before the 2Gb limit to 1-7 Mb/s after that limit.

The OS: HP-UX 11.00
Patches: QPK march 2003 + PHNE_29210 + its requirements.

Backup software: Legato Networker 7.1

NFS server: NetApp filer, but same issue with other NFS servers.

Anyone seen this before? Any thought on a solution? HP and Legato haven't come up with a solution in the last 2 weeks...
Every problem has at least one solution. Only some solutions are harder to find.
19 REPLIES 19
David Burgess
Esteemed Contributor

Re: Strange performance issue

Hi Elmar,

I had an issue with Networker 6.1.3 recently. I was trying to restore an AIX client from an HP server running 11.11. The recovery took hours. I had the network cards at both ends of a crossover cable set to auto negotiate. They were both showing 100 full duplex. It turned out they weren't. When I forced them both to 100 full duplex the recovery speed up and all was ok.

It may not be related, but may help.

I assume you have all the latest patches from Legato and HP.

Regards,

Dave.
Elena Leontieva
Esteemed Contributor

Re: Strange performance issue

Elmar,

Make sure that NFS_CLIENT=1 in the

/etc/rc.config.d/nfsconf

Elena.

Elmar P. Kolkman
Honored Contributor

Re: Strange performance issue

Thanx for the responses, but:

David, this is a known issue to us. If that was the cause, the copy action would drop in performance too.

Elena, if we didn't have NFS_CLIENT=1 in the config file, we wouldn't be able to nfsmount the directory at all...

Better luck next time.
Every problem has at least one solution. Only some solutions are harder to find.
Highlighted
Elmar P. Kolkman
Honored Contributor

Re: Strange performance issue

Something we tested too: ftp the 10Gb file to the NFS disk. Monitoring the growth of the file returns same behaviour.
If we start another ftp, paralel, to the same NFS disk but with another filename, the behaviour is the same, even though there is still a ftp session writing...

Only difference with cat and networker: the performance drops to less then 1 Mb/s in total.
Every problem has at least one solution. Only some solutions are harder to find.
James Murtagh
Honored Contributor

Re: Strange performance issue

Hi Elmar,

Sounds like an interesting one. Once the file reaches 2GB the filesystem will need to use indirection to manage the extra blocks, although I would be surprised if it caused that much performance degredation. Certainly 30MB/s seems good though before this. Its probably easier I try to replicate this and trace it at the various levels. A few things that would help though :

> the nfsstat's from the server and client, also the connection protocol
> size of nfs server memory and what is dedicated to the buffer cache
> type of filesystem used on the nfs server, also state the patches for this fs type
> have you tried any other OS's?
> have you changed any network parameters using ndd on the systems?

Also, what tool are you using to monitor the throughput?

Cheers,

James.
Ralph Haefner
Frequent Advisor

Re: Strange performance issue

Could it have something to do with caching on your disk array (if you're using one)?

I've seen before where a copy slows down once you've transferred enough data to fill up the cache.

Just a wild guess but maybe that will give you an idea to pursue.
Jeff Schussele
Honored Contributor

Re: Strange performance issue

Hi Elmar,

Sounds like a ptach issue to me. I see you already have the NFS perf patch. Do you have the latest LVM cumulative - PHCO_24437?

http://www2.itrc.hp.com/service/patch/patchDetail.do?BC=patch.breadcrumb.search|&patchid=PHCO_24437&context=hpux:800:11:00

Since the network is involved here, I'd also look at the latest ARPA cumulative - PHNE_26771:

http://www2.itrc.hp.com/service/patch/patchDetail.do?BC=patch.breadcrumb.search|&patchid=PHNE_26771&context=hpux:800:11:00

And while you're at it - the latest JFS 3.3 cumulative - PHCO_29258:

http://www2.itrc.hp.com/service/patch/patchDetail.do?BC=patch.breadcrumb.search|&patchid=PHCO_29258&context=hpux:800:11:00

And if fibre channel is involved here, the latest fibre channel cumulative - PHKL_23939:

http://www2.itrc.hp.com/service/patch/patchDetail.do?BC=patch.breadcrumb.search|&patchid=PHKL_23939&context=hpux:800:11:00

And probably should also look at the latest streams cumulative - PHNE_27902:

http://www2.itrc.hp.com/service/patch/patchDetail.do?BC=patch.breadcrumb.search|&patchid=PHNE_27902&context=hpux:800:11:00

I guess the point is if you're not current on patches, now would be a good time to do so as a significant portion of patches are performance related.

HTH,
Jeff

PERSEVERANCE -- Remember, whatever does not kill you only makes you stronger!
A. Clay Stephenson
Acclaimed Contributor

Re: Strange performance issue

I suggest that you read over this paper carefully -- hopefully something will "click" inside your head. Dave did an outstanding job of presenting NFS performance issues and techniques for diagnosing problems.

http://docs.hp.com/hpux/onlinedocs/1435/NFSPerformanceTuninginHP-UX11.0and11iSystems.pdf
If it ain't broke, I can fix that.
Sridhar Bhaskarla
Honored Contributor

Re: Strange performance issue

Hi Elmar,

You can see if this is with NFS in fact by probably doing an ftp or scp. If cache on the system or on the storage is playing a part, then you would see that behaviour through the other tools too.

-Sri
You may be disappointed if you fail, but you are doomed if you don't try