HP will not fix performance problem on 11.11 NFS-client

[Hartmut Lang]

We discovered a serious performance problem on our HPUX 11.11 NFS-clients.
Running our cross-assembler with debug option on a 11.11 box we observed runtimes to be about 100(!) times slower compared to runing the same program on a 10.20 box (both boxes with same HW). The cross-assembler creates an output file on a NFS-mounted directory.

HP support (asking the guys in the HPUX labs) could find out the reason for this poor performance. The cross-assembler issues a big number of write() and lseek(..., SEEK_END) system calls. On the 11.11 NFS-client each lseek()-call triggers the biod to flush its 32k write-cache to the server. This is not the case on 10.20 NFS-clients (i don't know about 11.0 NFS-clients).

The problem can be reproduced using the attached C-code. Creating and running an executable "nfstest" leads to the following results (running the executable on a nfs-mounted directory):

10.20 NFS-client:
> time nfstest 1
real 0.4
user 0.0
sys 0.2

11.11 NFS-client:
> time nfstest 1
real 35.2
user 0.0
sys 0.5

HP told me that they will not(!) fix this problem with a patch. The problem might(!) be fixed with later HPUX releases. Since the problem will not be fixed soon, he even could not give me a SR-number about this issue.
Why will this not be fixed?
HP does not expect that many customers are affected by this poor performance. And the fix is not easy.

So i ask the forum:
- is there anybody, who observed the same performance problems on 11.11 NFS-clients?
- what do you think about HP decision not to fix this issue?

Hartmut

[Alex Glennie]

Hartmut,

Well it would appear the problem is with lseek and was reported to HP end of last year, I've found an SR relating to this and quoting your exact eg code ? : fyi : SR = CR# JAGad958000.

It would be interesting to find out from other ITRC users whether this occurs on non hpux NFS clients eg Solaris.

I suspect the problem is not that HP will not fix the problem in a patch but that it's not possible to at least without adversely affecting other functionality or at least without significant changes which will take time and need to be tsted etc ...

My advise would have to be to get back to HP quoting the above SR and seeking an update to confirm this is still the case. You may not like the answer but at least you'll have a better understanding of why it was given.

[Hartmut Lang]

Alex,

thanks for your response.
It was HP support who told me that there is no CR (or SR) number available. I asked for such a number. He could not give me one.
But he told me the same thing: since it is a significant change (needs lot of testing, ...) and only few customers are affected, the decision was not to implement it in a patch.

I did tests with the same program on Linux 2.4.10, SunOS 5.8, HPUX10.20, HPUX11.11.
Only HPUX11.11 showed the bad performance problem.

Hartmut

[Alex Glennie]

I can only say publically so much via these forums ... there's definitely a SR raised and the issue appears to be under investigation still. I suspect the problem will occur on 11.00 as I *suspect* it's due to the version of NFS as opposed to the O/S ie ver 3.

I'd still suggest getting an update from HP support on the status of the above SR if you are still unhappy.

[Ravi_8]

Hi,

I have few HP machines (L-class) came from HP with pre-installed 11i, while powering on (for the first time at our site) all machines hangs while checking for NFS clients.

I have to reboot (power off), get into single user mode, turn off NFS-SERVER/CLIENT flags(/etc/rc.config.d/nfsconf) then boot again.
There is a bug wrt NFS in 11i.

[Brian Hackley]

Hartmut,

I looked up that status of CR # JAGad95800, which is listed as SR # 8606226738 (SR is the customer viewable doc #, CR is that HP Lab bug id) and confirmed the information in the discussion thread. I know it sounds like something that should be simple to resolve, belive me! Unfortunately it is not, so what I suggest to you is that escalation of the issue should be done through the HP Response Center and/or your local HP Customer Support contacts, rather than through the ITRC Forums.
Have you considered running with 0 biod's and observe the results?? Edit the /etc/rc.config.d/nfsconf file NUM_NFSIODS=0 and then reboot. I wish I had better news, suggestions or coding changes beyond this!!

Ravi as far as your specific issue is concerned, you might want to break that one out into a seperate discussion thread...

Hope this helps,
-> Brian Hackley

[harry d brown jr]

Have you tried using cifs9000 (samba) to replace NFS?

live free or die
harry

[Hartmut Lang]

Many thanks for your responses!

Alex,
i still have to call support about the status of this SR. But since support told me last week, that this issue will not be fixed soon (or at all), i did not expect that i should call again this week to check the current status.

Ravi,
i think your problem is not related to this issue.

Brian,
thank you giving me the SR#.
Is there a way for me to lookup the related document? I could not find it on ITRC.

I did try runing 0 biods, with same (bad) performance.

The reason why i posted in the forum was to find out, if maybe other HPUX11.11 users are affected. I think they are! But they might not be so heavily affected as we are (because their application makes less frequent lseek() calls).

Harry,
i will look into using CIFS instead of NFS. But since out development environment is build around NFS, i propably can not change. I think it will be easier to stay with HPUX10.20 or switch to Linux or Solaris.


Hartmut

[sven verhaegen]

Hi , I have to agree with Brian on his explanation about the fix , indeed there might be some very good reasons why the labs cannot implement the fix at first however I do feel that if your environment is enough at peril you might just return to your local responce center and ask for an escalation ... if it does affect your bussiness seriously and if you can prove that you stand to loose money on this HP might be entraved to see implementation differently and still consider pushing the JAG forward .. it doesn't hurt to try

[Hartmut Lang]

Sven,

our point is not that "we are loosing money" if this issue is not fixed.

The point is, that i have to tell my manager "well, using the HPUX 11.11 machine you have to wait 4min everytime you run the assembler, on the Linux box it will complete in less than 2 seconds".

The point is that we have to decide if HPUX is the "right" development environment or not. And we have some troubles with tools not being available for HPUX, expensive SW and HW, unfixed performance issues(!),...

There are many pieces, which lead to a decision to switch to Solaris or Linux, or stay with HPUX.


Hartmut

[Bill Hassell]

This is the type of issue that can be brought up with the Interex Advocacy survey as well as discussing it with lab management at HP World in a few weeks. It would seem to be a very big problem for a lot of people.

[Leif Halvarsson_2]

Hi

I read this because NFS-performance is important for us. I compiled the program on a rp2430 running HP-UX 11.11 and run it in directory on a local disk and on a NFS-mounted.
But I get almost exact the same performance:

### The local disk ####
# cd
# timex ./a.out

real 0.35
user 0.01
sys 0.05

### the NFS-mounted ####

# cd /home
# su cnt
% ./a.out
% timex ./a.out

real 0.35
user 0.01
sys 0.04

Do I somthing wrong.

[joe hepworth]

We are also having NFS Performance problems from an N-Class Server running HP-UX 11.00 to a NAS Filer over a Gigabit Ethernet link and we didn't know why although we did know that there were a lot of lseeks. We have run your test program on 10.20 and 11.00 over a 100meg link and gigabit link and it proves that it is 11.00 that is the problem. We will also be contacting HP Support about this to get it escalated as it is causing us serious problems.

Thanks for showing us where the tunnel starts.

[Hartmut Lang]

Bill,

I cannot be at HP world. But if someone else can bring up this issue, great!


Leif,

You have to use an argument (see program code), that the lseek() call is used.
Try this:
times ./a.out 1

How are your results?


Alan,

Did you run the program with and without argument (see my answer to Leif)?
Maybe you can tell us about the answer from HP support.


Hartmut

[Leif Halvarsson_2]

Hi

I run this test again using an argument (1). I also run on a Sun box. The mounted filesystem is the same for HP and Sun but The Sun computer is not of the same performance as the HP (300 MHz Ultra 2).

rp2430 local
real 0.37
user 0.01
sys 0.06

rp 2430 NFS
real 19.94
user 0.01
sys 0.32



Sun 450 local
real 0.11
user 0.02
sys 0.08

Sun 450 NFS
real 4.16
user 0.03
sys 0.08

[Hartmut Lang]

Leif,

it looks almost like the results i got.
What you should messure, is the difference between calling the executable with and without argument (both on a nfs-mounted filesystem).

From your previous posting i get your results
0.35 seconds without argument (no lseek)
19.94 seconds with argument (with lseek)

But both values should be almost the same (and they are on my HPUX10.20 box), that's the problem.

You can do a test on your sun box with and without argument.

Hartmut

[Hartmut Lang]

Last update:

HP has this issue (SR#8606226738) in the "enhancement" status.

It is unclear when/if it will be fixed!

Hartmut

[Leif Halvarsson_2]

Hi

Test on a Sun 450 300MHz with and without argument.


c170% timex ./a.out 1

real 2.57
user 0.00
sys 0.11

c170% timex ./a.out

real 2.70
user 0.00
sys 0.11

[H.Merijn Brand (procura]

We have lots of NFS performance issues too. We've already disabled all outgoing NFS from AIX-4.2

A. Compiled with HP-C -O2 on HP-UX 11.00
=========================================
1. HP-UX 11.00 => local machin e (no NFS)

real 0.36
user 0.01
sys 0.05

2. HP-UX 11.00 => other HP-UX 11.00

real 0.43
user 0.01
sys 0.03

3. HP-UX 11.00 => HP-UX 10.20

real 8.86
user 0.01
sys 0.03

4. HP-UX 11.00 => AIX 4.3

real 3.59
user 0.01
sys 0.03

5. HP-UX 11.00 => AIX 4.2

real 2:51.40
user 0.01
sys 0.03

B. Compiled with HP-C +DA2.0w -O2 on HP-UX 11.00
=========================================
1. HP-UX 11.00 => local machin e (no NFS)

real 0.38
user 0.01
sys 0.06

2. HP-UX 11.00 => other HP-UX 11.00

real 0.48
user 0.01
sys 0.04

3. HP-UX 11.00 => HP-UX 10.20

real 6.91
user 0.01
sys 0.03

4. HP-UX 11.00 => AIX 4.3

real 2.55
user 0.01
sys 0.03

5. HP-UX 11.00 => AIX 4.2

real 2:51.34
user 0.01
sys 0.04

C. Compiled with GNU gcc -O2 on HP-UX 11.00
=========================================
1. HP-UX 11.00 => local machin e (no NFS)

real 0.35
user 0.01
sys 0.05

2. HP-UX 11.00 => other HP-UX 11.00

real 0.31
user 0.01
sys 0.03

3. HP-UX 11.00 => HP-UX 10.20

real 8.36
user 0.01
sys 0.03

4. HP-UX 11.00 => AIX 4.3

real 2.42
user 0.01
sys 0.03

5. HP-UX 11.00 => AIX 4.2

real 2:51.34
user 0.01
sys 0.03

D. Compiled with GNU gcc -O2 on HP-UX 10.20
=========================================
1. HP-UX 10.20 => local machin e (no NFS)

real 0.65
user 0.01
sys 0.13

2. HP-UX 10.20 => HP-UX 11.00

real 3.32
user 0.01
sys 0.10

3. HP-UX 10.20 => AIX 4.3

real 2.55
user 0.01
sys 0.10

4. HP-UX 10.20 => AIX 4.2

real 3.99
user 0.01
sys 0.10

E. Compiled with vac -O2 on AIX 4.3
=========================================
1. AIX 4.3 => local machin e (no NFS)

real 0.12
user 0.01
sys 0.09

2. AIX 4.3 => HP-UX 11.00

real 0.43
user 0.01
sys 0.11

3. AIX 4.3 => HP-UX 10.20

real 2.52
user 0.01
sys 0.14

4. AIX 4.3 => AIX 4.2

real 19.69
user 0.01
sys 0.11

F. Compiled with gcc -O2 on AIX 4.3
=========================================
1. AIX 4.3 => local machin e (no NFS)

real 0.11
user 0.00
sys 0.08

2. AIX 4.3 => HP-UX 11.00

real 0.42
user 0.01
sys 0.12

3. AIX 4.3 => HP-UX 10.20

real 2.36
user 0.00
sys 0.10

4. AIX 4.3 => AIX 4.2

real 15.28
user 0.00
sys 0.17

All HP-UX filesystems used are VxFS, AIX filesystems used are JFS.

All this proves it's and NFS-3 to NFS-2 problem (AIX-4.3 is NFS-3, AIX-4.2 is NFS-2)

Did I miss any obvious tests?

HTH, still also waiting for better NFS times ...

[H.Merijn Brand (procura]

Previous list was without argument. This list is /with/ argument 1. Quite a difference!

A. Compiled with HP-C -O2 on HP-UX 11.00
=========================================
1. HP-UX 11.00 => local machin e (no NFS)

real 0.36
user 0.01
sys 0.06

2. HP-UX 11.00 => other HP-UX 11.00

real 1:06.69
user 0.02
sys 2.11

3. HP-UX 11.00 => HP-UX 10.20

real 1:36.11
user 0.02
sys 2.10

4. HP-UX 11.00 => AIX 4.3

real 2:13.08
user 0.02
sys 2.86

5. HP-UX 11.00 => AIX 4.2

real 1:58.02
user 0.02
sys 2.74

B. Compiled with HP-C +DA2.0w -O2 on HP-UX 11.00
=========================================
1. HP-UX 11.00 => local machin e (no NFS)

real 0.39
user 0.02
sys 0.06

2. HP-UX 11.00 => other HP-UX 11.00

real 1:08.71
user 0.03
sys 3.07

3. HP-UX 11.00 => HP-UX 10.20

real 1:36.83
user 0.03
sys 3.21

4. HP-UX 11.00 => AIX 4.3

real 2:13.01
user 0.03
sys 3.55

5. HP-UX 11.00 => AIX 4.2

real 1:49.56
user 0.03
sys 3.75

C. Compiled with GNU gcc -O2 on HP-UX 11.00
=========================================
1. HP-UX 11.00 => local machin e (no NFS)

real 0.35
user 0.01
sys 0.06

2. HP-UX 11.00 => other HP-UX 11.00

real 1:04.92
user 0.02
sys 0.81

3. HP-UX 11.00 => HP-UX 10.20

real 1:46.65
user 0.02
sys 0.66

4. HP-UX 11.00 => AIX 4.3

real 2:09.94
user 0.02
sys 0.97

5. HP-UX 11.00 => AIX 4.2

real 1:43.82
user 0.02
sys 0.89

D. Compiled with GNU gcc -O2 on HP-UX 10.20
=========================================
1. HP-UX 10.20 => local machin e (no NFS)

real 1.24
user 0.03
sys 0.18

2. HP-UX 10.20 => HP-UX 11.00

real 3.56
user 0.02
sys 0.17

3. HP-UX 10.20 => AIX 4.3

real 2.40
user 0.02
sys 0.16

4. HP-UX 10.20 => AIX 4.2

real 3.84
user 0.02
sys 0.16

E. Compiled with vac -O2 on AIX 4.3
=========================================
1. AIX 4.3 => local machin e (no NFS)

real 0.15
user 0.00
sys 0.11

2. AIX 4.3 => HP-UX 11.00

real 72.31
user 0.02
sys 4.54

3. AIX 4.3 => HP-UX 10.20

real 100.47
user 0.04
sys 3.42

4. AIX 4.3 => AIX 4.2

real 116.35
user 0.05
sys 4.45

F. Compiled with gcc -O2 on AIX 4.3
=========================================
1. AIX 4.3 => local machin e (no NFS)

real 0.12
user 0.00
sys 0.12

2. AIX 4.3 => HP-UX 11.00

real 71.54
user 0.04
sys 4.57

3. AIX 4.3 => HP-UX 10.20

real 100.75
user 0.09
sys 4.64

4. AIX 4.3 => AIX 4.2

real 118.27
user 0.05
sys 4.46

Now, come on HP ... *DO* something with this info!

[Cheryl Griffin]

Hartmut,
>>i still have to call support about the status of this SR. But since support told me last week, that this issue will not be fixed soon (or at all), i did not expect that i should call again this week to check the current status. >>

Actually you don't have to call support for a status. You can contact someone in support (easiest way is to log a support case under the ITRC - Software Call Manager, or use Web Collaboration {look for a new collab icon from the Knowledge Base) and ask the support person to log your information in the SR Notify Database. When the fix is released you'll be automatically notified by email. We will need:
Your Name
Company Name
Email address
SR# of interest

Cheryl

[H.Merijn Brand (procura]

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x6e9493e260b0d611abdb0090277a778c,00.html (this thread) will do quite nicely for HP's updates on the subject :)

Thanks anyway for sharing.

[Hartmut Lang]

Cheryl,

Thanks for this information. I will log a support case.



Procura,

Thanks for doing this intensive testing.
And for sharing my view, that something needs to be done!



Hartmut

[Pete Randall]

Just a thought: NFS has always been a bottle neck - do you suppose that this supports the case for SAN's? They eliminate NFS, yet still allow different servers to share data.

For What It's Worth,
Pete

[H.Merijn Brand (procura]

Pete, that would be no solution for us. We use NFS mainly for source synchronisation. Running a diff over NFS is quite informative before updating the sources to the other machine, especially if there are machine dependancies in the sources (#ifdef's and such)

If we were to drop the NFS for performance reasons, I guess rsync would be the next choice.

NFS performance has been OK for some time in HP-UX 11.00 world though. I don't recall the patch release that broke it's performance, but of top of my head my guesstimate would be March 2001