On the HP-UX NFS client, issue the command: showmount -e <NFS server hostname>. This will confirm the NFS client can send RPC requests to the NFS server and show the list of shared/exported filesystems. It should also show if the NFS client is in the access lists on the server, assuming there is one.
Dave
I work at HPE
HPE Support Center offers support for your HPE services and products when and how you need it. Get started with HPE Support Center today.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
So the server (Solaris) is sharing /projects, the NFS client (HP-UX) is mounting /projects. What specifically is not working? Is the problem that users on the client cannot read or write files in /projects? Is the problem that users cannot lock files in /projects? Can you be very specific about what the exact problem is and are you able to reproduce the problem outside of your application?
I work at HPE
HPE Support Center offers support for your HPE services and products when and how you need it. Get started with HPE Support Center today.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
> Because for every other system that .model file (inside /projects)working perfectly
> for catia application but only in this system at the time of using catia application
> when mounting to that particular project and using the .model file it is getting restarted
> by showing some s10 or s11 error .
I'm not familiar enough with the Catia application to know what an S10 or S11 error is or what it means. Have you checked with Catia to learn what these errors indicate? This is why I recommended you try to reproduce the *symptom* outside of Catia. Aside from the application errors, what is the actual symptom you experience? Does the application hang? Does it end?
The last time I dealt with a Catia problem the issue turned out to be a file locking problem. That might explain your situation, where users can read/write in /projects but Catia throws errors.
If you can explain exactly what the symptom is, and check with Catia to learn what an s10/s11 error means, it would help us narrow down the underlying cause of the problem.
Dave
I work at HPE
HPE Support Center offers support for your HPE services and products when and how you need it. Get started with HPE Support Center today.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
As I said, I have no idea what the CATIA errors mean. I'm surprised someone from CATIA cannot tell you what these errors mean, or at least offer some suggestions as to how to determine the root cause of the problem.
The fact that you're able to load the model file from a local directory but not an NFS directory leads me back to my original guess that this may be an NFS file locking problem. Since we have nothing else to go on we might as well rule out file locking as the problem.
Try this:
Log into the NFS client system (11.11) as root and issue the commands:
# ps -e | grep rpc.lockd
# ps -e | grep rpc.statd
Find the process ID numbers of both rpc.lockd and rpc.statd. Then send each of these running daemons a SIGUSR2 signal:
# kill -17 <lockd pid>
# kill -17 <statd pid>
That will enable debug logging of these daemons. Now reproduce the CATIA problem. Hopefully that will only take a minute or two to reproduce. Once you've reproduce the problem send another SIGUSR2 signal to the daemons:
# kill -17 <lockd pid>
# kill -17 <statd pid>
That should disable debug logging of these daemons. The debug logfiles should be in the /var/adm directory and they should be called something like rpc.lockd.log and rpc.statd.log. Collect those files and attach them here. Also, issue the command on the 11.11 system:
# ls -l /etc/sm
# ls -l /etc/sm.bak
Copy/paste the ls output here so I can see which files reside in these two directories. This information should at least tell us if NFS file locking is involved.
Regards,
Dave
I work at HPE
HPE Support Center offers support for your HPE services and products when and how you need it. Get started with HPE Support Center today.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
I looked at the logfiles and there is nothing obviously wrong in the log files. It does show a lot of lock/test/unlock activity, which confirms my suspicion that NFS file locking is likely involved. A couple of questions:
- Which server is "10.243.0.54"?
- Is there a reason why system argo cannot resolve this IP address to a hostname?
- From the collected data, do you know exactly when the problem occurs? I could focus on that time period in the log files to see what kind of activity was happening during the failure.
Dave
I work at HPE
HPE Support Center offers support for your HPE services and products when and how you need it. Get started with HPE Support Center today.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
Most NFS file locking problems are caused by hostname resolution problems.
bash-2.04# ll /var/statmon/sm
total 48
--w------- 1 root root 12 Nov 22 09:38 10.243.0.54
--w------- 1 root root 5 Nov 22 09:38 argo
--w------- 1 root root 5 Nov 22 09:38 zeus
The fact that the rpc.lockd and rpc.statd daemons on the HP-UX client are able to resolve zeus at one point but fail to resolve it moments later seems like something we need to fix before moving on. Some quick questions:
1) Does zeus have multiple IP addressess?
2) Are you NFS mounting filesystems from zeus using different IP addresses?
3) Are you NFS mounting some filesystems using the hostname "zeus" and other filesystems using zeus' IP address?
4) How do you have your hostname resolution configured in your /etc/nsswitch.conf file? I assume you have your "hosts" entry pointing to "files" first since that's what nslookup used in your example.
One thing to try:
A) Make sure argo resolves the hostname and IP address for zeus to the same IP address. (you already did this)
B) Make sure zeus resolves the hostname and IP address for argo to the same IP address.
C) Terminate the running rpc.lockd and rpc.statd daemons on argo
D) Terminate the running rpc.lockd and rpc.statd daemons on zeus
E) Remove the files in /var/statmon/sm on both systems
F) Re-start the rpc.lockd and rpc.statd daemons on both systems
G) Wait 1 minute for the grace period to expire
H) On argo, try locking a file in one of the NFS filesystems mounted from zeus using a program other than CATIA. If you do not have a simple lock/unlock program write to me at dave.olker@hp.com and I'll provide one for you.
Once you've made absolutely sure that the non-CATIA program can successfully lock and unlock files on zeus, then try running CATIA again and see if it works. If you are unable to successfully lock and unlock files on zeus using a test program then we can collect further data and figure out why.
Regards,
Dave
I work at HPE
HPE Support Center offers support for your HPE services and products when and how you need it. Get started with HPE Support Center today.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
> > Zeus is the NFS master server .. if I terminate the process then rest of the
> > NFS share may get disturbed / unmounted from client.
There's no chance that an NFS share will suddenly be unmounted on the NFS client, even if you did terminate and restart the locking daemons.
> Yes, that was always the quandary when cleaning up lock demons.
> Is it better to terminate everything to just fix one system?
> And if you remove the process/files on zeus, what about all of the
> other machines that mount from zeus, does it snowball to every machine?
Here's the 10,000 foot view of how file locking works with NFS version 3.
Lock Reqeust Time
1) Client requests a lock
a. Client creates an entry in /var/statmon/sm for itself and the NFS server it's locking with
2) Server starts processing the lock
b. Server creates an entry in /var/statmon/sm for itself and the NFS client requesting the lock
Over time the NFS client will have entries in /var/statmon/sm for itself and any NFS server it has requested a lock from. Similarly, the NFS server will have /var/statmon/sm entries for itself and any NFS clients it has serviced a lock request for.
Lock Recovery Time
When a client or server reboots (or a reboot is simulated by terminating and restarting the rpc.lockd and rpc.statd daemons) the rpc.statd daemon will take any files in /var/statmon/sm and move them to /var/statmon/sm.bak and then begin the process of notifying the remote systems that a recovery event has happened. Once the local system establishes a good connection with the rpc.statd on the remote system it removes the entry from /var/statmon/sm.bak. That's why any files remaining in /var/statmon/sm.bak after a lock recovery are suspect. It usually means either the remote system is no longer around, is not up, or the lock daemons are not started on that system.
If the local system performing the recovery is an NFS client, it is telling the remote NFS servers "I've just rebooted, so any locks you're holding for me are bogus. Release any of my stale locks." This makes sense in a client reboot case because any applications that were holding locks on the server are well and truly gone, so the server should release them and let others acquire them.
If the local system performing the recovery is an NFS server, it is telling the remote NFS clients "I've just rebooted so any locks you think you are holding with me are gone. I'll now give you 1 minute to reclaim those locks before I open the floodgates and let any NFS client grab locks." This is called the grace period, and only "reclaim" locks are allowed during the grace period.
What happens if you delete /var/statmon/sm (or sm.bak) entries and terminate/start lockd/statd?
If this is done on an NFS client, any applications holding locks are screwed because the locks they were holding are now gone. To make things worse, because the /var/statmon/sm entries were deleted, the client will not send a crash recovery notification to the NFS server. That means the NFS server could potentially be holding a bunch of locks for the client that will never be released (without terminating and restarting lockd/statd on the server). This means the original client cannot reclaim its lock (because the stale copy of the previous lock in the servers' queue is still there so the server considers the new request a conflicting lock request), but any other NFS clients also cannot get those file locks.
If this is done on the NFS server, any applications on remote NFS clients are screwed because the server just destroyed all of it's locks. Because the /var/statmon/sm entries were deleted, the server has no idea which NFS clients to contact to reclaim the locks they're still holding. This means clientA could be the legitimate lock owner but clientB could ask the server for the same lock and the server will grant it because clientA's lock was destroyed during this process of deleting /var/statmon/sm files and restarting the daemons.
Good times, no?
> Perhaps Dave has some suggestions.
As you can see from the above description, it's not too hard to imagine scenarios where NFS file locking can get wedged resulting in lock denial, application failure, etc. In crash & burn environments where you're just trying to get things working properly there's usually little harm in terminating/restarting the daemons and wiping out the statmon files. That typically wipes the slate clean and lets the daemons re-establish a good connection.
From the earlier description it sounded like this environment was more of a test/dev environment where things were being tested and systems were moving around. For that reason I suggested shaking the systems like an Etch-a-Sketch and starting over. Apparently that's not the situation.
Since Zeus is a production NFS server we can postpone any potentially destructive modifications to Zeus until other possibilities are ruled out. For the time being we can focus on the NFS client. Assuming CATIA is the only application doing file locking (which it appeared to be from the log files), you should be able to terminate/start the daemons on the client without hurting anything. Again, you'll have to wait for 60 seconds before testing since the grace period will be in effect.
I've attached a sample file locking program you can use to test outside of CATIA. The program should compile fine with default options. To try locking a file the syntax would be "lockprog -f <file>". You can try running this against a local file first to ensure the program compiled correctly. Then try running this against a file in an NFS-mounted filesystem on server Zeus.
Hope this helps,
Dave
I work at HPE
HPE Support Center offers support for your HPE services and products when and how you need it. Get started with HPE Support Center today.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
No - do NOT do a normal terminate. The SIGUSR2 is correct. It does not terminate the daemon, it merely tells the daemon to toggle debug logging on and off. We're not trying to terminate any daemons here, just capture logging.
Dave
I work at HPE
HPE Support Center offers support for your HPE services and products when and how you need it. Get started with HPE Support Center today.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]