service guard + "rpc" problem ..require immediate help...

[Sachin Soni_1]

Hi , all

I have a two node nfs cross mounted cluster.
server names are "x" and "y".
runnig 2 package "1" and "2" respectively.
package 2 failed and switched to server x.
now when i am trying to start in again on "y".
i am getting a lots of error mesgs.

now the scenario is different.
when i am starting cluster , package on y that is 2 is running while package 1 is not running.
it gives me rpc error.
it mounts all the file system , cluster comes up.
and than it gives me error "
"could not ping rpc.statd"
and than it starts unmounting all the file systems from "x".
y is still running with package 2.
when i try to check rpc daemons , all the daemons are running properly.
but when i am giving
#rpcinfo -u 127.0.0.1 (or localhost) 100020
it works on "y" but it gives me error on "x"
100020 is the port for rpc.lockd

but all th nfs daemons are running properly.
.but some time "y" shows high loadon it.
load is reaching to 6-7 in top output.
ninode is reaching to its maximum.

actually all are different issues, i am totally confused.

can any one help me on this.

Thanks in advance..
Sachin Soni

[Sanjay_6]

Hi sachin,

Look for the package log file in the diectory
/etc/cmcluster/pkg_name/pkg_name.log
also look for the error message in /var/adm/syslog/syslog.log

These two files log the messages involved in the startup of the cluster and package. Can you get something over there. Can you attach the pkg log file from /etc/cmcluster/pkg_name/pkg_name.log and syslog.log file.

Hope this helps.

Thanks

[Sanjay_6]

Hi Sachin,

What do you mean by NFS cross mounted cluster. Are you trying to convey that the disks are on shared paths.

Thanks

[Sachin Soni_1]

Hi,
The setup is of cross-mounting the filesystems i.e. both the servers export the file-systems and both of them mount all of the exported file systems.

The log file says what I mentioned in my last posting. I tried debugging the "mon"-script, it fails while doing -
rpcinfo -u 127.0.0.1 100020

Though the process rpc.lockd (program # 100020) is running, rpcinfo is not able to communicate with it! When the server fails to do so.. it stops the package on itself.


Thanks.

Sachin

[Rita C Workman · Accepted solution]

Take a look at this thread, note what Todd Morgan offered - - it may help you:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,1150,0x0d5d46ff9277d511abcd0090277a778c,00.html

Regards,
Rita

[Varghese Mathew]

Hi Sachin,
It seems u have no. of problems together.. I assume the meaning of ur description of cross mounted file systems in the cluster should be NFS automounted filesystems. Then if you feel the machine is overloaded in terms of nninode then you should be getting a message in the syslog.log file :file table full - verify the usage of ninode by "sar v 10 10" command.

I think Rita has given you a good hint how the rpc.lockd gives teouble.

Hope it helps..,
Cheers !!!
Varghese Mathew

[Sachin Soni_1]

HI All,
i solved problem , same way by removing host entry from /var/statmon/sm.
i was not able to understand whats this file,but it worked.
i though some kind of mess.now its clear.
thanks rita

actually someone tried to ** my machine.
he removed all the raw file entry from one vg's directory in /dev/vg05.
he changed maxswapchunks to very low value.
my secondry swap was not coming up.
changed ninode, nfiles.
and also removed sulogs and few more things.

i am sure it was not a "virus"


thanks to all.