yp client restart rebooted the MCSG node ?

[Mahesh Babbar]

Hi Gurus,

Environment is: HPUX 11i
2 node MCSG Cluster

Packages were running on one of the nodes.

At 9:11 AM, the nis client was tried to be restarted and following were the messages recorded in the /var/adm/syslog/syslog.log file.

Nov 2 09:11:49 ipdfnas2 syslog: ypbind: /var/yp/secureservers line 17: netmask does not match network
Nov 2 09:16:34 ipdfnas2 vmunix: msgcnt 492 vxfs: mesg 001: vx_nospace - /dev/sram/rdsramxp1 file system full (1 block extent
)
Nov 2 09:17:50 ipdfnas2 vmunix: msgcnt 494 vxfs: mesg 001: vx_nospace - /dev/sram/rdsramxp1 file system full (1 block extent
)
Nov 2 09:15:47 ipdfnas2 syslog: ypbind: /var/yp/secureservers line 17: netmask does not match network
Nov 2 09:20:06 ipdfnas2 syslogd: restart
Nov 2 09:25:08 ipdfnas2 rpcbind: check_netconfig: Found CLTS loopback transport
Nov 2 09:25:08 ipdfnas2 rpcbind: check_netconfig: Found COTS loopback transport
Nov 2 09:25:08 ipdfnas2 rpcbind: check_netconfig: Found COTS ORD loopback transport
Nov 2 09:25:08 ipdfnas2 rpcbind: init_transport: check binding for udp
Nov 2 09:25:08 ipdfnas2 rpcbind: init_transport: check binding for tcp
Nov 2 09:25:08 ipdfnas2 rpcbind: init_transport: check binding for ticlts
Nov 2 09:25:08 ipdfnas2 rpcbind: init_transport: check binding for ticotsord
Nov 2 09:25:08 ipdfnas2 rpcbind: init_transport: check binding for ticots
Nov 2 09:25:08 ipdfnas2 syslog: ypbind: /var/yp/secureservers line 17: netmask does not match network
Nov 2 09:25:15 ipdfnas2 syslog: svc_getreqset: No transport handle for fd 257
Nov 2 09:25:15 ipdfnas2 syslog: svc_getreqset: No transport handle for fd 259
Nov 2 09:25:15 ipdfnas2 syslog: svc_getreqset: No transport handle for fd 260
Nov 2 09:25:15 ipdfnas2 syslog: svc_getreqset: No transport handle for fd 261
Nov 2 09:25:15 ipdfnas2 syslog: svc_getreqset: No transport handle for fd 262
Nov 2 09:25:15 ipdfnas2 syslog: svc_getreqset: No transport handle for fd 263
Nov 2 09:25:15 ipdfnas2 syslog: svc_getreqset: No transport handle for fd 264
Nov 2 09:25:15 ipdfnas2 syslog: svc_getreqset: No transport handle for fd 265
Nov 2 09:25:15 ipdfnas2 syslog: svc_getreqset: No transport handle for fd 266
Nov 2 09:25:15 ipdfnas2 syslog: svc_getreqset: No transport handle for fd 268
Nov 2 09:25:15 ipdfnas2 syslog: svc_getreqset: No transport handle for fd 269
Nov 2 09:25:15 ipdfnas2 syslog: svc_getreqset: No transport handle for fd 270
Nov 2 09:25:15 ipdfnas2 syslog: svc_getreqset: No transport handle for fd 271
Nov 2 09:25:15 ipdfnas2 syslog: svc_getreqset: No transport handle for fd 273
Nov 2 09:25:15 ipdfnas2 syslog: svc_getreqset: No transport handle for fd 275
Nov 2 09:25:15 ipdfnas2 syslog: svc_getreqset: No transport handle for fd 276
Nov 2 09:25:15 ipdfnas2 syslog: svc_getreqset: No transport handle for fd 280
Nov 2 09:25:15 ipdfnas2 syslog: svc_getreqset: No transport handle for fd 284
Nov 2 09:25:15 ipdfnas2 syslog: svc_getreqset: No transport handle for fd 285
Nov 2 09:25:15 ipdfnas2 syslog: svc_getreqset: No transport handle for fd 287
Nov 2 09:25:15 ipdfnas2 syslog: svc_getreqset: No transport handle for fd 291
Nov 2 09:25:15 ipdfnas2 syslog: svc_getreqset: No transport handle for fd 292
Nov 2 09:25:15 ipdfnas2 syslog: svc_getreqset: No transport handle for fd 293
Nov 2 09:25:15 ipdfnas2 syslog: svc_getreqset: No transport handle for fd 294
Nov 2 09:25:15 ipdfnas2 syslog: svc_getreqset: No transport handle for fd 295
Nov 2 09:25:15 ipdfnas2 syslog: svc_getreqset: No transport handle for fd 298
Nov 2 09:25:15 ipdfnas2 syslog: svc_getreqset: No transport handle for fd 299
Nov 2 09:26:12 ipdfnas2 cmcld: Service NFSMON4 terminated due to an exit(1).
Nov 2 09:26:12 ipdfnas2 cmcld: Automatically restarted service NFSMON4 for the 1st time after failure.
Nov 2 09:26:13 ipdfnas2 cmcld: Service NFSMON3 terminated due to an exit(1).
Nov 2 09:26:13 ipdfnas2 cmcld: Automatically restarted service NFSMON3 for the 1st time after failure.
Nov 2 09:26:17 ipdfnas2 cmcld: Service NFSMON1 terminated due to an exit(1).

Also the messages in /etc/cmcluster//nfs.mon.log file is :
HANFS MON -- Nov 2 09:26:19 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/nfsd. Retrying 4 more times before giving up
HANFS MON -- Nov 2 09:26:19 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.lockd. Retrying 4 more times before giving up
HANFS MON -- Nov 2 09:26:19 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.statd. Retrying 4 more times before giving up
HANFS MON -- Nov 2 09:26:28 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.mountd. Retrying 3 more times before giving up
HANFS MON -- Nov 2 09:26:29 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/nfsd. Retrying 3 more times before giving up
HANFS MON -- Nov 2 09:26:29 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.lockd. Retrying 3 more times before giving up
HANFS MON -- Nov 2 09:26:29 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.statd. Retrying 3 more times before giving up
HANFS MON -- Nov 2 09:26:29 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.mountd. Retrying 3 more times before giving up
HANFS MON -- Nov 2 09:26:30 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/nfsd. Retrying 3 more times before giving up
HANFS MON -- Nov 2 09:26:30 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.lockd. Retrying 3 more times before giving up
HANFS MON -- Nov 2 09:26:30 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.statd. Retrying 3 more times before giving up
HANFS MON -- Nov 2 09:26:39 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.mountd. Retrying 2 more times before giving up
HANFS MON -- Nov 2 09:26:40 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/nfsd. Retrying 2 more times before giving up
HANFS MON -- Nov 2 09:26:40 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.lockd. Retrying 2 more times before giving up
HANFS MON -- Nov 2 09:26:40 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.statd. Retrying 2 more times before giving up
HANFS MON -- Nov 2 09:26:40 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.mountd. Retrying 2 more times before giving up
HANFS MON -- Nov 2 09:26:41 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/nfsd. Retrying 2 more times before giving up
HANFS MON -- Nov 2 09:26:41 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.lockd. Retrying 2 more times before giving up
HANFS MON -- Nov 2 09:26:41 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.statd. Retrying 2 more times before giving up
HANFS MON -- Nov 2 09:26:50 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.mountd. Retrying 1 more times before giving up
HANFS MON -- Nov 2 09:26:51 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/nfsd. Retrying 1 more times before giving up
HANFS MON -- Nov 2 09:26:51 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.lockd. Retrying 1 more times before giving up
HANFS MON -- Nov 2 09:26:52 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.statd. Retrying 1 more times before giving up
HANFS MON -- Nov 2 09:26:52 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.mountd. Retrying 1 more times before giving up
HANFS MON -- Nov 2 09:26:52 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/nfsd. Retrying 1 more times before giving up
HANFS MON -- Nov 2 09:26:53 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.lockd. Retrying 1 more times before giving up
HANFS MON -- Nov 2 09:26:53 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.statd. Retrying 1 more times before giving up
HANFS MON -- Nov 2 09:27:02 - Node "ipdfnas2" - Error: Could not ping /usr/sbin/rpc.mountd
HANFS MON -- Nov 2 09:27:03 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.mountd. Retrying 4 more times before giving up
HANFS MON -- Nov 2 09:27:03 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/nfsd. Retrying 4 more times before giving up
HANFS MON -- Nov 2 09:27:03 - Node "ipdfnas2" - Error: Could not ping /usr/sbin/rpc.mountd
HANFS MON -- Nov 2 09:27:04 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.lockd. Retrying 4 more times before giving up
HANFS MON -- Nov 2 09:27:04 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.mountd. Retrying 4 more times before giving up
HANFS MON -- Nov 2 09:27:04 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.statd. Retrying 4 more times before giving up
HANFS MON -- Nov 2 09:27:05 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/nfsd. Retrying 4 more times before giving up
HANFS MON -- Nov 2 09:27:05 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.lockd. Retrying 4 more times before giving up
HANFS MON -- Nov 2 09:27:06 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.statd. Retrying 4 more times before giving up
HANFS MON -- Nov 2 09:27:15 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.mountd. Retrying 3 more times before giving up
HANFS MON -- Nov 2 09:27:15 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/nfsd. Retrying 3 more times before giving up
HANFS MON -- Nov 2 09:27:15 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.lockd. Retrying 3 more times before giving up
HANFS MON -- Nov 2 09:27:16 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.statd. Retrying 3 more times before giving up
HANFS MON -- Nov 2 09:27:16 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.mountd. Retrying 3 more times before giving up
HANFS MON -- Nov 2 09:27:16 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/nfsd. Retrying 3 more times before giving up
HANFS MON -- Nov 2 09:27:16 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.lockd. Retrying 3 more times before giving up
HANFS MON -- Nov 2 09:27:17 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.statd. Retrying 3 more times before giving up
HANFS MON -- Nov 2 09:27:26 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.mountd. Retrying 2 more times before giving up
HANFS MON -- Nov 2 09:27:26 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/nfsd. Retrying 2 more times before giving up
HANFS MON -- Nov 2 09:27:26 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.lockd. Retrying 2 more times before giving up
HANFS MON -- Nov 2 09:27:27 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.statd. Retrying 2 more times before giving up
HANFS MON -- Nov 2 09:27:27 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.mountd. Retrying 2 more times before giving up
HANFS MON -- Nov 2 09:27:27 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/nfsd. Retrying 2 more times before giving up
HANFS MON -- Nov 2 09:27:27 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.lockd. Retrying 2 more times before giving up
HANFS MON -- Nov 2 09:27:28 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.statd. Retrying 2 more times before giving up
HANFS MON -- Nov 2 09:27:37 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.mountd. Retrying 1 more times before giving up
HANFS MON -- Nov 2 09:27:37 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/nfsd. Retrying 1 more times before giving up
HANFS MON -- Nov 2 09:27:37 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.lockd. Retrying 1 more times before giving up
HANFS MON -- Nov 2 09:27:38 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.statd. Retrying 1 more times before giving up
HANFS MON -- Nov 2 09:27:38 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.mountd. Retrying 1 more times before giving up
HANFS MON -- Nov 2 09:27:38 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/nfsd. Retrying 1 more times before giving up
HANFS MON -- Nov 2 09:27:38 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.lockd. Retrying 1 more times before giving up
HANFS MON -- Nov 2 09:27:39 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.statd. Retrying 1 more times before giving up
HANFS MON -- Nov 2 09:27:48 - Node "ipdfnas2" - Error: Could not ping /usr/sbin/rpc.mountd
HANFS MON -- Nov 2 09:27:49 - Node "ipdfnas2" - Error: Could not ping /usr/sbin/rpc.mountd

Needless to say , it rebooted the node and packages were shifted to second box.

Q is : why is that re-staring the NIS client resulted in rebooting of the node.

TIA,



Mahesh

[Sridhar Bhaskarla]

Hi Mahesh,

The messages of interest are

1. Nov 2 09:20:06 ipdfnas2 syslogd: restart
2. Nov 2 09:25:08 ipdfnas2 syslog: ypbind: /var/yp/secureservers line 17: netmask does not match network
3. Nov 2 09:26:12 ipdfnas2 cmcld: Service NFSMONX terminated due to an exit(1). (repeated)
4. HANFS MON -- Nov 2 09:27:39 - Node "ipdfnas2" - WARNING: couldn't ping /usr/sbin/rpc.statd. Retrying 1 more times before giving up (various)

Now the questions are

1. Is /etc/services one of the maps?.
2. Do you have "SERVICE_FAIL_FAST_ENABLED" set to YES in your package configuration file in serviceguard?.

If both the above are true, then it can explain why the node TOC'ed. Missing /etc/services or any network related component due to ypbind restart might have caused your NFS packages to go down and might have caused the node to go down due to 2 above.

However, I suggest you see the following

1. /var/tombstones/ts99 file. See if you there are any timestamps in it.
2. Look at /etc/shutdownlog file.
3. See if you have any core dump under /var/adm/crash directory. It's a good idea to run Q4 analysis and provide the output to HP.

-Sri

[Mahesh Babbar]

Sri,

/etc/services is not in map.

But SERVICE_FAIL_FAST_ENABLED is enabled.

Parallaly I have raised the ticket with HP.

Regards

Mahesh

[Sridhar Bhaskarla]

HI Mahesh,

It's not necessarily services file it can be anything like /etc/hosts or nsswitch.conf that affected the network services temporarily causing your packages to fail. Since you have enabled SERVICE_FAIL_FAST_ENABLED, serviceguard toc'ed the node.

-Sri