topic Re: Package is down in one node in Operating System - HP-UX

Package is down in one node

pkumar97 — Thu, 26 Feb 2009 20:34:45 GMT

Dear Team,
we are facing an issue on our one of the production box , the thing is when we tried to halt the package on one node and run the same package on another node, the pakcage successfully halted also the connection to the current node lost. now we are not able to login to the server and we cannot run the same package in another node

root@aagcapp / > cmviewcl

CLUSTER STATUS
AGIERPCL up

NODE STATUS STATE
aagcapp up running

PACKAGE STATUS STATE AUTO_RUN NODE
pkgap up running enabled aagcapp

NODE STATUS STATE
aagcdb up running

UNOWNED_PACKAGES

PACKAGE STATUS STATE AUTO_RUN NODE
pkgdb down halted disabled unowned

Please help me out on this, when i tried to login to the server using console at that time also helpless,console logs are full and not able to login

Re: Package is down in one node

Mel Burslan — Thu, 26 Feb 2009 20:51:44 GMT

Your package seems to take the physical IP address of your server down if you can not access the box from anywhere on the network. Mostly a misconfigured IP address situation.

Are you sure you are trying to ssh (or godforbid telnet) to the actual server name aagcapp not to the virtual IP address of the package ?

if you ping the server aagcapp from another system on the same network, are you able to get a response or is it totally dead as I expect ?

If you are not able to run the package from another cluster member node, you need to post the log section from the file /etc/cmcluster//.cntl.log, usually the last 100-200 lines, to give people to have some ideas why this could be happening to you. But more than likely, you cluster configuration is built or updated haphazardly and you were running by sheer luck until today.

You said you can not login from console because the console logs are full, doesn't make any sense at all, unless of course you are using a 3rd party application to manage the consoles connections. You still should be able to go to the back panel of the system and using a physical terminal or Hyperterminal on your PC with a serial cable, you should be able to aceess the server from the console.

Hope this helps

Re: Package is down in one node

pkumar97 — Thu, 26 Feb 2009 21:06:38 GMT

Dear Mel ,
Please find the package control log .

########### Node "aagcdb": Package start completed at Fri Jul 27 00:26:21 uae 2007 ###########

########### Node "aagcdb": Halting package at Thu Feb 26 23:28:09 uae 2009 ###########
Feb 26 23:28:09 - Node "aagcdb": Halting service service2
Feb 26 23:28:09 - Node "aagcdb": Remove IP address 192.168.5.12 from subnet 192.168.5.0
Feb 26 23:28:09 - Node "aagcdb": Unmounting filesystem on /dev/vg_backupdb/lvol1
Feb 26 23:28:10 - Node "aagcdb": Unmounting filesystem on /dev/vg_proddb/lvol1
umount: cannot unmount /ebizproddb : Device busy
umount: return error 1.
WARNING: Running fuser to remove anyone using the file system directly.
/dev/vg_proddb/lvol1: 23317c(oraprod) 22787c(oraprod)

Feb 26 23:28:11 - Node "aagcdb": Deactivating volume group vg_proddb
Deactivated volume group in Exclusive Mode.
Volume group "vg_proddb" has been successfully changed.
Feb 26 23:28:11 - Node "aagcdb": Deactivating volume group vg_backupdb
Deactivated volume group in Exclusive Mode.
Volume group "vg_backupdb" has been successfully changed.

also please find the attached login prompt here it is trying to login using ip address 12 and it is failed and connected via 13

login: Connection closed by foreign host.
root@aagcapp / > telnet aagcdb
Trying...

telnet: connect to address 192.168.5.12: Connection timed out
Trying 192.168.5.13...
Connected to aagcdb.
Escape character is '^]'.
Local flow control on
Telnet TERMINAL-SPEED option ON

HP-UX aagcdb B.11.23 U 9000/800 (ta)

how we can proceed to bring the pacakge now

Re: Package is down in one node

Mel Burslan — Thu, 26 Feb 2009 21:17:12 GMT

As I expected there is a problem with your IP address enabling. Lookslike, you dns resolver is resolving the name aagcdb first to .12 and then to .13 address and I am not sure how this is done at your site but it is not a preferred way to do this. Especiall tthe first address it resolves to, is the package virtual IP.

Once you are on the aagcapp server (assuming you can login to this server since the problem is on the aagcdb side), run the command :

cmrunpkg pkgdb

while this is running, open another terminal if poossible to the same system and run

tail -f

and observe the messages going into this log as they get generated. If you run into a problem, it will either help you figure out aht that problem is, or if you can not, post that output from beginning to the end, here for further assistance.

Re: Package is down in one node

melvyn burnard — Thu, 26 Feb 2009 22:58:55 GMT

do a cmviewcl -v -p pkgdb

Check the local switching for the package