VG not activating in Service Guard environment

[SYS ADMIN_7]

Hi All

I am facing some problems in MC/SG environment.
We are having MC/SG environment with 2 node setup. All of sudden the VGs are not actvating,(except root) so I am not able to mount the file system..
The OS is HP-UX 11.00

The two nodes are having one HDD with OS loaded.The storage box is having two HDDs of 9GB capacity and they are having mirroed volumes..This storage box connected to both nodes and contains the VG01.
The VG00 is root and file systems in it is mounting without any problem.

The HDDs in the Storage box are sensing in ioscan.

The some outputs of commands are

ioscan -funC disk
disk 8/4.11.0--- /dev/dsk/c0t11d0
disk 8/4.15.0---- /dev/dsk/c0t11d0

vgdisplay -v /dev/vg01
volume group can not be activated. cannot display volume group /dev/vg01

vgchange -a y /dev/vg01
vgchange: activation mode requested for the volume group " /dev/vg01" conflicts with configured mode.

vgscan:
couldn't query physical volume /dev/dsk/c0t15d0

I checked the HDD using the dd command, it doesn/t give any problem..
Being new to MC/SG this is creating lot of troubles..
Also following messages started flashing in both consoles

STREAMS /UX

error subsys: STREAMS loc:001231 718581 53069 or rput dlpi:DL UDERROR IND , dl dest address length 8 dl err no 4..

now the service guard is also not starting.


Please help


Rgds
Suraj

[RAC_1]

The VGs in MCSG are activated in exclusive mode. Try activating them as vgchange -a e vg01. Also check required settings are set in pkg.cntl/pkg.conf.

Also what does pkg.log/cluster log says?

Anil

[Sanjay_6]

Hi Suraj,

Under SG env, the package VGs are activated using the command vgchange -a e vg_name

-a e stands for activation in exclusive mode. The other data vg's not part of SG are activated using vgchange -a y vg_name. Sometime we use /etc/lvmrc to activate the non-sg vg's on the system. Check your package configuration and you should have the filesystem mount points defined over there.

http://docs.hp.com/cgi-bin/fsearch/framedisplay?top=/hpux/onlinedocs/B3936-90065/B3936-90065_top.html&con=/hpux/onlinedocs/B3936-90065/00/00/43-con.html&toc=/hpux/onlinedocs/B3936-90065/00/00/43-toc.html&searchterms=vgchange&queryid=20040525-121504

Hope this helps.

Regds

[Bharat Katkar]

Suraj,
I doubt something happened and your CLuster came down ungracefully.
One workaround would to try deactivtaing the VG in this way. See if that works.

On your primary node where VG were active before:

# vgchange -c n /dev/vgname
# vgchange -a n /dev/vgname

Repeat this for all VG's under Cluster.
then try halting Cluster

# cmhaltpkg -v pkgname
# cmhaltcl -f

Then reboot your System and see whether it comes up normally.

Regards,

[Rita C Workman]

Going along with the gentlemen above....

First -c n your vg's
[ I sometimes throw in a -a e just for fun ]
then -a n your vg's
then -a y your vg's....and see if you vgdisplay ok just one of them. If yes then your good...

Now before you reboot that server !!!!

vgchange -a n
vgchange -c y

Now since I doubt your package is up anyway...reboot and then try to restart your package.

Just a thought,
Rita

[Rita C Workman]

...Sorry for the reboot comment...

Why not just try to restart your package and see if it comes up ok...you shouldn't need to reboot a server because a pkg is having trouble getting up.....unless you really want to reboot that is...

Rita

[SYS ADMIN_7]

Hi

Thanks for your replies..

Dear Anil,
What settings should I check in pkg.cntl/pkg.conf.. Please give some more info.

Please also let me know how can I check the HDD ( Some doubt about it).

Bharat,
I will try your suggestion and let you know about it.
Can you please calrify one doubt.
When working with VG and MC/SG,whethter MC/SG servive starts after activating the VGs which are part of cluster or first service starts and then VG , which are part of cluster are activated.

Rgds
Suraj

[SYS ADMIN_7]

Hi all
Again.

Can anybody tell the what the STRAEM error message means, whcih starts flashing in the console.The message goes like this

STREAMS /UX

error subsys: STREAMS loc:001231 718581 53069 or rput dlpi:DL UDERROR IND , dl dest address length 8 dl err no 4..

Rgds
Suarj

[Isralyn Manalac_1]

Kindly post the pkg cntl log file as well as the syslog.log file.

Regards,
Ira

[Mel Burslan]

to answer your streams question, the messages you see on the screen is related to a part of networking subsystem.

first suggestion is to check the cables for an accidental loose connection or anything like that.

after this, make sure your network speed and duplex settings are the same at both ends, i.e., at the switch and at the card on your server. To check the status of your interface, use command

lanadmin -x 0 (for lan0)
lanadmin -x 1 (for lan1) and so on

another suggestion is to check if you have the latest streams patch for your server.

swlist -l product | grep PH | grep -i stream

and put this into the patch database search window to see if it is the latest or you are behind. If you are behind more than one revision level, it is strongly suggested to update.

but seeing the UDERROR is telling me that you have some faulty UDP packets floating on this network segment. Either there is an ill behaving application spitting out raw packets which are not constructed right, or somebody has released a virus on your network and it is causing this problem.

coming to your MC/SG question, without seeing your pkgname.ctl script it is going to be very hard to guess what might be wrong, but try activating the vg in exclusive mode with :

vgchange -a e vg01

command and see if this is able to activate the volume group. If not, you may have a bigger problem than just service guard. Your vg may have become corrupt for some reason.

Hope this helps

[Bharat Katkar]

Hi Suraj,
Before starting any service i.e. package, the VG/LV's/Filesystems used by that particular packages are started and mounted first. Unless and untill how they can access all required binaries and other files which are there on that filesystems. After that i.e. package ASCII file has completed its task then comes in picture your control script wherein you control the startup and halting of the package.
Similarly when you shut the cluster down all the services are brought down first and then the Filesystems unmounted and VG are deactivated.

Hope that helps.
All the besr
BHarat