TruCluster
Showing results for 
Search instead for 
Do you mean 

memchannel truCluster 5.1 no ip-adres

SOLVED
Go to Solution
Honored Contributor

memchannel truCluster 5.1 no ip-adres

i posted this question also on "Tru64.org"

Author: Pieter 't Hart (194.229.30.---)
Date: 08-04-03 04:37

for a long time we experinced some problems with our ES40 cluster running Tru64Trucluster V5.1 PK5
- T64V51B19AS0005-20020411 OSF510
- T64V51B19AS0005-20020411 TCR510
some caa-commands dont work as expected
at the moment out upgrade to 5.1A/5.1B is blocked bij this problem (upgrade thinks second node is unreachable.

I think i narrowed it down to
"memory channel doesn't get tcp/ip-adress at boottime"
mc0: flags=1100022
but there still can be another problem.

more info from /etc/sysconfigtab:
clubase:
cluster_expected_votes=3
cluster_name=spitfire
cluster_node_name=spitfire1
cluster_node_inter_name=spitfire1-mc0
cluster_node_votes=1
tnc_node_info=192.168.1.1
tnc_node_mask=255.255.255.0
tnc_interface=mc0
cluster_seqdisk_major=19
cluster_seqdisk_minor=63
cluster_qdisk_major=19
cluster_qdisk_minor=31
cluster_qdisk_votes=1

and bits from /etc/rc.config :
NETDEV_1="mc0"
export NETDEV_1
IFCONFIG_1="192.168.1.1 netmask 255.255.255.0"
export IFCONFIG_1
CLUSTER_NET="spitfire1-mc0"
export CLUSTER_NET

A similar ES40-cluster has been succesfully upgraded to V5.1A
i noticed the memory-channel is since differently configured

/etc/sysconfigtab
clubase:
cluster_expected_votes = 3
cluster_name = tornado
cluster_node_inter_name = tornado1-mc0
cluster_node_name = tornado1
cluster_node_votes = 1
cluster_qdisk_major = 19
cluster_qdisk_minor = 335
cluster_qdisk_votes = 1
cluster_seqdisk_major = 19
cluster_seqdisk_minor = 47
cluster_interconnect = mct

notice the "tnc_..." lines are missing

and /etc/rc.config :
CLUSTER_NET="tornado1-ics0"
export CLUSTER_NET


the interface is renamed to "ics0 instead of "mc0"

does anyone have a clue ???
had the memorychannel bee renamed to ics0 when one of the patchkits was installed ???
unfortunately not all of the changes/upgrades history is registered.

thanks for responses,
Pieter
20 REPLIES
Respected Contributor

Re: memchannel truCluster 5.1 no ip-adres

Hi Pieter,

Regarding the Memory Channel network name "mc0" versus "ics0". This was a change in the naming convention for TruCluster V5.1A. I believe the change was to support LAN cluster interconnect, since that is now a supported interconnect.


Additionally, the tnc_ sysconfig attributes where changes to TruCluster for V5.1A, this subsystem is now part of ics subsystem so I would _not_ expect to find those attributes in a V5.1A or higher TruCluster system. These attributes have been migrated in to the clubase, and ics_ll (low-level) subsystems.

Please provide further details on exactly how the Rolling Upgrade process fails when attempting to upgrade to V5.1A; i.e. what Step in the process does it fail and what is the symptom? Are there any error messages?

One common problem for an upgrade with respect to communications problems is that evmd, Event Manager, is not running on all members of the Cluster. Event Manager is required to shutdown the other members during a Rolling Upgrade. So, please check and make sure it's running. Also, run clu_check_config and resolve any problems before attempting to upgrade - CDSL's and processes not running which should be.

Hope this helps,
Dave Bechtold
Honored Contributor

Re: memchannel truCluster 5.1 no ip-adres

In Release 5.1a and 5.1b the cluster interconnect is named to "ics0" instead of "mc0". So don't worry about it.
Help() { FirstReadManual(urgently); Go_to_it;; }
Honored Contributor

Re: memchannel truCluster 5.1 no ip-adres

thanks for the responses
clu_upgrade stopped with messages about other node being "down" (don't have exact message becauese a collegue did this).
also rsh to the other node didn't work

clu_check_config reports no problems even with no ip-adresses on the mc0 !!!
cluster seems to funtion (clu_quorum reports normal status : 3 votes)
since we detected some mismatches in the rc.configs on both nodes;
like MAX_NETDEVS=24 instead of MAX_NETDEVS="24" this is corrected, but still after reboot there is no ip-adres on the mc0 (both nodes).

if i manualy give the mc0 an ipadres it seems to work
spitfire1:
ifconfig mc0 192.168.1.1
route change 192.168.1.0 192.168.1.1
route delete 192.168.1.1 10.1.254.253
route delete 192.168.1.2 10.1.254.253

spitfire2:
ifconfig mc0 192.168.1.2
route change 192.168.1.0 192.168.1.2
route delete 192.168.1.1 10.1.254.253
route delete 192.168.1.2 10.1.254.253
allso "clu_upgrade setup 1" has since completed succesfully

but it looks like i'll have to do this ifconfig/route over again after each reboot
that should not be neccesary ! where is this normally done at boottime ???

the routes to 10.1.254.253 found by "netstat -rn" should not have existed
i dont see a message about conflicting adresses and 192.168.1.1/192.168.1.2 are not "pingable"
here is an extract from /var/cluster/members/member1/tmp/gated.log wich is related to the route to 10.1.254.253
but i don't know what this message means

Pieter
Honored Contributor

Re: memchannel truCluster 5.1 no ip-adres

in the tru-cluster docu (V5.1) on the web i found in network troubleshooting:

>>>
Things to Check
Make sure that all cluster members are running gated.
Additionally, make sure that/etc/rc.config contains the following lines:
GATED="yes"
export GATED
Make sure that /etc/rc.config contains the following lines:
ROUTER="yes"
export ROUTER
<<<

in our fc.config i found :
GATED=YES, but ROUTED=NO and ROUTER=NO

in the web-docu i also found :

>>>
Note
The IP address for the cluster alias cannot be a 10 address. These addresses are reserved for use by cluster hardware. For example, if the IP address for the cluster alias is 10.1.0.9, problems will result.
<<<

this sounds strange to me sinc our cluster has from installation (dec-2000 from start V5.1 our) worked with a 10 adress with no problem since this was our first TruCluster we have read an enormous amount of documentation
without ever to have encountered such a note.

Pieter
Honored Contributor

Re: memchannel truCluster 5.1 no ip-adres

It is recommended to use a private network range for the memory channel and the range should differ if you are using also a private network for your official network.

So if using 10.0.0.x within your official network it should be wise to use 192.168.50.x for the interconnect (don't forget to set the resvok option in this case for the cluster aliases - see man cluamgr).

If rsh doesn't work, the cluster is not properly configured. Please run clu_check_config -v on all members to check proper installation.

If system was configured using graphical gui, have a look into the rc.config and manually re-arrange the IFCONFIG_ and NETDEV_ entries (the gui add them at the end without export statement and with duplicated entries). Move them to the top in the right order and restart network for testing.

Check the /etc/hosts for proper naming and ip-adresses. The memory channel interface is configured during boot by the sysconfigtab values and if network starts also within /etc/rc.config. Both depend on the correct values in /etc/hosts and /etc/rc.config.
Help() { FirstReadManual(urgently); Go_to_it;; }
Honored Contributor

Re: memchannel truCluster 5.1 no ip-adres

ralf thank you

in my prevous postings i mentioned clu_check_config reports no errors.
and specified some config files
sysconfigtab :
cluster_node_name=spitfire1
cluster_node_inter_name=spitfire1-mc0
tnc_node_info=192.168.1.1
tnc_node_mask=255.255.255.0
tnc_interface=mc0
and rc.config :
NETDEV_1="mc0"
export NETDEV_1
IFCONFIG_1="192.168.1.1 netmask 255.255.255.0"
export IFCONFIG_1
CLUSTER_NET="spitfire1-mc0"
export CLUSTER_NET
leaves as missing info /etc/hosts :
10.1.3.50 spitfire.bvh.nl spitfire
10.1.3.51 spitfire1.bvh.nl SPITFIRE1 spitfire1
10.1.3.52 spitfire2.bvh.nl SPITFIRE2 spitfire2

we used "rcmgr" to make the changes this deletes old references and places new references at the bottom.
the only manual change made was exchanging NETDEV_0 and NETDEV_1 to make the netrain-interface NETDEV_0 and the memory-channel NETDEV_1 this was done shortly after instllation and the cluster has worked ok for some time.
As this was our first unix-cluster at the time many tests and experiments including power-shudown, cable-disconnects disk-removals were performed and offcourse many reboots without the memory-channel losing its ip-adress.
The above problem was discovered years later

the resvok option is not used in the clu_alias.config file not on this system and not on the system that bevhaves OK.

we are closing in, but not solved yet

I have a few days off so i will not respond until after next week

Pieter
Honored Contributor

Re: memchannel truCluster 5.1 no ip-adres

Pieter,

the given informations are not suitable to solve your problem.

Please also check:

1. "cluster_node_inter_name" must match /etc/hosts name of memory channel interface e.g. tornado1-mc0. /etc/rc.config entry "CLUSTER_NET" must match the same name!
(this is not the case here!!)

2. If using private addresses like 10.x.x.x for the normal network requires the resvok option (mandatory), have a look into the cluster admin guide or the man pages for cluamgr!

the documentation is easy to read and contains all necessary informations, so have a look into the cluster admin guide.
(http://h30097.www3.hp.com/docs/cluster_doc/cluster_51A/TCR51A_DOC.HTM)
Help() { FirstReadManual(urgently); Go_to_it;; }
Honored Contributor

Re: memchannel truCluster 5.1 no ip-adres

ralf thanks, but i cant' see any mismatches in the configuration-files (see bottom of message)
i found your reference to "resvok" in another location (the suplied link was dead)

NB! i'm speaking about two different clusters.

We are in the progress to upgrade our "spitfire) from V5.1 to 5.1A, the 5.1 manual does NOT mention the necessity to use resvok as the 5.1A manual does (it does exist in the 5.1 version).
Also our "tornado"-cluster which is allready upgraded to 5.1A does NOT have resvok in any configuration file AND works as supposed to!
our third cluster the "mosquito" has from start been V5.1A (no memory-channel) and has from the start been 10.x.x.x as primary network and 192.168.1.x as cluster-interconnect network has no resvok in any configuration file AND works as expected!

Pieter


spitfire1 (root)# grep spitf hosts
10.1.3.50 spitfire.bvh.nl spitfire
10.1.3.51 spitfire1.bvh.nl SPITFIRE1 spitfire1
10.1.3.52 spitfire2.bvh.nl SPITFIRE2 spitfire2
192.168.1.1 spitfire1-mc0
192.168.1.2 spitfire2-mc0

spitfire1 (root)# grep CLUSTER_NET rc.config
CLUSTER_NET="spitfire1-mc0"
export CLUSTER_NET
spitfire1 (root)# grep cluster_node_inter_name sysconfigtab
cluster_node_inter_name=spitfire1-mc0
spitfire1 (root)#
==============================================
spitfire2 (root)# grep CLUSTER_NET rc.config
CLUSTER_NET="spitfire2-mc0"
export CLUSTER_NET
spitfire2 (root)# grep cluster_node_inter_name sysconfigtab
cluster_node_inter_name=spitfire2-mc0
spitfire2 (root)# grep spitf hosts
10.1.3.50 spitfire.bvh.nl spitfire
10.1.3.51 spitfire1.bvh.nl SPITFIRE1 spitfire1
10.1.3.52 spitfire2.bvh.nl SPITFIRE2 spitfire2
192.168.1.1 spitfire1-mc0
192.168.1.2 spitfire2-mc0
spitfire2 (root)#
Frequent Advisor

Re: memchannel truCluster 5.1 no ip-adres

In the /etc/rc.config file, make sure your NUM_NETCONFIG parameter has the number of interfaces that you need to configure, including the memory channel interface.

For instance, I have 2 NICs and 1 memory channel card, therefore my parameter is NUM_NETCONFIG="3".

--Blake
Honored Contributor

Re: memchannel truCluster 5.1 no ip-adres

Pieter,

resvok is recommended if using private addresses for the official network adapter.

By default, TruCluster will not advertise addresses in any of the private network ranges via RIP. If your cluster's alias addresses are in one of those ranges, you must issue "cluamgr -r resvok" to allow the alias (and interface) addresses in the
private ranges to advertised by RIP. So why discussing an issue already explained in the manual and man-pages?

The resvok option is not the cause of the not configured adapters but improves your configuration.

The given information is not suitable to solve the problem, because we are missing /etc/rc.config and /etc/rc.config.common.
Help() { FirstReadManual(urgently); Go_to_it;; }
Honored Contributor

Re: memchannel truCluster 5.1 no ip-adres

ralf, thanks again
i'm attaching the complete rc.config and rc.config.common to this thread
Honored Contributor

Re: memchannel truCluster 5.1 no ip-adres

there was something wrong withe the attachment rc.config
Honored Contributor

Re: memchannel truCluster 5.1 no ip-adres

rc.config.common
Honored Contributor

Re: memchannel truCluster 5.1 no ip-adres

Blake,
I looked again at your response and i think that is the pointer to the problem
apearantly we overlooked that one.

spitfire1 (root)# rcmgr get NUM_NETCONFIG
1

I'll adjust this and wil try a to schedule a reboot,
(operational system 24x7 ours availability cannot reboot at will )


Pieter
Honored Contributor

Re: memchannel truCluster 5.1 no ip-adres

Yes num_netdev equals 1 but the inet start script doesn't honor this variable. It uses MAX_NETDEVS instead. Have a look into the startup of the network:

a) first check is if num_netdev > 0
b) if so while loop until max_netdevs will be reached containing check of IFCONFIG_ and execution of the configuration

So get more information add "-x" to the shell statement within /sbin/init.d/inet. This will show you the lines and contents during execution of the network configuration.
(also add an ifconfig -a to the end of the inet file to be sure adapters are configured)


Help() { FirstReadManual(urgently); Go_to_it;; }
Honored Contributor

Re: memchannel truCluster 5.1 no ip-adres

i read and read again the documentation about the "resvok" option.
I still think we will CREATE a problem when we should activate this.

At this time the cluster-interconnect is intended ONLY for cluster-communication NOT for alternate routes, the cluster itself is NO router.
therefor the cluster-interconnect adresses are supposed to be LOCAL to the cluster (that is our interpretation about the mechanism)
if we should activate resvok, then the cluster-ip-adresses of cluster-1 and cluster-2 would both be known on the company-network

since both clusters use THE SAME adresses for
their cluster-interconnect then there certainly will be a problem!!!!!

spitfire1-mc0 192.168.1.1
spitfire2-mc0 192.168.1.2

tornado1-ics0 192.168.1.1
tornado2-ics0 192.168.1.2

all subnet 255.255.255.0

if we misinterpreted this than it should mean we have to allocate a different subnet to each 2-node cluster for its memory-channel.
(255.255.255.252 ?)

a test with different-ipadresses is allready done, this provided NO solution.

spitfire1-mc0 192.168.1.7
spitfire2-mc0 192.168.1.8

tornado1-ics0 192.168.1.1
tornado2-ics0 192.168.1.2

in the "best practices" i read an example, but in this document separate references to "private subnet" from "private-ip-adress".

Pieter
Honored Contributor

Re: memchannel truCluster 5.1 no ip-adres

you are right, this is the problem:

spitfire1-mc0 192.168.1.7
spitfire2-mc0 192.168.1.8

tornado1-ics0 192.168.1.1
tornado2-ics0 192.168.1.2

but changing to 1.7 and 1.8 is also a fault. you must change it to:

spitfire1-mc0 192.168.1.1
spitfire2-mc0 192.168.1.2

tornado1-ics0 192.168.2.1
tornado2-ics0 192.168.2.2

because the ics0 values must differ from the mc0 entries!

The interconnect between the members will be protected by the ifaccess.conf entries and the "filter" flag on the "outside" communication interface. The network is used for private communication between the members, the addresses must be unknown outside of the cluster.

resvok means, gated will advertise the cluster aliases within a private net. There is no interaction with the interconnect, this parameters affects the communication network. If not set, problems can occure depending on your configuration. If you uses official ip addresses on the communication network (this means the network to the other machines) gated will do that automatically without setting the resvok option! But using private addresses on the communication network REQUIRES this option!

Help() { FirstReadManual(urgently); Go_to_it;; }
Honored Contributor

Re: memchannel truCluster 5.1 no ip-adres

Ralf, Dave, Blake many thanks for your responses

Blake's response led to the solution: the misbehaviour was due to NUM_NETCONFIG not having the right value (1 instead of 2).
Ralf's suggestion to add "-x" to the shell statement in /sbin/init.d/inetd script made this visible when evaluating the captured output at boottime.
it took some time to switch from graphics to serial console because i used the wrong com-port (com1 is the MMJ-connector not the upper 9pin-subD).

Pieter
Honored Contributor

Re: memchannel truCluster 5.1 no ip-adres

Pieter, as written in the previous answers, the IP addresses of the ics and mc entries are not correct. Please correct this values also.

Btw. MAXNUMDEVS is the counter not the NUM_NETCONFIG variable. Have a look into the startup script - so I wonder why NUM_NETCONFIG solves the problem.....
Help() { FirstReadManual(urgently); Go_to_it;; }
Honored Contributor

Re: memchannel truCluster 5.1 no ip-adres

ralf thank you for your warning, but i think you misinterpreted the situation
the "spitfire" is a different two-node cluster from the "tornado" it is not a different alias on the same cluster.

the tornado has allready been succesfully upgraded to V5.1a and the upgrade-proces renamed the cluster-interconnect to "-ics0" instead of "-mc0".
the spitfire has this week been succesfully upgraded. to V5.1A PK4

regarding the NUM_NETCONFIG or MAX_NETDEVS issue i think the lines
if [ "$found" = "$NUM_NETCONFIG" ]; then
break
overrides the
while [ "$num" != "$MAX_NETDEVS" ]
do

i attached the output of the inetd startup where NUM_NETCONFIG was still set to "1"

Pieter
//Add this to "OnDomLoad" event