TruCluster
cancel
Showing results for 
Search instead for 
Did you mean: 

Cluster events not being processed

Cluster events not being processed

I have a 5 member cluster connected vie memory channel. When patches are apllied to the cluster and the initiating member sends the request to bring down the members to init level 2, all members except member 1 goe sto init 2. memeber 1 has to be forced manually. Then once the patches are applied and the initiating member sends the reboot request all members except member1 gets the command. The kernel then has to be manually built and copied.

What could be the cause of the member not receiving the commands?

I have removed the member from the cluster with clu_delete_member and then added it again with clu_add_member..this did not fix the problem.
6 REPLIES
Venkatesh BL
Honored Contributor

Re: Cluster events not being processed

Did you check the events on the member to see if it received the event? Do you have the console messages for this member?

Re: Cluster events not being processed

No console messages are displayed on the member that has to receive the event.

Where do I check if it has received the event? Surely if it has received the event it would execut it as well?

Nothing happens to the member and the initiating meber just waits for ever as it does not get any response back, hence the manual intervention.
Ivan Ferreira
Honored Contributor

Re: Cluster events not being processed

Can you sucessfuly rlogin to that member as root? What is the output of clu_check_config?
Por que hacerlo dificil si es posible hacerlo facil? - Why do it the hard way, when you can do it the easy way?
Victor Semaska_3
Esteemed Contributor

Re: Cluster events not being processed

We had a similar problem while installing patches on our cluster. I find it interesting that our cluster is also a 5 member cluster. Maybe just coincidence.

Anyway, on the cluster that refused to go to init 2 were these console messages:

evmpost: Failed to create EVM posting connection
evmpost: Error: Connection lost
Waiting for Event Management system to reconfigure...done
Waiting for all cluster members to complete event operation...

Did you check /var/adm/messages? We had to do pretty much the same thing you did.

Sent support output from '# sys_check -escalate' but they didn't find anything wrong. They suggested if it happens again to '# dumpsys' which copies a snapshot of memory to a dump file and then '# sys_check -escalate' again.

Vic
There are 10 kinds of people, one that understands binary and one that doesn't.

Re: Cluster events not being processed

Hi Vic..yes those are my exact mesages as well. Has anybody encountered this as well? The clientt is 600km's away from me and everytime we have to load patches it's a major issue. If the process worked as it should they can do it withoutt any problem and it will be quite painless. Ii hope we cann find the problem.
Rani sawade
Occasional Advisor

Re: Cluster events not being processed

Hi,
May be you can try reloading the evm on the
problematic member.

%evmstop
%evmreload

With two node cluster, I had got similar error messages given in Victor's update and reload solved this!

Regards,
Rani.