Windows is unable to start VMS EXEcutable

[Arch_Muthiah]

Hi,
I have posted similar question couple of months back, I still have the problem, but this time with other customer.

Env: Alpha Server/VMS 8.2 single node cluster, TCPIP V5.4 latest ECO, & Wind 2K. Both VMS & Windows side programs uses TCPIP socket routines. A Windows side program should start the VMS side application, and VMS returns the status back to windows.

No problem communicating from VMS to windows &
c:\ Telnet vms_ip 5601
c:\ Telnet vms_fqdn 5601
c:\ Telnet vms_node_name 5601
All the above commands work fine.

This application works fine with individual VMS node or two node cluster. The problem is only with this setup of single node cluster (plan of adding nodes in the future).

Error: when we start the windows program, it ends with the error "Unable to start the VMS' xxxx.EXE program"

In VMS side, a SERVICE is ENABLED to handle windows request and LISTENing at the preassigned port 5601.

As I found no proble in the communication side, now I concentrate on any problem with cluster VOTES, or QUORUM values.

I have attached VMS' TCPTRACE output taken during the windows program trying to start the VMS application.

Expecting your valuable suggestions

Thanks
Archunan

[Volker Halle · Accepted solution]

Archunan,

the full TCPTRACE shows the communication between node 10.91.2.52 Port 2888 (the Windows System ?) and node 10.91.1.61 Port 5601 (the VMS system ?).

Some user data is being exchanged, but it's not ASCII readable, so it would be up to you to decode the user data (after the second '|' in each packet captured).

The trace does not cover the connection establishment phase (SYN and SYN ACK), but it does capture the termination of the connection (FIN ACK sent from 10.91.1.61 in packet 4142) and acknowledged by 10.91.2.52 in packet 4147 (FIN ACK).

Ethereal seems to be unable to decode the FULL TCPTRACE format, so maybe next time you could run TCPTRACE without /FULL.

You've run TCPTRACE on the VMS system to capture the TCP data exchange, so what problem could there be with VOTES or QUORUM ?

Maybe you could run TCPTRACE in a working environment, so you have something to compare against and you might be able to spot the differences and find out where it goes wrong.

Is there any logfile of the VMS side TCP server program ? Is this server supposed to start a detached process ? Maybe check in accounting, if a process get's started but exits immediately ?

Volker.

[Robert Gezelter]

Archunan,

In the situation that you describe in your posting, I would be surprised if ANY of the cluster parameters are relevant.

The message "Unable to start the VMS'xxx.EXE program" appears, if I am reading your posting correctly, to be coming from the Windows program. I would start by looking at the source of that program to identify what can lead to that message.

- Bob Gezelter, http://www.rlgsc.com

[Robert Gezelter]

Archunan,

In re-reading my post, a clarification.

If the cluster parameters were incorrect, the cluster member would be non-responsive in an overall way. Your post seems to indicate that the cluster member is otherwise functional.

- Bob Gezelter, http://www.rlgsc.com

[Arch_Muthiah]

Volker, First my thanks for reading my lenthiest question and brief answer.

"The trace does not cover the connection establishment phase (SYN and SYN ACK),..."

ANS: do you mean that Windows connection request did not reach and acknowledged by VMS?, do you feel any netwpork issues involved in Windows side?

"but it does capture the termination of the connection..."
ANS: Yes it will be correct as there are no communication problem from VMS to Windows.

"You've run TCPTRACE on the VMS system to capture the TCP data exchange, so what problem could there be with VOTES or QUORUM ?"

ANS: If the Cluster's number of VOTES is less than QUORUM value, then Cluster member suspend all process activity and all I/O operations to cluster-accessible disks. Isn't it?
In that case, can I increase the EXPECTED VOTES or node VOTES ?

TCPTRACE command was taken from the working env, I will ask the customer for TCPTRACE without /FULL.

We have log file created by the intial startup COM procedure which actually start the EXE to be run, it contains only ....
$ ty DSA0:[xxxx.TMP]daemon_service.log
$ Set NoVerify
$ DEFINE/PROCESS UCX_MODE "1" !Define TCP/IP as UCX version
VMS_job_starter terminated at 26-APR-2006 14:40:56.12
Accounting information:
Buffered I/O count: 109 Peak working set size: 5616
Direct I/O count: 61 Peak virtual size: 177856
Page faults: 349 Mounted volumes: 0
Charged CPU time: 0 00:00:00.32 Elapsed time: 0 00:00:01.92

one more doubt: can windows able to start any VMS executable from single node node cluster?, any VOTE issues involved in this?

Archunan

[Arch_Muthiah]

Bob,
When I reponded to Volker, I saw your ans:

Thanks for reading my lenthiest question. I thought you people may not have time to read my lenghtiest question.

Volker said, "The trace does not cover the connection establishment phase", Is it anyway related with VOTES, because when cluster VOTES falls below the QUORUM value, the executable in the common disk can not be accessible. Is it true, Bob?


Archunan

[Art Wiens]

If your unsure, set VOTES and EXPECTED_VOTES to 1.

I believe Volker was pointing out that you haven't included the whole "conversation" between the two systems, he didn't see the initiation of the conversation in your trace.

Possibly an extra clue will be obtained with a REPLY/ENABLE on an existing VMS session when you try and start your Windows program. Maybe a login failure or permission problem or such.

Art

[Jan van den Ende]

Archunan,

I fully expect Bob to agree with Volker - as I do.
If anything is wrong with your VOTES and/or EXPECTED_VOTES, one way => your system will not do ANYTHING - except waiting for something to reset the wrong setting, or, the error(s) to the other wrong end => any semi-disturbance in connectivity and your disks go corrupt. But IF anything is funtioning (and it does - you DO get trace info) than the VOTES mechanism has _NO_influence!
Alas, -- I can operate TCPIP somewhat, but I am not a guru, and I know next to nothing in the M$ area. I consider myself unfit to debug mixed-platform TCP problems.
But I will monitor this thread to learn!

Proost.

Have one on me.

jpe

[Arch_Muthiah]

Thanks Jan and Art.

Every one knew Jan maintains his cluster environment UP for more than 9 years. I was expecting your answer for my previous similar thread also. Though you said you zero experience in mixed platform area, your posting gave some confirmed information.

Art: the VOTES and EXPECTED_VOTES values are 1 1 only (by default). I have given as much possible information as I can.

As Bob suggested, now I am looking into the code C++, it looks like one of the thread started by parent process (handles arrays of threads) takes time and failed in some type of testing (not sure, it may be time delay by a thread in releasing out the shared memory or some other).

I am wondering that the same application is running fine in my environment with same AS/VMS 8.2, mine is individual node.

I still expect more of your valuable suggestions and idea to where can I look into.

Thanks

[Steven Schweda]

Clearly, if the VMS system does _anything_,
then it does not have a cluster votes/quorum
problem.

> it ends with the error "Unable to start
> the VMS' xxxx.EXE program"

I'd spend more time looking at the program
which puts out this (pretty useless) error
message, to try to detrermine _why_ it
believes that it was unable to "start the
VMS" program. For example, what, exactly,
does it do, and what, exactly, fails?

[Arch_Muthiah]

Steven,

Yes it is useless meaning less message only. As you all said, this won't be cluster related issue, now I am looking into the code. As the processing is done using many threads, this message is there in tooo many places.

I have the similar environment as the customer who face this error, the only difference is that mine is individual node, but the customer said has configured their machine as single cluster node.

It works fine on my environment.

Archunan

[Volker Halle]

Archunan,

your trace does not show the complete TCP connection establishment and data exchange, something is missing at the beginning. The trace starts in the middle of an existing connection. A TCP connection starts with a message having the SYN bit set and a reply with SYN and ACK set. This is missing in your trace. Your trace only covers 510 milliseconds of TCP traffic between 2 hosts on 3-AUG-2006. The .LOG file you've shown is from 26-APR-2006. Please provide the .LOG file from the same time as the failing test.

Please verify the IP adresses shown, to confirm, that they are indeed from your VMS system and your Windows system involved in this test.

TCPTRACE was running on your VMS system to capture the failure, right ? If the VMS system would not have had quorum, the VMS system and TCPTRACE would have hung and not produced anything.


TCPTRACE command was taken from the working env.


Are you saying that the trace you've taken shows a successful TCP message exchange and a successful start of the program on VMS ? Next time, please ask the customer to start TCPTRACE, wait 10 seconds, start Windows program, wait until error message shows up and then stop TCPTRACE. By default, TCPTRACE will automatically stop after capturing 10 packets. You might need to run it with /PACKETS=100 and then CTRL-C out of it, once the error has been shown on the Windows GUI.

If it works on your standalone node, just temporarily re-configure that node with VAXCLUSTER=2, VOTES=1 and EXPECTED_VOTES=1, reboot and re-test. That should be easy.

Also consider to take a TCPTRACE in your working environment first, then you have something to compare the failing case with.

Volker.

[Arch_Muthiah]

Thanks volker for your long posting.

As per your suggestion, I asked the customer to change the cluster para and test the appl. once I get, I will post.

TCPTRACE and LOg file were taken the customer env only, the LOG file date is difft, but all the log file contains the same contents only.

Also I found that there is no problem in communication (wind to vms) as VMS application had been started by the COM procedure. But VMS application got delayed and windows errored out with "Unable to start the VMS' xxxx.EXE program" - it is windows side appl message. I have no idea about windows side source code.

I am still wondering why this application errored out only in single node cluster env!!!, while it runs fine in standalone system and two node cluster env?

Any known restrictions/issues running the application in single node cluster env?.
Or any two NIC (incomming and outgoing) config in wind or VMS?, I will have to ask about this to the customer.

Still hoping to get the suggestion from our Ian, volker, Uwe, Bob, Karl, Stewen, John, Wim, or especially our Cluster Guide - Jan.

Thanks
Archunan

[Steven Schweda]

> Any known restrictions/issues running the
> application in single node cluster env?.

Only if your application is particularly
weird. My primary system is an XP1000 which
is normally the only system in its cluster,
although other systems come and go from time
to time. I've never seen a normal
application which cared whether it was
running on a cluster member.

+--------------------------------------------------------+---------+
| SYSTEMS | MEMBERS |
+--------+--------------------------------+--------------+---------+
| NODE | HW_TYPE | SOFTWARE | STATUS |
+--------+--------------------------------+--------------+---------+
| ALP | COMPAQ Professional Workstatio | VMS V7.3-2 | MEMBER |
| ALP2 | COMPAQ Professional Workstatio | VMS V8.2 | BRK_NON |
| WUSS | VAXstation 3100/GPX | VMS V7.3 | BRK_NON |
| GIMP | VAX 4000-200 | VMS V7.3 | BRK_NON |
+--------+--------------------------------+--------------+---------+

[Wim Van den Wyngaert]

I would concentrate on tcpip config.

Check the file tcpip$etc:sysconfigtab.dat on a working and non-working system. May be a timeout is configured too low.

BTW : avoid giving more than 7 points without having the solution for your problem. This question has a "bunny" but no solution (yet).

Wim

[Wim Van den Wyngaert]

And check accounting for process termination status, operator log file for messages that might have been given by the application or tcpip and audit log file for violations.

Wim

[Phil.Howell]

I don't think cluster parameters would have anything to do with the problem.
What I would do...

Find the startup procedure that produces the log file and remove the set verify.

Add the line
$ set watch /class=major file
to the startup procedure

This should produce more detailed logs for further diagnosis
Phil

[Volker Halle]

Archunan,


As per your suggestion, I asked the customer to change the cluster para and test the appl. once I get, I will post.


I suggested that you temporarily change YOUR test system to become a single node cluster, so that we can easily rule out the 'application does not work on single node cluster' theory.

Volker.

[Arch_Muthiah]

Wim,

Thanks for the info about the ITRC pointing method. But I usually respect our members for their valuable time to read/understand my lenthiest questions described with my poor english knowledge. And I saw responses also composed in a big paras. So I feel I am right giving the points. Especially Volker, Jan and Steven spending their time to compose a big responses with broad range of informations, I really learn lot from their postings.

Anyway I followed your suggestions and not given 10 points this time.

Phil: I will try your sugestion today.

Wim: VMS application keeps trying to continue the processing, so the VMS application did not timedout. The timeout message is from windows application only. I even tested increasing the timeout env parameter in windows side as windows appl uses this para. Anyway I try your suggestion too.

Volker: This problem is in the customer's testing system only. So they can change the system to standalone system. So no problem.

Thanks
Archunan

[Volker Halle]

Archunan,


VMS application keeps trying to continue the processing, so the VMS application did not timedout. The timeout message is from windows application only.


If the VMS side continues to run and the Windows side reports a timeout, then I could only think of 2 scenarios:

- either the VMS side did not correctly indicate that it's working, before closing the TCP connection

- the Windows side expects additional information from the VMS side, which could only be through another TCP connection.

Maybe you need to trace ALL TCPIP traffic between the 2 nodes and you should also consider to take a TCPIP trace of a working configuration. Then you may be able to spot the difference and find out where it's going wrong and what is missing.

Volker.

[Jan van den Ende]

Archunan,

although I feel honored (and a bit shy) to be singled out as a cluster specialist, I have to admit to a real big hole in my knowledge: juat about anything concerning Micro$oft.

I am following this thread, and the responses, and
a) I tend to agree that it is not a VMS, but a M$ issue
b) should it prove in the end to _BE_ a VMS issue, than this is one I will be 100% learning, and 0 % problem-solving.
So, unless things take a turn that I do not expect, I fear that this will be all I can contribute to this issue.

Wishing you success non the less!

Proost.

Have one on me.

jpe

[Arch_Muthiah]

Jan/Volker,

I don't have any issue with this application (running fine in standalone system and two node clusters) from any other customers, except this european customer having single node cluster (the similar issue reported earlier by one canadian customer also kept unresolved). Now I am working with european customer. once I get the info, I will update here.

Stay with me.


Thanks
Archunan

[Steven Schweda]

Do everyone a favor. Stop worrying about the
cluster-related differences, and start
worrying about the differences which might
make a difference.

[Arch_Muthiah]

All,

I found the reason for this issue. As you all said, it is not a cluster issues.

This error is because our customer installed our application from one windows server and tried to start run the VMS application from the different windows server which has different IP. As our application updated first windows IP in the VMS application, it could not able authenticate the request from the second windows system with different IP.

As we all knew getting customer env details is not easy, they mostly won't like to disclose or accept their env changes (mistakes) as easily, because of this it took me long time.

Thanks for all your timing and suggestions to resolve this issue, especially my thanks to Volker, Jan, Bob, Art, Stewen, wim and Wim for you timings.


Archunan