Need help on Maestro Application

[Viswanadhan]

Hi Admins,

We have Maestro Software installed on Server-A, clients on Server-B,C...

Till yesterday everything is good. Today we got terrific problem on client machine.

In Client machine im able to start the service, but it's not communicating with Server-A.

But im able to ping from client system to server, and server to client. No changes in software, etc..

Regards,
Viswa.

[kevin_m · Accepted solution]

I'm assuming you are referring to IBM Tivoli Workload Scheduler (formerly named Maestro). If so, the problem you're describing can be caused by a few different issues. Lack of free disk space will prevent the communication so I'd check that first if you haven't already. Regardless, please provide the following details:

- operating systems of master (server A) and clients
- status of clients in admin console (specifically 'Jobman Running' and 'Link Status' columns)
- versions of software on master and clients (if known)

Kevin

[Viswanadhan]

Hi Kevin,

Sorry for delay.

We are using IBM Tivoli. Wehni run following, then i got following info.
# /sbin/init.d/maestro start
MAESTRO for UNIX (HPUX)/CONMAN 6.1 (6.8) (C) Tivoli Systems Inc. 1998
Installed for group 'DEFAULT'.
Schedule (Def) 04/29/08 (#4523) on CLIENT-A. Batchman down. Limit: 20, Fence: 0
start
MAESTRO for UNIX/STARTUP 6.1 (C) Unison Software Inc. 1997
TIVOLI UNIX (HPUX)/NETMAN 6.1 (6.3) (C) Tivoli Systems Inc. 1998
Program patch revision: 6.3
Netman installed under group 'DEFAULT'. [12100.57]
A start command was issued for CLIENT-A. [2202.507]
link SERVER-A;noask
Command forwarded to batchman for SERVER-A
#

If i see the status from GUI for other clients as " LINKED". For one client i donot see anything status as "LINKED" or " UNLINKED". It's blank status.

When i start from GUI, then i got error msg as follows.

CLIENT-A has not been initialixed [2202.21].

May i know the about the " [2202.21] " ?

Regards,
Viswa.

[Minoia]

Hi Viswa,
do you share the mozart database through NFS ? In this case the problem could be related to STREAMS DEVICE and NFS lock.
Please start the command conman, may be the command will not show the prompt.
Check the process with glance, may be you will see stop reason for STRMS.
In this case you can try to clear the NFS lock between client and server.
1. kill all maestro processes (netman,conman,mailman, batchman and writer)
2. with clear_locks clear all NFS lock between client and server
/usr/sbin/clear_locks -s YOUR_SERVER_NAME
3. from maestro client user
StartUP
conman start
If conman command works go to step 4
4. from maestro client server
conman
link YOUR_CLIENT_CPU
May be it will take a long time, the server have to deploy the Symphony.

In case step 3 doesn't work try to start maestro as root.

Good luck
Gianni

[Viswanadhan]

Hi,

When do from Server machine
# su - maestro
opt/maestro>StartUp
opt/maestro>conman
conman> start dscprhp1
Error: CLINET-A has not been initialized yet. [2202.21]

Same error msg i got from GUI.

Any guidence.

Regards,
Viswa.

[Minoia]

The problem is related to netman processes on client.

Please run this commands from client side :

./StartUp
conman> start

If the commands work you can link the client from the server, otherwise follows the procedure explained in my previous post.

Gianni

[kevin_m]

I agree with Gianni. Kill all the maestro processes on CLIENT-A, run ./StartUp on the client, and issue conman link CLIENT-A on the server. If the link is successful and your client is configured to autostart, all the client processes will eventually start by themselves.

If the link doesn't work however there could be corruption in the client's job and/or message files. There is a fix but it will wipe out the status of current and scheduled jobs on the client. WARNING: You won't be able to see which jobs finished and everything pending for CLIENT-A will be gone.

1) Stop or kill all maestro processes on the client.

2) Clear some files on CLIENT-A ONLY (assuming default path).
mv /opt/maestro/Sinfonia /opt/maestro/Sinfonia.bak
mv /opt/maestro/Symphony /opt/maestro/Symphony.bak
mv /opt/maestro/Jobtable /opt/maestro/Jobtable.bak
mv /opt/maestro/pobox/tomaster.msg /opt/maestro/pobox/tomaster.msg.bak

3) Clear the message file on the server but just for CLIENT-A.
mv /opt/maestro/pobox/CLIENT-A.msg /opt/maestro/pobox/CLIENT-A.msg.bak

4) Run this on CLIENT-A.
su - maestro
./StartUp

5) Run this on SERVER-A.
conman link CLIENT-A

It will push the current schedule from the server to the client and establish the link between them. Keep in mind the client may run jobs unexpectedly if their start times have already occurred.

Kevin

[Viswanadhan]

Hi Kevin,

I did according to ur information. I hope more than 90% i got resolved. But, when i see the clients status from server as follows.

Client-A LT W
Clinet-B LTI JW
Client-C LTI JW

But the correct status is " LTI JW".

When i do from CLI
# ps -ef|grep maestro( then i donot see jobman, and batchman ).

Other clients i able to see "jobman, batchman, mailman, netman, writer" deamons.

Regards,
Viswa.

[kevin_m]

Try running this on the server

conman start CLIENT-A
conman link CLIENT-A

or wait for the next schedule change when the jnextday job runs.

[Viswanadhan]

Hi Kevin,

I did it according to it. Im waiting for next schedule conformation from DBA team.

Regards,
Viswa.

[kevin_m]

If the start and link commands worked then jobman should be running on the client. If you check the status now does it include 'J'?

[Viswanadhan]

Hi Kevin,

I tried to stop, and start from client, but i got same status.

I asked my DBA team to look into jobs.

Regards,
Viswa.

[Viswanadhan]

Hi Kevin,

Problem was resolved. The main problem was problem with the /opt/maestro/bin/jobman owner permission, and /opt/maestro/Mailbox.msg file size.

# chown root jobman
# /opt/maestro/bin/evtsize /opt/maestro/Mailbox.msg 50000000
# /sbin/init.d/maestro stop
# /sbin/init.d/maestro start

Now i see 5 daemons are up " writer, jobman, netman, mailman, batcman".

Regards,
Viswa.

[Viswanadhan]

NA