topic Re: Need help on Maestro Application in Operating System - HP-UX

Need help on Maestro Application

Viswanadhan — Wed, 30 Apr 2008 06:24:46 GMT

Hi Admins,

We have Maestro Software installed on Server-A, clients on Server-B,C...

Till yesterday everything is good. Today we got terrific problem on client machine.

In Client machine im able to start the service, but it's not communicating with Server-A.

But im able to ping from client system to server, and server to client. No changes in software, etc..

Regards,
Viswa.

Re: Need help on Maestro Application

kevin_m — Wed, 30 Apr 2008 12:01:07 GMT

I'm assuming you are referring to IBM Tivoli Workload Scheduler (formerly named Maestro). If so, the problem you're describing can be caused by a few different issues. Lack of free disk space will prevent the communication so I'd check that first if you haven't already. Regardless, please provide the following details:

- operating systems of master (server A) and clients
- status of clients in admin console (specifically 'Jobman Running' and 'Link Status' columns)
- versions of software on master and clients (if known)

Kevin

Re: Need help on Maestro Application

Viswanadhan — Fri, 02 May 2008 03:24:05 GMT

Hi Kevin,

Sorry for delay.

We are using IBM Tivoli. Wehni run following, then i got following info.
# /sbin/init.d/maestro start
MAESTRO for UNIX (HPUX)/CONMAN 6.1 (6.8) (C) Tivoli Systems Inc. 1998
Installed for group 'DEFAULT'.
Schedule (Def) 04/29/08 (#4523) on CLIENT-A. Batchman down. Limit: 20, Fence: 0
start
MAESTRO for UNIX/STARTUP 6.1 (C) Unison Software Inc. 1997
TIVOLI UNIX (HPUX)/NETMAN 6.1 (6.3) (C) Tivoli Systems Inc. 1998
Program patch revision: 6.3
Netman installed under group 'DEFAULT'. [12100.57]
A start command was issued for CLIENT-A. [2202.507]
link SERVER-A;noask
Command forwarded to batchman for SERVER-A
#

If i see the status from GUI for other clients as " LINKED". For one client i donot see anything status as "LINKED" or " UNLINKED". It's blank status.

When i start from GUI, then i got error msg as follows.

CLIENT-A has not been initialixed [2202.21].

May i know the about the " [2202.21] " ?

Regards,
Viswa.

Re: Need help on Maestro Application

Minoia — Fri, 02 May 2008 08:59:30 GMT

Hi Viswa,
do you share the mozart database through NFS ? In this case the problem could be related to STREAMS DEVICE and NFS lock.
Please start the command conman, may be the command will not show the prompt.
Check the process with glance, may be you will see stop reason for STRMS.
In this case you can try to clear the NFS lock between client and server.
1. kill all maestro processes (netman,conman,mailman, batchman and writer)
2. with clear_locks clear all NFS lock between client and server
/usr/sbin/clear_locks -s YOUR_SERVER_NAME
3. from maestro client user
StartUP
conman start
If conman command works go to step 4
4. from maestro client server
conman
link YOUR_CLIENT_CPU
May be it will take a long time, the server have to deploy the Symphony.

In case step 3 doesn't work try to start maestro as root.

Good luck
Gianni

Re: Need help on Maestro Application

Viswanadhan — Fri, 02 May 2008 09:36:32 GMT

Hi,

When do from Server machine
# su - maestro
opt/maestro>StartUp
opt/maestro>conman
conman> start dscprhp1
Error: CLINET-A has not been initialized yet. [2202.21]

Same error msg i got from GUI.

Any guidence.

Regards,
Viswa.

Re: Need help on Maestro Application

Minoia — Fri, 02 May 2008 11:07:58 GMT

The problem is related to netman processes on client.

Please run this commands from client side :

./StartUp
conman> start

If the commands work you can link the client from the server, otherwise follows the procedure explained in my previous post.

Gianni

Re: Need help on Maestro Application

kevin_m — Fri, 02 May 2008 14:50:42 GMT

I agree with Gianni. Kill all the maestro processes on CLIENT-A, run ./StartUp on the client, and issue conman link CLIENT-A on the server. If the link is successful and your client is configured to autostart, all the client processes will eventually start by themselves.

If the link doesn't work however there could be corruption in the client's job and/or message files. There is a fix but it will wipe out the status of current and scheduled jobs on the client. WARNING: You won't be able to see which jobs finished and everything pending for CLIENT-A will be gone.

1) Stop or kill all maestro processes on the client.

2) Clear some files on CLIENT-A ONLY (assuming default path).
mv /opt/maestro/Sinfonia /opt/maestro/Sinfonia.bak
mv /opt/maestro/Symphony /opt/maestro/Symphony.bak
mv /opt/maestro/Jobtable /opt/maestro/Jobtable.bak
mv /opt/maestro/pobox/tomaster.msg /opt/maestro/pobox/tomaster.msg.bak

3) Clear the message file on the server but just for CLIENT-A.
mv /opt/maestro/pobox/CLIENT-A.msg /opt/maestro/pobox/CLIENT-A.msg.bak

4) Run this on CLIENT-A.
su - maestro
./StartUp

5) Run this on SERVER-A.
conman link CLIENT-A

It will push the current schedule from the server to the client and establish the link between them. Keep in mind the client may run jobs unexpectedly if their start times have already occurred.

Kevin

Re: Need help on Maestro Application

Viswanadhan — Mon, 05 May 2008 14:10:03 GMT

Hi Kevin,

I did according to ur information. I hope more than 90% i got resolved. But, when i see the clients status from server as follows.

Client-A LT W
Clinet-B LTI JW
Client-C LTI JW

But the correct status is " LTI JW".

When i do from CLI
# ps -ef|grep maestro( then i donot see jobman, and batchman ).

Other clients i able to see "jobman, batchman, mailman, netman, writer" deamons.

Regards,
Viswa.

Re: Need help on Maestro Application

kevin_m — Mon, 05 May 2008 14:28:30 GMT

Try running this on the server

conman start CLIENT-A
conman link CLIENT-A

or wait for the next schedule change when the jnextday job runs.

Re: Need help on Maestro Application

Viswanadhan — Mon, 05 May 2008 14:36:12 GMT

Hi Kevin,

I did it according to it. Im waiting for next schedule conformation from DBA team.

Regards,
Viswa.

Re: Need help on Maestro Application

kevin_m — Mon, 05 May 2008 16:45:12 GMT

If the start and link commands worked then jobman should be running on the client. If you check the status now does it include 'J'?

Re: Need help on Maestro Application

Viswanadhan — Mon, 05 May 2008 17:01:24 GMT

Hi Kevin,

I tried to stop, and start from client, but i got same status.

I asked my DBA team to look into jobs.

Regards,
Viswa.

Re: Need help on Maestro Application

Viswanadhan — Tue, 06 May 2008 15:17:34 GMT

Hi Kevin,

Problem was resolved. The main problem was problem with the /opt/maestro/bin/jobman owner permission, and /opt/maestro/Mailbox.msg file size.

# chown root jobman
# /opt/maestro/bin/evtsize /opt/maestro/Mailbox.msg 50000000
# /sbin/init.d/maestro stop
# /sbin/init.d/maestro start

Now i see 5 daemons are up " writer, jobman, netman, mailman, batcman".

Regards,
Viswa.

Re: Need help on Maestro Application

Viswanadhan — Tue, 06 May 2008 15:34:59 GMT