Can't get rid of these backup jobs (Omniback II A.3.50)!

[Matt Hearn]

I have an environment that we have been slowly but surely shrinking, so servers have been decommissioned left and right. As we do this, the Maestro jobs that run on them and that start the backups on our main backup server have been turned off.

Out of probably 20 servers I've decommissioned in two years, only three of them are giving me lingering problems in the backup environment, because something keeps firing off backups for them that inevitably fail every day because the servers have been turned off and shipped elsewhere.

They aren't running out of cron, as far as I can tell; there's nothing in any cronjobs to start them. They aren't running out of Maestro; the entire Maestro environment has disappeared, and I've disabled the agents on the backup server. They can't be triggered by the hosts themselves because they aren't there. I've checked omniback, and it doesn't have a schedule for any of these jobs. I even tried removing the backup specs, but something continues to fire them up every night at 8pm on the dot.

I can't, unfortunately, just ignore them, because my various managers get antsy when they see red items on their pretty green reports.

How can I kill these backups once and for all?

Thanks!

[Pedro Cirne]

Hi,

Probably there is a service that is started on boot...check

-/etc/rc.config.d directory if there is some configuration file for omniback

-/sbin/init.d directory if there is any script to start omniback

Enjoy :)

Pedro

[Rick Garland]

You can remove the job specifications.

[John Poff]

Hi,

Are the jobs kicking off at 15 minute increments (top of the hour, 15 minutes after, 30 minutes past, etc.), or are the times more random? If they are at 15 minute increments I would suspect a schedule buried somewhere in OmniBack, otherwise I'd look for a rogue cronjob doing a 'remsh omnib..." to the OmniBack server. I've outsmarted myself like that in the past.

JP

[Mel Burslan]

This usually happens in any large IT environment. Some clever programmer or sysadmin, at the lack of an official scheduler like maestro, sets up a rogue scheduler, or even better writes one of his own in the form of a shell script. Then this person leaves but his scheduled jobs keeps running because he/she did not bother to tellanyone what was done and of course documentation is for wimps :)

As you seem to pinpoint the timing of these backups, I would monitor these systems at the start time of the backups very closely and see see what processes were running just prior to these omnib processes get launched. It may track you back to the source.

My guess, they are launched by a process which stays dormant in the background and nobody notices it.

[Matt Hearn]

Hi Rick, I tried removing the specs once already. Something was still kicking them off! Truly disturbing.

Hi John, they're all kicking off at exactly 8pm every day, as if omniback was doing it. But there's no schedule for the backups. I've poked around all the servers I can think of to see if something is running a remsh or ssh to kick off the backups, but haven't found anything yet.

Thanks, guys!

[A. Clay Stephenson]

An omnib command would not require a datalist although that certainly makes things easier. While you said they are not being fired by cron you did not exclude "at". Have a look at your atjobs. If you can capture the ps -ef output, you would then have at least a portion of the command line and the PID and PPID of the process.

[Rick Garland]

What kind of enterprise-wide scheduler do you have in place?

If the jobs are starting at same time but they are not in cron or the omni scheduler, something is starting.

Note, you can have a cron configuration that is capable of running the enterprise. Question then becomes which system is doing the scheduling.

[Mel Burslan]

A suggestion:

to eliminate the possibility of a remote server launching these backups, I would suggest to disable the r-commands for a brief period of time, unles this is a production server and needs to be available 24x7, something like disabling remsh, rexec, ssh between 7:55 and 8:05, just to see if you are looking at local or remote launch.

[Steve Post]

I assume you checked the directory
/etc/opt/omni/schedules correct? If not....

As I recall, the ability to modify the schedules in omniback was pretty poor on the unix side. (Perhaps it's just my opinion).
But I bring it up because the original schedules may accidentally is set to still run the old backups. Why dont you just:

cd /etc/opt/omni/schedule
grep -il 2000 * | more
I say 2000 because that's 8pm in military time.

Program /opt/omni/sbin/omnitrig is probably running every 15 minutes via cron. The program looks at those schedule files to determine if stuff should run. At least that's how it appears to me.

steve