Shutdown script spanning more than one server

[Dan Sargis]

I am preparing to formulate a script which will start on my application server, shutting down all applications then telnet to the database server to stop the database.

Then when the DB backup is complete, start the database, exit back to the application server and start the applications.

Does anyone have an example script of how to accomplish this?

[Ivan Ferreira]

You can do this easily with secure shell (or an expect script).

If you use ssh, you should configure public key authentication without password, then just create a script:

stopapp
ssh databasehost stopdb
ssh databasehost startdb
startapp

[Dan Sargis]

Thank you Ivan for the reply.

I am uncertain what ssh is. I did a "man ssh" and nothing came back. You provide a little more info?

Thank you very much in advance for you reply.

[A. Clay Stephenson · Accepted solution]

Remsh or ssh commands make much more sense than telnet's. However, I can think of a way that's even better. Suppose that you stop your application(s), shutdown the database, take vxfs snapshots of the database files and of the application files; restart the database, and restart the applications. You then backup the snapshot mounts and unmount the snapshots when finsihed. You could then have "cold" backups with almost all the uptime of a "hot" backup because vxfs snapshot mounts take only seconds to complete.

I described this process along with the synchronization tools needed in this
article: http://www.samag.com/documents/s=7898/sam0304a/0304a.htm

The perl semaphore server and client tools are also available at that website.

[A. Clay Stephenson]

SSH is the secure shell. Get it from:
http://h20293.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=T1471AA

Essentially, it's the encrypted version of remsh and ftp in that the data sent over the network are encrypted. You may or may not need that level of security within your own network but in any event the scripting will be much easier using remsh or ssh than with telnet.

[Dan Sargis]

Thank you, Mr. Stephenson.

I am reading your article right now. I was originally leaning toward "remsh".

We have and HP SAN and have yet to implement snapshots but would LOVE to. Would you happen to have an example of your "createsnapshot.sh" script?

I am reading on .....

[Ivan Ferreira]

Probably, you don't have it installed, check it with:

# swlist -l product |grep -i secure

Secure_Shell A.03.10.007 HP-UX Secure Shell

You can download it from the hp software download page.

[A. Clay Stephenson]

You really don't need a script to show you how to create vxfs snapshot mounts; man mount_vxfs will tell you (almost) everything you need to know. The thing that it doesn't tell you is how big the snapshot buffer raw device should be. For backups completed in a reasonable amount of time (a few hours) and a filesystem that is undergoing moderate levels of updates during this interval (moderate means fairly heavy), a snapshot buffer 15% the size of the original filesystem should more than suffice.

My convention is that snapshot mountpoints are prefixed via "snap" so if your original filesystem is "/PrdOra/oradata1" then your snapshot mount point will be "/PrdOra/snaporadata1". Now for those mounts using a snapshot buffer, /dev/vg03/lvol7 the commands are:

mkdir /PrdOra/snaporadata1
mount -F vxfs -o snapof=/PrdOra/oradata1 /dev/vg03/lvol7 /PrdOra/snaporadata1

Now you backup /PrdOra/snaporadata1
and when finished:

umount /PrdOra/snaporadata1
rmdir /PrdOra/snaporadata1

[Dan Sargis]

What software package does "snapof" come with?

We have our SAN but evidently are missing some software.

[A. Clay Stephenson]

You need OnlineJFS but that is a package that no HP-UX box should be without. In additional to more mount options it also allows you to change the size of a filesystem "on the fly" while the filesystem is in use.

http://h20293.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=B3929CA

[A. Clay Stephenson]

Note that "snapof" is not a separate command; it is simply a vxfs mount option. Man mount_vxfs and you should see it described whether or not OnlineJFS is installed.

[Dan Sargis]

yeah, since my last post I realized that snapof was a continuation of the mount statement.

I think that the only thing I'm missing is how to create the snapshot buffer.

Here's my syntax and error:

atlas03:/s01# mount -F vxfs -o snapof=/s01/oradata /dev/vgs01/slvol1 /s01/snaporadata
mount: /dev/vgs01/slvol1 is already mounted on /s01/snaporadata
atlas03:/s01# bdf .
Filesystem kbytes used avail %used Mounted on
/dev/vgs01/slvol1 153600000 68066672 84865120 45% /s01
atlas03:/s01#

[A. Clay Stephenson]

The snapshot buffer can be any raw BLOCK (as opposed to character) device (e.g. a whole physical disk or LUN, /dev/dsk/c5t2d0; an LVM logical volume; /dev/vg04/lvol1; or a VxVM subdisk).

For example, if you are using LVM, you would lvcreate a new LVOL but not install a filesystem on it (or make an entry in /etc/fstab for it). Any raw block device
can be used -- just make sure that this device is not used for anything else. Every snapof mount will need its own snapshot buffer raw block device.

[Dan Sargis]

Wow this is a really cool option of which I was previously unaware.

Let me gain a little better understanding of "snapof". As you indicated, a great use of this is for cold back-up. However, would you not want to keep the DB down until the tape backup is completed? Otherwise, won't the /snaporadata buffer be updated as transactions continue to flow with the DB up and a tape backup going simultaneously?

[A. Clay Stephenson]

No, not at all. That why it's called a snapshot. It's a frozen image of the filesystem as it was at the time the snapshot was taken. Here's how it works. The snapshot doesn't actually copy anything initially; that's why it can complete in seconds even for a huge filesystem. Reads from the snapshot mountpoint are a composite of unchanged blocks in the original filesystem plus changed blocks read from the snapshot buffer. When a read() is done for, say, block 100. The change map is consulted. If block 100 has not been updated then the block is read directly from the original filesystem. Now suppose that block 303 has been updated by an operation in the original filesystem. Just before the write() operation occurs in the original filesystem for that block, the original unchanged contents of that block are written to the snapshot buffer and the change map reflects that block 303 has been changed and it also indicates what block in the change buffer to read instead. Only the first change to block 303 need be recorded in the snapshot buffer because all we care about from the point of view of the snapshot mount is what block 303 looked like when the snapshot was taken. Now when block 303 is being read by your backup software, the OS examines the change map to see if block 303 has been changed. Yes, so go fetch it from the indexed location in the snapshot buffer. All of this takes place in the background and is completely invisible to processes accessing eith the original filesystem or the snapshot filesystem. This means that your database can be restarted and in use while from the point of view of the snapshot, the database is still in the shutdown state.

It should be obvious that most of the read()'s for the backup occur in the original filesystem. That's why the snapshot buffer need only be about .15X the size of the original filesystem.

It is essential that all the filesystems associated with the database be snapshoted before restarting the database so that you have a valid "cold" backup. You really have all the "warm, fuzzy" of a cold backup for almost all the uptime of a "hot" backup.

One "gotcha" I should warn you about is that the snapshot buffer should be mirrored or otherwise highly available (e.g. an array LUN) because if for some reason the snapshot buffer device fails, the original filesystem can hang. Of course, if a disk in the original filesystem fails the same thing can occur. I'm just pointing out that
just as the original filesystem should be mirrored (or otherwise highly available) so should the snapshot buffer device.

[A. Clay Stephenson]

Now to really confuse you: It's possible to do a second snapof= mount using a different snapshot buffer and a different snapshot mountpoint of that same original filesystem while the first snapof= mount is still in effect. The first will be a snapshot as the
filesystem was at point in time it was snapped and the second will be a snapshot at the point in time that it was snapped. I've only used that feature a few times but it has great "coolness".

[Dan Sargis]

This is great news regarding "snapof". I discussed it with my systems staff and in the words of our senior sysadmin "I knew we should be getting more for what we're paying for JFS". They never showed us this in HP-UX System administration class.

Anyway, I'm going to review your article related to scripting using semaphores and I'll get back to you with questions.

Thanks again.

[A. Clay Stephenson]

I don't think snapshots are talked about until the "Ligical Volume Manager and Mirror/UX" class (or whatever the name actually is) but that's no excuse. The man pages (from end to end) are required reading for anyone who calls oneself a Sysadmin. My favorite is Section 2 (the system calls -- it's a real page turner) but in any event, it's amazing what one can learn by reading all the man pages rather than just consulting them when needed.

[Dan Sargis]

OK, back to the original question (I'm getting two for the price of one!).

I downloaded both server.pl and client.pl to one of my servers just to, as you put it, get a feel for how they work.

By the way the version of Perl on the Unix server is:

atlas03:/usr/contrib/bin# swlist | grep perl
perl B.5.6.1.C Perl Programming Language
atlas03:/usr/contrib/bin#

When I try running server.pl -P 7777 #, it just returns the following:

atlas03:/usr/contrib/bin# server.pl -P 7777
server.pl[4]: require: not found.
server.pl[10]: use: not found.
server.pl[11]: use: not found.
server.pl[12]: use: not found.
server.pl[13]: use: not found.
server.pl[14]: use: not found.
server.pl[15]: use: not found.
server.pl[16]: use: not found.
server.pl[18]: Syntax error at line 19 : `newline or ;' is not expected.
atlas03:/usr/contrib/bin#

Now, I admit that I'm very GREEN with Perl and it looks as if this will be my launching pad into the language but must I comment these lines out before the script will work?

[A. Clay Stephenson]

The version of Perl found under /usr/contrib/bin is very archaic. /usr/contrib/bin probably appears early in your PATH setting.

Probably the best fix is to create a symbolic link in /usr/bin for the "good" version of Perl.

cd /usr/bin
ln -s /opt/perl/bin/perl perl

That should fix you.

[Dan Sargis]

Hmmm, I still get the "require: not found" and the multiple "use: not found" errors after providing the symbolic link you listed.

[A. Clay Stephenson]

Oh, I assume that you set the execute bit on the Perl scripts as well.

e.g. chmod 755 Server.pl Client.pl

Also make sure that the "shebang", /usr/bin/perl -w, is the VERY FIRST line of the script. Perhaps when you downloaded the files, a leading blank line was inserted.

In any event, between setting the symbolic link, setting the execute bit on the file, cleaning up the shebang line, and possibly making sure that the current working directory ("./") is in your path or that you prefix Server.pl with the absolute path, you should be good to go.

The opther option is

perl Server.pl xxx yyy zzz.

[Dan Sargis]

Thanks that certainly helped it to execute. Now, here's where I'm at. The server.pl is started on the "boss" server ("atlas03"). I attempted to execute the client.pl on the client server and here's what I got:

titan03:/usr/contrib/bin# client.pl -P 7777 atlas03 -P 7777 Resource_1 INC 1
Invalid Tag atlas03 at /usr/contrib/bin/client.pl line 282.
titan03:/usr/contrib/bin#

[A. Clay Stephenson]

WRONG: client.pl -P 7777 atlas03 -P 7777 Resource_1 INC 1

RIGHT: client.pl -P 7777 -h atlas03 -P 7777 Resource_1 INC 1

NOTE: You can both server.pl and client.pl with a "-u" argument for full usage instructions.

[Dan Sargis]

Augghhh!

Thanks again!