Simpler Navigation coming for Servers and Operating Systems
Coming soon: a much simpler Servers and Operating Systems section of the Community. We will combine many of the older boards, and you won't have to click through so many levels to get at the information you need. If you are looking for an older board and do not find it, check the consolidated boards, as the posts are still there.
Showing results for 
Search instead for 
Did you mean: 

HPUX, RAC and rebalancing SAN ( Veritas )

Go to solution
Henrique Silva_3
Regular Advisor

HPUX, RAC and rebalancing SAN ( Veritas )

We have an issue that I am getting conflicting information on, so, I would like for somebody to point me to a link, or let me know what is going on here.

We have several rx7640 ( 8 cpus, 64 GB ), and these are setup in 4 node clusters for RAC, sharing 1.2 TB per system using Veritas.

On these systems, we are sharing many RAC DBs ( up to 50 in some of them ). The issue at hand is that when these were built, the archive log mount points were not broken down per DB, as we did not know which dbs were going to exist there, so, several 25 GB mount points were created, with generic names, so taht we could build the DBs, and later, we were supposed to break those down, and create unique mount points for each db. Needless to say this did not happen, and we are now, trying to address it.

SO, the way we are proposing to do this is to use some existing mount point ( such as reorg ), which is quite large and can accomodate all DBs ( 400 GB ). we would unmount that file system, give the storage back to the VG, and create the unique mount points from that, so that, and later, once we move all the archive log files to the new mount point, we can reclaim the generic ones, and rebuild the 400 GB reorg mount point.

Two issues came up, and I need your help.

First, we are being told that in order to recreate all these new mount points, we will need to remap the CRSP MCSG package, that controls the cluster mount points, and that down time is needed. I am not sure why yet. Just got a short email and I am meeting with the Unix guys tomorrow.

Second issue is that some other resource, and I am not sure if it is from the Storage or HPUX team, is telling us that the rebalancing of the SAN, once we are manipulating the storage, can take up to 24 hours. This is done so that they can keep the SAN optmized at all times.

So, the question is, with a high available cluster, with RAC db on 4 nodes, there is NO WAY to do this, without down time ? It seems a bit silly to me that you would spend all this money on something, and not be able to keep its uptime high !!!

Any info here will be appreciated !!!


"to be or not to be, what was the question ???? "
Honored Contributor

Re: HPUX, RAC and rebalancing SAN ( Veritas )

Re: downtime while remapping the package: true. And, it's quite a bit of work. Plus - I'd guess that the team wants a backup of the system before they take this on, before making changes.

That's the problem with clusters, since the level of interdependency goes up for the virtualization of the resources, also goes the complexity. Good solution, but it's a real amount of work to maintain it, and more than anything else, making sure it's right, and making sure it moves properly and correctly from server to server.

It's just not nearly as simple as manipulating a servers' own direct set of disk resources needed for use by only one server - not by a long shot.

Since the cost of doing it wrong is so high, proper time needs to be allocated for writing it, double checking it, and reviewing and testing it for all servers involved.

And, since it involves unrolling and rerolling disk areas that are actually part of the running package, the down time is higher, because it really can't be done in a separate test package outside of the uptime of the active package, it has to be done to the active package.

Now, it can be "mocked up" in advance as much as possible by writing scripts of what the package *should be* - which helps, but does not eliminate the big step of reconfiguring disk and package contents for each client server.
We are the people our parents warned us about --Jimmy Buffett
Steven E. Protter
Exalted Contributor

Re: HPUX, RAC and rebalancing SAN ( Veritas )

Shalom Henrique,

You are talking about changing the mount points of the RAC databases. Of course it requires downtime to do this.

Lets take a step back.

One gigantic area for the databases to set. Sounds like a great way to create hotspots on the SAN disks and not having a clue which database is having the problem.

Yes, having a very big mount point makes storage management simpler, however it makes performance management much more difficult in the future.

The Second issue about rebalancing the SAN makes sense. You will be changing SAN but not wanting to have downtime. So the load balancing will need some time to figure out the new I/O patterns and optimize the use of the various i/o channels in the SAN.

You mave serviceguard managing the mount points and you want to make changes. This is the cause of the need for downtime. If you have sufficient storage, you might be able to use mirror/ux to mirror the raw disk areas to new mount points and avoid some of the downtime.

It seems that the planning of the serviceguard configuration controlling the mountpoints is the issue thats come back to haunt you. No plan for making major changes to mountpoints has been done. Take a look at your setup and see if you can do hot, live mirroring to avoid the downtime. Mirror/UX can replicate the RAC raw disk areas and then theoretically you can kill the original and the database will still run.

Caveat. It will take a long time and put i/o stress on the SAN while you are replicating logical volumes(RAC raw disk). It may be faster to schedule and tolerate the downtime. Which is worse, giving the users a Saturday night off or a week of complaints in those whiney voices about how slow things are.

You may have tools on the SAN that permit you to do snapshots and duplicate LUNS that can also be used to avoid downtime. Same problem as mirror/ux but not as bad.

You made a long post, but there not enough detail to provide you a plan to totally avoid downtime. Looking at your actual setup and layout of available space with the ideas I've posted here may provide you a way of reducing the downtime.

Steven E Protter
Owner of ISN Corporation
Henrique Silva_3
Regular Advisor

Re: HPUX, RAC and rebalancing SAN ( Veritas )

Thanks guys.

Here is a bit more detail :

We are ONLY creating archive log mount points, not touching the database files at all.

So, we have several of these :

# bdf | grep ora | grep db_
25600000 150201 23859821 1% /u01/app/oracle/arch/db_8
51200000 29634 47972225 0% /u01/app/oracle/arch/db_16
51200000 56385 47947567 0% /u01/app/oracle/arch/db_14
51200000 89811
and so on ( 16 per box )
204800000 52254643 143011313 27% /u01/app/oracle/reorg

we want to give these back to the SAN, and in their place, create unique mount points such as

/u01/app/oracle/arch/DB1 which is 10 GB
/u01/app/oracle/arch/DB2 which is 20 GB
/u01/app/oracle/arch/DB3 which is 5 GB, and so a 50 GB db_14, might be broken up into 5 10 GB unique points.

In order to avoid down time, we were going to move the arch_log_dest to some other spot, maybe on reorg, trigger an rman archive log backup, so that the existing archive logs would be removed from the old ones, and when the mount point would be empty, de-allocate the storage, and re-allocate again, now, with the proper unique mount point and size.

I am not too concerned about performance, as we will be using the same VG that we were previously using, just breaking them up into smaller mount points.

So, if the only think we are touching, is the archive log mount point, and we can play around with the arch log dest, is there any way to avoid the down time ?

What if we are only ADDING new mount points, from unallocated storage ( we are trying to avoid this, because the RED tape is painfull ), to build the new ones, and then, unallocate the old mount points, can we bring one node down at a time, so that it would resync with the new mapping ?

I really appreciate the help here. Trying to learn as much as I can prior so that I can ask the proper questions.


"to be or not to be, what was the question ???? "