Mounting file-systems from other servers using a shared disk array

[FTSE SysAdmin]

Hi,

I'd like to know if would be possible to mount remote file-systems from other servers which are in a shared disk array.

I've got 2 stand alone rp7400 connected to a va7400 and both servers use and can see each other LUNs.

I've been trying to find a correct way to setup and mount a file-system from server A (as read only) on server B and then to mount a file-system from server B into server A (as read only). Well this in theory should work, however I've not been successefull to make it working.

I used SAM but far that I reached was... I created a complete new file-system on the same hardware path that is used by other server without any data loss ... what is really odd.

The hardware part on server A is 0/12/0/0.1.20.0.0.7.5 and on server B is 1/10/0/0.1.20.0.0.7.5

Would someone know if this would be possible and how can I do this? Thanks

Regards,
Miguel

[Doug Burton]

You may want to try exporting the filesystem read only on "System A" and nfs mount it on "System B". Do the same for the "B" to "A" mount.

[A. Clay Stephenson]

The to doing what you want to do is vgimport. As long as both filesystems are read-only this is perfectly legal. Don't use SAM for this.

"I created a complete new file-system on the same hardware path that is used by other server without any data loss."

If I translate the same hardware path to mean the same physical device/LUN, I absolutely assure that at least one part of your statement is false.

[SS_6]

Hi Miguel, I assume that your both servers can talk to each other if yes than NFS is the only solution.
Each server can act as NFS Server as well as NFS Client. I will give you few commands and let me know if further help is required.
On ServerA if want to export /test to ServerB
edit /etc/exports
make entry for
/test -ro=ServerB
nfs demons must be running
If already running you can do
exportfs -a
If there are some already exported file systems than do
#exportfs -i -o ro=ServerB /test


/sbin/init.d/ look for nfs.* and start them if not already running
Do man on exportfs if I made some error.
on server B
#mount ServerA:/test /test (or whatever mount point you want to use)
In above steps ServerA is NFS server and ServerB is nfs client.
Repeate the same steps on ServerB to mount FS on ServerA

[FTSE SysAdmin]

Hi,

Yes! NFS is one possible option. But I rather perfer not to go on that way due to performance reasons. The file-system in question is a DB dump area, which store large files with several Gbs...

Both servers are connected to a va7400, both have access to all LUNs created on the VA, they recognized all hardware paths of the VA. My problem here is how to mount one FS from a LUN that is used by one server (lets say server A) and mount it on server B as read only.

My last attempts always resulted on FS creation and I can't figured out how to make the OS recognize a new device and recognize that device has a FS on it...

On my previous post I forgot to mention one important thing... Server A is running HPUX 11.00 and server B is running HPUX 11.11 (11i)

[SS_6]

If you don't want to use NFS As A.Clay suggested do vgexport with -s -p options on serverA and vgimport with -s option on serverB and activate vg in read only mode. You can also do vgimport on ServerB using all device files manually.For this on ServerA do.
#vgimport -p -m /tmp/vgxx.map vgxx
rcp to vgxx.map to ServerB /tmp directory
On ServerB
do mkdir /dev/vgxxxx
use mknod to create group file
#mknod /dev/vgxxx/group c64 0xXX0000
(XX-use unique minor number)
vgimport -m /tmp/vgxx.map /dev/dsk/cxtxdx /dev/dsk/cxtxdx and so on...

You can alsl think to use least used VLAN(if you have more than one VLANs) for nfs and it will not cause performance issue.

[V.Tamilvanan · Accepted solution]

Hi,
YEs. It is very much possible.
Below is the procedure for mounting a FS on Server A into Server B as Readonly.

On Server A.

1.vgexport -s -p -m /tmp/vgXX.map vgXX
2.rcp /tmp/vgXX.map B:/tmp

On Server B.
1.mkdir /dev/vgXX
2.mknod /dev/vgXX/group c 64 0xXX0000
3.vgimport -s -m /tmp/vgXX.map vgXX
4.vgchange -a r vgXX
5.mount /dev/vgXX/lvolx on /Mountpoint

[Duncan Edmonstone]

Ok, so it can be done using vgexport/vgimport and vgchange -a r, BUT is it supported?

No - its not - HP won't guarantee that you won't have problems with this config (particularly on the system with the read-only mount). Even when a file-system is mounted read-only there are in-memory data structures associated with the file system - change it on one host, but not on another and this could potentially lead to core dumps/panics/corruptions.

The safe ways to do this are:

1. as mentioned, use NFS
2. spend lots of $$$ on Veritas Cluster Volume Manager & Cluster File System

HTH

Duncan

[FTSE SysAdmin]

Many thanks to you all that have replyed to my post :)

Since all servers are connected to this SAN at 2Gb bandwide, copy DB dumps between servers should be a fast process compared to NFS or other network process. All remote FS monted in one server would be read only just to prevent FS problems.

Since the FS are in the SAN I believe that shouldn't be a problem if one of the servers that share a FS to all others. However I'm going to test that.

I ran the following procedure, posted under my original post. Many thanks for it :)

On Server A.

vgexport -s -p -m /tmp/vg03.map vg03
rcp /tmp/vg03.map B:/tmp

On Server B.

mkdir /dev/vg11
mknod /dev/vg11/group c 64 0x110000
vgimport -s -m /tmp/vg03.map vg11
vgchange -a r vg11
mount -o ro /dev/vg11/lvolx /mountpoint


No major issues except one small thing that requires some attention... that is, file/directories ownership/group-id aren't the same and would require a standardization across the servers :)

Many thanks to you all,
Miguel

[A. Clay Stephenson]

I'm not sure you are doing this safely. You absolutely, positively cannot do this unless the filesystems are mounted as read-only on ALL the nodes. For example, you cannot mount the same filesystem read-write on hostA and read-only on hostB and expect hostB to safely access valid data. The problem is that hostB has no idea about what is happening in hostA's buffer cache.

Unless the filesystem is read-only on both hosts, your only option is NFS.

[Printaporn_1]

Hi Miguel,
Stupid answer from me is Buy Service Guard !

[FTSE SysAdmin]

After a couple of weeks working with other Hosts FS mounted as read only on other host (using a VA)

I never had a single issue with them :)

I'm going now to deploy the same structure to the main server. 3 servers would have 2 FS "read only" of the other servers... so copy dumps wouldn't be an issue.

HostB doesnt have any issue accessing the FS provided by HostA even if HostA is down. I've tested that... because its a VA providing a LUN and no problems regarding inods etc... Of course... I didn't test full access for that FS because isn't my objective...

But HostB does not have any problems accessing a FS (read only) from HostA using VA LUN what is great :)

Thanks for all ur tips

[A. Clay Stephenson]

I'm still not clear about what you are doing but unless ALL shared filesystems are mounted read-only on ALL hosts then chaos is going to result.

For example, suppose we have filesystem /u01 mounted read-write on Host A and read-only on hosts B and C. 99.9% of the time everything is fine. The problem occurs when you update, for example, block 17 of /u01/myfile. Now, suppose that /u01/myfile is actually open on all three hosts and that block 17 was in the cache of all three hosts. Hosts B and C wouldn't have a clue that Host A updated block 17 because their buffer caches have not been updated. Files newly created (or deleted) on Host A may or may not be detected on the other hosts, again depending upon the contents of their buffer caches. You could even get into cases where the buffer caches of the the two read-only hosts would not agree with each other so that literally to use my example above, Host A's (the accurate one) notion of the contents of block 17 might be "Tom" while host B's notion might be "Dick" and C's might be "Harry" depending upon the time at which the data was actually read by each.
You could lessen the impact of this by bypassing the buffer cache of each filesystem using the mount options convosync=direct,mincache=direct but you still would have synchronization problems.


Even if your tests to data have given you confidence, I assure that it is a false confidence.

Your only viable options are read-only on all filesystems or read-write on one system and NFS mounts for the others.

[Uwe Zessin]

Miguel,
you should really, really listen to Clay & Duncan.

I have seen such use of unsynchronized file system access on non-HP-UX systems about 20 years ago. It worked 'most of the time' when the caching was turned off, but from time to time it failed horribly. None of the rules have changed since then.

[FTSE SysAdmin]

Stephenson,

In fact its a good point and I'm not excluding possible problems.

What I'm trying to achieve is...

I've got 3 servers; main production server (HostA), a historical server (HostB) and a development server (HostC).

All of them are connected to a SAN where servers recognize all LUNs.

Each server has its own FS for nightly Sybase dumps. Usually we have requests do move and load dumps from HostA into HostC/HostB

I'm talking about sybase dump files with more than 30Gb, where the current process is: check space on the remote server, compressing the files, rcp to other server, uncompress and load. This process takes hours beside that all servers require that FS with the double of the space.

Now... with VA sharing those FS... I saw the following possible workaround...

HostA has its own FS (rw) where HostB and HostC have mounted as read-only.
HostB would also share its FS to HostA and HostC (as read-only)
HostC would share its FS to HostA and HostB (also read-only)

So the main benefits are:

Time reduction of a dump/load process (from hours into minutes)
No need to compress/decompress
Reduction of FS size (since each server will have its own dumps... so no need of extra space for other server dumps plus compression/decompression)

I've been testing that with our HostA providing dumps to HostB and until now I didn't have a single data corruption/syslog errors. But I agree with you.

This isn't a clean solution which guaranty data/system stability. But here I'm compromising some data stability to get speed, where managed to reduce the process from hours into just 45 minutes (time that sybase takes to load 30Gb of data).

I probably didn't have problems because the FS are only used at night and some times during the day... However I'm monitoring the logs and I'll rollback this if something really goes wrong

Spite be an odd method, this process provides a fast dump/load mechanism.

Now my question is... Would someone know a possible product that would provide the same function without data corruption (other than NFS)? Thanks

[A. Clay Stephenson]

Veritas makes something called a Clustered File System that is available for HP-UX; it's made to do what you are trying to do. It's normally intended to be used with Oracle RAC but I would contact them.
http://www.veritas.com/Products/www?c=keymessage&refId=145&psId=1786

Your present method would scare me to death. The last thing I would want are dumps and backups that I cannot absolutely, positively depend on. It would make much more sense in your case, to split mirrors and then mount the split mirrors.

[FTSE SysAdmin]

Thanks for the veritas tip, I'm going to look at it.

As I promised I'd give u feedback about this ...

Today I had my 1st case (like Clay & Duncan had mentioned)... The inod table from the server that has the FS mounted as read-only got several bad inods once a dump file was uncompressed on the server that owns the FS.

Since I'm not aware of any mechanism to sync the inod table of a mounting point... the best process was to remount the FS and the inod table of that FS got fine again...

This isn't a big issue for us because its a FS only used to retain dumps and to copy dumps between servers... so not used all the time.

But its right that the inod tables on the servers that have the remote FS mounted get corrupt (note: inod table referring that particular monting point) if there are inod changes on the server owner of the FS. However they don't create any panic on both servers, neither data corruption on the owner of the FS.

To be exact the host that has the FS mounted mark the inods as bad, showing the files with different size etc. However remounting the FS solves the table inod, refreshing the content of the FS mounted... so no data corruption what so ever (since its read-only FS).

Would someone know of a process to sync that mounting inod table without need to remount the FS? ;)

I can't believe that HP never thought about providing this simply mechanism since the only thing that is needed is to refresh the inod table of a mounting point once detected a change on the inod instead of mark the inods as bad :)

Right the way to go would be Veritas, spite to be a very expensive package just to solve a small refreshing mounting inod table issue :)

Our reality might not be a reality of any other company however data isn't compromised as I thought at beginning.

But until we have Veritas running... we would live with this remote mounting mechanism because refreshing mounting points its easy process to do and the benefits are great compared with NFS :)

I'll post here more info if I get new 'features' :)