NFS Mount Option to Propagate Client's rm Instantly?

[Ralph Grothe]

Hello,

in a cluster we etablished a kind of cross make_recovery_net between cluster nodes.
Due to storage limitations every node's Ignite server is only pulling one other node's image in a round-robin fashion to its local /var/opt/ignite/archives which is merely part of /var fs.
Because of these restrictions in the cronjob that performs the make_net_recovery twice a week on each node there is a prepended NFS mount of the Ignite server's archives just to delete previous images as the /var fs cannot hold more than one image at a time.
Although this is far from ideal, and we are still striving for a dedicated Ignite server with sufficiently attached storage, this used to work somehow.
Lately however the cronjobs tend to abort during make_sys_image due to no space errors.
Despite the prepended deletion NFS mount and umount the space changes don't seem to be propagated by the statd to the make_net_recovery NFS client in time.
I simply suggested to the colleague who wrote the cronjob to insert yet another NFS mount and umount to hopefully get statd "updated".
But this seems dubious.
I wonder if there exist special NFS mount options that would lead to faster propagation of space changes on NFS shares?

Regards
Ralph
the

[HorstB · Accepted solution]

Hello Ralph,

I'm not sure how to get NFS to respond as you require, but have another suggestion. Have you considered using a remsh or ssh to run the rm command on the host node? That should clean up the filesystem right away, I think.

Regards,
Horst

PS. FWIW, this is my first post on these forums.

[Steven E. Protter]

Shalom,

With Linux you can use a sync versus async option.

async might help.

It may also be a server load issue.

SEP

[A. Clay Stephenson]

I would think that nothing more complicated than a "sleep 10" would more than suffice for any latency. Is it possible that the real problem is that another process (possibly on a client far, faw away) has the file open so that the space is not freed eventhough the unlink() has been done?

[Dave Olker]

Hi Ralph,

If you're looking for an NFS mount option that will force the clients to always go over-the-wire to get the latest copy of file and directory attributes on the server (and not rely on cached attributes that may be out of date) then you'd want to use the "noac" option. This option disables the attribute cache on the client.

If this is not what you're looking for please let me know exactly what semantics you're hoping to get and I'll let you know if it's possible.

Regards,

Dave

[Ralph Grothe]

Hello Guys,

many thanks for your suggestions.

Horst,
I think your suggestion is probably the most obvious, and I don't know how we forgot that we are igniting within a trusted host environment as all Ignite clients and servers are nodes of the same cluster.
Therefore, I will assign you ten points,
especially as it is your first posting.
However, within a non-trusted environment I wouldn't want to either scatter .rhosts files or RSA/DSA public keys in authorized_keys files.
But maybe such is a prerequisite or at least common assumption for an Ignite strategy?

SEP,
I looked in the script, and yes my colleague had placed sleep as well as sync commands already.
Maybe you are right that another process could be holding a lock.
To tell, I know too little about Ignite's performance behind the scenes.
I also already had the vague idea that maybe a running automounter could have caused the problem.
To be honest, I don't know why automounter is running at all on these boxes because non of them or their application packages they host has the need to access NFS exports,
apart from the occasional make_net_recovery mounts.
I think I should set AUTOMOUNT=0 and restart nfs.client.
For instance with the current setting, make_net_recovery could as well access its needed share by something like

# ll /net/somenode/var/opt/ignite/recovery/archives
total 0

# bdf -t nfs
Filesystem kbytes used avail %used Mounted on
somenode:/var/opt/ignite/clients
6291456 2010976 4247272 32% /tmp_mnt/net/samoa/var/opt/ignite/clients

I am not sure if make_net_recovery is using such automounter net mounts if it sees that automounter is running and a /net mapping prevelant?

Dave,
I myself was already wondering when reading man mount_nfs if the noac option was what we possibly would require.
I will put it as an option in /etc/exports of the Ignite servers for the archives and clients shares, and will reexport.
Maybe this is doing the trick?

Strangely, the errors don't seem to occur regularily because yesterday a make_net_recovery finished without errors.
However, this was with the modified cronjob script where, as I suggested, the colleague inserted a second (idle) NFS mount and umount.

[Dave Olker]

Hi Ralph,

> I myself was already wondering when
> reading man mount_nfs if the noac option
> was what we possibly would require.
> I will put it as an option
> in /etc/exports of the Ignite servers for
> the archives and clients shares, and will
> reexport.
> Maybe this is doing the trick?

The "noac" option is a *mount* option, not an *export* option. It would not work if you put it in the /etc/exports file and re-export the filesystems. If anything it might break your environment since it's not a valid export option.

You would use "noac" when mounting the NFS filesystems on the client (i.e. mount -o noac ...). This option will allow the clients to immediately see any changes made on the server's filesystem because the client will not cache any file or directory attributes.

So if your problem is one where files/directories are modified or removed on the server and the clients are not seeing these changes quickly enough, the "noac" option should help as it will force the clients to always contact the server to get the latest information about the NFS filesystem without relying on what could be stale cached attributes.

Regards,

Dave

[Ralph Grothe]

Oh, sure it is a mount option.
Otherwise it wouldn't be mentioned in man mount_nfs.
I put it in the -o list of the mount command of the cronjob.
But I would have noticed on issuing "exportfs -a", anyway ;-)
I must have been a bit confused today.
I have been recovering a Solaris box with VxVM root disks, which is a morass compared to the laid-back Ignite recovery and LVM root disks on HP-UX.

[Dave Olker]

No worries Ralph. I'm sure you'll be dealing with VxVM root disks on HP-UX systems soon enough. :)