Could not make directory: Permission denied (errno = 13)

[DuKul]

Hi all,

I am facing strange issue.

I have made an Ignite server as a package within a cluster. This package has its own name (r_igdep) and via the package the following two dirs are exported to everyone (anon=2):

/var/opt/ignite/clients (everyone)
/var/opt/ignite/recovery/archives (everyone)

One of the nodes could be backuped easily and without any issue, while the another one is claiming following error when starting Ignite backup:

200812091443::OMN_002_003::normal::make_net_recovery started
ERROR: Could not make directory: /var/opt/ignite/recovery/client_mnt/0x001E0B834199: Permission denied (errno
= 13)
ERROR: Failed creating the directory /var/opt/ignite/recovery/client_mnt/0x001E0B834199:Permission denied
(errno = 13)
ERROR: Failed to create NFS mount directories for configuration files.
* Creating NFS mount directories for configuration files.

======= 12/09/08 14:43:20 EET make_net_recovery completed unsuccessfully
200812091443::OMN_002_001::critical::[ERR] Ignite Backup sapdbs2 (make_recovery) failed, with return code 1

Have you any hints?

[MarkSyder]

/usr/include/sys/errno.h helpfully tells us that errno 13 means Permission denied! I think we could probably have worked that out for ourselves.

What user are you running ignite as?

Mark Syder (like the drink but spelt different)

[DuKul]

Ignite is running as a root

[MarkSyder]

If you log on to the server with the problem and type:

ll -d /var/opt/ignite/recovery/client_mnt

What is the output?

Mark

[DuKul]

Yes, I have tried this by myself earlier, and this really confuses me:

#ll -d /var/opt/ignite/recovery/client_mnt
drwxr-xr-x 3 bin bin 96 Dec 9 14:44 /var/opt/ignite/recovery/client_mnt

[Scot Bean]

Before creating a recovery archive for the first time, you may need to do this

# mkdir -p /var/opt/ignite/recovery/archives/
# chown bin:bin /var/opt/ignite/recovery/archives/
Just a one-time setup.

[DuKul]

Yes, this was one of my first tricks to overcome it, but unfortunately to no avail :-(

[Dennis Handly]

>the following two dirs are exported to everyone (anon=2):
>/var/opt/ignite/clients (everyone)
>/var/opt/ignite/recovery/archives (everyone)

Didn't you want anon=-2?

Did you export root as root for those two filesystems? Otherwise root on the client can't create those directories.

Scot's chmod bin:bin and anon=2 would make sense.

[DuKul]

So I have tried the trick:

#mkdir /var/opt/ignite/recovery/client_mnt/0x001E0B834199
#chown bin:bin /var/opt/ignite/recovery/client_mnt/0x001E0B834199

but still the same error.

BTW when running the Ignite from the other node, everything is going fine, Ignite is managing its creating dirs by itself, as it should be.

Any idea?

[Scot Bean]

Is /var/opt/ignite a shared filesystem across the cluster?

[DuKul]

yes, the needed dirs are exported via HA-NFS:

XFS[0]="-o anon=2 /var/opt/ignite/clients"
XFS[1]="-o anon=2 /var/opt/ignite/recovery/archives"

[DuKul]

Hi all,

so the situation gets complicated:

now if trying to halt the associated package, the montpoint cannot be umounted:

Dec 18 15:11:33 - Node "sapdbs2": Remove IP address 10.20.17.100 from subnet 10.20.17.0
HANFS -- Dec 18 15:11:33 - Node "sapdbs2": Unexporting filesystem on /var/opt/ignite/clients
HANFS -- Dec 18 15:11:33 - Node "sapdbs2": Unexporting filesystem on /var/opt/ignite/recovery/archives
HANFS -- Dec 18 15:11:33 - Node "sapdbs2": Killing rpc.statd
HANFS -- Dec 18 15:11:33 - Node "sapdbs2": Killing rpc.lockd
HANFS -- Dec 18 15:11:33 - Node "sapdbs2": Restarting rpc.statd
HANFS -- Dec 18 15:11:33 - Node "sapdbs2": Restarting rpc.lockd
Dec 18 15:11:33 - Node "sapdbs2": Unmounting filesystem on /dev/vgigdep/lvvaroptignite
umount: cannot unmount /var/opt/ignite : Block device required
umount: return error 1.

WARNING: Running fuser to remove anyone using the file system directly.
/dev/vgigdep/lvvaroptignite:

umount: cannot unmount /var/opt/ignite : Block device required
umount: return error 1.
ERROR: Function umount_fs
ERROR: Failed to unmount /dev/vgigdep/lvvaroptignite
Dec 18 15:11:33 - Node "sapdbs2": Deactivating volume group vgigdep
vgchange: Couldn't deactivate volume group "vgigdep":
Device busy
Dec 18 15:11:33 - vgchange -a n vgigdep failed, trying again.
vgchange: Couldn't deactivate volume group "vgigdep":
Device busy
Dec 18 15:11:34 - vgchange -a n vgigdep failed, trying again.
The package script for igdep failed with no restart. igdep should not be restarted
Unable to halt package igdep
Check the syslog and pkg log files for more detailed information
sapdbs2:/#vgchange: Couldn't deactivate volume group "vgigdep":
Device busy
ERROR: Function deactivate_volume_group
ERROR: Failed to deactivate vgigdep

this is really bad - I have tried to bypass the issue with killing syncer daemon and alter mntab (while having the package up, down) but to no avail.

NFS part seems to work well (eporting, un-exporting dirs correctly), so I do not know how to proceed.

HELP please. But without reboot - it is impossible to reboot the system now before Christmas.

Dukul

[COSCON]

The ownership of "/var/opt/ignite/clients/0x*"should be bin:bin,not root:sys

Also check the permissions under
/var/opt/ignite/recovery/*/* the files here too should have bin:bin

Try it.