Filesystem unmount problem

[Kurt_32]

Hi,

I have unmounted all but one filesystem. When I do a bdf, I see the following:

bdf: /opt/oracle/yyccciidb/oraarch: No such device or address
bdf: /opt/oracle/yyccciidb/product: No such device or address
bdf: /opt/oracle/yyccciidb/admin/IC17: No such device or address
bdf: /opt/oracle/yyccciidb/oradata/IC17: No such device or address
bdf: /opt/oracle/yyccciidb/oradata/IC17/data: No such device or address
bdf: /opt/oracle/yyccciidb/oradata/IC17/index: No such device or address
/dev/vx/dsk/yyccciidbdg/orabase
512000 108767 378041 22% /opt/oracle/yyccciidb

I am unable to unmount the last filesystem.

These are veritas volume manager file systems and are under VCS control.

thanks

[Peter Godron]

Kurt,
what is the error message?
Are you sitting on the filesystem?
cd /
fuser -cu /opt/oracle/yyccciidb
Perhaps an oracle process is using the filesystem?

[Robert-Jan Goossens]

Hi Kurt,

Check if there is an active process keeping the filesystem mounted.

# lsof /opt/oracle/yyccciidb
# fuser /opt/oracle/yyccciidb

Regards,
Robert-Jan

[Arunvijai_4]

Hi Kurt,

In order to unmount, you need to kill all processes running in that file system. You can try the following commands,

# fuser -u /opt/oracle/yyccciidb
to list the processes running in /opt/oracle/yyccciidb

# fuser -ku /opt/oracle/yyccciidb
to kill them

-Arun

[Kurt_32]

I used the fuser command with -ck option on the mount point but no help. I also killed all processes using it. but still....

<>
# umount /opt/oracle/yyccciidb
umount: cannot unmount /opt/oracle/yyccciidb : Device busy
umount: return error 1.

Thanks

[Kurt_32]

And this is what syslog.log says for all the seven file systems:

Mar 21 11:03:42 ymxppdb0 vmunix: msgcnt 2440 vxfs: mesg 025: vx_wsuper - /dev/vx
/dsk/yyccciidbdg/orabase file system super-block update failed

[Peter Godron]

Kurt,
after you have done your:
fuser -ck /opt/oracle/yyccciidb
what does
fuser -cu /opt/oracle/yyccciidb
report?
Is there a daemon respawning new processes against this filesystem?

[Kurt_32]

It reports nothing.

[Robert-Jan Goossens]

Try to dd the disk and check if the disk is ok.
# dd if=/dev/rdsk/cxtydz of=/dev/null bs=1024k

WARNING: msgcnt x: vxfs: mesg 025:
vx_wsuper - mount_point file system super_block update failed

Message 025 Explanation:
An I/O error occurred while writing the super-block during a resize operation.
The file system is disabled.

Unmount the file system and use fsck to run a full structural check. Check the
console log for I/O errors. If the problem is a disk failure, replace the disk
before the file system is mounted for write access.

[Kurt_32]

This is what syslog.log file gives:

Mar 21 11:15:03 ymxppdb0 vmunix: msgcnt 2567 vxfs: mesg 008: vx_direrr: vx_dirsc
an_2 - /opt/oracle/yyccciidb file system inode 2 block 1285 error 6
Mar 21 11:15:03 ymxppdb0 vmunix: msgcnt 2568 vxfs: mesg 025: vx_wsuper - /dev/vx
/dsk/yyccciidbdg/orabase file system super-block update failed
Mar 21 11:15:03 ymxppdb0 vmunix: msgcnt 2569 vxfs: mesg 008: vx_direrr: vx_dirsc
an_2 - /opt/oracle/yyccciidb file system inode 2 block 1285 error 6
Mar 21 11:15:03 ymxppdb0 vmunix: msgcnt 2570 vxfs: mesg 025: vx_wsuper - /dev/vx
/dsk/yyccciidbdg/orabase file system super-block update failed
Mar 21 11:15:03 ymxppdb0 vmunix: msgcnt 2571 vxfs: mesg 008: vx_direrr: vx_dirsc
an_2 - /opt/oracle/yyccciidb file system inode 2 block 1285 error 6
Mar 21 11:15:03 ymxppdb0 vmunix: msgcnt 2572 vxfs: mesg 025: vx_wsuper - /dev/vx
/dsk/yyccciidbdg/orabase file system super-block update failed
Mar 21 11:15:03 ymxppdb0 vmunix: msgcnt 2573 vxfs: mesg 008: vx_direrr: vx_dirsc
an_2 - /opt/oracle/yyccciidb file system inode 2 block 1285 error 6
Mar 21 11:15:03 ymxppdb0 vmunix: msgcnt 2574 vxfs: mesg 025: vx_wsuper - /dev/vx
/dsk/yyccciidbdg/orabase file system super-block update failed
Mar 21 11:15:03 ymxppdb0 vmunix: msgcnt 2575 vxfs: mesg 008: vx_direrr: vx_dirsc
an_2 - /opt/oracle/yyccciidb file system inode 2 block 1285 error 6
Mar 21 11:15:03 ymxppdb0 vmunix: msgcnt 2576 vxfs: mesg 025: vx_wsuper - /dev/vx
/dsk/yyccciidbdg/orabase file system super-block update failed
Mar 21 11:15:03 ymxppdb0 vmunix: msgcnt 2577 vxfs: mesg 008: vx_direrr: vx_dirsc
an_2 - /opt/oracle/yyccciidb file system inode 2 block 1285 error 6
Mar 21 11:15:03 ymxppdb0 vmunix: msgcnt 2578 vxfs: mesg 025: vx_wsuper - /dev/vx
/dsk/yyccciidbdg/orabase file system super-block update failed
Mar 21 11:15:03 ymxppdb0 vmunix: msgcnt 2579 vxfs: mesg 008: vx_direrr: vx_dirsc
an_2 - /opt/oracle/yyccciidb file system inode 2 block 1285 error 6
Mar 21 11:15:03 ymxppdb0 vmunix: msgcnt 2580 vxfs: mesg 025: vx_wsuper - /dev/vx
/dsk/yyccciidbdg/orabase file system super-block update failed
Mar 21 11:15:37 ymxppdb0 Had[2030]: (ymxppdb0) VCS:13079:Resource(dg_YYCCCIIDBdg
): The last 90 invocations of the clean procedure have failed.
=======================

[Peter Godron]

Kurt,
I am not sure why you are getting the No such device messages unless they relat to systems mounted below the /opt/oracle/yyccciidb mount point?!

If there are any other mount-points starting /opt/oracle/yyccciidb, please unmount them first and retry the unmount of /opt/oracle/yyccciidb.

If fuser and lsof report no activity, but umount thinks it's busy, I suspect the only way is to re-boot.

By the way, were you trying to unmount all the disks in order to shut down?

[Deoncia Grayson_1]

Is the database still up on this server? This may be a reason why it won't let you unmount this filesystem.

[Kurt_32]

I have rebooted the server twice. After it comes back up, all filessytems get mounted without any problems... but at the time of unmounting them, I get these messages.

[Kurt_32]

This is what it says when I try to fsck on a filesystem:

# fsck -F vxfs -o full /dev/vx/rdsk/yyccciidbdg/orabase
vxfs fsck: Cannot open /dev/vx/rdsk/yyccciidbdg/orabase: No such device or addre
ss
file system check failure, aborting ...

[Deoncia Grayson_1]

By looking at the docs, it points to a faulty disk, have you ran an ioscan or run a dd on the disks that belongs to logical volume?

Try this link to read upon more about the errors you are getting:

http://docs.hp.com/en/B3929-90011/apas03.html

[Kurt_32]

The disks are sitting in SAN (EMC Clariion) and see to be ok. I just checked them by logging into Navisphere. Also I did ioscan -fnCdisk and it reported them CLAIMED.

[Kurt_32]

Thanks for all your help. A reboot fixed the problem....