topic Re: dmesg showing vxfs: mesg 008... error 6 on /var filesystem inode in Operating System - HP-UX

dmesg showing vxfs: mesg 008... error 6 on /var filesystem inode

JDR45 — Wed, 04 Apr 2018 16:09:28 GMT

Hello,

I have a HP RP3440 running 11.11. It reboots via cron twice a week around 3am. When the system comes up I get this message in the syslog. It sends out about 15 messages, all at 03:57, and then is fine the rest of the time-

Apr 4 03:57:25 vmunix: msgcnt 9 vxfs: mesg 008: vx_direrr: vx_dirscan_2
- /var file system inode 28171 block 4520501 error 6

I ran this command-
find /var -inum 2871 -print

And the output was
/var/dt

/var/dt is an empty directory dated March 19th at 15:29

Is there a way I can fix this?

I've done some searching on mesg 008 and error 6 and found some similar messages but none quite like the one I am seeing.

Thanks!

Re: dmesg showing vxfs: mesg 008... error 6 on /var filesystem inode

Patrick Wallek — Wed, 04 Apr 2018 19:01:57 GMT

This link comes from Veritas, the folks that write the VxFS file system, and seems to match your error exactly.

https://support.symantec.com/en_US/article.TECH1539.html

Even before reading this, I would recommend a FULL fsck of that file system.

# fsck -y -o full /dev/vg00/rlvol8

(or whatever the lvol of /var is on your system).

It may be best to boot into single-user mode to do the fsck since you really don't want the file system mounted while doing that.

Re: dmesg showing vxfs: mesg 008... error 6 on /var filesystem inode

JDR45 — Wed, 04 Apr 2018 19:16:40 GMT

Reading the link now, thanks!

Here's how /var looks on a bdf-

/dev/vg00/lvol7 36896768 1940896 34683616 5% /var

And once I'm inside /var, the empty "dt" directory looks like this-

drwxr-xr-x 4 root sys 8192 Mar 19 15:29 dt

As root I used vi to create and place a small test file inside /var/dt, and sure enough got the message "no such device or address" and it would not let my place the test file inside /var/dt. The rest of /var looks good.

Will look into doing the fsck soon.

Re: dmesg showing vxfs: mesg 008... error 6 on /var filesystem inode

Bill Hassell — Thu, 05 Apr 2018 13:55:08 GMT

This is an early warning that the disk may be going bad.

Do you have your boot disk mirrorred?
Are you using Ignite to create a recovery image?

Re: dmesg showing vxfs: mesg 008... error 6 on /var filesystem inode

JDR45 — Thu, 05 Apr 2018 14:00:01 GMT

So the full file system check ran last night in single user mode. It didn't find anything, but after booting back up the same message was waiting for us-

Apr 5 02:05:13 vmunix: msgcnt 32 vxfs: mesg 008: vx_direrr: vx_dirscan_2
- /var file system inode 28171 block 4520501 error 6

About one second before those messages appeared, I saw this in the syslog-

Apr 5 02:04:56 automount[742]: Automount abort!!!
Apr 5 02:04:56 automount[742]: Possible cause:
Apr 5 02:04:56 automount[742]: 1)No map provided
Apr 5 02:04:56 automount[742]: 2)Using the same map as the running aut
omount
Apr 5 02:04:56 automount[742]: 3)The maps contain invalid mount points
Apr 5 02:04:57 inetd[761]: Reading configuration

Basically, the server boots, puts out the automount messages, then 1 second later the "vx_direrr: vx_dirscan_2" messages come along, then it boots up, the users hop on the server, and everything runs fine and the syslog stays clear until the next reboot..

I wonder if something was configured wrong with /var/dt vs there being a hardware or filesystem problem?

Re: dmesg showing vxfs: mesg 008... error 6 on /var filesystem inode

JDR45 — Thu, 05 Apr 2018 14:01:57 GMT

And for Bill Hassell- yes, this is a mirrored pair of HDs and we do frequent backups.

Re: dmesg showing vxfs: mesg 008... error 6 on /var filesystem inode

JDR45 — Thu, 05 Apr 2018 14:59:31 GMT

Here's what I was able to pull from the console logs related to the fsck that ran overnight-

/dev/vg00/lvol7 :

vxfs fsck: sanity check: /dev/vg00/lvol7 needs checking

log replay in progress

file system is not clean, full fsck required

pass0 - checking structural files

pass1 - checking inode sanity and blocks

pass2 - checking directory linkage

pass3 - checking reference counts

pass4 - checking resource maps

OK to clear log? (ynq)y

set state to CLEAN? (ynq)y

vxfs mount: /dev/vg00/lvol7 is corrupted. needs checking

mount: /dev/vg00/lvol1 is already mounted on /stand

# bdf

Filesystem kbytes used avail %used Mounted on

/dev/vg00/lvol3 983040 138952 791339 15% /

/dev/vg00/lvol1 961051 49962 814983 6% /stand

/dev/vg00/lvol8 206503936 73208248 132256200 36% /JR

/dev/vg00/lvol6 7471104 1139336 5936139 16% /usr

/dev/vg00/lvol5 2097152 2007 1964260 0% /tmp

/dev/vg00/lvol4 5177344 1446421 3497758 29% /opt

# umount /JR

# fsck -y -o full /dev/vg00/rlvol7

fsck: /etc/default/fs is used for determining the file system type

log replay in progress

pass0 - checking structural files

pass1 - checking inode sanity and blocks

pass2 - checking directory linkage

pass3 - checking reference counts

pass4 - checking resource maps

OK to clear log? (ynq)y

# fsck -y -m -o full /dev/vg00/rlvol7

vxfs fsck: sanity check: /dev/vg00/rlvol7 OK

/dev/vg00/lvol7 :

vxfs fsck: sanity check: /dev/vg00/lvol7 OK

Re: dmesg showing vxfs: mesg 008... error 6 on /var filesystem inode

Bill Hassell — Thu, 05 Apr 2018 15:15:43 GMT

The HP-UX errno value for 6 means:

from /usr/include/sys/errno.h:
      errno 6 = ENXIO /* No such device or address      */

      [ENXIO]        No such device or address. I/O on a special file refers
                     to a subdevice that does not exist, or is beyond the
                     limits of the device.  It can also occur when, for
                     example, a tape drive is not on line or no disk pack is
                     loaded on a drive.

So this matches what you saw when creating a test file.
It is possibly a bookeeping error in the directory structure, but to verify that there are no bad sectors in /var, do this to read every sector:

# dd if=/dev/vg00/rlvol7 of=/dev/null bs=1024k

The dd comand will take several minutes as it appears that lvol7 is 37 GB in size.
I have attached a copy of the script bdfmegs in case you don't have it. It formats bdf for today's large filesystems:

File-System Mbytes Used Avail %Used Mounted on
/dev/vg00/lvol1 938 48 795 6% /stand
/dev/vg00/lvol3 960 135 772 15% /
/dev/vg00/lvol4 5056 1412 3415 29% /opt
/dev/vg00/lvol5 2048 1 1918 0% /tmp
/dev/vg00/lvol6 7296 1112 5797 16% /usr
/dev/vg00/lvol8 196.9g 69.8g 126.1g 36% /JR

NOTE: The bdfmegs file has a Windoze extension .jpg so it can be attached to this post. It is a simple text file, so remove the .jpg and copy it to /usr/contrib/bin.

I would rename the /var/dt directory to /var/dt_BAD and recreate /var/dt.
The dt directory is just temp storage for Xwindows.

Re: dmesg showing vxfs: mesg 008... error 6 on /var filesystem inode

JDR45 — Thu, 05 Apr 2018 17:39:57 GMT

Thanks for such a detailed answer!

# dd if=/dev/vg00/rlvol7 of=/dev/null bs=1024k

I'll give this a try afer hours, when there aren't so many users on the system.

Re: dmesg showing vxfs: mesg 008... error 6 on /var filesystem inode

JDR45 — Fri, 06 Apr 2018 12:43:00 GMT

So I ran this command early this morning before the users got on-

# dd if=/dev/vg00/rlvol7 of=/dev/null bs=1024k

And after about 6 minutes the result came back-

36032+0 records in

36032+0 records out

No errors, and the in / out number matched, so that's good news.

Also tried this to remove /var/dt and recreate it, but it would not let me remove /var/dt:

# cd /var

# ls -ld dt

drwxr-xr-x 4 root sys 8192 Mar 19 15:29 dt

# rmdir dt

rmdir: dt: Directory not empty

# cd dt

# ls -lar

total 0

# ls -la

total 0

# ll

total 0

# cd /var

# ls -laR dt

total 0

Looks like /var/dt is an odd little gremlin in the system.

Re: dmesg showing vxfs: mesg 008... error 6 on /var filesystem inode

Bill Hassell — Fri, 06 Apr 2018 16:06:13 GMT

You wrote:
>> # rmdir dt
>> rmdir: dt: Directory not empty
>> # cd dt
>> # ls -lar
>> total 0

Try: rm -rf /var/dt
You could also find the inode for /var/dt and remove the inode like this:

# ll -i /var/dt
(first column is the inode number)
# cd /var
# find . -inum ### -exec rm {} \;

Re: dmesg showing vxfs: mesg 008... error 6 on /var filesystem inode

JDR45 — Mon, 09 Apr 2018 13:47:22 GMT

Well between the fsck and the dd command, it is fixed. /var/dt is fine and the syslog is clear of messages, even after a reboot or two. So thanks for both recomendations.

Is there a way I can select two different posts as "accept as solution?"

Re: dmesg showing vxfs: mesg 008... error 6 on /var filesystem inode

Dennis Handly — Mon, 09 Apr 2018 18:17:30 GMT

> Is there a way I can select two different posts as "accept as solution"?

No, only the one. But you can give kudos to each.