dmesg showing vxfs: mesg 008... error 6 on /var filesystem inode

[JDR45]

Hello,

I have a HP RP3440 running 11.11. It reboots via cron twice a week around 3am. When the system comes up I get this message in the syslog. It sends out about 15 messages, all at 03:57, and then is fine the rest of the time-

Apr 4 03:57:25 vmunix: msgcnt 9 vxfs: mesg 008: vx_direrr: vx_dirscan_2
- /var file system inode 28171 block 4520501 error 6

I ran this command-
find /var -inum 2871 -print

And the output was
/var/dt

/var/dt is an empty directory dated March 19th at 15:29

Is there a way I can fix this?

I've done some searching on mesg 008 and error 6 and found some similar messages but none quite like the one I am seeing.

Thanks!

[Patrick Wallek]

This link comes from Veritas, the folks that write the VxFS file system, and seems to match your error exactly. 

https://support.symantec.com/en_US/article.TECH1539.html

Even before reading this, I would recommend a FULL fsck of that file system.

# fsck -y -o full /dev/vg00/rlvol8

(or whatever the lvol of /var is on your system). 

It may be best to boot into single-user mode to do the fsck since you really don't want the file system mounted while doing that.

[JDR45]

Reading the link now, thanks!

Here's how /var looks on a bdf-

/dev/vg00/lvol7    36896768 1940896 34683616    5% /var

And once I'm inside /var, the empty "dt" directory looks like this-

drwxr-xr-x   4 root       sys           8192 Mar 19 15:29 dt

As root I used vi to create and place a small test file inside /var/dt, and sure enough got the message "no such device or address" and it would not let my place the test file inside /var/dt.  The rest of /var looks good.

Will look into doing the fsck soon.  

[Bill Hassell]

This is an early warning that the disk may be going bad.

Do you have your boot disk mirrorred?
Are you using Ignite to create a recovery image?

[JDR45]

So the full file system check ran last night in single user mode.  It didn't find anything, but after booting back up the same message was waiting for us-

Apr 5 02:05:13 vmunix: msgcnt 32 vxfs: mesg 008: vx_direrr: vx_dirscan_2
- /var file system inode 28171 block 4520501 error 6

About one second before those messages appeared, I saw this in the syslog-

Apr 5 02:04:56  automount[742]: Automount abort!!!
Apr 5 02:04:56  automount[742]: Possible cause:
Apr 5 02:04:56 automount[742]: 1)No map provided
Apr 5 02:04:56 automount[742]: 2)Using the same map as the running aut
omount
Apr 5 02:04:56  automount[742]: 3)The maps contain invalid mount points
Apr 5 02:04:57  inetd[761]: Reading configuration

Basically, the server boots, puts out the automount messages, then 1 second later the "vx_direrr: vx_dirscan_2" messages come along, then it boots up, the users hop on the server, and everything runs fine and the syslog stays clear until the next reboot..

I wonder if something was configured wrong with /var/dt vs there being a hardware or filesystem problem?

[JDR45]

And for Bill Hassell- yes, this is a mirrored pair of HDs and we do frequent backups.

[JDR45]

Here's what I was able to pull from the console logs related to the fsck that ran overnight-

/dev/vg00/lvol7 :

vxfs fsck: sanity check: /dev/vg00/lvol7 needs checking

log replay in progress

file system is not clean, full fsck required

pass0 - checking structural files

pass1 - checking inode sanity and blocks

pass2 - checking directory linkage

pass3 - checking reference counts

pass4 - checking resource maps

OK to clear log? (ynq)y

set state to CLEAN? (ynq)y

vxfs mount: /dev/vg00/lvol7 is corrupted. needs checking

mount: /dev/vg00/lvol1 is already mounted on /stand

# bdf

Filesystem          kbytes    used   avail %used Mounted on

/dev/vg00/lvol3     983040  138952  791339   15% /

/dev/vg00/lvol1     961051   49962  814983    6% /stand

/dev/vg00/lvol8    206503936 73208248 132256200   36% /JR

/dev/vg00/lvol6    7471104 1139336 5936139   16% /usr

/dev/vg00/lvol5    2097152    2007 1964260    0% /tmp

/dev/vg00/lvol4    5177344 1446421 3497758   29% /opt

# umount /JR

# fsck -y -o full /dev/vg00/rlvol7

fsck: /etc/default/fs is used for determining the file system type

log replay in progress

pass0 - checking structural files

pass1 - checking inode sanity and blocks

pass2 - checking directory linkage

pass3 - checking reference counts

pass4 - checking resource maps

OK to clear log? (ynq)y

# fsck -y -m -o full /dev/vg00/rlvol7

vxfs fsck: sanity check: /dev/vg00/rlvol7 OK

/dev/vg00/lvol7 :

vxfs fsck: sanity check: /dev/vg00/lvol7 OK

[Bill Hassell · Accepted solution]

The HP-UX errno value for 6 means:

from /usr/include/sys/errno.h:
      errno 6 = ENXIO /* No such device or address      */

      [ENXIO]        No such device or address. I/O on a special file refers
                     to a subdevice that does not exist, or is beyond the
                     limits of the device.  It can also occur when, for
                     example, a tape drive is not on line or no disk pack is
                     loaded on a drive.

So this matches what you saw when creating a test file.
It is possibly a bookeeping error in the directory structure, but to verify that there are no bad sectors in /var, do this to read every sector:

# dd if=/dev/vg00/rlvol7 of=/dev/null bs=1024k

The dd comand will take several minutes as it appears that lvol7 is 37 GB in size.
I have attached a copy of the script bdfmegs in case you don't have it. It formats bdf for today's large filesystems:

File-System Mbytes Used Avail %Used Mounted on
/dev/vg00/lvol1 938 48 795 6% /stand
/dev/vg00/lvol3 960 135 772 15% /
/dev/vg00/lvol4 5056 1412 3415 29% /opt
/dev/vg00/lvol5 2048 1 1918 0% /tmp
/dev/vg00/lvol6 7296 1112 5797 16% /usr
/dev/vg00/lvol8 196.9g 69.8g 126.1g 36% /JR

 NOTE: The bdfmegs file has a Windoze extension .jpg so it can be attached to this post. It is a simple text file, so remove the .jpg and copy it to /usr/contrib/bin.

 I would rename the /var/dt directory to /var/dt_BAD and recreate /var/dt.
The dt directory is just temp storage for Xwindows.

[JDR45]

Thanks for such a detailed answer!

# dd if=/dev/vg00/rlvol7 of=/dev/null bs=1024k

I'll give this a try afer hours, when there aren't so many users on the system.  

[JDR45]

So I ran this command early this morning before the users got on-

# dd if=/dev/vg00/rlvol7 of=/dev/null bs=1024k

And after about 6 minutes the result came back-

36032+0 records in

36032+0 records out

No errors, and the in / out number matched, so that's good news.

Also tried this to remove /var/dt and recreate it, but it would not let me remove /var/dt:

 # cd /var

# ls -ld dt

drwxr-xr-x   4 root       sys           8192 Mar 19 15:29 dt

# rmdir dt

rmdir: dt: Directory not empty

# cd dt

# ls -lar

total 0

# ls -la

total 0

# ll

total 0

# cd /var

# ls -laR dt

total 0

#

Looks like /var/dt is an odd little gremlin in the system.  

[Bill Hassell]

You wrote:
>>      # rmdir dt
>>      rmdir: dt: Directory not empty
>>      # cd dt
>>      # ls -lar
>>       total 0

Try: rm -rf /var/dt
You could also find the inode for /var/dt and remove the inode like this:

# ll -i /var/dt
(first column is the inode number)
# cd /var
# find . -inum ### -exec rm {} \;

[JDR45]

Well between the fsck and the dd command, it is fixed.  /var/dt is fine and the syslog is clear of messages, even after a reboot or two.  So thanks for both recomendations.   

Is there a way I can select two different posts as "accept as solution?"

[Dennis Handly]

> Is there a way I can select two different posts as "accept as solution"?

No, only the one.  But you can give kudos to each.