nospace - /dev/root file system full (1 block extent)

[Bob Wallner]

I am getting this error message in the Syslog.

I can log in fine to the system

Everything appears to be ok.

But users can't save files.
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 540672 300912 237936 56% /
/dev/vg00/lvol1 311296 116672 193144 38% /stand
/dev/vg00/lvol8 7995392 2188136 5761936 28% /var
/dev/vg00/lvol7 6488064 3674648 2791464 57% /usr
/dev/vg00/lvol4 212992 40872 172120 19% /tmp
/dev/vg00/sysdata 5128192 2133847 2807210 43% /sysdata
/dev/vg00/lvol6 8192000 4178304 3982376 51% /opt
mscnas01.matsci.com:/vol/volapps/apps
157286400 4083208 153203192 3% /apps
msc00:/appl 8388608 6976464 1323944 84% /appl
mscnas01:/vol/vol0 419430400 290931688 128498712 69% /nas_root
mscnas01:/vol/vol0/home
419430400 290931688 128498712 69% /home
/scratch 540672 300912 237936 56% /soft/msc05

Syslog shows:

Jun 4 17:29:22 msc05 vmunix: vxfs: NOTICE: msgcnt 1 mesg 001: V-2-1: vx_nospace - /dev/root file system full (1 block extent)
Jun 4 17:36:27 msc05 vmunix: vxfs: NOTICE: msgcnt 3 mesg 001: V-2-1: vx_nospace - /dev/root file system full (1 block extent)
Jun 4 17:45:29 msc05 vmunix: vxfs: NOTICE: msgcnt 5 mesg 001: V-2-1: vx_nospace - /dev/root file system full (1 block extent)
Jun 4 17:56:03 msc05 vmunix: vxfs: NOTICE: msgcnt 7 mesg 001: V-2-1: vx_nospace - /dev/root file system full (1 block extent)
Jun 4 18:10:25 msc05 vmunix: vxfs: NOTICE: msgcnt 9 mesg 001: V-2-1: vx_nospace - /dev/root file system full (1 block extent)
Jun 5 08:51:50 msc05 vmunix: vxfs: NOTICE: msgcnt 11 mesg 001: V-2-1: vx_nospace - /dev/root file system full (1 block extent)

Du-k shows nothing of any value..

I don't see anything out of the ordinary in /dev

Any suggestions?

[Patrick Wallek]

"/dev/root" is actually your '/' file system. It appears that there is an intermittent problem. My guess would be some process that creates a temporary file, does stuff, then removes the file. Unless you can time it just right, it may be a challenge to find the culprit.

You also say: "But users can't save files." Where can't they save files? Do they get an error?

[Bob Wallner]

They save files on /home
ANd from what I see, they can save there.

As they run verious Nastran jobs it bombs out giving the error, not enough disk space

[A. Clay Stephenson]

After your second posting, my psychic, Miss Cleo, has this suggestion: "Look for a symbolic link that points to a directory in the / filesystem." She thinks that Nastran is creating temporary files using the symbolic link or the application tempdir is set to a directory within /. Most applications have an environment variable that will control the location of temporary and data files. That is whai I would look for.

[Bob Wallner]

I should add, that this one server is part of a server farm. there are 2 oter servers connected to the head node. All the other servers are running fine.

All the servers run the same application. all pointing to the head node.

THis server is the only one that having a problem.

I should add, that the code has not changed and the server has been running fine.

That said, I will search for symbolic links.

[A. Clay Stephenson]

Miss Cleo also said to ask you if all your filesystems are there? Her vision is a bit hazy but she said something about a filesystem is no longer getting mounted and that these files are now being written to the mountpoint directory which is now under /.

[Procnus]

I found this file system a bit strange:
/scratch 540672 300912 237936 56% /soft/msc05

It is the local root file system directory

[Bob Wallner]

Turns out. that one of the Hard Drives in the array took a dump. This was just a
syptom of the problem.

Thanks for your help!