topic Re: vx_nospace - /dev/root file system full (1 block extent) in Operating System - HP-UX

vx_nospace - /dev/root file system full (1 block extent)

BAUKnight — Wed, 30 Apr 2008 09:16:25 GMT

dears,
i have a problem in my file system, i am recieving the following message as part of the dmesg command

msgcnt 1 vxfs: mesg 001: vx_nospace - /dev/root file system full (1 block extent) msgcnt 3 vxfs: mesg 001: vx_nospace - /dev/root file system full (1 block extent)

I tried to delete any not needed files from the / directory, but it is still full, here is the bdf output

/ >bdf /
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 360448 360144 304 100% /

i am not sure what happened, but all the files i could delete still does not free enough space
i checked my log files, and daily maintenance mails, i found the following

in March 30, here is the bdf output
/dev/vg00/lvol3 360448 172096 186920 48% /

in March 31, here is the bdf output
/dev/vg00/lvol3 360448 360448 0 100% /

and since that time the system is saying
msgcnt 1 vxfs: mesg 001: vx_nospace - /dev/root file system full (1 block extent)

this is the parts of the syslog regarding the problem, with some event before the error and some events after it

Mar 30 07:56:41 prodx su: + ta root-aslam
Mar 30 08:04:16 prodx vmunix: msgcnt 1 vxfs: mesg 001: vx_nospace - /dev/root file system full (1 block extent)
Mar 30 08:19:04 prodx ftpd[24585]: FTP LOGIN FROM 10.100.1.8 [10.100.1.8], oracle
Mar 30 08:19:09 prodx ftpd[24587]: FTP LOGIN FROM 10.100.1.8 [10.100.1.8], oracle
Mar 30 08:19:10 prodx ftpd[24587]: FTP session closed
Mar 30 08:19:21 prodx ftpd[24585]: FTP session closed

seems there is something happened during the March 30,, but i could not find anything in my server poiting to this.

any help please

Re: vx_nospace - /dev/root file system full (1 block extent)

MarkSyder — Wed, 30 Apr 2008 09:46:21 GMT

cd /
ll -tr

This will show you the most recently updated files. Look for:

1. files that have grown very large, and

2. files that would be better in a dedicated lvol/filesystem.

Check your /dev directory for ordinary files. This problem is often caused by someone mistyping a tape drive name and creating a large text file in /dev. There should not be any ordinary files in /dev.

Mark Syder (like the drink but spelt different)

Re: vx_nospace - /dev/root file system full (1 block extent)

Rasheed Tamton — Wed, 30 Apr 2008 09:47:03 GMT

Hi,

Assume the problem happened after March 30 0700 am

Create a file with a date stamp of March 30 7am

cd /tmp
touch -mt 200803300700 aslam

then find the files created after that, sorted by size

find / -xdev -type f -newer /tmp/aslam -exec ls -l {} +|sort -rnk5|head

and remove the files or move the ones to another file system where you have enough space.

Regards,
Rasheed Tamton.

Re: vx_nospace - /dev/root file system full (1 block extent)

Matti_Kurkela — Wed, 30 Apr 2008 09:51:33 GMT

The standard way to find big files/directories:

du -kx / | sort -rn | more

The du option -x will restrict the command to display the root filesystem only. The sort command will list the numerically biggest entries first.

Examine the 10 or so largest directories: as the increase is so huge (>50%), this will probably allow you to find it.

If you have the "lsof" command installed, use "lsof +L1" to find any files that are deleted from directory listings but still being used by some process. If you find files like that, stopping the process that is hanging onto the files will cause the OS to automatically free the disk space.

Someone or something su'd to user "aslam" on Mar 30 07:56:41. If this was a human, he/she might know something about the cause. If this was a script (cron job or something?), you might wish to find out what the script was supposed to do and whether or not it creates a log about whatever it is doing.

MK

Re: vx_nospace - /dev/root file system full (1 block extent)

BAUKnight — Wed, 30 Apr 2008 10:13:12 GMT

i executed the commands RASHEED suggested

find / -xdev -type f -newer /tmp/aslam -exec ls -l {} +|sort -rnk5|head

this is the output

/ >find / -xdev -type f -newer /tmp/aslam -exec ls -l {} +|sort -rnk5|head
-rw------- 1 root root 192864256 Mar 30 08:04 /devahmed0m
-rw-r--r-- 1 root root 11480 Apr 30 13:45 /etc/utmpx
-rw-r--r-- 1 root root 1680 Apr 30 13:45 /etc/utmp
-r--r--r-- 1 root root 972 Apr 29 10:31 /etc/passwd
-rw-r--r-- 1 root root 0 Apr 30 08:25 /etc/dhcpdb
-rw------- 1 root sys 0 Apr 30 10:32 /.sh_history

seems the first file in the list is the problem, but i could not delete it

/ >rm /devahmed0m
rm: /devahmed0m non-existent

how can we remove it please,,,
thx for all your responses, but this was the best solution

Re: vx_nospace - /dev/root file system full (1 block extent)

BAUKnight — Wed, 30 Apr 2008 10:59:29 GMT

dears,,
this may be helpgul for you,,

i executed this command

/ >find / -xdev -type f -newer /tmp/aslam
/etc/passwd
/etc/utmp
/etc/utmpx
/etc/dhcpdb
/devahmed0m
/.sh_history

the main file is not listed here,, strange ??!!

Re: vx_nospace - /dev/root file system full (1 block extent)

Rasheed Tamton — Wed, 30 Apr 2008 11:04:32 GMT

The culprit is here:

-rw------- 1 root root 192864256 Mar 30 08:04 /devahmed0m

It is very huge and this should not be there in the root.

You can use file command on that
file /devahmed0m

Either you can delete it or move that file temporarily to another file system where you have enough space for later checking.

But you have to immediately rm or mv that.

Regards.

Re: vx_nospace - /dev/root file system full (1 block extent)

BAUKnight — Wed, 30 Apr 2008 11:07:26 GMT

/ >file /devahmed0m
/devahmed0m: cannot open
/ >rm /devahmed0m
rm: /devahmed0m non-existent

what do u think !!

Re: vx_nospace - /dev/root file system full (1 block extent)

Rasheed Tamton — Wed, 30 Apr 2008 11:13:47 GMT

Please check whether you have lsof installed. Then do what the above thread says.

what is now bdf / says.

Re: vx_nospace - /dev/root file system full (1 block extent)

BAUKnight — Wed, 30 Apr 2008 11:19:55 GMT

/ >bdf /
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 360448 360144 304 100% /
/ >swlist | grep lsof
/ >

Re: vx_nospace - /dev/root file system full (1 block extent)

Rasheed Tamton — Wed, 30 Apr 2008 11:25:02 GMT

Try with fuser

fuser /devahmed0m

if you can get process id kill it

or

do

fuser -ku /devahmed0m

Re: vx_nospace - /dev/root file system full (1 block extent)

MarkSyder — Wed, 30 Apr 2008 11:26:58 GMT

I think you have invisible characters in the file name. Try:

rm -i /dev*

This will list all files that start with dev one at a time and ask you to confirm that you want to delete them. Select yes only for the file that is causing the problem. Make sure you include the -i, otherwise you may delete something you want.

Mark

Re: vx_nospace - /dev/root file system full (1 block extent)

BAUKnight — Wed, 30 Apr 2008 11:27:09 GMT

i tried this before, this is the output

/ >fuser /devahmed0m
/devahmed0m: stat: No such file or directory
fuser: could not find or access file /devahmed0m.

/ >fuser -ku /devahmed0m
/devahmed0m: stat: No such file or directory
fuser: could not find or access file /devahmed0m.

any other ideas people.

Re: vx_nospace - /dev/root file system full (1 block extent)

BAUKnight — Wed, 30 Apr 2008 11:31:01 GMT

/ >rm -i /dev*
rm: /dev directory

/ >

sorry,, seems this did not work too.

Re: vx_nospace - /dev/root file system full (1 block extent)

Rasheed Tamton — Wed, 30 Apr 2008 11:31:27 GMT

download lsof from the below site for your HP-UX version, install and use it

http://hpux.cs.utah.edu/hppd/hpux/Sysadmin/lsof-4.78/

Re: vx_nospace - /dev/root file system full (1 block extent)

Rasheed Tamton — Wed, 30 Apr 2008 11:35:59 GMT

just a test

did you try

ps -ef|grep /devahmed

otherwise, go with lsof

the last option is a reboot, if lsof does not work.

Re: vx_nospace - /dev/root file system full (1 block extent)

BAUKnight — Wed, 30 Apr 2008 11:47:50 GMT

i could not install the application,, it gave an error saying there is no empty space in the (/),, which is out main problem

attached is the error from swinstall command

Re: vx_nospace - /dev/root file system full (1 block extent)

BAUKnight — Wed, 30 Apr 2008 12:04:01 GMT

/ >ps -ef|grep /devahmed
root 19916 13030 1 15:59:57 pts/ta 0:00 grep /devahmed
/ >

Re: vx_nospace - /dev/root file system full (1 block extent)

MarkSyder — Wed, 30 Apr 2008 12:07:41 GMT

If my earlier suggestion (rm -i /dev*) did not work, perhaps the invisible characters are at the beginning. Try:

rm -i *med

Mark

Re: vx_nospace - /dev/root file system full (1 block extent)

BAUKnight — Wed, 30 Apr 2008 12:09:20 GMT

/ >rm -i *med
rm: *med non-existent
/ >