vx_nospace - /dev/root file system full (1 block extent)

[BAUKnight]

dears,
i have a problem in my file system, i am recieving the following message as part of the dmesg command

msgcnt 1 vxfs: mesg 001: vx_nospace - /dev/root file system full (1 block extent) msgcnt 3 vxfs: mesg 001: vx_nospace - /dev/root file system full (1 block extent)


I tried to delete any not needed files from the / directory, but it is still full, here is the bdf output


/ >bdf /
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 360448 360144 304 100% /


i am not sure what happened, but all the files i could delete still does not free enough space
i checked my log files, and daily maintenance mails, i found the following

in March 30, here is the bdf output
/dev/vg00/lvol3 360448 172096 186920 48% /

in March 31, here is the bdf output
/dev/vg00/lvol3 360448 360448 0 100% /

and since that time the system is saying
msgcnt 1 vxfs: mesg 001: vx_nospace - /dev/root file system full (1 block extent)


this is the parts of the syslog regarding the problem, with some event before the error and some events after it

Mar 30 07:56:41 prodx su: + ta root-aslam
Mar 30 08:04:16 prodx vmunix: msgcnt 1 vxfs: mesg 001: vx_nospace - /dev/root file system full (1 block extent)
Mar 30 08:19:04 prodx ftpd[24585]: FTP LOGIN FROM 10.100.1.8 [10.100.1.8], oracle
Mar 30 08:19:09 prodx ftpd[24587]: FTP LOGIN FROM 10.100.1.8 [10.100.1.8], oracle
Mar 30 08:19:10 prodx ftpd[24587]: FTP session closed
Mar 30 08:19:21 prodx ftpd[24585]: FTP session closed



seems there is something happened during the March 30,, but i could not find anything in my server poiting to this.

any help please

[MarkSyder]

cd /
ll -tr

This will show you the most recently updated files. Look for:

1. files that have grown very large, and

2. files that would be better in a dedicated lvol/filesystem.

Check your /dev directory for ordinary files. This problem is often caused by someone mistyping a tape drive name and creating a large text file in /dev. There should not be any ordinary files in /dev.

Mark Syder (like the drink but spelt different)

[Rasheed Tamton]

Hi,

Assume the problem happened after March 30 0700 am

Create a file with a date stamp of March 30 7am

cd /tmp
touch -mt 200803300700 aslam

then find the files created after that, sorted by size

find / -xdev -type f -newer /tmp/aslam -exec ls -l {} +|sort -rnk5|head

and remove the files or move the ones to another file system where you have enough space.

Regards,
Rasheed Tamton.

[Matti_Kurkela]

The standard way to find big files/directories:

du -kx / | sort -rn | more

The du option -x will restrict the command to display the root filesystem only. The sort command will list the numerically biggest entries first.

Examine the 10 or so largest directories: as the increase is so huge (>50%), this will probably allow you to find it.

If you have the "lsof" command installed, use "lsof +L1" to find any files that are deleted from directory listings but still being used by some process. If you find files like that, stopping the process that is hanging onto the files will cause the OS to automatically free the disk space.

Someone or something su'd to user "aslam" on Mar 30 07:56:41. If this was a human, he/she might know something about the cause. If this was a script (cron job or something?), you might wish to find out what the script was supposed to do and whether or not it creates a log about whatever it is doing.

MK

[BAUKnight]

i executed the commands RASHEED suggested

find / -xdev -type f -newer /tmp/aslam -exec ls -l {} +|sort -rnk5|head

this is the output

/ >find / -xdev -type f -newer /tmp/aslam -exec ls -l {} +|sort -rnk5|head
-rw------- 1 root root 192864256 Mar 30 08:04 /devahmed0m
-rw-r--r-- 1 root root 11480 Apr 30 13:45 /etc/utmpx
-rw-r--r-- 1 root root 1680 Apr 30 13:45 /etc/utmp
-r--r--r-- 1 root root 972 Apr 29 10:31 /etc/passwd
-rw-r--r-- 1 root root 0 Apr 30 08:25 /etc/dhcpdb
-rw------- 1 root sys 0 Apr 30 10:32 /.sh_history

seems the first file in the list is the problem, but i could not delete it

/ >rm /devahmed0m
rm: /devahmed0m non-existent

how can we remove it please,,,
thx for all your responses, but this was the best solution

[BAUKnight]

dears,,
this may be helpgul for you,,

i executed this command

/ >find / -xdev -type f -newer /tmp/aslam
/etc/passwd
/etc/utmp
/etc/utmpx
/etc/dhcpdb
/devahmed0m
/.sh_history

the main file is not listed here,, strange ??!!

[Rasheed Tamton]

The culprit is here:

-rw------- 1 root root 192864256 Mar 30 08:04 /devahmed0m

It is very huge and this should not be there in the root.

You can use file command on that
file /devahmed0m

Either you can delete it or move that file temporarily to another file system where you have enough space for later checking.

But you have to immediately rm or mv that.

Regards.

[BAUKnight]

/ >file /devahmed0m
/devahmed0m: cannot open
/ >rm /devahmed0m
rm: /devahmed0m non-existent

what do u think !!

[Rasheed Tamton]

Please check whether you have lsof installed. Then do what the above thread says.

what is now bdf / says.

[BAUKnight]

/ >bdf /
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 360448 360144 304 100% /
/ >swlist | grep lsof
/ >

[Rasheed Tamton]

Try with fuser

fuser /devahmed0m

if you can get process id kill it

or

do

fuser -ku /devahmed0m

[MarkSyder]

I think you have invisible characters in the file name. Try:

rm -i /dev*

This will list all files that start with dev one at a time and ask you to confirm that you want to delete them. Select yes only for the file that is causing the problem. Make sure you include the -i, otherwise you may delete something you want.

Mark

[BAUKnight]

i tried this before, this is the output

/ >fuser /devahmed0m
/devahmed0m: stat: No such file or directory
fuser: could not find or access file /devahmed0m.

/ >fuser -ku /devahmed0m
/devahmed0m: stat: No such file or directory
fuser: could not find or access file /devahmed0m.

any other ideas people.

[BAUKnight]

/ >rm -i /dev*
rm: /dev directory

/ >

sorry,, seems this did not work too.

[Rasheed Tamton]

download lsof from the below site for your HP-UX version, install and use it

http://hpux.cs.utah.edu/hppd/hpux/Sysadmin/lsof-4.78/

[Rasheed Tamton]

just a test

did you try

ps -ef|grep /devahmed

otherwise, go with lsof

the last option is a reboot, if lsof does not work.

[BAUKnight]

i could not install the application,, it gave an error saying there is no empty space in the (/),, which is out main problem

attached is the error from swinstall command

[BAUKnight]

/ >ps -ef|grep /devahmed
root 19916 13030 1 15:59:57 pts/ta 0:00 grep /devahmed
/ >

[MarkSyder]

If my earlier suggestion (rm -i /dev*) did not work, perhaps the invisible characters are at the beginning. Try:

rm -i *med

Mark

[BAUKnight]

/ >rm -i *med
rm: *med non-existent
/ >

[MarkSyder · Accepted solution]

ls -b will show where the non-printing characters are. Adjust the position of the * to make sure it covers them.

Mark

[BAUKnight]

thank you alot mark, here what i did now

i added the -b to the original command

/ >find / -xdev -type f -newer /tmp/aslam -exec ls -l -b {} +|sort -rnk5|head
-rw------- 1 root root 192864256 Mar 30 08:04 /dev/rmt/0m\010\010\010\010\010\010\010ahmed
-rw-r--r-- 1 root root 11480 Apr 30 13:45 /etc/utmpx
-rw-r--r-- 1 root root 1680 Apr 30 13:45 /etc/utmp
-r--r--r-- 1 root root 972 Apr 29 10:31 /etc/passwd
-rw-r--r-- 1 root root 0 Apr 30 08:25 /etc/dhcpdb
-rw------- 1 root sys 0 Apr 30 10:32 /.sh_history

so i went to /dev/rmt then removed the file using the *ahm* combination

akayyali@prodx/dev/rmt >rm *ahm*
akayyali@prodx/dev/rmt >bdf /
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 360448 171792 187232 48% /


and as u can see now the file system has morer than enpugh space,
thxnks alot for all you help people.
marks assigned
regards