topic Re: no space left on the device in Operating System - Linux

no space left on the device

Piotr Kirklewski — Tue, 01 Dec 2009 16:40:51 GMT

Hi there
I'm using one of my machines for backup purposes only. There is one directory that holds the backups, scripts and logs. I called it /files. This directory is only 36GB in size so I don't get where is the missing 22GB ?

backup:/files# du -sh /files/
36G /files/

df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 3842048 78875 3763173 3% /
tmpfs 64537 6 64531 1% /lib/init/rw
udev 64537 1069 63468 2% /dev
tmpfs 64537 1 64536 1% /dev/shm
//10.15.3.1/data 0 0 0 - /mnt/PDM
//10.15.3.6/backup 0 0 0 - /mnt/bac001bri
//GDSBS/BACKUP/ 0 0 0 - /mnt/BACKUPgdsbs
//10.15.3.5/os43 rt 0 0 0 - /mnt/QUBE

df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 58G 58G 0 100% /
tmpfs 253M 0 253M 0% /lib/init/rw
udev 10M 568K 9.5M 6% /dev
tmpfs 253M 0 253M 0% /dev/shm
//10.15.3.1/data 256G 62G 195G 25% /mnt/PDM
//10.15.3.6/backup 2.8T 1.2T 1.6T 45% /mnt/bac001bri
//GDSBS/BACKUP/ 137G 35G 103G 26% /mnt/BACKUPgdsbs
//10.15.3.5/os43 rt 256G 16G 240G 7% /mnt/QUBE

backup:/# du -h |grep '^[0-9]*G'
...
35G /files/Projects
36G /files

Re: no space left on the device

Ivan Ferreira — Tue, 01 Dec 2009 18:21:41 GMT

I don't know where do you see 22GB missing.

According to your output, /files is under the / filesystem (sda1) and it's 100% used.

Re: no space left on the device

Piotr Kirklewski — Tue, 01 Dec 2009 18:45:33 GMT

sda1 = /dev/sda1 58G 58G 0 100% /
/files = 36G /files

So 58GB - 36GB = 22GB

backup:/# ls |sort
bin
boot
cdrom
dev
etc
files
home
initrd.img
lib
lost+found
media
mnt
opt
proc
root
save
sbin
selinux
srv
sys
tmp
usr
var
vmlinuz

du -sh /bin/
4.1M /bin/

backup:/# du -sh /boot
8.6M /boot

backup:/# du -sh /dev/
568K /dev/

backup:/# du -sh /etc/
5.0M /etc/

backup:/# du -sh /files
36G /files

backup:/# du -sh /home
36K /home

backup:/# du -sh /initrd.img
0 /initrd.img

backup:/# du -sh /lib/
68M /lib/

backup:/# du -sh /lost+found/
16K /lost+found/

backup:/# du -sh /media/
12K /media/

backup:/# du -sh /opt/
4.0K /opt/

backup:/# du -sh /proc/
du: cannot access `/proc/13909/task/13909/fd/4': No such file or directory
du: cannot access `/proc/13909/task/13909/fdinfo/4': No such file or directory
du: cannot access `/proc/13909/fd/4': No such file or directory
du: cannot access `/proc/13909/fdinfo/4': No such file or directory
0 /proc/

backup:/# du -sh /root/
76K /root/

backup:/# du -sh /save/
8.0K /save/

backup:/# du -sh /sbin/
3.6M /sbin/

backup:/# du -sh /selinux/
4.0K /selinux/

backup:/# du -sh /srv/
4.0K /srv/

backup:/# du -sh /sys/
0 /sys/

backup:/# du -sh /tmp/
20K /tmp/

backup:/# du -sh /usr/
549M /usr/

backup:/# du -sh /var/
213M /var/

backup:/# du -sh /vmlinuz
0 /vmlinuz

Re: no space left on the device

Ivan Ferreira — Tue, 01 Dec 2009 19:18:23 GMT

Maybe your missing space is under some of your directories acting as mount point. You should umount all filesystems and check to see if there is any files under the mounted filesystems. Remove the files from the directories acting as mount point.

Re: no space left on the device

macosta — Tue, 01 Dec 2009 23:23:45 GMT

Also look for processes that are still holding open file descriptors on a file that has since been deleted.

Re: no space left on the device

Piotr Kirklewski — Wed, 02 Dec 2009 00:25:38 GMT

How ?

Re: no space left on the device

Randy Jones_3 — Wed, 02 Dec 2009 02:26:53 GMT

Do a "touch /forcefsck" then restart the system. fsck should run at startup and that should recover any space that was somehow only partially deallocated.

Re: no space left on the device

macosta — Wed, 02 Dec 2009 02:50:34 GMT

I'm not certain a fsck is necessary, but shouldn't hurt anything.

The issue, if I recall correctly, is that a file can be held open, but deleted underneath the open() syscall. While the link count is dropped to 0, the links to the blocks haven't been cleared in the filesystem yet. This explains the discrepancy between 'df' which reads the filesystem metadata to give you a count of blocks, while 'du' actually crawls the filesystem.

Re: no space left on the device

Ivan Ferreira — Wed, 02 Dec 2009 05:15:07 GMT

It's true. You can use the lsof command to identify files deleted that was not released. I have found these kind of files on the /tmp file system sometimes.

Re: no space left on the device

Goran Koruga — Wed, 02 Dec 2009 08:02:18 GMT

Hello.

ls -l /proc/*/fd | grep '(deleted)'

I leave it up to you to expand on this (i.e. pipe to awk/sed/whatever and print filename only, then pipe to "xargs du ...").

Regards,
Goran

Re: no space left on the device

Piotr Kirklewski — Wed, 02 Dec 2009 13:56:37 GMT

Thanks everyone.