no space left on the device

[Piotr Kirklewski]

Hi there
I'm using one of my machines for backup purposes only. There is one directory that holds the backups, scripts and logs. I called it /files. This directory is only 36GB in size so I don't get where is the missing 22GB ?

backup:/files# du -sh /files/
36G /files/


df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 3842048 78875 3763173 3% /
tmpfs 64537 6 64531 1% /lib/init/rw
udev 64537 1069 63468 2% /dev
tmpfs 64537 1 64536 1% /dev/shm
//10.15.3.1/data 0 0 0 - /mnt/PDM
//10.15.3.6/backup 0 0 0 - /mnt/bac001bri
//GDSBS/BACKUP/ 0 0 0 - /mnt/BACKUPgdsbs
//10.15.3.5/os43 rt 0 0 0 - /mnt/QUBE




df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 58G 58G 0 100% /
tmpfs 253M 0 253M 0% /lib/init/rw
udev 10M 568K 9.5M 6% /dev
tmpfs 253M 0 253M 0% /dev/shm
//10.15.3.1/data 256G 62G 195G 25% /mnt/PDM
//10.15.3.6/backup 2.8T 1.2T 1.6T 45% /mnt/bac001bri
//GDSBS/BACKUP/ 137G 35G 103G 26% /mnt/BACKUPgdsbs
//10.15.3.5/os43 rt 256G 16G 240G 7% /mnt/QUBE

backup:/# du -h |grep '^[0-9]*G'
...
35G /files/Projects
36G /files

[Ivan Ferreira]

I don't know where do you see 22GB missing.

According to your output, /files is under the / filesystem (sda1) and it's 100% used.

[Piotr Kirklewski]

sda1 = /dev/sda1 58G 58G 0 100% /
/files = 36G /files

So 58GB - 36GB = 22GB

backup:/# ls |sort
bin
boot
cdrom
dev
etc
files
home
initrd.img
lib
lost+found
media
mnt
opt
proc
root
save
sbin
selinux
srv
sys
tmp
usr
var
vmlinuz




du -sh /bin/
4.1M /bin/

backup:/# du -sh /boot
8.6M /boot

backup:/# du -sh /dev/
568K /dev/

backup:/# du -sh /etc/
5.0M /etc/

backup:/# du -sh /files
36G /files

backup:/# du -sh /home
36K /home

backup:/# du -sh /initrd.img
0 /initrd.img

backup:/# du -sh /lib/
68M /lib/

backup:/# du -sh /lost+found/
16K /lost+found/

backup:/# du -sh /media/
12K /media/

backup:/# du -sh /opt/
4.0K /opt/

backup:/# du -sh /proc/
du: cannot access `/proc/13909/task/13909/fd/4': No such file or directory
du: cannot access `/proc/13909/task/13909/fdinfo/4': No such file or directory
du: cannot access `/proc/13909/fd/4': No such file or directory
du: cannot access `/proc/13909/fdinfo/4': No such file or directory
0 /proc/

backup:/# du -sh /root/
76K /root/

backup:/# du -sh /save/
8.0K /save/

backup:/# du -sh /sbin/
3.6M /sbin/

backup:/# du -sh /selinux/
4.0K /selinux/

backup:/# du -sh /srv/
4.0K /srv/

backup:/# du -sh /sys/
0 /sys/

backup:/# du -sh /tmp/
20K /tmp/

backup:/# du -sh /usr/
549M /usr/

backup:/# du -sh /var/
213M /var/

backup:/# du -sh /vmlinuz
0 /vmlinuz

[Ivan Ferreira]

Maybe your missing space is under some of your directories acting as mount point. You should umount all filesystems and check to see if there is any files under the mounted filesystems. Remove the files from the directories acting as mount point.

[macosta]

Also look for processes that are still holding open file descriptors on a file that has since been deleted.

[Piotr Kirklewski]

How ?

[Randy Jones_3]

Do a "touch /forcefsck" then restart the system. fsck should run at startup and that should recover any space that was somehow only partially deallocated.

[macosta]

I'm not certain a fsck is necessary, but shouldn't hurt anything.

The issue, if I recall correctly, is that a file can be held open, but deleted underneath the open() syscall. While the link count is dropped to 0, the links to the blocks haven't been cleared in the filesystem yet. This explains the discrepancy between 'df' which reads the filesystem metadata to give you a count of blocks, while 'du' actually crawls the filesystem.

[Ivan Ferreira]

It's true. You can use the lsof command to identify files deleted that was not released. I have found these kind of files on the /tmp file system sometimes.

[Goran Koruga]

Hello.

ls -l /proc/*/fd | grep '(deleted)'

I leave it up to you to expand on this (i.e. pipe to awk/sed/whatever and print filename only, then pipe to "xargs du ...").

Regards,
Goran