topic Re: No space left on device in Operating System - Linux

No space left on device

Md. Minhaz Khan — Sun, 16 May 2010 05:37:53 GMT

Dear All,

We are getting â No space left on device(28)â error during "Grid control Agent" software installation in one of our Linux Server under
"/u01" mount point.

From "df -h /u01" command output i have found
only "14%" space is in use & from "df -i /u01" command output i have found "80%" inode has already in use.

So what is the cause of using 80% inode as only 14% space used & how can i increase inode number of "/u01" mount point(/dev/vglmdb_app/lv_u01)

I have attached "df" & "vgdisplay" command output with this mail. Please help me to resolve this problem.

Thanks
Minhaz

Re: No space left on device

Dennis Handly — Sun, 16 May 2010 08:55:57 GMT

What type of filesystem is /u01?
On HP-UX, hfs filesystems have a fixed number of inodes and if you have lots of small files you'll run into this limit.
But vxfs filesystems are unlimited.

>how can I increase inode number

I assume you'll have to recreate the filesystem.

Re: No space left on device

Md. Minhaz Khan — Sun, 16 May 2010 09:31:43 GMT

Thanks for the reply. "/u01" has ext3 file-system.

# df -TH /u01
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/vglmdb_app-lv_u01 ext3 27G 3.8G 23G 14% /u01

If we need to recreate the file-system then
i have below query:

1)What is the RedHat Linux command to increase inode number after recreating the LV:

#mke2fs -i 4096 /dev/vglmdb_app/lv_u01
Or,
#mke2fs -j -N 150000 /dev/vglmdb_app/lv_u01

2) How could i found what is the present (current) "bytes-per-inode" value of "/dev/vglmdb_app/lv_u01" ? I have tried with
"tune2fs -l /dev/vglmdb_app/lv_u01" command but didn't find.

3)For my scenario, how could i determined what is the correct value of "bytes-per-inode" value? i have found below value in google:

*Less than or equal to 1 Gbyte = 2048
*Less than 2 Gbytes = 4096
*Less than 3 Gbytes = 6144
*3 Gbytes up to 1 Tbyte = 8192

Thanks
Minhaz

Re: No space left on device

Kapil Jha — Mon, 17 May 2010 02:41:45 GMT

80% inode usage means you have a lot of small files, you should delete these files is not used, and check the process which is creating these files.

for NO space left on device, there is a possibility that in /u01 some big files are open and they are showing it to be 100%

du -sk /u01 vs df -h /u01

#lsof /u01

If necessary you may have to stop the process seen in lsof which has too many big files open.
Or restart the service using /u01.

BR,
Kapil+

Re: No space left on device

Md. Minhaz Khan — Mon, 17 May 2010 03:03:33 GMT

Thanks a lot Dennis & kapil. Actuallly "/u01" was created with "-T" option & its size is 25GB. As a result number of inodes become fewer as bytes-per-inode is 1 MB.

Now please give me the guide-line to increase inodes of "/dev/vglmdb_app/lv_u01".

I have made an action plan. Please verify it & if any suggestion or modification required please assist me.

Action Plan1:

Increasing the size of the Logical Volume using "lvextend & ext2online".It will increases the total number of inodes without changing the inodes-per-bytes ratio.

Action Plan 2:

*Stop Application
*Stop Oracle DB
*Backup all data of /u01
*umount /u01
*mke2fs -i 8192 /dev/vglmdb_app/lv_u01
*mount /dev/vglmdb_app/lv_u01 /u01
*Restore Data from backup to /u01
*Start Oracle DB
*Start Application

Thanks
Minhaz

Re: No space left on device

Michal Kapalka (mikap) — Mon, 17 May 2010 12:02:41 GMT

hi,

if you increase the FS size its god solution, but if you use the inodes, you need again extend and them what is the threshold ( next step )???

i think the second plan seems to be better.

mikap

Re: No space left on device

Steven E. Protter — Mon, 17 May 2010 18:51:35 GMT

Shalom Minhaz,

cd /u01

du -k | sort -rn | more

Find those biggies and get rid of them. Do the fuser/lsof check first to see if there are open file handles.

SEP

Re: No space left on device

Dennis Handly — Sat, 22 May 2010 20:28:54 GMT

>Kapil: 80% inode usage means you have a lot of small files, you should delete these files if not used

You can also tar up 1000s of small files into one bigger file. And extract as needed.

Re: No space left on device

Md. Minhaz Khan — Sun, 23 May 2010 02:47:32 GMT

Thanks a lot every body for give me valuable suggestion

Minhaz