The root filesystem is on /dev/sda1, and is 28 gigabytes in size. Do you have any other filesystems at all?
Does /dev/sda2 exist? Is it your swap partition, or something else? Is there any unallocated space on /dev/sda? (Please show the output of "fdisk -l /dev/sda".)
When the inodes on a filesystem are 100% used, no new files or directories can be created. For the ext2/ext3/ext4 filesystems, the ratio of inodes per unit of disk space is chosen at filesystem creation time and cannot be changed afterwards without completely reinitializing the filesystem. (And since that's your root filesystem, it might be easier to backup everything, reinstall the OS with a better-planned filesystem layout, then restore.)
Because the inode ratio is fixed, extending the filesystem will also create new inodes. (But because you aren't using LVM and this is the first filesystem on the disk, extending the filesystem might be difficult.)
The default inode ratio is one inode per 16 Kbytes of disk space. The fact that you've run out of inodes while still having some disk space indicates your average file size is less than 16 Kbytes, i.e. you have a *lot* of tiny files lying around somewhere. This is probably a symptom of an application malfunction or some other problem.
This command will list 20 largest directories on the root filesystem:
du -kx / | sort -rn | head -20
If you run it, what does it say? (Can you even run it with the current situation?)
If the largest directories contain application data files, find someone who understands how the application works. Have him/her examine if the accumulated files are normal and necessary. Clean up if possible.
Your /var/log seems fairly normal in size.
Look into /var/mail and sub-directories of /var/spool: these might be common trouble spots.
If you find a huge number of small files in /var/spool/mqueue, it means you may have a problem with your Mail Transfer Agent (= usually sendmail in RHEL 5 and older, other distributions may use other software). Most likely it's a mis-configuration, perhaps made worse by a spammer attack. It could also be an often-repeating cron job that is trying to send an error message by email to and address that no longer exists. Identify the root cause, clean up the queue, then fix the root cause.
For the future, start planning a better filesystem configuration for this system.
If the application tends to produce an unlimited number of small files, you should plan for creating a separate filesystem for it: it can still fill to 100%, but it will no longer paralyze the rest of the system, and a separate filesystem will be easier to expand.
MK
MK
