Error 28: No space left on device

[Roro_2]

Hello,

Our cutomer had an rx8640 server running HPUX 11i v3. Many logical volumes were created on the XP storage connected to server.
We are not able to create any file on the following directory /db_ora_bckp. This directory is not fully used. An error appears whenever we try to create any file.
Error detail: HPUX-ia64 Error: 28: No space left on device
Kindly find attached more details related to syslog , vgdisplay , fstab and bdf.

Thanks and Regards
Roger

[Michal Kapalka (mikap)]

hi,

check this link :

http://ofdatabaseandotherdemons.blogspot.com/2008_04_02_archive.html

mikap

[Roro_2]

Hi,

The customer is not able to create any file even when using HPUX command touch.

The attached file includes more details about the errors.

Regards

Roger

[Dennis Handly]

>The attached file includes more details about the errors.
>file system error reading inode 3

Your disk has I/O errors, time to replace it.

[Matti_Kurkela]

An interesting message in the syslog:

> Feb 13 01:08:16 browning vmunix: vxfs: WARNING: msgcnt 236 mesg 056: V-2-56: vx_mapbad - vx_smap_stateupd - /dev/vg_SVbckp/lv_SVbckp file system extent allocation unit state bitmap number 10 marked bad

Your later attachment has more filesystem errors:
> Feb 13 01:10:09 browning vmunix: vxfs: WARNING: msgcnt 241 mesg 016: V-2-16: vx_ilisterr: vx_iread_2 - /db_ora_bckp file system error reading inode 5

The relevant part of bdf output:

> /dev/vg_SVbckp/lv_SVbckp
2023751680 576421914 1356871656 30% /db_ora_bckp

The syslog messages indicates your filesystem has errors. This might cause any amount of strange behavior.

If this is on XP storage, the status of the XP storage system should be checked very carefully. If it has RAID1, RAID5 or any other fault tolerant RAID configuration, this simply should never happen unless the XP has multiple failed disks. If that's the case, the situation is definitely very bad and some of your data may already be lost. I hope you have good backups.

If you want to try recovering your filesystem (instead of restoring everything from backups), here's something to try:

Unmount the filesystem, then run a full file system check on it:

umount /db_ora_bckp
fsck -F vxfs -o full /dev/vg_SVbckp/lv_SVbckp

A full filesystem check may take a long time.

MK

[Rita C Workman]

I fully agree with Matt K.
BUT....
...before you umount that file system, please ensure you have a solid backup of whatever is on there in case you need to restore something.

Kindest regards,
Rita

[Dennis Handly]

>ME: Your disk has I/O errors,

I suppose your filesystem could be corrupted if there was a recent crash or power failure.

You could use dd(1) to read the raw disk to see if you find any I/O errors. If not, MK's fsck(1m) may fix it.