topic Re: Error 28: No space left on device in Operating System - HP-UX

Error 28: No space left on device

Roro_2 — Mon, 15 Feb 2010 07:38:05 GMT

Hello,

Our cutomer had an rx8640 server running HPUX 11i v3. Many logical volumes were created on the XP storage connected to server.
We are not able to create any file on the following directory /db_ora_bckp. This directory is not fully used. An error appears whenever we try to create any file.
Error detail: HPUX-ia64 Error: 28: No space left on device
Kindly find attached more details related to syslog , vgdisplay , fstab and bdf.

Thanks and Regards
Roger

Re: Error 28: No space left on device

Michal Kapalka (mikap) — Mon, 15 Feb 2010 08:23:18 GMT

hi,

check this link :

http://ofdatabaseandotherdemons.blogspot.com/2008_04_02_archive.html

mikap

Re: Error 28: No space left on device

Roro_2 — Mon, 15 Feb 2010 09:57:09 GMT

Hi,

The customer is not able to create any file even when using HPUX command touch.

The attached file includes more details about the errors.

Regards

Roger

Re: Error 28: No space left on device

Dennis Handly — Mon, 15 Feb 2010 10:20:21 GMT

>The attached file includes more details about the errors.
>file system error reading inode 3

Your disk has I/O errors, time to replace it.

Re: Error 28: No space left on device

Matti_Kurkela — Mon, 15 Feb 2010 10:45:41 GMT

An interesting message in the syslog:

> Feb 13 01:08:16 browning vmunix: vxfs: WARNING: msgcnt 236 mesg 056: V-2-56: vx_mapbad - vx_smap_stateupd - /dev/vg_SVbckp/lv_SVbckp file system extent allocation unit state bitmap number 10 marked bad

Your later attachment has more filesystem errors:
> Feb 13 01:10:09 browning vmunix: vxfs: WARNING: msgcnt 241 mesg 016: V-2-16: vx_ilisterr: vx_iread_2 - /db_ora_bckp file system error reading inode 5

The relevant part of bdf output:

> /dev/vg_SVbckp/lv_SVbckp
2023751680 576421914 1356871656 30% /db_ora_bckp

The syslog messages indicates your filesystem has errors. This might cause any amount of strange behavior.

If this is on XP storage, the status of the XP storage system should be checked very carefully. If it has RAID1, RAID5 or any other fault tolerant RAID configuration, this simply should never happen unless the XP has multiple failed disks. If that's the case, the situation is definitely very bad and some of your data may already be lost. I hope you have good backups.

If you want to try recovering your filesystem (instead of restoring everything from backups), here's something to try:

Unmount the filesystem, then run a full file system check on it:

umount /db_ora_bckp
fsck -F vxfs -o full /dev/vg_SVbckp/lv_SVbckp

A full filesystem check may take a long time.

MK

Re: Error 28: No space left on device

Rita C Workman — Mon, 15 Feb 2010 15:21:23 GMT

I fully agree with Matt K.
BUT....
...before you umount that file system, please ensure you have a solid backup of whatever is on there in case you need to restore something.

Kindest regards,
Rita

Re: Error 28: No space left on device

Dennis Handly — Mon, 15 Feb 2010 23:48:03 GMT

>ME: Your disk has I/O errors,

I suppose your filesystem could be corrupted if there was a recent crash or power failure.

You could use dd(1) to read the raw disk to see if you find any I/O errors. If not, MK's fsck(1m) may fix it.