Memory/swap space errors preventing command line queries

[JGutfreund]

I'm running into a problem with my HPUX 10.20 server; however, I can't even run basic commands to get info from the system.

# bdf
/usr/lib/dld.sl: Call to mmap() failed - ZEROES /usr/lib/libc.1
/usr/lib/dld.sl: Not enough space
Abort(coredump)

any ideas? I haven't tried a shutdown command, but I may have to do a hard boot to get control again.

[James R. Ferguson]

Hi:

You are out of memory. You may need to increase your swap space and/or you kernel's 'maxdsiz' limit. Unfortunately, from what you describe, a reboot (hard if necessary) is probably the only way to gain control of your server.

What changes did you last make? Do you have a run-away process?

Regards!

...JRF...

[JGutfreund]

Looks like it is the root folder that is out of disk space. When I do a bdf, that's what it displays; however, I don't seem to be able to get a list of the files to be able to delete any.

# ls
Pid 21583 killed due to text modification or page I/O error


vxfs: mesg 001: vx_nospace - /dev/root file system full (1 block extent)
Bus error

# ls
Pid 21583 killed due to text modification or page I/O error

vxfs: mesg 001: vx_nospace - /dev/root file system full (1 block extent)
Bus error

[JGutfreund]

Here is the result of the bdf command:

# bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 86016 86016 0 100% /

[James R. Ferguson]

HI (again):

> /usr/lib/dld.sl: Not enough space

Is saying that you have exhausted memory and/or swap.

> vxfs: mesg 001: vx_nospace - /dev/root file system full (1 block extent)
Bus error

This is a critical filesystem (the '/') out-of-space.

I would reboot into single user mode and assess things from there. You are going to go nowhere otherwise.

Regards!

...JRF...

[JGutfreund]

Yes, that's kind of what I was thinking. Without being able to see the files, I can't determine what I can delete. I can't even FTP into it and get a file listing.

[Dennis Handly]

You are out of swap. You need to kill some processes.
You should also remove those corefiles.

>Looks like it is the root directory that is out of disk space.

This is unrelated to swap, unless the corefiles from the above abort filled up root.

>I don't seem to be able to get a list of the files to be able to delete any.

Hmm, I don't know if the tools in /sbin/ will still have the problem: /sbin/ls

Can you do ps(1) to kill processes?

>JRF: your kernel's 'maxdsiz' limit.

I doubt this is the problem for bdf, unless set trivially low.

[JGutfreund]

Thanks folks, but until I can get commands to be accepted, I can't make any changes, or free up any space. Looks like the only recourse is to do a shutdown/restart and then extend the root file system. God only knows why they set the default so low to start with.

[Dennis Handly]

>I can't make any changes, or free up any space.

Are there no users that you can ask to log off?
Any fork bombs running and eating up swap?

[James R. Ferguson]

Hi (again):

> I can't make any changes, or free up any space.

Then bite the bullet and hard boot the machine. If you can't do anything, I'd bet that your users are suffering too.

Amd, as far as extending the 'root' filesystem, if that is truly a crippling point, plan to take an Ignite image after you are back and running and use that image to cold-install from. You can choose the Ignite advanced installation mode and thus select the sizes for your vg00 mountpoints that best mach your environment.

Regards!

...JRF...