topic Re: Memory/swap space errors preventing command line queries in Operating System - HP-UX

Memory/swap space errors preventing command line queries

JGutfreund — Tue, 31 May 2011 17:13:35 GMT

I'm running into a problem with my HPUX 10.20 server; however, I can't even run basic commands to get info from the system.

# bdf
/usr/lib/dld.sl: Call to mmap() failed - ZEROES /usr/lib/libc.1
/usr/lib/dld.sl: Not enough space
Abort(coredump)

any ideas? I haven't tried a shutdown command, but I may have to do a hard boot to get control again.

Re: Memory/swap space errors preventing command line queries

James R. Ferguson — Tue, 31 May 2011 17:28:35 GMT

Hi:

You are out of memory. You may need to increase your swap space and/or you kernel's 'maxdsiz' limit. Unfortunately, from what you describe, a reboot (hard if necessary) is probably the only way to gain control of your server.

What changes did you last make? Do you have a run-away process?

Regards!

...JRF...

Re: Memory/swap space errors preventing command line queries

JGutfreund — Tue, 31 May 2011 19:27:59 GMT

Looks like it is the root folder that is out of disk space. When I do a bdf, that's what it displays; however, I don't seem to be able to get a list of the files to be able to delete any.

# ls
Pid 21583 killed due to text modification or page I/O error

vxfs: mesg 001: vx_nospace - /dev/root file system full (1 block extent)
Bus error

# ls
Pid 21583 killed due to text modification or page I/O error

vxfs: mesg 001: vx_nospace - /dev/root file system full (1 block extent)
Bus error

Re: Memory/swap space errors preventing command line queries

JGutfreund — Tue, 31 May 2011 19:48:04 GMT

Here is the result of the bdf command:

# bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 86016 86016 0 100% /

Re: Memory/swap space errors preventing command line queries

James R. Ferguson — Tue, 31 May 2011 20:09:26 GMT

HI (again):

> /usr/lib/dld.sl: Not enough space

Is saying that you have exhausted memory and/or swap.

> vxfs: mesg 001: vx_nospace - /dev/root file system full (1 block extent)
Bus error

This is a critical filesystem (the '/') out-of-space.

I would reboot into single user mode and assess things from there. You are going to go nowhere otherwise.

Regards!

...JRF...

Re: Memory/swap space errors preventing command line queries

JGutfreund — Tue, 31 May 2011 20:16:09 GMT

Yes, that's kind of what I was thinking. Without being able to see the files, I can't determine what I can delete. I can't even FTP into it and get a file listing.

Re: Memory/swap space errors preventing command line queries

Dennis Handly — Wed, 01 Jun 2011 01:31:31 GMT

You are out of swap. You need to kill some processes.
You should also remove those corefiles.

>Looks like it is the root directory that is out of disk space.

This is unrelated to swap, unless the corefiles from the above abort filled up root.

>I don't seem to be able to get a list of the files to be able to delete any.

Hmm, I don't know if the tools in /sbin/ will still have the problem: /sbin/ls

Can you do ps(1) to kill processes?

>JRF: your kernel's 'maxdsiz' limit.

I doubt this is the problem for bdf, unless set trivially low.

Re: Memory/swap space errors preventing command line queries

JGutfreund — Wed, 01 Jun 2011 13:13:25 GMT

Thanks folks, but until I can get commands to be accepted, I can't make any changes, or free up any space. Looks like the only recourse is to do a shutdown/restart and then extend the root file system. God only knows why they set the default so low to start with.

Re: Memory/swap space errors preventing command line queries

Dennis Handly — Wed, 01 Jun 2011 17:42:28 GMT

>I can't make any changes, or free up any space.

Are there no users that you can ask to log off?
Any fork bombs running and eating up swap?

Re: Memory/swap space errors preventing command line queries

James R. Ferguson — Wed, 01 Jun 2011 17:47:42 GMT

Hi (again):

> I can't make any changes, or free up any space.

Then bite the bullet and hard boot the machine. If you can't do anything, I'd bet that your users are suffering too.

Amd, as far as extending the 'root' filesystem, if that is truly a crippling point, plan to take an Ignite image after you are back and running and use that image to cold-install from. You can choose the Ignite advanced installation mode and thus select the sizes for your vg00 mountpoints that best mach your environment.

Regards!

...JRF...