malloc error

[David Gwin_1]

Some assistance please,

K260, 512mb ram, HPUX 10.20

I just attempted to restore a file set when I got the malloc error:

Starting file recovery...
frecover(5435): malloc error while attempting to read index
frecover(4301): /tmp/datafilename not recovered from backup media

I know enough to check the maxdsiz parm but that is where the problem really starts. maxdsiz is current 2063835136 (0x7B03A000) and if I attempt to increase I get the error that I have exceeded the maximum size for maxdsiz. I can recover the index file and it is rather large too, 74449920 (there are thousands of files).

So my question is, How can I restore my file if I can increase maxdsiz? Can I recover without the index file?

[TwoProc · Accepted solution]

Well, I don't think it has anything to do with maxdsize because your machine is at 1/2 gig, your maxdsize is 2G and you're trying to allocate 75meg. I just think you've used up all your free ram. I think you could make a larger swap area and let all of that memory allocation run from there. It would be slow, but you might be able to recover your file.

[harry d brown jr]

(1) get new hardware (your hardware is obsolete)
(2) Install 11.11 (or better: 11.23) (your OS is obsolete)

live free or die
harry d brown jr

[D Block 2]

what is your output from: ulimit ?

[A. Clay Stephenson]

Maxdsiz is already more than adequate and there is really no point in it being more than 1GB (and even that is ludicrous given your amount of physical memory). You are problem running out of virtual address space because you don't have enough swapspace. Malloc can also fail if ulimit has been applied.

Do a ulimit -Hd and a ulimit -Sd.

In any event, you only need enough shared memory to hold the index.

[David Gwin_1]

I would love to get new hardware/software however that is not an option.

ulimit -Hd
2015464

and the same for -Sd

swapinfo
Kb Kb Kb PCT
TYPE AVAIL USED FREE USED
dev 524288 13216 511072 3%
reserve - 151540 -151540
memory 389120 256508 132612 66%

[David Gwin_1]

swapinfo table didn't display too well,

_____________Kb______Kb______Kb___PCT_
TYPE______AVAIL____USED____FREE__USED_
dev_______524288___13216__511072____3%_
reserve_________-__151540__-151540
memory___389120__256508___132612___66%

[A. Clay Stephenson]

This is 32-bit HP-UX so both the stack and the data segment are allocated from the same 1GB quadrant. This means, for example, if you had set maxssiz to 800MB then at most only 200MB would actually be available for the data segment regardless of maxdsiz. (and actually about 40MB less than that). Check you setting of maxssiz. Anything bigger than 32MB is only needed for very badky written code.

[David Gwin_1]

Thank you for the information.

maxssiz is 83570688 (0x04FB3000)

[David Gwin_1]

I have made a few adjustments but I'm still not having any luck finding that magic answer.

Other than upgrading the HW or OS, are there any other suggestions as to how I might go about restoring this data tape. Any other parms that could be adjusted up or down?

I can see that I need to change my backup process but that's not going to help with the files that are already on backup tapes.

[Ermin Borovac]

Maybe you can try adding another swap device if have spare disk space.

If you have 'glance' installed on your system you can watch RSS/VSS (resident and virtual memory size) in process detail window (press s and select frecover PID) to see how much memory frecover is trying to allocate.

Alternatively you can run trace frecover with tusc.

# tusc -o tusc.log -fv -s brk frecover ...

You will see see number of brk() calls as frecover tries to increase the heap.

brk(0x400021e8) .......................................... = 0
brk(0x400041d0) .......................................... = 0
brk(0x40005000) .......................................... = 0
brk(0x40007000) .......................................... = 0
brk(0x40008000) .......................................... = 0
brk(0x4001e000) .......................................... = 0
brk(0x40022000) .......................................... = 0

In this example traced program successfully allocated 0x22000 bytes (139264 in decimal).

In your case you will see failed brk() call at the end when frecover fails. From there you can find out how much memory it's trying to allocate.

[A. Clay Stephenson]

At this point, I'm going to ask that you humor me and reduce maxdsiz to about 512MB, maxssiz to about 64MB (or 32MB) and maxtsiz to about 128MB. Since you already have 1X memory as swapspace, swapmem_on=1 is doing nothing for you so you might as well disable pseudoswap. When you reboot under the new kernel make sure that no other non-essential tasks are running. Given your machine's configuration, your present values are a little ridiculous so returning them to more or less sane levels couldn't hurt. I would also limit buffer cache to about 60MB by setting bufpages to 15360 and make sure nbuf is left at zero. This will turn off dynamic buffer cache.

At the very least, these values will make any errors more understandable.

[David Gwin_1]

I don't have glance or tusc installed on this system. I have the very basic system with the last and final hw/sw patches installed for 10.20.

Because this is a heavily used system, I am not able to shut it down during the day. I'll give the reduced maxdsiz, maxssiz and maxtsiz a try this weekend.

Could I use dd to dump the tape contents to another drive? What format is fbackup? Or is its own format?

[A. Clay Stephenson]

Dd is useless to recover this tape because it uses tape marks and variable blocking. You could use frecover on another HP-UX box but I think as soon as you can reduce the load on this box, you should be able to run frecover.

[harry d brown jr]

Auditers would kill us if we still had 10.20 running!

hopefully you are a non-profit!

live free or die
harry d brown jr

[Ermin Borovac]

Even if you don't have glance you can still run 'top' and watch SIZE and RES for frecover before it fails. I think it's important to find out amount of memory it's trying to allocate before making any changes. It also wouldn't hurt to capture 'Memory' line from 'top' and 'swapinfo -tam' before frecover failure.

PS Forgot this was 10.20 system, you wouldn't be able to run tusc anyway!

[D Block 2]

vmstat -n 5 100

I'm thinking vmstat should be around.. on 10.20

also, you can use 'adb' to print the 'freemem' in a loop, to just look at free memory in the kernel.

[David Gwin_1]

Just to give a bit more information,

This past weekend, made a number of attempts to restore the data tape. Changed parms (maxdsiz, maxssiz, maxtsiz, and others), watched the memory (the best that I could), reboots, single user mode, and anything else that i could think of and still no success.

I found someone here locally who has a D-box HP system w/ 11.x 2gb of memory. STILL no luck at restoring the tape.

Any other ideas?

[A. Clay Stephenson]

This really sounds as though you have a corrupt fbackup image on the tape. It sounds as though reading back your backup tapes was not a regular practice.

[David Gwin_1]

Does any one know if there is a limit to the size (space allowed) for the index header file on a tape. IE the first 64mb is reserved for the index header???
I just check the last know good backup is 46mb and there are 760,000 lines in the index file. The file is looks normal:
# 1 /path/filename


The backup that is failing is 74mb and 930,000 lines in the index file. This index is a bit different:
832 1 /path/filename

Each line has what looks like a filesize followed by about 20 spaces.

[David Gwin_1]

That last example didn't display as expected. It should be
832................1 /path/filename

but spaces instead of "."

After my backup is complete, i extract the index header file and parse that into a database for user reference. Then if a file is ever needed i can easily recover the file. The process had been working fine without any problems until just recently. Most of the time (but not this time) I will restore the last few file just as added insurance.

[Ermin Borovac]

At some point in time, fbackup was changed to list file size in the first column of index file (instead of just hash mark). Since you indicated that last valid backup had hash marks in index file, this means that your backups started failing when a certain fbackup/frecover patch was installed (not sure which patch changed the behaviour).

As for the index file size, there are some notes that it was limited to 100M, but you are not quite near that limit.

Just out of curiousity, what's the index file size as shown in the volume header.

# frecover -V - -f /dev/rmt/0m

Also can you verify files on the tape with -N option or it fails with malloc error?

# frecover -r -N -f /dev/rmt/0m

Does number of lines returned by the last command match with number of lines in the index file?

[vinod_25]

Hi david

Typically, this type of problem could be due to too small msxdsiz or
maxssiz, or it could be a swap issue. "/stand/system" indicated
that maxdsiz and maxssiz were default values. Upon checking swap:

# swapinfo -tm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 400 103 297 26% 0 - 1 /dev/vg00/lvol2
reserve - 297 -297
memory 197 178 19 90%
total 597 578 19 97% - 0 -


The choices were to add swap space (permanent or temporary) or
shutdown databases and/or applications to free up swap space.

Regards

Vinod

[David Gwin_1]

Thank you all for you information and help. Part of the problem was that maxdsiz was set too high for the system and the main problem was not enough swap space. After adding additional swap space the system was able to restore the tape.

Thank you again for all the help.

[David Gwin_1]

I guess those of you who suggested adding more swap space needed to use a BIGGER hammer. But I really thought that there was plenty.