realloc() and invalid memory refs.

[Rob Heard]

Hello!

I'm having trouble with some C code that was working fine on an AIX machine, but will not work on HPUX (11.00 32-bit install).

The problem occurs when populating an array of structs using realloc(). The array is described as follows???

struct VSPACE {
char *start;
char *end;
};

struct VARS {
char *fname;
struct VSPACE *point;
int store;
int retflag;
};

struct VARS vars[];

"vars[numvars].point->start" and "vars[numvars].point->end" are used to store the start and end locations in memory of sub-strings of a larger block of text. The sub-strings are identified by certain characters so all this section of code has to do is scan a block of memory, storing sub-string positions (for later reference) as it goes.

realloc() is being used (see text file attached) to dynamically create the array and most of the time works fine, but every now and then the prog will fail, and says 'Memory fault(coredump)'.

Doing a ???file??? on the core returns???
core: core file from 'newmerge' - received SIGSEGV

And doing a ???backtrace??? in GDB returns???

(gdb) bt
#0 0xc0182238 in _memcpy () from /usr/lib/libc.2
#1 0xc017ede8 in _memset () from /usr/lib/libc.2
#2 0xc01846e0 in realloc () from /usr/lib/libc.2
#3 0x1ff78 in setup_wp_vars (input=0x7f7dacf0) at setup_vars.c:129
#4 0x5618 in main (argc=10, argv=0x7f7f09f4) at newmerge.c:434

Any ideas?????? Sorry if my description of the code is a bit inconsistent, but it???s not my code and there are zero comments!

Thanks,

Rob.

[Hartmut Lang]

Only a small hint:
Are you running on a 64bit system?
Do you have a
#include
line in your code?

Otherwise you might get some wrong casting on the return values from malloc()/realloc().

[A. Clay Stephenson · Accepted solution]

Hi Rob:

I think I know what's wrong and you are not going to like my answer. Both your AIX and HP code has been working by accident. Most of the time, realloc is able to expand an allocation in place but sometimes it has to create a new block, copy the contents to the new block, and then return the old block to the heap for possible reallocation. Your pointers then point to data within the OLD block. The maddening part is that unless this data block is reused the data still SEEMS to be valid.

Whenever I faced this problem, I don't use pointers directly but rather let your structs store an array index. When that block is copied the array indexes remain valid even if the base pointer to the array to which they refer has been changed.

I'll try to do an example:

char *big_string; /* dynamically allocated */
/* give it a base address returned by realloc of 0x1000 for simplicity */

typedef struct
{
char my_string[30];
char *your_string;
} my_rec;

Now, suppose that *your_string points to the 3rd element in big_string (i.e. 0x1002); everything works great until at some point we need to grow big string. Most of the time big_string can expand and leave the base address at 0x1000 but suppose it can't and the new address is now 0x8000. your_string still refers to the OLD address and chances are that thast data has not even been overwritten YET. It still APPEARS fine.

The better answer is to change the struct to this:

typedef struct
{
char my_string[30];
int your_string;
} my_better_rec;

Now references to big_string use something
char *begin = &(big_string[your_string]) and not matter what the base of big_string all is well.

I hope I have been semi-clear, Clay

[Rob Heard]

Clay, thanks for your answer and no, I don't like it!

What your saying makes sense and fits in with what I've seen the code do. Just a quick question b/f we start re-writing anything (a big job).

The guy who wrote this code claims that he's run it on all types of systems and has never encountered this problem, and we've been running it for years without error (on AIX). As we're moving towards replacing this system in the near future anyway, is there anything I can do to increase the chance of this (dodgy) code continuing to work by accident? Increase the data segment size perhaps? Sell my soul to the devil? Etc.

Thanks again for ur time.

[A. Clay Stephenson]

Hi Rob:

Believe it or not I was brought in to fix this very bug is a MAJOR database vendor's code about 15 years ago. The problem was three libraries away from where I was looking and everything I did to probe the code looked great for an amazingly long time after the memory was realloc'ed - sometimes for many, many allocations. It depended on the underlying malloc() routines. In any event, this problem has been written to my NVRAM and I always look out for it. I suppose it you allocated a much larger chunk than you really need initially and add a counter for how much you really use; you could delay the problem.

I heard the same stories from the database vendor's programmers as well. "It works on our other platforms." Finally, when I made them carefully read the man pages, they saw that they had not quite played by the rules and realloc was doing just as advertised.

In the database case, eventhough it seem like a huge change, changing to an array index (eventhough is was really a pointer) completely solved the problem and I was able to make the changes in just a couple of hours. Finding that simple problem took a lot longer.

In my case, because this was in a library, the changes were made and a re-link fixed everything.

Food for thought, Clay

[A. Clay Stephenson]

Hi Rob:

I've had another thought for a quick fix. Using my 'big_string' example, suppose that you have an second pointer e.g. old_big_string. Whenever realloc is called to grow big_string, compare the returned value with old_big_string, if they differ, you can apply the difference between the old and the new to fixup your existing pointers and everything is hunky-dorey! That is probably much less intrusive to your code (though less efficient) than replacing pointer with array indices.

By the way, you are new to the Forums and it is considered polite to assign points for any responses you receive.

Regards, Clay

[Rob Heard]

Thanks again Clay, that's a good idea. Just going to try allocating more memory initially, as seems the easiest option. But if not will give your plans B & C a go!

Rob.