limitation of 2gb of data memory segment

[christian_derek]

Hi,

One of my client just migrate to a new server (Bl890c I6) running 11ive may 2020.  Since the migration we are getting some errors from our db supplier and after some investigation, it look like VSS is not able to grow beyond 2gb, ths installation of the O/S was done bye HPE services.  All my other client can grow beyond 2gb, we did increase maxdsiz_64bit to 32gb and restarted the database with no luck.  Here is the comment from my db supplier ...

"The glance output seems to indicate that the eloqdb process has a data VSS of 2 GB (amount of static and allocated memory).  The buffer cache is accounted as "other". So the memory allocation should not fail, given your maxdsiz_64bit limit.

The data VSS should be able to grow beyond 2 GB as necessary and get up to maxdsiz_64bit.

On my test system i can easily get a VSS of 4 GB before allocations fail (as expected due to my maxdsiz_64bit limit ).

I don't see why is your eloqdb seems limited to 2 GB, although it should not."

Any idea, why?

Thanks,

[KishJ]

Greetings,

If it is Java process, check if one of the argument being uses is "-Xss" and the size mentioned with it.

I would also check for the kernel parameters maxssiz (as well as mazssiz_64), of course maxtsiz, for text and maxdsiz for data (and their respective _64bit) are set to.

Hope it helps.

[Bill Hassell]

Make sure ulimit is set to unlimited. Set this for the program's environment.

# ulimit -d unlimited -m unlimited

[christian_derek]

Hi,

It's a C program that we launch with sudo, here are my kernel value...

maxdsiz 2147483647 2147483647 Immed
maxdsiz_64bit 34359738368 34359738368 Immed

maxrsessiz 8388608 Default
maxrsessiz_64bit 8388608 Default
maxssiz 8388608 Default Immed
maxssiz_64bit 268435456 Default Immed
maxtsiz 100663296 Default Immed
maxtsiz_64bit 1073741824 Default Immed

thanks,

[christian_derek]

this is the actual ulimit value I have

# ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 2097148
stack(kbytes) 8192
memory(kbytes) unlimited
coredump(blocks) 4194303
nofiles(descriptors) 8192

Thanks,

[Bill Hassell]

data(kbytes) 2097148 <--- This is 2GB

Use the command: ulimit -d unlimited
before you run the app, or better yet, put the ulimit -d command into /etc/profile.

[christian_derek]

the db process that we experience the issue is a 64bits program

On my dev server the ulimit -a was showing 1gb

# ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 1048576
stack(kbytes) 8192
memory(kbytes) unlimited
coredump(blocks) 4194303
nofiles(descriptors) 2048

I tried to increase the value with no luck

# ulimit -d 2097148
sh: ulimit: The specified value exceeds the user's allowable limit.

The only way I was able to increase the value was by changing the kernel parms maxdsiz to 2gb and it worked after

# ulimit -d 2097148
# ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 2097148
stack(kbytes) 8192
memory(kbytes) unlimited
coredump(blocks) 4194303
nofiles(descriptors) 2048

so, is the ulimit only for 32bits program?

We are using /sbin/init.d/eloq83 to start the program.

Thanks,

[Bill Hassell]

Is there a reason that you did not use the command:

ulimit -d unlimited
ulimit -a

[christian_derek]

Since, it's a production issue only that we can not reproduce.  I decided to play with your recommendation on my dev system when I saw that ulimit data was smaller than the prod.  I tried to bring the number to equal to the prod, but I was getting an error message and the only way I was able to increase the value was to modify maxdsiz.

the ulimit data seems to be related to maxdsiz

I will need to convince my client to increase the parameters to unlimited, but do you think that this could be the problem?  What I need to restart the database or this will be dynamic?

Thanks,

[Bill Hassell]

Since you are running 11.31 (required for the i6 blade you are using), compiling the program as a 64bit program removes any restrictions from the program and are limited by MAXDSIZ_64bit. NOTE: MAXDSIZ_64bit must be larger than MAXDSIZ. If MAXDSIZE_64bit = 1GB, then setting MAXDSIZ to 2GB has no effect -- both will limit data areas to 1GB.

Below is code for mallocmegs which can demonstrate all the compile options.
Start by setting MAXDSIZ to 3GB and MAXDSIZ_64bit to 32GB or larger. These are only runaway programming limits and have no effect on smaller programs.

kctune maxdsiz=3000000000
kctune maxdsiz_64bit=32000000000

Here's the program. Comments show how to compile it for different memory limits.

#include <stdio.h>
#include <stdlib.h>
#include <errno.h>

 /*
Program to request megs of RAM to demo malloc
=============================================

Compile with no flags to allow up to 900 megs.

     cc -Ae -o mallocmegs mallocmegs.c

Compile with -N to allow up to 1700 megs.

     cc -Ae -N -o mallocmegs mallocmegs.c

Use "chatr +q3p enable" to increase to about 2700 megs:

     chatr +q3p enable mallocmegs

and chatr +q4p enable mallocmegs to reach 3700 megs.

     +q4p = enable mallocmegs

For 11.23 (and later), the MPAS address model is available for
32bit programs. Use the option:

     -Wl,+as,mpas

and the result is an executable with 3800 meg capability
with no chatr needed and less restrictions on quadrant maps.

Compile with +DA2.0W or +DD64 to create a 64bit program.
With a 64-bit program, only maxdsiz_64 (and swap) limits
the total malloc area.

==============================

There can be problems with applying chatr +q3p and +q4p to some programs. A
program that has +q3p enabled cannot map shared memory segments that were
created by other processes in the third quadrant. A program using +q3p and
+q4p cannot map any shared memory segments created by other processes.

Code compiled with +Oaggressive will have +Osignedpointers assumptions in
the generated code. That can cause bad pointer comparisons between addresses
in the first two and the last two quadrants. +q3p makes that kind of pointer
comparison more likely.

*/

int main()
{
  extern int errno;
  int strerror_r(int errnum, char *buffer, int buflen);
  #define l_BUFLEN_MAX 80
  int nmeg,l_retval,l_errno,tot_megs;
  void *mptr;
  char *ptr,buf[128],l_errbuf[l_BUFLEN_MAX];
  char *t;
  int n;

  tot_megs=0;

while (1) {
  printf("Additional megs to malloc or CTRL-D: ");
  ptr=fgets(buf,20,stdin);
  if (ptr==NULL)
    {
    printf("\n\n");
    exit (0);
    }
  n=atoi(buf);
  nmeg=n*(1024*1024);
  tot_megs=(tot_megs+n);

/* clear errno before our call */
  errno = 0;
/* local errbuf is set to zero-length string */
  l_errbuf[0]=NULL;

/* let's try to get some more memory */
  mptr=malloc(nmeg);

/* capture errno before another call changes it */
  l_errno = errno;

  if (mptr == NULL)
    {
/* simple error message */

    printf("malloc %d more megs failed (%d total)\n",n,tot_megs);
    sprintf(l_errbuf,
    "errno %d from malloc getting %d megs more memory",l_errno,n)
    ;
    perror(l_errbuf);
    exit(1);
    }
  else
    {
    printf("Got %d megs more RAM, %d megs total\n\n",n,tot_megs);
    }

/* put ! chars in the array to make it occupy real RAM */

  for (n=0;n<nmeg;n+=1024)
    {
      *((char *)mptr+n)='!';
    }
  } /* while */

  printf("\n\n");
  return 0;
} /* main */

[christian_derek]

I'm trying to get the ok to compile and run the program on the prod system, we already try this version and no error was reported, only when we tried to reach 40gb (maxdsiz_64bit set at 32gb).  See the source code below ...

What is funny is in the memory regions of the process in glance, under VSS it shows 2.00gb and it's not growing.  At under client, we have higher value.

DATA /Priv 1 1.98gb 2.00gb 0kb /opt/.../8.3/bin/eloqdb

running tusc we are getting this error on the db process

04/02/21 08:43:57 [6095]{225969} brk(0x6000000081fc0000) ............................................................................................ ERR#12 ENOMEM

HPE also have us run a program rlimit (source code below) and the result was 

$ rlimit  (* with my user *)

cur: 0x800000000 max: 0x800000000

$ sudo rlimit

cur: 0x800000000 max: 0x800000000

$ sudo -u eloqdb rlimit

cur: 0x800000000 max: 0x800000000

$ su

Password:

# ./rlimit

cur: 0x800000000 max: 0x800000000

HPE also ask us to try this undocumented capture

#In this case, there is actually an obscure debug option we can turn #on.  As root, can the customer do this

        echo "mman_elog/W 1" | adb -w -o /stand/vmunix /dev/kmem

#That will turn on brk() debugging.  After they get the error, they #can turn this back off via

        echo "mman_elog/W 0" | adb -w -o /stand/vmunix /dev/kmem

And the result was

pr  2 16:07:07 sky vmunix: Process "eloqdb" Pid 6095 Uid 200 : brk/sbrk failure as exceeding RLIMIT_DATA - returning ENOMEM

Limit program

/*
test allocation limit
usage: limit [sz]
cc -Ae +DD64 -o limit limit.c
*/

#include <stdio.h>
#include <unistd.h>
#include <string.h>
#include <stdlib.h>
#include <sys/resource.h>

int main(int argc, char **argv)
{
int cc;
struct rlimit rl;
void *p;
size_t n;

cc = getrlimit (RLIMIT_DATA, &rl);
if(cc == -1)
perror("getrlimit");
else
printf("cur = %ld, max = %ld\n", rl.rlim_cur, rl.rlim_max);

if(argc == 1)
n = 1; /* default 1 GB */
else
n = atoi(argv[1]);

printf("allocating %ld Gb ...\n", n);

p = calloc(n, 1024*1024*1024);
if(p == NULL)
perror("calloc");
else
printf("done\n");
return 0;
}

rlimit program

#In case they are nervous, this is the source

#include <sys/resource.h>

main(argc, argv)

char argc;

char *argv[];

{

        struct rlimit rlp;

        if (getrlimit(RLIMIT_DATA, &rlp) < 0) {

                printf("syscall failed\n");

                exit(1);

        }

        printf("cur: 0x%lx max: 0x%lx\n", rlp.rlim_cur, rlp.rlim_max);

}

[KishJ]

Hello,

The hard limit for data, text etc. are indeed limited by their respective kernel parameters. You will have to increase them (and their dependencies, if any) to set them to the level your application needs to avoid memory related errors.