Problem with STR$CONCAT

[#1]

Hi,

We have a code for a product written in C.
A process make use of str$concat function to concatenate two strings.
Now when we are running that process it successfully doing the job for the first time but second time when we are doing it, its giving the following error:

STR$_INSVIRMEM

I already have OpenVMS RTL String Manipulation (STR$) Manual but I didn't find anything useful in it.

Could anyone help me to know why we got this error and why the process is running fine first time but giving problem second time.

Note: In the code we are trying to concatenate strings of very large length.

Best Regards,
ajaydec

[Steven Schweda]

> We have a code for a product written in C.

Perhaps so, but we can't see it. I can't
speak for others, but my psychic powers are
weak.

> Could anyone help me [...]

I don't see how.

> [...] strings of very large length.

How large is "very large"? 256? 2G?

You are using descriptors, right?

[Hein van den Heuvel]

STR$CONCAT requires descriptors.
Those are not native to C.
You can all too easily get them wrong.
Since you can do it once, the program must get it 'mostly' right. Still...

How does the product create the descriptors?

How large is a piece of string?
'very large' is NOT good enough as answer.
Magic numbers to think about: 32KB (2**15), 64KB (2**16).

Mind you... concatenating big pieces of string suggest 'sup-optimal' programming.

hth,
Hein.

[#4]

String size is more than 256 kb.
I want to know under what circumstances str$concat function can generate INSVIRMEM signal.

We are using extended descriptor as the destination and a character strings for source strings.

Best Regards,
ajaydec

[Wim Van den Wyngaert]

Help/mes/fac=str insvirmem

(just learned the /facility qualifier exist)

May be increase user quota pgflquo.

Wim

[Jon Pinkley]

RE:"I already have OpenVMS RTL String Manipulation (STR$) Manual but I didn't find anything useful in it."

Did you look at the end of the STR$CONCAT section?

STR$_INSVIRMEM Insufficient virtual memory. STR$CONCAT could not allocate heap storage for a dynamic or temporary string.

What you are asking the service to do is to take multiple input strings and create a new one from contiguous virtual address space. The system needs a place to put this copy. It allocates memory, copies your strings together, and returns the address of this new string in the descriptor.

If you are never freeing the memory used by these strings, the space they occupy will not be available for other strings. Even if you do free it, the pool will become fragmented if resultant string size is not a constant size, or if you are creating other strings in the pool. In this respect it is like fragmentation on a disks, and the diminishing size of the largest contiguous file that can be created on the disk.

Please skim chapter 1 and read chapter 2 of the STR$ manual.

Although the following thread isn't exactly related to your question, it discusses the underlying cause of the pool fragmentation:

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1198013

If you have plenty of memory, and privilege, you can increase your pgflquota and you will then be able to consume more resources before the system tells refuses to give you more.

Jon

[Willem Grooters]

Is the second call additional to the first?
I mean:

1: Concatenate A + B ==> Z
2: Concatenate C + Z ==> Y
(so Y = A + B + C)

of is the second one the same as the first with new variable content?

Is it possible that your strings are dynamic (requirement for STR$CONCAT), perhaps created with STR$GET1_DX? It could help if you did a STR$FREE1_DX first, to release allocated memory.

INSVIRMEM means you run out of virtual memory. Increasing PGFLQUOTA may help in that case.

[#8]

Yes, I am doing the following:
Is the second call additional to the first?
I mean:
1: Concatenate A + B ==> Z
2: Concatenate C + Z ==> Y
(so Y = A + B + C)
of is the second one the same as the first with new variable content?

And I am using lib$sfree1_dd to free the allocated memor.

[Jon Pinkley]

If you interested only in the A + B + C string and not the intermediate A + B string, then why not call STR$CONCAT with more than two arguments? The following is from the manual.

STR$CONCAT

The Concatenate Two or More Strings routine concatenates all specified source strings into a single destination string.

Format
STR$CONCAT destination-string ,source-string [,source-string...]

There really is useful information in the manual whether you found it or not. It does not seem you looked very hard.

[Hein van den Heuvel]

Ajadec,

So the program is using and building a 64 bit descriptor and using :
DSC64$Q_LENGTH / DSC64$PQ_POINTER

OS version / Platform?

Or are strings A + B possibly less than 64K in length and A + B + C > 64K?

I still suspect that concattenating large strigns is poor / lazy programig, but of course it may well be the optimal solution.


As others replied, INSVIRMEM suggest the program is out or PAGFILQUO.
Did you check?
Try: $SHOW PROC/CONT ... 'q'

Cheers,
Hein.

[#11]

Hi,

Following is the part of the code which show how we are using str$concat function to add two strings:

typedef struct dsc$descriptor_s desc;
typedef struct
{unsigned short XLENGTH;
unsigned char XDTYPE;
unsigned char XCLASS;
char *XPOINTER;
union
{struct
{char scl;
unsigned char dig;
} s;
short fac;
} u;
struct
{unsigned : 3;
unsigned bscl : 1;
unsigned : 4;
} sfl;
unsigned : 8;
} xdesc;

xdesc a, b, c;

str$concat(&a,&b,&c);
stat = lib$sfree1_dd ((desc *)&a);
stat = lib$sfree1_dd ((desc *)&b);


Best Regards,
ajaydec

[Hein van den Heuvel]

You are not serious are you?

You are just testing us to see how we react to silly questions right?

>>> unsigned short XLENGTH;

So that's a 16 bit quantity, allowing for a max of 64K as suggested migth be theproblem in earlier replies.

>> I already have OpenVMS RTL String Manipulation (STR$) Manual but I didn't find anything useful in it.

I strongly recommend you read that manual again. Twice. Seriously.
Notably:

1.1.1 64-Bit Addressing Support (Alpha Only)

http://h71000.www7.hp.com/doc/73final/5936/5936pro.html#str_64bit

and

2.2 Descriptor Classes and String Semantics
http://h71000.www7.hp.com/doc/73final/5936/5936pro_001.html#7_descriptorclassesandstrings


Jeez...

Best regards,
Hein.

[Hein van den Heuvel]

Ooops... I may have replied too quickly

I half stopped reading seeing dsc$descriptor_s
and the regular sounding lib$sfree1_dd

It looks like the application is hand-crafting a 64 bit descriptor, but thats hard to follow and the actually usage (initialization) is not shown.

Be sure to dig down into the code that fills in the fields for the xdesc (or the service call used to to so).

Sorry,
Hein.

[Hoff]

Do consider a move to system declarations, and away from local or private declarations. Makes for easier maintenance, and reduces errors. For this case, descrip.h and dscdef.h contain the system-provided data descriptor declarations.

Always verify the return status of a system or RTL call. Checking the return status of RTL and system service calls is considered good coding practice on OpenVMS. The contents of stsdef.h can be used here, or you can simply mask against the low bit of the return for standard OpenVMS condition values. If the low bit is set, the call worked.

Always initialize the contents of dynamic string descriptors. This (omission) looks to be the root cause of the str$concat error here, based on the short example C code posted. At its core, once a descriptor is initialized and then turned over to any of the OpenVMS routines, nothing other than OpenVMS routines should write to a dynamic descriptor. Not until after the descriptor has been freed. (And I usually then immediately zero it.)

When working to reproduce an error, code simplification is your friend. Unnecessary complexity leads to instability and increased maintenance costs. And here, that declaration is rather twisty.

Also consider a review the related discussion on using dynamic string descriptors present in the OpenVMS FAQ, and various of the available information on common C coding errors on OpenVMS. There are web sites with this material, and the OpenVMS Programming Concepts manual is a fairly good read.

http://www.hoffmanlabs.com/vmsfaq/
http://h71000.www7.hp.com/wizard/wiz_1661.html

Here's an example from the FAQ (< for { and > for }):

#include {descrip.h}
#include {lib$routines.h}
#include {stsdef.h}
int RetStat;
char TxtBuf[TXTSIZ]
struct dsc$descriptor StaticDsc =
{ 0, DSC$K_DTYPE_T, DSC$K_CLASS_S, NULL };
struct dsc$descriptor DynDsc =
{ 0, DSC$K_DTYPE_T, DSC$K_CLASS_D, NULL };
int DynDscLen = 255;
$DESCRIPTOR( ConstDsc, "This is a string" );

/* finish setting up a static descriptor */
StaticDsc.dsc$w_length = TXTSIZ;
StaticDsc.dsc$a_pointer = (void *) TxtBuf;

/* finish setting up a dynamic descriptor */
RetStat = lib$sget1_dd( &DynDscLen, &DynDsc );
if ( !$VMS_STATUS_SUCCESS( RetStat ) )
return RetStat;

/* release the dynamic storage */
RetStat = lib$sfree1_dd( &DynDsc );
if (!$VMS_STATUS_SUCCESS( RetStat ))
return RetStat;

[Bojan Nemec]

ajaydec,

It can be an alignement problem on your hand-crafted descriptor.

So, as Hoff mentioned, move to standard descriptor structs from descrip.h or try with alignement pragmas:

#pragma __member_alignment __save
#pragma __nomember_alignment

#pragma __member_alignment __restore

Bojan

[#16]

Thanks to everone for their efforts and time.

The problem is now solved.

Apart from this, I need one confirmation:

STR$CONCAT will allocate the memory dynamically by itself where it will store the resultant string. But nowhere in the str document they have mentioned that str$concat will allocate memory dynamically nor they have given advice/warning to free up that memory.

I just want to know whether my point is right or did I missed something in the manual to read.

Regards,
ajaydec

[Hein van den Heuvel]

I think this implied by selecting a "dynamic" string descriptor.
Surely the implication of that are well documented in the description of those descriptors, and probably for many, but possibly not all of its users.

Hein.

[Willem Grooters]

The STR$-library documentation clearly states that INSVERMEM is a status that can be returned. My assumtion therefore is that the routine will so some allocation for temprorary storage. And since I haven't heard of anything proclaming evidence of the opposite, I am sure that allocated memory is de-allocated after use.
Though I can live with that assumption, you may check the listings of the STR-libary routines to be sure. (I have no access to them)

WG

[Hoff]

String and the many other descriptors are a common area for coding errors, as the construct is usually entirely foreign to most programmers new to using C on OpenVMS.

As for the confirmation sought here, that and far more would be included in the early chapters of the STR$ manual. (This was a suggestion included earlier in this thread.) Also see the OpenVMS FAQ, as that contains some of the more common coding errors, including references to coding errors involving dynamic descriptors. Also see topic (1661) in the old Ask The Wizard http://www.hp.com/go/openvms/wizard area, as that has lists of common coding errors. Also see the Programming Concepts manual for the OpenVMS programming norms and concepts and interfaces, and the C manuals for the C norms and concepts associated with C on OpenVMS.

Yes, reading through all this material is work, and it'll slow you down for a little while while you work your way through it all, and while you digest and work toward understanding it. It'll be way more than one sitting, too. Down the road, however, this effort will help greatly speed your coding, and will improve the stability and maintainability of your code. And from a purely self-interest perspective, this sort of effort will also help you further your programming career.