Limitations on the number of mail recipients in UCX (TCP/IP Services) SMTP Client?

[Mark_Corcoran]

We have some emails that are sent to distribution lists, containing either a periodic report or a notification of an adverse event or condition having been detected.

OpenVMS MAIL allows you specify a distribution list file with the construction "@dev:[dir]filename.dis"

I've found in the past that when the number of characters in the file is "around" 1024, the distribution list gets truncated (I had thought that it was midway through an internet-style email address, but testing just now suggests not).

Initially, I had presumed it was something to do with the DCL CLI, with the maximum command line length (and that it was expanding the contents of the distribution list file into the command buffer), so I was looking at creating an executable that could be passed the filename, where it would then read one line at a time and use the callable mail routines (specifically MAIL$SEND_ADD_ADDRESS) to add each line one by one.

I created the main code body to call the wrapper routines I wrote years ago, and tried some testing this morning, only to find that even with MAIL$SEND_ADD_ADDRESS, the distribution list still gets truncated.

I then tried changing the destination addresses from smtp%"user@domain.com" to smtp%"nnn@a.com" after generating a .DIS file with nnn values 001 through 100 (this is on an internal test system which doesn't gave a gateway, so emails don't go anywhere except onto an SMTP queue for retrying before evenutally giving up).

With this distribution list, I actually ended up getting an access violation in the call to MAIL$SEND_ADD_ADDRESS - there was no traceback, just a register dump;  it was clear that the stack had been corrupted, because the virtual address that was reported was clearly a hex representation of 4 characters of the email address that was being attempted to be added.

A further test with the executable linked /DEBUG and run in the debugger reported the following before the ACCVIO:
%DEBUG-E-LASTCHANCE, stack exception handlers lost, re-initializing stack

I tried testing again, this time using hundreds of local OpenVMS user accounts rather than SMTP%-prefixed addresses, without an ACCVIO.

I'm not sure whether the ACCVIO really occurred in the callable mail routines, or if it was when callable mail spotted the SMTP% then handed it off to either UCX$SMTP_MAILSHR.EXE or UCX$SMTP_PARSESHR.EXE

[The test with local accounts repeated the same group of 30 account names 33 times, so I'm not sure if MAIL is checking to see whether or not the requested local account is already part of the list and if so, discarding it (so it never ends up overwriting the end of some buffer, or miscalculating an address), whereas it doesn't perhaps get the opportunity to do so with SMTP%-prefixed addresses (though I would assume that UCX eventually has to hand it back to callable mail to add to some structure or doubly-linked list)]

I've not been able to find any documented limitation on the number of email recipients that you can specify in the UCX SMTP client, though there obviously has to be an upper limit at some point on the local system (when you run out of memory/disk to store it, or you exceed the addressing limits of the CPU architecture).

Does anyone happen to know what (if any) documented limit there is on the number of characters that the recipient list can consume, and if this is in the Callable Mail Routines, or in UCX / TCP/IP Services (this is UCX v4.2 ECO 4 on OpenVMS/VAX v6.2, neither of which is likely to be upgraded, but I'd be curious to know whether or not there is a similar limitation in somewhat more recent versions of TCP/IP Services)?

Has anyone else had a similar problem, and if so, what have they done to work around it?

[When a list grows too big, we've been chopping it into two lists (thankfully, not had a need for three or more), but this does require the originating .COM to be modified to send an email to each list, and if someone does a "reply to all" to the list they are on, then the folk on the other list are in the dark about the fact an issue is being investigated/resolved.

I know that you can have lists within lists (i.e. inside the file that you specify as "@dev:[dir]filename.dis", you can specify @DEV:[dir]filename2.dis), but that would involve more work than the existing manual process.

Although distribution lists exist elsewhere (perhaps related to Active Directory), getting addresses added to/removed from those distribution lists is a lot more bureaucratic than just editing the .DIS file on OpenVMS, so that's probably also not a solution]

Mark

[Volker Halle]

Mark,

there are SMTP debug logicals described in the TCP/IP Service Management Guide chapter 18.5.2

TCPIP$SMTP_VMSMAIL_SEND MAIL$PROTOCOL Logs diagnostic messages to a file named DEBUG.TXT in the default directory. This logical name is reserved for use by HP.

TCPIP$SMTP_VMSMAIL_PARSE Symbiont,Receiver,MAIL$PROTOCOL

Causes the SMTP address parsing code to log diagnostics. This logical name is reserved for use by HP.

Defining those logicals in the context of your process may help you in your experiments to better understand the SMTP MAIL$PROTOCOL behaviour.

Volker.

[Volker Halle]

Mark,

in your case (TCPIP V4.2), the logicals need to be UCX$SMTP_VMSMAIL_PARSE and UCX$SMTP_VMSMAIL_SEND - just define them to a value of TRUE. You'll find the debug information in DEBUG.TXT

TCPIP V4.2 ECO 1 seems to truncate the To: field to around less than 1000 characters. Mail itself seems to remove duplicate addresses, so I had to make the addresses unique to force the error. And there is an error message:

build_path_to: cur_item: , HALLE33@nodexx.invenate.local
build_path_to: warning: TO: string overflow. Truncating...
build_path_to: exiting normally. TO: header is:
HALLE@axpvms.invenate.local, HALLE2@nodexx.invenate.local, HALLE3@nodexx.invena...

TCPIP V5.5 does not remove duplicate addresses in the .DIS file, but also truncates the To: field - also with an error message:

Entering append_sender_header
append_sender_header: cur_item: , HALLE@nodexx.invenate.local
append_sender_header: warning: To/Cc: string overflow. Truncating...

...

HEADER
h_field = 4, To:
h_length = 997
h_value = HALLE@nodexx.invenate.local, HALLE@nodexx.invenate.local,

...

TCPIP V5.5 is sending all the mails, it's just truncating the To: field.

Volker.

[Mark_Corcoran]

Thanks for your responses, Volker.

I had forgotten about the logical names (I had made notes on them years ago, in the 215pp document of various OpenVMS cribs I've collected over the last 30 years).

As you indicated, as UCX is the old name for TCP/IP services, the logical names are different (I'd already determined logical names after your first post by searching SYS$SHARE:*SMTP*.EXE for the right hand side of the logical name (on the basis that facility name wouldn't have been TCPIP)).

For some reason, I hadn't noticed the UCX$SMTP_VMSMAIL_SEND logical name when making a list in the .EXEs and then defining them /SYS /EXEC, so I was battling for a while before I saw your update, defined this logical, then saw the overflow message that you mentioned.

I carried out some more testing, inching the distribution list name by one character at a time, and it overflows if the resultant To: list (with ", " between each user@domain) exceeds 999 characters, but it doesn't truncate the user@domain which causes the overflow - it just doesn't add it to the list.

I had another look at the ACCVIO whilst stepping through the code in the debugger - it occurs at SHARE$MAILSHR+13486, and the instruction at that address is this:
SHARE$MAILSHR+13486: CALLS S^#0,W^SHARE$MAILSHR+13109

It's simply trying to call a procedure (with a stack argument list), but the problem is that the stack is hosed at this point.

I'll take another look at my own code to see if it's doing anything wrong (it is in C after all, and it's very easy to shoot yourself in the foot), but the sticking point here is that the To: list is limited to 999 characters.

Just to be clear Volker, are you saying that in v5.5, a To: list of more than 999 characters is actually sent to all the intended email addresses and it's just the To: field that is truncated?

Isn't the To: field part of the message header, and therefore if it is truncated, then recipients beyond the truncation point simply won't get the email?

Mark

[Volker Halle]

Just to be clear Volker, are you saying that in v5.5, a To: list of more than 999 characters is actually sent to all the intended email addresses and it's just the To: field that is truncated?

Yes, on OpenVMS Alpha V8.2 with TCPIP V5.5 I get 42 SMTP mails sent to myself (on the local node), if using a .DIS file with 42 addresses like halle@nodexx.invenate.local in 42 separate lines of the .DIS file. The To: list in the mail header does get truncated.

Interestingly the last 2 lines in the .DIS files are like volker.halle@domain.com and I only get ONE mail not TWO.

Volker.

[Steven Schweda]

> Isn't the To: field part of the message header, [...]

   It is.

> and therefore if it is truncated, then recipients beyond the
> truncation point simply won't get the email?

   No, precisely because the "To: " field is (only) part of the message
header.  The actual recipients are specified using "RCPT TO: ", not
"To: ".

   A Web search for terms like, say:
      smtp rcpt
should find more details. For example:

      https://stackoverflow.com/questions/10822190

   Everything's complicated.

[Mark_Corcoran]

>I'll take another look at my own code to see if it's doing anything wrong (it is in C after all, and it's very easy to shoot yourself in the foot)

I had another look, and the basics of what it was doing (once you stripped away all the parsing and sanity checking) was fairly simple.

I ran the executable in the debugger again, setting a breakpoint after the call to MAIL$SEND_ADD_ADDRESS(), so that I could repeatedly issue a SHOW CALLS then a GO - at no point were the call frames on the stack trashed until the final call to MAIL$SEND_MESSAGE(), at which point the ACCVIO occurred.

I've included a small reproducer below - apologies for the formatting - the web site always insists on stripping leading spaces when pasting, making it impossible to show correct indentation unless you enter spaces by hand AFTER pasting;  I'm none too keen on how it implements carriage returns/paragrpahs/double-spacing either (as bad as older versions of M$ Word).

#module REPRO /* REPRO.C */
#include <mail$routines.h>
#include <maildef.h>
#include <stdlib.h>

typedef struct itemlist3
{
unsigned short buflen ;
unsigned short code ;
void *bufadr ;
void *retlenaddr ;
} ITEMLIST3 ;

int main(int argc, char *argv[])
{
unsigned long fencepost1 = 0xFEEDFACE ;
int i, iterations, result ;
long context = 0L ;
unsigned short dummy = 0 ;

char dest[15] ;
char *subject = "Callable Mail Send Test" ;
char *file_to_send = "NL:" ;

ITEMLIST3 begin_inlist[3] =
{
{0, MAIL$_SEND_NO_PERS_NAME, NULL, 0L},
{0, MAIL$_NOSIGNAL, 0L, 0L},
{0, 0, 0L, 0L}
} ;

ITEMLIST3 attribute_inlist[3] =
{
{0, MAIL$_SEND_SUBJECT, subject, &dummy},
{0, MAIL$_NOSIGNAL, 0L, 0L},
{0, 0, 0L, 0L}
} ;

ITEMLIST3 address_inlist[3] =
{
{0, MAIL$_SEND_USERNAME, &dest[0], &dummy},
{0, MAIL$_NOSIGNAL, 0L, 0L},
{0, 0, 0L, 0L}
} ;

ITEMLIST3 add_bodypart_inlist[3] =
{
{0, MAIL$_SEND_FILENAME, file_to_send, &dummy},
{0, MAIL$_NOSIGNAL, 0L, 0L},
{0, 0, 0L, 0L}
} ;

ITEMLIST3 send_message_inlist = {0, 0, 0L, 0L} ;
ITEMLIST3 outlist = {0, 0, 0L, 0L} ;
unsigned long fencepost2 = 0xDEADBEEF ;

iterations = ((argc > 1) && !strcmp(argv[1], "ACCVIO")) ? 85 : 84 ;

/* Begin mail composition */
printf("\nCalling MAIL$SEND_BEGIN") ;
result = MAIL$SEND_BEGIN(&context, &begin_inlist, &outlist) ;
if (!(result & 1))
exit(result) ;

/* Add subject header */
attribute_inlist[0].buflen = strlen(subject) ;
printf("\nCalling MAIL$SEND_ADD_ATTRIBUTE") ;
result = MAIL$SEND_ADD_ATTRIBUTE(&context, &attribute_inlist, &outlist) ;
if (!(result & 1))
exit(result) ;

/*
Dest address always in the form SMTP%"nnn@a.a" so always the
same length.
*/
address_inlist[0].buflen = 14 ;

for (i=1; i<=iterations; i++)
{
/* Add all 85 email addresses... */
if (iterations == i)
printf("\nAbout to add last address (#%d)", i) ;

sprintf(dest, "SMTP%%\042%03d@a.a\042", i) ;
result = MAIL$SEND_ADD_ADDRESS(&context, &address_inlist, &outlist) ;
if (!(result & 1))
exit(result) ;
}

add_bodypart_inlist[0].buflen = 3 ; /* Length of "NL:" */
printf("\nCalling MAIL$SEND_ADD_BODYPART") ;
printf("\nFencepost #1: %%X%08lX", fencepost1) ;
printf("\nFencepost #2: %%X%08lX", fencepost2) ;

result = MAIL$SEND_ADD_BODYPART(&context, &add_bodypart_inlist, &outlist) ;
if (!(result & 1))
exit(result) ;

printf("\nCalling MAIL$SEND_MESSAGE") ;
result = MAIL$SEND_MESSAGE(&context, &send_message_inlist, &outlist) ;
if (!(result & 1))
exit(result) ;

printf("\nMail sent!") ;
return (1) ;
}

$ CC /VERSION NL:
DEC C V5.5-002 on OpenVMS VAX V6.2
$ CC /STANDARD=VAXC /DEBUG REPRO.C
$ LINK /DEBUG REPRO.OBJ
$ REPRO :== $SYS$LOGIN:REPRO.EXE
$ REPRO "NOACCVIO"

OpenVMS VAX DEBUG Version V6.2-000

%DEBUG-I-INITIAL, language is C, module set to REPRO

DBG> GO

Calling MAIL$SEND_BEGIN
Calling MAIL$SEND_ADD_ATTRIBUTE
About to add last address (#84)
Calling MAIL$SEND_ADD_BODYPART
Fencepost #1: %XFEEDFACE
Fencepost #2: %XDEADBEEF
Calling MAIL$SEND_MESSAGE
Mail sent!
%DEBUG-I-EXITSTATUS, is '%SYSTEM-S-NORMAL, normal successful completion'
DBG> ^Z
$
$
$
$ UCX SHOW MAIL

SMTP Mail Queue Entry: 1000 User: username
File: _nodename$DKA100:[USERS.username]19040401455698_username.UCX_nodename;1
Status: Pending

Message Headers:

Return Path: username@nodename.domain
Date: Thu, 4 Apr 2019 01:45:57 GMT
Message Id: <19040401455698@nodename.domain>
From: username@nodename.domain
To: 001@a.a, 002@a.a, 003@a.a, 004@a.a, 005@a.a, 006@a.a, 007@a.a, 008@a.a, 009@a.a, 010@a.a, 011@a.a, 012@a.a, 013@a
.a, 014@a.a, 015@a.a, 016@a.a, 017@a.a, 018@a.a, 019@a.a, 020@a.a, 021@a.a, 022@a.a, 023@a.a, 024@a.a, 025@a.a, 026@a.a, 027@a.a, 02
8@a.a, 029@a.a, 030@a.a, 031@a.a, 032@a.a, 033@a.a, 034@a.a, 035@a.a, 036@a.a, 037@a.a, 038@a.a, 039@a.a, 040@a.a, 041@a.a, 042@a.a,
043@a.a, 044@a.a, 045@a.a, 046@a.a, 047@a.a, 048@a.a, 049@a.a, 050@a.a, 051@a.a, 052@a.a, 053@a.a, 054@a.a, 055@a.a, 056@a.a, 057@a
.a, 058@a.a, 059@a.a, 060@a.a, 061@a.a, 062@a.a, 063@a.a, 064@a.a, 065@a.a, 066@a.a, 067@a.a, 068@a.a, 069@a.a, 070@a.a, 071@a.a, 07
2@a.a, 073@a.a, 074@a.a, 075@a.a, 076@a.a, 077@a.a, 078@a.a, 079@a.a, 080@a.a, 081@a.a, 082@a.a, 083@a.a, 084@a.a
Subject: Callable Mail Send Test
X-VMS-To: Line too long.
$
$
$
$
$ TYPE _nodename$DKA100:[USERS.username]19040401455698_username.UCX_nodename_TEXT
Date: Thu, 4 Apr 2019 01:45:57 GMT
Message-Id: <19040401455698@nodename.domain>
From: username@nodename.domain
To: 001@a.a, 002@a.a, 003@a.a, 004@a.a, 005@a.a, 006@a.a, 007@a.a, 008@a.a, 009@a.a, 010@a.a, 011@a.a, 012@a.a, 013@a.a, 014@a.a, 01
5@a.a, 016@a.a, 017@a.a, 018@a.a, 019@a.a, 020@a.a, 021@a.a, 022@a.a, 023@a.a, 024@a.a, 025@a.a, 026@a.a, 027@a.a, 028@a.a, 029@a.a,
030@a.a, 031@a.a, 032@a.a, 033@a.a, 034@a.a, 035@a.a, 036@a.a, 037@a.a, 038@a.a, 039@a.a, 040@a.a, 041@a.a, 042@a.a, 043@a.a, 044@a
.a, 045@a.a, 046@a.a, 047@a.a, 048@a.a, 049@a.a, 050@a.a, 051@a.a, 052@a.a, 053@a.a, 054@a.a, 055@a.a, 056@a.a, 057@a.a, 058@a.a, 05
9@a.a, 060@a.a, 061@a.a, 062@a.a, 063@a.a, 064@a.a, 065@a.a, 066@a.a, 067@a.a, 068@a.a, 069@a.a, 070@a.a, 071@a.a, 072@a.a, 073@a.a,
074@a.a, 075@a.a, 076@a.a, 077@a.a, 078@a.a, 079@a.a, 080@a.a, 081@a.a, 082@a.a, 083@a.a, 084@a.a
Subject: Callable Mail Send Test
X-VMS-To: Line too long.
$
$
$
$
$ REPRO "ACCVIO"

OpenVMS VAX DEBUG Version V6.2-000

%DEBUG-I-INITIAL, language is C, module set to REPRO

DBG> go

Calling MAIL$SEND_BEGIN
Calling MAIL$SEND_ADD_ATTRIBUTE
About to add last address (#85)
Calling MAIL$SEND_ADD_BODYPART
Fencepost #1: %XFEEDFACE
Fencepost #2: %XDEADBEEF
Calling MAIL$SEND_MESSAGE
%DEBUG-E-LASTCHANCE, stack exception handlers lost, re-initializing stack
%SYSTEM-F-ACCVIO, access violation, reason mask=01, virtual address=38302225, PC=80000010, PSL=03C00004
break on unhandled exception at 2147483664
DBG>


The virtual address of 38302225?

$ A = "1234"
$ A[0,8]=%X38
$ A[8,8]=%X30
$ A[16,8]=%X22
$ A[24,8]=%X25
$ SHOW SYMBOL A
A = "80"%"

So, that would appear to be %"08 from an address of SMTP%"08n@A.A" (in reverse byte order).

Since the memory fenceposts aren't being trashed* prior to the call to MAIL$SEND_MESSAGE, I would guess that the ACCVIO occurs in either SYS$SHARE:MAILSHR.EXE or SYS$SHARE:UCX$SMTP_MAILSHR.EXE

*I've also increased them to 16KB buffers, initialised them, then checked all of the bytes before the MAIL$SEND_MESSAGE call, and they are still intact.

$ ANALYZE /IMAGE SYS$SHARE:UCX$SMTP_MAILSHR.EXE /OUT=A.A
$ SEARCH A.A "image file id","link date"
image file identification: "UCX V4.2-21H"
link date/time: 12-JAN-2000 15:10:40.94
$ SEARCH A.A "Shareable Image List" /WINDOW=(0,13)
Shareable Image List

0) this image
1) "UCX$SMTP_PARSESHR"
2) "LIBRTL"
3) "SCNRTL"
4) "UCX$ACCESS_SHR"
5) "MTHRTL"
6) "CMA$TIS_SHR"
7) "DECC$SHR"
8) "MAILSHR"
9) "MAILSHRP"
10) "SECURESHR"
11) "SECURESHRP"

With the reference to DECC$SHR.EXE, my guess would be that it is UCX$SMTP_MAILSHR.EXE in which the ACCVIO occurs rather than in SYS$SHARE:MAILSHR.EXE, given the propensity for shooting oneself in the foot in C.

Whether or not more recent versions of UCX / TCP/IP Services suffer from this same ACCVIO is another matter.

Mark

[Mark_Corcoran]

>No, precisely because the "To: " field is (only) part of the message header.  The actual recipients are specified using "RCPT TO: ", not "To: ".

Thanks Steven.  I've never really had much call to look into the specifics of SMTP, and maybe only once a year look at the  headers of an email that I've received.

I was in a rush to get a response posted, but if I'd spent the time to search (as I don't imagine that there is a specific section of (a UCX) M that can be RTFed), I'd probably have answered my own question.

I may have to have a "play" with the SMTP server that emails are relayed through - it could be the internal SMTP relay that is barfing once the RCPT TO length (or number of recipients) exceeds a certain threshold, rather than it being a limitation within UCX.

My recollection from trying to do this in the past however, is that some SMTP servers might hang up the connection if they don't get commands within a small threshold amount of time (i.e. one has to be able to copy & paste;  even comparatively fast touch typists might struggle to type email addresses quickly enough).

The other issue is the ACCVIO when adding lots of addresses using callable mail - although if there were a limitation on the SMTP gateway and we were given an exception to exceed that limit, we could continue to use MAIL from the .COM files rather than run an executable image to call MAIL$ routines...

Mark

[Steven Schweda]

> [...] the web site always insists on stripping leading spaces on
> Paste, [...]

   It's worse.  More like F$EDIT( , "COMPRESS, TRIM").  It's annoying,
but there's a "</>" "Insert Code" control in the RICH TEXT editor which
might help.  (Still annoying, though.)

> [...] I'm none too keen on how it implements carriage
> returns/paragrpahs/double-spacing either [...]

   Try shift+return.  (Still annoying, though.)