using callable email to send a pdf attachment (uuencoded) to Outlook

[Rich Bordewick]

Hi,

I use uuencode to send text file reports to create Word attachments in Outlook, no problem.

I have recently modified my uuencode subroutine to created a pdf attachment of a file created by using PCL2PDF.  I am using callable email to send this to Outlook.

If the PDF attachment is small (less than six pages) I have success.  If it gets beyond six pages the pdf file becomes corrupt after around page six.  It may or may not always be six pages, but that is the general area of the corruption.

Does anyone have experience with uuencode and pdf files?  Any suggestions where I might look for a solution?

Thanks

[Dennis Handly]

Can you checksum your uuencoded attachment before you send it and after to see if it is the mail system or client that is corrupting it?

Can you uudecode the file on your source system and compare to your original file?

[Hoff]

There's far too much left out of this question.  I can't tell if this question involves OpenVMS except that you've asked a few related questions, or what OpenVMS version and patches, nor what code and tools and libraries are being used, nor what the raw message contents are, nor what generated the PDF and if that code has been tested...

Some options...

• Verify that this isn't some setting on the specific mail server involved; send the message to a Postfix or OpenSMTPd server, and see if that works.
• Dump out the message contents at the mail server, and see what's present in the file; analyze the MIME encoding and verify the PDF.  You'll want to ask the Microsoft folks how this logging can be done with Windows Server, but most mail servers can do this.  Or dump the messages from a different server.
• Send the ASCII text data somewhere else to be processed and mailed.

The MIME support on OpenVMS — if that's how you're generating these messages, followed by SFF or analogous  — has seen various patches to fix bugs in "less-than-current" OpenVMS versions.  Make sure you're patched to current. 

As an alternative writing and debugging and supporting your own code (if that's what has happened here), an alternative might involve use an open-source library to do this processing, maybe using Python tools or a mutt port.

Or post the MUA, MIME and PDF source code and some configuration details and steps to reproduce, and maybe somebody here can have a look...

[Steven Schweda]

> I use uuencode to send text file reports to create Word attachments in
> Outlook, no problem.

   What are "Word attachments"?  I'll assume that you mean e-mail
attachments, and that you're reading the e-mail using Outlook, not
creating the attachments in Outlook.  Uuencode is a little stone-age
these days, compared with, say, MIME, isn't it?

> I have recently modified my uuencode subroutine [...]

   The lack of details in that description is frightening.

> Does anyone have experience with uuencode and pdf files?

   Not really, but it should be relatively easy to run uuencode and
uudecode, and see if you get the original data back, before you drag
e-mail into the picture.  If uuencode+uudecode appears to work, then the
VMS MIME utility (which I use only for reading, not creating, so I know
nothing) or mpack could be used to create MIME instead of (stone-age)
uuencode stuff to send.

   A VMS-compatible mpack (which could work, but, again, I've used it
mostly for reading) should be available at:

      http://antinode.info/dec/sw/mpack.html

> Or post the MUA, MIME and PDF source code and some configuration
> details and steps to reproduce, [...]

   Yup.  Some actual information might make problem solving easier.

[Rich Bordewick]

Thanks for the replys,

Operating system is OpenVMS V8.3-1H1 on an Itanium 64

In house programs written with I64 Basic V1.7

PCL2PDF was purchased from a vendor.

I have a UUENCODE subroutine to attach text files to emails (MS Word, CSV, TXT, and now PDF attachments).  The PDF has success if less than six pages are created, and corrupts after that.

The form is a p/o created using pcl code.  I can transfer the output of the PCL2PDF by using my terminal emulator file transfer, and Adobe reader is good with that.  

I did send the output to Hotmail for a test and all the pages of the attachment were readable using MS Word Online.  Although, some of the form did not come through on three pages, and page seven was one of those pages.

Perhaps I am on the wrong forum.

Thanks

[Steven Schweda]

> Can you uudecode the file on your source system and compare to your
> original file?

   Still wondering.  As I read your description, something goes wrong if
you use uuencode and e-mail.  It's not clear to me whether the problem
lies in the uuencode part or the e-mail part of that operation.  (I'd be
trying to use MIME instead of uuencode, but uuencode may also work (if
done correctly).)

   None of the non-psychics here can see your data unless you exhibit
them, so, for now, only you can run the uuencode+uudecode test.

> I can transfer the output of the PCL2PDF by using my terminal emulator
> file transfer [...]

   Can you do that with the uuencoded data, too, and, after doing
uudecode at the destination, can you recover good data?

> I have a UUENCODE subroutine to attach text files to emails [...]

   Can't see the code.  Don't know if it works or not.  Can't see the

file sizes or other attributes.  Can't see the original data.  Can't see
the uuencoded data.  Can't see the e-mail message.  Can't see the
attachment after it's been extracted from the e-mail message.  Can't see
how I'm supposed to guess what the problem might be without any useful
information.

> I did send the output to Hotmail

   You did _what_, exactly?

>  for a test and all the pages of the
> attachment were readable using MS Word Online.  Although, some of the
> form did not come through on three pages, and page seven was one of
> those pages.

   Once again, the e-mail appears to be corrupt, but that's not news.
Rather than ask "MS Word Online" to verify the data, I might extract the
attachment, and use a differences program to compare it with the
original file.

> Perhaps I am on the wrong forum.

   Not certain, but possible.  If you can't provide any actual data, and
you can't break up the task into separately testable operations, and
test them, then you may need help from another spiritual plane.

[Hoff]

So to summarize, you're using some sort of probably-antique OpenVMS application which generates HP-printer-specific PCL code and where there's apparently have no access to something closer to useful than PCL; that this configuration intercepts the transport downstream of the PCL printer conversion, and converts that to PDF, and then that PDF goes through processing prior to being mailed, and that processing somehow involves uuencode.

I'll assume that there's no way to get at the file before the conversion to PCL here, though — if there is a way to do that, and you can use one of the available print symbionts to capture the text between the queue operations in the application and the conversion operation in the symbiont — if it's a symbiont where the conversion is being performed, and not the application — then there may be alternatives.  (I'd much rather capture and use ASCII text than post-process PCL back to something readable.  But I'll assume that there's a reason that this unspecified PCL2PDF tool was purchased, so...)

You're using a purchased third-party product for PCL to PDF conversion; this PCL2PDF tool, and this tool is operating with an unknown set of commands or sequence or operations, and somehow involving uuencode.   Given that what are apparently FTP or SFTP file transfers — I'm guessing that's what your "terminal emulator" part of the discussion references — seem to work, at least the basic operations of that tool appear to be working okay.

You're using unspecified mechanisms and commands for sending the files to the target system, though you're using uuencode as part of that sequence.   That sequence — were you're using uuencode and unspecified commands — to send the file to remote servers may or may not be the central issue here.

End summary...

Caveat...

Note that I'm *guessing* here.  I *really* dislike doing that too, as I could very well be repeating what you already know and/or have already tried and wasting everybody's time, or I could guess wrong and lead you or other subsequent readers down a completely wrong path.   This is why I and other folks ask for specific details.  Not to be nags.

Okay... Sending attachments on mail messages from OpenVMS.   That's usually performed with the MIME utility...

To invoke MIME, typical would be to use the DCL$PATH automatic foreign commands mechanism or to set up a traditional foreign command with the following DCL symbol:

$ MIME :== $MIME

Then invoke the MIME utility and use the following commands:

$ MIME

MIME> NEW {mime-filename}

MIME> ADD {pdf-filename} /ENCODING=BASE64 /CONTENT="APPLICATION/PDF"

MIME> CLOSE

$

Then using the SFF Send From File utility to send the {mime-filename} file to the intended recipient.   (Details on SFF; on the TCPIP$SYSTEM:TCPIP$SMTP_SFF.EXE utility.)

The encoding and content may need to be varied; the above attempts to set the details to what a PDF reader would expect. As an alternative, removing /ENCODING and /CONTENT and using /BINARY might also work, but I've not tried that.

There are some additional details on the MIME utility in the OpenVMS manuals, including in the OpenVMS User's Manual.  Also a few postings around the 'net.   Also see the available Overview section in the MIME utility HELP:

$ MIME

MIME> Help Overview

...

Note too that I'd use the SFF utility here and not the MAIL utility directly.   The MAIL utility and the associated pieces of TCP/IP Services that process the message on behalf of MAIL have also had some bugs around blank line handling over the years, and the SFF utility has been a somewhat more reliable path for sending out MIME-encoded messages.

It's also possible that the PCL2PDF conversion is somehow going wrong here, or that the conversion is also going wrong here, as different PDF viewers can have different PDF format expectations.   If using MIME doesn't get results, then discussions directly with the PCL2PDF vendor might be in order, particularly if you can provide a sequence that reproduces the error for the vendor.

You'll note that I'm not using uuencode here — that's not a tool I've particularly encountered in the context of MIME-encoded messages, though there's probably somebody somewhere using it for that.  AFAIK, tools or approaches or mechanisms such as MIME have largely replaced the use of uuencode and uudecode around the 'net.

Now as for testing this transfer and this sequence, that's been briefly discussed in previous replies — dump out the raw contents of mail messages that work and quite possibly messages from other mail clients, and dump out those messages and encodings that don't, and compare.   Look for differences with the MIME encoding and the content settings, in particular.  Also extract the sections of the MIME message attachments, and see if those work or if there are corruptions.  Test with a few formats and likely with some known-good PDF files other than the PCL2PDF-generated files, to try to isolate the (mis)behavior.  (There are many different pieces in play in this particular sequence, and that makes for much more complex discussions and also for walls-of-words replies such as this one.)

You're also on a rather older OpenVMS version.  You can see which patches are installed with PRODUCT commands; PRODUCT SHOW PRODUCT VMS /FULL and PRODUCT SHOW PRODUCT TCPIP /FULL will list the installed patches for OpenVMS and for TCP/IP Services — you'll need access to the patch area to see if those are current, and whether any of those patches apply to TCP/IP Services or to MIME or whatever it is that you're using here.   OpenVMS I64 V8.4 around UPDATE V10.0 with TCP/IP Services V5.7 around ECO 5 is current.   Why update to current?  The MIME utility has had some bugs.  As have various other parts of OpenVMS and TCP/IP Services, too.

I'd also generally encourage you to acquire some sort of escalation support here; an entity that specializes in OpenVMS and that can help you wade through these parts of OpenVMS and the core layered products, either to get these messages sent out, or that can help isolate and reproduce the bugs that you might find for HP or for VSI.

I'm fully expecting that the URLs referenced above will go stale shortly.   This given the impending corporate split into HP and HPE.  Hopefully there's enough context in this message to search for the information at HPE and — when they finally get their own web site sorted out — over at the VMS Software Inc (VSI) web site.

But again, all of the above is a guess — without the tools and command sequences that you're using, that's the best that I can do, and I may well guess wrong here.

Ah, hello my old friend "Your post has been changed because invalid HTML was found in the message body. The invalid HTML has been removed. Please review the message and submit the message when you are satisfied." — nice to that this forum software is as sterling as ever. 

[Steven Schweda]

> Note too that I'd use the SFF utility here and not the MAIL utility
> directly. [...]

   I was guessing that "callable email" meant that, but what do I know?
(Among the many vague bits of this problem description, that one seemed
less important than others.)

> You'll note that I'm not using uuencode here - that's not a tool I've
> particularly encountered in the context of MIME-encoded messages, [...]

   I haven't done it, so I know nothing, but I gather that uuencode was
a common encoding scheme for non-text in e-mail before MIME was
invented.  Apparently, many/most modern e-mail clients still support it.
I've never seen uuencode used _with_ MIME (but I don't get out much).
Base64 is the common encoding scheme used with MIME, I gather.

[Hoff]

Yes, uuencode and uudecode were used an eon or two ago, when you were trying to send a wad of data as the mail message, or through some other Unix-to-Unix means of file transfers that expected ASCII text.

With most "modern" mail clients, that's usually all been replaced by MIME-encoded messages, and often multi-part MIME messages.

As for whether the callable mail API or the Mail utility or SFF utility or something else is being here by the OP, that's not yet clear to me.

[MarkOfAus]

> haven't done it, so I know nothing, but I gather that uuencode was
>a common encoding scheme for non-text in e-mail before MIME was
>invented.  Apparently, many/most modern e-mail clients still support it.

If I transport from VMS to MS Exchange using a uuencoded message/embedded file, it places into the header:

MIME-Version: 1.0
...

X-ConvertedToMime: 1

So, yes, it does MIME it.

[MarkOfAus]

We have/had some legacy interactions that required uuencode and mail a while back and we found they did seem to play havoc when using PDF files.

For example, using the available command: $SYS$SYSTEM:TCPIP$UUENCODE.EXE

and an example PDF file: B13681.PDF (B13681.PDF;4                       3.36MB/3.37MB)

Create a symbol:  UUENCODE == "$SYS$SYSTEM:TCPIP$UUENCODE.EXE"

$ uuencode B13681.PDF B13681.UUE

Renders a file:

B13681.UUE;2                       2.15MB/2.25MB

Then transport it:

$ mail /subject="Test" B13681.UUE smtp%"my.email.address"

It arrives at MS Exchange as garbage (corrupted somewhere).

However, if I zip the pdf up (using zip from one of the Freeware CDs), and uuencode it again:

$ uuencode B13681.ZIP B13681.UUE

The resulting file is:
B13681.UUE;3                       1.58MB/1.68MB

I then transport it:

$ mail /subject="Test" B13681.UUE smtp%"my.email.address"

It arrives, is unzipped and looks/feels and acts like a PDF.

Conclusion? Something happens with pdfs and uuencode. Try to zip it up and see if it changes the outcome.

Alternatively, move to MIME as Hoff suggests.

[H.Becker]

What's the record format of the PDF?

I can reproduce a corruption with a PDF in "Stream" format.  After the uuencode-uudecode sequence the first difference is a 0x0d 0x0a in the original PDF and a 0x0a in the uudecoded PDF.

When I do a SET FILE/ATTRIBUTE with RFM=STMLF to the original file this difference no longer shows (the uudecoded file has an additional 0x0a at the end of the file, anything else is as in the original PDF).

Using RFM=FIX and LRL=512 seem sto have the same effect for the CR LF content, but then the additional trailing 0x0a is in its own record/disk block.

(Using UUxxCODE from HP TCP/IP Services for OpenVMS Alpha Version V5.7 - ECO 5)

[Rich Bordewick]

Hi,

I would like to appoligize for making you guess.  I was not really sure where my problem was originating.

The purpose of my changes are an attempt to get away from paper.  The users would print a p/o on a laser printer, scan it, and the email to a vendor.  The head office would also print copies of the purchase orders and mail them to other office locations.  Email makes this all simple.

The purchase order form report gets created with the p/o form macro number reference and the purchase order detail.  This p/o report file is then converted using an FDL to set the record format to Stream_LF (recommeded by the PCL2PDF vendor, and the test skipping this step produced unusable PDF's).  The converted report is then input into the PCL2PDF conversion program along with the PCL form macro file.  The PDF output file is read by my email subroutines and converted by a UUENCODE subroutine to create an email attachment.

What I referred to as "callable email"  are OpenVMS Mail Utility Routines.

The record format of the PCL2PDF output PDF file is Stream_LF, maximum 0 bytes, longest 32767.  I believe the longest physical record is around 5000-6000 characters long.  My test file had only 5 lines for 21 purchase orders.

I have spent some time changing the process to use MIME and have some output.  However, when it gets to Outlook the MIME file is not an attachment, but is in the body of the email and binary.

My I64 system is unlikely to be upgraded any time soon, and more likely to be replaced in the future by a new and unknown system.

Thank you again for your patience and suggestions.

[Rich Bordewick · Accepted solution]

---

@MarkOfAus wrote:

We have/had some legacy interactions that required uuencode and mail a while back and we found they did seem to play havoc when using PDF files.

For example, using the available command: $SYS$SYSTEM:TCPIP$UUENCODE.EXE

and an example PDF file: B13681.PDF (B13681.PDF;4                       3.36MB/3.37MB)

Create a symbol:  UUENCODE == "$SYS$SYSTEM:TCPIP$UUENCODE.EXE"

$ uuencode B13681.PDF B13681.UUE

Renders a file:

B13681.UUE;2                       2.15MB/2.25MB

Then transport it:

$ mail /subject="Test" B13681.UUE smtp%"my.email.address"

It arrives at MS Exchange as garbage (corrupted somewhere).

However, if I zip the pdf up (using zip from one of the Freeware CDs), and uuencode it again:

$ uuencode B13681.ZIP B13681.UUE

The resulting file is:
B13681.UUE;3                       1.58MB/1.68MB

I then transport it:

$ mail /subject="Test" B13681.UUE smtp%"my.email.address"

It arrives, is unzipped and looks/feels and acts like a PDF.

Conclusion? Something happens with pdfs and uuencode. Try to zip it up and see if it changes the outcome.

Alternatively, move to MIME as Hoff suggests.

---

Winner!

I don't have a zip on my I64 at this time.  There were some minor errors in the attachment, lost some of the form graphics. I will look into getting a zip program.

Again, thank you all.

[Hoff]

Please use the right tool: MIME.

Yes, you can use a hammer to drive in screws, but it just doesn't work very well.  

...zip for OpenVMS

[John Gillings]

Rich,

    This sounds hauntingly familiar... I can't find the details at the moment, but I remember hitting a strange magic number when using UUENCODE and MAIL. Messages over the magic size get extra characters inserted, which looks like corruption. I couldn't find a better workaround than using ZIP first.

ZIP can be relied on to correctly pack lines into predictable size chunks.

If you have some time to waste, you can determine the exact limit using a binary search, or digging through the binary contents of your messages. If your file is smaller than the magic number, you can post it as is, otherwise zip if first.