topic Re: Script help needed to parse /var/mail files in Operating System - HP-UX

Script help needed to parse /var/mail files

TheJuiceman — Thu, 19 Jun 2008 17:36:27 GMT

Hi gang,

I need a script that will go through /var/mail and pull "pertinent" information about failed email attempts that I can send to some I.S. personnel (info like the from user, the recipient, the subject, and the error message). I would like to extract only failed sent messages from /var/mail and exclude other messages in the mailboxes. And I would like to exclude all of the MIME info and that stuff that really is of no "value" to knowing what was sent that did not go. I would like to have a script that would do this once a day and only show the information I am looking for that was put there the current day (so once they are aware that an email was rejected that they are not notified about it everyday until the mailbox is emptied). It would also help if this information could be put into a single file (versus creating a file for each entry) and "prettied up" so that I can email it each day.

Something like this would be great....

*********************************
user1:
----- The following addresses had permanent fatal errors -----
recipient@somewhere.com... User unknown
Subject: This is my email
*********************************
user1:
----- The following addresses had permanent fatal errors -----
recipient@somewhere.com... User unknown
Subject: This is my email
*********************************
user2:
you see where this is going

Can any of you scripting gurus help me? (not asking for much am I? LOL) Thanks a million!!!

Re: Script help needed to parse /var/mail files

James R. Ferguson — Thu, 19 Jun 2008 17:43:53 GMT

Hi:

What have you attempted thus far? Why not post what you have and where you're stuck?

Regards!

...JRF...

Re: Script help needed to parse /var/mail files

Tim Nelson — Thu, 19 Jun 2008 19:50:06 GMT

How about looking in the mail log ?

/var/adm/syslog/mail.log

if it is not on then turn it on with the sylog config ?

Or.. take all the files and email them to the person requesting this and let them parse it on their own ( only joking :)

Re: Script help needed to parse /var/mail files

TheJuiceman — Thu, 19 Jun 2008 19:50:32 GMT

Hi James,

What I have is VERY crude, but this is basically what I have. It looks NOTHING like what I have above. That is what I would LIKE it to look like if possible. Here's roughly what I have....

find /var/mail -type f -print | xargs egrep -i 'Final-Recipient:|Subject:|THIS IS A WARN
ING MESSAGE ONLY|YOU DO NOT NEED TO RESEND YOUR MESSAGE|errors|User unknown|User
-Agent' | egrep -v 'Subject: at|root|cron|Returned' > /tmp/file

Then I check to see if the file exists and then email it. I pretty it up a little but not enough to make everyone happy.

Re: Script help needed to parse /var/mail files

TheJuiceman — Thu, 19 Jun 2008 20:06:55 GMT

Hi Tim,

I do have the mail.log enabled, but it does not give enough information for what is needed. They really need the subject line to be able to track down what did not go out and where it was generated.

Re: Script help needed to parse /var/mail files

James R. Ferguson — Thu, 19 Jun 2008 20:20:22 GMT

Hi (again):

As Tim hinted, you need to begin by parsing the '/var/adm/syslog/mail.log'. For example, if I parse one of mine for the word 'error' i find a line that (sanitized) looks like:

Jun 18 19:14:59 6C:myhostname sendmail[5900]: TAA05900: to=somebody@xyz.net', delay=00:01:15, mailer=relay, relay=excpap01.xyz.net, stat=Transient parse error -- message queued for future delivery

*Now*, if in turn, I parse out the '/var/spool/mqueue' file to which this record points ---- that's the "TAA05900" field --- I would find:

ls -l /var/spool/mqueue/*TAA05900
-rw------- 1 root mail 283 Jun 18 19:14 /var/spool/mqueue/dfTAA05900
-rw------- 1 root mail 494 Jan 18 19:14 /var/spool/mqueue/qfTAA05900

Then, the 'dfTAA05900' file contains the recipient(s) and the subject for which you are looking.

Regards!

...JRF...

Re: Script help needed to parse /var/mail files

TheJuiceman — Thu, 19 Jun 2008 21:05:32 GMT

Hi James,

The mqueue only shows items that have not sent. I need to get information on things that sent but failed. Is there a similar way to get this information? Thank you both again!!!

Oh, and I will submit points at the end. Thanks again.

Re: Script help needed to parse /var/mail files

James R. Ferguson — Thu, 19 Jun 2008 21:39:18 GMT

Hi (again):

> The mqueue only shows items that have not sent.

That would be true. Messages remain in this queue (directory) until they are successfull sent, at which time they are then removed.

Regards!

...JRF...

Re: Script help needed to parse /var/mail files

TheJuiceman — Thu, 19 Jun 2008 21:45:43 GMT

Hi again,

Yeah, I need to get messages that leave the server but do not get delivered due to things like the recipient is not valid, etc. /var/mail will populate with this information. I just need a way to sort through the mail and put it in a user friendly form.

Re: Script help needed to parse /var/mail files

OFC_EDM — Fri, 20 Jun 2008 06:32:53 GMT

Again I don't have access to a system to test this but....

Can you cross reference the failure messages in the mail.log file with the users mbox on the server?

I'm assuming sent mail is stored in the users mailbox.

So if you have the users name you know the users mailbox. The mailbox is a text file so you could search their mailbox for the original message.

I'm made a few assumptions. Hope they apply to your environment.

Cheers

Re: Script help needed to parse /var/mail files

OFC_EDM — Fri, 20 Jun 2008 06:50:54 GMT

Found this in the sendmail.cf readme. I've never used this setting so I don't know if it's what you need.

But at first glance it shows potential

confDEAD_LETTER_DROP DeadLetterDrop

[undefined] Filename to save bounce messages which could not be returned to the user or sent to postmaster. If not set, the queue file will be renamed.

I got this information from
http://www.sendmail.org/m4/tweaking_config.html

Re: Script help needed to parse /var/mail files

TheJuiceman — Fri, 20 Jun 2008 21:13:09 GMT

Well, the big problem isn't finding the messages. They are in /var/mail. The problem is when you look at them, there is a bunch of MIME information and stuff that the average user can't make heads or tails out of. All they want to know is 1) who sent an email that got bounced back 2) what was the email and 3) who was it suppose to go to. Doing the cross-referencing thing doesn't seem to get me anything more than I can get directly from /var/mail. I just need a way to get a user friendly way of pulling out the information they want.

Re: Script help needed to parse /var/mail files

Hein van den Heuvel — Sat, 21 Jun 2008 01:53:32 GMT

Well, you'll have to learn about the mail file layout. I never seriously looked at it, but it looks recognizable allright.

A script would have to look for stuff 'in order' as well as 'carefully'.

For example, it is not really interested in a line with "Subject:" unless it is at the start of a line, and not until it has seen a line with "permanent failure".
So the grep needs to be with 'anchored' strings as not to trigger on seemingly right pieces of strings in the wrong place.

The script will probably have to protect itself again NOT finding what it expects, and reset itself instead of blindly looking for strings which are not going to come anymore.

There are many, many ways to do this.
Below a sample implementation using perl and using a 'step' variable to remember what it is looking for next.

It works on my mail file, as per below.

It might only work on my mail file.
That's for you to find out.
And you would need to tweak it to go over all mail files, and format the report 'just so'. (That part is 'work', which I will gladly do.. for the right compensation. :-)

hth,
Hein.

$ perl permanent_fatal.pl /var/mail/hein
550 5.1.2 hein@nnosuchdomain.xyz... Host unknown (Name server: nosuchdomain.xyz: host not found)
Use of uninitialized value in print at permanent_fatal.pl line 29, <> line 75.
From: Hein van den Heuvel
Date: Fri, 20 Jun 2008 21:15:29 -0400 (EDT)

550 5.1.2 hein@wergfwegwe.org... Host unknown (Name server: wergfwegwe.org: host not found)
Subject: this is a test subject
From: Hein van den Heuvel
Date: Fri, 20 Jun 2008 21:20:18 -0400 (EDT)

$ cat permanent_fatal.pl
use strict;
use warnings;
my $step=0;
my $id=0;
my $old=1;
my ($problem, $subject, $from, $date);
while (<>) {
#debug print "$step $.\n" if $step != $old;
#debug old = $step;
if (0==$step) {
next unless /^ --.*permanent fatal.*-$/;
$step++;
}
elsif (1==$step) {
next unless /^ ----- Transcript.*-$/;
$problem = <>;
$step++;
}
elsif (2==$step) {
next unless /^--(\S+)$/;
$id = $1;
$step++;
}
elsif (3==$step) {
$subject = $_ if /^Subject:/;
$from = $_ if /^From:/;
$date = $_ if /^Date:/;
if (/^--${id}--$/) {
print $problem, $subject, $from, $date, "\n";
$step = 0; # NOT confused. Done. Start again.
}
if (/^--(\S+)$/) {
$step = 0 unless $1 eq $id ; # Confused? Start again.
}
}
}

Re: Script help needed to parse /var/mail files

Hein van den Heuvel — Sat, 21 Jun 2008 02:22:15 GMT

Oh, and in case you wonder about:

"Use of uninitialized value in print at permanent_fatal.pl line 29, <> line 75."

That is thanks to 'use warnings'.
The first refused message had no 'subject' line, so the $subjetc variable was not set up. A real solution would have to protect against that, reseting all message data after a print. Easy enough, but it needs to be done.

Hein.

Re: Script help needed to parse /var/mail files

TheJuiceman — Mon, 23 Jun 2008 23:48:58 GMT

Thanks Hein. I'm looking at your code and it is all pretty greek to me. Here is an example of what a returned mail might look like in /var/mail. Each message is separated by the line "From MAILER-DAEMON..."

From MAILER-DAEMON@domain.com Mon Jun 23 16:36:14 EDT 2008
Received: from localhost (localhost)
by local.domain.com (8.9.3 (PHNE_35950)/8.9.3) with internal id Q
AB24698;
Mon, 23 Jun 2008 16:36:14 -0400 (EDT)
Date: Mon, 23 Jun 2008 16:36:14 -0400 (EDT)
From: Mail Delivery Subsystem
Message-Id: <200806232036.QAB24698@local.domain.com>
To: sender@local.domain.com
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
boundary="QAB24698.1214253374/local.domain.com"
Subject: Returned mail: User unknown
Auto-Submitted: auto-generated (failure)

This is a MIME-encapsulated message

--QAB24698.1214253374/local.domain.com

The original message was received at Mon, 23 Jun 2008 16:36:14 -0400 (EDT)
from sender@localhost

----- The following addresses had permanent fatal errors -----
recipient@local.domain.com

----- Transcript of session follows -----
550 recipient@local.domain.com... User unknown

--QAB24698.1214253374/local.domain.com
Content-Type: message/delivery-status

Reporting-MTA: dns; local.domain.com
Arrival-Date: Mon, 23 Jun 2008 16:36:14 -0400 (EDT)

Final-Recipient: RFC822; recipient@local.domain.com
Action: failed
Status: 5.1.1
Last-Attempt-Date: Mon, 23 Jun 2008 16:36:14 -0400 (EDT)

--QAB24698.1214253374/local.domain.com
Content-Type: message/rfc822

Return-Path:
Received: (from sender@localhost)
by local.domain.com (8.9.3 (PHNE_35950)/8.9.3) id QAA24698;
Mon, 23 Jun 2008 16:36:14 -0400 (EDT)
Date: Mon, 23 Jun 2008 16:36:13 -0400
From: Sender's Name
To: recipient@foreigndomain.com
Subject: Email subject
Message-ID: <20080623203613.GA24696@local.domain.com>
Mime-Version: 1.0
Content-Type: multipart/mixed; boundary="qDbXVdCdHGoSgWSk"
Content-Disposition: inline
User-Agent: Mutt/1.4.2.1i

--qDbXVdCdHGoSgWSk
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Body of email

--qDbXVdCdHGoSgWSk
Content-Type: text/html; charset=us-ascii
Content-Disposition: attachment; filename="attachment.htm"

--qDbXVdCdHGoSgWSk--

--QAB24698.1214253374/local.domain.com--

From MAILER-DAEMON@local.domain.com Mon Jun 23 16:36:14 EDT 2008

How can I modify the script to look for information the user would want like ...

Subject: Returned mail: User unknown
From: Sender's Name
To: recipient@foreigndomain.com
Subject: Email subject

Thanks again.

Re: Script help needed to parse /var/mail files

Hein van den Heuvel — Tue, 24 Jun 2008 07:37:10 GMT

>> I'm looking at your code and it is all pretty greek to me.

If it was easy you would have had many more replies -). There are several lines starting with "From:" and "Subject:"... which ones to pick.

>> Here is an example of what a returned mail might look like in /var/mail.

Thanks. Cut&paste into the forum drops spaces, and maybe added spaces as the line end. I adapted my example to tolerate that, but an 'attached' text file would have been better.

>> Each message is separated by the line "c..."

That's what you observe/say. And it may be true. But like I said I knew nothing about the Email format and decided to trigger on the 'permanent fatal' line. :-). I thought, rightly or wrongly so, that those might perhaps come from other sources and also that maybe MAILER-DEAMON might send out other type message which need not be considered.

I changed the cript some to be more clear about looking for 'id' lines. In your case:
--QAB24698.1214253374/local.domain.com
--QAB24698.1214253374/local.domain.com--

Try it. Maybe it does what you want, maybe not. It works for me on your example and my file.

use strict;
use warnings;
my $step=0;
my $id=0;
my $old=1;
my ($problem, $subject, $from, $to, $real_subject);
while (<>) {
#debug print "$step $.\n" if $step != $old;
#debug $old = $step;

if (/^--(\S+\/\S+?)(-?-?)\s?$/) {
if ($1 ne $id) { # New ID? Clear all memory
$id = $1;
$step = 0;
$from = q()."\n";
$to = q()."\n";
$real_subject = q()."\n";
} else {
if ($2) { # Found closing marker
print $problem, $subject, $from, $to, $real_subject, "\n";
$step = 0; # Done. Start again.
}
}
}
if (0==$step) {
$subject = $_ if /^Subject:/;
next unless /^[ -]+.*permanent fatal.*-\s?$/;
$step++;
}
elsif (1==$step) {
next unless /^[ -]+Transcript.*-\s?$/;
$problem = <>;
$step++;
}
elsif (2==$step) {
$from = $_ if /^From:/;
$to = $_ if /^To:/;
$real_subject = $_ if /^Subject:/;
}
}

Re: Script help needed to parse /var/mail files

TheJuiceman — Wed, 25 Jun 2008 21:32:17 GMT

Hello Hein,

Sorry for the delay getting back. The updated script seems to be on the right track!!! I'm just wanting and checking for variations that I need to address. So far this is a good start!!! I will report my progress and reward points very soon. Thank you for your help

Re: Script help needed to parse /var/mail files

TheJuiceman — Wed, 02 Jul 2008 21:48:14 GMT

Hello Hein,

The first script seemed to work, it just did not get everything. The second one I am getting errors such as...

./permanent_fatal.pl[2]: use: not found.
./permanent_fatal.pl[3]: use: not found.
./permanent_fatal.pl[4]: my: not found.
./permanent_fatal.pl[5]: my: not found.
./permanent_fatal.pl[6]: my: not found.
./permanent_fatal.pl[7]: Syntax error at line 7 : `(' is not expected.

Can you send the script as an attachment so I can make sure it is correct? Thanks again.

Re: Script help needed to parse /var/mail files

James R. Ferguson — Wed, 02 Jul 2008 22:15:17 GMT

Hi :

> The first script seemed to work, it just did not get everything. The second one I am getting errors such as...

You need to put the "shebang" line at the head so that the Perl interpreter is used:

#!/usr/bin/perl

...assuming that '/usr/bin/perl' is where your Perl lives or is a symbolic link to it.

Without this, the shell thinks it is going to run another shell script. Hence the shell interpreter stumbles and chokes on the "my" and "use" tokens!

Regards!

...JRF...

Re: Script help needed to parse /var/mail files

Patrick Wallek — Wed, 02 Jul 2008 22:20:54 GMT

You can also run the script from the command line by using perl as the interpreter, such as:

# perl permanent_fatal.pl /var/mail/hein

(This was taken from Hein's first post.)