Script help needed to parse /var/mail files

[TheJuiceman]

Hi gang,

I need a script that will go through /var/mail and pull "pertinent" information about failed email attempts that I can send to some I.S. personnel (info like the from user, the recipient, the subject, and the error message). I would like to extract only failed sent messages from /var/mail and exclude other messages in the mailboxes. And I would like to exclude all of the MIME info and that stuff that really is of no "value" to knowing what was sent that did not go. I would like to have a script that would do this once a day and only show the information I am looking for that was put there the current day (so once they are aware that an email was rejected that they are not notified about it everyday until the mailbox is emptied). It would also help if this information could be put into a single file (versus creating a file for each entry) and "prettied up" so that I can email it each day.

Something like this would be great....

*********************************
user1:
----- The following addresses had permanent fatal errors -----
recipient@somewhere.com... User unknown
Subject: This is my email
*********************************
user1:
----- The following addresses had permanent fatal errors -----
recipient@somewhere.com... User unknown
Subject: This is my email
*********************************
user2:
you see where this is going


Can any of you scripting gurus help me? (not asking for much am I? LOL) Thanks a million!!!

[James R. Ferguson]

Hi:

What have you attempted thus far? Why not post what you have and where you're stuck?

Regards!

...JRF...

[Tim Nelson]

How about looking in the mail log ?

/var/adm/syslog/mail.log

if it is not on then turn it on with the sylog config ?

Or.. take all the files and email them to the person requesting this and let them parse it on their own ( only joking :)

[TheJuiceman]

Hi James,

What I have is VERY crude, but this is basically what I have. It looks NOTHING like what I have above. That is what I would LIKE it to look like if possible. Here's roughly what I have....

find /var/mail -type f -print | xargs egrep -i 'Final-Recipient:|Subject:|THIS IS A WARN
ING MESSAGE ONLY|YOU DO NOT NEED TO RESEND YOUR MESSAGE|errors|User unknown|User
-Agent' | egrep -v 'Subject: at|root|cron|Returned' > /tmp/file

Then I check to see if the file exists and then email it. I pretty it up a little but not enough to make everyone happy.

[TheJuiceman]

Hi Tim,

I do have the mail.log enabled, but it does not give enough information for what is needed. They really need the subject line to be able to track down what did not go out and where it was generated.

[James R. Ferguson]

Hi (again):

As Tim hinted, you need to begin by parsing the '/var/adm/syslog/mail.log'. For example, if I parse one of mine for the word 'error' i find a line that (sanitized) looks like:

Jun 18 19:14:59 6C:myhostname sendmail[5900]: TAA05900: to=somebody@xyz.net', delay=00:01:15, mailer=relay, relay=excpap01.xyz.net, stat=Transient parse error -- message queued for future delivery

*Now*, if in turn, I parse out the '/var/spool/mqueue' file to which this record points ---- that's the "TAA05900" field --- I would find:

ls -l /var/spool/mqueue/*TAA05900
-rw------- 1 root mail 283 Jun 18 19:14 /var/spool/mqueue/dfTAA05900
-rw------- 1 root mail 494 Jan 18 19:14 /var/spool/mqueue/qfTAA05900

Then, the 'dfTAA05900' file contains the recipient(s) and the subject for which you are looking.

Regards!

...JRF...

[TheJuiceman]

Hi James,

The mqueue only shows items that have not sent. I need to get information on things that sent but failed. Is there a similar way to get this information? Thank you both again!!!

Oh, and I will submit points at the end. Thanks again.

[James R. Ferguson]

Hi (again):

> The mqueue only shows items that have not sent.

That would be true. Messages remain in this queue (directory) until they are successfull sent, at which time they are then removed.

Regards!

...JRF...

[TheJuiceman]

Hi again,

Yeah, I need to get messages that leave the server but do not get delivered due to things like the recipient is not valid, etc. /var/mail will populate with this information. I just need a way to sort through the mail and put it in a user friendly form.

[OFC_EDM]

Again I don't have access to a system to test this but....

Can you cross reference the failure messages in the mail.log file with the users mbox on the server?

I'm assuming sent mail is stored in the users mailbox.

So if you have the users name you know the users mailbox. The mailbox is a text file so you could search their mailbox for the original message.

I'm made a few assumptions. Hope they apply to your environment.

Cheers

[OFC_EDM]

Found this in the sendmail.cf readme. I've never used this setting so I don't know if it's what you need.

But at first glance it shows potential

confDEAD_LETTER_DROP DeadLetterDrop

[undefined] Filename to save bounce messages which could not be returned to the user or sent to postmaster. If not set, the queue file will be renamed.

I got this information from
http://www.sendmail.org/m4/tweaking_config.html

[TheJuiceman]

Well, the big problem isn't finding the messages. They are in /var/mail. The problem is when you look at them, there is a bunch of MIME information and stuff that the average user can't make heads or tails out of. All they want to know is 1) who sent an email that got bounced back 2) what was the email and 3) who was it suppose to go to. Doing the cross-referencing thing doesn't seem to get me anything more than I can get directly from /var/mail. I just need a way to get a user friendly way of pulling out the information they want.

[Hein van den Heuvel]

Well, you'll have to learn about the mail file layout. I never seriously looked at it, but it looks recognizable allright.

A script would have to look for stuff 'in order' as well as 'carefully'.

For example, it is not really interested in a line with "Subject:" unless it is at the start of a line, and not until it has seen a line with "permanent failure".
So the grep needs to be with 'anchored' strings as not to trigger on seemingly right pieces of strings in the wrong place.

The script will probably have to protect itself again NOT finding what it expects, and reset itself instead of blindly looking for strings which are not going to come anymore.

There are many, many ways to do this.
Below a sample implementation using perl and using a 'step' variable to remember what it is looking for next.

It works on my mail file, as per below.

It might only work on my mail file.
That's for you to find out.
And you would need to tweak it to go over all mail files, and format the report 'just so'. (That part is 'work', which I will gladly do.. for the right compensation. :-)

hth,
Hein.

$ perl permanent_fatal.pl /var/mail/hein
550 5.1.2 hein@nnosuchdomain.xyz... Host unknown (Name server: nosuchdomain.xyz: host not found)
Use of uninitialized value in print at permanent_fatal.pl line 29, <> line 75.
From: Hein van den Heuvel
Date: Fri, 20 Jun 2008 21:15:29 -0400 (EDT)

550 5.1.2 hein@wergfwegwe.org... Host unknown (Name server: wergfwegwe.org: host not found)
Subject: this is a test subject
From: Hein van den Heuvel
Date: Fri, 20 Jun 2008 21:20:18 -0400 (EDT)


$ cat permanent_fatal.pl
use strict;
use warnings;
my $step=0;
my $id=0;
my $old=1;
my ($problem, $subject, $from, $date);
while (<>) {
#debug print "$step $.\n" if $step != $old;
#debug old = $step;
if (0==$step) {
next unless /^ --.*permanent fatal.*-$/;
$step++;
}
elsif (1==$step) {
next unless /^ ----- Transcript.*-$/;
$problem = <>;
$step++;
}
elsif (2==$step) {
next unless /^--(\S+)$/;
$id = $1;
$step++;
}
elsif (3==$step) {
$subject = $_ if /^Subject:/;
$from = $_ if /^From:/;
$date = $_ if /^Date:/;
if (/^--${id}--$/) {
print $problem, $subject, $from, $date, "\n";
$step = 0; # NOT confused. Done. Start again.
}
if (/^--(\S+)$/) {
$step = 0 unless $1 eq $id ; # Confused? Start again.
}
}
}

[Hein van den Heuvel]

Oh, and in case you wonder about:

"Use of uninitialized value in print at permanent_fatal.pl line 29, <> line 75."

That is thanks to 'use warnings'.
The first refused message had no 'subject' line, so the $subjetc variable was not set up. A real solution would have to protect against that, reseting all message data after a print. Easy enough, but it needs to be done.

Hein.

[TheJuiceman]

Thanks Hein. I'm looking at your code and it is all pretty greek to me. Here is an example of what a returned mail might look like in /var/mail. Each message is separated by the line "From MAILER-DAEMON..."

From MAILER-DAEMON@domain.com Mon Jun 23 16:36:14 EDT 2008
Received: from localhost (localhost)
by local.domain.com (8.9.3 (PHNE_35950)/8.9.3) with internal id Q
AB24698;
Mon, 23 Jun 2008 16:36:14 -0400 (EDT)
Date: Mon, 23 Jun 2008 16:36:14 -0400 (EDT)
From: Mail Delivery Subsystem
Message-Id: <200806232036.QAB24698@local.domain.com>
To: sender@local.domain.com
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
boundary="QAB24698.1214253374/local.domain.com"
Subject: Returned mail: User unknown
Auto-Submitted: auto-generated (failure)

This is a MIME-encapsulated message

--QAB24698.1214253374/local.domain.com

The original message was received at Mon, 23 Jun 2008 16:36:14 -0400 (EDT)
from sender@localhost

----- The following addresses had permanent fatal errors -----
recipient@local.domain.com

----- Transcript of session follows -----
550 recipient@local.domain.com... User unknown

--QAB24698.1214253374/local.domain.com
Content-Type: message/delivery-status

Reporting-MTA: dns; local.domain.com
Arrival-Date: Mon, 23 Jun 2008 16:36:14 -0400 (EDT)

Final-Recipient: RFC822; recipient@local.domain.com
Action: failed
Status: 5.1.1
Last-Attempt-Date: Mon, 23 Jun 2008 16:36:14 -0400 (EDT)

--QAB24698.1214253374/local.domain.com
Content-Type: message/rfc822

Return-Path:
Received: (from sender@localhost)
by local.domain.com (8.9.3 (PHNE_35950)/8.9.3) id QAA24698;
Mon, 23 Jun 2008 16:36:14 -0400 (EDT)
Date: Mon, 23 Jun 2008 16:36:13 -0400
From: Sender's Name
To: recipient@foreigndomain.com
Subject: Email subject
Message-ID: <20080623203613.GA24696@local.domain.com>
Mime-Version: 1.0
Content-Type: multipart/mixed; boundary="qDbXVdCdHGoSgWSk"
Content-Disposition: inline
User-Agent: Mutt/1.4.2.1i


--qDbXVdCdHGoSgWSk
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Body of email

--qDbXVdCdHGoSgWSk
Content-Type: text/html; charset=us-ascii
Content-Disposition: attachment; filename="attachment.htm"





--qDbXVdCdHGoSgWSk--

--QAB24698.1214253374/local.domain.com--


From MAILER-DAEMON@local.domain.com Mon Jun 23 16:36:14 EDT 2008

How can I modify the script to look for information the user would want like ...

Subject: Returned mail: User unknown
From: Sender's Name
To: recipient@foreigndomain.com
Subject: Email subject

Thanks again.

[Hein van den Heuvel]

>> I'm looking at your code and it is all pretty greek to me.

If it was easy you would have had many more replies -). There are several lines starting with "From:" and "Subject:"... which ones to pick.

>> Here is an example of what a returned mail might look like in /var/mail.

Thanks. Cut&paste into the forum drops spaces, and maybe added spaces as the line end. I adapted my example to tolerate that, but an 'attached' text file would have been better.

>> Each message is separated by the line "c..."

That's what you observe/say. And it may be true. But like I said I knew nothing about the Email format and decided to trigger on the 'permanent fatal' line. :-). I thought, rightly or wrongly so, that those might perhaps come from other sources and also that maybe MAILER-DEAMON might send out other type message which need not be considered.

I changed the cript some to be more clear about looking for 'id' lines. In your case:
--QAB24698.1214253374/local.domain.com
--QAB24698.1214253374/local.domain.com--

Try it. Maybe it does what you want, maybe not. It works for me on your example and my file.

use strict;
use warnings;
my $step=0;
my $id=0;
my $old=1;
my ($problem, $subject, $from, $to, $real_subject);
while (<>) {
#debug print "$step $.\n" if $step != $old;
#debug $old = $step;

if (/^--(\S+\/\S+?)(-?-?)\s?$/) {
if ($1 ne $id) { # New ID? Clear all memory
$id = $1;
$step = 0;
$from = q()."\n";
$to = q()."\n";
$real_subject = q()."\n";
} else {
if ($2) { # Found closing marker
print $problem, $subject, $from, $to, $real_subject, "\n";
$step = 0; # Done. Start again.
}
}
}
if (0==$step) {
$subject = $_ if /^Subject:/;
next unless /^[ -]+.*permanent fatal.*-\s?$/;
$step++;
}
elsif (1==$step) {
next unless /^[ -]+Transcript.*-\s?$/;
$problem = <>;
$step++;
}
elsif (2==$step) {
$from = $_ if /^From:/;
$to = $_ if /^To:/;
$real_subject = $_ if /^Subject:/;
}
}

[TheJuiceman]

Hello Hein,

Sorry for the delay getting back. The updated script seems to be on the right track!!! I'm just wanting and checking for variations that I need to address. So far this is a good start!!! I will report my progress and reward points very soon. Thank you for your help

[TheJuiceman]

Hello Hein,

The first script seemed to work, it just did not get everything. The second one I am getting errors such as...

./permanent_fatal.pl[2]: use: not found.
./permanent_fatal.pl[3]: use: not found.
./permanent_fatal.pl[4]: my: not found.
./permanent_fatal.pl[5]: my: not found.
./permanent_fatal.pl[6]: my: not found.
./permanent_fatal.pl[7]: Syntax error at line 7 : `(' is not expected.

Can you send the script as an attachment so I can make sure it is correct? Thanks again.

[James R. Ferguson]

Hi :

> The first script seemed to work, it just did not get everything. The second one I am getting errors such as...

You need to put the "shebang" line at the head so that the Perl interpreter is used:

#!/usr/bin/perl

...assuming that '/usr/bin/perl' is where your Perl lives or is a symbolic link to it.

Without this, the shell thinks it is going to run another shell script. Hence the shell interpreter stumbles and chokes on the "my" and "use" tokens!

Regards!

...JRF...

[Patrick Wallek]

You can also run the script from the command line by using perl as the interpreter, such as:

# perl permanent_fatal.pl /var/mail/hein

(This was taken from Hein's first post.)

[James R. Ferguson]

Hi (again):

Patrick has provided you a second way to run Hein's script --- without adding the 'shebang' line.

There is yet another trick that is useful if Perl resides in your PATH but isn't always in the same absolute directory; as for instance, when running on different servers. Then, add this 'shebang':

#!/usr/bin/env perl

As a general rule, however, one usually sets up a symbolic link from '/usr/bin/perl' to wherever the Perl interpreter lives. These musing apply to LINUX or UNIX of course.

Regards!

...JRF...

[TheJuiceman]

Doh...should have spotted that one.

OK...getting close here. How could I change this so that it will look for "Action:" instead of the FIRST "Subject:"? I think that would make this thing go!!!

Thanks again!!!

[Hein van den Heuvel]

Well, you look for the word "subject" and replace it by "action" or add a similar construct for "action".

Anyway... I still had the source so here is it changed to output the "Action:" line as well. Up to you to re-organize, re-format the output.

Enjoy,
Hein.

use strict;
use warnings;
my $step=0;
my $id=0;
my $old=1;
my ($problem, $action, $subject, $from, $to, $real_subject);
while (<>) {
#debug print "$step $.\n" if $step != $old;
#debug $old = $step;

if (/^--(\S+\/\S+?)(-?-?)\s?$/) {
if ($1 ne $id) { # New ID? Clear all memory
$id = $1;
$step = 0;
$from = q()."\n";
$to = q()."\n";
$action = q()."\n";
$real_subject = q()."\n";
} else {
if ($2) { # Found closing marker
print $problem, $action, $subject, $from, $to, $real_subject, "\n";
$step = 0; # Done. Start again.
}
}
}
if (0==$step) {
$subject = $_ if /^Subject:/;
next unless /^[ -]+.*permanent fatal.*-\s?$/;
$step++;
}
elsif (1==$step) {
next unless /^[ -]+Transcript.*-\s?$/;
$problem = <>;
$step++;
}
elsif (2==$step) {
$from = $_ if /^From:/;
$to = $_ if /^To:/;
$action = $_ if /^Action:/;
$real_subject = $_ if /^Subject:/;
}
}

[TheJuiceman]

Hello Hein!!!

Everything works except for one thing. Occasionally the script will return something like the following....

... while talking to localhost.com:
Action: failed
Subject: Returned mail: User unknown




However, here is the message that this generated from...

From MAILER-DAEMON@here.com Thu Jul 3 06:17:26 EDT 2008
Received: from localhost (localhost)
by here.com Thu, 3 Jul 2008 06:17:26 -0400 (EDT)
Date: Thu, 3 Jul 2008 06:17:26 -0400 (EDT)
From: Mail Delivery Subsystem
Message-Id: <200807031017.GAA00598@here.com>
To: sender@here.com
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
boundary="GAA00598.1215080246/here.com"
Subject: Returned mail: User unknown
Auto-Submitted: auto-generated (failure)

This is a MIME-encapsulated message

--GAA00598.1215080246/here.com

The original message was received at Thu, 3 Jul 2008 06:17:26 -0400 (EDT)
from sender@localhost

----- The following addresses had permanent fatal errors -----
recipient@here.com

----- Transcript of session follows -----
... while talking to here.com.:
>>> RCPT To:
<<< 550 No such recipient
550 recipient@here.com... User unknown

--GAA00598.1215080246/here.com
Content-Type: message/delivery-status

Reporting-MTA: dns; here.com
Arrival-Date: Thu, 3 Jul 2008 06:17:26 -0400 (EDT)

Final-Recipient: RFC822; recipient@here.com
Action: failed
Status: 5.1.1
Remote-MTA: DNS; here.com
Diagnostic-Code: SMTP; 550 No such recipient
Last-Attempt-Date: Thu, 3 Jul 2008 06:17:26 -0400 (EDT)

--GAA00598.1215080246/here.com
Content-Type: message/rfc822

Return-Path:
Received: (from sender@localhost)
by here.com id GAA00594
for recipient@here.com; Thu, 3 Jul 2008 06:17:26 -0400 (EDT)
Date: Thu, 3 Jul 2008 06:17:26 -0400
From: Sender Name
To: recipient@here.com
Subject: This is an email
Message-ID: <20080703101726.GA592@here.com>

I guess I don't understand why it would miss all of the "pertinent" information in this message and work fine for another?

Thanks again for your help and patience!!!

[TheJuiceman]

oops...obviously replace the first "localhost" in the script response with "here". Thanks again.