Script help needed to parse /var/mail files

[OFC_EDM]

Found this in the sendmail.cf readme. I've never used this setting so I don't know if it's what you need.

But at first glance it shows potential

confDEAD_LETTER_DROP DeadLetterDrop

[undefined] Filename to save bounce messages which could not be returned to the user or sent to postmaster. If not set, the queue file will be renamed.

I got this information from
http://www.sendmail.org/m4/tweaking_config.html

[TheJuiceman]

Well, the big problem isn't finding the messages. They are in /var/mail. The problem is when you look at them, there is a bunch of MIME information and stuff that the average user can't make heads or tails out of. All they want to know is 1) who sent an email that got bounced back 2) what was the email and 3) who was it suppose to go to. Doing the cross-referencing thing doesn't seem to get me anything more than I can get directly from /var/mail. I just need a way to get a user friendly way of pulling out the information they want.

[Hein van den Heuvel]

Well, you'll have to learn about the mail file layout. I never seriously looked at it, but it looks recognizable allright.

A script would have to look for stuff 'in order' as well as 'carefully'.

For example, it is not really interested in a line with "Subject:" unless it is at the start of a line, and not until it has seen a line with "permanent failure".
So the grep needs to be with 'anchored' strings as not to trigger on seemingly right pieces of strings in the wrong place.

The script will probably have to protect itself again NOT finding what it expects, and reset itself instead of blindly looking for strings which are not going to come anymore.

There are many, many ways to do this.
Below a sample implementation using perl and using a 'step' variable to remember what it is looking for next.

It works on my mail file, as per below.

It might only work on my mail file.
That's for you to find out.
And you would need to tweak it to go over all mail files, and format the report 'just so'. (That part is 'work', which I will gladly do.. for the right compensation. :-)

hth,
Hein.

$ perl permanent_fatal.pl /var/mail/hein
550 5.1.2 hein@nnosuchdomain.xyz... Host unknown (Name server: nosuchdomain.xyz: host not found)
Use of uninitialized value in print at permanent_fatal.pl line 29, <> line 75.
From: Hein van den Heuvel
Date: Fri, 20 Jun 2008 21:15:29 -0400 (EDT)

550 5.1.2 hein@wergfwegwe.org... Host unknown (Name server: wergfwegwe.org: host not found)
Subject: this is a test subject
From: Hein van den Heuvel
Date: Fri, 20 Jun 2008 21:20:18 -0400 (EDT)


$ cat permanent_fatal.pl
use strict;
use warnings;
my $step=0;
my $id=0;
my $old=1;
my ($problem, $subject, $from, $date);
while (<>) {
#debug print "$step $.\n" if $step != $old;
#debug old = $step;
if (0==$step) {
next unless /^ --.*permanent fatal.*-$/;
$step++;
}
elsif (1==$step) {
next unless /^ ----- Transcript.*-$/;
$problem = <>;
$step++;
}
elsif (2==$step) {
next unless /^--(\S+)$/;
$id = $1;
$step++;
}
elsif (3==$step) {
$subject = $_ if /^Subject:/;
$from = $_ if /^From:/;
$date = $_ if /^Date:/;
if (/^--${id}--$/) {
print $problem, $subject, $from, $date, "\n";
$step = 0; # NOT confused. Done. Start again.
}
if (/^--(\S+)$/) {
$step = 0 unless $1 eq $id ; # Confused? Start again.
}
}
}

[Hein van den Heuvel]

Oh, and in case you wonder about:

"Use of uninitialized value in print at permanent_fatal.pl line 29, <> line 75."

That is thanks to 'use warnings'.
The first refused message had no 'subject' line, so the $subjetc variable was not set up. A real solution would have to protect against that, reseting all message data after a print. Easy enough, but it needs to be done.

Hein.

[TheJuiceman]

Thanks Hein. I'm looking at your code and it is all pretty greek to me. Here is an example of what a returned mail might look like in /var/mail. Each message is separated by the line "From MAILER-DAEMON..."

From MAILER-DAEMON@domain.com Mon Jun 23 16:36:14 EDT 2008
Received: from localhost (localhost)
by local.domain.com (8.9.3 (PHNE_35950)/8.9.3) with internal id Q
AB24698;
Mon, 23 Jun 2008 16:36:14 -0400 (EDT)
Date: Mon, 23 Jun 2008 16:36:14 -0400 (EDT)
From: Mail Delivery Subsystem
Message-Id: <200806232036.QAB24698@local.domain.com>
To: sender@local.domain.com
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
boundary="QAB24698.1214253374/local.domain.com"
Subject: Returned mail: User unknown
Auto-Submitted: auto-generated (failure)

This is a MIME-encapsulated message

--QAB24698.1214253374/local.domain.com

The original message was received at Mon, 23 Jun 2008 16:36:14 -0400 (EDT)
from sender@localhost

----- The following addresses had permanent fatal errors -----
recipient@local.domain.com

----- Transcript of session follows -----
550 recipient@local.domain.com... User unknown

--QAB24698.1214253374/local.domain.com
Content-Type: message/delivery-status

Reporting-MTA: dns; local.domain.com
Arrival-Date: Mon, 23 Jun 2008 16:36:14 -0400 (EDT)

Final-Recipient: RFC822; recipient@local.domain.com
Action: failed
Status: 5.1.1
Last-Attempt-Date: Mon, 23 Jun 2008 16:36:14 -0400 (EDT)

--QAB24698.1214253374/local.domain.com
Content-Type: message/rfc822

Return-Path:
Received: (from sender@localhost)
by local.domain.com (8.9.3 (PHNE_35950)/8.9.3) id QAA24698;
Mon, 23 Jun 2008 16:36:14 -0400 (EDT)
Date: Mon, 23 Jun 2008 16:36:13 -0400
From: Sender's Name
To: recipient@foreigndomain.com
Subject: Email subject
Message-ID: <20080623203613.GA24696@local.domain.com>
Mime-Version: 1.0
Content-Type: multipart/mixed; boundary="qDbXVdCdHGoSgWSk"
Content-Disposition: inline
User-Agent: Mutt/1.4.2.1i


--qDbXVdCdHGoSgWSk
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Body of email

--qDbXVdCdHGoSgWSk
Content-Type: text/html; charset=us-ascii
Content-Disposition: attachment; filename="attachment.htm"





--qDbXVdCdHGoSgWSk--

--QAB24698.1214253374/local.domain.com--


From MAILER-DAEMON@local.domain.com Mon Jun 23 16:36:14 EDT 2008

How can I modify the script to look for information the user would want like ...

Subject: Returned mail: User unknown
From: Sender's Name
To: recipient@foreigndomain.com
Subject: Email subject

Thanks again.

[Hein van den Heuvel]

>> I'm looking at your code and it is all pretty greek to me.

If it was easy you would have had many more replies -). There are several lines starting with "From:" and "Subject:"... which ones to pick.

>> Here is an example of what a returned mail might look like in /var/mail.

Thanks. Cut&paste into the forum drops spaces, and maybe added spaces as the line end. I adapted my example to tolerate that, but an 'attached' text file would have been better.

>> Each message is separated by the line "c..."

That's what you observe/say. And it may be true. But like I said I knew nothing about the Email format and decided to trigger on the 'permanent fatal' line. :-). I thought, rightly or wrongly so, that those might perhaps come from other sources and also that maybe MAILER-DEAMON might send out other type message which need not be considered.

I changed the cript some to be more clear about looking for 'id' lines. In your case:
--QAB24698.1214253374/local.domain.com
--QAB24698.1214253374/local.domain.com--

Try it. Maybe it does what you want, maybe not. It works for me on your example and my file.

use strict;
use warnings;
my $step=0;
my $id=0;
my $old=1;
my ($problem, $subject, $from, $to, $real_subject);
while (<>) {
#debug print "$step $.\n" if $step != $old;
#debug $old = $step;

if (/^--(\S+\/\S+?)(-?-?)\s?$/) {
if ($1 ne $id) { # New ID? Clear all memory
$id = $1;
$step = 0;
$from = q()."\n";
$to = q()."\n";
$real_subject = q()."\n";
} else {
if ($2) { # Found closing marker
print $problem, $subject, $from, $to, $real_subject, "\n";
$step = 0; # Done. Start again.
}
}
}
if (0==$step) {
$subject = $_ if /^Subject:/;
next unless /^[ -]+.*permanent fatal.*-\s?$/;
$step++;
}
elsif (1==$step) {
next unless /^[ -]+Transcript.*-\s?$/;
$problem = <>;
$step++;
}
elsif (2==$step) {
$from = $_ if /^From:/;
$to = $_ if /^To:/;
$real_subject = $_ if /^Subject:/;
}
}

[TheJuiceman]

Hello Hein,

Sorry for the delay getting back. The updated script seems to be on the right track!!! I'm just wanting and checking for variations that I need to address. So far this is a good start!!! I will report my progress and reward points very soon. Thank you for your help

[TheJuiceman]

Hello Hein,

The first script seemed to work, it just did not get everything. The second one I am getting errors such as...

./permanent_fatal.pl[2]: use: not found.
./permanent_fatal.pl[3]: use: not found.
./permanent_fatal.pl[4]: my: not found.
./permanent_fatal.pl[5]: my: not found.
./permanent_fatal.pl[6]: my: not found.
./permanent_fatal.pl[7]: Syntax error at line 7 : `(' is not expected.

Can you send the script as an attachment so I can make sure it is correct? Thanks again.

[James R. Ferguson]

Hi :

> The first script seemed to work, it just did not get everything. The second one I am getting errors such as...

You need to put the "shebang" line at the head so that the Perl interpreter is used:

#!/usr/bin/perl

...assuming that '/usr/bin/perl' is where your Perl lives or is a symbolic link to it.

Without this, the shell thinks it is going to run another shell script. Hence the shell interpreter stumbles and chokes on the "my" and "use" tokens!

Regards!

...JRF...

[Patrick Wallek]

You can also run the script from the command line by using perl as the interpreter, such as:

# perl permanent_fatal.pl /var/mail/hein

(This was taken from Hein's first post.)