Print from datestamp marker in file to end of file

[Scott Lindstrom_2]

I datestamp the HISTFILE's of our root users on the 1st and 15th of the month with a unique character string (eg, "#@!#@!#@!#@! SOX datestamp `/usr/bin/date` #@!#@!#@!#@!").

For SOX reporting, I then want to email the contents of their HISTFILE from the *last* unique datestamp though the end of the file (and which point I write the datestamp again in prep for the next run in 15 days).

(We want to keep the HISTFILE intact up to its stated size, so I never want to permanently remove data that's already been emailed).

I have to imagine this is done with awk or sed, but I haven't found an example of this yet.

Any ideas?

Scott

[Alan Meyer_4]

Ok, here ya go

#grab the line number of the last datestamp
LN=`nl HISTFILE |grep "#@!#@!#@!#@!" |tail -1 |awk '{print $1}'`

#email all lines from the last datestamp onward
tail -n +$LN HISTFILE |mailx -s "SOX HISTFILE Report" sysadmin@bigcompany.com

[James R. Ferguson]

Hi Scott:

You could do something like:

# cat myextract
#!/usr/bin/perl -w
my $last =0;
die "Usage: $0 file\n" unless @ARGV;
open(FH, "<", "$ARGV[0]") or die "Can't open $!\n";
while () { $last = $. if m/#@!#@!/; }
seek(FH,0,0);
$. = 0;
do () until $. == $last || eof;
print while ();
1;
#

Run as ./myextract ${HOME}/.sh_history

Modify the matching expression in the script if you need to do so. The output is the everything after the last occurance of the marker pattern.

Regards!

...JRF...

[Scott Lindstrom_2]

James -

I get this when I run the perl script:

./myextract .xxxxx_history
Null filename used at ./myextract line 8, line 43747.

(.xxxxx_history is quite large).

Here is the script (in vi with :set nu on); only the first line was changed:

1 #!/opt/perl/bin/perl
2 my $last =0;
3 die "Usage: $0 file\n" unless @ARGV;
4 open(FH, "<", "$ARGV[0]") or die "Can't open $!\n";
5 while () { $last = $. if m/#@!#@!/; }
6 seek(FH,0,0);
7 $. = 0;
8 do () until $. == $last || eof;
9 print while ();
10 1;
11 #

Since I don't know perl at all, I'm at a loss.

Scott

[Scott Lindstrom_2]

Alan -

I'm not getting any output. I notice when I run the nl command against the HISTFILE, I don't get all the output I woudl expect:

nl .xxxxx_history|tail
51631 stm
51632 51633 51634
51635 cd /scripts/src
51636 51637 51638 51639
51640 cd /scripts/src
51641 51642 51643 51644 51645 51646 51647 51648 51649 51650 51651 51652 51653 51654 51655 51656 51657 51658 51659 51660
51661 cd /scripts/lvm
51662 51663 51664 51665 51666 51667 51668 51669 51670 51671

The numbered lines without commands I believe should have a command next to them, but I don't know why.

Scott

[Rodney Hills]

Try using "csplit" to split the HISTFILE based on your unique character strings.

This will create multiple files of the form-
xx00 xx01 xx02 ...
for each grouping of text that begins with #@#@#.

Then the last file in the sequence would be the text you want.

Here is a small script-

cd /tmp
mkdir roothist
cd roothist
n=`csplit /root/HISTFILE '/^#@#/' '{*}'| wc -l`
(( n = n - 1 ))
set -A a `ls -1`
cat ${a[$n]} >/tmp/HISTSNAP

n is set to the number of xx files created. We substract one since shell arrays are zero based.

HTH

-- Rod Hills

[Alan Meyer_4]

the numbered lines without commands are LF/CR type lines I believe. Did you try the tail command to see if the output was correct?

All the nl command does is get the point in the file from which to tail the output.

[Mel Burslan · Accepted solution]

MyPattern="#@!#@!#@!#@! SOX datestamp"
MyHistFile=/where/ever/my/file/is/histfile

LastStampLine=`grep -n"${MyPattern}" "${MyHistFile}" | tail -1 | cut -d: -f1`

cat ${MyHistFile}|sed -e "1,${LastStampLine}d"|sendmail -v myemail@mycompany.com



I hope the above commands will work for you.

[Alan Meyer_4]

from looking at my HISTFILE, is seems that some tty feedback and info has been fed into the histfile. This seems to come from login/profile execution.

[Alan Meyer_4]

Due to the corruption of the cr/lf lines in the history file, it seems that Mel's grep -n command works better than the nl method.

[Scott Lindstrom_2]

Mel -

I think that's going to do it.

I do need to simplify my datestamp. I think one or more of the characters #@! is causing trouble in the various solutions.

I will pick this up tomorrow and assign points!

Scott

[Alan Meyer_4]

Mel,
What do you make of the cr/lf's in the HISTFILE? Why does that affect the nl output?

[Bill Hassell]

Actually, the HISTFILE is really a binary datafile. 99% of the lines are plain ASCII but there are some binary codes inserted by the shell. That's why editing the HISTORY file (adding or removing lines) will render the file unuseable. To reinstate the shell history function, the file must be zeroed (null file).

[Arturo Galbiati]

Hi Scott,
this should work (I tested it on a plan ascii file and it's ok)

WrkFil=your_file
sed -n ''$(awk '/your_pattern/{N=NR}; END {print N+1}' $WrkFil)',$p' $WrkFil|mailx your_email_Address

the $(awk '/your_pattern/{N=NR}; END {print N+1}' $WrkFil) code, give you the last line in the file (+1) where your pattern is located

sed prints from the line to the end

HTH,
Art

[Scott Lindstrom_2]

Rodney -

No matter what I try, I keep getting "out of range" error messages.

A grep finds the string, but the csplit fails:

csplit /root/.xxxxx_history '/############ Sarbanes Oxley datestamp/' '{*}'

/############ Sarbanes Oxley datestamp/ - out of range

(I have simplified the timestamp to get rid of '@').

Scott

[Rodney Hills]

To quote the man page for csplit-

out of range means that the given argument did not reference a line between the current position and the end of the file. This warning also occurs if the file is exhausted before the repeat count is.

Which sounds like your timestamps as you supplied didn't match anything in the file.

It is a complex pattern. But you only have to match a portion, for instance-
/SOX datestamp/

might be sufficient...

Rod Hills

[Scott Lindstrom_2]

Rodney -

That's the strange part. This works:

grep Sarbanes /root/.xxxxx_history
############ Sarbanes Oxley datestamp Thu Nov 10 09:27:37 CST 2005 ############
############ Sarbanes Oxley datestamp Thu Nov 10 09:27:37 CST 2005 ############
etc, etc

But this does not:

csplit /root/.xxxxx_history '/Sarbanes/' '{*}'
/Sarbanes/ - out of range

Scott

[Scott Lindstrom_2]

Arturo -

It looks like this will work as well.

One part I forgot to mention is the timestamp could have rolled off of the HISTFILE (if there was a lot of command activity). So any solution I implement needs to take that into account.

Thanks to everyone for all the great ideas! I would never have come up with these.

Scott

[Mel Burslan]

to prevent scrolling off the active histfile, I would suggest setting up a daily or more frequent if you think even the daily duretion may be insufficient, cron job, which will copy the contents of the histfile to another location, something like

2 00 * * * /var/adm/HistFileCopier.sh

where HistFileCopier.sh is something like this

userlist="rootusr1 rootusr2 rootusrN"
storageLOCATION=/var/adm/histSTORE
timestamp=`date +%Y%m%d%H%M`

for user in ${userlist}
do
strings ~${user}/.sh_history > ${storageLOCATION}/${user}.file.${timestamp}
done



then you can run your email reports against these files instead of the actual history files. As these will be cumulative, you need to develop an intelligence to make a line by line comparison between the current version and the previous version in order not to duplicate the entries.

As Bill Hassell stated, messing with .sh_history file in any way, shape or form, will render the contents useless. So, you need to devise a method to date and timestamp those. I must not have read your question in detail in my first answer before putting my response.

Also, another point, if you are this much concerned with what your root users did fro sox audits, I belive you already know that what you are doing is not sufficient to monitor your root users effectively since by granting them root access rights, you are giving them the keys to the mint. Once they gain root access whatever monitoring mechanism you instate locally on this server, cen be circumvented untraceably. Since these Sarbanes Oxymoron audits are that much of importance to you, I presume your organizations financial reporting is at utmost level, which in turn makes you eligible to spend some money on this end. If this is the case, go to

http://www.symark.com

and read about their product powebroker and its capabilities for reporting and root power delegation. That would be my suggestion to end your woes rather than developing homebrew sherlock applications.

HTH