topic awk help in Operating System - HP-UX

awk help

Cem Tugrul — Thu, 06 Jul 2006 01:57:15 GMT

Hello,
i have a log file which contains thousands of records.i want to write a script which gives me
last 1 month's records...My log file keeps
the date information like;
yyyy/mm/dd
and example one of the record for my log file as below;
R MAN 954627 954632 2006/06/06 17:28:19 KSAD2565 /users/rvs/usrdat/KSAD2565 2953
so as you see the 5th colon is keeping date information..As a result,if i want to run the script today so this means i need records
from 2006/06/06--->today

All suggetions would be pointed...
:-)
Good Luck,

Re: awk help

Steven E. Protter — Thu, 06 Jul 2006 02:12:06 GMT

Shalom Cem,

Two step process.

Data looks like this:
R MAN 954627 954632 2006/06/06 17:28:19 KSAD2565 /users/rvs/usrdat/KSAD2565 2953

You can start with grep actually.

grep "2006/06" filename > newfile

You may need a special character to support the slash.

You can do it with awk.

while read -r dataline
do
datevar=$(echo $dataline | awk '{print $5}'
done < logfile

All you need to do now is parse the date variable.

yearvar=$(echo $datevar | awk -F\/ '{print $1}'
monthvar=$(echo $datevar | awk -F\/ '{print $2}'
dayvar=$(echo $datevar | awk -F\/ '{print $3}'

Not precisely sure about the -F statment. I'm assuming the special charcater treatment is required. I don't have time to test.

SEP

Re: awk help

H.Merijn Brand (procura — Thu, 06 Jul 2006 02:15:45 GMT

# perl -MDate::Calc=Delta_YMD -ne'BEGIN{@now=localtime;($Y,$M,$D)=($now[5]+1900,++$now[4],$now[3])}($y,$m,$d)=Delta_YMD(m{\s(\d+)/(\d+)/(\d+)\s},$Y,$M,$D);print if !$y' test.dta

prints the last year

# perl -MDate::Calc=Delta_YMD -ne'BEGIN{@now=localtime;($Y,$M,$D)=($now[5]+1900,++$now[4],$now[3])}($y,$m,$d)=Delta_YMD(m{\s(\d+)/(\d+)/(\d+)\s},$Y,$M,$D);print if !$y && !$m' test.dta

prints all from the last month

Enjoy, Have FUN! H.Merijn

Re: awk help

Cem Tugrul — Thu, 06 Jul 2006 02:37:56 GMT

Steven,

My 7th colon is the filename and changes...
As i said this file is log file which contains recieved files to my hp-ux so
i have to find out for some specific filenames but it does not matter i may grap it...the problem is getting records which belongs from last month's--->today
Anyway,i will test it...

Procura,

perl -MDate::Calc=Delta_YMD -ne'BEGIN{@now=localtime;($Y,$M,$D)=($now[5]+1900,++$now[4],$now[3])}($y,$m,$d)=Delta_YMD(m{\s(\d+)>
Can't locate Date/Calc.pm in @INC (@INC contains: /opt/perl5/lib/5.00502/PA-RISC1.1 /opt/perl5/lib/5.00502 /opt/perl5/lib/site_perl/5.005/PA-RISC1.1 /opt/perl5/lib/site_perl/5.005 .).
BEGIN failed--compilation aborted.

i think i made a mistake while running it

Re: awk help

Cem Tugrul — Thu, 06 Jul 2006 02:42:07 GMT

baan01:/baan/bse/etc#perl -v

This is perl, version 5.005_02 built for PA-RISC1.1

Copyright 1987-1998, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5.0 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using `man perl' or `perldoc perl'. If you have access to the
Internet, point your browser at http://www.perl.com/, the Perl Home Page.

Re: awk help

H.Merijn Brand (procura — Thu, 06 Jul 2006 02:48:31 GMT

Ah, oh, that is a rather old perl :)
And the problem in this case is not perl itself, but the fact that the module Date::Calc is not installed.

Would you consider a perl upgrade? (1),(2)
Would you like to install Date::Calc on the existing perl? (3),(4)
Or do you want a different solution altogether?

1) http://mirrors.develooper.com/hpux/#Perl
2) http://mirrors.develooper.com/hpux/downloads.html
3) http://search.cpan.org/~stbey/Date-Calc-5.4/
4) http://search.cpan.org/CPAN/authors/id/S/ST/STBEY/Date-Calc-5.4.tar.gz

Enjoy, Have FUN! H.Merijn

Re: awk help

Cem Tugrul — Thu, 06 Jul 2006 03:06:53 GMT

Procura,
Once upon a time you again helped me about perl&inst&upgrade...it was really helpful.
i can think upgrade perl or install the
date program but not now so is there any
other solution to manipulate with awk?

Re: awk help

spex — Thu, 06 Jul 2006 06:36:43 GMT

Cem,

First, use Clay's Date Hammer script (attached) to get the date a month in the past in the same format as your logfile. Then have awk output the first line in the logfile containing that date, along with all successive lines:

D=./caljd.sh -S "/" -y $(./caljd.sh -p 30)
awk "/${D}/,EOF" ${logfile}

If you only want certain lines returned by awk, pipe the output through grep.

PCS

Re: awk help

Peter Nikitka — Thu, 06 Jul 2006 08:29:53 GMT

Hi,

an awk solution seems possible to me.
If we assume the file is ordered by date, we can output from the first pattern found without further checking:

awk -v ym=$(date +%Y/%m/%d) 'BEGIN {split(ym,z,"/"); if(z[2]+0 == 1) {z[1]--;z[2]=12}
else z[2]--
lookfor=sprintf("%s/%02d/",z[1],z[2])}
$5 ~ "^"lookfor {split($5,dd,"/");if (dd[3]+0 >= z[3]+0) doit=1}
doit' logfile

If the date strings in logfile are not written sequentially you have to
- define a lookfor2 in BEGIN containing the original year and month, e.g.
...
BEGIN {split(ym,z,"/");
lookfor2=sprintf("%s/%02d/",z[1],z[2])
if(z[2]+0 == 1) {...

- decide in each match with 'lookfor' and 'lookfor2'

I'm shure you can do this by yourself.

mfG Peter

Re: awk help

Mel Burslan — Thu, 06 Jul 2006 09:42:02 GMT

Cem,

If you are looking for an awk one-liner, this posting is not it. And in my opinion, what you are trying to accomplish, requires a little more than a one-liner.

If you do not have it already, get Clay Stephenson's caljd.sh script from procura's web page http://mirrors.develooper.com/hpux/ (scroll to the bottom for the download links for both shell and perl versions. I personally use .sh version)

if you run caljd.sh without any arguments, you will get today in Julian date format, something like :

# ./caljd.sh
2453923
#

if you happen to run it like this

# caljd.sh 2453923
07 06 2006
#

so it basically toglles the julian format to regular date format. Nice thing about Julian date, you can add/subtract from that one.

I think you already figured out where I am going with this. All you need to do is something like this

todayjulian=`./caljd.sj`
monthlength=30 # modify it as you see fit to 28, 29 or 31

s=0

while [ $s -lt $monthlength ]
do
(( day=$todayjulian-$s ))
datestring=`./caljd.sh $day`

yr=`echo $datestring | cut -d" " -f3`
mo=`echo $datestring | cut -d" " -f2`
dy=`echo $datestring | cut -d" " -f1`

mydate=$yr"/"$mo"/"$dy

grep "$mydate" mylogfile >> last_month_log
# you can also do an awk {'print $5'} and compare it to $mydate if you prefer

(( s=$s+1 ))
done

Caveat emptors: It is not the most efficient way to do it and it is not tested, but as I follow your progress here, you can figure out the logic as a well experienced hp-ux person and modify it to your desires.

Kolay gelsin...

Re: awk help

Peter Nikitka — Thu, 06 Jul 2006 10:06:47 GMT

Hi Cem,

did you really try my awk-solution?
If not, do it!
If something was wrong in the code, you will tell us, I hope.

mfG Peter

Re: awk help

john korterman — Thu, 06 Jul 2006 11:02:15 GMT

Hi Cem,

try the attached script, using your logfile as $1

Start and end dates must both exist in your logfile.

regards,
John K.

Re: awk help

Jeff Colyer — Thu, 06 Jul 2006 23:08:14 GMT

Hi Cem,

Here's a simplified way that Peter was showing. This should work if you're date is always in the same format. Sorry I'm on my work pc and can't test this to verify it's working.

awk -v ymd=$(date +%Y/%m/%d) '{
if ($5==ymd) {
for (i=0;i print $i
}
}' log

explanation: ymd is grabbing the date from the shell and putting it in the format your log file is. $5 I believe, (hard to see from your copy and paste) is your date field. And the for statement should print your entire line. NF is number of fields.

good luck,
Jeff

Re: awk help

Cem Tugrul — Fri, 07 Jul 2006 01:02:56 GMT

Hello Again,

Sorry for late feedeback because unexpectly i had fatal problems over my SAN&production
servers and now it is fixed...

Peter,

awk -v ym=$(date +%Y/%m/%d) 'BEGIN {split(ym,z,"/"); if(z[2]+0 == 1) {z[1]--;z[2]=12}
else z[2]--
lookfor=sprintf("%s/%02d/",z[1],z[2])}
$5 ~ "^"lookfor {split($5,dd,"/");if (dd[3]+0 >= z[3]+0) doit=1}
doit' logfile

Your script done what i want so 10 goes to you...
i will test other replies's solution then give points so i still keep this thread open.

Thank's for all replies...
for you

Re: awk help

Arturo Galbiati — Fri, 07 Jul 2006 04:02:29 GMT

Hi Cem,
another solution using sed:

D="2006\/06\/06"
sed -n '/'$D'/,$p' f1

D="yyyy\/mm\/dd"
Put here your start date

sed -n '/'$D'/,$p' f1
Thsi print lines starting from line containing your date till the end of the file
($) that I suppose be today.

HTH,
Art

Re: awk help

Cem Tugrul — Wed, 03 Sep 2008 10:29:35 GMT

Peter's solution is the best...