topic Re: File consistency in Operating System - HP-UX

File consistency

heaman1 — Mon, 06 Jul 2009 02:13:14 GMT

I have two server in different site , each server have a directory , the files in the directories are as below .

Direcotry in server A
=====================
#ll
-rw-r--r-- 1 root root 68186 Jun 19 15:30 aaa.txt
-rw-r--r-- 1 root root 68186 Jun 19 15:30 bbb.txt
-rw-r--r-- 1 root root 68186 Jun 19 15:30 ccc.txt
-rw-r--r-- 1 root root 68186 Jun 19 15:30 ddd.txt
"

Directory in server B
=====================
#ll
-rw-r--r-- 1 root root 68186 Jun 19 15:30 Ora_aaa.txt
-rw-r--r-- 1 root root 68186 Jun 19 15:30 Ora_bbb.txt
-rw-r--r-- 1 root root 68186 Jun 19 15:30 Ora_ccc.txt
-rw-r--r-- 1 root root 68186 Jun 19 15:30 Ora_ddd.txt
"

Some files will be appended to these directories time by time .

The files size , date and no. of files in these directories should be the same ( except the file name , the files name in server B are begins with Ora_ )

Now , I would like to make sure the files in these two directories are the same and no file is missing , so I would like to regularly check the files is the same or not so that I can take action to correct it immediately .

can you advise the script to check the files in it are the same , eg. check the files regularly ( 4 hrs ) , if any files is missing then notify me so that I can take action to correct it .

Re: File consistency

Mel Burslan — Mon, 06 Jul 2009 03:29:36 GMT

First order of business for you is to setup trust between server A and server B, preferably not as root since everyone can read these files. root trust between servers will get you in trouble with auditors, so stay away from it.

So, let's assume server B trusts server A via ssh keys (you can choose to trust via .rhosts but I advise strongly against it)

OKflag=0
ran=$RANDOM
TMPDIR=/tmp/tempfile.${ran}
mkdir $TMPDIR

cd /my/directory/path
for file in `ls -1`
do
scp server_B:/my/directory/path/Ora_${file} $TMPDIR
r=${?}
if [ $r -ne 0 ]
then
echo "Do something ${file} is missing on server B"
OKflag=1
fi
diff /my/directory/path/${file} $TMPDIR/Ora_${file} > /dev/null
rr=${?}
if [ $rr -ne 0 ]
then
echo "Do some other thing: $file is different on server B"
OKflag=2
fi
rm $TMPDIR/$Ora_{file}
done

if [ $OKflag -eq 0 ]
then
echo "Everything checked out fine at `date`"
fi

Hope this helps

Beware of syntax and typographical errors as the script was not tested in any unix platform.

Re: File consistency

heaman1 — Mon, 06 Jul 2009 05:09:40 GMT

thx Mel Burslan ,

it works fine , for your method , sorry to have two more requirements .

1) as there are so many files in the directoies , it is not possible to copy all files to compare , can advise if only compare the file that created in a certain peroid of time ( eg. 4 hrs / 8 hrs ) what can i do ?

p.s assume the file creation date are the name .

2) can advise if I also want to compare the file size ( to make sure the file consistency ) , is it possible ?

thx

Re: File consistency

Jitesh purohit_1 — Mon, 06 Jul 2009 05:20:22 GMT

Hi heaman,

You can use scp/rcp to achive the above task

Thanks
Jitesh

Re: File consistency

Dennis Handly — Mon, 06 Jul 2009 05:22:14 GMT

>1) as there are so many files in the directories, it is not possible to copy all files to compare, can advise if only compare the file that created in a certain period of time (eg. 4 hrs / 8 hrs) what can I do?

You can create a reference file and only do files that are newer than that file.
If you do "ll -t", you can just stop when you see that file. (Or you can remember the time and do complex date arithmetic to see if newer.)

>2) I also want to compare the file size (to make sure the file consistency)?

From the ll(1) output, you can compare the sizes too.

Re: File consistency

likid0 — Mon, 06 Jul 2009 05:30:57 GMT

You could also use rsync. it does incremental updates of the files, if their hash has changed.

Re: File consistency

heaman1 — Mon, 06 Jul 2009 08:12:20 GMT

" You can create a reference file and only do files that are newer than that file.
If you do "ll -t", you can just stop when you see that file. (Or you can remember the time and do complex date arithmetic to see if newer.) " --->>

sorry , I am not too unstandard what you mean to use "ll -t" to compare the time ? maybe I state more , what I would like is to compare the files that created within 4 hrs because there are too many files in server B , it is not possible to copy , can advise what can i do ? thx

Re: File consistency

Dennis Handly — Mon, 06 Jul 2009 10:29:49 GMT

>I am not too understand what you mean to use "ll -t" to compare the time?

Every 4 hours you touch a file. Then you use the previous filename/time to stop looking.

Re: File consistency

Mel Burslan — Mon, 06 Jul 2009 13:39:20 GMT

In order to accomplish what you want, I see not other way than both servers trusting each other bidirectionally, i.e., you will need to be able to copy files from one to the other and execute commands via ssh from one to the other. This is necessary because you will have to do a remote command execution to create marker files as Dennis said.

you are on serverB:

touch /var/tmp/4hours_marker

...wait 4 hours or a little less

you are on serverA:

OKflag=0
ran=$RANDOM
TMPDIR=/tmp/tempfile.${ran}
mkdir $TMPDIR

ssh serverB "find /my/directory/path -newer /var/tmp/4hours_marker > /tmp/filelist; touch /var/tmp/4hours_marker"
ssh serverB "scp `cat /tmp/filelist` serverA:/$TMPDIR"

cd /my/directory/path
for file in `ls -1`
do

if [ -a Ora_${file} ]
then
#file exists, compare it to the local file
diff /my/directory/path/${file} $TMPDIR/Ora_${file} > /dev/null
rr=${?}
if [ $rr -ne 0 ]
then
echo "Do something: $file is different on server B"
OKflag=2
fi

else
echo "${file} is missing on serverB. Do something"
OKflag=1
fi

diff /my/directory/path/${file} $TMPDIR/Ora_${file} > /dev/null
rr=${?}
if [ $rr -ne 0 ]
then
echo "Do something: $file is different on server B"
OKflag=2
fi
done

if [ $OKflag -eq 0 ]
then
echo "Everything checked out fine at `date`"
fi

rm -r $TMPDIR

I know there are holes in this procedure, most important of which is lack of extensive error checking but nature of this forums is to guide you down the path. I hope my script helps you wind the way, on which you can build your own script with all the error handling and else.

Re: File consistency

heaman1 — Tue, 07 Jul 2009 05:28:47 GMT

thx Mel,

I tried your method

ssh serverB "find /my/directory/path -newer /var/tmp/4hours_marker > /tmp/filelist; touch /var/tmp/4hours_marker" .

I tried it but in do not create 4hours_marker file , it seems the command have error , if I just want to find the files created in 4 hours , how can i do it ? thx

Re: File consistency

Dennis Handly — Tue, 07 Jul 2009 07:42:35 GMT

>do not create 4hours_marker file, it seems the command have error

What had an error? The touch didn't create the file? Any error messages?

Re: File consistency

heaman1 — Wed, 08 Jul 2009 05:21:50 GMT

Thx Mel Burslan ,

for your latest program , does the -newer mean only compare the newer files so I need to setup a crontab job to run it in every 4 hour ? if yes , can advise if I want the program can be run at anytime to check the files in past 4 hours ( not only the newer file ) , what can i do ? thx

Re: File consistency

Dennis Handly — Wed, 08 Jul 2009 07:30:00 GMT

>does the -newer mean only compare the newer files so I need to setup a crontab job to run it in every 4 hour?

Yes.

>if I want the program can be run at anytime to check the files in past 4 hours (not only the newer file), what can I do?

Then you need to do date arithmetic to invoke touch. Or use perl. Or possibly gnu find that allows minutes in the -mtime.

Re: File consistency

heaman1 — Thu, 09 Jul 2009 06:12:53 GMT

thx reply ,

Then you need to do date arithmetic to invoke touch. Or use perl. Or possibly gnu find that allows minutes in the -mtime. -->

can you please advise how to do it ? thx

Re: File consistency

heaman1 — Thu, 09 Jul 2009 06:26:36 GMT

can advise how to use find to search the files on remote server ? thx

Re: File consistency

heaman1 — Thu, 09 Jul 2009 06:33:30 GMT

thx all,

Rather than copy the files back for comparison , is it possible to export the files to a list , then compare the lists in both servers , because copy the files back to local server , it seems increase the network traffic , can advise what can i do ? thx

Re: File consistency

Dennis Handly — Thu, 09 Jul 2009 09:54:04 GMT

>can you please advise how to do it?

It's called hand waving, you are suppose to give up. ;-) Or manually enter the touch time for 4 hours ago.
(I personally would just write a C program to subtract 4 hours.)

>date arithmetic to invoke touch.

I don't know if Clay's date script will do minutes?
Some threads that mention it and other time manipulation with perl:
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=951459
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1048731
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1120611

Gnu find:
http://hpux.connect.org.uk/hppd/hpux/Gnu/findutils-4.4.2/