Operating System - OpenVMS
1753670 Members
5352 Online
108799 Solutions
New Discussion юеВ

Re: Compare files with checksum and differences

 
Liu Tina
Occasional Advisor

Compare files with checksum and differences

Shanghai Stock Exchange customer compared two RMS files with recovery unit journaling named FTNTSPAR_NEW.RUJ, FTNTSPAR_OLD.RUJ with the checksum and differences.

They found the two files are same by тАЬchecksumтАЭ ( used default XOR ):

$ checksum FTNTSPAR_NEW.RUJ
$ show sym checksum$checksum
CHECKSUM$CHECKSUM = "796545170"
$ checksum FTNTSPAR_OLD.RUJ
$ show sym checksum$checksum
CHECKSUM$CHECKSUM = "796545170"

But when they used command тАЬdiffтАЭ, they found some of differences:
They asked why?

I used different ALGORITHM qualify with checksum command CRC, MD5 and default XOR get the different checksum result:

$ checksum/ALGORITHM=MD5 FTNTSPAR_NEW.RUJ
$ show sym checksum$checksum
CHECKSUM$CHECKSUM = "8CE744016224B055BCA2B88C67D85F81"

$ checksum/ALGORITHM=MD5 FTNTSPAR_old.ruj
$ show sym checksum$checksum
CHECKSUM$CHECKSUM = "F8268D402A770F0B56D6422AA3795020"

$ checksum/ALGORITHM=CRC FTNTSPAR_NEW.RUJ
$ show sym checksum$checksum
CHECKSUM$CHECKSUM = "7F88743C"

$ checksum/ALGORITHM=CRC FTNTSPAR_old.RUJ
$ show sym checksum$checksum
CHECKSUM$CHECKSUM = "98378C08"

Could you tell me:
1.what kind of algorithm we should be used for RMS files with recovery unit journaling?
2. What is the main different with the command тАЬdifferencesтАЭ and тАЬchecksumтАЭ?
3. Dose the тАЬdifferencesтАЭ just compare with the text file?
4. When we intend to compare two files what is the rule we should to follow to use to compare files?

Thanks for your help!
Best Regards,
Tina
18 REPLIES 18
P Muralidhar Kini
Honored Contributor

Re: Compare files with checksum and differences

Hi Tina,

>> used default XOR
Looks like the XOR algorithm used by VMS CHECKSUM can get a collision in
which case even though the files are different they can end up having the same
checksum value.

Check the following link for discussion on a similar topic -
http://www.mofeel.net/1152-comp-os-vms/3903.aspx

Hope this helps.

Regards,
Murali
Let There Be Rock - AC/DC
Steven Schweda
Honored Contributor

Re: Compare files with checksum and differences

> But when they used command "diff", they
> found some of differences:
> They asked why?

What did you tell them? I'd tell them that
it's probably because the files differ. If
your files contain more bits than the
checksum, then it must be possible to have
two files which have the same checksum.

http://en.wikipedia.org/wiki/Checksum

> 1. [...]

Used for what, exactly?

> 2. [...]

DIFFERENCES compares two files. CHECKSUM
calculates a checksum for a file.

HELP CHECKSUM
HELP DIFFERENCES

> 3. [...]

DIFFERENCES compares the two files which you
tell it to compare.

> 4. [...]

It depends on why you wish to compare them.

If you wish to be certain that two files
differ (or not) then you should compare them.
For large files, this may take a long time.
Comparing checksums may be faster, but is
less certain. (If the checksums differ, then
the files differ. If the checksums match,
then the files are unlikely to differ, where
the meaning of "unlikely" depends on the data
in the files, and on the checksum algorithm
used).

You left out BACKUP /COMPARE.
P Muralidhar Kini
Honored Contributor

Re: Compare files with checksum and differences

Hi Tina,

I carried out a simple test and was able to see a same checksum values for
two different backup savesets.

Check the attachment for the logs.

Regards,
Murali
Let There Be Rock - AC/DC
Steven Schweda
Honored Contributor

Re: Compare files with checksum and differences

> Looks like the XOR algorithm used by VMS
> CHECKSUM can get a collision [...]

Which algorithm can _never_ get a collision?

How many different 32-bit checksums are
possible? How many different 33-bit files
are possible?
Steven Schweda
Honored Contributor

Re: Compare files with checksum and differences

> [...] was able to see a same checksum
> values for two different backup savesets.

HELP BACKUP /CRC

If your files contains their own checksums,
then you might get more collisions than you
would otherwise, depending on which
algorithm(s) you use.
Shriniketan Bhagwat
Trusted Contributor

Re: Compare files with checksum and differences

Hi Tina,

I have seen, now a days more use of /ALGORITHM=MD5 qualifier when calculating the checksum of a file. Even for HP PCSI kits /ALGORITHM=MD5 is used to calculate the checksum.

Regards,
Ketan
Shriniketan Bhagwat
Trusted Contributor

Re: Compare files with checksum and differences

Liu Tina
Occasional Advisor

Re: Compare files with checksum and differences

The customer intend to know if that two files are same or different. They used checksum, it said the files are same. Then they used DCL command "differences". The out put of differences are:

HST103$ diff FTNTSPAR_OLD.RUJ FTNTSPAR_NEW.RUJ

************
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_OLD.RUJ;1
134 CNSH05190010OC? OCOC @├е ├з ┬╖E? @G├з ┬вG? ├з ┬в ├з ┬в ├й  H
135 CNSH05190010OD? ODOD @├е ├з ┬╖E? @G├з ┬вG?
******
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_NEW.RUJ;1
134 CNSH05190010OC? OCOC @├е ├з ┬╖E? @G├з ┬вG? ├з ┬в ├з ┬в `auH
135 CNSH05190010OD? ODOD @├е ├з ┬╖E? @G├з ┬вG?
************
************
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_OLD.RUJ;1
138 CNSH05190036OC? OCOC ├з┬╜ ├и┬╡ E? @G├з ┬вG? ├з ┬в ├з ┬в ├й  H
139 CNSH05190036OD? ODOD ├з┬╜ ├и┬╡ E? @G├з ┬вG?
******
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_NEW.RUJ;1
138 CNSH05190036OC? OCOC ├з┬╜ ├и┬╡ E? @G├з ┬вG? ├з ┬в ├з ┬в `auH
139 CNSH05190036OD? ODOD ├з┬╜ ├и┬╡ E? @G├з ┬вG?
************
************
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_OLD.RUJ;1
142 CNSH05190051OC? OCOC `├з ├з ┬╖E? @G├з ┬вG? ├з ┬в ├з ┬в ├й  H
143 CNSH05190051OD? ODOD `├з ├з ┬╖E? @G├з ┬вG?
******
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_NEW.RUJ;1
142 CNSH05190051OC? OCOC `├з ├з ┬╖E? @G├з ┬вG? ├з ┬в ├з ┬в `auH
143 CNSH05190051OD? ODOD `├з ├з ┬╖E? @G├з ┬вG?
************
************
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_OLD.RUJ;1
146 CNSH05190077OC? OCOC ├и┬╡ E? @G├з ┬вG? ├з ┬в ├з ┬в ├й  H
147 CNSH05190077OD? ODOD ├и┬╡ E? @G├з ┬вG?
******
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_NEW.RUJ;1
146 CNSH05190077OC? OCOC ├и┬╡ E? @G├з ┬вG? ├з ┬в ├з ┬в `auH
147 CNSH05190077OD? ODOD ├и┬╡ E? @G├з ┬вG?
************
************
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_OLD.RUJ;1
150 CNSH05190085OC? OCOC w├о ├й ? @G├з ┬вG? ├з ┬в ├з ┬в ├й  H
151 CNSH05190085OD? ODOD w├о ├й ? @G├з ┬вG?
******
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_NEW.RUJ;1
150 CNSH05190085OC? OCOC w├о ├й ? @G├з ┬вG? ├з ┬в ├з ┬в `auH
151 CNSH05190085OD? ODOD w├о ├й ? @G├з ┬вG?
************
************
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_OLD.RUJ;1
154 CNSH05190119OC? OCOC ?├о ┬░├й ? @G├з ┬вG? ├з ┬в ├з ┬в ├й  H
155 CNSH05190119OD? ODOD ?├о ┬░├й ? @G├з ┬вG?
******
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_NEW.RUJ;1
154 CNSH05190119OC? OCOC ?├о ┬░├й ? @G├з ┬вG? ├з ┬в ├з ┬в `auH
155 CNSH05190119OD? ODOD ?├о ┬░├й ? @G├з ┬вG?
************
************
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_OLD.RUJ;1
158 CNSH05190135OC? OCOC ├з┬╜ ├и┬╡ E? @G├з ┬вG? ├з ┬в ├з ┬в ├й  H
159 CNSH05190135OD? ODOD ├з┬╜ ├и┬╡ E? @G├з ┬вG?
******
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_NEW.RUJ;1
158 CNSH05190135OC? OCOC ├з┬╜ ├и┬╡ E? @G├з ┬вG? ├з ┬в ├з ┬в `auH
159 CNSH05190135OD? ODOD ├з┬╜ ├и┬╡ E? @G├з ┬вG?
************
************
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_OLD.RUJ;1
162 CNSH05190176OC? OCOC ├з┬╜ ├и┬╡ E? @G├з ┬вG? ├з ┬в ├з ┬в ├й  H
163 CNSH05190176OD? ODOD ├з┬╜ ├и┬╡ E? @G├з ┬вG?
******
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_NEW.RUJ;1
162 CNSH05190176OC? OCOC ├з┬╜ ├и┬╡ E? @G├з ┬вG? ├з ┬в ├з ┬в `auH
163 CNSH05190176OD? ODOD ├з┬╜ ├и┬╡ E? @G├з ┬вG?
************
************
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_OLD.RUJ;1
166 CNSH05190184OC? OCOC w├о ├й ? @G├з ┬вG? ├з ┬в ├з ┬в ├й  H
167 CNSH05190184OD? ODOD w├о ├й ? @G├з ┬вG?
******
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_NEW.RUJ;1
166 CNSH05190184OC? OCOC w├о ├й ? @G├з ┬вG? ├з ┬в ├з ┬в `auH
167 CNSH05190184OD? ODOD w├о ├й ? @G├з ┬вG?
************
************
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_OLD.RUJ;1
173 CNSH05190291OC? OCOC ├з┬╜ ├и┬╡ E? @G├з ┬вG? ├з ┬в ├з ┬в ├й  H
174 CNSH05190291OD? ODOD ├з┬╜ ├и┬╡ E? @G├з ┬вG?
******
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_NEW.RUJ;1
173 CNSH05190291OC? OCOC ├з┬╜ ├и┬╡ E? @G├з ┬вG? ├з ┬в ├з ┬в `auH
174 CNSH05190291OD? ODOD ├з┬╜ ├и┬╡ E? @G├з ┬вG?
************
************
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_OLD.RUJ;1
177 CNSH05190879OC? OCOC ?├о ┬░├й ? @G├з ┬вG? ├з ┬в ├з ┬в ├й  H
178 CNSH05190879OD? ODOD ?├о ┬░├й ? @G├з ┬вG?
******
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_NEW.RUJ;1
177 CNSH05190879OC? OCOC ?├о ┬░├й ? @G├з ┬вG? ├з ┬в ├з ┬в `auH
178 CNSH05190879OD? ODOD ?├о ┬░├й ? @G├з ┬вG?
************
************
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_OLD.RUJ;1
181 CNSH05191000OC? OCOC ├и┬╡ E? @G├з ┬вG? ├з ┬в ├з ┬в ├й  H
182 CNSH05191000OD? ODOD ├и┬╡ E? @G├з ┬вG?
******
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_NEW.RUJ;1
181 CNSH05191000OC? OCOC ├и┬╡ E? @G├з ┬вG? ├з ┬в ├з ┬в `auH
182 CNSH05191000OD? ODOD ├и┬╡ E? @G├з ┬вG?
************
************
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_OLD.RUJ;1
185 CNSH05191802OC? OCOC ├з ├о ┬╕├й ? @G├з ┬вG? ├з ┬в ├з ┬в ├й  H
186 CNSH05191802OD? ODOD ├з ├о ┬╕├й ? @G├з ┬вG?
******
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_NEW.RUJ;1
185 CNSH05191802OC? OCOC ├з ├о ┬╕├й ? @G├з ┬вG? ├з ┬в ├з ┬в `auH
186 CNSH05191802OD? ODOD ├з ├о ┬╕├й ? @G├з ┬вG?
************
************
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_OLD.RUJ;1
189 CNSH05191810OC? OCOC ├в ┬м?├з ┬╖E? @G├з ┬вG? ├з ┬в ├з ┬в ├й  H
190 CNSH05191810OD? ODOD ├в ┬м?├з ┬╖E? @G├з ┬вG?
******
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_NEW.RUJ;1
189 CNSH05191810OC? OCOC ├в ┬м?├з ┬╖E? @G├з ┬вG? ├з ┬в ├з ┬в `auH
190 CNSH05191810OD? ODOD ├в ┬м?├з ┬╖E? @G├з ┬вG?
************
************
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_OLD.RUJ;1
193 CNSH05193006OC? OCOC ├в ┬м?├з ┬╖E? @G├з ┬вG? ├з ┬в ├з ┬в ├й  H
194 CNSH05193006OD? ODOD ├в ┬м?├з ┬╖E? @G├з ┬вG?
******
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_NEW.RUJ;1
193 CNSH05193006OC? OCOC ├в ┬м?├з ┬╖E? @G├з ┬вG? ├з ┬в ├з ┬в `auH
194 CNSH05193006OD? ODOD ├в ┬м?├з ┬╖E? @G├з ┬вG?
************
************
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_OLD.RUJ;1
197 CNSH05195084OC? OCOC w├о ├й ? @G├з ┬вG? ├з ┬в ├з ┬в ├й  H
198 CNSH05195084OR? OROR w├о ├й ? @G├з ┬вG? ? ? ├й  H
199 CNSH05195084OT? OTOT w├о ├й ? @G├з ┬вG? ? ? ├й  H
200 CNSH05195183OC? OCOC ├з┬╜ ├и┬╡ E? @G├з ┬вG? ├з ┬в ├з ┬в ├й  H
201 CNSH05195183OR? OROR ├з┬╜ ├и┬╡ E? @G├з ┬вG? ? ? ├й  H
202 CNSH05195183OT? OTOT ├з┬╜ ├и┬╡ E? @G├з ┬вG? ? ? ├й  H
203 CNSH05195191OC? OCOC ?├о ┬░├й ? @G├з ┬вG? ├з ┬в ├з ┬в ├й  H
204 CNSH05195191OD? ODOD ?├о ┬░├й ? @G├з ┬вG?
******
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_NEW.RUJ;1
197 CNSH05195084OC? OCOC w├о ├й ? @G├з ┬вG? ├з ┬в ├з ┬в `auH
198 CNSH05195084OR? OROR w├о ├й ? @G├з ┬вG? ? ? ├й  H
199 CNSH05195084OT? OTOT w├о ├й ? @G├з ┬вG? ? ? ├й  H
200 CNSH05195183OC? OCOC ├з┬╜ ├и┬╡ E? @G├з ┬вG? ├з ┬в ├з ┬в `auH
201 CNSH05195183OR? OROR ├з┬╜ ├и┬╡ E? @G├з ┬вG? ? ? ├й  H
202 CNSH05195183OT? OTOT ├з┬╜ ├и┬╡ E? @G├з ┬вG? ? ? ├й  H
203 CNSH05195191OC? OCOC ?├о ┬░├й ? @G├з ┬вG? ├з ┬в ├з ┬в `auH
204 CNSH05195191OD? ODOD ?├о ┬░├й ? @G├з ┬вG?
************
************
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_OLD.RUJ;1
207 CNSH05196660OC? OCOC ├в ┬м?├з ┬╖E? @G├з ┬вG? ├з ┬в ├з ┬в ├й  H
208 CNSH05196660OR? OROR ├в ┬м?├з ┬╖E? @G├з ┬вG? ? ? ├й  H
******
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_NEW.RUJ;1
207 CNSH05196660OC? OCOC ├в ┬м?├з ┬╖E? @G├з ┬вG? ├з ┬в ├з ┬в `auH
208 CNSH05196660OR? OROR ├в ┬м?├з ┬╖E? @G├з ┬вG? ? ? ├й  H
************
************
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_OLD.RUJ;1
212 CNSH05196900OC? OCOC ├з ├о ┬╕├й ? @G├з ┬вG? ├з ┬в ├з ┬в ├й  H
213 CNSH05196900OD? ODOD ├з ├о ┬╕├й ? @G├з ┬вG?
******
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_NEW.RUJ;1
212 CNSH05196900OC? OCOC ├з ├о ┬╕├й ? @G├з ┬вG? ├з ┬в ├з ┬в `auH
213 CNSH05196900OD? ODOD ├з ├о ┬╕├й ? @G├з ┬вG?
************
************
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_OLD.RUJ;1
216 CNSH05196926OC? OCOC ├и┬╡ E? @G├з ┬вG? ├з ┬в ├з ┬в ├й  H
217 CNSH05196926OD? ODOD ├и┬╡ E? @G├з ┬вG?
******
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_NEW.RUJ;1
216 CNSH05196926OC? OCOC ├и┬╡ E? @G├з ┬вG? ├з ┬в ├з ┬в `auH
217 CNSH05196926OD? ODOD ├и┬╡ E? @G├з ┬вG?
************
************
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_OLD.RUJ;1
220 CNSH05199938OC? OCOC w├о ├й ? @G├з ┬вG? ├з ┬в ├з ┬в ├й  H
221 CNSH05199938OD? ODOD w├о ├й ? @G├з ┬вG?
******
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_NEW.RUJ;1
220 CNSH05199938OC? OCOC w├о ├й ? @G├з ┬вG? ├з ┬в ├з ┬в `auH
221 CNSH05199938OD? ODOD w├о ├й ? @G├з ┬вG?
************
************
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_OLD.RUJ;1
224 CNSH05199953OC? OCOC ?├о ┬░├й ? @G├з ┬вG? ├з ┬в ├з ┬в ├й  H
225 CNSH05199953OD? ODOD ?├о ┬░├й ? @G├з ┬вG?
******
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_NEW.RUJ;1
224 CNSH05199953OC? OCOC ?├о ┬░├й ? @G├з ┬вG? ├з ┬в ├з ┬в `auH
225 CNSH05199953OD? ODOD ?├о ┬░├й ? @G├з ┬вG?
************
************
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_OLD.RUJ;1
228 CNSH05199979OC? OCOC `% ├и┬╡ E? @G├з ┬вG? ├з ┬в ├з ┬в ├й  H
229 CNSH05199979OD? ODOD `% ├и┬╡ E? @G├з ┬вG?
******
File TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_NEW.RUJ;1
228 CNSH05199979OC? OCOC `% ├и┬╡ E? @G├з ┬вG? ├з ┬в ├з ┬в `auH
229 CNSH05199979OD? ODOD `% ├и┬╡ E? @G├з ┬вG?
************

Number of difference sections found: 22
Number of difference records found: 28

DIFFERENCES /IGNORE=()/MERGED=1-
TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_OLD.RUJ;1-
TEMP01$:[HP_MSE.CHECKSUM]FTNTSPAR_NEW.RUJ;1

HST103$ lo

SYSTEM logged out at 29-JUN-2010 13:45:00.50
I have put these two files into HP Lab:
RX66B 16.157.8.68, located in :

$ sh def

SYS$SYSROOT:[SYSMGR.TEST.CHECKSUM]

= SYS$SYSROOT:[SYSMGR.TEST.CHECKSUM]

$ dir */size/date=(c,m)

Directory SYS$SYSROOT:[SYSMGR.TEST.CHECKSUM]

FTNTSPAR_NEW.RUJ;1 4848 23-JUN-2010 17:50:27.71 23-JUN-2010 17:50:27.79

FTNTSPAR_OLD.RUJ;1 4848 23-JUN-2010 17:50:20.29 23-JUN-2010 17:50:20.50

Total of 2 files, 9696 blocks.

$

Please feel free to do your test on this OpenVMS module (16.157.8.68, system/newmanager)

Thanks for your help!
Tina



Shriniketan Bhagwat
Trusted Contributor

Re: Compare files with checksum and differences

Hi Tina,

MD5 algorithm is one of the most common scheme used to calculate checksum other than XOR. Please refer the OpenVMS FAQ from the below link. I would suggest you to use MD5 algorithm instead of default XOR.

http://www.uni-giessen.de/faq/archiv/dec-faq.vms.part1-11/msg00003.html


Regards,
Ketan