Compression of very big files

[Karl Rohwedder]

I tried ZIP V3.0E with different compiler options on a [small] RDB backup file (25MB) on a DS10/600MHz:

- standard compilation
- opt. for EV56
- opt. for EV67

See attached textfile for results.

ergards Kalle

[John Abbott_2]

Just some background info.
We spent some time about 3 yrs ago and found that bzip2 used slightly more cpu than zip(2.1) but bzip2 was better in compressing files. All our files selected were of fixed length. For our large compressions we ended up running them with a $set proc/prio=1 in an attempt not to upset the active system too much. Sorry, I don't have the test results anymore, the jobs run on a EV68 system.

John.

[Wim Van den Wyngaert]

Test results (on a 4100).

Input 447 K(blocks)

Old 2.3 (level / cpu / Kblocks)
1 62 120
2 66 113
3 74 110
4 91 107
5 105 100
6 170 97

New 2.31
1 62 119
2 64 114
3 74 110
4 89 107
5 110 100
6 165 98

Compared new with old for the 6 zips :
The wall time for 6 zips was 40 seconds lower
Total cpu time was 6 seconds lower.
*** Direct IO was almost cut in half ***

BTW : we found level 2 to be best buy.


Wim

[Steven Schweda]

> Test results (on a 4100).

With the programs built how? (Any fancy
compiler/linker options?)

> *** Direct IO was almost cut in half ***

As I said, it has some I/O improvements.
(But SET RMS_DEFAULT can help the older
version considerably.)

> The wall time for 6 zips was 40 seconds lower

Out of what? 50s v. 40s, or 10240s v. 10200s?

It's good to be careful about file caching
when running this kind of test. Otherwise,
the second program to run has an advantage.

Everything's complicated.

[Steven Schweda]

Oops. That should have been "50s v. 10s, or
10240s v. 10200s?".

Sigh. ("Preview >>" doesn't help much
unless one's brain is actually functional.)

[Wim Van den Wyngaert]

Steven,

Standard link.

6 seconds on a total of 9min18.
40 seconds on 14min40.

I did a pre-zip to fill the cache with (part of) the file.

Wim