topic Re: Too many resources required when I use "gzip" in Operating System - HP-UX

Too many resources required when I use "gzip"

Curtelin — Mon, 21 Aug 2006 11:46:28 GMT

Hello,

Every day, I need to compress a large text file (near 500 Mo for size).
I (must) use the command "gzip" to execute this task, but other applications do not appreciate...CPU is overloaded (near 100 % of WCPU for the process "gzip" when I display all processes with the command "top") and other processes may crash...

How to reduce (limit) the load of the CPU when I use "gzip" (it's not a problem for me if the task is longer), in order other processes don't crash ?

I use the HP-UX "RP2405" server with 2 CPU (650 Mhz each) and 2 gigas for memory.

Thanks a lot for your help and your answers !!!

Re: Too many resources required when I use "gzip"

DCE — Mon, 21 Aug 2006 11:58:09 GMT

One way is to use the nice command (man nice) to assign a lower priority to the gzip command.

Re: Too many resources required when I use "gzip"

Curtelin — Mon, 21 Aug 2006 12:01:17 GMT

Yes I've already tried using "nice -39 gzip my_file" to give the lower priority to the command "gzip" but the process is always the first one displayed in the "top" list, with 100% for WCPU...

Re: Too many resources required when I use "gzip"

A. Clay Stephenson — Mon, 21 Aug 2006 12:02:57 GMT

You can start the process under nice to lower priority or you can use the renice command to lower the priority of an already running process. However, no other processes should crash simply because the machine is heavily loaded. That behavior indicates a basic design flaw in the applications.

Re: Too many resources required when I use "gzip"

DCE — Mon, 21 Aug 2006 12:09:10 GMT

What does ps -l show under the NI column when you run the command with nice?

The reason I ask is that nice -39 problably will not work, since the number after the nice command is added to the existing nice value (usually a default of 20) and the resulting nice level of 59 is outside of nice's limits

nice -19 should get you to a nice level of 39 (the max)

Re: Too many resources required when I use "gzip"

Curtelin — Mon, 21 Aug 2006 12:09:40 GMT

I don't know all the processes used by the only application running on the server (how they work, etc.)...that's why I would like to forbid "gzip" command to load CPU at 100%.

(It's true that the application has probably problems with the OS...)

Re: Too many resources required when I use "gzip"

Tim Nelson — Mon, 21 Aug 2006 12:10:12 GMT

If no other processes are asking for CPU then your gzip will use all the CPU no matter what the nice value is. In this case because it is a 2 cpu system gzip will use only on CPU out of two and your load should be 50%.

If other processes need the CPU then the gzip will go to the bottom of the list as it's nice value is -19.

This is all normal.

Processes should not crash.

Now, IO is a different story, if the gzip has the disk pegged out and other processes need to IO to that disk...

Re: Too many resources required when I use "gzip"

Curtelin — Mon, 21 Aug 2006 12:18:58 GMT

Well the line "average" displays near 50% (for CPU, USER) when I use the command "gzip", but it seems that the problem is the "%WCPU" : it's always 98-100% (for example if I launch 2 gzip, both processes display 98% for WCPU)
How to reduce this WCPU percent ? what does it mean please ?
Thanks.

Re: Too many resources required when I use "gzip"

Bill Hassell — Mon, 21 Aug 2006 12:37:18 GMT

The man page for nice is confusing because a lower priority means that the HP-UX prioirity number will be larger. So nice -39 was the wrong way to go. nice 39 will make gzip run only when there are no other processes needing a CPU. Don't worry about 100% CPU usage. All flavors of Unix are timeshare systems and they can easily run 200% to 900% workloads with no problems at all You will see this with the uptime command. A workload of 2.00 for your 2 processor system means that both processors were fully used during the measurement periods. A workload of 4 means that each CPU-bound process will get about half the available time. That is what timesharing is all about. I have never heard of processes that crash due to heavy CPU usage unless these processes heavily depend on timing (rather than semaphores). In that case, find new programmers -- timing dependent code is a great example of bad programming.

So gzip will indeed consume some CPU cycles but like all data processing programs, it will relinquish the CPU each time it performs a read or write. This can happen dozens of times per second so everything will get it's fair share of disk and CPU. The nice priority only affects the compute time so if this large file is on a mountpoint that is heavily used by your applications, then gzip's I/O will slow things down and nice won't help.

Re: Too many resources required when I use "gzip"

A. Clay Stephenson — Mon, 21 Aug 2006 12:46:08 GMT

Well, if nice or renice isn't good enough for you then it would be rather easy to craft a very small perl script that would read stdin and output to stdout but be passed 2 parameters (e.g. -l count -s seconds) so that after every "count" lines the process would sleep "seconds" seconds. Gzip will read from stdin and write to stdout so your sleeper script would pipe its output to gzip. This would free up the CPU so that outher processes can run more often.

By the way, by any chance have you "improved" your timeslice and set it to a very low value (1) rather than the default 10?

Re: Too many resources required when I use "gzip"

Curtelin — Mon, 21 Aug 2006 12:49:46 GMT

"nice 39 gzip toto.tar" doesn't work...I must put a "-" before 39 and then I got "nice = 39" for the gzip process when I display processes with the command "top".

What is the timeslice ? :)

Re: Too many resources required when I use "gzip"

A. Clay Stephenson — Mon, 21 Aug 2006 12:55:46 GMT

It's a kernel tunable and if you have to ask, you are the wrong person to fix it.

Re: Too many resources required when I use "gzip"

Curtelin — Mon, 21 Aug 2006 12:57:52 GMT

Yes I don't know enough Unix kernel...
It's just to find a solution in order the application used on the server doesn't crash when I use gzip with a large file...

Re: Too many resources required when I use "gzip"

Curtelin — Mon, 21 Aug 2006 13:02:40 GMT

About the script, it's very interesting...
Is it difficult to create ? I only know ksh or csh scripts...
Could you help me please ?

You mean I can "cut" my large file, then compress each part of my file with a delay between each compression, and obtain only one compressed file at the end ?

Re: Too many resources required when I use "gzip"

A. Clay Stephenson — Mon, 21 Aug 2006 13:08:15 GMT

No, I mean read 5000 lines or so and then sleep 2 seconds or so and read another 5000 lines and ... until the file is completely read. You could do it in the shell but perl would be much more efficient.

This is a band-aid approach to fixing the underlying problem. Your applications should not crash.

Re: Too many resources required when I use "gzip"

Curtelin — Mon, 21 Aug 2006 13:11:17 GMT

Ok thanks, I "see" the mecanism...
The problem is that I "tar" the text file first before zipping it (because it's possible that there are more than one text file sometimes...)
So I don't know if I can "count" the lines of a tar file.

Re: Too many resources required when I use "gzip"

A. Clay Stephenson — Mon, 21 Aug 2006 13:17:47 GMT

No you cannot but I find it interesting that tar does not choke your box but gzip does. Gzip is more CPU intensive but should be no more i/o intensive than tar. I was leaning towards Perl because I thought there was more to the story but the nice thing about Perl is that it can handle binary data so rather than specifying lines you might specify a number of bytes.

Re: Too many resources required when I use "gzip"

Steven E. Protter — Mon, 21 Aug 2006 13:35:54 GMT

Shalom,

Bottom line is gzip is CPU intensive. There is no way around that.

I would suggest that you make sure any patches are available for gzip you get them.

Alternatively you may wish to stop using gzip altogether.

You've got valid solutions for lowering cpu use of gzip. What also might be causing problems and crashes is that while gzip is gzipping there are two copies of the file. filename and filname.gz

You may be stressing a critical filesystem. Perhaps try one with lots of extra space and low i/o.

SEP

Re: Too many resources required when I use "gzip"

Curtelin — Tue, 22 Aug 2006 03:06:20 GMT

It's better to have extra space using gzip ? even if the space is enough to store the 2 files ?

How to know my gzip version ? is it the version of "zlib" ? I got zlib v1.1.4 when I do "swlist"

Re: Too many resources required when I use "gzip"

Cam Diaz — Tue, 22 Aug 2006 04:36:06 GMT

If you have PRM (Process Resource Manager) installed with your OE you could configured it to do what you want.