Operating System - HP-UX
1748010 Members
4393 Online
108757 Solutions
New Discussion юеВ

Re: untar become slow after untarring 160000 files

 
SOLVED
Go to solution
haris m
Occasional Advisor

untar become slow after untarring 160000 files

HI experts,
System HP-UX 11.00 model n4000-44 with 4gb physical memory and two cpu @440Mhx.

Problem. When untarring a archive that contains more tha 160000 files it start to slow down after the 160k mark.

cpu usage is @ 80% then lower down to less than 20%. IO check show that it remain the same trough the whole process. Avwait 5 avserve less then 0.5.

command use to untar is tar xvf "file.tar"

Any idea why? Beside IO constrains.
14 REPLIES 14
Mel Burslan
Honored Contributor
Solution

Re: untar become slow after untarring 160000 files

Do you realize that you have directory with 160K files in it and everytime you want to perform a file operation, system has to scan all 160K of them. More the number of files in your working directory, slower the file i/o related functions. You can not avoid it. You might want to move the files as they get extracted from tar archive into a separate directory, preferably on a different disk device, using a different, parallel process, so that the i/o doesn't get bogged down.

hope this helps
________________________________
UNIX because I majored in cryptology...
Pete Randall
Outstanding Contributor

Re: untar become slow after untarring 160000 files

I would guess that your file system has become horribly fragmented and the OS is searching for space to write out the blocks of data. Try finding another spot to untar this.


Pete

Pete
Ganesan R
Honored Contributor

Re: untar become slow after untarring 160000 files

Hi,

Just a thought. check the patch level of tar. You might be using old version of tar.

#swlist -l patch |grep -i tar
Best wishes,

Ganesh.
James R. Ferguson
Acclaimed Contributor

Re: untar become slow after untarring 160000 files

Hi:

Regardless of whether you are extracting a single file or not, 'tar' will read from the beginning of its archive until the end. That is, even if the file to be extracted is satisfied at the very beginning, 'tar' keeps reading (and potentially extracting again) until the end of the archive is reached!

Regards!

...JRF...
Steven E. Protter
Exalted Contributor

Re: untar become slow after untarring 160000 files

Shalom,

Just as an ls command will be slow in a directory with that many files.

The reason why we can create sub directories is to avoid situations where simple operations like ls or your tar xvf will slow down to a crawl.

This is simply a fact of life for the OS, which is of course out of support. Same thing would happen with other OS' and newer versions of HP-UX.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Dennis Handly
Acclaimed Contributor

Re: untar become slow after untarring 160000 files

>SEP: Same thing would happen with other OS

Not if these other OSes implement each directory as a btree or some such, like MPE/iX.
haris m
Occasional Advisor

Re: untar become slow after untarring 160000 files

Tar patch on the server is as below.

PHCO_20028 1.0 Cummulative tar(1) patch
PHCO_29229 1.0 Cumulative tar(1) patch.

But if it is limited due to the numebr of file it rying to extracts , why does the tarring is not affected? Just want to know if there is a different in archiving and extracting?
haris m
Occasional Advisor

Re: untar become slow after untarring 160000 files

also will adding more resources like memory help? We have to untar ~400K of file from a tarball transfered from another server.
Dennis Handly
Acclaimed Contributor

Re: untar become slow after untarring 160000 files

>PHCO_29229

That's the last tar patch for 11.00.

>if it is limited due to the number of file it trying to extracts, why does the tarring is not affected?

As asked above, where is it putting all of those files? In same directory?

>Just want to know if there is a different in archiving and extracting?

Extracting has to get the OS involved in allocating space and directory entries. But archiving only has to find and copy the data.

>also will adding more resources like memory help?

It depends on where the problem is. What does "swapinfo -tam" and glance show when you have this slowdown?

You may also want to try this on a supported HP-UX version to see if it has the same problem.