General
cancel
Showing results for 
Search instead for 
Did you mean: 

untar become slow after untarring 160000 files

SOLVED
Go to solution
haris m
Occasional Advisor

untar become slow after untarring 160000 files

HI experts,
System HP-UX 11.00 model n4000-44 with 4gb physical memory and two cpu @440Mhx.

Problem. When untarring a archive that contains more tha 160000 files it start to slow down after the 160k mark.

cpu usage is @ 80% then lower down to less than 20%. IO check show that it remain the same trough the whole process. Avwait 5 avserve less then 0.5.

command use to untar is tar xvf "file.tar"

Any idea why? Beside IO constrains.
14 REPLIES
Mel Burslan
Honored Contributor
Solution

Re: untar become slow after untarring 160000 files

Do you realize that you have directory with 160K files in it and everytime you want to perform a file operation, system has to scan all 160K of them. More the number of files in your working directory, slower the file i/o related functions. You can not avoid it. You might want to move the files as they get extracted from tar archive into a separate directory, preferably on a different disk device, using a different, parallel process, so that the i/o doesn't get bogged down.

hope this helps
________________________________
UNIX because I majored in cryptology...
Pete Randall
Outstanding Contributor

Re: untar become slow after untarring 160000 files

I would guess that your file system has become horribly fragmented and the OS is searching for space to write out the blocks of data. Try finding another spot to untar this.


Pete

Pete
Ganesan R
Honored Contributor

Re: untar become slow after untarring 160000 files

Hi,

Just a thought. check the patch level of tar. You might be using old version of tar.

#swlist -l patch |grep -i tar
Best wishes,

Ganesh.
James R. Ferguson
Acclaimed Contributor

Re: untar become slow after untarring 160000 files

Hi:

Regardless of whether you are extracting a single file or not, 'tar' will read from the beginning of its archive until the end. That is, even if the file to be extracted is satisfied at the very beginning, 'tar' keeps reading (and potentially extracting again) until the end of the archive is reached!

Regards!

...JRF...
Steven E. Protter
Exalted Contributor

Re: untar become slow after untarring 160000 files

Shalom,

Just as an ls command will be slow in a directory with that many files.

The reason why we can create sub directories is to avoid situations where simple operations like ls or your tar xvf will slow down to a crawl.

This is simply a fact of life for the OS, which is of course out of support. Same thing would happen with other OS' and newer versions of HP-UX.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Dennis Handly
Acclaimed Contributor

Re: untar become slow after untarring 160000 files

>SEP: Same thing would happen with other OS

Not if these other OSes implement each directory as a btree or some such, like MPE/iX.
haris m
Occasional Advisor

Re: untar become slow after untarring 160000 files

Tar patch on the server is as below.

PHCO_20028 1.0 Cummulative tar(1) patch
PHCO_29229 1.0 Cumulative tar(1) patch.

But if it is limited due to the numebr of file it rying to extracts , why does the tarring is not affected? Just want to know if there is a different in archiving and extracting?
haris m
Occasional Advisor

Re: untar become slow after untarring 160000 files

also will adding more resources like memory help? We have to untar ~400K of file from a tarball transfered from another server.
Dennis Handly
Acclaimed Contributor

Re: untar become slow after untarring 160000 files

>PHCO_29229

That's the last tar patch for 11.00.

>if it is limited due to the number of file it trying to extracts, why does the tarring is not affected?

As asked above, where is it putting all of those files? In same directory?

>Just want to know if there is a different in archiving and extracting?

Extracting has to get the OS involved in allocating space and directory entries. But archiving only has to find and copy the data.

>also will adding more resources like memory help?

It depends on where the problem is. What does "swapinfo -tam" and glance show when you have this slowdown?

You may also want to try this on a supported HP-UX version to see if it has the same problem.
haris m
Occasional Advisor

Re: untar become slow after untarring 160000 files

Hi Dennis ,

it put all in the same directories, Is there an option to divide them into separate directories. Or I need to concurrently move the extracted files to differents folders while it continues to extract the remaining files?
Dennis Handly
Acclaimed Contributor

Re: untar become slow after untarring 160000 files

>it put all in the same directories,

In the same single directory?

>Is there an option to divide them into separate directories.

You can toss tar(1) and use pax(1).
With pax's -s, you can rename the directory path, based on the first few chars in the filenames. Do you have an example set of names?

You need to come up with a good hierarchy so the directory sizes aren't huge.

>Or I need to concurrently move the extracted files to different directories while it continues to extract the remaining files?

You could do that. I suppose you could use control-Z on the tar and then move as many files as you want, then resume.

Mel's comment about "preferably on a different disk device" will cause other issues, because it has to read/write each file. Better to move it to some subdirectory. Of course with pax's -s, you can put it where it you want to start with.
haris m
Occasional Advisor

Re: untar become slow after untarring 160000 files

Thanks Dennis . I will explore this options and yes currently all 400k files are extracted to the same single directory. In fact like SEP said, if one do an ls on the folder it take like hours to list them all out.

Will update all if this fix the problem.
haris m
Occasional Advisor

Re: untar become slow after untarring 160000 files

Hi , the solution is to use PAX as Dan sugested.
haris m
Occasional Advisor

Re: untar become slow after untarring 160000 files

Dennis was right. Using pax to archive to 3 or 4 smaller archives and dump it back to the destination servers.