topic Re: Housekeeping of old files on HP-UX in Operating System - HP-UX

Housekeeping of old files on HP-UX

ccchung — Sat, 28 Jan 2006 01:24:46 GMT

Hi all,

I need to remove files with mtime over 4 months inside a folder which contain over 40000 files in it.

This folder also contain 3 subfolders which also have over 30000 files.

Before I delete it, I have to provide the list of file to be delete and I use the following command in each folder to list out the name of file

find . -type f -mtime +120 -exec ls -l {} >> result \;

However, it is still running after 30 mins and not completed. It make the whole system response very slow and I have to stop it.

Please kindly help to provide some faster and better solution for my case ? Many Thanks.

Chung

Re: Housekeeping of old files on HP-UX

Devender Khatana — Sat, 28 Jan 2006 01:49:52 GMT

Hi,

You can do the same job in bunches by specifying the file names with wild characters. Something like

#find . -name "*.dbf" -mtime +120 -exec ll {} \;

Like this it will list only the files with the names matching.

HTH,
Devender

Re: Housekeeping of old files on HP-UX

ccchung — Sat, 28 Jan 2006 02:14:57 GMT

Hi Devender,

Thanks for the suggestion.There are only 5 types of files I can found in those folder.

But is it possible to limit the total number of output file when i run the command ?

Since my server is running slow and it have impact to the system performance if i delete all of them in same time , I can only set a cronjob to delete some old files ( say 2000 files ) every day, any idea ?

#find . -name "*.dbf" -mtime +120 -exec ll {} \;

Thanks,
Chung

Re: Housekeeping of old files on HP-UX

Devender Khatana — Sat, 28 Jan 2006 02:24:53 GMT

Hi,

What does this do
#find . -name "*.dbf" -mtime +120 -exec ll {} \; |pg

Will it not limit the output to be only one page at a time. ALlthough if the server is so heavily loaded you could search for some off time (night) to do this sort of jobs.

Also as the you have only 5 types of files are the nsames as well same for those files ? Instead of extentions you could also try various starting names like

"a*.*"
"b*.*"

Can you attach the output of ll in that directory?

HTH,
Devender

Re: Housekeeping of old files on HP-UX

ccchung — Sat, 28 Jan 2006 02:52:29 GMT

Re: Housekeeping of old files on HP-UX

ccchung — Sat, 28 Jan 2006 02:55:54 GMT

Re: Housekeeping of old files on HP-UX

ccchung — Sat, 28 Jan 2006 03:08:28 GMT

Hi Devender,

Sorry for misleading, i mean there exist 5 types of files with similar file name.

#find . -name "RevokeRequest*" -mtime +120 -exec ll {} \;

#find . -name "UploadRequest*" -mtime +120 -exec ll {} \;

However, only one of the folders have this pattern.Other folder contain different name with .zip format

Actually, I am planning to set a cronjob at night for this to avoid system performance.

Please find the below redult of ll in one of my folder.

some output of ll

-rw-rw-r-- 1 oracle oinstall 2576 Aug 9 21:12 ./RevokeRequest.2115603712460820.49695480.0809130351000.74
-rw-rw-r-- 1 oracle oinstall 2584 Aug 9 21:22 ./RevokeRequest.2115643440906802.98036990.0809131104000.74

-rw-rw-r-- 1 oracle oinstall 2598 Aug 9 21:22 ./UploadRequest.2115628945393296.21170640.0809131153000.74
-rw-rw-r-- 1 oracle oinstall 2573 Aug 9 21:42 ./UploadRequest.2115550562241390.44837920.0809133312000.74

-rw-rw-r-- 1 oracle oinstall 216 Aug 16 07:17 ./ReportWork.null.711601.1124147857917.8
-rw-rw-r-- 1 oracle oinstall 215 Aug 16 07:17 ./ReportWork.null.711602.1124147857918.8
-rw-rw-r-- 1 oracle oinstall 216 Aug 16 07:17 ./ReportWork.null.711600.1124147857916.8
-rw-rw-r-- 1 oracle oinstall 215 Aug 16 07:17 ./ReportWork.null.711599.1124147857912.4

-rw-rw-r-- 1 oracle oinstall 216 Aug 16 07:17 ./Schedule.null.711601.1124147857917.8
-rw-rw-r-- 1 oracle oinstall 215 Aug 16 07:17 ./Schedule.null.711602.1124147857918.8
-rw-rw-r-- 1 oracle oinstall 216 Aug 16 07:17 ./Schedule.null.711600.1124147857916.8
-rw-rw-r-- 1 oracle oinstall 215 Aug 16 07:17 ./Schedule.null.711599.1124147857912.4
Thanks,
chung

Re: Housekeeping of old files on HP-UX

ccchung — Sat, 28 Jan 2006 03:09:19 GMT

Re: Housekeeping of old files on HP-UX

Devender Khatana — Sat, 28 Jan 2006 03:44:06 GMT

Hi,
You could still differentiate these file as there is a number in file names which is different for all files. A complete listing would have given a better idea. Still you can do something like this

#find . -name "RevokeRequest.21156*" -mtime +120 -exec ll {} \;|wc -l

Note :wc -l to count the files listed.

#find . -name "UploadRequest..21155*" -mtime +120 -exec ll {} \;|wc -l

Also there are other files as well.
You can do this by cron but I do not suggest that as the process may consume all the resources due to some eventuality and could even cause a crash of application etc.

HTH,
Devender

Re: Housekeeping of old files on HP-UX

James R. Ferguson — Sat, 28 Jan 2006 10:03:14 GMT

Hi Chung:

You can greatly reduce the time your cleanup takes!

When you use '-exec' with 'find' you are spawning a new process for every file that is delivered to you! Process initiation, like birth, is a very, very intensive action.

The key to reducing the amount of work to do is to initiate one process to handle groups (or blocks) of data at a time. This is exactly what 'xargs' is designed to do.

# find . -xdev -type f -mtime +120 | xargs ls -l results.log1

Now, when you want to actually remove the files *and* would also like a trace of what was actually done, do:

# find . -xdev -type f -mtime +120 | xargs -t rm 2> results.log2

Note that the '-t' option to 'xargs' enables a trace of every command executed to be written to STDERR. Hence, note the redirection of file descriptor-2 to the log file.

Using 'xargs' instead of '-exec' will dramatically reduce your execution times!

Regards!

...JRF...

Re: Housekeeping of old files on HP-UX

RAC_1 — Sat, 28 Jan 2006 10:20:34 GMT

You can further reduce the time (after xargs use) by limiting the search to alphabetical splits. Remove files from a-d first and then so on and so forth.

find /dir -name "[aA-dD]*" -exec ll -d {} \; |xargs -t rm 2>>/dir/remove.log

Re: Housekeeping of old files on HP-UX

James R. Ferguson — Sat, 28 Jan 2006 10:56:02 GMT

Hi:

Removing files in groups (e.g. [A-F], [G-K] etc.) as opposed to in one pass will not gain anything.

By breaking one 'find' into several, you will end up walking the same trees, and 'stat'ing the *same* files multiple times before they are ultimately removed.

The use of 'xargs' in lieu of '-exec' will result in your user's thanking you.

Regards!

...JRF...

Re: Housekeeping of old files on HP-UX

dirk dierickx — Mon, 30 Jan 2006 08:13:26 GMT

well, yeah, something like that _can_ take a long time. as long as it gets done, i don't have a problem with it in general.
now, what your problem is seems to be that the process slows down your system. just let it run, but renice it to begin with. it might even take longer to complete, but at least it will not interfere (much) with the rest of your system.

nice

Re: Housekeeping of old files on HP-UX

Tor-Arne Nostdal — Mon, 30 Jan 2006 15:03:50 GMT

So you would:
* Optimize the find by using xargs instead of -exec
and
* Be nicer to the CPU with the "nice" command

A comment to Dirk's "nice" command.

For the command to really be nice to the system you would want to use a positive value!!!

Example:
nice 10 "command"

If you additionally put it to run in background it will be even slightly nicer.

nice 10 "command" &

An unsigned value increases the system nice value causing it to run at lower priority.

A negative value (less nice)=higher priority

Positive value = nicer :-))
because it get lower priority.

/2r