delete file, very very slow

[Scc_3]

Hello All,
I was wonder when I try to delete all the files out from one directory it took so long.

I even try to create a .com file and run it thru. batch run still very very slow.

examples :
$ set def dka400:[trs.data]
$ delete *.*;*

very very slow

other drive are the same
$ set def dka100:[trs.data]
$ delete *.*;*

samething, why I know is very slow, because I put /log on before, it delete one file in 30 sec. I set it as a batch run took 4 hrs still not complete delete the files start with an "a"

Thanks !
SCC

[Uwe Zessin · Accepted solution]

You must have a very large directory...

The problem is this:
- all files are stored alphabetically sorted in the directory file
- deletion starts at the front of the file
- when a block is 'empty' (consists no more filenames)
-- ALL remaining blocks will be copied forward
-- (data from block 1 is copied to block 0,
-- data from block 2 to block 1, and so on)
- deletion is continued (at the first block)

--

The most efficient way is to delete in reverse order - a somewhat more efficient way is to do this:
$ delete Z*.*;*
$ delete Y*.*;*
...
$ delete 1*.*;*
$ delete 0*.*;*

This reduces the forward shuffeling of directory blocks somewhat.

[Ian Miller.]

See also
DFU DELETE
http://h71000.www7.hp.com/freeware/freeware70/dfu/

[Andy Bustamante]

It all depends on what else the disk is doing
and how many files are in the directory. What version of VMS you're using may also apply, later versions support large directories better.

To start, please post the results of
$ dir dka400:[trs]data.dir/size=all
along with the version of VMS you're using. The magic number for pre (7.2) is 128 blocks. One work around is to delete from end of the directory first (delete az.dat then ay.dat then ax.dat). DFU also has an option to delete a directory. This is a freeware utilty that can be found on the VMS freeware CD kits.

To see how busy a disk is, use
$ monitor disk/item=que
For a single scsi disk, any value of 1 or more generally indicates the disk is saturated.


Andy

[Scc_3]

Hello,
I using delete the z*.*;*.....then 0*.*;*
it move more quicker.
Thannks !
Scc

[Jan van den Ende]

SCC,

Another possible cause may be, if your disk is inited ERASE_ON_DELETE (maybe not so likely, or, your disk has HIGHWATER_MARKING (much more likely, as it is the default).
And in combination with many-file directories it becomes doubly painful.

You can cure that quickly by simply
$ SET VOLUME xxx /NOHIGH

hth,

Proost.

Have one on me.

jpe

[Uwe Zessin]

The highwater mark (implicit erase) is used on allocating storage to prevent that a user can access data that (s)he has not written. I have never heard that it affects the speed of file deletions.

[Scc_3]

Hello,
I usually delete/purge the data files every months. But due to the user request, they would like to keep the data for a year.

Since I have some problem with the disk for the last few month. I have no option to more them to other disk and delete all the files from the orginal directory. Looks to me it slow maybe that disrectory have way way too many files in it.

Thanks !
SCC

[Uwe Zessin]

Storing many files in a directory is very problematic on VMS (as you have found out).

I don't know how your files look like, but if you like to store the data online, I suggest to put a group of files in their own directories - e.g.:

[TOP.YEAR.WEEK]file1, file2, file3

- use directory for work:

[TOP.CURRENT]

- after a week is over:

$ rename [TOP]CURRENT.DIR;1 [TOP.2005]46.DIR
$ create/directory [TOP.CURRENT]

- if you want to delete old data from the first 9 weeks of last year:

$ delete [TOP.2004.0%]*.*;*
$ delete [TOP.2004]0%.DIR;1

[Scc_3]

Hello,

This is not that easy that you think. Due to our business here. We support or selling a lot of building material like roof and floor to our customers (builder). Their quotation is valid for 90 days, and if expire, the program will re-run it and update the selling prices. Usually I delete all the quotes which is 180 days old. And the girls can't get the quotes information anymore since already deleted. That's is why they want to keep in the system longer, if not they have to input it from scratch, and usually they don't have or lost the information already.
Yes there are some other way to keep the old quotes by copy them to another directory like your idea. I can setup the login command for the user to looking for date file
at dka100:[trs.data1] first, if file not found goto dka100:[trs.data2]....
data2 - can be jan, data3 - feb.....

We are average running 75 - 100 quotes per day. That's is alot.

Scc

[Uwe Zessin]

Ah, OK. Thanks for the information - it is always interesting to learn how a system is working.

How about another idea:

- periodically move old data that has not been touched

$ rename [TRS.DATA]*.*;* [TRS.OLD.2005_05] -
/before= "-180-" /CREATED

(/CREATED assumes that a re-run of a quote does create a new file)


- in case you need access to old data, you could RENAME it back to [TRS.DATA]

[Lawrence Czlapinski]

SCC: Another alternative might be to put the files into 12 monthly directories:[TOP.YEAR.month]
Lawrence

[Jan van den Ende]

Re Lawrence:


Another alternative might be to put the files into 12 monthly directories:[TOP.YEAR.month]


.. and define a logical name searchlist over these directories for retrieval!

Proost.

Have one on me.

jpe

[Lawrence Czlapinski]

SCC: You could prompt user for year (with default of current year) and have a menu for month of quote with press ENTER for current month. Most of the requested quotes would be for recent months.
Lawrence

[Hein van den Heuvel]

Scc,

Sounds like you have since deleted the files as needed. For future reference you may want to check out this discussion:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=625667

It also points out DFU as solution, and I posted a little DCL script to 'reverse delete' the files much faster.

> We are average running 75 - 100 quotes per day. That's is alot

That suggests to me that you want to limit directories to hold about a month worth of data. They'll be big, but manageable.

>>> Yes there are some other way to keep the old quotes by copy them to another directory like your idea. I can setup the login command for the user to looking for date file
at dka100:[trs.data1] first, if file not found goto dka100:[trs.data2]....
data2 - can be jan, data3 - feb.....

NO NO! Do not copy them, and that looking for data file is a simple, system wide, search list defintion:

For example, for 3 months ending 2005/11
current directories are:
dka100:[trs.current]
dka100:[trc.200510]
dka100:[trs.200509]

Now when December is to roll around you do
$rename/log dka100:[trs.current] dka100:[trs.200511]
$create/dir dka100:[trs.current]
and in change the SYLOGICAL.COM to:
$DEFINE/SYS TRS_DATA dka100:[trs.current],dka100:[trs.200511],dka100:[trc.200510],dka100:[trs.200509]

Any new file will be created in dka100:[trs.current]
Old files are automagically accessible in their old directories.
The 'flip' is instantaneous. Just rename the directory, do not move files or directory entries around.

When the decision is taken to purge, then do a final backup the oldest directory in the list and deleted the files (or have backup do so). You can drop that directory from the search list, but that is nto critical. I would keep the directory itself, or a fresh empty directory with the same name and put a README file in there indicating who purged, when, and where to find the backup.

Hope this helps,

Hein.


$create/dir

[Steven Schweda]

> $rename/log dka100:[trs.current] dka100:[trs.200511]

In your dreams. (Or is this a new feature?)
I assume that you meant:

$rename/log dka100:[trs]current.dir dka100:[trs]200511.dir

> $create/dir dka100:[trs.current]

No problem there.

> $DEFINE/SYS TRS_DATA dka100:[trs.current],dka100:[trs.200511],dka100:[trc.200510],dka100:[trs.200509]

I'd do that first.

[John Gillings]

The DCL DELETE command deletes files in the "wrong" order for efficient removal of large numbers of files. Rules for directory file manipulation exacerbate the problem.

DELETE/TREE is proposed for a future version of OpenVMS to address this problem (but no promises!)

In the mean time, the fastest way to DELETE *.*;* is to use DFU DELETE/DIRECTORY [/TREE].

[labadie_1]

I suppose that your appli puts files in a directory named disk$appli,
and that your appli is started every morning and shut every evening.

May be you should do the following, define disk$appli as a search-list

The following will automagically roll

def disk$appli disk:,-
disk:,-
disk:,-
disk:,-
disk:,-
disk:,-
disk:

Of course you have to create your directory <.monday> and so

You can of course use a little dcl to have a search-list with many more
elements (day of month comes to mind), and make it roll.

On monday you can quietly move the file in <.tuesday> and other
directories to otherwhere, while not disturbing the application.

have fun

[Sebastian Bazley]

Another way to delete files that may be quicker is to use BACKUP/DELETE.

For example

BACKUP/DELETE *.*;* nl:a.a/SAVE

==

Alternatively, create a list of the file names, sort them in reverse order, and then read the file and pass it to the delete command.

[Jan van den Ende]

Sebastian,

-- BACKUP/DELETE essentially has the same drawbacks on directory emptying as DELETE has.
-- reverse-order DELETEs has the disadvantage of VERY many image activations (although it still will be faster than the directory shuffle!

Proost.

Have one on me.

jpe

[Robert_Boyd]

John Gillings,

Why hasn't anything been done to make DELETE more efficient? Is there some architectural limitation that prevents someone from changing the code to analyze what order/mechanism(s) to use to perform the delete?

It seems like if nothing else if the old behavior MUST be preserved for some arcane reason then add a /FAST qualifier to the utility at least. Why should VMS continue to be tarnished by this agravating feature of one of the oldest utilities on the system?

I can't for the life of me see why this awful old behavior must be maintained since there is no guarantee about the status of the files once you initiate a delete command -- i.e. you don't get to pick the order for how they disappear from the directory.

Robert

[Uwe Zessin]

Present a business case (i.e. 100 big AlphaServer GS) and you might get heard...

[Uwe Zessin]

I forgot...
of course that won't help SCC, who is riding an old VAX.

[Scc_3]

Hello All,

Thank you all your people for the help and great ideas. I will keep a copies and will try to test them all and see which one is best fit for me. I already delete all the files last weekend and copy all the data too to another dir. and call it trs_databu21nov2005.dir

Thank !
SCC

[Ian Miller.]

Issue 1973 - Improve DELETE command efficency

http://www.hpuseradvocacy.org/node/1973

Go and vote today. only five votes needed to get escalated to hp.