topic Re: Multiple files??? in Operating System - OpenVMS

Multiple files???

Peter Quodling — Thu, 14 Apr 2005 01:56:54 GMT

I am seeing multiple apparent versions of the same files, turning up in a directory.
NETSERVER.LOG;13 15/15
NETSERVER.LOG;12 15/15
NETSERVER.LOG;11 15/15
NETSERVER.LOG;10 15/15
NETSERVER.LOG;9 15/15
NETSERVER.LOG;8 15/15
NETSERVER.LOG;12 15/15
NETSERVER.LOG;11 6/10
NETSERVER.LOG;10 6/10
NETSERVER.LOG;9 6/10
NETSERVER.LOG;8 6/10

Yet, Dir/full returns

Prd> dir/full n*.log;1

Directory WWW_PRD_ROOT:[000000]

NETSERVER.LOG;1 no such file

NETSERVER.LOG;1 no such file

Total of 2 files, 0/0 blocks.

It's a busy large Raid set (6 drive 0+1)
and people are getting worried.

ALso getting a lot of warnings (badhiblk, freespadrift, bad_nameorder and related...)

Mister Q

Re: Multiple files???

Antoniov. — Thu, 14 Apr 2005 02:09:40 GMT

Hi mister Q,
No such file are the file marked to deleted but not yet really deleted. I guess the deleting process is not terminate. ANAL/DISK could report same information (file marked to delete) and /REPAIR could solve it.
However, I believe you have some network process keep opened these file while some other try to delete them.

Antonio Vigliotti

Re: Multiple files???

David B Sneddon — Thu, 14 Apr 2005 02:24:02 GMT

Peter,

The last time I saw something like that was on a 4 member RAID 0+1
volume where 2 of the members had dropped out.
Are all 6 disks still OK?
The directory looks to be screwed. DFU might be able to help.
Does the directory command loop on that block?
Try using DUMP/DIRECTORY to examine the contents of the
blocks after the one containing the listed files.
I seem to recall that ANALYZE/DISK was unable to repair
the problem I came across.
How big is the directory containing these files?
It looks like possibly a directory shuffle was
interrupted.

Dave.

Re: Multiple files???

Wim Van den Wyngaert — Thu, 14 Apr 2005 02:37:43 GMT

Is it possible that WWW_PRD_ROOT is a search list ? Do sho log WWW_PRD_ROOT to find out.
Then the multiple files may be normal (do a dir for each part of WWW_PRD_ROOT to find out where the files really are).

That combined with open files that are deleted could be very normal.

Wim

Re: Multiple files???

Uwe Zessin — Thu, 14 Apr 2005 02:49:40 GMT

> No such file are the file marked to deleted but not yet really deleted.

Sorry, but I don't agree. This error message implies a 'dangeling directory entry'. It means that the directory entry points to a file header/ revision number combination that is no longer in use.

A simple 'DELETE NETSERVER.LOG;vers' should remove the dangeling entry in recent versions of OpenVMS - else, use '$ set FILE/REMOVE'.

File names or file revisions out of order indicate a corruption of the directory file. It is possible that a maintanance operation was aborted by a system crash.

The people have a good reason to be worried!

You might want to dismount the disk and take a BACKUP/PHYSICAL copy before you start any repairs.

After that, I agree with Antonio to further analyze the problem with '$ ANALYZE /DISK_STRUCTURE'.

Re: Multiple files???

Willem Grooters — Thu, 14 Apr 2005 03:00:16 GMT

Just can answer on your first concern, having no real experience on the second. Others have already given theirs.

is logical WWW_PRD_ROOT: a searchlist? if you SET DEF WW_PRD_ROOT before, DIR will run down each entry - each directory - and you'll get multiple lines, as shown.
This is standard behaviour; it has nothing to do with the storage architecture.

If not, there IS a serious problem with your system, but then you would be too busy to consult ITRC....

Re: Multiple files???

Jan van den Ende — Thu, 14 Apr 2005 03:06:52 GMT

Q,

do not yet discards Willem's idea.
If you defined your WWW_PRD_ROOT (obviously a Concealed Device) to be the concealed logical name for multiple directories, and files of the same name appear in different dirs, then the output looks to be out-of-sequence, like what you get. Defining a concealed logical for each directory tree, and defining a (NOT concealed) search list over those ROOTs, will generate dir headers between the various listings.

Your DIR/FULL presenting TWO files with the same name also points in that direction.

The "no such file" issue may well be a separate thing, which DOES need further attention, as indicated by Uwe.

Proost.

Have one on me.

Jan

Re: Multiple files???

Uwe Zessin — Thu, 14 Apr 2005 03:10:47 GMT

Peter wrote:
> ALso getting a lot of warnings (badhiblk, freespadrift, bad_nameorder and related...)

So, let's take a look:
$ help/message bad_nameorder

Re: Multiple files???

Wim Van den Wyngaert — Thu, 14 Apr 2005 03:26:17 GMT

Did a test : try to delete a file that is open (shared). And try to find file afterwards with dir.

6.2, 7.3, 8.2 : no files found

So, the message "no such file" could indicate problems.

Wim

Re: Multiple files???

Uwe Zessin — Thu, 14 Apr 2005 04:05:26 GMT

Of course. If you delete an open file, the system removes the directory link, but keeps the file header. Should the system now crash, you will have a 'lost file' - a header without a directory entry pointing to it -- the opposite of a dangeling directory entry ;-)

Re: Multiple files???

Jan van den Ende — Thu, 14 Apr 2005 04:36:27 GMT

To quickly generate a "no such file":

$ CREATE X.X
^Z
$ SET FILE /ENTER=Y.Y X.X
$ DELETE X.X.*
$ DIR Y.Y

Proost.

Have one on me.

Jan

Re: Multiple files???

Uwe Zessin — Thu, 14 Apr 2005 04:47:58 GMT

You would have to ammend your directory command with a qualifier that forces a request of some data from the file header, e.g.
$ directory /size Y.Y

else, directory will just display the file name.

I've just tried this under OpenVMS Alpha V7.3-2, but this games does not work any longer on a disk that has hard links enabled ;-)

Re: Multiple files???

Hein van den Heuvel — Thu, 14 Apr 2005 07:15:03 GMT

Hi there mister Q. Long time no see.
(Per still lives on Queens way)

Considering that the sizes are different, we must be speaking about different file. But why speculate if you can be sure... for your next DIR command add in /FILE this will report the FILE ID (which points to the file header). This is the only information in the directory itself. Any other detail (size, date) will come form the header and may cause 'no such file' even if the directory entry is there.

Now in your example you use: "NETSERVER.LOG;1", and you give no 'proof' that this file might simple not exist. That would be a very good reason for "no files found", but not for "no such file", the latter message comes from the broken link, as mentioned earlier. I suspect that for a file it an extension header, a missing extention could cause this also.

DUMP is your friend to analyze further. DUMP/DIR as mentioned, but also DUMP/IDEN to go after the reported file ID and see what it there now.

You may want to blow away the directory (in a soft, recoverable way: set file /nodir, rename) and have DFU or ANA/DISK find back the files.

>>> ALso getting a lot of warnings (badhiblk, freespadrift, bad_nameorder and related

That would worry me all right!

Start doublechecking the backup procedures, logs. Start reviewing the recovery procedure.

Good luck

Hein.

Re: Multiple files???

Lawrence Czlapinski — Thu, 14 Apr 2005 12:57:06 GMT

Peter,
1. www_prd_root probably has multiple translations. The multiple versions are probably from 2 different directories especially since they are in 2 groups of desending sequence with different sizes. So that part is probably ok. As previously suggested do show log/sys of www_prd_root and do individual dir n*.log.
SYS$SYSROOT is an example of a multiple translation logical.
2. The no such file may be a real problem.
Perhaps a delete of the files was interrupted. Do another dir n*.log;1 just to be sure that you didn't catch it while it was being deleted. It's not like, but just to cover that possibility. If you get the same results, than do as others have suggested to correct it.

Re: Multiple files???

John Gillings — Thu, 14 Apr 2005 22:40:13 GMT

Hey Q!

I'd look at DIR/FILE then examine the files by FID. This might be an artifact of a search list, or it could be directory corruption (bad_nameorder doesn't look good).

If it is directory corruption, I'd recommend repairing the directories with ANALYZE/DISK/REPAIR *AND* DFU. Make sure your disk is clean, then watch the directory carefully for a few weeks to see if the problem returns.

As you know, there are many possible causes of this type of problem, and it's all but impossible to diagnose retrospectively. We need to at least narrow down a time window to have a good crack at figuring out the cause.

Re: Multiple files???

Ian Miller. — Fri, 15 Apr 2005 04:47:14 GMT

It looks like a bad directory file.

The repair for bad_nameorder errors is to get rid of the directory, use ANAL/DISK/REP, or DFU to recover the files now not in a directory and then recreate the directory.

Re: Multiple files???

Peter Quodling — Mon, 18 Apr 2005 02:21:29 GMT

re all...

www_prd_root is not a search list.

show log www_prd_root
"WWW_PRD_ROOT" = "DATA1:[PROD04.WWW_PROD.]" (LNM$SYSTEM_TABLE)
Prd> show log www_prd_root/full
"WWW_PRD_ROOT" [exec] = "DATA1:[PROD04.WWW_PROD.]" [concealed] (LNM$SYSTEM_TABLE)

By File ID I see...
NETSERVER.LOG;13 (402,44510,0) 15/15
NETSERVER.LOG;12 (3109,25741,0) 15/15
NETSERVER.LOG;11 (2686,1568,0) 15/15
NETSERVER.LOG;10 (1444,16908,0) 15/15
NETSERVER.LOG;9 (1030,57182,0) 15/15
NETSERVER.LOG;8 (929,16481,0) 15/15
NETSERVER.LOG;7 (3715,57855,0) 15/15
NETSERVER.LOG;6 (932,29654,0) 15/15
NETSERVER.LOG;5 (3544,5997,0) 15/15
NETSERVER.LOG;4 (1758,20663,0) 15/15
NETSERVER.LOG;3 (479,6900,0) 15/15
NETSERVER.LOG;2 (3511,13613,0) 15/15
NETSERVER.LOG;1 (3466,1304,0) 15/15
NETSERVER.LOG;12 (5752,21700,0) 15/15
NETSERVER.LOG;11 (3534,8418,0) 6/10
NETSERVER.LOG;10 (5112,2341,0) 6/10
NETSERVER.LOG;9 (3425,11024,0) 6/10
NETSERVER.LOG;8 (5127,49434,0) 6/10
NETSERVER.LOG;7 (3825,23594,0) 6/10
NETSERVER.LOG;6 (3379,1160,0) 6/10
NETSERVER.LOG;5 (3378,24912,0) 6/10
NETSERVER.LOG;4 (3375,6237,0) 6/10
NETSERVER.LOG;3 (2858,19727,0) 6/10
NETSERVER.LOG;2 (4097,7333,0) 6/10
NETSERVER.LOG;1 (4118,7704,0) 6/10

Total of 82 files, 451980/452230 blocks.
P

Looks like there goes the upcoming long weekend.

Peter

Re: Multiple files???

Paul Jerrom — Mon, 18 Apr 2005 18:32:52 GMT

So is DATA1: a logical name search list?

I've seen the "no such file" problems when restoring incremental backups. Until all the incrementals are restored, missing files are flagged as existing in the directory, but give a "no such file" error when you try to look at the file itself (because there is no such file!!).

Re: Multiple files???

Peter Quodling — Mon, 18 Apr 2005 20:33:33 GMT

No such luck.

Prd> show log/full data1
"DATA1" [exec] = "$2$DRA1:" [concealed,terminal] (LNM$SYSTEM_TABLE)

And there haven't been any restores to this disk in recent times.

q

Re: Multiple files???

Antoniov. — Tue, 19 Apr 2005 02:44:11 GMT

Hi Q,
attached file is an example how you can see "No such file" message.
The command file creates a lot of TEMP*.TMP files, then spawn a subprocess to delete them while executes dir command.
While subprocess is deleting files, the dir catches some entry of deleted file.
Obviously, this is a merely temporary situation but for some weird reason in your case is not temporary.

Antonio Vigliotti
P.S. If you run attached example, please, execute it in a temporary directory.