Open VMS 6.2 block loss on root disk.

[Kenneth Rudgers]

I am experiencing a slow block loss on my shadow set root disk. It seems to be losing blocks every 5 to 10 minutes on the root disk. I have done searches on files for new files existing files that are growing and I cannot seem to find where these blocks are going... is there a search mechanism to find where these blocks are going? its a slow process but I keep running out of space on the root disk and have to reboot the cluster. Any insight to why I am having this block loss would be greatly appreciated. If you need further info please let me know and I will provide whatever you may need to diagnose. Thank you.

[Robert Gezelter · Accepted solution]

Kenneth,

From your description, it sounds like something is writing a log file of some sort on an ongoing basis. The file is not created anew, it is extended as needed.

I would look for files that are MODIFIED recently. I would then check to see which of them is growing.

Another approach is to turn on disk quotas (over allocate the quotas to prevent problems). Then see which UIC has constantly increasing usage.

Diagnosing this may require more than can be done over the forum. If I can be of assistance, please contact me via email.

- Bob Gezelter, http://www.rlgsc.com

[Hein van den Heuvel]

Ken, When I read the title it got me really worried. Loosing blocks leaves holes you can trip over :-).

But I think you just mean that the free space it reducing slowly and unexplanably.

On a system disk the classical suspects would be ACCOUNTNG.DAT, OPERATOR.LOG, ERRLOG.SYS and maybe some MAIL.MAI
New files could be PDMF, FTP, TELNET, FAL logs.

For a file to grow, it must be modified.
So that would be your primary search option.

Your search tool could be good old "DIRECTORY", but that could be slow as it trashed through all directories and file headers: $DIRECT/MODI/SINCE=TODAY

I would hightly recommend using DFU instead.
$DFU SEAR/MODI=SINC=TODAY SYS$SYSDEVICE:
Save it to a file, Do it again tomorrow and DIFF (actually.. may need a double-diff to get back the common file. Conv/excep to an indexed file with no-dup primary? )

A slowly growing file is likely to become highly fragmented. Again, use DFU:

$DFU SEAR/fragm=mini=200 ...
Save output and compare tomorrow if need be

New version of the same file appearing all the time would cause hight version numbers:

$ dfu sea/versi=mini=1000 sys$sysdevice:

Good luck!
Hein van den Heuvel
HvdH Performance Consulting

[Hoff]

The massively brute-force approach:

DIRECTORY /SIZE=ALL /OUTPUT=list1.txt /NOHEAD /NOTRAIL /COLUMN=1 ddcu:[*...]*.*;*

wait

DIRECTORY /SIZE=ALL /OUTPUT=list2.txt /NOHEAD /NOTRAIL /COLUMN=1 ddcu:[*...]*.*;*

DIFFERENCES list1.txt,list2.txt

DFU and such can list directory stuff faster, but DIRECTORY will work well enough for needs here.

The list1 and list2 files should best be on another disk.

There will be matches from a number of files that are active in normal operations, but this will potentially narrow down the list of suspects.

I'd also use ANALYZE/DISK/REPAIR and would look in SYSLOST, and would look at the files that are open and ex-directory. For the latter bits, SHOW DEVICE /FILES ddcu: can potentially be used to spot active channels and applications that have channels open.

Though as Robert G. correctly indicates, this can be more involved than an ITRC discussion.

Stephen Hoffman
HoffmanLabs

[Kenneth Rudgers]

Thanks to all you give me plenty to look at. Greatly appreciated... hopefully I can find the culprit...

[Kenneth Rudgers]

Gents,
I performed a ANALYZE/DISK/REPAIR DSA265: and recieved a ton of these messages... ideas what these are? Is this a clue?


%ANALDISK-W-ALLOCEXT, blocks allocated to lost extension file header
LBN 3232544 to 3232595, RVN 1
%ANALDISK-W-ALLOCEXT, blocks allocated to lost extension file header
LBN 3232604 to 3232651, RVN 1
%ANALDISK-W-ALLOCEXT, blocks allocated to lost extension file header
LBN 3232708 to 3232827, RVN 1
%ANALDISK-W-ALLOCEXT, blocks allocated to lost extension file header
LBN 3232832 to 3232879, RVN 1
%ANALDISK-W-ALLOCEXT, blocks allocated to lost extension file header
LBN 3233080 to 3233083, RVN 1
%ANALDISK-W-FREESPADRIFT, free block count of 1153172 is incorrect (RVN 1);
the correct value is 1196604

[Hoff]

ALLOCEXT? My initial suspicion is simply that this report results from an operation on an active disk volume. It's corrected by the ANALYZE /DISK /REPAIR command.

ALLOCEXT is a very rare error.

On V7.3-1 and later, you can better quiesce the volume with the ANALYZE /DISK /REPAIR /LOCK mechanisms, but V6.2 is too far back for that.

FWIW, I'd be looking to ensure I had a reliable and complete "standalone" BACKUP /IMAGE of the disk involved. This not because of a problem I see here, but simply due to inherent paranoia.

I'd also be looking for ECOs, as this could conceivably be a disk corruption, or a disk error of some sort. ECOs can address known matters, whether with the file system or with shadowing. Not that I see that here. Again, paranoia.

There are tools around to compare shadowset member volumes for differences, as well.

If you see these errors arising regularly (and particularly if these might be the blocks lost), then I'd be looking at what might be trigging the loss. And that would bring me back to the file system, shadowing and all mandatory ECO kits. (The shotgun answer, yes...) I'd run a series of ANALYZE passes over time, and see if this is transient or on-going, or if it's the loss.)

[Hoff]

ALLOCEXT? My initial suspicion is simply that this error report results from an ANALYZE /DISK /REPAIR operation occurring on an active disk volume. It's corrected by the ANALYZE /DISK /REPAIR command.

ALLOCEXT is a very rare error. Not one I can recall having seen before, either.

On V7.3-1 and later, you can better quiesce the volume with the ANALYZE /DISK /REPAIR /LOCK mechanisms, but V6.2 is too far back for that.

FWIW, I'd be looking to ensure I had a reliable and complete "standalone" BACKUP /IMAGE of the disk involved. This not because of a problem I see here, but simply due to inherent paranoia.

I'd also be looking for ECOs, as this could conceivably be a disk corruption, or a disk error of some sort. ECOs can address known matters, whether with the file system or with shadowing. Not that I see that here. Again, paranoia.

There are tools around to compare shadowset member volumes for differences, as well.

If you see these errors arising regularly (and particularly if these might be the blocks lost), then I'd be looking at what might be trigging the loss. And that would bring me back to the file system, shadowing and all mandatory ECO kits. (The "shotgun" answer, yes...) I'd run a series of ANALYZE /DISK /REPAIR passes over time, and see if this is transient or on-going, or if it's the loss.

And I would (for grins?) ensure I had a good BACKUP /IMAGE of the disk, if it contains valuable data, and that I can restore the BACKUP somewhere. (I've had a few surprises over the years here, with write-only tapes and tape errors. Again, paranoia. Not that I see something wrong here.)

[Kenneth Rudgers]

Could it have anything to do with a degraded Raid drive?

[Kenneth Rudgers]

Here is the listing comparison... anything stand out in your eyes? SECURITY_AUDIT$JOURNAL looks like it could be a culprit...

BUZZ[RUDGERSK]>dif list1.txt list2.txt
************
File IBMGS_SA:[RUDGERSK]LIST1.TXT;1
1015 DSA265:[ORACLE.V5122]SAS$WORK204BE4D7.DIR;1
******
File IBMGS_SA:[RUDGERSK]LIST2.TXT;1
1015 DSA265:[ORACLE.V5122]SAS$WORK20400D65.DIR;1
1016 1/4
1017 DSA265:[ORACLE.V5122]SAS$WORK204BE4D7.DIR;1
************
************
File IBMGS_SA:[RUDGERSK]LIST1.TXT;1
1032 263/264
1033 DSA265:[ORACLE.V5122]SPIKECOM7_JOB.LOG;164
******
File IBMGS_SA:[RUDGERSK]LIST2.TXT;1
1034 269/272
1035 DSA265:[ORACLE.V5122]SPIKECOM7_JOB.LOG;164
************
************
File IBMGS_SA:[RUDGERSK]LIST1.TXT;1
1036 263/264
1037 DSA265:[ORACLE.V5122]SPIKECORP7_JOB.LOG;250
******
File IBMGS_SA:[RUDGERSK]LIST2.TXT;1
1038 269/272
1039 DSA265:[ORACLE.V5122]SPIKECORP7_JOB.LOG;250
************
************
File IBMGS_SA:[RUDGERSK]LIST1.TXT;1
1099 DSA265:[ORACLE.V5122.SAS$WORK204005CA]LOADFILE.LOCK$SASEB$DATA;1
1100 0/132
1101 DSA265:[ORACLE.V5122.SAS$WORK204005CA]S$000001.SASEB$UTILITY;1
1102 1/4
1103 DSA265:[ORACLE.V5122.SAS$WORK204005CA]S$000002.SASEB$PUTILITY;1
1104 17/20
1105 DSA265:[ORACLE.V5122.SSH2]AUTHORIZATION.;1
******
File IBMGS_SA:[RUDGERSK]LIST2.TXT;1
1101 DSA265:[ORACLE.V5122.SAS$WORK20400D65]MTRPTS.LOCK$SASEB$DATA;1
1102 0/132
1103 DSA265:[ORACLE.V5122.SAS$WORK20400D65]S$000001.SASEB$UTILITY;1
1104 1/4
1105 DSA265:[ORACLE.V5122.SAS$WORK20400D65]S$000002.SASEB$PUTILITY;1
1106 17/20
1107 DSA265:[ORACLE.V5122.SAS$WORK20400D65]S$000003.LOCK$SASEB$UTILITY;1
1108 0/132
1109 DSA265:[ORACLE.V5122.SSH2]AUTHORIZATION.;1
************
************
File IBMGS_SA:[RUDGERSK]LIST1.TXT;1
9844 726/5372
9845 DSA265:[SYS0.SYSCOMMON.SYSMGR]SECURITY_AUDIT$JOURNAL.OLD;1297
******
File IBMGS_SA:[RUDGERSK]LIST2.TXT;1
9848 740/5372
9849 DSA265:[SYS0.SYSCOMMON.SYSMGR]SECURITY_AUDIT$JOURNAL.OLD;1297
************
************
File IBMGS_SA:[RUDGERSK]LIST1.TXT;1
12490 65045/65048
12491 DSA265:[SYS1.CA_LIC]LICENSEIT.EXE;1
******
File IBMGS_SA:[RUDGERSK]LIST2.TXT;1
12494 65046/65048
12495 DSA265:[SYS1.CA_LIC]LICENSEIT.EXE;1
************
************
File IBMGS_SA:[RUDGERSK]LIST1.TXT;1
21030 726/5372
21031 DSA265:[SYS1.SYSCOMMON.SYSMGR]SECURITY_AUDIT$JOURNAL.OLD;1297
******
File IBMGS_SA:[RUDGERSK]LIST2.TXT;1
21034 740/5372
21035 DSA265:[SYS1.SYSCOMMON.SYSMGR]SECURITY_AUDIT$JOURNAL.OLD;1297
************
************
File IBMGS_SA:[RUDGERSK]LIST1.TXT;1
23610 40/44
23611 DSA265:[SYS1.SYSEXE]UCX$TELNETSYM_TD$P_MVY_CE.LOG;5
******
File IBMGS_SA:[RUDGERSK]LIST2.TXT;1
23614 41/44
23615 DSA265:[SYS1.SYSEXE]UCX$TELNETSYM_TD$P_MVY_CE.LOG;5
************
************
File IBMGS_SA:[RUDGERSK]LIST1.TXT;1
23970 128/272
23971 DSA265:[SYS1.SYSMGR]OPERATOR.LOG;2982
******
File IBMGS_SA:[RUDGERSK]LIST2.TXT;1
23974 130/272
23975 DSA265:[SYS1.SYSMGR]OPERATOR.LOG;2982
************
************
File IBMGS_SA:[RUDGERSK]LIST1.TXT;1
32996 726/5372
32997 DSA265:[SYS2.SYSCOMMON.SYSMGR]SECURITY_AUDIT$JOURNAL.OLD;1297
******
File IBMGS_SA:[RUDGERSK]LIST2.TXT;1
33000 745/5372
33001 DSA265:[SYS2.SYSCOMMON.SYSMGR]SECURITY_AUDIT$JOURNAL.OLD;1297
************
************
File IBMGS_SA:[RUDGERSK]LIST1.TXT;1
35670 66238/66240
35671 DSA265:[SYS3.SYS$STARTUP]CRASH.DAT;1
******
File IBMGS_SA:[RUDGERSK]LIST2.TXT;1
35674 66239/66240
35675 DSA265:[SYS3.SYS$STARTUP]CRASH.DAT;1
************
************
File IBMGS_SA:[RUDGERSK]LIST1.TXT;1
44084 726/5372
44085 DSA265:[SYS3.SYSCOMMON.SYSMGR]SECURITY_AUDIT$JOURNAL.OLD;1297
******
File IBMGS_SA:[RUDGERSK]LIST2.TXT;1
44088 745/5372
44089 DSA265:[SYS3.SYSCOMMON.SYSMGR]SECURITY_AUDIT$JOURNAL.OLD;1297
************
************
File IBMGS_SA:[RUDGERSK]LIST1.TXT;1
46750 128/272
46751 DSA265:[SYS3.SYSMGR]OPERATOR.LOG;2860
******
File IBMGS_SA:[RUDGERSK]LIST2.TXT;1
46754 132/272
46755 DSA265:[SYS3.SYSMGR]OPERATOR.LOG;2860
************
************
File IBMGS_SA:[RUDGERSK]LIST1.TXT;1
55440 731/5372
55441 DSA265:[VMS$COMMON.SYSMGR]SECURITY_AUDIT$JOURNAL.OLD;1297
******
File IBMGS_SA:[RUDGERSK]LIST2.TXT;1
55444 745/5372
55445 DSA265:[VMS$COMMON.SYSMGR]SECURITY_AUDIT$JOURNAL.OLD;1297
************

Number of difference sections found: 14
Number of difference records found: 22

DIFFERENCES /IGNORE=()/MERGED=1-
IBMGS_SA:[RUDGERSK]LIST1.TXT;1-
IBMGS_SA:[RUDGERSK]LIST2.TXT;1

[Hein van den Heuvel]

>> 55440 731/5372
>> 55441 DSA265:[VMS$COMMON.SYSMGR]SECURITY_AUDIT$JOURNAL.OLD;1297
******

Bad luck on the DIFF!
You have the SIZE that changed, but not the name.
Unlile SEARCH you can not make DIFF list the pre-diff record (best I recall)
Fix DIR to use DIR/WID=FILE=xx
(or use DFU!, and why not filter by date?)

Hein.

[Kenneth Rudgers]

Trying to run the DFU commands and it keeps erroring out... was this utility avail in 6.2?

[Hoff]

DFU is a Freeware package.

You need V3.1 from the OpenVMS Freeware distro (or from Jur's site); V3.2 requires V7.3-1.

[Martin Hughes]

You should try to prevent unnecessary writes to your system disk where possible. As previously mentioned, the most common culprits are the accounting file, operator log, and audit journal. If you have another disk with adequate free space, you should consider redirecting these files to another disk. Minimising processes writing to your system disk is desirable for performance reasons, as well as reducing the likelyhood of disk space issues.

You can find instructions on how to redirect these files in the System Manager's Manual volume 2 at;

http://h71000.www7.hp.com/doc/os83_index.html