Output of "show dev d" seems not correct

[Ricky Pardede]

Hi All,

I suspect I have untraced disk usage on my system. It is on the system disk.

From "show dev $1$DGA100/size=byte/full" :
Total size 100.00GB
Free space 30.16GB
=> means around 70 GB is used.

Meanwhile, from "dir $1$DGA100:[000000...]*.* /size=unit=byte/total" :
Grand total of 3025 directories, 91108 files, 21.73GB
=> it says disk usage is only 21.73 GB

I can not find any folders that take disk usage excessively.

Can anyone suggest why "show dev d" indicates high disk usage on my system ?
Why the output is so much different with "dir /total"

Thanks a lot guys.

[Jan van den Ende]

Ricky,

multiple reasons, but you very probably already get much better agreement if you do those measurements again after
$ SET VOLUME/REBUILD
That will clear (most importantly) any remaining incorrectly closed extended files.
A next step would be
$ ANAL/DISK/REPAIR
That will (most importantly) clear any references to INSTALLed files that were deleted before deINSTALL, or upon INSTALL/REPLACE will still being referenced (ANAL/DISK will report those as Delete Pending)
Furthermore, in an active system some disk space is already "reserved" for quick allocation when wished for. That amount gets multiplied by the number of nodes in a cluster.
Furthermore,
$ dir $1$DGA100:[000000...]*.*
reports only ONE version of any file, if you need to see them all, then
$ dir $1$DGA100:[000000...]*.*;*
or
$ dir $1$DGA100:[000000...]
would give more relevant info.

... and the actual situation can easily be reversed for the system disk, because you see many directories twice!
Each [SYS*]SYSCOMMON.DIR is an alias for [000000]VMS$COMMON.DIR, and all files in there are counted (at least!) twice.

hth.

Proost.

Have one on me.

jpe

[Joseph Huber_1]

I recommend to install and use the DFU utility for this purpose: its REPORT command show all relevant usage/allocation/free data.

[Kris Clippeleyr · Accepted solution]

Hi,

DIRECTORY/SIZE gives the "size used" not the "size allocated". That also might explain (partially) the difference you've calculated.

Regards,
Kris (aka Qkcl)

[Hein van den Heuvel]

Ricky,

This this OpenVMS. It is correct.
Certainly for common, simple, operations such as "SHOW DEV"

So when you write "Output of "show dev d" seems not correct" you should have been thinking "what am I missing in my understanding of the system"

In this case the verification seemed flawed.
The command issued, only reports the USED bytes, not the ALLOCATED bytes.
Change to : /size=(alloc,unit=byte)
Also change to : /size=(all,unit=byte)

Try this on a small directory first, not the whole disk.

Of course it could also be that there files on the disk which are not in any directory, and other tricky situations, but for now let's just assume the difference is in allocated versus used because allocated was not reported / an unknown concept.

Joseph: use the DFU utility
YES! for serious investigation, but this is just basics.


Jan: $ SET VOLUME/REBUILD ... $ ANAL/DISK/REPAIR
YES! for serious investigation, but this is probably just basics.

Jan: *.* reports only ONE version of any file
NO! *.*; reports only one.

Jan: situation can easily be reversed for the system disk
YES!, but let's keep it simple for now.

Good luck!
Hein

[Jan van den Ende]

@Hein:

>>>
Jan: *.* reports only ONE version of any file
NO! *.*; reports only one.
<<<

Oops! My bad! Thank for the correction.

Proost.

Have one on me.

jpe

[Suresh Hegde]

Ricky,

I think, i had this situation too.

I believe this might be because of the two different links (VMS$COMMON and SYSCOMMON) for same directory SYS$SYSDEVICE:[SYS0.SYSCOMMON] in system disk.

For example:

When i do directory for single file from my login directory, i get like below...

1.
>dir [-...]ABS$BACKUP.EXE/size/grand/total

Grand total of 2 directories, 2 files, 2378 blocks.
>

2.

>dir [-...]ABS$BACKUP.EXE/size

Directory SYS$SYSDEVICE:[SYS0.SYSCOMMON.SYSEXE]

ABS$BACKUP.EXE;1 1189

Total of 1 file, 1189 blocks.

Directory SYS$SYSDEVICE:[VMS$COMMON.SYSEXE]

ABS$BACKUP.EXE;1 1189

Total of 1 file, 1189 blocks.

Grand total of 2 directories, 2 files, 2378 blocks.
>

Hope this helps you understand the difference.

/Suresh

[Jon Pinkley]

@Hein,

Although I agree with you that the most likely cause of the discrepancy is that the command was displaying used instead of allocated space, OpenVMS 8.3 (Alpha) still occasionally gets confused. The following shows the results of analyze/disk/repair after I tried to backup about 4 million blocks (2 GB) of files to a device that was showing over 5 million block free. I was surprised when the operation failed with an insufficient space error. This is all I saved, which occurred after the error.

$ show dev DISK$BJLARC01

Device Device Error Volume Free Trans Mnt
Name Status Count Label Blocks Count Cnt
$1$DGA3305: (SIGMA) Mounted 0 BJLARC01 2089392 1 2
$ anal/disk/rep disk$bjlarc01
Analyze/Disk_Structure/Repair for _$1$DGA3305: started on 1-MAR-2010 03:36:14.93

%ANALDISK-I-OPENQUOTA, error opening QUOTA.SYS
-SYSTEM-W-NOSUCHFILE, no such file
%ANALDISK-W-FREESPADRIFT, free block count of 2089392 is incorrect (RVN 1);
the correct value is 0
$

One of the two cluster nodes had been recently rebooted, and the disk mounted with the following mount command:

$ mount/system/noassi/rebuild/subsystem/wind=80 $1$dga3305: bjlarc01/nowrite

but the mount failed because the disk wash mounted for write on the other node. I possibly had recently deleted files from the disk, I can't say for sure. When I mounted it manually, I probably used that exact command without the /nowrite (cut & pasted from our mount_disks.com file)

What is odd is that this disk has very little activity, it is just used to archive old backup journal files. Perhaps the inactivity in some way contributed to the free space drift. Other disks that were frequently modified did not have the problem.

The reason I saved the output was because it was something that happens infrequently. And I wasn't able to reproduce the problem (but I didn't try real hard).

If /size=allocated is used, and there are no alias files on the disk, then the numbers should add. Up.

I just tried on a disk with a relatively small number of large files, no aliases, and in this case, the numbers free + allocated = total (as expected)

$ dir/size=allocated disk$data4:[000000...]/grand

Grand total of 11 directories, 12719 files, 215333504 blocks.
$ show dev disk$data4/ful ! output edited to remove non-relevant info

Disk DSA2200:, device type Generic SCSI disk, is online, mounted, file-oriented
device, shareable, available to cluster, error logging is enabled, device
supports bitmaps (no bitmaps active).

Total blocks 251658240 Sectors per track 128
Logical Volume Size 251658240 Expansion Size Limit 2147475456
Free blocks 36324736 Maximum files allowed 12549313
$ write sys$output 251658240-36324736
215333504
$ write sys$output 251658240-36324736-215333504
0
$

Jon

[Steve Reece_3]

Hi Ricky,

As others have commented, the most likely cause for this (I'm assuming that it's not a system disk) is that the space you're reporting is the used disk space rather than the allocated space. It works like this:

I have a disk of 36GB. Because I want to save space in the index files and bitmaps and such like, I initialize the disk with a cluster factor equivalent to 5MB. That means that the smallest space that I'm able to allocate to a file is 5MB - no matter what the contents of the file are. So, if I have a 2KB file it will have 2KB used of 5MB allocated. If I have a 6MB file it'll have 6MB used of 10MB allocated. Only files that are nice multiples of 5MB will use all of the space allocated to them. The rest of the space will be allocated but not actually used.

If you have any BIG log files that are still open for write (e.g. SET HOST 0/LOG) then this will also allocate space but not use it until the file is closed. On a DIR/SIZ=ALL you'll see things like 0/1778 or 0/9234567 with the used figure on the left and allocated on the right.

An ANALYZE/DISK pass across the device might uncover some lost files or files that the system thinks should be deleted but haven't been and stuff like that, but it sounds like the disk was just badly set up to start with compared with the present use of it.

[Jon Pinkley]

@Suresh,

You make a valid point; that directory /grand will include the contributions from alias files or alias directories, thus overstating the total size that directory reports. However, in this case, Ricky was wondering why show device disk was showing a higher utilization than directory /grand. Alias directories (as exist on the system disk) will make the output of directory [000000...] larger, not smaller. That's why the most likely cause is the discrepancy between directory/size=allocated vs. the default /size=used.

Steve just provided a good explanation of why allocated size is always .ge. used size.

Jon

[Ricky Pardede]

@all : Thanks for mentioning about used and allocated. I didn't aware on this concept.
I believe my case was caused by this wrong reading.

By checking "allocated", I have found a directory that has high usage of allocated space. It's smsc$root:[log]

XXXX:SYSTEM> dir /size=(alloc,unit=byte)/total
Directory SMSC$ROOT:[LOG]
Total of 2656 files, 59.59GB

Then, I try to copy this whole directory to another one but in different disk.
The result :
XXXX:SYSTEM> dir /size=(alloc,unit=byte)/total
Directory SMSC_TRC0:[LOG]
Total of 2599 files, 287.56MB

I was surprised with the comparison result.

The original takes allocated 60GB, and the copied takes allocated just 288MB.

Can please suggest me which related setting I need to adjust on $1$DGA100 ?
Or may be I need to Jan suggestion :
SET VOLUME/REBUILD
ANAL/DISK/REPAIR ?

Thank a lot.

[Ricky Pardede]

I suspect the system disk $1$DGA100 needs to repair.

I deleted the whole files in smsc$root:[log], but still it is showing allocated space 60GB.

XXXX:SYSTEM> dir /size=(alloc,unit=byte)/total
Directory SMSC$ROOT:[LOG]
Total of 58 files, 59.40GB

I am not sure what's the cause.
Can help to suggest me the safest step to try troubleshooting ?

Thanks a lot.

[Joseph Huber_1]

Leave out the /total, and do a dir/size=all to see which files remained.
If there are none, then a rebuild and analyze/repair are the way to recover.
rebuild will free allocated/cached blocks.
analyze/repair finds files which are not in any directory, and puts them into [syslost] directory.

[Joseph Huber_1]

and by the way:
[
dir /size=(alloc,unit=byte)/total
Directory SMSC$ROOT:[LOG]
Total of 58 files, 59.40GB
]

shows there ARE still files, not just allocated
Maybe You did not delete all versions of files ?

[Ricky Pardede]

@Joseph :
Some files can't be deleted because it's locked by another process.

So you suggest me to execute :
anal/disk/repair $1$DGA100 ?

To execute this, I think I need to shutdown all applications and running queue.
Isn't it ?

[Joseph Huber_1]

[
Some files can't be deleted because it's locked by another process.
]

No, in this case a /repair will not change the situation.
Only a stop of the application locking the files will unlock them, then delete.

[p.balamurugan]

show dev d/full. you need to see the cluster_size. For example if cluster_size=200 and you are file is only 10. As per VMS it will allocate the 200 block and content may be 10 block. To reduce this while initialize you need to give the cluster_size parameters. So your disk usage will not be wasted. Here I pasted the example.

25MAY.LIS;1 6/16(it is allocated space and this value will be taken from cluster_size). In the above example you could see size of the file is 6 block. But allocated is 16 block. So you need to reduce the cluster_size value. your issue will be resolved.

[Shriniketan Bhagwat]

Hi,

If your requirement is to have larger cluster factor and want to reduce the allocated but unused blocks of the file, then you can use the $ set file/truncate command. This command truncates the file at the end of the block containing the end-of-file (EOF) marker and releases allocated but unused blocks of the file. Try this on some huge file which has larger unused and allocated blocks. Truncate the file and then check the free space on the disk. This could be better approach than deleting the file (in my opinion).

Regards,
Ketan

[Jan van den Ende]

@Ketan,

not exactly.

SET FILE/TRUNC reduces the file down to the disk cluster containing the EOF.
Then again, so does a normal close of a file, as well as SET VOLUME/REBUILD.
What you refer to is only slightly related: If a file needs to grow, the chunk of growth is the EXTEND size (can be set per file, if not uses the volume's setting) - rounded up to the next cluster size border.
If a file is NOT closed appropriately (program crash, process crash, system crash) then the extended chunk is not rounded down.

hth

Proost.

Have one on me.

jpe

[Joseph Huber_1]

The cluster size is completely irrelevant here:
the OP has 58 files summing up to a total of 59.6 GB, hold locked by an application.
Even if each of the file has once cluster allocated unused, that's nothing compared to the total size.

[Hoff]

If the disk is quiescent, then the reported behavior would be atypical, particularly if it persists after both a forced rebuild (which resets the counts) and an analyze rebuild (which purges old caches and marked-for-delete files).

If the disk is active and with application I/O channels open here, then it's entirely possible to have temporary files active within the file system but that are not visible to DIRECTORY. Caches and scratch files and such.

Active OpenVMS system disks are expected to have various active files and cache activity and locked files.

[Hein van den Heuvel]

I suspect you already know which files have trouble, and possibly why.

But if you need some further clarification, then get DFU installed from http://www.digiater.nl/dfu
(Hint : $ DEFINE DFU$NOSMG YES )

With that in place you ask for the big files:

$ DFU SEARC/SIZE=MINI=1000000 $1$DGA100

You will find this will finish in seconds, not minutes compared to the $ DIR/SIZE=ALL/SELE=SIZE=UNUSED=nnnn

An interesting alternative:

$ DFU SEARCH / OVER_ALLOCATED = n

I also like /FRAG=MINI=nnn
And of course the simple REPORT $1$DGA100

The directory tree deletion is nice also, although I find it too verbose, and I find it odd not being able to specify a directory ( [xxx.yyy] ), but only a directory file ( [xxx]yyy.dir )

$ DFU DELE /DIRE/TREE/NOLOG xxx.DIR

fwiw,
Hein

[Jon Pinkley]

Hoff has a valid point about some files that can use space but are not cataloged in any directory, for example spool files. These are more the exception than the rule, but can account for space that doesn't show up in the output of directory. Analyze/disk/repair will put these files into the [syslost] directory.

That's another reason for using DFU, it can see lost files, since it looks directly in the INDEXF.SYS file (where the file headers live), instead of traversing the directories.

See attachment for an example of creating an over allocated file, "losing it", and showing that directory won't find it, but DFU will.

I agree with Hein, DFU is well worth installing. It is much faster than directory, especially on a device that has 3025 directories and 91108 files.

Note, that if you want to use DFU to find files that are consuming space, you need to use the /allocated switch.

Here's an example of the difference. (trimmed, see attachment for complete output)

$ dfu search /size=min:400000/sort sys$sysdevice

DSA1407:[TCPIP$FTP]TCPIP$FTP_RUN.LOG;697 433780/433792

%DFU-S-FND , Files found : 1, Size : 433780/433792

$ dfu search /size=min:400000/allo/sort sys$sysdevice

DSA1407:[TCPIP$FTP]TCPIP$FTP_RUN.LOG;697 433780/433792
DSA1407:[VMS$COMMON.SYSEXE]SYS$QUEUE_MANAGER.QMAN$JOURNAL;1 2/421792

%DFU-S-FND , Files found : 2, Size : 433782/855584

$

[Jon Pinkley]

As Jan pointed out, truncating a file can't reduce the amount of space occupied by a file to something less than a multiple of the cluster size. In other words, if the clutser size is 16380, and you create a text file with 1 character in it, it will consume 16380 blocks. Truncating it will not reduce the allocated size. Think about it, the bitmap.sys file has a bit for every cluster on the disk, not for every block on the disk, so the allocation "quantum" is one cluster. The EOF is just a record of where the last portion of data is.

Truncation will help if the the difference between used and allocated is .ge. one cluster. However, you can only truncate a file with set file/trucate if the file is not opened by any process (in the cluster). Otherwise you will get a file access conflict error.

This true even if the file is opened for write sharing.

$ copy sys$login:login.com disk$uio:[000000]/allo=100000
$ dir /siz=all disk$uio:[000000]login.com

Directory DISK$UIO:[000000]

LOGIN.COM;245 22/100000

Total of 1 file, 22/100000 blocks.
$ open/app/share=write foo disk$uio:[000000]login.com

From another process..

$ set file/trun disk$uio:[000000]login.com
%SET-E-READERR, error reading DISK$UIO:[000000]LOGIN.COM;245
-SYSTEM-W-ACCONFLICT, file access conflict

Also, analyze disk/repair does not truncate files.

$ close foo ! back to original process that had file open
$ anal/disk/rep disk$uio
Analyze/Disk_Structure/Repair for _DSA3406: started on 6-JUL-2010 16:50:08.87

%ANALDISK-I-SHORTBITMAP, storage bitmap on RVN 1 does not cover the entire device
%ANALDISK-I-OPENQUOTA, error opening QUOTA.SYS
-SYSTEM-W-NOSUCHFILE, no such file
$ dir /siz=all disk$uio:[000000]login.com

Directory DISK$UIO:[000000]

LOGIN.COM;245 22/100000

Total of 1 file, 22/100000 blocks.
$ set file/trun disk$uio:[000000]login.com
$ dir /siz=all disk$uio:[000000]login.com

Directory DISK$UIO:[000000]

LOGIN.COM;245 22/24

Total of 1 file, 22/24 blocks.
$ del disk$uio:[000000]login.com;
$

Jon

[Jon Pinkley]

@Ricky

XXXX:SYSTEM> dir /size=(alloc,unit=byte)/total
Directory SMSC$ROOT:[LOG]
Total of 58 files, 59.40GB

I am not sure what's the cause.
Can help to suggest me the safest step to try troubleshooting ?

-----------------------------------------

These files are open by some application, and you will need to determine what that is. The easiest way is to use the command:

$ show device/files SMSC$ROOT /out=sys$scratch:smsc.files
$ search sys$scratch:smsc.files ".LOG]"

You will need to do this from every member of the VMS cluster, since each node will only display files that are opened on the current node.

This will display the process name and process id (8 hex digits) that has the file open. If the process name does not help, you can use analyze/system to see more about the process.

Suppose this was the output:

SMSC1 20800411 [SMSC.LOG]ABCD.LOG;32

$ analyze/system
SDA> set process/ind=20800411
SDA> show process/chan ! this will display all the files open by the process and will give a good clue
SDA> exit

Now you need to determine how to get that process to close the file. Then you can copy it somewhere else, and restart the process with an new log file.

Jon

a Google search for smsc$root found this

http://www.scribd.com/SMSC-52MR4-Installation-Manual/d/20342007

which appears to be software from Acision that uses RdB. SMSC is an abbreviation for Short Message Service Centre