System disk is getting filled continuously

[Joslovevms]

Im having a problem.

 System disk is keep on getting filled continuously. But no file is updated with the time stamp to identify which file is growing in size.

   $ pipe sho dev sys$sysdevice/full | sea tt free

******************************
_MPA8341:[].;

    Free blocks              3238208    Maximum files allowed            500801
$ sho ti
  20-OCT-2011 21:49:41
$ pipe sho dev sys$sysdevice/full | sea tt free

******************************
_MPA8342:[].;

    Free blocks              3237904    Maximum files allowed            500801
$ sho ti
  20-OCT-2011 21:49:45
$ pipe sho dev sys$sysdevice/full | sea tt free

******************************
_MPA8343:[].;

    Free blocks              3237600    Maximum files allowed            500801
$ sho ti
  20-OCT-2011 21:49:49
$ pipe sho dev sys$sysdevice/full | sea tt free

******************************
_MPA8344:[].;

    Free blocks              3237220    Maximum files allowed            500801
$ sho ti
  20-OCT-2011 21:49:52
$

When I checked which file is getting updated

$ dir [*...]*.*;*/sinc=20-OCT-2011:21:49:37/modified/da=(c,m)/size

Directory SYS$SYSROOT:[SYSCOMMON.SYSEXE]

SYSUAF.DAT;2           33782  29-MAR-2000 08:30:39.45  20-OCT-2011 21:52:56.59
VMSMAIL_PROFILE.DATA;1
                        1254   8-DEC-2000 13:07:44.14  20-OCT-2011 21:51:23.30

Total of 2 files, 35036 blocks.

Directory SYS$SYSROOT:[SYSMGR]

XXXXXEXTRACT.RPT;1
                           1  20-OCT-2011 21:51:42.34  20-OCT-2011 21:51:42.36
SSSSSSEXTRACT.RPT;1
                           1  20-OCT-2011 21:51:42.60  20-OCT-2011 21:51:50.33

Total of 2 files, 2 blocks.

Directory SYS$COMMON:[SYSEXE]

SYSUAF.DAT;2           33782  29-MAR-2000 08:30:39.45  20-OCT-2011 21:52:56.59
VMSMAIL_PROFILE.DATA;1
                        1254   8-DEC-2000 13:07:44.14  20-OCT-2011 21:51:23.30

Total of 2 files, 35036 blocks.

Grand total of 3 directories, 6 files, 70074 blocks.
$

I can understand that some file is getting updated but the time stamp is not modified..

COuld this be these files??

$ sho dev sys$sysdevice/files

Files accessed on device $1$DGA4102: (HADES PGA, HADESK) on 20-OCT-2011 21:55:13.44

Process name      PID     File name
                00000000  [000000]INDEXF.SYS;1
                00000000  [SYS0.SYSEXE]PAGEFILE.SYS;6
                00000000  [SYS0.SYSEXE]SWAPFILE.SYS;3
                00000000  [SYS0.SYSEXE]SYS$ERRLOG.DMP;2
                00000000  [VMS$COMMON.SYSEXE]SYSDUMP-CMN.DMP;1

This is a common system disk hosting 5 nodes at the moment. Cant lose this now.

$ sho sys/noproc
OpenVMS V7.3-2  on node XXXXXXX  20-OCT-2011 21:57:05.12  Uptime  100 23:23:46
$

[Bob Blunt]

That is a pretty healthy SYSUAF.DAT...  but any login could be changing the modified date.  The last output, SHOW DEV/FI lists files that are usually not modified in size on an indiscriminate basis.  Identifying files that are increasing may not be as simple as looking at what files have the most recent modified dates.

For starters I'd look at logical names for common files that are prone to growth.  Check to see where the Audit server journal is kept, where OPERATOR.LOG and ACCOUNTNG.DAT live.  All three can be moved to someplace else if 3M blocks free is causing problems.  Another problem child can be ERRLOG.SYS (there have been conditions where it filled the disk because of a bug or two but most of those have been fixed).  There can also be "invisible" temporary files that can affect free space so you might be chasing a phantom or two.  I'd also keep a long-term watch on the files that you've asked about.  Some of them can be moved if you have a problem with them expanding.

In reality the changes in free space that you've posted aren't that strange for a system disk for the amount of time indicated.  One thousand blocks in under 20 seconds isn't much (with 3M+ free).  If that rate sustains for several hours and never recovers it might be alarming but on a system with plenty of OPCOM messages or logging plenty of device/system/memory errors 1K VMS disk blocks isn't that bad unless it never comes back.

bob

[John McL]

The problem could be from any of your five nodes, so you'll have to check all 5 systsems.

Some thoughts:

1 - It's something weriting to an error log.  Check all 5 machines

2 - Try a $ SHOW DEV/FILES SYS$SYSDEVICE and looking at the list, ignoring any .EXE and .SYS files.  Some file might stand out as a likely candidate.  (use /OUTPUT= of maybe a PIPE and SEARCH to make life easier.)

3 - Has image level accounting been enabled on one or more nodes?  In environments that activate lots of images that can chew up disk space.

[John McL]

Bob seems to have posted while I was typing.  His comments prompted me to recall that the Audit log is another file that might be growing, especially if you have attempted break-ins.

By the way, you don't happen to have application log files being written to the system disk, do you ??

[John Gillings]

> I can understand that some file is getting updated but the time stamp is not modified..

  The timestamps won't be updated until the file is closed. In your case the file(s) being extended are probably open, so the dates won't be changing.

Note that SHOW DEVICE/FILES only shows you files open from the current node. You need to look at it on ALL nodes in the cluster. I'd also be astonished if I only saw 5 files open on my system disk. There MUST be more.

Try this command on each node in the cluster

$ pipe show dev/file sys$sysdevice: | search sys$pipe ".EXE;",".COM;","RIGHTSLIST"/MAT=NOR

This will give you a list of candidate files. Check the allocated size of the files over time and see what's growing. You can also use DIRECTORY commands to find suspects. For example:

$ dir/size=all sys$sysdevice:[000000...]/select=size=min=50000

$ dir/size=all sys$sysdevice:[000000...]/select=size=unused=10000

The attachment contains a command procedure which will determine the sizes of open files on the system disk across a cluster (subject to the filtering suggested above). Output is placed in a file called "GROWING.SAMPLE". Execute it twice to compare with the previous sample. The following example shows thhat the ACCOUNTNG log and the DECPS data base both grew between samples.

@GROWING

(wait a while)

@GROWING

************
File DISK_USER0:[JOHNG]GROWING.SAMPLE;3
    6   SYS$SYSDEVICE:[SYS0.SYSMGR]ACCOUNTNG.DAT;5283                                                     136/168
******
File DISK_USER0:[JOHNG]GROWING.SAMPLE;2
    6   SYS$SYSDEVICE:[SYS0.SYSMGR]ACCOUNTNG.DAT;5283                                                     132/168
************
************
File DISK_USER0:[JOHNG]GROWING.SAMPLE;3
   22   SYS$SYSDEVICE:[VMS$COMMON.DECPS-DATABASE]PSDC$JG_2011OCT21.CPD;1                                               2383/7014
******
File DISK_USER0:[JOHNG]GROWING.SAMPLE;2
   22   SYS$SYSDEVICE:[VMS$COMMON.DECPS-DATABASE]PSDC$JG_2011OCT21.CPD;1                                               2367/7014
************

Number of difference sections found: 2
Number of difference records found: 2

DIFFERENCES /MATCH=0/MERGED=0-
    DISK_USER0:[JOHNG]GROWING.SAMPLE;3-
    DISK_USER0:[JOHNG]GROWING.SAMPLE;2

[Hoff]

Some general background: OpenVMS Tips: Your System Disk is Full? DEVICEFULL?

Use DIRECTORY/SIZE=ALLOCATED (or /SIZE=ALL) to go shopping for files that are allocating blocks, but that have not updated the written (used) size.

I'd also try the following:

DIRECTORY/SIZE=ALL XXXXXEXTRACT.RPT;1,SSSSSSEXTRACT.RPT;1

to confirm that those two files aren't the culprits.  And watch for processes that are tossing out I/O requests.

The DFU tool (using its SEARCH command and probably with /ALLOCATE, /SIZE and /SORT) can also be a friend in your quest.

[Joslovevms]

Many thanks for all your replies.

My bad.. I did not give you all the information you needed.

This is the guy.

SYSTEM> dir/size=all TCPIP$SMTP_LOGFILE.LOG;83

Directory SYS$SYSDEVICE:[SYS2.TCPIP$SMTP]

TCPIP$SMTP_LOGFILE.LOG;83
                     4405763/4405796

Total of 1 file, 4405763/4405796 blocks.
SYSTEM> sho ti
  21-OCT-2011 03:23:32
SYSTEM> dir/size=all TCPIP$SMTP_LOGFILE.LOG;83

Directory SYS$SYSDEVICE:[SYS2.TCPIP$SMTP]

TCPIP$SMTP_LOGFILE.LOG;83
                     4405987/4406024

Total of 1 file, 4405987/4406024 blocks.
SYSTEM> sho ti
  21-OCT-2011 03:23:35
SYSTEM> dir/size=all TCPIP$SMTP_LOGFILE.LOG;83

Directory SYS$SYSDEVICE:[SYS2.TCPIP$SMTP]

TCPIP$SMTP_LOGFILE.LOG;83
                     4406224/4406252

Total of 1 file, 4406224/4406252 blocks.
SYSTEM> sho ti
  21-OCT-2011 03:23:39
SYSTEM>

with this error inside

SYSTEM> ty/tail TCPIP$SMTP_LOGFILE.LOG;83
%SYSTEM-F-ACCVIO, access violation, reason mask=00, virtual address=000000000000
0000, PC=FFFFFFFF802D2AB0, PS=0000001B

%%%%%%%%%%%%                   21-OCT-2011 03:21:40.63  %%%%%%%%%%%%
%SYSTEM-F-ACCVIO, access violation, reason mask=00, virtual address=000000000000
0000, PC=FFFFFFFF802D2AB0, PS=0000001B

%%%%%%%%%%%%                   21-OCT-2011 03:21:40.63  %%%%%%%%%%%%
%SYSTEM-F-ACCVIO, access violation, reason mask=00, virtual address=000000000000
0000, PC=FFFFFFFF802D2AB0, PS=0000001B

%%%%%%%%%%%%                   21-OCT-2011 03:21:40.64  %%%%%%%%%%%%
%SYSTEM-F-ACCVIO, access violation, reason mask=00, virtual address=000000000000
0000, PC=FFFFFFFF802D2AB0, PS=0000001B

%%%%%%%%%%%%                   21-OCT-2011 03:21:40.64  %%%%%%%%%%%%
%SYSTEM-F-ACCVIO, access violation, reason mask=00, virtual address=000000000000
0000, PC=FFFFFFFF802D2AB0, PS=0000001B

%%%%%%%%%%%%                   21-OCT-2011 03:21:40.65  %%%%%%%%%%%%
%SYSTEM-F-ACCVIO, access violation, reason mask=00, virtual address=000000000000
0000, PC=FFFFFFFF802D2AB0, PS=0000001B

%%%%%%%%%%%%                   21-OCT-2011 03:21:40.65  %%%%%%%%%%%%
%SYSTEM-F-ACCVIO, access violation, reason mask=00, virtual address=000000000000
0000, PC=FFFFFFFF802D2AB0, PS=0000001B

%%%%%%%%%%%%                   21-OCT-2011 03:21:40.65  %%%%%%%%%%%%
%SYSTEM-F-ACCVIO, access violation, reason mask=00, virtual address=000000000000
0000, PC=FFFFFFFF802D2AB0, PS=0000001B

I will try to stop and start smtp to chk whether this file is getting released.

Will update you all

[Joslovevms]

It did not mov out on smooth exit. I killed the process and started the SMTP through the startup command procedure.

The file released and I have move the file to another directory..

 Now what was the actual problem?

 Any idea from the last few lines of that log which I pasted in the last post?

Thanks once again to all of you, to help me identify the file...

[John Gillings]

That's a common-or-garden ACCVIO. Some code is attempting to dereference a null pointer. Shouldn't happen in a layered product. You should log a case with HP Customer Support.

If you're curious, you might get a clue as to what's happening by symbolising the address. Try this (with CMKRNL privilege)

$ ANALYZE/SYSTEM

SDA> TCPIP SHOW MEMORY/SUMMARY     ! this will drag in TCPIP symbols

SDA>  EXAMINE 802D2AB0

RMS$INIT_EXTENSION_C+009B0:  A04A003C.E6000031   "1..æ<.J."

On my system the address maps into RMS, but yours may be different, and may help identify the problem.

I'd also look at the log file in the region BEFORE all the messages.

Has restarting SMTP stopped the messages?

[Joslovevms]

Hi John,

  Yes, the restart of SMTP has stopped that message. The log file is fine now.

Yes,  I wanted to what was the exact problem.

Here is the output.

$ ana/system

OpenVMS (TM) system analyzer

SDA> TCPIP SHOW MEMORY/SUMMARY
%TCPIP-I-LINKDATE, TCPIP$TNDRIVER linked on  2-OCT-2007 19:12:11.63
%TCPIP-I-LINKDATE, TCPIP$BGDRIVER linked on  2-OCT-2007 19:12:03.38
%TCPIP-I-LINKDATE, TCPIP$PROXY_SERVICES linked on  2-OCT-2007 19:13:19.36
%TCPIP-I-LINKDATE, TCPIP$INTERNET_SERVICES linked on  2-OCT-2007 19:12:05.23
%TCPIP-E-IMAGENF, SYS$SYSTEM:TCPIP$INTERNET_SERVICES_PERF.STB not found; check current process or EXEC list
%TCPIP-W-NOSYMBOLS, Symbols from SYS$SYSTEM:TCPIP$INTERNET_SERVICES_PERF.STB are not available
-SDA-E-IMAGENF, !AS not found; check current process or EXEC list
%TCPIP-E-IMAGENF, SYS$SYSTEM:TCPIP$BGDRIVER_PERF.STB not found; check current process or EXEC list
%TCPIP-W-NOSYMBOLS, Symbols from SYS$SYSTEM:TCPIP$BGDRIVER_PERF.STB are not available
-SDA-E-IMAGENF, !AS not found; check current process or EXEC list
%TCPIP-E-NOPWIP, Pathworks-over-IP (PWIP) driver is not present
%TCPIP-E-NONFSSVCS, NFS Services execlet is not present
%TCPIP-W-NOREAD, unable to access location FFFFFFFF.FFFFFFFF
%TCPIP-W-NOREAD, unable to access location FFFFFFFF.FFFFFFFF
%TCPIP-W-NOREAD, unable to access location FFFFFFFF.FFFFFFFF
%TCPIP-E-ERROR, Block #1 at 7B574508 bad blink 84C87B78 != FFFFFFFF, reversing direction
%TCPIP-W-NOREAD, unable to access location 00000000.00000000
%TCPIP-E-ERROR, Unable to access memory block #2 at 00000000, terminating display
    DESCRIPTION       TYPE    COUNT     SUBTOTAL ALLOCATION
--------------------  ----  ---------  ----------------------
%TCPIP-F-ACCVIO, access violation, reason mask=00, virtual address=0000000000000000, PC=FFFFFFFF80DAA8E8, PS=00000000
SDA> examin 84C87B78
FFFFFFFF.84C87B78:  00000000.00000000   "........"
SDA> examine 7B574508
LIBRTL+E0508:  84C87B78.84C87C38   "8|È.x{È."
SDA>