bdf shows same value after deleting stuff in /var

[NDO]

Hi All

something unusual is occurring since two weeks ago I´ve beed deleting big files in /var/spool/cron/tmp but I can only see the efect of using "rm" when rebooting the server.
Please can somebody explain to me why, why this big files are all of sudden there. I appreciate your help on this.
Although I pass the CSA exame I dont have a lot of experience on the field, so please help if you can.

regards

F.R.

[vishnu.khandare]

Hi Nandinho,

Have u deleted the big files before?
Please check the syslog.
Please check whether u have deleted the correct file?
have u tried to delete the files by rm -rf?
If not please check this.

Hope this solves ur issue
Dont forget to assign points

Regards
Vishnu Khandare

[NDO]

Hi!

I´ve beed deleting file by file, e.g. ( rm croutHLAa01568 ). There is no reference in syslog.log

note: I always assign points when posting queries on the forum.

regards

F.R.

[Duncan Edmonstone]

You have no doubt deleted (a) file(s) that some process(es) still had open...

Your challenge now is to try and identify those processes and stop them if you are able or it is sensible to do so. The easiest way to do that is to get hold of a copy of lsof if you don't have it already:

http://hpux.connect.org.uk/hppd/hpux/Sysadmin/lsof-4.84/

You can then run:

lsof +aL1 /filesystem

To find all "unlinked" files on the filesystem which are still open, and the processes holding them open..

HTH

Duncan

[ani007]

Try to use the find command to search the big files.

#cd /var
#find . -xdev -size +10000 -exec ls -lrt {} \;

before deleting the file check the timestamp
#ll
remove the file. If you see that again the file is there then check the time stamp. May be after deletion again its created automatically. so please check the timestamp before &after deletion.

[ani007]

MR Duncan is right may be you have deleted but due to open files it can happen ,as he suggest pls use lsof.Thank you Ducan

[NDO]

Hi
I´ve used the lsof, this is the output of the command:

/usr/local/bin #./lsof +aL1 /var
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
sh 699 ora10g 1w REG 64,0xa 586861075 0 9780 /var (/dev/vg00/lvol10)
sh 699 ora10g 2w REG 64,0xa 586861075 0 9780 /var (/dev/vg00/lvol10)
sh 703 ora10g 1w REG 64,0xa 586861075 0 9780 /var (/dev/vg00/lvol10)
sh 703 ora10g 2w REG 64,0xa 586861075 0 9780 /var (/dev/vg00/lvol10)
sh 744 ora10g 1w REG 64,0xa 586936815 0 8460 /var (/dev/vg00/lvol10)
sh 744 ora10g 2w REG 64,0xa 586936815 0 8460 /var (/dev/vg00/lvol10)
sshd 916 root 3u unix 64,0xa 0t0 0 4099 /var/spool/sockets/pwgr/client915 (0xbc6173c0)
lpsched 1560 lp 3u unix 64,0xa 0t0 0 4137 /var/spool/sockets/pwgr/client1557 (0xbf6c8c80)
swagentd 1722 root 5u REG 64,0xa 60 0 7327 /var (/dev/vg00/lvol10)
sh 5277 ora10g 1w REG 64,0xa 586861075 0 9780 /var (/dev/vg00/lvol10)
sh 5277 ora10g 2w REG 64,0xa 586861075 0 9780 /var (/dev/vg00/lvol10)
sh 7597 ora10g 1w REG 64,0xa 586897644 0 9090 /var (/dev/vg00/lvol10)
sh 7597 ora10g 2w REG 64,0xa 586897644 0 9090 /var (/dev/vg00/lvol10)
sh 7605 ora10g 1w REG 64,0xa 586897644 0 9090 /var (/dev/vg00/lvol10)
sh 7605 ora10g 2w REG 64,0xa 586897644 0 9090 /var (/dev/vg00/lvol10)
sh 9460 ora10g 1w REG 64,0xa 570483566 0 9355 /var (/dev/vg00/lvol10)
sh 9460 ora10g 2w REG 64,0xa 570483566 0 9355 /var (/dev/vg00/lvol10)
sh 9464 ora10g 1w REG 64,0xa 570483566 0 9355 /var (/dev/vg00/lvol10)
sh 9464 ora10g 2w REG 64,0xa 570483566 0 9355 /var (/dev/vg00/lvol10)
sh 10365 ora10g 1w REG 64,0xa 570483566 0 9355 /var (/dev/vg00/lvol10)
sh 10365 ora10g 2w REG 64,0xa 570483566 0 9355 /var (/dev/vg00/lvol10)
sh 11330 ora10g 1w REG 64,0xa 586897644 0 9090 /var (/dev/vg00/lvol10)
sh 11330 ora10g 2w REG 64,0xa 586897644 0 9090 /var (/dev/vg00/lvol10)
bch 16750 bscsprod 3u REG 64,0xa 4418 0 9473 /var (/dev/vg00/lvol10)
bch 16752 bscsprod 3u REG 64,0xa 4418 0 9541 /var (/dev/vg00/lvol10)
ftp 17820 ora10g 0u REG 64,0xa 209 0 10302 /var (/dev/vg00/lvol10)
ftp 17820 ora10g 2w REG 64,0xa 570483566 0 9355 /var (/dev/vg00/lvol10)
ftp 17824 ora10g 0u REG 64,0xa 119 0 10298 /var (/dev/vg00/lvol10)
ftp 17824 ora10g 2w REG 64,0xa 586861075 0 9780 /var (/dev/vg00/lvol10)
ftp 17829 ora10g 0u REG 64,0xa 209 0 7714 /var (/dev/vg00/lvol10)
ftp 17836 ora10g 0u REG 64,0xa 119 0 8583 /var (/dev/vg00/lvol10)
ftp 17836 ora10g 2w REG 64,0xa 586897644 0 9090 /var (/dev/vg00/lvol10)
ftp 17840 ora10g 0u REG 64,0xa 209 0 9144 /var (/dev/vg00/lvol10)
ftp 17847 ora10g 0u REG 64,0xa 119 0 7611 /var (/dev/vg00/lvol10)
ftp 17847 ora10g 2w REG 64,0xa 586936815 0 8460 /var (/dev/vg00/lvol10)
sh 28456 ora10g 1w REG 64,0xa 586936815 0 8460 /var (/dev/vg00/lvol10)
sh 28456 ora10g 2w REG 64,0xa 586936815 0 8460 /var (/dev/vg00/lvol10)
sh 28461 ora10g 1w REG 64,0xa 586936815 0 8460 /var (/dev/vg00/lvol10)
sh 28461 ora10g 2w REG 64,0xa 586936815 0 8460 /var (/dev/vg00/lvol10)


Would be right to say that the files that I deleted are related to:


sshd 916 root 3u unix 64,0xa 0t0 0 4099 /var/spool/sockets/pwgr/client915 (0xbc6173c0)
lpsched 1560 lp 3u unix 64,0xa 0t0 0 4137 /var/spool/sockets/pwgr/client1557 (0xbf6c8c80)


So what will be the next step?

Regards

R.R.

[NDO]

Hi

I´ve read on another post in the forum that files with link counts = 0 (NKINK) they will be ones that are still in use by a process and are thus holding disk space. If this is correct all those processes shown by lsof command must be killed?
Am I right? Please help

regards

F.R.

[Dennis Handly · Accepted solution]

>If this is correct all those processes shown by lsof command must be killed?

Well, only the ones with large files.
You'll have to determine whether the space is more important than the process.

You seem to have a lot of process by ora10g referencing these large stdout files:
586861075 0 9780
586936815 0 8460
586897644 0 9090
570483566 0 9355

>Would be right to say that the files that I deleted are related to:

Not really. Those seem to be sockets?

[NDO]

Hi Dennis!

Space is more important, so I´ve killed just one, the biggest, but still when I do bdf it shows /var as 97% as it was before.

I´ve used "kill PID number" to kill the process.

I need that space to go down up to 70%,


Regards

Fernando

[Dennis Handly]

>I've killed just one, the biggest

Which? If the file is opened by multiple PIDs, you need to get each.
What does lsof show again?
17847 ora10g 2w 586936815 8460

This is PID 17847, FD 2 (stderr) offset 586936815, inode 8460.

[NDO]

Dennis

/var was 100% now:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
sh 744 ora10g 1w REG 64,0xa 586943780 0 8460 /var (/dev/vg00/lvol10)
sh 744 ora10g 2w REG 64,0xa 586943780 0 8460 /var (/dev/vg00/lvol10)
sshd 916 root 3u unix 64,0xa 0t0 0 4099 /var/spool/sockets/pwgr/client915 (0xbc6173c0)
swagentd 1722 root 5u REG 64,0xa 60 0 7327 /var (/dev/vg00/lvol10)
sh 5277 ora10g 1w REG 64,0xa 586868075 0 9780 /var (/dev/vg00/lvol10)
sh 5277 ora10g 2w REG 64,0xa 586868075 0 9780 /var (/dev/vg00/lvol10)
sh 7597 ora10g 1w REG 64,0xa 586904609 0 9090 /var (/dev/vg00/lvol10)
sh 7597 ora10g 2w REG 64,0xa 586904609 0 9090 /var (/dev/vg00/lvol10)
sh 7605 ora10g 1w REG 64,0xa 586904609 0 9090 /var (/dev/vg00/lvol10)
sh 7605 ora10g 2w REG 64,0xa 586904609 0 9090 /var (/dev/vg00/lvol10)
sh 9460 ora10g 1w REG 64,0xa 570490566 0 9355 /var (/dev/vg00/lvol10)
sh 9460 ora10g 2w REG 64,0xa 570490566 0 9355 /var (/dev/vg00/lvol10)
sh 9464 ora10g 1w REG 64,0xa 570490566 0 9355 /var (/dev/vg00/lvol10)
sh 9464 ora10g 2w REG 64,0xa 570490566 0 9355 /var (/dev/vg00/lvol10)
sh 10365 ora10g 1w REG 64,0xa 570490566 0 9355 /var (/dev/vg00/lvol10)
sh 10365 ora10g 2w REG 64,0xa 570490566 0 9355 /var (/dev/vg00/lvol10)
sh 11330 ora10g 1w REG 64,0xa 586904609 0 9090 /var (/dev/vg00/lvol10)
sh 11330 ora10g 2w REG 64,0xa 586904609 0 9090 /var (/dev/vg00/lvol10)
ftp 27476 ora10g 0u REG 64,0xa 119 0 8464 /var (/dev/vg00/lvol10)
ftp 27476 ora10g 2w REG 64,0xa 586904609 0 9090 /var (/dev/vg00/lvol10)
ftp 27480 ora10g 0u REG 64,0xa 209 0 9144 /var (/dev/vg00/lvol10)
ftp 27507 ora10g 0u REG 64,0xa 119 0 9406 /var (/dev/vg00/lvol10)
ftp 27507 ora10g 2w REG 64,0xa 586943780 0 8460 /var (/dev/vg00/lvol10)
ftp 27517 ora10g 0u REG 64,0xa 209 0 15987 /var (/dev/vg00/lvol10)
ftp 27517 ora10g 2w REG 64,0xa 570490566 0 9355 /var (/dev/vg00/lvol10)
ftp 27521 ora10g 0u REG 64,0xa 119 0 15989 /var (/dev/vg00/lvol10)
ftp 27521 ora10g 2w REG 64,0xa 586868075 0 9780 /var (/dev/vg00/lvol10)
ftp 27525 ora10g 0u REG 64,0xa 209 0 4357 /var (/dev/vg00/lvol10)
sh 28456 ora10g 1w REG 64,0xa 586943780 0 8460 /var (/dev/vg00/lvol10)
sh 28456 ora10g 2w REG 64,0xa 586943780 0 8460 /var (/dev/vg00/lvol10)
sh 28461 ora10g 1w REG 64,0xa 586943780 0 8460 /var (/dev/vg00/lvol10)
sh 28461 ora10g 2w REG 64,0xa 586943780 0 8460 /var (/dev/vg00/lvol10)

I´ve killed pid 744 and 5277 , so bdf went down to 88%

My questions why all this files are created, (croutQEAa01568) . This issue started 2 weeks ago.

Regards

F.R.

[NDO]

Hi!

If I´m not supposed to rm files inside /var/spool/cron/tmp ( croutQEAa01568 ), so what is the procedure to clean that directory, because clearly those files are the ones that are filling up that directory.

regards

F.R.

[Dennis Handly]

>why all this files are created, (croutQEAa01568)?

Ah, you said that at the top.

>If I'm not supposed to rm files inside /var/spool/cron/tmp (croutQEAa01568), so what is the procedure to clean that directory, because clearly those files are the ones that are filling up that directory.

You have some crontab jobs that are going crazy and producing massive output. You might want to do a tail on one of those files next time. They may have an error repeating over and over. Or there may be too much logging going on.

If you want to delete these large files, you have to kill all of the PIDs created under that crontab entry:
UNIX95=EXTENDED_PS ps -H -fu ora10g

Match these trees up with your lsof output.

>I've killed pid 744 and 5277

17824 has the same file open that 5277 did.
17847 28456 8461 have the same file open that 744 did.

[NDO]

Dennis!

Are you saying that these ftp sessions (see below) are the ones that creating all this trouble?:

UNIX95=EXTENDED_PS ps -H -fu ora10g
UID PID PPID C STIME TTY TIME CMD
ora10g 29626 1568 0 Nov 9 ? 00:00 sh -c sh /data1/MONETA/postpaid/cron_BSCSIX_POSTPAID_RETAIL_BILLING.sh
ora10g 29629 29626 0 Nov 9 ? 00:00 sh /data1/MONETA/postpaid/cron_BSCSIX_POSTPAID_RETAIL_BILLING.sh
ora10g 154 29629 0 Nov 9 ? 00:03 sh ftp.sh
ora10g 29120 154 0 14:35:42 ? 00:00 ftp -n -v 192.168.0.85
ora10g 3257 1568 0 02:00:00 ? 00:00 sh -c sh /data1/MONETA/postpaid/cron_BSCSIX_POSTPAID_RETAIL_BILLING.sh
ora10g 3261 3257 0 02:00:00 ? 00:00 sh /data1/MONETA/postpaid/cron_BSCSIX_POSTPAID_RETAIL_BILLING.sh
ora10g 4288 3261 0 02:36:49 ? 00:00 sh ftp.sh
ora10g 29111 4288 0 14:35:23 ? 00:00 ftp -n -v 192.168.0.85
ora10g 3783 1 0 Nov 8 ? 00:07 ora_mman_bscsrtx
ora10g 3787 1 0 Nov 8 ? 00:13 ora_lgwr_bscsrtx
ora10g 2071 1 0 Nov 8 ? 02:50 /fs2/ora10g/bin/ocssd.bin
ora10g 9464 1 0 Nov 10 ? 00:00 sh /data1/MONETA/postpaid/cron_BSCSIX_POSTPAID_RETAIL_BILLING.sh
ora10g 10365 9464 0 Nov 10 ? 00:02 sh ftp.sh
ora10g 29107 10365 0 14:35:19 ? 00:00 ftp -n -v 192.168.0.85
ora10g 3803 1 0 Nov 8 ? 00:05 ora_p001_bscsrtx
ora10g 3807 1 0 Nov 8 ? 00:05 ora_p003_bscsrtx
ora10g 3805 1 0 Nov 8 ? 00:05 ora_p002_bscsrtx
ora10g 3801 1 0 Nov 8 ? 00:05 ora_p000_bscsrtx
ora10g 3795 1 0 Nov 8 ? 00:49 ora_cjq0_bscsrtx
ora10g 3779 1 0 Nov 8 ? 00:49 ora_pmon_bscsrtx
ora10g 3789 1 0 Nov 8 ? 01:00 ora_ckpt_bscsrtx
ora10g 3785 1 0 Nov 8 ? 00:17 ora_dbw0_bscsrtx
ora10g 3781 1 0 Nov 8 ? 00:07 ora_psp0_bscsrtx
ora10g 3797 1 0 Nov 8 ? 00:40 ora_mmon_bscsrtx
ora10g 3793 1 0 Nov 8 ? 00:01 ora_reco_bscsrtx
ora10g 29087 1 0 14:34:22 ? 00:00 ora_j000_bscsrtx
ora10g 3799 1 0 Nov 8 ? 01:37 ora_mmnl_bscsrtx
ora10g 3791 1 0 Nov 8 ? 00:10 ora_smon_bscsrtx
ora10g 3820 1 0 Nov 8 ? 00:05 /fs2/ora10g_2/bin/tnslsnr LISTENER -inherit
ora10g 11330 1 0 Nov 10 ? 00:02 sh ftp.sh
ora10g 29116 11330 0 14:35:42 ? 00:00 ftp -n -v 192.168.0.85
mcelbill[432]/ #


regards

F.R.

[Shibin_2]

Hi Nandinho,

Is it possible to restart your cron ? I guess, you might have deleted the file, which was using by some cron daemon. Stop and start cron might help you.

Thanks.

[NDO]

Hi

I´ve once rebooted the system, but after 3 days I was starting to have the sane issue, so I dont think stopping/starting cron will help. But I´ve done so.

Regards

F.R.

[NDO]

Hi

Any ideas for this problem please


regards

F.R.

[Dennis Handly]

>Are you saying that these ftp sessions (see below) are the ones that creating all this trouble?:

It's possible. That's why you have to look IN those files before you remove them.
But those ftp processes are keeping them open.
I suppose you could use tusc to see what's being written to those files.

>Any ideas for this problem please

Since stdout/stderr can be shared in the whole process tree, you need to find which process is writing all that output.

[NDO]

Hi all!

I´m closing the thread, thanks to all, Dennis in particular, your suggestions did not solve my problem, but get me close to find a solution.

regards

F.R.