bdf shows same value after deleting stuff in /var

[Dennis Handly]

>I've killed just one, the biggest

Which? If the file is opened by multiple PIDs, you need to get each.
What does lsof show again?
17847 ora10g 2w 586936815 8460

This is PID 17847, FD 2 (stderr) offset 586936815, inode 8460.

[NDO]

Dennis

/var was 100% now:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
sh 744 ora10g 1w REG 64,0xa 586943780 0 8460 /var (/dev/vg00/lvol10)
sh 744 ora10g 2w REG 64,0xa 586943780 0 8460 /var (/dev/vg00/lvol10)
sshd 916 root 3u unix 64,0xa 0t0 0 4099 /var/spool/sockets/pwgr/client915 (0xbc6173c0)
swagentd 1722 root 5u REG 64,0xa 60 0 7327 /var (/dev/vg00/lvol10)
sh 5277 ora10g 1w REG 64,0xa 586868075 0 9780 /var (/dev/vg00/lvol10)
sh 5277 ora10g 2w REG 64,0xa 586868075 0 9780 /var (/dev/vg00/lvol10)
sh 7597 ora10g 1w REG 64,0xa 586904609 0 9090 /var (/dev/vg00/lvol10)
sh 7597 ora10g 2w REG 64,0xa 586904609 0 9090 /var (/dev/vg00/lvol10)
sh 7605 ora10g 1w REG 64,0xa 586904609 0 9090 /var (/dev/vg00/lvol10)
sh 7605 ora10g 2w REG 64,0xa 586904609 0 9090 /var (/dev/vg00/lvol10)
sh 9460 ora10g 1w REG 64,0xa 570490566 0 9355 /var (/dev/vg00/lvol10)
sh 9460 ora10g 2w REG 64,0xa 570490566 0 9355 /var (/dev/vg00/lvol10)
sh 9464 ora10g 1w REG 64,0xa 570490566 0 9355 /var (/dev/vg00/lvol10)
sh 9464 ora10g 2w REG 64,0xa 570490566 0 9355 /var (/dev/vg00/lvol10)
sh 10365 ora10g 1w REG 64,0xa 570490566 0 9355 /var (/dev/vg00/lvol10)
sh 10365 ora10g 2w REG 64,0xa 570490566 0 9355 /var (/dev/vg00/lvol10)
sh 11330 ora10g 1w REG 64,0xa 586904609 0 9090 /var (/dev/vg00/lvol10)
sh 11330 ora10g 2w REG 64,0xa 586904609 0 9090 /var (/dev/vg00/lvol10)
ftp 27476 ora10g 0u REG 64,0xa 119 0 8464 /var (/dev/vg00/lvol10)
ftp 27476 ora10g 2w REG 64,0xa 586904609 0 9090 /var (/dev/vg00/lvol10)
ftp 27480 ora10g 0u REG 64,0xa 209 0 9144 /var (/dev/vg00/lvol10)
ftp 27507 ora10g 0u REG 64,0xa 119 0 9406 /var (/dev/vg00/lvol10)
ftp 27507 ora10g 2w REG 64,0xa 586943780 0 8460 /var (/dev/vg00/lvol10)
ftp 27517 ora10g 0u REG 64,0xa 209 0 15987 /var (/dev/vg00/lvol10)
ftp 27517 ora10g 2w REG 64,0xa 570490566 0 9355 /var (/dev/vg00/lvol10)
ftp 27521 ora10g 0u REG 64,0xa 119 0 15989 /var (/dev/vg00/lvol10)
ftp 27521 ora10g 2w REG 64,0xa 586868075 0 9780 /var (/dev/vg00/lvol10)
ftp 27525 ora10g 0u REG 64,0xa 209 0 4357 /var (/dev/vg00/lvol10)
sh 28456 ora10g 1w REG 64,0xa 586943780 0 8460 /var (/dev/vg00/lvol10)
sh 28456 ora10g 2w REG 64,0xa 586943780 0 8460 /var (/dev/vg00/lvol10)
sh 28461 ora10g 1w REG 64,0xa 586943780 0 8460 /var (/dev/vg00/lvol10)
sh 28461 ora10g 2w REG 64,0xa 586943780 0 8460 /var (/dev/vg00/lvol10)

I´ve killed pid 744 and 5277 , so bdf went down to 88%

My questions why all this files are created, (croutQEAa01568) . This issue started 2 weeks ago.

Regards

F.R.

[NDO]

Hi!

If I´m not supposed to rm files inside /var/spool/cron/tmp ( croutQEAa01568 ), so what is the procedure to clean that directory, because clearly those files are the ones that are filling up that directory.

regards

F.R.

[Dennis Handly]

>why all this files are created, (croutQEAa01568)?

Ah, you said that at the top.

>If I'm not supposed to rm files inside /var/spool/cron/tmp (croutQEAa01568), so what is the procedure to clean that directory, because clearly those files are the ones that are filling up that directory.

You have some crontab jobs that are going crazy and producing massive output. You might want to do a tail on one of those files next time. They may have an error repeating over and over. Or there may be too much logging going on.

If you want to delete these large files, you have to kill all of the PIDs created under that crontab entry:
UNIX95=EXTENDED_PS ps -H -fu ora10g

Match these trees up with your lsof output.

>I've killed pid 744 and 5277

17824 has the same file open that 5277 did.
17847 28456 8461 have the same file open that 744 did.

[NDO]

Dennis!

Are you saying that these ftp sessions (see below) are the ones that creating all this trouble?:

UNIX95=EXTENDED_PS ps -H -fu ora10g
UID PID PPID C STIME TTY TIME CMD
ora10g 29626 1568 0 Nov 9 ? 00:00 sh -c sh /data1/MONETA/postpaid/cron_BSCSIX_POSTPAID_RETAIL_BILLING.sh
ora10g 29629 29626 0 Nov 9 ? 00:00 sh /data1/MONETA/postpaid/cron_BSCSIX_POSTPAID_RETAIL_BILLING.sh
ora10g 154 29629 0 Nov 9 ? 00:03 sh ftp.sh
ora10g 29120 154 0 14:35:42 ? 00:00 ftp -n -v 192.168.0.85
ora10g 3257 1568 0 02:00:00 ? 00:00 sh -c sh /data1/MONETA/postpaid/cron_BSCSIX_POSTPAID_RETAIL_BILLING.sh
ora10g 3261 3257 0 02:00:00 ? 00:00 sh /data1/MONETA/postpaid/cron_BSCSIX_POSTPAID_RETAIL_BILLING.sh
ora10g 4288 3261 0 02:36:49 ? 00:00 sh ftp.sh
ora10g 29111 4288 0 14:35:23 ? 00:00 ftp -n -v 192.168.0.85
ora10g 3783 1 0 Nov 8 ? 00:07 ora_mman_bscsrtx
ora10g 3787 1 0 Nov 8 ? 00:13 ora_lgwr_bscsrtx
ora10g 2071 1 0 Nov 8 ? 02:50 /fs2/ora10g/bin/ocssd.bin
ora10g 9464 1 0 Nov 10 ? 00:00 sh /data1/MONETA/postpaid/cron_BSCSIX_POSTPAID_RETAIL_BILLING.sh
ora10g 10365 9464 0 Nov 10 ? 00:02 sh ftp.sh
ora10g 29107 10365 0 14:35:19 ? 00:00 ftp -n -v 192.168.0.85
ora10g 3803 1 0 Nov 8 ? 00:05 ora_p001_bscsrtx
ora10g 3807 1 0 Nov 8 ? 00:05 ora_p003_bscsrtx
ora10g 3805 1 0 Nov 8 ? 00:05 ora_p002_bscsrtx
ora10g 3801 1 0 Nov 8 ? 00:05 ora_p000_bscsrtx
ora10g 3795 1 0 Nov 8 ? 00:49 ora_cjq0_bscsrtx
ora10g 3779 1 0 Nov 8 ? 00:49 ora_pmon_bscsrtx
ora10g 3789 1 0 Nov 8 ? 01:00 ora_ckpt_bscsrtx
ora10g 3785 1 0 Nov 8 ? 00:17 ora_dbw0_bscsrtx
ora10g 3781 1 0 Nov 8 ? 00:07 ora_psp0_bscsrtx
ora10g 3797 1 0 Nov 8 ? 00:40 ora_mmon_bscsrtx
ora10g 3793 1 0 Nov 8 ? 00:01 ora_reco_bscsrtx
ora10g 29087 1 0 14:34:22 ? 00:00 ora_j000_bscsrtx
ora10g 3799 1 0 Nov 8 ? 01:37 ora_mmnl_bscsrtx
ora10g 3791 1 0 Nov 8 ? 00:10 ora_smon_bscsrtx
ora10g 3820 1 0 Nov 8 ? 00:05 /fs2/ora10g_2/bin/tnslsnr LISTENER -inherit
ora10g 11330 1 0 Nov 10 ? 00:02 sh ftp.sh
ora10g 29116 11330 0 14:35:42 ? 00:00 ftp -n -v 192.168.0.85
mcelbill[432]/ #


regards

F.R.

[Shibin_2]

Hi Nandinho,

Is it possible to restart your cron ? I guess, you might have deleted the file, which was using by some cron daemon. Stop and start cron might help you.

Thanks.

[NDO]

Hi

I´ve once rebooted the system, but after 3 days I was starting to have the sane issue, so I dont think stopping/starting cron will help. But I´ve done so.

Regards

F.R.

[NDO]

Hi

Any ideas for this problem please


regards

F.R.

[Dennis Handly]

>Are you saying that these ftp sessions (see below) are the ones that creating all this trouble?:

It's possible. That's why you have to look IN those files before you remove them.
But those ftp processes are keeping them open.
I suppose you could use tusc to see what's being written to those files.

>Any ideas for this problem please

Since stdout/stderr can be shared in the whole process tree, you need to find which process is writing all that output.

[NDO]

Hi all!

I´m closing the thread, thanks to all, Dennis in particular, your suggestions did not solve my problem, but get me close to find a solution.

regards

F.R.