topic Re: / area getting full. in Operating System - HP-UX

/ area getting full.

kpatel786 — Fri, 20 Feb 2009 10:03:10 GMT

On one of my hpux box in past few weeks the / area gets full upto 100% after further investigation it was found that there is a dsm process which hogs up the space and restarting the same releases the space.

The details of process are as below:

server2 (/) # ps -ef |grep dsm
root 12794 1 0 Feb 13 ? 0:11 dsmc schedule
root 19877 4989 9 09:50:51 ? 0:02 /usr/bin/dsmc selective /apcl/oradata/nlhelma1/nlhelma1_sl_data
root 20250 25585 0 09:52:10 pts/4 0:00 grep dsm

server2 (/) # lsof / |grep dsm
dsmc 12794 root cwd DIR 64,0x3 8192 2 /
dsmc 12794 root 0u STR 157,0x8 0t221575 946 /dev/pts/8->ldterm->ptem->pts
dsmc 12794 root 1u REG 64,0x3 7988722 3479 / (/dev/vg00/lvol3)
dsmc 12794 root 2u CHR 3,0x2 0t82841 66 /dev/null
dsmc 12794 root 9u REG 64,0x3 5460 1016 /etc/utmp
dsmc 12794 root 10u REG 64,0x3 14140 2242 /etc/utmpx
dsmc 12794 root 12u STSO 21,0xd0000e 0t0 97 /dev/tcp->tcpm->ip
dsmc 19877 root cwd DIR 64,0x3 8192 2 /
dsmc 19877 root 0u STR 157,0x8 0t221575 946 /dev/pts/8->ldterm->ptem->pts
dsmc 19877 root 9u REG 64,0x3 5460 1016 /etc/utmp
dsmc 19877 root 10u REG 64,0x3 14140 2242 /etc/utmpx
dsmc 19877 root 12u STSO 21,0xd0000e 0t0 97 /dev/tcp->tcpm->ip

My question is
1) I need to know exactly what is causing the issue. I tried searching could not find anything.
2) Also TSM process is creating file nohup.out in / area. How can I ammend this?

Awaiting you valuable inputs. Thanking you in advance.

Re: / area getting full.

Ganesan R — Fri, 20 Feb 2009 10:17:53 GMT

Hi,

First you need to find what this process is? What is it doing?

I think this process is related to IBM-Tivoli storage.

Have you installed client component of IBM-Tivoli on hp-ux server?

Re: / area getting full.

sujit kumar singh — Fri, 20 Feb 2009 10:19:14 GMT

hi

can have a look at the contab, the contents of the nohup.out itself, the script that might be being executed.
regards
sujit

Re: / area getting full.

kpatel786 — Fri, 20 Feb 2009 10:19:21 GMT

Yes this is TSM process.

Re: / area getting full.

kpatel786 — Fri, 20 Feb 2009 10:21:15 GMT

Below is the content of nohuo.ut file

A short contect of the nohup.out file:

IBM Tivoli Storage Manager

Command Line Backup/Archive Client Interface

Client Version 5, Release 5, Level 1.0

Client date/time: 02/13/09 23:02:37

(c) Copyright by IBM Corporation and other(s) 1990, 2008. All Rights Reserved.

TSM Backup-Archive Client Version 5, Release 5, Level 1.0

Querying server for next scheduled event.

Re: / area getting full.

Ganesan R — Fri, 20 Feb 2009 10:37:14 GMT

Hi,

This link may helpful to you..

http://www-01.ibm.com/support/docview.wss?rs=663&tc=SSGSG7&uid=swg21249079

Re: / area getting full.

kpatel786 — Fri, 20 Feb 2009 10:56:18 GMT

My current kernel value for maxdsiz is

1073741824 which is more than specified in the doc.

My OS version is:

server2(/) # uname -a
HP-UX server2 B.11.11 U 9000/800 2474625751 unlimited-user license

Re: / area getting full.

Ganesan R — Fri, 20 Feb 2009 11:41:11 GMT

Hi,

Check what are the files created on root filesystem and how fast it grows?

Based on that you need to decide the following.

Is the file growth normal or something upnormal? In somecases I have seen that these backup team put high debug level. Due to that, log files will grow rapidly. So check with backup team about the files getting created.

If need change the files location from root to somewhere else like /var

Otherwise you can create soft link to the files which is pointing to some other locations like /var. So root will not grow unnecessaryly

Re: / area getting full.

kpatel786 — Fri, 20 Feb 2009 11:45:07 GMT

The usage of the area usually remains around 59 % however at the time of problem it is 100 % removing any files from this area does not help it hogs the free space immediately becoming 100 % again. Only restarting the dsmc process resolves the problem.

Re: / area getting full.

john korterman — Fri, 20 Feb 2009 11:52:12 GMT

Hi,

>a dsm process which hogs up the space and restarting the same releases the space<

could be an "unlink" situation:
# man 2 unlink
for an explanation.
The dsm process may write to a file that (for some reason) has been deleted. Try
# lsof| grep -i dsm
and check that the files held open by dsm actually exist.

regards,
John K.

Re: / area getting full.

Ganesan R — Fri, 20 Feb 2009 12:00:51 GMT

Hi,

>>>The usage of the area usually remains around 59 % however at the time of problem it is 100 % <<<

Can you say, When is "at the time of problem"? Is it during the backup operation?

If the filesystem gets filled up duing backup, then I would prefer you to work with backup team(Tivoli team) to check the debug level of log files or other client backup configurations.

Re: / area getting full.

T G Manikandan — Fri, 20 Feb 2009 12:06:12 GMT

when you notice your file system is filling up , As suggested please use 'lsof' to find which are the processes and directories used by TSM.

I would also suggest to check the home directory of the user managing this application

DSMC_DIR Specifies the location of the Tivoli Storage Manager configuration directory by default at /home/padmin/tivoli/tsm/

Re: / area getting full.

kpatel786 — Fri, 20 Feb 2009 12:28:16 GMT

I have spoken with the backup team they have made some changes and told me to monitor the area for some days.
Thank you all for your invaluable timely response.

Re: / area getting full.

kpatel786 — Fri, 20 Feb 2009 12:29:25 GMT

Thanks.