Why my / root file system keep increasing and almost full ?

[Kenn Chen]

i found a syslog in syslog.log

Aug 22 00:09:07 acs_lpa EMS [1731]: ------ EMS Event Notification ------ Value: "MAJORWARNING (3)" for Resource: "/storage/events/
enclosures/ses_enclosure/0_4_0_0.14.0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/re
smon/bin/resdata -R 113442858 -r /storage/events/enclosures/ses_enclosure/0_4_0_0.14.0 -n 113442825 -a
Aug 22 00:09:08 acs_lpa EMS [1731]: ------ EMS Event Notification ------ Value: "MAJORWARNING (3)" for Resource: "/storage/events/
enclosures/ses_enclosure/0_4_0_0.15.0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/re
smon/bin/resdata -R 113442862 -r /storage/events/enclosures/ses_enclosure/0_4_0_0.15.0 -n 113442826 -a
Aug 22 00:09:09 acs_lpa EMS [1731]: ------ EMS Event Notification ------ Value: "MAJORWARNING (3)" for Resource: "/storage/events/
enclosures/ses_enclosure/0_7_0_0.14.0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/re
smon/bin/resdata -R 113442866 -r /storage/events/enclosures/ses_enclosure/0_7_0_0.14.0 -n 113442827 -a
Aug 22 00:09:09 acs_lpa EMS [1731]: ------ EMS Event Notification ------ Value: "MAJORWARNING (3)" for Resource: "/storage/events/
enclosures/ses_enclosure/0_7_0_0.15.0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/re
smon/bin/resdata -R 113442870 -r /storage/events/enclosures/ses_enclosure/0_7_0_0.15.0 -n 113442828 -a

[Bill Hassell]

You need to execute the referenced command in order to see the details. There is no useful information other than the command to run.

[linuxfan]

Hi Idris,

If the / (root) filesystem keeps getting full, there is some process writing to / filesystem when it shouldn't be.

You can do a "du -kx / |sort -n" to see which directory in the / filesystem is the culprit.

-HTH
Ramesh

[Kenn Chen]

after issue du -kx command and cannot identify the file which can be deleted. all files are small and not up-to-date. There is no core files or other dummy files.

[Sridhar Bhaskarla]

Idris,

I am not able to correlate between question and the details you gave.

You just need to run the command

/opt/re
smon/bin/resdata -R 113442866 -r /storage/events/enclosures/ses_enclosure/0_7_0_0.14.0 -n 113442827 -a

This will print some good output. Most of the times, it would be self-explanatory else HP can easily determine and fix it.

Regarding the root file system full messages, find out for core files. Usually some applications dump core while exiting - I have been having experience with netscape on my workstation.


-Sri

[Vincent Stedema]

Hi,

It looks like ems is constantly dumping error messages into /var/opt/resmon/log/event.log . That's why your root filesystem keeps filling up.

tail the event.log (last 100 lines) to find out what the problem is and then try to solve it. I'm not sure if it's possible to stop the ems monitors from the command line as normally they are started from inittab and are respawned after a kill.

A device is probably failing and ems is warning you about this.

Regards,

Vincent

[Joseph Chakkery]

Hello,
Probably Vincent is right. U check /var/opt/resmon/log/event.log file If it is very big. then u may have to remap the diagnostic.
I have a doc send from HP for a similar problem go through it.

RESOLUTION

EMS keeps track in various configuration files of hardware configured on
the system. When EMS is installed and configured it creates these configuration
files.
When you copy the EMS configuration files onto another system, EMS will
try to start monitors which do not match the current system. You may see
p_client consuming CPU time, and excessive file creation in
/etc/opt/resmon as a result.

As a general rule do not copy EMS configuration files between systems with
different hardware.
If you have copied EMS configuration files and experience above symptoms,
follow these steps to clean up your EMS configuration :

Stop monitoring

# /etc/opt/resmon/lbin/monconfig
Select (K)ill (disable) monitoring

Stop p_client

Comment out the p_client entry in /etc/inittab :

# vi /etc/inittab

Change the following line
ems4:3456:respawn:/etc/opt/resmon/lbin/p_client
to
#ems4:3456:respawn:/etc/opt/resmon/lbin/p_client

Reread /etc/inittab :

# init q

Check if p_client is still running, if yes kill the process.

Delete the persistence files

# cd /etc/opt/resmon/persistence
# rm p*
# rm m*

Have diagnostics remap the system hardware

# cstm
cstm>remap
cstm>quit

Allow enough time for remap to finish before restarting the monitors.

Clean up monitor flags in the diagnostics directory

# cd /var/stm/data/tools/monitor
# rm *.hwa

Restart p_client

# vi /etc/inittab
Change the following line
#ems4:3456:respawn:/etc/opt/resmon/lbin/p_client
back to
ems4:3456:respawn:/etc/opt/resmon/lbin/p_client

Reread /etc/inittab :

# init q

Restart the EMS monitors

# /etc/opt/resmon/lbin/monconfig
Select (E)nable Monitoring


Hope this may help U.

Regards
Joe.

[AlHassani Mohamed]

Hi Idris,
we had resembled that one problem according to introduction XP256. The problem was solved through the installation the newest version the diagnostic tools. Version A.26.00. This on the June01 diagnostic CD (june 2001)
regards
Mohamed

[Brent Fortman]

We are using 'Golden Image' Ignite installs with our systems. I have noticed on the systems that we don't have Fibre-channel addapters in, we get a huge number of p.* files in the /etc/opt/resmon/persistence directory, and they do reference the fibre-channel cards. I will give Joseph C.'s advice a try, to see if it resolves our issue here.

[Victor_5]

The / filesystem is static and all directories should be writable only by root. If it is growing, something is wrong and you'll need to remove or move the offending files and directories that don't belong there.

Check /var/adm/syslog/syslog.log for some hints.