cmomd.log is too big

[ma_7]

The /var/opt/cmom/cmomd.log grows very fast and now is
-rw-rw-rw- 1 root root 2147483647 Dec 21 20:04 cmomd.log

what is the matter ?

Thanks in advance.

[Peter Godron]

Hi,
Please read:
http://forums1.itrc.hp.com/service/forums/helptips.do?#33 on how to reward any useful answers given to your questions.

So far you have not awarded any points !

check for errors:
tail -100 /var/opt/cmom/cmomd.log

[ma_7]

Sorry, I am not very familiar with this forum.

Indeed, tail outputs:

....
instance of LogEntryInstTable {
instance of LogEntryInstTable {
instance of LogEntryInstTable {
instance of LogEntryInstTable {
instance of LogEntryInstTable {
instance of LogEntryInstTable {
instance of LogEntryInstTable {
instance of LogEntryInstTable {

all of these strings.

[Peter Godron]

Hi,
looks to me as if you have a largefile problem (i.e. your file system does not support filesizes above 2Gb).Does the file size now stay the same.

I would stop the application, rename the log file and restart the application.

Then analyse the old logfile ignoring the "instance of LogEntryInstTable" entries.

Also keep an eye on the new log file.

Please read, but not necessarily act upon:
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=866569

[ma_7]

Absolutely the cmomd.log meets the large file limits, but the problem is that if I empty the file with:

#cat /dev/null > cmomd.log
then the size will be zero and then in half an hour, the cmomd.log will grow to 2G again.

I think there maybe something else.

[Denver Osborn]

Is this an mc/sg clustered node?

Can you also post the hacl entries from /etc/inted.conf?

grep ha /etc/inetd.conf

This info might help troubleshoot.

-denver

[ma_7]

hacl-probe stream tcp nowait root /opt/cmom/lbin/cmomd /opt/cmom/lbin/cmomd -f /var/opt/cmom/cmomd.log
hacl-cfg dgram udp wait root /usr/lbin/cmclconfd cmclconfd -p
hacl-cfg stream tcp nowait root /usr/lbin/cmclconfd cmclconfd -c

[George Liu_4]

lsof to find the applications

[Rita C Workman]

What version of MC/SG are you running ?
Are you patched up ?

...and my real question....is it possible some software is running a port scan that is hitting anywhere on your cluster ??????

Rita

[ma_7]

These nodes are in an MC/SG cluster.
by use fuser, I got:

#fuser /var/opt/cmom/cmomd.log
/var/opt/cmom/cmomd.log: 9699o 12872o 12866o 9675o

#ps -ef | grep cmom
root 9699 910 255 11:33:40 ? 768:06 /opt/cmom/lbin/cmomd -f /var/opt/cmom/cmomd.log
root 12872 910 255 11:44:52 ? 812:33 /opt/cmom/lbin/cmomd -f /var/opt/cmom/cmomd.log
root 12866 910 255 11:44:52 ? 811:55 /opt/cmom/lbin/cmomd -f /var/opt/cmom/cmomd.log
root 9675 910 255 11:33:39 ? 775:38 /opt/cmom/lbin/cmomd -f /var/opt/cmom/cmomd.log
root 8127 8097 1 08:19:17 pts/0 0:00 grep cmom

[ma_7]

Throught swlist:

#

#
# Bundle(s):
#

B2491BA B.11.11 MirrorDisk/UX
B5161FA A.11.13 ServiceGuard OPS Edition Bundle
B5725AA B.3.5.89 HP-UX Installation Utilities (Ignite-UX)
B8324BA B.01.04 HP Cluster Object Manager
B8325BA A.02.00 ServiceGuard Manager
B9788AA 1.3.1.02.00 Java 2 SDK 1.3 for HP-UX (700/800), PA1.1 + PA2.0 Add On
BUNDLE B.11.00 Patch Bundle
BUNDLE11i B.11.11.0102.2 Required Patch Bundle for HP-UX 11i, February 2001
Base-VXVM B.03.20.1 Base VERITAS Volume Manager 3.2 for HP-UX
CDE-ChineseS B.11.11 Simplified Chinese CDE Environment
CMDVIEWSDM A.1.06.00 hp StorageWorks Command View SDM
FDDI-00 B.11.11.02 PCI FDDI;Supptd HW=A3739A/A3739B;SW=J3626AA
FibrChanl-00 B.11.11.09 PCI/HSC FibreChannel;Supptd HW=A6684A,A6685A,A5158A,A6795A
GOLDAPPS11i B.11.11.0212.4 Gold Applications Patches for HP-UX 11i, December 2002
GOLDBASE11i B.11.11.0212.4 Gold Base Patches for HP-UX 11i, December 2002
GigEther-00 B.11.11.14 PCI/HSC GigEther;Supptd HW=A4926A/A4929A/A4924A/A4925A;SW=J1642AA
GigEther-01 B.11.11.04 PCI GigEther;Supptd HW=A6794A/A6825A/A6847A
HPUX11i-OE B.11.11.0206 HP-UX 11i Operating Environment Component
HPUXBase64 B.11.11 HP-UX 64-bit Base OS
HPUXBaseAux B.11.11.0206 HP-UX Base OS Auxiliary
HWEnable11i B.11.11.0206.5 Hardware Enablement Patches for HP-UX 11i, June 2002
Ignite-UX-10-20 B.3.5.89 HP-UX Installation Utilities for Installing 10.20 Systems
Ignite-UX-11-00 B.3.5.89 HP-UX Installation Utilities for Installing 11.00 Systems
Ignite-UX-11-11 B.3.5.89 HP-UX Installation Utilities for Installing 11.11 Systems
OnlineDiag B.11.11.07.11 HPUX 11.11 Support Tools Bundle, Jun 2002
RAID-00 B.11.11.01 PCI RAID; Supptd HW=A5856A
perl B.5.6.1.C Perl Programming Language
#
# Product(s) not contained in a Bundle:
#

CETool 1.0
EMCpower HP.4.3.0_b110 PowerPath
HPOVSAMHA 03.00.00.0307 hp OpenView storage area manager hostagent framework
HPOVSAMJR 01.03.01.05 HP OpenView Storage Area Manager Embedded JRE
LSM 5.5.lsm.Build.55 LSM for HP-UX 10.XX and 11.xx
NAVICLI 6.16.0.4.63 Navisphere Disk Array Management Tool (CLI)
PHKL_27225 1.0 IDE/ATAPI cumulative patch
PHKL_27304 1.0 SCSI Tape (stape) cumulative
PHKL_27751 1.0 Fibre Channel Mass Storage Patch
PHKL_28695 1.0 Cumulative VM, Psets, Preemption, PRM, MRG
PHNE_26369 1.0 IGELAN 1000Base-SX/T cumul. patch
PHSS_26057 1.0 Cluster Object Manager B.01.04
PHSS_26138 1.0 OV EMANATE14.2 Agent Consolidated Patch
PHSS_27087 1.0 MC/ServiceGuard and SG-OPS Edition A.11.13

[Carsten Krege]

The following logs are symptoms for a bug that is caused by a portscan against an HPUX node. The fix came with patches Cluster Object Manager, namely

PHSS_26675 'Cluster Object Manager A.01.03.01' 11.00 A.01.03
PHSS_27991 'Cluster Object Manager B.01.04.01' 11.00 B.01.04


------------------------------------------------------------
instance of LogEntryInstTable {
timestamp = "Aug 28 21:33:23";
source = "11324";
category = "ERROR";
path = "om.object";
message = "Failed object (0x0) type assertion unknown != name value
pair (file
cm_name_value_pair.c line 105)";
};
-------------------------------------------------------------

You should also install patches for SG A.11.13. Please note that SG A.11.13 is not supported anymore. Youi should update SG to a supported version.

Carsten

[Stephen Doud]

I helped 2 Serviceguard administrators with portscan-related issues last week. If you have inetd logging increased (inetd -l), you may see the offender's IP along with more cmclconfd messages.
Since Carsten corroborated a port-scan issue...
0) install the COM patch for your version
1) set up a firewall
2) block the external IP with /var/adm/inetd.sec

[ma_7]

Thank you very much ,
I will try it.