Odd messages in syslog

[Gary Glick]

I'm having a problem with one of my servers. There's an Oracle Express application running on it that keeps aborting, but not right away. It may stay up for a few hours then stop. It does produce a core dump, but I don't know how to look at it I've not had much success with debuggers etc.
I have noticed that there are some messages that keep getting entered into syslog, I've included them at the end of this post. The app is acting like there might be a bad disk but, I can't find anything wrong with the disks that are still in the system. I did have a disk that was not part of any of the volume groups that was bad but that was removed last Friday and I'm still experiencing problems today.

This is a vpar environment running on an rp54xx class server. There are a total of 2 vpars on the system. I'm rebooting the environment now.

Any thoughts or ideas?

Thank you

Gary Glick

I'm getting lots of these.
Error messages from syslog.log:

May 22 12:13:55 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/5/0/0.3.0.42.0.2.7;
May 22 12:14:01 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.8.0;
May 22 12:14:08 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.14.0;
May 22 12:14:15 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.12.0;
May 22 12:14:24 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.10.0;
May 22 12:14:31 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/0/2/0.2.0;
May 22 12:14:38 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/0/2/0.0.0;
May 22 12:14:45 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/0/1/1.2.0;
May 22 12:14:54 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/0/1/1.0.0;
May 22 12:15:12 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.8.0;
May 22 12:15:26 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.14.0;
May 22 12:15:44 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.12.0;
May 22 12:15:58 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.10.0;
May 22 12:35:27 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/5/0/0.3.0.42.0.2.7;
May 22 12:35:36 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.8.0;
May 22 12:35:43 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.14.0;
May 22 12:35:50 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.12.0;
May 22 12:35:57 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.10.0;
May 22 12:36:06 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/0/2/0.2.0;
May 22 12:36:13 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/0/2/0.0.0;
May 22 12:36:20 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/0/1/1.2.0;
May 22 12:36:27 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/0/1/1.0.0;
May 22 12:36:40 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.8.0;
May 22 12:36:53 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.14.0;
May 22 12:37:06 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.12.0;
May 22 12:37:19 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.10.0;
May 22 12:56:45 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/5/0/0.3.0.42.0.2.7;
May 22 12:56:54 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.8.0;
May 22 12:57:01 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.14.0;
May 22 12:57:08 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.12.0;
May 22 12:57:15 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.10.0;
May 22 12:57:25 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/0/2/0.2.0;
May 22 12:57:32 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/0/2/0.0.0;
May 22 12:57:39 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/0/1/1.2.0;
May 22 12:57:46 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/0/1/1.0.0;
May 22 12:57:59 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.8.0;
May 22 12:58:12 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.14.0;
May 22 12:58:25 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.12.0;
May 22 12:58:38 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.10.0;
May 22 13:17:54 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/5/0/0.3.0.42.0.2.7;
May 22 13:18:00 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.8.0;
May 22 13:18:07 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.14.0;
May 22 13:18:14 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.12.0;
May 22 13:18:23 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.10.0;
May 22 13:18:30 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/0/2/0.2.0;
May 22 13:18:37 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/0/2/0.0.0;
May 22 13:18:44 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/0/1/1.2.0;
May 22 13:18:53 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/0/1/1.0.0;
May 22 13:19:07 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.8.0;
May 22 13:19:21 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.14.0;
May 22 13:19:39 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.12.0;
May 22 13:19:53 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.10.0;
May 22 13:34:11 gobds3 su: - ta ggadmin-root
May 22 13:34:23 gobds3 su: + ta ggadmin-root
May 22 13:39:08 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/5/0/0.3.0.42.0.2.7;
May 22 13:39:17 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.8.0;
May 22 13:39:24 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.14.0;
May 22 13:39:26 gobds3 sudo: mdbdba : TTY=pts/tb ; PWD=/ibc/oracle/olap/bin ;
USER=root ; COMMAND=./express stop
May 22 13:39:31 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.12.0;
May 22 13:39:38 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/10/0/0/4/1.10.0;
May 22 13:39:47 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/0/2/0.2.0;
May 22 13:39:54 gobds3 syslog: CVSDM; DEVICE ADDED Event Code=198; Description=D
evice Added; Hardware Address=0/0/2/0.0.0;

[Torsten. · Accepted solution]

This is coming from commandview sdm, the VA management software.

What version you are using?
Could be some old.

[Gary Glick]

Interesting, I hadn't thought of that. The system that was cloned had commandview installed to manage or VA7410 but I don't need it on this one as it's not connected. I can go ahead an remove it.

I don't know if that would be causing my problem though. This env was up for many weeks without a problem until last week.

FYI: The reboot didn't help matters.

[Torsten.]

You can manage other disks like the DS series (e.g. DS2405) with commandview, but you don't really need it. Depending on your version, you have to remove some other filesets too, just read the swremove logfile.

Any other entries in the logs?

[Steven E. Protter]

Shalom,

Oracle should be able to anaylze the core dump, but you can look at it with the strings command.

SEP

[Gary Glick]

I went ahead and removed the command view software, since I didn't need it and it was logging so much.

I can't find anything on the system side.
I cleaned up the /var/stm/logs/os files and nothing new has been logged even though program aborts have happened since. stm indicatates that memory & disks are fine. EMS indicatates that one of the power supplies is unplugged, but it's never been plugged in. I'm not sure where else to check, any thoughts would much appreciated. I'm still thinking that it's a hardware issue of some sort since nothing has changed in the environemnt

I'm working through the core dump now I'll probably have to have our dba & oracle give me a hand.

Thanks again.