Locating source of console messages

[TheJuiceman]

Hey gang,

Periodically I am getting strange messages on my system console (like "rm: f non-existent"). What is the best method to track down the source of this error? Thanks!!!!

[Jov]

Hi,

It could be anything directed to the console, but most of the time its syslog. Check your latest syslog to see if it has the same error or message.

From the error it seems there is a script calling the 'rm' command with 'f' as a parameter option. It probably meant to be 'rm -f' or the likes.

Its likely to be a cron job. Grep for the message under /var/spool/cron (hope i did get this missed up with Solaris).


Jov

[TheJuiceman]

Grepping for this string has not given any results. Any other advice? There has to be a way of determining where this is coming from. I tried editing the /etc/syslog.conf file to send alert messages to a file instead of the console, but I'm still getting the messages on the console and nothing in the file.

[A. Clay Stephenson]

This looks more like an application that has redirected stderr to /dev/console that a standard utility that is using syslogd. Are the messages appearing with some periodicity? If so, it suggests a cron'ed job or a script with a fixed sleep interval.

If you have lsof installed then "lsof /dev/console" might prove useful.

[TheJuiceman]

I am thinking it is a script too. I do not have lsof unfortunately. Any other suggestions? Thank you again!!!

[Jov]

Clay has given me an idea. Instead of lsof on the console, do an lsof on rm (assuming there is only one and its the 1st in $PATH).

Log/redirect the output to a file until the message comes up again. That should give you the program calling rm (hope you dont have a large number of scripts/apps calling 'rm').


Jov

[TheJuiceman]

Nope...that won't work. We have WAY too many things that call rm (just checking my rm wrapper tells me that). And I do not have lsof unfortunately.

[A. Clay Stephenson]

Without lsof, you are shooting in the dark. Fuser might help but it's a poor substitute.

Get lsof from here and install it. It's a utility no HP-UX box should be without.

http://hpux.connect.org.uk/hppd/hpux/Sysadmin/lsof-4.77/

[TheJuiceman]

awesome!!! i will get it installed and then go from there. hopefully i can post on here the results (better yet, close the thread and hand out a bunch of points!!!) Thanks again!!!

[TheJuiceman]

I have lsof now. How would be the best way to run it so that it will record the error messages when I get them on the console? Thanks again.

[A. Clay Stephenson]

I trust that you are aware the the lsof packages comes with its on man page. I'm sure that you have at least done a "man lsof" --- pay attention to the -r option.

lsof -r 30 /dev/console >> /var/tmp/lsof.log

This will run the command at 30 second interval; sooner or later you should hit paydirt.

[TheJuiceman]

Thanks. I did check out the man page, but it is VERY detailed with options. Running the command you suggested will generate a HUGE file. Is there a way to modify it to look for the error output? Thanks!!!

[A. Clay Stephenson]

Think about it. Lsof lists open files. The file that cannot be found by definition isn't open. All you have a prayer of doing is spotting an unfamilar process that matches the times of your console messages.

You could aosi get lucking by simply doing
date commands followed by ps -ef's and sleep 25 in a loop logging to a file.

[Jov]

>> Running the command you suggested will generate a HUGE file. Is there a way to modify it to look for the error output? Thanks!!!

You can create a perl or shell wrapper for 'rm' which spits out the caller procces id/name with something like 'ps' and grep for rm into a log file. This *should* work assumming the caller program/script is waiting for a return/exit code from 'rm'.

I've never tried it, just a theory at the moment.


Jov

[Dennis Handly]

>Jov: You can create a perl or shell wrapper for 'rm' which spits out the caller

This seems like a more refined way to find the evil user doing that rm, than using lsof.

Also, does the author know what file occurs in the message? If so, that would make the script even more targeted.

>assumming the caller program/script is waiting for a return/exit code from 'rm'.

By that, only log it if there is a bad exit from the real rm?