server inaccessible via telnet (((Help plz)))

[meekrob]

Hi All,

im facing the following issue with hp-ux 11iv3 installed on rx6600 server.

The fact is today after restarting the server, it becomes inaccessible via "Telnet" however, imstill accessing it through SSH.

I tried to check the /etc/inetd.conf    file (if anyone commented by a hash mark the line containing "telnet") and when i tried to issue the following command:   # inetd -c       i received the output:  "inetd: There is no inetd running" .

I tried to restart the server without ant results.

I also tried to disable the firewall and i got the below output:

# ipfilter -d
/sbin/init.d/ipfboot[5]: Syntax error at line 5 : `{' is not matched.
/sbin/init.d/pfilboot[5]: Syntax error at line 5 : `{' is not matched.
ERROR:   lan5 interface: IP address already in use
IPFilter Disabled

In addition and the weird thing is when i issued the "hostname" command, i received the output:

# hostname
unknown

For more details you can check the attached output.

Any  troubleshooting suggestions are so precious to me.

Thanks in advance

[Patrick Wallek]

It sounds like you  may have a garbage file in your /etc/rc.config.d directory.  Make sure that you DO NOT have any back up files, or any files that do not belong in the /etc/rc.config.d directory.

As far as inetd goes, what happens if you try to start it manually?

# /sbin/init.d/inted start

Does it error?

[meekrob]

Hello Patrick and thanks for your reply,

how can i identify garbage files in order to move/remove them without affecting any other services?

and how to tell that specific files do not belong to /etc/rc.config.d ?

Your help is much appreciated

[Bill Hassell]

I am assuming that this system worked correctly in the past. So whatever happened occurred recently. That's where sorting the list of files in /etc/rc.config.d in time order is VERY useful:

ls -lrt /etc/rc.config.d

The most recently changed files are at the bottom of the list. None of the standard configuration scripts will have extensions such as .old or .prev or .20120314, etc. Filenames like junk or test or yy or zz definitely do not belong. If you insist on making copies of your old files, create a directory called /etc/rc.config.d/old and move all the junk files there.This directory is unlike any other where everything in it will be sourced at boot up.

The original files can be seen in /usr/newconfig/etc/rc.config.d

You can test every file in this directory like this:

for RCFILE in /etc/rc.config.d/*
do
echo $RCFILE
. $RCFILE
done

 Look for error messages during the above loop.

Then go through the rc start log: /etc/rc.log looking for errors, failures, other problems.

[meekrob]

Hello,

this situation is becoming frustrating and i really need your help/opinion.

The fact is that i tried to check the /etc/rc.config.d  directory for garbage files and it seems to be fine as you can see in the attached output. I also checked the //etc/rc.config.d/netcong  for any missing parameter and i checked the /etc/inetd.conf  file (see attachment), 

Whenever i try to start inetd by issuing the command:   # /sbin/init.d/inetd start       im getting the following output:

/sbin/init.d/inetd[38]: Syntax error at line 38 : `{' is not matched.

Also noticed that whenever i tried to shutdown the server, im getting the below errors:

Broadcast Message from root (pts/1) Thu Mar 22 11:06:21...
SYSTEM BEING BROUGHT DOWN NOW ! ! !

/sbin/rc[75]: Syntax error at line 75 : `{' is not matched.
/sbin/rc2.d/K177hpws_apache[46]: Syntax error at line 46 : `{' is not matched.
/sbin/rc2.d/K177hpws_tomcat[46]: Syntax error at line 46 : `{' is not matched.
/sbin/rc2.d/K177hpws_webmin[45]: Syntax error at line 45 : `{' is not matched.
/sbin/rc2.d/K177hpws_webproxy[44]: Syntax error at line 44 : `{' is not matched.
/sbin/rc2.d/K177hpws_xmltools[42]: Syntax error at line 42 : `{' is not matched.
/sbin/rc1.d/K300acct[39]: Syntax error at line 39 : `{' is not matched.
/sbin/rc1.d/K410Rpcd[30]: Syntax error at line 30 : `{' is not matched.

     System shutdown in progress

i also printed /etc/rc.log   file for reference

Your help / suggestions are much appreciated.

Thanks in advance

[Torsten.]

This is a bit strange, there is always a line like

Syntax error at line 45 : `{' is not matched.

I compared some of the files and found almost everywhere a line like this


export PATH=$PATH:/usr/sbin:/usr/bin:/sbin


There must be a bad line or character in a very central location, maybe a typo "{" instead of "/".


Look for "{" in

/etc/PATH

and do a

# echo $PATH | grep {

Also check /etc/profile and /.profile for such typos.

[meekrob]

Hello,

could you please be more specific? and how can it be that those files were modified?

Regards

[Torsten.]

It is just an idea.

I suspect there is anywhere a typo in a file that is everytime sourced when a script is executed.

If you review your last attachment - what is in the specific lines when the error is reported?

[Torsten.]

Another idea, look at this:

-r--r--r-- 1 bin bin 4584 Nov 16 12:45 SnmpMaster

changed, recently, more or less.
Then this:

/etc/rc.config.d/SnmpMaster
sh[4]: Syntax error: `{' is not matched.


So please check this file too, but I still asume there is something wrong in a "profile" file.

[meekrob]

Hello,

I have a similar running server without problems on the subnet so if i just copy the profile files from the 1st one to the 2nd showing problems and replace its files does it work instead of comparing each file by itself? or are these server/profile files specific to each server?

Any other hint/suggestion that helps in that issue is appreciated.

Thanks in advance

[Torsten.]

In your situation I would feel better if I can find the source of the problem ... did you check the suggested 4 files?

[meekrob]

Hello Torsten,

i understand what you are saying but going through each file and check each character is really frustrating, that's why i was searchin for a fastest solution especially that this is the only sever i have which is intended for backups.

So, in your opinion (and correct me if im wrong), i need to check the following 4 files:

• /sbin/init.d/inetd
• /etc/rc.config.d/SnmpMaster
• /etc/profile
• /.profile

Thanks for your help

[Torsten.]

Only

/etc/rc.config.d/SnmpMaster
/etc/profile
/.profile

and

/etc/PATH



Just look if there is a { instead of /

(on my keyboard both are on the same key)


If you start replacing files from another system, you may produce more problems (if the files are different) without solving the initial problem ...

If you think this too much time consuming, consider to restore your ignite backup have have a consistent status again ...

[meekrob]

OK i will do what you already suggested and i will reply back when done.

Many Thanks

[meekrob]

Hello,

still have one concern.

Suppose that one of the root's user profile is missing something as you already supposed so, in that case only the root user will be prevented from accessing the server via telnet and not other non superusers or, in this case the whole inetd daemon is not starting. And how can it be that the sever is still accessible via SSH protocol?  isn't the inetd who manages all network services?

Thanks in advance

[Torsten.]

I think sshd is started as a standalone daemon, not via inetd.

Why not attach the 4 files in question?

[meekrob]

OK i will attach these files asap

[Bill Hassell]

These errors:

     /sbin/rc2.d/K177hpws_apache[46]: Syntax error at line 46 : `{' is not matched.

explain the problem fairly well. If you vi the hpws_apache file and look at line 46, you will see:

         . /etc/rc.config

So the error is coming from /etc/rc.config. This file does exactly what my small sample script above does. It sources all the files in /etc/rc.config.d. Normally, this script runs without any messages. But in your case, one of the scripts has an error and *every* startup and shutdown step will run rc.config -- thus the large number of error messages. And when the error is detected, the startup script stops, therefore most of your services are not being started. So here is the rc.config script with one modification: it reports the name of each config script. When the bad script is sourced, the error message will appear under the filename.

for CFG_FILE in /etc/rc.config.d/* /etc/TIMEZONE
do
echo $CFG_FILE   ## This was added to show the bad filename
        fname=${CFG_FILE##*/}           # get file basename
        if [ -f $CFG_FILE -a "$fname" != "core" -a "${fname##*[.,~\#]}" = "$fname" ]
        then . $CFG_FILE                # source a valid config file
        fi
done

 Just save this text in a file, perhaps /tmp/rctest. Then run it like this:

       sh /tmp/rctest

It would be a good idea for you to post the results from this script so we can help you.

A few posts back, I suggested that you list the files in /etc/rc.config.d in timestamp order. The purpose of this command was to show the most recently edited file at the bottom of the list. My guess is that the failing file in the script above will be one of the mostly recently changed.

ls -lrt /etc/rc.config.d

[Dennis Handly]

>I tried to check the /etc/rc.config.d directory for garbage files and it seems to be fine as you can see ...

No, it still says there is garbage there:

/etc/rc.config.d/SnmpMaster
sh[4]: Syntax error: `{' is not matched.

You need to look into SnmpMaster.

You may have to do:

set -x

. /etc/rc.config.d/SnmpMaster

set +x

And you may get a better trace of where it goes wrong.

Also try to syntax check the file:

sh -n /etc/rc.config.d/SnmpMaster

>I compared some of the files and found almost everywhere a line like this
>export PATH=$PATH:/usr/sbin:/usr/bin:/sbin

On mine it was nearly always the suspected line: . /etc/rc.config

>-r--r--r-- 1 bin bin 4584 Nov 16 12:45 SnmpMaster
>changed, recently, more or less.

This file is very suspect.  It is way bigger than mine and /usr/newconfig/etc/rc.config.d/SnmpMaster

>I still assume there is something wrong in a "profile" file.

I'm not sure why?  These config files are self contained.  If your .profile had a dangling "{", you should see your $PS1 prompt on each line.

>The purpose of this command was to show the most recently edited file at the bottom of the list. My guess is that the failing file in the script above will be one of the mostly recently changed.

Except that "most recently" is back in Nov.  :-(

[meekrob]

Hi All,

kindly find attached the requested output.

@ Torsten, i checked the files you mentioned as you can see in output and tried to output with vi using line numbers to be more clear, however with line numbers and once copied to notepad i got some unreadable text.

It is clear that the /etc/rc.config.d/SnmpMaster file's size is weird, we are investigating if someone brought some modifications to this file without our knowledge.

@ Bill, ive tried the script you posted as you can see its output below:

# ./rctest
/etc/rc.config.d/LANG
/etc/rc.config.d/Rpcd
/etc/rc.config.d/SnmpHpunix
/etc/rc.config.d/SnmpMaster
./rctest[6]: Syntax error at line 6 : `{' is not matched.

In addition i tried to open the  /etc/rc.config.d/SnmpMaster   file  to see the line 6 as mentionned by the error message above, however, this line in vi is commented by a # mark and it belongs to a comment at the beginning of the  SnmpMaster file.

What do you suggest in that case?

Thanks in advance for your precious help

[Matti_Kurkela]

> ./rctest[6]: Syntax error at line 6 : `{' is not matched.

This error message refers to line 6 of the rctest script. It's the line that sources (reads) the /etc/rc.config.d/SnmpMaster file. So the error indicates there is a problem in the file the rctest script is currently reading (i.e. the SnmpMaster file) but unfortunately does not identify the exact line within that file.

On line 97 of the SnmpMaster script is the opening curly brace of the StartMasterAgent() function. It does not seem to have a corresponding closing curly brace. It looks like someone has (somewhat inexpertly) modified the StartMasterAgent function to start the HP Internet Express Net-SNMP agent instead of the standard HP-UX OV SNMP agent.

The minimal fix might be to add a closing curly brace to line 101.

[meekrob]

Thanks MK for your opinion, i will try to search within that file.

If i replace the  /etc/rc.config.d/SnmpMaster    by  the  /usr/newconfig/etc/rc.config.d/SnmpMaster   and after that i rename it, is this going to train more problems? 

Thanks in advance

[Dennis Handly]

>It is clear that the /etc/rc.config.d/SnmpMaster file's size is weird,

If you look at the file contents, it says it is really /sbin/init.d/SnmpMaster!  This is the script that will process the config file above.

But it seems there are lines in this script that have been modified, causing your error.

>we are investigating if someone brought some modifications to this file without our knowledge.

Can you check the directory modification dates of /etc/rc.config.d?

Also, take a diff with that config file and /sbin/init.d/SnmpMaster.

>If I replace the /etc/rc.config.d/SnmpMaster by the /usr/newconfig/etc/rc.config.d/SnmpMaster, is this going to cause more problems?

Not any more than you have now.   ;-)  It is useless as it is.

Do you have a back up of SnmpMaster?  Otherwise, did you ever enable SnmpMaster?

(Do you have an old rc.log in your backups?)

>It looks like someone has (somewhat inexpertly) modified the StartMasterAgent function

And put the file into the wrong place.  :-(

[meekrob]

Many Thanks for your opinion, but could you clarify more plz?

> If you look at the file contents, it says it is really /sbin/init.d/SnmpMaster!  This is the script that will process the config file above.

You mean that  the /sbin/init.d/SnmpMaster  is the script that is having syntax error and not the  /etc/rc.config.d/SnmpMaster file?

>Can you check the directory modification dates of /etc/rc.config.d?

kindly find attached corresponding output.

>Otherwise, did you ever enable SnmpMaster?  (Do you have an old rc.log in your backups?)

It's a backup server that each dept can use, how can i find if someone enabled SnmpMaster? As for an older rc.log  i will try to post the output of an older one stored in /etc/rc.log.old  

>And put the file into the wrong place.

Could you clarify more regarding this upper point of view? I really need to know what happens indeed.

In conclusion, what can be the best solution to re-enable inetd? with less problems on the server?

Thanks in advance

[Bill Hassell]

Your server has been down for many days because of this error. Rather than try to find the bad code added by another root user, make an archive directory, move the Snmp files into old and recover the original files from /usr/newconfig/etc/rc.config.d, like this:

umask 022
cd /etc/rc.config.d
mkdir old
mv Snmp* old
cp -p /usr/newconfig/etc/rc.config.d/Snmp* .

 and then reboot. Although the most recent changes to rc.config.d were back in November, these changes were probably made without a reboot so all the changed files in November are suspect. Snmp programs are used primarily for monitoring so your system will at least boot and run. As for finding out why the files were changed and whether Snmp is required, you need to look at your change control documents. If changes are made in secret and root users do not write down what they have done, then you'll just have to muddle along and wait for someone to ask that Snmp be enabled. I suspect that this will not happen since the Snmp config scripts haven't worked since they were changed last November.