memory fault (core dump) with telnet, SAM, ...

[Ahmad S. Al-Amer]

Dear,

We have a server that we used to telnet. After rebooting it we can't.

The error it says while we try to telnet the localhost is "Memory fault (core dump)".

While rebooting it mention that the FC port is offline, I tried to run SAM but again "Memory fault (core dump)".

The server keeps saying the following message:

INIT: Command is respawning too rapidly.
Will try again in 5 minutes.
Check for possible errors.
id:emient -> /opt/cifsclient/bin/cifsclient
s4 "/etc/opt/resmon/lbin/p_client"


Any help plz?

Thank you in advance

[RAC_1]

What else do you see?
Can you log onto console?

[Ahmad S. Al-Amer]

Yes, I can login to the console.

[Peter Godron]

Ahmad,
did you chamge anything before the reboot?
From another thread:
"
Firstly check /etc/inittab file.
samd entry should look like this.
samd:23456:respawn:/usr/sam/lbin/samd # system mgmt daemon
Once you change /etc/inittab file, run 'init q' to force init to re-read the file.
"

[Ahmad S. Al-Amer]

Actually, we recieved the server to deactivate all the volume groups and some checkup. We don't know really if something was changed.

I checked "/etc/inittab": it does not have the mentioned line.

Should I add it?

[Muthukumar_5]

May be /etc/inittab file is corrupted?! Try to use the configuration from some other machine with same configuration. If you have a backup use that else.

--
Muthu

[RAC_1]

Log onto console. Modify the line that has emient in it. Set respawn/bootwait or whatever it is to off. Do init -q

Try logging in now. check the error. Check syslog.log

[Ahmad S. Al-Amer]

I'll try it now

[Ahmad S. Al-Amer]

RAC,
I tried it but still can not telnet remotly.

anyway I can login from the console.

but my issue here is Why giving me "memory fault (core dump)" error message.

thanx

[RAC_1]

After loggin into console, post following.

swapinfo -mat
dmesg
ps -ef|grep [t]elnet
glance -m
vmstat 1 3
Any errors in syslog.log?

[Peter Godron]

Ahmad,
the core dumps are probably caused by not having enough free memory to keep these processes (or associated processes) running.
Each process running requires a certain amount of memory, so if you have constantly increasing number of processes....

[Muthukumar_5]

Kernel parameters are needed to tuned.

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=436510

Refer Jeff's reply.

--
Muthu

[Ahmad S. Al-Amer]

RAC,

==> swapinfo -mat

Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 8208 0 8208 0% 0 - 1 /dev/vg00/lvol2
dev 8192 0 8192 0% 0 - 1 /dev/vg00/lvol3
reserve - 41 -41
memory 25463 72 25391 0%
total 41863 113 41750 0% - 0 -



==> dmesg

Mar 29 13:21
...
is OFFLINE, can not scan

fcd: Claimed Dual Port 2Gb HBA Port 1 at 1/0/10/0/1
1/0/10/0/1 fcd
1/0/10/0/1: Fibre channel host port is OFFLINE, can not scan

1/0/12 lba
1/0/12/0/0 igelan
1/0/14 lba
c8xx BUS: 8 SCSI C1010 Ultra160 Wide LVD A6829-60101 assigned CPU: 3
1/0/14/0/0 c8xx
1/0/14/0/0.7 tgt
1/0/14/0/0.7.0 sctl
c8xx BUS: 9 SCSI C1010 Ultra160 Wide LVD A6829-60101 assigned CPU: 4
1/0/14/0/1 c8xx
1/0/14/0/1.7 tgt
1/0/14/0/1.7.0 sctl
1/5 memory
1/10 processor
1/11 processor
1/12 processor
1/13 processor
2 cell
2/5 memory
2/10 processor
2/11 processor
2/12 processor
2/13 processor
3 cell
3/5 memory
3/10 processor
3/11 processor
3/12 processor
3/13 processor

System Console is on the Built-In Serial Interface
igelan0: INITIALIZING HP A7109-60001 PCI 1000Base-T Core at hardware path 0/0/0/1/0
igelan2: INITIALIZING HP A7109-60001 PCI 1000Base-T Core at hardware path 1/0/0/1/0
igelan3: INITIALIZING HP A6847-60101 PCI 1000Base-SX Adapter at hardware path 1/0/12/0/0
igelan1: INITIALIZING HP A6847-60101 PCI 1000Base-SX Adapter at hardware path 0/0/14/0/0
Logical volume 64, 0x4 configured as ROOT
Logical volume 64, 0x2 configured as SWAP
Logical volume 64, 0x2 configured as DUMP
Logical volume 64, 0x3 configured as DUMP
Swap device table: (start & size given in 512-byte blocks)
entry 0 - major is 64, minor is 0x2; start = 0, size = 16809984
Dump device table: (start & size given in 1-Kbyte blocks)
entry 0000000000000000 - major is 31, minor is 0x6000; start = 265056, size = 8404992
entry 0000000000000001 - major is 31, minor is 0x6000; start = 8670048, size = 8388608
Starting the STREAMS daemons-phase 1
Create STCP device files
Starting the STREAMS daemons-phase 2
$Revision: vmunix: vw: -proj selectors: CUPI80_BL2000_1108 -c 'Vw for CUPI80_BL2000_1108 build' -- cupi80_bl2000_1108 'CUPI80_BL2000_1108' Wed Nov 8 19:24:56 PST 2000 $
Memory Information:
physical page size = 4096 bytes, logical page size = 4096 bytes
Physical: 33521664 Kbytes, lockable: 31427420 Kbytes, available: 29866832 Kbytes

emcp:Mp:Error: All paths to Symm 000284700672 vol 031a are dead.
emcp:Mp:Error: All paths to Symm 000284700672 vol 0326 are dead.
emcp:Mp:Error: All paths to Symm 000284700672 vol 0334 are dead.
emcp:Mp:Error: All paths to Symm 000284700672 vol 0338 are dead.
emcp:Mp:Error: All paths to Symm 000284700672 vol 033c are dead.
emcp:Mp:Error: All paths to Symm 000284700672 vol 036e are dead.
emcp:Mp:Error: All paths to Symm 000284700672 vol 032a are dead.
emcp:Mp:Error: All paths to Symm 000284700672 vol 032b are dead.
emcp:Mp:Error: All paths to Symm 000284700672 vol 0342 are dead.
emcp:Mp:Error: All paths to Symm 000284700672 vol 034c are dead.
emcp:Mp:Error: All paths to Symm 000284700672 vol 0356 are dead.
emcp:Mp:Error: All paths to Symm 000284700672 vol 035b are dead.
emcp:Mp:Error: All paths to Symm 000284700672 vol 0360 are dead.
emcp:Mp:Error: All paths to Symm 000284700672 vol 0365 are dead.
emcp:Mp:Error: All paths to Symm 000284700672 vol 0368 are dead.
emcp:Mp:Error: All paths to Symm 000284700672 vol 041b are dead.
emcp:Mp:Error: All paths to Symm 000284700672 vol 0413 are dead.
emcp:Mp:Error: All paths to Symm 000284700672 vol 0417 are dead.
emcp:Mp:Error: All paths to Symm 000284700672 vol 041f are dead.
emcp:Mp:Error: All paths to Symm 000284700672 vol 0420 are dead.
emcp:Mp:Error: All paths to Symm 000284700672 vol 038c are dead.
emcp:Mp:Error: All paths to Symm 000284700672 vol 0468 are dead.
emcp:Mp:Error: All paths to Symm 000284700672 vol 0480 are dead.
0/0/8/0/0: Fibre channel host port is OFFLINE, can not scan

0/0/10/0/0: Fibre channel host port is OFFLINE, can not scan

0/0/10/0/1: Fibre channel host port is OFFLINE, can not scan

0/0/12/0/0: 0/0/12/0/1: Fibre channel host port is OFFLINE, can not scan

Fibre channel host port is OFFLINE, can not scan

1/0/8/0/1: Fibre channel host port is OFFLINE, can not scan

1/0/10/0/0: Fibre channel host port is OFFLINE, can not scan

1/0/10/0/1: Fibre channel host port is OFFLINE, can not scan




==> ps -ef|grep [t]elnet
nothing shown


==> glance -m

glance is not installed


==> vmstat 1 3

root@BSS03-/ # vmstat 1 3
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
1 1 0 3172 7439367 39 41 36 0 0 0 5 3467 1617 130 0 0 99
1 1 0 3172 7439320 2 0 1 0 0 0 2 3364 73 70 0 0 100
1 1 0 3172 7439320 1 0 1 0 0 0 1 3382 68 70 0 0 100


regards

[RAC_1]

What does following say?

file "core_file"

[Ahmad S. Al-Amer]

RAC,

it says:
core: core file from 'telnetd' - received SIGSEGV

[RAC_1]

This is on account of segmentation fault. What are your settings for following?

shmmax
kmtune -lq shmmax

Also, do you have tusc tool? If not get it here.
http://hpux.connect.org.uk
Now run telnet as follows and post.

tusc -vfpz "telnet localhost"

[Ahmad S. Al-Amer]

RAC,

Parameter: shmmax
Value: 0X40000000
Default: 0X4000000
Minimum: -
Module: -




I'm installing the tusc tool
regards

[Ahmad S. Al-Amer]

RAC,

root@BSS03-/tmp # ./tusc -vfpz telnet localhost
[2396] open("/opt/graphics/OpenGL/lib/libogltls.sl", O_RDONLY, 0) ERR#2 ENOENT
[2396] mmap(NULL, 1335296, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_SHLIB, 3, 0x6d000) ERR#12 ENOMEM
[2396] stat("/usr/lib/libsis.0", 0x7b042e68) ............. ERR#2 ENOENT
[2396] open("/usr/lib/libsis.0", O_RDONLY, 0100) ......... ERR#2 ENOENT
[2396] mmap(NULL, 12288, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_SHLIB, 3, 0x2000) ERR#12 ENOMEM
[2396] stat("/dev/ip6", 0x7b040740) ...................... ERR#2 ENOENT
[2396] ioctl(3, TCGETA, 0x7b0413d8) ...................... ERR#25 ENOTTY
command: _IOR('T', 1, 18)
[2396] mmap(NULL, 12288, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_SHLIB, 3, 0x2000) ERR#12 ENOMEM
[2396] open("/etc/resolv.conf", O_RDONLY, 0666) .......... ERR#2 ENOENT
[2396] recvfrom(3, 0x7b040e24, 1024, 0, 0x7b042208, 0x7b0421e4) ERR#227 EADDRNOTAVAIL
[2396] recvfrom(3, 0x7b040e24, 1024, 0, 0x7b042208, 0x7b0421e4) ERR#227 EADDRNOTAVAIL
[2396] recvfrom(3, 0x7b040e24, 1024, 0, 0x7b042208, 0x7b0421e4) ERR#227 EADDRNOTAVAIL
[2396] recvfrom(3, 0x7b040e24, 1024, 0, 0x7b042208, 0x7b0421e4) ERR#227 EADDRNOTAVAIL
[2396] mmap(NULL, 12288, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_SHLIB, 3, 0x2000) ERR#12 ENOMEM
[2396] mmap(NULL, 1335296, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_SHLIB, 3, 0x6d000) ERR#12 ENOMEM
[2396] mmap(NULL, 12288, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_SHLIB, 3, 0x2000) ERR#12 ENOMEM
[2396] ioctl(3, TCGETA, 0x7b041358) ...................... ERR#25 ENOTTY
command: _IOR('T', 1, 18)
[2396] Received signal 11, SIGSEGV, in user mode, [SIG_DFL], partial siginfo
[2396] Siginfo: si_code: I_NONEXIST, faulting address: 0xfffffffe, si_errno: 0
[2396] PC: 0xc004a993, instruction: 0x0c872001
[2396] exit(11) [implicit] ..................... WIFSIGNALED (SIGSEGV) |
WCOREDUMP

[RAC_1]

Do you get error for all commands or only for telnet and SAM? Post ulimit -Sa
and ulimit -Ha

[Ahmad S. Al-Amer]

RAC,

I'm not sure but I tried "ftp" also the same memory failure error.

root@trains2: / # ulimit -Sa
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 1310720
stack(kbytes) 8192
memory(kbytes) unlimited
coredump(blocks) 4194303
nofiles(descriptors) 2048
root@trains2: / # ulimit -Ha
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 1310720
stack(kbytes) 8192
memory(kbytes) unlimited
coredump(blocks) unlimited
nofiles(descriptors) 2048

[RAC_1]

Any other command that you try?
did you reboot the machine?

All looks ok and nothing suspicios with memory usage, ulimit etc. Run tusc on ftp also and post.

[Ahmad S. Al-Amer]

RAC,

emmmmmm...
What kind of commands you mean, name some?

Yes, I did reboot the machine several times.

[Arunvijai_4]

Hi Ahmed,

You can try installing latest patches available for telnetd,

PHNE_30695, PHNE_28841

You can find these patchs at ITRC patch database,

http://www1.itrc.hp.com/service/patch/search.do?BC=patch.breadcrumb.main|&pageContextName=hpux:::

-Arun

[Ahmad S. Al-Amer]

Dear,

I tried to install the patch but I failed due to the same problem "Memory fault (coredump)".

Is there a way to install it without walking throgh the GUI?

Regards

[RAC_1]

swinstall -s "full_path_of_depot" \*

Will do express install.