network problem.. lanadmin -> lan -> reset

[Paul Torp]

hi.. we have a stange situation..
each night at 02.30 the interface hangs. and we have to do a reset.

HP is looking at the prolem, but in the meanwhile i REALLY could use a script that checks if the subnet 10.100.0.0 is up and executes commandline reset/selftest (havent found doc on it).

I am going on vacation and i wish my substitute easy nights.

can anyone help?

[Ludovic Derlyn]

hi,

I have encountered this problem on a natwork card, each time i have executed an save or restore by DP
cards hangs but still up, monitored by cluster, up --> no switch on standby card

I have try to check cable connectivity and so , i have changed the network card

Now it's ok

What is your interface model ? me , it's a combo card HP AB465-60001 on hpux 1123 itanium

what is the result of :

netfmt -v -f /var/adm/nettl.LOG000 ?

Regards

L-DERLYN

[Jaime Bolanos Rojas.]

Paul, I am not much into scripts, but for a script like this you would have to ping the interface all the time, and this is something not very nice in networking.
But if you already know that it happens every night at the same time, then run just a cron job at that time every night, and just execute the command that you use to reset the interface.

Regards,

Jaime.

[Paul Torp]

Class I H/W Path Driver S/W State H/W Type Description
===================================================================
lan 0 0/1/2/0 igelan CLAIMED INTERFACE HP A6825-60101 PCI 1000Base-T Adapter
lan 1 0/4/1/0 igelan CLAIMED INTERFACE HP A6825-60101 PCI 1000Base-T Adapter
lan 2 0/5/2/0 igelan CLAIMED INTERFACE HP A6825-60101 PCI 1000Base-T Adapter


its on an RP4440 machine.. newest driver is installed.

[Ludovic Derlyn]

paul

what is the result of netfmt ?
Have you a save running at this time ?

Have you change the network card ?

regards

L-DERLYN

[Paul Torp]

hi..

the problem is that it indicates NOTHIN at the time it hangs..

the log entries is when i do the reset.
**********************Gigabit Ethernet LAN/9000 Networking******************@#%
Timestamp : Wed Jul 19 METDST 2006 07:51:27.802689
Process ID : [ICS] Subsystem : IGELAN
User ID ( UID ) : -1 Log Class : ERROR
Device ID : 0 Path ID : 0
Connection ID : 0 Log Instance : 0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

<2004> 1000Base-T in path 0/1/2/0
Detected a faulty or disconnected cable.

Verify that the cable from this adapter to its link partner is not
defective. If this adapter is directly connected to another Ethernet
adapter, ensure that a cross-over cable is used for speeds less than
1000 Mb/s. Using lanadmin -x card_info , verify that the link
settings such as Flow Control and Auto-negotiation on this adapter
are compatible with the corresponding settings on its link partner.
Also note that a transient link down may be detected during a reset
to this device or its link partner.

**********************Gigabit Ethernet LAN/9000 Networking******************@#%
Timestamp : Thu Jul 20 METDST 2006 08:00:38.719195
Process ID : [ICS] Subsystem : IGELAN
User ID ( UID ) : -1 Log Class : ERROR
Device ID : 0 Path ID : 0
Connection ID : 0 Log Instance : 0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

<2010> 1000Base-T in path 0/1/2/0
Experienced a link down event. A reset that was issued 2 seconds ago
may have caused this.

(Error) A transient link down event is expected during a reset.
If the link doesn't come back up on its own please check the cable
and verify the configuration on this interface and its link partner.

**********************Gigabit Ethernet LAN/9000 Networking******************@#%
Timestamp : Fri Jul 21 METDST 2006 02:38:53.245154
Process ID : [ICS] Subsystem : IGELAN
User ID ( UID ) : -1 Log Class : ERROR
Device ID : 0 Path ID : 0
Connection ID : 0 Log Instance : 0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

<2010> 1000Base-T in path 0/1/2/0
Experienced a link down event. A reset that was issued 2 seconds ago
may have caused this.

(Error) A transient link down event is expected during a reset.
If the link doesn't come back up on its own please check the cable
and verify the configuration on this interface and its link partner.

============================= LOG File Summary =============================

Node: hpfciio
HP-UX Version: B.11.11 U Machine Type: 9000/800

Total number of messages: 159
Messages dropped: 0 Data dropped(bytes): 0

First Message Last Message
Time: 13:42:22.550105 Time: 02:38:53.245154
Date: 06/30/04 Date: 07/21/06

Message distribution:
Disaster: 0 Error: 159
Warning: 0 Informative: 0


~~~~~~~~~~~~~Message distribution by Subsystem~~~~~~~~~~~~~

Subsystem Name: STREAMS Group Name: STREAMS/UX
Disaster: 0 Error: 129
Warning: 0 Informative: 0

Subsystem Name: IGELAN Group Name: Gigabit Ethernet LAN
Disaster: 0 Error: 30
Warning: 0 Informative: 0


its much more form earlier day..
the hang happeneds exactly at 02.31 every day..


i am doing a workaround now and i wonder what the commandline commad is for :lanadmin -> lan -> reset

[Ludovic Derlyn]

hi

i have exactely this error before change my network card, at regular time also

We have see that a specific save running , and cards hang

After, change no problem are appears

regards

L-DERLYN

[Paul Torp]

what i didnt tell you is that happened on the other machine in the cluster as well.. ONE time.. same hours..

[Ludovic Derlyn]

Ok, for me it's a cluster , but this problem are located only on a node
Have you a specific job running at this time ?

[Paul Torp]

yep look like it.. the customers competence is on vacation..

and tyhey cantr answer anything.. that is annoying..

looks like they run Brightstore uagentd.

they claim (not sure) that it should start at 02.02 but my last TOP from the night before the hang showed otherwise..


System: hpux2 Fri Jul 21 02:28:46 2006
Load averages: 1.24, 1.49, 1.80
148 processes: 126 sleeping, 21 running, 1 zombie
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 1.58 24.0% 0.0% 26.7% 49.3% 0.0% 0.0% 0.0% 0.0%
1 0.90 7.2% 0.0% 66.1% 26.7% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 1.24 15.6% 0.0% 46.5% 37.9% 0.0% 0.0% 0.0% 0.0%

Memory: 3635972K (2993888K) real, 4028932K (3320896K) virtual, 57176K free Page
# 1/8

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
1 ? 7164 ca 154 20 16956K 1304K sleep 0:25 1.68 1.68 uagentd
1 ? 36 root 152 20 7424K 7424K run 0:46 1.61 1.60 vxfsd
0 ? 1792 root -27 20 11444K 7632K run 2:21 0.67 0.67 cmcld
0 ? 1810 root 152 20 10404K 3488K run 0:24 0.26 0.26 aws_orb
0 ? 1405 root -16 20 27456K 6456K run 1:44 0.22 0.22 midaemon
0 ? 0 root 127 20 32K 0K sleep 1:54 0.21 0.21 swapper
0 ? 1891 root 152 20 9636K 2316K run 0:15 0.21 0.21 caiLogA2
1 ? 1550 root 152 20 16096K 4344K run 0:04 0.18 0.18 rep_serve
0 ? 1552 root 152 20 13532K 1828K run 0:02 0.18 0.18 agdbserve
1 ? 1893 root 152 20 11988K 4196K run 1:32 0.18 0.18 caiUxOs
0 ? 1886 r
0 ? 21134 oracle 156 20 597M 10432K sleep 12:02 0.63 0.63 ora_j003_IN
0 ? 21128 oracle 156 20 597M 9032K sleep 17:32 0.56 0.56 ora_j001_IN

[Ludovic Derlyn · Accepted solution]

hi,

perhaps a job in crontab.
Have you checked the log of crontab ?
(/var/adm/cron/log)


for reset, I didn't know if lanadmin will be automatized

One solution could be to shut lan by ifconfig lan x down
ifconfig lan x up

Caution:

if you are in cluster, normally standby card will be become up (cmviewcl -v give card and standby card), else you risk to cut access a few time to the server

try it with caution before apply !

[Paul Torp]

i have made a workaround..

tnx for all replies..

I let HP work on the solution while i am on vacation..

Paul

[Ludovic Derlyn]

hi Paul

if i can help you, see :

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1047978

regards

L-DERLYN