v3001 R6_0_55 locks up on boot every time

[rythan73]

v3001
R6_0_55

My phone system locked up last Thursday. There was no power outage and the system has a UPS. The phones just stopped responding and a console connection didn't display a prompt. I had to force reboot and every time it boots, the system locks up at the exact same spot (log below).

I tried swapping the ram, removing the FXO boards, pulling the bios battery, and booting in to R6_0_54 (the drive said it had 54 and 55 on it from the factory) but every time it stops at the exact same spot.

Any ideas? At this point I am not sure if it is the DB, the HD, or the Hardware. I used to be an installer for NBX back in the R2 to R4 nbx100-v5000 days but it seems that a lot of my tricks no longer work. I do have a recent backup and the install is rather small so even if I have to start from a clean db, I am fine with that. I just need to get this thing back up and running.

The best part is the unit went out of warranty ~45 days ago

Thanks much,

Bob

---- LOG ----
Insyde Software XpressROM NX BIOS Version 4.00.01
Copyright 1983 - 2005 Insyde Software Corp. All Rights Reserved
AMD Geode NX SBC Logic Platform BIOS 4.00.02
Project ID : 3214 (AMD Geode NX with SiS 746/964)
Base Memory 0000640 Kb
Extended Memory 0261120 Kb
Total Memory 0262144 Kb
Auto Detecting IDE Devices
Primary Master : ST380815AS
Primary Slave : NONE
Secondary Master : NONE
Secondary Slave : NONE


666 Mhz Mobile AMD Athlon (tm) XP 0000+ CPU
External Cache: 256K installed


Copyright 2005 3Com Corp. All Rights Reserved

ERROR - Zarlink switch not responding
Hit any key to break into boot monitor
Hit any key to break into boot monitor
Hit any key to break into boot monitor
Hit any key to break into boot monitor
Hit any key to break into boot monitor
Loading OS ....

VxWorks System Boot

Copyright 1984-1998 Wind River Systems, Inc.

CPU: PC PENTIUM
Version: 5.4.2
BSP version: 1.2/0
Creation date: Nov 30 2006, 11:45:36

Attaching to ATA disk device... done.

Flash type ef(M29W400BB) found, start address = 0xfffc0000.
Checking NCP Flash Code Revision
Read 0x40000 bytes from /dos0/MMARBIOS.bin
Prom version MM V1_0_1
File version MM V1_0_1
PROM at correct revision
No NCP Flash Upgrade Needed
Read version R6_0_55 from /dos0/NBXVer.cfg file

Press any key to stop auto-boot...
7
6
5
4
3
2
1
0

auto-booting...

boot device : ata=0,0
unit number : 0
processor number : 0
host name : host
file name : /dos0/R6_0_55/vxWorks
inet on ethernet (e) : 192.0.0.1
flags (f) : 0x0
target name (tn) : NBOSS
other (o) : ndp

Loading /dos0/R6_0_55/vxWorks...16315608 + 5933328 + 1317008
Starting at 0x108000...

0xfffee7c (tRootTask): GPIO Initialize: platform = MONTIMARE
User: target
Attached TCP/IP interface to ndp unit 0
Attaching network interface lo0... done.
NFS client support not included.
0xfffee7c (tRootTask): Disk 0: (pri) Model ST380815AS, SN 9QZBN3HR, Firmware 4.AAB, 80031 MB
0xfffee7c (tRootTask): Mirroring Not Enabled for any disk
0xfffee7c (tRootTask): ataDevCreate(0, 0, 0, 63): no mirroring
Loading symbol table from /dos0/R6_0_55/vxWorks.sym ...Opening /dos0/R6_0_55/vxWorks.sym: status = 0x380003


]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]
]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]
]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]
]]]]]]]]]]] ]]]] ]]]]]]]]]] ]] ]]]] (R)
] ]]]]]]]]] ]]]]]] ]]]]]]]] ]] ]]]]
]] ]]]]]]] ]]]]]]]] ]]]]]] ] ]] ]]]]
]]] ]]]]] ] ]]] ] ]]]] ]]] ]]]]]]]]] ]]]] ]] ]]]] ]] ]]]]]
]]]] ]]] ]] ] ]]] ]] ]]]]] ]]]]]] ]] ]]]]]]] ]]]] ]] ]]]]
]]]]] ] ]]]] ]]]]] ]]]]]]]] ]]]] ]] ]]]] ]]]]]]] ]]]]
]]]]]] ]]]]] ]]]]]] ] ]]]]] ]]]] ]] ]]]] ]]]]]]]] ]]]]
]]]]]]] ]]]]] ] ]]]]]] ] ]]] ]]]] ]] ]]]] ]]]] ]]]] ]]]]
]]]]]]]] ]]]]] ]]] ]]]]]]] ] ]]]]]]] ]]]] ]]]] ]]]] ]]]]]
]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]
]]]]]]]]]]]]]]]]]]]]]]]]]]]]] Development System
]]]]]]]]]]]]]]]]]]]]]]]]]]]]
]]]]]]]]]]]]]]]]]]]]]]]]]]] VxWorks version 5.4.2
]]]]]]]]]]]]]]]]]]]]]]]]]] KERNEL: WIND version 2.5
]]]]]]]]]]]]]]]]]]]]]]]]] Copyright Wind River Systems, Inc., 1984-1999

CPU: PC PENTIUM. Processor #0.
Memory Size: 0xffff000. BSP version 1.2/0.
WDB: Ready.

0xfffee7c (tRootTask-> ): bootDev: ata=0,0
0xfffee7c (): task dead===== /dos0 partition start 0x3f, 0x961a1 sectors ===
0xfffee7c (): task deadataDevCreate(0, 0, 614817, 63): no mirroring
0xfffee7c (): task deadataDevCreate(0, 0, 0, 614881): no mirroring
0xfffee7c (): task dead===== /htfs0 start 0x961e1, 0x94796cf sectors (to end of disk) ===
0xf948db0 (nbxTaskInit):
********************************************************************************

COPYRIGHT 3COM CORPORATION 1997-2006, ALL RIGHTS RESERVED

NOTICE

PROPRIETARY - CONFIDENTIAL

This document contains copyrighted and/or trade secret and confidential
information which is the property of 3Com Corporation, and is furnished for
the sole purpose of the operation and the maintenance of products of
3Com Corporation. No part of this publication is to be used for any other
purpose, and is not to be reproduced, copied, disclosed, transmitted,
stored in a retrieval system or translated into any human or computer
language, in any form, by any means, in whole or in part, without the prior
express written consent of 3Com Corporation.


********************************************************************************
Datalight Reli0xance v1.20.0395
f948db0Copyright (c) 2 (003 - 2005 DatalignbxTaskInitht, Inc.
): Hostname = NBOSS
0xf948db0 (nbxTaskInit): IP Address = 192.0.0.1

NBXFS stat.0xf_blocks f948db0 (nbxTaskInit38921651): stat.f_bsize Ethernet Address = 00:e0:bb:39:3d:35
20480xf948db0
(nbxTaskInit):
***** Type nbxBreak now to have booting stop after /htfs0 partition initialized *****
0xf948db0 (nbxTaskInit): Got nbxLogger initialization
0xf948db0 (nbxTaskInit): NBossConfigInit() was successful
0xf948db0 (nbxTaskInit): NBossConfigGetIsDownloadEnabled() 1
0xf948db0 (nbxTaskInit): NBossConfigGetDatabaseSelection() 0
0xf948db0 (nbxTaskInit): NBossConfigGetDoHtfsInit() 0
0xf948db0 (nbxTaskInit): NBossConfigGetIPHostname() V3001R
0xf948db0 (nbxTaskInit): NBossConfigGetIPAddress() 10.1.10.250
0xf948db0 (nbxTaskInit): NBossConfigGetIPGateway() 10.1.10.1
0xf948db0 (nbxTaskInit): NBossConfigGetSubnetMask() 255.255.255.0
0xf948db0 (nbxTaskInit): nbxMountNBXFSV2() Mounted NBXFSV2 file system on /htfs0 partition
0xf948db0 (nbxTaskInit): ç0xf948db0 (nbxTaskInit):
NBXFSV2 file system is in use on /htfs0 partition.
0xf948db0 (nbxTaskInit): nbxFsckFSV2() Skipping fsck.
0xf948db0 (nbxTaskInit): Completed fsck.
0xf948db0 (nbxTaskInit): DbConfigInit() was successful
0xf948db0 (nbxTaskInit): Hostname = V3001R
0xf948db0 (nbxTaskInit): IP Address = 10.1.10.250
0xf948db0 (nbxTaskInit): Ethernet Address = 00:e0:bb:39:3d:35
Determining schema match on /htfs0/R6_0_55/NBoss/DB/Import/schemaid.dat
==================================================================
NBX Database Initialization
Schema Version: 000002037
Content Version: 000002037
Date: 07/08/2006
==================================================================
Initializing RAIMA task f948db0 on=201a0 off=2 lock=1 files=40 multi=1...
Task f948db0 (nbxTaskInit) opened DB connection. Active cnxns = 1
Retrieved system profile [Valued Customer 0]...
Retrieving vpim profile... Found 1.
Retrieving feature and feature codes... Found 53.
Retrieving dial plan exceptions... Found 0.
Retrieving device engineering data... Found 90.
Retrieving NBSet,TLIM and Attendant Console device configurations... Found 145.
Retrieving trunk group data... Found 6.
Retrieving device timer data... Found 40.
Retrieving user group profiles... Found 3.
Retrieving user profiles... Found 21.
Retrieving system and user offsite notification profiles... Found 0.
Retrieving auto-attendant menu... Found 2.
Retrieving extension profiles... Found 335.
Retrieving hunt group members... Found 45.
Retrieving acd group members... Found 0.
Retrieving acd group announcements... Found 0.
Retrieving acd custom hours... Found 0.
Initializing multi-lingual prompt server...
Initializing billing server...
Retrieving offsite notification devices... Found 4.
Retrieving call pickup data... Found 50.
Retrieving zone page members... Found 0.
Placing users into default pickup group... Found 0.
Retrieving dial plan... Found 3.
Retrieving account codes... Found 0.
Retrieving VM Distribution Lists... Found 1.
Retrieving VM Distribution List Members... Found 21.
Retrieving Generic Gateway Profiles... Found 0.
Retrieving SIP Associated Devices... Found 0.
Retrieving Custom Compressions... Found 0.
Retrieving Whisper Page Domains... Found 0.
Retrieving WhisperPage Domain Members... Found 0.
Retrieving WhisperPage Domain Member Properties... Found 0.
Retrieving MWB Domains... Found 1.
Retrieving MWB Domain Members... Found 0.
Grouping common devices... Found 6.
==================================================================
NBX Database Initialization complete.
Version: 0.81.02.01
==================================================================
i18n system initializing
i18n system initialized
0xf948db0 (nbxTaskInit): initFTP: Invalid administrator password from system profile
Task edbfe98 (nbxCCMDBserver) opened DB connection. Active cnxns = 2
Task edb783c (nbxACDRTS) opened DB connection. Active cnxns = 3
Task ed2a7c0 (tEmWeb) opened DB connection. Active cnxns = 4
Task ec93814 (nbxCallProcessing) opened DB connection. Active cnxns = 5

CallP : Wait for a max of 120 sec for the link to come up
0xef86d20 (nbxH3initialization):
NBX H3 Started, H3 Version: 0x202 Date: 11/11/1999 9:00am
Task ef8405c (nbxVMAA0) opened DB connection. Active cnxns = 6
Task dbdb960 (nbxVMAA1) opened DB connection. Active cnxns = 7
Task dba6d90 (nbxVMAA2) opened DB connection. Active cnxns = 8
Task db721c0 (nbxVMAA3) opened DB connection. Active cnxns = 9
Task db2a428 (nb

^^^ System locks there every time on the exact same character.

[bosoxfan]

Something in the 6.0.55 db is keeping this from booting.. what you need to do is boot to a previous version, then delete the db on 6.0.55, then boot back to 6.0.55 and restore your backup db to the system.

To boot to a prevous version:

1. Reboot the NBX, watch the console, you need to stop the boot After seeing "Press any key for auto-reboot"..

2. type "c", then return

3. continue to scroll down the vxworks prompt until you see

file name : /dos0/R6_0_55/vxWorks - right next to this type "/dos0/R6_0_54/vxWorks" (type everything in the quotes, but not the quotes), hit return (you did say you have 6.0.54 on the disk, right)

4. hit return multilple times until you see the vxworks prompt, then type @

5. the boot will continue to 6.0.54

now you have a console prompt and can delete the db on 6.0.55. to do this, type the following:

cd "/htfs0/R6_0_55/NBoss/DB" <ret> (you must type the quotes)

vxtar "-xvf","cleandb.tar" (you must type the quotes)

Now reboot the box. .it will come up on 6.0.55 with a blank database and you can restore you original db.

[merlin215]

why are you running R6.0.55 code ? It has a lot of bugs . I would suggest upgrading to R6.0.63 or R6.5.22P03 . Bother are free and have the latest fixes in them . 

[rythan73]

---

@bosoxfan wrote:

Something in the 6.0.55 db is keeping this from booting.. what you need to do is boot to a previous version, then delete the db on 6.0.55, then boot back to 6.0.55 and restore your backup db to the system.

To boot to a prevous version:

1. Reboot the NBX, watch the console, you need to stop the boot After seeing "Press any key for auto-reboot"..

2. type "c", then return

3. continue to scroll down the vxworks prompt until you see

file name : /dos0/R6_0_55/vxWorks - right next to this type "/dos0/R6_0_54/vxWorks" (type everything in the quotes, but not the quotes), hit return (you did say you have 6.0.54 on the disk, right)

4. hit return multilple times until you see the vxworks prompt, then type @

5. the boot will continue to 6.0.54

now you have a console prompt and can delete the db on 6.0.55. to do this, type the following:

cd "/htfs0/R6_0_55/NBoss/DB" <ret> (you must type the quotes)

vxtar "-xvf","cleandb.tar" (you must type the quotes)

Now reboot the box. .it will come up on 6.0.55 with a blank database and you can restore you original db.

---

I tried booting to 6_0_54 and it locked at the same place.  I will try it again, maybe I misstyped something.  The HD says it has 6_0_54 & 6_0_55 but I cannot say if that version was deleted at some point.

Thanks for the help, I will reply to let you know what happens.

[rythan73]

---

@merlin215 wrote:

why are you running R6.0.55 code ? It has a lot of bugs . I would suggest upgrading to R6.0.63 or R6.5.22P03 . Bother are free and have the latest fixes in them . 

---

I shall if I can ever get back in to the system.  That is one downfall I always hated about the NBX.  It was able to be bricked by database corruption unless you had previous versions to boot from.

I guess worst case if it is a DB problem, I just need a new replacement HD.

I used to back up these drives with dd and a linux live cd, anyone ever try to mount the volume and extract the cleandb.tar manually? :)

[merlin215]

if you cannot get back to a workable code then you may have to replace the drive . it sounds like the code is corrupt . what type of system ? V3001 's came with R6.0.55 or 6.0.57 I believe and not 2 codes . 

V3000's came with older R4.4.7  I think ....

[bosoxfan]

Wow! That is quite strange that you would have a corrupted db on two versions... you would not have been able to upgrade on 6.0.54 if you had a corrupt db.... so this is sounding like a disk issue.....

can you get to the prompt at all??? try control-c....

if you can get to the prompt, then you can delete the db

cd "/htfs0/R6_0_55/NBoss/DB" <ret> (you must type the quotes)

vxtar "-xvf","cleandb.tar" (you must type the quotes)

....

[rythan73]

I was able to get to a prompt by running the "nbxBreak" command during boot and I was able to install the Clean database via vxtar but the problem still exists.  It still freezes at the exact same point.

I am thinking that something else non DB related is corrupt on the drive.  Anyone had any luck mounting the drive on a linux livecd?

[merlin215]

To change versions , go to the message " press any key to stop auto reboot " then you will see a timer and you hit any key to stop the boot , then you will be able to follow BoSoxFans instructions to either delete the db or change versions of code . 

If you change versions of code , may be easier ,  you can then go in to netset tab to and remove the software version that you have an issue with .

Then re install your R6.5 code and  your db . 

Either way , I am not sure what went wrong .. Did you go into the correct directroy ? You may have to change versions of code as I have seen this before as well . 

[rythan73]

I was able to boot to the 6_0_54 and I was able to overwrite the DB with the clean DB but it still locks at exactly the same spot.  I think either something else on the drive that isn't database related got corrupt or something in the hardware has failed and is locking the system up when the driver loads.

Looks like I will be ordering some hardware but I am not sure if I need to get a HD or an FRU for the chassis.  I just wish I knew which one....