IP Telephony - NBX
cancel
Showing results for 
Search instead for 
Did you mean: 

3001R crashed- Ideas?

DanCarroll
Frequent Advisor

3001R crashed- Ideas?

My system stopped working yesterday at 132 PM, hyperterm  showed this line--



** Illegal Index (57) for collection with 57 elements


** Processing Suspended.



nboss had these entries also---



  


0304:094757:0826 MediaServer    W No Flow Ctrl from MAC 00:00:00 cleaning up - Task_03                                                                   


0304:094827:0961 SNTP           W LI VN Mode do not conform in the SNTP packet received from the Server. ucLIVNMode: 2, ucStratum: 0                     


0304:094827:0961 SNTP           W Invalid LI VN Mode received from the Server.                                                                           


0304:094827:0961 DBI            E findAndWrite Error k 1000 v 524 r 10001 v 524 b -52 l 0 f 0 w 0 e 0                                                    


0304:094827:0961 DBI            E findAndWrite Error k 1000 v 153 r 10001 v 153 b -52 l 0 f 0 w 0 e 0                                                    


0304:094827:0961 DBI            E findAndWrite Error k 1000 v 657 r 10001 v 657 b -52 l 0 f 0 w 0 e 0                                                    


0304:094827:0961 DBI            E findAndWrite Error k 1000 v 11251 r 10001 v 11251 b -52 l 0 f 0 w 0 e 0                                                


0304:094827:0961 DBI            E findAndWrite Error k 1000 v 167 r 10001 v 167 b -52 l 0 f 0 w 0 e 0                                                    


0304:094827:0961 DBI            E findAndWrite Error k 1000 v 213 r 10001 v 213 b -52 l 0 f 0 w 0 e 0                                                    


0304:094827:0961 DBI            E findAndWrite Error k 1000 v 656 r 10001 v 656 b -52 l 0 f 0 w 0 e 0                                                    


0304:094827:0961 DBI            E findAndWrite Error k 1000 v 248 r 10001 v 248 b -52 l 0 f 0 w 0 e 0                                                    


0304:094827:0961 DBI            E findAndWrite Error k 1000 v 172 r 10001 v 172 b -52 l 0 f 0 w 0 e 0                                                    


0304:094827:0961 DBI            E findAndWrite Error k 1000 v 424 r 10001 v 424 b -52 l 0 f 0 w 0 e 0                                                    


0304:094827:0961 DBI            E findAndWrite Error k 1000 v 659 r 10001 v 659 b -52 l 0 f 0 w 0 e 0                                                    


0304:094827:0961 DBI            E findAndWrite Error k 1000 v 133 r 10001 v 133 b -52 l 0 f 0 w 0 e 0                                                    


0304:094827:0961 DBI            E findAndWrite Error k 1000 v 675 r 10001 v 675 b -52 l 0 f 0 w 0 e 0                                                    


0304:094827:0961 DBI            E findAndWrite Error k 1000 v 134 r 10001 v 134 b -52 l 0 f 0 w 0 e 0                                                    


0304:094836:0211 DBI            W Failure bSetAvgWaitTimeRTS.                                                                                            


03



What makes a Vailseesion do a log dump as seen below?



** START OF VAILSess4 LOG DUMP **                                                                                    


0304:130250:0019 VAILSess4      E 1267725705 EV DTMFRcvd(1) in ST MediaPlay: NEXT ST MediaPlay                                                           


0304:130250:0019 VAILSess4      E 1267725705 EV PlayCompl in ST MediaPlay: NEXT ST Connected                                                             


0304:130250:0019 VAILSess4      E 1267725705 EV PlayFile in ST Connected: NEXT ST PlayConfig                                                             


0304:130250:0019 VAILSess4      E 1267725706 EV PlayDevConfiged in ST PlayConfig: NEXT ST MediaPlay                                                      


0304:130250:0019 VAILSess4      E 1267725707 State MediaPlay; call StopMediaPlay() due to term DTMF 2                                                    


0304:130250:0019 VAILSess4      E 1267725707 EV DTMFRcvd(2) in ST MediaPlay: NEXT ST MediaPlay                                                           


0304:130250:0019 VAILSess4      E 1267725707 EV PlayCompl in ST MediaPlay: NEXT ST Connected                                                             


0304:130250:0019 VAILSess4      E 1267725708 EV DTMFRcvd(7) in ST Connected: NEXT ST Connected                                                           


0304:130250:0019 VAILSess4      E 1267725708 EV DTMFRcvd(3) in ST Connected: NEXT ST Connected                                                           


0304:130250:0019 VAILSess4      E 1267725709 EV DTMFRcvd(2) in ST Connected: NEXT ST Connected                                                           


0304:130250:0019 VAILSess4      E 1267725712 EV PlayFile in ST Connected: NEXT ST PlayConfig                                                             


0304:130250:0019 VAILSess4      E 1267725712 EV PlayFile in ST PlayConfig: NEXT ST PlayConfig                                                            


0304:130250:0019 VAILSess4      E 1267725712 EV PlayDevConfiged in ST PlayConfig: NEXT ST MediaPlay                                                      


0304:130250:0019 VAILSess4      E 1267725716 State MediaPlay PauseAudio(): calling StopMediaPlay()                                                       


0304:130250:0019 VAILSess4      E 1267725716 EV PauseAudio in ST MediaPlay: NEXT ST Hold                                                                 


0304:130250:0019 VAILSess4      E 1267725716 EV PlayCompl in ST Hold: NEXT ST Hold                                                                       


0304:130250:0019 VAILSess4      E 1267725717 EV CallReleased in ST Hold: NEXT ST ReleaseCall                                                             


0304:130250:0019 VAILSess4      E 1267725717 Unreserved Media Handle 4                                                                                   


0304:130250:0019 VAILSess4      E 1267725717 EV ReleaseCall in ST ReleaseCall: NEXT ST Idle                                                              


0304:130250:0019 VAILSess4      E 1267725732 Device 76 Reserved Media Session with handle 4                                                              


0304:130250:0019 VAILSess4      E 1267725732 EV Reserve in ST Idle: NEXT ST Reserved                                                                     


0304:130250:0019 VAILSess4      E 1267725735 IncomingCall: Remote MAC Addr = 00:e0:bb:22:7c:7a                                                           


0304:130250:0019 VAILSess4      E 1267725735 App AA, Calling , Called , FwdReason 6                                                                      


0304:130250:0019 VAILSess4      E 1267725735 EV IncomingCall in ST Reserved: NEXT ST IncomingCall                                                        


0304:130250:0019 VAILSess4      E 1267725735 EV AnsCall in ST IncomingCall: NEXT ST AnswerCall                                                           


0304:130250:0019 VAILSess4      E 1267725735 EV CallConnected in ST AnswerCall: NEXT ST Connected                                                        


0304:130250:0019 VAILSess4      E 1267725735 EV DelayedAnn in ST Connected: NEXT ST Connected                                                            


0304:130250:0019 VAILSess4      E 1267725735 EV PlayFile in ST Connected: NEXT ST PlayConfig                                                             


0304:130250:0019 VAILSess4      E 1267725735 EV PlayDevConfiged in ST PlayConfig: NEXT ST MediaPlay                                                      


0304:130250:0019 VAILSess4      E 1267725735 EV PlayDevConfiged in ST MediaPlay: Ignored                                                                 


0304:130250:0019 VAILSess4      E 1267725747 EV PlayCompl in ST MediaPlay: NEXT ST Connected                                                             


0304:130250:0019 VAILSess4      E 1267725747 EV PauseAudio in ST Connected: NEXT ST Hold                                                                 


0304:130250:0019 VAILSess4      E 1267725747 EV CallReleased in ST Hold: NEXT ST ReleaseCall                                                             


0304:130250:0019 VAILSess4      E 1267725747 Unreserved Media Handle 4                                                                                   


0304:130250:0019 VAILSess4      E 1267725747 EV ReleaseCall in ST ReleaseCall: NEXT ST Idle                                                              


0304:130250:0019 VAILSess4      E 1267725760 Device 54 Reserved Media Session with handle 4                                                              


0304:130250:0019 VAILSess4      E 1267725760 EV Reserve in ST Idle: NEXT ST Reserved                                                                     


0304:130250:0019 VAILSess4      E 1267725760 Caller ID name Allan Bredice, number 1201                                                                   


0304:130250:0019 VAILSess4      E 1267725760 EV SetCallerId in ST Reserved: NEXT ST Reserved                                                             


0304:130250:0019 VAILSess4      E 1267725760 IncomingCall: Remote MAC Addr = 00:00:00:00:00:00                                                           


0304:130250:0019 VAILSess4      E 1267725760 App VM, Calling 4408461404, Called 1201, FwdReason 2                                                        


0304:130250:0019 VAILSess4      E 1267725760 EV IncomingCall in ST Reserved: NEXT ST IncomingCall                                                        


0304:130250:0019 VAILSess4      E 1267725760 EV AnsCall in ST IncomingCall: NEXT ST AnswerCall                                                           


0304:130250:0019 VAILSess4      E 1267725760 Caller ID name RH1-4, number 4408461404                                                                     


0304:130250:0019 VAILSess4      E 1267725760 EV SetCallerId in ST AnswerCall: NEXT ST AnswerCall                                                         


0304:130250:0019 VAILSess4      E 1267725761 MAC Addr of remote device changed to 00:e0:bb:22:7c:7a, Ip Addr 0.0.0.0                                     


0304:130250:0019 VAILSess4      E 1267725761 EV SetDevAddr in ST AnswerCall: NEXT ST AnswerCall                                                          


0304:130250:0019 VAILSess4      E 1267725761 EV CallConnected in ST AnswerCall: NEXT ST Connected                                                        


0304:130250:0019 VAILSess4      E 1267725761 EV PlayFile in ST Connected: NEXT ST PlayConfig                                                             


0304:130250:0019 VAILSess4      E 1267725761 MAC Addr of remote device changed to 00:e0:bb:22:7c:7a, Ip Addr 0.0.0.0                                     


0304:130250:0019 VAILSess4      E 1267725761 EV SetDevAddr in ST PlayConfig: NEXT ST PlayConfig                                                          


0304:130250:0019 VAILSess4      E 1267725761 Caller ID name RH1-4, number 4408461404                                                                     


0304:130250:0019 VAILSess4      E 1267725761 EV SetCallerId in ST PlayConfig: NEXT ST PlayConfig                                                         


0304:130250:0019 VAILSess4      E 1267725761 Caller ID name RH1-4, number 4408461404                                                                     


0304:130250:0019 VAILSess4      E 1267725761 EV SetCallerId in ST PlayConfig: NEXT ST PlayConfig                                                         


0304:130250:0019 VAILSess4      E 1267725761 EV PlayDevConfiged in ST PlayConfig: NEXT ST MediaPlay                                                      


0304:130250:0019 VAILSess4      E 1267725768 EV PlayCompl in ST MediaPlay: NEXT ST Connected                                                             


0304:130250:0019 VAILSess4      E 1267725768 EV PlayFile in ST Connected: NEXT ST PlayConfig                                                             


0304:130250:0019 VAILSess4      E 1267725768 EV PlayDevConfiged in ST PlayConfig: NEXT ST MediaPlay                                                      


0304:130250:0019 VAILSess4      E 1267725769 State MediaPlay; call StopMediaPlay() due to term DTMF 1                                                    


0304:130250:0019 VAILSess4      E 1267725769 EV DTMFRcvd(1) in ST MediaPlay: NEXT ST MediaPlay                                                           


0304:130250:0019 VAILSess4      E 1267725770 EV PlayCompl in ST MediaPlay: NEXT ST Connected                                                             


0304:130250:0019 VAILSess4      E 1267725770 EV RecordFile in ST Connected: NEXT ST RecordConfig                                                         


0304:130250:0019 VAILSess4      E 1267725770 EV RecDevConfiged in ST RecordConfig: NEXT ST MediaRecord                                                   


0304:130250:0019 VAILSess4      E ** END OF VAILSess4 LOG DUMP **



Any thoughts as to what was happenning at the time?



The log dump happened 30 minutes before the suspension, no other entries shown in nboss at the time.



8 REPLIES
merlin_1
Super Advisor

Re: 3001R crashed- Ideas?

** Illegal Index (57) for collection with 57 elements





** Processing Suspended.



 



What process suspended  ?



 



When the system goes down , the vmail will do those vail sessions automatically



 



The other errors look like software corruption errors . After hours reboot and see if they clean up



This message was edited by merlin on 3-5-10 @ 7:11 AM
bosoxfan_1
Super Advisor

Re: 3001R crashed- Ideas?

Are you using ACD groups? we have seen this with some ACD groups being corrupted.



Go to the Status page on all your ACD groups and check the Total Number of Members.. make



sure that the "Logged in" members plus the "logged out" members equal the Total number of members. if not, then delete the ACD group and rebuild it



DanCarroll
Frequent Advisor

Re: 3001R crashed- Ideas?

Merlin-nbxcallprocessor suspended



I did a clean reboot this am from Netset. I will monitor and see if those entries reappear.



Bosoxfan-



Yes I have acd groups, all members counts match,



But



I had 3 hunt groups that did not match.,meaning the rank is off. 



I used the change rank feature at bottom of page -Changed ranks ----all match now.



 



 



DanCarroll
Frequent Advisor

Re: 3001R crashed- Ideas?

down again today.



Time to purge data base  I guess.



 



merlin_1
Super Advisor

Re: 3001R crashed- Ideas?

You should open a case . If call p is suspending then you need to do a trace and a dump core . I forget what code you are running but I know you have a V5000 system .



DanCarroll
Frequent Advisor

Re: 3001R crashed- Ideas?

was a v5000- now a v3001r with same database.



I thought corruption was cleared. and had done everything I clould to find and clean the database..



The only pickup group I have active is Call Pickup group  0 . It has 2 members. But no extensionsn are  is listed...



bosoxfan_1
Super Advisor

Re: 3001R crashed- Ideas?



Problem: System crashes every few days with "illegal index error" on the CLI every time. Rebooting the system brings it back up.



Cause: The NBX database is corrupted



Fix: Restore from backup before the system crashed the first time



Alternate workaround: You may attempt to find and correct the corruption. We have found that ACD groups and pickup groups corrupted as seen in netset have caused this illegal index error, and in prior cases rebuilding the groups (deleting and adding back again) has resolved the crashes and illegal index errors. Please note this is NOT the recommended or sure way to fix the problem and should only be used if there are NO prior backups and a full system rebuild is unwieldy.





Look for corruption in pickup groups and correct if found:





- In Netset, go to Feature Settings, then Call Pickup groups.



- For each pickup group, note the number of members stated, then click on each pickup group, and COUNT the members to confirm the total actual members matches the total for that group on the summary page. If there is a descrepancy, delete ALL members for that pickup group, and add them back again, and confirm the counts match



- Do for each pickup group





Look for ACD group member corruption and fix:





- In Netset, go to Call Distribution Groups, ACD groups



- For each ACD group, note the number of members stated, then click on each ACD group extension, then click Membership tab, and COUNT the members. If there is a discrepancy, remove all members of that ACD group and add them back, confirm totals are now equal - actual member count equals number shown on main page



- Note the rank numbers. For example if there are 2 members then ranks should be 1 and 2 only, you shouldn't see a rank #3 or higher if there are only 2 members



- If the number of ACD group agents shown in the Membership tab matches the reported total on the main page, then click "Show All" to show all phones, and now look for any DUPLICATE names. If there are any duplicate names , remove all acd members from this group and add them back, then again click "Show All" and confirm no more duplicates. If there are still duplicate names you must delete any duplicate users and re-create them. You can assign them the same extension number after you rebuild them, but this will require them to lose all of their voicemail, and personal extension settings, and custom button mappings



- If the number of ACD group agents shown in the Membership tab does not match the reported total on the main page, remove all acd agents, add them back, and confirm that this time the number of members matches the total reported on the main page.



 





Look for Hunt group member corruption and fix:





- In Netset, go to Call Distribution Groups, Hunt Groups



The procedure is a bit different for hunt groups. The main page of hunt group list does not show membership totals, please follow these steps:



- Click on each hunt group extension. Below the settings, it lists the members. The list should default to just members with check boxes (members). If instead the "show members only" button is grayed out, and none of the boxes are checked in the list of phones, this hunt group is corrupted, and must be manually rebuilt by deleting the entire hunt group and adding it back. You can't just remove all members and add them back since you can't see the checked members on the page in order to delete them. Delete the entire hunt group after recording all of the group parameters (forwarding, number of rings, etc).



- If there is at least one member showing as checked when you click to get into the hunt group below the hunt group settings, then click on the Status tab. This will show you the members from a different view for status. Manually count the number of members on this screen, make sure there are no duplicates, and compare this total to the number of members with green check boxes under the Modify tab list - they should be identical. If they are not, this hunt group is corrupted, and you must delete the entire hunt group, and rebuild it. You can assign the same hunt group extension when you rebuild it



- Do this for ALL hunt groups





DanCarroll
Frequent Advisor

Re: 3001R crashed- Ideas?

UPDATE-



Well I did purge and rebuild the data base.



The size went from 1.080 Gig tto 280 meg.



Finished reprograming the last of it yesterday.