Nimble logs

[Cloudcover]

Hello Nimble Gurus

can you pls assist us with the assessment of the logs? We are trying to determine if this is a pre-existing issue

We see in the logs that multiple unexpected controller reboots have started from Aug 2020 but the customer is saying it is due to power outages

Then in Oct of 2023, the customer updated their software version from  5.2.1.1100-1027043-op to 6.1.1.300-1028597-opt

Less than a month later, Nov 11 2023 controller B has been rebooting unexpectedly every week.

Can some nimble guru advise whether this issue is pre-existing or not?

will attach log files but trying to figure out how at the moment

Thank you Gurus

[SEENIVASANP]

if the customer says it is due to a power event you could see power-related events in the logs. else it could be due to some other underlaying issue.

[Cloudcover]

Hi SEENIVASANP

Thank you for your reply. If you will be kind enough to review the logs and let us know your assessment if this a pre-existing or not

       583 INFO     Mar  5 2022 08:42:37  12201 hardware    AF-213000  IP interface 172.17.1.197 up on controller B NIC port eth0a                    581 WARNING  Mar  5 2022 08:43:10   2028 hardware    AF-213000  Controller B rebooted unexpectedly. Contact Nimble Storage Support.                582 INFO     Mar  5 2022 08:43:10   2101 service     AF-213000  System services started. Software version is 5.0.7.300-644174-opt                  578 INFO     Mar  5 2022 08:43:20  12343 hardware    AF-213000  Fan "fan1a" on shelf "AF-213000" at controller B is now running at speed 10584 rpm              579 INFO     Mar  5 2022 08:43:20  12343 hardware    AF-213000  Fan "fan1b" on shelf "AF-213000" at controller B is now running at speed 10976 rpm              580 INFO     Mar  5 2022 08:43:20  12343 hardware    AF-213000  Fan "fan2a" on shelf "AF-213000" at controller B is now running at speed 10486 rpm              597 INFO     Mar  5 2022 08:43:20  12343 hardware    AF-213000  Fan "fan2b" on shelf "AF-213000" at controller B is now running at speed 11270 rpm              584 INFO     Mar  5 2022 08:43:21  12356 hardware    AF-213000  Temperature sensor "outlet-temp" on shelf "AF-213000" at outlet temp of controller B is operational: current temperature is 48 C        585 INFO     Mar  5 2022 08:43:21  12356 hardware    AF-213000  Temperature sensor "RightEar" on shelf "AF-213000" at right-side ambient of controller B is operational: current temperature is 28 C        586 INFO     Mar  5 2022 08:43:21  12348 hardware    AF-213000  Power supply "power-supply1" on shelf "AF-213000" at lower right rear is OK                587 INFO     Mar  5 2022 08:43:21  12348 hardware    AF-213000  Power supply "power-supply2" on shelf "AF-213000" at upper right rear is OK                588 INFO     Mar  5 2022 08:43:21  12356 hardware    AF-213000  Temperature sensor "LeftEar" on shelf "AF-213000" at left-side ambient of controller A is operational: current temperature is 29 C        598 INFO     Mar  5 2022 08:43:21  12343 hardware    AF-213000  Fan "fan3a" on shelf "AF-213000" at controller B is now running at speed 10388 rpm              608 INFO     Mar  5 2022 08:43:21  12343 hardware    AF-213000  Fan "fan3b" on shelf "AF-213000" at controller B is now running at speed 10976 rpm              609 INFO     Mar  5 2022 08:43:21  12356 hardware    AF-213000  Temperature sensor "inlet-temp" on shelf "AF-213000" at inlet temp of controller B is operational: current temperature is 40 C        589 INFO     Mar  5 2022 08:43:56  12354 hardware    AF-213000  System temperature is now within range: current temperature for sensor "inlet-temp" on shelf "AF-213000" at inlet temp of controller B is 40 C        606 INFO     Mar  5 2022 08:43:56  12354 hardware    AF-213000  System temperature is now within range: current temperature for sensor "outlet-temp" on shelf "AF-213000" at outlet temp of controller B is 48 C        607 INFO     Mar  5 2022 08:43:56  12354 hardware    AF-213000  System temperature is now within range: current temperature for sensor "LeftEar" on shelf "AF-213000" at left-side ambient of controller A is 29 C        610 INFO     Mar  5 2022 08:43:56  12354 hardware    AF-213000  System temperature is now within range: current temperature for sensor "RightEar" on shelf "AF-213000" at right-side ambient of controller B is 28 C        596 INFO     Mar  5 2022 08:44:20  12201 hardware    AF-213000  IP interface 172.17.1.196 up on controller A NIC port eth0a                    590 INFO     Mar  5 2022 08:44:53   2101 service     AF-213000  System services started. Software version is 5.0.7.300-644174-opt                  591 WARNING  Mar  5 2022 08:44:53   2028 hardware    AF-213000  Controller A rebooted unexpectedly. Contact Nimble Storage Support.                592 INFO     Mar  5 2022 08:45:01  12201 hardware    AF-213000  IP interface 172.17.1.195 up on controller A NIC port eth0a                    593 INFO     Mar  5 2022 08:45:01  12201 hardware    AF-213000  IP interface 172.25.1.60 up on controller A NIC port tg1a                    594 INFO     Mar  5 2022 08:45:01  12201 hardware    AF-213000  IP interface 172.25.1.63 up on controller A NIC port tg1b                    595 INFO     Mar  5 2022 08:45:01  12201 hardware    AF-213000  IP interface 172.25.1.61 up on controller A NIC port tg1a                    575 INFO     Mar  5 2022 08:45:08  12343 hardware    AF-213000  Fan "fan1a" on shelf "AF-213000" at controller A is now running at speed 10584 rpm              576 INFO     Mar  5 2022 08:45:09  12343 hardware    AF-213000  Fan "fan1b" on shelf "AF-213000" at controller A is now running at speed 11270 rpm              577 INFO     Mar  5 2022 08:45:09  12343 hardware    AF-213000  Fan "fan2a" on shelf "AF-213000" at controller A is now running at speed 10682 rpm              599 INFO     Mar  5 2022 08:45:09  12343 hardware    AF-213000  Fan "fan2b" on shelf "AF-213000" at controller A is now running at speed 11074 rpm              600 INFO     Mar  5 2022 08:45:09  12356 hardware    AF-213000  Temperature sensor "inlet-temp" on shelf "AF-213000" at inlet temp of controller A is operational: current temperature is 39 C        601 INFO     Mar  5 2022 08:45:09  12356 hardware    AF-213000  Temperature sensor "outlet-temp" on shelf "AF-213000" at outlet temp of controller A is operational: current temperature is 47 C        603 INFO     Mar  5 2022 08:45:09  12343 hardware    AF-213000  Fan "fan3b" on shelf "AF-213000" at controller A is now running at speed 11270 rpm              604 INFO     Mar  5 2022 08:45:09  12343 hardware    AF-213000  Fan "fan3a" on shelf "AF-213000" at controller A is now running at speed 10584 rpm              602 INFO     Mar  5 2022 08:45:41  12354 hardware    AF-213000  System temperature is now within range: current temperature for sensor "outlet-temp" on shelf "AF-213000" at outlet temp of controller A is 47 C        605 INFO     Mar  5 2022 08:45:41  12354 hardware    AF-213000  System temperature is now within range: current temperature for sensor "inlet-temp" on shelf "AF-213000" at inlet temp of controller A is 39 C        612 INFO     Mar  5 2022 08:46:56  12308 hardware    AF-213000  NVRAM Battery 0 is OK on controller B                        611 INFO     Mar  5 2022 08:48:41  12308 hardware    AF-213000  NVRAM Battery 0 is OK on controller A                        614 INFO     Mar  5 2022 08:48:56  14023 hardware    AF-213000  Temperature sensor  "outlet-temp" on shelf "AF-213000" at controller B is and below critical level Celsius and below critical level        615 INFO     Mar  5 2022 08:48:56  14019 hardware    AF-213000  Ambient temperature on shelf AF-213000 is at 28 Celsius and is below warning level.              613 INFO     Mar  5 2022 08:50:41  14023 hardware    AF-213000  Temperature sensor  "outlet-temp" on shelf "AF-213000" at controller A is and below critical level Celsius and below critical level        616 INFO     Mar  5 2022 08:54:56  14019 hardware    AF-213000  Ambient temperature on shelf AF-213000 is at 28 Celsius and is below critical level.              652 WARNING  Mar 11 2022 16:52:58  12226 hardware    AF-213000  NIC port eth0b is down on controller B                      655 WARNING  Mar 11 2022 16:52:58  12226 hardware    AF-213000  NIC port eth0a is down on controller B                      617 WARNING  Mar 11 2022 16:55:10  12226 hardware    AF-213000  NIC port eth0a is down on controller A                      618 WARNING  Mar 11 2022 16:55:10  12226 hardware    AF-213000  NIC port eth0b is down on controller A                      649 INFO     Mar 11 2022 16:55:10  12225 hardware    AF-213000  NIC port eth0b is up on controller B                        651 INFO     Mar 11 2022 16:55:10  12225 hardware    AF-213000  NIC port eth0a is up on controller B                        619 CRITICAL Mar 11 2022 16:55:11  12223 hardware    AF-213000  Network connectivity lost, all links down on subnet with label management of type mgmt.            620 WARNING  Mar 11 2022 16:55:29   2109 service     -          Failed to send alert e-mail to SMTP server. Verify SMTP server and port configuration.              621 INFO     Mar 11 2022 16:57:14  12225 hardware    AF-213000  NIC port eth0a is up on controller A                        622 INFO     Mar 11 2022 16:57:14  12225 hardware    AF-213000  NIC port eth0b is up on controller A                        654 INFO     Mar 11 2022 17:22:20  12201 hardware    AF-213000  IP interface 172.17.1.197 up on controller B NIC port eth0a                    648 WARNING  Mar 11 2022 17:22:53   2028 hardware    AF-213000  Controller B rebooted unexpectedly. Contact Nimble Storage Support.                657 INFO     Mar 11 2022 17:22:53   2101 service     AF-213000  System services started. Software version is 5.0.7.300-644174-opt                  650 INFO     Mar 11 2022 17:23:03  12343 hardware    AF-213000  Fan "fan1b" on shelf "AF-213000" at controller B is now running at speed 9604 rpm              653 INFO     Mar 11 2022 17:23:03  12343 hardware    AF-213000  Fan "fan2a" on shelf "AF-213000" at controller B is now running at speed 9212 rpm              656 INFO     Mar 11 2022 17:23:03  12343 hardware    AF-213000  Fan "fan1a" on shelf "AF-213000" at controller B is now running at speed 9408 rpm              658 INFO     Mar 11 2022 17:23:03  12343 hardware    AF-213000  Fan "fan3a" on shelf "AF-213000" at controller B is now running at speed 9212 rpm              659 INFO     Mar 11 2022 17:23:03  12343 hardware    AF-213000  Fan "fan3b" on shelf "AF-213000" at controller B is now running at speed 9604 rpm              660 INFO     Mar 11 2022 17:23:03  12356 hardware    AF-213000  Temperature sensor "inlet-temp" on shelf "AF-213000" at inlet temp of controller B is operational: current temperature is 38 C        662 INFO     Mar 11 2022 17:23:03  12343 hardware    AF-213000  Fan "fan2b" on shelf "AF-213000" at controller B is now running at speed 9800 rpm              663 INFO     Mar 11 2022 17:23:03  12356 hardware    AF-213000  Temperature sensor "outlet-temp" on shelf "AF-213000" at outlet temp of controller B is operational: current temperature is 49 C        661 INFO     Mar 11 2022 17:23:40  12354 hardware    AF-213000  System temperature is now within range: current temperature for sensor "inlet-temp" on shelf "AF-213000" at inlet temp of controller B is 38 C        664 INFO     Mar 11 2022 17:23:40  12354 hardware    AF-213000  System temperature is now within range: current temperature for sensor "outlet-temp" on shelf "AF-213000" at outlet temp of controller B is 49 C        624 INFO     Mar 11 2022 17:24:26  12201 hardware    AF-213000  IP interface 172.17.1.196 up on controller A NIC port eth0a                    628 INFO     Mar 11 2022 17:24:59   2101 service     AF-213000  System services started. Software version is 5.0.7.300-644174-opt                  629 WARNING  Mar 11 2022 17:24:59   2028 hardware    AF-213000  Controller A rebooted unexpectedly. Contact Nimble Storage Support.                625 INFO     Mar 11 2022 17:25:00  12201 hardware    AF-213000  IP interface 172.25.1.60 up on controller A NIC port tg1a                    626 INFO     Mar 11 2022 17:25:00  12201 hardware    AF-213000  IP interface 172.25.1.63 up on controller A NIC port tg1b                    630 INFO     Mar 11 2022 17:25:00  12201 hardware    AF-213000  IP interface 172.17.1.195 up on controller A NIC port eth0a                    631 INFO     Mar 11 2022 17:25:00  12201 hardware    AF-213000  IP interface 172.25.1.61 up on controller A NIC port tg1a                    623 INFO     Mar 11 2022 17:25:07  12343 hardware    AF-213000  Fan "fan2a" on shelf "AF-213000" at controller A is now running at speed 9310 rpm              627 INFO     Mar 11 2022 17:25:07  12343 hardware    AF-213000  Fan "fan2b" on shelf "AF-213000" at controller A is now running at speed 9800 rpm              632 INFO     Mar 11 2022 17:25:07  12343 hardware    AF-213000  Fan "fan3a" on shelf "AF-213000" at controller A is now running at speed 9310 rpm              633 INFO     Mar 11 2022 17:25:07  12343 hardware    AF-213000  Fan "fan3b" on shelf "AF-213000" at controller A is now running at speed 9898 rpm              634 INFO     Mar 11 2022 17:25:07  12356 hardware    AF-213000  Temperature sensor "inlet-temp" on shelf "AF-213000" at inlet temp of controller A is operational: current temperature is 37 C        638 INFO     Mar 11 2022 17:25:07  12343 hardware    AF-213000  Fan "fan1a" on shelf "AF-213000" at controller A is now running at speed 9408 rpm              639 INFO     Mar 11 2022 17:25:07  12343 hardware    AF-213000  Fan "fan1b" on shelf "AF-213000" at controller A is now running at speed 9800 rpm              635 INFO     Mar 11 2022 17:25:08  12356 hardware    AF-213000  Temperature sensor "outlet-temp" on shelf "AF-213000" at outlet temp of controller A is operational: current temperature is 49 C        636 INFO     Mar 11 2022 17:25:08  12356 hardware    AF-213000  Temperature sensor "LeftEar" on shelf "AF-213000" at left-side ambient of controller A is operational: current temperature is 25 C        637 INFO     Mar 11 2022 17:25:08  12356 hardware    AF-213000  Temperature sensor "RightEar" on shelf "AF-213000" at right-side ambient of controller B is operational: current temperature is 24 C        640 INFO     Mar 11 2022 17:25:08  12348 hardware    AF-213000  Power supply "power-supply1" on shelf "AF-213000" at lower right rear is OK                641 INFO     Mar 11 2022 17:25:08  12348 hardware    AF-213000  Power supply "power-supply2" on shelf "AF-213000" at upper right rear is OK                665 INFO     Mar 11 2022 17:25:40  12356 hardware    AF-213000  Temperature sensor "LeftEar" on shelf "AF-213000" at left-side ambient of controller A is operational: current temperature is 24 C        666 INFO     Mar 11 2022 17:25:40  12356 hardware    AF-213000  Temperature sensor "RightEar" on shelf "AF-213000" at right-side ambient of controller B is operational: current temperature is 23 C        667 INFO     Mar 11 2022 17:25:40  12348 hardware    AF-213000  Power supply "power-supply1" on shelf "AF-213000" at lower right rear is OK                668 INFO     Mar 11 2022 17:25:40  12348 hardware    AF-213000  Power supply "power-supply2" on shelf "AF-213000" at upper right rear is OK                642 INFO     Mar 11 2022 17:25:46  12354 hardware    AF-213000  System temperature is now within range: current temperature for sensor "inlet-temp" on shelf "AF-213000" at inlet temp of controller A is 37 C        643 INFO     Mar 11 2022 17:25:46  12354 hardware    AF-213000  System temperature is now within range: current temperature for sensor "outlet-temp" on shelf "AF-213000" at outlet temp of controller A is 48 C        644 INFO     Mar 11 2022 17:25:46  12354 hardware    AF-213000  System temperature is now within range: current temperature for sensor "RightEar" on shelf "AF-213000" at right-side ambient of controller B is 23 C        645 INFO     Mar 11 2022 17:25:46  12354 hardware    AF-213000  System temperature is now within range: current temperature for sensor "LeftEar" on shelf "AF-213000" at left-side ambient of controller A is 24 C        669 INFO     Mar 11 2022 17:26:40  12354 hardware    AF-213000  System temperature is now within range: current temperature for sensor "RightEar" on shelf "AF-213000" at right-side ambient of controller B is 23 C        670 INFO     Mar 11 2022 17:26:40  12354 hardware    AF-213000  System temperature is now within range: current temperature for sensor "LeftEar" on shelf "AF-213000" at left-side ambient of controller A is 24 C        671 INFO     Mar 11 2022 17:26:40  12308 hardware    AF-213000  NVRAM Battery 0 is OK on controller B                        673 INFO     Mar 11 2022 17:28:40  14023 hardware    AF-213000  Temperature sensor  "outlet-temp" on shelf "AF-213000" at controller B is and below critical level Celsius and below critical level        646 INFO     Mar 11 2022 17:28:46  12308 hardware    AF-213000  NVRAM Battery 0 is OK on controller A                        647 INFO     Mar 11 2022 17:30:46  14023 hardware    AF-213000  Temperature sensor  "outlet-temp" on shelf "AF-213000" at controller A is and below critical level Celsius and below critical level        672 INFO     Mar 11 2022 17:31:40  14019 hardware    AF-213000  Ambient temperature on shelf AF-213000 is at 24 Celsius and is below warning level.              674 INFO     Mar 11 2022 17:34:40  14019 hardware    AF-213000  Ambient temperature on shelf AF-213000 is at 24 Celsius and is below critical level.              675 WARNING  Mar 11 2022 18:47:19   2109 service     -          Failed to send alert e-mail to SMTP server. Verify SMTP server and port configuration.              676 WARNING  Jun 20 2022 10:34:38  12226 hardware    AF-213000  NIC port eth0b is down on controller B                      677 WARNING  Jun 20 2022 10:34:38  12226 hardware    AF-213000  NIC port eth0a is down on controller B                      678 INFO     Jun 20 2022 10:36:08  12225 hardware    AF-213000  NIC port eth0a is up on controller B                        679 INFO     Jun 20 2022 10:36:08  12225 hardware    AF-213000  NIC port eth0b is up on controller B                        680 WARNING  Jun 20 2022 10:45:10   2109 service     -          Failed to send alert e-mail to SMTP server. Verify SMTP server and port configuration.              684 WARNING  Mar  3 2023 11:15:14  12226 hardware    AF-213000  NIC port eth0a is down on controller B                      683 WARNING  Mar  3 2023 11:15:15  12226 hardware    AF-213000  NIC port eth0b is down on controller B                      681 INFO     Mar  3 2023 11:15:21  12225 hardware    AF-213000  NIC port eth0a is up on controller B                        682 INFO     Mar  3 2023 11:15:21  12225 hardware    AF-213000  NIC port eth0b is up on controller B                        685 WARNING  Mar  3 2023 11:40:37   2109 service     -          Failed to send alert e-mail to SMTP server. Verify SMTP server and port configuration.              686 WARNING  Mar  3 2023 11:41:01  12226 hardware    AF-213000  NIC port eth0b is down on controller A                      687 INFO     Mar  3 2023 11:41:08  12225 hardware    AF-213000  NIC port eth0b is up on controller A                        688 NOTICE   Oct 12 2023 07:59:06   6025 update      AF-213000  Software update to version 5.2.1.1100-1027043-opt started on controller B                690 INFO     Oct 12 2023 07:59:13   6008 update      AF-213000  Unpacking version 5.2.1.1100-1027043-opt update package on controller B                692 INFO     Oct 12 2023 08:01:59   6009 update      AF-213000  Unpacked version 5.2.1.1100-1027043-opt update package on controller B                693 INFO     Oct 12 2023 08:02:02   6010 update      AF-213000  Reboot to 5.2.1.1100-1027043-opt on controller B              

Then the weekly reboot on controller B started on Nov

       979 INFO     Nov 11 2023 17:38:06  12201 hardware    AF-213000  IP interface 172.17.1.197 up on controller B NIC port eth0a                  977 WARNING  Nov 11 2023 17:38:10   2028 hardware    AF-213000  Controller B rebooted unexpectedly. Contact Nimble Storage Support.              978 INFO     Nov 11 2023 17:38:11   2101 service     AF-213000  System services started. Software version is 6.1.1.300-1028597-opt                980 INFO     Nov 11 2023 17:40:00  12343 hardware    AF-213000  Fan "fan1a" on shelf "AF-213000" at controller B is now running at speed 9310 rpm            981 INFO     Nov 11 2023 17:40:00  12343 hardware    AF-213000  Fan "fan1b" on shelf "AF-213000" at controller B is now running at speed 9506 rpm            982 INFO     Nov 11 2023 17:40:00  12343 hardware    AF-213000  Fan "fan2a" on shelf "AF-213000" at controller B is now running at speed 9212 rpm            983 INFO     Nov 11 2023 17:40:00  12343 hardware    AF-213000  Fan "fan2b" on shelf "AF-213000" at controller B is now running at speed 9800 rpm            984 INFO     Nov 11 2023 17:40:00  12343 hardware    AF-213000  Fan "fan3a" on shelf "AF-213000" at controller B is now running at speed 9310 rpm            985 INFO     Nov 11 2023 17:40:01  12343 hardware    AF-213000  Fan "fan3b" on shelf "AF-213000" at controller B is now running at speed 9506 rpm            986 INFO     Nov 11 2023 17:40:01  12356 hardware    AF-213000  Temperature sensor "inlet-temp" on shelf "AF-213000" at inlet temp of controller B is operational: current temperature is 37 C        987 INFO     Nov 11 2023 17:40:01  12356 hardware    AF-213000  Temperature sensor "outlet-temp" on shelf "AF-213000" at outlet temp of controller B is operational: current temperature is 49 C        988 INFO     Nov 11 2023 17:41:20  12354 hardware    AF-213000  System temperature is now within range: current temperature for sensor "inlet-temp" on shelf "AF-213000" at inlet temp of controller B is 37 C        989 INFO     Nov 11 2023 17:41:21  12354 hardware    AF-213000  System temperature is now within range: current temperature for sensor "outlet-temp" on shelf "AF-213000" at outlet temp of controller B is 49 C        990 INFO     Nov 11 2023 17:41:21  12356 hardware    AF-213000  Temperature sensor "LeftEar" on shelf "AF-213000" at left-side ambient of controller A is operational: current temperature is 24 C        991 INFO     Nov 11 2023 17:41:21  12356 hardware    AF-213000  Temperature sensor "RightEar" on shelf "AF-213000" at right-side ambient of controller B is operational: current temperature is 23 C        992 INFO     Nov 11 2023 17:41:21  12348 hardware    AF-213000  Power supply "power-supply1" on shelf "AF-213000" at lower right rear is OK              993 INFO     Nov 11 2023 17:41:21  12348 hardware    AF-213000  Power supply "power-supply2" on shelf "AF-213000" at upper right rear is OK              994 INFO     Nov 11 2023 17:42:20  12354 hardware    AF-213000  System temperature is now within range: current temperature for sensor "LeftEar" on shelf "AF-213000" at left-side ambient of controller A is 24 C        995 INFO     Nov 11 2023 17:42:21  12354 hardware    AF-213000  System temperature is now within range: current temperature for sensor "RightEar" on shelf "AF-213000" at right-side ambient of controller B is 23 C        996 INFO     Nov 11 2023 17:43:21  12359 hardware    AF-213000  NVRAM-0 Battery 0 is OK on controller B                    997 INFO     Nov 11 2023 17:47:20  14019 hardware    AF-213000  Ambient temperature on shelf AF-213000 is at 24 Celsius and is below critical level.            998 INFO     Nov 11 2023 17:47:21  14019 hardware    AF-213000  Ambient temperature on shelf AF-213000 is at 24 Celsius and is below warning level.    

[DaveOb]

The logs posted are really just alerts.Kinda of like asking a mechanic to diagnose a car fault by just seeing the check engine light.

Power loss or a hardware issue will present in the same way.Nimble support would be able to tell you.

An educated guess  its a hardware issue if its only effecting one controller repeatedly but without looking at actual logs its impossibe to say.