System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

Memory dump during server boot up

SOLVED
Go to solution
Satish Kumar_5
Regular Advisor

Memory dump during server boot up

Dear All,

 

I have one Rp 3440 with HPUX 11iV2. It gives the error during boot up of file system corruption (i am attaching the entire console output for your refference). I have below question:

 

1. Is there any way we can repair the file system? HOw with steps?

2. If not then shall i reformat the entire server or a update/upgrade with fix it?

 

Any other suggestion is welcome. One more thing, i have another same server whose helth LED showing amber but i checked all logs (syslog, boot msg) but found nothing. How to go about this? Any way we can know why its amber?

 

Thanks a lot for the help..

10 REPLIES
Bill Hassell
Honored Contributor

Re: Memory dump during server boot up

Your boot disk has failed. The relevant messages are:

Checking root file system.
vxfs fsck: file system had I/O error(s) on meta-data.
log replay in progress
log replay failed to clean file system
file system is not clean, full fsck required
pass0 - checking structural files
pass1 - checking inode sanity and blocks
vxfs fsck: fsck read failure bno = 22544, off = 0, len = 0
file system check failure, aborting ...

The I/O errors and fsck read failure cannot be fixed because the disk has gone bad.

 

It appears that the disk may not have been mirrored so you'll have to restore from your last Ignite backup.

 

...amber light

 

This indicates that a hardware error condition has been detected. You must connect to the console (HP-UX cannot access the MP--maintenance processor). If you setup the MP with a LAN cable and IP, you can login and read the logs with SL and then E

 

If you have no LAN setup, you'll have to connect a serial terminal (or PC with serial port and terminal emulator).



Bill Hassell, sysadmin
Satish Kumar_5
Regular Advisor

Re: Memory dump during server boot up

Hi Bill,

 

Thank you so much for the reply. I didnt have any ignite backup, so in that case i think i have to re-install the OS from DVD but tell me one thing, if the disk has failed how come it going tell fsck? Also i didnt have any error in front of HDD, so still you think its disk failure?

 

On Amber--

 

Would i have to restart for checking this to reach the MP console? Please advise accordingly as i cant restrt this server..

 

Thanks..

Dennis Handly
Acclaimed Contributor

Re: Memory dump during server boot up

>if the disk has failed how come it going tell fsck? Also I didn't have any error in front of HDD, so still you think it's disk failure?

 

If you start getting I/O errors to the disk, it's time to replace it.

Bill Hassell
Honored Contributor

Re: Memory dump during server boot up

fsck has reported the I/O error. fsck cannot repair a bad disk. It's sole purpose is to repair bookweeping errors in the filesystem structures. Disks have hundreds of ways to fail, electronic or physical damage. Sometimes they simply stop talking, but not always. And the error LED in front is kust another piece of electronics...sometimes it glows red or yellow and sometimes not.  The errors would show in syslog.log but since you cannot bootup, there is no way to look at the logs.

 

So replace the boot disk. Then if you have only one boot disk, order another disk immediately and if you have the disk mirroring software installed, reinstall the OS from your DVD and immediately mirror the boot disk -- and test the ability to boot from it. Only then should you load all the current patches, then applications and database code.

 

Finally, Ignite backups are mandatory for production. Whether there is a disk failure in the future or a bad sysadmin error, you need to be able to restore the system quickly. Reinstalling from the DVD means days of downtime. An ignite restore would take just a couple of hours to recover.

 

---

 

The MP is a separate computer. You connect to the console, type ctrl-B and login to the MP. Once you are logged in, type the SL and E commands to look at the logs. These will be hardware logs and not very easy to understand. But once you read the first log, the amber light will go out. This has no effect on the HP-UX server.

 

All of your production systems should have remote console connections. This is not HP-UX, it is the hardware console. In the Intel world, this is commonly called the iLO port. If you don't have the remote console LAN connected, you will have to walk into the computer room, locate a terminal, the proper cables and hook up to the serial console port. This can mean multiple hours of delay when you need to use the console.



Bill Hassell, sysadmin
Satish Kumar_5
Regular Advisor

Re: Memory dump during server boot up

Hi Bill,

 

Thanks for the help. It seems some bad blocks are cropped up in my boot disk as i ran "Expert recovery Shell" and tried to fix the corrupted volume but it through blow error and didnt fix even with "-o full" option.

 

vxfs fsck: V-3-25432: fsck read failure devid = 0

 

So then i tried reinstalling 11iv2 from media i have and it through error during same. Suspecting that my drive/media may have to some issue, i tried with 11iv1 dvd to narror down but there it dropped me to ISL prompt without teling what to do. Then i thought may be its not allowing to downgrade and i tried to isntall 11iv3 and it installed. But at last i need to install 11iv2 only. So can you tell me with steps what i have to do to downgrade it back  to 11iv2. Also is there any website i can download 11iv2 media?

 

Thanks again..

Dennis Handly
Acclaimed Contributor

Re: Memory dump during server boot up

>is there any website I can download 11.23 media?

 

I don't think so.  I think e-Delivery is only for 11.31:

https://h30406.www3.hp.com/campaigns/2009/promo/1-6MU53/index.php

Satish Kumar_5
Regular Advisor

Re: Memory dump during server boot up

Hi Bill,

 

Thanks for same. Can you tell me how can i get my HP-UX access from xmanager? I tried it but it didnt connect.. below is list of s/w instaled. please advise if something is missed..

 


# swlist X11
# Initializing...
# Contacting target "erp21"...
#
# Target:  erp21:/
#

# X11                                   B.11.31        HP-UX X Window Software
  X11.MKFONTDIR                         B.11.31        Utility for making fonts
  X11.MKFONTDIR-COM                     B.11.31        Utility for making fonts
  X11.MOTIF-SHLIB                       B.11.31        Motif PA shared libraries
  X11.X11-FONT-AUX                      B.11.31        Fonts for non-minimal runtime system
  X11.X11-FONT-MIN                      B.11.31        Fonts for minimal runtime system
  X11.X11-FONTSRV                       B.11.31        X11 Font Server
  X11.X11-FONTSRV-CM                    B.11.31        X11 Font Server
  X11.X11-FRE-I-MSG                     B.11.31        French ISO message catalogs for X11
  X11.X11-GER-I-MSG                     B.11.31        German ISO message catalogs for X11
  X11.X11-ITA-I-MSG                     B.11.31        Italian ISO message catalogs for X11
  X11.X11-JPN-E-MSG                     B.11.31        Japanese EUC message catalogs for X11
  X11.X11-JPN-S-MSG                     B.11.31        Japanese SJIS message catalogs for X11
  X11.X11-KOR-E-MSG                     B.11.31        Korean EUC message catalogs for X11
  X11.X11-RUN                           B.11.31        X11 clients for minimal runtime system
  X11.X11-RUN-CL                        B.11.31        X11 clients for client-side environment
  X11.X11-RUN-CL-COM                    B.11.31        X11 clients for client-side environment
  X11.X11-RUN-CL-MAN                    B.11.31        Online manual pages for X11 clients
  X11.X11-RUN-COM                       B.11.31        X11 clients for minimal runtime system
  X11.X11-RUN-CT-MAN                    B.11.31        Online manual pages for X11 contrib clients
  X11.X11-RUN-CTRB                      B.11.31        Contributed X11 clients for runtime system
  X11.X11-RUN-MAN                       B.11.31        Online manual pages for X11 clients
  X11.X11-SCH-H-MSG                     B.11.31        S-Chinese HP15 message catalogs for X11
  X11.X11-SPA-I-MSG                     B.11.31        Spanish ISO message catalogs for X11
  X11.X11-SWE-I-MSG                     B.11.31        Swedish ISO message catalogs for X11
  X11.X11-TCH-B-MSG                     B.11.31        T-Chinese Big5 message catalogs for X11
  X11.X11-TCH-E-MSG                     B.11.31        T-Chinese EUC message catalogs for X11
  X11.X11R4-SHLIBS                      B.11.31        X11R4 and Motif 1.1 shared libraries
  X11.X11R5-SHLIBS                      B.11.31        X11R5 shared libraries
  X11.X11R6-SHLIBS                      B.11.31        X11R6 PA shared libraries
# swlist CDE
# Initializing...
# Contacting target "erp21"...
#
# Target:  erp21:/
#

# CDE                           B.11.31        HP-UX CDE User Interface
  CDE.CDE-SHLIBS                B.11.31        CDE Shared Libraries

 

Regs,

Satish Kumar_5
Regular Advisor

Re: Memory dump during server boot up

Hi Bill,

 

Thanks i got the xmanager to get the server. the required s/w not installed as u will see from swlist output. Now 1 things only, how can we downgrade the OS to 11iv2 back. Please help what needs to be done to do the same with steps if any..

 

As the same HDD now working, i hope there is no drive issue and we had just file system corruption..

 

Regs,

Bill Hassell
Honored Contributor
Solution

Re: Memory dump during server boot up

There is no downgrade possibility. Just install directly from the DVD. The install will erase everything on the boot disk. I would never use Xwindows to perform installations -- it far too complicated to get working and is useless if HP-UX is not running correctly with correct networking and CDE running. Do you have a copy of PuTTY that ou can use for installation?



Bill Hassell, sysadmin
Satish Kumar_5
Regular Advisor

Re: Memory dump during server boot up

Thanks Bill.. i got it working

 

Thanks for your time and help..