Ignite problems (SOFTWARE FAILURE)

[Manuales]

Hi all:
I'm reviewing a new system with the following skill talking about ignites:

version: HP-UX B.11.00 U 9000/800
ignite into cron: 00 20 * * 4 /opt/ignite/bin/make_recovery -Av >/tmp/make_recovery.log 2>&1

QUESTION NUMBER 1:
i'd like to know if is correct the paremeters used with make_recovery command.

QUESTION NUMBER 2:
I was reviewing the tapes which it guess contains ignite with the following steps:

1.- shutdownSERVERNAME -R 0
2.- wait to message 10 seconds
3.- enter anyword
4.- and i arrived a menu named: MAIN MENU (see picture named main menu attached)
5.- i choosen SEA option
6.- its shows "Searching for potential boot device"
7.- there i verify the path # for sequential acess media (this is for tape)
8.- its shows:
Main menu: enter command or menu> bo p0
9.- interact with IPL (Y,N , or cancel): n
10.- and begin with the following:
Booting ...
Boot IO Dependen code (IODC) revision 1 ...
................................
................................
................................
................................

and is showed the attached picture named "ignite" ...

SYSTEM ALERT
DATE:
ALERT LEVER: 12 = Software failure
REASON FOR ALERT
SOURCE: 1 = processor
SOURCE DETAIL: 1 = processor general SOURCE ID: 0
PROBLEM DETAIL: 0 = no problem detail

LED State: Unexpected Reboot, Running non-OS code Non -critical error dtected
Check Chassis and Console logs for error messages ....

QUESTION 2:
Could you tell me why appears the following error: Sofware failure?

QUESTION 3:
What must i expect without error?

Thanks.
Manuales.

[Manuales]

Does anybody know something about it?

Thanks.

[Jaime Bolanos Rojas. · Accepted solution]

Manuales, unless I am mistaken it, you were to supposed to specified the tape device to run the command to make the ignite backup

ej.

#make_tape_recovery -Ava /dev/rmt/xmn -x inc_entire=vg00 -I


Regards,

Jaime.

[Manuales]

ok thanks for answer to question 1.

but, do you know what is the meaning to the message shown when i want boot the tape?

SYSTEM ALERT
DATE:
ALERT LEVER: 12 = Software failure
REASON FOR ALERT
SOURCE: 1 = processor
SOURCE DETAIL: 1 = processor general SOURCE ID: 0
PROBLEM DETAIL: 0 = no problem detail

Thanks ...

[Dave Hutton]

Its been awhile since I've actually made an ignite tape.

Did you use the command you posted above to make the tape? Like suggested above, if you don't specify the tape I doubt it wrote anything to the tape.

So could it be that there really isn't an image on the tape? You just have the ignite header information on the tape? I think theres quite a few steps in creating the tape.

[Jaime Bolanos Rojas.]

Dave is right, please do an ignite tape with the #make_tape_recovery -Ava /dev/rmt/xmn -x inc_entire=vg00 -I

Then reboot your machine again and see if you get the error this time.

Regards,

jaime.

[Manuales]

thanks friends, but ...

my problem right now is that one of the servers can not restart ...

i want started from a ignite (tape} because it appears that there are problems with operating system, but when i try to do this one using the tape wich in theory contains the ignite done with above command shown,

is shown the following message:
SYSTEM ALERT
DATE:
ALERT LEVER: 12 = Software failure
REASON FOR ALERT
SOURCE: 1 = processor
SOURCE DETAIL: 1 = processor general SOURCE ID: 0
PROBLEM DETAIL: 0 = no problem detail

I'd like to know why appears that message?

am i doing in a bad way the ignite?
(/opt/ignite/bin/make_recovery -Av >/tmp/make_recovery.log 2>&1)

why is that error: Software failure ?

Thanks.


Thanks.

[Dave Hutton]

I take it, its too late to see what is in your make_recovery.log. I'm guessing thats on the server you are trying to recreate. Any chance you can recover that file from tape to see if your make_recovery was actually working?

My *guess* is, your tape is blank.

[OldSchool]

in cron: /opt/ignite/bin/make_recovery -Av >/tmp/make_recovery.log 2>&1



OK,

1) make_recovery is waayy old, should be make_tape_recovery. You need to update your installed version of ignite. Make_recovery was supported on 10.20 and some versions on 11.0 as well. According to the docs on the web "-A" includes the entire root volume group and "-v" specifies verbose messages, so those are good. Since you did not use -d , the tape defaults to /dev/rmt/0mn. Is this the correct device for your tape drive? If not, the cron job failed and you don't have anything on the backup.

2) Will the system boot from the normal boot device at all?

3) Was the ignite tape perhaps created after you started having problems with the system?

4) You didn't specify, but I assume you are trying to Ignite the same server that created the tape. If not, and they boxes are different models, you may not be able to use that tape.

[Lynn Hsu_2]

Assume your ignite tape is good...
I had same "software failure" error and
it turned out was GSP firmware glitch. Reset
the GSP helped. Either press the GSP bottom
or SW reset via management port.

Lynn

[Manuales]

ANSWERING QUESTIONS:
2) Will the system boot from the normal boot device at all?
No.

3) Was the ignite tape perhaps created after you started having problems with the system?
I do not know, how can i know if there were problems during ignite creation?

4) You didn't specify, but I assume you are trying to Ignite the same server that created the tape. If not, and they boxes are different models, you may not be able to use that tape.
No, i'm trying to start the server with an ingnite done there.

Lynn:
do i need the ignite anyway?
..
after doing you told me above, what do i need to do? only start server?

Thanks.

[AGUSTIN LIZARRAGA]

El mensaje de SW failure es genérico siendo la más probable causa la definición del path utilizado para el boot o bien la imposibilidad de realizar el boot de éste.

-un principio será validar la integridad de la cinta -se anexa procedimiento.


Ignite: verifying a good make_tape_recovery tape
PROBLEM
How can I verify that my Ignite make_tape_recovery tape is good?
CONFIGURATION
Operating System - HP-UX
Application - Ignite/UX

RESOLUTION
Here are the steps to follow to verify whether your Ignite make_recovery
tape is good:

1. Check the log generated during the Ignite backup:

/var/opt/ignite/recovery/latest/recovery.log

2. Verify whether the tape is bootable:

Boot from the tape and interrupt the boot process before
non-interactive recovery commences, and then abort the install.

OR

Extract the boot area (LIF) from the tape in to /tmp/bootimage file:

# copy_boot_tape -u /dev/rmt/0mn -b -d /tmp >/tmp/lif.out 2>&1

NOTE: Be sure to use a non-rewinding tape device file.

After copy_boot_tape command completes select 'E' to exit at
"[C]opy to new tape or [E]xit:" prompt.

Check /tmp/lif.out file for LIF filenames sizes.
Example list of filenames:
ISL AUTO INDEX CONFIG HPUX
INSTALL INSTALLFS FWWKAR INSTCMDS SYSCMDS
SCRIPTS

3. The second image on the tape is the tar archive of the OS and other
files.

a) Skip over the first image with 'mt' command using the
appropriate tape device file:

mt -t /dev/rmt/0mn rew
mt -t /dev/rmt/0mn fsf 1

b) Verify the contents of the tar image with the tar
command using the appropriate device file:

tar tvf /dev/rmt/0m

NOTE: This can take more then 30 minutes depending on the size of
the archive.

[Manuales]

Good afternoon Agustin .... :)
thanks a lot .....!!!!!!

I will verify the tape in other server ...
I will notice you what is happening ..

Thanks.
Manuales.

[Manuales]

Hi Agustin:
This is the out of the file named /var/opt/ignite/recovery/logs/

********
Started - Thu Jul 6 20:00:00 2006
********
/opt/ignite/bin/make_recovery -A -v
/usr/sbin/vgdisplay -v "vg00"
/usr/sbin/vgdisplay -v "vg00"
/usr/sbin/vgdisplay -v "vg01"
/usr/sbin/vgdisplay -v "vgboltons1"
/usr/sbin/vgdisplay -v
"/opt/ignite/bin/save_config" -f "/var/opt/ignite/recovery/config.recover" "vg00" >> "/var/
opt/ignite/logs/makrec.log1" 2>&1
lssf: Must specify a special_file
usage: lssf special_file ...
save_config: error - unknown disk type for , not SCSI or HPFL
save_config: error - cannot determine root disk

********
Ended Unsuccessfully - Thu Jul 6 20:04:26 2006
********

I can see that has been done in a bad way ...
why?
what must i do to do it well?

Thanks.

[AGUSTIN LIZARRAGA]

Demos un paso atrás y aseguremos la inexistencia de alguna inconsistencia reflejada por el uso de los comandos:

lvlnboot
strings /etc/lvmtab
vgdisplay -v

+Intenta realizar un preview de la creación del lvmtab con vgscan -p ; siempre es buena idea respaldar el lvmtab: e.g. cp -p /etc/lvmtab /var/tmp/lvmtab.save
+ Si decides recrear el lvmtab muévelo y procede con el "vgscan -v"

Saludos.

Agustín.

[AGUSTIN LIZARRAGA]

Hola Daniela, saludos!

Hablemos del Servidor A
El mensaje de Software failure debe probar total ausencia cuando intentes realizar el boot desde otra media: e.g. CDROM Install Media. o bien en el BCH deberás intentar realizar el boot de aquel dispositivo que te indique tener IPL cargado.i.e. BCH> sea ipl

Ahora del servidor B que tiene problemas para generar la cinta de ignite con los mensajes:
save_config: error - cannot determine root disk

De los mensajes la línea de arriba es la que me parece más significativa y promueve que resuelvas primero el problema por el que los mensajes en el dmesg confirman tienes ahora -"un disco del vg00 está probablemente dañado"; a continuación la salida del vg00 nos lo dice:

Jul 13 13:06:59 torpfop vmunix: SCSI: Abort abandoned -- lbolt: 15557, dev: 1f026000, io_id: 20006a6, status: 200

LV Name /dev/vg00/lvol1
LV Status available/stale

LV Name /dev/vg00/lvol2
LV Status available/stale

LV Name /dev/vg00/lvol3
LV Status available/stale


LV Name /dev/vg00/lvol4
LV Status available/stale


LV Name /dev/vg00/lvol5
LV Status available/stale


LV Name /dev/vg00/lvol6
LV Status available/stale


LV Name /dev/vg00/lvol7
LV Status available/stale


LV Name /dev/vg00/lvol8
LV Status available/stale


LV Name /dev/vg00/lvol9
LV Status available/stale


--- Physical volumes ---
PV Name /dev/dsk/c1t6d0
PV Status available
Total PE 4340
Free PE 0
Autoswitch On

PV Name /dev/dsk/c2t6d0
PV Status unavailable
Total PE 4340
Free PE 0
Autoswitch On

A menos que la previsión de la etiqueta de arranque en cada uno de los discos ( en el que queda disponible) utilice la opción -lq ; el equipo deberá levantar sin problemas a menos que se manifieste otra situación actualmente inadvertida -e.g. modificaciones al sistema operativo (parámetros del kernel -dinámicos, otros).

Opciones:
1) Sacar el disco que no responde y darle unos leves (muy leves) golpes con la palma de la mano en su costado esperar 10 segundos e insertarlo de nuevo;
Verificar que realice el spin-up.
Realizar un ioscan y si lo ve satisfactoriamente activar el vg00 nuevamente
La salida del vgdisplay deberá indicar que ambos discos ahora estan disponibles.
Si todo va como aquí se describe será necesario iniciar el vgsync del vg00.
Cuando veas que todos los lvols están en syncd (not stale) ; es momento que vuelvas a lanzar la creación de la cinta de ignite.

2) Levantar en modo de mantenimiento para remover los extends asociados al dispositivo que actualmente ya no responde. si para esto no hay fecha próxima para su reemplazo.
lvreduce -k -m 0 /dev/vg00/
Una vez hecho lo anterior para todos y cada uno de los lvols en mirror del vg00
Aquí es imperativo que en el server no se active el vg00 -es necesario un reboot.
NOTA: Ante cualquier situación no contemplada e inesperada en este procedimiento, su ejercicio conlleva un riesgo de que el equipo no levante. Por lo que lo más recomendable es proceder con el reemplazo del disco en primer lugar y la restauración de las estructuras asociadas.

P.S. ¿tiene contrato el equipo?

Saludos.

Agustín.

[AGUSTIN LIZARRAGA]

Veamos a continuación un ejemplo de la ejecución del comando "lvreduce -k -m 0 /dev/vg00/

NOTA: Esto lo hice estando en multiuser....


# lvdisplay -v /dev/vg00/lvol2 | more
--- Logical volumes ---
LV Name /dev/vg00/lvol2
VG Name /dev/vg00
LV Permission read/write
LV Status available/stale
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 2560
Current LE 640
Allocated PE 1280
Stripes 0
Stripe Size (Kbytes) 0
Bad block off
Allocation strict/contiguous
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c1t2d0 640 640
/dev/dsk/c2t2d0 640 640

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c1t2d0 00075 current /dev/dsk/c2t2d0 00075 stale
00001 /dev/dsk/c1t2d0 00076 current /dev/dsk/c2t2d0 00076 stale
00002 /dev/dsk/c1t2d0 00077 current /dev/dsk/c2t2d0 00077 stale


# lvreduce -k -m 0 /dev/vg00/lvol2
Logical volume "/dev/vg00/lvol2" has been successfully reduced.
Unable to query the I/O interface: I/O tree node does not exist.
lvlnboot: Logical Volume has no extents.
vgcfgbackup: Unable to read the physical volume.: No such device or address
vgcfgbackup: Invalid LVMREC on Physical Volume /dev/rdsk/c2t2d0


# lvdisplay -v /dev/vg00/lvol2 | more
--- Logical volumes ---
LV Name /dev/vg00/lvol2
VG Name /dev/vg00
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 2560
Current LE 640
Allocated PE 640
Stripes 0
Stripe Size (Kbytes) 0
Bad block off
Allocation strict/contiguous
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c1t2d0 640 640

--- Logical extents ---
LE PV1 PE1 Status 1
00000 /dev/dsk/c1t2d0 00075 current
00001 /dev/dsk/c1t2d0 00076 current
00002 /dev/dsk/c1t2d0 00077 current
00003 /dev/dsk/c1t2d0 00078 current
#