topic Re: facing problem while taking ignite backup in Operating System - HP-UX

facing problem while taking ignite backup

NEVATDR — Sat, 29 Aug 2009 00:45:33 GMT

./make _tape _recovery -Av -a /dev/rmt/2mn
* Creating local directories for configuration files and archive.

======= 08/28/09 23:31:49 IST Started ./make_tape_recovery. (Fri Aug 28 23:31:49 IST 2009)
@(#)Ignite-UX Revision C.7.8.201
@(#)ignite/net_recovery (opt) Revision: /branches/IUX_RA0903/ignite/src@76987 Last Modified: 2009-02-05 15:45:55
-0700 (Thu, 05 Feb 2009)

* Recovery Archive Description = Recovery Archive

* Recovery Archive Location = /dev/rmt/2mn

* Number of Archives to Save = 2

* Pax type = tar

In? dsk/vg name minor# Associated disks/mountpoints
2 v /dev/vg00 0x00 /dev/dsk/c2t0d0
/dev/vg00/lvol1 /stand 2
/dev/vg00/lvol2
/dev/vg00/lvol3 / 2
/dev/vg00/lvol4 /home 2
/dev/vg00/lvol5 /opt 2
/dev/vg00/lvol6 /tmp 2
/dev/vg00/lvol7 /usr 2
/dev/vg00/lvol8 /var 2
0 v /dev/vg01 0x01 /dev/dsk/c2t1d0
/dev/vg01/lvol12 /oracledb 0
/dev/vg01/lvol13 /oracleas 0
/dev/vg01/archive /archive 0
0 v /dev/vg02 0x02 /dev/dsk/c14t0d1 /dev/dsk/c12t0d1
/dev/vg02/lvol20 /oradata 0

** 0 - The Volume Group or Filesystem is Not included in the
System Recovery Archive
** 1 - The Volume Group or Filesystem is Partially included in the
System Recovery Archive
** 2 - The Volume Group or Filesystem is Fully included in the
System Recovery Archive

* Checking Versions of Ignite-UX filesets
* Scanning system for IO devices...
* Boot device is: 0/1/1/0.0.0
* Creating System Configuration.
* /opt/ignite/bin/save_config -f /var/opt/ignite/recovery/2009-08-28,23:31/system_cfg vg00
Pid 29548 killed due to text modification or page I/O error

Pid 29551 killed due to text modification or page I/O error

Pid 29599 killed due to text modification or page I/O error

Pid 29602 killed due to text modification or page I/O error

Pid 29650 killed due to text modification or page I/O error

Pid 29653 killed due to text modification or page I/O error

Pid 29701 killed due to text modification or page I/O error

Pid 29704 killed due to text modification or page I/O error

Pid 29752 killed due to text modification or page I/O error

Pid 29755 killed due to text modification or page I/O error

Pid 29803 killed due to text modification or page I/O error

Pid 29806 killed due to text modification or page I/O error

ERROR: "/var/opt/ignite/recovery/2009-08-28,23:31/system_cfg", line 71: Illegal character: "K"
ERROR: "/var/opt/ignite/recovery/2009-08-28,23:31/system_cfg", line 71: syntax error
ERROR: Problems were encountered while parsing config file: "/var/opt/ignite/recovery/2009-08-28,23:31/system_cfg".
instl_adm: Config file: "/var/opt/ignite/recovery/2009-08-28,23:31/system_cfg" cannot be parsed.
ERROR: System error, /opt/ignite/bin/instl_adm failed checking /var/opt/ignite/recovery/2009-08-28,23:31/system_cfg

======= 08/28/09 23:32:09 IST make_tape_recovery completed unsuccessfully

Re: facing problem while taking ignite backup

R.K. # — Sat, 29 Aug 2009 01:49:07 GMT

Hello,

To start with, can you check following o/p:
# swverify Ignite-UX

Regds..

Re: facing problem while taking ignite backup

TTr — Sat, 29 Aug 2009 12:21:46 GMT

> ERROR: "/var/opt/ignite/recovery/2009-08-28,23:31/system_cfg", line 71: Illegal character: "K"

Binary files being killed and the config file appears corrupt. Check the root disk and the disk where /var/opt is for errors. Most likely a disk is failing.

If the disk is OK, then in your attachment there are inconsistencies in the versions of your igniteUX filesets. You may want to swremove all ignite filesets and install the same (latest) version.

Re: facing problem while taking ignite backup

NEVATDR — Mon, 31 Aug 2009 04:20:36 GMT

# swverify Ignite-UX

======= 08/31/09 10:23:49 IST BEGIN swverify SESSION
(non-interactive) (jobid=vcmsar-0256)

* Session started for user "root@vcmsar".

* Beginning Selection
* Target connection succeeded for "vcmsar:/".
* Software selections:
Ignite-UX.BOOT-COMMON-PA,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.BOOT-KRN-11-11,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.BOOT-SERVICES,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.CD-TOOLS,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.CD-TOOLS-DOC,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.CD-TOOLS-SRC,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.DVD-TOOLS,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.DVD-TOOLS-DOC,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.DVD-TOOLS-SRC,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.FILE-SRV-11-11,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.IGNITE,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.IGNT-ENG-A-MAN,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.IGNT-JPN-E-MAN,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.IGNT-JPN-S-MAN,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.MGMT-TOOLS,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.RECOVERY,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
* Selection succeeded.

* Beginning Analysis
* Session selections have been saved in the file
"/.sw/sessions/swverify.last".
WARNING: "vcmsar:/": 82 files had warnings during this operation.
* Verification succeeded.

NOTE: More information may be found in the agent logfile using the
command "swjob -a log vcmsar-0256 @ vcmsar:/".

======= 08/31/09 10:23:56 IST END swverify SESSION (non-interactive)
(jobid=vcmsar-0256)

After that I again remove the ignite installer and then again install the depot after copying the file from other where the backup is working properly but still facing same problem . Please help me out and laso it is live server so I have execute the work asap.

Regards
DP

Re: facing problem while taking ignite backup

Hakki Aydin Ucar — Mon, 31 Aug 2009 05:08:15 GMT

you seem that you have latest version of ignite. After you installed ignite ,have you ever been faced this problem ? if so, I recommend first check system generally, in this case any disk error noticed ? and if your disk seem clean, you can try re-install ignite.

You can check these out by the way:

cd /var/opt/ignite/recovery/
ll
cd latest/
ll
wc -l flist < to see how many files in your archieve
and
grep '^[0-9][0-9]* oracledb/' flist < to see if really /oracledb be included in your archieve

Re: facing problem while taking ignite backup

Sudeesh — Mon, 31 Aug 2009 05:25:24 GMT

First impression is that some ignite files are currupt. I would recommend to remove the ignite files completely and re-install.

swlist|grep -i ignite

swremove ...

rm -rf /opt/ignite /var/opt/ignite
(keep a backup before you do this)

Install ignite sw again and try.

Re: facing problem while taking ignite backup

NEVATDR — Mon, 31 Aug 2009 06:33:29 GMT

Dear Sudeesh,
Thanks for your response I did accordingly ie. I remove the ignite folder contents for both the url and then re-install the ignite s/w now also it's status remains same as follows with less errors

# swverify Ignite-UX

======= 08/31/09 12:26:19 IST BEGIN swverify SESSION
(non-interactive) (jobid=vcmsar-0263)

* Session started for user "root@vcmsar".

* Beginning Selection
* Target connection succeeded for "vcmsar:/".
* Software selections:
Ignite-UX.BOOT-COMMON-PA,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.BOOT-KRN-11-11,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.BOOT-SERVICES,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.CD-TOOLS,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.CD-TOOLS-DOC,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.CD-TOOLS-SRC,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.DVD-TOOLS,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.DVD-TOOLS-DOC,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.DVD-TOOLS-SRC,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.FILE-SRV-11-11,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.IGNITE,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.IGNT-ENG-A-MAN,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.IGNT-JPN-E-MAN,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.IGNT-JPN-S-MAN,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.MGMT-TOOLS,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.RECOVERY,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
* Selection succeeded.

* Beginning Analysis
* Session selections have been saved in the file
"/.sw/sessions/swverify.last".
WARNING: "vcmsar:/": 21 files had warnings during this operation.
* Verification succeeded.

NOTE: More information may be found in the agent logfile using the
command "swjob -a log vcmsar-0263 @ vcmsar:/".

======= 08/31/09 12:26:24 IST END swverify SESSION (non-interactive)
(jobid=vcmsar-0263)

Re: facing problem while taking ignite backup

Fabio Ettore — Mon, 31 Aug 2009 08:15:32 GMT

Hi,

I think your issue is not regarding the product of Ignite-UX, so I believe it's useless to continue to reinstall it.

I suggest checking syslog.log to verify other possible indications of disk or general system problems.
Also last part of message suggests to check system_cfg file, check what it reports on line 71. Command instl_adm is good to verify system_cfg file (man instl_adm).

Another thing which is not directly about your issue. I suggest using make_tape_recovery -x inc_entire=vg00 -v -a /dev/rmt/2mn if you want to create a bootable tape. -A option has a different meaning between make_tape_recovery and make_recovery.

HTH.

Best regards,
Fabio

Re: facing problem while taking ignite backup

NEVATDR — Mon, 31 Aug 2009 09:24:49 GMT

Thanks all but still I am waiting for the positive response. Even after executing the following command the result remains the same.

make_tape_recovery -x inc_entire=vg00 -v -a /dev/rmt/2mn

Please help me out

Re: facing problem while taking ignite backup

Fabio Ettore — Mon, 31 Aug 2009 09:32:34 GMT

I think I need to explain better what I wrote in my answer.

First of all I suggested using make_tape_recovery -x inc_entire=vg00 -a /dev/rmt/2mn not to solve your main problem, it was just to say that -A option on make_tape_recovery has not the same behaviour of -A option on make_recovery.

About the main problem I suggest checking syslog.log for possible system problem and system_cfg file for syntax issues (do that by instl_adm command). Consider that message "killed due to text modification or page I/O error" does indicate something not directly about Ignite-UX.

I hope it's well explained now what I meant in my post.

Best regards,
Fabio

Re: facing problem while taking ignite backup

Turgay Cavdar — Mon, 31 Aug 2009 11:04:25 GMT

Any possibility for bad swap space? Can you check it with dd read from swap device?

Re: facing problem while taking ignite backup

TTr — Tue, 01 Sep 2009 11:07:18 GMT

> WARNING: "vcmsar:/": 82 files had warnings...
> WARNING: "vcmsar:/": 21 files had warnings...

You may start by looking at the NOTE below the two warnings for more information about which files had warnings. Two installations of the same (same?) ignite fileset gave two different numbers of warnings.

You never mentioned anything about checking the syslog for disk errors or actually checking the disk yourself for errors.

And also you never showed us what the illegal characters are in the "/var/opt/ignite/recovery/2009-08-28,23:31/system_cfg" file. It might give you and us some hints as to what happened.

Re: facing problem while taking ignite backup

NEVATDR — Tue, 01 Sep 2009 11:40:24 GMT

Thanks a lot to all for your continuous support I am attaching the syslog.log file for your reference and I have checked the contents it's showing some binding address problem which is due to one of the server on the same network is having changed to same hostname after first login then we have to change the hostname by giving command uname -S and hostname
I have already edited two files 1)/etc/hosts 2)/etc/rc.config.d/netconf but that seems to be not sufficient for setting the hostname

Regards
DP

Re: facing problem while taking ignite backup

TTr — Tue, 01 Sep 2009 11:53:41 GMT

So far you only retured one of the things I asked for. And the syslog is FULL of didk errors of at least two kinds. A fiber based disk is dissappearing and coming back and "Synchronous Page I/O error occurred while paging to/from disk". There is definitely something wrong in your server. You need to check your disks out.

It should be very easy to grep out all the bind error so that you are left with a clean syslog to look at
grep -v "inetd[840]: shell/tcp" /var/adm/syslog/syslog.log

Re: facing problem while taking ignite backup

NEVATDR — Tue, 01 Sep 2009 12:33:00 GMT

Please go through with the attachment file hope this will help you to see the contents more clear.

Re: facing problem while taking ignite backup

TTr — Tue, 01 Sep 2009 16:39:07 GMT

There are many errors here.

1. An important patch that can lead to a panic is missing.
2. This is probably atape drive.
> 0/4/1/0/4/0: Device at device id 0x610ee4 has disappeared from Name Server GPN_FT
3. I am not sure which device this is but you can look for a device with this minor number 0x40000003
> Synchronous Page I/O error occurred while paging to/from dis
> device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes

4. This is probably the /var filesystem
> msgcnt 3 vxfs: mesg 038: vx_dataioerr - /dev/vg00/lvol5 file system file data read error

There is a disk that either is failing or is giving out read errors. Most likely you need to recover the /var volume/filesystem and possibly the whole boot disk.

Re: facing problem while taking ignite backup

TTr — Tue, 01 Sep 2009 16:40:53 GMT

Run "grep vmunix /var/adm/syslog/syslog.log > /tmp/errors.syslog"
and read in detail the file /tmp/errors.syslog to see the errors.

Re: facing problem while taking ignite backup

NEVATDR — Wed, 02 Sep 2009 14:39:39 GMT

Thanks all for your continuous support now I checked all other servers on the network on which SAN disk is mounted having same message as below

# ./usr/sbin/dmesg

Sep 2 20:36
...
bytes not paged, page addr =3642000.30000, bflag =9f, bp = 5caa9800
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =1541400.30000, bflag =9f, bp = 5caa9800
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =e3af800.30000, bflag =9f, bp = 5caa9800
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =3095c00.30000, bflag =9f, bp = 5caa9800
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =cc84400.30000, bflag =9f, bp = 5caa9800
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =bd20800.30000, bflag =9f, bp = 5caa9800
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =4fd6800.30000, bflag =9f, bp = 5caa9800
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =6f2e000.30000, bflag =9f, bp = 48686080
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =8293c00.30000, bflag =9f, bp = 48686080
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =a725000.30000, bflag =9f, bp = 5b698340
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =a09e400.30000, bflag =9f, bp = 5b698340
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =2187c00.30000, bflag =9f, bp = 5b698340
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =a5ee400.30000, bflag =9f, bp = 5b698340
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =b3af000.30000, bflag =9f, bp = 5b698340
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =d50d000.30000, bflag =9f, bp = 5b698340
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =8b0f000.30000, bflag =9f, bp = 5b698340
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =d9bf400.30000, bflag =9f, bp = 5b698340
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =2490800.30000, bflag =9f, bp = 5b698340
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =9cdd800.30000, bflag =9f, bp = 5b698340
0/4/1/0/4/0: Device at device id 0x610ee4 has disappeared from Name Server GPN_FT
(FCP type) response, or its 'Port World-Wide Name' has changed.
device id = loop id, for private loop devices
device id = nport ID, for fabric/public-loop devices
System won't be able to see LUNs behind this port.

0/4/1/0/4/0: Device at device id 0x610ee4 is back in Name Server GPN_FT (FCP type)
response, and its 'Port World-Wide Name' remains the same as
original.
device id = loop id, for private loop devices
device id = nport ID, for fabric/public-loop devices
System will be able to see LUNs behind this port
(might need to run 'ioscan' first).

Also I am attaching the text file as asked. Please help me out

Re: facing problem while taking ignite backup

Torsten. — Wed, 02 Sep 2009 14:57:38 GMT

Your system has problem with disk at 0/4/1/0/4/0 - hardware problems?

Re: facing problem while taking ignite backup

Fabio Ettore — Wed, 02 Sep 2009 15:36:22 GMT

Hi,

you can see Page I/O error messages that you saw in Ignite-UX backup.
Definitely I think it's the reason of problem, suggest checking what's going on with that disk.

Best regards,
Fabio