facing problem while taking ignite backup

[NEVATDR]

./make _tape _recovery -Av -a /dev/rmt/2mn
* Creating local directories for configuration files and archive.

======= 08/28/09 23:31:49 IST Started ./make_tape_recovery. (Fri Aug 28 23:31:49 IST 2009)
@(#)Ignite-UX Revision C.7.8.201
@(#)ignite/net_recovery (opt) Revision: /branches/IUX_RA0903/ignite/src@76987 Last Modified: 2009-02-05 15:45:55
-0700 (Thu, 05 Feb 2009)

* Recovery Archive Description = Recovery Archive

* Recovery Archive Location = /dev/rmt/2mn

* Number of Archives to Save = 2

* Pax type = tar


In? dsk/vg name minor# Associated disks/mountpoints
2 v /dev/vg00 0x00 /dev/dsk/c2t0d0
/dev/vg00/lvol1 /stand 2
/dev/vg00/lvol2
/dev/vg00/lvol3 / 2
/dev/vg00/lvol4 /home 2
/dev/vg00/lvol5 /opt 2
/dev/vg00/lvol6 /tmp 2
/dev/vg00/lvol7 /usr 2
/dev/vg00/lvol8 /var 2
0 v /dev/vg01 0x01 /dev/dsk/c2t1d0
/dev/vg01/lvol12 /oracledb 0
/dev/vg01/lvol13 /oracleas 0
/dev/vg01/archive /archive 0
0 v /dev/vg02 0x02 /dev/dsk/c14t0d1 /dev/dsk/c12t0d1
/dev/vg02/lvol20 /oradata 0

** 0 - The Volume Group or Filesystem is Not included in the
System Recovery Archive
** 1 - The Volume Group or Filesystem is Partially included in the
System Recovery Archive
** 2 - The Volume Group or Filesystem is Fully included in the
System Recovery Archive

* Checking Versions of Ignite-UX filesets
* Scanning system for IO devices...
* Boot device is: 0/1/1/0.0.0
* Creating System Configuration.
* /opt/ignite/bin/save_config -f /var/opt/ignite/recovery/2009-08-28,23:31/system_cfg vg00
Pid 29548 killed due to text modification or page I/O error

Pid 29551 killed due to text modification or page I/O error

Pid 29599 killed due to text modification or page I/O error

Pid 29602 killed due to text modification or page I/O error

Pid 29650 killed due to text modification or page I/O error

Pid 29653 killed due to text modification or page I/O error

Pid 29701 killed due to text modification or page I/O error

Pid 29704 killed due to text modification or page I/O error

Pid 29752 killed due to text modification or page I/O error

Pid 29755 killed due to text modification or page I/O error

Pid 29803 killed due to text modification or page I/O error

Pid 29806 killed due to text modification or page I/O error

ERROR: "/var/opt/ignite/recovery/2009-08-28,23:31/system_cfg", line 71: Illegal character: "K"
ERROR: "/var/opt/ignite/recovery/2009-08-28,23:31/system_cfg", line 71: syntax error
ERROR: Problems were encountered while parsing config file: "/var/opt/ignite/recovery/2009-08-28,23:31/system_cfg".
instl_adm: Config file: "/var/opt/ignite/recovery/2009-08-28,23:31/system_cfg" cannot be parsed.
ERROR: System error, /opt/ignite/bin/instl_adm failed checking /var/opt/ignite/recovery/2009-08-28,23:31/system_cfg

======= 08/28/09 23:32:09 IST make_tape_recovery completed unsuccessfully

[R.K. #]

Hello,

To start with, can you check following o/p:
# swverify Ignite-UX

Regds..

[TTr]

> ERROR: "/var/opt/ignite/recovery/2009-08-28,23:31/system_cfg", line 71: Illegal character: "K"

Binary files being killed and the config file appears corrupt. Check the root disk and the disk where /var/opt is for errors. Most likely a disk is failing.

If the disk is OK, then in your attachment there are inconsistencies in the versions of your igniteUX filesets. You may want to swremove all ignite filesets and install the same (latest) version.

[NEVATDR]

# swverify Ignite-UX

======= 08/31/09 10:23:49 IST BEGIN swverify SESSION
(non-interactive) (jobid=vcmsar-0256)

* Session started for user "root@vcmsar".

* Beginning Selection
* Target connection succeeded for "vcmsar:/".
* Software selections:
Ignite-UX.BOOT-COMMON-PA,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.BOOT-KRN-11-11,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.BOOT-SERVICES,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.CD-TOOLS,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.CD-TOOLS-DOC,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.CD-TOOLS-SRC,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.DVD-TOOLS,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.DVD-TOOLS-DOC,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.DVD-TOOLS-SRC,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.FILE-SRV-11-11,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.IGNITE,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.IGNT-ENG-A-MAN,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.IGNT-JPN-E-MAN,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.IGNT-JPN-S-MAN,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.MGMT-TOOLS,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.RECOVERY,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
* Selection succeeded.


* Beginning Analysis
* Session selections have been saved in the file
"/.sw/sessions/swverify.last".
WARNING: "vcmsar:/": 82 files had warnings during this operation.
* Verification succeeded.


NOTE: More information may be found in the agent logfile using the
command "swjob -a log vcmsar-0256 @ vcmsar:/".

======= 08/31/09 10:23:56 IST END swverify SESSION (non-interactive)
(jobid=vcmsar-0256)

After that I again remove the ignite installer and then again install the depot after copying the file from other where the backup is working properly but still facing same problem . Please help me out and laso it is live server so I have execute the work asap.

Regards
DP

[Hakki Aydin Ucar]

you seem that you have latest version of ignite. After you installed ignite ,have you ever been faced this problem ? if so, I recommend first check system generally, in this case any disk error noticed ? and if your disk seem clean, you can try re-install ignite.

You can check these out by the way:

cd /var/opt/ignite/recovery/
ll
cd latest/
ll
wc -l flist < to see how many files in your archieve
and
grep '^[0-9][0-9]* oracledb/' flist < to see if really /oracledb be included in your archieve

[Sudeesh]

First impression is that some ignite files are currupt. I would recommend to remove the ignite files completely and re-install.

swlist|grep -i ignite

swremove ...

rm -rf /opt/ignite /var/opt/ignite
(keep a backup before you do this)

Install ignite sw again and try.

[NEVATDR]

Dear Sudeesh,
Thanks for your response I did accordingly ie. I remove the ignite folder contents for both the url and then re-install the ignite s/w now also it's status remains same as follows with less errors

# swverify Ignite-UX

======= 08/31/09 12:26:19 IST BEGIN swverify SESSION
(non-interactive) (jobid=vcmsar-0263)

* Session started for user "root@vcmsar".

* Beginning Selection
* Target connection succeeded for "vcmsar:/".
* Software selections:
Ignite-UX.BOOT-COMMON-PA,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.BOOT-KRN-11-11,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.BOOT-SERVICES,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.CD-TOOLS,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.CD-TOOLS-DOC,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.CD-TOOLS-SRC,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.DVD-TOOLS,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.DVD-TOOLS-DOC,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.DVD-TOOLS-SRC,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.FILE-SRV-11-11,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.IGNITE,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.IGNT-ENG-A-MAN,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.IGNT-JPN-E-MAN,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.IGNT-JPN-S-MAN,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.MGMT-TOOLS,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
Ignite-UX.RECOVERY,l=/,r=C.7.8.201,a=HP-UX_B.11.11_32/64,v=HP,fr=C.7.8.201,fa=HP-UX_B.11.11_32/64
* Selection succeeded.


* Beginning Analysis
* Session selections have been saved in the file
"/.sw/sessions/swverify.last".
WARNING: "vcmsar:/": 21 files had warnings during this operation.
* Verification succeeded.


NOTE: More information may be found in the agent logfile using the
command "swjob -a log vcmsar-0263 @ vcmsar:/".

======= 08/31/09 12:26:24 IST END swverify SESSION (non-interactive)
(jobid=vcmsar-0263)

[Fabio Ettore]

Hi,

I think your issue is not regarding the product of Ignite-UX, so I believe it's useless to continue to reinstall it.

I suggest checking syslog.log to verify other possible indications of disk or general system problems.
Also last part of message suggests to check system_cfg file, check what it reports on line 71. Command instl_adm is good to verify system_cfg file (man instl_adm).

Another thing which is not directly about your issue. I suggest using make_tape_recovery -x inc_entire=vg00 -v -a /dev/rmt/2mn if you want to create a bootable tape. -A option has a different meaning between make_tape_recovery and make_recovery.

HTH.

Best regards,
Fabio

[NEVATDR]

Thanks all but still I am waiting for the positive response. Even after executing the following command the result remains the same.

make_tape_recovery -x inc_entire=vg00 -v -a /dev/rmt/2mn

Please help me out

[Fabio Ettore]

I think I need to explain better what I wrote in my answer.

First of all I suggested using make_tape_recovery -x inc_entire=vg00 -a /dev/rmt/2mn not to solve your main problem, it was just to say that -A option on make_tape_recovery has not the same behaviour of -A option on make_recovery.

About the main problem I suggest checking syslog.log for possible system problem and system_cfg file for syntax issues (do that by instl_adm command). Consider that message "killed due to text modification or page I/O error" does indicate something not directly about Ignite-UX.

I hope it's well explained now what I meant in my post.

Best regards,
Fabio

[Turgay Cavdar]

Any possibility for bad swap space? Can you check it with dd read from swap device?

[TTr]

> WARNING: "vcmsar:/": 82 files had warnings...
> WARNING: "vcmsar:/": 21 files had warnings...

You may start by looking at the NOTE below the two warnings for more information about which files had warnings. Two installations of the same (same?) ignite fileset gave two different numbers of warnings.

You never mentioned anything about checking the syslog for disk errors or actually checking the disk yourself for errors.

And also you never showed us what the illegal characters are in the "/var/opt/ignite/recovery/2009-08-28,23:31/system_cfg" file. It might give you and us some hints as to what happened.

[NEVATDR]

Thanks a lot to all for your continuous support I am attaching the syslog.log file for your reference and I have checked the contents it's showing some binding address problem which is due to one of the server on the same network is having changed to same hostname after first login then we have to change the hostname by giving command uname -S and hostname
I have already edited two files 1)/etc/hosts 2)/etc/rc.config.d/netconf but that seems to be not sufficient for setting the hostname

Regards
DP

[TTr]

So far you only retured one of the things I asked for. And the syslog is FULL of didk errors of at least two kinds. A fiber based disk is dissappearing and coming back and "Synchronous Page I/O error occurred while paging to/from disk". There is definitely something wrong in your server. You need to check your disks out.

It should be very easy to grep out all the bind error so that you are left with a clean syslog to look at
grep -v "inetd[840]: shell/tcp" /var/adm/syslog/syslog.log

[NEVATDR]

Please go through with the attachment file hope this will help you to see the contents more clear.

[TTr]

There are many errors here.

1. An important patch that can lead to a panic is missing.
2. This is probably atape drive.
> 0/4/1/0/4/0: Device at device id 0x610ee4 has disappeared from Name Server GPN_FT
3. I am not sure which device this is but you can look for a device with this minor number 0x40000003
> Synchronous Page I/O error occurred while paging to/from dis
> device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes

4. This is probably the /var filesystem
> msgcnt 3 vxfs: mesg 038: vx_dataioerr - /dev/vg00/lvol5 file system file data read error

There is a disk that either is failing or is giving out read errors. Most likely you need to recover the /var volume/filesystem and possibly the whole boot disk.

[TTr]

Run "grep vmunix /var/adm/syslog/syslog.log > /tmp/errors.syslog"
and read in detail the file /tmp/errors.syslog to see the errors.

[NEVATDR]

Thanks all for your continuous support now I checked all other servers on the network on which SAN disk is mounted having same message as below

# ./usr/sbin/dmesg

Sep 2 20:36
...
bytes not paged, page addr =3642000.30000, bflag =9f, bp = 5caa9800
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =1541400.30000, bflag =9f, bp = 5caa9800
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =e3af800.30000, bflag =9f, bp = 5caa9800
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =3095c00.30000, bflag =9f, bp = 5caa9800
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =cc84400.30000, bflag =9f, bp = 5caa9800
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =bd20800.30000, bflag =9f, bp = 5caa9800
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =4fd6800.30000, bflag =9f, bp = 5caa9800
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =6f2e000.30000, bflag =9f, bp = 48686080
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =8293c00.30000, bflag =9f, bp = 48686080
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =a725000.30000, bflag =9f, bp = 5b698340
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =a09e400.30000, bflag =9f, bp = 5b698340
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =2187c00.30000, bflag =9f, bp = 5b698340
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =a5ee400.30000, bflag =9f, bp = 5b698340
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =b3af000.30000, bflag =9f, bp = 5b698340
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =d50d000.30000, bflag =9f, bp = 5b698340
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =8b0f000.30000, bflag =9f, bp = 5b698340
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =d9bf400.30000, bflag =9f, bp = 5b698340
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =2490800.30000, bflag =9f, bp = 5b698340
Synchronous Page I/O error occurred while paging to/from disk
device 0x40000003, 1K block #2980, 28672 bytes of 28672 bytes not paged, page addr =9cdd800.30000, bflag =9f, bp = 5b698340
0/4/1/0/4/0: Device at device id 0x610ee4 has disappeared from Name Server GPN_FT
(FCP type) response, or its 'Port World-Wide Name' has changed.
device id = loop id, for private loop devices
device id = nport ID, for fabric/public-loop devices
System won't be able to see LUNs behind this port.

0/4/1/0/4/0: Device at device id 0x610ee4 is back in Name Server GPN_FT (FCP type)
response, and its 'Port World-Wide Name' remains the same as
original.
device id = loop id, for private loop devices
device id = nport ID, for fabric/public-loop devices
System will be able to see LUNs behind this port
(might need to run 'ioscan' first).

Also I am attaching the text file as asked. Please help me out

[Torsten.]

Your system has problem with disk at 0/4/1/0/4/0 - hardware problems?

[Fabio Ettore]

Hi,

you can see Page I/O error messages that you saw in Ignite-UX backup.
Definitely I think it's the reason of problem, suggest checking what's going on with that disk.

Best regards,
Fabio

[NEVATDR]

But on that harware path disk I can access all the file system easily and morever I have again after your message initially while take ignite backup some other problem I have observed if you see my initial log file

Regards
DP

[TTr]

From your syslog...
1. Aug 20 09:46:08 vcmsar vmunix: Variable size pages used to map 1000 graf pages at ffffffffb1000000 PROBLEM DETECTED! The system is susceptible to a panic if memory near 0xb0200000 is accessed for something other than an access of this graphics device. You need to apply the kernel vm patch PHKL_23203, or a superceding version, ASAP

This can be safely ignore if you don't use the graphics device on this server.

2. (Bottom of Aug 20 09:46:08 listing)
Aug 20 09:46:08 vcmsar vmunix: msgcnt 1 vxfs: mesg 071: cleared data I/O error flag in/dev/root file system
Aug 20 09:46:08 vcmsar vmunix: msgcnt 2 vxfs: mesg 038: vx_dataioerr - /dev/root file system file data read error

There is definitely a disk read error here.

3. Synchronous Page I/O error occurred while paging to/from disk

I am not sure what this refers to but it may have to do with the swap volume. Your server has only 4GB of memory and with a database running on it I suspecting that it is swapping heavily. This error could be the result of that with possibly disk errors in the swap volume.

4. Aug 23 00:35:29 vcmsar vmunix: 0/4/1/0/4/0: Device at device id 0x610ee4 has disappeared from Name Server

This is probably a tape device because of the tape devices listed on the same hardware path on Aug 20 19:30:00

5. Aug 31 12:15:36 vcmsar vmunix: msgcnt 3 vxfs: mesg 038: vx_dataioerr - /dev/vg00/lvol5 file system file data read error

This is definitely a disk read error on the /opt filesystem.

You should run a read test on your disk volumkes such as the swap volume

dd if=/dev/vg00/lvol2 of=/dev/null bs=1024k

and see if you get read errors. Repeat the command for /dev/vg00/rlvol5 (/opt)

Run a "vgdisplay -v /dev/vg00" and look for errors. Run "lvdisplay -v /dev/vgXX/lvolYY" for vg00 and vg01 and all their lvols.

As I said earlier you probably need to replace your disk and recover your server

[Torsten.]

Your system clearly says there is a problem with disk at 0/4/1/0/4/0 - it is disappearing, so you likely have a hardware problem with this.

[Torsten.]

Please execute

/opt/resmon/bin/resdata -R 144834562 -r /adapters/events/ql_adapter/0_4_1_0_4_0 -n 144834572 -a

Looks like the server is accessing both disks and tapes over this connection - did you create separate zones for them?

[NEVATDR]

Hi,
First of all I would like to thank all of you for your continuous support. I have observed that all the servers on the network having same problem. Yes you are right we are using same path for both tape library and SAN switch. But with this type of connection we are running the setup for last 4 years and not facing any problem. I am also attaching the log files for another server for your reference. Please make quick resolve of the issue