Does this mean gzip on Ignite client needs patching?

[Ralph Grothe]

Hello,

when I checked this morning the latest recovery.log of an Ignite client on the server
I discovered this unsuccessful abort
owe to a too large file for gzip to be handled.
This sounds to me as if gzip on the client needs to be patched to support large files.


# tail /var/opt/ignite/clients/samos/recovery/latest/recovery.log
gzip: stdout: File too large
ERROR: The gzip command failed (exit status 1).
ERROR: The gzip command failed (exit status 1).

ERROR: The make_sys_image command failed. The system recovery archive will
not be created.


======= 11/07/06 19:32:21 MET make_net_recovery completed unsuccessfully





In the readme of PHCO_34539
http://www4.itrc.hp.com/service/patch/patchDetail.do?patchid=PHCO_34539&sel={hpux:11.11,}&BC=main|search|
I discovered this snippet:

PHCO_32116:

JAGae63284, JAGaf19697: gzip(1) and gunzip(1) do not
handle large files greater than 2 gigabytes.


On the Ignite client at least none of those patches seems to be installed


$ /usr/sbin/swlist PHCO_34539 PHCO_32116
# Initializing...
# Contacting target "samos"...
ERROR: Software "PHCO_34539" was not found on host "samos:/".
ERROR: Software "PHCO_32116" was not found on host "samos:/".


Would you agree that gzip needs patching?

Regards
Ralph

[Peter Godron]

Ralph,
what version of gzip is your recovery using, if it is 1.3.5 then it already patched.
I assume all your relevant filesystem are largefile enabled.

You may also want to scan through:
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=576751
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=611148
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=657601

[Ralph Grothe]

Hi Peter,

the patch PHCO_34539 seems to upgrade the gzip command from 1.2.4 to 1.3.5
But patching the Ignite client didn't help.
So I assume that the gzip-ing of the stdin stream is performed on the Ignite server,
why I also now applied PHCO_34539 on it.
I rescheduled make_net_recovery and will have to wait what happens now...

[Steven E. Protter]

Shalom,

Certainly seems you need gzip and then to patch gzip.

Creating archives with largefiles has given problems unzipping images in the past. pax_iux has been known to have problems with files bigger than 2 GB.

For an OS root backup you don't need files bigger than 2 GB. Ignite is not a good tool for large database files and those volume groups should be exluded and that data tranferred after imaging is complete on the target system.

SEP

[Ralph Grothe]

Still the same,
make_net_recovery failed again despite patched gzip commands on Ignite server and client.


# tail /var/opt/ignite/clients/samos/recovery/latest/recovery.log
gzip: stdout: File too large
ERROR: The gzip command failed (exit status 1).
ERROR: The gzip command failed (exit status 1).

ERROR: The make_sys_image command failed. The system recovery archive will
not be created.


======= 11/08/06 10:32:28 MET make_net_recovery completed unsuccessfully


The Ignite server's archives dir resides on a separate filesystem (to prevent /var from being filled by ignite images) which I created with largefiles support, and for which I put this option in fstab.
Therefore the current mount of this filesystem bears this feature

# mount -p|grep recovery
/dev/vg01/lvrecovery /var/opt/ignite/recovery vxfs delaylog,nodatainlog,largefiles 0 0


As you can see gzip version on the server

# gzip -V|head -1
gzip 1.3.5

is after patching the same as on the client

# ssh samos /usr/contrib/bin/gzip -V\|head -1
root@samos's password:
gzip 1.3.5


So, I cannot see why the handled file is still too large for gzip?

[Ralph Grothe]

Hi SEP,

when firing up make_net_recovery I took care to only include vg00 and exclude temp stuff.
This is the command I executed on the Ignite client


# echo /opt/ignite/bin/make_net_recovery -v -P s -s elba -x inc_entire=vg00 -x exclude=/tmp -x exclude=/var/tmp|batch

[Peter Godron]

Ralph,
somebody else also had the same problem whilst using NFS:
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1004714
Problem was resolved by:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=713940

Could you please post a bit more info from
/var/opt/ignite/clients/samos/recovery/latest/recovery.log . Double check the link to gzip from usr/local to /usr/contrib etc.

[john korterman]

Hi Ralph,

my understanding of the last "0" in your mount options imply that the file system is ignored by fsck - are you sure that the file system is ok?


regards,
John K.

[Ralph Grothe]

John,

the filesystem is pretty new.
So I assume it didn't have more than 3 mounts yet.
Nevertheless I placed a secondary fsck prioority option in fstab and afterwards mounted it manually by giving mount only the mountpoint without any options or type specifier.

# grep recovery /etc/fstab
/dev/vg01/lvrecovery /var/opt/ignite/recovery vxfs delaylog,largefiles 0 2

So I gather that the mount -p only designates a clean state with the trailing 0.

But since the filesystem is only used during Ignite operations I can however run a check manually

# fuser -c /var/opt/ignite/recovery
/var/opt/ignite/recovery:

# umount /var/opt/ignite/recovery
# fsck -F vxfs -m /dev/vg01/rlvrecovery
vxfs fsck: sanity check: /dev/vg01/rlvrecovery OK
# fsck -F vxfs -o full /dev/vg01/rlvrecovery
log replay in progress
pass0 - checking structural files
pass1 - checking inode sanity and blocks
pass2 - checking directory linkage
pass3 - checking reference counts
pass4 - checking resource maps
OK to clear log? (ynq)y
set state to CLEAN? (ynq)y
# mount /var/opt/ignite/recovery
# fsadm -F vxfs /var/opt/ignite/recovery
largefiles
# mount -p|grep /var/opt/ignite/recovery
/dev/vg01/lvrecovery /var/opt/ignite/recovery vxfs delaylog,nodatainlog,largefiles 0 0

[Ralph Grothe]

Thank you Peter for supplying me with the other threads.
Unfortunately I only receive a 404 from the webserver when I follow the second of them,
the one you considered to hold the solution.
I guess that the one of the European HP Forums' proxies that I am using is either not having this URL in its cache, or isn't redirecting. :-(

However, I can follow the first thread which I still have to go through.
What kind of bemuses me is that one is required to set a symlink from /usr/contrib/bin/gzip to /usr/local/bin,
but if it helps...

[Ralph Grothe]

Hm, did provide the link

# find /usr /etc /opt -follow -name gzip|xargs ll -i
768 -r-xr-xr-x 1 bin bin 139264 May 7 2002 /usr/adm/sw/save/PHCO_34539/GZIP/usr/contrib/bin/gzip
39268 -r-xr-xr-x 3 bin bin 114688 Aug 19 2004 /usr/contrib/bin/gzip
# ln -s /usr/contrib/bin/gzip /usr/local/bin
# find /usr /etc /opt -follow -name gzip|xargs ll -i
15065 lrwxr-xr-x 1 root sys 21 Nov 8 13:29 /opt/Xaw3d/bin/gzip -> /usr/contrib/bin/gzip
15065 lrwxr-xr-x 1 root sys 21 Nov 8 13:29 /opt/jpeg-6/bin/gzip -> /usr/contrib/bin/gzip
15065 lrwxr-xr-x 1 root sys 21 Nov 8 13:29 /opt/libpng/bin/gzip -> /usr/contrib/bin/gzip
15065 lrwxr-xr-x 1 root sys 21 Nov 8 13:29 /opt/tiff-3.5/bin/gzip -> /usr/contrib/bin/gzip
15065 lrwxr-xr-x 1 root sys 21 Nov 8 13:29 /opt/xpm/bin/gzip -> /usr/contrib/bin/gzip
15065 lrwxr-xr-x 1 root sys 21 Nov 8 13:29 /opt/zlib/bin/gzip -> /usr/contrib/bin/gzip
768 -r-xr-xr-x 1 bin bin 139264 May 7 2002 /usr/adm/sw/save/PHCO_34539/GZIP/usr/contrib/bin/gzip
39268 -r-xr-xr-x 3 bin bin 114688 Aug 19 2004 /usr/contrib/bin/gzip
15065 lrwxr-xr-x 1 root sys 21 Nov 8 13:29 /usr/gnu/bin/gzip -> /usr/contrib/bin/gzip
15065 lrwxr-xr-x 1 root sys 21 Nov 8 13:29 /usr/local/bin/gzip -> /usr/contrib/bin/gzip


but wait, last responder of first thread is telling something about inappropiate NFS settings...

[Peter Godron]

Ralph,
the relevant section from the second link:
"The only known reasons for large file errors in Ignite-UX is from
make_net_recovery (except when make_sys_image is used directly).
And typically, there errors are caused by:

1) NFSv2 is being used instead of NFSv3, NFSv2 only supports
2Gb files and does not support large files at all.

2) The local or remote file system the archive or golden image
is being written to does not have large file support enabled.

When the file system does not support large files, it is gzip (from
within make_sys_image and make_net_recovery) that will print an error
about the file. Although gzip itself could successfully write a large
file via a shell pipe, an error can occur when a write is attempted
to a file system which does not support large files. This is not a
problem with gzip but rather a problem with the file system being
written to."

[Ralph Grothe]

Ah, think I made the same mistake

# grep ^AUTOFS /etc/rc.config.d/nfsconf
AUTOFS=0

So let's restart the nfs.server with AUTOFS=1
...

[Ralph Grothe]

I set AUTOFS=1 (although I suspect this to be relevant only for the NFS client),
restarted the Ignite server's NFS server,
and flushed nfsstats

# nfsstat -zn >/dev/null
# nfsstat -sn|grep -i version
Version 2: (0 calls)
Version 3: (0 calls)


On Ignite client's NFS server I set these

# grep ^[^#] /etc/rc.config.d/nfsconf
NFS_CLIENT=1
NFS_SERVER=0
NUM_NFSD=16
NUM_NFSIOD=16
PCNFS_SERVER=0
LOCKD_OPTIONS=""
STATD_OPTIONS=""
MOUNTD_OPTIONS=""
AUTOMOUNT=0
AUTO_MASTER="/etc/auto_master"
AUTO_OPTIONS="-f $AUTO_MASTER"
START_MOUNTD=1
AUTOFS=1
AUTOMOUNT_OPTIONS=""
AUTOMOUNTD_OPTIONS=""


and restarted nfs.core & nfs.client

# nfsstat -z >/dev/null

# nfsstat -zcn|grep -i version
Version 2: (0 calls)
Version 3: (0 calls)

and mounting the Ignite archive share manually on the client

# very/archives/samos mount -F nfs elba:/var/opt/ignite/recovery/archives/samos /var/opt/ignite/recovery/arch_mnt

# bdf -t nfs
Filesystem kbytes used avail %used Mounted on
elba:/var/opt/ignite/recovery/archives/samos
10485760 1142008 9270760 11% /var/opt/ignite/recovery/arch_mnt


# nfsstat -cm
/var/opt/ignite/recovery/arch_mnt from elba:/var/opt/ignite/recovery/archives/samos (Addr n.n.n.n)
Flags: vers=3,proto=tcp,auth=unix,hard,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)


# nfsstat -nc|grep -i version
Version 2: (0 calls)
Version 3: (8 calls)


Well, looks we are exclusively using Version 3 calls now.
So let's try the make_net_recovery again...


# umount /var/opt/ignite/recovery/arch_mnt

# bdf -t nfs

# echo /opt/ignite/bin/make_net_recovery -v -P s -s elba -x inc_entire=vg00 -x exclude=/tmp -x exclude=/var/tmp|bat
warning: commands will be executed using /usr/bin/sh
job 1162991299.b at Wed Nov 8 14:08:19 2006

[Ralph Grothe]

Even though this time every prerequisite seemed to be met
the make_net_recovery failed again with the same error.

# tail /var/opt/ignite/clients/samos/recovery/latest/recovery.log
gzip: stdout: File too large
ERROR: The gzip command failed (exit status 1).
ERROR: The gzip command failed (exit status 1).

ERROR: The make_sys_image command failed. The system recovery archive will
not be created.


======= 11/08/06 14:26:14 MET make_net_recovery completed unsuccessfully



This time I *had* NFS V3,
and the other requirements had been met
like before, viz. Ignite server NFS archive share being mounted there *with* largefile option, and the exportfs of it after that had taken place.

Anything else I have forgotten?

[Ralph Grothe]

Sorry Guys,
it somehow has escaped my notice that vg00 of the Ignite client contais an LV of some 10 Gig that was mounted on /export/oracle and filled to abt. 7.5 Gig.
Even for largefile capable tools this seems a bit over the top, I suppose.
I think this LV should better be recovered by OmniBack and doesn't belong on an OS disaster recovery backup.
So I additionally inserted an "-x exclude=/export/oracle" and reran make_net_recovery.
This time I sucessfully passed the make_sys_image hurdle, and the dump is currently written into this clients archive dir on the Ignite server.

Sorry, for bothering you.

[Ralph Grothe]

Sorry Guys,
it somehow has escaped my notice that vg00 of the Ignite client contais an LV of some 10 Gig that was mounted on /export/oracle and filled to abt. 7.5 Gig.
Even for largefile capable tools this seems a bit over the top, I suppose.
I think this LV should better be recovered by OmniBack and doesn't belong on an OS disaster recovery backup.
So I additionally inserted an "-x exclude=/export/oracle" and reran make_net_recovery.
This time I sucessfully passed the make_sys_image hurdle, and the dump is currently written into this client's archive dir on the Ignite server.

Sorry, for bothering you.

[Ralph Grothe]

Epilogue:

# tail /var/opt/ignite/clients/kos/recovery/latest/recovery.log
* Creating CINDEX Configuration File

* /opt/ignite/bin/manage_index -q -c 2006-11-08,17:02\ Recovery\ Archive
-i /var/opt/ignite/recovery/client_mnt/0x00306E49C51A/CINDEX -u
Recovery\ Archive



======= 11/08/06 17:15:16 MET make_net_recovery completed successfully!

[Ralph Grothe]

Epilogue:

# tail /var/opt/ignite/clients/kos/recovery/latest/recovery.log
* Creating CINDEX Configuration File

* /opt/ignite/bin/manage_index -q -c 2006-11-08,17:02\ Recovery\ Archive
-i /var/opt/ignite/recovery/client_mnt/0x00306E49C51A/CINDEX -u
Recovery\ Archive



======= 11/08/06 17:15:16 MET make_net_recovery completed successfully!

[Ralph Grothe]

Epilogue:

# tail -3 /var/opt/ignite/clients/kos/recovery/latest/recovery.log

======= 11/08/06 17:15:16 MET make_net_recovery completed successfully!

[Ralph Grothe]

Oops, my POST request inadvertently was caught trice by the webserver