frecover slow on new itanium server 11.31

[Charles McCary]

Group,

hey - I am attempting to run an frecover process on a new server (rx6600) running 11.31 with direct attached p2000. The frecover process is taking 4-5 times as long to complete as on an older rp7420 server running 11.11 with a v7410 array.

I've run some dd perf tests on the new server and I'm seeing 390mbps writes and 400mpbs reads with concurrent reads and writes, so I don't think it's an IO issue.

The fbackup file was created on the 11.11 server if that matters.

Here's the frecover command I'm using:

/usr/contrib/bin/gzcat /backup/test/file1.fbackup.data1.gz | /usr/sbin/frecover -xvF \
-i /data1/datapath \
-f - \
> restore_data1.log 2> restore_data1.err

[Steven E. Protter]

Shalom,

Commonly these issues are dealt with in bi-annual patch sets for the OS.

You might also want to make sure you are getting good disk throughput on the p2000, if its a disk device.

SEP

[Charles McCary]

Steven,

thanks. We are on September 2010 patch sets for the itanium server.

As mentioned in the original post, we are seeing 390mpbs(write) and 400mpbs(read) on concurrent dd tests.

[Charles McCary]

should bd mbps...sorry.

[Steven E. Protter]

Shalom,

Try running iostat while running frecover.

Lets get some hard data and compare it to other disk operations on the p2000.

Check the HP-UX patch database for frecover/fbackup patches.

SEP

[Charles McCary]

latest 11.31 patch is PHCO_40460 and it's installed.

I will re-run the frecover and post some output.

c

[Charles McCary]

iostat info:

copying a large file from a p400 LUN to the p2000 lun:

disk24 is the p400 lun, disk28 is the p2000 R5 lun.

disk24 198464 200.2 1.0
disk28 198544 197.4 1.0

disk24 199486 200.8 1.0
disk28 199584 198.2 1.0


running the frecover from the p400 lun to the p2000 r5 lun. Again, disk24 is p400, disk28 is p2000 r5
disk24 0 0.0 1.0
disk28 7303 5489.6 1.0

disk24 0 0.0 1.0
disk28 7230 5428.9 1.0

disk24 568 10.8 1.0
disk28 7327 5490.0 1.0

disk24 1348 25.5 1.0
disk28 7239 5441.9 1.0

disk24 958 18.2 1.0
disk28 7319 5489.0 1.0

disk24 1344 25.6 1.0
disk28 7279 5466.2 1.0

disk24 829 15.7 1.0
disk28 7225 5421.3 1.0

disk24 1147 22.1 1.0
disk28 7274 5472.1 1.0

disk24 960 18.2 1.0
disk28 7290 5463.6 1.0

disk24 960 18.2 1.0
disk28 7310 5491.6 1.0

disk24 1082 20.5 1.0
disk28 7193 5405.0 1.0

disk24 1088 20.8 1.0
disk28 7263 5448.4 1.0

disk24 64 1.2 1.0
disk28 7222 5422.0 1.0

disk24 0 0.0 1.0
disk28 7288 5477.9 1.0

disk24 192 3.8 1.0
disk28 7221 5418.6 1.0

disk24 703 13.2 1.0
disk28 7296 5476.4 1.0


running restore from the r5 lun to the r5 lun: disk28 7859 5214.6 1.0

disk28 7320 5488.8 1.0

disk28 7296 5479.8 1.0

disk28 7232 5422.7 1.0

disk28 7240 5431.1 1.0

disk28 7297 5466.5 1.0

disk28 7268 5460.2 1.0

disk28 7298 5478.8 1.0

disk28 7239 5426.5 1.0

disk28 7303 5477.4 1.0

disk28 7245 5436.9 1.0

disk28 7515 5464.0 1.0

disk28 8438 5428.3 1.0


running the restore from a p2000 raid10 lun to the r5 lun. the raid10 lun is disk27

disk27 674 21.1 1.0
disk28 7270 5463.2 1.0

disk27 622 19.6 1.0
disk28 7256 5447.0 1.0

disk27 606 18.9 1.0
disk28 7289 5480.9 1.0

disk27 492 15.4 1.0
disk28 7229 5440.5 1.0

disk27 13 0.6 1.0
disk28 7327 5496.4 1.0

disk27 6 0.2 1.0
disk28 7263 5467.3 1.0

disk27 13 0.4 1.0
disk28 7280 5464.7 1.0

disk27 6 0.2 1.0
disk28 7219 5430.9 1.0

disk27 13 0.4 1.0
disk28 7310 5489.0 1.0

disk27 13 0.4 1.0
disk28 7269 5464.8 1.0

disk27 990 31.1 1.0
disk28 7334 5497.8 1.0

[Charles McCary]

more testing results:



1) First I did a copy of the same file on two different servers to compare the copy times:

OLD_SERVER
# time cp finindex01.dbf testcopyfile
real 1:39.0
user 0.1
sys 34.0


NEW ITANIUM SERVER
# time cp finindex01.dbf testcopyfile
real 32.8
user 0.0
sys 0.3

Conclusions â the copy times are much faster on the new infrastructure.


2) Next I ran an fbackup on both servers to compare:

OLD SERVER:
fbackup(1004): session begins on Thu Nov 18 10:46:20 2010
fbackup(3024): writing volume 1 to the output file -
1: / 16
2: /dir1 16
3: /dir1/dir2 16
4: /dir1/dir2/file 8396816
fbackup(1005): run time: 600 seconds
fbackup(3055): total file blocks read for backup: 8396864
fbackup(3056): total blocks written to output file -: 8396890


NEW ITANIUM SERVER:
fbackup(1004): session begins on Thu Nov 18 10:29:52 2010
fbackup(3024): writing volume 1 to the output file -
1: / 16
2: /dir1 1
3: /dir1/dir2 1
4: /dir1/dir2/file 8396816
fbackup(1005): run time: 432 seconds
fbackup(3055): total file blocks read for backup: 8396850
fbackup(3056): total blocks written to output file -: 8396882


conclusions â fbackup runs faster on the new infrastructure.


3) Next I ran frestores on both servers to compare:

Restores on NEW ITANIUM server
1st restore was attempted from the fbackup file created on the old server.

# ./restore_test.sh
START: Thu Nov 18 10:05:16 CST 2010
FINISH: Thu Nov 18 10:18:12 CST 2010

2nd restore was attempted from the fbackup file created on the new ITANIUM SERVER

# ./restore_test.sh
START: Thu Nov 18 10:39:37 CST 2010
FINISH: Thu Nov 18 10:52:32 CST 2010


Hereâ s the same restore on the OLD Server
# ./restore_test.sh
START: Thu Nov 18 11:21:48 EST 2010
FINISH: Thu Nov 18 11:23:55 EST 2010


Conclusions
a) No significant difference in restore times on the new server when restoring from the fbackup file created on OLD 11.11 server or 11.31 server.
b) The restore process on the OLD server is significantly (5-6 times) faster.

__________________
Hereâ s the fbackup commands used in the work above:
/usr/sbin/fbackup -v -c ${BDIR}/fbackup.conf -i ${BDIR}/finindex01.dbf \
-f - 2> test.fbackup.err | /usr/contrib/bin/gzip - > /${BDIR}/test.fbackup.gz
__________________

Hereâ s the frecover command used in the work above.
# cat restore_test.sh
#!/bin/ksh

RESTORE_DIR=/dir1/dir2/restore
RESTORE_FILE=test.fbackup.gz

cd ${RESTORE_DIR}


echo "START: `date`"
(
/usr/contrib/bin/gzcat ${RESTORE_DIR}/${RESTORE_FILE} | /usr/sbin/frecover -xvF \
-i /dir1/dir2/ \
-f - \
> restore_test.log 2> restore_test.err
)
echo "FINISH: `date`"
__________________

Hereâ s the fbackup.conf file used in the fbackup command.

blocksperrecord 4096
records 64
checkpointfreq 4096
readerprocesses 6
maxretries 5
retrylimit 5000000
maxvoluses 200
filesperfsm 2000

[Bill Hassell]

> The restore process on the OLD server is significantly (5-6 times) faster.

What model tape drives are you using? fbackup/frecover have a high performance feature for DDS drives called high speed search using special DDS codes called search marks. Using search marks, the tape runs about 10-20x faster to skip over unselected files. But DLT and LTO drives lack this feature and fbackup emulates search marks with file marks, with lower performance.

And fbackup/frecover underwent significant hacking (code changes) for 11.31 which resulted in compatibility issues with earlier versions of fbackup/frecover. Since 11.31 is the end of these tools, I think the performance may be suspect too.