fbackup slower after patch installation

[Raynald Boucher]

Hello all,

On Saturday we installed the following patches:
PHNE_35950 sendmail(1M) 8.9.3 patch
PHSS_34389 Xserver cumulative patch
PHSS_36286 KRB5-Client Version 1.0 cumulative patch

We are running an rp3440 with hp-ux 11.11.

Since then, out backups are taking 30 minutes or 30% more time to complete.

We did not detect this on our development server as we use fbackup in production but Omniback in development.

Output is directed to an Ultrium 2 tape drive on an SCSI Ultra320 A6961-60011 bus.

As these patches do not appear to have anything to do with backup...

Can anyone provide a hint or a reason why the backup is taking so must longer?
I have checked all installation logs and found no errors.

Thanks

RayB

[Steven E. Protter]

Shalom RayB,

No I can't give you a hint.

I can say that sometimes patches have unintended consequences. These patches are all three star I believe and should not have caused the slowdown.

It's good to look at the patches, because when something goes wrong, it usually is the last thing you did. Or something you did not do.

You say you installed three patches. What about the last bi-annual patch set? Lots of performance related patches in both of those.

What you have before you is an investigation.

I assume you want those patches installed, bu t if you can live without them for a while, see if using swremove to remove them one at a time helps.

Note, it also could be something that someone else did. Perhaps you are backing up san data and while you were installing patches the san team was working magic and slowing down your I/O.

SEP

[A. Clay Stephenson]

Nothing jumps out as obvious although you should note that your X server patch has been superceded with a later version (PHSS_34391) marked as critical.

The only thing that occurs to me is to back out each patch one at the time to see if the problem disappears -- I'm betting it won't.

I would look to other things as the cause of the bottleneck during backup but you are going to have to gather some metrics.

I do find it ironic that you are using fbackup for production and OB2 for production. I would have thought that it would have been the other way around or OB2/DP for both. Of course, if you really meant OB2 rather than DP then all versions of OB2 are out of support so perhaps that's why you are using fbackup.

Did someone "improve" the fbackup config file over the weekend as well?

[Bill Hassell]

And to amplify Clay's question: are you using the mandatory fbackup config file? Without it, the default values are designed for archaic reel-to-reel tapes, circa 1980's. Use this as a starting point:

blocksperrecord 4096
records 64
checkpointfreq 4096
readerprocesses 6
maxretries 5
retrylimit 5000000
maxvoluses 200
filesperfsm 2000

NOTE: records 64 will need a fairly large chunk of shared memory. You can reduce this to 32 or 24 with a drop in performance.

[tkc]

hi ray,

i suggest you to run fbackup with the -c option to the file /etc/sam/br/fbackup_config. you can edit the fbackup_config file with different value for blocksperrecord to find out the best block size for the fbackup performance to the LTO2 drive. the default fbackup parameter will not get the best performance out from the LTO2 drive.

[whiteknight]

hi RayB,

Here is another tips.

There are two steps required to increase the write performance of fbackup:

First the parameters defined in the config file should be adapted for better performance:

blocksperrecord 64
records 64
checkpointfreq 128
readerprocesses 1
maxretries 5
retrylimit 5000000
maxvoluses 100
filesperfsm 500


If disk striping is used the number of reader processes should be increased to the number of disks:

readerprocesses number_of_disks


Note:

The config file is specified at the command line via the '-c' switch. If the backup is setup as an automated backup via SAM then the config file /etc/sam/br/fbackup_config is used.


WK
please assign points

[skt_skt]

Could you paste the output of

ll /usr/sbin/sendmail
what /usr/sbin/sendmail
ll /etc/inetd.conf

[Raynald Boucher]

Hello all,

the command we use is:
/usr/sbin/fbackup -y -ug $BKUPROOT/graph2.files -V $hdr_file -I $inf_file -c $BKUPROOT/fb_config.parms -f $BKUPDEV | tee -a $LOGFILE
where
BKUPROOT=/usr/local/backup

ll /usr/local/backup gives:
-r-xr-xr-x 1 root software 271 Oct 9 1997 error
-rw-r--r-- 1 root software 73 Jul 7 2002 graph.files
-rw-r--r-- 1 root software 32 Jun 23 2005 graph2.files
-r-xr-xr-x 1 root sys 600 Jul 20 2005 chgvol_dat
-rw-r--r-- 1 root software 204 Aug 25 2005 fb_config.parms
-r-xr-xr-x 1 root software 647 Nov 9 2005 chgvol

and fb_config.parms contains:
blocksperrecord 128
records 32
checkpointfreq 256
readerprocesses 6
maxretries 3
chgvol /usr/local/backup/chgvol
error /usr/local/backup/error

Looking at the dates, nothing has changed in nearly 2 years.


For you Santhosh,
$ ll /usr/sbin/sendmail
-r-sr-sr-t 1 root mail 864256 May 10 05:21 /usr/sbin/sendmail
$ what /usr/sbin/sendmail
/usr/sbin/sendmail:
Copyright (c) 1998 HEWLETT PACKARD COMPANY and its licensors,
including Sendmail, Inc., and the Regents of the
University of California. All rights reserved.
version.c 8.9.3 (Berkeley) 10 May 2007 (PHNE_35950)
$ ll /etc/inetd.conf
-r--r--r-- 1 bin bin 4840 Mar 31 13:35 /etc/inetd.conf

I rather suspected patch 34389 (Xserver) because it is the only one labelled with "hardware enablement".

Thanks all,

RayB

[TY 007]

Hello Ray,

>> On Saturday we installed the following patches:
>> PHNE_35950 sendmail(1M) 8.9.3 patch
>> PHSS_34389 Xserver cumulative patch
>> PHSS_36286 KRB5-Client Version 1.0 cumulative patch
>> Since then, out backups are taking 30 minutes or
>> 30% more time to complete

Stop Sendmail & try to perform backup again:
# /sbin/init.d/sendmail stop

Thanks

[Raynald Boucher]

We'll consider...

but what do sendmail and fbackup have in common?

It's definately an fbackup problem because, as shown below, ftio writes to the same hardware immediately after and it's run time has not increased:

$ for file in `ls bkup* | tail -10`
> do
> zcat $file | head | grep finished
> done
08/17/07,04:29:51: fbackup finished
08/17/07,05:19:05: ftio finished
08/19/07,04:29:44: fbackup finished
08/19/07,05:19:00: ftio finished
08/20/07,04:29:55: fbackup finished
08/20/07,05:19:01: ftio finished
08/21/07,04:30:26: fbackup finished
08/21/07,05:19:39: ftio finished
08/22/07,04:28:52: fbackup finished
08/22/07,05:18:06: ftio finished
08/23/07,04:28:56: fbackup finished
08/23/07,05:17:58: ftio finished
08/24/07,04:30:00: fbackup finished
08/24/07,05:18:57: ftio finished
08/26/07,05:02:55: fbackup finished
08/26/07,05:51:47: ftio finished
08/27/07,05:03:31: fbackup finished
08/27/07,05:52:28: ftio finished
08/28/07,05:03:06: fbackup finished
08/28/07,05:52:14: ftio finished
$

[Steve Lewis]

Had anyone changed any kernel parameters and not rebooted before these patches were installed? A reboot would put a new kernel in place and this may contain parameters that would affect your backup.

Also, had anyone changed your scsi_queue_depth parameters or primary paths to disks but not made the changes permanent?

A reboot would put you back to the state before any on-the-fly changes had been made possibly affecting the performance of your backup.

Another possibility is that maybe before these patches were applied, your system may have only just been keeping up to speed with your LTO2 (which has no tape speed slow-down on data starvation, IIRC). The new patches may have caused a slight increase in CPU consumption, enough to slow down the data rate to the tape and tip it over the edge, with a busier system causing occasional data starvation and resultant tape shoe-shining. Thats a long shot though.

[tkc]

hi ray,

if writing to the LTO2 drive is ok, there could possibly be a bottleneck on the i/o read from the disk. do u think there is a move of data reading from a higher performance disk to a lower one?

[skt_skt]

Here is the patch level i have in one of the server where i use LTO
. make sure that you are not lagging in patch level--just a try

PHCO_25831 1.0 SCSI Ultra160 driver Online Addition script
PHCO_29098 1.0 scsictl(1M) cumulative patch
PHCO_33205 1.0 mountall, Dev IDs enabler, iSCSI support
PHCO_33221 1.0 /sbin/fs/hfs/bcheckrc patch;iSCSI
PHCO_33236 1.0 mount_vxfs(1M) cumulative patch, iSCSI
PHCO_33237 1.0 sync switch;invalid sparse inode;iSCSI
PHCO_33242 1.0 fsck_hfs(1M) cumulative patch, iSCSI support
PHCO_33327 1.0 Device IDs enab, mount(1M), iSCSI support
PHKL_23292 1.0 scsi_surface.h header file patch
PHKL_23666 1.0 SCSI IO Subsystem Cumulative Patch
PHKL_23810 1.0 Enable SCSI floppy for 64 bit computers
PHKL_30523 1.0 PDC_SCSI_PARMS,iCOD,vPars,IPMI,OLA/R
PHKL_32089 1.0 SCSI Ultra160 Cumulative Patch
PHKL_32090 1.0 SCSI IO Cumulative Patch
PHKL_33425 1.0 SCSI Tape (stape) cumulative