Degraded backup and Oracle dump performance after 11.23 to 11.31 upgrade

[Martin Roende]

Hello.
I am seeking for a solution of this i/o performance problem after upgrading from 11.23 to 11.31.

System rx6600 with EVA (HSV210) storage.
Now running HP-UX v. 11.31 Sep09 Dist.
Before running HP-UX v. 11.23 Sep06 Dist.

The nightly RMAN backup job og aprox. 2 Tbyte into separate disk are took before upgrade 5 hours.
Now the same job takes 8 hours.

I have testet the /exp with dd on another machine (same setup, just movig the vg04 over) )and get the same dd result.

Kernel params are set after Oracle recommandations.

I am looking for inputs from other sysadms with sort of problem.
Right now I am looking into vxtunefs.

Regards MArtin Rønde Andersen Miracle A/S miracleas.dk

[butti]

Hi,

did you have changed the Mount Options from the volumes (cache or without)?

Did you have change any Oracle Paramter (SGA etc..)?

Could you monitor the Disk queue during the Backup (with glance)?

by,
butti

[TTr]

How much memory does the server have? The 11.31 might be using more memory and may be running out of physical memory.

Also did oracle get upgraded as well or stayed the same?

[TwoProc]

My question is:

What else, besides the OS has changed? SGA? Kernel Params?

[Martin Roende]

The system is the same ..
rx6600 ,Memory 32G , Cpu 2 x dual core Itanium

The test config (a similar machine), has been running clones of the database for 16 months.

The production database setup is the same:
- no changes in SGA, oracle
- I have not used any special mount options.
(what do you mean ?) I tested before the upgrade, moving the mount point between the prod (11.23) and test (11.31) and all application functionality was aproved.

- After going into production, we experienced that nfile=30000 was throttling the system and we changed it to 0.

- I am at the moment investigating disk que figures from the glance collections I have (1 minut interval) from before and after the upgrade.

- I tried (with no success to change vxtunefs params: for the /exp area. (now changed back after test overnight.
read_ahead=2 (now back to 1)
max_buf_data_size=64k (now back to 8k)
scsimgr param for the disk182 (/exp):
leg_mpath_enable=false (now back to true)

-I have attached the current varux01 (prod) kctune output, in which you also will see the changes I am planning to get changed at next reboot.

Thanks for the comments, sorry for the late answer back,I was hung up at another customer.
Regards, Martin Roende Andersen

[Martin Roende]

For the reference.
Things done, that takes longer:'
RMAN backup BEFORE:
runs from 18:40 - 23:59

RMAN backup AFTER:
runs from 18:40 - 01:30
Changed in schedule
runs from 17:00 - 00:00 (aprox)

----
After RMAN backup to /exp, the /exp mount point is backup up.
BEFORE:
DP backup to fiber connected Tape:
Runtime 5:09:21
Backup speed 106.181,44 KB/s
Endtime 07:16:37

AFTER:
Runtime 8:05:55
Backup speed 66.29 MB/s
Endtime 10:05:55

Thats how the customer sees the problem.

Regards Martin Roende Andersen

[TTr]

> moving the mount point between the prod (11.23) and test (11.31)...After going into production

Is the 11.31 production rx6600 a different server than the rx6600 11.23 production?

Also unless you have a typo, the DP times are the same (actually the AFTER is shorter but the speed is different. How is that happening?

The /exp is the common factor for both the RMAN and the DP backups but is that really the issue? Have you done any disk tests on /exp?

Also whether the hardware changed or not, I would look into the fiber interfaces (and the fiber in general). This is common to both the /exp RMAN writes and DP reads as well as the tape writes.

Have you noticed anything in the glance metrics? It might be better to observe the glance during the RMAN and DP runs so that you will see any alerts.

[Duncan Edmonstone]

>> The test config (a similar machine),

- How similar? Completely identical (same model, same number and type of CPU, same memory, same FC cards etc...)

These jobs look like big sequential IO jobs - in my experience nothing much you might do with vxtunefs will change that - it would be helpful to see the mount options on the 2 servers:

mount -p

HTH

Duncan

[Duncan Edmonstone]

The other thing to look at is the EVA...

It's possible all the ownership of LUNs has moved to just one controller (whcih would tally with about half the sequential IO performance you are seeing)

on 11.11/11.23 the EVA Vdisk controller preference should be left at â No Preferenceâ and the LVM Primary/Alternate paths should be set carefully to be balanced evenly across the two controllers and ports (because the UX IO will always be down one path only). (Iâ m assuming Securepath was not used). This way UX determines the balance of IO and handles failover/failback, and the EVA should match Vdisk ownership to the prevailing IO pattern.

in contrast, with 11iv3, the EVA Vdisk preferences should be set deliberately to a 50/50 mix of â Controller A failover/failbackâ and â Controller B failover/failbackâ and Multipathing should be set to RR with ALUA (the defaults). In this case the UX server respects the EVA preferences/balance.

SO you need to look at the EVA CommandView config for these LUNs and then compare to what you see from evainfo and "scsimgr lun_map" on the 11.31 host, to be sure that all your IO is using both EVA controllers (any given LUN will just use 1 controller, but you should have an even number of LUNs spread across both controllers.)

HTH

Duncan

[Martin Roende]

Duncan ..
They are exactly the equal machines ..
Your point about the EVA has not been into consideration, and I will talk to the storage manager..

With /exp on varux01 machine:
/dev/vg04/exp1vol /exp vxfs ioerror=mwdisable,largefiles,delaylog,dev=40040001 0 0

From fstab:
varux01 # grep /exp /etc/fstab
/dev/vg04/exp1vol /exp vxfs rw,suid,largefiles,delaylog,datainlog 0 2


See the atttached file for vginfo, dsf, scsimgr info ..