System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

RE: Disk IO shooting to 100%

Henry Manda
Advisor

RE: Disk IO shooting to 100%

Every time I run COB in T24, the disk IO shoots to 100%.I created a new file system with PVGs to enable load balancing, disk stripping is also enabled, but still Im experiencing the same problem.COB takes 12 hrs.Kindly give me an idea about creating a memfs and use it with T24 application.or is there anything else i can do to reduce the IO??
Im running HP-UX 11i V3, T24R06.005, jbase 4.1,15branches, 2000 trxn per branch per day.
15 REPLIES
Sunny123_1
Esteemed Contributor

Re: RE: Disk IO shooting to 100%

Hi

Refer following link

http://h20000.www2.hp.com/bc/docs/support/SupportManual/c01917506/c01917506.pdf (BSC link updated by admin)



Regards
Sunny
ManojK_1
Valued Contributor

Re: RE: Disk IO shooting to 100%

Hi,

Please provide the following details.

1) server Model
2) vgdisplay -v
3) Disks are local disk or disk presented from
storage?
4) Whether it is an application Server or DB
Server?
5) out put of the command "sar -d 2 20"
6) out put of the command "swapinfo -tam"

Manoj
Thanks and Regards,
Manoj K
Viktor Balogh
Honored Contributor

Re: RE: Disk IO shooting to 100%

FYI: T24 is a special banking application from company Temenos.

http://www.temenos.com/Software/Core-Banking-Solution-System-T24/

and COB must be referring to "Close of Business", a routine at the end of the day.

Could you confirm it, Henry?
****
Unix operates with beer.
Henry Manda
Advisor

Re: RE: Disk IO shooting to 100%

Server Model: rx8640
vgdisplay output:

--- Logical volumes ---
LV Name /dev/vg00/lvol1
LV Status available/syncd
LV Size (Mbytes) 1792
Current LE 28
Allocated PE 28
Used PV 1

LV Name /dev/vg00/lvol2
LV Status available/syncd
LV Size (Mbytes) 32768
Current LE 512
Allocated PE 512
Used PV 1

LV Name /dev/vg00/lvol3
LV Status available/syncd
LV Size (Mbytes) 10240
Current LE 160
Allocated PE 160
Used PV 1

LV Name /dev/vg00/lvol4
LV Status available/syncd
LV Size (Mbytes) 5120
Current LE 80
Allocated PE 80
Used PV 1

LV Name /dev/vg00/lvol5
LV Status available/syncd
LV Size (Mbytes) 30720
Current LE 480
Allocated PE 480
Used PV 1

LV Name /dev/vg00/lvol6
LV Status available/syncd
LV Size (Mbytes) 10240
Current LE 160
Allocated PE 160
Used PV 1

LV Name /dev/vg00/lvol7
LV Status available/syncd
LV Size (Mbytes) 20480
Current LE 320
Allocated PE 320
Used PV 1

LV Name /dev/vg00/lvol8
LV Status available/syncd
LV Size (Mbytes) 33280
Current LE 520
Allocated PE 520
Used PV 1


--- Physical volumes ---
PV Name /dev/disk/disk4_p2
PV Status available
Total PE 4785
Free PE 2525
Autoswitch On
Proactive Polling On



VG Name /dev/vgt24
VG Write Access read/write
VG Status available, exclusive
Max LV 16
Cur LV 1
Open LV 1
Max PV 8
Cur PV 4
Act PV 4
Max PE per PV 32000
VGDA 8
PE Size (Mbytes) 16
Total PE 51196
Alloc PE 51136
Free PE 60
Total PVG 1
Total Spare PVs 0
Total Spare PVs in use 0
VG Version 1.0
VG Max Size 4000g
VG Max Extents 256000

--- Logical volumes ---
LV Name /dev/vgt24/lvt24
LV Status available/syncd
LV Size (Mbytes) 818176
Current LE 51136
Allocated PE 51136
Used PV 4


--- Physical volumes ---
PV Name /dev/dsk/c14t0d2
PV Name /dev/dsk/c7t0d2 Alternate Link
PV Name /dev/dsk/c9t0d2 Alternate Link
PV Name /dev/dsk/c12t0d2 Alternate Link
PV Status available
Total PE 12799
Free PE 15
Autoswitch On
Proactive Polling On

PV Name /dev/dsk/c7t0d3
PV Name /dev/dsk/c14t0d3 Alternate Link
PV Name /dev/dsk/c9t0d3 Alternate Link
PV Name /dev/dsk/c12t0d3 Alternate Link
PV Status available
Total PE 12799
Free PE 15
Autoswitch On
Proactive Polling On

PV Name /dev/dsk/c9t0d4
PV Name /dev/dsk/c7t0d4 Alternate Link
PV Name /dev/dsk/c14t0d4 Alternate Link
PV Name /dev/dsk/c12t0d4 Alternate Link
PV Status available
Total PE 12799
Free PE 15
Autoswitch On
Proactive Polling On

PV Name /dev/dsk/c12t0d5
PV Name /dev/dsk/c7t0d5 Alternate Link
PV Name /dev/dsk/c9t0d5 Alternate Link
PV Name /dev/dsk/c14t0d5 Alternate Link
PV Status available
Total PE 12799
Free PE 15
Autoswitch On
Proactive Polling On


--- Physical volume groups ---
PVG Name pvg01
PV Name /dev/dsk/c14t0d2
PV Name /dev/dsk/c7t0d3
PV Name /dev/dsk/c9t0d4
PV Name /dev/dsk/c12t0d5
PV Name /dev/dsk/c7t0d2
PV Name /dev/dsk/c9t0d2
PV Name /dev/dsk/c12t0d2
PV Name /dev/dsk/c14t0d3
PV Name /dev/dsk/c9t0d3
PV Name /dev/dsk/c12t0d3
PV Name /dev/dsk/c7t0d4
PV Name /dev/dsk/c14t0d4
PV Name /dev/dsk/c12t0d4
PV Name /dev/dsk/c7t0d5
PV Name /dev/dsk/c9t0d5
PV Name /dev/dsk/c14t0d5


--- Volume groups ---
VG Name /dev/vgt24b
VG Write Access read/write
VG Status available
Max LV 2047
Cur LV 1
Open LV 1
Max PV 2048
Cur PV 3
Act PV 3
Max PE per PV 230391
VGDA 6
PE Size (Mbytes) 4
Total PE 230391
Alloc PE 230391
Free PE 0
Total PVG 1
Total Spare PVs 0
Total Spare PVs in use 0
VG Version 2.1
VG Max Size 921564m
VG Max Extents 230391

--- Logical volumes ---
LV Name /dev/vgt24b/lvt24b
LV Status available/syncd
LV Size (Mbytes) 921564
Current LE 230391
Allocated PE 230391
Used PV 3


--- Physical volumes ---
PV Name /dev/disk/disk43
PV Status available
Total PE 76797
Free PE 0
Autoswitch On
Proactive Polling On

PV Name /dev/disk/disk44
PV Status available
Total PE 76797
Free PE 0
Autoswitch On
Proactive Polling On

PV Name /dev/disk/disk45
PV Status available
Total PE 76797
Free PE 0
Autoswitch On
Proactive Polling On


--- Physical volume groups ---
PVG Name pvg1
PV Name /dev/disk/disk43
PV Name /dev/disk/disk44
PV Name /dev/disk/disk45

Disk is presented from EVA4400 storage
Its a database server.

# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 32768 0 32768 0% 0 - 1 /dev/vg00/lvol2
reserve - 893 -893
memory 15520 5859 9661 38%
total 48288 6752 41536 14% - 0 -
#
YES, ITS FROM TEMENOS AND YOU ARE 100%CORRECT,
Pete Randall
Outstanding Contributor

Re: RE: Disk IO shooting to 100%

100% of what? Where are you seeing this output? And what does it really mean? Is it really a problem or does the high number (that we have no frame of reference for) concern you?


Pete

Pete
Tim Nelson
Honored Contributor

Re: RE: Disk IO shooting to 100%

A few thoughts:

Just because a device is at 100% utility does not neccesarily mean there is a performance issue. If a disk is being accessed then what more is expected ?

a RAM disk is certainly ok, but look at a couple things first.

Is there really an issue ?

Keep in mind how the performance gathering works, take a sample wait, take another sample and average. If every time a sample is taken the device is busy then 100% is the number. ( this is usually the case with single devices leading to a SAN).
When devices are busy you are more interested in the response time of the requests ( e.g. disk service time from either sar or glance ).
IF these are poor then sure, start trying to make the device faster by any number of methods, faster HW, striping, data layout, contention reduction etc..

if responses are around 3-5ms on SAN storage then you are not going to get much faster except moving to solid state disk or your RAM disk solution.

Keep in mind there are trade offs with everything. A RAM disk has a very high risk of data loss in the event of a system crash. And, a is a RAM disk that much different from the fs buffer cache mechanism.

Maybe some tuning in the right places could be a better solution ? Maybe moving the data around to reduce contention ? Maybe splitting up of processes ? Maybe ...

Just some things to think about.
Henry Manda
Advisor

Re: RE: Disk IO shooting to 100%


To me there is a problem because the COB process takes longer than usual.
with 15 branches and around 2000 trxns per day and a database size of 200GB, COB should take at most 4 hours, instead its taking 13hrs, and infact Temenos consultants have done all that pertains to performance tuning on the database.
Please Assist me.
Shibin_2
Honored Contributor

Re: RE: Disk IO shooting to 100%

Can we think something out with respect to the OS and application?

T24 released long ago, before HP-UX 11.31 comes out. Even, I worked on it.

Could this be an issue with the database version of jBASE that you are running?
Regards
Shibin
Torsten.
Acclaimed Contributor

Re: RE: Disk IO shooting to 100%

You should always use the new device files (e.g. /dev/disk/diskx) instead of legacy.
Consider to convert!

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Henry Manda
Advisor

Re: RE: Disk IO shooting to 100%

No, Jbase version is ok.

JBCPORTNO : Not Set
JBCRELEASEDIR : '/usr/jbc'
JBCGLOBALDIR : '/usr/jbc'
WARNING: JBCDATADIR is not set, Default '/usr/jbc/jbase_data'
WARNING: JBCDATADIR directory '/usr/jbc/jbase_data' not valid, error 2
WARNING: JBCDATADIR is subdirectory of JBCGLOBALDIR
HOME : '/T24/live/bnk.run'
JEDIFILEPATH : '/T24/live/bnk.run'
JEDIFILENAME_MD : '/T24/live/bnk.run/VOC'
JEDIFILENAME_SYSTEM : '/usr/jbc/src/SYSTEM'
SYSTEM File is (DICT) : '/usr/jbc/src/SYSTEM]D'
RELEASE Information : Major 4.1 , Minor 5.18 , Patch 5711 (Change 56523)
Spooler dir (JBCSPOOLERDIR) : '/usr/jspooler'
JBCEMULATE : 'prime'
Object path (JBCOBJECTLIST) : '/T24/live/bnk.run/locallib:/T24/live/bnk.run/lib:/T24/live/bnk.run/globuslib:/T24/live/bnk.run/GR0800005lib'
jBASE Compiler Run-time : '/usr/jbc/config/system.properties'
Program dir (JBCDEV_BIN) : '/T24/live/bnk.run/localbin'
Subroutine dir (JBCDEV_LIB) : '/T24/live/bnk.run/locallib'
Max open files : 2048


......and its working elsewhere.
Torsten.
Acclaimed Contributor

Re: RE: Disk IO shooting to 100%

Have a look at

/usr/contrib/bin/vgdsf

http://h20000.www2.hp.com/bc/docs/support/SupportManual/c01916036/c01916036.pdf (BSC link updated by admin)

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Henry Manda
Advisor

Re: RE: Disk IO shooting to 100%

my interest is /dev/vgt24b/lvt24b
dont bother about other vgs
TTr
Honored Contributor

Re: RE: Disk IO shooting to 100%

> my interest is /dev/vgt24b/lvt24b

Then you need to provide more details on this volume group. All 3 disks are used by one LV maybe this is not ideal for the type of i/o that you have. Verify that you are using dual/multiple paths for the 3 disks. Also what do the 3 disks translate to in the EVA? How many spindles, raid type etc?
Rita C Workman
Honored Contributor

Re: RE: Disk IO shooting to 100%

I'm going to drop to some serious basics.

One I don't see any device swap.

Two....one nice big logical volume.
This may not be your situation, but I do have to ask since I've seen folks do this, and I never cease to be tickled at what is being done on these kind of setups.

Here's what I've seen....
/fs-24b
/fs-24b/oracle (with all of oracle software
/fs-24b/oracle/logs (with all oracle redo;cntl logfiles, not to mention exports, tracefiles, archivelogs....you name it!!!)

/fs-24b/data (...see a pattern yet...?)
/fs-24b/data/.dbf
/fs-24b/data/.dbf and lots more dbf files

/fs-24b/other-data/tons of textfiles, report files, batch job output files, some developer programs/scripts...you name it...

All the read/write hits; all the querries; all the new information updating; all the report data writing output.....all on the same disk/lvol that is truly being beaten up.

...So....what's under that lvol/fs???

Just wondering...
Rita
ManojK_1
Valued Contributor

Re: RE: Disk IO shooting to 100%

Hi Henry,

Which database you are using? Is it a oracle or jbase itself?

If you are using oracle are you using ASM, lvm raw devices or File systems.

Pleae paste the bdf output.

You are using legacy dsf.

Please paste the autopath display output and verify Load balancing is enabled (Round Robin).
Better to use persistent dsf files in place of legacy device files and it will do a load balancing by default.

COB in t24 should not take this much time with only this much daily transcations and 15 Branches.

Try to involve the application team also for the fine tuning.

What about your application server?

Please take a sar -d 2 20 output during COB and paste it.

Manoj K


Thanks and Regards,
Manoj K