slow performance on logical volumes

[msmalik0101]

its a SUSE 11 SP2 xen virtual machine with 8 GB Memory, kernel version 3.0.58-0.6.6-xen #1 SMP

we recently changed the traditional/static partitioning(/dev/xvdc{1,2,3} to lvm based volumes, and after that we are observing serious performance issues. e.g earlier(when we were using static/traditional partitioning) backup took only 45 minutes, now backup takes 2 hours or more. Also our end users are complaining about performance too.

following are storage details:
df -hT
Filesystem                                      Type      Size  Used Avail Use% Mounted on
/dev/xvda3                                      ext3       47G  7.6G   38G  17% /
devtmpfs                                        devtmpfs  4.1G  140K  4.1G   1% /dev
tmpfs                                           tmpfs     8.0G  179M  7.9G   3% /dev/shm
/dev/xvda1                                      ext3      243M   24M  207M  11% /boot
/dev/mapper/VG--ORG--ORACLE-sap--oracle         ext3      245G  222G   11G  96% /oracle
/dev/mapper/VG--ORG--ORACLE-sap--ora--sapmnt    ext3       15G  1.7G   13G  12% /sapmnt
/dev/mapper/VG--ORG--ORACLE-sap--ora--usrsap    ext3       30G   24G  4.3G  85% /usr/sap

# pvscan
  PV /dev/xvdc    VG VG-ORG-ORACLE   lvm2 [350.00 GiB / 7.00 GiB free]

# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "VG-ORG-ORACLE" using metadata type lvm2

# /sbin/lvs
  sap-ora-sapmnt    VG-ORG-ORACLE -wi-ao  15.00g                                      
  sap-ora-usrsap    VG-ORG-ORACLE -wi-ao  30.00g                                      
  sap-oracle        VG-ORG-ORACLE -wi-ao 248.00g                                      

running "hdparm -t" on the raw disk /dev/xvdc returns far better results then running "hdparm -t" on logical volumes

/dev/xvdc:
 Timing buffered disk reads:  742 MB in  3.00 seconds = 247.31 MB/sec

/dev/xvdc:
 Timing buffered disk reads:  670 MB in  3.00 seconds = 223.31 MB/sec

/dev/xvdc:
 Timing buffered disk reads:  718 MB in  3.00 seconds = 239.23 MB/sec
 [/code]

while running "hdparm -t" on logical volume returns very slow results:

/dev/mapper/VG--ORG--ORACLE-sap--oracle:
 Timing buffered disk reads:  148 MB in  3.01 seconds =  49.17 MB/sec

/dev/mapper/VG--ORG--ORACLE-sap--oracle:
 Timing buffered disk reads:  192 MB in  3.10 seconds =  62.02 MB/sec

/dev/mapper/VG--ORG--ORACLE-sap--oracle:
 Timing buffered disk reads:  154 MB in  3.03 seconds =  50.87 MB/sec

/dev/mapper/VG--ORG--ORACLE-sap--oracle:
 Timing buffered disk reads:  184 MB in  3.03 seconds =  60.77 MB/sec
 
Please help and suggest

[msmalik0101]

also for your kind information

pvdisplay
  --- Physical volume ---
  PV Name               /dev/xvdc
  VG Name               VG-ORG-ORACLE
  PV Size               350.00 GiB / not usable 4.00 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              89599
  Free PE               1791
  Allocated PE          87808
  PV UUID               jhypNN-fPpy-2uzc-nsxP-qZIl-9JAg-O9aenE
 
 
vgdisplay
  --- Volume group ---
  VG Name               VG-ORG-ORACLE
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  6
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                4
  Open LV               4
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               350.00 GiB
  PE Size               4.00 MiB
  Total PE              89599
  Alloc PE / Size       87808 / 343.00 GiB
  Free  PE / Size       1791 / 7.00 GiB
  VG UUID               MSnb2T-A5Pf-3i7B-5sz2-f10M-2T2s-uCbdJn
 
lvdisplay
  --- Logical volume ---
  LV Name                /dev/VG-ORG-ORACLE/sap-ora-sapmnt
  VG Name                VG-ORG-ORACLE
  LV UUID                mSM4Mg-jxoV-phr5-8p5S-i5XY-VSbN-p6McFk
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                15.00 GiB
  Current LE             3840
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     1024
  Block device           253:0

  --- Logical volume ---
  LV Name                /dev/VG-ORG-ORACLE/sap-ora-usrsap
  VG Name                VG-ORG-ORACLE
  LV UUID                zbwf4e-E3HL-6Vnw-ayXK-19PH-Dtir-uqi52g
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                30.00 GiB
  Current LE             7680
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     1024
  Block device           253:1

  --- Logical volume ---
  LV Name                /dev/VG-ORG-ORACLE/sap-oracle
  VG Name                VG-ORG-ORACLE
  LV UUID                4zt2BF-BhIJ-a4I7-mr9i-KVBk-c6Dt-BvztDQ
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                248.00 GiB
  Current LE             63488
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:2

  --- Logical volume ---
  LV Name                /dev/VG-ORG-ORACLE/oracle-sap-backup
  VG Name                VG-ORG-ORACLE
  LV UUID                vXL33N-CaUY-ILsT-YwqE-Hn0Z-xLRV-Pm2aRS
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                50.00 GiB
  Current LE             12800
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     1024
  Block device           253:3

[msmalik0101]

I think I have found the solution.

our XEN hypervisor(dom0) is using LVM, and we are also using lvm on our virtual machines(domU).. which is not recommend

as per https://www.suse.com/support/kb/doc.php?id=3559698

Only use Logical Volume Manager (LVM) on Dom0 or DomU, but not both

Regards