SLES + HP DL380: poor performance with cciss driver?

[A.Brandt]

Hello!

At the moment i'm configuring my Service Guard Cluster. Everything work fine, but the io performance seems to be very low.
The problems occurs when using the postgres database. When I import an 1,1GB sql dump, the postmaster process takes about 100% cpu usage n one processor. The problem is, that only 5-10% of these 100% are user or system usage. 90-95% are wait io... so i think, that i have a big problem with the scsi drivers... does anybody know this problem?

In order to see, where the problem might be, i tested the harddisk with hdparm, but i get an error:

/dev/cciss/c0d0:
Timing buffered disk reads: 456 MB in 3.02 seconds = 151.12 MB/sec
BLKFLSBUF failed: Inappropriate ioctl for device

The System is running on an HP Package Cluster ( HP DL380 with Smart Array 500 ) 2x 3,4Ghz Xeon with 6GB Ram per node .

OS is SLES - without any service packs yet..

[Steven E. Protter]

The problem is, that only 5-10% of these 100% are user or system usage. 90-95% are wait io... so i think, that i have a big problem with the scsi drivers... does anybody know this problem?

I don't think the CPU usage for I/O is a problem. Most of what the system is doing is I/O, so it makes sense that the CPU is working hard to process the data.

You might find a tool like Glance for Linux helps you figure out if there is a disk performance problem.

Please post more informaiton about your disk layout and raid setup. If you are raid 5, you should expect poor perfromance during large/heavy writes because many copies of the data have to be written at the same time.

SEP

[A.Brandt]

The internal Harddisks ( 15k rpm, 36GB) are running as an Raid1 at the Moment.
3 disks at the Smart Array are configured as Raid5.

I used bonnie to test the filesystem performance. This is what bonnie tells me:

ClusterNode2:~ # bonnie -s 1999 -v 4
Bonnie 1.4: File './Bonnie.31398', size: 2096103424, volumes: 4
Writing with putc()... done: 43350 kB/s 95.9 %CPU
Rewriting... done: 47017 kB/s 14.1 %CPU
Writing intelligently... done: 91943 kB/s 32.1 %CPU
Reading with getc()... done: 48360 kB/s 88.8 %CPU
Reading intelligently... done: 160632 kB/s 19.4 %CPU
Seek numbers calculated on first volume only
Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
---Sequential Output (nosync)--- ---Sequential Input-- --Rnd Seek-
-Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --04k (03)-
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
Cluste 4*1999 43350 95.9 91943 32.1 47017 14.1 48360 88.8160632 19.4 786.3 1.6

Seems to be okay, right? I've nothing to compare with :(

To come back to the io. My Problem is, that a postgres database (8.0.3), which should run on the server, is very slow.
On old Server ( Celeron 2Ghz, 2GB Ram ) it takes 10 minutes to import an 1,1GB sql dump. On out new DL380 G4 it take ~1,5h.
Select Statements are also very slow... something that takes 3 Seconds on the Celeron will take more than 30 oder 40 seconds on the DL380...
And i absolutely don't know why :(

[David Claypool]

We've noticed some things 'broken' in PostgreSQL 8. You might fall back to a prior version if you can.

Some additional information that you might find useful in our practical experiences with HP SIM.

We have a 'rule-of-thumb' guideline for customers that is truly a YMMV, but does illustrate the issue a bit. With HP SIM 5.0, on a system with recommended configuration for Windows (1.5GHz, 2GB RAM, 10K/Ultra3 drives in RAID 0), 'acceptable' (totally subjective, I know, but this is a practical measurement) performance with MSDE is up to approximately 500 devices and 5,000 events. That same system with MS SQL 2000 raises the bar to 5,000 devices and 50,000 events.

A similarly configured system using HP SIM on RHEL or SLES with PostgreSQL starts becoming 'painful' to use at 300 devices.

Interestingly enough, we also support PostgreSQL for HP SIM when hosted on HP-UX, and the number of devices isn't markedly different in that case, even when running on a 64-bit Itanium.

Because of that, we have most recently qualified Oracle as an option for Windows, Linux and HP-UX.

[Franco_24]

Hi,

I got the exact same issue, but with OpenLDAP on my DL380 G4
P4 3GHz CPU
1Gb RAM
Linux
CCISS driver v.2.6.6

I encounter this while adding a lot of entries in my LDAP tree.

I recently upgraded to the last SmartArray6i firmaware without any improvements

Upgrading from kernel 2.6.12.4 to 2.6.13 made things work a little faster.

Any idea ?

[Franco_24]

Hi again,

In the meanwhile I found this very interesting google thread:
http://groups.google.ch/group/pgsql.performance/browse_frm/thread/545d77410e24a2a6/e2aa4e32e124671a

It seems the solution would be to buy another controller :\