Performance drop in storage blade over time.

[Small Clanger]

Hi All,

We've just completed a migration of services form a G1 BL460c to a G6 of the same model; moving from Debian Etch to Debian Lenny in the process.

In both servers, the OS was installed to a mirror set on the blade's internal drives. After prepping the new server in advance, we switched the attached storage blade across on a weekend.

The storage blade is set up with as 6-disk RAID6 array using 150GB SAS drives for around 560GB usable space. It's set up with a single partition and was/is mounted as /var.

The migration went well. We merged the various parts of /var in single-user mode and the server booted on the second go (For some reason, the controller IDs are switched on the new server, so I had to alter the root= lines in the grub config.)

While in single user mode, however, I ran the hdparm as a rough test of the throughput on the storage blade and found it was running a sequential transfer of around 180MB/s

When I ran these benchmarks around 2 years ago on the fresh server, the same test gave me around 450MB/s.

Though the server still seems to be performing well enough under its normal load, I'm concerned about this drop off in performance.

I think it's more likely theat the drop-off was happening slowly over a couple of years rather than as a result of the upgrade, but I wanted to ask if anyone else had seen a similar problem and if there's anything that can be done about it?

Cheers,

Terry.

[Steven E. Protter]

Shalom,

A lot can change over time.

Check for memory leaks.

http://www.hpux.ws/?p=8

Check with the application people, maybe some new program is reading data that needs to be indexed.

SEP

[Small Clanger]

Hi Stephen,

I'd considered some application problem, but the server had been booted for all of 30 seconds and was running in single-user mode. There was nothing running but my shell. Also, The server has 12GB of RAM and was unlikely to be paging so soon into the boot (and the swap partition is on the other array).

The filesystem is JFS and has seen a lot of use over time, but hdparm performs block-level reads on the device, rather than filesystem reads, so I'm not sure that has any effect either.

Over this timeframe I'd expect raw disk performance to drop maybe 10-15%. It's what rotational disks do, but a 60% drop in two years seems too much.