Exchange 2003 Performance - Avg Disk Queue Length

[Jon White]

If this is not the correct forum - my apologies. I am just running out of troubleshooting steps to try.

I have Exchange 2003/SP2 on Windows 2003 baseline running on a ML370 G3 with dual 3.2GHz processors and 4GB of RAM.

Recently, the Outlook 2003 clients are getting the "Outlook is retrieving data from the Exchange server" message VERY frequently.

At the time of these delays, the Avg Disk Queue Length on the Exchange server's Performance Monitor spikes for the E: / Database (PhysicalDisk counter) drive. However, the Avg Disk/sec Read and Avg Disk/sec Write on the E: drive remain low.

I used the 7.40 firmware CD to update the firmware on the hard drives which the POST was alerting me to.

The HP Array Diagnostics is not reporting any errors.

I have downloaded the 7.51 Support Pack and updated the drivers. The Exchange Best Practices Utility pointed out that the array drivers were over two years old.

I updated 1 or 2 of the registry memory settings which the EBPU also recommended.

With each of these changes, performance improves for 1 to 4 days. After that, the Outlook clients start getting the message again.

Note - the Public Folders for the entire organization are homed on this server with partial replication to two remote Exchange servers in the organization. There are plans to move these to a separate server, but this won't happen any time soon.

At this point, I am wondering if it might be bad (or potentially bad) hardware. Any better diagnostics tools for determining this?

I am also open to ANY suggestions [software or hardware] as this has been going on for 3+ weeks now.

[Igor Karasik]

Jon,
several questions:
-do you use Outlook 2003 cached mode ?
-which SmartArray controller you have on ML 370?
-do you use 10K SCSI drives or 15K SCSI drives?
-how many concurrent exchange users you have on the server?

Did you run Microsoft Exchange Server Performance Troubleshooting Analyzer Tool?
http://www.microsoft.com/downloads/details.aspx?FamilyID=4bdc1d6b-de34-4f1c-aeba-fed1256caf9a&DisplayLang=en

P.S. Did you read these articles
http://support.microsoft.com/?id=892764
http://support.microsoft.com/?id=839862

[Alan_152]

How big is your swap file, and is it contiguous? Does your exchange server run any other services? What are the speeds of your disks, and are they in any kind of RAID configuration? How much free space do you have on your disks?

[Jon White]

Most of our clients are using Cached mode. I am using "classic" mode of an ost file and offline Outlook usage. All systems are receiving the notification.

The ML370 has a 6400 Smart Array controller.

The drives are 15K rpm drives.

We have about 200 users on the Exchange server.

The MESPTAT tool reported a few items. I updated a registry setting on one the Heap memory settings. It also flagged the array drivers being over two years old which have already been updated.

I have been working with both articles that were mentioned.

I have moved the page file to two different volumes and set it for System Managed. The adjustments have not made any difference.

The Exchange server does not provide any other services. It is running Diskeeper, antivirus and a backup agent. None of these are active during the day when the problem is showing up.

The system drive and database log logical partitions are on a RAID1 in drive cage A. The database drive is on a RAID5 array.

The system partition has 5GB of free space, the log partition has 10+ GB of space while the database drive has 130+ GB on it.

[Alan_152]

All this sounds good to me. I'd move the swap file back to c:\ and set it to min 4095, max 4095 instead of system managed. Then give c:\ a good defrag.

[Jon White]

The page file is back on C:, but it is set to System Managed.

The drive, queue and IO counters that are pegging are all on the E: drive - the database store. The C: drive (system) and D: drive (logs) are both very low utilization.

[Rune J. Winje]

- have you tried offline defragment of the database(s) ? (Remember full backup first).
- have you ruled out WINS, DNS, "ping" problems (ICMP echo is used by clients to determine network speed - no response and the client/server can authenticate slowly or assume a slow network...)
- have you tried sp2 for Exchange 2003 (more efficient Offline Address book download etc)
- do you have a baseline performance before the problems occured that shows the disk queue lenght was less?

http://support.microsoft.com/kb/918006/en-us might be worth looking into (unresponsive GC and/or authentication problems).


Cheers,
Rune

[Jon White]

- have you tried offline defragment of the database(s) ? (Remember full backup first).

It has been a while. ESEUTIL and ISINTEG could probably both stand to be completed.


- have you ruled out WINS, DNS, "ping" problems....

It is not a name resolution issue. WINS not used. DNS lookups work fine. I have used WildPacket's EtherPeek product to do network analysis. Network utilization to the Exchange server is 2 to 5 percent. Very low.


- have you tried sp2 for Exchange 2003...

Already running Exchange 2003 Service Pack 2.


- do you have a baseline performance before the problems occured that shows the disk
queue lenght was less?

I have opened a case with Microsoft support. They have have me going through the article that has the ExBPA, ExPTA, MSRpt utility, etc. I am currently running a 2 hour capture on the ExPTA monitoring right now during the morning workload - when the Outlook balloon tends to show up the most.


I am running my own diags on Avg Disk Queue Length and several disk I/O counters. The ADQL has quieted down quite a bit over the past two weeks - no changes on my part to cause that.
The disk I/O counters are bouncing like ping pong balls though.

[Paul Slijkhuis]

I wonder if the server itself is the real bottleneck. It may be a network performance issue when you say that the Exchange server is low on network utilisation.

I usually get these messages when running on a VPN tunnel from at home. Or when network traffic is high in general.

I also wonder what kind of disks you are using (? RPM, ? RAID) to put things in perspective.

h2h
- Paul

[GBH]

Jon
Did you ever get this resolved? We are having the same issue and would be interested in what you came up with.

Thanks

[Jon White]

I have not checked Performance Monitor on the Disk Queues for the server. However, I think the problem was enhanced by another issue we had going on.

The Outlook clients were getting the balloon of 'Contacting ' or 'Contact '. I ran Exchange Performance Troubleshooting Analyzer on the Exchange server. It rebooted LDAP latency to the domain controllers. The 2nd domain controller was about 495ms (should be less than 100) while the 1st domain controller was 11950ms (way over 100ms).

I transferred all roles from the DC1 to DC2. I then demoted DC1 (which I suspected to be corrupt or failing hardware). I decommissioned the server and rebuilt it from the beginning. I added it back to the domain. I did NOT have the server take back its roles. However, the Exchange server is using it as its main GC. ExPTA is now reporting almost no LDAP latency (the most DC1 has is 109ms which I can live with while DC2 has no latency over 100ms). The Outlook clients are also not getting the balloons (very infrequently now if at all).

I will have to go back and check the PerfMon on this Exchange server and see what impact the rebuild of the DC had on it.

[GBH]

Thanks Jon ... I'll let you know what we find

GBH