Beyond API latency: Designing high-throughput asynchronous systems

[HPE_Experts]

A fast API response does not guarantee fast processing. Learn how queues, backpressure, circuit breakers, observability, and operation-level metrics help asynchronous systems scale reliably.

Modern enterprise applications increasingly rely on asynchronous processing. A client submits a request, the application validates it, creates a job, places work into a queue, and immediately returns a successful response. Meanwhile, background workers continue processing the actual workload.

This model is widely used for file processing, data pipelines, report generation, artificial intelligence workloads, infrastructure automation, and large-scale batch operations. It provides scalability and responsiveness, but it also creates an important engineering challenge:

A fast application programming interface (API) does not necessarily mean a fast system.

An API may return HTTP 202 Accepted within milliseconds while thousands or millions of operations are still waiting to be completed in the background. Understanding this distinction is critical when designing and performance-testing high-throughput asynchronous platforms.

Understanding the difference between request throughput and work throughput

Consider an application that accepts requests containing large batches of filesystem operations.

A load-testing tool might send 100 requests per second. If each request contains 1000 operations, the system is actually receiving a potential workload of 100,000 operations per second. Measuring only HTTP requests per second would therefore provide an incomplete picture of performance. The more meaningful measurement is: Completed business operations per second. This could represent completed file operations, database transactions, infrastructure actions, events, or any other unit of meaningful work.

For an asynchronous architecture, engineers should ask two separate questions:

• How quickly can the API accept requests?

• How quickly can the system complete the work represented by those requests?

The difference between these two rates determines whether the system is truly keeping up.

The queue is more than a buffer

Message queues are commonly described as mechanisms that decouple producers from consumers. However, they also reveal one of the most important characteristics of a distributed system: the relationship between incoming workload and processing capacity. Assume:

λ = Incoming operation rate

μ = Sustainable processing rate

If λ remains lower than μ, workers can generally keep up with incoming workload. If λ becomes consistently higher than μ, backlog begins to grow.

For example, imagine a platform receiving 12,000 operations per second but completing only 10,000. The difference is 2000 operations every second. After one minute, approximately 120,000 additional operations are waiting. After ten minutes, that backlog can reach 1.2 million operations.

The API itself may still appear completely healthy. Requests may continue receiving successful responses. CPU and memory may even remain within operational thresholds. Yet the system is falling further behind every second. This is why queue depth should be treated as a first-class performance metric rather than simply an implementation detail.

Designing backpressure into the system

Queues absorb short bursts of traffic effectively, but unlimited buffering can create another problem.

If a producer continues submitting work faster than workers can process it, the queue eventually becomes a bottleneck. Backpressure provides a way for the system to respond to this condition. Instead of blindly accepting unlimited work, the application can make decisions based on implementation-side signals such as:

• Queue depth

• Worker utilization

• Processing latency

• Failure rate

• Downstream service availability

• Circuit-breaker state

This creates an important architectural principle:

Processing capacity should be governed by the slowest critical component in the implementation path, not by the fastest component. For example, increasing API concurrency provides little benefit if the downstream filesystem, database, or external service cannot handle the additional workload.

Why tail latency matters

Average response time is another metric that can hide problems. Suppose most operations complete in 20 milliseconds, but a small percentage require several seconds. The average might still appear acceptable. However, in a job involving hundreds of thousands of operations, those slower calls can significantly increase total implementation time. Percentile metrics provide better visibility. Common measurements include:

• P50–typical latency

• P90–latency experienced by 90% of operations

• P95–useful for identifying degradation

• P99–long-tail latency experienced by the slowest operations

Tail latency becomes increasingly important as workload size grows. Even a small percentage of slow operations can become significant when millions of operations are processed.

Circuit breakers and failure isolation

High-throughput systems must also consider what happens when a downstream dependency becomes unhealthy. Without protection, workers may repeatedly call a failing system. Failures then trigger retries. Retries generate additional workload. The already degraded dependency receives even more traffic, potentially creating a failure-amplification loop. A circuit breaker interrupts this pattern. It commonly operates through three states:

Closed–requests flow normally

Open–calls to the unhealthy dependency are temporarily blocked

Half-open–a limited number of requests are allowed to determine whether the dependency has recovered

Once recovery is confirmed, the circuit returns to the closed state. However, the scope of the circuit breaker is equally important. A single global circuit breaker may create an unnecessarily large failure domain. If only one filesystem path, service, or downstream target is unhealthy, unrelated workloads may still be able to continue. A stronger design principle is therefore: Align resilience boundaries with failure boundaries. Where possible, circuit breakers can be scoped to individual downstream targets rather than disabling an entire implementation pipeline.

Avoiding retry storms

Retries are useful for temporary failures, but poorly designed retries can become a source of additional instability.

Consider 10,000 failed operations configured to retry three times. The original 10,000 operations can generate up to 30,000 additional attempts. If thousands of workers retry simultaneously, the recovering dependency may immediately become overloaded again.

A safer retry strategy usually combines:

• Limited retry attempts

• Exponential backoff

• Randomized jitter

• Circuit breakers

Jitter is particularly valuable because it prevents large numbers of workers from retrying at exactly the same time. The goal is not merely to retry failed work. The goal is to retry without amplifying the original failure.

Observability must follow the work

Traditional API monitoring commonly tracks HTTP request rates, response codes, CPU, and memory. These metrics remain valuable, but they are not sufficient for asynchronous systems. Observability should follow the lifecycle of the workload.

Useful metrics include:

• Successfully completed operations per second
• Failed operations per second
• Queue depth
• P50, P90, P95, and P99 operation latency
• Job completion duration
• Worker utilization
• Circuit-breaker state
• Retry volume

Together, these metrics answer a much more important question:

Is useful work actually being completed at the expected rate?

Load testing for system behavior

Performance testing should not only determine whether an application survives a target load. It should reveal how the system behaves as load increases.

A useful approach is to gradually increase workload through several stages while continuously observing throughput, latency, failures, queue depth, and infrastructure utilization. Eventually, the system may reach a point where additional incoming workload no longer produces proportional increases in completed operations. Instead:

• Latency rises rapidly
• Queue depth grows continuously
• Failure rates increase
• Resource contention appears
• Circuit breakers may begin opening

This point represents the practical capacity boundary of the current architecture. Understanding where and why that boundary occurs provides far more engineering value than simply knowing whether a service remained available during a load test.

Engineering for completion, not acceptance

Asynchronous architectures enable enterprise applications to process large workloads while keeping client-facing APIs responsive. But they also require engineers to look beyond traditional API metrics. A successful request means that the system accepted the work. It does not necessarily mean that the work was completed quickly, successfully, or sustainably.

For high-throughput platforms, system health must therefore be measured across the complete implementation pipeline—from request acceptance to queueing, processing, dependency interaction, retries, resilience controls, and final completion.

The most important performance question is not:

How quickly did the API respond?

It is:

How reliably can the platform complete useful work at scale?

That distinction becomes increasingly important as modern applications move toward asynchronous, distributed, and event-driven architectures. Because accepting work is easy, completing it reliably under pressure is where the real engineering begins.

Learn how HPE application engineering and modernization services can help organizations design resilient, scalable, and cloud-native applications for demanding enterprise workloads.

Meet the author:

Nachammai Nagappan, Cloud Developer – Application Engineering, HPE
Linkedin.com/in/nachammai-nagappan