Outages caused when system activities exceed the system’s capacity is a leading concern in system design. The ability to handle system activity efficiently, and gracefully limit the execution of activities before the system is under stress is a fundamental to system resiliency. .NET does not have a standardized means for expressing and managing resource limiting logic needed to produce a resilient system. This adds complexity to designing and developing resilient software in .NET by introducing an easy vector for competing resource limiting logic and anti-patterns. A standardized interface in .NET for limiting activities will make it easier for developers to build resilient systems for all scales of deployment and workload.
Users will interact with the proposed APIs in order to ensure rate and/or concurrency limits are enforced. This abstraction require explicit release semantics to accommodate non self-replenishing (i.e. concurrency) resource limits similar to how Semaphores operate. The abstraction also accounts for self-replenishing (i.e. rate) resource limits where no explicit release semantics are needed as the resource is replenished automatically over time. This component encompasses the TryAcquire/AcquireAsync mechanics (i.e. check vs wait behaviours) and default implementatinos will be provided for select accounting method (fixed window, sliding window, token bucket, simple concurrency). The return type is a Resource type which manages the lifecycle of the aquired resources.
public interface IResourceLimiter
{
// An estimated count of resources.
long EstimatedCount { get; }
// Fast synchronous attempt to acquire resources.
// Set requestedCount to 0 to get whether resource limit has been reached.
bool TryAcquire(long requestedCount, out Resource resource);
// Wait until the requested resources are available.
// Set requestedCount to 0 to wait until resource is replenished.
// An exception is thrown if resources cannot be obtained.
ValueTask<Resource> AcquireAsync(long requestedCount, CancellationToken cancellationToken = default);
}
public struct Resource : IDisposable
{
private Action<Resource>? _onDispose;
// This represents additional metadata that can be returned as part of a call to TryAcquire/AcquireAsync
// Potential uses could include a RetryAfter value.
public object? State { get; init; }
public Resource(long count, object? state, Action<Resource>? onDispose)
{
State = state;
_onDispose = onDispose;
}
public void Dispose()
{
// TODO: Ensure Dispose will no-op on subsequent invocations
_onDispose?.Invoke(this);
}
// This static field can be used for rate limiters that do not require release semantics or for failed concurrency limiter acquisition requests.
public static Resource NoopResource = new Resource(null, null);
}
public static class ResourceLimiterExtensions
{
public static bool TryAcquire(this IResourceLimiter limiter, out Resource resource)
{
return limiter.TryAcquire(1, out resource);
}
public static ValueTask<Resource> AcquireAsync(this IResourceLimiter limiter, CancellationToken cancellationToken = default)
{
return limiter.AcquireAsync(1, cancellationToken);
}
}The struct Resource is used to facilitate the release semantics of resource limiters. That is, for non self-replenishing, the returning of the resources obtained via TryAcquire/AcquireAsync is achieved by disposing the Resource. This enables the ability to ensure that the user can't release more resources than was obtained.
For components enforcing limits, the standard usage pattern will be:
if (limiter.TryAcquire(1, out var resource))
{
// Resource obtained successfully.
using (resource)
{
// Continue with processing
// Resource released when disposed
}
}
else
{
// Limit exceeded, no resources obtained
}In cases where it is known that the resource limiter is a rate limit with no-op release semantics, the usage can be simplified to:
if (limiter.TryAcquire(1, out _))
{
// Resource obtained successfully.
// Continue with processing
}
else
{
// Limit exceeded, no resources obtained
}This API will be useful in implementing limits for various BCL types including:
- Channels
- Pipelines
- Streams
- HttpClient
For example, a rate limit applied to a BoundedChannel:
// Rate limiter added to options
var rateLimiter = new FixedWindowRateLimiter(resourcePerSecond: 5);
var rateLimitedChannel = Channel.CreateBounded<string>(new BoundedChannelOptions(5) { WriteRateLimiter = rateLimiter });
// This channel will now only write 5 times per second
rateLimitedChannel.Writer.TryWrite("New message");Ongoing experiments in ASP.NET Core for application in Kestrel server limits and a middleware for enforcing limits on request processing is ongoing at https://github.com/dotnet/aspnetcore/tree/johluo/rate-limits.
We also adoption for enforcing limits in YARP as well as conversion of existing implementations in ATS and ACR.
Major variants considered
A design where rate limits and concurrency limits were expressed by separate abstractions was considered. The design more clearly express the intended use pattern where rate limits do not need to return a Resource and does not possess release semantics. In comparison, the proposed design where the release semantics for rate limits will no-op.
However, this design has the drawback for consumers of resource limits since there are two possible limiter typess that can be specified by the user. To alleviate some of the complexity, a wrapper for rate limits was considered. However, the complexity of this design was deemed undesirable and a unified abstraction for rate and concurrency limits was preferred.
Instead of using the Resource struct to track release of resources an alternative approach proposes adding a void Release(long releaseCount) method on IResourceLimiter and require users to call this method explicitly. However, this requires the user to call release with the correct count which can be error prone and the Resource approach was preferred.
This approach allows for subclassing to include additional metadata instead of an object? State property on the struct. However, it was deemed that potentially allocating a new Resource for each acquisition request is too much and a struct was preferred.
Currently, the acquisition and release of resources is all-or-nothing.
Addtional APIs will be needed to allow for the ability to acquire a part of the requested resources. For example, 5 resources were requested but willing to accept a subset of the requested resources if not all 5 is available.
Similarly, additional APIs can be added to Resource to facilitate the release a part of the acquired resource. For example, 5 resources are obtained, but as processing continues, each resource can be released sequentially.
These APIs are not included in this proposal since no concrete use cases has been currently identified.
This is a proposal for new API and main concerns include:
- Consumption patterns of resource limiters should be simple and idiomatic to prevent pitfalls.
- The default rate and/or concurrency limiters should suffice in most general use cases.
- The abstraction should should be expressive enought to allow for customized resource limiters.