HTTP Retry with Polly

Polly allows http retries with exponential backoff so if the resource you are trying to reach throws a transient error (an error that should resolve itself) like 500 (Server Error) or 408 (Request Timeout) then the request will auto-magically be re-tried x times with an increased back-off (the period between re-tries) before giving up.

We can include 404 (Not Found) but that depends on the use case, in some API’s 404 means the data you were looking for is not avalible. Example if GET /person/1 responded in 404 it COULD mean 1 doesnt exist but the resource is still there.

Polly can also do other cool things listed below but I’ll focus on simple retry.

  • Circuit Breaker
  • Fallback
  • Timeout
  • Bulkhead Isolation

Create the retry policy

  1. Install nuget Microsoft.Extensions.Http.Polly

  2. In the DI container set the handler to be applied to the injected http client, this will be avalible to the constructor of FooService. The microsoft example also sets .SetHandlerLifetime(TimeSpan.FromMinutes(5)).

1
2
3
//ConfigureServices()  - Startup.cs
services.AddHttpClient<IFooService, FooService>()
        .AddPolicyHandler(GetRetryPolicy());
  1. Define the handler which will cater for
  • Retry policy for 5xx, 408 and 404.
  • Retry based on RetryCount and exponentially back-off by retryAttempt starting at RetrySleepDuration.

Here onRetryAsync is passed a deligate inline method that just writes out a message. This was helpful when manually testing my worker as its a console application. 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
public static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy()
{
    var retryCount = 3;
    var retrySleepDuration = 2;

    return HttpPolicyExtensions
        .HandleTransientHttpError()
        .OrResult(msg => msg.StatusCode == System.Net.HttpStatusCode.NotFound)
        .WaitAndRetryAsync(
            retryCount, 
            retryAttempt => TimeSpan.FromSeconds(Math.Pow(retrySleepDuration, retryAttempt)),
            onRetryAsync: (_, _) =>
            {
                Console.WriteLine("GetRetryPolicy retrying...");
                return Task.CompletedTask;
            });
}

Building on WaitAndRetryAsync => onRetry/onRetryAsync

The 3rd parameter of onRetry is an int which represents retryAttempt, this can be added to logs.

I also wasnt sure on the value of using async when its just a log, so I switched it out.

1
2
3
4
5
6
7
.WaitAndRetryAsync(
    retryCount,
    retryAttempt => TimeSpan.FromSeconds(Math.Pow(retrySleepDuration, retryAttempt)),
    onRetry: (_, _, retryAttempt, _) =>
    {
        Console.WriteLine($"Retry attempt ({retryAttempt} of {retryCount})");
    });

Unit tests

You may be tempted to create additional infastructure and unit test an injected HttpClient with mocked out http responses but its simpler to just unit test the extension method. This can be done with a simple DummyMethod that keeps track of its invocations and has a sorted and predefined collection of response http status codes.

For this test the following should be true per invocation

  • 4th => HttpStatusCode.OK (200)
  • 3rd => HttpStatusCode.NotFound (404)
  • 2nd => HttpStatusCode.RequestTimeout (408)
  • 1st => HttpStatusCode.InternalServerError (500)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
public class PollyExtensionTests
{
    private int _invocationCount;

    private readonly HttpStatusCode[] _httpStatusCodes = {
            HttpStatusCode.InternalServerError,
            HttpStatusCode.RequestTimeout,
            HttpStatusCode.NotFound,
            HttpStatusCode.OK
        };

    public PollyExtensionTests()
    {
        _invocationCount = 0;
    }

    private Task<HttpResponseMessage> DummyMethod()
    {
        _invocationCount++;

        return _httpStatusCodes.Length >= _invocationCount
            ? Task.FromResult(new HttpResponseMessage(_httpStatusCodes[_invocationCount - 1]))
            : Task.FromResult(new HttpResponseMessage(HttpStatusCode.OK));
    }

    [Theory]
    [InlineData(3, HttpStatusCode.OK)]
    [InlineData(2, HttpStatusCode.NotFound)]
    [InlineData(1, HttpStatusCode.RequestTimeout)]
    [InlineData(0, HttpStatusCode.InternalServerError)]
    public void GetRetryPolicy_Retries_Transient_And_NotFound_Http_Errors(int retryCount, HttpStatusCode expectedHttpStatusCode)
    {
        // Arrange
        var httpResponseMessage = PollyExtension.GetRetryPolicy();

        //Act
        var result = httpResponseMessage.ExecuteAsync(() => DummyMethod()).Result;

        //Assert
        result.StatusCode.Should().Be(expectedHttpStatusCode);
    }
}

References