TCP Socket Action Probe In Worker (Liveness)

TCP Socket Action Probe In Worker

Motivation

The examples in this post are for a TCP health check see Health Checks for detailed motivation and alternatives like HTTP.

“Kubernetes relies on probes in your application to assess whether your application is healthy.” - dzone.com

If the worker instance is un-healthy then K8s will tear it down and spin up a new instance.

Code Example

  1. Install the following package : Microsoft.Extensions.Diagnostics.HealthChecks

  2. Create a simple health check that implements IHealthCheck and always returns healthy.

1
2
3
4
5
6
7
8
9
10
11
using Microsoft.Extensions.Diagnostics.HealthChecks;

namespace MyWorker.HealthChecks;

public class HealthCheck : IHealthCheck
{
    public Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context, 
        CancellationToken token = default) =>
            Task.FromResult(new HealthCheckResult(HealthStatus.Healthy));
}
  1. Create a new background service to accept the TCP socket liveness probe, this code was adapted from dzone.com and is intentionally simple without logging and error handling. So turn piss off Karen 😛
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
public class TcpLivenessProbeService : BackgroundService
{
    private readonly TcpListener _listener;
    private readonly HealthCheckService _service;

    public TcpLivenessProbeService(HealthCheckService service)
    {
        _listener = new TcpListener(5000);
        _service = service;
    }

    protected override async Task ExecuteAsync(CancellationToken token)
    {
        _listener.Start();
        while (!token.IsCancellationRequested)
        {
            await UpdateHeartbeatAsync(token);
            await Task.Delay(TimeSpan.FromSeconds(1), token);
        }
        _listener.Stop();
    }

    private async Task UpdateHeartbeatAsync(CancellationToken token)
    {
        var result = await _service.CheckHealthAsync(token);
        var isHealthy = result.Status == HealthStatus.Healthy;

        if (!isHealthy)
            return;

        while (_listener.Server.IsBound && _listener.Pending())
        {
            using var client = await _listener.AcceptTcpClientAsync(token);
        }
    }     
}
  1. Now resolve the dependencies in Program.cs
1
2
3
// Liveness check
services.AddHealthChecks().AddCheck<HealthCheck>("liveness");
services.AddHostedService<TcpLivenessProbeService>();
  1. You will then need to configure the liveness probe in K8s to query port 5000
1
2
3
4
5
6
7
8
9
spec:
    containers:
        ports:
            - name: liveness-check
              containerPort: 5000
              hostPort: 5000
        livenessProbe:
            tcpSocket:
                port: liveness-check

References