K6 Performance Testing

WIP

“k6 is an open-source load testing tool”

The k6 documentation is amazing and a massive shout out to Chris Alwin James for his github examples and YouTube video.

Test life cycle

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
// 1. init code

export function setup() {
  // 2. setup code
  // Anything returned here is passed into life cycle steps 3 and 4
  // This is useful for getting access tokens
}

export default function (data) {
  // 3. VU code
}

export function teardown(data) {
  // 4. teardown code
}

Terminology

  • VU (Virtual User), an entity that executes a test and makes requests. These are run concurrently and repeat the test over and over.
  • Checks, are like asserts but differ in that they dont halt execution.
  • Thresholds, are global pass/fail criteria for k6 to use, this can fail a load test.

Types of tests

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
//baseline
export let options = {
    stages: [
        { duration: "1m", target: 10 },
        { duration: "10m", target: 10 },
        { duration: "1m", target: 10 }
    ]
};

//load
export let options = {
    stages: [
        { duration: "5m", target: 2000 },
        { duration: "15m", target: 2000 },
        { duration: "5m", target: 0 }
    ]
};

//spike
export let options = {
    stages: [
        { duration: "1m", target: 2000 },
        { duration: "9m", target: 2000 },
        { duration: "3m", target: 10000 },
        { duration: "7m", target: 10000 },
        { duration: "10m", target: 0 }
    ]
};

Metrics & Example test

  • Counter - A metric that cumulatively sums added values.
  • Gauge - A metric that stores the min, max and last values added to it.
  • Rate - A metric that tracks the percentage of added values that are non-zero.
  • Trend - A metric that allows for calculating statistics on the added values (min, max, average and percentiles).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
import http from "k6/http";
import { check, sleep } from "k6";
import { Counter, Gauge, Rate, Trend  } from "k6/metrics";

let ErrorCount = new Counter("foo_error_count");
let ErrorGauge = new Gauge("foo_gauge");
let ErrorRate = new Rate("foo_error_rate");
let ErrorTrend = new Trend("foo_trend");

let statusPercentage = __ENV.STATUS_PERCENTAGE;
let counter = 0;

// Up to 100 looping VUs for 1m over 3 stages
export let options = {
  stages: [
    { duration: "15s", target: 50 }, // ramp up from 0 to 50vu in 50s
    { duration: "30s", target: 50 }, // stay at 50vu for 30s
    { duration: "15s", target: 0 }   // over the last 15s ramp down from 50 to 0vu
  ],
  thresholds: {
    foo_error_count: ["count<10"],    // error count needs to be less than 10, else fail the custom count metric `foo_error_count`
    foo_error_rate: ["rate<0.1"],     // error rate needs to be less than 10%, if its more than 10% then fail the custom rate metric `foo_error_rate`
    http_req_duration: ['p(99)<1100'] // 99% of requests must complete below 1.1s, else fail the built in metric `http_req_duration`
  }
};

export function setup() {
  console.log("XXXXXXXXXXXXXXXXXXXXXXXXXXXXX");
  console.log("STATUS_PERCENTAGE=" + statusPercentage);
  console.log("XXXXXXXXXXXXXXXXXXXXXXXXXXXXX");
}

export default function() {
  const status = Math.random() < statusPercentage ? "200" : "500";

  let response = http.get(`http://httpbin.org/status/${status}`);
  let success = check(response, {
    "status is 200": r => r.status === 200
  });
  counter++;

  if (!success) {
    ErrorCount.add(1);
    ErrorRate.add(true);
  } else {
    ErrorRate.add(false);
  }
  ErrorTrend.add(status);
  ErrorGauge.add(counter); // the last value will be kept

  sleep(0.5);
}

Understanding the metric results

The following are results based on simple-poll.test.js with status percentage set at 0.1 and 0.9

1
2
3
-------------- status percentage 0.1

-------------- status percentage 0.9

Looking at status percentage 0.1 results its a little confusing as the ✗ are the passes and the ✓ are the failures (except for the checks metric!)

Built-in metric

Metric name Comments
checks
http_req_duration
iterations
http_reqs Difference in iterations and http_reqs

Custom metric

Metric name Comments
foo_error_count
foo_error_rate
foo_gauge
foo_trend

Extensions & Typescript support

k6 Browser Recorder can be used to simulate load and generate a baseline test.

JS is great but type safety is great(er) :D