New Relic With Terraform

Updated 17/04/2025

“Terraform is a popular infrastructure-as-code software tool built by HashiCorp. You use it to provision all kinds of infrastructure and services, including New Relic dashboards and alerts.” - docs.newrelic.com

In this post I will create an Error New Relic alert via Terraform, I see this Error Rate as Availability and the resulting New Relic resources would be a provider, alert policy which is the parent, alert condition which are children to the parent and alert trigger

Four Golden Signals

For your alerts you need to think about whats sensible to alert on, Goole SRE is the golden standard, because you know… its Google :D … so you can re-invent the wheel or learn from their Golden Signals:

High Level Commands & Flow

Required Terraform Config

The Newrelic docs have a great example which I based the below on, you will notice they are also focused on the Goole SRE Golden Signals

The high level config that I used to create my .tf files are

  • providers.tf
  • alerts.tf
    • alert policy (this is the parent)
    • alert conditions (these are the children)
    • alert triggers
  • locals.tf (optional)
  • variables.tf (optional)
Additional abstractions

Locals

Locals are like constants in terraform, example locals.tf file with property newrelic_account_id

1
2
3
locals {
  newrelic_account_id = "123456789"
}

To access newrelic_account_id in another file like providers.tf

1
2
3
provider "newrelic" {
  account_id = local.newrelic_account_id
}

Variables

1
2
3
4
5
6
7
8
9
10
11
variable "environment" {
  type = string
}

variable "newrelic_slack_channels" {
  type = map(string)
  default = {
    uat  = "X05LBNSKMHU"
    prod = "X05LIBNLBYJ"
  }
}

I set the environment in prod.tfvars, uat.tfvars files, example value environment = "uat"

You can access the map newrelic_slack_channels in a file, example locals.tf

1
2
3
locals {
  newrelic_slack_channel_id = var.newrelic_slack_channels[var.environment]
}

Example Setup

  1. Install Terraform

Chocolatey makes this easy for windows users like me … sudo apt-get this 🖕

1
2
choco install terraform
terraform -version
  1. Create your initial config with providers.tf and locals.tf

providers.tf

1
2
3
4
5
6
7
8
9
10
11
12
13
terraform {
  required_version = "~> 1.9.8"
  required_providers {
    newrelic = {
      source  = "newrelic/newrelic"
    }
  }
}

provider "newrelic" {
  account_id = local.newrelic_account_id
  api_key    = local.api_key
}

locals.tf

You can get this from your NR account under Administration -> API Keys

1
2
3
4
locals {
  newrelic_account_id = "0000000"
  api_key             = "NRAK-00000000000000000000000000"
}
  1. Run terraform init which will initialize the backend and provider plugins, mine created these files/folder locally. I manually modified my locals.tf with my account id and api key.

I then also updated my .gitignore with the config below

1
2
.terraform/*
.terraform.lock.hcl
  1. Create Alert Policy (parent) and Alert Condition (children) inside alerts.tf, here the alert condition is selecting from Transaction which would be for web traffic originating from a NR client.

alerts.tf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
// *** Alert policy

resource "newrelic_alert_policy" "hoonapi_alert_policy" {
  name  = "Hoon API Alert Policy ${var.environment}"
}

// *** Alert conditions

resource "newrelic_nrql_alert_condition" "error_rate" {
  name                         = "Golden Shower Error Rate"
  account_id                   = local.newrelic_account_id
  policy_id                    = newrelic_alert_policy.hoonapi_alert_policy[count.index].id
  type                         = "static"
  enabled                      = true
  violation_time_limit_seconds = 259200
  runbook_url                  = "https://carlpaton.github.io/2023/01/newrelic/"

  nrql {
    query = "FROM Transaction SELECT percentage(count(*), WHERE `response.status` LIKE '5%') WHERE appName in ('NewRelicHoon') AND request.uri NOT IN ('/foo', '/bar')"
  }

  critical {
    operator              = "above"
    threshold             = 2
    threshold_duration    = 300
    threshold_occurrences = "all"
  }

  fill_option        = "none"
  aggregation_window = 60
  aggregation_method = "event_flow"
  aggregation_delay  = 120
  slide_by           = 30
}

variables.tf

Additionally create variables.tf

1
2
3
variable "environment" {
  type = string
}

  1. Run terraform validate and fix any errors.

  2. Run terraform plan and specify the environment as test

  3. Run terraform apply

The console will confirm the actions

Additionall the local terraform.tfstate file will be created

  1. Log in to New Relic and navigate to Alert Policies to confirm that Terraform created your new policy.

Alert Parent policy

Child Alert Condition

  1. Add triggers, the example below is of type EMAIL, others exist like SLACK

trigger.tf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
resource "newrelic_notification_destination" "team_email_destination" {
  name = "email-example"
  type = "EMAIL"

  property {
    key = "email"
    value = "team.member1@email.com,team.member2@email.com,team.member3@email.com"
  }
}

resource "newrelic_notification_channel" "team_email_channel" {
  name = "email-example"
  type = "EMAIL"
  destination_id = newrelic_notification_destination.team_email_destination.id
  product = "IINT"

  property {
    key = "subject"
    value = "New Subject"
  }
}

resource "newrelic_workflow" "team_workflow" {
  name = "Hoon API Workflow ${var.environment}"
  muting_rules_handling = "NOTIFY_ALL_ISSUES"

  issues_filter {
    name = "alerts_filter"
    type = "FILTER"

    predicate {
      attribute = "labels.policyIds"
      operator  = "EXACTLY_MATCHES"
      values    = [newrelic_alert_policy.hoonapi_alert_policy.id]
    }

    predicate {
      attribute = "priority"
      operator  = "EQUAL"
      values    = ["CRITICAL"]
    }
  }

  destination {
    channel_id = newrelic_notification_channel.team_email_channel.id
  }
}

  1. Run terraform validate and fix any errors.

  2. Run terraform plan and specify the environment as test

  3. Run terraform apply

You will notice the state updates and it creates a backup file for its internal process

Then the notification can be seen under the policy

Synthetic Monitoring

“You can think of our synthetic monitors as crash test dummies for your websites, applications, and API endpoints.” - docs.newrelic.com

There are 3 types of monitors

Ping

  • Would generally call a /ping endpoint not needing any authentication but I have seen teams call /health endpoints with this type
  • Doesnt have a synthetic script it runs
  • The default is type="SIMPLE" but can also have a type="BROWSER", the docs call it a SIMPLE BROWSER
  • Terraform resource is newrelic_synthetics_monitor

Scripted API

Example Scripted API From New Relic 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
var assert = require('assert');

$http.post('http://httpbin.org/post',
  // Post data
  {
    json: {
      widgetType: 'gear',
      widgetCount: 10
    }
  },
  // Callback
  function (err, response, body) {
    assert.equal(response.statusCode, 200, 'Expected a 200 OK response');

    console.log('Response:', body.json);
    assert.equal(body.json.widgetType, 'gear', 'Expected a gear widget type');
    assert.equal(body.json.widgetCount, 10, 'Expected 10 widgets');
  }
);
Real world Example Scripted API

This is still doing the same post and callback like the above but factors in how you could use this with an authentication mechanism (example OAuth2 Client Credentials Flow) and pass potential required headers ect.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
let assert = require("assert");
let request = require("request");
const { v4: uuidv4 } = require('uuid');

const auth = {
  uri: "https://authorisation-service.local/token",
  headers: {
    "Content-Type": "application/x-www-form-urlencoded",
  },
  form: {
    grant_type: "client_credentials",
    client_id: "my-client-id",
    client_secret: "password",
    scope: "foo.read",
  },
};

let callback = function (err, response, body) {
  const correlationId = uuidv4();
  let options = {
    uri: "https://app-service.local/foo/bar",
    headers: {
      Authorization: `Bearer ${body.access_token}`,
      "Correlation-Id": correlationId,
      "User-Id": "123",
    },
  };

  request.get(options, function (err, response, body) {
    assert.equal(response.statusCode, 200);
  });
};

request.post(auth, callback);

Scripted Browser

  • Can be used to call a script, the script then calls your website and performs actions like login, asssert elements exist
  • Terraform resource is newrelic_synthetics_script_monitor with type="SCRIPT_BROWSER"
Example Scripted Browser From New Relic

This example is written using Selenium Webdriver version 3.6, so variables like $driver and $browser The current NR docs for Synthetic scripted browser use Selenium Webdriver version 4.1, so variables like $webDriver

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
var assert = require('assert');

$browser.get('http://example.com').then(function(){
  // Check the H1 title matches "Example Domain"
  return $browser.findElement($driver.By.css('h1')).then(function(element){
    return element.getText().then(function(text){
      assert.equal('Example Domain', text, 'Page H1 title did not match');
    });
  });
}).then(function(){
  // Check that the external link matches "https://www.iana.org/domains/example"
  return $browser.findElement($driver.By.css('div > p > a')).then(function(element){
    return element.getAttribute('href').then(function(link){
      assert.equal('https://www.iana.org/domains/example', link, 'More information link did not match');
    });
  });
});
Real world Example Scripted Browser

This too is the same as the above but adapted something closer to real world.

Its sensible for the script to assert the existance of elements by a HTML property like data-automationid. While the data-automationid HTML property is not a formal web standard recognized by bodies like the W3C, it has become a widely adopted best practice and a de facto standard within the software testing and quality assurance community, especially for synthetic testing.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
$browser
  .addHostnamesToBlacklist([ // can blacklist anything
    "*.newrelic.com", 
    "*.facebook.net",
    "*.linkedin.com",
  ])
  .then(() => $browser.get("https://app-service.local/foo/bar")) // expect auth to be challanged, the app redirects to its authority
  .then(() =>
    $browser.wait(
      () =>
        $browser
          .getCurrentUrl()
          .then((url) => url.startsWith("https://authorisation-service.local")), // wait to be at authority url
      10000
    )
  )
  .then(() =>
    $browser
      .findElement($driver.By.xpath('//input[data-automationid="passwordField-input"]')) // provide the password
      .sendKeys("password")
  )
  .then(() =>
    $browser
      .findElement($driver.By.xpath('//input[data-automationid="usernameField-input"]')) // provide the username
      .sendKeys("username@example.com")
  )
  .then(() =>
    $browser
      .findElement($driver.By.xpath('//button[@data-automationid="submitLogin-button"]')) // click the login button
      .click()
  )
  .then(() =>
    $browser.wait(
      () =>
        $browser
          .getCurrentUrl()
          .then((url) => url == "https://app-service.local/foo/bar"), // wait for the authorised user to be redirected back to the app
      10000
    )
  )
  .then(() =>
    $browser.waitForElement(
      $driver.By.xpath('//*[@data-automationid="Foo-button"]'), // validate the existance of an element
      10000
    )
  )
  .then(() =>
    $browser.waitForElement(
      $driver.By.xpath('//*[@data-automationid="Foo-element"]'), // validate the existance of another element
      10000
    )
  );

Secrets Security

We dont want secrets commited to source control, the examples above were password

We can use a resource newrelic_synthetics_secure_credential to instead manage the secret, the example below is from the docs, you would then reference it in the alerts above using $secure.MY_KEY

1
2
3
4
5
resource "newrelic_synthetics_secure_credential" "foo" {
  key = "MY_KEY"
  value = "My value"
  description = "My description"
}

After applying the above, from NR UI -> Synthetics Monitoring -> Secure credentials -> locate by name -> Edit secure credential

Alerts

Synthetic Monitoring Alerts use the same resource type newrelic_nrql_alert_condition as mentioned above for the alerts.tf file, the only difference is where they select their data from:

Example Ping (The simple one)

1
nrql_query  = "FROM SyntheticCheck SELECT percentage(count(*),WHERE result != 'SUCCESS') WHERE monitorName = 'Foo - Synthetic Availability'"

Example Scripted Browser or Scripted API

1
2
nrql_query  = "SELECT count(result) FROM SyntheticCheck WHERE result = 'FAILED' AND monitorName = 'Foo - Page Load Synthetic'"
nrql_query  = "SELECT count(result) FROM SyntheticCheck WHERE result = 'FAILED' AND monitorName = 'Foo - Endpoint Synthetic'"