New Relic With Terraform

“Terraform is a popular infrastructure-as-code software tool built by HashiCorp. You use it to provision all kinds of infrastructure and services, including New Relic dashboards and alerts.” - docs.newrelic.com

In this post I will create an Error New Relic alert via Terraform, I see this Error Rate as Availability and the resulting New Relic resources would be a provider, alert policy which is the parent, alert condition which are children to the parent and alert trigger

Four Golden Signals

For your alerts you need to think about whats sensible to alert on, Goole SRE is the golden standard, because you know… its Google :D … so you can re-invent the wheel or learn from their Golden Signals:

High Level Commands & Flow

Required Terraform Config

The Newrelic docs have a great example which I based the below on, you will notice they are also focused on the Goole SRE Golden Signals

The high level config that I used to create my .tf files are

  • providers.tf
  • alerts.tf
    • alert policy (this is the parent)
    • alert conditions (these are the children)
    • alert triggers
  • locals.tf (optional)
  • variables.tf (optional)
Additional abstractions

Locals

Locals are like constants in terraform, example locals.tf file with property newrelic_account_id

1
2
3
locals {
  newrelic_account_id = "123456789"
}

To access newrelic_account_id in another file like providers.tf

1
2
3
provider "newrelic" {
  account_id = local.newrelic_account_id
}

Variables

1
2
3
4
5
6
7
8
9
10
11
variable "environment" {
  type = string
}

variable "newrelic_slack_channels" {
  type = map(string)
  default = {
    uat  = "X05LBNSKMHU"
    prod = "X05LIBNLBYJ"
  }
}

I set the environment in prod.tfvars, uat.tfvars files, example value environment = "uat"

You can access the map newrelic_slack_channels in a file, example locals.tf

1
2
3
locals {
  newrelic_slack_channel_id = var.newrelic_slack_channels[var.environment]
}

Example Setup

  1. Install Terraform

Chocolatey makes this easy for windows users like me … sudo apt-get this 🖕

1
2
choco install terraform
terraform -version
  1. Create your initial config with providers.tf and locals.tf

providers.tf

1
2
3
4
5
6
7
8
9
10
11
12
13
terraform {
  required_version = "~> 1.9.8"
  required_providers {
    newrelic = {
      source  = "newrelic/newrelic"
    }
  }
}

provider "newrelic" {
  account_id = local.newrelic_account_id
  api_key    = local.api_key
}

locals.tf

You can get this from your NR account under Administration -> API Keys

1
2
3
4
locals {
  newrelic_account_id = "0000000"
  api_key             = "NRAK-00000000000000000000000000"
}
  1. Run terraform init which will initialize the backend and provider plugins, mine created these files/folder locally. I manually modified my locals.tf with my account id and api key.

I then also updated my .gitignore with the config below

1
2
.terraform/*
.terraform.lock.hcl
  1. Create Alert Policy (parent) and Alert Condition (children) inside alerts.tf

locals.tf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
// *** Alert policy

resource "newrelic_alert_policy" "hoonapi_alert_policy" {
  name  = "Hoon API Alert Policy ${var.environment}"
}

// *** Alert conditions

resource "newrelic_nrql_alert_condition" "error_rate" {
  name                         = "Golden Shower Error Rate"
  account_id                   = local.newrelic_account_id
  policy_id                    = newrelic_alert_policy.hoonapi_alert_policy[count.index].id
  type                         = "static"
  enabled                      = true
  violation_time_limit_seconds = 259200
  runbook_url                  = "https://carlpaton.github.io/2023/01/newrelic/"

  nrql {
    query = "FROM Transaction SELECT percentage(count(*), WHERE `response.status` LIKE '5%') WHERE appName in ('NewRelicHoon') AND request.uri NOT IN ('/foo', '/bar')"
  }

  critical {
    operator              = "above"
    threshold             = 2
    threshold_duration    = 300
    threshold_occurrences = "all"
  }

  fill_option        = "none"
  aggregation_window = 60
  aggregation_method = "event_flow"
  aggregation_delay  = 120
  slide_by           = 30
}

variables.tf

Additionally create variables.tf

1
2
3
variable "environment" {
  type = string
}

  1. Run terraform validate and fix any errors.

  2. Run terraform plan and specify the environment as test

  3. Run terraform apply

The console will confirm the actions

Additionall the local terraform.tfstate file will be created

  1. Log in to New Relic and navigate to Alert Policies to confirm that Terraform created your new policy.

Alert Parent policy

Child Alert Condition

  1. Add triggers, the example below is of type EMAIL, others exist like SLACK

trigger.tf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
resource "newrelic_notification_destination" "team_email_destination" {
  name = "email-example"
  type = "EMAIL"

  property {
    key = "email"
    value = "team.member1@email.com,team.member2@email.com,team.member3@email.com"
  }
}

resource "newrelic_notification_channel" "team_email_channel" {
  name = "email-example"
  type = "EMAIL"
  destination_id = newrelic_notification_destination.team_email_destination.id
  product = "IINT"

  property {
    key = "subject"
    value = "New Subject"
  }
}

resource "newrelic_workflow" "team_workflow" {
  name = "Hoon API Workflow ${var.environment}"
  muting_rules_handling = "NOTIFY_ALL_ISSUES"

  issues_filter {
    name = "alerts_filter"
    type = "FILTER"

    predicate {
      attribute = "labels.policyIds"
      operator  = "EXACTLY_MATCHES"
      values    = [newrelic_alert_policy.hoonapi_alert_policy.id]
    }

    predicate {
      attribute = "priority"
      operator  = "EQUAL"
      values    = ["CRITICAL"]
    }
  }

  destination {
    channel_id = newrelic_notification_channel.team_email_channel.id
  }
}

  1. Run terraform validate and fix any errors.

  2. Run terraform plan and specify the environment as test

  3. Run terraform apply

You will notice the state updates and it creates a backup file for its internal process

Then the notification can be seen under the policy

Synthetic Monitor

This would be

1
2
newrelic_alert_condition
newrelic_synthetics_alert_condition