Background

One of the cards I took on last week, was the task of adding a low-no-data alert for one of our services. This was an Action Item from an incident post-mortem when we were manually alerted to one of our endpoints being down. Having this alert would ensure that we are notified more quickly should this happen again.

Using Lightstep, it was pretty easy to create the alert based on a metric that we had. (We use a different metric IRL for traffic, but for the sake of this tutorial, I'm using scrape_samples_scraped)

But I wasn't quite done yet.

• What about making the same alert in all the other environments ? I would need to recreate this alert for all of those environments by hand.

• What happens if we have the worst day ever ™️ and all of our alerts are gone? I would need to remember the queries and details to recreate them.

This is where Infrastructure-as-Code comes in handy, and specifically Terraform - which allows us to codify our resources such as lightstep_alert's. By using Terraform, we can:

1. Focus on being DRY (Don't Repeat Yourself) - Create one alert and programmatically generate alerts in our other environments.

2. Have our alerts codified - Be able to recover from a disaster and restore to our current state.

And so I set out to write the Terraform with a file that looked like this:

resource "lightstep_alert" "low-no-requests-api-terraform" {

}

Staring at this blank resource stanza made me think:

But wait...I already created the alert in the UI..do I now need to flip back and forth between the Terraform provider documentation and re-create the alert in Terraform?

This would be like the equivalent of getting a transparency paper and attempting to redraw a reference image on a different piece of paper.

I thought this was the only way until I learned a very valuable pattern from one of my very wise coworkers.

[You already have the alert,] Why don't you just import the alert and then use state show to get the code for it?

I..had never thought of that. But, I tried it and it worked! And was a huge time saver.

Here's the tutorial.

Tutorial

1. Set up Teraform Provider

First up was to ensure I had set up my Terraform provider to use my Lightstep organization and an api_key with a minimum level of Member

provider "lightstep" {
  api_key         = var.ls_api_key
  organization    = "LightStep"
}

1. Get ID of previously created Resource

To import the lightstep_alert, I needed need to grab the ID of the existing alert. (This can be easily gathered from the browser, see below).

1. Import the Resource

With the alert ID in hand, I simply had to run a terraform import lightstep_alert.<name_of_alert> <project>.<id>

• In my example it was:terraform import lightstep_alert.low-no-requests-api dev-tratnayake.mXgC4cG1f

terraform import lightstep_alert.low-no-requests-api dev-tratnayake.mXgC4cG1f         1 ✘  at 20:34:20  
lightstep_alert.low-no-requests-api: Importing from ID "dev-tratnayake.mXgC4cG1f"...
lightstep_alert.low-no-requests-api: Import prepared!
  Prepared lightstep_alert for import
lightstep_alert.low-no-requests-api: Refreshing state... [id=mXgC4cG1f]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

1. Show the imported Resource

Now here's the magic part where you can use Terraform like a 🔬 microscope 🔬 to break down an existing resource into its Terraform. With the resource imported into Terraform state, I could simply use terraform state show to output the alert in Terraform.

terraform state show lightstep_alert.low-no-requests-api                              1 ✘  at 20:35:27  
# lightstep_alert.low-no-requests-api:
resource "lightstep_alert" "low-no-requests-api" {
    id           = "mXgC4cG1f"
    name         = "Low-no-data-alert (UI)"
    project_name = "dev-tratnayake"
    type         = "metric_alert"

    expression {
        is_multi   = false
        is_no_data = false
        operand    = "below"

        thresholds {
            critical = "1"
            warning  = "5000"
        }
    }

    query {
        display        = "line"
        hidden         = false
        hidden_queries = {}
        query_name     = "a"
        query_string   = "metric scrape_samples_scraped | filter (job == \"apiserver\") | latest | group_by [\"job\"], sum"
    }
}

1. Use the Code from the imported Resource

With the alert now codified, I could simply copy-pasta it as a new alert in my Terraform file as follows (ensuring to strip out the id and metric_type as those are computed when the Terraform is applied).

resource "lightstep_alert" "low-no-requests-api-terraform" {
    # id           = "mXgC4cG1f"
    name         = "Low-no-data-alert (Terraform)"
    project_name = "dev-tratnayake"
    # type         = "metric_alert"

[... Rest of the Terraform from `terraform state show` here ... ]

1. Apply the Terraform to create the Resource

Finally, I ran a terraform apply which would:

1. Create the new alert (low-no-requests-api-terraform); and

2. Delete the old alert created via UI (low-no-requests) because that's not in the Terraform file, and therefore doesn't match the Terraform state.

terraform apply                                                                         ✔  at 20:37:03  
lightstep_alert.low-no-requests-api: Refreshing state... [id=mXgC4cG1f]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
  - destroy

Terraform will perform the following actions:

  # lightstep_alert.low-no-requests-api will be destroyed
  # (because lightstep_alert.low-no-requests-api is not in configuration)
  - resource "lightstep_alert" "low-no-requests-api" {
      - id           = "mXgC4cG1f" -> null
      - name         = "Low-no-data-alert (UI)" -> null
      - project_name = "dev-tratnayake" -> null
      - type         = "metric_alert" -> null

      - expression {
          - is_multi   = false -> null
          - is_no_data = false -> null
          - operand    = "below" -> null

          - thresholds {
              - critical = "1" -> null
              - warning  = "5000" -> null
            }
        }

      - query {
          - display        = "line" -> null
          - hidden         = false -> null
          - hidden_queries = {} -> null
          - query_name     = "a" -> null
          - query_string   = "metric scrape_samples_scraped | filter (job == \"apiserver\") | latest | group_by [\"job\"], sum" -> null
        }
    }

  # lightstep_alert.low-no-requests-api-terraform will be created
  + resource "lightstep_alert" "low-no-requests-api-terraform" {
      + id           = (known after apply)
      + name         = "Low-no-data-alert (Terraform)"
      + project_name = "dev-tratnayake"
      + type         = (known after apply)

      + expression {
          + is_multi   = false
          + is_no_data = false
          + operand    = "below"

          + thresholds {
              + critical = "1"
              + warning  = "5000"
            }
        }

      + query {
          + display      = "line"
          + hidden       = false
          + query_name   = "a"
          + query_string = "metric scrape_samples_scraped | filter (job == \"apiserver\") | latest | group_by [\"job\"], sum"
        }
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

lightstep_alert.low-no-requests-api: Destroying... [id=mXgC4cG1f]
lightstep_alert.low-no-requests-api-terraform: Creating...
lightstep_alert.low-no-requests-api: Destruction complete after 1s
lightstep_alert.low-no-requests-api-terraform: Creation complete after 2s [id=mw5HVFtch]

Apply complete! Resources: 1 added, 0 changed, 1 destroyed

The end result is the Alert that I created in UI - now codified and managed by Terraform ☑️

Conclusion

There are many times that Terraform is not the first thing we reach for when creating a resource. Many times we may reach for a GUI or specialized tool to get create resources, especially during rapid prototyping. Using terraform import and terraform state show allows you to keep that momentum by using Terraform to codify the resources you've already created (in a more user-friendly tool) which can then easily be modified to fit your needs.

Doing this is a huge time-saver that almost felt like cheating, and I will definitely be using this in my future Terraforming travels 🚀