Detecting Terraform Drift with Github actions

22 Nov 2023 on Development, Terraform

Like dichotomous branching, or a ‘Back to the Future’-esque timeline; drift is gonna happen

I’ve been using terraform for a while and what I’ve discovered is that there doesn’t seem to be a whole wealth of information about what to do when cold hard reality smacks your utopian infrastructure as code in the face. A classic example might be a support engineer adding LOG_LEVEL=super-verbose as an environment variable via the AWS console to some particular runtime because they’re getting reports that something blahblahblah doesn’t work.

If the first thought in your head is “That’ll never happen because IaC means they just have to modify some file; raise a PR; and then it’ll get auto deployed after going through the appropriate approval gates”; this blog post is not for you.

I’ve been using a pipeline that’s based on this blog post. That’s not a pre-requisite; you just have to be be able to generate repository dispatch events after doing the terraform plan. Post terraform plan we generate a repository dispatch event so that we can trigger other activity¹. We can have a downstream consumer that raises github issues in the event that a scheduled terraform plan shows changes.

The repository dispatch event

There’s no right answer here; capture the information that you think is most useful. We extract a oneliner from the terraform plan via this JQ snippet and turn it into something very similar to what you see on the commandline. It doesn’t have to be a one-liner; tf-summarize can generate nice output for inclusion as a PR comment.

# shellcheck disable=SC2016,SC2034
JQ_JSON_PLAN_ONELINER='
    (
      [
        .resource_changes[]?.change |
        [
          .actions?,
          if has("importing") then
            "importing"
          else
            empty
          end
        ]
      ] | flatten
    ) | {
      "plan_outcome_import":(map(select(.=="importing")) | length),
      "plan_outcome_add":(map(select(.=="create")) | length),
      "plan_outcome_change":(map(select(.=="update")) | length),
      "plan_outcome_destroy":(map(select(.=="delete")) | length)
    } | to_entries | map("\(.key)=\(.value)") | @sh
'
plan_oneliner() {
  exitcode="${PLAN_EXITCODE}"
  plan_outcome_add="0"
  plan_outcome_change="0"
  plan_outcome_destroy="0"
  plan_outcome_import="0"

  if [ "$exitcode" == "0" ]; then
    echo "has-changes=no" >>"$GITHUB_OUTPUT"
    echo "Terraform plan has no changes"
  elif [ "$exitcode" == "2" ]; then
    # shellcheck disable=SC2016
    eval "export $(terraform show -json "$PLAN_JSON_FILE" | jq -r "${JQ_JSON_PLAN_ONELINER}")"
    {
      printf "has-changes=yes\n"
      printf "add=%s\n" "${plan_outcome_add}"
      printf "change=%s\n" "${plan_outcome_change}"
      printf "destroy=%s\n" "${plan_outcome_destroy}"
    } >>"$GITHUB_OUTPUT"
    oneliner="Terraform plan has changes: ${plan_outcome_import} to import, ${plan_outcome_add} to add, ${plan_outcome_change} to change, ${plan_outcome_destroy} to destroy."
    echo "summary-oneliner=$oneliner" >>"$GITHUB_OUTPUT"
    echo "$oneliner"
  fi
}

You have full control over the client_payload part of the repository dispatch event. Bear in mind that the JSON forming the client_payload may only have 10 top level properties so we’re nesting into custom_data to avoid the limit. The other values in the client_payload are standardised values that we’ve found useful for other downstream consumers but won’t contain the right information in this context. We’ll be using event_name as an additional filter and our client payload looks something like this:

{
  ...
  "event": {
    // other bits skipped for brevity.
    "client_payload": {
      "repository" : "quotidian-ennui/infrastructure",
      "workflow" : "terraform.yml",
      "event_name" : "schedule|pull_request|...", // event will be ignored if it's not schedule
      "actor" : "...", // likely to be the last committer on the main branch, so not very useful
      "ref" : "refs/heads/main", // it's a schedule so default branch
      "sha": "commit sha", // it's a schedule so latest on default branch
      "custom_data": {
        "plan_has_changes": "yes || no || unknown", // from terraform-plan
        "plan_result": "success || failure || skipped || cancelled", // Whether terraform plan actually finished
        "plan_oneliner": "Terraform plan has changes: [n] to import, [n] to add, [n] to change, [n] to destroy. || 'n/a'",
        "workflow_run_url": "https://github.com/... pointing to the workflow run that triggered this"
      }
    }
  }
}

The format of your client_payload starts becoming important because you’re going to be using it in other workflows that live on the default branch of the repository. You need to orchestrate the modification the client_payload (generating new payloads while living on feature/new-client-payload) and the downstream workflows to match (all executing from refs/head/main). It’s overkill right now, and the words ‘schema validation’ aren’t my favourite words in the world, but it’s something to think about.