Use Kubernetes Service Accounts in Terraform for AKS clusters with AAD integration

When enabling Azure Active Directory Authentication on your Azure Kubernetes Service (AKS) cluster, users will be prompted to login interactively, the first time they authenticate against the cluster. While this works great for humans, CI/CD pipelines and non-interactive scripts obviously struggle with this approach.

Currently, the AAD integration in AKS does not support Service Principals. Only Users can authenticate interactively. While we can bypass the AAD authentication using the --admin flag with the az aks get-credentials command, it is a very bad practice to give all non-interactive scripts full cluster-admin access to the cluster.

az aks get-credentials -n AKS_CLUSTER -g RESOURCE_GROUP --admin

Service Accounts to the rescue

A Kubernetes Service Account can help here. They can be used to authenticate against Kubernetes Clusters and can bypass the AAD authentication.

The idea is to only use the cluster-admin credentials we get when using the --admin flag only once, to create the required Service Accounts, Roles and Role Bindings. From that moment on, all scripts and pipelines can use these Service Accounts.

Create a Kubernetes Service Account for Terraform in Terraform

Let's assume, we have created an AKS cluster with the following Terraform script.

resource "azurerm_kubernetes_cluster" "example" {
  name                = "mycluster"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
  dns_prefix          = "mycluster"
  kubernetes_version  = "1.15.7"

  default_node_pool {
    name       = "default"
    node_count = 2
    vm_size    = "Standard_DS2_v2"
  }
  
  identity {
    type = "SystemAssigned"
  }

  role_based_access_control {
    enabled = true
    azure_active_directory {
        client_app_id = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
        server_app_id = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
        server_app_secret = "xxxxxxxx"
    }
  }
}

This let's us access the Admin Kube-Config at the azurerm_kubernetes_cluster.example.kube_admin_config variable. We can use this Admin Kube-Config, to authenticate the Kubernetes Provider in Terraform and create the Service Principals, Roles and Role Bindings we need.

provider "kubernetes" {
  alias = "admin"
  load_config_file       = "false"
  host                   = azurerm_kubernetes_cluster.example.kube_admin_config.0.host
  username               = azurerm_kubernetes_cluster.example.kube_admin_config.0.username
  password               = azurerm_kubernetes_cluster.example.kube_admin_config.0.password
  client_certificate     = base64decode(azurerm_kubernetes_cluster.example.kube_admin_config.0.client_certificate)
  client_key             = base64decode(azurerm_kubernetes_cluster.example.kube_admin_config.0.client_key)
  cluster_ca_certificate = base64decode(azurerm_kubernetes_cluster.example.kube_admin_config.0.cluster_ca_certificate)
}

# Create a Service Account
resource "kubernetes_service_account" "example" {
  provider = kubernetes.admin
  automount_service_account_token = true
  
  metadata {
    name = "terraform-example"
  }
}

# Add the Secret, that holds the Service Account Token as a data source
data "kubernetes_secret" "example" {
  provider = kubernetes.admin

  metadata {
    name = "${kubernetes_service_account.example.default_secret_name}"
  }
}

# Create a new Role for the Service Account
resource "kubernetes_cluster_role" "example" {
  provider = kubernetes.admin
  metadata {
    name = "terraform-example"
  }

  rule {
    api_groups = [""]
    resources  = ["namespaces"]
    verbs      = ["get", "list", "update", "create", "patch"]
  }
}

# Assign the Role to the Service Account
resource "kubernetes_cluster_role_binding" "example" {
  provider = kubernetes.admin

  metadata {
    name = "terraform-example"
  }

  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = kubernetes_cluster_role.example.metadata[0].name
  }

  subject {
    kind      = "ServiceAccount"
    name      = kubernetes_service_account.example.metadata[0].name
  }
}

That will be the last time, we used the the Admin Kube-Config. From now on, we will authenticate the Kubernetes Provider in Terraform with our Service Account.

Authenticate against Kubernetes in Terraform with a Service Account

To use the Service Account for the rest of our Kubernetes operations in Terraform, we need to create a second provider block. Remember, that when using multiple providers of the same kind in Terraform, we need to give them aliases to distinguish them.

This second Kubernetes Provider uses an Access Token to authenticate, which it can get from a Kubernetes Secret, that has been automatically created with the Service Account.

provider "kubernetes" {
  alias = "service_account"
  load_config_file       = "false"
  host                   = azurerm_kubernetes_cluster.example.kube_config.0.host
  cluster_ca_certificate = lookup(data.kubernetes_secret.example.data, "ca.crt")
  token                  = lookup(data.kubernetes_secret.example.data, "token")
}

resource "kubernetes_namespace" "example" {
  provider = kubernetes.service_account
  depends_on = [kubernetes_cluster_role.example, kubernetes_cluster_role_binding.example]

  metadata {
    name = "hello-terraform"
  }
}

As you can see, the Namespace resource is using the provider = kubernetes.service_account Kubernetes Provider and only has the rights to operate within the boundaries of its Role Bindings.

Important: Please note, that this unfortunately does not get rid of the need have at least Contributor rights over the AKS cluster and the theoretical rights to perform admin tasks. It just lets your Terraform script execute its work using a non-admin Service Account. The script executor still needs admin rights.