Use Azure KeyVault with ASP.NET Core running in an AKS cluster using AAD Pod Identity

When dealing with secrets like Connection Strings or API Keys, it's best practice to keep them out of your code files or environment variables. Azure Key Vault provides a great way of receiving those secrets at runtime. It allows storing them securely in the cloud and allows fine-grained access using Azure Active Directory Identities, so even no sensitive connection information for accessing Azure Key Vault needs to be given to the application.

In container land, this can become difficult as most un-orchestrated containers don't have an assigned identity. In those cases, an Azure Service Principal ID and Secret can be given to the container, which can be used to authenticate against Key Vault. However, this should only be considered for local test environments. As we already mentioned, storing any passwords in environment variables should be avoided. The managed Azure Kubernetes introduced the AAD Pod Identity project, to assign a Managed Identity to specific pods, so that they can authenticate against Azure Key Vault.

Implementation in ASP.NET Core

When using ASP.NET core, we usually add Azure Key Vault Secrets to the Configuration at Startup Time. The following implementation of the CreateWebHostBuilder method inside Program.cs works best for me.

using Microsoft.AspNetCore;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;

namespace KeyVaultDemo
{
    public class Program
    {
        public static void Main(string[] args)
        {
            CreateWebHostBuilder(args).Build().Run();
        }

        public static IWebHostBuilder CreateWebHostBuilder(string[] args)
        {
            var configBuilder = new ConfigurationBuilder().AddEnvironmentVariables();
            var configuration = configBuilder.Build();

            // Configure Azure Key Vault Connection
            var uri = configuration["AzureKeyVault:Uri"];
            var clientId = configuration["AzureKeyVault:ClientId"];
            var clientSecret = configuration["AzureKeyVault:ClientSecret"];

            // Check, if Client ID and Client Secret credentials for a Service Principal
            // have been provided. If so, use them to connect, otherwise let the connection 
            // be done automatically in the background
            if (!string.IsNullOrEmpty(clientId) && !string.IsNullOrEmpty(clientSecret))
                configBuilder.AddAzureKeyVault(uri, clientId, clientSecret);
            else
                configBuilder.AddAzureKeyVault(uri);

            return WebHost.CreateDefaultBuilder(args)
                .UseConfiguration(configBuilder.Build())
                .UseStartup<Startup>();
        }
    }
}

As you can see, I am checking for some environment variables for connection details. I am expecting to see AzureKeyVaultUri there, as the application always needs to know, which Key Vault to connect with. If no other variables are provided, the Key Vault SDK will try to figure out how to connect on its own, including trying to use a Managed Identity.

To make this work, ensure to add the latest version of the Microsoft.Azure.Services.AppAuthentication NuGet package to your project, otherwise the Azure Key Vault SDK will use an older version to connect, that does not support Managed Identities.

$ dotnet add package Microsoft.Azure.Services.AppAuthentication --version 1.3.1

I like this solution because it is so flexible. When a Managed Identity is available, there's nothing for you to do. Otherwise, simply provide a Service Principal Client Id and Secret and you are also good to go. Again, try to avoid the second in production environments.

Create a Managed Identity

Now we need an Identity, which we can assign to our Pods later and which we can give access to Azure KeyVault. The easiest way of doing this is the Azure CLI.

$ az identity create -g <resource-group> -n <name> -o json

Warning: The Identity name has some very tight restrictions and may not be shorter than 3 and longer than 24 characters 128 characters (limits changed recently). I experienced situations, where hurting these restrictions don't put out error messages.

This should give you a similar result to this:

{
  "clientId": "00000000-0000-0000-0000-000000000000",
  "clientSecretUrl": "https://control-eastus.identity.azure.net/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/myresourcegroup/providers/Microsoft.ManagedIdentity/userAssignedIdentities/myidentity/credentials?tid=00000000-0000-0000-0000-000000000000&oid=00000000-0000-0000-0000-000000000000&aid=00000000-0000-0000-0000-000000000000",
  "id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/myresourcegroup/providers/Microsoft.ManagedIdentity/userAssignedIdentities/myidentity",
  "location": "eastus",
  "name": "myidentity",
  "principalId": "00000000-0000-0000-0000-000000000000",
  "resourceGroup": "myresourcegroup",
  "tags": {},
  "tenantId": "00000000-0000-0000-0000-000000000000",
  "type": "Microsoft.ManagedIdentity/userAssignedIdentities"
}

The Identity we just created needs to have read-rights to the Resource Group where your Azure KeyVault is located at. To grant these rights, run the following command.

$ az role assignment create \
    --role Reader \
    --assignee <identity-principalId> \
    --scope /subscriptions/<subscriptionId>/resourcegroups/<resourceGroupName>

Grant access to Azure Key Vault

Of course, we should not forget to grant permissions to read Key Vault Secrets to our Managed Identity!

$ az keyvault set-policy \
    --name <name-of-the-key-vault> \
    --secret-permissions list get
    --object-id <identity-principalId>

Configure the AKS Cluster

Now it's time to configure the cluster to assign the Managed Identity to our Pods. We first need to allow AKS to use the Managed Identity. When creating the AKS cluster, we provided a Service Principal, which we now give permissions to use our Identity.

$ az role assignment create \
    --role "Managed Identity Operator" \
    --assignee <aksServicePrincipalClientId> \
    --scope <identiy-id> # subscriptions/.../userAssignedIdentities/XXXX

If you need to look up, what the Service Principal was you assigned to your cluster, you can do this by running az aks show command.

Time to install the AAD Pod Identity plugin!

$ kubectl apply -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment-rbac.yaml

This allows us to create an AzureIdentity as Kubernetes resource.

apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentity
metadata:
  name: my-identity
spec:
  type: 0
  ResourceID: <identity-id> # subscriptions/.../userAssignedIdentities/XXXX
  ClientID: <identity-clientId>

Once Kubernetes knows about our identity, we can create an AzureIdentityBinding for it, to define which of our Pods should have this identity assigned.

apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentityBinding
metadata:
  name: my-identity-binding
spec:
  AzureIdentity: my-identity
  Selector: bind-my-identity

The Selector is key here. All Pods with a matching label will have this Identity assigned. All we need to do is add the aadpodidbinding label to it. A Pod with that label would look like this.

apiVersion: v1
kind: Pod
metadata:
  labels:
    aadpodidbinding: bind-my-identity
  name: my-pod
spec:
  containers:
    - name: my-container
      image: nginx

That's it

Now, this Pod has access to Azure Key Vault secrets without a single Connection String or Service Principal password attached to it. Simply by assigning a Managed Identity and (if you are using .NET) running the initialization code from above. Super cool!

If you need help or run into issues, check out the Troubleshooting Guide and don't hesitate to leave a question in the comments.

☝️ Advertisement Block: I will buy myself a pizza every time I make enough money with these ads to do so. So please feed a hungry developer and consider disabling your Ad Blocker.