Azure Pipeline Agent Container App

This repo contains an experiment to run Azure Pipeline Agents in Azure Container Apps. For production use, consider Scale set agents (sample repo).

Features (see limitations below):

• KEDA Azure Pipelines scaler
• Diagnostics logs saved on Azure Files
• Ubuntu based image with core set of tools e.g. Azure CLI, Bash, Helm, Kubectl, Packer, PowerShell, Terraform (Dockerfile)

Instructions

There are a number of scripts and pipelines you can use to get going. Below, I'll describe a local and pipeline approach, but you can blend these.

Pre-requisites

• AKS: Register the AKS-KedaPreview feature flag

Local setup

• You'll need Azure CLI, Docker, PowerShell and Terraform
• You can use an existing Azure Container Registry (if you already have a shared registry) or let Terraform create one. In case Terraform creates the ACR, there is no opportunity to build and push the container image to the ACR before the Container App will use it.
  Either let Terraform fail -> build & push the image -> retry Terraform apply, or pre-create the ACR. In case you pre-create the ACR, you also need to pre-create a User-assigned Managed Identity with AcrPull role on the ACR.
• Build and push the agent container image using build_image.ps1 script (alternatively, use the build-image.yml pipeline in case you don't have Docker locally)
• Create a Personal Access Token with Agent Pools read & manage scope
• Create a config.auto.tfvars file (example) in the terraform directory, and use it to override the following variables:
  agent_identity_resource_id
container_registry_id
devops_pat
devops_url (Organization url https://dev.azure.com/<org>)
• Provision infrastructure by running terraform apply

Pipeline setup

This approach uses the deploy-container-agents.yml pipeline to build the container image, provision Container App infrastructure and run a test job on a newly created agent.

• You'll need an existing Azure Container Registry (the assumption is that the Service Connection identity does not have the Azure Owner role required to configure RBAC and the ACR is a shared component anyway)
• Create an User-assigned Managed Identity with AcrPush role on the Azure Container Registry
• Create an Terraform azurerm backend
• Create a Docker Registry Service Connection to the ACR
• Create a Personal Access Token with Agent Pools read & manage scope
• Create a variable group build-container-agent-image with the following variable:
  containerRegistry (ACR Service Connection)
• Create a variable group pipeline-container-agents with the following variables:
  subscriptionConnection (Azure Service Connection)
  TF_STATE_CONTAINER_NAME (Terraform azurerm backend storage container)
  TF_STATE_RESOURCE_GROUP_NAME (Terraform azurerm backend storage account resource group)
  TF_STATE_STORAGE_ACCOUNT_NAME (Terraform azurerm backend storage account)
  TF_VAR_agent_identity_resource_id
TF_VAR_container_registry_id
TF_VAR_devops_pat
• Make sure you have the Terraform extension installed
• Use the deploy-container-agents.yml to build the agent container image, provision infrastructure and run a test job on a newly created agent. Override the destroy parameter to prevent the Container App infrastructure from being destroyed at the end of the pipeline run

Testing

By default, the agents will be created in the Default agent pool with system capability CONTAINER_APP_NAME. Use the image-info.yml pipeline to test the agents. You can override the numberOfJobs parameter to test elasticity

Limitations

This repo is an experiment, you may have various stability issues. Here are some known issues:

• The container image is not a general purpose image that works with all of the standard Azure Pipeline Tasks.
• Using Container Registry Private Endpoints has an issue.