Azure Automation with Ansible

Ansible with its huge community and broad platform support is one of the tools you should have a look at to address your cloud automation demands.

Azure Automation with Ansible can address all layers of the automation stack introduced by Gregor Hohpe in his AWS Innovate Keynote Talk. But it would be best if you decided for yourself whether it is the best tool for all these challenges.

automation stack introduced by Gregor Hohpe

# Development Machine

I created a new Development Machine for my first steps in Azure Automation with Ansible and chose Ubuntu as the Guest OS. Of course, an Azure virtual machine was the best choice here.

# Install Ansible

Installing Ansible on Ubuntu is straightforward using the PPA.

1
2
3
4
sudo apt update
sudo apt install software-properties-common
sudo add-apt-repository --yes --update ppa:ansible/ansible
sudo apt install ansible

# Installing Ansible Azure Collection

I recommend installing the Azure.Azcollection from the Ansible Galaxy.

1
ansible-galaxy collection install azure.azcollection

Azure Automation with Ansible - Azure.Azcollection

# Install dependencies for the collection

To avoid incompatibility install the dependencies required by the collection with the requirements-file shipped with the Azure.Azcollection.

1
sudo pip3 install -r ~/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt

# Storing Azure credentials

On my development machines, I prefer storing the credentials in a password file. For CI/CD Pipelines or environments like Ansible Automation Platform, environment variables are the way to go (will be covered later).

Using the Azure Resource Manager modules requires authenticating with the Azure API. You can choose from two authentication strategies:

Active Directory Username/Password
Service Principal Credentials

I chose Service Principal Credentials.

Create Service Principal:

1
az ad sp create-for-rbac --name ansible --role Contributor --scopes /subscriptions/<subscription_id>

Store Credentials:

1
2
3
4
5
6
7
8
mkdir ~/.azure
vi ~/.azure/credentials

[default]
subscription_id=<subscription_id>
client_id=<service_principal_app_id>
secret=<service_principal_password>
tenant=<service_principal_tenant_id>

# Run the first playbook

To test the newly created setup, I run a simple playbook that creates an empty Azure Resource Group.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
- name: Azure Storage Account
  hosts: localhost
  connection: local
  gather_facts: no
  collections:
    - azure.azcollection
  tasks:
    - name: Create resource group
      azure.azcollection.azure_rm_resourcegroup:
        auth_source: auto
        name: test00881155
        location: northeurope
        tags:
          testing: testing

1
ansible-playbook az-test.yml

Azure Automation with Ansible - Playbook

Azure Automation with Ansible - Result

# Ansible dynamic inventories

Using dynamic inventories of your Azure resources using Ansible opens up a whole new range of areas of application and possibilities for Azure Automation with Ansible. The underlying idea is to create Ansible inventory groups based on the Azure configuration (e.g. Dependency) and the attributes of the resources (e.g. Tags). Playbooks can be started against these groups.

To use the inventory, create a configuration based on the Azure dynamic-inventory plug-in.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
plugin: azure.azcollection.azure_rm

include_vm_resource_groups:
  - rg-inventory
auth_source: auto
conditional_groups:
  linux: "'canonical' in image.publisher"
  windows: "'WindowsServer' in image.offer"
keyed_groups:
 - key: tags.application

Azure Automation with Ansible - Inventory

In this example, you can see the empty default group “ungrouped”, because all running Azure VMs are grouped. Two conditional groups (Windows and Linux) are created based on the Guest OS, derived from different attributes. Both running Azure have the same tag “application:testApp” and therefore are members of the same keyed group. Keyed groups are mighty and flexible because you don’t need to know all the future Application Tags, as new applications are created a new group is created in the dynamic inventory,

To see all attributes of your inventory objects, use the --list parameter.

1
 ansible-inventory -i inventory.azure_rm.yml --list

# Using the inventory

You can reference all groups of the dynamic inventory like other Ansible Hosts.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
- name: Linux-Inventory Ping
  hosts: linux
  gather_facts: no
  vars_prompt:
    - name: ansible_user
      prompt: What is your username?
      private: false
    - name: ansible_ssh_pass
      prompt: What is your password?
      private: true
  tasks:
    - name: Example from an Ansible Playbook
      ansible.builtin.ping:

Azure Automation with Ansible - Inventory Usage

# Azure DevOps Pipeline

Azure DevOps is a great platform for managing or consuming your cloud platform through IaC. Azure DevOps Pipelines offer a secure environment to apply your Ansible playbooks.

# Credential handover

As mentioned earlier in this post, for CI/CD pipelines, environment variables are the preferred way to hand over the credentials. I decided to use a variable group that is imported into the pipeline.

Azure DevOps Variable Group

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
[...]
variables:
  - group: ansible
[...]
- script: ansible-inventory -i turned-on-inventory.azure_rm.yml --graph
  displayName: 'Show Inventory Graph'
  env:
    AZURE_CLIENT_ID: $(ARM_CLIENT_ID)
    AZURE_SECRET: $(ARM_CLIENT_SECRET)
    AZURE_TENANT: $(ARM_TENANT_ID)
    AZURE_SUBSCRIPTION_ID: $(ARM_SUBSCRIPTION_ID)
[...]

# Ansible extension

Azure Pipeline capabilities can be extended with additional (third party) extensions. For Azure Automation with Ansible, the Ansible extension from the Marketplace is required.

This extension includes a build/release task to integrate with Ansible. The task executes a given Ansible playbook on a specified list of inventory nodes via command line interface.
https://marketplace.visualstudio.com/items?itemName=ms-vscs-rm.vss-services-ansible&targetId=a9216eef-c81a-4493-b6c1-5f69d79898a0

# Pipeline with Microsoft Hosted Agents

With Microsoft Hosted Agents you have minimal effort in operations and maintenance. Each time you run a pipeline, you get a fresh virtual machine for all Tasks within the pipeline. Microsoft takes care of the updates of the image.

Image	Classic Editor Agent Specification	YAML VM Image Label	Included Software
Windows Server 2022 with Visual Studio 2022	windows-2022	`windows-latest` OR `windows-2022`	Link
Windows Server 2019 with Visual Studio 2019	windows-2019	`windows-2019`	Link
Ubuntu 22.04	ubuntu-22.04	`ubuntu-latest` OR `ubuntu-22.04`	Link
Ubuntu 20.04	ubuntu-20.04	`ubuntu-20.04`	Link
macOS 13 Ventura	macOS-13	`macOS-13`	Link
macOS 12 Monterey	macOS-12	`macOS-latest` OR `macOS-12`	Link
macOS 11 Big Sur	macOS-11	`macOS-11`	Link

Reference: Microsoft-hosted agents

# Pipeline definition

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
name: build_$(BuildDefinitionName)_$(date:yyyyMMdd)$(rev:.r)

trigger: none

pool:
  vmImage: ubuntu-latest

parameters:
  - name: resourcegroup_name
    displayName: resourcegroup_name
    type: string
  - name: storageaccount_name
    displayName: storageaccount_name
    type: string

variables:
  - group: ansible
  - name: RgName
    value: ${{ parameters.resourcegroup_name }}
  - name: StoName
    value: ${{ parameters.storageaccount_name }}

steps:
- script: pip3 install ansible
  displayName: 'Install Ansible'
- script: ansible-galaxy collection install azure.azcollection --force
  displayName: 'Install Azure Collection'
- script: pip3 install -r ~/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt
  displayName: 'Install modules needed for Azure'
- script: ansible-inventory -i turned-on-inventory.azure_rm.yml --graph
  displayName: 'Show Inventory Graph'
  env:
    AZURE_CLIENT_ID: $(ARM_CLIENT_ID)
    AZURE_SECRET: $(ARM_CLIENT_SECRET)
    AZURE_TENANT: $(ARM_TENANT_ID)
    AZURE_SUBSCRIPTION_ID: $(ARM_SUBSCRIPTION_ID)
- script: ansible-inventory -i turned-on-inventory.azure_rm.yml --list
  displayName: 'Show Inventory Details'
  env:
    AZURE_CLIENT_ID: $(ARM_CLIENT_ID)
    AZURE_SECRET: $(ARM_CLIENT_SECRET)
    AZURE_TENANT: $(ARM_TENANT_ID)
    AZURE_SUBSCRIPTION_ID: $(ARM_SUBSCRIPTION_ID)
- task: Ansible@0
  displayName: 'Run Ansible Playbook'
  inputs:
    ansibleInterface: 'agentMachine'
    playbookPathOnAgentMachine: 'az-storage.yml'
    inventoriesAgentMachine: 'file'
    inventoryFileOnAgentMachine: 'turned-on-inventory.azure_rm.yml'
    args: --extra-vars '{"resourcegroup_name":${{ variables['RgName'] }}, "storageaccount_name":${{ variables['StoName'] }}}'
  env:
    AZURE_CLIENT_ID: $(ARM_CLIENT_ID)
    AZURE_SECRET: $(ARM_CLIENT_SECRET)
    AZURE_TENANT: $(ARM_TENANT_ID)
    AZURE_SUBSCRIPTION_ID: $(ARM_SUBSCRIPTION_ID)

# Pipeline execution

What is prepared for the actual execution?

Install the latest Ansible version (runner-image Ansible version is too old)
Install Ansible Azure Collection
Install all requirements of the Ansible Azure Collection

Azure Automation with Ansible - Pipeline

Azure Automation with Ansible - Task Inventoty Groups

Azure Automation with Ansible - Task Playbook

# Pipeline with Self Hosted Agents

Unfortunately, there are also disadvantages to using pipelines with Microsoft Hosted Agents. The fact that each run uses a new image has a great disadvantage if you need many of your tools not included in the runner-image (or newer versions) in the execution environment. Installing the requirements of the Ansible Azure Collection takes at least 70 seconds. With a Self Hosted Agent, you can persistently prepare your execution environment.

# Agent preparation

The Ubuntu Azure VM I use was prepared similarly to the Development Machine in the first chapter, plus the Agent Package. Please check the detailed instructions for additional information.

After the manual run or the service-start, the agent initiates an SSL connection to your Azure DevOps project, destinations are listed in the FAQ of the detailed instructions.

1
2
3
4
5
6
7
wget https://vstsagentpackage.azureedge.net/agent/3.232.0/vsts-agent-linux-x64-3.232.0.tar.gz
mkdir myagent && cd myagent
tar zxvf ../vsts-agent-linux-x64-3.232.0.tar.gz
sudo ./config.sh
sudo ./svc.sh install
sudo ./svc.sh start
sudo ./svc.sh status

Azure Automation with Ansible - Agent Service

Azure DevOps Agent Group

# Pipeline definition

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
name: build_$(BuildDefinitionName)_$(date:yyyyMMdd)$(rev:.r)

trigger: none

pool: 
  name: Default


parameters:
  - name: resourcegroup_name
    displayName: resourcegroup_name
    type: string
  - name: storageaccount_name
    displayName: storageaccount_name
    type: string

variables:
  - group: ansible
  - name: RgName
    value: ${{ parameters.resourcegroup_name }}
  - name: StoName
    value: ${{ parameters.storageaccount_name }}

steps:
- script: ansible-inventory -i turned-on-inventory.azure_rm.yml --graph
  displayName: 'Show Inventory Graph'
  env:
    AZURE_CLIENT_ID: $(ARM_CLIENT_ID)
    AZURE_SECRET: $(ARM_CLIENT_SECRET)
    AZURE_TENANT: $(ARM_TENANT_ID)
    AZURE_SUBSCRIPTION_ID: $(ARM_SUBSCRIPTION_ID)
- task: Ansible@0
  displayName: 'Run Ansible Playbook'
  inputs:
    ansibleInterface: 'agentMachine'
    playbookPathOnAgentMachine: 'az-storage.yml'
    inventoriesAgentMachine: 'file'
    inventoryFileOnAgentMachine: 'turned-on-inventory.azure_rm.yml'
    args: --extra-vars '{"resourcegroup_name":${{ variables['RgName'] }}, "storageaccount_name":${{ variables['StoName'] }}}'
  env:
    AZURE_CLIENT_ID: $(ARM_CLIENT_ID)
    AZURE_SECRET: $(ARM_CLIENT_SECRET)
    AZURE_TENANT: $(ARM_TENANT_ID)
    AZURE_SUBSCRIPTION_ID: $(ARM_SUBSCRIPTION_ID)