10.8 Ansible Advanced Features
Learning Objectives
By the end of this chapter, you will be able to:
Design and implement Ansible roles for modular and reusable automation
Organize complex playbooks using role dependencies and inheritance
Secure sensitive data using Ansible Vault for production environments
Master advanced Jinja2 templating with filters, tests, and custom functions
Implement dynamic inventories and advanced host targeting strategies
Optimize playbook performance with parallel execution and caching
Integrate Ansible with external systems and APIs
Apply advanced error handling and workflow control patterns
Create custom modules and plugins for specialized requirements
Prerequisites: Solid understanding of Ansible core concepts from Chapter 10.7, including inventory, playbooks, variables, and basic modules.
Advanced Focus: This chapter covers production-ready Ansible patterns and advanced techniques used in enterprise environments.
Ansible Roles: Modular Automation
Roles are Ansible’s way of organizing and reusing automation code. They provide a structured approach to breaking down complex playbooks into manageable, reusable components.
Role Directory Structure:
roles/
└── webserver/
├── tasks/
│ ├── main.yml # Main task list for the role
│ ├── install.yml # Package installation tasks
│ ├── configure.yml # Configuration tasks
│ └── security.yml # Security hardening tasks
├── handlers/
│ └── main.yml # Event handlers (restarts, reloads)
├── templates/
│ ├── nginx.conf.j2 # Jinja2 configuration templates
│ ├── ssl.conf.j2 # SSL configuration template
│ └── logrotate.j2 # Log rotation configuration
├── files/
│ ├── index.html # Static files to copy
│ └── ssl-cert.pem # SSL certificates
├── vars/
│ └── main.yml # Role-specific variables
├── defaults/
│ └── main.yml # Default variables (lowest precedence)
├── meta/
│ └── main.yml # Role metadata and dependencies
└── README.md # Role documentation
Creating a Web Server Role:
tasks/main.yml - Main task orchestration:
---
# Main task file for webserver role
- name: Include OS-specific variables
include_vars: "{{ ansible_os_family }}.yml"
tags: [always]
# Include task files in logical order
- name: Install web server packages
include_tasks: install.yml
tags: [installation, packages]
- name: Configure web server
include_tasks: configure.yml
tags: [configuration]
- name: Apply security hardening
include_tasks: security.yml
tags: [security, hardening]
when: apply_security_hardening | default(true)
- name: Verify web server is running
include_tasks: verify.yml
tags: [verification]
tasks/install.yml - Package installation:
---
- name: Update package cache (Debian/Ubuntu)
apt:
update_cache: yes
cache_valid_time: 3600
when: ansible_os_family == "Debian"
- name: Install web server and dependencies
package:
name: "{{ webserver_packages }}"
state: present
- name: Install additional packages
package:
name: "{{ item }}"
state: present
loop: "{{ additional_packages | default([]) }}"
when: additional_packages is defined
- name: Create web server user
user:
name: "{{ webserver_user }}"
system: yes
shell: /bin/false
home: "{{ webserver_home }}"
create_home: no
when: create_webserver_user | default(true)
tasks/configure.yml - Configuration management:
---
- name: Create web server directories
file:
path: "{{ item }}"
state: directory
owner: "{{ webserver_user }}"
group: "{{ webserver_group }}"
mode: '0755'
loop:
- "{{ webserver_document_root }}"
- "{{ webserver_log_dir }}"
- "{{ webserver_config_dir }}/sites-available"
- "{{ webserver_config_dir }}/sites-enabled"
- name: Generate main configuration file
template:
src: "{{ webserver_main_config_template }}"
dest: "{{ webserver_config_dir }}/{{ webserver_main_config_file }}"
owner: root
group: root
mode: '0644'
backup: yes
notify: restart webserver
- name: Configure virtual hosts
template:
src: vhost.conf.j2
dest: "{{ webserver_config_dir }}/sites-available/{{ item.name }}"
owner: root
group: root
mode: '0644'
loop: "{{ virtual_hosts | default([]) }}"
notify: restart webserver
- name: Enable virtual hosts
file:
src: "{{ webserver_config_dir }}/sites-available/{{ item.name }}"
dest: "{{ webserver_config_dir }}/sites-enabled/{{ item.name }}"
state: link
loop: "{{ virtual_hosts | default([]) }}"
when: item.enabled | default(true)
notify: restart webserver
- name: Remove default site
file:
path: "{{ webserver_config_dir }}/sites-enabled/default"
state: absent
notify: restart webserver
when: remove_default_site | default(true)
defaults/main.yml - Default variables:
---
# Web server package and service names
webserver_packages:
- nginx
- nginx-common
webserver_service: nginx
webserver_user: www-data
webserver_group: www-data
# Paths and directories
webserver_config_dir: /etc/nginx
webserver_document_root: /var/www/html
webserver_log_dir: /var/log/nginx
webserver_home: /var/www
# Configuration files
webserver_main_config_file: nginx.conf
webserver_main_config_template: nginx.conf.j2
# Default settings
nginx_worker_processes: auto
nginx_worker_connections: 1024
nginx_keepalive_timeout: 65
nginx_client_max_body_size: 64m
# Security settings
apply_security_hardening: true
remove_default_site: true
create_webserver_user: false # User usually exists
# SSL/TLS settings
ssl_protocols: "TLSv1.2 TLSv1.3"
ssl_ciphers: "ECDHE+AESGCM:ECDHE+AES256:ECDHE+AES128:!aNULL:!MD5:!DSS"
ssl_prefer_server_ciphers: "off"
handlers/main.yml - Event handlers:
---
- name: restart webserver
systemd:
name: "{{ webserver_service }}"
state: restarted
daemon_reload: yes
listen: restart webserver
- name: reload webserver
systemd:
name: "{{ webserver_service }}"
state: reloaded
listen: reload webserver
- name: restart webserver
systemd:
name: "{{ webserver_service }}"
state: restarted
- name: reload webserver
systemd:
name: "{{ webserver_service }}"
state: reloaded
Using Roles in Playbooks:
---
- name: Configure web infrastructure
hosts: web_servers
become: true
vars:
virtual_hosts:
- name: example.com
document_root: /var/www/example.com
enabled: true
- name: api.example.com
document_root: /var/www/api
enabled: true
additional_packages:
- curl
- htop
- vim
roles:
- role: common
tags: [common, base]
- role: security
tags: [security]
- role: webserver
tags: [webserver, nginx]
nginx_worker_processes: 4
apply_security_hardening: true
- role: monitoring
tags: [monitoring]
when: environment == "production"
Role Dependencies:
meta/main.yml - Define role dependencies:
---
dependencies:
- role: common
vars:
install_development_tools: false
- role: security
vars:
configure_firewall: true
ssh_hardening: true
when: apply_security_hardening | default(true)
- role: ssl-certificates
vars:
ssl_domains: "{{ virtual_hosts | map(attribute='name') | list }}"
when: enable_ssl | default(false)
galaxy_info:
role_name: webserver
author: DevOps Team
description: Configures Nginx web server with security hardening
company: Example Corp
license: MIT
min_ansible_version: 2.9
platforms:
- name: Ubuntu
versions:
- focal
- jammy
- name: Debian
versions:
- buster
- bullseye
galaxy_tags:
- webserver
- nginx
- security
- ssl
Ansible Vault: Secrets Management
Ansible Vault provides encryption for sensitive data like passwords, API keys, and certificates, allowing you to store secrets safely in version control.
Creating Vault Files:
# Create a new encrypted file
ansible-vault create secrets.yml
# Create encrypted variable file for production
ansible-vault create group_vars/production/vault.yml
# Encrypt existing file
ansible-vault encrypt existing-secrets.yml
# Edit encrypted file
ansible-vault edit secrets.yml
Vault File Content:
# group_vars/production/vault.yml (encrypted)
---
vault_database_password: "super_secret_db_password_123"
vault_api_key: "sk-1234567890abcdef..."
vault_ssl_private_key: |
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDXyZ...
-----END PRIVATE KEY-----
vault_oauth_credentials:
client_id: "oauth_client_123"
client_secret: "oauth_secret_xyz789"
vault_smtp_credentials:
username: "notification@example.com"
password: "smtp_password_456"
Using Vault Variables:
# group_vars/production/vars.yml (unencrypted)
---
database_host: "db.production.example.com"
database_port: 5432
database_name: "myapp_production"
database_user: "myapp_user"
database_password: "{{ vault_database_password }}"
api_key: "{{ vault_api_key }}"
ssl_private_key: "{{ vault_ssl_private_key }}"
oauth_config: "{{ vault_oauth_credentials }}"
smtp_config: "{{ vault_smtp_credentials }}"
Running Playbooks with Vault:
# Prompt for vault password
ansible-playbook site.yml --ask-vault-pass
# Use password file
ansible-playbook site.yml --vault-password-file ~/.ansible/vault-pass
# Use multiple vault passwords
ansible-playbook site.yml --vault-id prod@~/.ansible/vault-prod-pass --vault-id stage@~/.ansible/vault-stage-pass
# Environment variable for password
export ANSIBLE_VAULT_PASSWORD_FILE=~/.ansible/vault-pass
ansible-playbook site.yml
Vault Best Practices:
# Separate encrypted and unencrypted variables
# group_vars/production/
# ├── vars.yml # Unencrypted variables
# └── vault.yml # Encrypted secrets (prefixed with vault_)
# Use consistent naming convention
database_password: "{{ vault_database_password }}"
api_secret: "{{ vault_api_secret }}"
ssl_key: "{{ vault_ssl_key }}"
# Template usage with vault variables
- name: Generate application configuration
template:
src: app_config.py.j2
dest: /var/www/app/config.py
mode: '0600'
vars:
db_connection_string: "postgresql://{{ database_user }}:{{ vault_database_password }}@{{ database_host }}/{{ database_name }}"
Vault ID Management:
# Create separate vaults for different purposes
ansible-vault create --vault-id database@prompt database_secrets.yml
ansible-vault create --vault-id api@prompt api_secrets.yml
# Use in playbooks
ansible-playbook deploy.yml --vault-id database@~/.vault/db-pass --vault-id api@~/.vault/api-pass
Advanced Jinja2 Templating
Jinja2 is the templating engine used by Ansible for generating dynamic content. Advanced templating techniques enable sophisticated configuration management.
Template Filters:
{# templates/nginx.conf.j2 #}
# String manipulation
server_name {{ domain_name | lower }};
root {{ document_root | default('/var/www/html') }};
# List operations
{% for upstream in backend_servers | list %}
server {{ upstream.host }}:{{ upstream.port }} weight={{ upstream.weight | default(1) }};
{% endfor %}
# Dictionary operations
{% for key, value in ssl_config | dictsort %}
{{ key }} {{ value }};
{% endfor %}
# Numeric operations
worker_connections {{ (ansible_processor_vcpus * 1024) | int }};
client_max_body_size {{ max_upload_size | filesizeformat }};
# Date and time
# Generated on {{ ansible_date_time.iso8601 }}
# Expires: {{ (ansible_date_time.epoch | int + 86400) | strftime('%a, %d %b %Y %H:%M:%S GMT') }}
# JSON operations
{% set config_json = app_config | to_nice_json %}
var appConfig = {{ config_json }};
# URL operations
proxy_pass {{ backend_url | urlencode }};
# Regular expressions
{% if server_name | regex_match('^www\.') %}
# WWW subdomain configuration
{% endif %}
Advanced Template Logic:
{# Complex conditional logic #}
{% if ssl_enabled and environment == 'production' %}
{% set ssl_config = production_ssl_config %}
{% elif ssl_enabled and environment == 'staging' %}
{% set ssl_config = staging_ssl_config %}
{% else %}
{% set ssl_config = {} %}
{% endif %}
# SSL Configuration
{% for directive, value in ssl_config.items() %}
{{ directive }} {{ value }};
{% endfor %}
{# Loop controls and variables #}
upstream backend {
{% for server in backend_servers %}
{% if loop.index0 == 0 %}
# Primary server
{% endif %}
server {{ server.host }}:{{ server.port }}{% if server.backup | default(false) %} backup{% endif %};
{% if loop.last %}
# Total servers: {{ loop.length }}
{% endif %}
{% endfor %}
}
{# Macros for reusable template snippets #}
{% macro ssl_block(domain, cert_path, key_path) %}
ssl_certificate {{ cert_path }}/{{ domain }}.crt;
ssl_certificate_key {{ key_path }}/{{ domain }}.key;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE+AESGCM:ECDHE+AES256:ECDHE+AES128;
{% endmacro %}
# Virtual hosts with SSL
{% for vhost in virtual_hosts %}
server {
listen 443 ssl http2;
server_name {{ vhost.domain }};
{{ ssl_block(vhost.domain, ssl_cert_path, ssl_key_path) }}
location / {
root {{ vhost.document_root }};
index index.html index.php;
}
}
{% endfor %}
Custom Filters and Tests:
# filter_plugins/custom_filters.py
def format_memory(value):
"""Convert bytes to human readable format"""
for unit in ['B', 'KB', 'MB', 'GB', 'TB']:
if value < 1024.0:
return f"{value:3.1f} {unit}"
value /= 1024.0
return f"{value:.1f} PB"
def validate_ip(value):
"""Validate IP address format"""
import ipaddress
try:
ipaddress.ip_address(value)
return True
except ValueError:
return False
class FilterModule(object):
def filters(self):
return {
'format_memory': format_memory,
'validate_ip': validate_ip,
}
Using Custom Filters:
- name: Configure system with custom filters
template:
src: system.conf.j2
dest: /etc/system.conf
vars:
memory_limit: "{{ ansible_memtotal_mb * 1024 * 1024 | format_memory }}"
valid_servers: "{{ server_list | selectattr('ip', 'validate_ip') | list }}"
Dynamic Inventories and Plugins
Dynamic inventories allow Ansible to automatically discover and manage infrastructure from external sources like cloud providers, CMDBs, or custom APIs.
Google Cloud Platform Dynamic Inventory:
# inventory/gcp.yml
plugin: google.cloud.gcp_compute
projects:
- my-project-id
- my-other-project-id
regions:
- us-central1
- us-east1
- europe-west1
zones:
- us-central1-a
- us-central1-b
auth_kind: serviceaccount
service_account_file: ~/.gcp/service-account.json
# Group instances by labels and metadata
groups:
# Create groups based on instance labels
web_servers: "'web' in (labels.role | default(''))"
databases: "'db' in (labels.role | default(''))"
# Environment-based groups
production: "'prod' in (labels.env | default(''))"
staging: "'stage' in (labels.env | default(''))"
development: "'dev' in (labels.env | default(''))"
# Location-based groups
us_central: "zone.startswith('us-central')"
us_east: "zone.startswith('us-east')"
europe: "zone.startswith('europe')"
# Compose host variables from instance data
compose:
ansible_host: networkInterfaces[0].accessConfigs[0].natIP | default(networkInterfaces[0].networkIP)
internal_ip: networkInterfaces[0].networkIP
instance_type: machineType.split('/')[-1]
instance_zone: zone.split('/')[-1]
environment: labels.env | default('unknown')
role: labels.role | default('generic')
# Create keyed groups for easier targeting
keyed_groups:
- key: labels.env | default('unknown')
prefix: env
- key: labels.role | default('generic')
prefix: role
- key: zone.split('/')[-1]
prefix: zone
Custom Dynamic Inventory Script:
#!/usr/bin/env python3
# inventory/custom_inventory.py
import json
import argparse
import requests
from typing import Dict, List, Any
class CustomInventory:
def __init__(self):
self.inventory = {
'_meta': {
'hostvars': {}
}
}
def fetch_from_api(self) -> List[Dict[str, Any]]:
"""Fetch server data from custom API or CMDB"""
api_url = "https://cmdb.example.com/api/servers"
headers = {"Authorization": "Bearer YOUR_API_TOKEN"}
response = requests.get(api_url, headers=headers)
response.raise_for_status()
return response.json()
def build_inventory(self):
"""Build Ansible inventory from API data"""
servers = self.fetch_from_api()
for server in servers:
hostname = server['hostname']
# Add to _meta hostvars
self.inventory['_meta']['hostvars'][hostname] = {
'ansible_host': server['ip_address'],
'ansible_user': server.get('ssh_user', 'ubuntu'),
'server_id': server['id'],
'environment': server['environment'],
'role': server['role'],
'datacenter': server['datacenter']
}
# Create groups based on server attributes
for attr in ['environment', 'role', 'datacenter']:
group_name = server[attr]
if group_name not in self.inventory:
self.inventory[group_name] = {
'hosts': [],
'vars': {}
}
self.inventory[group_name]['hosts'].append(hostname)
def get_inventory(self) -> Dict[str, Any]:
"""Return complete inventory"""
self.build_inventory()
return self.inventory
def get_host(self, hostname: str) -> Dict[str, Any]:
"""Return host-specific variables"""
self.build_inventory()
return self.inventory['_meta']['hostvars'].get(hostname, {})
def main():
parser = argparse.ArgumentParser(description='Custom Ansible Dynamic Inventory')
parser.add_argument('--list', action='store_true', help='List all groups')
parser.add_argument('--host', help='Get variables for specific host')
args = parser.parse_args()
inventory = CustomInventory()
if args.list:
print(json.dumps(inventory.get_inventory(), indent=2))
elif args.host:
print(json.dumps(inventory.get_host(args.host), indent=2))
else:
parser.print_help()
if __name__ == '__main__':
main()
Using Dynamic Inventories:
# Test dynamic inventory
ansible-inventory -i inventory/gcp.yml --list
ansible-inventory -i inventory/custom_inventory.py --graph
# Use with playbooks
ansible-playbook -i inventory/gcp.yml site.yml
ansible-playbook -i inventory/custom_inventory.py deploy.yml
# Combine static and dynamic inventories
mkdir inventory
cp static_hosts.yml inventory/
cp gcp.yml inventory/
ansible-playbook -i inventory/ site.yml
Performance Optimization
Optimize Ansible performance for large-scale deployments and complex automation workflows.
Parallel Execution Configuration:
# ansible.cfg
[defaults]
# Increase parallel processes
forks = 50
# Reduce SSH overhead
host_key_checking = False
ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o UserKnownHostsFile=/dev/null
# Enable pipelining for faster execution
pipelining = True
# Timeout settings
timeout = 30
gather_timeout = 30
# Fact caching for performance
fact_caching = redis
fact_caching_connection = localhost:6379:0
fact_caching_timeout = 86400
# Reduce output verbosity in production
stdout_callback = minimal
[ssh_connection]
# SSH multiplexing for performance
ssh_args = -o ControlMaster=auto -o ControlPersist=600s
control_path_dir = ~/.ansible/cp
control_path = %(directory)s/%%h-%%p-%%r
Optimized Playbook Patterns:
---
- name: High-performance deployment
hosts: all
gather_facts: false # Skip fact gathering when not needed
strategy: mitogen_linear # Use mitogen for faster execution
tasks:
# Gather only required facts
- setup:
filter: "ansible_os_family"
when: need_os_info | default(false)
# Use async for long-running tasks
- name: Download large files
get_url:
url: "{{ item.url }}"
dest: "{{ item.dest }}"
async: 300
poll: 0
loop: "{{ large_files }}"
register: download_jobs
# Batch operations efficiently
- name: Install packages in batch
package:
name: "{{ packages | list }}"
state: present
vars:
packages:
- nginx
- git
- curl
- vim
# Wait for async tasks to complete
- name: Wait for downloads to complete
async_status:
jid: "{{ item.ansible_job_id }}"
register: download_results
until: download_results.finished
retries: 30
delay: 10
loop: "{{ download_jobs.results }}"
Caching Strategies:
# Use fact caching to avoid repeated fact gathering
- name: Cache expensive operations
set_fact:
expensive_computation: "{{ some_complex_calculation }}"
cacheable: yes
when: expensive_computation is not defined
# Cache external API calls
- name: Get API data
uri:
url: "https://api.example.com/data"
return_content: yes
register: api_response
cache_for: "{{ 3600 }}" # Cache for 1 hour
run_once: true
delegate_to: localhost
Integration Patterns
Integrate Ansible with external systems, APIs, and other DevOps tools for comprehensive automation workflows.
Terraform Integration:
# terraform/outputs.tf
output "web_servers" {
value = {
for instance in google_compute_instance.web_servers :
instance.name => {
internal_ip = instance.network_interface[0].network_ip
external_ip = instance.network_interface[0].access_config[0].nat_ip
zone = instance.zone
}
}
}
# Generate Ansible inventory from Terraform state
- name: Generate inventory from Terraform
hosts: localhost
gather_facts: false
tasks:
- name: Get Terraform outputs
shell: terraform output -json
register: tf_output
changed_when: false
- name: Parse Terraform outputs
set_fact:
terraform_data: "{{ tf_output.stdout | from_json }}"
- name: Generate Ansible inventory
template:
src: inventory.yml.j2
dest: ./inventory/terraform.yml
vars:
web_servers: "{{ terraform_data.web_servers.value }}"
API Integration:
# Integration with external APIs
- name: Deploy with external service integration
hosts: localhost
tasks:
- name: Notify deployment start
uri:
url: "https://api.slack.com/api/chat.postMessage"
method: POST
headers:
Authorization: "Bearer {{ slack_bot_token }}"
Content-Type: "application/json"
body_format: json
body:
channel: "#deployments"
text: "Starting deployment of {{ app_name }} version {{ app_version }}"
- name: Update deployment status in monitoring system
uri:
url: "https://monitoring.example.com/api/deployments"
method: POST
headers:
Authorization: "Bearer {{ monitoring_api_token }}"
body_format: json
body:
application: "{{ app_name }}"
version: "{{ app_version }}"
status: "in_progress"
environment: "{{ environment }}"
- name: Register with service discovery
uri:
url: "https://consul.example.com/v1/agent/service/register"
method: PUT
body_format: json
body:
ID: "{{ service_id }}"
Name: "{{ service_name }}"
Address: "{{ ansible_default_ipv4.address }}"
Port: "{{ service_port }}"
Check:
HTTP: "https://{{ ansible_default_ipv4.address }}:{{ service_port }}/health"
Interval: "10s"
CI/CD Pipeline Integration:
# .github/workflows/deploy.yml
name: Deploy with Ansible
on:
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Setup Ansible
run: |
pip install ansible google-auth requests
ansible-galaxy collection install google.cloud
- name: Configure GCP credentials
run: echo '${{ secrets.GCP_SERVICE_ACCOUNT }}' > ~/.gcp/service-account.json
- name: Deploy infrastructure
run: |
cd infrastructure/
terraform init
terraform apply -auto-approve
- name: Configure infrastructure
run: |
cd configuration/
ansible-playbook -i inventory/gcp.yml site.yml
env:
ANSIBLE_VAULT_PASSWORD_FILE: ${{ secrets.VAULT_PASSWORD_FILE }}
Note
The advanced features in this chapter form the foundation for enterprise-grade automation. Focus on understanding the concepts and patterns rather than memorizing syntax - the goal is to build maintainable, scalable automation that grows with your infrastructure needs.