CI/CD Pipelines
Automated software delivery pipelines that test, validate, and deploy code changes safely to production.
Continuous Integration (CI)
Core Workflow:
name: Basic CI Pipeline
# Simple pipeline demonstrating modern Python CI/CD with GitHub Actions
on:
push: # Run on every push
branches: [ main, develop ] # Target branches
pull_request: # Run on PRs
branches: [ main ] # PR target branch
jobs:
ci:
runs-on: ubuntu-latest # Use Ubuntu runner
strategy:
matrix: # Test multiple versions
python-version: ["3.11", "3.12", "3.13"]
steps:
- name: Checkout code # Get repository code
uses: actions/checkout@v4 # Latest stable version
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5 # Install Python
with:
python-version: ${{ matrix.python-version }}
- name: Install uv (modern Python package manager)
uses: astral-sh/setup-uv@v3 # Ultra-fast package manager
with:
enable-cache: true # Cache dependencies
- name: Install dependencies
run: |
uv venv # Create virtual environment
uv pip install pytest ruff mypy # Install dev tools
- name: Code formatting check
run: |
uv run ruff format --check . # Check code formatting
- name: Linting
run: |
uv run ruff check . # Run linter
- name: Type checking
run: |
uv run mypy *.py # Static type checking
- name: Run tests
run: |
uv run python hello_world.py # Test our script
echo "✅ All tests passed!" # Success message
Key Components:
Automated builds on every commit
Multi-version testing (Python 3.11, 3.12, 3.13)
Code quality checks (formatting, linting, type checking)
Fast feedback (<5 minutes for basic validation)
Continuous Deployment (CD)
Production Pipeline:
name: Production Python CI/CD Pipeline
# Comprehensive pipeline with modern tooling, security, and deployment
on:
push:
branches: [ main, develop ] # Deploy branches
tags: [ 'v*' ] # Release tags
pull_request:
branches: [ main ] # PR validation
env:
PYTHON_VERSION: "3.12" # Default Python version
jobs:
# Stage 1: Fast quality checks (fail fast principle)
quality:
runs-on: ubuntu-latest
outputs:
cache-hit: ${{ steps.cache.outputs.cache-hit }}
steps:
- name: Checkout code
uses: actions/checkout@v4 # Get repository code
- name: Set up Python ${{ env.PYTHON_VERSION }}
uses: actions/setup-python@v5 # Install Python
with:
python-version: ${{ env.PYTHON_VERSION }}
- name: Install uv (modern package manager)
uses: astral-sh/setup-uv@v3 # Fast dependency management
with:
enable-cache: true # Cache for speed
cache-dependency-glob: "pyproject.toml"
- name: Cache dependencies
id: cache # Cache Python packages
uses: actions/cache@v4
with:
path: ~/.cache/uv
key: ${{ runner.os }}-uv-${{ hashFiles('pyproject.toml') }}
restore-keys: ${{ runner.os }}-uv-
- name: Install dependencies
run: |
uv sync --dev # Install all dependencies
- name: Code formatting check
run: |
uv run ruff format --check . # Check code formatting
Pipeline Stages:
Quality checks (linting, formatting, type checking)
Multi-platform testing (Ubuntu, Windows, macOS)
Security scanning (vulnerability detection, dependency audit)
Package building (wheel and source distributions)
Environment deployment (staging → production)
Monitoring and cleanup
Modern Python CI/CD Stack
Essential Tools (2024):
# Modern pyproject.toml configuration
[project.optional-dependencies]
dev = [
"uv>=0.1.0", # Fast package manager (replaces pip)
"ruff>=0.1.0", # Fast linter and formatter (replaces flake8/black/isort)
"mypy>=1.5.0", # Static type checking
"pytest>=7.0.0", # Testing framework
"bandit>=1.7.0", # Security scanning
]
Platform Comparison
GitHub Actions (Recommended for most teams)
Native GitHub integration
2,000 free minutes/month for private repos
Excellent marketplace ecosystem
Simple YAML configuration
GitLab CI/CD
All-in-one DevOps platform
Built-in container registry
Advanced deployment features
Self-hosted options
Jenkins
Maximum customization
Large plugin ecosystem
Higher maintenance overhead
Self-hosted only
Key Metrics
Deployment Frequency
Target: Daily deployments
Measure: Commits per day reaching production
Lead Time
Target: <4 hours commit to production
Measure: Time from code commit to user availability
Change Failure Rate
Target: <5% of deployments require rollback
Measure: Failed deployments / total deployments
Mean Time to Recovery
Target: <1 hour for rollbacks
Measure: Detection to resolution time
Container Integration
Docker + CI/CD Pattern:
# Production-ready Dockerfile for Python CLI application
# Multi-stage build for optimized image size and security
# Stage 1: Build environment with all development tools
FROM python:3.12-slim as builder
# Set environment variables for build stage
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
ENV UV_COMPILE_BYTECODE=1
ENV UV_LINK_MODE=copy
# Install system dependencies needed for building
RUN apt-get update && apt-get install -y \
build-essential \
curl \
--no-install-recommends && \
rm -rf /var/lib/apt/lists/*
# Install uv (modern Python package manager)
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
# Set working directory
WORKDIR /app
# Copy dependency files first (for better caching)
COPY pyproject.toml uv.lock ./
# Install dependencies in virtual environment
RUN uv venv /opt/venv
Key Features:
Multi-stage builds for minimal image size
Non-root user for security
Health checks for deployment validation
Dependency caching for faster builds
Common Solutions
Slow Tests
Use test parallelization and caching
Implement smart test selection
Container-based consistent environments
Security Integration
Automated vulnerability scanning with bandit
Dependency security checks with safety
Built-in compliance validation
Cost Optimization
Intelligent caching strategies
Conditional job execution
Right-sized runner selection
Quick Start
1. Basic Pipeline (5 minutes):
#!/usr/bin/env python3
"""
Basic Python script demonstrating CI/CD pipeline testing.
This simple example shows proper function structure, error handling,
and exit codes that work well with automated testing.
"""
import sys
from typing import Optional
def greet(name: str, greeting: str = "Hello") -> str:
"""
Generate a greeting message.
Args:
name: The name to greet
greeting: The greeting word (default: "Hello")
Returns:
Formatted greeting string
"""
if not name or not name.strip(): # Validate input
raise ValueError("Name cannot be empty")
return f"{greeting}, {name.strip()}!" # Clean and format
def main() -> int:
"""
Main function with proper exit codes for CI/CD.
Returns:
0 for success, 1 for error (standard Unix conventions)
"""
try:
# Get name from command line args or use default
name: Optional[str] = sys.argv[1] if len(sys.argv) > 1 else "World"
if name is None: # Handle None case
print("Error: No name provided")
return 1
message = greet(name) # Generate greeting
print(message) # Output result
return 0 # Success exit code
except ValueError as e: # Handle validation errors
print(f"Error: {e}", file=sys.stderr)
return 1
except Exception as e: # Handle unexpected errors
print(f"Unexpected error: {e}", file=sys.stderr)
return 1
if __name__ == "__main__":
sys.exit(main()) # Proper exit code handling
2. GitHub Actions Setup:
name: Basic CI Pipeline
# Simple pipeline demonstrating modern Python CI/CD with GitHub Actions
on:
push: # Run on every push
branches: [ main, develop ] # Target branches
pull_request: # Run on PRs
branches: [ main ] # PR target branch
jobs:
ci:
runs-on: ubuntu-latest # Use Ubuntu runner
strategy:
matrix: # Test multiple versions
python-version: ["3.11", "3.12", "3.13"]
steps:
- name: Checkout code # Get repository code
uses: actions/checkout@v4 # Latest stable version
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5 # Install Python
with:
python-version: ${{ matrix.python-version }}
3. Production Template:
Full production-ready examples available in source_code/pipelines/
basic/- Getting started examplesadvanced/- Production patternstemplates/- Reusable configurationsexamples/- Complete applications