Forge — User Guide

1. Quick Start

Forge is Cisco's open-source Terraform/Terragrunt framework for provisioning and managing self-hosted GitHub Actions runners on AWS. It supports two runner backends — EC2 (VM-based) and ARC (Actions Runner Controller on EKS/Kubernetes) — with multi-tenant isolation, observability integrations, and automated lifecycle management.

Key features:

• Multi-tenant GitHub Actions runner infrastructure on AWS
• EC2 runners with auto-scaling, AMI management, and spot/on-demand support
• EKS-based ARC runners with Helm-managed scale sets and Karpenter autoscaling
• Splunk Cloud and Splunk Observability integrations (billing, logs, dashboards)
• GitHub webhook relay via EventBridge for cross-account event forwarding
• Teleport integration for secure cluster access
• Terragrunt-based hierarchical configuration (global → environment → region → VPC → tenant)
• Comprehensive pre-commit hooks for code quality and security scanning

Prerequisites:

• Terraform ~> 1.11 or OpenTofu
• Terragrunt
• AWS CLI configured with appropriate credentials
• A GitHub App configured for your organization

Minimal setup:

git clone git@github.com:cisco-open/forge.git
cd forge/examples/infra-forge

Tip

Start with the examples/infra-forge deployment — it provisions the foundational infrastructure (EKS, ECR, storage, AMI policies) needed before deploying runners.

What to explore next:

• Configuration section for Terragrunt hierarchy details
• Common Tasks section for adding your first tenant
• Admin Guide for full architecture and module reference

2. Installation

System Requirements

Clone the Repository

git clone git@github.com:cisco-open/forge.git
cd forge

Install Pre-Commit Hooks

Forge uses an extensive set of pre-commit hooks for code quality:

pip install pre-commit
pre-commit install --install-hooks
pre-commit install --hook-type commit-msg

Important

The pre-commit configuration requires Python 3.12. Hooks include Terraform formatting and validation, YAML linting, shell checking (shellcheck, shfmt), security scanning (Gitleaks), Ansible linting, and Dockerfile linting (hadolint).

Verify Provider Access

Ensure your AWS credentials are configured:

aws sts get-caller-identity

{
    "UserId": "AIDAEXAMPLE",
    "Account": "123456789012",
    "Arn": "arn:aws:iam::123456789012:user/deployer"
}

Post-Install Verification

Validate a module to confirm your toolchain works:

cd modules/core/arc
terraform init -backend=false
terraform validate

Success! The configuration is valid.

Warning

Do not run terraform apply on individual modules directly. Forge is designed to be deployed through Terragrunt, which handles state management, dependency ordering, and hierarchical configuration injection.

3. Configuration

Forge uses a Terragrunt hierarchical configuration pattern. Settings cascade from global to specific, with each level able to override or extend the parent.

Configuration Hierarchy

terragrunt/
├── root.hcl                           # Root Terragrunt config
├── terragrunt.hcl                     # Top-level includes
├── _global_settings/
│   ├── _global.hcl                    # Global HCL settings
│   ├── _global.yaml                   # Global YAML values
│   ├── <module>.hcl                   # Per-module source references
│   └── terragrunt.hcl
├── environments/
│   ├── safety_hook.sh                 # Pre-apply safety check
│   ├── terragrunt.hcl
│   └── <env>/                         # e.g., dev, prod
│       ├── _environment_wide_settings/
│       │   ├── _environment.hcl
│       │   └── _environment.yaml
│       └── regions/
│           └── <region>/              # e.g., eu-west-1
│               ├── _region_wide_settings/
│               │   └── _region.hcl
│               └── <module>/          # Module deployment
│                   ├── config.hcl
│                   ├── config.yaml
│                   └── terragrunt.hcl

Release Versions

Each deployment references module versions through a release_versions.yaml file:

api_version: v1
product: Forge
cloud_provider: aws
version: v0.0.0
spec:
  iac:
    modules:
      forge_runners:
        local_path: modules/forge-modules
        repo: git@github.com:cisco-open/forge.git
        ref: main
      teleport:
        local_path: modules/forge-modules
        repo: git@github.com:cisco-open/forge.git
        ref: main
      splunk_cloud_data_manager:
        local_path: modules/infra-modules
        repo: git@github.com:cisco-open/forge.git
        ref: main
      splunk_conf:
        local_path: modules/forge-modules
        repo: git@github.com:cisco-open/forge.git
        ref: main

Note

Set use_local_repos: true in release_versions.yaml during development to reference local module paths instead of fetching from the remote repository.

Deployment Stacks

Forge organizes deployments into logical stacks:

Tip

Use the examples/templates/ directory as a starting point for new deployments. Templates provide pre-configured YAML files for each stack type.

Configuration Templates

Templates for each deployment type live under examples/templates/:

See the Admin Guide for the full configuration reference covering every module variable.

4. Core Usage

Deploying Foundation Infrastructure

The infra-forge stack provisions the AWS infrastructure that runners depend on:

cd examples/infra-forge/terragrunt/environments/dev
terragrunt run-all plan

[INFO] Planning modules in dependency order...

After reviewing the plan:

terragrunt run-all apply

Note

The safety_hook.sh script runs before every apply as a Terragrunt before_hook. It provides environment validation to prevent accidental production changes.

Deploying Runners

Once the foundation infrastructure is in place, deploy the tenant runner stack:

cd examples/deployments/forge-tenant/terragrunt/environments/prod
terragrunt run-all plan

Forge supports two runner backends simultaneously:

EC2 Runners — VM-based runners with AMI management:

• Auto-scaling based on GitHub Actions demand
• Spot and on-demand instance support via instance_target_capacity_type
• Custom AMI with configurable EBS volumes
• Platform support for Linux, macOS, and Windows (dedicated user data templates for each)

ARC Runners — Kubernetes-based runners on EKS:

• Helm-managed scale sets with configurable min_runners / max_runners
• Docker-in-Docker (DinD) or Kubernetes mode via scale_set_type
• Karpenter-based node autoscaling
• Per-runner IAM via EKS Pod Identity

Warning

EC2 and ARC runners have different IAM requirement profiles. EC2 runners use instance profiles while ARC runners use EKS Pod Identity associations. Ensure the correct policies are attached for each backend.

Managing the GitHub App

Forge uses a GitHub App for runner registration and webhook integration. The app configuration is stored in AWS SSM Parameter Store:

The webhook secret auto-rotates via a time_rotating Terraform resource. A null_resource provisioner calls scripts/generate_and_patch_github_app.sh to update the GitHub App webhook URL and secret.

5. Common Tasks

Adding a New Tenant

1. Copy the tenant template:

cp -r examples/templates/tenant/tenant/ \
  examples/deployments/forge-tenant/terragrunt/environments/prod/regions/eu-west-1/vpcs/sl/tenants/<new-tenant>/

2. Edit config.yaml with the tenant's settings (runner labels, instance types, scaling parameters).

3. Update the VPC-wide settings (_vpc.yaml, _vpc.hcl) if the new tenant requires network changes.

4. Plan and apply:

cd examples/deployments/forge-tenant/terragrunt/environments/prod
terragrunt run-all plan
terragrunt run-all apply

Updating Runner AMIs

AMI updates are handled by the ec2_update_runner_ssm_ami Lambda, which runs on a schedule via EventBridge. It:

1. Queries for the latest AMI matching configured filters (ami_filter.name, ami_filter.state)
2. Updates the SSM parameter with the new AMI ID
3. New runners launched after the update use the new AMI automatically

Scaling Runner Pools

EC2 runners support scheduled scaling via pool_config:

pool_config:
  - size: 2
    schedule_expression: "cron(0 8 ? * MON-FRI *)"
    schedule_expression_timezone: "America/New_York"
  - size: 0
    schedule_expression: "cron(0 20 ? * MON-FRI *)"
    schedule_expression_timezone: "America/New_York"

ARC runners scale between min_runners and max_runners based on GitHub Actions queue demand, managed by the ARC scale set controller.

Setting Up Splunk Observability

Deploy the Splunk deployment stack for full observability:

cd examples/deployments/splunk-deployment/terragrunt/environments/prod
terragrunt run-all apply

This provisions:

• Splunk Cloud Data Manager for AWS data ingestion
• Splunk O11y AWS integration with SignalFx
• OTel collector on EKS via Helm
• AWS billing data pipeline (per-service and per-resource CUR data) to Splunk via Lambda + S3

Tip

Forge provisions seven SignalFx dashboard types automatically: billing, DynamoDB, EBS, Lambda, runner EC2, runner K8s, and SQS. Each dashboard supports dynamic variables for filtering by tenant.

Enabling GitHub Webhook Relay

The webhook relay forwards GitHub events across AWS accounts via EventBridge:

github_webhook_relay = {
  enabled                    = true
  destination_account_id     = "987654321098"
  destination_event_bus_name = "webhook-relay-destination"
  destination_region         = "eu-west-1"
}

The source module creates an API Gateway + Lambda that validates webhook signatures before forwarding events to EventBridge. Destination modules (e.g., Webex webhook relay) receive events via EventBridge rules.

Migrating Tenants

Use the provided migration scripts:

# Migrate a single tenant
./scripts/migrate-tenant.sh <tenant-name>

# Migrate all tenants
./scripts/migrate-all-tenants.sh

Updating Terraform Documentation

./scripts/update-terraform-docs.sh

This regenerates all README.md files in modules/ using terraform-docs with the configuration in .terraform-docs.yml.

6. Troubleshooting

Error: terraform init fails with provider version conflicts

Cause: Forge modules require specific provider versions (e.g., AWS >= 6.25, Helm >= 3.0.0). Older cached providers may conflict.

Solution:

1. Clear the provider cache:

   rm -rf .terraform .terraform.lock.hcl
   

2. Re-initialize:

   terraform init -upgrade
   

Prevention: Use the OpenTofu provider registry as configured in renovate.json for consistent resolution.

Error: Terragrunt run-all apply fails with dependency errors

Cause: Modules have implicit dependencies that Terragrunt may not resolve automatically if dependency blocks are misconfigured.

Solution:

1. Visualize the dependency tree:

   terragrunt graph-dependencies
   

2. Apply dependent modules individually in correct order
3. Verify all dependency blocks in terragrunt.hcl files point to valid paths

Prevention: Always run terragrunt run-all plan before apply to catch dependency issues early.

Error: GitHub App authentication failures in Lambda functions

Cause: SSM parameters containing GitHub App credentials may be stale, or the private key may have been rotated without updating the parameter.

Solution:

1. Verify the SSM parameters:

   aws ssm get-parameter --name "/forge/<prefix>/github_app_id" --with-decryption
   

2. Re-run the update script:

   ./scripts/update-github-app-secrets.sh
   

Prevention: Monitor Lambda CloudWatch logs for jwt_error or installation_token_error patterns.

Warning

Never store GitHub App private keys in plaintext configuration files. Forge stores them as SecureString SSM parameters.

Error: EKS cluster creation timeout

Cause: The null_resource.wait_for_cluster polls the cluster endpoint. Network issues or security group misconfiguration can cause timeouts.

Solution:

1. Check EKS cluster status in the AWS Console
2. Retry the apply — Terraform resumes from the last successful state
3. Verify VPC networking and security group rules allow the Terraform execution environment to reach the EKS API endpoint

Prevention: Ensure the VPC has NAT gateway configuration and the external_access_cidr_blocks variable includes the CIDR of your deployment environment.

Error: Karpenter node provisioning fails

Cause: Karpenter requires specific IAM permissions and the EC2NodeClass/NodePool custom resources must be applied after the cluster is ready.

Solution:

1. Verify the Karpenter IAM role has required policies
2. Check that null_resource.karpenter, null_resource.apply_ec2_node_class, and null_resource.apply_node_pool completed
3. Review Karpenter controller logs:

   kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter
   

Error: Pre-commit hooks fail on commit

Cause: Missing tool dependencies or version mismatches.

Solution:

1. Ensure required tools are installed: shellcheck, shfmt, tflint, hadolint, ansible-lint, checkmake
2. Update hooks:

   pre-commit autoupdate
   pre-commit run --all-files
   

Prevention: The .pre-commit-config.yaml pins specific versions. Use Renovate to keep hooks updated.

Caution

Forge uses commitizen with cz-conventional-gitmoji for commit message validation. Commits that do not follow the conventional commit format will be rejected at the commit-msg stage.

Error: Splunk HEC ingestion failures in billing Lambdas

Cause: The Splunk HEC endpoint URL or token may be misconfigured, or the target index may not exist.

Solution:

1. Verify Splunk secrets in AWS Secrets Manager
2. Check Lambda CloudWatch logs for HTTP response codes
3. Test the HEC endpoint:

   curl -k https://<splunk-hec-url>/services/collector \
     -H "Authorization: Splunk <token>" \
     -d '{"event": "test"}'
   

Prevention: Use the splunk_secrets module to manage HEC tokens consistently across environments.

Issue: Branch protection violation (GitHub #287)

Cause: Security policy requires branch protection rules that may not be configured for all branches.

Solution: Ensure branch protection is enabled on the main branch with required status checks and review requirements as specified by your organization's security policy.

Issue: Cost safeguards for runner environments (GitHub #296)

Cause: Runner environments can incur significant costs without proper guardrails.

Solution: Configure max_instances for EC2 runner pools and max_runners for ARC scale sets. Consider implementing Cloud Custodian policies via the cloud_custodian module for automated cost controls.

7. FAQ

Q: Can I use Forge with GitHub.com or only GitHub Enterprise Server?

Forge supports both. Set ghes_url to an empty string "" for public GitHub.com, or provide your GHES base URL (e.g., https://github.mycompany.com).

Q: Can I run EC2 and ARC runners simultaneously for the same tenant?

Yes. The forge_runners module composes both ec2_deployment and arc_deployment submodules. Configure ec2_deployment_specs and arc_deployment_specs in the same deployment to run both backends.

Q: What runner operating systems are supported for EC2?

Linux, macOS, and Windows. Each has a dedicated user data template (user_data_linux.tftpl, user_data_osx.tftpl, user_data_windows.tftpl) in modules/platform/ec2_deployment/template_files/.

Q: How does the multi-tenant isolation work?

Each tenant gets its own GitHub App installation, runner group, IAM roles, and optionally dedicated subnets. The forge_trust_validator Lambda periodically validates IAM trust relationships to ensure tenants cannot escalate privileges across boundaries.

Q: What is the safety_hook.sh script?

A Terragrunt before_hook that runs before every apply. It validates the target environment and provides a confirmation checkpoint, preventing accidental production changes.

Q: How are Splunk dashboards managed?

Forge provisions SignalFx dashboards via the splunk_o11y_conf_shared module using the signalfx Terraform provider (< 10.0.0). Seven dashboard types are included: billing, DynamoDB, EBS, Lambda, runner EC2, runner K8s, and SQS. Each supports dynamic variables and tenant-based filtering.

Q: How do I update Terraform module documentation?

Run ./scripts/update-terraform-docs.sh. This uses terraform-docs with the .terraform-docs.yml config to inject auto-generated docs between <!-- BEGIN_TF_DOCS --> and <!-- END_TF_DOCS --> markers.

Tip

Do not manually edit content between the TF_DOCS markers — it will be overwritten on the next documentation generation run.

Q: What dependency management does Forge use?

Renovate, configured via renovate.json. It includes custom regex managers for Terraform provider versions, GitHub runner Lambda versions, and Dockerfile versions. Key behaviors: security patches auto-merge, patch/pin/digest updates auto-merge, major updates require 7-day stability, and GitHub Actions are pinned to commit SHAs for security.

Q: What are the open feature requests I should know about?

As of commit 1a8b9a7, open feature requests include: lifecycle field refactoring (#236), Grafana/PagerDuty alerting integration (#125, #124), AWS IAM Roles Anywhere for multi-cloud runners (#122), and open-source monitoring stack support (#121).