# Forge — User Guide

## 1. Quick Start

Forge is Cisco's open-source Terraform/Terragrunt framework for provisioning and managing self-hosted GitHub Actions runners on AWS. It supports two runner backends — **EC2** (VM-based) and **ARC** (Actions Runner Controller on EKS/Kubernetes) — with multi-tenant isolation, observability integrations, and automated lifecycle management.

**Key features:**

- Multi-tenant GitHub Actions runner infrastructure on AWS
- EC2 runners with auto-scaling, AMI management, and spot/on-demand support
- EKS-based ARC runners with Helm-managed scale sets and Karpenter autoscaling
- Splunk Cloud and Splunk Observability integrations (billing, logs, dashboards)
- GitHub webhook relay via EventBridge for cross-account event forwarding
- Teleport integration for secure cluster access
- Terragrunt-based hierarchical configuration (global → environment → region → VPC → tenant)
- Comprehensive pre-commit hooks for code quality and security scanning

**Prerequisites:**

- Terraform ~> 1.11 or OpenTofu
- Terragrunt
- AWS CLI configured with appropriate credentials
- A GitHub App configured for your organization

**Minimal setup:**

```bash
git clone git@github.com:cisco-open/forge.git
cd forge/examples/infra-forge
```

> [!TIP]
> Start with the `examples/infra-forge` deployment — it provisions the foundational infrastructure (EKS, ECR, storage, AMI policies) needed before deploying runners.

**What to explore next:**

- Configuration section for Terragrunt hierarchy details
- Common Tasks section for adding your first tenant
- Admin Guide for full architecture and module reference

## 2. Installation

### System Requirements

| Requirement | Version |
|---|---|
| Terraform / OpenTofu | ~> 1.11 |
| Terragrunt | Latest stable |
| AWS Provider | >= 6.25 |
| Helm Provider | >= 3.0.0 |
| Kubernetes Provider | >= 3.0 |
| Python | 3.12 (for pre-commit hooks and Lambda functions) |
| AWS CLI | v2 |
| `kubectl` | Compatible with your EKS cluster version |

### Clone the Repository

```bash
git clone git@github.com:cisco-open/forge.git
cd forge
```

### Install Pre-Commit Hooks

Forge uses an extensive set of pre-commit hooks for code quality:

```bash
pip install pre-commit
pre-commit install --install-hooks
pre-commit install --hook-type commit-msg
```

> [!IMPORTANT]
> The pre-commit configuration requires Python 3.12. Hooks include Terraform formatting and validation, YAML linting, shell checking (shellcheck, shfmt), security scanning (Gitleaks), Ansible linting, and Dockerfile linting (hadolint).

### Verify Provider Access

Ensure your AWS credentials are configured:

```bash
aws sts get-caller-identity
```

```text
{
    "UserId": "AIDAEXAMPLE",
    "Account": "123456789012",
    "Arn": "arn:aws:iam::123456789012:user/deployer"
}
```

### Post-Install Verification

Validate a module to confirm your toolchain works:

```bash
cd modules/core/arc
terraform init -backend=false
terraform validate
```

```text
Success! The configuration is valid.
```

> [!WARNING]
> Do not run `terraform apply` on individual modules directly. Forge is designed to be deployed through Terragrunt, which handles state management, dependency ordering, and hierarchical configuration injection.

## 3. Configuration

Forge uses a **Terragrunt hierarchical configuration** pattern. Settings cascade from global to specific, with each level able to override or extend the parent.

### Configuration Hierarchy

```text
terragrunt/
├── root.hcl                           # Root Terragrunt config
├── terragrunt.hcl                     # Top-level includes
├── _global_settings/
│   ├── _global.hcl                    # Global HCL settings
│   ├── _global.yaml                   # Global YAML values
│   ├── <module>.hcl                   # Per-module source references
│   └── terragrunt.hcl
├── environments/
│   ├── safety_hook.sh                 # Pre-apply safety check
│   ├── terragrunt.hcl
│   └── <env>/                         # e.g., dev, prod
│       ├── _environment_wide_settings/
│       │   ├── _environment.hcl
│       │   └── _environment.yaml
│       └── regions/
│           └── <region>/              # e.g., eu-west-1
│               ├── _region_wide_settings/
│               │   └── _region.hcl
│               └── <module>/          # Module deployment
│                   ├── config.hcl
│                   ├── config.yaml
│                   └── terragrunt.hcl
```

### Release Versions

Each deployment references module versions through a `release_versions.yaml` file:

```yaml
api_version: v1
product: Forge
cloud_provider: aws
version: v0.0.0
spec:
  iac:
    modules:
      forge_runners:
        local_path: modules/forge-modules
        repo: git@github.com:cisco-open/forge.git
        ref: main
      teleport:
        local_path: modules/forge-modules
        repo: git@github.com:cisco-open/forge.git
        ref: main
      splunk_cloud_data_manager:
        local_path: modules/infra-modules
        repo: git@github.com:cisco-open/forge.git
        ref: main
      splunk_conf:
        local_path: modules/forge-modules
        repo: git@github.com:cisco-open/forge.git
        ref: main
```

> [!NOTE]
> Set `use_local_repos: true` in `release_versions.yaml` during development to reference local module paths instead of fetching from the remote repository.

### Deployment Stacks

Forge organizes deployments into logical stacks:

| Stack | Purpose | Key Modules |
|---|---|---|
| `infra-forge` | Foundation infrastructure | EKS, ECR, storage, AMI policies, CloudFormation, service-linked roles |
| `forge-tenant` | Per-tenant runner deployment | Runner configuration, GitHub App, webhook relay |
| `forge-eks` | EKS cluster configuration | EKS cluster, Karpenter, Calico networking |
| `forge-extras` | Optional add-ons | Cloud Custodian, ECR, Forge subscription |
| `forge-integrations` | Third-party integrations | Splunk Cloud configuration |
| `splunk-deployment` | Splunk observability | Splunk O11y, OTel, Data Manager, billing |

> [!TIP]
> Use the `examples/templates/` directory as a starting point for new deployments. Templates provide pre-configured YAML files for each stack type.

### Configuration Templates

Templates for each deployment type live under `examples/templates/`:

| Template Path | Purpose |
|---|---|
| `templates/tenant/` | New tenant configuration (config.yaml, runner_settings.hcl, terragrunt.hcl) |
| `templates/eks/` | EKS cluster settings |
| `templates/extras/` | Optional infrastructure add-ons (Cloud Custodian, ECR, Forge subscription) |
| `templates/splunk/` | Splunk integration settings (Data Manager, O11y, OTel, secrets, billing) |

See the Admin Guide for the full configuration reference covering every module variable.

## 4. Core Usage

### Deploying Foundation Infrastructure

The `infra-forge` stack provisions the AWS infrastructure that runners depend on:

```bash
cd examples/infra-forge/terragrunt/environments/dev
terragrunt run-all plan
```

```text
[INFO] Planning modules in dependency order...
```

After reviewing the plan:

```bash
terragrunt run-all apply
```

> [!NOTE]
> The `safety_hook.sh` script runs before every apply as a Terragrunt `before_hook`. It provides environment validation to prevent accidental production changes.

### Deploying Runners

Once the foundation infrastructure is in place, deploy the tenant runner stack:

```bash
cd examples/deployments/forge-tenant/terragrunt/environments/prod
terragrunt run-all plan
```

Forge supports two runner backends simultaneously:

**EC2 Runners** — VM-based runners with AMI management:

- Auto-scaling based on GitHub Actions demand
- Spot and on-demand instance support via `instance_target_capacity_type`
- Custom AMI with configurable EBS volumes
- Platform support for Linux, macOS, and Windows (dedicated user data templates for each)

**ARC Runners** — Kubernetes-based runners on EKS:

- Helm-managed scale sets with configurable `min_runners` / `max_runners`
- Docker-in-Docker (DinD) or Kubernetes mode via `scale_set_type`
- Karpenter-based node autoscaling
- Per-runner IAM via EKS Pod Identity

> [!WARNING]
> EC2 and ARC runners have different IAM requirement profiles. EC2 runners use instance profiles while ARC runners use EKS Pod Identity associations. Ensure the correct policies are attached for each backend.

### Managing the GitHub App

Forge uses a GitHub App for runner registration and webhook integration. The app configuration is stored in AWS SSM Parameter Store:

| Parameter | Purpose |
|---|---|
| `github_app_id` | GitHub App numeric ID |
| `github_app_client_id` | OAuth client ID |
| `github_app_installation_id` | Installation ID for the organization |
| `github_app_key` | Private key (base64-encoded, SecureString) |
| `github_app_name` | App display name |
| `github_app_webhook_secret` | Webhook signature secret (auto-rotated every 30 days) |

The webhook secret auto-rotates via a `time_rotating` Terraform resource. A `null_resource` provisioner calls `scripts/generate_and_patch_github_app.sh` to update the GitHub App webhook URL and secret.

## 5. Common Tasks

### Adding a New Tenant

1. Copy the tenant template:

```bash
cp -r examples/templates/tenant/tenant/ \
  examples/deployments/forge-tenant/terragrunt/environments/prod/regions/eu-west-1/vpcs/sl/tenants/<new-tenant>/
```

2. Edit `config.yaml` with the tenant's settings (runner labels, instance types, scaling parameters).

3. Update the VPC-wide settings (`_vpc.yaml`, `_vpc.hcl`) if the new tenant requires network changes.

4. Plan and apply:

```bash
cd examples/deployments/forge-tenant/terragrunt/environments/prod
terragrunt run-all plan
terragrunt run-all apply
```

### Updating Runner AMIs

AMI updates are handled by the `ec2_update_runner_ssm_ami` Lambda, which runs on a schedule via EventBridge. It:

1. Queries for the latest AMI matching configured filters (`ami_filter.name`, `ami_filter.state`)
2. Updates the SSM parameter with the new AMI ID
3. New runners launched after the update use the new AMI automatically

### Scaling Runner Pools

**EC2 runners** support scheduled scaling via `pool_config`:

```yaml
pool_config:
  - size: 2
    schedule_expression: "cron(0 8 ? * MON-FRI *)"
    schedule_expression_timezone: "America/New_York"
  - size: 0
    schedule_expression: "cron(0 20 ? * MON-FRI *)"
    schedule_expression_timezone: "America/New_York"
```

**ARC runners** scale between `min_runners` and `max_runners` based on GitHub Actions queue demand, managed by the ARC scale set controller.

### Setting Up Splunk Observability

Deploy the Splunk deployment stack for full observability:

```bash
cd examples/deployments/splunk-deployment/terragrunt/environments/prod
terragrunt run-all apply
```

This provisions:

- Splunk Cloud Data Manager for AWS data ingestion
- Splunk O11y AWS integration with SignalFx
- OTel collector on EKS via Helm
- AWS billing data pipeline (per-service and per-resource CUR data) to Splunk via Lambda + S3

> [!TIP]
> Forge provisions seven SignalFx dashboard types automatically: billing, DynamoDB, EBS, Lambda, runner EC2, runner K8s, and SQS. Each dashboard supports dynamic variables for filtering by tenant.

### Enabling GitHub Webhook Relay

The webhook relay forwards GitHub events across AWS accounts via EventBridge:

```hcl
github_webhook_relay = {
  enabled                    = true
  destination_account_id     = "987654321098"
  destination_event_bus_name = "webhook-relay-destination"
  destination_region         = "eu-west-1"
}
```

The source module creates an API Gateway + Lambda that validates webhook signatures before forwarding events to EventBridge. Destination modules (e.g., Webex webhook relay) receive events via EventBridge rules.

### Migrating Tenants

Use the provided migration scripts:

```bash
# Migrate a single tenant
./scripts/migrate-tenant.sh <tenant-name>

# Migrate all tenants
./scripts/migrate-all-tenants.sh
```

### Updating Terraform Documentation

```bash
./scripts/update-terraform-docs.sh
```

This regenerates all `README.md` files in `modules/` using `terraform-docs` with the configuration in `.terraform-docs.yml`.

## 6. Troubleshooting

### Error: `terraform init` fails with provider version conflicts

**Cause:** Forge modules require specific provider versions (e.g., AWS >= 6.25, Helm >= 3.0.0). Older cached providers may conflict.

**Solution:**
1. Clear the provider cache:
   ```bash
   rm -rf .terraform .terraform.lock.hcl
   ```
2. Re-initialize:
   ```bash
   terraform init -upgrade
   ```

**Prevention:** Use the OpenTofu provider registry as configured in `renovate.json` for consistent resolution.

### Error: Terragrunt `run-all apply` fails with dependency errors

**Cause:** Modules have implicit dependencies that Terragrunt may not resolve automatically if `dependency` blocks are misconfigured.

**Solution:**
1. Visualize the dependency tree:
   ```bash
   terragrunt graph-dependencies
   ```
2. Apply dependent modules individually in correct order
3. Verify all `dependency` blocks in `terragrunt.hcl` files point to valid paths

**Prevention:** Always run `terragrunt run-all plan` before apply to catch dependency issues early.

### Error: GitHub App authentication failures in Lambda functions

**Cause:** SSM parameters containing GitHub App credentials may be stale, or the private key may have been rotated without updating the parameter.

**Solution:**
1. Verify the SSM parameters:
   ```bash
   aws ssm get-parameter --name "/forge/<prefix>/github_app_id" --with-decryption
   ```
2. Re-run the update script:
   ```bash
   ./scripts/update-github-app-secrets.sh
   ```

**Prevention:** Monitor Lambda CloudWatch logs for `jwt_error` or `installation_token_error` patterns.

> [!WARNING]
> Never store GitHub App private keys in plaintext configuration files. Forge stores them as SecureString SSM parameters.

### Error: EKS cluster creation timeout

**Cause:** The `null_resource.wait_for_cluster` polls the cluster endpoint. Network issues or security group misconfiguration can cause timeouts.

**Solution:**
1. Check EKS cluster status in the AWS Console
2. Retry the apply — Terraform resumes from the last successful state
3. Verify VPC networking and security group rules allow the Terraform execution environment to reach the EKS API endpoint

**Prevention:** Ensure the VPC has NAT gateway configuration and the `external_access_cidr_blocks` variable includes the CIDR of your deployment environment.

### Error: Karpenter node provisioning fails

**Cause:** Karpenter requires specific IAM permissions and the EC2NodeClass/NodePool custom resources must be applied after the cluster is ready.

**Solution:**
1. Verify the Karpenter IAM role has required policies
2. Check that `null_resource.karpenter`, `null_resource.apply_ec2_node_class`, and `null_resource.apply_node_pool` completed
3. Review Karpenter controller logs:
   ```bash
   kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter
   ```

### Error: Pre-commit hooks fail on commit

**Cause:** Missing tool dependencies or version mismatches.

**Solution:**
1. Ensure required tools are installed: `shellcheck`, `shfmt`, `tflint`, `hadolint`, `ansible-lint`, `checkmake`
2. Update hooks:
   ```bash
   pre-commit autoupdate
   pre-commit run --all-files
   ```

**Prevention:** The `.pre-commit-config.yaml` pins specific versions. Use Renovate to keep hooks updated.

> [!CAUTION]
> Forge uses `commitizen` with `cz-conventional-gitmoji` for commit message validation. Commits that do not follow the conventional commit format will be rejected at the `commit-msg` stage.

### Error: Splunk HEC ingestion failures in billing Lambdas

**Cause:** The Splunk HEC endpoint URL or token may be misconfigured, or the target index may not exist.

**Solution:**
1. Verify Splunk secrets in AWS Secrets Manager
2. Check Lambda CloudWatch logs for HTTP response codes
3. Test the HEC endpoint:
   ```bash
   curl -k https://<splunk-hec-url>/services/collector \
     -H "Authorization: Splunk <token>" \
     -d '{"event": "test"}'
   ```

**Prevention:** Use the `splunk_secrets` module to manage HEC tokens consistently across environments.

### Issue: Branch protection violation (GitHub #287)

**Cause:** Security policy requires branch protection rules that may not be configured for all branches.

**Solution:** Ensure branch protection is enabled on the `main` branch with required status checks and review requirements as specified by your organization's security policy.

### Issue: Cost safeguards for runner environments (GitHub #296)

**Cause:** Runner environments can incur significant costs without proper guardrails.

**Solution:** Configure `max_instances` for EC2 runner pools and `max_runners` for ARC scale sets. Consider implementing Cloud Custodian policies via the `cloud_custodian` module for automated cost controls.

## 7. FAQ

**Q: Can I use Forge with GitHub.com or only GitHub Enterprise Server?**

Forge supports both. Set `ghes_url` to an empty string `""` for public GitHub.com, or provide your GHES base URL (e.g., `https://github.mycompany.com`).

**Q: Can I run EC2 and ARC runners simultaneously for the same tenant?**

Yes. The `forge_runners` module composes both `ec2_deployment` and `arc_deployment` submodules. Configure `ec2_deployment_specs` and `arc_deployment_specs` in the same deployment to run both backends.

**Q: What runner operating systems are supported for EC2?**

Linux, macOS, and Windows. Each has a dedicated user data template (`user_data_linux.tftpl`, `user_data_osx.tftpl`, `user_data_windows.tftpl`) in `modules/platform/ec2_deployment/template_files/`.

**Q: How does the multi-tenant isolation work?**

Each tenant gets its own GitHub App installation, runner group, IAM roles, and optionally dedicated subnets. The `forge_trust_validator` Lambda periodically validates IAM trust relationships to ensure tenants cannot escalate privileges across boundaries.

**Q: What is the `safety_hook.sh` script?**

A Terragrunt `before_hook` that runs before every `apply`. It validates the target environment and provides a confirmation checkpoint, preventing accidental production changes.

**Q: How are Splunk dashboards managed?**

Forge provisions SignalFx dashboards via the `splunk_o11y_conf_shared` module using the `signalfx` Terraform provider (< 10.0.0). Seven dashboard types are included: billing, DynamoDB, EBS, Lambda, runner EC2, runner K8s, and SQS. Each supports dynamic variables and tenant-based filtering.

**Q: How do I update Terraform module documentation?**

Run `./scripts/update-terraform-docs.sh`. This uses `terraform-docs` with the `.terraform-docs.yml` config to inject auto-generated docs between `<!-- BEGIN_TF_DOCS -->` and `<!-- END_TF_DOCS -->` markers.

> [!TIP]
> Do not manually edit content between the TF_DOCS markers — it will be overwritten on the next documentation generation run.

**Q: What dependency management does Forge use?**

Renovate, configured via `renovate.json`. It includes custom regex managers for Terraform provider versions, GitHub runner Lambda versions, and Dockerfile versions. Key behaviors: security patches auto-merge, patch/pin/digest updates auto-merge, major updates require 7-day stability, and GitHub Actions are pinned to commit SHAs for security.

**Q: What are the open feature requests I should know about?**

As of commit `1a8b9a7`, open feature requests include: lifecycle field refactoring (#236), Grafana/PagerDuty alerting integration (#125, #124), AWS IAM Roles Anywhere for multi-cloud runners (#122), and open-source monitoring stack support (#121).

---

_Generated May 2026 from cisco-open/forge at 1a8b9a7. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._