# Forge — Administrator & Developer Guide

## 1. Architecture Overview

Forge is Cisco's open-source Infrastructure-as-Code (IaC) framework for deploying and operating multi-tenant, self-hosted GitHub Actions runners on AWS. It provides a modular Terraform/Terragrunt architecture that provisions both EC2-based and Kubernetes-based (ARC) runner backends, along with comprehensive observability, security, and lifecycle management infrastructure.

The framework is designed around three architectural layers:

- **Core** — GitHub Actions Runner Controller (ARC) on EKS, including scale set management and Helm-based deployment
- **Infrastructure** — Foundational AWS resources: EKS clusters, ECR registries, S3 storage, AMI lifecycle policies, CloudFormation stacks, and cross-account subscription roles
- **Platform** — Runner orchestration: EC2 and ARC deployment, GitHub App management, webhook relay, job log archival, trust validation, and dead-letter queue redriving
- **Integrations** — Third-party services: Splunk Cloud (data ingestion, dashboards, props/transforms), Splunk Observability (SignalFx, OTel), Teleport (secure access), and GitHub webhook relay (EventBridge cross-account forwarding with Webex notification support)

### Architecture Diagram

```text
┌──────────────────────────────────────────────────────────────────────┐
│                        GitHub (GHES / github.com)                    │
│  ┌─────────┐   ┌──────────┐   ┌───────────────┐                    │
│  │ Webhooks │   │ Runner   │   │ GitHub App    │                    │
│  │          │   │ Groups   │   │ (JWT + Install│                    │
│  └────┬─────┘   └─────┬────┘   │  Token Auth)  │                    │
│       │               │        └───────┬───────┘                    │
└───────┼───────────────┼────────────────┼────────────────────────────┘
        │               │                │
        ▼               │                ▼
┌───────────────────────┼──────────────────────────────────────────────┐
│  AWS Account          │                                              │
│                       │                                              │
│  ┌──────────────┐     │     ┌─────────────────────────────────────┐  │
│  │ API Gateway  │─────┼────▶│ Webhook Relay Lambda               │  │
│  │ (webhook     │     │     │ (validate_signature.py)            │  │
│  │  ingress)    │     │     └──────────────┬──────────────────────┘  │
│  └──────────────┘     │                    │                         │
│                       │                    ▼                         │
│                       │     ┌─────────────────────────────────────┐  │
│                       │     │ EventBridge                         │  │
│                       │     │ ┌─────────┐  ┌──────────────────┐  │  │
│                       │     │ │ Source   │─▶│ Destination Bus  │  │  │
│                       │     │ │ Rules    │  │ (cross-account)  │  │  │
│                       │     │ └─────────┘  └──────┬───────────┘  │  │
│                       │     └─────────────────────┼──────────────┘  │
│                       │                           │                  │
│           ┌───────────┴───────────┐               ▼                  │
│           │                       │    ┌──────────────────┐          │
│  ┌────────▼─────┐  ┌─────────────▼┐   │ Webex Relay      │          │
│  │ EC2 Runners  │  │ ARC Runners  │   │ Lambda           │          │
│  │ (ASG/Fleet)  │  │ (EKS Pods)   │   └──────────────────┘          │
│  │ ┌──────────┐ │  │ ┌──────────┐ │                                  │
│  │ │ Linux    │ │  │ │ DinD     │ │   ┌──────────────────────────┐  │
│  │ │ macOS    │ │  │ │ K8s Mode │ │   │ Support Lambdas          │  │
│  │ │ Windows  │ │  │ └──────────┘ │   │ ┌──────────────────────┐ │  │
│  │ └──────────┘ │  └──────────────┘   │ │ trust_validator      │ │  │
│  └──────────────┘                     │ │ ssm_ami_updater      │ │  │
│                                       │ │ runner_tag_updater   │ │  │
│  ┌────────────────────────────────┐   │ │ global_lock_cleaner  │ │  │
│  │ EKS Cluster                   │   │ │ app_runner_group     │ │  │
│  │ ┌────────┐ ┌────────────────┐ │   │ │ job_log_archiver     │ │  │
│  │ │Karpenter│ │ARC Controller │ │   │ │ job_log_dispatcher   │ │  │
│  │ │        │ │(Helm)         │ │   │ │ redrive_deadletter   │ │  │
│  │ └────────┘ └────────────────┘ │   │ └──────────────────────┘ │  │
│  └────────────────────────────────┘   └──────────────────────────┘  │
│                                                                      │
│  ┌───────────────────────────────────────────────────────────────┐   │
│  │ Observability (Splunk)                                        │   │
│  │ ┌──────────┐ ┌──────────┐ ┌─────────┐ ┌────────────────────┐│   │
│  │ │Data Mgr  │ │O11y/SFx  │ │OTel EKS │ │Billing (CUR→HEC)  ││   │
│  │ │(CloudFmt)│ │(Dashbds) │ │(Helm)   │ │(S3→Lambda→Splunk) ││   │
│  │ └──────────┘ └──────────┘ └─────────┘ └────────────────────┘│   │
│  └───────────────────────────────────────────────────────────────┘   │
│                                                                      │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐               │
│  │ ECR      │ │ S3       │ │ SSM      │ │ Secrets  │               │
│  │ (images) │ │ (logs,   │ │ (params) │ │ Manager  │               │
│  │          │ │  billing)│ │          │ │          │               │
│  └──────────┘ └──────────┘ └──────────┘ └──────────┘               │
└──────────────────────────────────────────────────────────────────────┘
```

### Component Inventory

| Component | Path | Role |
|---|---|---|
| ARC Core | `modules/core/arc/` | Scale set controller, scale sets, Karpenter, storage classes |
| ARC Deployment | `modules/platform/arc_deployment/` | ARC runner deployment wrapper |
| EC2 Deployment | `modules/platform/ec2_deployment/` | EC2 runner instances, AMI management, security groups, user data |
| Forge Runners | `modules/platform/forge_runners/` | Main orchestration: composes EC2 + ARC, GitHub App, webhooks, logs |
| EKS | `modules/infra/eks/` | EKS cluster, addons, Karpenter, Calico, node groups |
| ECR | `modules/infra/ecr/` | Container registry for runner images |
| Storage | `modules/infra/storage/` | S3 buckets for artifacts and logs |
| AMI Policy | `modules/infra/ami_policy/` | DLM lifecycle policies, EBS encryption |
| AMI Sharing | `modules/infra/ami_sharing/` | Cross-account AMI sharing |
| Cloud Custodian | `modules/infra/cloud_custodian/` | Cost governance policies |
| CloudFormation | `modules/infra/cloud_formation/` | CloudFormation stack management |
| Forge Subscription | `modules/infra/forge_subscription/` | Cross-account IAM roles and ECR policies for subscribers |
| Opt-in Regions | `modules/infra/opt_in_regions/` | AWS opt-in region enablement |
| Service Linked Roles | `modules/infra/service_linked_roles/` | AWS service-linked role provisioning |
| Splunk Cloud Conf | `modules/integrations/splunk_cloud_conf_shared/` | Splunk props, transforms, dashboards |
| Splunk Data Manager | `modules/integrations/splunk_cloud_data_manager/` | CloudWatch + custom data ingestion to Splunk |
| Splunk Data Manager Common | `modules/integrations/splunk_cloud_data_manager_common/` | Shared Splunk DM IAM roles and config |
| Splunk AWS Billing | `modules/integrations/splunk_aws_billing/` | CUR billing data pipeline to Splunk |
| Splunk O11y Integration | `modules/integrations/splunk_o11y_aws_integration/` | CloudFormation-based AWS integration |
| Splunk O11y Common | `modules/integrations/splunk_o11y_aws_integration_common/` | SignalFx API integration + IAM |
| Splunk O11y Dashboards | `modules/integrations/splunk_o11y_conf_shared/` | SignalFx dashboard group + 7 dashboard types |
| Splunk OTel EKS | `modules/integrations/splunk_otel_eks/` | Splunk OTel collector Helm deployment |
| Splunk S3 Runner Logs | `modules/integrations/splunk_cloud_s3_runner_logs/` | Runner log shipping via Kinesis Firehose |
| Splunk Secrets | `modules/integrations/splunk_secrets/` | Secrets Manager entries for Splunk credentials |
| Teleport | `modules/integrations/teleport/` | Teleport agent for secure cluster access |
| Teleport Tenant | `modules/integrations/teleport/tenant/` | Per-tenant Teleport RBAC |
| Webhook Relay Source | `modules/integrations/github_webhook_relay_source/` | API Gateway + Lambda webhook ingress |
| Webhook Relay Destination | `modules/integrations/github_webhook_relay_destination/` | EventBridge destination bus + rules |
| Webhook Relay Receivers | `modules/integrations/github_webhook_relay_destination_receivers/` | Lambda receivers (e.g., Webex relay) |

### Technology Stack

| Technology | Version | Purpose |
|---|---|---|
| Terraform / OpenTofu | ~> 1.11 | Infrastructure-as-Code engine |
| Terragrunt | Latest | Configuration hierarchy, dependency management |
| AWS Provider | >= 6.25 | AWS resource provisioning |
| Helm Provider | >= 3.0.0 | Kubernetes Helm chart management |
| Kubernetes Provider | >= 3.0 | Kubernetes resource management |
| Splunk Provider | >= 1.4.30 | Splunk Cloud configuration |
| SignalFx Provider | < 10.0.0 | Splunk Observability dashboards |
| Python | 3.12 | Lambda functions, pre-commit hooks |
| terraform-aws-modules/eks | 21.15.1 | EKS cluster module |
| terraform-aws-modules/lambda | 8.7.0 | Lambda function module |
| terraform-aws-modules/iam | 6.4.0 | IAM IRSA roles |

## 2. System Requirements

### AWS Account Prerequisites

| Resource | Requirement |
|---|---|
| VPC | With private subnets for runners, NAT gateway for internet access |
| IAM | Permissions to create roles, policies, Lambda functions, EKS clusters |
| Secrets Manager | For Splunk credentials, GitHub App secrets |
| SSM Parameter Store | For GitHub App configuration |
| S3 | For Terraform state, runner logs, billing data, CloudFormation templates |
| EKS | For ARC runners (if using Kubernetes backend) |
| ECR | For custom runner container images |
| EventBridge | For webhook relay and scheduled Lambda triggers |
| CloudWatch | For Lambda logging and metrics |
| KMS | For encryption of logs, SQS queues, and S3 buckets |

### GitHub App Requirements

The GitHub App must have these permissions:

| Permission | Level | Purpose |
|---|---|---|
| Actions | Read | Job log archival |
| Metadata | Read | Repository metadata access |
| Self-hosted runners | Read & Write | Runner registration/deregistration |
| Organization self-hosted runners | Read & Write | Runner group management |

> [!IMPORTANT]
> The GitHub App must be installed on the target organization. The installation ID, app ID, client ID, and base64-encoded private key must be available for Terraform to store in SSM.

### Toolchain

| Tool | Version | Required For |
|---|---|---|
| Terraform / OpenTofu | ~> 1.11 | All modules |
| Terragrunt | Latest | Deployment orchestration |
| AWS CLI v2 | Latest | Provider authentication, scripts |
| kubectl | EKS-version compatible | EKS management, Karpenter, ARC |
| Helm | >= 3.0.0 | ARC controller + scale sets, OTel |
| Python | 3.12 | Pre-commit hooks, Lambda development |
| pre-commit | Latest | Code quality enforcement |
| shellcheck, shfmt | Latest | Shell script linting |
| tflint | Latest | Terraform linting |
| hadolint | Latest | Dockerfile linting |
| ansible-lint | v26.3.0 | Ansible linting |
| Gitleaks | v8.30.0 | Secret detection |
| commitizen | v4.13.9 | Commit message validation |

## 3. Configuration Reference

### Global Settings (`_global_settings/`)

**`_global.yaml`** — shared values inherited by all environments:

```yaml
# Example from templates/tenant/_global_settings/_global.yaml
# Contains: aws_profile, aws_region, default_tags, product metadata
```

**`_global.hcl`** — shared HCL configuration for provider setup, backend, and default inputs.

**Per-module `.hcl` files** — each module has a dedicated HCL file (e.g., `tenant.hcl`, `eks.hcl`, `storage.hcl`) that specifies the module source path and version:

```hcl
# Example: _global_settings/eks.hcl
locals {
  source = "${get_repo_root()}/modules/infra/eks"
}
```

### Environment Settings (`_environment_wide_settings/`)

**`_environment.yaml`** — environment-specific overrides (e.g., dev vs. prod instance sizes, scaling limits).

**`_environment.hcl`** — environment-level HCL (e.g., different AWS profiles per environment).

### Region Settings (`_region_wide_settings/`)

**`_region.hcl`** — region-specific configuration (subnet IDs, VPC IDs, availability zones).

### VPC Settings (`_vpc_wide_settings/`)

**`_vpc.yaml`** / **`_vpc.hcl`** — VPC-level configuration for multi-VPC deployments, including subnet assignments and security group references.

### Forge Runners Module — Key Variables

The `forge_runners` module (`modules/platform/forge_runners/`) is the main orchestration module. Key input variables:

#### `deployment_config`

```hcl
variable "deployment_config" {
  type = object({
    deployment_prefix = string
    secret_suffix     = string
    env               = string
    github_app = object({
      id              = string
      client_id       = string
      installation_id = string
      name            = string
    })
    github = object({
      ghes_org             = string
      ghes_url             = string
      repository_selection = string  # "all" or "selected"
      runner_group_name    = string
    })
    tenant = object({
      name                         = string
      iam_roles_to_assume          = optional(list(string), [])
      ecr_registries               = optional(list(string), [])
      github_logs_reader_role_arns = optional(list(string), [])
    })
  })
}
```

> [!IMPORTANT]
> The `deployment_prefix` is used to namespace all SSM parameters, log groups, and resource names. It must be unique per tenant deployment within an AWS account.

#### `ec2_deployment_specs`

```hcl
variable "ec2_deployment_specs" {
  type = object({
    lambda_subnet_ids = list(string)
    subnet_ids        = list(string)
    lambda_vpc_id     = string
    vpc_id            = string
    scale_errors      = optional(list(string), [])
    runner_specs = map(object({
      ami_filter                    = object({ name = list(string), state = list(string) })
      ami_kms_key_arn               = string
      ami_owners                    = list(string)
      runner_labels                 = list(string)
      runner_os                     = string    # "linux", "osx", "windows"
      runner_architecture           = string    # "x86_64", "arm64"
      extra_labels                  = list(string)
      max_instances                 = number
      min_run_time                  = number
      instance_types                = list(string)
      pool_config                   = list(object({
        size                         = number
        schedule_expression          = string
        schedule_expression_timezone = string
      }))
      runner_user                   = string
      enable_userdata               = bool
      instance_target_capacity_type = string  # "spot" or "on-demand"
      block_device_mappings         = list(object({...}))
    }))
  })
}
```

#### `arc_deployment_specs`

```hcl
variable "arc_deployment_specs" {
  type = object({
    cluster_name    = string
    migrate_cluster = optional(bool, false)
    runner_specs = map(object({
      runner_size = object({
        max_runners = number
        min_runners = number
      })
      scale_set_name               = string
      scale_set_type               = string  # "k8s" or "dind"
      container_actions_runner     = string
      container_limits_cpu         = string
      container_limits_memory      = string
      container_requests_cpu       = string
      container_requests_memory    = string
      volume_requests_storage_size = string
      volume_requests_storage_type = string
    }))
  })
}
```

#### `github_webhook_relay`

```hcl
variable "github_webhook_relay" {
  type = object({
    enabled                     = bool
    destination_account_id      = optional(string)
    destination_event_bus_name  = optional(string)
    destination_region          = optional(string)
    destination_reader_role_arn = optional(string)
  })
  default = { enabled = false }
}
```

### EKS Module — Key Variables

| Variable | Type | Default | Description |
|---|---|---|---|
| `cluster_name` | `string` | required | EKS cluster name |
| `cluster_version` | `string` | required | Kubernetes version |
| `cluster_size` | `object` | required | Instance type, min/max/desired node count |
| `cluster_volume` | `object` | required | EBS size, IOPS, throughput, type |
| `cluster_endpoint_public_access` | `bool` | `false` | Whether the API endpoint is public |
| `external_access_cidr_blocks` | `list(string)` | `[]` | CIDRs allowed to access the K8s API |
| `cluster_admin_role_arn` | `string` | `""` | IAM role for cluster admin access |
| `subnet_ids` | `list(string)` | required | Private subnets for worker nodes |
| `vpc_id` | `string` | required | VPC ID |

### Common Variables (shared across modules)

| Variable | Type | Description |
|---|---|---|
| `aws_profile` | `string` | AWS CLI profile name |
| `aws_region` | `string` | Target AWS region |
| `default_tags` | `map(string)` | Default tags applied to all resources |
| `tags` | `map(string)` | Additional resource-specific tags |
| `log_level` | `string` | Lambda log level (INFO, DEBUG, WARN, ERROR) |
| `logging_retention_in_days` | `number` | CloudWatch log retention (default varies: 3 or 30 days) |

### Splunk Configuration Variables

| Module | Key Variable | Description |
|---|---|---|
| `splunk_aws_billing` | `splunk_aws_billing_config.splunk_hec_url` | Splunk HEC endpoint URL |
| `splunk_aws_billing` | `splunk_aws_billing_config.splunk_index` | Target Splunk index |
| `splunk_aws_billing` | `splunk_aws_billing_config.splunk_metrics_url` | Splunk metrics endpoint |
| `splunk_cloud_conf_shared` | `splunk_conf.splunk_cloud` | Splunk Cloud endpoint |
| `splunk_cloud_conf_shared` | `splunk_conf.index` | Splunk index name |
| `splunk_cloud_conf_shared` | `splunk_conf.tenant_names` | List of tenant names |
| `splunk_cloud_data_manager` | `splunk_cloud` | Splunk Cloud endpoint |
| `splunk_o11y_aws_integration_common` | `splunk_api_url` | SignalFx API URL |
| `splunk_o11y_aws_integration_common` | `splunk_organization_id` | SignalFx org ID |
| `splunk_o11y_aws_integration_common` | `integration_regions` | AWS regions to monitor |
| `splunk_otel_eks` | `splunk_otel_collector` | OTel collector config object |
| `splunk_cloud_s3_runner_logs` | `splunk_hec_endpoint` | Splunk HEC URL for runner logs |

> [!NOTE]
> All Splunk secrets (HEC tokens, API tokens, Splunk Cloud credentials) are stored in AWS Secrets Manager and referenced by the modules via `aws_secretsmanager_secret` data sources. Never hardcode credentials in Terraform variables.

## 4. Deployment & Scaling

### Deployment Order

Forge deployments follow a strict dependency order:

1. **`infra-forge`** — Foundation: EKS, ECR, storage, AMI policies, service-linked roles, opt-in regions
2. **`forge-extras`** — Optional: Cloud Custodian, additional ECR repos, Forge subscription (cross-account)
3. **`forge-eks`** — EKS cluster with Karpenter and Calico (if using ARC runners)
4. **`forge-integrations`** — Splunk Cloud configuration (props, transforms, dashboards)
5. **`splunk-deployment`** — Splunk O11y, OTel, Data Manager, billing pipeline
6. **`forge-tenant`** — Per-tenant runner deployment (last, depends on all above)

### Deploying a Full Stack

```bash
# 1. Foundation
cd examples/infra-forge/terragrunt/environments/dev
terragrunt run-all apply

# 2. EKS cluster (if using ARC)
cd examples/deployments/forge-eks/terragrunt/environments/prod
terragrunt run-all apply

# 3. Integrations
cd examples/deployments/forge-integrations/terragrunt/environments/prod
terragrunt run-all apply

# 4. Splunk
cd examples/deployments/splunk-deployment/terragrunt/environments/prod
terragrunt run-all apply

# 5. Tenants
cd examples/deployments/forge-tenant/terragrunt/environments/prod
terragrunt run-all apply
```

### Scaling EC2 Runners

EC2 runner pools scale via two mechanisms:

1. **Reactive scaling** — The upstream `terraform-aws-github-runner` module responds to GitHub Actions webhook events to launch/terminate instances based on demand.

2. **Scheduled scaling** — The `pool_config` variable defines baseline capacity on a cron schedule:

```hcl
pool_config = [
  {
    size                         = 2
    schedule_expression          = "cron(0 8 ? * MON-FRI *)"
    schedule_expression_timezone = "America/New_York"
  },
  {
    size                         = 0
    schedule_expression          = "cron(0 20 ? * MON-FRI *)"
    schedule_expression_timezone = "America/New_York"
  }
]
```

Key scaling parameters per runner spec:

| Parameter | Description |
|---|---|
| `max_instances` | Hard ceiling on concurrent runners |
| `min_run_time` | Minimum job duration (minutes) before scale-down eligibility |
| `instance_target_capacity_type` | `spot` or `on-demand` |
| `instance_types` | List of allowed EC2 instance types |

### Scaling ARC Runners

ARC runner pools scale via the GitHub Actions Runner Controller:

| Parameter | Description |
|---|---|
| `max_runners` | Maximum concurrent runners in the scale set |
| `min_runners` | Minimum warm runners (always running) |
| `container_limits_cpu` / `memory` | Pod resource limits |
| `container_requests_cpu` / `memory` | Pod resource requests (baseline) |

Karpenter manages the underlying EKS node capacity, provisioning nodes as pods demand increases.

> [!TIP]
> Set `min_runners` to at least 1 for latency-sensitive workflows. Cold starts (launching a new runner pod + node) can add 2-5 minutes to job startup.

### ARC Cluster Migration

Set `migrate_arc_cluster = true` when performing a blue/green cluster cutover. This flag is propagated to the scale set controller and scale set modules to coordinate the migration.

## 5. Operations

### Lambda Functions

Forge deploys several operational Lambda functions:

| Lambda | Module | Trigger | Purpose |
|---|---|---|---|
| `forge_trust_validator` | `forge_runners/forge_trust_validator` | EventBridge schedule | Validates IAM trust relationships for runner roles |
| `ec2_update_runner_ssm_ami` | `ec2_deployment/ec2_update_runner_ssm_ami` | EventBridge schedule | Updates SSM with latest AMI matching filters |
| `ec2_update_runner_tags` | `ec2_deployment/ec2_update_runner_tags` | EventBridge (event bus) | Tags EC2 runner instances on state changes |
| `github_app_runner_group` | `forge_runners/github_app_runner_group` | EventBridge schedule | Creates/updates GitHub runner groups, attaches repos |
| `github_clean_global_lock` | `forge_runners/github_global_lock` | EventBridge schedule | Cleans stale GitHub global locks |
| `redrive_deadletter` | `forge_runners/redrive_deadletter` | EventBridge schedule | Redrives messages from DLQ back to main queue |
| `job_log_dispatcher` | `forge_runners/github_actions_job_logs` | EventBridge (webhook) | Dispatches completed job events to SQS |
| `job_log_archiver` | `forge_runners/github_actions_job_logs` | SQS | Downloads and archives GitHub Actions job logs to S3 |
| `validate_signature` | `github_webhook_relay_source` | API Gateway | Validates GitHub webhook HMAC signatures |
| `webex_webhook_relay` | `github_webhook_relay_destination_receivers` | EventBridge | Forwards failure notifications to Webex rooms |
| `splunk_s3_runner_logs` | `splunk_cloud_s3_runner_logs` | SQS (S3 notifications) | Ships runner logs to Splunk via Kinesis Firehose |
| `cur_per_resource` | `splunk_aws_billing` | S3 notification | Processes per-resource CUR billing data |
| `cur_per_resource_process` | `splunk_aws_billing` | S3 notification | Aggregates and sends per-resource costs to Splunk HEC |
| `cur_per_service` | `splunk_aws_billing` | S3 notification | Processes per-service CUR billing data to Splunk |
| `sec_meta_ec2_tags` | `splunk_cloud_data_manager/sec_meta_ec2_tags` | (data manager) | EC2 tag metadata for Splunk security enrichment |

> [!NOTE]
> All Lambda functions use a configurable `log_level` variable (default `INFO`) and `logging_retention_in_days` for CloudWatch log group retention. Common values: 3 days for integration Lambdas, 30 days for platform Lambdas.

### GitHub Actions Job Log Archival

The job log archiver supports two modes:

**Queue Pipeline (default, recommended):**

```text
GitHub Webhook → EventBridge → Dispatcher Lambda → SQS → Archiver Lambda → S3
```

- Better isolation, retry semantics, and backpressure handling
- SQS dead-letter queue for failed downloads
- Configurable batch size (`downloader_batch_size`) and concurrency (`downloader_max_concurrency`)

**Direct Mode (fallback):**

```text
GitHub Webhook → EventBridge → Archiver Lambda → S3
```

- Simpler, fewer moving parts
- Set `enable_queue_pipeline = false` to use

Logs are stored at: `s3://<bucket>/<workflow_name>/<run_attempt>/<job_id>.zip`

### Monitoring

Forge provides built-in monitoring through Splunk Observability (SignalFx) dashboards:

| Dashboard | Module Path | Metrics |
|---|---|---|
| Runner EC2 | `dashboards/runner_ec2/` | CPU, memory, disk, network utilization; active hosts by AZ and instance type |
| Runner K8s | `dashboards/runner_k8s/` | Active pods, memory usage, network bytes, pod phases |
| Billing | `dashboards/billing/` | Cost per service, cost per tenant, net costs |
| Lambda | `dashboards/lambda/` | Invocations, errors, duration, throttles, concurrency |
| SQS | `dashboards/sqs/` | Queue depth, message age, processing trends |
| EBS | `dashboards/ebs/` | IOPS, throughput, latency, queue length, utilization |
| DynamoDB | `dashboards/dynamodb/` | Request latency, throttles, capacity utilization |

Each dashboard supports dynamic variables for filtering by tenant via the `dashboard_variables` input.

> [!TIP]
> The Splunk Cloud Conf module (`splunk_cloud_conf_shared`) provisions three Splunk dashboards (CI Jobs, Tenant, Trust Relationship Validation) as saved searches using `splunk_data_ui_views` resources.

### Secrets Management

| Secret Type | Storage | Modules |
|---|---|---|
| GitHub App private key | SSM Parameter Store (SecureString) | `forge_runners` |
| GitHub App webhook secret | SSM Parameter Store | `forge_runners` (auto-rotated every 30 days) |
| Splunk Cloud credentials | Secrets Manager | All Splunk integration modules |
| Splunk HEC tokens | Secrets Manager | `splunk_aws_billing`, `splunk_cloud_s3_runner_logs` |
| SignalFx API token | Secrets Manager | `splunk_o11y_aws_integration_common`, `splunk_o11y_conf_shared` |
| Webex bot token | Secrets Manager | `webex_webhook_relay` |

> [!WARNING]
> The Webex webhook relay Lambda expects a Secrets Manager secret at `/cicd/common/webex_webhook_relay_bot_token` containing JSON with `token` and `room_id` keys. Both are required. The Lambda auto-prepends `Bearer ` to the token if not present.

### Operational Scripts

| Script | Purpose |
|---|---|
| `scripts/migrate-tenant.sh` | Migrate a single tenant to new configuration |
| `scripts/migrate-all-tenants.sh` | Batch tenant migration |
| `scripts/reinstall-eks-with-deps.sh` | Reinstall EKS cluster with all dependencies |
| `scripts/update-github-app-secrets.sh` | Update GitHub App credentials in SSM |
| `scripts/update-terraform-docs.sh` | Regenerate terraform-docs for all modules |
| `scripts/generate_and_patch_github_app.sh` | Generate and patch GitHub App webhook configuration |
| `scripts/download_lambdas.sh` | Download Lambda deployment packages |

## 6. Performance

### EKS Cluster Sizing

The EKS module exposes `cluster_size` for the self-managed node group:

```hcl
cluster_size = {
  instance_type = "m5.large"
  min_size      = 2
  max_size      = 10
  desired_size  = 3
}
```

And `cluster_volume` for EBS configuration:

```hcl
cluster_volume = {
  size       = 100
  iops       = 3000
  throughput = 125
  type       = "gp3"
}
```

Karpenter handles additional node provisioning beyond the base node group, using NodePool and EC2NodeClass custom resources applied via `null_resource` provisioners.

### ARC Scale Set Resource Tuning

Each ARC runner pod's resources are individually configurable:

| Setting | Example | Impact |
|---|---|---|
| `container_requests_cpu` | `"500m"` | Baseline CPU reservation |
| `container_requests_memory` | `"1Gi"` | Baseline memory reservation |
| `container_limits_cpu` | `"2000m"` | CPU burst ceiling |
| `container_limits_memory` | `"4Gi"` | Memory ceiling (OOM-kill above) |
| `volume_requests_storage_size` | `"20Gi"` | Persistent volume for workspace |
| `volume_requests_storage_type` | `"gp3"` | Storage class |

### EC2 Runner Instance Selection

Configure instance types per runner pool for cost/performance optimization:

```hcl
instance_types = ["m5.xlarge", "m5a.xlarge", "m5d.xlarge"]
instance_target_capacity_type = "spot"  # or "on-demand"
```

> [!TIP]
> Use multiple instance types in the `instance_types` list when using spot capacity to improve availability. The runner module selects from the list based on current spot pricing and capacity.

## 7. Security

### IAM Architecture

Forge implements a layered IAM model:

- **Runner IAM roles** — Each tenant's runners assume a dedicated IAM role (`role_for_forge_runners`) with scoped permissions for S3, Secrets Manager, ECR, and Packer operations
- **Forge trust validator** — A scheduled Lambda that validates IAM trust policies on runner roles, ensuring no unauthorized principals can assume them
- **ECR access policies** — Per-tenant ECR repository policies granting pull access only to authorized runner roles
- **Cross-account roles** — The `forge_subscription` module creates IAM roles and ECR policies for subscriber accounts

### Trust Validation

The `forge_trust_validator` Lambda (`modules/platform/forge_runners/forge_trust_validator/lambda/forge_trust_validator.py`) runs on a schedule and:

1. Reads the trust policy of each configured Forge IAM role
2. Validates that only expected principals appear in the trust relationship
3. Reports violations (the Splunk Cloud Conf module provides a Trust Relationship Validation dashboard)

### Webhook Signature Validation

The GitHub webhook relay source Lambda (`modules/integrations/github_webhook_relay_source/lambda/validate_signature.py`) validates incoming webhook payloads using HMAC-SHA256 with the GitHub App webhook secret before forwarding events to EventBridge.

### Encryption

| Resource | Encryption |
|---|---|
| EBS volumes | Default encryption enabled via `aws_ebs_encryption_by_default` (in `ami_policy` module) |
| S3 buckets | Server-side encryption (SSE-S3 or SSE-KMS depending on module) |
| SQS queues | KMS encryption for job log queues |
| Kinesis streams | KMS encryption for runner log streaming |
| SSM parameters | SecureString type for sensitive values |

> [!CAUTION]
> The `ami_policy` module enables EBS encryption by default at the account level via `aws_ebs_encryption_by_default`. This affects ALL EBS volumes in the region, not just Forge resources. Coordinate with other teams before applying.

### Network Security

- EC2 runners are deployed in private subnets with security groups managed by the `ec2_deployment` module
- EKS clusters default to `cluster_endpoint_public_access = false` (private API endpoint)
- `external_access_cidr_blocks` must be explicitly configured for any external access to the K8s API
- Calico network policies are applied via the EKS module for pod-level network segmentation

### Pre-Commit Security Hooks

| Hook | Purpose |
|---|---|
| `detect-private-key` | Prevents committing private keys |
| `gitleaks` (v8.30.0) | Scans for secrets and credentials |
| `check-added-large-files` | Prevents accidentally committing large binaries |

## 8. Developer Internals

### Repository Structure

```text
cisco-open/forge/
├── modules/
│   ├── core/
│   │   └── arc/                    # ARC scale set controller + scale sets
│   │       ├── scale_set/          # Individual scale set (Helm + K8s + IAM)
│   │       ├── scale_set_controller/  # ARC controller (Helm + namespace + secret)
│   │       └── templates/          # Karpenter NodePool template
│   ├── infra/
│   │   ├── ami_policy/             # DLM lifecycle, EBS encryption
│   │   ├── ami_sharing/            # Cross-account AMI sharing
│   │   ├── cloud_custodian/        # Cost governance
│   │   ├── cloud_formation/        # CF stack management
│   │   ├── ecr/                    # Container registry
│   │   ├── eks/                    # EKS cluster + addons + Karpenter
│   │   ├── forge_subscription/     # Cross-account subscription
│   │   ├── opt_in_regions/         # AWS region opt-in
│   │   ├── service_linked_roles/   # SLR provisioning
│   │   └── storage/                # S3 buckets
│   ├── integrations/
│   │   ├── github_webhook_relay_source/        # Webhook ingress (API GW + Lambda)
│   │   ├── github_webhook_relay_destination/   # EventBridge destination bus
│   │   ├── github_webhook_relay_destination_receivers/  # Lambda receivers
│   │   ├── splunk_aws_billing/                 # CUR billing pipeline
│   │   ├── splunk_cloud_conf_shared/           # Splunk props/transforms/dashboards
│   │   ├── splunk_cloud_data_manager/          # CloudWatch data ingestion
│   │   ├── splunk_cloud_data_manager_common/   # Shared DM config
│   │   ├── splunk_cloud_s3_runner_logs/        # Runner log shipping
│   │   ├── splunk_o11y_aws_integration/        # AWS integration (CloudFormation)
│   │   ├── splunk_o11y_aws_integration_common/ # SignalFx API integration
│   │   ├── splunk_o11y_conf_shared/            # SignalFx dashboards (7 types)
│   │   ├── splunk_otel_eks/                    # OTel collector Helm
│   │   ├── splunk_secrets/                     # Secrets Manager entries
│   │   └── teleport/                           # Teleport agent + tenant RBAC
│   └── platform/
│       ├── arc_deployment/         # ARC deployment wrapper
│       ├── ec2_deployment/         # EC2 runners (AMI, SG, user data, Lambdas)
│       └── forge_runners/          # Main orchestration module
│           ├── forge_trust_validator/     # IAM trust validation Lambda
│           ├── github_actions_job_logs/   # Job log archiver (dispatcher + archiver)
│           ├── github_app_runner_group/   # Runner group management Lambda
│           ├── github_global_lock/        # Global lock cleanup Lambda
│           ├── github_webhook_relay/      # Webhook relay orchestration
│           ├── redrive_deadletter/        # DLQ redrive Lambda
│           └── scripts/                   # GitHub App patching script
├── examples/
│   ├── deployments/                # Full deployment examples per stack
│   │   ├── forge-eks/
│   │   ├── forge-extras/
│   │   ├── forge-integrations/
│   │   ├── forge-tenant/
│   │   └── splunk-deployment/
│   ├── infra-forge/                # Foundation infrastructure deployment
│   └── templates/                  # Configuration templates per stack type
├── docs/                           # Internal documentation
│   ├── configurations/             # Deployment configuration guides
│   └── tenant-usage/               # Tenant onboarding and Renovate
├── scripts/                        # Operational scripts
├── .pre-commit-config.yaml         # 30+ pre-commit hooks
├── .terraform-docs.yml             # terraform-docs configuration
├── renovate.json                   # Renovate dependency management
├── .ansible-lint                   # Ansible linting config
└── .yamllint                       # YAML linting config
```

### Module Composition Pattern

The `forge_runners` module demonstrates the primary composition pattern:

```hcl
# modules/platform/forge_runners/
module "ec2_runners"              { source = "../ec2_deployment" }
module "arc_runners"              { source = "../arc_deployment" }
module "forge_trust_validator"    { source = "./forge_trust_validator" }
module "github_actions_job_logs"  { source = "./github_actions_job_logs" }
module "github_app_runner_group"  { source = "./github_app_runner_group" }
module "github_global_lock"       { source = "./github_global_lock" }
module "github_webhook_relay"     { source = "./github_webhook_relay" }
module "redrive_deadletter"       { source = "./redrive_deadletter" }
```

This single module composes all runner-related infrastructure, creating a cohesive deployment unit per tenant.

### Code Patterns and Conventions

**State management:** Each module uses a `backend.tf` with S3 backend configuration. State paths follow the Terragrunt hierarchy.

**Provider configuration:** Providers are declared in `providers.tf` per module, with versions pinned in `versions.tf`.

**Tagging:** Every module accepts `default_tags` and `tags` variables. The AWS provider `default_tags` feature applies organization-wide tags, while `tags` adds module-specific tags.

**Naming:** Resources use a `prefix` or `deployment_prefix` variable for namespace isolation. SSM parameters follow `/forge/<prefix>/<parameter_name>` convention.

**Lambda packaging:** Lambda functions are Python 3.12, packaged in-module under `lambda/` directories. The `terraform-aws-modules/lambda/aws` module (v8.7.0) handles packaging and deployment.

**Template files:** Terraform template files use `.tftpl` extension and live in `template_files/` directories within their respective modules.

**External data sources:** Several modules use `external_external` data sources to call shell scripts (e.g., `fetch_secret_value.sh`, `list_buckets.sh`) for operations that lack native Terraform providers.

### Webhook Relay Architecture

The webhook relay spans three modules:

1. **Source** (`github_webhook_relay_source/`) — API Gateway endpoint receives GitHub webhooks → Lambda validates HMAC signature → forwards valid events to EventBridge source bus

2. **Destination** (`github_webhook_relay_destination/`) — EventBridge destination bus in another account receives cross-account events → rules route to Lambda targets based on event patterns

3. **Receivers** (`github_webhook_relay_destination_receivers/`) — Lambda functions that process forwarded events. The included `webex_webhook_relay` receiver:
   - Filters for `main` branch failures (`cancelled`, `failure`, `timed_out`, `timeout`, `startup_failure`)
   - Builds a Slack-format card, converts to Webex Adaptive Card format
   - Sends to Webex via the Messages API using a bot token from Secrets Manager

### Splunk Billing Pipeline

The billing integration (`splunk_aws_billing/`) implements a three-Lambda pipeline:

1. **`cur_per_resource`** — Triggered by S3 notifications when new CUR (Cost and Usage Report) parquet files arrive. Splits data by date and re-uploads to S3 for processing.

2. **`cur_per_resource_process`** — Groups billing data by resource, service, and application. Sends events to Splunk HEC in batches (respecting `MAX_BATCH_COUNT` and `MAX_BATCH_SIZE_BYTES`). Also sends metrics to SignalFx (`forge.per_resource.cost_usd`, `forge.per_resource.net_cost_usd`).

3. **`cur_per_service`** — Similar to per-resource but aggregated at the service level. Sends `forge.per_service.cost_usd` and `forge.per_service.net_cost_usd` metrics.

All three Lambdas use a shared `common.py` module for Splunk HEC/SignalFx communication and data preprocessing.

### Testing Strategy

Forge relies on:

- **Pre-commit validation** — `terraform validate`, `terraform fmt`, `tflint`, `tofu-validate` run on every commit
- **Terraform plan** — `terragrunt run-all plan` serves as the primary integration test
- **Safety hooks** — `safety_hook.sh` provides environment-aware guardrails before apply

### Code Quality Toolchain

The `.pre-commit-config.yaml` configures 30+ hooks across categories:

| Category | Hooks |
|---|---|
| General | YAML/JSON/TOML/XML syntax, merge conflict markers, large file guard, line endings |
| Python | autopep8, isort, autoflake, flake8 (E501 ignored), pyupgrade, validate-pyproject |
| Shell | shellcheck, shfmt (4-space indent), bashate, script-must-have-extension |
| Terraform/IaC | terragrunt-hcl-fmt, tofu-fmt, tofu-validate, terraform_fmt, terraform_tflint, terraform_validate, packer-validate |
| Security | gitleaks, detect-private-key |
| Docker | hadolint |
| Ansible | ansible-lint (v26.3.0, Python 3.12) |
| Markdown | mdformat (with GFM, ruff, frontmatter plugins, excludes `modules/`) |
| JSON Schema | GitHub workflows, GitHub Actions, Docker Compose, Dependabot, Renovate |
| Commit | commitizen with cz-conventional-gitmoji |

### Dependency Management

Renovate (`renovate.json`) manages all dependency updates:

| Rule | Behavior |
|---|---|
| Security patches | Immediate processing, auto-merge, `security` + `critical` labels |
| Patch/pin/digest | Auto-merge with `simple-review` label |
| Minor updates | `simple-review` label, no auto-merge |
| Major updates | 7-day stability period, `major-update` + `breaking-change` labels |
| AWS providers/modules | Grouped as "AWS providers and modules" |
| GitHub Actions | Pinned to commit SHA, grouped |
| Pre-commit hooks | Grouped as "Pre-commit hooks" |
| Docker images | Digest pinning |
| Terraform providers | Resolved from OpenTofu registry |

Custom regex managers extract versions from:
- `.tf` files (`required_version`)
- `download_lambdas.sh` references (GitHub runner version)
- Dockerfiles (with renovate comments)

### Contributing

Forge follows conventional commits with gitmoji (enforced by `commitizen` pre-commit hook). Key practices:

- All Terraform code must pass `terraform validate`, `terraform fmt`, and `tflint`
- Shell scripts must pass `shellcheck` and `shfmt`
- Python code must pass `flake8` (E501 ignored), `autopep8`, `isort`, and `autoflake`
- Module documentation is auto-generated — update source code, then run `scripts/update-terraform-docs.sh`
- YAML files are formatted with `yamlfmt` (2-space mapping, 4-space sequence, 2-space offset, 180 char width)

### Potential Improvements

Based on open GitHub issues:

- **Lifecycle field refactoring** (#236) — Replace `count` conditionals with the `lifecycle.enabled` field for cleaner resource toggling
- **Grafana/PagerDuty integration** (#125, #124) — Add alerting pipeline from Grafana and Splunk Cloud to PagerDuty
- **AWS IAM Roles Anywhere** (#122) — Enable multi-cloud ephemeral runners via certificate-based authentication
- **Open-source monitoring** (#121) — Add Loki, Prometheus, and Grafana as alternatives to Splunk
- **Cost safeguards** (#296) — Default cost guardrails and hardening for runner environments

## 9. Integration Points

### AWS Service Integration

| AWS Service | Forge Usage |
|---|---|
| EC2 | Runner instances (Linux, macOS, Windows) |
| EKS | Kubernetes clusters for ARC runners |
| ECR | Container image registry for runner images |
| S3 | Terraform state, runner logs, billing data, CloudFormation templates, Firehose backup |
| SSM Parameter Store | GitHub App configuration, AMI IDs |
| Secrets Manager | Splunk credentials, Webex bot tokens |
| EventBridge | Webhook relay, Lambda scheduling, EC2 state change events |
| API Gateway | GitHub webhook ingress endpoint |
| Lambda | 14+ operational functions |
| CloudWatch | Lambda logs, metrics, event rules |
| KMS | Encryption for S3, SQS, Kinesis, EBS |
| IAM | Runner roles, service roles, cross-account access |
| CloudFormation | Splunk O11y AWS integration |
| DLM | EBS snapshot lifecycle policies |
| Kinesis Data Streams | Runner log streaming |
| Kinesis Firehose | Splunk HEC delivery |
| SQS | Job log dispatch queue, dead-letter queues |
| Service Catalog | AppRegistry for Forge application tracking |
| BCM Data Exports | Cost and Usage Report (CUR) exports |
| Calico | EKS network policies |

### Third-Party Integrations

| Service | Integration Module | Purpose |
|---|---|---|
| Splunk Cloud | `splunk_cloud_conf_shared`, `splunk_cloud_data_manager` | Log ingestion, props/transforms, dashboards |
| Splunk Observability (SignalFx) | `splunk_o11y_aws_integration_common`, `splunk_o11y_conf_shared` | Metrics, dashboards |
| Splunk OTel | `splunk_otel_eks` | Kubernetes observability via Helm |
| Teleport | `teleport/`, `teleport/tenant/` | Secure cluster access with per-tenant RBAC |
| Webex | `webex_webhook_relay` | CI failure notifications |
| GitHub (API) | `forge_runners`, `github_app_runner_group` | Runner management, App webhook configuration |

### Terraform Module Dependencies

| External Module | Version | Used By |
|---|---|---|
| `terraform-aws-modules/eks/aws` | 21.15.1 | `modules/infra/eks/` |
| `terraform-aws-modules/eks/aws//modules/karpenter` | 21.15.1 | `modules/infra/eks/` |
| `terraform-aws-modules/eks/aws//modules/self-managed-node-group` | 21.15.1 | `modules/infra/eks/` |
| `terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts` | 6.4.0 | `modules/infra/eks/` |
| `terraform-aws-modules/lambda/aws` | 8.7.0 | Multiple Lambda-based modules |

---

_Generated May 2026 from cisco-open/forge at 1a8b9a7. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._