# Project N.O.M.A.D. — Administrator & Developer Guide

This guide is for the people who run, extend, deploy, and audit Project
N.O.M.A.D. — system administrators, DevOps, security reviewers, and developers
contributing patches. End-user task-flow documentation lives in the User
Guide; this document focuses on architecture, full configuration reference,
operations, security, and internals.

## 1. Executive Summary

N.O.M.A.D. is an **AdonisJS 6** TypeScript application served behind a Docker
Compose stack of seven containers. The admin server speaks Inertia.js to a
React 19 SPA, persists structured state in MySQL 8 via Lucid ORM, queues
background work in Redis 7 via BullMQ, and orchestrates an arbitrary number
of *managed services* (Kiwix, Ollama, Qdrant, Kolibri, etc.) by talking to
the host Docker socket through the `dockerode` library. Real-time UI updates
are pushed to the browser via `@adonisjs/transmit` (SSE over Redis pub/sub).

Two sidecar containers wrap operational concerns the admin can't safely do
itself:

- **`disk-collector`** — privileged read-only mount of the host filesystem,
  publishes JSON disk telemetry every 120 seconds.
- **`updater`** — receives signed update requests via a shared volume, edits
  `compose.yml`, pulls images, and recreates the admin container (so the
  admin can update itself without a chicken-and-egg problem).

The product target is a single-tenant LAN appliance, so the design
deliberately skips authentication, multi-user accounts, and TLS termination
inside the container. SSRF protection is the main exposed-surface defense.

| Aspect | Choice | Rationale |
|--------|--------|-----------|
| Backend framework | AdonisJS 6 | Opinionated DI container, TypeScript-native, mature MVC |
| Frontend | React 19 + Inertia.js | Server-side routing with SPA UX; no separate JSON API to maintain |
| Database | MySQL 8 | Familiar, ships with healthcheck, lockstep with `mysql2` driver |
| Queue | Redis 7 + BullMQ | Persistent, scalable, supports queue priorities |
| Realtime | `@adonisjs/transmit` over Redis | SSE; survives reverse proxies; no WebSocket upkeep |
| Container control | `dockerode` against host socket | Lets admin install/start/stop/remove sibling services |
| Vector DB (RAG) | Qdrant | Filterable, self-hosted, simple HTTP API |
| Embeddings | `nomic-embed-text` via Ollama | Self-contained; no separate embedding service |
| Maps engine | maplibre-gl + Protomaps PMTiles | Vector tiles, fully offline once cached |
| Doc renderer | Markdoc | Sandboxed, structured authoring of in-app docs |

## 2. Architecture Overview

### 2.1 Process & container topology

```text
                ┌───────────────────────────────────────────────┐
                │                Host (Debian/Ubuntu)           │
                │                                               │
                │   /var/run/docker.sock                        │
                │            ▲                                  │
                │            │ (bind-mounted)                   │
                │   ┌────────┴────────┐                         │
                │   │ nomad_admin     │ ─ AdonisJS:8080 ─┐      │
                │   │ (this app)      │                  │      │
                │   └─┬───────┬─────┬─┘                  │      │
                │     │       │     │  ┌─────────────────▼────┐ │
                │     │       │     └─►│ nomad_dozzle         │ │
                │     │       │        │ (read-only logs)     │ │
                │     │       │        └──────────────────────┘ │
                │   ┌─▼───┐ ┌─▼───┐                             │
                │   │mysql│ │redis│                             │
                │   └─────┘ └─────┘                             │
                │                                               │
                │   ┌──────────────┐  ┌────────────────────┐    │
                │   │ disk-collec- │  │ updater (sidecar)  │    │
                │   │ tor sidecar  │  │ shared volume IPC  │    │
                │   └──────────────┘  └────────────────────┘    │
                │                                               │
                │   Managed services (created on demand)        │
                │   ┌──────────┐ ┌──────────┐ ┌──────────┐      │
                │   │  kiwix   │ │  ollama  │ │  qdrant  │ …    │
                │   └──────────┘ └──────────┘ └──────────┘      │
                └───────────────────────────────────────────────┘
                          ▲
                          │ HTTP :8080  (Inertia + REST)
                ┌─────────┴─────────┐
                │ Browsers on LAN   │
                └───────────────────┘
```

Every container shares the `project-nomad_default` Docker network, named by
Docker Compose. Managed services are joined to the same network on creation
so the admin can reach them by container name (`nomad_kiwix_server`,
`nomad_ollama`, …).

> [!NOTE]
> The admin container is the *only* component with access to the Docker
> socket. Sidecars get scoped capabilities (read-only host mount for
> `disk-collector`, write access to `compose.yml` for `updater`). Managed
> services never touch the socket.

### 2.2 Component inventory

| Component | Path | Role |
|-----------|------|------|
| HTTP server | `admin/bin/server.ts` (compiled) | AdonisJS HTTP entrypoint, `node bin/server.js` |
| Routes | `admin/start/routes.ts` | All 104 Inertia + REST routes |
| Kernel / middleware | `admin/start/kernel.ts` | Server + router middleware stacks |
| Env validator | `admin/start/env.ts` | Vine schema for required env vars; fails fast on bad config |
| Controllers | `admin/app/controllers/*.ts` | 13 thin HTTP layer classes |
| Services | `admin/app/services/*.ts` | 17 service classes (business logic, ~296 KB of code) |
| Models | `admin/app/models/*.ts` | 10 Lucid ORM models |
| Validators | `admin/app/validators/*.ts` | 10 Vine validation schemas |
| Background jobs | `admin/app/jobs/*.ts` | BullMQ job processors |
| Constants | `admin/constants/*.ts` | Service names, Ollama prompts, Kiwix command, broadcast channels |
| Migrations | `admin/database/migrations/*.ts` | 23 migrations (numerically sorted) |
| Frontend | `admin/inertia/**/*.tsx` | React 19 SPA (Inertia adapter) |
| Install scripts | `install/*.sh` | Bash one-shots for install/update/uninstall/migrate |
| Sidecar (disk) | `install/sidecar-disk-collector/` | Bash + `lsblk` polling loop |
| Sidecar (updater) | `install/sidecar-updater/` | Bash polls `/shared/update-request` |

### 2.3 Technology stack

| Layer | Library / runtime | Version pin |
|-------|-------------------|-------------|
| Runtime | Node.js | (image-pinned in GHCR build) |
| Framework | `@adonisjs/core` | ^6.18.0 |
| HTTP | AdonisJS HTTP server | bundled |
| ORM | `@adonisjs/lucid` + `mysql2` | ^21.8.2 / ^3.14.1 |
| Realtime | `@adonisjs/transmit` (SSE) + `react-adonis-transmit` | ^2.0.2 / ^1.0.1 |
| Queues | `bullmq` | ^5.65.1 |
| Validation | `@vinejs/vine` | ^3.0.1 |
| Inertia | `@adonisjs/inertia` + `@inertiajs/react` | ^3.1.1 / ^2.0.13 |
| UI | React 19 + Tailwind 4 + Headless UI + Tabler Icons | — |
| Docs | `@markdoc/markdoc` | ^0.5.2 |
| LLM client | `ollama` SDK + `openai` SDK | ^0.6.3 / ^6.27.0 |
| Embeddings | `@chonkiejs/core` (chunking) + Ollama `nomic-embed-text` | — |
| Vector DB | `@qdrant/js-client-rest` | ^1.16.2 |
| Maps | `maplibre-gl` + `@protomaps/basemaps` + `pmtiles` | ^4.7.1 / ^5.7.0 / ^4.4.0 |
| ZIM | `@openzim/libzim` | ^4.0.0 |
| File parsing | `pdf-parse`, `tesseract.js`, `pdf2pic`, `cheerio`, `fast-xml-parser`, `jszip`, `tar`, `yaml` | — |
| System info | `systeminformation`, `dockerode` | ^5.31.0 / ^4.0.7 |
| Image work | `sharp` | ^0.34.5 |
| Misc | `axios`, `luxon`, `compression`, `fuse.js`, `stopword`, `url-join` | — |

> [!NOTE]
> The `package.json` in the repo root carries only metadata (`name`,
> `version`, `author`). All runtime deps live in `admin/package.json`. The
> root `version` field is what semantic-release bumps; the admin one is
> pinned at `0.0.0`.

### 2.4 Request lifecycle

```text
Browser  ──HTTP──►  nomad_admin :8080
                     │
                     ▼
        Server middleware stack
        ├── ContainerBindingsMiddleware (binds HttpContext)
        ├── @adonisjs/cors (origin '*', credentials true)
        ├── @adonisjs/vite (asset reload in dev)
        ├── @adonisjs/inertia (Inertia handshake)
        ├── @adonisjs/static (public/)
        └── MapsStaticMiddleware (storage/maps/*)
                     │
                     ▼
        Router middleware stack (matched routes only)
        ├── BodyParser (multipart limit 110 MB)
        ├── @adonisjs/shield (XFrame=DENY, CSP/CSRF disabled)
        └── CompressionMiddleware (compression library, gzip)
                     │
                     ▼
        Controller method  ──►  Service layer  ──►  Lucid model / Docker / Ollama
                     │
                     ▼
        Response: Inertia HTML, JSON, SSE stream, or static file
```

Streaming chat responses bypass the response object's normal flow: the
controller flushes SSE headers (`Content-Type: text/event-stream`,
`Cache-Control: no-cache`, `Connection: keep-alive`) before any RAG
preprocessing so the client connection stays open during the (potentially
slow) query rewrite + Qdrant search phase.

### 2.5 Data flow: AI chat with RAG

```text
1. POST /api/ollama/chat  (validated by chatSchema)
2. OllamaController.chat
   ├── Inject SYSTEM_PROMPTS.default if no system message
   ├── rewriteQueryWithContext()
   │     └─► OllamaService.chat with SYSTEM_PROMPTS.query_rewrite
   │         (skips when <3 user turns or qwen2.5:3b not installed)
   ├── ragService.searchSimilarDocuments(rewritten, 5, 0.3)
   │     └─► Qdrant via @qdrant/js-client-rest
   ├── trim by RAG_CONTEXT_LIMITS based on model size
   ├── splice SYSTEM_PROMPTS.rag_context(joined) before first user msg
   ├── checkModelHasThinking() decides "think" param
   ├── numCtx auto-bump if system prompt > 3k tokens
   ├── chatStream() → for await chunk → response.write SSE frames
   └── on end → chatService.addMessage(assistant, fullContent)
```

## 3. System Requirements

| Resource | Minimum | Recommended | Notes |
|----------|---------|-------------|-------|
| OS | Debian 12 / Ubuntu 22.04 | Debian 12 / Ubuntu 24.04 | `install_nomad.sh` checks `/etc/debian_version` |
| Kernel | 5.15+ | 6.x | Required for cgroup v2 + recent overlayfs |
| Docker Engine | 24.x | 27.x | Compose v2 plugin **required** (v1 is rejected) |
| CPU | 4 cores | 8+ cores (AVX2) | sysbench scoring assumes modern x86_64 |
| RAM | 8 GB | 32 GB | 16 GB bare-minimum once a 7B model and Wikipedia load |
| Storage | 100 GB | 1 TB+ NVMe | Wikipedia full + maps + AI models = 200–500 GB |
| GPU | None | NVIDIA, CUDA 12+ | AMD ROCm not supported; toolkit auto-installed when detected |
| Open ports | 8080 (admin), 9999 (Dozzle) | + 11434 Ollama, + others on demand | `installer` does not configure firewall |
| Outbound network | One-time only | One-time only | After content download, fully air-gappable |

> [!NOTE]
> The installer **requires** `bash` (not `sh`) and **requires** `sudo` —
> `check_is_bash` and `check_has_sudo` are mandatory pre-flight gates. The
> Debian-family check is also enforced (`/etc/debian_version` must exist).

## 4. Configuration Reference

### 4.1 Environment variables (validated in `admin/start/env.ts`)

| Variable | Type | Default | Required | Notes |
|----------|------|---------|:-------:|-------|
| `NODE_ENV` | enum: `development`/`production`/`test` | — | yes | Toggles dev tooling, logger pretty-print, cookie secure flag |
| `PORT` | number | — | yes | In-container HTTP port (do not change; map host port instead) |
| `APP_KEY` | string | — | yes | ≥ 16 chars; encrypts cookies, signs URLs (`@adonisjs/core/helpers` Secret) |
| `HOST` | host string | — | yes | Bind interface; production value is `0.0.0.0` |
| `URL` | string | — | yes | Public URL the UI is reached at (e.g. `http://192.168.1.42:8080`) |
| `LOG_LEVEL` | string | — | yes | `debug` / `info` / `warn` / `error` (`pino` levels) |
| `INTERNET_STATUS_TEST_URL` | string | optional | no | Override the URL used by the internet-status indicator |
| `DISABLE_COMPRESSION` | boolean | optional | no | Set true behind a compressing reverse proxy |
| `NOMAD_STORAGE_PATH` | string | optional | no | Reserved for storage relocation; default is `/app/storage` |
| `NOMAD_API_URL` | string | optional | no | Override `https://api.projectnomad.us` for catalog fetching |
| `DB_HOST` | host | — | yes | `mysql` in default compose |
| `DB_PORT` | number | — | yes | `3306` |
| `DB_USER` | string | — | yes | `nomad_user` in default compose |
| `DB_PASSWORD` | string | optional | no | Required when DB enforces auth (default config does) |
| `DB_DATABASE` | string | — | yes | `nomad` |
| `DB_SSL` | boolean | optional | no | Enable when MySQL TLS is configured |
| `REDIS_HOST` | host | — | yes | `redis` |
| `REDIS_PORT` | number | — | yes | `6379` |
| `APP_NAME` | string | optional | no | Used for the pino logger `name` |

> [!IMPORTANT]
> `APP_KEY` must be **≥ 16 characters** or AdonisJS refuses to boot —
> validation happens during `Env.create` at startup. The installer generates
> 32 random alphanumeric chars via `tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 32`.

### 4.2 Container environment (from `install/management_compose.yaml`)

| Variable | Default | Notes |
|----------|---------|-------|
| `NODE_ENV` | `production` | Compose overrides Adonis runtime env |
| `PORT` | `8080` | In-container only; remap on host side |
| `HOST` | `0.0.0.0` | Listen on all interfaces in the container |
| `LOG_LEVEL` | `info` | Switch to `debug` while triaging |
| `URL` | (templated) | Replaced by installer with `http://<lan-ip>:8080` |
| `APP_KEY` | (templated) | 32-char random; replace before first start |
| `DB_HOST` | `mysql` | Compose service name resolution |
| `DB_PORT` | `3306` | Internal MySQL port |
| `DB_DATABASE` | `nomad` | Database name |
| `DB_USER` | `nomad_user` | Non-root MySQL user |
| `DB_PASSWORD` | (templated) | Must equal MySQL service `MYSQL_PASSWORD` |
| `DB_NAME` | `nomad` | Legacy alias for `DB_DATABASE` |
| `DB_SSL` | `false` | TLS disabled by default |
| `REDIS_HOST` | `redis` | |
| `REDIS_PORT` | `6379` | |
| `DISABLE_COMPRESSION` | `false` | Skip the in-process gzip middleware |

### 4.3 In-app settings (KV store + benchmark settings)

The `kv_store` table holds free-form, schema-validated settings. Defined keys
(from `admin/constants/kv_store.ts`):

| Key | Type | Purpose |
|-----|------|---------|
| `chat.suggestionsEnabled` | boolean | Show prompt-starter chips on the chat home |
| `chat.lastModel` | string | Persist last used Ollama model per server (UI also remembers per-browser) |
| `ui.hasVisitedEasySetup` | boolean | Suppresses the easy-setup nag once first run completes |
| `ui.theme` | string | Server-side fallback for theme; React provider owns the actual choice |
| `system.earlyAccess` | boolean | Reveals beta UI |
| `ai.assistantCustomName` | string | User-facing rename of "AI Assistant" |
| `ai.remoteOllamaUrl` | string | Switch to remote LLM mode |
| `ai.ollamaFlashAttention` | boolean | Pass `OLLAMA_FLASH_ATTENTION=1` to the local Ollama container |

The `benchmark_settings` table holds keys with their own type system:

| Key | Type | Purpose |
|-----|------|---------|
| `allow_anonymous_submission` | string ('true'/'false') | Default for the submit-anonymous toggle |
| `installation_id` | UUID string | Persistent per-install identifier sent with submissions |
| `last_benchmark_run` | ISO 8601 string | When the last benchmark completed |

### 4.4 AdonisJS config files (`admin/config/*.ts`)

| File | Purpose | Notable values |
|------|---------|----------------|
| `app.ts` | HTTP cookie defaults | `maxAge: '2h'`, `httpOnly: true`, `secure: app.inProduction`, `sameSite: 'lax'` |
| `bodyparser.ts` | Body parsing | `multipart.limit = '110mb'` (knowledge-base file ceiling = 100 MB) |
| `cors.ts` | CORS policy | `origin: ['*']`, `credentials: true`, `maxAge: 90`, methods incl. `DELETE`/`PUT` |
| `database.ts` | Lucid + MySQL | Pool: min 2, max 15, `acquireTimeoutMillis: 10_000`; debug enabled when `NODE_ENV=development` |
| `hash.ts` | Password hashing | scrypt: `cost: 16384`, `blockSize: 8`, `parallelization: 1`, `maxMemory: 33_554_432` |
| `inertia.ts` | Inertia shared props | `appVersion`, `environment`, `aiAssistantName` (cached 60s server-side) |
| `logger.ts` | Pino logger | Pretty in dev; in production writes to `/app/storage/logs/admin.log` |
| `queue.ts` | BullMQ connection | Reads `REDIS_HOST` + `REDIS_PORT` |
| `session.ts` | Sessions | **Disabled** (entire file commented out — N.O.M.A.D. is sessionless) |
| `shield.ts` | Security headers | `csp.enabled: false`, `csrf.enabled: false`, `xFrame.enabled: true / DENY`, `hsts.enabled: false`, `contentTypeSniffing.enabled: true` |
| `static.ts` | Static asset middleware | `etag: true`, `lastModified: true`, `dotFiles: 'ignore'`, `acceptRanges: true` |
| `transmit.ts` | SSE transport | Redis driver, `pingInterval: '30s'`, `keyPrefix: 'transmit:'` |
| `vite.ts` | Vite asset bundler | Build output → `public/assets`, manifest → `public/assets/.vite/manifest.json` |

### 4.5 REST + Inertia route map

A consolidated table; see `admin/start/routes.ts:1` for source of truth.

#### Inertia (HTML) routes

| Path | Controller |
|------|-----------|
| `GET /` | `HomeController.index` (redirects to `/home`) |
| `GET /home` | `HomeController.home` |
| `GET /about` | inline render |
| `GET /chat` | `ChatsController.inertia` |
| `GET /maps` | `MapsController.index` |
| `GET /knowledge-base` | redirect → `/chat?knowledge_base=true` |
| `GET /easy-setup` | `EasySetupController.index` |
| `GET /easy-setup/complete` | `EasySetupController.complete` |
| `GET /docs/:slug` | `DocsController.show` |
| `GET /docs` | redirect → `/docs/home` |
| `GET /settings/{system,apps,legal,maps,models,update,zim,zim/remote-explorer,benchmark,support}` | `SettingsController.<page>` |

#### JSON API routes (104 endpoints across 9 prefixes)

| Prefix | Controller | Method count |
|--------|-----------|--------------|
| `/api/easy-setup/*` + `/api/manifests/*` | `EasySetupController` | 4 |
| `/api/content-updates/*` | `CollectionUpdatesController` | 3 |
| `/api/maps/*` | `MapsController` | 14 |
| `/api/docs/*` | `DocsController` | 1 |
| `/api/downloads/*` | `DownloadsController` | 4 |
| `/api/health` | inline | 1 |
| `/api/ollama/*` | `OllamaController` | 7 |
| `/api/chat/sessions/*` + `/api/chat/suggestions` | `ChatsController` | 8 |
| `/api/rag/*` | `RagController` | 8 |
| `/api/system/*` | `SystemController` + `SettingsController` | 17 |
| `/api/zim/*` | `ZimController` | 8 |
| `/api/benchmark/*` | `BenchmarkController` | 11 |

### 4.6 Database migrations (in execution order)

| Filename prefix | Purpose |
|-----------------|---------|
| `1751086751801_create_services_table` | Core service catalog |
| `1763499145832_update_services_table` | Add columns |
| `1764912210741_create_curated_collections_table` | Legacy curated content (later dropped) |
| `1764912270123_create_curated_collection_resources_table` | (later dropped) |
| `1768170944482_update_services_add_installation_statuses_table` | `installation_status` enum |
| `1768453747522_update_services_add_icon` | Tabler icon name on service rows |
| `1769097600001_create_benchmark_results_table` | Benchmark history |
| `1769097600002_create_benchmark_settings_table` | Benchmark KV |
| `1769300000001_add_powered_by_and_display_order_to_services` | UI metadata |
| `1769300000002_update_services_friendly_names` | Backfill display names |
| `1769324448000_add_builder_tag_to_benchmark_results` | Builder-tag column |
| `1769400000001_create_installed_tiers_table` | Legacy (dropped later) |
| `1769400000002_create_kv_store_table` | KV store |
| `1769500000001_create_wikipedia_selection_table` | Single-row Wikipedia selection |
| `1769646771604_create_create_chat_sessions_table` | Chat sessions |
| `1769646798266_create_create_chat_messages_table` | Chat messages |
| `1769700000001_create_zim_file_metadata_table` | ZIM metadata cache |
| `1770269324176_add_unique_constraint_to_curated_collection_resources_table` | (legacy, before drop) |
| `1770273423670_drop_installed_tiers_table` | Cleanup |
| `1770849108030_create_create_collection_manifests_table` | Spec snapshot store |
| `1770849119787_create_create_installed_resources_table` | Per-resource installs |
| `1770850092871_create_drop_legacy_curated_tables_table` | Drop legacy curated tables |
| `1771000000001_add_update_fields_to_services` | Available-update tracking |
| `1771000000002_pin_latest_service_images` | Pin image tags |
| `1771100000001_migrate_kiwix_to_library_mode` | Switch Kiwix command line |
| `1771200000001_create_map_markers_table` | Map markers |

`naturalSort: true` is set in `database.ts` so the millisecond prefix is
respected.

## 5. Deployment & Scaling

### 5.1 Reference deployment (single host)

The supplied `install/management_compose.yaml` is the canonical layout and
starts six containers:

| Container | Image | Restart | Healthcheck |
|-----------|-------|---------|-------------|
| `nomad_admin` | `ghcr.io/crosstalk-solutions/project-nomad:latest` | `unless-stopped` | `curl -f http://localhost:8080/api/health` every 30s |
| `nomad_dozzle` | `amir20/dozzle:v10.0` | `unless-stopped` | none |
| `nomad_mysql` | `mysql:8.0` | `unless-stopped` | `mysqladmin ping -h localhost` every 30s |
| `nomad_redis` | `redis:7-alpine` | `unless-stopped` | `redis-cli ping` every 30s |
| `nomad_updater` | `ghcr.io/crosstalk-solutions/project-nomad-sidecar-updater:latest` | `unless-stopped` | none |
| `nomad_disk_collector` | `ghcr.io/crosstalk-solutions/project-nomad-disk-collector:latest` | `unless-stopped` | none |

Volumes:

```yaml
volumes:
  /opt/project-nomad/storage  -> /app/storage             # admin
  /var/run/docker.sock        -> /var/run/docker.sock     # admin + dozzle + updater
  /opt/project-nomad/mysql    -> /var/lib/mysql           # mysql
  /opt/project-nomad/redis    -> /data                    # redis
  /opt/project-nomad          -> /opt/project-nomad       # updater (RW)
  /                           -> /host (ro,rslave)        # disk-collector
  nomad-update-shared (named) -> /app/update-shared       # admin <-> updater IPC
```

`extra_hosts: ["host.docker.internal:host-gateway"]` is set so the admin can
reach services bound to the host (Linux equivalent of the macOS shortcut).

### 5.2 Scaling considerations

| Bottleneck | Symptom | Fix |
|------------|---------|-----|
| MySQL pool exhaustion | `acquireTimeoutMillis` errors after ~10s | Bump `pool.max` in `database.ts`, scale MySQL vertically |
| BullMQ throughput | Backlog in `nomad_admin` worker logs | Run dedicated worker processes: `node ace queue:work --queue=downloads`, `--queue=model-downloads`, `--queue=benchmarks` |
| Ollama inference | Slow tokens/sec | Add NVIDIA GPU; or move to remote Ollama on a beefier box |
| Disk full | `Failed to dispatch download job` | Move `/opt/project-nomad/storage` to a bigger volume; rerun `migrate-disk-collector.sh` if needed |
| Network bandwidth | Slow ZIM downloads | Use the resumable downloader's auto-retry; pre-stage ZIMs from a LAN mirror with the **Custom URL** option |

> [!TIP]
> Run multiple queue workers per host to parallelize background work without
> running multiple admin instances. The admin process auto-spawns one
> `queue:work --all` worker; bring up additional workers as separate compose
> services pointing at the same `compose.yml`.

### 5.3 Updates

Production updates flow through the sidecar pattern:

1. UI button (`POST /api/system/update`) → `SystemUpdateService.requestSystemUpdate`
2. Writes `{target_tag, ...}` JSON to `/app/update-shared/update-request`
3. `nomad_updater` polls `/shared/update-request` once per second
4. On request: `sed -i` rewrites the admin image tag in `compose.yml`,
   `docker compose pull`, then sequential `stop`/`rm`/`up -d --no-deps` for
   `admin`, `mysql`, `redis`, `dozzle`
5. Status JSON is written to `/shared/update-status`; admin reads it for UI
   progress

If you've manually customized `compose.yml`, **avoid the in-app update
button** — sed-based rewrites can leave indentation broken on heavily-edited
files. Use `update_nomad.sh` instead.

> [!WARNING]
> A botched update can wedge `compose.yml`. The updater always backs up to
> `compose.yml.bak.<timestamp>` before editing — keep `/opt/project-nomad`
> backed up so you can roll back.

## 6. Operations

### 6.1 Day-2 runbooks

#### Restart the admin only

```bash
docker compose -p project-nomad -f /opt/project-nomad/compose.yml restart admin
```

#### Tail logs

```bash
docker logs -f --tail 200 nomad_admin
docker compose -p project-nomad -f /opt/project-nomad/compose.yml logs -f
# Or use the embedded Dozzle UI at http://<host>:9999
```

#### Force-rerun migrations

The admin runs `node ace migration:run --force` from `install/entrypoint.sh`
on every container start. To re-trigger out-of-band:

```bash
docker exec -it nomad_admin node ace migration:run --force
```

#### Re-seed the service catalog

```bash
docker exec -it nomad_admin node ace db:seed
```

Idempotent: the seeder uses upserts on `service_name`.

#### Back up

```bash
sudo /opt/project-nomad/stop_nomad.sh
sudo tar czf nomad-$(date +%F).tar.gz -C /opt project-nomad
sudo /opt/project-nomad/start_nomad.sh
```

For a faster, app-consistent backup of just the metadata (excluding bulky
content), use `mysqldump`:

```bash
docker exec nomad_mysql mysqldump -u nomad_user -p nomad > nomad.sql
```

#### Rotate `APP_KEY`

`APP_KEY` rotation will invalidate every signed URL in flight (Maps assets,
download tokens). Plan a brief outage:

1. Stop the stack.
2. Edit `/opt/project-nomad/compose.yml` → set new `APP_KEY=...`.
3. Start the stack. `Env.create` will validate; admin restarts.

> [!NOTE]
> Cookies are also encrypted with `APP_KEY` (`@adonisjs/core/helpers Secret`),
> so users may need to re-load their tabs. The stack does not currently use
> server sessions, so there's no session blow-up to worry about.

### 6.2 Monitoring & telemetry

- **Health**: `GET /api/health` returns `{"status":"ok"}` plain — wire to
  Uptime Kuma / Healthchecks.io.
- **System info**: `GET /api/system/info` returns disk, RAM, CPU, GPU, and
  internet-status data structured for the dashboard. Expensive — limit
  polling to once every 5–15s in custom dashboards.
- **Internet status**: `GET /api/system/internet-status` is the lightweight
  endpoint the UI polls. Override the test URL via `INTERNET_STATUS_TEST_URL`.
- **Container state**: read directly from Dozzle (`:9999`) or `docker ps`.
- **Update status**: `GET /api/system/update/status` returns the JSON the
  updater sidecar wrote to the shared volume — `idle | starting | pulling |
  pulled | recreating | complete | error`.
- **Queue**: BullMQ stores state in Redis. Use `bullmq-cli` or any BullMQ
  dashboard pointed at `redis://nomad_redis:6379`.

> [!TIP]
> The admin's pino logger writes to `/app/storage/logs/admin.log` in
> production. Mounted to the host at `/opt/project-nomad/storage/logs/`. Hook
> Promtail/Vector here to centralize.

### 6.3 Background job queues

| Queue | Worker class | Purpose |
|-------|--------------|---------|
| `downloads` | `RunDownloadJob` | Resumable ZIM/PMTiles downloads with stall detection |
| `model-downloads` | `DownloadModelJob` | Ollama `pull` with progress streaming |
| `benchmarks` | `RunBenchmarkJob` | Async benchmark runs (sysbench + Ollama) |
| (no name) | `EmbedFileJob` | Knowledge-base file embedding |
| (no name) | `CheckUpdateJob` | Periodic GitHub release check |
| (no name) | `CheckServiceUpdatesJob` | Container image update polling |

Run dedicated workers via the npm scripts in `admin/package.json`:

```bash
node ace queue:work --queue=downloads
node ace queue:work --queue=model-downloads
node ace queue:work --queue=benchmarks
node ace queue:work --all     # one worker handling every queue
```

### 6.4 Pre-deploy checklist

- [ ] Snapshot `/opt/project-nomad` (or at least `mysql`, `redis`, `compose.yml`)
- [ ] Confirm `docker compose version` reports v2
- [ ] Verify `nvidia-smi` works on the host (if GPU-using)
- [ ] Confirm `curl http://localhost:8080/api/health` returns `{"status":"ok"}`
- [ ] Set `LOG_LEVEL=info` (not `debug`) for production
- [ ] Plan firewall rules for `8080` (admin) and `9999` (Dozzle)
- [ ] Update Crosstalk's GHCR PAT in your CI mirror if you build from source
- [ ] Run a Full benchmark and record the score for hardware regression checks

## 7. Security

### 7.1 Threat model

N.O.M.A.D. is built for a **trusted LAN appliance**:

- No authentication on the admin UI or REST API.
- No HTTPS termination inside the container.
- The admin container has full access to the host Docker socket.
- Sidecar containers have specific elevated privileges
  (`disk-collector` reads the host root filesystem; `updater` writes
  `compose.yml`).

This design is acceptable on a private LAN behind a network firewall but
**unacceptable on the open internet**. Do not expose port 8080 publicly.

> [!CAUTION]
> Anyone on the LAN can stop services, install/uninstall containers, browse
> chat history, and trigger updates. If you need network segmentation, put
> N.O.M.A.D. on a dedicated VLAN.

### 7.2 SSRF protection

The platform blocks loopback and link-local URLs at the validator layer:

```ts
// admin/app/validators/common.ts
const blockedPatterns = [
  /^localhost$/,
  /^127\.\d+\.\d+\.\d+$/,
  /^0\.0\.0\.0$/,
  /^169\.254\.\d+\.\d+$/,        // AWS/GCP/Azure metadata
  /^\[::1\]$/,
  /^\[?fe80:/i,                  // IPv6 link-local
  /^\[::ffff:/i,                 // IPv4-mapped IPv6 (e.g. [::ffff:7f00:1])
  /^\[::\]$/,                    // IPv6 all-zeros
]
```

Triggered through `assertNotPrivateUrl(url)` in every download path
(maps, ZIM, content-update). RFC1918 ranges (`10.x`, `172.16-31.x`,
`192.168.x`) are intentionally **allowed** because LAN mirrors of content are
a supported use case.

### 7.3 Network exposure

| Port | Service | Reachable |
|------|---------|-----------|
| 8080 | admin | LAN |
| 9999 | dozzle | LAN |
| 11434 | ollama (when installed) | container network only by default |
| 6333 | qdrant (when installed) | container network only by default |
| MySQL 3306 | nomad_mysql | container network only |
| Redis 6379 | nomad_redis | container network only |

> [!WARNING]
> Even with default install, `9999` (Dozzle) is reachable on the LAN. Logs
> can leak credentials, internal paths, and PII. If your LAN isn't fully
> trusted, drop the `dozzle` service from `compose.yml`.

### 7.4 Headers and middleware (`admin/config/shield.ts`)

| Header / control | State | Note |
|------------------|-------|------|
| `X-Frame-Options: DENY` | Enabled | Set via shield |
| Content-Type sniffing | Disabled in browsers (`X-Content-Type-Options: nosniff`) | Enabled |
| CSP | **Disabled** | Inertia + Vite asset reloading made strict CSP painful; tracked TODO |
| CSRF | **Disabled** | TODO comment in source — sessionless API; would need re-architecture |
| HSTS | **Disabled** | Marked TODO — enable when fronting with TLS |

> [!IMPORTANT]
> When you put N.O.M.A.D. behind a reverse proxy (Caddy, Traefik, nginx),
> the proxy is responsible for HSTS, TLS, and any auth gate. The app trusts
> `X-Forwarded-Proto` for generating the maps `styles.json` URL.

### 7.5 Secrets handling

- **Where**: `compose.yml` contains `APP_KEY`, `DB_PASSWORD`,
  `MYSQL_ROOT_PASSWORD`, and `MYSQL_PASSWORD` in plain text.
- **Generation**: install script uses `tr -dc 'A-Za-z0-9' < /dev/urandom |
  head -c 32` for each.
- **Hashing**: `admin/config/hash.ts` uses scrypt with cost 16384, blockSize
  8, parallelization 1, max memory 32 MiB. Currently unused (no end-user
  passwords) but available for future auth.
- **Benchmark HMAC**: `BENCHMARK_HMAC_SECRET` is a *hardcoded* shared secret
  in `admin/app/services/benchmark_service.ts`. Acceptable — N.O.M.A.D. is
  open source; the secret only deters casual abuse of the public benchmark
  endpoint, and a determined attacker can extract it from source.

### 7.6 Container security posture

| Container | Privilege |
|-----------|-----------|
| `nomad_admin` | Standard, but **bind-mounted Docker socket** = root-equivalent on host |
| `nomad_mysql` | Standard |
| `nomad_redis` | Standard |
| `nomad_dozzle` | Bind-mounted Docker socket (read-only logs); `DOZZLE_ENABLE_ACTIONS=false` and `DOZZLE_ENABLE_SHELL=false` to suppress unauth privilege escalation |
| `nomad_disk_collector` | Bind-mount `/:/host:ro,rslave` (read-only host root) — same pattern as Prometheus node-exporter; no `privileged`, no `SYS_ADMIN` |
| `nomad_updater` | Bind-mount Docker socket + write access to `/opt/project-nomad/compose.yml` |

> [!CAUTION]
> Anyone who can drop a malicious image inside `compose.yml` and trigger the
> in-app update can run code as root on the host. Treat write access to
> `/opt/project-nomad/` as equivalent to root.

### 7.7 Input validation

Vine schemas guard every JSON-accepting endpoint. Patterns to be aware of:

- Filenames: `vine.string().trim().minLength(1).maxLength(4096)` — the
  controllers also pass through `sanitizeFilename` in `admin/app/utils/fs.ts`.
- URLs: `vine.string().url({require_tld: false})` to permit LAN hostnames,
  paired with `assertNotPrivateUrl` for SSRF defense.
- Builder-tag format: `^[A-Za-z]+-[A-Za-z]+-\d{4}$` enforced before save.
- Validator schemas for chat, settings, ollama models, ZIM lists, downloads,
  benchmarks, content updates, and Wikipedia selection. Required keys are
  documented in §10.

### 7.8 Dependency hygiene

- `npm audit` should be clean as of the latest tag — repo carries
  `package-lock.json` for reproducible installs.
- Heavy parsers (`pdf-parse`, `tesseract.js`, `cheerio`, `fast-xml-parser`,
  `jszip`, `tar`) are sandboxed only by Node's process isolation. The admin
  process is the only thing parsing user-supplied uploads. Untrusted file
  content is a real attack surface — keep the host updated.

## 8. Developer Internals

### 8.1 Repository layout

```
project-nomad/
├── package.json                 (metadata only)
├── package-lock.json            (root lock — corresponds to bare metadata)
├── Dockerfile                   (admin image, multi-stage, copies into /app)
├── README.md / FAQ.md / CODE_OF_CONDUCT.md / CONTRIBUTING.md
├── .releaserc.json              (semantic-release config)
├── admin/                       AdonisJS application
│   ├── ace.js                   (TypeScript-aware ace launcher)
│   ├── adonisrc.ts              (providers, preloads, command list)
│   ├── package.json             (full dependency set)
│   ├── package-lock.json
│   ├── tsconfig.json / tailwind.config.ts / vite.config.ts / eslint.config.js
│   ├── start/                   (env.ts, kernel.ts, routes.ts)
│   ├── config/                  (13 config files — see §4.4)
│   ├── constants/               (broadcast, kiwix, kv_store, misc, ollama, service_names, zim_extraction)
│   ├── app/
│   │   ├── controllers/         (13)
│   │   ├── exceptions/          (handler.ts + InternalServerErrorException)
│   │   ├── jobs/                (BullMQ workers)
│   │   ├── middleware/          (compression, container_bindings, force_json_response, maps_static)
│   │   ├── models/              (10 Lucid models)
│   │   ├── services/            (17 services — see §8.2)
│   │   ├── utils/               (downloads, fs, misc, version)
│   │   └── validators/          (10 Vine schemas)
│   ├── database/migrations/     (23 migrations + service_seeder.ts)
│   ├── docs/                    (Markdoc files for the in-app docs page)
│   ├── inertia/
│   │   ├── app/app.tsx          (React + Inertia bootstrap)
│   │   ├── pages/               (page-level components)
│   │   ├── components/          (shared UI: chat, maps, file-uploader, markdoc renderers, system info, etc.)
│   │   ├── hooks/               (useDebounce, useSystemInfo, useTheme, …)
│   │   ├── layouts/             (AppLayout, DocsLayout, MapsLayout, SettingsLayout)
│   │   ├── lib/                 (api.ts, classNames, navigation, util, icons, builderTagWords)
│   │   ├── providers/           (ModalProvider, NotificationProvider, ThemeProvider)
│   │   ├── context/             (ModalContext, NotificationContext)
│   │   └── css/app.css
│   ├── public/                  (favicons, branding webp/svg, in-app docs images)
│   ├── resources/views/         (Edge inertia_layout.edge)
│   ├── tests/                   (Japa bootstrap; suites: unit, functional)
│   ├── types/                   (benchmark, chat, collections, docker, downloads, files, kv_store, maps, ollama, rag, services, system, util, zim)
│   └── views/                   (deprecated location of inertia_layout.edge)
├── collections/                 (Markdown todo list + JSON specs for kiwix-categories, maps, wikipedia)
└── install/
    ├── *.sh                     (install/start/stop/uninstall/update + migrate-disk-collector + run_updater_fixes)
    ├── management_compose.yaml  (canonical compose definition)
    ├── sidecar-disk-collector/  (Dockerfile + collect-disk-info.sh)
    ├── sidecar-updater/         (Dockerfile + update-watcher.sh)
    └── wikipedia_en_100_mini_*.zim  (small test ZIMs shipped for offline first-run)
```

### 8.2 Service catalog (`admin/app/services/`)

| Service | Responsibility |
|---------|----------------|
| `BenchmarkService` | Hardware detection (`systeminformation`), sysbench orchestration, Ollama timing, NOMAD-score calculation, repository submission with HMAC |
| `ChatService` | Sessions/messages CRUD, default-suggestion list, title generation via the active model |
| `CollectionManifestService` | Fetch + cache `zim_categories`, `maps`, `wikipedia` JSON specs from `https://api.projectnomad.us` (or `NOMAD_API_URL`); spec_version-aware |
| `CollectionUpdateService` | Per-resource update detection (compare installed_version to latest_version) and apply via dispatched download jobs |
| `ContainerRegistryService` | Token-based GHCR/Docker Hub auth, tag listing, retry-with-backoff |
| `DockerService` | The Docker socket abstraction: createContainer, inspect, pull, start/stop/restart, image cleanup, network attach; also handles the Kiwix command migration |
| `DocsService` | Reads `admin/docs/*.md`, validates with Markdoc, returns parsed AST for the in-app docs viewer |
| `DownloadService` | Generic resumable downloader (HEAD probe, range requests, stall timer, atomic move via tmp suffix) |
| `KiwixLibraryService` | Builds and refreshes `/data/kiwix-library.xml` from installed ZIMs |
| `MapService` | Protomaps base-asset download, region listing, custom PMTiles ingest, marker CRUD passthrough, generates dynamic `styles.json` |
| `OllamaService` | Wraps the official `ollama` SDK; model catalog, pull progress, stream chat, "thinking" detection, remote-mode probing |
| `QueueService` | Thin wrapper around BullMQ Queue/Worker constructors |
| `RagService` | File chunking via `@chonkiejs/core`, embedding via Ollama `nomic-embed-text`, Qdrant CRUD, similarity search, query rewriting |
| `SystemService` | Service catalog operations (install/affect/force-reinstall/check-updates), debug info, NVIDIA SMI passthrough, internet-status check |
| `SystemUpdateService` | Writes update-request JSON, reads update-status, retrieves admin app version |
| `ZIMExtractionService` | Streams ZIM articles via `@openzim/libzim`, applies `HTML_SELECTORS_TO_REMOVE` and `NON_CONTENT_HEADING_PATTERNS` filters |
| `ZimService` | ZIM list/list-remote/curated-categories, Wikipedia selection state, deletion with cleanup |

### 8.3 Inertia frontend stack

- **Bootstrap** in `admin/inertia/app/app.tsx`:
  - Patches `crypto.randomUUID` for non-HTTPS contexts (LAN deploys often
    don't terminate TLS at the container).
  - Wraps the app in QueryClientProvider, ThemeProvider, TransmitProvider
    (real-time SSE), NotificationsProvider, ModalsProvider.
  - Shows React Query Devtools only when `environment in ['development',
    'staging']`.
- **Page resolution**: `resolvePageComponent` uses a glob over
  `admin/inertia/pages/**/*.tsx` so adding a new page is just dropping a file.
- **Shared props**: `inertia.ts` config injects `appVersion`, `environment`,
  and `aiAssistantName` (cached one minute on the server, invalidated when
  `ai.assistantCustomName` is updated).
- **Routing**: server-side via Adonis routes; the React side is a fully
  Inertia-style SPA — no React Router, no client-side routing.
- **Real-time channels**: `admin/constants/broadcast.ts` defines:

```ts
BENCHMARK_PROGRESS:    'benchmark-progress'
OLLAMA_MODEL_DOWNLOAD: 'ollama-model-download'
SERVICE_INSTALLATION:  'service-installation'
SERVICE_UPDATES:       'service-updates'
```

The frontend subscribes via `react-adonis-transmit` hooks; the backend
publishes via `transmit.broadcast(channel, payload)`.

### 8.4 Lucid models (`admin/app/models/`)

| Model | Table | Notes |
|-------|-------|-------|
| `BenchmarkResult` | `benchmark_results` | All score columns + `submitted_to_repository` |
| `BenchmarkSetting` | `benchmark_settings` | `getValue` / `setValue` / `getAllSettings` typed accessors |
| `ChatSession` | `chat_sessions` | hasMany `messages` |
| `ChatMessage` | `chat_messages` | belongsTo `session` (custom FK mapping `session_id`) |
| `CollectionManifest` | `collection_manifests` | Stores cached spec JSON; `consume`/`prepare` JSON.parse round-tripping |
| `InstalledResource` | `installed_resources` | Type union `'zim' | 'map'`, `version`, `file_path` |
| `KVStore` | `kv_store` | Generic KV with type-aware getters via `KV_STORE_SCHEMA` |
| `MapMarker` | `map_markers` | name, longitude, latitude, color, route_id (future) |
| `Service` | `services` | The service catalog row; `dependency` and `dependencies` self-referential relations |
| `WikipediaSelection` | `wikipedia_selection` | Single user choice; status enum |

All use `SnakeCaseNamingStrategy` and `@column.dateTime({ autoCreate: true,
autoUpdate: true })` for timestamps.

### 8.5 Code patterns and conventions

- **Path aliases**: `admin/package.json` `imports` field maps `#controllers/*`
  → `./app/controllers/*.js`, `#services/*`, `#middleware/*`, `#models/*`,
  `#validators/*`, `#jobs/*`, `#config/*`, `#start/*`, `#exceptions/*`,
  `#database/*`, `#tests/*`, `#providers/*`, `#policies/*`, `#abilities/*`,
  `#mails/*`, `#listeners/*`, `#events/*`. Always import via these aliases.
- **Hot-reload boundaries** (development only): `hotHook.boundaries` in
  `admin/package.json` lists `./app/controllers/**/*.ts` and
  `./app/middleware/*.ts` — these get HMR'd via `hot-hook`.
- **DI**: services are decorated `@inject()` and constructor-injected.
  Controllers take services in the constructor.
- **Streaming**: never use Adonis's `response.send()` for SSE; flush headers
  manually and write to `response.response` (the underlying Node res).
- **Validation first**: every controller method that takes JSON should call
  `request.validateUsing(schema)` before touching services.
- **Errors**: throw `InternalServerErrorException` (`admin/app/exceptions/`)
  to surface a 500 with a structured payload; everything else is a plain
  Error. The custom handler in `admin/app/exceptions/handler.ts` formats
  output as JSON when `Accept` includes `application/json`.
- **Transactions**: use `Database.transaction()` in services; controllers
  shouldn't open transactions directly.
- **Logging**: use the injected `logger` (or `import logger from
  '@adonisjs/core/services/logger'`) and tag with the class name in square
  brackets (`logger.info('[BenchmarkService] ...')`). Pino formatting follows
  this convention everywhere.

### 8.6 Testing strategy

- Test runner: **Japa 4** with `@japa/assert` and `@japa/plugin-adonisjs`.
- Suites in `admin/adonisrc.ts`:
  - `unit` — `tests/unit/**/*.spec(.ts|.js)`, 2s timeout
  - `functional` — `tests/functional/**/*.spec(.ts|.js)`, 30s timeout
- Bootstrap (`admin/tests/bootstrap.ts`) starts the HTTP server for any suite
  with `name in ['browser', 'functional', 'e2e']`.
- Run tests:

```bash
cd admin
node ace test                     # all suites
node ace test --suites=unit       # single suite
```

> [!NOTE]
> The test suite is sparse at present — community contributions to expand
> coverage on services and validators are welcome. CI runs whatever exists
> on every PR.

### 8.7 Build pipeline

- **Image**: `Dockerfile` in repo root produces the admin GHCR image. Multi-
  stage: install deps → run `node ace build` (compiles TypeScript + Vite
  bundles the React app) → copy minimal runtime layer.
- **Local dev**: `cd admin && npm run dev` runs `node ace serve --hmr`. Vite
  serves the SPA in middleware mode through `@adonisjs/vite/vite_middleware`.
- **Type-check**: `npm run typecheck` (`tsc --noEmit`).
- **Lint**: `npm run lint` (ESLint flat config plus `@tanstack/eslint-plugin-query`
  recommended).
- **Format**: `npm run format` (Prettier with `@adonisjs/prettier-config`).
- **Release**: semantic-release on `main` (stable) and `rc` (prerelease).
  Each merged commit's conventional-commit type drives version bumps; CI
  publishes the GHCR image plus a GitHub Release.

### 8.8 Adding a new managed service

1. Add the entry to `admin/database/seeders/service_seeder.ts` (or a new
   migration if you want it added without a re-seed).
2. Add the constant to `admin/constants/service_names.ts`.
3. If it needs a special install action (volumes, pre-pull steps), wire it
   in `DockerService.createContainerPreflight`.
4. Add a Tabler icon name to the `icon` column so the UI renders it.
5. If it's a UI-driven service (not a dependency), set `is_dependency_service
   = false` and provide `ui_location`.
6. Restart the admin container and verify it appears under
   **Settings → Apps**.

### 8.9 Potential improvements (open opportunities)

- Re-enable CSP (currently `csp.enabled: false`) once the Vite asset URL
  pattern is settled.
- Re-enable CSRF protection — would need a token-vending route for the SPA.
- Add structured tracing (OpenTelemetry) — pino is great for log correlation
  but tracing across BullMQ workers is currently log-only.
- Pluggable embedding model (today the `nomic-embed-text` choice is
  hardcoded in RAG service).
- ROCm GPU support (NVIDIA-only today, see `setup_nvidia_container_toolkit`
  in `install_nomad.sh`).
- True multi-tenant mode with auth — would touch routes, controllers, and
  the (currently disabled) session/auth providers.
- Service install rollback semantics (currently best-effort; failed installs
  leave a partial container behind).

### 8.10 Contributing

- File issues at the GitHub tracker; PRs against `main` for stable, `rc`
  for upcoming features.
- Keep PRs scoped — one logical change per PR.
- Run `npm run lint && npm run typecheck && node ace test` before opening.
- Commit messages follow Conventional Commits (`feat:`, `fix:`, `chore:`,
  `docs:`, `refactor:`, `test:`) — semantic-release reads them.

## 9. Troubleshooting

### Symptom: Admin container exits at boot with `Cannot validate environment variable APP_KEY`

**Cause:** `APP_KEY` is unset, empty, or shorter than 16 characters.

**Solution:** Edit `compose.yml` and ensure the `APP_KEY=` line under the
admin service has a 16+ character value. Restart the stack.

**Prevention:** Don't hand-edit `compose.yml` without running through
`Env.create` validation — start the admin and watch logs.

### Symptom: `Failed to detect hardware: command not found`

**Cause:** `systeminformation` falls back to OS commands; on a slim base
image something like `lspci` may be missing.

**Solution:** The Crosstalk-published image already bundles the deps — but
if you've built your own image, add `apt-get install -y pciutils` to the
Dockerfile.

### Symptom: AI benchmark returns `Ollama is not running or not accessible`

**Cause:** Reaching the `nomad_ollama` service is blocked, the container is
down, or the model isn't pulled.

**Solution:**

```bash
docker logs nomad_ollama --tail 50
docker exec nomad_admin curl -s http://nomad_ollama:11434/api/tags
```

If the second command 404s on `/api/tags`, Ollama itself is unhealthy.
Restart the container; a fresh `nomad_ollama` will replay the pulled-models
volume.

### Symptom: `Markdoc validation failed`

**Cause:** A `.md` doc in `admin/docs/` uses Markdoc tag syntax that doesn't
parse, or the user uploaded a Markdown file with non-Markdoc tags.

**Solution:** Run `node ace doc:validate` if you're shipping new in-app
docs. For user uploads, the system catches the error and surfaces it through
`InternalServerErrorException` — the Knowledge Base UI shows the failure on
the Files tab.

### Symptom: `Container is unhealthy` after pull

**Cause:** Healthcheck failed within the configured `interval` × `retries`
window. For `nomad_admin` that's 30s × 3 = 90s for `/api/health` to come
back 200.

**Solution:** Tail logs (`docker logs --tail 200 nomad_admin`); look for env
validation errors, MySQL connection refusals, or migration failures. The
admin's entrypoint runs `migration:run --force` then `db:seed` then
`queue:work --all &` then `bin/server.js`; failure in any step will show in
the logs.

### Symptom: Update sidecar reports `Failed to update compose.yml image tag`

**Cause:** `compose.yml` is structurally different from what the
`sed -i 's|\(image: ghcr\.io/crosstalk-solutions/project-nomad\):.*|\1:${target_tag}|'`
pattern expects (e.g., you reformatted YAML or pinned a private registry).

**Solution:** Run `update_nomad.sh` from the host instead, which uses
`docker compose pull` + `up -d --force-recreate` and doesn't edit YAML.
Restore from a `compose.yml.bak.<timestamp>` if the sidecar partially edited
the file.

### Symptom: `extractStructuredContent parses only direct body children`

**Cause:** Modern HTML ZIMs nest article content inside multiple wrapper
divs; the v1.x extractor only walked direct children. Issue
[#720](https://github.com/Crosstalk-Solutions/project-nomad/issues/720).

**Solution:** Update to a release where this is fixed (track the issue);
otherwise use older-format ZIMs (Wikipedia mwoffliner pre-2024).

### Symptom: ZIM batch processing stops at 50 articles

**Cause:** `ZIM_BATCH_SIZE = 50` is the per-batch ceiling and `hasMoreBatches`
used the wrong counter, ending iteration early. Issue
[#719](https://github.com/Crosstalk-Solutions/project-nomad/issues/719).

**Solution:** Update; this is fixed in current `main`. Check
`admin/constants/zim_extraction.ts` for the constant.

### Symptom: `scanAndSyncStorage queues non-embeddable files`

**Cause:** The sync routine queued every file in `kb_uploads` regardless of
extension, including `kiwix-library.xml`. Issue
[#718](https://github.com/Crosstalk-Solutions/project-nomad/issues/718).

**Solution:** Update; the embedder now filters by extension list in
`RagService`. Quick fix: move the offending file out of `kb_uploads`.

### Symptom: SSE stream hangs / `chat` endpoint never sends data

**Cause:** A reverse proxy is buffering. nginx without `proxy_buffering off`
will hold SSE chunks until they flush.

**Solution:** Configure your reverse proxy:

```nginx
location /api/ollama/chat {
  proxy_pass http://nomad_admin:8080;
  proxy_buffering off;
  proxy_http_version 1.1;
  chunked_transfer_encoding off;
}
```

Caddy and Traefik handle this correctly out of the box.

> [!WARNING]
> Don't put a CDN in front of N.O.M.A.D. — most CDNs buffer SSE and the
> chat experience will look broken.

## 10. Quick Reference

### 10.1 Important file paths

```
HOST                                            CONTAINER
/opt/project-nomad/compose.yml          ─►      (n/a — host file)
/opt/project-nomad/storage/             ─►      /app/storage
/opt/project-nomad/storage/logs/admin.log       /app/storage/logs/admin.log
/opt/project-nomad/storage/maps/                /app/storage/maps
/opt/project-nomad/storage/zim/                 /app/storage/zim
/opt/project-nomad/storage/kb_uploads/          /app/storage/kb_uploads
/opt/project-nomad/storage/nomad-disk-info.json /app/storage/nomad-disk-info.json (read by admin)
/opt/project-nomad/mysql/                       /var/lib/mysql
/opt/project-nomad/redis/                       /data
named volume: nomad-update-shared               /app/update-shared (admin) + /shared (updater)
```

### 10.2 Service names (`admin/constants/service_names.ts`)

```
KIWIX:      nomad_kiwix_server
OLLAMA:     nomad_ollama
QDRANT:     nomad_qdrant
CYBERCHEF:  nomad_cyberchef
FLATNOTES:  nomad_flatnotes
KOLIBRI:    nomad_kolibri
```

### 10.3 Key constants (`admin/constants/`)

| Constant | Value | File |
|----------|-------|------|
| `KIWIX_LIBRARY_CMD` | `--library /data/kiwix-library.xml --monitorLibrary --address=all` | `kiwix.ts` |
| `NOMAD_API_DEFAULT_BASE_URL` | `https://api.projectnomad.us` | `misc.ts` |
| `DEFAULT_QUERY_REWRITE_MODEL` | `qwen2.5:3b` | `ollama.ts` |
| `RAG_CONTEXT_LIMITS` | `[{≤3B: 2/1k}, {≤8B: 4/2.5k}, {>8B: 5/no-cap}]` | `ollama.ts` |
| `ZIM_BATCH_SIZE` | `50` | `zim_extraction.ts` |
| `BROADCAST_CHANNELS.BENCHMARK_PROGRESS` | `benchmark-progress` | `broadcast.ts` |
| `BROADCAST_CHANNELS.OLLAMA_MODEL_DOWNLOAD` | `ollama-model-download` | `broadcast.ts` |
| `BROADCAST_CHANNELS.SERVICE_INSTALLATION` | `service-installation` | `broadcast.ts` |
| `BROADCAST_CHANNELS.SERVICE_UPDATES` | `service-updates` | `broadcast.ts` |
| `SETTINGS_KEYS` | 8 valid kv-store keys | `kv_store.ts` |

### 10.4 Useful one-liners

| Task | Command |
|------|---------|
| Tail admin logs | `docker logs -f --tail 200 nomad_admin` |
| Open MySQL CLI | `docker exec -it nomad_mysql mysql -u nomad_user -p nomad` |
| Open Redis CLI | `docker exec -it nomad_redis redis-cli` |
| List installed Ollama models | `docker exec nomad_ollama ollama list` |
| Pull model manually | `docker exec nomad_ollama ollama pull <model>` |
| Force re-pull all images | `docker compose -p project-nomad -f /opt/project-nomad/compose.yml pull --no-cache` |
| Force-recreate one service | `docker compose -p project-nomad -f /opt/project-nomad/compose.yml up -d --force-recreate <service>` |
| Inspect Docker network | `docker network inspect project-nomad_default` |
| Watch update progress | `cat /var/lib/docker/volumes/project-nomad_nomad-update-shared/_data/update-status` |
| Run all queue workers | `docker exec -d nomad_admin node ace queue:work --all` |
| Generate Adonis route table | `docker exec nomad_admin node ace list:routes` |
| Generate debug bundle | `curl -s http://localhost:8080/api/system/debug-info > debug.json` |

### 10.5 Versioning

- **App version**: `package.json` (`version`) — bumped by semantic-release.
- **Image tags on GHCR**: `latest` (stable), `rc` (release candidate),
  exact `vX.Y.Z` from each tagged release.
- **Migrations**: monotonically-numbered; never edit a migration after it's
  shipped — write a new one.
- **DB schema version**: implicit (highest applied migration). No
  `schema_version` table; AdonisJS Lucid manages this internally.

### 10.6 Default ports

| Port | Service | Default-installed? |
|------|---------|:------------------:|
| 8080 | admin | yes |
| 9999 | dozzle | yes |
| 11434 | nomad_ollama | when AI Assistant installed |
| 6333 | nomad_qdrant | when Knowledge Base used |
| 8086 | nomad_kiwix_server | when any ZIM installed |
| 8000 | nomad_cyberchef | optional |
| 8200 | nomad_flatnotes | optional |
| 8080 (mapped) | nomad_kolibri | optional |
| 3306 | nomad_mysql | yes (internal only) |
| 6379 | nomad_redis | yes (internal only) |

---

_Generated April 2026 from Crosstalk-Solutions/project-nomad at 9cbf8c2. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._