MiroFish Administrator & Developer Guide

This guide is the operational and architectural reference for MiroFish — a multi-agent swarm-intelligence prediction engine. It is written for the people who deploy, operate, extend, or audit the system: administrators, DevOps engineers, backend developers, and architects. End-user instructions live in the companion User Guide.

MiroFish turns unstructured seed documents into large-scale social simulations. A Vue 3 single-page app drives a Flask backend, which orchestrates an LLM (via the OpenAI SDK), a Zep Cloud knowledge graph, and the OASIS multi-agent social-simulation framework. There is no database server and no message broker — state is persisted as JSON and CSV files on disk, and long-running work is handled by background threads plus subprocesses.

1. Executive Summary

Important

MiroFish as shipped is a single-tenant, unauthenticated developer/research tool. CORS is wide open (origins: * on /api/*), there is no login, and the Flask dev server is the runtime even in the Docker image. Treat any deployment as trusted-network-only unless you add an auth and reverse-proxy layer yourself.

2. Architecture

2.1 System diagram

┌──────────────────────────────────────────────────────────────────┐
│                       Frontend — Vue 3 SPA                         │
│  Vite dev server :3000  ·  Vue Router  ·  vue-i18n  ·  D3 ·  axios │
│  Views: Home → MainView(Process) → SimulationView →                │
│         SimulationRunView → ReportView → InteractionView           │
└───────────────────────────┬──────────────────────────────────────┘
                            │ HTTP/JSON (axios; 5-min timeout; auto-retry)
                            │ dev: Vite proxies /api → :5001
                            ▼
┌──────────────────────────────────────────────────────────────────┐
│                      Backend — Flask :5001                         │
│  CORS(*) on /api/*   ·   before/after request logging              │
│                                                                    │
│  Blueprints:                                                       │
│    /api/graph/*       → graph_bp        (ontology, graph build)    │
│    /api/simulation/*  → simulation_bp   (entities, prepare, run)   │
│    /api/report/*      → report_bp       (report gen, chat, logs)   │
│    /health            → liveness                                   │
│                                                                    │
│  ┌───────────────────────── Service layer ─────────────────────┐  │
│  │ OntologyGenerator      GraphBuilderService                  │  │
│  │ ZepEntityReader        OasisProfileGenerator                │  │
│  │ SimulationManager      SimulationConfigGenerator            │  │
│  │ SimulationRunner ──────► OASIS subprocess (scripts/run_*.py)│  │
│  │ SimulationIPC (client/server, file-based command channel)  │  │
│  │ ReportAgent (ReAct) ──► ZepToolsService (graph toolset)     │  │
│  │ ZepGraphMemoryUpdater (dynamic temporal memory)             │  │
│  │ TextProcessor                                               │  │
│  └────────────────────────────────────────────────────────────┘  │
│  ┌─────────── Models ──────────┐  ┌──────────── Utils ─────────┐  │
│  │ Project   (JSON on disk)    │  │ LLMClient (OpenAI SDK)     │  │
│  │ Task      (in-memory single)│  │ FileParser (PDF/MD/TXT)    │  │
│  │ Simulation/Report state (fs)│  │ retry / locale / logger    │  │
│  └─────────────────────────────┘  │ zep_paging                 │  │
│                                    └────────────────────────────┘  │
└──────────────┬─────────────────────────────────┬──────────────────┘
               │ OpenAI SDK                       │ zep-cloud SDK
               ▼                                  ▼
        ┌────────────┐                      ┌─────────────┐
        │ LLM API    │                      │  Zep Cloud  │
        │ (OpenAI-   │                      │ graph +     │
        │  compatible)│                     │ memory      │
        └────────────┘                      └─────────────┘

2.2 Request lifecycle for long operations

The three expensive operations — graph build, simulation prepare, and report generate — follow the same asynchronous pattern:

Client POST → handler validates → creates a Task (TaskManager) →
spawns daemon threading.Thread → returns {task_id} immediately (HTTP 200) →
client polls GET task/status → background thread updates Task progress →
on success: Task COMPLETED with result; on error: Task FAILED with message

Note

Because progress lives in the in-memory TaskManager singleton, restarting the backend loses all in-flight task state. Persisted artifacts (the project, simulation files, report sections on disk) survive; the task status objects do not. The prepare/report endpoints mitigate this by also reconstructing "already done?" answers from on-disk files (_check_simulation_prepared, get_report_by_simulation).

2.3 The five-stage pipeline mapped to code

3. Requirements

3.1 Host requirements

3.2 External service requirements

• LLM endpoint — any service speaking the OpenAI Chat Completions API. Must support response_format={"type": "json_object"} for the generation steps to be reliable.
• Zep Cloud — an account and API key. Used for graph storage, entity/edge extraction, and temporal memory.

3.3 Python dependency inventory

Warning

zep-cloud, camel-oasis, and camel-ai are exact-pinned (==). Do not float these — MiroFish calls version-specific APIs (e.g. client.graph.node.get_by_graph_id with uuid_cursor paging). Upgrading them is a deliberate, tested change, not a routine bump.

3.4 Frontend dependency inventory

4. Configuration Mastery

All backend configuration is environment-variable driven and centralized in backend/app/config.py (class Config). The .env file is loaded from the project root (MiroFish/.env), resolved relative to config.py; if absent, environment variables are read directly (the production/Docker path).

4.1 Complete environment variable reference

4.2 Non-env (code-level) configuration

These are set in Config and not env-overridable:

4.3 OASIS platform action sets

The available agent actions per platform are defined in Config and injected into the simulation config:

4.4 Simulation configuration schema

SimulationConfigGenerator (an LLM call) produces a SimulationParameters object composed of:

• AgentActivityConfig — how active agents are.
• TimeSimulationConfig — total_simulation_hours, minutes_per_round; the recommended round count is derived as total_hours × 60 / minutes_per_round.
• EventConfig — scheduled events injected into the world.
• PlatformConfig — per-platform settings.

This serializes to simulation_config.json in the simulation directory.

Tip

To force-regenerate personas or config after changing the model or requirement, pass force_regenerate: true to /api/simulation/prepare. Without it, MiroFish detects completed preparation via _check_simulation_prepared() and skips regeneration to save tokens.

5. Deployment

5.1 Source deployment

git clone https://github.com/666ghj/MiroFish.git
cd MiroFish
cp .env.example .env          # set LLM_API_KEY, ZEP_API_KEY
npm run setup:all             # root + frontend npm install, backend uv sync
npm run dev                   # concurrently runs backend (:5001) + frontend (:3000)

The root package.json scripts are the operational contract:

5.2 Docker deployment

# docker-compose.yml (as shipped)
services:
  mirofish:
    image: ghcr.io/666ghj/mirofish:latest
    container_name: mirofish
    env_file:
      - .env
    ports:
      - "3000:3000"
      - "5001:5001"
    restart: unless-stopped
    volumes:
      - ./backend/uploads:/app/backend/uploads

cp .env.example .env          # set your keys
docker compose up -d

The image (Dockerfile) is python:3.11 with Node/npm added and uv copied from ghcr.io/astral-sh/uv:0.9.26. It installs Node deps (npm ci), frontend deps, and backend deps (uv sync --frozen), then runs CMD ["npm", "run", "dev"].

Caution

The container's command is npm run dev — the Vite dev server and the Flask dev server, not a production WSGI server or a built static bundle. This is convenient but is not a hardened production setup: no Gunicorn/uvicorn, FLASK_DEBUG may be on, and the dev server is single-process. For production, front it with a reverse proxy (TLS, auth), set FLASK_DEBUG=False and a strong SECRET_KEY, serve the built frontend (npm run build) as static files, and run Flask under a real WSGI server.

5.3 The frontend↔backend connection

• Dev: vite.config.js proxies /api → http://localhost:5001 (with changeOrigin). The axios client also has baseURL = import.meta.env.VITE_API_BASE_URL || 'http://localhost:5001'.
• Vite aliases: @ → frontend/src, @locales → repo-root locales/.
• To point the frontend at a non-local backend, set VITE_API_BASE_URL at build/dev time.

Note

Locales are shared between frontend and backend: both read the repo-root locales/ directory (en.json, zh.json, languages.json). The frontend imports them via the @locales alias and import.meta.glob; the backend loads them in utils/locale.py. Keep them in sync when adding a language.

6. API Reference

All endpoints are JSON over HTTP. Success responses follow {"success": true, "data": ...}; errors follow {"success": false, "error": "...", "traceback": "..."} with an appropriate status code (400 validation, 404 not found, 500 internal). There is no authentication.

Note

Paths below include the blueprint prefix. In the source, route decorators are blueprint-relative; the prefixes (/api/graph, /api/simulation, /api/report) are attached in app/__init__.py via register_blueprint.

6.1 Health

6.2 Graph API (/api/graph)

6.3 Simulation API (/api/simulation)

Entity reading

Lifecycle

Profiles & config inspection

Run control

Interaction (Step 5)

6.4 Report API (/api/report)

7. Operations

7.1 Starting and stopping

npm run dev          # both services (foreground, color-coded)
npm run backend      # backend only
npm run frontend     # frontend only
docker compose up -d # detached Docker
docker compose down  # stop Docker

The backend registers a cleanup hook (SimulationRunner.register_cleanup()) on startup so that all OASIS subprocesses are terminated when the server shuts down. A clean shutdown of the Flask process therefore reaps child simulations.

7.2 Data layout on disk

Everything persists under backend/uploads/ (the Docker bind mount). Knowing this layout is essential for backup, debugging, and cleanup.

backend/uploads/
├── projects/
│   └── <project_id>/
│       ├── project.json          # Project metadata + ontology
│       ├── extracted_text.txt    # Concatenated seed text
│       └── files/                # Uploaded seed files (renamed)
├── simulations/
│   └── <simulation_id>/
│       ├── state.json            # SimulationState
│       ├── simulation_config.json
│       ├── reddit_profiles.json
│       ├── twitter_profiles.csv
│       └── run_state.json        # Round/action progress
└── reports/
    └── <report_id>/
        ├── meta.json
        └── section_NN.md         # Streamed report sections

• Project IDs: proj_<12 hex> · Reports: report_<12 hex> · Tasks: UUID4 · Simulations: sim_*.

7.3 Backup and restore

Because all durable state is files, backup is a directory copy:

tar czf mirofish-backup-$(date +%F).tgz backend/uploads/

Restore by extracting back into backend/uploads/. There is no database to dump. Note that the Zep graphs live in Zep Cloud, not on disk — a local backup captures projects/simulations/reports but not the remote graph data.

7.4 Logging

utils/logger.py configures a logger named mirofish with two sinks:

• File: backend/logs/YYYY-MM-DD.log, RotatingFileHandler, 10 MB × 5 backups, DEBUG level, detailed format.
• Console: INFO and above, concise format, forced UTF-8 on Windows.

Every request is logged (before_request/after_request); request bodies are logged at DEBUG when JSON.

Warning

Request-body DEBUG logging writes inbound JSON to disk. With FLASK_DEBUG=True (the default) this can persist user-supplied content and, potentially, secrets passed in payloads. Lower the log level and disable debug for any shared deployment.

7.5 Housekeeping

• TaskManager.cleanup_old_tasks(max_age_hours=24) prunes completed/failed tasks older than a day from memory.
• SimulationRunner exposes cleanup_simulation_logs / cleanup_all_simulations for log hygiene.
• Old projects/, simulations/, and reports/ directories are not auto-purged — manage disk usage yourself.

8. Internals — Module Reference

8.1 Services (backend/app/services/)

8.2 Models (backend/app/models/)

• Project (project.py) — a dataclass persisted as project.json. Tracks status (created → ontology_generated → graph_building → graph_completed / failed), files, ontology, graph_id, chunk settings. ProjectManager handles filesystem CRUD and file storage (uploaded files renamed to <8 hex><ext>).
• Task (task.py) — an in-memory async-task record (status, progress 0–100, message, result, error). TaskManager is a thread-safe singleton (__new__ + threading.Lock). Not persisted — see the restart caveat in §2.2.

8.3 Utilities (backend/app/utils/)

8.4 Frontend structure (frontend/src/)

• Entry: main.js mounts App.vue with router + i18n. App.vue is just <router-view/>.
• Routing (router/index.js): Home (/), Process (/process/:projectId, = MainView), Simulation (/simulation/:simulationId), SimulationRun (/simulation/:simulationId/start), Report (/report/:reportId), Interaction (/interaction/:reportId).
• API layer (api/): index.js is the axios instance (5-min timeout, Accept-Language injection, response unwrapping, requestWithRetry with exponential backoff). graph.js, simulation.js, report.js are typed wrappers over the backend endpoints.
• Components: Step1GraphBuild … Step5Interaction, plus GraphPanel (D3), HistoryDatabase, LanguageSwitcher.

Note

The axios client unwraps response.data and rejects when res.success === false, so frontend callers receive the data object directly and catch errors as thrown Errors. Any new endpoint must follow the {success, data|error} envelope to stay compatible.

9. Security Analysis

Caution

MiroFish ships with no authentication, no authorization, and permissive CORS. The following are not theoretical — they are properties of the code as written. Deploy accordingly.

10. Code Quality & Conventions

Strengths

• Clean layering: API → services → models/utils with single-responsibility modules. The 60 endpoints group cleanly into three blueprints.
• Consistent async pattern: every long task returns a task_id and runs in a daemon thread with a uniform progress-callback shape.
• Robustness: centralized retry/backoff (retry.py, RetryableAPIClient), Zep paging with per-page retry, encoding fallbacks for text, and idempotent prepare/report (skip-if-done from on-disk state).
• i18n done thread-aware: locale captured before spawning background threads (current_locale = get_locale(); set_locale(current_locale)), a subtle correctness detail many codebases miss.
• LLM defensiveness: stripping <think> blocks and Markdown fences before JSON parsing acknowledges real model behavior.

Risks & rough edges

• In-memory TaskManager loses progress on restart; partially mitigated by on-disk reconstruction but not for arbitrary in-flight tasks.
• No tests are wired into a pipeline beyond the pytest/pytest-asyncio dev deps; coverage of the service layer is unverified from the source tree.
• Large handler files: simulation.py is ~2,600 lines; route handlers carry nontrivial business logic that could move into services.
• Tracebacks leak to clients by default, coupling debuggability with information disclosure.
• Bilingual code: error strings and many docstrings are Chinese; non-Chinese-reading maintainers should rely on the locales/ keys.

11. Testing

The backend declares dev tooling (pytest, pytest-asyncio, pipreqs) under [project.optional-dependencies].dev and [dependency-groups].dev. There are test scripts under backend/scripts/ (e.g. test_profile_format.py).

To run tests after installing dev deps:

cd backend
uv sync --extra dev
uv run pytest

Note

Because the system depends on live external services (LLM, Zep) and on the OASIS subprocess, meaningful end-to-end testing requires either real credentials or mocks for the OpenAI and zep-cloud clients. Unit-test the pure logic (TextProcessor chunking, FileParser, retry, locale, the dataclass to_dict/from_dict round-trips) without external calls.

12. Troubleshooting (Operations)

Backend exits immediately with 配置错误

Config.validate() returns errors when LLM_API_KEY or ZEP_API_KEY is unset. The process prints the missing keys and sys.exit(1). Fix .env (project root) and restart.

Graph build task fails

Check backend/logs/ for the mirofish.build logger. Common causes: missing/invalid ZEP_API_KEY, Zep rate limits, or LLM JSON failures during ontology use. The project status flips to failed with the error stored; reset and rebuild.

500 on ontology generation with reasoning models

Reasoning models (Qwen3, DeepSeek-R1) emit content that breaks strict JSON (community issue #642). LLMClient strips <think> and fences, but switch LLM_MODEL_NAME to a standard chat model for the generation steps.

PDF upload → Unexpected non-whitespace character after JSON

Reported in issues #672/#670 (byte-identical error position across attempts). Workaround: upload Markdown/text instead of the offending PDF, or regenerate the PDF. Keep files under 50 MB.

Simulation never reaches ready

Inspect the simulation directory under backend/uploads/simulations/<id>/. _check_simulation_prepared requires state.json, simulation_config.json, reddit_profiles.json, and twitter_profiles.csv to all exist, with config_generated: true. Missing files indicate the prepare task failed mid-way — re-run prepare (optionally force_regenerate).

Interviews fail with "environment not running"

Live interviews go over IPC to the OASIS subprocess (无法执行Interview errors come from simulation_runner.py). If the env was closed or crashed, restart the simulation. Use /api/simulation/env-status to check liveness.

Repeated 429/500 under load

Provider rate limiting (issue #662). Reduce rounds/agents, throttle, or upgrade your LLM tier. The built-in backoff helps but cannot exceed your provider's quota.

Orphaned OASIS processes

A clean Flask shutdown reaps children via the registered cleanup hook. If the server was killed -9, check for stray Python processes from backend/scripts/run_*_simulation.py and terminate them; then use cleanup_all_simulations housekeeping.

13. Potential Improvements

These are derived from the code and from community issues; none are implemented as of 96096ea.

• Persist task state (SQLite or a JSON journal) so progress survives restarts instead of living only in the TaskManager singleton.
• Production runtime: ship a built frontend served as static assets and run Flask under a WSGI server (Gunicorn/uvicorn-worker) rather than npm run dev in the container.
• Add an auth layer and tighten CORS before any multi-user or public exposure.
• Stop leaking tracebacks to clients when not in debug.
• Report export formats (issue #636 requests saving the final report) — a Markdown download exists; PDF/HTML export and a clearer in-UI "save report" affordance would close the gap.
• OpenZep / self-hosted Zep support (issue #183) to escape Zep Cloud quota limits.
• Thin the route handlers by moving business logic from simulation.py/report.py into the service layer for testability.
• Resilience for reasoning models: a stricter JSON-repair or function-calling path would reduce the #642 class of failures.

14. Quick Reference

Ports & URLs

Key files

ID formats

Operational commands

npm run dev                       # run both services
docker compose up -d              # run via Docker
curl localhost:5001/health        # liveness
tar czf backup.tgz backend/uploads/   # back up all state