# BettaFish Admin & Architecture Guide

This guide is for the engineers, architects, and operators who deploy, extend, and maintain BettaFish. It documents the system's architecture, configuration surface, security posture, operational procedures, and internals in enough depth to run it reliably or rebuild it from scratch.

For installation and day-to-day usage, see the companion **User Guide**.

---

## Table of Contents

1. [Executive Summary](#1-executive-summary)
2. [System Architecture](#2-system-architecture)
3. [Component Reference](#3-component-reference)
4. [Requirements & Dependencies](#4-requirements--dependencies)
5. [Configuration Mastery](#5-configuration-mastery)
6. [Data Layer & Schema](#6-data-layer--schema)
7. [HTTP API Reference](#7-http-api-reference)
8. [Deployment](#8-deployment)
9. [Operations](#9-operations)
10. [Security Analysis](#10-security-analysis)
11. [Code Patterns & Conventions](#11-code-patterns--conventions)
12. [Testing](#12-testing)
13. [Extending the System](#13-extending-the-system)
14. [Potential Improvements & Tech Debt](#14-potential-improvements--tech-debt)
15. [Quick Reference](#15-quick-reference)

---

## 1. Executive Summary

BettaFish is a **multi-agent public-opinion analysis platform** written in pure Python. It orchestrates four LLM-driven "deep search" agents, a moderator LLM, an AI crawler, and a collection of local sentiment models to turn a natural-language question into a chart-rich analysis report.

**Architectural character:**

- **Modular monolith.** A single Flask + Socket.IO application (`app.py`) supervises everything: it launches three Streamlit sub-applications as child processes, runs the ForumEngine in a background thread, and mounts the ReportEngine as a Flask blueprint. There is no microservice mesh and no message broker.
- **Hand-rolled agent framework.** The agents do **not** use LangChain/LangGraph. Each engine implements its own node-graph (`nodes/`, `state/`, `llms/`, `prompts/`, `tools/`) — the `__init__` describes it as a "framework-less deep search AI agent." All four analysis engines share the same node structure and differ only in their tools and data source.
- **Collaboration via shared log files.** The agents do not call each other. Each writes to its own log; the **ForumEngine** tails those logs, extracts summaries, and republishes them to a shared `forum.log`, where a moderator LLM periodically posts synthesis. This file-as-message-bus design is the system's defining (and most unusual) pattern.
- **Provider-agnostic.** Every LLM interaction uses the OpenAI request format. Each of seven logical "slots" can point at a different provider via `api_key` / `base_url` / `model`.
- **Configuration as a single Pydantic settings object** loaded from one root `.env`.

| Attribute | Value |
|-----------|-------|
| Language | Python 3.9+ (3.11 recommended) |
| Web framework | Flask 2.3 + Flask-SocketIO 5.3 (eventlet) |
| Sub-app UI | Streamlit 1.28 |
| LLM interface | `openai` SDK, OpenAI-compatible |
| Crawler | Playwright + MediaCrawler (git submodule) |
| Database | PostgreSQL (recommended) or MySQL, via SQLAlchemy (async drivers) |
| Config | `pydantic-settings` ← root `.env` |
| Report rendering | WeasyPrint (PDF), inlined Chart.js/MathJax (HTML) |
| License | GPL-2.0 |
| Version | v1.2.1 |

---

## 2. System Architecture

### 2.1 Process & component topology

```text
                         ┌─────────────────────────────────────────────┐
                         │   app.py  — Flask + Socket.IO  (port 5000)   │
                         │   • supervises sub-apps  • /api/* control     │
                         │   • /api/report/* (ReportEngine blueprint)    │
                         │   • config editor → writes root .env          │
                         └───────┬───────────────┬──────────────┬───────┘
            launches as          │               │              │  mounts
            subprocesses ┌───────┴──────┐ ┌──────┴───────┐ ┌────┴────────┐
                         │ QueryEngine  │ │ MediaEngine  │ │ InsightEngine│
                         │ Streamlit    │ │ Streamlit    │ │ Streamlit    │
                         │ :8503        │ │ :8502        │ │ :8501        │
                         └──────┬───────┘ └──────┬───────┘ └──────┬───────┘
        writes logs/query.log   │  logs/media.log│ logs/insight.log│
                                ▼                ▼                 ▼
                         ┌──────────────────────────────────────────────┐
                         │  ForumEngine (background thread in app.py)    │
                         │  • tails the 3 engine logs                    │
                         │  • extracts SummaryNode JSON                  │
                         │  • writes logs/forum.log  ([INSIGHT]/[MEDIA]/ │
                         │    [QUERY]/[HOST]/[SYSTEM])                    │
                         │  • every 5 speeches → ForumHost LLM ([HOST])  │
                         └───────────────────────┬──────────────────────┘
                  agents read forum.log via      │
                  utils/forum_reader.py ◄─────────┘
                                ▼
                         ┌──────────────────────────────────────────────┐
                         │  ReportEngine (blueprint + flask_interface)   │
                         │  template → layout → budget → chapters → IR   │
                         │  → renderers (HTML / PDF / Markdown)          │
                         └──────────────────────────────────────────────┘

   MindSpider (standalone CLI) ──► Database ◄── InsightEngine queries (read-only)
   BroadTopicExtraction → daily_topics ; DeepSentimentCrawling → platform tables
```

### 2.2 The deep-search agent node graph

All four analysis engines (`QueryEngine`, `MediaEngine`, `InsightEngine`, and structurally `ReportEngine`) implement the same package layout:

```text
<Engine>/
├── agent.py        # orchestrator — drives the node sequence
├── llms/base.py    # OpenAI-compatible LLM client (per-engine key/url/model)
├── nodes/          # processing nodes (the "graph")
│   ├── base_node.py
│   ├── search_node.py
│   ├── summary_node.py
│   ├── report_structure_node.py
│   └── formatting_node.py
├── state/state.py  # mutable run state (paragraphs, search history, summaries)
├── prompts/prompts.py
├── tools/          # the data-source differentiator
└── utils/          # config.py, db.py (Insight only), text_processing.py
```

The execution sequence (from `InsightEngine/agent.py::research`):

```text
ReportStructureNode          → decompose query into N paragraphs/sections
  for each paragraph:
    FirstSearchNode          → choose a search tool + query, execute it
    FirstSummaryNode         → summarize results into the paragraph
    repeat MAX_REFLECTIONS×:  (default 3)
      ReflectionNode         → decide what's missing, search again
      ReflectionSummaryNode  → revise the paragraph
ReportFormattingNode         → assemble all paragraphs into a Markdown report
```

> [!NOTE]
> State mutation is explicit: each node exposes `mutate_state(input, state, paragraph_index)` and returns a new `State`. There is no implicit global graph runtime — `agent.py` is the scheduler. This is why the same node names appear in every engine but behave according to that engine's tools.

The engines differ only in `tools/`:

- **InsightEngine** → `MediaCrawlerDB` (5 read-only SQL query tools) + `keyword_optimizer` (a Qwen middleware that expands a query into multiple optimized keywords) + KMeans clustering over `sentence-transformers` embeddings (`paraphrase-multilingual-MiniLM-L12-v2`) to sample representative results + `multilingual_sentiment_analyzer` (auto-scores results, 22 languages).
- **QueryEngine** → domestic/international news + web search.
- **MediaEngine** → multimodal search (video/image/rich cards).

### 2.3 The ForumEngine collaboration mechanism

`ForumEngine/monitor.py` (`LogMonitor`) is a polling loop (1 s tick) that:

1. Tails `logs/insight.log`, `logs/media.log`, `logs/query.log` by tracking byte offsets.
2. Identifies **target lines** — output from `FirstSummaryNode`/`ReflectionSummaryNode` (matched by class name, module path, or the marker text `正在生成首次段落总结`/`正在生成反思总结`). It **excludes** `ERROR`-level lines and search-node output, and handles loguru's multi-line JSON by buffering until the closing brace.
3. Extracts the `updated_paragraph_latest_state` / `paragraph_latest_state` field from each summary's JSON (with a hand-written `fix_json_string` state machine to repair unescaped quotes).
4. Writes the cleaned content to `logs/forum.log` tagged `[INSIGHT]`/`[MEDIA]`/`[QUERY]` under a write lock.
5. Buffers agent speeches; once the buffer reaches `host_speech_threshold = 5`, it calls `ForumHost.generate_host_speech()` (LLM) and writes the result tagged `[HOST]`.

A forum **session** begins when a `FirstSummaryNode` first appears (the monitor clears `forum.log` and starts a new session) and ends on log shrink or after `7200` inactive ticks (~2 hours).

> [!IMPORTANT]
> The "multi-agent debate" is implemented entirely through the filesystem. There is no IPC, queue, or shared memory between agents — only log files and a poller. This is robust (a crashed agent doesn't break the others) but couples the system tightly to exact log formats; see [§14](#14-potential-improvements--tech-debt).

### 2.4 The ReportEngine pipeline

`ReportEngine` converts findings into a finished document through a node pipeline plus an intermediate representation (IR):

```text
template_selection_node  → pick 1 of 6 templates (LLM-assisted)
document_layout_node     → title, TOC, theme design
word_budget_node         → per-chapter length plan + directives
chapter_generation_node  → generate each chapter as validated JSON blocks
       │
       ▼
core/stitcher.py         → assemble chapters into a Document IR (+ anchors/metadata)
ir/schema.py + validator → block/mark schema contract + structural validation
       │
       ▼
renderers/
  html_renderer.py       → interactive HTML (Chart.js, MathJax, html2canvas,
                            jsPDF inlined from renderers/libs/ for offline use)
  pdf_renderer.py        → HTML → PDF via WeasyPrint (+ pdf_layout_optimizer,
                            chart_to_svg, math_to_svg for vector charts)
  markdown_renderer.py   → Markdown export
```

It exposes its own Flask interface (`ReportEngine/flask_interface.py`) with SSE streaming, mounted by `app.py` under `/api/report`.

### 2.5 MindSpider crawler

A standalone CLI subsystem (`MindSpider/main.py`) with two stages:

- **BroadTopicExtraction** — collects hot news from ~13 sources (`get_today_news.py`), runs LLM topic extraction (`topic_extractor.py`, DeepSeek by default), persists to `daily_topics`.
- **DeepSentimentCrawling** — loads keywords (`keyword_manager.py`), drives platform crawls (`platform_crawler.py`) over the vendored **MediaCrawler** submodule using Playwright across 7 platforms.

---

## 3. Component Reference

| Module | Path | Responsibility |
|--------|------|----------------|
| Main app | `app.py` | Flask+SocketIO supervisor; sub-app lifecycle; config editor; control API |
| Global config | `config.py` | Pydantic `Settings` loaded from `.env`; `reload_settings()` |
| Query Agent | `QueryEngine/` | Web/news deep-search agent |
| Media Agent | `MediaEngine/` | Multimodal deep-search agent |
| Insight Agent | `InsightEngine/` | Private-DB deep-search agent (clustering, sentiment) |
| Report Agent | `ReportEngine/` | Report assembly, IR, renderers, SSE Flask interface |
| Forum | `ForumEngine/` | `monitor.py` (LogMonitor) + `llm_host.py` (ForumHost) |
| Crawler | `MindSpider/` | Topic extraction + multi-platform crawling |
| Sentiment models | `SentimentAnalysisModel/` | 5 model families, each train/predict CLI |
| Single-agent apps | `SingleEngineApp/` | Streamlit front-ends for each analysis engine |
| Shared utilities | `utils/` | `forum_reader.py`, `github_issues.py`, `retry_helper.py` |
| Report CLI | `report_engine_only.py` | Headless report build from latest agent outputs |
| Re-render CLIs | `regenerate_latest_{html,md,pdf}.py` | Re-render latest report without regeneration |
| Templates | `templates/index.html` | Flask web UI |
| Tests | `tests/` | ForumEngine + ReportEngine sanitization tests |

---

## 4. Requirements & Dependencies

### 4.1 Runtime requirements

| Requirement | Minimum | Notes |
|-------------|---------|-------|
| Python | 3.9 | 3.11 recommended |
| Database | PostgreSQL or MySQL 5.7+ | PostgreSQL recommended |
| RAM | 2 GB+ | More for local sentiment models |
| Playwright | Chromium | `playwright install chromium` |
| OS | Windows / Linux / macOS | Crawler login is interactive (QR) |

### 4.2 Dependency groups

The root `requirements.txt` is organized by concern. Key pins:

| Group | Notable packages |
|-------|------------------|
| Web framework | `flask==2.3.3`, `flask-socketio==5.3.6`, `streamlit==1.28.1`, `python-socketio==5.8.0`, `eventlet==0.33.3` |
| HTTP / async | `requests==2.31.0`, `httpx==0.28.1`, `aiohttp>=3.8.0`, `aiofiles`, `PySocks` |
| LLM | `openai>=1.3.0` (DeepSeek and others via OpenAI format) |
| Search | `tavily-python>=0.3.0` |
| Data | `pandas>=2.0.0`, `numpy>=1.24.0`, `jieba==0.42.1`, `regex` |
| Databases | `pymysql`, `aiomysql`, `aiosqlite`, `asyncpg`, `psycopg[binary]`, `SQLAlchemy==2.0.35`, `motor`, `redis` |
| Crawler | `playwright==1.45.0`, `beautifulsoup4`, `lxml`, `parsel`, `pyexecjs`, `xhshow`, `opencv-python`, `Pillow==9.5.0` |
| Visualization | `plotly`, `matplotlib==3.9.0`, `wordcloud==1.9.3` |
| PDF | `weasyprint>=60.0` |
| ML | `torch>=2.0.0` (CPU), `transformers>=4.30.0`, `sentence-transformers`, `scikit-learn`, `xgboost` |
| Tooling | `typer`, `python-dotenv`, `tenacity==8.2.2`, `loguru`, `pydantic==2.5.2`, `pydantic-settings==2.2.1`, `json-repair` |
| Server | `fastapi==0.110.2`, `uvicorn==0.29.0` |

> [!WARNING]
> `Pillow` is pinned at `9.5.0` (and `<11` under MindSpider) **for Streamlit 1.28.1 compatibility** (Streamlit requires `pillow<11`). Do not bump Pillow without testing the Streamlit sub-apps.

> [!NOTE]
> **Split requirements.** Each sentiment-model family and MindSpider ships its own `requirements.txt` (e.g. `MindSpider/requirements.txt` adds `asyncmy`, `pyhumps`; `SentimentAnalysisModel/*/requirements.txt` add `peft`, `accelerate`, `bitsandbytes`, `datasets`). The effective dependency set is the union, assembled per use case. MediaCrawler's own deps are auto-installed on first crawl.

### 4.3 GPU (optional)

The local sentiment models default to CPU `torch`. For a GPU build:

```bash
pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu126
```

---

## 5. Configuration Mastery

All configuration is a single Pydantic `Settings` object (`config.py`) loaded from `.env`. Precedence: a `.env` in the current working directory wins, otherwise the project-root `.env`. `model_config` sets `case_sensitive=False` and `extra="allow"`.

### 5.1 Server & database

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| `HOST` | str | `0.0.0.0` | Flask bind address |
| `PORT` | int | `5000` | Web UI port |
| `DB_DIALECT` | str | `postgresql` | `postgresql` or `mysql` |
| `DB_HOST` | str | `your_db_host` | Database host |
| `DB_PORT` | int | `5432` | `5432` (PG) / `3306` (MySQL) |
| `DB_USER` | str | `your_db_user` | Username |
| `DB_PASSWORD` | str | `your_db_password` | Password |
| `DB_NAME` | str | `your_db_name` | Database name |
| `DB_CHARSET` | str | `utf8mb4` | Charset (MySQL); emoji-safe |

### 5.2 LLM slots

Seven slots, each with `*_API_KEY`, `*_BASE_URL`, `*_MODEL_NAME`:

| Slot prefix | Default model | Default base URL |
|-------------|---------------|------------------|
| `INSIGHT_ENGINE_` | `kimi-k2-0711-preview` | `https://api.moonshot.cn/v1` |
| `MEDIA_ENGINE_` | `gemini-2.5-pro` | `https://aihubmix.com/v1` |
| `QUERY_ENGINE_` | `deepseek-chat` | `https://api.deepseek.com` |
| `REPORT_ENGINE_` | `gemini-2.5-pro` | `https://aihubmix.com/v1` |
| `MINDSPIDER_` | (provider default) | (provider default) |
| `FORUM_HOST_` | `qwen-plus` | (provider default) |
| `KEYWORD_OPTIMIZER_` | `qwen-plus` | (provider default) |

> [!IMPORTANT]
> The Report Agent requires a strong model. Under-powered models cause `ChapterJsonParseError` / `ChapterContentError` and the report-regeneration loop (issue #666). Size this slot generously.

### 5.3 Web-search tools

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| `SEARCH_TOOL_TYPE` | enum | `AnspireAPI` | `AnspireAPI` or `BochaAPI` |
| `ANSPIRE_BASE_URL` | str | `https://plugin.anspire.cn/api/ntsearch/search` | Anspire endpoint |
| `ANSPIRE_API_KEY` | str | — | Anspire key |
| `BOCHA_BASE_URL` | str | `https://api.bocha.cn/v1/ai-search` | Bocha endpoint |
| `BOCHA_WEB_SEARCH_API_KEY` | str | — | Bocha key |
| `TAVILY_API_KEY` | str | — | Tavily key (optional) |

### 5.4 Insight-engine search tuning

These tune retrieval volume and reflection depth for the Insight agent:

| Key | Default | Description |
|-----|---------|-------------|
| `DEFAULT_SEARCH_HOT_CONTENT_LIMIT` | 100 | Hot-list items |
| `DEFAULT_SEARCH_TOPIC_GLOBALLY_LIMIT_PER_TABLE` | 50 | Per-table global topic cap |
| `DEFAULT_SEARCH_TOPIC_BY_DATE_LIMIT_PER_TABLE` | 100 | Per-table date-scoped cap |
| `DEFAULT_GET_COMMENTS_FOR_TOPIC_LIMIT` | 500 | Comments per topic |
| `DEFAULT_SEARCH_TOPIC_ON_PLATFORM_LIMIT` | 200 | Platform-scoped topic cap |
| `MAX_SEARCH_RESULTS_FOR_LLM` | 0 | Results passed to LLM (`0` = unlimited) |
| `MAX_HIGH_CONFIDENCE_SENTIMENT_RESULTS` | 0 | High-confidence sentiment cap (`0` = unlimited) |
| `MAX_REFLECTIONS` | 3 | Reflection rounds per paragraph |
| `MAX_PARAGRAPHS` | 6 | Max report paragraphs |
| `SEARCH_TIMEOUT` | 240 | Per-search timeout (s) |
| `MAX_CONTENT_LENGTH` | 500000 | Max content chars per search |

> [!NOTE]
> The agent deliberately **ignores** LLM-supplied `limit` parameters and substitutes these config values — this is a guardrail so a model can't request unbounded retrieval. To change retrieval volume, edit config, not prompts.

### 5.5 Per-engine overrides & request timeouts

Each engine also reads `*_REQUEST_TIMEOUT` (and a shared `LLM_REQUEST_TIMEOUT`) via `os.getenv` in its `llms/base.py`. Legacy per-engine `utils/config.py` knobs (`max_reflections`, `max_search_results`, `comprehensive_search_limit`, etc.) still exist but configuration has been **unified to the root `.env`**; sub-agents inherit it automatically.

### 5.6 Live config editing

`app.py` exposes a `CONFIG_KEYS` allow-list (the DB, LLM, and search keys). `GET /api/config` reads them from the live `Settings`; `POST /api/config` writes them back to `.env` (`write_config_values`) and calls `reload_settings()` — changes apply without a restart.

> [!CAUTION]
> The config editor reads and writes plaintext API keys and DB credentials to `.env` over an unauthenticated endpoint (see [§10](#10-security-analysis)). Never expose port 5000 to an untrusted network.

---

## 6. Data Layer & Schema

### 6.1 Connectivity

- The Insight agent connects via SQLAlchemy async engine (`InsightEngine/utils/db.py`), driven by `DATABASE_URL`/`DB_*`, and performs **read-only** queries (the `MediaCrawlerDB` tool wraps `SELECT`-style operations only).
- MindSpider writes via its own managers (`MindSpider/schema/db_manager.py`, `models_sa.py`, `models_bigdata.py`).

### 6.2 Core tables (MindSpider)

| Table | Purpose |
|-------|---------|
| `daily_news` | Hot news collected per day (title, link, description, rank) |
| `daily_topics` | LLM-extracted topics + keyword lists |
| `topic_news_relation` | Topic↔news links with relevance scores |
| `crawling_tasks` | Per-platform crawl task status/progress |
| Platform tables | `xhs_note`, `douyin_aweme`, `kuaishou_video`, `bilibili_video`, `weibo_note`, `tieba_note`, `zhihu_content` (inherited from MediaCrawler) |

Schema DDL lives in `MindSpider/schema/mindspider_tables.sql`; apply/extend with `python MindSpider/schema/init_database.py`.

### 6.3 Insight agent query tools

The Insight agent's `MediaCrawlerDB` exposes five retrieval tools (the agent's LLM picks one per step):

| Tool | Purpose |
|------|---------|
| `search_hot_content` | Recent hot content (by `time_period`) |
| `search_topic_globally` | Cross-table topic search |
| `search_topic_by_date` | Date-bounded topic search |
| `get_comments_for_topic` | Comment-level retrieval |
| `search_topic_on_platform` | Platform-scoped topic search |

Plus `analyze_sentiment` as a standalone scoring tool. Results pass through keyword optimization, KMeans clustering, deduplication (by URL or content prefix), and optional sentiment scoring before reaching the LLM.

### 6.4 Database inspection CLI

`MindSpider/schema/db_manager.py` offers admin commands:

```bash
python MindSpider/schema/db_manager.py --tables           # list tables
python MindSpider/schema/db_manager.py --stats            # data statistics
python MindSpider/schema/db_manager.py --recent 7         # last N days
python MindSpider/schema/db_manager.py --cleanup 30 --execute   # purge >30d
```

---

## 7. HTTP API Reference

### 7.1 Main app (`app.py`, port 5000)

| Method | Path | Purpose |
|--------|------|---------|
| GET | `/` | Web UI |
| GET | `/api/status` | Sub-app process status |
| GET | `/api/start/<app_name>` | Start a Streamlit sub-app |
| GET | `/api/stop/<app_name>` | Stop a sub-app |
| GET | `/api/output/<app_name>` | Recent sub-app output |
| GET | `/api/test_log/<app_name>` | Test-log endpoint |
| GET | `/api/forum/start` | Start ForumEngine monitoring |
| GET | `/api/forum/stop` | Stop ForumEngine monitoring |
| GET | `/api/forum/log` | Current forum log |
| POST | `/api/forum/log/history` | Forum log history |
| POST | `/api/search` | Submit an analysis query |
| GET | `/api/config` | Read configuration (allow-listed keys) |
| POST | `/api/config` | Write configuration to `.env` + reload |
| GET | `/api/system/status` | Overall system state |
| POST | `/api/system/start` | Start all components |
| POST | `/api/system/shutdown` | Shut down the system |

### 7.2 ReportEngine (`flask_interface.py`, mounted at `/api/report`)

| Method | Path | Purpose |
|--------|------|---------|
| GET | `/status` | Report engine status |
| POST | `/generate` | Start report generation |
| GET | `/progress/<task_id>` | Progress for a task |
| GET | `/stream/<task_id>` | SSE event stream |
| GET | `/result/<task_id>` | Final result (HTML) |
| GET | `/result/<task_id>/json` | Result as JSON/IR |
| GET | `/download/<task_id>` | Download report |
| POST | `/cancel/<task_id>` | Cancel a task |
| GET | `/templates` | List templates |
| GET | `/log` / POST `/log/clear` | Report log read/clear |
| GET | `/export/md/<task_id>` | Export Markdown |
| GET | `/export/pdf/<task_id>` | Export PDF |
| POST | `/export/pdf-from-ir` | Export PDF directly from an IR payload |

> [!NOTE]
> `SECRET_KEY` is hard-coded in `app.py` and Socket.IO sets `cors_allowed_origins="*"`. Both are fine for local single-user use and unsafe for public exposure — see [§10](#10-security-analysis).

---

## 8. Deployment

### 8.1 Docker Compose (recommended)

```bash
docker compose up -d
```

The stack (`docker-compose.yml` + `Dockerfile`) runs the app and a PostgreSQL service (`db`, user/pass/name `bettafish`, port 5432). Alternative registry mirrors are provided as comments in the compose file for slow pulls.

```text
DB_HOST=db  DB_PORT=5432  DB_USER=bettafish  DB_PASSWORD=bettafish  DB_NAME=bettafish  DB_DIALECT=postgresql
```

### 8.2 Source deployment

Deployment checklist:

- [ ] Clone with `--recurse-submodules` (or `git submodule update --init --recursive`)
- [ ] Python 3.11 environment (conda or uv)
- [ ] (Optional) PDF system libraries for WeasyPrint
- [ ] `pip install -r requirements.txt`
- [ ] `playwright install chromium`
- [ ] Populate `.env` (DB + 7 LLM slots + search keys)
- [ ] Verify DB connectivity: `python MindSpider/main.py --status`
- [ ] Log in to each crawl platform once (QR) before unattended crawling
- [ ] `python app.py` → confirm all three sub-apps + ForumEngine start

### 8.3 Port map

| Port | Service |
|------|---------|
| 5000 | Flask main app / web UI |
| 8501 | InsightEngine Streamlit |
| 8502 | MediaEngine Streamlit |
| 8503 | QueryEngine Streamlit |

> [!WARNING]
> After a stopped run, a Streamlit sub-app may keep holding its port. If startup reports a busy port, find and kill the orphaned process before relaunching.

---

## 9. Operations

### 9.1 Start / stop

- **Start everything:** `python app.py` (or `POST /api/system/start`). `initialize_system_components()` initializes the DB, stops any stale ForumEngine, launches each Streamlit sub-app (waiting up to 30 s per app for startup), then starts ForumEngine and ReportEngine.
- **Stop:** `POST /api/system/shutdown`, or terminate `app.py` (an `atexit` handler and the supervisor clean up child processes).

### 9.2 Logs

`loguru` is used throughout. Per-engine logs live in `logs/`:

| File | Producer |
|------|----------|
| `logs/insight.log` | Insight agent |
| `logs/media.log` | Media agent |
| `logs/query.log` | Query agent |
| `logs/forum.log` | ForumEngine (aggregated discussion) |
| `logs/report*.log` | Report engine |

> [!IMPORTANT]
> ForumEngine parses these logs by exact format. It supports both legacy `[HH:MM:SS]` and loguru `YYYY-MM-DD HH:mm:ss.SSS | LEVEL |` formats. If you change the log format/handler, re-validate `tests/test_monitor.py` — a format drift silently breaks forum collaboration.

### 9.3 Routine maintenance

- Purge old crawl data: `python MindSpider/schema/db_manager.py --cleanup <days> --execute`.
- Index high-traffic query columns; consider partitioning large platform tables.
- Use a proxy pool and conservative crawl intervals to avoid rate-limiting/bans.
- Re-render reports without re-running agents via `regenerate_latest_{html,md,pdf}.py`.

### 9.4 Output locations

| Output | Location |
|--------|----------|
| Web/CLI reports | `final_reports/` (+ `pdf/`, `md/`) |
| Report IR | `final_reports/ir/` |
| Per-agent reports | `query_engine_streamlit_reports/`, `media_engine_streamlit_reports/`, `insight_engine_streamlit_reports/` |

---

## 10. Security Analysis

> [!CAUTION]
> BettaFish is designed as a **local, single-operator research tool**. Several defaults are unsafe on an untrusted network. Treat the following as negative-knowledge security facts — they are intentional design points, not bugs, but they constrain how you may deploy it.

### 10.1 No authentication

None of the HTTP endpoints (`app.py` or ReportEngine) require authentication. Anyone who can reach port 5000 can submit queries, start/stop sub-apps, **read and overwrite all API keys and DB credentials** via `/api/config`, and shut the system down. **Do not bind to a public interface.** Keep `HOST` at `127.0.0.1` for anything beyond a trusted LAN, or place the app behind an authenticating reverse proxy + firewall.

### 10.2 Permissive CORS and a hard-coded secret

Socket.IO is configured with `cors_allowed_origins="*"`, and `app.config['SECRET_KEY']` is a hard-coded string committed to the repository. For any non-local deployment, override the secret and restrict CORS origins.

### 10.3 Secrets at rest

All credentials live in plaintext in `.env`. The `/api/config` endpoint reads and rewrites that file. Restrict file permissions on `.env`, keep it out of version control (it is, by `.gitignore`), and never expose the config API.

### 10.4 Browser automation & login state

The crawler drives real logged-in sessions via Playwright and persists session state under `MediaCrawler/browser_data/`. This directory contains authenticated cookies for your social accounts — treat it as a secret and exclude it from backups/images that leave your machine.

### 10.5 LLM output handling

The ReportEngine ingests LLM-generated JSON and renders HTML. The repository includes a dedicated sanitization test (`tests/test_report_engine_sanitization.py`), and the JSON parser (`utils/json_parser.py`) and `json-repair` defensively repair malformed model output. When extending renderers, preserve sanitization — rendered reports may be shared.

### 10.6 Legal/compliance posture

The crawler scrapes third-party platforms. The project is explicitly **research/education only**, GPL-2.0, and forbids commercial use and large-scale scraping. Operators are responsible for compliance with each platform's `robots.txt` and terms, and with local law.

---

## 11. Code Patterns & Conventions

- **Pydantic settings singleton.** One global `settings` object; `reload_settings()` rebuilds it after `.env` edits. New config is a typed `Field` on `Settings`.
- **Explicit node/state machine.** Agents schedule nodes by hand (`agent.py`); nodes mutate an explicit `State`. No graph-runtime magic — easy to trace, easy to extend.
- **OpenAI-compatible LLM clients.** Every engine's `llms/base.py` validates `api_key`/`model_name` and raises a localized error if absent (`... API key is required.`). Adding a provider = setting a `base_url`.
- **Graceful retry.** `utils/retry_helper.py` provides `with_graceful_retry(CONFIG, default_return=...)` decorators (e.g. `SEARCH_API_RETRY_CONFIG`) that return safe defaults on total failure instead of crashing a run.
- **Defensive JSON.** Both ForumEngine (`fix_json_string`) and ReportEngine (`json_parser.py` + `json-repair`) repair malformed LLM JSON; typed exceptions (`ChapterJsonParseError`, `StageOutputFormatError`, `JSONParseError`) carry stage context.
- **Config-over-prompt guardrails.** Retrieval limits are forced from config, ignoring model-supplied values.
- **Localization.** All log messages, prompts, and error strings are Chinese; identifiers and structure are English.
- **Concurrency.** Thread locks guard shared state (`write_lock` for `forum.log`, `system_state_lock` for lifecycle flags); the eventlet WSGI `finish` is monkey-patched to swallow benign client-disconnect exceptions.

---

## 12. Testing

Tests live in `tests/` and run with pytest:

```bash
pytest tests/test_monitor.py -v
python tests/run_tests.py
```

| Test | Covers |
|------|--------|
| `tests/test_monitor.py` | ForumEngine log parsing: target-line detection, JSON start/end detection, single- and multi-line JSON extraction, content formatting, value filtering |
| `tests/test_report_engine_sanitization.py` | ReportEngine output sanitization |
| `tests/forum_log_test_data.py` | Fixtures for old `[HH:MM:SS]` and new loguru log formats |

> [!IMPORTANT]
> `tests/README.md` documents a known gap: the ForumEngine timestamp regexes were written for the legacy `[HH:MM:SS]` format and may not fully handle loguru's `YYYY-MM-DD HH:mm:ss.SSS` format. These tests exist specifically to surface and guide fixes for that mismatch. Run them after any logging change.

---

## 13. Extending the System

### 13.1 Add an LLM provider

Set the slot's `*_BASE_URL` to any OpenAI-compatible endpoint and `*_MODEL_NAME` to a model it serves. No code change — the `openai` client handles the rest.

### 13.2 Integrate a custom business database (Insight agent)

1. Add `BUSINESS_DB_*` settings to `config.py`.
2. Create a tool class (e.g. `InsightEngine/tools/custom_db_tool.py`) wrapping your queries.
3. Wire it into `InsightEngine/agent.py` and expose it as a selectable search tool.

This is the documented path to "external trends + internal insights" analysis.

### 13.3 Add a crawl platform

Under `MindSpider/DeepSentimentCrawling/MediaCrawler/media_platform/<platform>/`, implement `client.py`, `core.py`, `login.py`, `field.py`, then register the platform code.

### 13.4 Add a news source (topic extraction)

Add an async collector method in `MindSpider/BroadTopicExtraction/get_today_news.py`.

### 13.5 Add a report template

Drop a `.md`/`.txt` file into `ReportEngine/report_template/` (or upload via the web UI). The template-selection node will consider it. Six ship by default: brand-reputation, market-competition, routine-monitoring, policy/industry, social-hot-event, and crisis-PR.

### 13.6 Repurpose the whole system

Because tools are OpenAI-compatible and behavior is prompt-driven, swapping the agent prompts (`*/prompts/prompts.py`) and the Insight DB tools retargets BettaFish to a different analytical domain (e.g. financial markets) with no orchestration changes.

---

## 14. Potential Improvements & Tech Debt

- **Log-format coupling.** Agent collaboration depends on parsing exact log strings. A logging change can silently break the forum without raising an error. Consider a structured side-channel (a JSONL events file or an in-process queue) decoupled from human-readable logs. The known loguru-vs-legacy regex gap (`tests/README.md`) is a concrete instance.
- **No authentication / open config API.** The biggest deployment risk. An auth layer (even a single shared token) on `app.py` would make non-localhost use viable. CORS `*` and the hard-coded `SECRET_KEY` should be configurable.
- **Hard-coded clustering knobs.** `ENABLE_CLUSTERING`, `MAX_CLUSTERED_RESULTS`, `RESULTS_PER_CLUSTER` are module constants in `InsightEngine/agent.py`; promote them to `Settings`.
- **Report regeneration loop.** Weak Report-Agent models can loop on token limits (#666). A hard cap on chapter-retry attempts with a graceful degradation path would bound failure (`ChapterJsonParseError` after `chapter_max_attempts` already exists; surfacing it to the UI sooner would help).
- **Dual legacy config.** Per-engine `utils/config.py` knobs coexist with the unified root config. Removing the legacy path would cut confusion.
- **Insight dedup precision.** Issue #688 notes deduplication can drop distinct URL-less results (it keys on a 100-char content prefix). A stronger identity (hash + platform + author) would reduce false collisions.
- **Subprocess sub-apps.** Running Streamlit apps as child processes complicates lifecycle (orphaned ports) and observability. A consolidated runtime or container-per-engine would be cleaner.

---

## 15. Quick Reference

### Entry points

```bash
python app.py                       # full system (web UI :5000)
docker compose up -d                # app + PostgreSQL
python report_engine_only.py        # headless report from latest agent outputs
python MindSpider/main.py --complete --test   # crawl pipeline (test volume)
streamlit run SingleEngineApp/insight_engine_streamlit_app.py --server.port 8501
```

### Key paths

| Path | What |
|------|------|
| `config.py` / `.env` | Global configuration |
| `app.py` | Supervisor + control API |
| `<Engine>/agent.py` | Agent orchestrator |
| `ForumEngine/monitor.py` | Forum collaboration |
| `ReportEngine/flask_interface.py` | Report SSE API |
| `MindSpider/main.py` | Crawler CLI |
| `MindSpider/schema/mindspider_tables.sql` | DB schema |
| `logs/` | Per-engine + forum logs |
| `final_reports/` | Output reports |

### Config slot cheat-sheet

```text
DB:      DB_DIALECT DB_HOST DB_PORT DB_USER DB_PASSWORD DB_NAME DB_CHARSET
LLM:     {INSIGHT,MEDIA,QUERY,REPORT,MINDSPIDER}_ENGINE_* + FORUM_HOST_* + KEYWORD_OPTIMIZER_*
         (each: _API_KEY _BASE_URL _MODEL_NAME)
Search:  SEARCH_TOOL_TYPE  ANSPIRE_*  BOCHA_*  TAVILY_API_KEY
Server:  HOST PORT
```

### Platform codes

`xhs` Xiaohongshu · `dy` Douyin · `ks` Kuaishou · `bili` Bilibili · `wb` Weibo · `tieba` Tieba · `zhihu` Zhihu

---

_Generated June 2026 from 666ghj/BettaFish at 40327d7. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._