BettaFish Admin & Architecture Guide

This guide is for the engineers, architects, and operators who deploy, extend, and maintain BettaFish. It documents the system's architecture, configuration surface, security posture, operational procedures, and internals in enough depth to run it reliably or rebuild it from scratch.

For installation and day-to-day usage, see the companion User Guide.

Table of Contents

1. Executive Summary
2. System Architecture
3. Component Reference
4. Requirements & Dependencies
5. Configuration Mastery
6. Data Layer & Schema
7. HTTP API Reference
8. Deployment
9. Operations
10. Security Analysis
11. Code Patterns & Conventions
12. Testing
13. Extending the System
14. Potential Improvements & Tech Debt
15. Quick Reference

1. Executive Summary

BettaFish is a multi-agent public-opinion analysis platform written in pure Python. It orchestrates four LLM-driven "deep search" agents, a moderator LLM, an AI crawler, and a collection of local sentiment models to turn a natural-language question into a chart-rich analysis report.

Architectural character:

• Modular monolith. A single Flask + Socket.IO application (app.py) supervises everything: it launches three Streamlit sub-applications as child processes, runs the ForumEngine in a background thread, and mounts the ReportEngine as a Flask blueprint. There is no microservice mesh and no message broker.
• Hand-rolled agent framework. The agents do not use LangChain/LangGraph. Each engine implements its own node-graph (nodes/, state/, llms/, prompts/, tools/) — the __init__ describes it as a "framework-less deep search AI agent." All four analysis engines share the same node structure and differ only in their tools and data source.
• Collaboration via shared log files. The agents do not call each other. Each writes to its own log; the ForumEngine tails those logs, extracts summaries, and republishes them to a shared forum.log, where a moderator LLM periodically posts synthesis. This file-as-message-bus design is the system's defining (and most unusual) pattern.
• Provider-agnostic. Every LLM interaction uses the OpenAI request format. Each of seven logical "slots" can point at a different provider via api_key / base_url / model.
• Configuration as a single Pydantic settings object loaded from one root .env.

2. System Architecture

2.1 Process & component topology

                         ┌─────────────────────────────────────────────┐
                         │   app.py  — Flask + Socket.IO  (port 5000)   │
                         │   • supervises sub-apps  • /api/* control     │
                         │   • /api/report/* (ReportEngine blueprint)    │
                         │   • config editor → writes root .env          │
                         └───────┬───────────────┬──────────────┬───────┘
            launches as          │               │              │  mounts
            subprocesses ┌───────┴──────┐ ┌──────┴───────┐ ┌────┴────────┐
                         │ QueryEngine  │ │ MediaEngine  │ │ InsightEngine│
                         │ Streamlit    │ │ Streamlit    │ │ Streamlit    │
                         │ :8503        │ │ :8502        │ │ :8501        │
                         └──────┬───────┘ └──────┬───────┘ └──────┬───────┘
        writes logs/query.log   │  logs/media.log│ logs/insight.log│
                                ▼                ▼                 ▼
                         ┌──────────────────────────────────────────────┐
                         │  ForumEngine (background thread in app.py)    │
                         │  • tails the 3 engine logs                    │
                         │  • extracts SummaryNode JSON                  │
                         │  • writes logs/forum.log  ([INSIGHT]/[MEDIA]/ │
                         │    [QUERY]/[HOST]/[SYSTEM])                    │
                         │  • every 5 speeches → ForumHost LLM ([HOST])  │
                         └───────────────────────┬──────────────────────┘
                  agents read forum.log via      │
                  utils/forum_reader.py ◄─────────┘
                                ▼
                         ┌──────────────────────────────────────────────┐
                         │  ReportEngine (blueprint + flask_interface)   │
                         │  template → layout → budget → chapters → IR   │
                         │  → renderers (HTML / PDF / Markdown)          │
                         └──────────────────────────────────────────────┘

   MindSpider (standalone CLI) ──► Database ◄── InsightEngine queries (read-only)
   BroadTopicExtraction → daily_topics ; DeepSentimentCrawling → platform tables

2.2 The deep-search agent node graph

All four analysis engines (QueryEngine, MediaEngine, InsightEngine, and structurally ReportEngine) implement the same package layout:

<Engine>/
├── agent.py        # orchestrator — drives the node sequence
├── llms/base.py    # OpenAI-compatible LLM client (per-engine key/url/model)
├── nodes/          # processing nodes (the "graph")
│   ├── base_node.py
│   ├── search_node.py
│   ├── summary_node.py
│   ├── report_structure_node.py
│   └── formatting_node.py
├── state/state.py  # mutable run state (paragraphs, search history, summaries)
├── prompts/prompts.py
├── tools/          # the data-source differentiator
└── utils/          # config.py, db.py (Insight only), text_processing.py

The execution sequence (from InsightEngine/agent.py::research):

ReportStructureNode          → decompose query into N paragraphs/sections
  for each paragraph:
    FirstSearchNode          → choose a search tool + query, execute it
    FirstSummaryNode         → summarize results into the paragraph
    repeat MAX_REFLECTIONS×:  (default 3)
      ReflectionNode         → decide what's missing, search again
      ReflectionSummaryNode  → revise the paragraph
ReportFormattingNode         → assemble all paragraphs into a Markdown report

Note

State mutation is explicit: each node exposes mutate_state(input, state, paragraph_index) and returns a new State. There is no implicit global graph runtime — agent.py is the scheduler. This is why the same node names appear in every engine but behave according to that engine's tools.

The engines differ only in tools/:

• InsightEngine → MediaCrawlerDB (5 read-only SQL query tools) + keyword_optimizer (a Qwen middleware that expands a query into multiple optimized keywords) + KMeans clustering over sentence-transformers embeddings (paraphrase-multilingual-MiniLM-L12-v2) to sample representative results + multilingual_sentiment_analyzer (auto-scores results, 22 languages).
• QueryEngine → domestic/international news + web search.
• MediaEngine → multimodal search (video/image/rich cards).

2.3 The ForumEngine collaboration mechanism

ForumEngine/monitor.py (LogMonitor) is a polling loop (1 s tick) that:

1. Tails logs/insight.log, logs/media.log, logs/query.log by tracking byte offsets.
2. Identifies target lines — output from FirstSummaryNode/ReflectionSummaryNode (matched by class name, module path, or the marker text 正在生成首次段落总结/正在生成反思总结). It excludes ERROR-level lines and search-node output, and handles loguru's multi-line JSON by buffering until the closing brace.
3. Extracts the updated_paragraph_latest_state / paragraph_latest_state field from each summary's JSON (with a hand-written fix_json_string state machine to repair unescaped quotes).
4. Writes the cleaned content to logs/forum.log tagged [INSIGHT]/[MEDIA]/[QUERY] under a write lock.
5. Buffers agent speeches; once the buffer reaches host_speech_threshold = 5, it calls ForumHost.generate_host_speech() (LLM) and writes the result tagged [HOST].

A forum session begins when a FirstSummaryNode first appears (the monitor clears forum.log and starts a new session) and ends on log shrink or after 7200 inactive ticks (~2 hours).

Important

The "multi-agent debate" is implemented entirely through the filesystem. There is no IPC, queue, or shared memory between agents — only log files and a poller. This is robust (a crashed agent doesn't break the others) but couples the system tightly to exact log formats; see §14.

2.4 The ReportEngine pipeline

ReportEngine converts findings into a finished document through a node pipeline plus an intermediate representation (IR):

template_selection_node  → pick 1 of 6 templates (LLM-assisted)
document_layout_node     → title, TOC, theme design
word_budget_node         → per-chapter length plan + directives
chapter_generation_node  → generate each chapter as validated JSON blocks
       │
       ▼
core/stitcher.py         → assemble chapters into a Document IR (+ anchors/metadata)
ir/schema.py + validator → block/mark schema contract + structural validation
       │
       ▼
renderers/
  html_renderer.py       → interactive HTML (Chart.js, MathJax, html2canvas,
                            jsPDF inlined from renderers/libs/ for offline use)
  pdf_renderer.py        → HTML → PDF via WeasyPrint (+ pdf_layout_optimizer,
                            chart_to_svg, math_to_svg for vector charts)
  markdown_renderer.py   → Markdown export

It exposes its own Flask interface (ReportEngine/flask_interface.py) with SSE streaming, mounted by app.py under /api/report.

2.5 MindSpider crawler

A standalone CLI subsystem (MindSpider/main.py) with two stages:

• BroadTopicExtraction — collects hot news from ~13 sources (get_today_news.py), runs LLM topic extraction (topic_extractor.py, DeepSeek by default), persists to daily_topics.
• DeepSentimentCrawling — loads keywords (keyword_manager.py), drives platform crawls (platform_crawler.py) over the vendored MediaCrawler submodule using Playwright across 7 platforms.

3. Component Reference

4. Requirements & Dependencies

4.1 Runtime requirements

4.2 Dependency groups

The root requirements.txt is organized by concern. Key pins:

Warning

Pillow is pinned at 9.5.0 (and <11 under MindSpider) for Streamlit 1.28.1 compatibility (Streamlit requires pillow<11). Do not bump Pillow without testing the Streamlit sub-apps.

Note

Split requirements. Each sentiment-model family and MindSpider ships its own requirements.txt (e.g. MindSpider/requirements.txt adds asyncmy, pyhumps; SentimentAnalysisModel/*/requirements.txt add peft, accelerate, bitsandbytes, datasets). The effective dependency set is the union, assembled per use case. MediaCrawler's own deps are auto-installed on first crawl.

4.3 GPU (optional)

The local sentiment models default to CPU torch. For a GPU build:

pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu126

5. Configuration Mastery

All configuration is a single Pydantic Settings object (config.py) loaded from .env. Precedence: a .env in the current working directory wins, otherwise the project-root .env. model_config sets case_sensitive=False and extra="allow".

5.1 Server & database

5.2 LLM slots

Seven slots, each with *_API_KEY, *_BASE_URL, *_MODEL_NAME:

Important

The Report Agent requires a strong model. Under-powered models cause ChapterJsonParseError / ChapterContentError and the report-regeneration loop (issue #666). Size this slot generously.

5.3 Web-search tools

5.4 Insight-engine search tuning

These tune retrieval volume and reflection depth for the Insight agent:

Note

The agent deliberately ignores LLM-supplied limit parameters and substitutes these config values — this is a guardrail so a model can't request unbounded retrieval. To change retrieval volume, edit config, not prompts.

5.5 Per-engine overrides & request timeouts

Each engine also reads *_REQUEST_TIMEOUT (and a shared LLM_REQUEST_TIMEOUT) via os.getenv in its llms/base.py. Legacy per-engine utils/config.py knobs (max_reflections, max_search_results, comprehensive_search_limit, etc.) still exist but configuration has been unified to the root .env; sub-agents inherit it automatically.

5.6 Live config editing

app.py exposes a CONFIG_KEYS allow-list (the DB, LLM, and search keys). GET /api/config reads them from the live Settings; POST /api/config writes them back to .env (write_config_values) and calls reload_settings() — changes apply without a restart.

Caution

The config editor reads and writes plaintext API keys and DB credentials to .env over an unauthenticated endpoint (see §10). Never expose port 5000 to an untrusted network.

6. Data Layer & Schema

6.1 Connectivity

• The Insight agent connects via SQLAlchemy async engine (InsightEngine/utils/db.py), driven by DATABASE_URL/DB_*, and performs read-only queries (the MediaCrawlerDB tool wraps SELECT-style operations only).
• MindSpider writes via its own managers (MindSpider/schema/db_manager.py, models_sa.py, models_bigdata.py).

6.2 Core tables (MindSpider)

Schema DDL lives in MindSpider/schema/mindspider_tables.sql; apply/extend with python MindSpider/schema/init_database.py.

6.3 Insight agent query tools

The Insight agent's MediaCrawlerDB exposes five retrieval tools (the agent's LLM picks one per step):

Plus analyze_sentiment as a standalone scoring tool. Results pass through keyword optimization, KMeans clustering, deduplication (by URL or content prefix), and optional sentiment scoring before reaching the LLM.

6.4 Database inspection CLI

MindSpider/schema/db_manager.py offers admin commands:

python MindSpider/schema/db_manager.py --tables           # list tables
python MindSpider/schema/db_manager.py --stats            # data statistics
python MindSpider/schema/db_manager.py --recent 7         # last N days
python MindSpider/schema/db_manager.py --cleanup 30 --execute   # purge >30d

7. HTTP API Reference

7.1 Main app (app.py, port 5000)

7.2 ReportEngine (flask_interface.py, mounted at /api/report)

Note

SECRET_KEY is hard-coded in app.py and Socket.IO sets cors_allowed_origins="*". Both are fine for local single-user use and unsafe for public exposure — see §10.

8. Deployment

8.1 Docker Compose (recommended)

docker compose up -d

The stack (docker-compose.yml + Dockerfile) runs the app and a PostgreSQL service (db, user/pass/name bettafish, port 5432). Alternative registry mirrors are provided as comments in the compose file for slow pulls.

DB_HOST=db  DB_PORT=5432  DB_USER=bettafish  DB_PASSWORD=bettafish  DB_NAME=bettafish  DB_DIALECT=postgresql

8.2 Source deployment

Deployment checklist:

• [ ] Clone with --recurse-submodules (or git submodule update --init --recursive)
• [ ] Python 3.11 environment (conda or uv)
• [ ] (Optional) PDF system libraries for WeasyPrint
• [ ] pip install -r requirements.txt
• [ ] playwright install chromium
• [ ] Populate .env (DB + 7 LLM slots + search keys)
• [ ] Verify DB connectivity: python MindSpider/main.py --status
• [ ] Log in to each crawl platform once (QR) before unattended crawling
• [ ] python app.py → confirm all three sub-apps + ForumEngine start

8.3 Port map

Warning

After a stopped run, a Streamlit sub-app may keep holding its port. If startup reports a busy port, find and kill the orphaned process before relaunching.

9. Operations

9.1 Start / stop

• Start everything: python app.py (or POST /api/system/start). initialize_system_components() initializes the DB, stops any stale ForumEngine, launches each Streamlit sub-app (waiting up to 30 s per app for startup), then starts ForumEngine and ReportEngine.
• Stop: POST /api/system/shutdown, or terminate app.py (an atexit handler and the supervisor clean up child processes).

9.2 Logs

loguru is used throughout. Per-engine logs live in logs/:

Important

ForumEngine parses these logs by exact format. It supports both legacy [HH:MM:SS] and loguru YYYY-MM-DD HH:mm:ss.SSS | LEVEL | formats. If you change the log format/handler, re-validate tests/test_monitor.py — a format drift silently breaks forum collaboration.

9.3 Routine maintenance

• Purge old crawl data: python MindSpider/schema/db_manager.py --cleanup <days> --execute.
• Index high-traffic query columns; consider partitioning large platform tables.
• Use a proxy pool and conservative crawl intervals to avoid rate-limiting/bans.
• Re-render reports without re-running agents via regenerate_latest_{html,md,pdf}.py.

9.4 Output locations

10. Security Analysis

Caution

BettaFish is designed as a local, single-operator research tool. Several defaults are unsafe on an untrusted network. Treat the following as negative-knowledge security facts — they are intentional design points, not bugs, but they constrain how you may deploy it.

10.1 No authentication

None of the HTTP endpoints (app.py or ReportEngine) require authentication. Anyone who can reach port 5000 can submit queries, start/stop sub-apps, read and overwrite all API keys and DB credentials via /api/config, and shut the system down. Do not bind to a public interface. Keep HOST at 127.0.0.1 for anything beyond a trusted LAN, or place the app behind an authenticating reverse proxy + firewall.

10.2 Permissive CORS and a hard-coded secret

Socket.IO is configured with cors_allowed_origins="*", and app.config['SECRET_KEY'] is a hard-coded string committed to the repository. For any non-local deployment, override the secret and restrict CORS origins.

10.3 Secrets at rest

All credentials live in plaintext in .env. The /api/config endpoint reads and rewrites that file. Restrict file permissions on .env, keep it out of version control (it is, by .gitignore), and never expose the config API.

10.4 Browser automation & login state

The crawler drives real logged-in sessions via Playwright and persists session state under MediaCrawler/browser_data/. This directory contains authenticated cookies for your social accounts — treat it as a secret and exclude it from backups/images that leave your machine.

10.5 LLM output handling

The ReportEngine ingests LLM-generated JSON and renders HTML. The repository includes a dedicated sanitization test (tests/test_report_engine_sanitization.py), and the JSON parser (utils/json_parser.py) and json-repair defensively repair malformed model output. When extending renderers, preserve sanitization — rendered reports may be shared.

10.6 Legal/compliance posture

The crawler scrapes third-party platforms. The project is explicitly research/education only, GPL-2.0, and forbids commercial use and large-scale scraping. Operators are responsible for compliance with each platform's robots.txt and terms, and with local law.

11. Code Patterns & Conventions

• Pydantic settings singleton. One global settings object; reload_settings() rebuilds it after .env edits. New config is a typed Field on Settings.
• Explicit node/state machine. Agents schedule nodes by hand (agent.py); nodes mutate an explicit State. No graph-runtime magic — easy to trace, easy to extend.
• OpenAI-compatible LLM clients. Every engine's llms/base.py validates api_key/model_name and raises a localized error if absent (... API key is required.). Adding a provider = setting a base_url.
• Graceful retry. utils/retry_helper.py provides with_graceful_retry(CONFIG, default_return=...) decorators (e.g. SEARCH_API_RETRY_CONFIG) that return safe defaults on total failure instead of crashing a run.
• Defensive JSON. Both ForumEngine (fix_json_string) and ReportEngine (json_parser.py + json-repair) repair malformed LLM JSON; typed exceptions (ChapterJsonParseError, StageOutputFormatError, JSONParseError) carry stage context.
• Config-over-prompt guardrails. Retrieval limits are forced from config, ignoring model-supplied values.
• Localization. All log messages, prompts, and error strings are Chinese; identifiers and structure are English.
• Concurrency. Thread locks guard shared state (write_lock for forum.log, system_state_lock for lifecycle flags); the eventlet WSGI finish is monkey-patched to swallow benign client-disconnect exceptions.

12. Testing

Tests live in tests/ and run with pytest:

pytest tests/test_monitor.py -v
python tests/run_tests.py

Important

tests/README.md documents a known gap: the ForumEngine timestamp regexes were written for the legacy [HH:MM:SS] format and may not fully handle loguru's YYYY-MM-DD HH:mm:ss.SSS format. These tests exist specifically to surface and guide fixes for that mismatch. Run them after any logging change.

13. Extending the System

13.1 Add an LLM provider

Set the slot's *_BASE_URL to any OpenAI-compatible endpoint and *_MODEL_NAME to a model it serves. No code change — the openai client handles the rest.

13.2 Integrate a custom business database (Insight agent)

1. Add BUSINESS_DB_* settings to config.py.
2. Create a tool class (e.g. InsightEngine/tools/custom_db_tool.py) wrapping your queries.
3. Wire it into InsightEngine/agent.py and expose it as a selectable search tool.

This is the documented path to "external trends + internal insights" analysis.

13.3 Add a crawl platform

Under MindSpider/DeepSentimentCrawling/MediaCrawler/media_platform/<platform>/, implement client.py, core.py, login.py, field.py, then register the platform code.

13.4 Add a news source (topic extraction)

Add an async collector method in MindSpider/BroadTopicExtraction/get_today_news.py.

13.5 Add a report template

Drop a .md/.txt file into ReportEngine/report_template/ (or upload via the web UI). The template-selection node will consider it. Six ship by default: brand-reputation, market-competition, routine-monitoring, policy/industry, social-hot-event, and crisis-PR.

13.6 Repurpose the whole system

Because tools are OpenAI-compatible and behavior is prompt-driven, swapping the agent prompts (*/prompts/prompts.py) and the Insight DB tools retargets BettaFish to a different analytical domain (e.g. financial markets) with no orchestration changes.

14. Potential Improvements & Tech Debt

• Log-format coupling. Agent collaboration depends on parsing exact log strings. A logging change can silently break the forum without raising an error. Consider a structured side-channel (a JSONL events file or an in-process queue) decoupled from human-readable logs. The known loguru-vs-legacy regex gap (tests/README.md) is a concrete instance.
• No authentication / open config API. The biggest deployment risk. An auth layer (even a single shared token) on app.py would make non-localhost use viable. CORS * and the hard-coded SECRET_KEY should be configurable.
• Hard-coded clustering knobs. ENABLE_CLUSTERING, MAX_CLUSTERED_RESULTS, RESULTS_PER_CLUSTER are module constants in InsightEngine/agent.py; promote them to Settings.
• Report regeneration loop. Weak Report-Agent models can loop on token limits (#666). A hard cap on chapter-retry attempts with a graceful degradation path would bound failure (ChapterJsonParseError after chapter_max_attempts already exists; surfacing it to the UI sooner would help).
• Dual legacy config. Per-engine utils/config.py knobs coexist with the unified root config. Removing the legacy path would cut confusion.
• Insight dedup precision. Issue #688 notes deduplication can drop distinct URL-less results (it keys on a 100-char content prefix). A stronger identity (hash + platform + author) would reduce false collisions.
• Subprocess sub-apps. Running Streamlit apps as child processes complicates lifecycle (orphaned ports) and observability. A consolidated runtime or container-per-engine would be cleaner.

15. Quick Reference

Entry points

python app.py                       # full system (web UI :5000)
docker compose up -d                # app + PostgreSQL
python report_engine_only.py        # headless report from latest agent outputs
python MindSpider/main.py --complete --test   # crawl pipeline (test volume)
streamlit run SingleEngineApp/insight_engine_streamlit_app.py --server.port 8501

Key paths

Config slot cheat-sheet

DB:      DB_DIALECT DB_HOST DB_PORT DB_USER DB_PASSWORD DB_NAME DB_CHARSET
LLM:     {INSIGHT,MEDIA,QUERY,REPORT,MINDSPIDER}_ENGINE_* + FORUM_HOST_* + KEYWORD_OPTIMIZER_*
         (each: _API_KEY _BASE_URL _MODEL_NAME)
Search:  SEARCH_TOOL_TYPE  ANSPIRE_*  BOCHA_*  TAVILY_API_KEY
Server:  HOST PORT

Platform codes

xhs Xiaohongshu · dy Douyin · ks Kuaishou · bili Bilibili · wb Weibo · tieba Tieba · zhihu Zhihu