n8n-workflows — Administrator & Developer Guide

A self-hosted search engine and documentation platform for a large curated collection of n8n workflow .json files. At its core it is a Python FastAPI service backed by a SQLite FTS5 index, with a parallel Node.js (Express) implementation of the same API, a static single-page front end, and container/Kubernetes deployment assets. This guide covers architecture, configuration, security, operations, and internals in enough depth to deploy, harden, extend, or reimplement the system.

Tip

For end-user search syntax, the web UI, and everyday workflow discovery, see the companion User Guide. This document assumes you operate or develop the platform.

Important

Scope note — what is and isn't the product. The shipping, wired-together application is the core search server (run.py → api_server.py → workflow_db.py) plus its static front end, with a Node.js equivalent under src/. The repository also contains several standalone, experimental feature modules (src/ai_assistant.py, src/analytics_engine.py, src/enhanced_api.py, src/integration_hub.py, src/performance_monitor.py, src/user_management.py, src/community_features.py) that each build their own FastAPI app and are not mounted into the main server. They are documented in Section 9 and clearly labeled as optional/standalone. The repository additionally bundles an unrelated subproject, medcards-ai/ (a Next.js application), which is not part of this platform and is out of scope here.

1. Executive Summary

Design goals reflected in the code:

• Speed — SQLite FTS5 + targeted indexes + WAL/PRAGMA tuning for sub-100 ms search.
• Self-contained — no external database, message broker, or cloud dependency.
• Incremental indexing — per-file MD5 hashing skips unchanged files on re-index.
• Defense at the file boundary — strict filename validation guards the only file-serving endpoints against path traversal.

2. Architecture Overview

2.1 System Topology

                         Browser / API client
                                  │
                                  ▼
        ┌──────────────────────────────────────────────────┐
        │  Core Server (choose ONE stack)                    │
        │                                                    │
        │   Python: run.py ──► api_server.py (FastAPI)       │
        │                          │   Uvicorn/Gunicorn      │
        │                          ▼                         │
        │                   workflow_db.py                   │
        │                  (WorkflowDatabase)                │
        │                                                    │
        │   Node:   src/server.js (Express)                  │
        │                          │                         │
        │                          ▼                         │
        │                   src/database.js                  │
        └──────────────────────────┬─────────────────────────┘
                                   │
                ┌──────────────────┼───────────────────┐
                ▼                  ▼                   ▼
        database/workflows.db   workflows/**.json   static/ (+ docs/ mirror)
        (SQLite, FTS5, WAL)     (source of truth)   (front end assets)
                                                          ▲
                                            context/*.json (category maps)

The workflow .json files in workflows/ are the source of truth. The database is a derived, rebuildable index — deleting it and re-indexing reproduces all state.

2.2 Request flow (core search)

1. Client calls GET /api/workflows?q=...&trigger=....
2. api_server.py parses/validates query params via Pydantic-typed FastAPI signatures.
3. It delegates to WorkflowDatabase.search_workflows() in workflow_db.py.
4. If a query string is present, the DB runs an FTS5 MATCH against workflows_fts, joined to the workflows table, ordered by FTS rank; otherwise it runs a filtered table scan ordered by analyzed_at.
5. Results are validated into WorkflowSummary Pydantic models and returned as a paginated SearchResponse.

2.3 Component inventory

3. Requirements

3.1 Runtime

3.2 Python dependencies (requirements.txt)

Note

Several of these packages (PyJWT, passlib, psutil, email-validator, python-multipart) are required only by the standalone feature modules in Section 9, not by the core search server. The core server needs only fastapi, uvicorn, pydantic, and the Python standard library (sqlite3).

4. Configuration Reference

All configuration is via environment variables and command-line flags. There is no config file for the core server.

4.1 Environment variables

The Docker Compose overlays set additional process-level variables (ENVIRONMENT, LOG_LEVEL, DEBUG, RELOAD, ENABLE_METRICS, MAX_WORKERS) consumed by the container entrypoint/compose rather than the Python code directly.

Caution

ADMIN_TOKEN is checked with a plain string comparison and passed as a query parameter (?admin_token=...) to /api/reindex. Query parameters are commonly logged by proxies and servers. If you enable reindexing, terminate TLS in front of the service, restrict the endpoint at the proxy, and rotate the token if it may have been logged.

4.2 Command-line flags

run.py (primary launcher):

api_server.py (run directly, bypassing run.py):

workflow_db.py (DB CLI): --index, --force, --search "TEXT", --stats.

src/index-workflows.js (Node indexer): -f/--force, --stats.

src/server.js (Node server): -p/--port, -h/--host, --dev.

5. Database & Indexing Internals

5.1 Schema

The database (workflow_db.py → init_database()) creates one base table, one FTS5 virtual table, five indexes, and three sync triggers.

workflows table:

workflows_fts — FTS5 virtual table over filename, name, description, integrations, tags, configured as an external-content table (content=workflows, content_rowid=id). Three triggers (workflows_ai/workflows_ad/workflows_au) keep it in sync on insert/delete/update.

Indexes: idx_trigger_type, idx_complexity, idx_active, idx_node_count, idx_filename.

Performance PRAGMAs set at init: journal_mode=WAL, synchronous=NORMAL, cache_size=10000, temp_store=MEMORY.

5.2 Indexing pipeline (index_all_workflows)

1. Recursively glob workflows/**/*.json.
2. For each file: unless force_reindex, compute MD5 and compare to the stored file_hash; skip if unchanged.
3. analyze_workflow_file() parses JSON and extracts: node count → complexity bucket; trigger type and integrations from node types (analyze_nodes()); a name (JSON name, or format_workflow_name() from the filename); and a description (JSON description, or generate_description()).
4. INSERT OR REPLACE into workflows; triggers update the FTS index.
5. Returns {processed, skipped, errors}.

Tip

Indexing is idempotent and incremental. Re-running after adding a handful of files reprocesses only those files. Use --force to rebuild after changing the analysis logic itself (e.g. the service-mapping table), since unchanged files would otherwise be skipped.

5.3 Integration & trigger detection

analyze_nodes() maps node type strings to friendly service names via a large service_mappings dictionary (e.g. n8n-nodes-base.slack → "Slack", @n8n/... namespaced nodes handled too). Utility nodes (set, if, merge, code, stickyNote, wait, filter, etc.) map to None and are excluded from the integration list. Trigger type is inferred from node types: a webhook node → Webhook, cron/schedule → Scheduled, otherwise Manual; a large multi-integration workflow (>10 nodes and >3 integrations) is promoted to Complex.

get_service_categories() groups services into broad buckets (messaging, email, cloud_storage, database, project_management, ai_ml, social_media, ecommerce, analytics, calendar_tasks, forms, development) used by search_by_category() and the /api/workflows/category/{category} endpoint.

Note

The user-facing display categories shown in the UI (AI Agent Development, CRM & Sales, …) come from context/*.json and the /api/categories endpoint. The internal get_service_categories() buckets above are a separate, code-level grouping used by the category-search endpoint. They are related but not identical lists.

6. Core API Reference

Served by api_server.py (FastAPI app titled "N8N Workflow Documentation API", version 2.0.0). All responses are JSON unless noted. Interactive docs at /docs, OpenAPI schema at /openapi.json.

per_page is bounded 1–100; page ≥ 1. The search endpoint returns a SearchResponse envelope (workflows, total, page, per_page, pages, query, filters).

Middleware: GZipMiddleware (compresses responses ≥ 1000 bytes) and CORSMiddleware (restricted — see Section 7). A global exception handler returns {"detail": "Internal server error: ..."} with HTTP 500.

Warning

The global exception handler echoes the exception string into the response body. This can leak internal detail (paths, SQL fragments) to clients. Consider replacing it with a generic message and server-side logging before exposing the service publicly. See Section 11 (Improvements).

6.1 Status codes you'll see

7. Security

7.1 Path-traversal protection (validate_filename)

The three filename-bearing endpoints (detail, download, diagram) all call validate_filename() before touching the filesystem. It:

• Decodes the input up to 3 times to defeat nested URL-encoding (%252e%252e).
• Rejects any string containing .., slashes/backslashes, null bytes, newlines, ~, :, |, <, >, *, ?, $, ;, &.
• Rejects absolute paths and Windows drive letters.
• Requires the name to match ^[a-zA-Z0-9_\-]+\.json$.

As defense in depth, after locating a candidate file the handlers call Path.resolve().relative_to(workflows_path) and refuse anything resolving outside the workflows/ tree. The repository ships test_security.sh, which fires a battery of traversal payloads and asserts they're blocked (400/404), plus a positive-path download test.

Important

Because filenames are restricted to [A-Za-z0-9_-].json, any workflow file whose name contains spaces, dots (beyond the extension), or other characters will be un-downloadable through the API even though it indexes fine. Keep workflow filenames within that charset.

7.2 Rate limiting

check_rate_limit() keeps a per-IP in-memory list of request timestamps and enforces MAX_REQUESTS_PER_MINUTE = 60. It is applied to the detail, download, diagram, and reindex endpoints.

Caution

Rate-limit state is an in-process dict. It is not shared across workers or replicas and resets on restart, and it keys on request.client.host — which behind a reverse proxy is the proxy's IP unless you configure trusted forwarded headers. For real protection at scale, enforce rate limiting at the proxy/edge instead of relying on this.

7.3 CORS

api_server.py restricts CORS to an explicit allowlist: http://localhost:3000, :8000, :8080, https://zie619.github.io, and a Render community URL — with methods limited to GET/POST and headers to Content-Type/Authorization. Edit ALLOWED_ORIGINS to add your own front-end origin.

Note

The standalone enhanced_api.py module, by contrast, uses allow_origins=["*"]. That is another reason to treat the standalone modules (Section 9) as not production-ready as-is.

7.4 Authentication posture

The core server has no authentication for browsing or for the read endpoints — it is designed to serve a public, read-only catalog. The only privileged action is /api/reindex, gated by ADMIN_TOKEN. Login/JWT functionality exists only in the optional src/user_management.py module (Section 9), which is not mounted by default.

7.5 Hardening checklist

• [ ] Run behind a TLS-terminating reverse proxy (the prod Compose overlay uses Traefik).
• [ ] Add proxy-level auth (e.g. Traefik basicauth) if the catalog should not be public.
• [ ] Leave ADMIN_TOKEN unset unless you actively need remote reindexing; prefer reindexing via CLI on the host.
• [ ] Restrict /api/reindex at the proxy even when ADMIN_TOKEN is set.
• [ ] Replace the verbose global exception handler before public exposure.
• [ ] Enforce rate limiting at the edge, not just in-process.
• [ ] Mount workflows/ read-only into the container (the prod overlay already does: ./workflows:/app/workflows:ro).

8. Deployment & Operations

8.1 Docker / Docker Compose

The repo provides a base docker-compose.yml plus two overlays:

• docker-compose.dev.yml — bind-mounts the source for live editing, runs python run.py --host 0.0.0.0 --port 8000 --dev, exposes 8000/8001, and offers optional db-admin (Adminer) and file-watcher profiles.
• docker-compose.prod.yml — restart: always, production env vars (ENVIRONMENT=production, LOG_LEVEL=warning, ENABLE_METRICS=true, MAX_WORKERS=4), named volumes for the DB and logs, a read-only mount of workflows/, resource limits (512 MB / 0.5 CPU), Traefik labels for TLS + basicauth, and optional Prometheus monitoring.

One-command bring-up (build, start, health-check, open browser):

./run-as-docker-container.sh

Manual:

docker compose up -d --build                                   # base
docker compose -f docker-compose.yml -f docker-compose.dev.yml up   # dev
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d # prod
docker compose logs -f
docker compose down

8.2 Kubernetes & Helm

• k8s/ contains raw Kubernetes manifests.
• helm/workflows-docs/ is a Helm chart (with templates/) for templated deployment.

Use the Helm chart for parameterized installs; fall back to the raw manifests for a quick kubectl apply -f k8s/.

8.3 Production server

For production, prefer Gunicorn (pinned in requirements.txt) managing Uvicorn workers rather than the single-process uvicorn.run() used by run.py:

gunicorn api_server:app -k uvicorn.workers.UvicornWorker -w 4 -b 0.0.0.0:8000

Warning

The in-process rate limiter and the empty-DB auto-index logic assume a single process. With multiple Gunicorn workers, ensure the database is already indexed before starting (run python workflow_db.py --index first, or start once with run.py to build it), so workers don't race to index on first request.

8.4 Routine operations

Tip

The database is fully derivable from workflows/. The simplest disaster recovery is to keep the workflow JSON files under version control and re-index — no database backup strategy is strictly required.

8.5 The static / GitHub Pages mirror

docs/ is a static export intended for GitHub Pages (zie619.github.io). Its docs/js/search.js loads a pre-built api/search-index.json and does client-side filtering — there is no live server in that mode. docs/api/*.json (stats.json, integrations.json, categories.json, metadata.json) are the static data snapshots that front end consumes. Regenerate them when you publish a new static build.

9. Optional / Standalone Feature Modules

Important

Everything in this section lives under src/*.py, builds its own FastAPI application object, and is not imported or mounted by api_server.py. Running the core server does not expose these endpoints. They appear to be experimental/auxiliary and are documented here for completeness and for developers who may wish to wire them in. Treat them as not production-ready without review (permissive CORS, query-string secrets, simulated data — noted below).

9.1 Community features data model

If wired in, community_features.py adds these SQLite tables alongside workflows: workflow_ratings (1–5 star, unique per (workflow_id,user_id)), workflow_stats (aggregate ratings/views/downloads), user_profiles, workflow_collections (favorites, JSON id array), and workflow_comments (threaded via parent_id). Ratings outside 1–5 raise ValueError('Rating must be between 1 and 5').

9.2 Wiring one in (developer note)

These modules expose either a module-level app or a create_* helper. To integrate, e.g., community endpoints into the main server you would import and call create_community_api_endpoints(app) against the api_server.py app and run a single process. Before doing so, review CORS, auth, secret handling, and the SQL-string building in ai_assistant.search_workflows_intelligent() (see Section 11).

10. Code Quality & Conventions

Strengths observed in the source:

• Consistent, well-commented modules with docstrings on public functions.
• Pydantic models (WorkflowSummary, SearchResponse, StatsResponse) give the core API a typed, self-documenting contract and power the /docs UI.
• Parameterized SQL in the core search/stats/category paths (no string interpolation of user input).
• Deliberate security work at the file boundary, with a dedicated test script.
• The Python and Node implementations are kept conceptually aligned (same schema, same complexity thresholds, same format_workflow_name rules).

Patterns & conventions:

• The filename (basename) is the public identity of a workflow throughout the API.
• Complexity is a pure function of node count (≤5 / ≤15 / else).
• JSON-encoded arrays (integrations, tags) are stored as TEXT and parsed on read.
• Connections open and close per query (no shared pool) — fine at this scale, simple to reason about.

11. Potential Improvements & Tech Debt

These are observations from the source, prioritized for an operator/maintainer.

Caution

Bug — search_workflows_intelligent() in ai_assistant.py. When the detected intent is "integration", the code does params.append('Webhook') but params is never defined in that scope (and the subsequent query is executed without parameters), which will raise a NameError/parameter mismatch. This standalone module's intent-based branch is effectively broken. Fix before wiring the AI assistant in.

Warning

SQL string interpolation in the standalone AI module. ai_assistant.py builds its WHERE clause by interpolating extracted keywords directly into SQL (name LIKE '%{keyword}%'). Keywords are derived from a fixed vocabulary so exploitation is limited, but this pattern should be converted to parameterized queries before any production use. The core server does not have this issue.

Other items:

• Verbose global exception handler (api_server.py) leaks exception text to clients — replace with a generic message + server-side logging.
• /api/reindex token via query string — move to a header (Authorization/X-Admin-Token) to avoid proxy/access-log leakage, and use a constant-time comparison.
• In-process rate limiting — does not survive restarts or scale across workers; move to the edge or a shared store.
• /api/integrations returns an empty list with only a count — it could surface the ranked integration list already present in docs/api/integrations.json.
• Simulated analytics — analytics_engine.get_trend_analysis() returns hardcoded values; either implement real historical tracking or label it clearly as a demo.
• Repository hygiene — the unrelated medcards-ai/ Next.js subproject and several experimental src/ modules inflate the repo and dependency surface; consider splitting them into separate repositories.
• Known issue #156 (No module named 'workflow_db') stems from running outside the repo root; documenting the working directory requirement (done in the User Guide) mitigates it.

12. Testing

Run them against a locally running instance:

python run.py --skip-index &     # or with a built DB
bash test_api.sh
bash test_security.sh
python test_workflows.py

Note

These are shell/script smoke tests, not a unit-test suite. There is no pytest configuration in the core project; coverage of the indexing/search logic is via the live API scripts above.

13. Quick Reference

Launch (Python):

python run.py [--host H] [--port P] [--reindex] [--dev] [--skip-index]
python api_server.py [--host H] [--port P] [--reload]

Database CLI:

python workflow_db.py [--index] [--force] [--search "TEXT"] [--stats]

Node stack:

node src/index-workflows.js [--force] [--stats]
node src/server.js [-p PORT] [-h HOST] [--dev]

Production (Gunicorn):

python workflow_db.py --index      # build DB first
gunicorn api_server:app -k uvicorn.workers.UvicornWorker -w 4 -b 0.0.0.0:8000

Containers:

./run-as-docker-container.sh
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d

Environment variables:

Core endpoints: /, /health, /api/stats, /api/workflows, /api/workflows/{filename}[/download|/diagram], /api/reindex, /api/integrations, /api/categories, /api/category-mappings, /api/workflows/category/{category}, /docs.