PrivateGPT Administrator & Developer Guide

1. Architecture Overview

PrivateGPT is a privacy-focused RAG (Retrieval-Augmented Generation) platform that wraps LlamaIndex behind an OpenAI-compatible FastAPI server and an optional Gradio web UI. It enables document ingestion, embedding, vector search, and LLM-powered Q&A entirely within a private infrastructure — no data leaves the host environment.

The system follows a dependency-injection architecture using the injector library, with all component wiring centralized in the application bootstrapper. Settings are loaded from mergeable YAML profiles with environment variable substitution. The server exposes a stateless REST API that delegates to service classes for chat, ingestion, chunk retrieval, summarization, and embeddings.

Architecture Diagram

                    ┌─────────────────────────┐
                    │     Gradio Web UI        │
                    │   (optional, /path)      │
                    └────────────┬────────────┘
                                 │ mount_in_app()
┌────────────────────────────────┼────────────────────────────────┐
│                          FastAPI App                             │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐          │
│  │ /v1/chat │ │/v1/ingest│ │/v1/chunks│ │  /health │   ...    │
│  └────┬─────┘ └────┬─────┘ └────┬─────┘ └──────────┘          │
│       │             │            │                               │
│  ┌────┴─────┐ ┌────┴─────┐ ┌───┴──────┐                       │
│  │ChatService│ │IngestSvc │ │ChunksSvc │  (DI via Injector)    │
│  └────┬─────┘ └────┬─────┘ └───┬──────┘                       │
└───────┼─────────────┼───────────┼──────────────────────────────┘
        │             │           │
┌───────┴─────────────┴───────────┴──────────────────────────────┐
│                      LlamaIndex Core                            │
│  ┌──────────┐  ┌──────────────┐  ┌────────────┐               │
│  │LLM Component│ │Embedding     │ │VectorStore │               │
│  │(pluggable) │ │Component     │ │Component   │               │
│  └────────────┘ └──────────────┘ └────────────┘               │
│     llamacpp       huggingface      qdrant                      │
│     ollama         ollama           chroma                      │
│     openai         openai           postgres                    │
│     azopenai       azopenai         clickhouse                  │
│     gemini         gemini           milvus                      │
│     sagemaker      sagemaker                                    │
│     mock           mock                                         │
└─────────────────────────────────────────────────────────────────┘

Component Inventory

Technology Stack

Data Flow: RAG Query

User query → FastAPI chat_router
  → ChatService.stream_chat(messages, use_context=True)
    → LlamaIndex query engine
      → EmbeddingComponent → vector similarity search
      → VectorStoreComponent → retrieve top-k chunks
      → (optional) RerankSettings → cross-encoder rerank
      → LLMComponent → generate response with context
    ← streaming ChatResponse + source Chunks
  ← SSE stream (OpenAI format) or JSON response

Data Flow: Document Ingestion

File upload → FastAPI ingest_router
  → IngestService.ingest(file)
    → LlamaIndex document parser
      → Split into chunks/nodes
      → EmbeddingComponent → generate embeddings
      → VectorStoreComponent → store vectors
      → NodeStoreComponent → store document nodes
    ← IngestResponse with doc_ids + metadata

2. System Requirements

Note

When using cloud LLM backends (OpenAI, Azure, Gemini), the RAM and GPU requirements drop significantly since inference runs remotely. Local llama.cpp and HuggingFace embeddings are the primary consumers of local resources.

Dependencies (Key Libraries)

3. Configuration Reference

Configuration Architecture

Settings are loaded through a profile-merging system in private_gpt/settings/settings_loader.py:

1. The default profile (settings.yaml) is always loaded first
2. Profiles from PGPT_PROFILES env var are loaded in order and deep-merged
3. If running tests (tests.fixtures in sys.modules), the test profile is appended
4. The merged dict is validated against the Settings Pydantic model

Important

Profile merging uses deep update — nested keys are merged, not replaced wholesale. A profile only needs to specify the keys it wants to override.

Environment Variables

Environment Variable Substitution Syntax

The YAML loader (private_gpt/settings/yaml.py) supports ${VAR} and ${VAR:default}:

api_key: ${OPENAI_API_KEY}        # Required — fails if not set
api_key: ${OPENAI_API_KEY:}       # Optional — empty string default
model: ${MODEL:gpt-3.5-turbo}     # Optional — with specific default

The substitution is implemented as a custom PyYAML implicit resolver that matches the pattern \$\{(\w|_)+:?.*\}.

Complete Settings Schema

All settings are defined as Pydantic BaseModel classes in private_gpt/settings/settings.py.

Server Settings (server)

Warning

The default settings.yaml ships with server.cors.allow_origins: ["*"] and server.cors.allow_methods: ["*"]. Restrict these in production deployments.

Data Settings (data)

LLM Settings (llm)

Embedding Settings (embedding)

RAG Settings (rag)

UI Settings (ui)

Summarize Settings (summarize)

Vector Store Settings (vectorstore)

Node Store Settings (nodestore)

llama.cpp Settings (llamacpp)

HuggingFace Settings (huggingface)

Caution

Setting trust_remote_code: true allows arbitrary code execution from the model repository. The default settings.yaml enables this for nomic-ai/nomic-embed-text-v1.5 which requires it. Verify the model source before enabling.

Ollama Settings (ollama)

OpenAI Settings (openai)

Azure OpenAI Settings (azopenai)

Gemini Settings (gemini)

SageMaker Settings (sagemaker)

Qdrant Settings (qdrant)

PostgreSQL Settings (postgres)

ClickHouse Settings (clickhouse)

Milvus Settings (milvus)

Configuration Files

4. Deployment & Scaling

Deployment Options

Direct Python

PGPT_PROFILES=ollama python -m private_gpt

The entry point (private_gpt/__main__.py) starts uvicorn bound to 0.0.0.0 on the configured port with custom log configuration.

Note

Uvicorn's default logging is disabled (log_config=None) in favor of PrivateGPT's own logging format: HH:MM:SS.mmm [LEVEL ] module_name - message.

Docker Compose

The docker-compose.yaml and Docker profile (settings-docker.yaml) configure:

• Port: 8080 (via ${PORT:8080})
• LLM/embedding modes: configurable via env vars (PGPT_MODE, PGPT_EMBED_MODE)
• Ollama auto-pull: enabled by default

docker compose up -d

Docker with Ollama

docker build -t private-gpt -f Dockerfile.ollama .
docker run -p 8001:8080 \
  -e PGPT_OLLAMA_API_BASE=http://host.docker.internal:11434 \
  private-gpt

Docker with llama.cpp (CPU)

docker build -t private-gpt-cpu -f Dockerfile.llamacpp-cpu .
docker run -p 8001:8080 private-gpt-cpu

Scaling Considerations

Tip

For production deployments handling multiple concurrent users, consider:

• Using a client-server vector store (Qdrant server, PostgreSQL) instead of local file-based storage
• Running the Ollama service on a separate host with GPU acceleration
• Setting embedding.ingest_mode: parallel with appropriate count_workers for bulk ingestion
• Placing a reverse proxy (nginx, Traefik) in front of PrivateGPT for TLS termination

The FastAPI application is single-process by default. For horizontal scaling:

1. Use an external vector store that supports concurrent connections
2. Run multiple uvicorn workers or deploy behind a load balancer
3. Ensure the Ollama/LLM backend can handle concurrent requests

5. Operations

Health Check

curl http://localhost:8001/health

{"status": "ok"}

Tip

Use the /health endpoint for container orchestration liveness probes. It returns HTTP 200 when the FastAPI application is running and accepting connections.

Monitoring

PrivateGPT uses Python's standard logging module at INFO level by default. The log format is:

HH:MM:SS.mmm [LEVEL   ] module_name - message

Set ROOT_LOG_LEVEL = "DEBUG" in private_gpt/__init__.py for verbose output including library logs.

LlamaIndex observability is enabled via the simple global handler in launcher.py, which provides callback-based tracing of LLM calls, embedding operations, and retrieval steps.

Ollama Model Management

When ollama.autopull_models is true, PrivateGPT automatically pulls missing models at startup. The pull process:

1. Checks installed models via client.list()
2. If the configured model is not found, pulls with streaming progress bars
3. Retries up to 5 times with jitter (3–10 seconds) on connection errors

Note

The Ollama connection check uses a retry decorator with 5 attempts and randomized jitter between 3.0 and 10.0 seconds to handle slow Ollama startups.

Local Data Management

Default local data paths (derived from data.local_data_folder):

• local_data/private_gpt/ — main data directory
• local_data/private_gpt/qdrant/ — Qdrant local persistence (when using local Qdrant)
• local_data/private_gpt/milvus/ — Milvus local persistence
• models/ — downloaded model files
• models/cache/ — model cache

Backup and Recovery

For local Qdrant deployments, back up the local_data/ directory. For PostgreSQL deployments, use standard pg_dump. Vector store data and node store data must be backed up together to maintain consistency.

6. Performance

Ingestion Mode Comparison

The count_workers setting (default: 2) controls parallelism for non-simple modes. Optimal values depend on CPU threads and available RAM.

Warning

Setting count_workers too high in parallel mode can cause memory exhaustion, especially with large embedding models. Stay at or below your CPU thread count.

LLM Tuning Parameters

RAG Tuning

• rag.similarity_top_k: Number of chunks retrieved before reranking (default: 2)
• rag.similarity_value: Minimum similarity score filter — disabled by default; set to 0.45–0.7 to filter low-relevance results
• Reranking: Enable for better precision at the cost of additional compute (loads a cross-encoder model)

7. Security

Authentication

Authentication is implemented in private_gpt/server/utils/auth.py as a FastAPI dependency:

• When server.auth.enabled is false (default): a dummy dependency always returns True
• When enabled: validates the Authorization header against server.auth.secret using secrets.compare_digest() (constant-time comparison)

Failed authentication returns HTTP 401 with a WWW-Authenticate: Basic realm="All the API" header.

Caution

The default settings.yaml contains a hardcoded auth secret: Basic c2VjcmV0OmtleQ== (which decodes to secret:key). This is a known security issue (#2204). Always change this value and enable authentication before any network exposure.

CORS Configuration

CORS is handled via FastAPI's CORSMiddleware, configured under server.cors. The default settings.yaml enables CORS with permissive settings:

cors:
  enabled: true
  allow_origins: ["*"]
  allow_methods: ["*"]
  allow_headers: ["*"]

Warning

Wildcard CORS origins allow any website to make API requests to your PrivateGPT instance. Restrict allow_origins to your specific frontend domains in production.

Data Privacy

• No telemetry: Gradio analytics is explicitly disabled (GRADIO_ANALYTICS_ENABLED=False)
• No ChromaDB telemetry: Anonymized telemetry is disabled
• Offline tokenization: The tiktoken_cache/ directory is bundled for offline token counting
• Local-only processing: When using local LLM and embedding backends, no data leaves the host
• Local ingestion control: The data.local_ingestion.allow_ingest_from setting restricts which filesystem paths can be ingested

Security Recommendations

• [ ] Change the default auth secret before deployment
• [ ] Enable authentication (server.auth.enabled: true)
• [ ] Restrict CORS origins to known frontends
• [ ] Use TLS termination via a reverse proxy
• [ ] Restrict local_ingestion.allow_ingest_from to specific directories
• [ ] Review huggingface.trust_remote_code — disable if not required by your model
• [ ] Consider integrity and malware checks for local embeddings (#2201)

8. Developer Internals

Repository Structure

private-gpt/
├── private_gpt/                  # Main application package
│   ├── __init__.py               # Logging setup, telemetry disabling
│   ├── __main__.py               # Uvicorn entry point
│   ├── main.py                   # App singleton
│   ├── launcher.py               # FastAPI app factory
│   ├── di.py                     # Dependency injection container
│   ├── constants.py              # PROJECT_ROOT_PATH
│   ├── paths.py                  # Derived filesystem paths
│   ├── components/               # Pluggable backend components
│   │   ├── embedding/            # Embedding backends
│   │   │   └── custom/           # SageMaker custom embedding
│   │   ├── ingest/               # Document ingestion
│   │   ├── llm/                  # LLM backends
│   │   │   ├── prompt_helper.py  # Prompt formatting
│   │   │   └── custom/           # SageMaker custom LLM
│   │   ├── node_store/           # Node storage backends
│   │   └── vector_store/         # Vector storage backends
│   │       └── batched_chroma.py # Batched Chroma operations
│   ├── open_ai/                  # OpenAI-compatible models
│   │   ├── openai_models.py      # Request/response Pydantic models
│   │   └── extensions/
│   │       └── context_filter.py # Document ID filtering
│   ├── server/                   # FastAPI routers and services
│   │   ├── chat/                 # Chat completions
│   │   ├── chunks/               # Chunk retrieval
│   │   ├── completions/          # Text completions
│   │   ├── embeddings/           # Embedding generation
│   │   ├── health/               # Health check
│   │   ├── ingest/               # Document ingestion + watcher
│   │   ├── recipes/summarize/    # Summarization recipe
│   │   └── utils/auth.py         # Authentication
│   ├── settings/                 # Configuration system
│   │   ├── settings.py           # Pydantic settings models
│   │   ├── settings_loader.py    # Profile loading and merging
│   │   └── yaml.py               # YAML loader with env var substitution
│   ├── ui/                       # Gradio web interface
│   │   ├── ui.py                 # UI logic (4 modes, file management)
│   │   ├── images.py             # Logo SVG data
│   │   └── avatar-bot.ico        # Bot avatar
│   └── utils/                    # Utilities
│       ├── eta.py                # ETA calculator with Pearson regression
│       ├── ollama.py             # Ollama connection, model pulling
│       ├── retry.py              # Retry decorator wrapper
│       └── typing.py             # TypeVar definitions
├── fern/                         # API documentation (Fern)
│   ├── docs.yml                  # Fern docs configuration
│   ├── fern.config.json          # Fern project config
│   ├── generators.yml            # SDK generator config
│   ├── openapi/openapi.json      # OpenAPI 3.1 spec
│   └── docs/                     # Documentation pages
├── scripts/                      # Utility scripts
│   ├── extract_openapi.py        # Export OpenAPI spec
│   ├── ingest_folder.py          # Bulk folder ingestion
│   └── setup                     # Setup script
├── tests/                        # Test suite
│   ├── conftest.py               # Shared fixtures
│   ├── fixtures/                 # Test fixtures (mock injector, etc.)
│   ├── server/                   # Router tests
│   ├── settings/                 # Settings tests
│   └── ui/                       # UI tests
├── settings.yaml                 # Default settings
├── settings-*.yaml               # Profile settings
├── docker-compose.yaml           # Docker stack
├── Dockerfile.llamacpp-cpu       # llama.cpp CPU image
├── Dockerfile.ollama             # Ollama image
├── Makefile                      # Build/test targets
└── pyproject.toml                # Poetry project definition

Dependency Injection Architecture

PrivateGPT uses the injector library for dependency injection. The DI container is created in private_gpt/di.py:

def create_application_injector() -> Injector:
    _injector = Injector(auto_bind=True)
    _injector.binder.bind(Settings, to=unsafe_typed_settings)
    return _injector

Key behaviors:

• auto_bind=True allows automatic resolution of dependencies not explicitly bound
• Settings is bound as a singleton to the loaded settings object
• The global injector is created at module import time
• Each FastAPI request gets the injector via request.state.injector
• Service classes (ChatService, IngestService, etc.) use @inject decorators

Note

The global_injector is a module-level singleton. For testing, use MockInjector to override bindings. The test fixtures in tests/fixtures/mock_injector.py demonstrate this pattern.

Application Startup Sequence

1. private_gpt/__init__.py — configures logging (INFO level), disables Gradio analytics and ChromaDB telemetry, sets tiktoken cache path
2. private_gpt/settings/settings_loader.py — loads active profiles, merges settings
3. private_gpt/settings/settings.py — validates merged settings against Pydantic models
4. private_gpt/di.py — creates the global Injector and binds Settings
5. private_gpt/launcher.py — create_app() creates the FastAPI app:
   • Registers bind_injector_to_request dependency
   • Includes all routers (completions, chat, chunks, ingest, summarize, embeddings, health)
   • Sets up LlamaIndex simple observability handler
   • Configures CORS middleware if enabled
   • Mounts Gradio UI if ui.enabled
6. private_gpt/__main__.py — starts uvicorn on 0.0.0.0:{port}

API Endpoints (Complete Reference)

"Auth: Yes" means the endpoint respects the authenticated dependency — when server.auth.enabled is false, authentication is effectively a no-op.

OpenAI Compatibility Layer

The private_gpt/open_ai/openai_models.py module defines Pydantic models that mirror the OpenAI API response format:

Helper functions:

• to_openai_response() — converts a LlamaIndex response to OpenAICompletion
• to_openai_sse_stream() — converts a LlamaIndex streaming generator to SSE format with data: prefixed JSON chunks, terminated by data: [DONE]

Context Filtering

The ContextFilter model (private_gpt/open_ai/extensions/context_filter.py) allows filtering RAG queries to specific documents by passing their doc_ids. This is used by both the API and the Gradio UI when a specific file is selected.

Gradio UI Internals

PrivateGptUi (private_gpt/ui/ui.py) is a @singleton class that:

• Receives services via @inject: IngestService, ChatService, ChunksService, SummarizeService
• Builds a Gradio Blocks interface with a mode radio, file management sidebar, system prompt input, and chat interface
• Dispatches to different service methods based on the selected Modes enum (RAG, Search, Basic, Summarize)
• Limits chat history to 20 messages to prevent context overflow
• Strips source separators from history to keep context clean
• Mounts into FastAPI via gr.mount_gradio_app()

The UI tab title is hardcoded as "My Private GPT".

ETA Calculator

private_gpt/utils/eta.py provides an ETA class that predicts completion time using Pearson product-moment correlation coefficient on timing data. It maintains a sliding window of 100 samples and reports progress in human-readable format (e.g., "2m 30s @ 15/min").

Retry Mechanism

private_gpt/utils/retry.py wraps the retry-async library's retry decorator with typed parameters. Used for Ollama connection retries with configurable:

• tries: max attempts (5 for Ollama)
• jitter: random delay range (3.0–10.0 seconds for Ollama)
• exceptions: exception types to catch
• backoff: multiplicative backoff factor

Module Reference

private_gpt/settings/settings.py

The central settings module defines 27 Pydantic BaseModel classes representing every configurable aspect of the system. The top-level Settings model aggregates all sub-models. Settings are loaded once at import time via unsafe_typed_settings and made available through DI or the settings() function.

private_gpt/launcher.py

The create_app() factory function is the single place where all routers are registered, middleware is configured, and the UI is mounted. It takes an Injector and returns a fully configured FastAPI application.

private_gpt/ui/ui.py

At 598 lines, this is the largest module. It implements the complete Gradio interface including file upload/delete, 4 chat modes, streaming response rendering with source attribution, system prompt management, and model label display.

private_gpt/open_ai/openai_models.py

Implements the OpenAI-compatible response format. The OpenAICompletion class provides from_text() for single responses and json_from_delta() for streaming chunks.

private_gpt/server/utils/auth.py

Implements a switchable authentication pattern: at module load time, it checks settings().server.auth.enabled and defines either a no-op authenticated() function or one that validates the Authorization header via secrets.compare_digest().

Code Patterns and Conventions

• Dependency injection everywhere: Services are @inject-decorated, DI container resolves all dependencies
• Pydantic for all data: Settings, API request/response bodies, and internal models use Pydantic BaseModel
• Profile-based configuration: No hardcoded values — everything flows through the YAML profile system
• Module-level initialization: Settings are loaded and validated at import time, before the server starts
• Match statements: Python 3.10+ match/case used for mode dispatching in the UI
• Type hints throughout: Full type annotations with mypy checking
• Pre-commit hooks: Black (formatting), Ruff (linting), mypy (types), pytest (tests on push)

Testing

Tests use pytest with custom fixtures:

The test profile (settings-test.yaml) configures:

• Auth disabled with dummy secret "foo bar; dummy secret"
• Mock LLM and embedding modes
• Local data stored in local_data/tests
• UI disabled

Test coverage includes:

• Authentication (default no-auth and enabled basic auth)
• Settings loading, merging, and env var substitution
• Summarize routes (streaming, single, with/without context)
• UI endpoint mounting

Potential Improvements

• Per-user document isolation (#1796) — currently all ingested documents are global
• Integrity and malware checks for local embeddings (#2201)
• Additional LLM backend support (Claude requested in #2199)
• OceanBase vector storage support (#2214)
• Session-based ingestion with stop-and-start capability (#2190, #1914)
• Bulk ingestion from Dockerfile with --watch support (#1914)

Contributing

The project uses:

• Poetry for dependency management
• Black for code formatting
• Ruff for linting
• mypy for type checking
• pytest for testing
• pre-commit for automated hooks (format on commit, test on push)

Setup the development environment:

git clone https://github.com/zylon-ai/private-gpt
cd private-gpt
poetry install --all-extras
pre-commit install

Run checks:

make test    # Run pytest
make mypy    # Run type checking

The API documentation is generated with Fern from the OpenAPI spec at fern/openapi/openapi.json. The Python SDK is generated via fernapi/fern-python-sdk (version 0.6.2).

9. Integration Points

OpenAI SDK Compatibility

PrivateGPT's API is compatible with OpenAI's client libraries. Point the OpenAI SDK at your PrivateGPT instance:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8001/v1", api_key="not-needed")
response = client.chat.completions.create(
    model="private-gpt",
    messages=[{"role": "user", "content": "Hello"}]
)

Fern SDK

An official Python SDK is generated from the OpenAPI spec using Fern. The SDK output path is ../../pgpt-sdk/python relative to the fern/ directory. The Fern documentation is hosted at docs.privategpt.dev.

Ingestion Watcher

The private_gpt/server/ingest/ingest_watcher.py module provides filesystem watching capability for automatic ingestion of new files placed in monitored directories. This requires data.local_ingestion.enabled: true.

LlamaIndex Ecosystem

PrivateGPT builds on LlamaIndex for:

• Document parsing and chunking
• Embedding model abstraction
• Vector store abstraction
• Query engine and chat engine
• Callback-based observability

All LlamaIndex components are accessed through PrivateGPT's component wrappers in private_gpt/components/, which handle configuration injection and backend selection.