# PrivateGPT Administrator & Developer Guide

## 1. Architecture Overview

PrivateGPT is a privacy-focused RAG (Retrieval-Augmented Generation) platform that wraps LlamaIndex behind an OpenAI-compatible FastAPI server and an optional Gradio web UI. It enables document ingestion, embedding, vector search, and LLM-powered Q&A entirely within a private infrastructure — no data leaves the host environment.

The system follows a dependency-injection architecture using the `injector` library, with all component wiring centralized in the application bootstrapper. Settings are loaded from mergeable YAML profiles with environment variable substitution. The server exposes a stateless REST API that delegates to service classes for chat, ingestion, chunk retrieval, summarization, and embeddings.

### Architecture Diagram

```text
                    ┌─────────────────────────┐
                    │     Gradio Web UI        │
                    │   (optional, /path)      │
                    └────────────┬────────────┘
                                 │ mount_in_app()
┌────────────────────────────────┼────────────────────────────────┐
│                          FastAPI App                             │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐          │
│  │ /v1/chat │ │/v1/ingest│ │/v1/chunks│ │  /health │   ...    │
│  └────┬─────┘ └────┬─────┘ └────┬─────┘ └──────────┘          │
│       │             │            │                               │
│  ┌────┴─────┐ ┌────┴─────┐ ┌───┴──────┐                       │
│  │ChatService│ │IngestSvc │ │ChunksSvc │  (DI via Injector)    │
│  └────┬─────┘ └────┬─────┘ └───┬──────┘                       │
└───────┼─────────────┼───────────┼──────────────────────────────┘
        │             │           │
┌───────┴─────────────┴───────────┴──────────────────────────────┐
│                      LlamaIndex Core                            │
│  ┌──────────┐  ┌──────────────┐  ┌────────────┐               │
│  │LLM Component│ │Embedding     │ │VectorStore │               │
│  │(pluggable) │ │Component     │ │Component   │               │
│  └────────────┘ └──────────────┘ └────────────┘               │
│     llamacpp       huggingface      qdrant                      │
│     ollama         ollama           chroma                      │
│     openai         openai           postgres                    │
│     azopenai       azopenai         clickhouse                  │
│     gemini         gemini           milvus                      │
│     sagemaker      sagemaker                                    │
│     mock           mock                                         │
└─────────────────────────────────────────────────────────────────┘
```

### Component Inventory

| Component | Path | Role |
|---|---|---|
| App entry point | `private_gpt/__main__.py` | Starts uvicorn with the FastAPI app |
| App factory | `private_gpt/launcher.py` | Creates FastAPI app, registers routers, mounts UI |
| App singleton | `private_gpt/main.py` | Exports the `app` object for uvicorn |
| DI container | `private_gpt/di.py` | Creates and configures the `Injector` instance |
| Settings | `private_gpt/settings/` | YAML-based config with profile merging |
| Server routers | `private_gpt/server/` | FastAPI route handlers |
| Components | `private_gpt/components/` | LLM, embedding, ingest, vector/node store wrappers |
| OpenAI compat | `private_gpt/open_ai/` | OpenAI response format models |
| Gradio UI | `private_gpt/ui/ui.py` | Web interface with 4 interaction modes |
| Utilities | `private_gpt/utils/` | ETA, Ollama helpers, retry decorator |
| Paths | `private_gpt/paths.py` | Derived paths for models, docs, local_data |
| Constants | `private_gpt/constants.py` | `PROJECT_ROOT_PATH` |

### Technology Stack

| Layer | Technology |
|---|---|
| Language | Python 3.11+ |
| Web framework | FastAPI + Uvicorn |
| AI framework | LlamaIndex |
| DI framework | Injector |
| Settings | Pydantic v2 models + YAML (PyYAML SafeLoader) |
| Web UI | Gradio |
| LLM backends | llama.cpp, Ollama, OpenAI, Azure OpenAI, Gemini, SageMaker |
| Embedding backends | HuggingFace, Ollama, OpenAI, Azure OpenAI, Gemini, SageMaker, Mistral AI |
| Vector stores | Qdrant, Chroma, PostgreSQL, ClickHouse, Milvus |
| Node stores | Simple (file-based), PostgreSQL |
| Package management | Poetry |
| Code quality | Black, Ruff, mypy, pre-commit |
| Testing | pytest |
| API docs | Fern (OpenAPI-driven) |

### Data Flow: RAG Query

```text
User query → FastAPI chat_router
  → ChatService.stream_chat(messages, use_context=True)
    → LlamaIndex query engine
      → EmbeddingComponent → vector similarity search
      → VectorStoreComponent → retrieve top-k chunks
      → (optional) RerankSettings → cross-encoder rerank
      → LLMComponent → generate response with context
    ← streaming ChatResponse + source Chunks
  ← SSE stream (OpenAI format) or JSON response
```

### Data Flow: Document Ingestion

```text
File upload → FastAPI ingest_router
  → IngestService.ingest(file)
    → LlamaIndex document parser
      → Split into chunks/nodes
      → EmbeddingComponent → generate embeddings
      → VectorStoreComponent → store vectors
      → NodeStoreComponent → store document nodes
    ← IngestResponse with doc_ids + metadata
```

## 2. System Requirements

| Resource | Development | Production |
|---|---|---|
| Python | 3.11+ | 3.11+ |
| RAM | 8 GB | 16 GB+ (32 GB for large models) |
| CPU | 4 cores | 8+ cores (for parallel ingestion) |
| Disk | 10 GB | 50 GB+ (models + vector data) |
| GPU | Optional (llama.cpp) | Recommended for local LLMs |
| Network | Required for model downloads | Optional (fully offline after setup) |

> [!NOTE]
> When using cloud LLM backends (OpenAI, Azure, Gemini), the RAM and GPU requirements drop significantly since inference runs remotely. Local llama.cpp and HuggingFace embeddings are the primary consumers of local resources.

### Dependencies (Key Libraries)

| Library | Role |
|---|---|
| `fastapi` | REST API framework |
| `uvicorn` | ASGI server |
| `llama-index-core` | RAG orchestration |
| `injector` | Dependency injection |
| `pydantic` | Settings validation |
| `gradio` | Web UI (optional extra) |
| `ollama` | Ollama client (optional extra) |
| `qdrant-client` | Qdrant vector store (optional extra) |
| `chromadb` | Chroma vector store (optional extra) |
| `tqdm` | Progress bars for model downloads |
| `retry-async` | Retry decorator with jitter |
| `httpx` | HTTP client (Ollama connectivity) |
| `tiktoken` | Tokenization (bundled cache for offline use) |

## 3. Configuration Reference

### Configuration Architecture

Settings are loaded through a profile-merging system in `private_gpt/settings/settings_loader.py`:

1. The `default` profile (`settings.yaml`) is always loaded first
2. Profiles from `PGPT_PROFILES` env var are loaded in order and deep-merged
3. If running tests (`tests.fixtures` in `sys.modules`), the `test` profile is appended
4. The merged dict is validated against the `Settings` Pydantic model

> [!IMPORTANT]
> Profile merging uses deep update — nested keys are merged, not replaced wholesale. A profile only needs to specify the keys it wants to override.

### Environment Variables

| Variable | Default | Description |
|---|---|---|
| `PGPT_PROFILES` | (empty) | Comma-separated list of profiles to activate |
| `PGPT_SETTINGS_FOLDER` | project root | Directory containing settings YAML files |
| `APP_ENV` | `prod` | Environment name (used in `${APP_ENV:default}`) |
| `PORT` | `8001` | Server port |
| `LOCAL_INGESTION_ENABLED` | `false` | Enable local folder ingestion |
| `OPENAI_API_KEY` | (empty) | OpenAI API key |
| `AZ_OPENAI_API_KEY` | (empty) | Azure OpenAI API key |
| `AZ_OPENAI_ENDPOINT` | (empty) | Azure OpenAI endpoint URL |
| `AZ_OPENAI_EMBEDDING_DEPLOYMENT_NAME` | (empty) | Azure embedding deployment name |
| `AZ_OPENAI_LLM_DEPLOYMENT_NAME` | (empty) | Azure LLM deployment name |
| `GOOGLE_API_KEY` | (empty) | Google Gemini API key |
| `HF_TOKEN` | (empty) | HuggingFace access token |
| `PGPT_MODE` | `mock` | LLM mode for Docker profile |
| `PGPT_EMBED_MODE` | `mock` | Embedding mode for Docker profile |
| `PGPT_HF_REPO_ID` | `lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF` | llama.cpp HuggingFace repo |
| `PGPT_HF_MODEL_FILE` | `Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf` | llama.cpp model file |
| `PGPT_EMBEDDING_HF_MODEL_NAME` | `nomic-ai/nomic-embed-text-v1.5` | HuggingFace embedding model |
| `PGPT_OLLAMA_API_BASE` | `http://ollama:11434` | Ollama API base for Docker |
| `PGPT_OLLAMA_LLM_MODEL` | `llama3.1` | Ollama LLM model for Docker |
| `PGPT_OLLAMA_EMBEDDING_MODEL` | `nomic-embed-text` | Ollama embedding model for Docker |
| `PGPT_OLLAMA_AUTOPULL_MODELS` | `true` | Auto-pull Ollama models in Docker |

### Environment Variable Substitution Syntax

The YAML loader (`private_gpt/settings/yaml.py`) supports `${VAR}` and `${VAR:default}`:

```yaml
api_key: ${OPENAI_API_KEY}        # Required — fails if not set
api_key: ${OPENAI_API_KEY:}       # Optional — empty string default
model: ${MODEL:gpt-3.5-turbo}     # Optional — with specific default
```

The substitution is implemented as a custom PyYAML implicit resolver that matches the pattern `\$\{(\w|_)+:?.*\}`.

### Complete Settings Schema

All settings are defined as Pydantic `BaseModel` classes in `private_gpt/settings/settings.py`.

#### Server Settings (`server`)

| Key | Type | Default | Description |
|---|---|---|---|
| `server.env_name` | `str` | — (required) | Environment name |
| `server.port` | `int` | — (required) | Server port |
| `server.cors.enabled` | `bool` | `false` | Enable CORS headers |
| `server.cors.allow_credentials` | `bool` | `false` | Allow cookies in cross-origin requests |
| `server.cors.allow_origins` | `list[str]` | `[]` | Allowed origins |
| `server.cors.allow_origin_regex` | `list[str]` | `null` | Origin regex patterns |
| `server.cors.allow_methods` | `list[str]` | `["GET"]` | Allowed HTTP methods |
| `server.cors.allow_headers` | `list[str]` | `[]` | Allowed request headers |
| `server.auth.enabled` | `bool` | `false` | Enable API authentication |
| `server.auth.secret` | `str` | — (required when enabled) | Full `Authorization` header value |

> [!WARNING]
> The default `settings.yaml` ships with `server.cors.allow_origins: ["*"]` and `server.cors.allow_methods: ["*"]`. Restrict these in production deployments.

#### Data Settings (`data`)

| Key | Type | Default | Description |
|---|---|---|---|
| `data.local_data_folder` | `str` | — (required) | Path to local storage (absolute or relative to project root) |
| `data.local_ingestion.enabled` | `bool` | `false` | Enable local folder ingestion |
| `data.local_ingestion.allow_ingest_from` | `list[str]` | `[]` | Allowed folders for local ingestion |

#### LLM Settings (`llm`)

| Key | Type | Default | Description |
|---|---|---|---|
| `llm.mode` | `str` | — (required) | One of: `llamacpp`, `openai`, `openailike`, `azopenai`, `sagemaker`, `mock`, `ollama`, `gemini` |
| `llm.max_new_tokens` | `int` | `256` | Max tokens per completion |
| `llm.context_window` | `int` | `3900` | Max context tokens |
| `llm.tokenizer` | `str` | `null` | HuggingFace tokenizer model ID |
| `llm.temperature` | `float` | `0.1` | Generation temperature |
| `llm.prompt_style` | `str` | `llama2` | One of: `default`, `llama2`, `llama3`, `tag`, `mistral`, `chatml` |

#### Embedding Settings (`embedding`)

| Key | Type | Default | Description |
|---|---|---|---|
| `embedding.mode` | `str` | — (required) | One of: `huggingface`, `openai`, `azopenai`, `sagemaker`, `ollama`, `mock`, `gemini`, `mistralai` |
| `embedding.ingest_mode` | `str` | `simple` | One of: `simple`, `batch`, `parallel`, `pipeline` |
| `embedding.count_workers` | `int` | `2` | Worker count for non-simple modes |
| `embedding.embed_dim` | `int` | `384` | Embedding vector dimensions |

#### RAG Settings (`rag`)

| Key | Type | Default | Description |
|---|---|---|---|
| `rag.similarity_top_k` | `int` | `2` | Number of chunks for RAG / reranker input |
| `rag.similarity_value` | `float` | `null` | Minimum similarity score threshold (0–1) |
| `rag.rerank.enabled` | `bool` | `false` | Enable cross-encoder reranking |
| `rag.rerank.model` | `str` | `cross-encoder/ms-marco-MiniLM-L-2-v2` | Reranker model |
| `rag.rerank.top_n` | `int` | `2` | Number of documents after reranking |

#### UI Settings (`ui`)

| Key | Type | Default | Description |
|---|---|---|---|
| `ui.enabled` | `bool` | — (required) | Enable Gradio web UI |
| `ui.path` | `str` | — (required) | Mount path for UI |
| `ui.default_mode` | `str` | `RAG` | One of: `RAG`, `Search`, `Basic`, `Summarize` |
| `ui.default_chat_system_prompt` | `str` | `null` | Default system prompt for Basic mode |
| `ui.default_query_system_prompt` | `str` | `null` | Default system prompt for RAG mode |
| `ui.default_summarization_system_prompt` | `str` | `null` | Default system prompt for Summarize mode |
| `ui.delete_file_button_enabled` | `bool` | `true` | Show delete-file button |
| `ui.delete_all_files_button_enabled` | `bool` | `false` | Show delete-all button |

#### Summarize Settings (`summarize`)

| Key | Type | Default | Description |
|---|---|---|---|
| `summarize.use_async` | `bool` | `true` | Async summarization |

#### Vector Store Settings (`vectorstore`)

| Key | Type | Default | Description |
|---|---|---|---|
| `vectorstore.database` | `str` | — (required) | One of: `chroma`, `qdrant`, `postgres`, `clickhouse`, `milvus` |

#### Node Store Settings (`nodestore`)

| Key | Type | Default | Description |
|---|---|---|---|
| `nodestore.database` | `str` | — (required) | One of: `simple`, `postgres` |

#### llama.cpp Settings (`llamacpp`)

| Key | Type | Default | Description |
|---|---|---|---|
| `llamacpp.llm_hf_repo_id` | `str` | — (required) | HuggingFace repo ID for model |
| `llamacpp.llm_hf_model_file` | `str` | — (required) | GGUF model filename |
| `llamacpp.tfs_z` | `float` | `1.0` | Tail free sampling (1.0 = disabled) |
| `llamacpp.top_k` | `int` | `40` | Top-K sampling |
| `llamacpp.top_p` | `float` | `0.9` | Nucleus sampling |
| `llamacpp.repeat_penalty` | `float` | `1.1` | Repetition penalty |

#### HuggingFace Settings (`huggingface`)

| Key | Type | Default | Description |
|---|---|---|---|
| `huggingface.embedding_hf_model_name` | `str` | — (required) | HuggingFace embedding model name |
| `huggingface.access_token` | `str` | `null` | HuggingFace access token |
| `huggingface.trust_remote_code` | `bool` | `false` | Trust and execute remote model code |

> [!CAUTION]
> Setting `trust_remote_code: true` allows arbitrary code execution from the model repository. The default `settings.yaml` enables this for `nomic-ai/nomic-embed-text-v1.5` which requires it. Verify the model source before enabling.

#### Ollama Settings (`ollama`)

| Key | Type | Default | Description |
|---|---|---|---|
| `ollama.api_base` | `str` | `http://localhost:11434` | Ollama API endpoint |
| `ollama.embedding_api_base` | `str` | `http://localhost:11434` | Ollama embedding endpoint |
| `ollama.llm_model` | `str` | `null` | LLM model name |
| `ollama.embedding_model` | `str` | `null` | Embedding model name |
| `ollama.keep_alive` | `str` | `5m` | Time model stays loaded after request |
| `ollama.tfs_z` | `float` | `1.0` | Tail free sampling |
| `ollama.num_predict` | `int` | `null` | Max tokens (-1 = infinite, -2 = fill context) |
| `ollama.top_k` | `int` | `40` | Top-K sampling |
| `ollama.top_p` | `float` | `0.9` | Nucleus sampling |
| `ollama.repeat_last_n` | `int` | `64` | Repetition lookback window |
| `ollama.repeat_penalty` | `float` | `1.1` | Repetition penalty |
| `ollama.request_timeout` | `float` | `120.0` | Request timeout in seconds |
| `ollama.autopull_models` | `bool` | `false` | Auto-pull missing models |

#### OpenAI Settings (`openai`)

| Key | Type | Default | Description |
|---|---|---|---|
| `openai.api_base` | `str` | `null` | Custom API base URL |
| `openai.api_key` | `str` | — (required) | API key |
| `openai.model` | `str` | `gpt-3.5-turbo` | LLM model |
| `openai.request_timeout` | `float` | `120.0` | Request timeout |
| `openai.embedding_api_base` | `str` | `null` | Custom embedding API base |
| `openai.embedding_api_key` | `str` | — (required) | Embedding API key |
| `openai.embedding_model` | `str` | `text-embedding-ada-002` | Embedding model |

#### Azure OpenAI Settings (`azopenai`)

| Key | Type | Default | Description |
|---|---|---|---|
| `azopenai.api_key` | `str` | — (required) | Azure API key |
| `azopenai.azure_endpoint` | `str` | — (required) | Azure endpoint URL |
| `azopenai.api_version` | `str` | `2023_05_15` | API version (YYYY-MM-DD format) |
| `azopenai.embedding_deployment_name` | `str` | — (required) | Embedding deployment name |
| `azopenai.embedding_model` | `str` | `text-embedding-ada-002` | Embedding model |
| `azopenai.llm_deployment_name` | `str` | — (required) | LLM deployment name |
| `azopenai.llm_model` | `str` | `gpt-35-turbo` | LLM model |

#### Gemini Settings (`gemini`)

| Key | Type | Default | Description |
|---|---|---|---|
| `gemini.api_key` | `str` | — (required) | Google API key |
| `gemini.model` | `str` | `models/gemini-pro` | LLM model |
| `gemini.embedding_model` | `str` | `models/embedding-001` | Embedding model |

#### SageMaker Settings (`sagemaker`)

| Key | Type | Default | Description |
|---|---|---|---|
| `sagemaker.llm_endpoint_name` | `str` | — (required) | SageMaker LLM endpoint |
| `sagemaker.embedding_endpoint_name` | `str` | — (required) | SageMaker embedding endpoint |

#### Qdrant Settings (`qdrant`)

| Key | Type | Default | Description |
|---|---|---|---|
| `qdrant.location` | `str` | `null` | `:memory:` for in-memory, or URL |
| `qdrant.url` | `str` | `null` | Qdrant server URL |
| `qdrant.port` | `int` | `6333` | REST API port |
| `qdrant.grpc_port` | `int` | `6334` | gRPC port |
| `qdrant.prefer_grpc` | `bool` | `false` | Prefer gRPC over REST |
| `qdrant.https` | `bool` | `null` | Use HTTPS |
| `qdrant.api_key` | `str` | `null` | API key for Qdrant Cloud |
| `qdrant.prefix` | `str` | `null` | REST URL path prefix |
| `qdrant.timeout` | `float` | `null` | Request timeout |
| `qdrant.host` | `str` | `null` | Hostname (defaults to localhost) |
| `qdrant.path` | `str` | `null` | Local persistence path |
| `qdrant.force_disable_check_same_thread` | `bool` | `true` | Disable SQLite thread check for local Qdrant |

#### PostgreSQL Settings (`postgres`)

| Key | Type | Default | Description |
|---|---|---|---|
| `postgres.host` | `str` | `localhost` | Database host |
| `postgres.port` | `int` | `5432` | Database port |
| `postgres.user` | `str` | `postgres` | Database user |
| `postgres.password` | `str` | `postgres` | Database password |
| `postgres.database` | `str` | `postgres` | Database name |
| `postgres.schema_name` | `str` | `public` | Schema name |

#### ClickHouse Settings (`clickhouse`)

| Key | Type | Default | Description |
|---|---|---|---|
| `clickhouse.host` | `str` | `localhost` | Server host |
| `clickhouse.port` | `int` | `8443` | Server port |
| `clickhouse.username` | `str` | `default` | Username |
| `clickhouse.password` | `str` | (empty) | Password |
| `clickhouse.database` | `str` | `__default__` | Database name |
| `clickhouse.secure` | `bool\|str` | `false` | Use HTTPS/TLS |
| `clickhouse.interface` | `str` | `null` | `http` or `https` |
| `clickhouse.settings` | `dict` | `null` | ClickHouse session settings |
| `clickhouse.connect_timeout` | `int` | `null` | Connection timeout |
| `clickhouse.send_receive_timeout` | `int` | `null` | Read timeout |
| `clickhouse.verify` | `bool` | `null` | Verify server certificate |
| `clickhouse.ca_cert` | `str` | `null` | CA root certificate path |
| `clickhouse.client_cert` | `str` | `null` | Client certificate path |
| `clickhouse.client_cert_key` | `str` | `null` | Client certificate key path |
| `clickhouse.http_proxy` | `str` | `null` | HTTP proxy |
| `clickhouse.https_proxy` | `str` | `null` | HTTPS proxy |
| `clickhouse.server_host_name` | `str` | `null` | TLS server hostname |

#### Milvus Settings (`milvus`)

| Key | Type | Default | Description |
|---|---|---|---|
| `milvus.uri` | `str` | `local_data/private_gpt/milvus/milvus_local.db` | Milvus URI |
| `milvus.token` | `str` | (empty) | Access token |
| `milvus.collection_name` | `str` | `make_this_parameterizable_per_api_call` | Collection name |
| `milvus.overwrite` | `bool` | `true` | Overwrite existing collection schema |

### Configuration Files

| File | Format | Location | Purpose |
|---|---|---|---|
| `settings.yaml` | YAML | Project root | Base/default configuration |
| `settings-{profile}.yaml` | YAML | Project root | Profile overrides |
| `pyproject.toml` | TOML | Project root | Poetry dependencies and extras |
| `.pre-commit-config.yaml` | YAML | Project root | Pre-commit hooks |
| `docker-compose.yaml` | YAML | Project root | Docker Compose stack |
| `Dockerfile.llamacpp-cpu` | Dockerfile | Project root | llama.cpp CPU Docker image |
| `Dockerfile.ollama` | Dockerfile | Project root | Ollama Docker image |

## 4. Deployment & Scaling

### Deployment Options

#### Direct Python

```bash
PGPT_PROFILES=ollama python -m private_gpt
```

The entry point (`private_gpt/__main__.py`) starts uvicorn bound to `0.0.0.0` on the configured port with custom log configuration.

> [!NOTE]
> Uvicorn's default logging is disabled (`log_config=None`) in favor of PrivateGPT's own logging format: `HH:MM:SS.mmm [LEVEL   ] module_name - message`.

#### Docker Compose

The `docker-compose.yaml` and Docker profile (`settings-docker.yaml`) configure:
- Port: `8080` (via `${PORT:8080}`)
- LLM/embedding modes: configurable via env vars (`PGPT_MODE`, `PGPT_EMBED_MODE`)
- Ollama auto-pull: enabled by default

```bash
docker compose up -d
```

#### Docker with Ollama

```bash
docker build -t private-gpt -f Dockerfile.ollama .
docker run -p 8001:8080 \
  -e PGPT_OLLAMA_API_BASE=http://host.docker.internal:11434 \
  private-gpt
```

#### Docker with llama.cpp (CPU)

```bash
docker build -t private-gpt-cpu -f Dockerfile.llamacpp-cpu .
docker run -p 8001:8080 private-gpt-cpu
```

### Scaling Considerations

> [!TIP]
> For production deployments handling multiple concurrent users, consider:
> - Using a client-server vector store (Qdrant server, PostgreSQL) instead of local file-based storage
> - Running the Ollama service on a separate host with GPU acceleration
> - Setting `embedding.ingest_mode: parallel` with appropriate `count_workers` for bulk ingestion
> - Placing a reverse proxy (nginx, Traefik) in front of PrivateGPT for TLS termination

The FastAPI application is single-process by default. For horizontal scaling:
1. Use an external vector store that supports concurrent connections
2. Run multiple uvicorn workers or deploy behind a load balancer
3. Ensure the Ollama/LLM backend can handle concurrent requests

## 5. Operations

### Health Check

```bash
curl http://localhost:8001/health
```

```json
{"status": "ok"}
```

> [!TIP]
> Use the `/health` endpoint for container orchestration liveness probes. It returns HTTP 200 when the FastAPI application is running and accepting connections.

### Monitoring

PrivateGPT uses Python's standard `logging` module at `INFO` level by default. The log format is:

```text
HH:MM:SS.mmm [LEVEL   ] module_name - message
```

Set `ROOT_LOG_LEVEL = "DEBUG"` in `private_gpt/__init__.py` for verbose output including library logs.

LlamaIndex observability is enabled via the `simple` global handler in `launcher.py`, which provides callback-based tracing of LLM calls, embedding operations, and retrieval steps.

### Ollama Model Management

When `ollama.autopull_models` is `true`, PrivateGPT automatically pulls missing models at startup. The pull process:
1. Checks installed models via `client.list()`
2. If the configured model is not found, pulls with streaming progress bars
3. Retries up to 5 times with jitter (3–10 seconds) on connection errors

> [!NOTE]
> The Ollama connection check uses a retry decorator with 5 attempts and randomized jitter between 3.0 and 10.0 seconds to handle slow Ollama startups.

### Local Data Management

Default local data paths (derived from `data.local_data_folder`):
- `local_data/private_gpt/` — main data directory
- `local_data/private_gpt/qdrant/` — Qdrant local persistence (when using local Qdrant)
- `local_data/private_gpt/milvus/` — Milvus local persistence
- `models/` — downloaded model files
- `models/cache/` — model cache

### Backup and Recovery

For local Qdrant deployments, back up the `local_data/` directory. For PostgreSQL deployments, use standard `pg_dump`. Vector store data and node store data must be backed up together to maintain consistency.

## 6. Performance

### Ingestion Mode Comparison

| Mode | Behavior | Best For |
|---|---|---|
| `simple` | Sequential, one file at a time | Small datasets, debugging |
| `batch` | Parse in parallel, embed in batch | Mixed file sizes |
| `parallel` | Parse and embed in parallel across workers | Local setups with multi-core CPU |
| `pipeline` | Continuous embedding pipeline | High-throughput ingestion |

The `count_workers` setting (default: `2`) controls parallelism for non-simple modes. Optimal values depend on CPU threads and available RAM.

> [!WARNING]
> Setting `count_workers` too high in `parallel` mode can cause memory exhaustion, especially with large embedding models. Stay at or below your CPU thread count.

### LLM Tuning Parameters

| Parameter | Effect | Default |
|---|---|---|
| `temperature` | Higher = more creative, lower = more factual | `0.1` |
| `max_new_tokens` | Maximum response length | `256` (base), `512` (ollama/local profiles) |
| `context_window` | Maximum input context | `3900` |
| `top_k` | Vocabulary diversity filter | `40` |
| `top_p` | Nucleus sampling threshold | `0.9` |
| `repeat_penalty` | Repetition avoidance strength | `1.1` |
| `tfs_z` | Tail free sampling (1.0 = disabled) | `1.0` |

### RAG Tuning

- `rag.similarity_top_k`: Number of chunks retrieved before reranking (default: `2`)
- `rag.similarity_value`: Minimum similarity score filter — disabled by default; set to `0.45`–`0.7` to filter low-relevance results
- Reranking: Enable for better precision at the cost of additional compute (loads a cross-encoder model)

## 7. Security

### Authentication

Authentication is implemented in `private_gpt/server/utils/auth.py` as a FastAPI dependency:

- When `server.auth.enabled` is `false` (default): a dummy dependency always returns `True`
- When enabled: validates the `Authorization` header against `server.auth.secret` using `secrets.compare_digest()` (constant-time comparison)

Failed authentication returns HTTP 401 with a `WWW-Authenticate: Basic realm="All the API"` header.

> [!CAUTION]
> The default `settings.yaml` contains a hardcoded auth secret: `Basic c2VjcmV0OmtleQ==` (which decodes to `secret:key`). This is a known security issue (#2204). Always change this value and enable authentication before any network exposure.

### CORS Configuration

CORS is handled via FastAPI's `CORSMiddleware`, configured under `server.cors`. The default `settings.yaml` enables CORS with permissive settings:

```yaml
cors:
  enabled: true
  allow_origins: ["*"]
  allow_methods: ["*"]
  allow_headers: ["*"]
```

> [!WARNING]
> Wildcard CORS origins allow any website to make API requests to your PrivateGPT instance. Restrict `allow_origins` to your specific frontend domains in production.

### Data Privacy

- **No telemetry:** Gradio analytics is explicitly disabled (`GRADIO_ANALYTICS_ENABLED=False`)
- **No ChromaDB telemetry:** Anonymized telemetry is disabled
- **Offline tokenization:** The `tiktoken_cache/` directory is bundled for offline token counting
- **Local-only processing:** When using local LLM and embedding backends, no data leaves the host
- **Local ingestion control:** The `data.local_ingestion.allow_ingest_from` setting restricts which filesystem paths can be ingested

### Security Recommendations

- [ ] Change the default auth secret before deployment
- [ ] Enable authentication (`server.auth.enabled: true`)
- [ ] Restrict CORS origins to known frontends
- [ ] Use TLS termination via a reverse proxy
- [ ] Restrict `local_ingestion.allow_ingest_from` to specific directories
- [ ] Review `huggingface.trust_remote_code` — disable if not required by your model
- [ ] Consider integrity and malware checks for local embeddings (#2201)

## 8. Developer Internals

### Repository Structure

```text
private-gpt/
├── private_gpt/                  # Main application package
│   ├── __init__.py               # Logging setup, telemetry disabling
│   ├── __main__.py               # Uvicorn entry point
│   ├── main.py                   # App singleton
│   ├── launcher.py               # FastAPI app factory
│   ├── di.py                     # Dependency injection container
│   ├── constants.py              # PROJECT_ROOT_PATH
│   ├── paths.py                  # Derived filesystem paths
│   ├── components/               # Pluggable backend components
│   │   ├── embedding/            # Embedding backends
│   │   │   └── custom/           # SageMaker custom embedding
│   │   ├── ingest/               # Document ingestion
│   │   ├── llm/                  # LLM backends
│   │   │   ├── prompt_helper.py  # Prompt formatting
│   │   │   └── custom/           # SageMaker custom LLM
│   │   ├── node_store/           # Node storage backends
│   │   └── vector_store/         # Vector storage backends
│   │       └── batched_chroma.py # Batched Chroma operations
│   ├── open_ai/                  # OpenAI-compatible models
│   │   ├── openai_models.py      # Request/response Pydantic models
│   │   └── extensions/
│   │       └── context_filter.py # Document ID filtering
│   ├── server/                   # FastAPI routers and services
│   │   ├── chat/                 # Chat completions
│   │   ├── chunks/               # Chunk retrieval
│   │   ├── completions/          # Text completions
│   │   ├── embeddings/           # Embedding generation
│   │   ├── health/               # Health check
│   │   ├── ingest/               # Document ingestion + watcher
│   │   ├── recipes/summarize/    # Summarization recipe
│   │   └── utils/auth.py         # Authentication
│   ├── settings/                 # Configuration system
│   │   ├── settings.py           # Pydantic settings models
│   │   ├── settings_loader.py    # Profile loading and merging
│   │   └── yaml.py               # YAML loader with env var substitution
│   ├── ui/                       # Gradio web interface
│   │   ├── ui.py                 # UI logic (4 modes, file management)
│   │   ├── images.py             # Logo SVG data
│   │   └── avatar-bot.ico        # Bot avatar
│   └── utils/                    # Utilities
│       ├── eta.py                # ETA calculator with Pearson regression
│       ├── ollama.py             # Ollama connection, model pulling
│       ├── retry.py              # Retry decorator wrapper
│       └── typing.py             # TypeVar definitions
├── fern/                         # API documentation (Fern)
│   ├── docs.yml                  # Fern docs configuration
│   ├── fern.config.json          # Fern project config
│   ├── generators.yml            # SDK generator config
│   ├── openapi/openapi.json      # OpenAPI 3.1 spec
│   └── docs/                     # Documentation pages
├── scripts/                      # Utility scripts
│   ├── extract_openapi.py        # Export OpenAPI spec
│   ├── ingest_folder.py          # Bulk folder ingestion
│   └── setup                     # Setup script
├── tests/                        # Test suite
│   ├── conftest.py               # Shared fixtures
│   ├── fixtures/                 # Test fixtures (mock injector, etc.)
│   ├── server/                   # Router tests
│   ├── settings/                 # Settings tests
│   └── ui/                       # UI tests
├── settings.yaml                 # Default settings
├── settings-*.yaml               # Profile settings
├── docker-compose.yaml           # Docker stack
├── Dockerfile.llamacpp-cpu       # llama.cpp CPU image
├── Dockerfile.ollama             # Ollama image
├── Makefile                      # Build/test targets
└── pyproject.toml                # Poetry project definition
```

### Dependency Injection Architecture

PrivateGPT uses the `injector` library for dependency injection. The DI container is created in `private_gpt/di.py`:

```python
def create_application_injector() -> Injector:
    _injector = Injector(auto_bind=True)
    _injector.binder.bind(Settings, to=unsafe_typed_settings)
    return _injector
```

Key behaviors:
- `auto_bind=True` allows automatic resolution of dependencies not explicitly bound
- `Settings` is bound as a singleton to the loaded settings object
- The global injector is created at module import time
- Each FastAPI request gets the injector via `request.state.injector`
- Service classes (`ChatService`, `IngestService`, etc.) use `@inject` decorators

> [!NOTE]
> The `global_injector` is a module-level singleton. For testing, use `MockInjector` to override bindings. The test fixtures in `tests/fixtures/mock_injector.py` demonstrate this pattern.

### Application Startup Sequence

1. `private_gpt/__init__.py` — configures logging (`INFO` level), disables Gradio analytics and ChromaDB telemetry, sets tiktoken cache path
2. `private_gpt/settings/settings_loader.py` — loads active profiles, merges settings
3. `private_gpt/settings/settings.py` — validates merged settings against Pydantic models
4. `private_gpt/di.py` — creates the global `Injector` and binds `Settings`
5. `private_gpt/launcher.py` — `create_app()` creates the FastAPI app:
   - Registers `bind_injector_to_request` dependency
   - Includes all routers (completions, chat, chunks, ingest, summarize, embeddings, health)
   - Sets up LlamaIndex `simple` observability handler
   - Configures CORS middleware if enabled
   - Mounts Gradio UI if `ui.enabled`
6. `private_gpt/__main__.py` — starts uvicorn on `0.0.0.0:{port}`

### API Endpoints (Complete Reference)

| Method | Path | Auth | Description |
|---|---|---|---|
| `POST` | `/v1/completions` | Yes | Text completion with optional context |
| `POST` | `/v1/chat/completions` | Yes | Chat completion with optional context |
| `POST` | `/v1/chunks` | Yes | Retrieve relevant document chunks |
| `POST` | `/v1/ingest` | Yes | Ingest file (deprecated — use `/v1/ingest/file`) |
| `POST` | `/v1/ingest/file` | Yes | Ingest and process a file |
| `POST` | `/v1/ingest/text` | Yes | Ingest raw text |
| `GET` | `/v1/ingest/list` | Yes | List all ingested documents |
| `DELETE` | `/v1/ingest/{doc_id}` | Yes | Delete an ingested document |
| `POST` | `/v1/summarize` | Yes | Summarize text or documents |
| `POST` | `/v1/embeddings` | Yes | Generate embeddings for input text |
| `GET` | `/health` | No | System health check |

"Auth: Yes" means the endpoint respects the `authenticated` dependency — when `server.auth.enabled` is `false`, authentication is effectively a no-op.

### OpenAI Compatibility Layer

The `private_gpt/open_ai/openai_models.py` module defines Pydantic models that mirror the OpenAI API response format:

| Class | Purpose |
|---|---|
| `OpenAIDelta` | Streaming chunk content |
| `OpenAIMessage` | Message with role (`assistant`/`system`/`user`) and content |
| `OpenAIChoice` | Response choice (either `delta` for streaming or `message` for non-streaming) |
| `OpenAICompletion` | Top-level response with `id`, `object`, `created`, `model`, `choices` |

Helper functions:
- `to_openai_response()` — converts a LlamaIndex response to `OpenAICompletion`
- `to_openai_sse_stream()` — converts a LlamaIndex streaming generator to SSE format with `data:` prefixed JSON chunks, terminated by `data: [DONE]`

### Context Filtering

The `ContextFilter` model (`private_gpt/open_ai/extensions/context_filter.py`) allows filtering RAG queries to specific documents by passing their `doc_ids`. This is used by both the API and the Gradio UI when a specific file is selected.

### Gradio UI Internals

`PrivateGptUi` (`private_gpt/ui/ui.py`) is a `@singleton` class that:
- Receives services via `@inject`: `IngestService`, `ChatService`, `ChunksService`, `SummarizeService`
- Builds a Gradio `Blocks` interface with a mode radio, file management sidebar, system prompt input, and chat interface
- Dispatches to different service methods based on the selected `Modes` enum (`RAG`, `Search`, `Basic`, `Summarize`)
- Limits chat history to 20 messages to prevent context overflow
- Strips source separators from history to keep context clean
- Mounts into FastAPI via `gr.mount_gradio_app()`

The UI tab title is hardcoded as `"My Private GPT"`.

### ETA Calculator

`private_gpt/utils/eta.py` provides an `ETA` class that predicts completion time using Pearson product-moment correlation coefficient on timing data. It maintains a sliding window of 100 samples and reports progress in human-readable format (e.g., `"2m 30s @ 15/min"`).

### Retry Mechanism

`private_gpt/utils/retry.py` wraps the `retry-async` library's `retry` decorator with typed parameters. Used for Ollama connection retries with configurable:
- `tries`: max attempts (5 for Ollama)
- `jitter`: random delay range (3.0–10.0 seconds for Ollama)
- `exceptions`: exception types to catch
- `backoff`: multiplicative backoff factor

### Module Reference

#### `private_gpt/settings/settings.py`

The central settings module defines 27 Pydantic `BaseModel` classes representing every configurable aspect of the system. The top-level `Settings` model aggregates all sub-models. Settings are loaded once at import time via `unsafe_typed_settings` and made available through DI or the `settings()` function.

#### `private_gpt/launcher.py`

The `create_app()` factory function is the single place where all routers are registered, middleware is configured, and the UI is mounted. It takes an `Injector` and returns a fully configured `FastAPI` application.

#### `private_gpt/ui/ui.py`

At 598 lines, this is the largest module. It implements the complete Gradio interface including file upload/delete, 4 chat modes, streaming response rendering with source attribution, system prompt management, and model label display.

#### `private_gpt/open_ai/openai_models.py`

Implements the OpenAI-compatible response format. The `OpenAICompletion` class provides `from_text()` for single responses and `json_from_delta()` for streaming chunks.

#### `private_gpt/server/utils/auth.py`

Implements a switchable authentication pattern: at module load time, it checks `settings().server.auth.enabled` and defines either a no-op `authenticated()` function or one that validates the `Authorization` header via `secrets.compare_digest()`.

### Code Patterns and Conventions

- **Dependency injection everywhere:** Services are `@inject`-decorated, DI container resolves all dependencies
- **Pydantic for all data:** Settings, API request/response bodies, and internal models use Pydantic `BaseModel`
- **Profile-based configuration:** No hardcoded values — everything flows through the YAML profile system
- **Module-level initialization:** Settings are loaded and validated at import time, before the server starts
- **Match statements:** Python 3.10+ `match`/`case` used for mode dispatching in the UI
- **Type hints throughout:** Full type annotations with `mypy` checking
- **Pre-commit hooks:** Black (formatting), Ruff (linting), mypy (types), pytest (tests on push)

### Testing

Tests use `pytest` with custom fixtures:

| Fixture | Purpose |
|---|---|
| `test_client` | FastAPI `TestClient` (from `tests/fixtures/fast_api_test_client.py`) |
| `MockInjector` | DI container with overridable bindings |
| `auto_close_qdrant` | Qdrant cleanup fixture |
| `ingest_helper` | Test file ingestion |

The `test` profile (`settings-test.yaml`) configures:
- Auth disabled with dummy secret `"foo bar; dummy secret"`
- Mock LLM and embedding modes
- Local data stored in `local_data/tests`
- UI disabled

Test coverage includes:
- Authentication (default no-auth and enabled basic auth)
- Settings loading, merging, and env var substitution
- Summarize routes (streaming, single, with/without context)
- UI endpoint mounting

### Potential Improvements

- Per-user document isolation (#1796) — currently all ingested documents are global
- Integrity and malware checks for local embeddings (#2201)
- Additional LLM backend support (Claude requested in #2199)
- OceanBase vector storage support (#2214)
- Session-based ingestion with stop-and-start capability (#2190, #1914)
- Bulk ingestion from Dockerfile with `--watch` support (#1914)

### Contributing

The project uses:

- **Poetry** for dependency management
- **Black** for code formatting
- **Ruff** for linting
- **mypy** for type checking
- **pytest** for testing
- **pre-commit** for automated hooks (format on commit, test on push)

Setup the development environment:

```bash
git clone https://github.com/zylon-ai/private-gpt
cd private-gpt
poetry install --all-extras
pre-commit install
```

Run checks:

```bash
make test    # Run pytest
make mypy    # Run type checking
```

The API documentation is generated with Fern from the OpenAPI spec at `fern/openapi/openapi.json`. The Python SDK is generated via `fernapi/fern-python-sdk` (version 0.6.2).

## 9. Integration Points

### OpenAI SDK Compatibility

PrivateGPT's API is compatible with OpenAI's client libraries. Point the OpenAI SDK at your PrivateGPT instance:

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8001/v1", api_key="not-needed")
response = client.chat.completions.create(
    model="private-gpt",
    messages=[{"role": "user", "content": "Hello"}]
)
```

### Fern SDK

An official Python SDK is generated from the OpenAPI spec using Fern. The SDK output path is `../../pgpt-sdk/python` relative to the `fern/` directory. The Fern documentation is hosted at `docs.privategpt.dev`.

### Ingestion Watcher

The `private_gpt/server/ingest/ingest_watcher.py` module provides filesystem watching capability for automatic ingestion of new files placed in monitored directories. This requires `data.local_ingestion.enabled: true`.

### LlamaIndex Ecosystem

PrivateGPT builds on LlamaIndex for:
- Document parsing and chunking
- Embedding model abstraction
- Vector store abstraction
- Query engine and chat engine
- Callback-based observability

All LlamaIndex components are accessed through PrivateGPT's component wrappers in `private_gpt/components/`, which handle configuration injection and backend selection.

---

_Generated May 2026 from zylon-ai/private-gpt at 7effd57. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._