# Pathway LLM App Templates User Guide

`pathwaycom/llm-app` is a **catalogue of ten production-shape reference applications** that sit on top of the [Pathway](https://pathway.com/) streaming-dataflow engine. It is not a library, not a single app, and not a framework — it is a curated set of working RAG (Retrieval-Augmented Generation), document-indexing, and alerting services, each one already wired with a Dockerfile, an `app.yaml` configuration, sample data, and (for most templates) a Streamlit UI. Pick the template that matches your use case, point it at your documents, set your API keys, and run.

This guide walks you through choosing, configuring, and running the templates. For architecture, deployment hardening, and the shared App-class internals, see the **Pathway LLM App Templates Administrator & Developer Guide**.

## 1. Quick Start

### What this repository ships

- **8 deployable templates** under `templates/` — each a Pathway pipeline with a REST or MCP server.
- **2 Jupyter cookbooks** under `cookbooks/self-rag-agents/` — interactive tutorials for self-RAG agentic workflows with LangGraph; these are *learning* notebooks, not deployable apps.
- **Per-template `Dockerfile`, `docker-compose.yml`, `app.yaml`, and seed data** so each template runs end-to-end in one command.
- **Streamlit UIs** for four of the templates (`question_answering_rag/ui`, `slides_ai_search/ui`, `drive_alert/ui`, `unstructured_to_sql_on_the_fly/ui`) so you can chat with your data in a browser.

### Key features

- Real-time, always-fresh document indexes — Pathway watches your data sources and updates the vector index as files change.
- Pluggable LLM, embedder, parser, splitter, and retriever — swap any component by editing one YAML block.
- Built-in connectors for the local filesystem, Google Drive, and Microsoft SharePoint (configured via YAML, no code changes).
- Cloud or fully-private deployment — drop `OpenAIChat` for `LiteLLMChat` pointed at Ollama and run with zero outbound calls.
- REST API and Model Context Protocol (MCP) servers exposed by the templates' Pathway xpacks.
- Optional persistent caching so a restart re-uses already-computed embeddings.

### Prerequisites

Docker 24+ with the Compose plugin (or Python 3.10+ with `pip` if you prefer to run templates natively). An OpenAI API key for templates that default to GPT — or local Ollama for the private template.

### One-command Quick Start (the canonical template)

The reference template — the one whose shape every other App-class template mirrors — is `question_answering_rag`. Run it like this:

```bash
git clone https://github.com/pathwaycom/llm-app.git
cd llm-app/templates/question_answering_rag

# Provide your OpenAI key
echo "OPENAI_API_KEY=sk-..." > .env

# Build and run with the bundled compose file
docker compose up --build
```

```text
[+] Building 78.4s (12/12) FINISHED
[+] Running 2/2
 ✔ Container question_answering_rag-app-1  Started
 ✔ Container question_answering_rag-ui-1   Started
2026-01-07 12:00:00 pathway INFO Indexed 1 documents
2026-01-07 12:00:01 streamlit INFO Running on http://0.0.0.0:8501
```

Open `http://localhost:8501` in a browser. You will see the Streamlit RAG UI with the bundled sample PDF (`IdeanomicsInc_..._Content License Agreement.pdf`) already indexed. Ask a question; the answer comes back with the source documents that were used as context.

> [!TIP]
> You can hit the Pathway REST API directly without the Streamlit UI:
> `curl -X POST http://localhost:8000/v2/answer -H "Content-Type: application/json" -d '{"prompt": "What is this document about?"}'`. Every App-class template exposes the same endpoint shape on port 8000 by default.

### What to try next

- **Pick a different template** — see the selector table in section 4 to choose by use case.
- **Index your own documents** — drop PDFs into `templates/question_answering_rag/data/` and the index updates automatically (no restart).
- **Switch to local LLMs** — try `templates/private_rag/` to run with Ollama instead of OpenAI.
- **Read the per-template README** — every `templates/<name>/README.md` documents that specific template in depth.

## 2. Installation

### System requirements

| Component | Minimum | Recommended | Notes |
|---|---|---|---|
| Docker | 24.0 | 26.0+ | Compose plugin required (`docker compose`, not legacy `docker-compose`) |
| RAM | 4 GB | 8+ GB | Multimodal/Slides templates push higher; private RAG with Ollama needs more |
| Disk | 2 GB | 10+ GB | For Docker images, document cache, and Pathway persistence directory |
| Python (native run) | 3.10 | 3.11 | Only needed if you skip Docker; templates use `pathway[all]` |
| OpenAI account | API key | Project key | Default for all templates except `private_rag` |

> [!IMPORTANT]
> The templates default to OpenAI models (`gpt-4.1-mini`, `gpt-4o`, `text-embedding-3-small`) and will fail to start without `OPENAI_API_KEY` unless you switch the LLM block in `app.yaml` to a local provider. Keep the key in `.env` next to `app.yaml`; never commit it.

### Install methods

=== "Docker Compose (recommended)"
    ```bash
    git clone https://github.com/pathwaycom/llm-app.git
    cd llm-app/templates/<template_name>
    cp .env.example .env  # (when present) — fill in your secrets
    docker compose up --build
    ```

=== "Native Python"
    ```bash
    git clone https://github.com/pathwaycom/llm-app.git
    cd llm-app/templates/<template_name>
    python -m venv .venv && source .venv/bin/activate
    pip install pathway[all] python-dotenv
    pip install -r requirements.txt  # adds template-specific extras
    cp .env.example .env  # when present
    python app.py
    ```

=== "Single Dockerfile"
    ```bash
    cd llm-app/templates/<template_name>
    docker build -t pathway-llm-app .
    docker run --rm -p 8000:8000 \
      -v "$PWD/data:/app/data" \
      --env-file .env \
      pathway-llm-app
    ```

### Pathway license key

Each template's `app.py` calls `pw.set_license_key("demo-license-key-with-telemetry")` so the demo runs out of the box with anonymous telemetry. To unlock advanced features (multi-machine clustering via `--addresses`, autoscaling, certain enterprise persistence backends), get a free Scale key from <https://pathway.com/features> and either edit the line in `app.py` or set `PATHWAY_LICENSE_KEY` in `.env` and replace the call with `pw.set_license_key(os.environ["PATHWAY_LICENSE_KEY"])`. Comment the call out entirely to use Pathway Community (no telemetry, no advanced features).

> [!NOTE]
> The demo license key is real and works — it just enables anonymous usage telemetry. There is no quota or expiration. Switch to a personal Scale key if your environment cannot send telemetry, or remove the call to opt out entirely.

### Post-install verification

After `docker compose up --build` completes for `question_answering_rag`:

```bash
# REST API: ask a question
curl -X POST http://localhost:8000/v2/answer \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Summarise the document"}'

# REST API: list indexed documents
curl http://localhost:8000/v2/list_documents

# REST API: retrieve top-k chunks for a query (no LLM call)
curl -X POST http://localhost:8000/v1/retrieve \
  -H "Content-Type: application/json" \
  -d '{"query": "license agreement", "k": 4}'
```

```text
{"response": "This document is a content license agreement between..."}
```

If you see a JSON response, the pipeline is live, the documents are indexed, and your API key works.

### Install troubleshooting

| Symptom | Likely cause | Fix |
|---|---|---|
| `RuntimeError: OpenAI API key not provided` | `.env` not loaded, file missing, wrong key | Confirm `.env` sits next to `app.py`; key starts with `sk-`; restart container |
| `Cannot connect to API on localhost:8000` | Container not yet ready, port collision | Wait 30-90 s for first index build; check `docker ps` for port mapping |
| `pathway.set_license_key: invalid key` | Edited `app.py` and pasted a malformed key | Use the demo key string verbatim, or unset and rely on Community mode |
| `cuda not available` (private_rag, document_indexing) | SentenceTransformer falling back to CPU | Acceptable; embeddings just take longer. Add a GPU-enabled image for production |
| `pulling image takes forever` on first build | Pathway base image (`pathwaycom/pathway:latest`) is large | Normal first-pull; subsequent builds reuse the layer |
| `Connection refused: ollama:11434` (private_rag) | Ollama service not started or unreachable | Start `ollama serve` on the host and use `host.docker.internal:11434` from a Mac/Windows container |
| `No documents found, index empty` | Data directory empty or path mismatch | Drop PDFs into the template's `data/` (or `files-for-indexing/`) folder; Pathway picks them up live |
| `psycopg.OperationalError: connection refused` (unstructured_to_sql_on_the_fly) | Postgres service not running | This template needs a running Postgres; use the included `docker-compose.yml` instead of `docker run` |
| `KeyError: 'SHAREPOINT_URL'` | Switched on a SharePoint connector without filling its env vars | See section 3.3 for the full SharePoint variable inventory |
| `Slack 401 unauthorized` (drive_alert) | `SLACK_ALERT_TOKEN` missing or revoked | Re-issue the bot token; Slack tokens start with `xoxb-` |
| Streamlit UI shows "Connection error" | Pathway app container not ready or `PATHWAY_HOST`/`PATHWAY_PORT` mismatch | When using compose, the UI defaults `PATHWAY_HOST=app` (the service name); native runs need `localhost` |

## 3. Configuration

### YAML-first, environment-second

Every modern template is configured by exactly one file: `app.yaml` next to `app.py`. The YAML uses Pathway's `!pw.<class>` tag syntax to instantiate Python objects directly. `$variables` (anchors) let you define an embedder once and reference it from both the parser and the retriever. The two legacy templates (`drive_alert` and `unstructured_to_sql_on_the_fly`) instead use procedural Python kwargs and `os.environ.get` defaults — they predate the App-class refactor.

### Annotated minimal config (Question-Answering RAG)

The smallest meaningful `app.yaml`:

```yaml
# Where to read documents from
$sources:
  - !pw.io.fs.read
    path: data
    format: binary
    with_metadata: true

# The chat model
$llm: !pw.xpacks.llm.llms.OpenAIChat
  model: "gpt-4.1-mini"
  cache_strategy: !pw.udfs.DefaultCache {}
  temperature: 0
  capacity: 8
  async_mode: "fully_async"

# The embedding model
$embedder: !pw.xpacks.llm.embedders.OpenAIEmbedder
  model: "text-embedding-3-small"
  cache_strategy: !pw.udfs.DefaultCache {}

# Chunking + parsing
$splitter: !pw.xpacks.llm.splitters.TokenCountSplitter
  max_tokens: 400
$parser: !pw.xpacks.llm.parsers.DoclingParser
  table_parsing_strategy: "llm"
  async_mode: "fully_async"
  chunk: false
  cache_strategy: !pw.udfs.DefaultCache {}

# Vector index
$retriever_factory: !pw.indexing.UsearchKnnFactory
  reserved_space: 1000
  embedder: $embedder
  metric: !pw.indexing.USearchMetricKind.COS

# The actual document store + question-answerer wired together
$document_store: !pw.xpacks.llm.document_store.DocumentStore
  docs: $sources
  parser: $parser
  splitter: $splitter
  retriever_factory: $retriever_factory

question_answerer: !pw.xpacks.llm.question_answering.BaseRAGQuestionAnswerer
  llm: $llm
  indexer: $document_store
```

Every YAML field maps directly to a Pydantic field on `App` in `app.py` — `extra="forbid"` is set, so a typo in a key fails fast at startup with a Pydantic validation error.

> [!TIP]
> The `$` prefix is a YAML anchor: `$llm:` defines, `llm: $llm` references. This lets one component (e.g. `$embedder`) be reused by both the parser and retriever without duplication. Only fields that are *not* prefixed with `$` (top-level: `question_answerer`, `host`, `port`, `persistence_mode`, `terminate_on_error`) are read by the App-class itself.

### Common option combinations

| Goal | Edit |
|---|---|
| Use a different OpenAI model | Change `model:` under `$llm:` (e.g. `"gpt-4o"` or `"gpt-4o-mini"`) |
| Switch to a local LLM | Replace `$llm` block with `!pw.xpacks.llm.llms.LiteLLMChat` and set `model: "ollama/<name>"` and `api_base: "http://host.docker.internal:11434"` |
| Add SharePoint as a source | Uncomment the `!pw.xpacks.connectors.sharepoint.read` block under `$sources:` and fill in tenant + cert |
| Add Google Drive as a source | Uncomment the `!pw.io.gdrive.read` block under `$sources:` and place `gdrive_indexer.json` next to `app.yaml` |
| Persist the cache across restarts | Set `persistence_mode: !pw.PersistenceMode.PERSISTING` at the top of `app.yaml` |
| Move the cache to a custom path | Add `persistence_backend: !pw.persistence.Backend.filesystem` with `path: "/data/cache"` |
| Halt on first error | Set `terminate_on_error: true` (default false) |
| Change listening port | Set `port: 9000` (default 8000 for App-class; 8080 for legacy) |
| Customise the RAG prompt | Add `prompt_template: "Given these documents: {context}, please answer the question: {query}"` under `question_answerer:` |
| Cap the number of context docs | Add `search_topk: 6` under `question_answerer:` |

### Environment variables you typically set

These are referenced inside `app.yaml` (with `$VARIABLE`) or by `os.environ.get` in legacy templates:

| Variable | Used by | Default | Purpose |
|---|---|---|---|
| `OPENAI_API_KEY` | All OpenAI-backed templates | — | OpenAI auth |
| `PATHWAY_LICENSE_KEY` | App, when read at runtime | demo key | Unlock Scale features |
| `PATHWAY_HOST` | Streamlit UIs | `app` (compose) / `localhost` | Where the UI reaches the Pathway app |
| `PATHWAY_PORT` | Streamlit UIs | `8000` | Port for the Pathway REST API |
| `PATHWAY_REST_CONNECTOR_HOST` | Legacy templates | `0.0.0.0` | Bind address for legacy REST connector |
| `PATHWAY_REST_CONNECTOR_PORT` | Legacy templates | `8080` | Port for legacy REST connector |
| `FILE_OR_DIRECTORY_ID` | `drive_alert` | — | Google Drive folder/file ID to monitor |
| `GOOGLE_CREDS` | `drive_alert` | `templates/drive_alert/secrets.json` | Path to GCP service-account JSON |
| `SLACK_ALERT_CHANNEL_ID` | `drive_alert` | — | Slack channel for change alerts |
| `SLACK_ALERT_TOKEN` | `drive_alert` | — | Slack bot token (`xoxb-...`) |
| `SHAREPOINT_URL` / `SHAREPOINT_TENANT` / `SHAREPOINT_CLIENT_ID` / `SHAREPOINT_THUMBPRINT` / `SHAREPOINT_ROOT` | Templates with SharePoint connector enabled | — | Microsoft SharePoint app-only auth |
| `DRIVE_ID` | Templates with Google Drive connector enabled | — | Google Drive folder ID |
| `PATHWAY_DATA_DIR` | `unstructured_to_sql_on_the_fly` | `./data/quarterly_earnings` | Where to read source PDFs |
| `POSTGRESQL_HOST` / `POSTGRESQL_PORT` / `POSTGRESQL_DB` / `POSTGRESQL_USER` / `POSTGRESQL_PASSWORD` / `POSTGRESQL_TABLE` | `unstructured_to_sql_on_the_fly` | `localhost` / `5432` / `STRUCTUREDDB` / `user` / `password` / `quarterly_earnings` | Target Postgres for structured output |
| `FILE_SERVER_URL` | `slides_ai_search/ui` | `http://localhost:8080/` | nginx file server inside the slides stack |

> [!NOTE]
> Legacy templates (`drive_alert`, `unstructured_to_sql_on_the_fly`) get every parameter from `os.environ.get` calls inside `app.py`. Modern App-class templates pull values from `app.yaml`, so the equivalent of "setting an env var" there is editing the YAML.

For deeper config — including persistence-backend tuning, retry strategies, and connector-specific knobs — see the **Pathway LLM App Templates Administrator & Developer Guide**, section 3.

## 4. Core Usage

### 4.1 Pick the template that fits your problem

This is the most important table in the guide. Each template is a complete, self-contained service with its own folder, README, Dockerfile, and YAML. Pick by use case, not by name.

| Template | Best when you need… | Default LLM | Default embedder | Default port | UI? | Connector(s) used |
|---|---|---|---|---|---|---|
| `question_answering_rag` | A general-purpose RAG over PDFs/PPTX with a Streamlit chat UI — the canonical reference | `gpt-4.1-mini` | `text-embedding-3-small` | 8000 | yes | filesystem (default), SharePoint/GDrive optional |
| `adaptive_rag` | RAG that retrieves *as few documents as possible* per question to save tokens (Pathway's adaptive retrieval algorithm) | `gpt-4.1-mini` | `text-embedding-3-small` | 8000 | no | filesystem (default), SharePoint/GDrive optional |
| `private_rag` | Fully air-gapped RAG — every model runs locally via Ollama + sentence-transformers, no outbound calls | `ollama/mistral` | `avsolatorio/GIST-small-Embedding-v0` | 8000 | no | filesystem (default), SharePoint/GDrive optional |
| `multimodal_rag` | RAG over PDFs whose tables and images carry meaning — uses GPT-4o to parse images/tables | `gpt-4.1-mini` (answers) + `gpt-4o` (image parsing) | `text-embedding-3-small` | 8000 | no | filesystem (default), SharePoint/GDrive optional |
| `document_indexing` | Just an indexing pipeline + retrieval API — no LLM, no answering, just a vector store you can query yourself | none (no LLM) | `mixedbread-ai/mxbai-embed-large-v1` (local) | 8000 | no | filesystem (default) |
| `document_store_mcp_server` | An MCP server (Model Context Protocol, streamable HTTP) that exposes a live document store as a tool for MCP-aware clients (Claude Desktop, etc.) | none (MCP serves retrieval) | `mixedbread-ai/mxbai-embed-large-v1` (local) | 8000 (App) / 8068 (MCP) | no | filesystem (default) |
| `slides_ai_search` | Searching presentation decks — parses each slide as an image, extracts categories/languages, lets you filter | `gpt-4o` | `text-embedding-3-small` | 8000 (App), 8501 (UI), 8080 (nginx) | yes | filesystem (default), SharePoint/GDrive optional |
| `drive_alert` *(legacy)* | RAG over Google Drive that *also* sends Slack alerts when a doc change materially shifts the answer | `gpt-3.5-turbo` | `text-embedding-ada-002` | 8080 | yes | Google Drive (required) + Slack (required) |
| `unstructured_to_sql_on_the_fly` *(legacy)* | Continuously parse PDFs into a Postgres table, then translate natural-language questions to SQL against that table | `gpt-3.5-turbo` | n/a (LLM-only extraction) | 8080 | yes | filesystem in, Postgres out |
| **Cookbook** `cookbooks/self-rag-agents/*.ipynb` | Learning self-RAG and agentic patterns interactively with LangGraph — *not* a deployable app | n/a | n/a | n/a | Jupyter | n/a |

> [!TIP]
> When in doubt, start with `question_answering_rag`. It is the canonical template; every other modern template is a variation on its `App(BaseModel)` shape. Once you understand it, switching to `adaptive_rag` or `multimodal_rag` is a YAML diff away.

### 4.2 The shared shape (every modern template)

Every App-class template follows the same lifecycle:

```text
app.yaml ──► pw.load_yaml(f) ──► App(**config) ──► app.run()
                                       │
                                       ├── instantiate xpack server (QASummaryRestServer / DocumentStoreServer / PathwayMcp)
                                       ├── resolve persistence_mode + persistence_backend
                                       └── pw.run(persistence_config=…, terminate_on_error=…, monitoring_level=NONE)
```

The Pydantic shell carries seven public fields (`question_answerer` or `document_store` or `mcp_http`, `host`, `port`, `with_cache` *(deprecated)*, `persistence_backend`, `persistence_mode`, `terminate_on_error`) and one method (`run`). When you start the container, `app.py` reads `app.yaml`, builds the `App`, and calls `pw.run` to boot the streaming dataflow.

### 4.3 Run the canonical RAG (`question_answering_rag`)

```bash
cd templates/question_answering_rag
echo "OPENAI_API_KEY=sk-..." > .env
docker compose up --build
```

The compose file starts two containers: `app` (port 8000, the Pathway REST API + index) and `ui` (port 8501, the Streamlit UI). The UI uses `RAGClient(url=f"http://{PATHWAY_HOST}:{PATHWAY_PORT}")` — by default `app:8000` inside the compose network — to talk to the API.

#### REST endpoints (canonical RAG and adaptive/multimodal/private RAG)

| Method | Path | Purpose |
|---|---|---|
| POST | `/v2/answer` | Ask a question; receive `{response, context_docs}` |
| POST | `/v1/retrieve` | Top-k chunk retrieval; no LLM call |
| POST | `/v1/statistics` | Index size, last-update timestamp, document count |
| GET | `/v2/list_documents` | List every document the indexer knows about, with `_indexing_status` per file |
| POST | `/v2/summarize` | LLM summary of a list of texts (used by `SummaryQuestionAnswerer`) |

```bash
# Ask a question
curl -X POST http://localhost:8000/v2/answer \
  -H "Content-Type: application/json" \
  -d '{"prompt": "What are the contract terms?"}'

# Retrieve only — useful for debugging the index
curl -X POST http://localhost:8000/v1/retrieve \
  -H "Content-Type: application/json" \
  -d '{"query": "license", "k": 4}'

# Health-check the indexer
curl -X POST http://localhost:8000/v1/statistics
```

> [!NOTE]
> The endpoints above belong to Pathway's xpacks (`QASummaryRestServer`, `DocumentStoreServer`). They are not implemented in the templates — the templates configure them. For exhaustive REST-shape details (every JSON field, every status code) consult Pathway's API docs at <https://pathway.com/developers/api-docs/pathway-xpacks-llm/>.

### 4.4 Adaptive RAG — same shape, smarter retrieval

`adaptive_rag` substitutes `BaseRAGQuestionAnswerer` for `AdaptiveRAGQuestionAnswerer`, which retrieves documents iteratively: it starts with `n_starting_documents: 2`, and if the model is not confident enough, it requests `factor: 2`× more, up to `max_iterations: 4`. This usually halves token spend on questions where the answer sits in 1-2 chunks while still escalating for hard questions.

```yaml
question_answerer: !pw.xpacks.llm.question_answering.AdaptiveRAGQuestionAnswerer
  llm: $llm
  indexer: $document_store
  n_starting_documents: 2
  factor: 2
  max_iterations: 4
```

Run identically to the canonical template — same endpoints, same UI compatibility.

### 4.5 Multimodal RAG — image and table understanding

`multimodal_rag` upgrades the parser to read tables and images by way of a *second* OpenAI client (`gpt-4o`) used only inside `DoclingParser`:

```yaml
$parsing_llm: !pw.xpacks.llm.llms.OpenAIChat
  model: "gpt-4o"
  ...

$parser: !pw.xpacks.llm.parsers.DoclingParser
  multimodal_llm: $parsing_llm
  image_parsing_strategy: "llm"
  table_parsing_strategy: "llm"
  ...
```

The answering LLM stays as `gpt-4.1-mini` — only parsing is multimodal. Use this for finance reports, scientific papers, slide-derived PDFs, anywhere a chart or a table holds the actual answer.

### 4.6 Private RAG — fully local

`private_rag` keeps the same shape but swaps every external service for a local one:

```yaml
$llm_model: "ollama/mistral"

$llm: !pw.xpacks.llm.llms.LiteLLMChat
  model: $llm_model
  api_base: "http://localhost:11434"
  # api_base: "http://host.docker.internal:11434"  # for Docker-on-Mac/Windows
  async_mode: "fully_async"

$embedding_model: "avsolatorio/GIST-small-Embedding-v0"
$embedder: !pw.xpacks.llm.embedders.SentenceTransformerEmbedder
  model: $embedding_model
```

Prerequisites:

1. Install [Ollama](https://ollama.com/) on the host.
2. `ollama pull mistral` (or any model your `$llm_model` references).
3. `ollama serve` (or run the official Ollama Docker image on the host network).
4. From inside the Docker container, `localhost` won't work — use `host.docker.internal` on Mac/Windows, or `--network host` on Linux.

```bash
ollama pull mistral
ollama serve &
cd templates/private_rag
docker compose up --build
```

> [!IMPORTANT]
> Private RAG uses `AdaptiveRAGQuestionAnswerer` with `strict_prompt: true`. The strict prompt is more careful about saying "I don't know" when the context isn't sufficient — usually desirable for sensitive data, occasionally surprising if you expect the model to extrapolate.

### 4.7 Document indexing only — no LLM

`document_indexing` skips the answerer entirely and exposes a pure document store on REST. Use it when you want Pathway to maintain the index but call your own LLM downstream:

| Method | Path | Purpose |
|---|---|---|
| POST | `/v1/retrieve` | Top-k chunk retrieval |
| POST | `/v1/statistics` | Index stats |
| GET | `/v2/list_documents` | All known documents and their indexing status |

The default embedder is `mixedbread-ai/mxbai-embed-large-v1` running locally via SentenceTransformer — no OpenAI key needed. Drop files into `templates/document_indexing/files-for-indexing/` and they get indexed live.

### 4.8 MCP server — feed Pathway to MCP clients

`document_store_mcp_server` extends the document_indexing shape with an MCP (Model Context Protocol) server that streams over HTTP:

```yaml
mcp_http: !pw.xpacks.llm.mcp_server.PathwayMcp
  name: "Streamable MCP Server"
  transport: "streamable-http"
  host: "0.0.0.0"
  port: 8068
  serve:
    - $document_store
```

Point any MCP-aware client (Claude Desktop, an MCP-aware coding agent, etc.) at `http://localhost:8068` to access the live, always-current index as a tool.

> [!TIP]
> The MCP server lives on **port 8068**, not 8000. The App container itself still binds 8000, but in the MCP template that port carries no useful endpoints — clients should target 8068.

### 4.9 Slides search — semantic search over presentations

`slides_ai_search` is the most elaborate stack. It indexes `.pdf` and `.pptx` files, parses every slide as an image, classifies each slide along a 15-category taxonomy (Problem Statement, Solution Overview, Pricing and Packaging, …), tags it (e.g. `['price', 'shopping']`), detects language, and exposes search filtered by category and language.

The template ships **three containers** in its `docker-compose.yml`: the Pathway app (8000), a Streamlit UI (8501), and an nginx file server (8080) that serves the slide images and original PDFs back to the UI for rendering.

```bash
cd templates/slides_ai_search
cp .env.example .env  # add OPENAI_API_KEY
docker compose up --build
```

UI at `http://localhost:8501` — it shows a query box, two multi-selects (categories and languages), and renders the top-k slides as images you can click through.

### 4.10 Drive Alert (legacy) — RAG with Slack notifications on document change

`drive_alert` is one of the two legacy templates — it predates the App-class refactor and uses procedural `run(*, ...)` kwargs with `os.environ.get` defaults. It indexes Google Drive, exposes a `POST /` REST endpoint on **port 8080** (not 8000), and on top of that sends a Slack alert whenever a document change makes the answer to a previously-asked alerting query materially different.

Set up:

1. Create a GCP service account, download `secrets.json`, share the target Drive folder with the service-account email.
2. Set `FILE_OR_DIRECTORY_ID` to the Drive folder/file ID, `GOOGLE_CREDS` to `secrets.json`'s path.
3. Create a Slack bot, install it to your workspace, copy the bot token to `SLACK_ALERT_TOKEN`, set `SLACK_ALERT_CHANNEL_ID`.
4. `OPENAI_API_KEY` for the LLM.

Ask the assistant `Tell me and alert about the start date of the campaign for Magic Cola`. The keyword "alert" registers the query for monitoring; the next time a Drive doc change shifts the answer, you receive a Slack notification.

> [!WARNING]
> The legacy templates default to `gpt-3.5-turbo` and `text-embedding-ada-002`, which are older than the App-class defaults. They also bind to **port 8080** — if you run them next to a modern template that uses 8000, the ports do not collide, but if you reverse-proxy be aware of the mismatch.

### 4.11 Unstructured-to-SQL (legacy) — turn PDFs into a Postgres table you can query

`unstructured_to_sql_on_the_fly` is the second legacy template. Two pipelines run side by side inside one process:

1. **Pipeline 1 (extraction):** Watch `data/quarterly_earnings/`, send each PDF to OpenAI with a prompt that extracts the schema below into JSON, write the rows to Postgres and to `./data/quarterly_earnings.csv`.
2. **Pipeline 2 (query):** Expose a REST endpoint on port 8080 that accepts natural-language questions, asks GPT to translate them to a constrained `SELECT` against the `quarterly_earnings` table, executes the SQL, returns the result.

The hard-coded extraction schema (you cannot change this without editing `app.py`):

| Field | Type | Unit |
|---|---|---|
| `company_symbol` | str | stock ticker |
| `year` | int | calendar year |
| `quarter` | str | one of `Q1`, `Q2`, `Q3`, `Q4` |
| `revenue_md` | float | millions of dollars |
| `eps` | float | dollars per share |
| `net_income_md` | float | millions of dollars |

Run it with the bundled compose file (which also starts a Postgres container with `postgres/init-db.sql`):

```bash
cd templates/unstructured_to_sql_on_the_fly
docker compose up --build
```

Browse `http://localhost:8501` for the Streamlit UI, or `curl -X POST http://localhost:8080/ -d '{"user":"u","query":"What is the average net income in Q2 2023?"}' -H "Content-Type: application/json"`.

> [!CAUTION]
> Pipeline 2 builds and executes SQL produced by an LLM. The prompt constrains the grammar (`SELECT` only, restricted aggregators, no DDL) but you are still trusting an LLM with database access. Run against a read-only role and a database that only contains the extracted financial data — never against your production warehouse.

### 4.12 Cookbooks — self-RAG agents (Jupyter, learning-only)

`cookbooks/self-rag-agents/` ships two notebooks:

- `pathway_langgraph_agentic_rag.ipynb` — building an agentic RAG with LangGraph and Pathway.
- `pathway_deploy_langgraph_agents.ipynb` — deploying the same graph as a service.

These are interactive tutorials. They are not deployable (no Dockerfile, no compose, no `app.yaml`) and they import from LangGraph, which is not in any template's `requirements.txt`. Treat them as walkthroughs, not products. Open them in Jupyter Lab with `pip install pathway[all] langgraph jupyter && jupyter lab`.

## 5. Common Tasks

### 5.1 Add your own PDFs to a RAG template

```bash
cp /path/to/your.pdf templates/question_answering_rag/data/
# Pathway picks it up live; no restart needed
curl -X POST http://localhost:8000/v2/list_documents
```

### 5.2 Swap to a cheaper or stronger OpenAI model

```yaml
# in app.yaml
$llm: !pw.xpacks.llm.llms.OpenAIChat
  model: "gpt-4o-mini"   # was "gpt-4.1-mini"
```

Restart the container; the cache will need to rebuild for any prompts that hit the LLM differently.

### 5.3 Index a Google Drive folder instead of local files

In `app.yaml`, comment out the filesystem source, uncomment the Google Drive source, fill in `$DRIVE_ID`, and place `gdrive_indexer.json` (your service-account JSON) next to `app.yaml`:

```yaml
$sources:
  - !pw.io.gdrive.read
    object_id: $DRIVE_ID
    service_user_credentials_file: gdrive_indexer.json
    file_name_pattern:
      - "*.pdf"
      - "*.pptx"
    object_size_limit: null
    with_metadata: true
    refresh_interval: 30
```

Set `DRIVE_ID` in `.env`.

### 5.4 Index SharePoint instead of local files

```yaml
$sources:
  - !pw.xpacks.connectors.sharepoint.read
    url: $SHAREPOINT_URL
    tenant: $SHAREPOINT_TENANT
    client_id: $SHAREPOINT_CLIENT_ID
    cert_path: sharepointcert.pem
    thumbprint: $SHAREPOINT_THUMBPRINT
    root_path: $SHAREPOINT_ROOT
    with_metadata: true
    refresh_interval: 30
```

Place `sharepointcert.pem` next to `app.yaml`. Set the five env vars in `.env`.

### 5.5 Mix multiple sources in one index

`$sources:` is a list — repeat as many `!pw.io...` entries as you like; the document store unions all of them.

```yaml
$sources:
  - !pw.io.fs.read
    path: data
    format: binary
    with_metadata: true
  - !pw.io.gdrive.read
    object_id: $DRIVE_ID
    service_user_credentials_file: gdrive_indexer.json
```

### 5.6 Persist the cache between restarts

```yaml
# At the top level of app.yaml, alongside `host:` and `port:`
persistence_mode: !pw.PersistenceMode.PERSISTING
persistence_backend: !pw.persistence.Backend.filesystem
  path: ".Cache"
```

Mount the path as a Docker volume so it survives container rebuilds.

### 5.7 Customize the RAG prompt

```yaml
question_answerer: !pw.xpacks.llm.question_answering.BaseRAGQuestionAnswerer
  llm: $llm
  indexer: $document_store
  prompt_template: |
    You are a contracts assistant. Use ONLY the provided context.
    Context: {context}
    Question: {query}
    Answer (cite the document path):
```

### 5.8 Run with a different port

```yaml
host: "0.0.0.0"
port: 9000
```

Update the compose file's port mapping to match.

### 5.9 Wire the Streamlit UI to a remote Pathway app

The bundled UIs default `PATHWAY_HOST=app` (the compose service name). To point a UI at a Pathway service running elsewhere:

```bash
PATHWAY_HOST=rag.example.com PATHWAY_PORT=443 streamlit run ui.py
```

### 5.10 Halt on first error

For CI or strict production deployments where partial answers are worse than failure:

```yaml
terminate_on_error: true
```

### 5.11 Use a local embedding model (cost / privacy)

Replace `OpenAIEmbedder` with `SentenceTransformerEmbedder`:

```yaml
$embedder: !pw.xpacks.llm.embedders.SentenceTransformerEmbedder
  model: "mixedbread-ai/mxbai-embed-large-v1"
  call_kwargs:
    show_progress_bar: False
```

The first run downloads the model (~700 MB). After that it is fully local.

### 5.12 Build a private, end-to-end air-gapped stack

Use `private_rag`'s `app.yaml` as a starting point. Confirm:

- `$llm` uses `LiteLLMChat` with an Ollama `api_base`.
- `$embedder` uses `SentenceTransformerEmbedder`.
- `$parser` is `DoclingParser` with `table_parsing_strategy: "llm"` *only if* you point `multimodal_llm` at a local model — otherwise drop it to `"text"`.
- `$sources:` lists only the local filesystem.
- The Docker container runs with `--network host` (Linux) so it can reach `localhost:11434`.

### 5.13 Deploy to a cluster (multi-machine)

Multi-machine deployment uses Pathway's `--addresses` flag and requires a `PATHWAY_LICENSE_KEY` (Scale tier). Each machine runs the same `app.py` binary; the framework coordinates work across them. See the Admin Guide for the full deployment story.

> [!WARNING]
> Multi-machine clustering is gated by the Pathway license. Without a Scale key, `--addresses` raises a license error at startup.

## 6. Troubleshooting

### Diagnostic decision tree

1. **Container exits immediately.** Check `docker compose logs app`. 90 % of the time it's a missing `OPENAI_API_KEY` or a YAML typo (`extra="forbid"` rejects unknown fields).
2. **API responds but answers are empty.** `curl http://localhost:8000/v2/list_documents` — if the list is empty, Pathway hasn't found your data; check `$sources:` paths and Docker volume mounts.
3. **Answers are wrong but documents are indexed.** `curl -X POST http://localhost:8000/v1/retrieve` — if retrieval is wrong, the embedder is the problem (try a stronger model or smaller chunks). If retrieval is right but the answer is wrong, the LLM is the problem (try `gpt-4o`).
4. **Answers are slow.** First-time queries pay full LLM + embedding cost. Cached results are sub-second; cold runs are seconds. If consistently slow, check the OpenAI capacity limit — `capacity: 8` in `$llm` is a per-model rate limit.
5. **Streamlit shows "Connection error".** The UI cannot reach the Pathway app. Confirm `PATHWAY_HOST` matches your topology (compose: `app`; native: `localhost`).

### Error: `pydantic.ValidationError: 1 validation error for App ... extra inputs are not permitted`

**Cause:** Your `app.yaml` includes a key that the `App` Pydantic model does not declare. `extra="forbid"` is set on every modern template.

**Solution:**
1. Compare your `app.yaml` against the bundled one for that template.
2. Remove any unknown top-level keys.
3. Confirm the spelling: `question_answerer` (not `question_answer`), `persistence_mode` (not `persistance_mode`).

**Prevention:** Start from the shipped `app.yaml` and edit only what you need.

### Error: `RuntimeError: OpenAI API key not provided`

**Cause:** `OPENAI_API_KEY` is missing from the environment when `OpenAIChat` or `OpenAIEmbedder` instantiates.

**Solution:**
1. Confirm a `.env` file sits next to `app.py` (or `app.yaml`) and contains `OPENAI_API_KEY=sk-...`.
2. For Docker, the compose file or `--env-file .env` flag must explicitly load it.
3. Restart the container — `dotenv.load_dotenv()` runs at process start.

**Prevention:** Use `cp .env.example .env` when an `.env.example` is present and immediately fill in the keys.

### Error: `KeyError: 'SHAREPOINT_URL'`

**Cause:** You uncommented the SharePoint connector block in `app.yaml` but did not set the SharePoint env vars.

**Solution:**
1. Set `SHAREPOINT_URL`, `SHAREPOINT_TENANT`, `SHAREPOINT_CLIENT_ID`, `SHAREPOINT_THUMBPRINT`, `SHAREPOINT_ROOT` in `.env`.
2. Place the cert file at the path given by `cert_path:` (default `sharepointcert.pem` next to `app.yaml`).
3. Confirm the SharePoint app registration has *application* (not delegated) permissions for `Sites.Read.All`.

**Prevention:** The relevant `.env.example` files (when shipped) list every required variable.

### Error: `psycopg.OperationalError: connection to server on socket failed`

**Cause:** `unstructured_to_sql_on_the_fly` cannot reach Postgres. Most often you ran `docker run` instead of the bundled compose file.

**Solution:** Run `docker compose up --build` from the template's directory. The compose file starts a Postgres container with `postgres/init-db.sql` already loaded.

**Prevention:** This template is the only one that *requires* a database; do not skip the compose file.

### Error: `ConnectionRefusedError: [Errno 111] ... ollama:11434`

**Cause:** `private_rag`'s container cannot reach the Ollama service.

**Solution:**
1. On the host: `ollama serve` (or run the official Docker image with `-p 11434:11434`).
2. On Mac/Windows Docker, edit `app.yaml`'s `api_base` to `http://host.docker.internal:11434`.
3. On Linux Docker, run the container with `--network host` and keep `api_base: "http://localhost:11434"`.

**Prevention:** Pull the Ollama model and start the daemon before `docker compose up`.

### Error: Streamlit page loads but says "Connection error"

**Cause:** The UI's `RAGClient(url=f"http://{PATHWAY_HOST}:{PATHWAY_PORT}")` cannot reach the app.

**Solution:**
1. With compose: ensure `PATHWAY_HOST=app` and the app service is named `app` (or update both).
2. Native run: set `PATHWAY_HOST=localhost`.
3. Confirm the app container actually finished its first index build — the UI hits `list_documents` on load and that endpoint isn't ready until indexing completes.

**Prevention:** Always wait for the app log line `"INFO Indexed N documents"` before opening the UI.

### Error: Slack returns `401 invalid_auth`

**Cause:** `drive_alert` cannot post to Slack.

**Solution:**
1. Re-issue the bot token from the Slack app settings page.
2. Confirm the bot user has been *installed* to the workspace and *invited* to the channel.
3. Token must start with `xoxb-`, not `xapp-` or `xoxp-`.

**Prevention:** Slack rotates tokens on app reinstall — keep `.env` in sync.

### Error: Index is stale, doesn't pick up new files

**Cause:** Pathway's filesystem connector watches the path you configured in `$sources:` — if Docker's bind mount points elsewhere or the path is wrong, changes are invisible.

**Solution:**
1. Confirm `$sources:` `path:` matches the in-container path.
2. Confirm the compose file mounts the host directory to that container path.
3. For Google Drive, `refresh_interval: 30` controls the poll interval; lower it for faster updates.

**Prevention:** Always test by adding a small file and watching the app logs for the `"Indexed N documents"` increment.

### Issue: GitHub #124 — Getting Started instructions are fragmented across templates

**Cause:** Each template has its own README, and there is no single entry-point doc.

**Solution:** Use this guide's selector table (section 4.1) to pick a template, then the per-template README for deep details. Treat `templates/<name>/README.md` as the canonical source for that template.

**Prevention:** Until upstream consolidates, this guide is the index — link or bookmark section 4.1.

### Issue: GitHub #46 — OpenAI streaming response support requested

**Cause:** The shipped templates use `OpenAIChat` in non-streaming mode; the response is delivered as a single JSON body.

**Solution:** Streaming is a Pathway xpacks feature, not a template feature. If your downstream client requires SSE or chunked transfer, write a thin proxy in front of the Pathway REST endpoint that re-streams the response. Track the upstream issue on the Pathway core repo.

### Issue: GitHub #82 — How to use a local embedding model

**Cause:** Default templates use OpenAI embeddings; users want fully local.

**Solution:** Either start from `private_rag` (local Ollama LLM + SentenceTransformer embedder) or copy its `$embedder:` block into any other template's `app.yaml`.

**Prevention:** Cross-reference the template defaults table (section 4.1) before starting a new project.

## 7. FAQ

### General

**Q: Is `pathwaycom/llm-app` a library?**

No. It is a *catalogue* of ten reference applications (eight templates plus two cookbooks). The library doing the heavy lifting is [Pathway](https://github.com/pathwaycom/pathway), specifically its `pathway.xpacks.llm` module. The templates here are recipes that wire Pathway's xpacks pieces into deployable services.

**Q: Where do I find Pathway's full API?**

<https://pathway.com/developers/api-docs/pathway-xpacks-llm/>. The Admin Guide covers the slice of Pathway exercised by the templates in section 8.

**Q: Can I use a model other than OpenAI?**

Yes. The `$llm:` block can be `OpenAIChat`, `LiteLLMChat` (Ollama, Anthropic, Bedrock, Vertex, etc.), or any class under `pathway.xpacks.llm.llms`. `private_rag` shows the LiteLLM pattern.

**Q: Are the templates production-ready?**

They are production-shape — real Dockerfiles, real config, real connectors — but they default to demo settings (small `capacity:`, single-machine, demo license key). For production: rotate to a Scale license, raise capacity, add persistence, and put the API behind a reverse proxy with auth. See the Admin Guide.

**Q: What is a cookbook vs a template?**

A *template* is a deployable app (Dockerfile + compose + `app.yaml`). A *cookbook* is a Jupyter notebook tutorial. The cookbooks live under `cookbooks/self-rag-agents/` and demonstrate LangGraph integration; they are for learning, not deployment.

### Configuration

**Q: How do I switch from `gpt-4.1-mini` to `gpt-4o`?**

Edit `app.yaml`, change `model: "gpt-4.1-mini"` to `model: "gpt-4o"` under `$llm:`. Restart the container. Cached cache entries for the previous model still apply (the cache key includes the prompt, not the model — verify before caring about token cost).

**Q: How do I disable caching?**

In `app.yaml` set `persistence_mode: null` at the top level. Per-component caches (`cache_strategy: !pw.udfs.DefaultCache {}`) cache LLM and embedder calls in-process for the lifetime of the run regardless.

**Q: Where does the cache live?**

By default, the container's working directory under `./Cache`. Change with `persistence_backend: !pw.persistence.Backend.filesystem` and `path: "/data/cache"`. Mount that path as a Docker volume to make it persistent.

**Q: How do I customize the RAG prompt?**

Add `prompt_template:` under `question_answerer:`. Use `{query}` and `{context}` placeholders. See section 5.7.

**Q: What does `extra="forbid"` mean?**

The Pydantic `App` model rejects YAML keys it doesn't recognise — a typo fails fast at startup with a clear error rather than silently ignoring your override.

### Connectors and data sources

**Q: How many connectors do the templates use?**

Five: filesystem (`pw.io.fs.read`), Google Drive (`pw.io.gdrive.read`), SharePoint (`pw.xpacks.connectors.sharepoint.read`), Postgres (`pw.io.postgres.write`, output side, `unstructured_to_sql_on_the_fly`), and Slack (`pw.io.slack.send_alerts`, output, `drive_alert`). The parent Pathway library has dozens more — these templates use a subset.

**Q: Can I use S3 as a source?**

Pathway supports `pw.io.s3.read` (mentioned in `drive_alert`'s code comments). It is not used in any shipped `app.yaml`, but you can add it under `$sources:` the same way the GDrive and SharePoint blocks are added.

**Q: Does it support Postgres as a source?**

Pathway supports `pw.io.postgres.read`; none of the templates use it as input. `unstructured_to_sql_on_the_fly` writes *to* Postgres, not reads from it.

**Q: How fresh is the index?**

The filesystem connector polls effectively continuously (Pathway uses a streaming dataflow). Google Drive defaults to `refresh_interval: 30` seconds. SharePoint same.

### Deployment

**Q: Do I need Docker?**

No, but it is the recommended path. Native Python works — `pip install pathway[all] python-dotenv && python app.py` — but you are responsible for matching system libraries (Pathway includes a Rust core that links against system OpenSSL and similar).

**Q: Is GPU required?**

No. Templates default to CPU-friendly settings. `private_rag` benefits from a GPU because of the local embedder, but does not require one.

**Q: What ports do the templates use?**

App-class templates: 8000 for the Pathway REST API. The MCP template adds 8068. Streamlit UIs: 8501. Legacy templates: 8080. The slides template adds an nginx file server on 8080 internally.

**Q: How do I deploy on Kubernetes?**

The templates do not ship K8s manifests. Build the bundled Dockerfile, push to a registry, write a Deployment + Service per template. For multi-machine clustering, see the Admin Guide on `--addresses` and the Pathway license.

### Cost and limits

**Q: How expensive is a default RAG run?**

Indexing a small PDF set with `text-embedding-3-small` is fractions of a cent. Each query at `gpt-4.1-mini` is roughly the cost of one prompt + ~500-token response. The first answer is most expensive (cold cache); repeated queries hit the cache and cost nothing.

**Q: What is the `capacity:` field on `$llm:`?**

A simple in-process rate limit on concurrent OpenAI calls. The default is 8. Lower it if you hit OpenAI rate limits; raise it for faster indexing on a high-tier account.

**Q: Why does the app retry?**

`retry_strategy: !pw.udfs.ExponentialBackoffRetryStrategy max_retries: 6` is set on every LLM and embedder. Transient OpenAI errors are retried with exponential backoff before the pipeline fails.

### Security and privacy

**Q: Are documents sent to OpenAI?**

By default, yes — both for embedding (so the index can be built) and for question answering. `private_rag` and `document_indexing` (with `mxbai-embed-large-v1`) keep documents local.

**Q: Can I run this in a regulated environment?**

Yes — start from `private_rag` and confirm every external service is replaced with a local one. The Admin Guide section 7 enumerates the exact things to audit.

**Q: How do I rotate the OpenAI key?**

Update `.env`, then `docker compose restart app`. The key is read once at process start.

### Cookbooks and integrations

**Q: Why does `cookbooks/self-rag-agents/` import LangGraph?**

The cookbooks demonstrate combining Pathway (for the live document store) with LangGraph (for the agent state machine). LangGraph is a dependency only of the notebooks, not of any template's `requirements.txt`.

**Q: Is there an integration with LangChain?**

LangChain is not bundled. You can use `pw.xpacks.llm.question_answering.RAGClient` (used by the Streamlit UIs) from any Python client, including LangChain.

---

_Generated April 2026 from pathwaycom/llm-app at 5fa97b7. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._