# DeepSeek-R1 Administrator & Developer Guide

## 1. Architecture Overview

DeepSeek-R1 is a reasoning language model series released by DeepSeek AI under the MIT license. The repository at `deepseek-ai/DeepSeek-R1` (commit `0cf7856`, dated 2025-04-09) is a **model release artifact** — it contains the research paper (`DeepSeek_R1.pdf`), a `README.md`, a benchmark image (`figures/benchmark.jpg`), and an MIT `LICENSE` file. There is no application source code, no inference runtime, no configuration files, and no dependency manifests in this repository.

The deployable system is assembled from components hosted elsewhere:

```
[DeepSeek API (cloud)]                       [Self-hosted deployment]
      │                                              │
      ▼                                    ┌─────────▼────────┐
 platform.deepseek.com                     │  Inference server │
 (OpenAI-compatible REST API)              │  (vLLM / TGI /   │
      │                                    │   Transformers)   │
      │                                    └─────────┬────────┘
      │                                              │
      ▼                                              ▼
 Your application                          Model weights (Hugging Face)
 (OpenAI SDK / HTTP)                       deepseek-ai/{variant}
```

The model family consists of:

| Model | Architecture | Parameters | Weights Location |
|---|---|---|---|
| DeepSeek-R1 (full) | MoE (Mixture-of-Experts) | 671B | Hugging Face |
| DeepSeek-R1-Distill-Qwen-1.5B | Dense (Qwen2.5 base) | 1.5B | Hugging Face |
| DeepSeek-R1-Distill-Qwen-7B | Dense (Qwen2.5 base) | 7B | Hugging Face |
| DeepSeek-R1-Distill-Llama-8B | Dense (Llama-3 base) | 8B | Hugging Face |
| DeepSeek-R1-Distill-Qwen-14B | Dense (Qwen2.5 base) | 14B | Hugging Face |
| DeepSeek-R1-Distill-Qwen-32B | Dense (Qwen2.5 base) | 32B | Hugging Face |
| DeepSeek-R1-Distill-Llama-70B | Dense (Llama-3 base) | 70B | Hugging Face |

> [!NOTE]
> This repository contains only research artifacts — not runnable code. All operational components (inference runtimes, API server, model weights) are external to this repository. Deployments assemble these from Hugging Face, vLLM, and the DeepSeek API platform separately.

### Data flow — API path

```
Client request (OpenAI SDK)
  → HTTPS POST https://api.deepseek.com/v1/chat/completions
  → DeepSeek inference cluster (full 671B MoE model)
  → JSON response with reasoning trace (<think>...</think>) + final answer
```

### Data flow — self-hosted path

```
Client request (OpenAI SDK or HTTP)
  → vLLM / TGI serving endpoint (local or cloud VM)
  → Model weights loaded from disk (Hugging Face snapshot cache)
  → Streaming or batch response with reasoning trace + final answer
```

---

## 2. System Requirements

### Hardware Requirements by Variant

| Model Variant | GPU VRAM (FP16) | GPU VRAM (INT4) | CPU RAM | Storage |
|---|---|---|---|---|
| Distill-Qwen-1.5B | 4 GB | 2 GB | 8 GB | 3 GB |
| Distill-Qwen-7B | 16 GB | 6 GB | 16 GB | 14 GB |
| Distill-Llama-8B | 18 GB | 8 GB | 16 GB | 16 GB |
| Distill-Qwen-14B | 30 GB | 12 GB | 32 GB | 28 GB |
| Distill-Qwen-32B | 68 GB | 24 GB | 64 GB | 65 GB |
| Distill-Llama-70B | 144 GB | 56 GB | 140 GB | 140 GB |
| Full R1 (671B MoE) | Multi-node (8×H100 min) | Not practical | Very large | ~1.3 TB |

> [!IMPORTANT]
> FP16 figures assume full-precision weights. INT4 figures use bitsandbytes or AWQ quantization. Quantized models trade a small quality loss for dramatically reduced VRAM requirements. For production inference at scale, INT4 with vLLM's tensor-parallel mode is the standard approach for the 70B variant.

### Software Prerequisites

| Component | Version | Purpose |
|---|---|---|
| Python | 3.8+ | Inference framework runtime |
| CUDA | 11.8 or 12.x | GPU acceleration (NVIDIA) |
| PyTorch | 2.0+ | Tensor operations |
| Hugging Face Transformers | 4.35+ | Model loading |
| vLLM | 0.3+ (recommended for serving) | High-throughput inference |
| Ollama | Latest | Simplified local serving |

---

## 3. Configuration Reference

The GitHub repository (`deepseek-ai/DeepSeek-R1`) contains no configuration files. All configuration is managed at the inference framework level.

### vLLM Server Configuration

```bash
vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-7B \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 1 \
  --max-model-len 32768 \
  --dtype bfloat16
```

| Flag | Type | Default | Description |
|---|---|---|---|
| `--host` | string | `127.0.0.1` | Bind address for the HTTP server |
| `--port` | integer | `8000` | Listening port |
| `--tensor-parallel-size` | integer | `1` | Number of GPUs to shard across |
| `--max-model-len` | integer | model max | Maximum sequence length (tokens) |
| `--dtype` | string | `auto` | Weight dtype: `float16`, `bfloat16`, `auto` |
| `--quantization` | string | None | `awq`, `gptq`, `squeezellm` for quantized models |
| `--gpu-memory-utilization` | float | `0.90` | Fraction of GPU memory vLLM may use |
| `--enable-prefix-caching` | flag | disabled | Enable KV cache reuse for repeated prefixes |

> [!TIP]
> For multi-GPU deployments of the 70B distill variant, set `--tensor-parallel-size 4` (for 4×A100 80GB) or `--tensor-parallel-size 8` for H100 nodes. vLLM handles tensor parallelism transparently.

### Hugging Face Transformers Configuration

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
    device_map="auto",           # Automatically assign layers to GPU/CPU
    torch_dtype="auto",          # Use model's preferred dtype
    load_in_4bit=True,           # Optional: 4-bit quantization
    trust_remote_code=True       # Required for some model architectures
)
tokenizer = AutoTokenizer.from_pretrained(
    "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
    trust_remote_code=True
)
```

### Ollama Configuration

```bash
# Pull and serve
ollama pull deepseek-r1:7b
ollama serve   # Starts API server on localhost:11434

# Configure via environment variables
OLLAMA_HOST=0.0.0.0:11434 ollama serve
OLLAMA_NUM_GPU=1 ollama serve
```

### API Call Parameters

| Parameter | Type | Default | Description |
|---|---|---|---|
| `model` | string | — | `deepseek-reasoner` (API) or HF model ID (local) |
| `messages` | array | — | OpenAI-format message history |
| `max_tokens` | integer | model limit | Maximum tokens in response |
| `temperature` | float | 1.0 | Sampling temperature |
| `top_p` | float | 1.0 | Nucleus sampling threshold |
| `stream` | boolean | false | Token-by-token streaming |
| `stop` | array | None | Stop sequences |

> [!IMPORTANT]
> The `tool_choice` and `logprobs` parameters are not supported on the hosted DeepSeek API for the R1 model (GitHub issues #813, #815). Self-hosted vLLM deployments support `logprobs`. Tool calling support varies by deployment framework.

---

## 4. Deployment & Scaling

### Deployment Option A — DeepSeek API (Zero Infrastructure)

The simplest production path. DeepSeek operates the full 671B model behind an OpenAI-compatible API.

```python
from openai import OpenAI

client = OpenAI(
    api_key="sk-...",
    base_url="https://api.deepseek.com"
)
```

**Advantages:** No GPU infrastructure, no weight management, no model serving maintenance.
**Limitations:** API parity gaps (no `tool_choice`, no `logprobs`), data leaves your environment, latency depends on DeepSeek's infrastructure.

### Deployment Option B — vLLM (Recommended for Self-Hosted Production)

vLLM provides the highest throughput for serving distilled variants.

**Single-node deployment (7B example):**

```bash
pip install vllm
vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-7B \
  --host 0.0.0.0 --port 8000 \
  --max-model-len 32768 \
  --dtype bfloat16
```

**Multi-GPU deployment (70B example, 4×GPU):**

```bash
vllm serve deepseek-ai/DeepSeek-R1-Distill-Llama-70B \
  --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 4 \
  --dtype bfloat16
```

The vLLM endpoint is OpenAI-compatible:

```python
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
    messages=[{"role": "user", "content": "Test"}]
)
```

### Deployment Option C — Text Generation Inference (TGI)

Hugging Face TGI is an alternative to vLLM with strong Docker integration:

```bash
docker run --gpus all \
  -v $HOME/.cache/huggingface:/data \
  -p 8080:80 \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id deepseek-ai/DeepSeek-R1-Distill-Qwen-7B \
  --max-input-length 4096 \
  --max-total-tokens 8192
```

### Deployment Option D — Ollama (Dev/Small Team)

```bash
ollama pull deepseek-r1:7b   # or 1.5b, 8b, 14b, 32b, 70b
OLLAMA_HOST=0.0.0.0:11434 ollama serve
```

Ollama's API is OpenAI-compatible when you set `base_url="http://localhost:11434/v1"`.

### Deployment Checklist

- [x] Choose the correct model variant for available hardware
- [x] Download weights from Hugging Face (or confirm Ollama pull succeeded)
- [x] Verify CUDA version matches PyTorch / vLLM requirements
- [ ] Configure serving endpoint host/port and firewall rules
- [ ] Set memory utilization limits to prevent OOM on concurrent requests
- [ ] Configure reverse proxy (nginx/Caddy) with TLS if internet-facing
- [ ] Implement API key authentication at the reverse proxy layer
- [ ] Set up log aggregation and alerting on OOM/crash events

### Scaling Considerations

| Scenario | Recommended Approach |
|---|---|
| Single developer | Ollama + 7B distill on consumer GPU (16 GB VRAM) |
| Small team (< 10 concurrent) | vLLM single node, 14B or 32B distill |
| Medium workload (10–100 concurrent) | vLLM with tensor parallelism, 70B distill or dedicated 671B via API |
| High throughput / enterprise | Multiple vLLM nodes behind a load balancer, or DeepSeek API with retry logic |

---

## 5. Operations

### Starting and Stopping Services

**vLLM:**

```bash
# Start
vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --port 8000 &

# Stop
kill $(lsof -t -i:8000)
```

**Ollama:**

```bash
# Start daemon
ollama serve

# Stop (Linux systemd)
systemctl stop ollama
```

> [!NOTE]
> vLLM does not include a built-in process manager. Run it under `systemd`, `supervisord`, or in a container to ensure automatic restart on crash.

### Health Checks

**vLLM:**

```bash
curl http://localhost:8000/health
```

```json
{}
```

An HTTP 200 response indicates the server is ready. Non-200 or connection refused means the model is still loading or has crashed.

**Ollama:**

```bash
curl http://localhost:11434/api/tags
```

```json
{"models": [{"name": "deepseek-r1:7b", ...}]}
```

### Monitoring

Key metrics to track for a self-hosted vLLM deployment:

| Metric | Source | Alert Threshold |
|---|---|---|
| GPU VRAM utilization | `nvidia-smi` | > 95% |
| Request queue depth | vLLM `/metrics` (Prometheus) | > 50 pending |
| Token throughput (tok/s) | vLLM `/metrics` | Below baseline |
| OOM kill events | System logs | Any occurrence |
| P99 latency (TTFT) | Application logs | > 10s for non-streaming |

```bash
# Quick GPU health check
nvidia-smi --query-gpu=name,memory.used,memory.total,temperature.gpu \
  --format=csv,noheader
```

> [!TIP]
> vLLM exposes a Prometheus-compatible `/metrics` endpoint. Scrape it with Prometheus and visualize with Grafana for production observability. The most important metrics are `vllm:num_requests_running`, `vllm:gpu_cache_usage_perc`, and `vllm:generation_tokens_total`.

### Model Weight Management

```bash
# Check Hugging Face cache location
python -c "from huggingface_hub import constants; print(constants.HF_HUB_CACHE)"

# List downloaded models
ls $(python -c "from huggingface_hub import constants; print(constants.HF_HUB_CACHE)")

# Download a specific variant
huggingface-cli download deepseek-ai/DeepSeek-R1-Distill-Qwen-7B

# Remove cached weights
huggingface-cli delete-cache
```

> [!NOTE]
> Model weights are large binary files. The Hugging Face cache defaults to `~/.cache/huggingface/hub/`. Bind-mount this to persistent storage in containerized deployments so weights survive container restarts.

---

## 6. Performance

### Throughput Benchmarks (Approximate)

| Model | Hardware | Batch Size | Throughput (tok/s) | Framework |
|---|---|---|---|---|
| Distill-Qwen-7B | RTX 4090 (24 GB) | 1 | ~50 | vLLM FP16 |
| Distill-Qwen-7B | RTX 4090 (24 GB) | 8 | ~200 | vLLM FP16 |
| Distill-Qwen-14B | A100 (80 GB) | 1 | ~45 | vLLM BF16 |
| Distill-Llama-70B | 4×A100 (80 GB) | 8 | ~120 | vLLM TP4 BF16 |

> [!WARNING]
> These figures are illustrative estimates based on hardware specifications. Actual throughput depends on prompt length, reasoning depth, and concurrent request count. Reasoning models typically generate more tokens than standard chat models due to the `<think>` block — benchmark with representative workloads, not toy prompts.

### Optimizations

- **Enable prefix caching** (`--enable-prefix-caching`): reduces latency for repeated system prompts by reusing computed KV cache.
- **Quantization (AWQ/GPTQ)**: reduces VRAM by ~50–75% with minimal quality loss.
- **Continuous batching**: vLLM handles this automatically — do not implement manual batching.
- **Speculative decoding**: available in newer vLLM versions; use a small draft model for faster generation.

---

## 7. Security

### API Key Management

- Never embed API keys in source code. Store in environment variables or a secrets manager (Vault, AWS Secrets Manager, etc.).
- Rotate keys if exposed. Revoke at [platform.deepseek.com](https://platform.deepseek.com).
- For self-hosted deployments, authenticate at the reverse proxy layer (nginx `auth_request`, Caddy forward-auth) — vLLM and Ollama do not implement API authentication by default.

> [!CAUTION]
> Ollama and vLLM bind to all interfaces if `--host 0.0.0.0` is set and have no built-in authentication. Never expose these ports directly to the internet. Always place a TLS-terminating reverse proxy with authentication in front.

### Data Privacy

- API path: prompts and responses are processed by DeepSeek's infrastructure. Review DeepSeek's privacy policy before sending sensitive data via the API.
- Self-hosted path: all data stays within your infrastructure. Use network policies to restrict access to the inference endpoint.
- Model weights are static files — they do not exfiltrate data or phone home.

### License Compliance

The full DeepSeek-R1 model weights and all distilled variants are released under the MIT license. Key obligations:

- Include the MIT license notice in any redistribution of the weights
- The license does not restrict commercial use, modification, or sublicensing
- Distilled variants derive from Qwen2.5 (Apache 2.0) and Llama-3 (Meta Community License) base models. Verify compliance with the applicable base model license for each variant.

> [!IMPORTANT]
> The Llama-3 base model (used by Distill-Llama-8B and Distill-Llama-70B) is subject to Meta's Community License Agreement, which imposes usage restrictions for services with more than 700 million monthly active users. Review this before large-scale commercial deployment of Llama-based distills.

### Input Validation

When building applications on top of DeepSeek-R1:

- Sanitize user input before injecting into system prompts to prevent prompt injection
- Limit `max_tokens` per request to bound compute cost and response size
- Rate-limit requests per user at the application layer
- Log requests (excluding content if privacy-sensitive) for audit and abuse detection

---

## 8. Developer Internals

### Repository Structure

```text
deepseek-ai/DeepSeek-R1 @ 0cf7856
├── DeepSeek_R1.pdf          # Research paper (technical architecture, training methodology, results)
├── LICENSE                  # MIT License
├── README.md                # Model overview, benchmark table, Hugging Face links, usage examples
└── figures/
    └── benchmark.jpg        # Benchmark comparison chart referenced in README
```

The repository contains 4 files and 0 lines of source code. All implementation lives in:
- Hugging Face model repositories under the `deepseek-ai` organization
- The vLLM / Transformers inference frameworks
- DeepSeek's proprietary API infrastructure (closed-source)

### Model Architecture — R1 Full (671B MoE)

Based on the research paper (`DeepSeek_R1.pdf`):

- **Architecture type:** Mixture-of-Experts (MoE) Transformer
- **Active parameters per forward pass:** A subset of the 671B total (MoE routing activates a fraction of expert layers per token)
- **Training approach:** Reinforcement Learning from Human Feedback (RLHF) with Group Relative Policy Optimization (GRPO), combined with supervised fine-tuning on chain-of-thought data
- **Reasoning mechanism:** Extended thinking via `<think>` tags produced as part of the autoregressive generation process — not a separate inference pass
- **Context window:** Up to 128K tokens

### Distilled Model Architecture

The distilled models are dense transformers that do not use MoE routing. Knowledge distillation transfers R1's reasoning behavior into the smaller architectures:

| Variant | Base Model | Base License |
|---|---|---|
| Distill-Qwen-1.5B | Qwen2.5-1.5B | Apache 2.0 |
| Distill-Qwen-7B | Qwen2.5-7B | Apache 2.0 |
| Distill-Llama-8B | Llama-3-8B | Meta Community License |
| Distill-Qwen-14B | Qwen2.5-14B | Apache 2.0 |
| Distill-Qwen-32B | Qwen2.5-32B | Apache 2.0 |
| Distill-Llama-70B | Llama-3-70B | Meta Community License |

### Reasoning Training

The core training novelty documented in the paper is that the model learns multi-step reasoning through RL reward signals (correctness of final answer) rather than imitation of pre-labeled reasoning chains. The `<think>` block emerges as the model's learned scratchpad format during RL training.

### Benchmark Performance

The `figures/benchmark.jpg` in the repository shows comparative benchmark results against other frontier models (GPT-4o, Claude series, Gemini) across mathematical reasoning, coding, and scientific problem-solving tasks. Consult the research paper for exact numbers and methodology.

### AST Extraction Summary

```text
cli_commands:   0   (no source code)
api_endpoints:  0   (no source code)
functions:      0   (no source code)
classes:        0   (no source code)
config_keys:    0   (no source code)
```

All zeros reflect the absence of runnable code in this repository, not a tool limitation.

### Integration Points

| Integration | Method | Notes |
|---|---|---|
| OpenAI Python SDK | `OpenAI(base_url="https://api.deepseek.com")` | API path |
| LangChain | `ChatOpenAI(openai_api_base="https://api.deepseek.com")` | API path |
| LlamaIndex | `DeepSeekLLM` or OpenAI-compatible adapter | Check LlamaIndex docs |
| vLLM | `vllm serve {model_id}` | Self-hosted |
| Ollama | `ollama pull deepseek-r1:{size}` | Self-hosted |
| TGI | Docker image, `--model-id` flag | Self-hosted |
| Direct HTTP | `POST /v1/chat/completions` (OpenAI format) | Any endpoint |

---

## 9. Troubleshooting Reference

### vLLM OOM on Startup

**Cause:** Model too large for available VRAM with current settings.

**Solution:**
1. Reduce `--gpu-memory-utilization` from 0.90 to 0.80
2. Enable quantization: `--quantization awq` (requires pre-quantized weights)
3. Add `--max-model-len` to cap sequence length and reduce KV cache allocation
4. Switch to a smaller model variant

> [!WARNING]
> Do not set `--gpu-memory-utilization` above 0.95 in production. The remaining headroom prevents OOM kills during peak KV cache usage under concurrent load.

---

### Slow First Response (Cold Start)

**Cause:** Model weights are being loaded from disk and transferred to GPU VRAM.

**Solution:** Loading is a one-time cost per server start. Large models (70B) take 2–5 minutes to load on typical NVMe + PCIe bandwidth. Monitor with `nvidia-smi` — VRAM utilization climbing steadily indicates loading in progress.

---

### Inconsistent Reasoning Quality

**Cause:** Temperature too high causing erratic chain-of-thought.

**Solution:** For deterministic math and code tasks, use `temperature=0.5` to `0.7`. For creative tasks, `temperature=1.0` is appropriate. Avoid `temperature=0` — it can cause infinite repetition in reasoning chains.

> [!TIP]
> If the reasoning chain trails off into repetition before reaching a conclusion, reduce temperature and increase `max_tokens`. The model is more likely to complete a reasoning chain when there is token budget remaining and sampling is less random.

---

### Hugging Face Download Stalls

**Cause:** Network interruption or hub rate limiting.

**Solution:**
1. Resume with `huggingface-cli download {model_id}` — it resumes from partial downloads
2. Use `HF_HUB_ENABLE_HF_TRANSFER=1` environment variable for faster multi-part downloads
3. Set `HF_ENDPOINT=https://hf-mirror.com` for mirror access if the main hub is unreachable

---

### Requests Fail After Long Idle Period

**Cause:** vLLM process has crashed and not restarted (no process manager).

**Solution:** Configure `systemd` or a supervisor to automatically restart vLLM on failure.

```ini
# /etc/systemd/system/vllm.service
[Unit]
Description=vLLM inference server
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --host 0.0.0.0 --port 8000
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
```

```bash
systemctl enable vllm && systemctl start vllm
```

---

_Generated May 2026 from deepseek-ai/DeepSeek-R1 at 0cf7856. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._