ChatGLM-6B Admin Guide

This guide is for the engineers, operators, and architects who deploy, harden, fine-tune, and maintain ChatGLM-6B in production or research environments. It covers the repository's architecture, the full fine-tuning configuration surface, the security implications of how the model is loaded and served, deployment patterns the repository itself does not prescribe, and the internals you need to operate it reliably.

For installation and day-to-day usage, see the companion User Guide.

1. Executive Summary

ChatGLM-6B is an open-source, bilingual (Chinese–English) conversational large language model with 6.2 billion parameters, built on the GLM (General Language Model) architecture. It was released by the Knowledge Engineering Group (KEG) of Tsinghua University / Zhipu AI (the THUDM / zai-org organization).

What this repository actually contains. This is a model-release repository, not a self-contained application. The model weights and the model's architecture code live on the Hugging Face Hub (THUDM/chatglm-6b); this repository ships:

• A handful of demo entry points — a CLI chat (cli_demo.py), three web UIs (web_demo.py, web_demo2.py, web_demo_old.py), a REST API (api.py), and two multimodal VisualGLM demos.
• A multi-GPU helper (utils.py).
• A P-Tuning v2 fine-tuning harness (ptuning/) — the largest body of code in the repo.

Every demo is a thin wrapper that calls AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True) and then model.chat(...) / model.stream_chat(...). The substantive model logic is downloaded and executed from the Hub at runtime.

Important

Two security facts dominate the threat model and recur throughout this guide. (1) Loading the model executes arbitrary remote Python via trust_remote_code=True, and deserializes pickled checkpoints via torch.load. (2) The bundled REST API (api.py) binds to 0.0.0.0:8000 with no authentication. Treat both as load-bearing operational constraints, not footnotes. See Section 5: Security.

Lifecycle status. ChatGLM-6B is the first-generation GLM chat model. The maintainers have since shipped ChatGLM2-6B, ChatGLM3, and GLM-4, and the README routes new users to them. The repository is effectively in maintenance: stable and runnable, but not actively developed. Plan to pin your dependency versions (especially transformers==4.27.1) and carry any patches internally rather than expecting upstream fixes.

2. Architecture

2.1 System overview

                        Hugging Face Hub
                 ┌──────────────────────────────┐
                 │  THUDM/chatglm-6b             │
                 │   ├── modeling_chatglm.py ────┼──► executed locally
                 │   ├── tokenization_chatglm.py │    (trust_remote_code=True)
                 │   ├── quantization.py         │
                 │   └── pytorch_model-*.bin ────┼──► torch.load (pickle)
                 └──────────────┬───────────────┘
                                │  AutoModel.from_pretrained(...)
                                ▼
  ┌─────────────────────────────────────────────────────────────────┐
  │                      THIS REPOSITORY (glue)                       │
  │                                                                   │
  │   Inference demos            Fine-tuning harness (ptuning/)       │
  │   ┌──────────────┐           ┌─────────────────────────────────┐ │
  │   │ cli_demo.py  │           │ main.py  (HfArgumentParser)      │ │
  │   │ web_demo*.py │           │   ├── arguments.py (dataclasses) │ │
  │   │ api.py       │           │   ├── trainer.py        (vendored│ │
  │   │ *_vision.py  │           │   ├── trainer_seq2seq.py  HF code)│ │
  │   └──────┬───────┘           │   └── deepspeed.json (ZeRO-2)    │ │
  │          │ model.chat /      └─────────────────────────────────┘ │
  │          │ stream_chat        produces: PrefixEncoder checkpoint  │
  │   ┌──────▼───────┐                                                │
  │   │ utils.py     │  load_model_on_gpus() → accelerate dispatch    │
  │   └──────────────┘                                                │
  └─────────────────────────────────────────────────────────────────┘

2.2 Component inventory

Note

ptuning/trainer.py and ptuning/trainer_seq2seq.py are vendored copies of Hugging Face internals (~234 KB of the repo's ~253 KB of source). They are not project-authored logic — they are pinned to guarantee compatibility with transformers==4.27.1. Do not treat their classes (Trainer, Seq2SeqTrainer) or their SageMaker-specific environment-variable reads as part of the project's own configuration surface.

2.3 The model-loading path

All demos converge on the same sequence:

tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True)
model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).half().cuda()
model = model.eval()

• trust_remote_code=True causes transformers to download and execute modeling_chatglm.py, tokenization_chatglm.py, and quantization.py from the Hub.
• .half() casts to FP16; .quantize(4|8) (when chained) builds INT4/INT8 kernels via cpm_kernels.
• .cuda() / .float() / .to('mps') selects the device.

2.4 Multi-GPU device map (utils.py)

load_model_on_gpus() splits the model across cards. The model has a fixed layer budget that auto_configure_device_map() distributes:

• transformer.word_embeddings, transformer.final_layernorm, and lm_head are always pinned to GPU 0 (counted as 2 layers).
• The 28 transformer layers (transformer.layers.{0..27}) are spread evenly across the remaining capacity (30 / num_gpus layers per card).

Tip

The embeddings, final layernorm, and head are forced onto card 0 deliberately. On Linux, model.device resolves to lm_head.device; if word_embeddings lived on a different card, model.chat/stream_chat would move input_ids to the wrong device and raise a RuntimeError. This is a deliberate workaround, not an arbitrary choice — preserve it if you write a custom device_map.

3. System Requirements

3.1 Hardware (inference)

Important

On-the-fly quantization (.quantize(n)) loads the FP16 model into RAM (~13 GB) before compressing. On RAM-constrained hosts, load a pre-quantized checkpoint (THUDM/chatglm-6b-int4) instead, which needs only ~5.2 GB.

3.2 Software

Fine-tuning adds rouge_chinese, nltk, jieba, datasets, and — for full fine-tuning — deepspeed.

Warning

The transformers version is a hard compatibility gate, not a recommendation. The model's remote code is written against 4.27.1; newer releases surface as runtime errors (ChatGLMTokenizer has no attribute vocab_size, KeyError: '<|endoftext|>', .generate() incompatibilities). Pin it in your deployment image.

4. Configuration Reference

This repository has no application config files (no YAML/TOML/.env). The two configuration surfaces are: (1) the generation parameters passed to model.chat, and (2) the fine-tuning arguments in ptuning/.

4.1 Generation parameters

4.2 Fine-tuning: ModelArguments (ptuning/arguments.py)

4.3 Fine-tuning: DataTrainingArguments (ptuning/arguments.py)

main.py parses these three dataclasses (ModelArguments, DataTrainingArguments, Seq2SeqTrainingArguments) via Hugging Face's HfArgumentParser. You can also pass a single .json file argument and it will be parsed as the full config.

4.4 The launch scripts (ptuning/*.sh)

PRE_SEQ_LEN=128 is the soft-prompt length used across the P-Tuning scripts.

4.5 DeepSpeed config (ptuning/deepspeed.json)

{
  "train_micro_batch_size_per_gpu": "auto",
  "zero_allow_untested_optimizer": true,
  "fp16": { "enabled": "auto", "loss_scale": 0, "initial_scale_power": 16,
            "loss_scale_window": 1000, "hysteresis": 2, "min_loss_scale": 1 },
  "zero_optimization": {
    "stage": 2, "allgather_partitions": true, "allgather_bucket_size": 5e8,
    "overlap_comm": false, "reduce_scatter": true, "reduce_bucket_size": 5e8,
    "contiguous_gradients": true
  }
}

ZeRO stage 2 partitions optimizer state and gradients (but not parameters) across the four GPUs. This is what makes full-parameter fine-tuning fit on 4× consumer/datacenter cards.

5. Security Analysis

Caution

ChatGLM-6B's loading and serving model carries three concrete, exploitable risks. Review all three before any networked deployment. A community security report (issue #1517, "Insecure Deserialization and Remote Code Execution via trust_remote_code and torch.load") documents the first two.

5.1 Remote code execution via trust_remote_code=True

Every demo loads the model with trust_remote_code=True, which downloads and executes Python from the Hugging Face Hub (modeling_chatglm.py, tokenization_chatglm.py, quantization.py). If the Hub repository, your network path to it, or your local model cache is compromised, that code runs with your process's full privileges.

Mitigations:

• Pin a known-good revision. Use revision="<commit-sha>" (not a moving branch) so an upstream change cannot silently alter the executed code.
• Vendor the weights and code. Download once to a local, integrity-checked folder and load from that path. Audit the model's .py files before first use.
• Run unprivileged and sandboxed. Use a dedicated non-root user, a container with no host mounts, and no outbound network beyond what's required.

5.2 Pickle deserialization via torch.load

The P-Tuning inference path calls torch.load(os.path.join(CHECKPOINT_PATH, "pytorch_model.bin")). torch.load uses Python pickle, which can execute arbitrary code during deserialization. Loading a checkpoint from an untrusted source is equivalent to running an untrusted program.

Mitigations: only load checkpoints you produced or trust; store them with restrictive permissions; prefer safetensors where a conversion is available; never load a user-uploaded .bin directly.

5.3 Unauthenticated network service (api.py)

uvicorn.run(app, host='0.0.0.0', port=8000, workers=1)

api.py binds to all interfaces with no authentication, no rate limiting, and no TLS. The single POST / endpoint runs inference for anyone who can reach the port, and it prints every prompt to stdout (the server logs user input in cleartext).

Mitigations:

• Bind to 127.0.0.1 and front the service with a reverse proxy (nginx/Caddy/Traefik) that enforces TLS, authentication, and rate limits.
• Place the service on a private network or behind a firewall; never expose port 8000 to the internet directly.
• Review the stdout logging if prompts may contain sensitive data — it logs prompt and response verbatim.
• workers=1 means the API serves one request at a time on a single GPU; do not assume concurrency.

5.4 Model-level safety limitations

The model itself can produce harmful, biased, or factually wrong content, is easily misled (including about its own identity), and is weak at math/logic. These are documented limitations, not bugs. Any user-facing deployment needs its own content moderation and output validation — do not rely on the base model for safety.

6. Deployment Patterns

The repository ships demos, not a production server — it is deliberately unopinionated about deployment. The patterns below are operator guidance for taking it to production.

6.1 Single-GPU inference service

The simplest production shape: harden api.py and run it as a managed service.

• Change host='0.0.0.0' to host='127.0.0.1'; add a reverse proxy for TLS + auth.
• Load at the quantization level that fits your card with headroom (INT8 on an 8 GB card, INT4 on 6 GB).
• Run under a process manager (systemd, supervisor) that restarts on crash.

Example systemd unit (sketch):

[Unit]
Description=ChatGLM-6B API
After=network.target

[Service]
User=chatglm
WorkingDirectory=/opt/chatglm-6b
ExecStart=/opt/chatglm-6b/venv/bin/python api.py
Restart=on-failure
Environment=CUDA_VISIBLE_DEVICES=0

[Install]
WantedBy=multi-user.target

6.2 Containerized deployment

• Base image: an nvidia/cuda runtime image matching your driver, plus Python 3.10.
• Bake transformers==4.27.1 and the rest of requirements.txt into the image.
• Bake or volume-mount the weights so the container does not download from the Hub at start (also closes the RCE-on-pull window).
• Run with --gpus all, a non-root user, and no host mounts beyond the weights.

Tip

Pin every dependency in the image. Because the upstream repo is in maintenance, an unpinned pip install can pull a transformers version the model's remote code does not support.

6.3 Multi-GPU for large/long contexts

Use utils.load_model_on_gpus(..., num_gpus=N) (Section 2.4) when one card cannot hold the model. Note this splits one model across cards for capacity — it does not increase throughput. For throughput, run independent single-GPU replicas behind a load balancer.

6.4 Capacity notes

• api.py is single-worker and synchronous: one in-flight request per process. Scale by running multiple replicas (one per GPU) behind a proxy, not by raising workers on one GPU.
• VRAM grows with conversation length; enforce a max_length ceiling and consider trimming history server-side for long sessions.

7. Operations

7.1 Starting and stopping

7.2 GPU memory management

api.py calls torch_gc() after each request to release cache and IPC handles:

def torch_gc():
    if torch.cuda.is_available():
        with torch.cuda.device(CUDA_DEVICE):
            torch.cuda.empty_cache()
            torch.cuda.ipc_collect()

If you build your own server, replicate this between requests. If VRAM stays pinned despite empty_cache(), the only reliable recovery is to restart the process (a known issue — see #250, #1144).

7.3 Monitoring

There is no built-in metrics endpoint. For production, add:

• GPU utilization/VRAM monitoring (nvidia-smi, DCGM exporter).
• Latency and error-rate metrics at the reverse-proxy layer.
• Log rotation — api.py prints every prompt/response to stdout; capture and rotate it, and scrub if prompts are sensitive.

7.4 Fine-tuning operations runbook

1. Install fine-tuning deps: pip install rouge_chinese nltk jieba datasets.
2. Stage the dataset so AdvertiseGen/ (or your data) sits in ptuning/.
3. cd ptuning && bash train.sh — checkpoints land in output/adgen-chatglm-6b-pt-128-2e-2/checkpoint-* every 1000 steps.
4. bash evaluate.sh — predictions and Rouge/BLEU-4 scores in generated_predictions.txt.
5. Deploy the prefix checkpoint by loading the base model + PrefixEncoder weights (User Guide §7.5).

8. Internals

8.1 Inference flow

user text → tokenizer.encode → model.stream_chat → (28 GLM layers) → logits
          → sampling (top_p, temperature) → detokenize → yield partial response

stream_chat yields the cumulative response on each step, which is what produces the typewriter effect in the demos. chat is the non-streaming equivalent.

8.2 P-Tuning v2 mechanism

P-Tuning v2 freezes the entire base model and trains a small PrefixEncoder that injects learned key/value prefixes into every attention layer (length pre_seq_len). This is why:

• A P-Tuning checkpoint contains only transformer.prefix_encoder.* weights (a few MB), not a full model.
• Inference must load the base model first, then graft the prefix weights on top (model.transformer.prefix_encoder.load_state_dict(...)).
• prefix_projection=False (default) uses a plain embedding prefix; setting it True adds an MLP reparameterization.

8.3 Quantization

.quantize(4|8) is implemented in the model's remote code and depends on cpm_kernels for the GPU path. On CPU, the kernels are compiled at runtime via gcc/OpenMP. The prefix encoder is kept in FP32 even when the base model is quantized (model.transformer.prefix_encoder.float()), because the small prefix is cheap and benefits from full precision.

8.4 Repository structure

ChatGLM-6B/
├── api.py                 FastAPI REST server (POST /)
├── cli_demo.py            Terminal streaming chat
├── cli_demo_vision.py     VisualGLM terminal demo
├── web_demo.py            Gradio UI (current)
├── web_demo2.py           Streamlit UI (community)
├── web_demo_old.py        Gradio UI (legacy, no typewriter)
├── web_demo_vision.py     VisualGLM Gradio UI
├── utils.py               Multi-GPU device-map helper
├── requirements.txt       Runtime dependencies
├── FAQ.md / UPDATE.md / PROJECT.md
├── examples/              Capability screenshots
├── limitations/           Failure-case screenshots
├── improve/               Community fine-tuning data sample
└── ptuning/
    ├── main.py            Seq2seq train/eval driver
    ├── arguments.py       Model/Data argument dataclasses
    ├── trainer.py         Vendored HF Trainer (4.27.1)
    ├── trainer_seq2seq.py Vendored HF Seq2SeqTrainer
    ├── deepspeed.json     ZeRO stage-2 config
    ├── train.sh / train_chat.sh / ds_train_finetune.sh
    ├── evaluate.sh / evaluate_finetune.sh / web_demo.sh
    └── web_demo.py        Web demo that loads a prefix checkpoint

9. Troubleshooting (operational)

10. Suggested Improvements & Technical Debt

These observations are derived from the source and the issue tracker. They are guidance for operators forking or maintaining the code internally — upstream is in maintenance and unlikely to action them.

• Pin transformers explicitly in requirements.txt. It currently lists transformers==4.27.1, which is correct — keep it; do not relax to a range. A surprising number of issues stem from version drift.
• Harden api.py before any networked use. Add authentication, bind to localhost, add TLS via a proxy, and stop logging full prompts to stdout. (See Section 5.3.)
• Prefer safetensors over torch.load for checkpoints to remove the pickle RCE surface where conversions exist.
• Migrate off trust_remote_code by vendoring and auditing the model code, then loading from a fixed local path with a pinned revision.
• De-duplicate the demos. web_demo.py, web_demo_old.py, and ptuning/web_demo.py share most of their logic; cli_demo.py/cli_demo_vision.py likewise. A small shared module would reduce drift.
• Consider the newer models. For new work, ChatGLM2-6B/ChatGLM3/GLM-4 supersede this model on quality, context length (32K+), and inference speed. Document the migration path for downstream consumers.
• Add a metrics/health endpoint to api.py for production observability (currently only the inference POST / exists).

11. Quick Reference

Loading recipes

Key facts