Qlib Admin & Developer Guide

This guide is for the engineers, architects, and researchers who deploy Qlib, configure it for production research, extend its components, and reason about its internals. It assumes familiarity with the day-to-day usage covered in the User Guide and focuses on architecture, the full configuration surface, security posture, operations, and the code-level structure needed to extend or replicate the platform.

Note

Package identity: PyPI distribution pyqlib; import name qlib. Source is a src-less flat layout rooted at qlib/. The platform is pure Python with two compiled Cython extensions (qlib.data._libs.rolling, qlib.data._libs.expanding) built from setup.py.

1. Executive Summary

Qlib is an AI-oriented quantitative investment platform that covers the full research chain: data processing, feature engineering, model training, signal generation, portfolio construction, back-testing, and analysis. It is architected as a stack of loosely-coupled components, each usable stand-alone, wired together either declaratively (a single workflow YAML executed by qrun) or imperatively (Python "workflow by code").

Operational stance: Qlib is a library and research framework, not a long-running service. There is no API server, no authentication layer, and no network listener in the core. Its "production" footprint is the data directory, the MLflow tracking store, and optionally Redis/MongoDB. Security and operations therefore center on data integrity, dependency management, and reproducibility rather than on hardening a service surface.

2. Architecture

2.1 Component layers

┌─────────────────────────────────────────────────────────────┐
│ qlib.init() → QlibConfig (global C) → register providers/ops │
└─────────────────────────────────────────────────────────────┘
        │
        ▼
┌───────────────┐   Data Layer  (qlib/data)
│ Providers     │   LocalProvider → Calendar/Instrument/Feature/PIT/Expression/Dataset providers
│ Expression    │   $field raw access + operator graph (Ref, Mean, ...) compiled via _libs
│ Caches        │   ExpressionCache, DatasetCache (Disk* variants depend on Redis)
└───────────────┘
        │
        ▼
┌───────────────┐   Dataset Layer  (qlib/data/dataset)
│ DataHandler   │   Alpha158 / Alpha360 (qlib/contrib/data/handler)
│ DatasetH      │   train/valid/test segmentation, DataHandlerLP processing
└───────────────┘
        │
        ▼
┌───────────────┐   Model Layer  (qlib/model, qlib/contrib/model)
│ Model.fit     │   LGBModel + ~20 pytorch/tensorflow SOTA models
│ Model.predict │   Trainer / TrainerR / TrainerRM (MongoDB TaskManager)
└───────────────┘
        │
        ▼
┌───────────────┐   Workflow Layer  (qlib/workflow)
│ R (recorder)  │   QlibRecorder → MLflowExpManager → MLflowExperiment → MLflowRecorder
│ RecordTemp    │   SignalRecord → SigAnaRecord → PortAnaRecord
└───────────────┘
        │
        ▼
┌───────────────┐   Strategy + Execution  (qlib/strategy, qlib/backtest)
│ BaseStrategy  │   TopkDropoutStrategy, RL strategies
│ Executor      │   SimulatorExecutor, NestedExecutor, Exchange, Position, Account
└───────────────┘
        │
        ▼
┌───────────────┐   Analysis  (qlib/contrib/eva, contrib/evaluate, contrib/report)
│ IC / risk     │   calc_ic, risk_analysis, indicator_analysis
└───────────────┘

Every box is independently importable. The glue is config-driven instantiation: throughout Qlib, components are described by {class, module_path, kwargs} dicts and built at runtime by init_instance_by_config (qlib/utils). This is why a workflow YAML can swap any model, dataset, or strategy without code changes.

2.2 Initialization and the global config object

qlib.init() (qlib/__init__.py) is the single entry point that activates the framework:

1. Resets the global config C (a QlibConfig) from a mode template (client or server).
2. Applies region defaults (cn/us/tw) and any keyword overrides.
3. Resolves and (optionally) mounts the data path — including NFS auto-mount logic for shared deployments (_mount_nfs_uri).
4. Calls C.register(), which registers all expression operators (register_all_ops), data wrappers (register_all_wrappers), and constructs the QlibRecorder (R) backed by the configured experiment manager.

import qlib
from qlib.constant import REG_CN
qlib.init(provider_uri="~/.qlib/qlib_data/cn_data", region=REG_CN)

Alternative initializers:

Note

The config object C is a singleton (qlib/config.py, C = QlibConfig(_default_config)). qlib.init() is not safe to call twice while a recorder is active — the RecorderWrapper raises RecorderInitializationError to prevent silently relocating experiment storage.

2.3 Data path management

QlibConfig.DataPathManager resolves provider_uri (and mount_path) into concrete data URIs, distinguishing local paths from NFS paths by regex (get_uri_type). provider_uri may be a string or a per-frequency dict:

qlib_init:
    provider_uri:
        day:  "~/.qlib/qlib_data/cn_data"
        1min: "~/.qlib/qlib_data/cn_data_1min"

For NFS deployments, Qlib can auto-mount (auto_mount: True) on Windows (mount -o anon) or Linux/Unix (mount.nfs, requires nfs-common and sudo). This is the mechanism behind shared, multi-client data servers.

3. System Requirements

Important

Each benchmark model under examples/benchmarks/ pins its own dependency versions in a local requirements.txt (often specific torch/tensorflow/lightgbm versions). Treat these as authoritative per-model environments — run_all_model.py enforces this by building an isolated virtualenv per model.

4. Configuration Mastery

There are three distinct configuration surfaces in Qlib. Conflating them is the most common operational mistake.

1. Runtime config — the qlib.init() keyword arguments, layered over the _default_config dict in qlib/config.py.
2. Environment settings — the QSettings (pydantic-settings) object, prefix QLIB_.
3. Workflow config — the per-experiment YAML consumed by qrun (model/dataset/strategy/backtest).

4.1 Runtime config (_default_config)

These are the keys you can pass to qlib.init() or set in qlib_init:. Reproduced from qlib/config.py:

Mode templates (MODE_CONF) differ in cache defaults:

• client (default): caches disabled by default, region: cn, auto_mount: False, timeout: 100, custom_ops: [].
• server: DiskExpressionCache + DiskDatasetCache enabled (these depend on Redis).

Warning

DiskExpressionCache and DiskDatasetCache require a reachable Redis (DEPENDENCY_REDIS_CACHE). At init, if Redis is unreachable, Qlib silently disables these caches and logs a warning — it will not fail. Monitor for this warning in production; a disabled cache can silently 10–25× your data-loading time.

4.2 Region defaults

set_region() applies market-specific trading parameters:

4.3 Environment settings (QLIB_ prefix)

QSettings (qlib/config.py) reads from the environment with prefix QLIB_ and nested delimiter _:

Note

Settings precedence is: explicit qlib.init() kwargs > QLIB_* environment > built-in defaults. Use environment variables for deployment-level config (data location, MLflow store) and qlib.init() kwargs for per-run overrides.

4.4 Workflow config schema

The qrun YAML is the primary research configuration. Its sections (qlib_init, market, benchmark, data_handler_config, port_analysis_config, task.model, task.dataset, task.record) are documented in the User Guide. Two operational features worth highlighting:

• BASE_CONFIG_PATH — inherit from a base config and override only changed fields (update_config merge in qlib/cli/run.py). Enables a base-template + experiment-delta pattern across a model sweep.
• Jinja2 templating — configs are rendered as Jinja2 before parsing; undeclared {{ VAR }} variables are filled from matching environment variables (render_template). Enables parameter sweeps from a single file driven by env vars.

The qrun workflow handler (workflow() in qlib/cli/run.py) signature: workflow(config_path, experiment_name="workflow", uri_folder="mlruns"). If exp_manager is present under qlib_init, it is used directly; otherwise the MLflow URI is set to file:<cwd>/<uri_folder>.

5. Security

Qlib's core ships no network service, no auth, and no secret management — its threat surface is the data pipeline and the dependency graph, not an exposed endpoint. Security review should focus on the following.

5.1 Data provenance and integrity

• Bundled/collected market data is sourced from third parties (Yahoo Finance, Coingecko, baostock, East Money). The project explicitly warns this data "might not be perfect" and may contain anomalies even after price adjustment. Do not trust downloaded data blindly — run scripts/check_data_health.py and consider the cross-validated crowd_source data to reduce survivorship bias.
• The community data source (chenditc/investment_data) is a third-party GitHub release fetched over wget. Verify checksums/releases before ingestion in a controlled environment.

5.2 Code-execution surfaces

Caution

Qlib's config system is powerful by design and trusting by default. Several mechanisms execute or import arbitrary code from configuration:

• init_instance_by_config imports any module_path and instantiates any class named in a config — including loading a pickle object from a file:// path (InstConf in qlib/typehint.py).
• R.save_objects / load_object and model serialization use Python pickle. Never load a recorder artifact or a config from an untrusted source — pickle deserialization is arbitrary code execution.
• The sys section of a workflow config appends arbitrary paths to sys.path (sys_config in qlib/cli/run.py).
• Jinja2 config rendering pulls values from the process environment.

Treat workflow YAML files and saved artifacts as trusted code, not data. Source them only from your own pipeline.

5.3 Network and credentials

• NFS auto-mount (_mount_nfs_uri) shells out to sudo mount.nfs (Linux) or mount -o anon (Windows). In shared deployments this requires elevated privileges; scope it carefully and prefer pre-mounting volumes over auto_mount: True.
• Redis (redis_host/redis_port/redis_password) and MongoDB (mongo.task_url) connection details live in plain config. In production, supply credentials via environment/secret injection and bind these services to private networks — they are unauthenticated by default in the example configs (mongodb://localhost:27017/, redis 127.0.0.1).
• Data collectors make outbound HTTP requests to financial data providers; some (Yahoo) are geo-blocked from China.

5.4 Recommended hardening

• [ ] Run model sweeps (run_all_model.py) in isolated virtualenvs (the default) and ideally isolated containers.
• [ ] Pin and audit per-model requirements.txt — many pin old torch/tensorflow/numpy with known CVEs.
• [ ] Never deserialize third-party recorder artifacts or configs.
• [ ] Bind Redis/MongoDB to localhost or a private subnet; set redis_password and Mongo auth.
• [ ] Validate downloaded data with the health checker before training.

6. Operations

6.1 Deployment modes

Online mode is documented under the separate Qlib-Server project and supports one-click Azure deployment. In online mode, multiple clients share data and cache, improving cache-hit rate and reducing per-client disk usage.

6.2 The experiment store (MLflow)

All run state — params, metrics, tags, and artifacts (models, predictions, reports) — is persisted to the MLflow tracking URI (default file:./mlruns). Operationally:

• The URI can be a local file: path or a remote MLflow tracking server. For file URIs, use absolute paths (the backend rejects ~-style strings via set_uri).
• Concurrency: experiment creation is guarded by a filelock on file URIs (_get_or_create_exp) because MLflow does not lock multi-process experiment creation. For non-file schemes, Qlib double-checks via ExpAlreadyExistError.
• MLflow version compatibility is handled (search_experiments for v2+, list_experiments for v1).

from qlib.workflow import R
# Audit all runs of an experiment as a DataFrame
df = R.search_records([exp_id], order_by=["metrics.ic DESC"])
# Inspect a specific run's artifacts
rec = R.get_recorder(recorder_id=rid, experiment_name="my_exp")

Note

MLflow's search_runs caps results at 50,000 (UNLIMITED = 50000 in qlib/workflow/exp.py). For very large experiment stores, page or filter with filter_string.

6.3 Parallel & distributed training

Two trainer families (qlib/model/trainer.py):

For TrainerRM, run the orchestrator (train) on one node and worker() on others, all sharing one task_pool (MongoDB collection). This is Qlib's mechanism for scaling a model sweep across machines.

6.4 Data lifecycle operations

Tip

dump_bin.py dump_all parallelizes with --max_workers (default 16) and selects fields via --include_fields / --exclude_fields. For high-frequency data, set maxtasksperchild: 1 in the runtime config to bound per-process memory.

6.5 Logging

Logging is fully configurable through the logging_config runtime key (a logging.dictConfig). Notable built-ins (qlib/log.py):

• A custom LogFilter suppresses noisy "data not found" warnings by regex.
• set_global_logger_level(level) and the set_global_logger_level_cm context manager adjust handler levels at runtime.
• TimeInspector (logt context manager) instruments code blocks for timing — useful for profiling data-loading bottlenecks.

7. Repository Structure

qlib/
├── __init__.py            init(), auto_init(), NFS mount logic
├── config.py              QlibConfig, _default_config, MODE_CONF, region config, QSettings
├── constant.py            REG_CN/US/TW, EPS, INF
├── log.py                 logging, LogFilter, TimeInspector
├── typehint.py            InstConf / InstDictConf (config-as-instance contract)
├── cli/
│   ├── run.py             `qrun` workflow runner (fire-based)
│   └── data.py            `python -m qlib.cli.data` data downloader
├── data/                  Data layer: providers, expression engine, caches, _libs (Cython)
│   └── dataset/           DatasetH, DataHandler(LP), processors, samplers
├── model/                 base.py (Model/ModelFT), trainer.py (TrainerR/RM)
├── workflow/              R recorder, ExpManager, RecordTemp, task/ (TaskManager)
├── strategy/              BaseStrategy, RLStrategy, RLIntStrategy
├── backtest/              Exchange, Executor, Position, Account, decision
├── rl/                    Reinforcement-learning framework (interpreters, trainer, order_execution)
├── contrib/               Batteries-included: models, data handlers (Alpha158/360),
│                          strategies (TopkDropout), evaluation, reporting, rolling, tuner
├── utils/                 init_instance_by_config, serialization, parallel, time
└── tests/                 GetData, canned configs (CSI300_GBDT_TASK, CSI300_BENCH)

scripts/                   get_data.py, dump_bin.py, dump_pit.py, check_data_health.py,
└── data_collector/        per-source collectors (yahoo, cn_index, us_index, crypto, fund, pit, ...)

examples/                  workflow_by_code.py, run_all_model.py, benchmarks/<MODEL>/,
                           benchmarks_dynamic/, rl_order_execution/, nested_decision_execution/,
                           highfreq/, tutorial/

docs/                      Sphinx documentation (readthedocs)

7.1 Module reference (HIGH-priority)

8. Internals & Extension Points

8.1 Config-driven instantiation (the core idiom)

init_instance_by_config(config, accept_types=..., default_module=...) is the universal factory. A config can be:

• A dict {class, module_path, kwargs} (the common case).
• A class object directly, with kwargs.
• A string: a class name, a dotted a.b.c.ClassName, or a file://.../obj.pkl pickle path.
• An already-constructed instance (passed through).

The task executor (_exe_task in qlib/model/trainer.py) shows the full lifecycle: instantiate model + dataset, auto_filter_kwargs(model.fit)(dataset, reweighter=...), save the model and (config-only) dataset, fill the <MODEL>/<DATASET> placeholders via fill_placeholder, then run each record template's generate().

8.2 Writing a custom model

Subclass qlib.model.base.Model and implement fit and predict:

from qlib.model.base import Model
from qlib.data.dataset import Dataset
from qlib.data.dataset.handler import DataHandlerLP

class MyModel(Model):
    def fit(self, dataset: Dataset, reweighter=None):
        df_train, df_valid = dataset.prepare(
            ["train", "valid"], col_set=["feature", "label"], data_key=DataHandlerLP.DK_L
        )
        x_train, y_train = df_train["feature"], df_train["label"]
        # ... train self ...

    def predict(self, dataset: Dataset, segment="test"):
        x = dataset.prepare(segment, col_set="feature", data_key=DataHandlerLP.DK_I)
        return ...  # pandas.Series indexed by (datetime, instrument)

Note

Learned attributes must not start with _ so the model serializes cleanly (Serializable contract). For fine-tunable models, subclass ModelFT and add finetune(dataset).

8.3 Writing a custom strategy

Subclass qlib.strategy.base.BaseStrategy and implement generate_trade_decision(execute_result). The base provides executor, trade_calendar, trade_position, and trade_exchange via the shared level_infra / common_infra infrastructure objects. For RL strategies, subclass RLStrategy (carries a policy) or RLIntStrategy (adds state_interpreter / action_interpreter that translate between Qlib execution results and the RL env).

8.4 Custom expression operators

Pass custom_ops to qlib.init() — each element is either an ExpressionOps subclass or a {class, module_path} dict. They are registered by register_all_ops at init, extending the $field/operator expression language.

8.5 Record templates

Custom analysis = subclass RecordTemp (qlib/workflow/record_temp.py) and implement generate(). Templates declare artifact_path and an optional depend_cls so dependent records (e.g. SigAnaRecord depending on SignalRecord) can locate upstream artifacts. The built-in chain is SignalRecord → SigAnaRecord → PortAnaRecord.

9. Code Quality & Conventions

• Consistent factory pattern. Nearly every component is built through init_instance_by_config, giving a uniform, declarative, swap-anything design. This is the platform's defining architectural strength.
• MLflow wrapper rationale. qlib/workflow/__init__.py documents why Qlib wraps MLflow rather than using it directly: object-oriented recorder handles instead of bare run_ids, code-diff logging at run start, and log_object/load_object for Python objects. The cost is the up-front need to create an experiment before getting a recorder.
• Abstract base + concrete impl is used consistently (Experiment/MLflowExperiment, ExpManager/MLflowExpManager, Trainer/TrainerR/TrainerRM), keeping the backend swappable.
• Pydantic settings (QSettings) provide typed, env-overridable configuration.
• Cython hot paths. Rolling/expanding window operators are compiled C++ (qlib/data/_libs/*.pyx) for the expression engine — the source of Qlib's data-throughput advantage.

9.1 Performance characteristics

Qlib's own storage benchmark (14 features over ~800 stocks/day, 2007–2020) demonstrates the compact-binary + cache design:

(+E = ExpressionCache, +D = DatasetCache.) The takeaway for operators: enable both caches in any throughput-sensitive deployment.

10. Known Issues & Compatibility

These are confirmed from the project's issue tracker and release notes — track them when pinning dependencies.

11. Potential Improvements

For maintainers and contributors, the following are visible technical-debt and enhancement areas (several flagged in-repo):

• LightGBM 4.0+ support across all GBDT models and the DDG-DA pipeline — the most impactful compatibility gap.
• run_all_model.py portability — currently Linux-only and cannot run the same model in parallel across iterations; both are noted as planned.
• SeriesDFilter optimization — deduplicate filter-pipe execution across field groups to fix the slow-loading path.
• GeneralPtNN consolidation — migrate existing per-architecture NN models onto the unified GeneralPtNN design so swapping a PyTorch module is the only change needed.
• Data robustness — the project itself recommends users supply their own high-quality data; the crowd-sourced/cross-validated pipeline reduces survivorship bias and could become the default.
• Find rough edges — the README explicitly suggests rg 'TODO|FIXME' qlib to surface incomplete implementations.

12. Contributing

• Repo: https://github.com/microsoft/qlib (MIT, Microsoft Open Source CoC). A CLA is required on first PR.
• Dev install: pip install -e .[dev]; see docs/developer/code_standard_and_dev_guide.rst.
• Good entry points: the good first issue label, doc fixes, new models (examples/benchmarks/#contributing), new datasets, and bug fixes.
• Build docs: cd docs/ && make html (requires sphinx + sphinx_rtd_theme).
• CI reference: if local install/examples fail, compare against .github/workflows/test_qlib_from_source.yml.

Tip

When adding a model, follow the existing examples/benchmarks/<MODEL>/ convention: a workflow_config_*.yaml, a requirements.txt with pinned deps, and a README.md citing the source paper. This keeps the model zoo reproducible and run_all_model.py-compatible.

13. Quick Reference

Initialization

import qlib
from qlib.constant import REG_CN
qlib.init(provider_uri="~/.qlib/qlib_data/cn_data", region=REG_CN)

Key globals & factories

Operational commands

Service endpoints (optional, default insecure)