funNLP — User Guide

1. Quick Start

funNLP is a curated collection of Chinese Natural Language Processing (NLP) datasets, dictionaries, and corpora. It provides ready-to-use lexical resources for text segmentation, named entity recognition, sentiment analysis, and other NLP tasks — organized by domain and immediately usable in any Chinese NLP pipeline.

Key resources included:

• 24+ domain-specific Chinese dictionaries (legal, medical, financial, IT, food, animals, and more)
• Chinese/English/Japanese name corpora with 120W+ entries and gender annotations
• Multiple stop word lists from leading Chinese universities
• Synonym, antonym, and negation word dictionaries
• Classical Chinese poetry organized by 30 themes
• Chinese abbreviation and idiom databases
• Company and organization name corpora (480W+ entries)
• Chinese rumor detection dataset
• Traditional/Simplified Chinese conversion tables
• Character decomposition (拆字) dictionaries
• Chinese word segmentation dictionaries (300K+ entries across multiple formats)

Prerequisites: Git (to clone the repository)

git clone https://github.com/fighting41love/funNLP.git
cd funNLP/data

Browse the data/ directory — each subdirectory contains a themed dictionary or corpus in plain text format, ready for direct use.

Tip

Most dictionary files are plain UTF-8 text with one entry per line. You can load them directly into any NLP framework (jieba, HanLP, pkuseg, etc.) as custom dictionaries.

What to try next:

• Browse the dataset inventory below to find resources for your task
• See "Loading Dictionaries" in Core Usage for integration examples
• Check the file format reference for each dataset type

2. Installation

System Requirements

Clone the Repository

git clone https://github.com/fighting41love/funNLP.git

Cloning into 'funNLP'...
remote: Enumerating objects: ...
Resolving deltas: 100% ... done.

Important

Some files contain Chinese characters in their filenames. Ensure your filesystem and terminal support UTF-8 encoding. On Windows, use Git Bash or WSL to avoid path encoding issues.

Verify Installation

ls data/

You should see 24+ subdirectories, each containing domain-specific dictionary files:

IT词库          NLP_BOOK        中文分词词库整理  中文缩写库
中文谣言数据    中英日文名字库  停用词            公司名字词库
动物词库        医学词库        历史名人词库      古诗词库
同义词库、反义词库、否定词库    地名词库          成语词库
拆字词库        汽车品牌、零件词库  法律词库      繁简体转换词库
职业词库        诗词短句词库    财经词库          食物词库
paper

Download Specific Datasets Only

If you only need certain dictionaries, use a sparse checkout:

git clone --no-checkout https://github.com/fighting41love/funNLP.git
cd funNLP
git sparse-checkout init --cone
git sparse-checkout set data/停用词 data/成语词库
git checkout

Note

The data/公司名字词库 directory contains .rar archives. You need unrar or a compatible tool to extract them.

3. Core Usage

Dataset Inventory

Loading Dictionaries

Most dictionaries are plain text files with one entry per line. Here are common integration patterns:

Load a dictionary into Python:

with open("data/停用词/中文停用词库.txt", encoding="utf-8") as f:
    stopwords = set(line.strip() for line in f if line.strip())

>>> len(stopwords)
1893

Use as a jieba custom dictionary:

import jieba

jieba.load_userdict("data/IT词库/THUOCL_it.txt")
words = jieba.lcut("深度学习框架TensorFlow非常流行")
print(words)

Tip

THUOCL dictionary files use tab-separated format: word\tfrequency. Most Chinese NLP tools (jieba, HanLP, pkuseg) accept this format directly as custom dictionaries.

Load the rumor dataset:

import json

with open("data/中文谣言数据/rumors_v170613.json", encoding="utf-8") as f:
    rumors = json.load(f)

Note

The rumor dataset is dated June 2017. Treat it as a historical snapshot for model training, not as a current rumor database.

THUOCL Dictionary Format

Many dictionaries in this collection originate from the Tsinghua University Open Chinese Lexicon (THUOCL). These files follow a consistent format:

深度学习	�1234
机器学习	5678
自然语言处理	2345

Each line contains a word and its frequency count, separated by a tab character. Higher frequency indicates more common usage in the Tsinghua corpus.

Stop Word Lists

Four curated stop word lists are provided, each from a different academic source:

Warning

These lists have different coverage and aggressiveness. The Harbin Institute list is conservative (fewer words removed), while the general Chinese list is broader. Choose based on your task — aggressive filtering helps topic modeling but harms tasks like sentiment analysis where negation words matter.

Classical Poetry Collection

The data/古诗词库/ directory contains classical Chinese poetry organized by theme:

Name Corpora

The data/中英日文名字库/ directory provides multilingual name datasets:

Chinese Names (Chinese_Names_Corpus/):

• Chinese_Names_Corpus（120W）.txt — 1.2 million Chinese names
• Chinese_Names_Corpus_Gender（120W）.txt — Names with gender labels
• Ancient_Names_Corpus（25W）.txt — 250K historical/classical names
• Chinese_Family_Name（1k）.xlsx — ~1,000 Chinese family names
• Chinese_Relationship（4.8k）.xlsx — 4,800 relationship terms

English Names (English_Names_Corpus/):

• English_Names_Corpus（2W）.txt — 20K English names
• English_Cn_Name_Corpus（48W）.txt — 480K English names with Chinese translations
• English_Cn_Name_Corpus_Gender（48W）.txt — With gender labels

Japanese Names (Japanese_Names_Corpus/):

• Japanese_Names_Corpus（18W）.txt — 180K Japanese names
• Japanese_Names_Corpus（1W）.xlsx — 10K names in spreadsheet format

Tip

The gender-annotated name files are particularly useful for training NER (Named Entity Recognition) models and gender prediction classifiers.

Character Decomposition Dictionary

The data/拆字词库/ directory contains character-to-radical decomposition tables:

• chaizi-jt.txt — Simplified Chinese character decomposition
• chaizi-ft.txt — Traditional Chinese character decomposition

These are useful for character-level NLP tasks, Chinese input method development, and educational applications.

4. Common Tasks

Build a Domain-Specific NER System

Goal: Create a named entity recognizer using the provided dictionaries.

Steps:

1. Load relevant dictionaries as custom word lists:

import jieba

# Load domain dictionaries
jieba.load_userdict("data/医学词库/THUOCL_medical.txt")
jieba.load_userdict("data/地名词库/THUOCL_diming.txt")
jieba.load_userdict("data/历史名人词库/THUOCL_lishimingren.txt")

2. Segment text with domain awareness:

text = "张仲景在南阳研究伤寒杂病论"
words = jieba.lcut(text)
print(words)

3. Match segmented tokens against dictionary sets for entity classification.

Create a Chinese Text Preprocessing Pipeline

Goal: Build a standard preprocessing pipeline using funNLP resources.

Steps:

1. Load stop words:

with open("data/停用词/哈工大停用词表.txt", encoding="utf-8") as f:
    stopwords = set(line.strip() for line in f)

2. Load a segmentation dictionary:

import jieba
jieba.load_userdict("data/中文分词词库整理/30wdict_utf8.txt")

3. Process text:

def preprocess(text):
    words = jieba.lcut(text)
    return [w for w in words if w.strip() and w not in stopwords]

Build a Sentiment Lexicon

Goal: Combine synonym, antonym, and negation dictionaries for sentiment analysis.

Steps:

1. Load the three lexical relation files:

def load_dict(path):
    with open(path, encoding="utf-8") as f:
        return [line.strip() for line in f if line.strip()]

synonyms = load_dict("data/同义词库、反义词库、否定词库/同义词库.txt")
antonyms = load_dict("data/同义词库、反义词库、否定词库/反义词库.txt")
negations = load_dict("data/同义词库、反义词库、否定词库/否定词库.txt")

2. Use negation words to flip sentiment polarity during analysis.

Warning

The negation dictionary is essential for accurate Chinese sentiment analysis. Removing negation words during stop word filtering will degrade sentiment classification accuracy. Always exclude negation words from your stop word list when building sentiment models.

Generate Random Chinese Names

Goal: Sample realistic Chinese names for test data generation.

Steps:

import random

with open("data/中英日文名字库/Chinese_Names_Corpus/Chinese_Names_Corpus（120W）.txt",
          encoding="utf-8") as f:
    names = [line.strip() for line in f if line.strip()]

sample = random.sample(names, 10)
print(sample)

Convert Between Traditional and Simplified Chinese

Goal: Use the conversion dictionary for character mapping.

conversion = {}
with open("data/繁简体转换词库/fanjian_suoyin.txt", encoding="utf-8") as f:
    for line in f:
        parts = line.strip().split()
        if len(parts) >= 2:
            conversion[parts[0]] = parts[1]

5. Troubleshooting

Error: UnicodeDecodeError when reading dictionary files

Cause: Files are encoded in UTF-8 but your script uses the system default encoding.

Solution:

1. Always specify encoding="utf-8" when opening files:

with open("data/停用词/中文停用词库.txt", encoding="utf-8") as f:
    content = f.read()

2. On Windows, set the environment variable PYTHONUTF8=1 for system-wide UTF-8 default.

Prevention: Always use explicit encoding="utf-8" in every open() call when working with this dataset.

Error: Cannot extract .rar archives in company name corpus

Cause: The data/公司名字词库/ directory contains .rar compressed files that require a RAR-compatible extractor.

Solution:

# Linux/macOS
sudo apt install unrar  # or: brew install unrar
unrar x "data/公司名字词库/Company-Names-Corpus（480W）.rar"

# Windows
# Use 7-Zip or WinRAR to extract

Prevention: Consider converting extracted data to .txt for easier reuse.

Error: Chinese characters display as garbled text in filenames

Cause: The filesystem or terminal does not support UTF-8 encoded filenames.

Solution:

1. On Windows, use Windows Terminal or Git Bash instead of CMD
2. On Linux, ensure locale is set: export LANG=en_US.UTF-8
3. Use ls with proper locale settings to verify directory names display correctly

Prevention: Use a UTF-8 compatible terminal and filesystem from the start.

Warning

Avoid renaming directories to ASCII equivalents — this breaks relative path references and makes it harder to match files back to the dataset inventory.

Issue: Dictionary file appears empty or has only headers

Cause: Some dictionary files may use different line endings (CR/LF vs LF) or BOM markers.

Solution:

with open(filepath, encoding="utf-8-sig") as f:  # handles BOM
    lines = [line.strip() for line in f if line.strip()]

Tip

The utf-8-sig encoding in Python automatically strips the BOM (Byte Order Mark) that some Windows-created text files include.

Issue: thirtyw.py fails to run

Cause: This script uses Python 2 syntax (print without parentheses) and is a one-off data cleaning utility, not a general-purpose tool.

Solution: This script is a historical artifact for processing the 30wChinsesSeqDic.txt file. Its output (30wdict.txt) is already included in the repository. You do not need to run it.

6. FAQ

Q: What does "funNLP" stand for?

The name combines "fun" with "NLP" (Natural Language Processing). The repository is a curated collection of Chinese NLP resources — dictionaries, corpora, datasets, and academic materials — gathered for practical use in NLP tasks.

Q: What are the THUOCL files?

THUOCL stands for THUinghua University Open Chinese Lexicon. These are domain-specific word frequency lists released by Tsinghua University's NLP lab. Each file contains words and their corpus frequency counts in tab-separated format.

Q: Are these datasets suitable for commercial use?

Licensing varies by dataset. The THUOCL dictionaries are released by Tsinghua University for research use. The name corpora and other datasets may have different terms. Check the original source of each dataset before commercial use.

Q: How current are these datasets?

The repository's last source commit is dated 2023-08-24. The rumor dataset is from 2017. Most dictionaries are static lexical resources that do not require frequent updates, though domain-specific terminology (IT, finance) may benefit from supplementation with newer terms.

Q: Can I use these with tools other than jieba?

Yes. The plain-text, one-word-per-line format is compatible with virtually any Chinese NLP tool including HanLP, pkuseg, LAC, LTP, and custom implementations. The THUOCL tab-separated format (word + frequency) is directly accepted by most segmentation tools.

Q: Which stop word list should I use?

It depends on your task:

For most tasks, start with the general Chinese stop word list and adjust based on results.

Q: The repository mentions many more resources in the README. Where are they?

The README serves as a curated directory of external NLP resources across the web. The data/ directory contains the datasets that are directly included in this repository. External resources linked in the README must be obtained from their respective sources.

Note

Community issues (#456, #457) indicate the repository is no longer actively updated. The included datasets remain useful as static lexical resources. For the latest NLP tools and models, supplement with actively maintained projects.

Q: How do I contribute a new dictionary?

Open a GitHub issue or pull request with your dataset placed in a new subdirectory under data/, following the existing naming convention (Chinese category name). Include a brief description of the source and scope.