# funNLP — Administrator & Developer Guide

## 1. Architecture Overview

funNLP is a static dataset repository — a curated collection of Chinese NLP lexical resources organized into 24+ domain-specific directories under `data/`. There is no runtime application, no server, no build system, and no API. The repository's architecture is its file organization: a flat hierarchy of themed dictionary directories containing plain-text corpus files.

### Repository Architecture

```text
funNLP/
├── README.md              ← Curated directory of external NLP resources (links)
└── data/                  ← All included datasets
    ├── IT词库/            ← THUOCL domain dictionaries (×10 domains)
    ├── 医学词库/
    ├── 法律词库/
    ├── 财经词库/
    ├── 食物词库/
    ├── 动物词库/
    ├── 地名词库/
    ├── 历史名人词库/
    ├── 诗词短句词库/
    ├── 汽车品牌、零件词库/
    ├── 古诗词库/          ← 30 themed classical poetry files
    ├── 停用词/            ← 4 university-curated stop word lists
    ├── 同义词库、反义词库、否定词库/  ← Lexical relation dictionaries
    ├── 成语词库/          ← Idiom corpus (5W entries)
    ├── 中英日文名字库/    ← Multilingual name corpora (120W+ entries)
    │   ├── Chinese_Names_Corpus/
    │   ├── English_Names_Corpus/
    │   └── Japanese_Names_Corpus/
    ├── 公司名字词库/      ← Company/org names (480W+, RAR archives)
    ├── 中文缩写库/        ← Abbreviation expansion (train/dev/test split)
    ├── 职业词库/          ← Occupation classification
    ├── 拆字词库/          ← Character decomposition (trad. + simp.)
    ├── 繁简体转换词库/    ← Traditional↔Simplified conversion index
    ├── 中文分词词库整理/  ← Segmentation dictionaries (300K+ entries)
    ├── 中文谣言数据/      ← Rumor detection dataset (JSON, 2017)
    ├── NLP_BOOK/          ← NLP reference textbooks (PDF)
    └── paper/             ← Research papers (PDF)
```

> [!NOTE]
> This is a data-only repository. There is no application runtime, no deployment pipeline, and no service infrastructure. Administration concerns are limited to storage, data integrity, and contribution management.

### Component Inventory

| Component | Path | Role | Format |
|---|---|---|---|
| THUOCL Domain Dicts | `data/{domain}词库/THUOCL_*.txt` | Domain-specific word frequency lists from Tsinghua University | TSV: `word\tfrequency` |
| Stop Word Lists | `data/停用词/*.txt` | Text preprocessing — common word removal | One word per line |
| Lexical Relations | `data/同义词库、反义词库、否定词库/` | Synonym, antonym, and negation word sets | One word/pair per line |
| Name Corpora | `data/中英日文名字库/` | Chinese/English/Japanese name datasets with metadata | TXT (one per line), XLSX |
| Company Names | `data/公司名字词库/` | Organization and company name datasets | RAR-compressed TXT |
| Classical Poetry | `data/古诗词库/` | Tang/Song dynasty poetry by theme (30 categories) | One poem per entry |
| Idiom Corpus | `data/成语词库/` | Chinese four-character idioms | TXT |
| Segmentation Dicts | `data/中文分词词库整理/` | Multiple word segmentation dictionaries | TXT, various encodings |
| Abbreviation Data | `data/中文缩写库/` | Chinese abbreviation expansion (ML-ready split) | Train/dev/test TXT |
| Character Decomp. | `data/拆字词库/` | Radical decomposition tables | TXT: `char\tradicals` |
| Trad./Simp. Conv. | `data/繁简体转换词库/` | Character conversion index | TXT mapping |
| Rumor Dataset | `data/中文谣言数据/` | Rumor detection training data | JSON |
| Occupation Dict | `data/职业词库/` | National occupation classification | TXT + DOC |
| NLP Books | `data/NLP_BOOK/` | Reference texts (Eisenstein NLP notes) | PDF |
| Research Papers | `data/paper/` | Academic papers (LLM survey) | PDF |
| Dict Cleaner Script | `data/中文分词词库整理/thirtyw.py` | One-off Python 2 script for dict processing | Python 2 |

### Technology Stack

| Layer | Technology |
|---|---|
| Version control | Git / GitHub |
| Data format | Plain text (UTF-8), JSON, XLSX, DOC, PDF, RAR |
| Only code | Python 2 (single utility script) |
| No runtime deps | No package manager, no build system |

## 2. System Requirements

| Resource | Minimum | Notes |
|---|---|---|
| Disk space | ~500 MB | Includes RAR archives; extracted company name corpus adds ~1 GB |
| Git | Any version | LFS not required — all files are committed directly |
| Python | 2.7+ (only for `thirtyw.py`) | Not needed for general dataset use |
| RAR extractor | `unrar` or equivalent | Only for `data/公司名字词库/*.rar` |
| Filesystem | UTF-8 support | Directory names contain Chinese characters |

> [!IMPORTANT]
> The repository uses Chinese characters in directory and file names extensively. Git on Windows may require `core.quotePath=false` to display these correctly:
> ```bash
> git config --global core.quotePath false
> ```

## 3. Configuration Reference

This repository has no configuration files. There are no settings, environment variables, or config formats to manage.

The only data-processing code is `data/中文分词词库整理/thirtyw.py`, a self-contained Python 2 script with hardcoded input/output paths:

| Parameter | Value | Hardcoded In |
|---|---|---|
| Input file | `30wChinsesSeqDic.txt` | `thirtyw.py` line 2 |
| Output file | `30wdict.txt` | `thirtyw.py` line 3 |
| Filter threshold | `len > 2` characters | `thirtyw.py` line 9 |

> [!NOTE]
> The script's outputs (`30wdict.txt`, `30wdict_utf8.txt`) are already committed to the repository. There is no need to run this script unless reprocessing the source dictionary.

## 4. Deployment & Scaling

### Distribution

This is a static data repository. "Deployment" means making the data available to consumers:

**Direct clone:**

```bash
git clone https://github.com/fighting41love/funNLP.git
```

**Sparse checkout (selective datasets):**

```bash
git clone --no-checkout https://github.com/fighting41love/funNLP.git
cd funNLP
git sparse-checkout init --cone
git sparse-checkout set data/停用词 data/成语词库
git checkout
```

**Mirror for internal use:**

```bash
git clone --mirror https://github.com/fighting41love/funNLP.git funNLP.git
```

### Storage Considerations

| Dataset | Approximate Size | Notes |
|---|---|---|
| THUOCL dictionaries (all) | ~5 MB | Small text files |
| Name corpora | ~200 MB | Large text files (120W+ entries) |
| Company name archives | ~300 MB | RAR compressed; ~1 GB extracted |
| Segmentation dicts | ~100 MB | Multiple dictionary formats |
| Classical poetry | ~2 MB | 30 small text files |
| PDF materials | ~50 MB | Books and papers |

> [!TIP]
> For CI/CD pipelines that only need specific dictionaries, use Git sparse checkout to minimize clone size and build time.

## 5. Operations

### Data Integrity Verification

Since this is a static dataset, verify integrity after cloning:

```bash
# Count files per directory
find data -type f | wc -l
```

```text
Expected: ~100 files
```

```bash
# Verify no corrupted text files (check for valid UTF-8)
find data -name "*.txt" -exec file {} \; | grep -v "UTF-8\|ASCII"
```

> [!NOTE]
> The `data/中文分词词库整理/.DS_Store` file is a macOS artifact and can be safely ignored or removed.

> [!TIP]
> Some text files in `data/中文分词词库整理/` may use GB2312 or GBK encoding rather than UTF-8. The `30wdict_utf8.txt` variant is the UTF-8 conversion of `30wdict.txt`.

### Archive Extraction

The company name corpus requires extraction:

```bash
cd data/公司名字词库
unrar x "Company-Names-Corpus（480W）.rar"
unrar x "Organization-Names-Corpus（110W）.rar"
```

```text
Extracting from Company-Names-Corpus（480W）.rar
...
All OK
```

### Monitoring Repository Updates

The repository has not been actively updated since 2023 (source commit `29f4ac8`, dated 2023-08-24). Community issues #456 and #457 ask about update status.

```bash
# Check for upstream changes
git fetch origin
git log HEAD..origin/master --oneline
```

## 6. Performance

### File I/O Optimization

When loading large dictionary files (name corpora, company names) into memory:

```python
# Efficient: generator for large files
def iter_dict(path):
    with open(path, encoding="utf-8") as f:
        for line in f:
            word = line.strip()
            if word:
                yield word

# Use set for O(1) lookup
medical_terms = set(iter_dict("data/医学词库/THUOCL_medical.txt"))
```

For the largest files (120W+ name entries), consider memory-mapped I/O or database loading rather than holding everything in memory simultaneously.

> [!TIP]
> For THUOCL files, parse both word and frequency if you need weighted matching:
> ```python
> def load_thuocl(path):
>     result = {}
>     with open(path, encoding="utf-8") as f:
>         for line in f:
>             parts = line.strip().split("\t")
>             if len(parts) == 2:
>                 result[parts[0]] = int(parts[1])
>     return result
> ```

## 7. Security

### Data Provenance

| Dataset Category | Source | Trust Level |
|---|---|---|
| THUOCL dictionaries | Tsinghua University NLP Lab | Academic institution — high trust |
| Name corpora | Community contributed | Verify before use in production |
| Company names | Community contributed | Contains real entity names — PII considerations |
| Rumor dataset | Community contributed | Contains real rumor text — content sensitivity |
| Stop words | Chinese universities (HIT, SCU) | Academic institution — high trust |

> [!WARNING]
> The company name corpus (`480W+ entries`) and name corpora (`120W+ entries`) contain real-world personally identifiable information (PII). If using these in applications subject to data protection regulations (GDPR, China's PIPL), ensure compliance with applicable data handling requirements.

> [!CAUTION]
> The rumor dataset (`data/中文谣言数据/rumors_v170613.json`) contains actual rumor text. Do not surface this content to end users without appropriate labeling. It is intended for training rumor detection models, not as a content source.

### Supply Chain Considerations

- No executable dependencies — the repository contains only data files and one inert Python 2 script
- No package manager artifacts — no risk of dependency confusion or supply chain attacks
- The `.pyc` file (`thirtyw.pyc`) is a compiled Python 2 bytecode artifact and is harmless but unnecessary

## 8. Developer Internals

### Repository Structure

```text
funNLP/
├── README.md                          ← Main content: curated NLP resource links
├── data/                              ← All included datasets (24+ subdirectories)
│   ├── IT词库/THUOCL_it.txt
│   ├── NLP_BOOK/eisenstein-nlp-notes.pdf
│   ├── paper/LLM_Survey_Chinese_0418.pdf
│   ├── 中文分词词库整理/             ← Largest module: segmentation dicts + utility
│   │   ├── thirtyw.py                ← Python 2 dict cleaner (only code in repo)
│   │   ├── thirtyw.pyc               ← Compiled bytecode artifact
│   │   ├── 30wChinsesSeqDic.txt      ← Source dictionary (input to thirtyw.py)
│   │   ├── 30wdict.txt               ← Cleaned output (words >2 chars)
│   │   ├── 30wdict_utf8.txt          ← UTF-8 conversion
│   │   └── ... (15+ dictionary files)
│   ├── 中文缩写库/                   ← ML-ready train/dev/test split
│   │   ├── train_set.txt
│   │   ├── dev_set.txt
│   │   └── test_set.txt
│   ├── 中文谣言数据/
│   │   ├── README.md                 ← Dataset-specific documentation
│   │   └── rumors_v170613.json       ← Rumor entries (JSON)
│   ├── 中英日文名字库/
│   │   ├── 萌名-中文人名生成器V1.1.xlsx
│   │   ├── Chinese_Names_Corpus/     ← 120W entries, gender labels
│   │   ├── English_Names_Corpus/     ← 48W entries, CN translations
│   │   └── Japanese_Names_Corpus/    ← 18W entries
│   ├── 停用词/                       ← 4 stop word lists
│   ├── 公司名字词库/                 ← RAR archives (480W+ companies, 110W+ orgs)
│   ├── 古诗词库/                     ← 30 themed poetry files
│   ├── 同义词库、反义词库、否定词库/ ← 3 lexical relation files
│   ├── 成语词库/                     ← 5W idioms + THUOCL
│   ├── 拆字词库/                     ← Simplified + Traditional decomposition
│   ├── 繁简体转换词库/              ← Conversion index
│   └── 职业词库/                     ← Occupation classification (TXT + DOC)
```

### Module Reference

#### `data/中文分词词库整理/thirtyw.py` — Dictionary Cleaner

The only executable code in the repository. A Python 2 script that:

1. Reads `30wChinsesSeqDic.txt` (a raw 300K-entry Chinese dictionary)
2. Parses each line by splitting on space then tab
3. Filters entries to only those where the word is longer than 2 characters
4. Writes matching words to `30wdict.txt` (append mode)

```python
# Source: data/中文分词词库整理/thirtyw.py (complete, 15 lines)
f = open('30wChinsesSeqDic.txt')
fout = open('30wdict.txt','a')
count = 0
for line in f:
    temp = line.strip()
    temp_list = temp.split(' ')
    temp_sublist = temp_list[1].split('\t')
    if len(temp_sublist[1]) > 2:
        count = count + 1
        print temp_sublist[1]        # Python 2 print statement
        fout.write(temp_sublist[1] + '\n')
f.close()
fout.close()
```

**Issues:**
- Python 2 only (`print` without parentheses)
- Uses append mode (`'a'`) — running twice doubles the output
- No encoding specification (relies on system default)
- No error handling for malformed lines
- Hardcoded file paths (not parameterized)

> [!NOTE]
> This script is a historical artifact. Its output files are already committed. A modern equivalent would take ~5 lines of Python 3 with proper encoding handling.

### Data Format Reference

| Format | Files | Structure | Encoding |
|---|---|---|---|
| THUOCL TSV | `THUOCL_*.txt` | `word\tfrequency` per line | UTF-8 |
| Plain word list | Most `*.txt` | One word per line | UTF-8 (some GB2312) |
| Name + gender | `*_Gender*.txt` | `name\tgender` per line | UTF-8 |
| JSON dataset | `rumors_v170613.json` | JSON array of rumor objects | UTF-8 |
| Character decomp. | `chaizi-*.txt` | `character\tradical1 radical2` | UTF-8 |
| Conversion index | `fanjian_suoyin.txt` | `traditional\tsimplified` per line | UTF-8 |
| ML split | `train_set.txt`, `dev_set.txt`, `test_set.txt` | Abbreviation expansion pairs | UTF-8 |
| Compressed | `*.rar` | RAR archive containing TXT | N/A |
| Binary docs | `*.xlsx`, `*.doc` | Microsoft Office formats | N/A |
| Academic | `*.pdf` | PDF documents | N/A |

### Code Patterns

**Pattern: Flat data organization.** Each dataset lives in its own descriptively-named Chinese directory. No nesting beyond one level (except name corpora, which split by language). This makes discovery trivial but requires UTF-8 filesystem support.

**Pattern: THUOCL as baseline.** Ten domain dictionaries follow the identical THUOCL format (word + tab + frequency). This consistency enables generic loaders that work across all domains.

**Pattern: No build/test infrastructure.** The repository has no tests, no CI, no linting, no packaging. This is appropriate for a static data collection but means data quality depends entirely on manual curation.

### Testing Strategy

There are no automated tests. For a data repository, quality assurance would involve:

- **Encoding validation:** Ensure all `.txt` files are valid UTF-8
- **Format consistency:** Verify THUOCL files follow `word\tfrequency` format
- **Duplicate detection:** Check for duplicate entries within dictionaries
- **Completeness checks:** Verify RAR archives extract successfully

### Potential Improvements

1. **Add a `requirements.txt` or `pyproject.toml`** — Even if minimal, it signals the project is Python-ecosystem adjacent
2. **Convert `thirtyw.py` to Python 3** — The Python 2 script is non-functional on modern installations
3. **Replace RAR archives with open formats** — ZIP or gzip would eliminate the `unrar` dependency
4. **Add data validation CI** — GitHub Actions workflow to check encoding and format consistency on PR
5. **Standardize encoding** — Convert all files to UTF-8 (some segmentation dicts use GB2312)
6. **Remove `.DS_Store` and `.pyc` files** — Add a `.gitignore` to prevent OS/build artifacts
7. **Add dataset metadata** — A `manifest.json` describing each dataset's source, license, size, and format
8. **Provide unified loader** — A Python module that loads any dictionary by category name with proper encoding handling
9. **Document licensing per dataset** — THUOCL, community-contributed, and scraped data have different terms

### Contributing

To add a new dataset:

1. Create a directory under `data/` with a descriptive Chinese name
2. Place data files in plain text (UTF-8), one entry per line where applicable
3. Follow the THUOCL format (`word\tfrequency`) for frequency-annotated dictionaries
4. Add a `README.md` in the dataset directory if the data requires explanation (see `data/中文谣言数据/README.md` for an example)
5. Open a pull request with a description of the data source and scope

> [!IMPORTANT]
> Do not commit datasets containing sensitive personal data (ID numbers, phone numbers, addresses) or data scraped in violation of terms of service. Name corpora and company name lists are acceptable as they are publicly available information.

## 9. Integration Points

### NLP Framework Compatibility

| Framework | Dictionary Loading | Notes |
|---|---|---|
| jieba | `jieba.load_userdict(path)` | Accepts THUOCL format directly |
| HanLP | Custom dictionary configuration | Requires conversion for some formats |
| pkuseg | `pkuseg.pkuseg(user_dict=path)` | One-word-per-line format |
| LAC (Baidu) | Custom dictionary support | Accepts word list format |
| LTP | Lexicon augmentation | Requires specific format conversion |

### Data Pipeline Integration

For ETL pipelines that consume these dictionaries:

```python
import os
import glob

def load_all_thuocl(data_dir="data"):
    """Load all THUOCL dictionaries into a unified frequency map."""
    combined = {}
    for path in glob.glob(os.path.join(data_dir, "**/THUOCL_*.txt"), recursive=True):
        domain = os.path.basename(os.path.dirname(path))
        with open(path, encoding="utf-8") as f:
            for line in f:
                parts = line.strip().split("\t")
                if len(parts) == 2:
                    word, freq = parts[0], int(parts[1])
                    combined[word] = combined.get(word, 0) + freq
    return combined
```

### Database Loading

For applications that need indexed search over the dictionaries:

```sql
CREATE TABLE dictionary (
    word TEXT NOT NULL,
    frequency INTEGER DEFAULT 0,
    domain TEXT NOT NULL,
    source TEXT DEFAULT 'THUOCL'
);

CREATE INDEX idx_word ON dictionary(word);
CREATE INDEX idx_domain ON dictionary(domain);
```

---

_Generated May 2026 from fighting41love/funNLP at 29f4ac8. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._