# Protocol Buffers (protobuf) — Administrator & Developer Guide

## 1. Architecture Overview

Protocol Buffers is Google's language-neutral, platform-neutral serialization framework consisting of three major architectural layers: a schema definition language (the `.proto` format), a compiler (`protoc`) that generates language-specific code from schemas, and per-language runtime libraries that provide serialization, deserialization, reflection, and message manipulation.

The project is organized as a multi-language monorepo with a shared C++ core, a micro protobuf C library (upb) used as the high-performance backend for several language runtimes, and individual language-specific implementations that may use native code, upb, or the C++ runtime as their backend.

```text
                          .proto files
                              │
                         ┌────▼────┐
                         │  protoc  │ (C++ compiler)
                         └────┬────┘
            ┌─────┬─────┬────┼────┬─────┬─────┬─────┐
            ▼     ▼     ▼    ▼    ▼     ▼     ▼     ▼
          C++   Java  Python C#  Ruby  PHP   Rust   Go*
          │      │     │     │    │     │     │      │
          ▼      ▼     ▼     ▼    ▼     ▼     ▼      ▼
       libprotobuf  protobuf  Google. google- PHP   protobuf
       (C++ lib)    -java     Protobuf protobuf      (crate)
                     │         (NuGet) (gem)   │      │
                     │                  │      │      │
                     ▼                  ▼      ▼      ▼
                  JVM runtime      upb (C)  upb (C)  upb/C++
                                  backend   backend  backend

  * Go runtime lives in separate repo: protobuf-go
```

> [!NOTE]
> The Go, Dart, and JavaScript protobuf implementations live in separate repositories (`protobuf-go`, `dart-lang/protobuf`, `protobuf-javascript`). This repository contains C++, Java, Python, C#, Ruby, PHP, Kotlin, Objective-C, and Rust implementations.

### Component Inventory

| Component | Path | Role |
|-----------|------|------|
| **protoc compiler** | `src/google/protobuf/compiler/` | Parses `.proto` files, runs code generators |
| **C++ runtime** | `src/google/protobuf/` | Core serialization library (171 source files, 3M chars) |
| **upb (micro protobuf)** | `upb/` | Lightweight C implementation used as backend for Python, Ruby, PHP, Rust |
| **upb code generator** | `upb_generator/` | Generates upb C code from `.proto` files |
| **Java runtime** | `java/` | Full + Lite + Util + Kotlin DSL |
| **Python runtime** | `python/` | C extension (upb-backed) + pure Python fallback |
| **C# runtime** | `csharp/` | .NET library targeting multiple frameworks |
| **Ruby runtime** | `ruby/` | C extension (CRuby) + FFI (JRuby) |
| **PHP runtime** | `php/` | Pure PHP + optional C extension |
| **Objective-C runtime** | `objectivec/` | iOS/macOS library (66 source files, 1.4M chars) |
| **Rust runtime** | `rust/` | V4 Google-supported, upb or C++ backend |
| **hpb (header protobuf)** | `hpb/` | C++ header-only protobuf API |
| **Conformance tests** | `conformance/` | Cross-language serialization compliance tests |
| **Editions** | `editions/` | Feature-based proto evolution system |
| **Examples** | `examples/` | AddressBook tutorial in Java, Python, Dart, Ruby, Go |
| **Build (Bazel)** | `bazel/`, `BUILD.bazel` | Primary build system rules |
| **Build (CMake)** | `cmake/`, `CMakeLists.txt` | Alternative build system |
| **CI** | `ci/` | Continuous integration scripts |
| **Benchmarks** | `benchmarks/` | Performance benchmarking suite |

### Technology Stack

| Layer | Technology |
|-------|-----------|
| Core language | C++ (C++17, ~3M chars in `src/google/protobuf/`) |
| Micro runtime | C (upb, ~1.3M chars across `upb/` subsystem) |
| Build systems | Bazel (primary), CMake (alternative) |
| Testing | Google Test (C++), JUnit 4 (Java), unittest (Python), PHPUnit (PHP), test-unit (Ruby) |
| Dependencies | abseil-cpp (C++ utilities), utf8_range, Guava (Java), JRuby (Ruby FFI) |
| CI | Kokoro (Google internal), AppVeyor (Windows) |
| Package distribution | Maven Central (Java), PyPI (Python), NuGet (.NET), RubyGems (Ruby), Composer/PECL (PHP), crates.io (Rust) |

### Data Flow: Protobuf Compilation and Serialization

```text
1. Schema Definition
   developer writes .proto file
        │
2. Code Generation
   protoc --<lang>_out=./gen schema.proto
   protoc reads .proto → builds FileDescriptor → invokes language-specific CodeGenerator
        │
3. Compilation
   language compiler compiles generated code + runtime library
        │
4. Serialization (at runtime)
   app creates message → sets fields → SerializeToString()
   message → field-by-field encoding → varint tags + wire-format values → byte stream
        │
5. Deserialization
   byte stream → ParseFromString() → decoded message object
   parser reads tag → identifies field number + wire type → decodes value → populates message
```

### Wire Format

The binary wire format uses field tags (field number + wire type) followed by the encoded value:

| Wire Type | ID | Used For |
|-----------|-----|----------|
| Varint | 0 | int32, int64, uint32, uint64, sint32, sint64, bool, enum |
| 64-bit | 1 | fixed64, sfixed64, double |
| Length-delimited | 2 | string, bytes, embedded messages, packed repeated fields |
| Start group | 3 | groups (deprecated) |
| End group | 4 | groups (deprecated) |
| 32-bit | 5 | fixed32, sfixed32, float |

Tag encoding: `(field_number << 3) | wire_type`

---

## 2. System Requirements

### Building from Source

| Requirement | Bazel Build | CMake Build |
|-------------|-------------|-------------|
| C++ compiler | GCC 7+ / Clang 7+ / MSVC 2019+ | Same |
| Build tool | Bazel 7+ | CMake 3.16+ |
| Python | 3.x (for code generators) | 3.x |
| Java | JDK 8+ (for Java artifacts) | JDK 8+ |
| Git | For submodule init | For submodule init |

### Runtime Requirements by Language

| Language | Minimum Version | Key Dependencies |
|----------|----------------|-----------------|
| C++ | C++17 | abseil-cpp, utf8_range |
| Java | JDK 8 | Guava 32.0.1-android (runtime) |
| Java (Kotlin) | Kotlin 1.6.0+ | kotlin-stdlib |
| Python | 3.x | numpy ≤ 2.3.4, setuptools ≤ 78.1.1 (build) |
| C# | .NET 4.5 / netstandard1.1 / net5+ | — |
| Ruby | CRuby 2.7+ or JRuby 9.2+ | rake, rake-compiler (build); FFI (JRuby) |
| PHP | 8.2.0+ | ext-bcmath (optional, for JSON deserialization) |
| Rust | Stable | protoc binary version must match crate version |
| Go | 1.23+ | google.golang.org/protobuf v1.36.9 |

> [!IMPORTANT]
> For Rust, the `protoc` binary version **must exactly match** the `protobuf` crate version. If you're using Rust protobuf `x.y.z`, you need protoc `y.z`. This is enforced by the codegen crate.

---

## 3. Configuration Reference

### protoc Compiler Flags

| Flag | Type | Description |
|------|------|-------------|
| `--proto_path=PATH` / `-I=PATH` | string | Directory to search for imported `.proto` files. Can be specified multiple times. |
| `--cpp_out=DIR` | string | Generate C++ source and header files |
| `--java_out=DIR` | string | Generate Java source files |
| `--python_out=DIR` | string | Generate Python `_pb2.py` files |
| `--csharp_out=DIR` | string | Generate C# `.cs` files |
| `--ruby_out=DIR` | string | Generate Ruby `_pb.rb` files |
| `--php_out=DIR` | string | Generate PHP files |
| `--kotlin_out=DIR` | string | Generate Kotlin DSL files (requires `--java_out` on JVM) |
| `--objc_out=DIR` | string | Generate Objective-C `.pbobjc.h`/`.m` files |
| `--rust_out=DIR` | string | Generate Rust source files |
| `--descriptor_set_out=FILE` | string | Write a `FileDescriptorSet` binary proto of all input files |
| `--include_imports` | flag | Include all transitive imports in the descriptor set |
| `--include_source_info` | flag | Include source code info in the descriptor set |
| `--experimental_allow_proto3_optional` | flag | Enable proto3 optional field support (now stable) |
| `--experimental_editions` | flag | Enable Editions syntax support |
| `--encode=MESSAGE_TYPE` | string | Read text-format message from stdin, write binary to stdout |
| `--decode=MESSAGE_TYPE` | string | Read binary message from stdin, write text-format to stdout |
| `--decode_raw` | flag | Read binary message from stdin, print raw field values |
| `--version` | flag | Print version info and exit |
| `--help` | flag | Print usage information |

**Language-specific sub-options (passed via `--<lang>_opt=`):**

| Option | Language | Description |
|--------|----------|-------------|
| `file_extension=.pb.cs` | C# | Change generated file extension |
| `base_namespace=Google.Protobuf` | C# | Set base namespace |
| `lite` | Java | Generate lite-runtime compatible code |

### Proto File Options

| Option | Scope | Description |
|--------|-------|-------------|
| `syntax = "proto3"` | File | Declare proto3 syntax |
| `syntax = "proto2"` | File | Declare proto2 syntax |
| `edition = "2023"` | File | Declare edition-based syntax |
| `package` | File | Namespace for proto definitions |
| `import` | File | Import another `.proto` file |
| `import public` | File | Transitive import |
| `option optimize_for` | File | `SPEED` (default), `CODE_SIZE`, `LITE_RUNTIME` |
| `option java_package` | File | Java package name |
| `option java_outer_classname` | File | Java outer class name |
| `option java_multiple_files` | File | Generate separate `.java` per message |
| `option go_package` | File | Go package import path |
| `option csharp_namespace` | File | C# namespace |
| `option objc_class_prefix` | File | Objective-C class prefix |
| `option ruby_package` | File | Ruby module path |
| `option php_namespace` | File | PHP namespace |
| `option php_metadata_namespace` | File | PHP metadata namespace |
| `option cc_enable_arenas` | File | Enable arena allocation for C++ |
| `option deprecated` | Field/Message/Enum | Mark as deprecated |
| `option packed` | Field | Pack repeated scalar fields (default in proto3) |
| `option map_entry` | Message | Internal: marks message as map entry |
| `reserved` | Message/Enum | Reserve field numbers/names to prevent reuse |
| `extensions` | Message (proto2) | Declare extension range |

### Environment Variables

| Variable | Language | Default | Description |
|----------|----------|---------|-------------|
| `PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION` | Python | `upb` | Backend selection: `upb`, `python`, `cpp` |
| `OUT_DIR` | Rust | (build) | Output directory for Rust codegen (set by Cargo) |
| `KOKORO_PYTHON_VERSION` | CI | — | Python version override for CI testing |

> [!WARNING]
> Setting `PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION` to an unsupported value raises `ValueError` at import time. Only `python`, `cpp`, and `upb` are valid.

### Editions Feature Flags

The Editions system replaces proto2/proto3 syntax with per-feature configuration. Edition 2023 is the first stable edition.

| Feature | Options | Edition 2023 Default |
|---------|---------|---------------------|
| Field presence | `EXPLICIT`, `IMPLICIT`, `LEGACY_REQUIRED` | `EXPLICIT` |
| Enum type | `OPEN`, `CLOSED` | `OPEN` |
| Repeated field encoding | `PACKED`, `EXPANDED` | `PACKED` |
| Message encoding | `LENGTH_PREFIXED`, `DELIMITED` | `LENGTH_PREFIXED` |
| JSON format | `ALLOW`, `LEGACY_BEST_EFFORT` | `ALLOW` |
| UTF-8 validation | `VERIFY`, `NONE` | `VERIFY` |

---

## 4. Deployment & Scaling

### Building protoc from Source (Bazel)

```bash
git clone https://github.com/protocolbuffers/protobuf.git
cd protobuf
git submodule update --init --recursive

# Build protoc
bazel build //:protoc

# Build all C++ libraries
bazel build //src/google/protobuf:protobuf

# Build Java artifacts
bazel build //java/core:core

# Run all tests
bazel test //...
```

### Building protoc from Source (CMake)

```bash
git clone https://github.com/protocolbuffers/protobuf.git
cd protobuf
git submodule update --init --recursive

cmake -S . -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -Dprotobuf_BUILD_TESTS=OFF \
  -Dprotobuf_ABSL_PROVIDER=module

cmake --build build --parallel $(nproc)
cmake --install build --prefix /usr/local
```

> [!TIP]
> Use `-Dprotobuf_BUILD_TESTS=OFF` to skip building tests and significantly reduce build time. Use `-Dprotobuf_ABSL_PROVIDER=module` to build abseil-cpp from the bundled submodule.

### CMake Integration in Downstream Projects

```cmake
find_package(Protobuf REQUIRED)

protobuf_generate_cpp(PROTO_SRCS PROTO_HDRS person.proto)
add_executable(myapp main.cpp ${PROTO_SRCS})
target_link_libraries(myapp protobuf::libprotobuf)
```

### Building Java Artifacts

```bash
cd java
mvn clean install -DskipTests
```

The Java build produces these Maven artifacts:

| Artifact | GroupId | ArtifactId |
|----------|---------|------------|
| Core runtime | `com.google.protobuf` | `protobuf-java` |
| Lite runtime | `com.google.protobuf` | `protobuf-javalite` |
| Util (JSON, etc.) | `com.google.protobuf` | `protobuf-java-util` |
| Kotlin DSL | `com.google.protobuf` | `protobuf-kotlin` |
| Kotlin Lite DSL | `com.google.protobuf` | `protobuf-kotlin-lite` |
| BOM | `com.google.protobuf` | `protobuf-bom` |

### Building Python Package

```bash
cd python
python setup.py build
pip install .
```

### Building C# Package

```bash
cd csharp
dotnet restore src/Google.Protobuf/Google.Protobuf.csproj
dotnet pack --no-restore -c Release src/Google.Protobuf.sln -p:ContinuousIntegrationBuild=true
```

> [!NOTE]
> The C# build script sets `DOTNET_SKIP_FIRST_TIME_EXPERIENCE=true` and `DOTNET_CLI_TELEMETRY_OPTOUT=true` to disable telemetry and first-run behavior.

### Building Ruby Gem

```bash
cd ruby
bundle install
rake compile
rake build
```

Ruby uses `rake-compiler` for building the C extension and supports cross-compilation via `rake-compiler-dock`.

### Building PHP Extension

```bash
cd php
# Pure PHP (via Composer)
composer install

# C extension
cd ext/google/protobuf
phpize
./configure
make
make install
```

### Building Rust Crates

The Rust implementation is published as multiple crates:

| Crate | Purpose |
|-------|---------|
| `protobuf` | Runtime library |
| `protobuf_codegen` | Build-time code generator |
| `protobuf_example` | Example usage |
| `protobuf_macros` | `proto!` macro |
| `protobuf_well_known_types` | Well-known type definitions |

```bash
cd rust/release_crates
cargo build --workspace
```

### Release Process

Pre-built binaries and packages are published via:

| Channel | Artifacts |
|---------|-----------|
| GitHub Releases | `protoc` binaries for all platforms |
| Maven Central | `protobuf-java`, `protobuf-javalite`, `protobuf-kotlin`, `protobuf-java-util`, `protobuf-bom` |
| PyPI | `protobuf` |
| NuGet | `Google.Protobuf`, `Google.Protobuf.Tools` |
| RubyGems | `google-protobuf` |
| Composer/PECL | `google/protobuf` |
| crates.io | `protobuf`, `protobuf_codegen`, `protobuf_macros`, `protobuf_well_known_types` |

---

## 5. Operations

### Conformance Testing

The conformance testing framework validates that all language implementations produce identical serialization/deserialization behavior:

```bash
# Build and run conformance tests
bazel test //conformance/...
```

The conformance test runner (`conformance/conformance_test_runner`) communicates with language-specific test harnesses via stdin/stdout using length-prefixed binary messages. Each request specifies an input format (binary, JSON, text), a message type, and a desired output format. The harness serializes/deserializes and returns the result.

**Supported conformance harnesses:**

| Language | Harness |
|----------|---------|
| C++ | Built into `conformance_test_runner` |
| Java | `conformance/ConformanceJava.java` |
| Python | `conformance/conformance_python.py` |
| Ruby | `conformance/ruby/conformance.rb` |
| Rust | `conformance/conformance_rust.rs` |

**Failure list management:**

Known failures are tracked in failure list files. The `conformance/update_failure_list.py` tool manages these lists:

```bash
python conformance/update_failure_list.py failure_list.txt --add "TestName" --remove "FixedTest"
```

> [!TIP]
> Run conformance tests after any change to serialization logic to catch cross-language regressions early. The conformance suite covers proto2, proto3, edition2023, and edition-unstable message types.

### Compatibility Testing

The `compatibility/` directory contains smoke tests that verify backward compatibility with previous protobuf versions (v3.0.0, v3.8.0, v3.11.0, v3.19.0, v3.20.0, v3.25.0, v25.8, v26.0, v32.1). These tests ensure that generated code from older protoc versions can be compiled against the current runtime.

### Version Compatibility Matrix

| Component Version | Compatible With |
|-------------------|----------------|
| protoc 4.36.0 | All runtimes 4.36.x |
| protobuf-java 4.36.0 | protoc 4.36.x |
| Python protobuf 4.36.x | protoc 4.36.x |
| Rust protobuf 0.x.y.z | protoc y.z exactly |

> [!IMPORTANT]
> Always match the protoc compiler version with the runtime library version. Cross-version combinations may produce subtle bugs or compilation errors.

### Benchmark Suite

```bash
# Run benchmarks
bazel run //benchmarks:benchmark -- --benchmark_filter=".*"
```

The benchmark suite measures serialization and deserialization performance across message sizes and complexities.

---

## 6. Performance

### upb: The Micro Protobuf Library

upb is a small, fast C library for parsing and serializing protobuf messages. It serves as the backend for several language runtimes:

| Runtime | upb Usage |
|---------|-----------|
| Python (default) | Primary backend via `PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=upb` |
| Ruby (CRuby) | C extension wraps upb |
| PHP (C ext) | C extension wraps upb |
| Rust | One of two backend options (`upb_kernel`) |

**upb architecture:**

| Component | Path | Purpose |
|-----------|------|---------|
| Base types | `upb/base/` | Fundamental types, status codes |
| Memory management | `upb/mem/` | Arena allocator (bump allocation, no individual frees) |
| Wire encoding | `upb/wire/` | Binary format encode/decode, fast-path decoder |
| JSON encoding | `upb/json/` | JSON ↔ protobuf conversion |
| Text encoding | `upb/text/` | Text format ↔ protobuf conversion |
| Mini tables | `upb/mini_table/` | Compact schema representation for fast parsing |
| Mini descriptors | `upb/mini_descriptor/` | Compact binary schema encoding |
| Reflection | `upb/reflection/` | Full descriptor-based reflection API |
| Message operations | `upb/message/` | Message creation, field access, comparison |
| Hash tables | `upb/hash/` | High-performance hash table for maps |
| Lexer | `upb/lex/` | Number/string parsing utilities |
| Port layer | `upb/port/` | Platform abstraction |
| Utilities | `upb/util/` | Required fields check, comparison, debug utilities |

> [!NOTE]
> upb uses **arena allocation** exclusively — all message memory is allocated from arenas and freed in bulk when the arena is destroyed. This provides excellent performance for typical protobuf usage patterns (create, populate, serialize, discard) but means individual message fields cannot be freed independently.

### Performance Considerations

**Serialization size optimization:**

- Use `sint32`/`sint64` for frequently negative values (ZigZag encoding)
- Use `fixed32`/`fixed64` for values typically > 2^28 / 2^56
- Assign field numbers 1–15 to frequently set fields (1-byte tag)
- Use `packed` encoding for repeated scalars (default in proto3)

**Runtime performance:**

- Python: ensure the `upb` backend is active (default); pure Python is 10–50x slower
- Java: the full runtime includes reflection and text format; use `javalite` for size-sensitive applications
- C++: enable arena allocation (`option cc_enable_arenas = true`) for batch allocation patterns
- Rust: choose between `upb_kernel` (C backend, smaller binary) and `cpp_kernel` (C++ backend, feature parity)

**Wire format efficiency compared to alternatives:**

| Format | Relative Size | Relative Speed |
|--------|--------------|----------------|
| Protocol Buffers | 1x (baseline) | 1x (baseline) |
| JSON | 3–10x larger | 10–50x slower |
| XML | 5–20x larger | 20–100x slower |

---

## 7. Security

### Input Validation

Protobuf parsers perform structural validation of incoming data but do **not** validate application-level semantics:

| Validation | Performed | Not Performed |
|-----------|-----------|---------------|
| Wire format integrity | Yes | — |
| Field type matching | Yes | — |
| UTF-8 string validation | Configurable | — |
| Required fields (proto2) | Yes | — |
| Value range constraints | — | No |
| Business logic validation | — | No |
| Message size limits | Configurable | Not enforced by default |

> [!CAUTION]
> Always set message size limits when parsing untrusted input. The default limit is 64 MB, but a maliciously crafted message can consume excessive memory (see GitHub issue #27104 — OOM in packed field decoders).

### Known Vulnerabilities in Current Source

Based on open GitHub issues:

| Issue | Severity | Component | Description |
|-------|----------|-----------|-------------|
| #27104 | High | Wire decoder | OOM in packed field decoders with crafted input |
| #27099 | Medium | C++ runtime | Null pointer dereference in `EpsCopyOutputStream::EnableAliasing` with array-only constructor |
| #27097 | Medium | C++ runtime | Unvalidated negative `buffer_size` in `GzipOutputStream::Init` causes crash |

### Security Best Practices

1. **Set message size limits** — use `CodedInputStream::SetTotalBytesLimit()` (C++) or `CodedInputStream.setSizeLimit()` (Java)
2. **Set recursion depth limits** — the default is 100 levels; reduce for untrusted input
3. **Validate after parsing** — protobuf doesn't enforce business rules; validate field values in application code
4. **Don't expose descriptors** — reflection APIs can reveal schema details; avoid exposing them in production
5. **Use deterministic serialization** for signing — default serialization is non-deterministic for map fields
6. **Sanitize text format input** — text format parsing is more permissive than binary; treat text input as untrusted

> [!WARNING]
> Protobuf's binary format provides no authentication, encryption, or integrity checking. Use TLS for transport security and sign/encrypt at the application level if needed.

### UTF-8 Validation

Proto3 requires `string` fields to be valid UTF-8. In Edition 2023, this is controlled by the `utf8_validation` feature:

- `VERIFY` (default): reject invalid UTF-8 during parsing
- `NONE`: accept any byte sequence in string fields

---

## 8. Developer Internals

### Repository Structure

```text
protocolbuffers/protobuf/
├── src/                          # C++ core library
│   └── google/protobuf/          # 171 source files, 3M chars
│       ├── compiler/             # protoc code generators (C++, Java, Python, C#, etc.)
│       ├── io/                   # CodedInputStream/OutputStream, zero-copy streams
│       ├── json/                 # JSON serialization (C++)
│       ├── stubs/                # Compatibility stubs
│       ├── util/                 # Utility functions
│       ├── any.h/cc              # Any type helpers
│       ├── descriptor.h/cc       # Schema descriptor (reflection)
│       ├── message.h/cc          # Full message interface
│       ├── message_lite.h/cc     # Lite message interface (no reflection)
│       └── wire_format.h/cc      # Wire format encoding/decoding
├── upb/                          # Micro protobuf C library
│   ├── base/                     # Fundamental types
│   ├── mem/                      # Arena memory allocator
│   ├── wire/                     # Binary encode/decode
│   ├── json/                     # JSON encode/decode
│   ├── text/                     # Text format encode/decode
│   ├── mini_table/               # Compact schema tables
│   ├── mini_descriptor/          # Binary schema descriptors
│   ├── reflection/               # Descriptor-based reflection
│   ├── message/                  # Message operations
│   ├── hash/                     # Hash map implementation
│   ├── lex/                      # Lexer utilities
│   ├── port/                     # Platform abstraction
│   ├── util/                     # Utilities
│   └── conformance/              # upb conformance harness
├── upb_generator/                # upb code generator
│   ├── c/                        # C code generator
│   ├── minitable/                # Mini table generator
│   ├── reflection/               # Reflection generator
│   └── common/                   # Shared generator utilities
├── hpb/                          # Header protobuf (C++ API over upb)
│   ├── backend/                  # upb/C++ backend implementations
│   └── internal/                 # Internal utilities
├── hpb_generator/                # hpb code generator
├── java/                         # Java runtime
│   ├── core/                     # protobuf-java
│   ├── lite/                     # protobuf-javalite
│   ├── util/                     # protobuf-java-util (JSON, etc.)
│   ├── kotlin/                   # protobuf-kotlin DSL
│   ├── kotlin-lite/              # protobuf-kotlin-lite DSL
│   ├── bom/                      # Bill of Materials POM
│   ├── osgi/                     # OSGi bundle
│   ├── protoc/                   # protoc Maven artifact POM
│   └── internal/                 # Internal shared code
├── python/                       # Python runtime (C extension + pure Python)
│   ├── google/protobuf/          # Main package
│   │   └── internal/             # Internal implementation (65 files, 1M chars)
│   ├── protobuf_distutils/       # setuptools plugin for protoc integration
│   └── docs/                     # Sphinx documentation config
├── csharp/                       # C# runtime
│   ├── src/                      # Source projects
│   │   ├── Google.Protobuf/      # Main library
│   │   └── Google.Protobuf.Test/ # Tests
│   ├── protos/                   # Test protos
│   └── compatibility_tests/      # v3.0.0 compatibility tests
├── ruby/                         # Ruby runtime
│   ├── ext/                      # C extension (CRuby)
│   ├── lib/                      # Ruby library + FFI (JRuby)
│   └── src/                      # JRuby Java source
├── php/                          # PHP runtime
│   ├── src/                      # PHP classes
│   ├── ext/                      # C extension
│   └── tests/                    # Test suite
├── objectivec/                   # Objective-C runtime (66 files, 1.4M chars)
│   └── DevTools/                 # PDDM macro preprocessor
├── rust/                         # Rust runtime
│   ├── cpp_kernel/               # C++ backend for Rust
│   ├── upb_kernel/               # upb backend for Rust
│   ├── upb/                      # Low-level upb bindings
│   ├── release_crates/           # Published crate sources
│   │   ├── protobuf/             # Main runtime crate
│   │   ├── protobuf_codegen/     # Code generator crate
│   │   ├── protobuf_macros/      # proto! macro crate
│   │   └── protobuf_well_known_types/ # Well-known types
│   └── test/                     # Test infrastructure
├── conformance/                  # Cross-language conformance tests
│   ├── test_protos/              # Test message definitions
│   └── ruby/                     # Ruby conformance harness
├── editions/                     # Editions system
│   ├── golden/                   # Golden test files
│   ├── input/                    # Edition transform inputs
│   └── codegen_tests/            # Code generation tests
├── examples/                     # Tutorial examples (AddressBook)
│   └── go/                       # Go example
├── bazel/                        # Bazel build rules
│   ├── common/                   # Common rules
│   ├── flags/                    # Build flag definitions
│   ├── toolchains/               # Toolchain configs
│   └── private/                  # Internal rules
├── cmake/                        # CMake build support
├── ci/                           # CI configuration
├── benchmarks/                   # Performance benchmarks
├── compatibility/                # Version compatibility tests
├── patches/                      # Patch files for older versions
├── build_defs/                   # Build definition helpers
├── editors/                      # Editor support files
├── pkg/                          # Package configuration
└── toolchain/                    # Toolchain definitions
```

### Core C++ Architecture

The C++ runtime (`src/google/protobuf/`) is organized around these key abstractions:

**Message hierarchy:**

```text
MessageLite (minimal interface, no reflection)
  └── Message (full interface with reflection + text format)
       └── Generated classes (e.g., Person, AddressBook)
```

- `MessageLite` — base class for all messages; provides `SerializeToString()`, `ParseFromString()`, `ByteSizeLong()`, `MergeFrom()`
- `Message` — extends `MessageLite` with reflection (`GetDescriptor()`, `GetReflection()`), text format, and `SpaceUsedLong()`
- Generated classes — concrete implementations with typed accessors

**Descriptor system (reflection):**

| Class | Purpose |
|-------|---------|
| `FileDescriptor` | Represents a `.proto` file |
| `Descriptor` | Represents a message type |
| `FieldDescriptor` | Represents a single field |
| `EnumDescriptor` | Represents an enum type |
| `ServiceDescriptor` | Represents a service definition |
| `MethodDescriptor` | Represents an RPC method |
| `DescriptorPool` | Registry of all known descriptors |

**I/O layer:**

| Class | Purpose |
|-------|---------|
| `CodedInputStream` | Low-level varint/fixed decoding from byte streams |
| `CodedOutputStream` | Low-level varint/fixed encoding to byte streams |
| `ZeroCopyInputStream` | Interface for zero-copy byte access |
| `ZeroCopyOutputStream` | Interface for zero-copy byte output |
| `GzipInputStream` | Gzip-decompressing stream wrapper |
| `GzipOutputStream` | Gzip-compressing stream wrapper |

### upb Architecture Deep Dive

upb is designed for maximum performance with minimal memory overhead:

**Arena allocator (`upb/mem/`):**

- All message allocations come from an arena — a bump allocator that frees all memory at once
- Arenas can be fused: two arenas become one, sharing a lifetime
- Arena allocation is ~3x faster than malloc for typical protobuf patterns

**Mini tables (`upb/mini_table/`):**

- Compact, cache-friendly representation of message schemas
- Generated at compile time by the upb code generator
- Used by the wire decoder for fast field dispatch
- Internal layout: array of field entries indexed by field number, each entry encodes type, offset, and presence information

**Fast decoder (`upb/wire/`):**

- Two-stage decoder: fast path (inline, no function calls for common cases) + slow path (fallback for uncommon wire types)
- `decode_fast` uses computed-goto dispatch on x86/ARM for ~2 ns per field
- Handles packed repeated fields, submessages, and groups

**Reflection (`upb/reflection/`):**

- Full descriptor-based reflection API for dynamic message access
- Used by JSON and text format encoders that need to enumerate fields at runtime
- Significantly more memory than mini tables — only loaded when reflection is needed

### Python Runtime Internals

The Python protobuf library (`python/`) has three backends:

1. **upb backend (default):** C extension in `python/convert.c`, `python/message.c`, `python/protobuf.c`. Wraps the upb C library for message storage and serialization. Python objects are thin wrappers around upb messages allocated in arenas.

2. **Pure Python backend:** `python/google/protobuf/internal/python_message.py`. Full Python implementation using `__slots__` and descriptors. ~1450 lines. Supports all features but is 10–50x slower than the upb backend.

3. **C++ backend (deprecated):** Previously linked against the C++ protobuf library. Being phased out in favor of upb.

**Key internal modules:**

| Module | Path | Purpose |
|--------|------|---------|
| `containers.py` | `python/google/protobuf/internal/` | `RepeatedScalarFieldContainer`, `RepeatedCompositeFieldContainer`, `ScalarMap`, `MessageMap` |
| `decoder.py` | `python/google/protobuf/internal/` | Binary format decoder — varint, fixed, delimited, groups |
| `encoder.py` | `python/google/protobuf/internal/` | Binary format encoder |
| `type_checkers.py` | `python/google/protobuf/internal/` | Field value type checking and validation |
| `well_known_types.py` | `python/google/protobuf/internal/` | Timestamp, Duration, Any, FieldMask helpers |
| `python_message.py` | `python/google/protobuf/internal/` | Pure Python message metaclass and implementation |
| `enum_type_wrapper.py` | `python/google/protobuf/internal/` | Enum value lookup by name/number |
| `extension_dict.py` | `python/google/protobuf/internal/` | Extension field access |
| `field_mask.py` | `python/google/protobuf/internal/` | FieldMask path operations |
| `api_implementation.py` | `python/google/protobuf/internal/` | Backend selection logic |

> [!NOTE]
> The Python `api_implementation.py` module raises `ValueError` if `api_version=1` is requested (removed in v4) and raises `ValueError` if `PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION` is set to an unrecognized value.

### Rust Runtime Internals

The Rust protobuf implementation (V4) provides a safe API with ownership semantics:

**Core types:**

| Type | Purpose |
|------|---------|
| `Map<K, V>` | Owned protobuf map |
| `MapView<'msg, K, V>` | Borrowed read-only map view |
| `MapMut<'msg, K, V>` | Borrowed mutable map reference |
| `Repeated<T>` | Owned repeated field |
| `RepeatedView<'msg, T>` | Borrowed read-only repeated view |
| `RepeatedMut<'msg, T>` | Borrowed mutable repeated reference |
| `ProtoString` | Owned protobuf string |
| `ProtoStr` | Borrowed protobuf string slice |
| `ProtoBytes` | Owned protobuf bytes |
| `Arena` | upb arena wrapper (memory allocator) |

**Backend architecture:**

The Rust runtime supports two backends selected at compile time:

| Backend | Module | Binary Size | Features |
|---------|--------|-------------|----------|
| upb kernel | `rust/upb_kernel/` | Smaller | Full message, map, repeated, extension support |
| C++ kernel | `rust/cpp_kernel/` | Larger | Full feature parity with C++ runtime |

Both backends implement the same traits (`Proxied`, `ViewProxy`, `MutProxy`) behind `Private` visibility barriers, ensuring the public API is identical regardless of backend.

**Extension API:**

```rust
// Extensions use ExtensionId<Extendee, T> for type-safe access
let ext = ExtensionId::<MyMessage, i32>::new(/* ... */);
ext.set(&mut msg, 42);
let val: i32 = ext.get(&msg);
```

### Objective-C Runtime

The Objective-C runtime (`objectivec/`, 66 files, 1.4M chars) is one of the largest per-language implementations. It includes:

- **PDDM macro preprocessor** (`objectivec/DevTools/pddm.py`): A custom macro expansion system used to generate repetitive Objective-C code for each field type. PDDM definitions in source files are expanded in-place during development.
- **Xcode project files** for macOS, iOS, and tvOS targets
- Full descriptor and reflection support
- Dynamic message creation

### PHP Runtime

| Component | Description |
|-----------|-------------|
| Pure PHP | `php/src/Google/Protobuf/Internal/` (53 files, 469K chars) — full message, descriptor, and JSON support |
| C extension | `php/ext/google/protobuf/` — upb-backed C extension for performance |
| Metadata | `php/src/GPBMetadata/` — generated descriptor metadata |

PHP minimum version: 8.2.0. The `ext-bcmath` extension is suggested for proper JSON deserialization of large integers.

### Java Runtime

The Java implementation is organized as a Maven multi-module project:

| Module | Artifact | Purpose |
|--------|----------|---------|
| `java/core` | `protobuf-java` | Full runtime with reflection, text format |
| `java/lite` | `protobuf-javalite` | Lite runtime without reflection |
| `java/util` | `protobuf-java-util` | JSON serialization (`JsonFormat`), field mask utilities |
| `java/kotlin` | `protobuf-kotlin` | Kotlin DSL extensions |
| `java/kotlin-lite` | `protobuf-kotlin-lite` | Kotlin DSL for lite runtime |
| `java/bom` | `protobuf-bom` | Bill of Materials for dependency management |
| `java/osgi` | — | OSGi bundle manifest |
| `java/protoc` | `protoc` | protoc binary Maven artifact |

Java target compatibility: Java 8 (source + target `1.8`). Build uses `maven-antrun-plugin` to invoke `protoc` during test source generation.

### Conformance Test Architecture

The conformance testing framework uses a two-process architecture:

```text
┌─────────────────────────┐     stdin/stdout      ┌───────────────────────┐
│ conformance_test_runner │ ◄──── length-prefixed ──► language_harness    │
│ (C++ driver)            │      binary messages   │ (Java/Python/Ruby/   │
│                         │                        │  Rust/C#)            │
│ - Generates test cases  │                        │ - Receives request   │
│ - Validates responses   │                        │ - Parses/serializes  │
│ - Reports pass/fail     │                        │ - Returns response   │
└─────────────────────────┘                        └───────────────────────┘
```

Communication protocol: 4-byte little-endian length prefix + serialized `ConformanceRequest` / `ConformanceResponse` messages.

Test coverage includes:

- Proto2, proto3, edition2023, edition-unstable message types
- Binary ↔ JSON ↔ text format conversions
- Edge cases: default values, unknown fields, oneof clearing, map ordering
- Required field validation (proto2)
- UTF-8 validation
- Group encoding (legacy)

### Code Patterns and Conventions

**Naming conventions:**

| Element | Convention | Example |
|---------|-----------|---------|
| Proto package | `lowercase.dotted` | `google.protobuf` |
| Message name | `PascalCase` | `AddressBook` |
| Field name | `snake_case` | `phone_number` |
| Enum value | `SCREAMING_SNAKE_CASE` | `PHONE_TYPE_MOBILE` |
| Service name | `PascalCase` | `SearchService` |
| RPC method | `PascalCase` | `SearchRequest` |

**Field number allocation:**

- 1–15: single-byte tag — use for frequently set fields
- 16–2047: two-byte tag — use for less common fields
- 19000–19999: reserved by protobuf implementation — never use
- 1–536870911 (2^29 - 1): valid range

**Generated code naming (by language):**

| Language | Message → Class | Field → Accessor | Enum → |
|----------|----------------|-------------------|--------|
| Java | `Person` class + `Person.Builder` | `getName()` / `setName()` | `Person.PhoneType.MOBILE` |
| Python | `Person` class | `person.name` | `Person.MOBILE` |
| C++ | `Person` class | `name()` / `set_name()` | `Person::MOBILE` |
| C# | `Person` class | `person.Name` | `Person.Types.PhoneType.Mobile` |
| Ruby | `Person` class | `person.name` | `:MOBILE` (symbol) |
| Go | `Person` struct | `person.Name` / `person.GetName()` | `Person_MOBILE` |

### Testing Strategy

| Test Category | Framework | Location |
|---------------|-----------|----------|
| C++ unit tests | Google Test | `src/google/protobuf/*_test.cc` |
| C++ JSON tests | Google Test | `src/google/protobuf/json/internal/*_test.cc` |
| Java tests | JUnit 4 | `java/core/src/test/`, `java/lite/src/test/` |
| Python tests | unittest | `python/google/protobuf/internal/*_test.py` |
| PHP tests | PHPUnit 11.5+ | `php/tests/` |
| Ruby tests | test-unit | `ruby/tests/` |
| Objective-C tests | XCTest | `objectivec/Tests/` |
| Conformance tests | Custom runner | `conformance/` |
| Compatibility tests | Bazel | `compatibility/` |
| Editions tests | Google Test | `editions/codegen_tests/` |
| upb tests | Google Test | `upb/test/` |
| CMake staleness | unittest | `upb/cmake/staleness_test.py` |

Test infrastructure uses `protoc` to generate test protos during the build process (via Maven Antrun for Java, Bazel genrules for C++/others).

> [!TIP]
> The Python test suite at `python/google/protobuf/internal/` contains 65 files with ~1M chars of tests — one of the most comprehensive test suites in the project. Run with `python -m pytest python/google/protobuf/internal/` or via Bazel.

### Build System Details

**Bazel (primary):**

- `MODULE.bazel` defines external dependencies (abseil-cpp, utf8_range, etc.)
- `BUILD.bazel` files throughout the tree define targets
- `bazel/` contains custom rules: `proto_library`, `java_proto_library`, `py_proto_library`, etc.
- `bazel/toolchains/` configures cross-compilation toolchains
- `bazel/flags/` defines build configuration flags (C++ standards, Java settings)

**CMake (alternative):**

- `CMakeLists.txt` at root
- `cmake/dependencies_generator.py` converts `MODULE.bazel` dependencies to CMake variables
- External dependency handling via `FetchContent` or pre-installed packages
- `upb/cmake/` contains CMake-specific build files for the upb library
- Staleness tests (`upb/cmake/staleness_test.py`) verify generated CMake files match Bazel definitions

### Potential Improvements

Based on source analysis:

1. **Packed field decoder safety** — the OOM issue (#27104) in packed field decoders needs bounds checking before allocation
2. **GzipOutputStream validation** — buffer_size parameter in `GzipOutputStream::Init` should validate non-negative values (#27097)
3. **Null safety in EpsCopyOutputStream** — the array-only constructor should validate internal state before use (#27099)
4. **GCC compatibility** — build failures with certain GCC versions (#26385) need compiler-specific workarounds
5. **CMake + abseil-cpp integration** — version mismatch issues (#22209) between protobuf and abseil-cpp builds need smoother dependency resolution

### Contributing

The project uses a standard GitHub workflow:

1. Fork the repository
2. Create a feature branch
3. Make changes with tests
4. Run conformance tests: `bazel test //conformance/...`
5. Run language-specific tests: `bazel test //java/...`, `bazel test //python/...`, etc.
6. Submit a pull request

**Code style:** Follow existing patterns in each language directory. C++ follows Google C++ Style Guide. Java follows Google Java Style Guide. Python follows PEP 8.

**License:** BSD-3-Clause. All source files include the license header.

**CI:** Kokoro (Google internal) runs the primary CI. AppVeyor covers Windows builds (Visual Studio 2017, MinGW). CI configurations are in `ci/` and `appveyor.yml`.

---

## 9. Integration Points

### gRPC Integration

Protobuf is the default serialization format for gRPC. Service definitions in `.proto` files generate RPC stubs:

```protobuf
service RouteGuide {
  rpc GetFeature(Point) returns (Feature);
  rpc ListFeatures(Rectangle) returns (stream Feature);
  rpc RecordRoute(stream Point) returns (RouteSummary);
  rpc RouteChat(stream RouteNote) returns (stream RouteNote);
}
```

gRPC plugins are separate from the protobuf compiler but invoked via `protoc`'s plugin system.

### Maven Integration (Java)

Use the `protobuf-maven-plugin` to integrate protoc into Maven builds:

```xml
<plugin>
  <groupId>org.xolstice.maven.plugins</groupId>
  <artifactId>protobuf-maven-plugin</artifactId>
  <configuration>
    <protocArtifact>com.google.protobuf:protoc:4.36.0:exe:${os.detected.classifier}</protocArtifact>
  </configuration>
</plugin>
```

### Gradle Integration (Java/Kotlin)

```groovy
plugins {
    id 'com.google.protobuf' version '0.9.4'
}

protobuf {
    protoc {
        artifact = 'com.google.protobuf:protoc:4.36.0'
    }
}
```

### Python setuptools Integration

The `protobuf_distutils` package (`python/protobuf_distutils/`) provides a setuptools command for running `protoc` during Python package builds:

```python
# setup.py
from setuptools import setup
setup(
    # ...
    options={'generate_py_protobufs': {
        'source_dir': 'protos',
        'output_dir': 'mypackage/protos',
    }},
)
```

### Bazel Integration

```python
load("@protobuf//bazel:proto_library.bzl", "proto_library")
load("@protobuf//bazel:cc_proto_library.bzl", "cc_proto_library")
load("@protobuf//bazel:java_proto_library.bzl", "java_proto_library")
load("@protobuf//bazel:py_proto_library.bzl", "py_proto_library")

proto_library(
    name = "person_proto",
    srcs = ["person.proto"],
    deps = ["@protobuf//:timestamp_proto"],
)

cc_proto_library(name = "person_cc_proto", deps = [":person_proto"])
java_proto_library(name = "person_java_proto", deps = [":person_proto"])
py_proto_library(name = "person_py_proto", deps = [":person_proto"])
```

### NuGet/MSBuild Integration (C#)

The `Google.Protobuf.Tools` NuGet package bundles `protoc` and well-known type `.proto` files for MSBuild integration:

```xml
<PackageReference Include="Google.Protobuf" Version="4.36.0" />
<PackageReference Include="Google.Protobuf.Tools" Version="4.36.0" />
```

Platform binaries included: `windows_x86`, `windows_x64`, `macosx_x64`, `linux_x86`, `linux_x64`, `linux_aarch64`.

### Rust Cargo Integration

Use the `protobuf_codegen` crate in your `build.rs`:

```rust
// build.rs
fn main() {
    protobuf_codegen::CodeGen::new()
        .inputs(&["src/protos/person.proto"])
        .include("src/protos")
        .cargo_out_dir("protobuf_generated")
        .generate()
        .expect("protoc codegen failed");
}
```

```rust
// main.rs
mod protos {
    include!(concat!(env!("OUT_DIR"), "/protobuf_generated/person.rs"));
}
```

### Read the Docs Integration

The Python documentation uses Sphinx with Read the Docs hosting. Configuration:

- `.readthedocs.yml` at repo root — uses conda environment, Sphinx, Python 3.8
- `python/docs/conf.py` — Sphinx configuration
- `python/docs/requirements.txt` — Sphinx dependencies (sphinx 3.0.4, sphinx_rtd_theme 0.4.3)

---

_Generated May 2026 from protocolbuffers/protobuf at 7bb6178. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._