# Protocol Buffers (protobuf) — User Guide

## 1. Quick Start

Protocol Buffers (protobuf) is Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data. You define how your data is structured once in `.proto` files, then use the `protoc` compiler to generate source code in your target language — C++, Java, Python, C#, Go, Ruby, PHP, Kotlin, Objective-C, Rust, or Dart. The generated code provides typed accessors for reading and writing your data efficiently in a compact binary format.

**Key features:**

- Compact binary serialization — 3–10x smaller and 20–100x faster than XML
- Strongly typed schema with forward and backward compatibility
- Code generation for 12+ programming languages from a single `.proto` definition
- Built-in JSON mapping for interoperability with web services
- Well-known types for common patterns (timestamps, durations, field masks, etc.)
- Editions system for evolving language features without breaking changes
- Conformance testing framework to verify cross-language compatibility
- Lite runtime for resource-constrained environments (Android, embedded)

**Prerequisites:** A working compiler toolchain for your target language (e.g., JDK 8+ for Java, Python 3.x for Python, .NET SDK for C#).

**Install protoc and try it:**

```bash
# Download the latest release for your platform
# from https://github.com/protocolbuffers/protobuf/releases
# Extract and add to PATH

protoc --version
```

```text
libprotoc 4.36.0
```

**Minimal example — define a message, generate code, use it:**

Create `person.proto`:

```protobuf
syntax = "proto3";
package example;

message Person {
  string name = 1;
  int32 id = 2;
  string email = 3;
}
```

Generate Python code:

```bash
protoc --python_out=. person.proto
```

Use the generated code:

```python
import person_pb2

person = person_pb2.Person()
person.name = "Alice"
person.id = 42
person.email = "[email removed]"

# Serialize to binary
data = person.SerializeToString()

# Deserialize
person2 = person_pb2.Person()
person2.ParseFromString(data)
print(person2.name)  # Alice
```

```text
Alice
```

> [!TIP]
> The fastest way to get started is to download a prebuilt `protoc` binary from the [GitHub Releases](https://github.com/protocolbuffers/protobuf/releases) page and install the runtime library for your language via its package manager.

**What to try next:**

- See the Installation section for all supported install methods
- See Core Usage for the complete `.proto` language guide
- See Common Tasks for real-world recipes (JSON conversion, well-known types, maps)

---

## 2. Installation

### System Requirements

| Component | Requirement |
|-----------|-------------|
| `protoc` compiler | Any 64-bit OS (Linux, macOS, Windows) |
| C++ runtime | C++17 compiler (GCC 7+, Clang 7+, MSVC 2019+), CMake 3.16+ or Bazel 7+ |
| Java runtime | JDK 8+ |
| Python runtime | Python 3.x, `pip` |
| C# runtime | .NET 4.5+ / .NET Standard 1.1+ / .NET 5+ |
| Ruby runtime | Ruby 2.7+ (CRuby or JRuby) |
| PHP runtime | PHP 8.2.0+ |
| Go runtime | Go 1.23+ |
| Kotlin runtime | Kotlin 1.6.0+, JDK 8+ |
| Rust runtime | Rust stable, `protoc` version must exactly match crate version |

### Installing protoc (the Protocol Buffer Compiler)

=== "Pre-built Binary (recommended)"
    ```bash
    # Download from GitHub releases
    # https://github.com/protocolbuffers/protobuf/releases
    # Choose protoc-<version>-<platform>.zip
    
    # Linux x86_64 example:
    curl -LO https://github.com/protocolbuffers/protobuf/releases/download/v36.0/protoc-36.0-linux-x86_64.zip
    unzip protoc-36.0-linux-x86_64.zip -d $HOME/.local
    export PATH="$HOME/.local/bin:$PATH"
    
    protoc --version
    ```

    ```text
    libprotoc 4.36.0
    ```

=== "Homebrew (macOS/Linux)"
    ```bash
    brew install protobuf
    protoc --version
    ```

=== "apt (Debian/Ubuntu)"
    ```bash
    sudo apt install -y protobuf-compiler
    protoc --version
    ```

=== "Build from Source (Bazel)"
    ```bash
    git clone https://github.com/protocolbuffers/protobuf.git
    cd protobuf
    git submodule update --init --recursive
    bazel build //:protoc
    # Binary at bazel-bin/protoc
    ```

=== "Build from Source (CMake)"
    ```bash
    git clone https://github.com/protocolbuffers/protobuf.git
    cd protobuf
    git submodule update --init --recursive
    cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
    cmake --build build --target protoc
    # Binary at build/protoc
    ```

> [!IMPORTANT]
> When using the Rust crate, the `protoc` binary version must **exactly** match the `protobuf` crate version. If you are using Rust protobuf `x.y.z`, you need protoc `y.z`. Download the matching release from the [GitHub Releases](https://github.com/protocolbuffers/protobuf/releases) page.

### Installing Language-Specific Runtime Libraries

| Language | Package Manager | Install Command |
|----------|----------------|-----------------|
| Python | pip | `pip install protobuf` |
| Java | Maven | `com.google.protobuf:protobuf-java:4.36.0` |
| Java (Lite) | Maven | `com.google.protobuf:protobuf-javalite:4.36.0` |
| Java (Util) | Maven | `com.google.protobuf:protobuf-java-util:4.36.0` |
| Kotlin | Maven | `com.google.protobuf:protobuf-kotlin:4.36.0` |
| C# | NuGet | `dotnet add package Google.Protobuf` |
| C# (Tools) | NuGet | `dotnet add package Google.Protobuf.Tools` |
| Ruby | Gem | `gem install google-protobuf` |
| PHP (pure) | Composer | `composer require google/protobuf` |
| PHP (C ext) | PECL | `pecl install protobuf` |
| Go | Go modules | `go get google.golang.org/protobuf` |
| Rust | Cargo | `cargo add protobuf` |

**Java Maven dependency:**

```xml
<dependency>
  <groupId>com.google.protobuf</groupId>
  <artifactId>protobuf-java</artifactId>
  <version>4.36.0</version>
</dependency>
```

**Java BOM (Bill of Materials):**

```xml
<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>com.google.protobuf</groupId>
      <artifactId>protobuf-bom</artifactId>
      <version>4.36.0</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>
```

> [!WARNING]
> Distribution packages (e.g., `apt install protobuf-compiler`) often ship **older** versions of protoc. For production use, prefer the pre-built binaries from GitHub Releases to ensure version compatibility between the compiler and runtime libraries.

### Post-Install Verification

```bash
protoc --version
```

```text
libprotoc 4.36.0
```

Verify language runtimes:

```bash
# Python
python -c "import google.protobuf; print(google.protobuf.__version__)"
```

```text
4.36.0
```

```bash
# Ruby
ruby -e "require 'google/protobuf'; puts Google::Protobuf::VERSION"
```

### Install Troubleshooting

**`protoc: command not found`** — the binary is not in your `PATH`. Add the directory containing `protoc` to your `PATH` environment variable.

**`protoc: error while loading shared libraries`** — on Linux, you may need to run `ldconfig` after installing or set `LD_LIBRARY_PATH` to the library directory.

**Version mismatch between protoc and runtime** — always ensure the protoc compiler version matches the runtime library version. Mixing versions can cause subtle serialization bugs or compilation errors.

**`pip install protobuf` fails on Apple Silicon** — try `pip install --no-binary protobuf protobuf` to build from source, or ensure you have the latest pip and setuptools.

**CMake build fails with `abseil-cpp` errors** — protobuf depends on abseil-cpp. Ensure you initialized submodules: `git submodule update --init --recursive`. See GitHub issue [#22209](https://github.com/protocolbuffers/protobuf/issues/22209).

**`npm install google-protobuf`** — the JavaScript implementation lives in the separate `protobuf-javascript` repository.

**Go module errors** — the Go runtime lives in the separate `protobuf-go` repository at `google.golang.org/protobuf`.

**Dart protobuf** — the Dart implementation lives in the separate `dart-lang/protobuf` repository.

**Windows: `protoc.exe` is blocked by antivirus** — add an exception for the protoc binary or download from the official GitHub Releases page.

**`gcc` compilation failures (issue #26385)** — building protobuf from source with certain GCC versions may fail. Use a supported compiler version or build with Clang.

**Ruby JRuby vs CRuby** — on JRuby, the Ruby gem uses FFI to access the protobuf library. On CRuby, it uses a compiled C extension backed by upb. Both are installed via `gem install google-protobuf`.

---

## 3. Configuration

### Proto File Syntax

Protocol Buffers uses `.proto` files to define message schemas. The current recommended syntax is `proto3`, with `proto2` supported for legacy schemas and the new Editions system for evolving features.

**Minimal proto file:**

```protobuf
syntax = "proto3";
package mypackage;

message MyMessage {
  string name = 1;
  int32 count = 2;
  bool active = 3;
}
```

### Scalar Types

| Proto Type | C++ | Java | Python | C# | Go | Ruby | Notes |
|-----------|-----|------|--------|----|----|------|-------|
| `double` | `double` | `double` | `float` | `double` | `float64` | `Float` | 64-bit IEEE 754 |
| `float` | `float` | `float` | `float` | `float` | `float32` | `Float` | 32-bit IEEE 754 |
| `int32` | `int32_t` | `int` | `int` | `int` | `int32` | `Fixnum` | Variable-length |
| `int64` | `int64_t` | `long` | `int` | `long` | `int64` | `Bignum` | Variable-length |
| `uint32` | `uint32_t` | `int` | `int` | `uint` | `uint32` | `Fixnum` | Variable-length |
| `uint64` | `uint64_t` | `long` | `int` | `ulong` | `uint64` | `Bignum` | Variable-length |
| `sint32` | `int32_t` | `int` | `int` | `int` | `int32` | `Fixnum` | ZigZag encoded |
| `sint64` | `int64_t` | `long` | `int` | `long` | `int64` | `Bignum` | ZigZag encoded |
| `fixed32` | `uint32_t` | `int` | `int` | `uint` | `uint32` | `Fixnum` | Always 4 bytes |
| `fixed64` | `uint64_t` | `long` | `int` | `ulong` | `uint64` | `Bignum` | Always 8 bytes |
| `sfixed32` | `int32_t` | `int` | `int` | `int` | `int32` | `Fixnum` | Always 4 bytes |
| `sfixed64` | `int64_t` | `long` | `int` | `long` | `int64` | `Bignum` | Always 8 bytes |
| `bool` | `bool` | `boolean` | `bool` | `bool` | `bool` | `TrueClass/FalseClass` | |
| `string` | `string` | `String` | `str` | `string` | `string` | `String` | UTF-8 encoded |
| `bytes` | `string` | `ByteString` | `bytes` | `ByteString` | `[]byte` | `String` | Arbitrary bytes |

### Language-Specific Options

Common file-level options you can set in `.proto` files:

```protobuf
// Java
option java_package = "com.example.myproject";
option java_outer_classname = "MyProtos";
option java_multiple_files = true;

// C#
option csharp_namespace = "MyCompany.MyProject";

// Go
option go_package = "github.com/myorg/myrepo/mypb";

// Objective-C
option objc_class_prefix = "MYP";

// Ruby
option ruby_package = "MyCompany::MyProject";

// PHP
option php_namespace = "MyCompany\\MyProject";
option php_metadata_namespace = "MyCompany\\MyProject\\Metadata";
```

### Python Backend Selection

The Python protobuf library supports multiple backends. The default backend (upb) is the fastest:

| Backend | Set Via | Performance | Notes |
|---------|---------|-------------|-------|
| `upb` (default) | Default, no env var needed | Fastest | C-based, recommended |
| `python` | `PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python` | Slowest | Pure Python, useful for debugging |
| `cpp` | `PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp` | Fast | Deprecated, use upb |

> [!NOTE]
> The `cpp` backend is deprecated. The `upb` backend (default since protobuf v4) offers equivalent or better performance. Setting `PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION` to any value other than `python`, `cpp`, or `upb` raises a `ValueError`.

See the Admin Guide for the full configuration reference including all protoc flags, edition features, and build system options.

---

## 4. Core Usage

### Defining Messages

Messages are the fundamental building blocks of protobuf schemas. Each field has a type, a name, and a unique field number.

```protobuf
syntax = "proto3";
package tutorial;

import "google/protobuf/timestamp.proto";

message Person {
  string name = 1;
  int32 id = 2;
  string email = 3;

  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }

  message PhoneNumber {
    string number = 1;
    PhoneType type = 2;
  }

  repeated PhoneNumber phones = 4;
  google.protobuf.Timestamp last_updated = 5;
}

message AddressBook {
  repeated Person people = 1;
}
```

> [!TIP]
> Field numbers 1–15 use a single byte in the wire format, so assign them to frequently populated fields for optimal serialization size.

### Field Rules

| Keyword | Meaning | Proto3 Default |
|---------|---------|----------------|
| (none) | Singular field — zero or one value | Yes |
| `repeated` | Zero or more values (ordered list) | — |
| `map<K, V>` | Key-value pairs | — |
| `optional` | Explicit presence tracking | — |
| `oneof` | At most one field in the group is set | — |

### Enumerations

```protobuf
enum Status {
  STATUS_UNSPECIFIED = 0;  // First value MUST be zero
  ACTIVE = 1;
  INACTIVE = 2;
  DELETED = 3;
}
```

> [!IMPORTANT]
> In proto3, the first enum value **must** be zero and serves as the default. Name it with an `_UNSPECIFIED` suffix by convention.

### Maps

```protobuf
message Project {
  map<string, int32> scores = 1;
  map<int32, Person> people_by_id = 2;
}
```

Map key types can be any integer or string type. Map value types can be any type except another map.

### Oneof Fields

```protobuf
message Profile {
  string name = 1;
  oneof avatar {
    string image_url = 2;
    bytes image_data = 3;
  }
}
```

Setting any field within a `oneof` clears all other fields in the group.

### Importing Other Proto Files

```protobuf
import "google/protobuf/timestamp.proto";
import "myother/messages.proto";
```

### Generating Code

Use `protoc` to compile `.proto` files into language-specific source code:

```bash
# Python
protoc --python_out=./gen person.proto

# Java
protoc --java_out=./gen person.proto

# C++
protoc --cpp_out=./gen person.proto

# C#
protoc --csharp_out=./gen person.proto

# Ruby
protoc --ruby_out=./gen person.proto

# PHP
protoc --php_out=./gen person.proto

# Kotlin (requires --java_out too on JVM)
protoc --java_out=./gen --kotlin_out=./gen person.proto

# Objective-C
protoc --objc_out=./gen person.proto

# Multiple outputs at once
protoc --java_out=./java_gen --python_out=./py_gen person.proto
```

**Specifying import paths:**

```bash
protoc -I=./protos -I=./third_party --python_out=./gen ./protos/person.proto
```

> [!NOTE]
> The `-I` (or `--proto_path`) flag tells protoc where to search for imported `.proto` files. You can specify it multiple times for multiple search paths.

### Using Generated Code

#### Python

```python
import person_pb2

# Create and populate
person = person_pb2.Person()
person.name = "Bob"
person.id = 1234
person.email = "[email removed]"

phone = person.phones.add()
phone.number = "555-4321"
phone.type = person_pb2.Person.HOME

# Serialize to binary
data = person.SerializeToString()

# Deserialize from binary
person2 = person_pb2.Person()
person2.ParseFromString(data)
print(person2.name)
```

```text
Bob
```

> [!WARNING]
> In Python, assigning to a repeated or composite field directly (e.g., `person.phones = [...]`) raises an `AttributeError`. Use `.add()`, `.append()`, `.extend()`, or `.CopyFrom()` instead.

#### Java

```java
import com.example.tutorial.protos.Person;
import com.example.tutorial.protos.AddressBook;
import com.google.protobuf.util.Timestamps;

// Create and populate
Person person = Person.newBuilder()
    .setName("Bob")
    .setId(1234)
    .setEmail("[email removed]")
    .addPhones(Person.PhoneNumber.newBuilder()
        .setNumber("555-4321")
        .setType(Person.PhoneType.HOME))
    .setLastUpdated(Timestamps.now())
    .build();

// Serialize to bytes
byte[] data = person.toByteArray();

// Deserialize from bytes
Person person2 = Person.parseFrom(data);

// Write to file
try (FileOutputStream output = new FileOutputStream("person.bin")) {
    person.writeTo(output);
}

// Read from file
Person person3 = Person.parseFrom(new FileInputStream("person.bin"));
```

#### C++

```cpp
#include "person.pb.h"
#include <fstream>

tutorial::Person person;
person.set_name("Bob");
person.set_id(1234);
person.set_email("[email removed]");

auto* phone = person.add_phones();
phone->set_number("555-4321");
phone->set_type(tutorial::Person::HOME);

// Serialize
std::string data;
person.SerializeToString(&data);

// Deserialize
tutorial::Person person2;
person2.ParseFromString(data);

// Write to file
std::ofstream output("person.bin", std::ios::binary);
person.SerializeToOstream(&output);
```

#### C#

```csharp
using Google.Protobuf;
using Tutorial;

var person = new Person
{
    Name = "Bob",
    Id = 1234,
    Email = "[email removed]",
    Phones = { new Person.Types.PhoneNumber
    {
        Number = "555-4321",
        Type = Person.Types.PhoneType.Home
    }}
};

// Serialize
byte[] data = person.ToByteArray();

// Deserialize
Person person2 = Person.Parser.ParseFrom(data);
```

#### Ruby

```ruby
require 'google/protobuf'
require 'addressbook_pb'

person = Tutorial::Person.new(
  name: "Bob",
  id: 1234,
  email: "[email removed]"
)
person.phones << Tutorial::Person::PhoneNumber.new(
  number: "555-4321",
  type: :HOME
)

# Serialize
data = Tutorial::Person.encode(person)

# Deserialize
person2 = Tutorial::Person.decode(data)
```

#### PHP

```php
use Google\Protobuf\Internal\Message;
use Tutorial\Person;
use Tutorial\Person\PhoneNumber;
use Tutorial\Person\PhoneType;

$person = new Person();
$person->setName("Bob");
$person->setId(1234);
$person->setEmail("[email removed]");

$phone = new PhoneNumber();
$phone->setNumber("555-4321");
$phone->setType(PhoneType::HOME);
$person->setPhones([$phone]);

// Serialize
$data = $person->serializeToString();

// Deserialize
$person2 = new Person();
$person2->mergeFromString($data);
```

#### Rust

```rust
use protobuf::proto;

// Using the proto! macro
let person = proto!(Person {
    name: "Bob",
    id: 1234,
    email: "[email removed]"
});

// Serialize
let data = person.serialize().unwrap();

// Deserialize
let person2 = Person::parse(&data).unwrap();
assert_eq!(person2.name(), "Bob");
```

> [!NOTE]
> The Rust protobuf crate (V4) is backed by either the upb C library or the C++ protobuf implementation. It provides a safe Rust API with `View`, `Mut`, and owned message types for zero-copy access patterns.

#### Go

```go
import (
    "google.golang.org/protobuf/proto"
    tutorialpb "github.com/protocolbuffers/protobuf/examples/go/tutorialpb"
)

person := &tutorialpb.Person{
    Name:  "Bob",
    Id:    1234,
    Email: "[email removed]",
}

// Serialize
data, err := proto.Marshal(person)

// Deserialize
person2 := &tutorialpb.Person{}
err = proto.Unmarshal(data, person2)
```

---

## 5. Common Tasks

### Task 1: Converting Between Protobuf and JSON

**Goal:** Serialize protobuf messages to JSON for REST APIs or debugging.

**Python:**

```python
from google.protobuf.json_format import MessageToJson, Parse

# To JSON
json_string = MessageToJson(person)
print(json_string)

# From JSON
person2 = person_pb2.Person()
Parse(json_string, person2)
```

```text
{
  "name": "Bob",
  "id": 1234,
  "email": "[email removed]"
}
```

**Java:**

```java
import com.google.protobuf.util.JsonFormat;

// To JSON
String json = JsonFormat.printer().print(person);

// From JSON
Person.Builder builder = Person.newBuilder();
JsonFormat.parser().merge(json, builder);
Person person2 = builder.build();
```

**PHP:**

```php
// To JSON
$json = $person->serializeToJsonString();

// From JSON
$person2 = new Person();
$person2->mergeFromJsonString($json);
```

> [!TIP]
> JSON output uses camelCase field names by default (e.g., `last_updated` becomes `lastUpdated`). Both naming conventions are accepted when parsing.

### Task 2: Using Well-Known Types

Protobuf includes a set of well-known types for common patterns. Import them from `google/protobuf/`:

| Type | Import | Use Case |
|------|--------|----------|
| `Timestamp` | `google/protobuf/timestamp.proto` | Point in time (seconds + nanos since epoch) |
| `Duration` | `google/protobuf/duration.proto` | Span of time |
| `Any` | `google/protobuf/any.proto` | Arbitrary message wrapping |
| `Struct` | `google/protobuf/struct.proto` | Dynamic JSON-like structures |
| `Empty` | `google/protobuf/empty.proto` | No-data placeholder (RPC returns) |
| `FieldMask` | `google/protobuf/field_mask.proto` | Partial update specification |
| `Wrappers` | `google/protobuf/wrappers.proto` | Nullable scalar wrappers |

**Using Timestamp (Python):**

```python
from google.protobuf.timestamp_pb2 import Timestamp
from datetime import datetime

ts = Timestamp()
ts.GetCurrentTime()

# From datetime
ts.FromDatetime(datetime(2026, 1, 15, 10, 30, 0))

# To datetime
dt = ts.ToDatetime()
```

> [!WARNING]
> `Timestamp` seconds must be in range `[-62135596800, 253402300799]` and nanos in `[0, 999999999]`. Values outside these ranges raise a `ValueError`.

**Using Duration (Python):**

```python
from google.protobuf.duration_pb2 import Duration
from datetime import timedelta

dur = Duration()
dur.FromTimedelta(timedelta(hours=2, minutes=30))
td = dur.ToTimedelta()
```

**Using Any (Java):**

```java
import com.google.protobuf.Any;

Any any = Any.pack(person);

if (any.is(Person.class)) {
    Person unpacked = any.unpack(Person.class);
}
```

### Task 3: Working with Maps

```protobuf
message GameState {
  map<string, int32> player_scores = 1;
  map<int32, string> id_to_name = 2;
}
```

**Python:**

```python
game = game_pb2.GameState()
game.player_scores["alice"] = 100
game.player_scores["bob"] = 85

for name, score in game.player_scores.items():
    print(f"{name}: {score}")
```

> [!NOTE]
> Map fields cannot be `repeated`. Map iteration order is not guaranteed to be consistent across languages or serialization round-trips.

### Task 4: Using Oneof for Variant Types

```protobuf
message PaymentMethod {
  oneof method {
    CreditCard card = 1;
    BankTransfer transfer = 2;
    string voucher_code = 3;
  }
}
```

**Python:**

```python
payment = PaymentMethod()
payment.voucher_code = "SAVE20"

# Check which field is set
which = payment.WhichOneof("method")
print(which)  # "voucher_code"

# Setting a different oneof field clears the previous one
payment.card.CopyFrom(some_card)
print(payment.HasField("voucher_code"))  # False
```

### Task 5: Schema Evolution — Adding and Removing Fields

**Adding a new field (backward compatible):**

```protobuf
message Person {
  string name = 1;
  int32 id = 2;
  string email = 3;
  string nickname = 4;  // NEW — old code ignores this field
}
```

**Removing a field (use `reserved` to prevent reuse):**

```protobuf
message Person {
  reserved 3;
  reserved "email";
  string name = 1;
  int32 id = 2;
  // email was field 3, now removed
}
```

> [!IMPORTANT]
> Never reuse a field number after removing a field — old serialized data still uses those numbers. Always mark removed field numbers and names as `reserved`.

### Task 6: Writing Proto Files with Imports and Packages

```protobuf
syntax = "proto3";
package mycompany.myproject;

import "google/protobuf/timestamp.proto";
import "common/shared_types.proto";

message Order {
  string id = 1;
  common.Customer customer = 2;
  repeated common.LineItem items = 3;
  google.protobuf.Timestamp created_at = 4;
}
```

Compile with import paths:

```bash
protoc \
  -I=./protos \
  -I=./third_party \
  --java_out=./gen \
  ./protos/mycompany/myproject/order.proto
```

### Task 7: Working with Repeated Fields (Lists)

**Python:**

```python
person = person_pb2.Person()

# Add items
phone = person.phones.add()
phone.number = "555-1234"
phone.type = person_pb2.Person.MOBILE

# Add multiple
person.phones.extend([
    person_pb2.Person.PhoneNumber(number="555-5678", type=person_pb2.Person.WORK)
])

# Iterate
for phone in person.phones:
    print(phone.number)

# Length
print(len(person.phones))
```

**Java:**

```java
Person.Builder builder = Person.newBuilder();
builder.addPhones(Person.PhoneNumber.newBuilder()
    .setNumber("555-1234")
    .setType(Person.PhoneType.MOBILE));

// Get count
int count = builder.getPhonesCount();

// Get by index
Person.PhoneNumber first = builder.getPhones(0);

// Get list
List<Person.PhoneNumber> all = builder.getPhonesList();
```

### Task 8: Checking Field Presence

In proto3, singular scalar fields return their default value when not set. Use `optional` to track presence explicitly:

```protobuf
message Settings {
  optional int32 timeout = 1;     // Has presence
  int32 retries = 2;              // No presence (default = 0)
}
```

**Python:**

```python
settings = settings_pb2.Settings()

# Check if optional field was explicitly set
if settings.HasField("timeout"):
    print(settings.timeout)

# For oneof
which = msg.WhichOneof("my_oneof")
```

**Java:**

```java
if (settings.hasTimeout()) {
    int timeout = settings.getTimeout();
}
```

### Task 9: Using Extensions (proto2 only)

```protobuf
syntax = "proto2";

message MyMessage {
  extensions 100 to 199;
}

extend MyMessage {
  optional string my_extension = 100;
}
```

### Task 10: Serializing to and from Files

**Python:**

```python
# Write
with open("data.bin", "wb") as f:
    f.write(address_book.SerializeToString())

# Read
address_book = addressbook_pb2.AddressBook()
with open("data.bin", "rb") as f:
    address_book.ParseFromString(f.read())
```

**Java:**

```java
// Write
try (FileOutputStream out = new FileOutputStream("data.bin")) {
    addressBook.writeTo(out);
}

// Read
AddressBook book = AddressBook.parseFrom(new FileInputStream("data.bin"));
```

**Ruby:**

```ruby
# Write
File.open("data.bin", "wb") { |f| f.write(Tutorial::AddressBook.encode(book)) }

# Read
book = Tutorial::AddressBook.decode(File.read("data.bin", mode: "rb"))
```

### Task 11: Using Kotlin DSL

When targeting the JVM, Kotlin provides a DSL for building messages:

```kotlin
val person = person {
    name = "Alice"
    id = 42
    email = "[email removed]"
    phones += phoneNumber {
        number = "555-1234"
        type = Person.PhoneType.MOBILE
    }
}
```

> [!NOTE]
> The Kotlin DSL requires both `--java_out` and `--kotlin_out` flags when running protoc. It sits on top of the Java generated code.

### Task 12: Merging Messages

**Python:**

```python
# MergeFrom: existing fields are overwritten, repeated fields are appended
target = person_pb2.Person()
target.MergeFrom(source)

# CopyFrom: clears target first, then copies
target.CopyFrom(source)
```

**Java:**

```java
Person.Builder builder = Person.newBuilder();
builder.mergeFrom(existingPerson);
```

### Task 13: Comparing Messages

**Python:**

```python
if person1 == person2:
    print("Messages are equal")
```

**Java:**

```java
if (person1.equals(person2)) {
    System.out.println("Messages are equal");
}
```

### Task 14: Working with Unknown Fields

When a message is deserialized with fields that don't match the schema, those fields are preserved as unknown fields and re-serialized. This is key to forward compatibility.

**Python:**

```python
# Access unknown fields (read-only)
from google.protobuf.internal.containers import UnknownFieldSet

unknown = UnknownFieldSet(message)
for field in unknown:
    print(field.field_number, field.wire_type, field.data)
```

### Task 15: Using the Lite Runtime (Android/Embedded)

For resource-constrained environments, use the lite runtime which produces smaller generated code:

**Java (Maven):**

```xml
<dependency>
  <groupId>com.google.protobuf</groupId>
  <artifactId>protobuf-javalite</artifactId>
  <version>4.36.0</version>
</dependency>
```

The lite runtime omits reflection, descriptors, and text format support but provides full serialization/deserialization.

### Task 16: Text Format (Debugging)

**Python:**

```python
from google.protobuf import text_format

# To text format
text = text_format.MessageToString(person)
print(text)

# From text format
person2 = person_pb2.Person()
text_format.Parse(text, person2)
```

```text
name: "Bob"
id: 1234
email: "[email removed]"
phones {
  number: "555-4321"
  type: HOME
}
```

### Task 17: Defining Services (for gRPC)

```protobuf
syntax = "proto3";

service Greeter {
  rpc SayHello (HelloRequest) returns (HelloReply);
  rpc SayHelloStream (HelloRequest) returns (stream HelloReply);
}

message HelloRequest {
  string name = 1;
}

message HelloReply {
  string message = 1;
}
```

Generate gRPC stubs alongside protobuf code using language-specific gRPC plugins.

### Task 18: Using Editions (replacing proto2/proto3 syntax)

Editions are the modern way to evolve protobuf language features:

```protobuf
edition = "2023";
package mypackage;

message MyMessage {
  string name = 1;
  int32 id = 2;
}
```

Editions replace the `syntax` keyword and allow gradual feature migration via feature flags.

### Task 19: Building with Bazel

```python
# BUILD file
load("@rules_proto//proto:proto_library.bzl", "proto_library")
load("@protobuf//bazel:java_proto_library.bzl", "java_proto_library")

proto_library(
    name = "person_proto",
    srcs = ["person.proto"],
)

java_proto_library(
    name = "person_java_proto",
    deps = [":person_proto"],
)
```

### Task 20: Default Values

In proto3, fields have language-specific default values when not set:

| Type | Default Value |
|------|--------------|
| `string` | `""` (empty string) |
| `bytes` | `b""` (empty bytes) |
| `bool` | `false` |
| `int32/64`, `uint32/64`, etc. | `0` |
| `float`, `double` | `0.0` |
| `enum` | First defined value (must be 0) |
| `message` | Language-dependent (null/nil/None) |
| `repeated` | Empty list |
| `map` | Empty map |

---

## 6. Troubleshooting

### Diagnostic Decision Tree

```text
Problem with protobuf?
├── Compilation error in .proto file?
│   ├── Check syntax declaration (proto2/proto3/edition)
│   ├── Check field numbers are unique within message
│   ├── Check enum first value is zero (proto3)
│   └── Check import paths match -I flags
├── Generated code won't compile?
│   ├── Version mismatch: protoc vs runtime library
│   ├── Missing imports in generated code
│   └── Language-specific build config issue
├── Serialization/deserialization failure?
│   ├── Schema mismatch between writer and reader
│   ├── Truncated data (incomplete read)
│   └── Encoding issue (string not valid UTF-8)
├── Performance issue?
│   ├── Python: check backend (upb vs pure python)
│   ├── Large messages: consider streaming
│   └── Repeated parsing: reuse message objects
└── Runtime error?
    ├── Check error message in Troubleshooting below
    └── Check GitHub issues
```

### Error: `Truncated message.`

**Cause:** The binary data being parsed is incomplete — the parser reached the end of input before the message was fully decoded.

**Solution:**
1. Verify the data was fully written (check file size, network transfer)
2. Ensure you're reading in binary mode (`"rb"` in Python)
3. Check for buffer size issues when reading from streams

**Prevention:** Always verify the byte count matches between serializer and parser.

### Error: `Too many bytes when decoding varint.`

**Cause:** A variable-length integer (varint) in the binary data exceeded the maximum 10-byte encoding.

**Solution:**
1. Check for data corruption
2. Verify the data is actually protobuf-encoded (not JSON or text format)
3. Ensure you're not accidentally reading text-format data as binary

### Error: `DecodeError: Field number 0 is illegal.`

**Cause:** The parser encountered a field with number 0, which is reserved by the wire format.

**Solution:**
1. Check for data corruption or incorrect offset when reading
2. Ensure the data is protobuf binary format, not JSON or text

### Error: `Missing group end tag.`

**Cause:** A group-encoded field (legacy proto2 feature) is missing its terminator.

**Solution:**
1. Verify data integrity
2. If migrating from proto2, ensure groups are properly closed

### Error: `Unexpected end-group tag.`

**Cause:** An end-group wire type appeared where the parser didn't expect one.

**Solution:**
1. Check for schema mismatch between serializer and deserializer
2. Verify data isn't truncated or corrupted

### Error: `Error parsing message: too many levels of nesting.`

**Cause:** The message exceeds the recursion depth limit (default: 100 levels).

**Solution:**
1. Flatten deeply nested message structures
2. Increase the recursion limit via `CodedInputStream.setRecursionLimit()` (Java) if the nesting is intentional

**Prevention:** Design schemas with reasonable nesting depth — deeply nested structures indicate a design problem.

### Error: `Message is missing required fields` (proto2)

**Cause:** A proto2 `required` field was not set before serialization.

**Solution:**
1. Set all required fields before calling `SerializeToString()`
2. Use `IsInitialized()` to check before serializing

### Error: `Protocol message has no "X" field.`

**Cause:** Attempting to access a field that doesn't exist in the message descriptor.

**Solution:**
1. Check for typos in field names
2. Verify you're using the correct generated class version
3. Regenerate code if the `.proto` file was recently updated

### Error: `Cannot set X to Y: TypeError`

**Cause:** Attempting to assign a value of the wrong type to a field (e.g., string to int32).

**Solution:**
1. Check the field type in the `.proto` file
2. Convert the value to the correct type before assignment

### Error: `Assignment not allowed to repeated field`

**Cause:** In Python, directly assigning to a repeated field (e.g., `msg.items = [...]`).

**Solution:**
Use mutation methods instead:
```python
msg.items.extend([item1, item2])
# or
msg.items.append(item)
# or
del msg.items[:]
msg.items.extend(new_items)
```

### Error: `Assignment not allowed to composite field`

**Cause:** In Python, directly assigning to a sub-message field.

**Solution:**
Use `CopyFrom()` or `MergeFrom()`:
```python
msg.sub_message.CopyFrom(other_sub_message)
```

### Error: `Enum has no value defined for name`

**Cause:** Looking up an enum value by a name that doesn't exist.

**Solution:**
1. Check for typos in the enum value name
2. Verify the `.proto` file contains the expected enum values
3. Regenerate code after proto file changes

### Error: `Value out of range` for integer fields

**Cause:** Assigning a value outside the valid range for the field type.

**Solution:**
Check the range for your field type:
- `int32`: -2,147,483,648 to 2,147,483,647
- `uint32`: 0 to 4,294,967,295
- `int64`: -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807

### Error: OOM in packed field decoders (GitHub issue #27104)

**Cause:** Maliciously crafted messages with extremely large packed field counts can cause out-of-memory conditions.

**Solution:**
1. Set message size limits when parsing untrusted input
2. Use `CodedInputStream.setSizeLimit()` (Java/C++) to cap message size

> [!CAUTION]
> Always set size limits when parsing protobuf messages from untrusted sources. Without limits, a malicious message can consume all available memory.

### Error: Null Pointer Dereference in `EpsCopyOutputStream::EnableAliasing` (GitHub issue #27099)

**Cause:** Using the array-only constructor with `EnableAliasing` can trigger a null pointer dereference.

**Solution:**
1. Avoid the array-only constructor pattern when aliasing is enabled
2. Update to a patched version when available

### Error: Allocation-size-too-big crash in `GzipOutputStream::Init` (GitHub issue #27097)

**Cause:** A negative `buffer_size` parameter in `GzipOutputStream::Init` is not validated, causing an allocation failure.

**Solution:**
1. Ensure buffer sizes are always positive
2. Update to a patched version when available

### Error: PHP `GPBDecodeException: Fail to push limit.`

**Cause:** Malformed or truncated protobuf data being parsed by the PHP runtime.

**Solution:**
1. Verify the data source is producing valid protobuf binary format
2. Check for data truncation during transmission

### Error: PHP `GPBDecodeException: Illegal field number zero.`

**Cause:** Corrupted protobuf data containing an invalid field number.

**Solution:**
1. Verify data integrity
2. Ensure the correct binary data is being passed to `mergeFromString()`

### Error: PHP `GPBDecodeException: Unexpected wire type.`

**Cause:** The wire type in the serialized data doesn't match the expected type for the field.

**Solution:**
1. Check for schema version mismatch between producer and consumer
2. Verify data isn't corrupted or mixed with non-protobuf data

### Error: PHP `GPBDecodeException: Invalid base64 characters`

**Cause:** Base64-encoded protobuf data contains invalid characters when parsed via JSON.

**Solution:**
1. Ensure proper base64 encoding of bytes fields
2. Use standard base64 encoding (not URL-safe) unless the parser expects it

### Error: `RuntimeError: The generated code only work with python cpp extension`

**Cause:** Generated Python code was produced for the C++ extension backend but the pure Python runtime is active.

**Solution:**
1. Regenerate the Python code
2. Or set `PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=upb` (default)

### Error: `PickleError: Can't pickle repeated scalar fields`

**Cause:** Attempting to pickle a `RepeatedScalarFieldContainer` directly.

**Solution:**
Convert to a list first:
```python
import pickle
my_list = list(msg.repeated_field)
pickle.dumps(my_list)
```

### Error: FieldMask validation errors

**Cause:** Invalid field mask path names — paths must use lowercase with underscores.

**Solution:**
1. Path names must not contain uppercase letters when printing to JSON
2. Path names must not contain underscores when parsing from JSON (use camelCase)
3. Trailing underscores in path names are not allowed

### Error: `Duration is not valid: Sign mismatch.`

**Cause:** Duration seconds and nanos have different signs.

**Solution:**
Ensure both `seconds` and `nanos` are either both non-negative or both non-positive:
- Valid: `seconds=5, nanos=500000000`
- Invalid: `seconds=5, nanos=-500000000`

> [!WARNING]
> Duration seconds must be in range `[-315576000000, 315576000000]` (approximately ±10,000 years) and nanos in `[-999999999, 999999999]`.

### Error: `Timestamp JSON value not a string`

**Cause:** Parsing a Timestamp from JSON but the value is not a string.

**Solution:**
Timestamp JSON format requires RFC 3339 strings:
```json
"2026-01-15T10:30:00Z"
```

### Error: `Duration must end with letter "s"`

**Cause:** Parsing a Duration from JSON but the value doesn't end with `s`.

**Solution:**
Duration JSON format requires a string ending in `s`:
```json
"3600s"
```

---

## 7. FAQ

### General

**Q: What is the difference between proto2 and proto3?**

Proto3 simplifies the language by removing required fields, making all fields optional by default, and dropping default value customization. Proto3 is the recommended syntax for new projects. Proto2 remains supported for existing schemas. The new Editions system is the forward-looking replacement for both.

**Q: What are Editions and should I use them?**

Editions replace the `syntax = "proto2"` / `syntax = "proto3"` declaration with `edition = "2023"` (or later). They allow gradual, per-feature migration without a hard syntax boundary. For new projects, either proto3 or edition 2023 is recommended.

**Q: Which languages does protobuf support?**

First-party support (in this repository): C++, Java, Python, C#, Objective-C, Ruby, PHP, Kotlin, Rust. First-party but separate repos: Go (`protobuf-go`), Dart (`dart-lang/protobuf`), JavaScript (`protobuf-javascript`). Many community-maintained implementations exist for other languages.

**Q: Is protobuf backward/forward compatible?**

Yes, by design. New fields can be added without breaking old readers (they ignore unknown fields). Old fields can be removed if their numbers are reserved. Never reuse field numbers.

**Q: What is the maximum message size?**

The default limit is 64 MB (configurable). For larger payloads, consider streaming or splitting into multiple messages.

### Schema Design

**Q: Should I use `int32` or `sint32`?**

Use `int32` for values that are usually non-negative. Use `sint32` for values that are frequently negative — it uses ZigZag encoding which is more efficient for negative numbers. Same applies to `int64` vs `sint64`.

**Q: When should I use `fixed32`/`fixed64` instead of `int32`/`int64`?**

Use `fixed32`/`fixed64` when values are typically large (> 2^28 or 2^56) — they use a constant 4 or 8 bytes, which is more efficient than varint encoding for large values.

**Q: Can I use `float` or `double` as map keys?**

No. Map keys can only be integer types (`int32`, `int64`, `uint32`, `uint64`, `sint32`, `sint64`, `fixed32`, `fixed64`, `sfixed32`, `sfixed64`), `bool`, or `string`.

**Q: How do I represent nullable fields in proto3?**

Use `optional` to track field presence, or use wrapper types from `google/protobuf/wrappers.proto` (e.g., `google.protobuf.Int32Value`).

**Q: What field numbers should I avoid?**

Field numbers 19000–19999 are reserved by the protobuf implementation. Field number 0 is invalid. Numbers 1–15 use 1 byte on the wire, 16–2047 use 2 bytes.

**Q: Can map fields be repeated?**

No. `repeated map<K,V>` is not valid protobuf syntax. Use a message wrapper if you need a list of maps.

### Runtime and Performance

**Q: Which Python backend should I use?**

Use the default `upb` backend. It's C-based and provides the best performance. The `cpp` backend is deprecated and the `python` (pure Python) backend is significantly slower.

**Q: What is the Lite runtime?**

The lite runtime (`protobuf-javalite` for Java) produces smaller generated code by omitting reflection, descriptors, and text format support. Use it for Android apps and other size-sensitive environments.

**Q: How do I improve serialization performance?**

1. Reuse message objects instead of creating new ones per operation
2. Use the appropriate backend (upb for Python, C++ runtime for JVM)
3. Consider the lite runtime for mobile/embedded
4. Use `ByteString` (Java) to avoid copying byte arrays

**Q: Are protobuf messages thread-safe?**

Immutable message objects (e.g., built Java messages) are thread-safe for reading. Mutable builders and Python message objects are NOT thread-safe — use external synchronization if sharing across threads.

### Interoperability

**Q: Can I mix proto2 and proto3 messages?**

Yes, proto2 and proto3 messages can import and reference each other. However, some features like required fields (proto2-only) won't be available when accessed from proto3 code.

**Q: How do I send protobuf over HTTP?**

Serialize to bytes and send with `Content-Type: application/x-protobuf` (binary) or use JSON serialization with `Content-Type: application/json`.

**Q: Does protobuf support streaming?**

The wire format itself doesn't include message framing — you need to prepend length headers when sending multiple messages over a stream. Use `writeDelimitedTo()`/`parseDelimitedFrom()` (Java) or implement length-prefixed framing.

**Q: Can I convert between proto2 and proto3?**

Use the Editions migration tool (`protoc --editions_migration`) or manually convert. Proto3 drops required fields and custom defaults, so review those semantics before converting.

### Build and Tooling

**Q: Do I need Bazel to use protobuf?**

No. Protobuf supports both Bazel and CMake for building from source. Most users only need the `protoc` binary (available as pre-built downloads) and a language-specific runtime installed via their package manager.

**Q: How do I use protobuf with gRPC?**

Define `service` blocks in your `.proto` files and use the gRPC protoc plugin for your language (e.g., `--grpc-java_out`, `--grpc_python_out`). The gRPC framework handles the RPC transport.

**Q: Why does `protoc` not find my imported `.proto` files?**

Use the `-I` (or `--proto_path`) flag to specify directories where `protoc` should search for imports. Multiple `-I` flags can be specified. The import path in the `.proto` file must be relative to one of the `-I` directories.

**Q: How do I generate code for multiple languages at once?**

Specify multiple output flags in a single `protoc` invocation:
```bash
protoc --java_out=./java --python_out=./python --cpp_out=./cpp myproto.proto
```

**Q: Where are the well-known type `.proto` files?**

They're bundled with the `protoc` binary in the `include/` directory of the release archive. The compiler automatically finds them when you use `-I` to point at that directory, or when they're installed to a system include path.

---

_Generated May 2026 from protocolbuffers/protobuf at 7bb6178. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._