# Apache Dubbo User Guide

Apache Dubbo is a high-performance, Java-native **Web and RPC framework** for building enterprise-grade microservices. You define an ordinary Java interface, publish its implementation as a **service provider**, and call it from a **service consumer** as if it were a local method — Dubbo handles the networking, serialization, service discovery, load balancing, and fault tolerance for you.

This guide is for the people who **build applications with Dubbo**: backend and microservice developers writing providers and consumers in Java or Spring Boot. It covers installation, the everyday API and configuration surface, the protocols and registries you will choose between, traffic-management features, day-to-day operations, troubleshooting, and a complete configuration reference.

> [!NOTE]
> Dubbo is polyglot. This guide documents the **Java SDK** (the `apache/dubbo` repository). Sibling implementations exist for [Go](https://github.com/apache/dubbo-go), [Python](https://github.com/dubbo/py-client-for-apache-dubbo), [PHP](https://github.com/apache/dubbo-php-framework), [Rust](https://github.com/apache/dubbo-rust), [Erlang](https://github.com/apache/dubbo-erlang), and [Node.js/Web](https://github.com/apache/dubbo-js). The concepts carry over, but the code in this guide is Java.

---

## Table of Contents

1. [What Dubbo Does](#1-what-dubbo-does)
2. [Quick Start (5 minutes)](#2-quick-start-5-minutes)
3. [Installation](#3-installation)
4. [Core Concepts](#4-core-concepts)
5. [Defining a Service Provider](#5-defining-a-service-provider)
6. [Consuming a Service](#6-consuming-a-service)
7. [Configuration Essentials](#7-configuration-essentials)
8. [Protocols](#8-protocols)
9. [Registries and Service Discovery](#9-registries-and-service-discovery)
10. [Traffic Management](#10-traffic-management)
11. [Common Tasks (Recipes)](#11-common-tasks-recipes)
12. [Operating a Running Application (QOS)](#12-operating-a-running-application-qos)
13. [Troubleshooting](#13-troubleshooting)
14. [Frequently Asked Questions](#14-frequently-asked-questions)
15. [Configuration Reference](#15-configuration-reference)

---

## 1. What Dubbo Does

A microservice system has two recurring needs: **services need to call each other**, and **callers need to find callees** that move around as instances scale up, down, and redeploy. Dubbo solves both, plus the operational concerns that come with them.

| Capability | What you get |
|---|---|
| **Remote invocation (RPC)** | Call a remote service through a normal Java interface. Multiple wire protocols: Triple (gRPC-compatible, HTTP/2), Dubbo2 (TCP), REST, and pluggable custom protocols. |
| **Service discovery** | Providers register themselves in a registry (Zookeeper, Nacos, …); consumers subscribe and discover live instances automatically. |
| **Traffic management** | Cluster fault-tolerance (failover, failfast, …), six load-balancing strategies, routing rules, and method-level configuration. |
| **Observability** | Built-in metrics (Micrometer/Prometheus) and distributed tracing (OpenTelemetry, Zipkin, Brave). |
| **Security** | Token authentication, TLS, serialization allow-listing, and a permission-gated operations port. |
| **Extensibility** | Almost every component is a swappable SPI extension — you can replace protocols, registries, serializers, filters, routers, and load balancers without forking. |

> [!TIP]
> If you are starting a brand-new service, prefer the **Triple protocol**. It runs over HTTP/2, is gRPC- and cURL-compatible, works through service meshes, and is the protocol Dubbo invests in going forward. The classic **`dubbo`** TCP protocol remains the default for interface-based RPC and is fully supported.

---

## 2. Quick Start (5 minutes)

This is the smallest possible Dubbo system: one shared interface, one provider, one consumer, all wired with the programmatic (Bootstrap) API and an in-process registry-free setup is not used — a registry makes the example realistic. We use Zookeeper.

### 2.1 The shared interface

Put this in a module both sides depend on:

```java
package org.apache.dubbo.demo;

public interface GreetingService {
    String sayHello(String name);
}
```

### 2.2 The provider

```java
package org.apache.dubbo.demo.provider;

import org.apache.dubbo.config.ApplicationConfig;
import org.apache.dubbo.config.RegistryConfig;
import org.apache.dubbo.config.ServiceConfig;
import org.apache.dubbo.config.bootstrap.DubboBootstrap;
import org.apache.dubbo.demo.GreetingService;

public class Provider {
    public static void main(String[] args) {
        ServiceConfig<GreetingService> service = new ServiceConfig<>();
        service.setInterface(GreetingService.class);
        service.setRef((name) -> "Hello, " + name);

        DubboBootstrap.getInstance()
                .application(new ApplicationConfig("first-dubbo-provider"))
                .registry(new RegistryConfig("zookeeper://127.0.0.1:2181"))
                .service(service)
                .start()
                .await();
    }
}
```

### 2.3 The consumer

```java
package org.apache.dubbo.demo.consumer;

import org.apache.dubbo.config.ApplicationConfig;
import org.apache.dubbo.config.ReferenceConfig;
import org.apache.dubbo.config.RegistryConfig;
import org.apache.dubbo.config.bootstrap.DubboBootstrap;
import org.apache.dubbo.demo.GreetingService;

public class Consumer {
    public static void main(String[] args) {
        ReferenceConfig<GreetingService> reference = new ReferenceConfig<>();
        reference.setInterface(GreetingService.class);

        DubboBootstrap.getInstance()
                .application(new ApplicationConfig("first-dubbo-consumer"))
                .registry(new RegistryConfig("zookeeper://127.0.0.1:2181"))
                .reference(reference)
                .start();

        GreetingService greetingService = reference.get();
        System.out.println(greetingService.sayHello("Dubbo"));
    }
}
```

### 2.4 Run it

```bash
# 1. Start a Zookeeper registry (Docker is the fastest path)
docker run -d --name zookeeper -p 2181:2181 zookeeper:3.9

# 2. Start the provider, then the consumer (two JVMs)
java org.apache.dubbo.demo.provider.Provider
java org.apache.dubbo.demo.consumer.Consumer
```

Expected consumer output:

```text
Hello, Dubbo
```

> [!IMPORTANT]
> The interface (`GreetingService`) must be on the **classpath of both** provider and consumer. Dubbo binds providers and consumers by the service's fully-qualified interface name (plus optional `group` and `version`), not by a generated stub. Keep the interface in a shared artifact.

---

## 3. Installation

Dubbo is consumed as a set of Maven artifacts under the `org.apache.dubbo` group. You do **not** "install Dubbo" as a standalone program — you add it to your application's build.

### 3.1 Version and compatibility

| Dubbo line | JDK support | Status |
|---|---|---|
| `3.3.x` | 1.8 – 25 | **Current**, actively maintained — Triple, REST, Spring Boot starters, native-image, JDK 25 |
| `3.2.x` | 1.8 – 17 | Maintained — metrics & tracing, thread-pool isolation, native image |
| `3.1.x` | 1.8 – 17 | Stable, not actively maintained |

> [!NOTE]
> This guide is generated from source at commit `ee9c190` (development line `3.3.7-SNAPSHOT`). For application dependencies, use the **latest stable `3.3.x` release** (`3.3.6` at time of writing) rather than a `-SNAPSHOT`. Snapshot builds are for contributors, not production.

### 3.2 Plain Maven / Gradle

The cleanest way to pin a consistent set of versions is the Dubbo BOM.

=== "Maven (BOM + dependency)"

    ```xml
    <dependencyManagement>
      <dependencies>
        <dependency>
          <groupId>org.apache.dubbo</groupId>
          <artifactId>dubbo-bom</artifactId>
          <version>3.3.6</version>
          <type>pom</type>
          <scope>import</scope>
        </dependency>
      </dependencies>
    </dependencyManagement>

    <dependencies>
      <dependency>
        <groupId>org.apache.dubbo</groupId>
        <artifactId>dubbo</artifactId>
      </dependency>
      <!-- pick a registry -->
      <dependency>
        <groupId>org.apache.dubbo</groupId>
        <artifactId>dubbo-registry-zookeeper</artifactId>
      </dependency>
    </dependencies>
    ```

=== "Gradle (Kotlin DSL)"

    ```kotlin
    dependencies {
        implementation(platform("org.apache.dubbo:dubbo-bom:3.3.6"))
        implementation("org.apache.dubbo:dubbo")
        implementation("org.apache.dubbo:dubbo-registry-zookeeper")
    }
    ```

> [!TIP]
> The umbrella `dubbo` artifact pulls in the core plus the common protocols and serializers. As you move to production you will usually depend on **specific** modules instead (`dubbo-rpc-triple`, `dubbo-remoting-netty4`, `dubbo-serialization-hessian2`, a registry, a config center, a metadata report) to keep the dependency tree lean. See the [Admin Guide](apache__dubbo__AdminGuide.md) for the full module map.

### 3.3 Spring Boot

For Spring Boot applications, add a starter and configure everything in `application.yml`.

```xml
<dependency>
  <groupId>org.apache.dubbo</groupId>
  <artifactId>dubbo-spring-boot-starter</artifactId>
  <version>3.3.6</version>
</dependency>
<dependency>
  <groupId>org.apache.dubbo</groupId>
  <artifactId>dubbo-zookeeper-curator5-spring-boot-starter</artifactId>
  <version>3.3.6</version>
</dependency>
```

> [!NOTE]
> On JDK 17+, the build automatically uses `dubbo-spring-boot-3-autoconfigure` (Spring Boot 3 / Jakarta). On JDK 8–11 it uses the Spring Boot 2 autoconfigure path. You add the same `dubbo-spring-boot-starter` either way.

### 3.4 Which add-on modules?

Dubbo's functionality is split across optional modules so you only ship what you use. The most common ones:

| Need | Add this artifact |
|---|---|
| Zookeeper registry / config center | `dubbo-registry-zookeeper`, `dubbo-zookeeper-curator5-spring-boot-starter` |
| Nacos registry | `dubbo-registry-nacos`, `dubbo-nacos-spring-boot-starter` |
| Triple (gRPC/HTTP-2) protocol | `dubbo-rpc-triple` |
| Classic Dubbo TCP protocol | `dubbo-rpc-dubbo`, `dubbo-remoting-netty4` |
| Hessian2 serialization (default) | `dubbo-serialization-hessian2` |
| Fastjson2 serialization | `dubbo-serialization-fastjson2` |
| Metrics + tracing | `dubbo-observability-spring-boot-starter`, `dubbo-tracing-otel-zipkin-spring-boot-starter` |
| Operations port (telnet/HTTP) | `dubbo-qos` |
| Sentinel flow control | `dubbo-sentinel-spring-boot-starter` |
| Seata distributed transactions | `dubbo-seata-spring-boot-starter` |

---

## 4. Core Concepts

You only need a handful of concepts to be productive. Deeper internals live in the [Admin Guide](apache__dubbo__AdminGuide.md).

**Provider**
: An application that **exposes** a service implementation. It registers its address in the registry.

**Consumer**
: An application that **calls** a service. It discovers providers from the registry and invokes them.

**Service interface**
: A plain Java interface, shared by both sides, that defines the remotable methods. The interface's fully-qualified name (optionally narrowed by `group` and `version`) is the service's identity.

**Registry**
: The directory where providers publish their addresses and consumers look them up — Zookeeper, Nacos, and others. Dubbo 3 defaults to **application-level service discovery** (instances register once per application, not once per interface), which scales to far larger clusters than the older interface-level model.

**Protocol**
: The wire format and transport for a remote call (`dubbo`, `tri`, `rest2`, `injvm`, …). A single provider can expose the same service over several protocols at once.

**Invoker**
: Dubbo's internal abstraction of "a callable target." Your interface proxy ultimately calls an `Invoker`; cluster, routing, and load balancing all manipulate invokers.

**URL**
: The universal configuration carrier. Every component is configured by a `URL` such as `dubbo://10.0.0.1:20880/org.apache.dubbo.demo.GreetingService?timeout=3000&version=1.0.0`. You rarely write these by hand, but you will see them in logs and the operations port.

> [!NOTE]
> A call flows: **your proxy → Cluster (fault tolerance) → Router (filter candidates) → LoadBalance (pick one) → Filter chain → Protocol → network → provider**. Each stage is a configurable, swappable extension.

---

## 5. Defining a Service Provider

There are three ways to publish a service. Pick the one that matches your stack.

### 5.1 Spring Boot — `@DubboService` (recommended)

Annotate your implementation with `@DubboService`. Component-scanning picks it up and Dubbo exports it on startup.

```java
import org.apache.dubbo.config.annotation.DubboService;
import org.apache.dubbo.demo.GreetingService;

@DubboService(version = "1.0.0", group = "demo", timeout = 3000)
public class GreetingServiceImpl implements GreetingService {
    @Override
    public String sayHello(String name) {
        return "Hello, " + name;
    }
}
```

Enable Dubbo and point component-scanning at your packages:

```java
import org.apache.dubbo.config.spring.context.annotation.EnableDubbo;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
@EnableDubbo(scanBasePackages = "org.apache.dubbo.demo.provider")
public class ProviderApplication {
    public static void main(String[] args) {
        SpringApplication.run(ProviderApplication.class, args);
    }
}
```

```yaml
# application.yml
dubbo:
  application:
    name: demo-provider
  registry:
    address: zookeeper://127.0.0.1:2181
  protocol:
    name: tri
    port: 50051
```

> [!TIP]
> `@EnableDubbo` is the umbrella annotation — it combines `@DubboComponentScan` (find `@DubboService`/`@DubboReference` beans) and `@EnableDubboConfig` (bind the `dubbo.*` properties). Use `@DubboComponentScan` alone if you only need scanning.

### 5.2 Programmatic — `ServiceConfig` + `DubboBootstrap`

When you are not on Spring, use the Bootstrap API (shown in the [Quick Start](#2-quick-start-5-minutes)). Build a `ServiceConfig`, set the interface and the implementation `ref`, and register it with `DubboBootstrap`.

```java
ServiceConfig<GreetingService> service = new ServiceConfig<>();
service.setInterface(GreetingService.class);
service.setRef(new GreetingServiceImpl());
service.setVersion("1.0.0");
service.setGroup("demo");
service.setTimeout(3000);

DubboBootstrap.getInstance()
        .application(new ApplicationConfig("demo-provider"))
        .registry(new RegistryConfig("zookeeper://127.0.0.1:2181"))
        .protocol(new ProtocolConfig("tri", 50051))
        .service(service)
        .start()
        .await();
```

### 5.3 XML

For legacy Spring XML applications:

```xml
<dubbo:application name="demo-provider"/>
<dubbo:registry address="zookeeper://127.0.0.1:2181"/>
<dubbo:protocol name="tri" port="50051"/>

<bean id="greetingService" class="org.apache.dubbo.demo.provider.GreetingServiceImpl"/>
<dubbo:service interface="org.apache.dubbo.demo.GreetingService"
               ref="greetingService" version="1.0.0" group="demo"/>
```

### 5.4 Per-service options worth knowing

Every attribute below is available on `@DubboService`, `ServiceConfig`, and `<dubbo:service>`:

| Attribute | Meaning | Default |
|---|---|---|
| `version` | Service version; consumers must match | `""` |
| `group` | Logical grouping for the same interface | `""` |
| `timeout` | Per-call timeout in ms | `-1` (inherit/1000) |
| `retries` | Retry count on failover (excludes first try) | `-1` (→ 2) |
| `loadbalance` | Default load balancer for this service | empty (→ `random`) |
| `cluster` | Fault-tolerance strategy | empty (→ `failover`) |
| `weight` | Static weight for load balancing | `-1` |
| `executes` | Max concurrent executions on the provider | `-1` (unlimited) |
| `token` | Require a token from consumers (auth) | `""` |
| `deprecated` | Mark service deprecated (warns consumers) | `false` |
| `register` | Whether to register in the registry | `true` |
| `accesslog` | Enable access logging (`true` or a file path) | `""` |
| `filter` | Extra filters in the invocation chain | `{}` |
| `serialization` | Serialization for this service | inherit |
| `mcpEnabled` | Expose as an MCP (Model Context Protocol) tool | `false` |

> [!TIP]
> Set common values once on `dubbo.provider.*` (see [§7](#7-configuration-essentials)) and override per-service only when needed. A `@DubboService(timeout = 5000)` overrides `dubbo.provider.timeout`.

---

## 6. Consuming a Service

### 6.1 Spring Boot — `@DubboReference`

Inject a remote service into any Spring bean with `@DubboReference`:

```java
import org.apache.dubbo.config.annotation.DubboReference;
import org.springframework.stereotype.Service;

@Service
public class GreetingClient {

    @DubboReference(version = "1.0.0", group = "demo", timeout = 3000, check = false)
    private GreetingService greetingService;

    public String greet(String name) {
        return greetingService.sayHello(name);
    }
}
```

> [!IMPORTANT]
> The `version` and `group` on `@DubboReference` **must match** the provider's `@DubboService`. A mismatch is the single most common cause of `No provider available for the service …` (see [Troubleshooting](#13-troubleshooting)).

### 6.2 Programmatic — `ReferenceConfig`

```java
ReferenceConfig<GreetingService> reference = new ReferenceConfig<>();
reference.setInterface(GreetingService.class);
reference.setVersion("1.0.0");
reference.setGroup("demo");
reference.setCheck(false);

GreetingService service = reference.get();   // returns the proxy
String result = service.sayHello("Dubbo");
```

> [!CAUTION]
> `ReferenceConfig` is heavyweight: it holds a live invoker and a connection. **Cache and reuse** the proxy returned by `get()`. Never create a new `ReferenceConfig` per call — that leaks connections. (The class itself is documented as expensive; for new code the Spring/Bootstrap paths manage its lifecycle for you.)

### 6.3 Asynchronous calls

Declare your interface method to return `CompletableFuture`, or use `RpcContext` to retrieve the future of a synchronous-looking method:

```java
// Interface-declared async
CompletableFuture<String> future = greetingService.sayHelloAsync("Dubbo");
future.whenComplete((reply, ex) -> System.out.println(reply));

// Context-based async on a sync signature
RpcContext.getServiceContext().asyncCall(() -> greetingService.sayHello("Dubbo"))
          .whenComplete((reply, ex) -> System.out.println(reply));
```

### 6.4 Generic calls (no interface on the classpath)

When the consumer does not have the interface class (gateways, test tools), use a generic reference:

```java
ReferenceConfig<GenericService> reference = new ReferenceConfig<>();
reference.setInterface("org.apache.dubbo.demo.GreetingService");
reference.setGeneric("true");

GenericService generic = reference.get();
Object result = generic.$invoke(
        "sayHello",
        new String[]{"java.lang.String"},
        new Object[]{"Dubbo"});
```

### 6.5 The `check` flag

By default, a consumer **checks at startup** that at least one provider exists, and fails fast if not. Set `check = false` to allow the consumer to start before its providers do (common in dev and in systems that boot in arbitrary order).

---

## 7. Configuration Essentials

Dubbo configuration is organized into **config components**, each with a `dubbo.<component>.*` property prefix. For Spring Boot you set them in `application.yml`; the same keys exist as XML elements and config-object setters.

```yaml
dubbo:
  application:
    name: order-service          # required: the application identity
    qos-enable: true             # operations port on/off
    qos-port: 22222
  registry:
    address: nacos://127.0.0.1:8848
  protocol:
    name: tri                    # tri | dubbo | rest2 | grpc
    port: 50051
  provider:
    timeout: 3000                # default timeout for all exported services
    retries: 2
    loadbalance: roundrobin
  consumer:
    check: false                 # don't fail consumer startup if provider absent
    timeout: 3000
  config-center:
    address: nacos://127.0.0.1:8848
  metadata-report:
    address: nacos://127.0.0.1:8848
```

### 7.1 The configuration components

| Prefix | Config object | Purpose |
|---|---|---|
| `dubbo.application` | `ApplicationConfig` | Application name, QOS, metadata, probes, shutdown |
| `dubbo.registry` / `dubbo.registries` | `RegistryConfig` | Where to register/discover services |
| `dubbo.protocol` / `dubbo.protocols` | `ProtocolConfig` | Wire protocol(s) and server tuning |
| `dubbo.provider` | `ProviderConfig` | Defaults applied to all exported services |
| `dubbo.consumer` | `ConsumerConfig` | Defaults applied to all references |
| `dubbo.config-center` | `ConfigCenterConfig` | External dynamic configuration source |
| `dubbo.metadata-report` | `MetadataReportConfig` | Where service metadata is published |
| `dubbo.module` | `ModuleConfig` | Module-scoped settings |
| `dubbo.metrics` | `MetricsConfig` | Metrics collection |
| `dubbo.tracing` | `TracingConfig` | Distributed tracing |
| `dubbo.ssl` | `SslConfig` | TLS certificates |

> [!NOTE]
> Use the **singular** prefix (`dubbo.registry`) for a single instance, and the **plural** prefix with an id (`dubbo.registries.zk.address: …`) when you have more than one registry or protocol. A service can then point at a specific one by id.

### 7.2 Configuration precedence

When the same setting appears in multiple places, Dubbo resolves it in this order (highest wins):

1. **JVM `-D` system properties** (e.g. `-Ddubbo.protocol.port=50052`)
2. **External config center** (Nacos/Zookeeper/Apollo dynamic config)
3. **Environment variables** (`DUBBO_PROTOCOL_PORT`)
4. **`application.yml` / `dubbo.properties`** in your application
5. **Per-service/reference annotation or XML attributes**
6. **Built-in defaults**

> [!TIP]
> A bare `dubbo.properties` file on the classpath is still honored — it's the lowest-friction way to set a handful of values without Spring. The property key is `dubbo.<component>.<field>`.

A complete field-by-field reference for every config component is in [§15](#15-configuration-reference).

---

## 8. Protocols

A protocol is the wire format plus transport. You choose it with `dubbo.protocol.name` (or per service). A provider can expose **multiple** protocols simultaneously by configuring several `dubbo.protocols.*`.

| Name | Transport | When to use |
|---|---|---|
| `dubbo` | Single long-lived TCP connection, Hessian2 by default | The classic, highest-throughput interface RPC. Default protocol. |
| `tri` (Triple) | HTTP/2 | gRPC- and cURL-compatible, mesh-friendly, streaming. **Recommended for new services.** |
| `grpc` | HTTP/2 | Native gRPC interop. |
| `rest2` | HTTP/1.1 | Expose services as REST endpoints. |
| `injvm` | In-process | Local calls within the same JVM — no network. Chosen automatically when provider and consumer share a JVM. |

> [!NOTE]
> The default protocol is **`dubbo`**, the default serialization is **`hessian2`**, the default proxy factory is **`javassist`**, and the default cluster strategy is **`failover`** — all confirmed from the framework's SPI definitions.

### 8.1 Triple, the gRPC/HTTP-friendly protocol

Triple speaks HTTP/2 and accepts plain HTTP/JSON, so you can call a Dubbo service with `curl` — no Dubbo client required:

```bash
curl -v -d '{"name":"dubbo"}' \
     -H 'Content-Type: application/json' \
     http://127.0.0.1:50051/org.apache.dubbo.demo.hello.GreeterService/sayHello
```

Server-streaming works the same way (the method returns a stream):

```bash
curl -v -d '{"name":"dubbo"}' \
     -H 'Content-Type: application/json' \
     http://127.0.0.1:50051/org.apache.dubbo.demo.hello.GreeterService/sayHelloStream
```

> [!TIP]
> If you use Protobuf IDL, the `dubbo-maven-plugin` generates the Java stubs. Add the plugin to your build and run `mvn dubbo:compile` to turn `.proto` files into Dubbo Triple stubs.

### 8.2 Exposing two protocols at once

```yaml
dubbo:
  protocols:
    tri:
      name: tri
      port: 50051
    dubbo:
      name: dubbo
      port: 20880
```

```java
@DubboService(protocol = {"tri", "dubbo"})
public class GreetingServiceImpl implements GreetingService { /* ... */ }
```

---

## 9. Registries and Service Discovery

The registry is how consumers find providers. Configure it with `dubbo.registry.address` using a `protocol://host:port` URL.

| Registry | Address scheme | Module |
|---|---|---|
| **Zookeeper** | `zookeeper://host:2181` | `dubbo-registry-zookeeper` |
| **Nacos** | `nacos://host:8848` | `dubbo-registry-nacos` |
| **Multicast** | `multicast://224.5.6.7:1234` | `dubbo-registry-multicast` (dev/testing only) |
| **Multiple** | composite of several registries | `dubbo-registry-multiple` |

```yaml
dubbo:
  registry:
    address: zookeeper://127.0.0.1:2181
    timeout: 10000          # connect timeout (ms)
    check: true             # fail startup if registry unreachable
    group: dubbo            # namespace within the registry
```

### 9.1 Application-level vs interface-level discovery

Dubbo 3 introduced **application-level service discovery**: each application instance registers **once**, and the mapping from interface to application is stored as metadata. This dramatically reduces the number of registry entries compared to Dubbo 2's interface-level model (one entry per interface per instance), which is what lets Dubbo scale to very large clusters. Dubbo 3 supports both and migrates transparently; you generally do not need to configure it.

> [!WARNING]
> A registry being briefly unreachable should **not** take your application down. By default consumers cache the last-known provider list and keep working. However, with `dubbo.registry.check: true`, **startup** will fail if the registry is down. In environments where the registry (e.g. Zookeeper) is a *weak* dependency, set `check: false` so a registry outage during boot doesn't crash the app — this is a known sharp edge (see issue #16178).

### 9.2 Multiple registries

```yaml
dubbo:
  registries:
    beijing:
      address: zookeeper://bj-zk:2181
    shanghai:
      address: nacos://sh-nacos:8848
```

A service can register to a subset: `@DubboService(registry = {"beijing"})`. With multiple subscriptions, consumers use the **`zone-aware`** cluster by default to prefer the nearest zone.

---

## 10. Traffic Management

### 10.1 Cluster fault tolerance

The **cluster** strategy decides what happens when a call fails. Set it with `cluster` on a service/reference, or `dubbo.provider.cluster` / `dubbo.consumer.cluster`.

| Strategy | Behavior | Use for |
|---|---|---|
| `failover` *(default)* | Retry on another provider (up to `retries`) | Idempotent reads |
| `failfast` | Fail immediately, no retry | Non-idempotent writes |
| `failsafe` | Swallow the exception, log it, return null | Best-effort (audit logs) |
| `failback` | Return immediately, retry failed calls in background | Notifications, eventually-consistent |
| `forking` | Call several providers in parallel, take the first success | Low-latency reads, costlier |
| `broadcast` | Call **all** providers; fail if any fails | Cache/config invalidation |
| `available` | Call the first available provider | Simple |
| `mergeable` | Call a group and merge results | Aggregation across groups |

```java
@DubboReference(cluster = "failfast", retries = 0)
private OrderWriteService orderWriteService;
```

> [!CAUTION]
> `failover` retries on failure. For **non-idempotent** operations (create order, charge card) this can cause duplicate side effects. Always use `failfast` (or `retries = 0`) for writes.

### 10.2 Load balancing

When more than one provider qualifies, the **load balancer** picks one. Set it with `loadbalance`.

| Strategy | How it picks |
|---|---|
| `random` *(default)* | Weighted random |
| `roundrobin` | Weighted round-robin |
| `leastactive` | Provider with the fewest in-flight calls |
| `shortestresponse` | Provider with the lowest recent response time |
| `consistenthash` | Same arguments → same provider (sticky) |
| `adaptive` | Adaptive, load-aware selection |

```yaml
dubbo:
  consumer:
    loadbalance: leastactive
```

### 10.3 Method-level configuration

You can tune individual methods, not just whole services:

```java
@DubboReference(
    methods = {
        @Method(name = "sayHello", timeout = 5000, retries = 0),
        @Method(name = "sayHelloStream", timeout = 30000)
    }
)
private GreetingService greetingService;
```

### 10.4 Routing and dynamic rules

Dubbo supports **condition**, **tag**, **script**, **affinity**, and **mesh** routers for canary releases, traffic isolation, and gray deployments. These rules are typically pushed through the config center at runtime rather than hard-coded. The router architecture and rule syntax are covered in depth in the [Admin Guide](apache__dubbo__AdminGuide.md).

> [!TIP]
> Tag routing is the simplest canary mechanism: tag a provider (`dubbo.provider.tag: gray`) and have a consumer request that tag at runtime via `RpcContext.getClientAttachment().setAttachment("dubbo.tag", "gray")`. Only `gray`-tagged providers will serve the call.

---

## 11. Common Tasks (Recipes)

### 11.1 Set a global default timeout and retries

```yaml
dubbo:
  provider:
    timeout: 3000
    retries: 2
  consumer:
    timeout: 3000
```

### 11.2 Require token authentication on a service

```java
@DubboService(token = "true")   // Dubbo generates a random token
public class SecureServiceImpl implements SecureService { }
```

Consumers that discover the provider through the registry receive the token automatically; direct (peer-to-peer) callers without it are rejected.

### 11.3 Enable access logging

```yaml
dubbo:
  provider:
    accesslog: /var/log/dubbo/access.log   # or `true` to log via the framework logger
```

### 11.4 Enable metrics and tracing

Add `dubbo-observability-spring-boot-starter` and a tracing starter, then:

```yaml
dubbo:
  metrics:
    protocol: prometheus
    enable-jvm-metrics: true
  tracing:
    enabled: true
    sampling:
      probability: 0.1
    tracing-exporter:
      zipkin-config:
        endpoint: http://localhost:9411/api/v2/spans
```

### 11.5 Graceful shutdown

Dubbo registers a JVM shutdown hook that drains in-flight requests before exiting. Tune the drain window:

```yaml
dubbo:
  application:
    shutwait: 10s
```

> [!NOTE]
> When Dubbo runs inside Spring, the framework coordinates shutdown with Spring's lifecycle so it doesn't tear down beans Spring still needs. You can disable Dubbo's own hook entirely with `-Ddubbo.service.shutdown.wait` set appropriately or the `IGNORE_LISTEN_SHUTDOWN_HOOK` property when an external supervisor owns shutdown.

### 11.6 Direct (peer-to-peer) invocation — skip the registry

Handy for local testing against a known provider:

```java
@DubboReference(url = "tri://127.0.0.1:50051")
private GreetingService greetingService;
```

### 11.7 Kubernetes / service mesh

Set `dubbo.consumer.mesh-enable: true` (or `meshEnable` on a reference). In mesh mode, Triple is required and Dubbo resolves the provider through the Kubernetes service DNS (`<service>.<namespace>.svc.<cluster-domain>`), letting Envoy/Istio handle routing. The cluster domain defaults to `cluster.local` and the mesh port to `80`.

---

## 12. Operating a Running Application (QOS)

Dubbo ships an **operations port** (the "QOS" port) that exposes runtime commands over telnet **and** HTTP. Enable it via `dubbo.application.qos-enable: true` (default port `22222`).

```yaml
dubbo:
  application:
    qos-enable: true
    qos-port: 22222
    qos-accept-foreign-ip: false   # only localhost by default — keep it that way in prod
```

Connect with telnet and run `help` to list commands:

```bash
telnet localhost 22222
```

Or call a command over HTTP:

```bash
curl http://localhost:22222/ls
```

### 12.1 The command set

| Command | What it does |
|---|---|
| `ls` | List exported (provider) and subscribed (consumer) services and their status |
| `ps` | Show ports and their bound services |
| `online` / `offline` | Register / deregister all services (graceful in/out of rotation) |
| `onlineApp` / `offlineApp` | Online/offline by application name |
| `onlineInterface` / `offlineInterface` | Online/offline a single interface |
| `getConfig` | Dump effective configuration |
| `getAddress` | Show the resolved address list for a service |
| `invoke` | Invoke a service method from the console |
| `select` | Choose among candidate providers for a manual `invoke` |
| `count` | Count invocations over an interval |
| `version` | Print the Dubbo version |
| `live` / `ready` / `startup` | Liveness / readiness / startup probes (Kubernetes-friendly) |
| `shutdown` / `gracefulShutdown` | Shut the application down (gracefully drains first) |
| `publishMetadata` | Re-publish service metadata to the metadata center |
| `getOpenAPI` | Export the OpenAPI document for REST/Triple services |
| `loggerInfo` / `switchLogger` / `switchLogLevel` | Inspect and change logging at runtime |
| `enableSimpleProfiler` / `enableDetailProfiler` / `disableSimpleProfiler` / `disableDetailProfiler` | Toggle the invocation profiler |
| `setProfilerWarnPercent` | Tune the profiler's slow-call warning threshold |
| `enableRouterSnapshot` / `disableRouterSnapshot` / `getRouterSnapshot` / `getEnabledRouterSnapshot` / `getRecentRouterSnapshot` | Capture routing decisions for debugging |
| `serializeCheckStatus` / `serializeWarnedClasses` | Inspect serialization allow-list status |
| `quit` | Close the telnet session |

> [!CAUTION]
> The QOS port can change your application's runtime behavior (`offline`, `shutdown`, `switchLogLevel`). **Never expose it to untrusted networks.** Keep `qos-accept-foreign-ip: false` (localhost only) in production and restrict it further with `qos-anonymous-access-permission-level` and `qos-anonymous-allow-commands` if you must open it. See the [Admin Guide](apache__dubbo__AdminGuide.md) security section.

### 12.2 Kubernetes probes

Wire the QOS HTTP probes into your pod spec:

```yaml
livenessProbe:
  httpGet: { path: /live, port: 22222 }
readinessProbe:
  httpGet: { path: /ready, port: 22222 }
startupProbe:
  httpGet: { path: /startup, port: 22222 }
```

---

## 13. Troubleshooting

### Error: `No provider available for the service <interface>`

**Cause:** No provider is registered for that exact interface + `group` + `version`, or the consumer started before any provider registered, or the registry is unreachable.

**Solution:**
1. Confirm the provider is up and registered (`telnet localhost 22222` → `ls` on the provider).
2. Verify `group` and `version` match **exactly** between `@DubboService` and `@DubboReference`.
3. Check the registry address is identical on both sides.
4. For boot-order issues, set `check = false` on the reference so it resolves lazily.

**Prevention:** Keep `group`/`version` in shared constants; use `check: false` in environments where boot order is not guaranteed.

### Error: `Failed to check the status of the service … No provider available … from the url …`

**Cause:** The consumer's startup availability check timed out — same root causes as above, surfaced at `ReferenceConfig.get()`.

**Solution:** Same as above. To stop startup from blocking on this check, set `check = false`.

### Error: `In mesh mode, a triple protocol must be specified`

**Cause:** `mesh-enable` is on but no Triple protocol is configured.

**Solution:** Configure `dubbo.protocol.name: tri` (mesh mode requires HTTP/2 Triple) and ensure `providedBy` is set.

### Error: `dubbo.consumer.timeout setting too long leads to OOM`

**Cause:** A very large consumer-side timeout lets requests pile up faster than they complete, exhausting heap (reported in issue #16294).

**Solution:** Keep timeouts realistic (seconds, not minutes). Combine with `actives`/`executes` limits to bound concurrency, and use `failfast` for writes so failed calls don't accumulate.

### Error: `NoClassDefFoundError: com/google/gson/JsonSyntaxException` on JDK 25

**Cause:** A known JDK 25 + ServiceLoader interaction where an optional Gson dependency isn't on the classpath (issues #16295, #16296).

**Solution:** Use a Dubbo `3.3.x` build with JDK 25 support and ensure Gson is present if you use components that need it; otherwise pin to JDK 21 for now.

### Hessian2 deserialization sets fields to default/null

**Cause:** Hessian2 may not run field-initialization code during deserialization, assigning defaults instead (issue #15805). This is a serialization-format behavior, not a Dubbo bug.

**Solution:** Don't rely on field initializers running on the deserialized object; set values explicitly, or switch the affected service to `serialization: fastjson2`.

### Provider start fails when Zookeeper is down

**Cause:** Zookeeper treated as a hard dependency at startup (issue #16178).

**Solution:** Set `dubbo.registry.check: false` so a weakly-dependent registry being down doesn't block startup; the app will register once the registry recovers.

### `NetUtils.matchIpExpression(...) fails to parse a normal IPv4 address with port`

**Cause:** Edge case in IP-expression matching with a port suffix (issue #16308).

**Solution:** Specify host and port separately in config rather than embedding a port in an IP-match expression; upgrade to a release where it's fixed.

> [!TIP]
> Most "it won't connect" problems are a `group`/`version` mismatch or a registry-address typo. Use the QOS `ls` and `getAddress` commands on both sides before digging deeper — they show exactly what each side thinks it is exporting and subscribing to.

---

## 14. Frequently Asked Questions

**Q: Do I need Spring to use Dubbo?**
No. The programmatic `DubboBootstrap` API works without Spring. Spring Boot is the most popular path, but it's optional.

**Q: Which protocol should I choose, `dubbo` or `tri`?**
For brand-new services, `tri` (Triple) — it's HTTP/2, gRPC- and cURL-compatible, and mesh-friendly. For maximum-throughput interface RPC between Java services, the classic `dubbo` TCP protocol is excellent and remains the default.

**Q: Can one provider expose several protocols?**
Yes. Configure multiple `dubbo.protocols.*` and list them on the service (`protocol = {"tri","dubbo"}`).

**Q: How do consumers find providers?**
Through the registry (Zookeeper, Nacos, …). Providers register their address; consumers subscribe and receive live updates. For testing you can bypass the registry with a direct `url`.

**Q: What's the difference between `group` and `version`?**
`version` is for evolving the same logical service over time (e.g. `1.0.0` vs `2.0.0`); `group` is for parallel variants of the same interface (e.g. tenant A vs B, or read vs write groups). Both participate in service identity, so consumer and provider must agree.

**Q: Is the registry a single point of failure?**
No. Consumers cache the provider list and keep calling even if the registry goes down — as long as you don't force `check: true` at startup. Treat the registry as needed for *discovery changes*, not for every call.

**Q: How do I call a Dubbo service without the Java interface?**
Use a generic reference (`setGeneric("true")` + `$invoke`), or expose the service over Triple/REST and call it with plain HTTP/JSON.

**Q: What serialization does Dubbo use?**
`hessian2` by default. `fastjson2` is also bundled. Serialization is an SPI, so you can add others. Dubbo enforces a serialization allow-list for security (see Admin Guide).

**Q: How do I take an instance out of rotation for a deploy?**
Use the QOS `offline` command (or `offlineApp`) to deregister gracefully, deploy, then `online` again. Combine with `gracefulShutdown` to drain in-flight calls.

**Q: Does Dubbo work on the latest JDK?**
Dubbo `3.3.x` supports JDK 8 through 25. Some very new JDK versions have known edge cases (see Troubleshooting); track the version-compatibility table in the README.

**Q: Can I call a Dubbo (Java) service from Go/Python/Node?**
Yes — use the Triple protocol (gRPC-compatible) and the matching Dubbo SDK for that language, or plain gRPC/HTTP clients.

---

## 15. Configuration Reference

Every field below is a real property on its config object, exposed as `dubbo.<prefix>.<field>` (kebab-case in YAML), as an attribute on the corresponding annotation/XML element, and as a setter on the config class.

### 15.1 `dubbo.application` (`ApplicationConfig`)

| Field | Purpose |
|---|---|
| `name` | **Required.** Application identity (used for app-level discovery) |
| `version` | Application version |
| `owner`, `organization`, `architecture`, `environment` | Descriptive metadata |
| `logger` | Logging framework adapter (`slf4j`, `log4j2`, `jcl`, …) |
| `compiler` | Bytecode compiler (`javassist`, `jdk`) |
| `registry-ids` | Which registries this app uses |
| `qos-enable` | Turn the operations port on/off |
| `qos-port` | Operations port (default `22222`) |
| `qos-host` | Bind host for the operations port |
| `qos-accept-foreign-ip` | Allow non-localhost connections to QOS (keep `false` in prod) |
| `qos-accept-foreign-ip-whitelist` | Allowed remote IPs for QOS |
| `qos-anonymous-access-permission-level` | Permission level for unauthenticated QOS access |
| `qos-anonymous-allow-commands` | Commands permitted anonymously |
| `qos-check` | Validate QOS config |
| `metadata-type` | `local` or `remote` metadata storage |
| `metadata-service-protocol`, `metadata-service-port` | Metadata service exposure |
| `shutwait` | Graceful-shutdown drain window |
| `register-mode` | `interface`, `instance`, or `all` (discovery model) |
| `register-consumer` | Whether consumers register themselves |
| `enable-empty-protection` | Ignore empty provider pushes (avoid wiping the address list) |
| `serialize-check-status`, `auto-trust-serialize-class`, `trust-serialize-class-level`, `check-serializable` | Serialization security controls |
| `executor-management-mode` | Thread-pool management (`default`, `isolation`) |
| `liveness-probe`, `readiness-probe`, `startup-probe` | Custom probe implementations |
| `dump-enable`, `dump-directory` | Thread/heap dump on overload |
| `parameters` | Arbitrary extra parameters |

### 15.2 `dubbo.registry` (`RegistryConfig`)

| Field | Purpose |
|---|---|
| `address` | **Required.** `protocol://host:port` registry URL |
| `protocol` | Registry protocol (`zookeeper`, `nacos`, `multicast`, …) |
| `username`, `password` | Registry credentials |
| `port` | Registry port (if not in `address`) |
| `group`, `version` | Namespace within the registry |
| `timeout` | Connect timeout (ms) |
| `session` | Session timeout (ms) |
| `file` | Local cache file for the provider list |
| `check` | Fail startup if the registry is unreachable |
| `register`, `subscribe` | Whether to register / subscribe |
| `dynamic` | Auto-deregister on shutdown |
| `simplified` | Register a reduced URL (less registry data) |
| `use-as-config-center`, `use-as-metadata-center` | Reuse this registry for config / metadata |
| `zone`, `weight`, `preferred` | Multi-registry zone routing |
| `register-mode` | Discovery model override |
| `enable-empty-protection` | Empty-push protection |
| `secure` | TLS to the registry |
| `parameters` | Extra parameters |

### 15.3 `dubbo.protocol` (`ProtocolConfig`)

| Field | Purpose |
|---|---|
| `name` | Protocol id (`dubbo`, `tri`, `rest2`, `grpc`, `injvm`) |
| `host` | Bind/advertise host |
| `port` | Listen port |
| `contextpath` | Context path (HTTP-based protocols) |
| `serialization` | Serializer (`hessian2`, `fastjson2`) |
| `prefer-serialization` | Preferred serializer negotiation |
| `threadpool` | Thread-pool type (`fixed`, `cached`, `limited`, `eager`) |
| `corethreads`, `threads`, `iothreads` | Thread-pool sizing |
| `alive`, `queues` | Pool keep-alive and queue depth |
| `accepts` | Max accepted connections |
| `payload` | Max request/response size (bytes) |
| `buffer` | Network buffer size |
| `heartbeat` | Heartbeat interval (ms) |
| `accesslog` | Access logging |
| `transporter`, `exchanger`, `dispatcher`, `networker` | Remoting layer plug points |
| `server`, `client` | Server/client transport impls |
| `codec` | Codec |
| `telnet`, `prompt` | Telnet handler and prompt |
| `status` | Status checkers |
| `keep-alive` | Keep connections alive |
| `ssl-enabled` | TLS for this protocol |
| `register` | Register this protocol's services |
| `ext-protocol`, `preferred-protocol` | Extended/preferred protocol negotiation |
| `json-check-level` | JSON validation strictness |

### 15.4 `dubbo.provider` (`ProviderConfig`)

Provider-side defaults inherited by every exported service. In addition to the service attributes in [§5.4](#54-per-service-options-worth-knowing): `host`, `port`, `contextpath`, `threadpool`, `threadname`, `threads`, `iothreads`, `alive`, `queues`, `accepts`, `codec`, `charset`, `payload`, `buffer`, `transporter`, `exchanger`, `dispatcher`, `networker`, `server`, `client`, `telnet`, `prompt`, `status`, `wait`, `export-thread-num`, `export-background`.

### 15.5 `dubbo.consumer` (`ConsumerConfig`)

Consumer-side defaults inherited by every reference (plus the reference attributes from [§5.4](#54-per-service-options-worth-knowing)): `threadpool`, `corethreads`, `threads`, `queues`, `shareconnections` (connections shared across references), `url-merge-processor`, `refer-thread-num`, `refer-background`, `mesh-enable`.

### 15.6 Other components

- **`dubbo.config-center` (`ConfigCenterConfig`)** — external dynamic configuration (`address`, `protocol`, `group`, `namespace`, `timeout`, `check`, `highest-priority`).
- **`dubbo.metadata-report` (`MetadataReportConfig`)** — where service metadata is published (`address`, `username`, `password`, `timeout`, `group`).
- **`dubbo.metrics` (`MetricsConfig`)** — metrics protocol and JVM/aggregation toggles.
- **`dubbo.tracing` (`TracingConfig`)** — `enabled`, sampling probability, baggage, and exporter (Zipkin/OTLP) endpoints.
- **`dubbo.ssl` (`SslConfig`)** — certificate/key/trust-cert paths for TLS.

> [!NOTE]
> Spring Boot generates IDE auto-completion metadata for these properties from the Javadoc on the config classes, so your editor will suggest and document them as you type. The authoritative source for any field's exact semantics is always the config class in `dubbo-common`.

---

_Generated June 2026 from apache/dubbo at ee9c190. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._