# Ray Admin & Architecture Guide

This guide is for the engineers who **operate, deploy, secure, and extend** Ray:
platform/ML-infra teams, SREs, and architects. It documents Ray's internal
architecture, the full configuration and environment-variable surface, ports and
networking, deployment topologies (bare-metal, VM autoscaler, and Kubernetes),
security hardening, day-2 operations, observability, and the repository layout
needed to navigate or contribute to the codebase.

For end-user API and CLI usage, see the companion **User Guide**. This guide
assumes you already understand Ray's three core abstractions (Tasks, Actors,
Objects).

---

## 1. Executive Summary

Ray is a distributed execution framework: a **C++ core runtime** wrapped by
Python (primary), Java, and C++ client APIs, plus five higher-level AI libraries
(Data, Train, Tune, Serve, RLlib) and an LLM stack — all of which compile down to
the same Tasks/Actors/Objects primitives.

| Aspect | Summary |
|--------|---------|
| **Core runtime** | C++ (`src/ray`): GCS, raylet, core worker, object manager (Plasma) |
| **Client APIs** | Python (`python/ray`), Java (`java/`), C++ (`cpp/`) |
| **State backend** | GCS (Global Control Store) on the head node; no external DB required by default |
| **Object transport** | Shared-memory object store (Plasma) per node + spilling |
| **Scheduling** | Distributed; each raylet schedules locally with resource accounting |
| **Cluster mgmt** | Built-in autoscaler; cluster launcher for clouds; KubeRay operator for Kubernetes |
| **Observability** | Dashboard (port 8265), state API/CLI, Prometheus metrics, OpenTelemetry tracing |
| **Trust model** | Assumes a **trusted network**; not multi-tenant-secure by default |

> [!IMPORTANT]
> Ray's default security posture is "trusted network." There is no
> authentication on the core RPC ports unless you explicitly enable it. Treat a
> Ray cluster like a distributed shell: anyone who can reach its ports can
> execute arbitrary code. Section 7 covers hardening.

---

## 2. Architecture

### 2.1 Process topology

```
                     HEAD NODE
   +-------------------------------------------------+
   |  GCS (Global Control Store)   :6379             |
   |    - cluster metadata, actor registry,          |
   |      placement groups, KV store, pub/sub        |
   |  Autoscaler / Monitor                           |
   |  Dashboard (API server)       :8265             |
   |  Raylet (scheduler + object manager)            |
   |  Object Store (Plasma, shared memory)           |
   |  Dashboard agent / runtime-env agent  :52365    |
   |  Driver + Worker processes                      |
   +-------------------------------------------------+
        ^                         ^
        | gRPC                    | gRPC / object transfer
        v                         v
   +------------------+     +------------------+
   |   WORKER NODE    | ... |   WORKER NODE    |
   |  Raylet          |     |  Raylet          |
   |  Object Store    |     |  Object Store    |
   |  Workers         |     |  Workers         |
   |  Dashboard agent |     |  Dashboard agent |
   +------------------+     +------------------+
```

### 2.2 Components

| Component | Role |
|-----------|------|
| **GCS** (Global Control Store) | Cluster-wide control plane on the head node: node registry, actor registry, placement groups, resource view, an internal key-value store, and pub/sub. Source: `src/ray/gcs`. |
| **Raylet** | Per-node agent. Contains the **local scheduler** (resource accounting, task dispatch) and the **object manager** (locating and transferring objects between nodes). Source: `src/ray/raylet`, `src/ray/object_manager`. |
| **Core worker** | The runtime embedded in every driver and worker process. Submits tasks, manages object references and ownership, and executes actor methods. Source: `src/ray/core_worker`. |
| **Object store (Plasma)** | Per-node shared-memory store for immutable objects, with **spilling** to disk when full. Part of the object manager. |
| **Driver** | Your program — the process that calls `ray.init()` and submits work. |
| **Worker** | A process that executes tasks and hosts actors; pooled and reused. |
| **Autoscaler / Monitor** | Runs on the head node, watches resource demand, and adds/removes worker nodes (v1 and v2 implementations). Source: `python/ray/autoscaler`. |
| **Dashboard** | FastAPI/aiohttp API server + React frontend on the head node (port 8265). Source: `python/ray/dashboard`. |
| **Dashboard agent / runtime-env agent** | Per-node HTTP agents that serve metrics, logs, and set up runtime environments (listen port 52365). |

### 2.3 Object ownership and references

Ray uses a **distributed ownership** model: the worker that creates an
`ObjectRef` *owns* it and tracks its reference count and lineage. This enables
fine-grained garbage collection and **lineage-based reconstruction** — if a
worker dies, Ray can re-execute the task that produced a lost object (when
reconstruction is enabled). This is why holding many `ObjectRef`s pins memory:
the owner cannot collect an object while live references exist.

### 2.4 Scheduling

Each raylet schedules tasks locally against its known resource totals
(CPU, GPU, `memory`, `object_store_memory`, and custom resources). Resources are
**logical reservations**, not OS-enforced limits — a task that requests
`num_cpus=2` is accounted as using two cores but is not cgroup-limited unless
**resource isolation** is enabled (see Section 4.5). Scheduling strategies
include `DEFAULT` (pack), `SPREAD`, node affinity, and **placement groups** for
gang scheduling.

### 2.5 Language workers

Ray is polyglot. The Python frontend (`python/ray`) is primary; `java/` provides
the Java API and runtime (Maven, `io.ray`), and `cpp/` provides the C++ API. All
three speak to the same C++ core over gRPC and share the Protobuf message
definitions in `src/ray/protobuf`.

---

## 3. Requirements

| Resource | Recommendation |
|----------|----------------|
| **OS** | Linux (production), macOS (dev). Windows supported with limits. |
| **Python** | A supported CPython (3.9+). All nodes in a cluster must run the **same** Python and Ray version. |
| **CPU/RAM** | Head node carries GCS + dashboard + autoscaler — give it headroom; don't schedule heavy work on it in large clusters. |
| **Shared memory** | The object store is backed by `/dev/shm`. Size it appropriately (see Section 4.4); under-provisioned `/dev/shm` forces spilling. |
| **GPUs** | Optional; auto-detected. Drivers/CUDA must be present on GPU nodes. |
| **Network** | Low-latency, high-bandwidth intra-cluster network. Many ports must be open **between** nodes (Section 5). |

> [!WARNING]
> Version skew is a frequent cause of mysterious failures. The Ray version,
> Python version, and key serialization libraries must match across the head and
> all workers. Pin them in your container image or `runtime_env`.

---

## 4. Configuration

Ray is configured through three channels, in increasing specificity:

1. **`ray start` flags** / `ray.init()` arguments — cluster and node setup.
2. **Environment variables** (`RAY_*`) — runtime behavior tuning.
3. **Per-task/actor options** and **`runtime_env`** — workload-level.

### 4.1 Head-node startup reference

The most operationally significant `ray start --head` flags:

| Flag | Default | Purpose |
|------|---------|---------|
| `--port` | `6379` | GCS server port (the cluster's bootstrap address). `0` = auto-pick. |
| `--node-ip-address` | autodetect | Address other nodes use to reach this one |
| `--num-cpus` / `--num-gpus` | autodetect | Override advertised resources |
| `--resources` | `{}` | JSON map of custom resources |
| `--object-store-memory` | ~30% RAM | Object-store size in bytes (capped by shm and 200 GB) |
| `--memory` | available | Heap memory advertised to workers |
| `--min-worker-port` / `--max-worker-port` | `10002` / `19999` | Worker port range |
| `--ray-client-server-port` | `10001` | Ray Client (`ray://`) endpoint |
| `--include-dashboard` / `--dashboard-host` / `--dashboard-port` | on / `localhost` / `8265` | Dashboard controls |
| `--metrics-export-port` | auto | Prometheus scrape port |
| `--temp-dir` | `/tmp/ray` | Root for logs, sockets, spilled objects |
| `--plasma-directory` | `/dev/shm` | Backing store for the object store |
| `--object-spilling-directory` | temp dir | Where objects spill when the store is full |
| `--system-config` | — | JSON overrides for internal `_system_config` knobs |
| `--block` | off | Stay in foreground (for systemd/containers) |
| `--autoscaling-config` | — | Path to an autoscaler config to run the monitor |
| `--enable-resource-isolation` | off | cgroup v2 reservation for system processes (Section 4.5) |

### 4.2 Runtime environment variables

Ray reads many `RAY_*` environment variables at process start. The
operationally important runtime ones:

| Variable | Effect |
|----------|--------|
| `RAY_ADDRESS` | Default cluster/job address for `ray.init()` and the jobs CLI |
| `RAY_REDIS_ADDRESS` | Use an **external** Redis as the GCS backing store (HA) |
| `RAY_GCS_SERVER_PORT` | GCS port (alternative to `--port`) |
| `RAY_enable_autoscaler_v2` | Opt into the v2 autoscaler |
| `RAY_TASK_MAX_RETRIES` | Default task retry count |
| `RAY_STORAGE` | URI for cluster-wide persistent storage |
| `RAY_DEBUG` | `legacy` re-enables the old `ray debug` PDB flow |
| `RAY_START_HOOK` / `RAY_RUNTIME_ENV_HOOK` | Pluggable hooks for startup / runtime-env resolution |
| `RAY_DASHBOARD_MAX_EVENTS_TO_CACHE` | Dashboard event cache size (default 10000) |
| `RAY_ROTATION_MAX_BYTES` | Worker log rotation size |
| `RAY_TQDM` | Toggle distributed progress bars |

> [!NOTE]
> Each library exposes its own large family of tuning variables: `RAY_SERVE_*`
> (proxy/router/replica behavior, gRPC, HA proxy, autoscaling windows),
> `RAY_DATA_*` (shuffle strategy, exception propagation), and the
> `RAY_TLS_*` / `RAY_AUTH_*` security variables (Section 7). Treat these as
> advanced knobs — change them deliberately and document the reason.

### 4.3 Logs and the session directory

By default Ray writes everything under `--temp-dir` (`/tmp/ray`), with a
per-session subdirectory (`/tmp/ray/session_latest`). Inside it: `logs/` (driver,
worker, raylet, gcs_server, dashboard, autoscaler logs), Unix domain sockets, and
spilled objects. Point `--temp-dir` at fast local storage, and ship `logs/` to
your aggregator. Worker logs rotate at `RAY_ROTATION_MAX_BYTES`.

### 4.4 Object store and spilling

The object store lives in shared memory (`--plasma-directory`, default
`/dev/shm`). When it fills with objects that are still referenced, Ray **spills**
the least-recently-used objects to `--object-spilling-directory` and restores
them on demand. Symptoms of under-sizing: high spill I/O, `ObjectStoreFullError`,
and stalled tasks. Tune `--object-store-memory`, ensure `/dev/shm` is large
enough (containers often default to 64 MB — raise it), and spill to fast disk.

### 4.5 Resource isolation (cgroup v2)

`--enable-resource-isolation` reserves CPU and memory for Ray's **system**
processes using cgroup v2, so a runaway workload can't starve the raylet/GCS.

| Flag | Purpose |
|------|---------|
| `--system-reserved-cpu` | Cores reserved for system processes (default `min(3, max(1, 0.05·cores))`) |
| `--system-reserved-memory` | Bytes reserved (default `min(10 GB, max(500 MB, 0.10·RAM))`) |
| `--cgroup-path` | cgroup the raylet manages (default `/sys/fs/cgroup`) |

Requires cgroup v2 with the memory and cpu controllers enabled and read/write
permission for the raylet.

---

## 5. Networking & Ports

| Port (default) | Component | Scope |
|----------------|-----------|-------|
| `6379` | GCS server (bootstrap address) | Head; reachable by all nodes |
| `8265` | Dashboard / Jobs API | Head; **keep private** |
| `10001` | Ray Client server (`ray://`) | Head; clients |
| `52365` | Dashboard agent (HTTP) | Every node |
| `10002–19999` | Worker ports | Every node, node-to-node |
| object manager / node manager | object transfer, raylet RPC | Node-to-node |
| `--metrics-export-port` | Prometheus metrics | Scraper |

> [!CAUTION]
> A Ray cluster requires **wide-open node-to-node connectivity** on the worker
> and manager port ranges. Put the entire cluster inside one security group /
> private subnet and expose **nothing** to the public internet. The jobs/dash
> port (8265) and GCS port (6379) are the highest-value targets — never bind them
> to `0.0.0.0` on a public interface.

---

## 6. Deployment

### 6.1 Manual / bare-metal

```bash
# Head:
ray start --head --port=6379 --dashboard-host=127.0.0.1

# Each worker:
ray start --address='<head-ip>:6379'
```

Run under a process supervisor (systemd, supervisord) using `--block` so the
process stays in the foreground and restarts on failure.

```ini
# /etc/systemd/system/ray-head.service (sketch)
[Service]
ExecStart=/usr/bin/ray start --head --block --port=6379 --dashboard-host=127.0.0.1
Restart=always
```

### 6.2 VM clusters via the cluster launcher

Define a cluster YAML and let Ray provision and autoscale cloud VMs (AWS, GCP,
Azure, vSphere, on-prem). Providers live in `python/ray/autoscaler/<provider>`.

```bash
ray up cluster.yaml --yes        # provision + start
ray monitor cluster.yaml         # tail autoscaler logs
ray submit cluster.yaml job.py   # run a script
ray down cluster.yaml --yes      # tear down
```

The YAML declares the head/worker node types, `min_workers`/`max_workers`, the
container/AMI image, and setup commands. The autoscaler scales worker nodes
between the min and max based on pending resource demand.

> [!TIP]
> `ray up` is idempotent — re-running it applies config changes. Use
> `--no-restart` to update worker config without bouncing running Ray processes,
> and `--restart-only` to restart without re-running setup commands.

### 6.3 Kubernetes (KubeRay)

The supported way to run Ray on Kubernetes is the **KubeRay** operator, which
provides three CRDs:

- **RayCluster** — a Ray cluster (head + worker pod groups) with autoscaling.
- **RayJob** — run a job to completion against an ephemeral or existing cluster.
- **RayService** — a highly-available Ray Serve deployment with zero-downtime
  upgrades.

In-tree support lives in `python/ray/autoscaler/kuberay` and
`python/ray/autoscaler/_private/kuberay`; the head node runs the
`ray kuberay-autoscaler` entrypoint. The KubeRay operator itself is a separate
project that you install into the cluster.

> [!IMPORTANT]
> On Kubernetes, give the head pod its own node or generous requests/limits, set
> `/dev/shm` large enough for the object store (the default `emptyDir` memory
> medium is small), and use RayService (not a bare Deployment) for serving so you
> get health checks and rolling upgrades.

---

## 7. Security

Ray is **not** secure by default and is **not** designed for untrusted
multi-tenancy. Harden it with network controls plus the optional auth/TLS
features.

### 7.1 The trust boundary

- Anyone who can reach the GCS port, a raylet, or the dashboard/jobs API can run
  arbitrary code on the cluster.
- The dashboard's Jobs API accepts code submissions; the state API exposes logs.
- **Primary control: network isolation.** Keep every Ray port on a private
  network; expose the dashboard only via SSH port-forwarding or an
  authenticating reverse proxy.

### 7.2 Token authentication

Ray supports an authentication token on its RPC layer:

| Variable / command | Purpose |
|--------------------|---------|
| `RAY_AUTH_MODE` | Enable/select the authentication mode |
| `RAY_AUTH_TOKEN` / `RAY_AUTH_TOKEN_PATH` | The shared token (inline or from a file) |
| `RAY_AUTHORIZATION_HEADER_NAME` | Header carrying the token |
| `ray get-auth-token [--generate]` | Print (or generate) the cluster auth token |

Distribute the same token to every node and to clients that connect.

### 7.3 TLS for gRPC

Encrypt intra-cluster gRPC traffic:

| Variable | Purpose |
|----------|---------|
| `RAY_USE_TLS` | Enable TLS on Ray gRPC channels |
| `RAY_TLS_SERVER_CERT` / `RAY_TLS_SERVER_KEY` | Server certificate and key |
| `RAY_TLS_CA_CERT` | CA certificate for peer verification |

Provision the same CA across all nodes and mount per-node certs.

### 7.4 Redis hardening

When using an external Redis as the GCS backend (`RAY_REDIS_ADDRESS`), secure it
with `--redis-username` / `--redis-password` (or the corresponding env vars) and
network ACLs. Never run an internet-reachable unauthenticated Redis.

### 7.5 Usage telemetry

Ray collects anonymous usage statistics by default. Disable it for sensitive
environments with `ray disable-usage-stats`, `ray start --disable-usage-stats`,
or the usage-stats environment variable.

> [!CAUTION]
> Do **not** rely on the dashboard's `--dashboard-host` binding alone for
> security. Combine network isolation, token auth, and TLS. A reachable,
> unauthenticated dashboard is a remote-code-execution vector.

---

## 8. Operations

### 8.1 Health and status

```bash
ray status                       # autoscaler + resource view
ray status --verbose
ray list nodes                   # node health
ray list nodes --filter "state=ALIVE"
ray health-check --component gcs # hidden, NOT a public API — for liveness probes
```

### 8.2 Inspecting workloads

```bash
ray list tasks --filter "state=FAILED" --detail
ray list actors --filter "state=ALIVE"
ray summary tasks
ray memory                       # object references and store usage
ray logs job --id <SUBMISSION_ID>
```

### 8.3 Draining and scaling down

```bash
ray drain-node --node-id <ID> --reason <REASON> --reason-message "<msg>"
```

`ray drain-node` gracefully removes a worker node (hidden command used by the
autoscaler and KubeRay). For autoscaler-managed clusters, prefer adjusting
`min_workers`/`max_workers` and letting the autoscaler converge.

### 8.4 Diagnostics collection

```bash
ray cluster-dump                 # collect logs/state from nodes into an archive
ray local-dump                   # this node only (hidden)
ray timeline                     # Chrome tracing timeline
ray stack                        # py-spy stacks of local workers
ray global-gc                    # force GC across all workers (hidden)
```

`ray cluster-dump` is the first thing to run when filing a bug or analyzing an
incident — it bundles logs and state from one or more nodes.

### 8.5 Backup and persistence

GCS state is in-memory by default and is **lost** when the head node dies. For
fault tolerance:

- Back the GCS with **external Redis** (`RAY_REDIS_ADDRESS`) so head-node restarts
  recover cluster state (GCS fault tolerance).
- Use `RAY_STORAGE` for a cluster-wide persistent storage URI (checkpoints,
  Train/Tune artifacts) on durable object storage (S3/GCS/NFS).

> [!WARNING]
> Without external-Redis GCS fault tolerance, a head-node failure tears down the
> whole cluster — all actors and in-flight work are lost. Treat the head node as
> a single point of failure and plan accordingly (Redis HA + RayService on K8s).

---

## 9. Observability & Metrics

| Channel | What you get | How |
|---------|--------------|-----|
| **Dashboard** | Cluster, jobs, actors, tasks, logs, per-library views | Port 8265 |
| **State API/CLI** | Structured task/actor/node/object/job state | `ray list/get/summary`, `ray.util.state` Python API |
| **Prometheus metrics** | Resource utilization, task/actor counts, Serve latencies | `--metrics-export-port`, scrape each node |
| **Grafana** | Prebuilt dashboards | Point Grafana at Prometheus; dashboards ship in `python/ray/dashboard/modules/metrics/dashboards` |
| **OpenTelemetry tracing** | Distributed traces | `--tracing-startup-hook`; OTel SDK is a base dependency |
| **Structured logging** | JSON logs per job | `LoggingConfig` on `ray.init()` |

> [!NOTE]
> Ray exports metrics **per node** via the dashboard agent. Configure Prometheus
> to discover all node agents (not just the head). A known issue (#63219) ships a
> Serve LLM Grafana dashboard with malformed PromQL — validate panel queries
> after import.

---

## 10. Troubleshooting (operational)

### Symptom: head node restart loses the entire cluster

**Cause:** GCS state is in-memory.
**Fix:** Enable GCS fault tolerance with external Redis (`RAY_REDIS_ADDRESS`);
on K8s use RayService + a managed Redis.

### Symptom: workers can't join the head

**Cause:** Blocked ports or wrong advertised IP.
**Fix:** Open the worker/manager port ranges node-to-node; set
`--node-ip-address` to the address peers can actually reach; confirm the
`--address='<head>:6379'` value.

### Symptom: heavy object spilling / `ObjectStoreFullError`

**Cause:** Object store too small or `/dev/shm` undersized (common in containers).
**Fix:** Increase `--object-store-memory`, enlarge `/dev/shm`, spill to fast
disk via `--object-spilling-directory`, and stop pinning large objects.

### Symptom: GCS memory grows unbounded over a long job

**Cause:** The GCS KV store has no eviction for pickled functions during a job's
lifetime (community issue #63666).
**Fix:** Avoid re-registering many distinct remote functions in long-lived jobs;
restart long-running drivers periodically; track the upstream issue.

### Symptom: Serve controller RSS grows after autoscaling cycles

**Cause:** Known controller memory growth (#58815).
**Fix:** Reduce autoscaling churn, restart the controller as a stopgap, and
upgrade once a fix lands.

### Symptom: autoscaler not adding nodes

**Cause:** No pending demand visible, provider quota/permissions, or `max_workers`
reached.
**Fix:** Check `ray monitor <config>` and `ray status`; verify cloud
credentials/quotas and the node-type resource definitions.

---

## 11. Repository Layout (for contributors)

| Path | Contents |
|------|----------|
| `src/ray/` | **C++ core.** `gcs/` (control store), `raylet/` (scheduler + node manager), `core_worker/` (per-process runtime), `object_manager/` (Plasma + transfer), `rpc/`, `pubsub/`, `ray_syncer/`, `protobuf/` (message schemas), `common/`, `util/`. |
| `python/ray/` | **Python frontend + libraries.** Core API (`_private/worker.py`, `actor.py`, `remote_function.py`), `scripts/scripts.py` (the `ray` CLI), `serve/`, `data/`, `train/`, `tune/`, `air/`, `llm/`, `dashboard/`, `autoscaler/`, `job_submission/`, `runtime_env/`, `util/state/`. |
| `rllib/` | RLlib — algorithms, connectors, models, policies, offline RL, evaluation. |
| `java/` | Java API and runtime (`io.ray`, Maven multi-module: api/runtime/serve/test). |
| `cpp/` | C++ application API. |
| `dashboard/client/` | React/TypeScript dashboard frontend. |
| `doc/` | Documentation sources and runnable examples. |
| `ci/`, `bazel/`, `docker/` | Build (Bazel), CI pipelines, and container image definitions. |

### 11.1 Build system

Ray's core is built with **Bazel** (C++), wrapped by a `setup.py` that drives the
Bazel build during `pip install` from source. Key build env vars:
`RAY_BUILD_CORE`, `RAY_INSTALL_JAVA`, `RAY_INSTALL_CPP`, `RAY_DISABLE_EXTRA_CPP`,
`SKIP_BAZEL_BUILD`, `BAZEL_PATH`. Java uses Maven (`java/pom.xml`); the dashboard
client uses `react-scripts` (`python/ray/dashboard/client/package.json`).

### 11.2 Public-API surface

The Python public API is the `__all__` of `python/ray/__init__.py`
(`init`, `remote`, `get`, `put`, `wait`, `cancel`, `kill`, `method`,
`get_actor`, `nodes`, `cluster_resources`, `available_resources`, `shutdown`,
`get_runtime_context`, `LoggingConfig`, `ClientBuilder`, …). Internal APIs live
under `_private`/`_internal` and are not stability-guaranteed. The canonical set
of `@ray.remote` / `.options()` keys is defined in
`python/ray/_common/ray_option_utils.py` — that file is the source of truth when
documenting or validating task/actor options.

---

## 12. Improvements & Technical Debt

Observations drawn from the source and the live issue tracker:

- **Head-node SPOF by default.** GCS fault tolerance (external Redis) is opt-in;
  many deployments run without it and are one head-node failure from total loss.
  Make Redis-backed GCS the documented default for production.
- **GCS KV store has no eviction for pickled functions** during a job's lifetime
  (#63666) — a slow leak for long-lived drivers that register many functions.
- **Serve controller memory growth** after repeated autoscaling cycles (#58815).
- **Serve LLM Grafana dashboard ships malformed PromQL** (#63219) — observability
  templates need CI validation.
- **Ray Data Parquet pushdown** crashes on generic `UDFExpr` filter predicates
  (#63761) and lacks `unnest` for expressions/`with_columns` (#63838).
- **Fractional-GPU plumbing for Serve LLM** — auto-derived
  `VLLM_RAY_PER_WORKER_GPUS` doesn't reach the vLLM engine (#63875).
- **Security is opt-in and easy to misconfigure.** Auth (`RAY_AUTH_*`) and TLS
  (`RAY_TLS_*`) exist but are off by default; the dashboard is a standing RCE
  vector if exposed. Ship secure-by-default templates for KubeRay/RayService.
- **Operational ergonomics around `/dev/shm`** in containers remains a common
  foot-gun (object-store sizing) — worth surfacing more loudly in the K8s path.

---

## 13. Quick Reference

**Start / stop**

```bash
ray start --head --port=6379 --dashboard-host=127.0.0.1
ray start --address='<head>:6379'
ray stop [--force] [--grace-period N]
ray status [--verbose]
```

**Cluster launcher**

```bash
ray up|down|attach|monitor <config.yaml>
ray exec <config.yaml> "<cmd>"
ray submit <config.yaml> <script.py> [--start]
```

**Jobs**

```bash
ray job submit --working-dir . -- python main.py
ray job status|logs|stop|delete <id>
ray job list
```

**State / observability**

```bash
ray list tasks|actors|nodes|jobs|objects|placement-groups [--filter ... --detail]
ray get <resource> <id>
ray summary tasks|actors|objects
ray logs cluster|actor|worker|job|task [--id ...]
ray memory ; ray timeline ; ray cluster-dump
```

**Key ports:** GCS `6379` · Dashboard/Jobs `8265` · Ray Client `10001` ·
Dashboard agent `52365` · Workers `10002–19999`.

**Key env vars:** `RAY_ADDRESS` · `RAY_REDIS_ADDRESS` · `RAY_STORAGE` ·
`RAY_USE_TLS` + `RAY_TLS_*` · `RAY_AUTH_MODE` + `RAY_AUTH_TOKEN*` ·
`RAY_enable_autoscaler_v2`.

---

_Generated June 2026 from ray-project/ray at ef43e3f. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._