Ray Admin & Architecture Guide

This guide is for the engineers who operate, deploy, secure, and extend Ray: platform/ML-infra teams, SREs, and architects. It documents Ray's internal architecture, the full configuration and environment-variable surface, ports and networking, deployment topologies (bare-metal, VM autoscaler, and Kubernetes), security hardening, day-2 operations, observability, and the repository layout needed to navigate or contribute to the codebase.

For end-user API and CLI usage, see the companion User Guide. This guide assumes you already understand Ray's three core abstractions (Tasks, Actors, Objects).

1. Executive Summary

Ray is a distributed execution framework: a C++ core runtime wrapped by Python (primary), Java, and C++ client APIs, plus five higher-level AI libraries (Data, Train, Tune, Serve, RLlib) and an LLM stack — all of which compile down to the same Tasks/Actors/Objects primitives.

Important

Ray's default security posture is "trusted network." There is no authentication on the core RPC ports unless you explicitly enable it. Treat a Ray cluster like a distributed shell: anyone who can reach its ports can execute arbitrary code. Section 7 covers hardening.

2. Architecture

2.1 Process topology

                     HEAD NODE
   +-------------------------------------------------+
   |  GCS (Global Control Store)   :6379             |
   |    - cluster metadata, actor registry,          |
   |      placement groups, KV store, pub/sub        |
   |  Autoscaler / Monitor                           |
   |  Dashboard (API server)       :8265             |
   |  Raylet (scheduler + object manager)            |
   |  Object Store (Plasma, shared memory)           |
   |  Dashboard agent / runtime-env agent  :52365    |
   |  Driver + Worker processes                      |
   +-------------------------------------------------+
        ^                         ^
        | gRPC                    | gRPC / object transfer
        v                         v
   +------------------+     +------------------+
   |   WORKER NODE    | ... |   WORKER NODE    |
   |  Raylet          |     |  Raylet          |
   |  Object Store    |     |  Object Store    |
   |  Workers         |     |  Workers         |
   |  Dashboard agent |     |  Dashboard agent |
   +------------------+     +------------------+

2.2 Components

2.3 Object ownership and references

Ray uses a distributed ownership model: the worker that creates an ObjectRef owns it and tracks its reference count and lineage. This enables fine-grained garbage collection and lineage-based reconstruction — if a worker dies, Ray can re-execute the task that produced a lost object (when reconstruction is enabled). This is why holding many ObjectRefs pins memory: the owner cannot collect an object while live references exist.

2.4 Scheduling

Each raylet schedules tasks locally against its known resource totals (CPU, GPU, memory, object_store_memory, and custom resources). Resources are logical reservations, not OS-enforced limits — a task that requests num_cpus=2 is accounted as using two cores but is not cgroup-limited unless resource isolation is enabled (see Section 4.5). Scheduling strategies include DEFAULT (pack), SPREAD, node affinity, and placement groups for gang scheduling.

2.5 Language workers

Ray is polyglot. The Python frontend (python/ray) is primary; java/ provides the Java API and runtime (Maven, io.ray), and cpp/ provides the C++ API. All three speak to the same C++ core over gRPC and share the Protobuf message definitions in src/ray/protobuf.

3. Requirements

Warning

Version skew is a frequent cause of mysterious failures. The Ray version, Python version, and key serialization libraries must match across the head and all workers. Pin them in your container image or runtime_env.

4. Configuration

Ray is configured through three channels, in increasing specificity:

1. ray start flags / ray.init() arguments — cluster and node setup.
2. Environment variables (RAY_*) — runtime behavior tuning.
3. Per-task/actor options and runtime_env — workload-level.

4.1 Head-node startup reference

The most operationally significant ray start --head flags:

4.2 Runtime environment variables

Ray reads many RAY_* environment variables at process start. The operationally important runtime ones:

Note

Each library exposes its own large family of tuning variables: RAY_SERVE_* (proxy/router/replica behavior, gRPC, HA proxy, autoscaling windows), RAY_DATA_* (shuffle strategy, exception propagation), and the RAY_TLS_* / RAY_AUTH_* security variables (Section 7). Treat these as advanced knobs — change them deliberately and document the reason.

4.3 Logs and the session directory

By default Ray writes everything under --temp-dir (/tmp/ray), with a per-session subdirectory (/tmp/ray/session_latest). Inside it: logs/ (driver, worker, raylet, gcs_server, dashboard, autoscaler logs), Unix domain sockets, and spilled objects. Point --temp-dir at fast local storage, and ship logs/ to your aggregator. Worker logs rotate at RAY_ROTATION_MAX_BYTES.

4.4 Object store and spilling

The object store lives in shared memory (--plasma-directory, default /dev/shm). When it fills with objects that are still referenced, Ray spills the least-recently-used objects to --object-spilling-directory and restores them on demand. Symptoms of under-sizing: high spill I/O, ObjectStoreFullError, and stalled tasks. Tune --object-store-memory, ensure /dev/shm is large enough (containers often default to 64 MB — raise it), and spill to fast disk.

4.5 Resource isolation (cgroup v2)

--enable-resource-isolation reserves CPU and memory for Ray's system processes using cgroup v2, so a runaway workload can't starve the raylet/GCS.

Requires cgroup v2 with the memory and cpu controllers enabled and read/write permission for the raylet.

5. Networking & Ports

Caution

A Ray cluster requires wide-open node-to-node connectivity on the worker and manager port ranges. Put the entire cluster inside one security group / private subnet and expose nothing to the public internet. The jobs/dash port (8265) and GCS port (6379) are the highest-value targets — never bind them to 0.0.0.0 on a public interface.

6. Deployment

6.1 Manual / bare-metal

# Head:
ray start --head --port=6379 --dashboard-host=127.0.0.1

# Each worker:
ray start --address='<head-ip>:6379'

Run under a process supervisor (systemd, supervisord) using --block so the process stays in the foreground and restarts on failure.

# /etc/systemd/system/ray-head.service (sketch)
[Service]
ExecStart=/usr/bin/ray start --head --block --port=6379 --dashboard-host=127.0.0.1
Restart=always

6.2 VM clusters via the cluster launcher

Define a cluster YAML and let Ray provision and autoscale cloud VMs (AWS, GCP, Azure, vSphere, on-prem). Providers live in python/ray/autoscaler/<provider>.

ray up cluster.yaml --yes        # provision + start
ray monitor cluster.yaml         # tail autoscaler logs
ray submit cluster.yaml job.py   # run a script
ray down cluster.yaml --yes      # tear down

The YAML declares the head/worker node types, min_workers/max_workers, the container/AMI image, and setup commands. The autoscaler scales worker nodes between the min and max based on pending resource demand.

Tip

ray up is idempotent — re-running it applies config changes. Use --no-restart to update worker config without bouncing running Ray processes, and --restart-only to restart without re-running setup commands.

6.3 Kubernetes (KubeRay)

The supported way to run Ray on Kubernetes is the KubeRay operator, which provides three CRDs:

• RayCluster — a Ray cluster (head + worker pod groups) with autoscaling.
• RayJob — run a job to completion against an ephemeral or existing cluster.
• RayService — a highly-available Ray Serve deployment with zero-downtime upgrades.

In-tree support lives in python/ray/autoscaler/kuberay and python/ray/autoscaler/_private/kuberay; the head node runs the ray kuberay-autoscaler entrypoint. The KubeRay operator itself is a separate project that you install into the cluster.

Important

On Kubernetes, give the head pod its own node or generous requests/limits, set /dev/shm large enough for the object store (the default emptyDir memory medium is small), and use RayService (not a bare Deployment) for serving so you get health checks and rolling upgrades.

7. Security

Ray is not secure by default and is not designed for untrusted multi-tenancy. Harden it with network controls plus the optional auth/TLS features.

7.1 The trust boundary

• Anyone who can reach the GCS port, a raylet, or the dashboard/jobs API can run arbitrary code on the cluster.
• The dashboard's Jobs API accepts code submissions; the state API exposes logs.
• Primary control: network isolation. Keep every Ray port on a private network; expose the dashboard only via SSH port-forwarding or an authenticating reverse proxy.

7.2 Token authentication

Ray supports an authentication token on its RPC layer:

Distribute the same token to every node and to clients that connect.

7.3 TLS for gRPC

Encrypt intra-cluster gRPC traffic:

Provision the same CA across all nodes and mount per-node certs.

7.4 Redis hardening

When using an external Redis as the GCS backend (RAY_REDIS_ADDRESS), secure it with --redis-username / --redis-password (or the corresponding env vars) and network ACLs. Never run an internet-reachable unauthenticated Redis.

7.5 Usage telemetry

Ray collects anonymous usage statistics by default. Disable it for sensitive environments with ray disable-usage-stats, ray start --disable-usage-stats, or the usage-stats environment variable.

Caution

Do not rely on the dashboard's --dashboard-host binding alone for security. Combine network isolation, token auth, and TLS. A reachable, unauthenticated dashboard is a remote-code-execution vector.

8. Operations

8.1 Health and status

ray status                       # autoscaler + resource view
ray status --verbose
ray list nodes                   # node health
ray list nodes --filter "state=ALIVE"
ray health-check --component gcs # hidden, NOT a public API — for liveness probes

8.2 Inspecting workloads

ray list tasks --filter "state=FAILED" --detail
ray list actors --filter "state=ALIVE"
ray summary tasks
ray memory                       # object references and store usage
ray logs job --id <SUBMISSION_ID>

8.3 Draining and scaling down

ray drain-node --node-id <ID> --reason <REASON> --reason-message "<msg>"

ray drain-node gracefully removes a worker node (hidden command used by the autoscaler and KubeRay). For autoscaler-managed clusters, prefer adjusting min_workers/max_workers and letting the autoscaler converge.

8.4 Diagnostics collection

ray cluster-dump                 # collect logs/state from nodes into an archive
ray local-dump                   # this node only (hidden)
ray timeline                     # Chrome tracing timeline
ray stack                        # py-spy stacks of local workers
ray global-gc                    # force GC across all workers (hidden)

ray cluster-dump is the first thing to run when filing a bug or analyzing an incident — it bundles logs and state from one or more nodes.

8.5 Backup and persistence

GCS state is in-memory by default and is lost when the head node dies. For fault tolerance:

• Back the GCS with external Redis (RAY_REDIS_ADDRESS) so head-node restarts recover cluster state (GCS fault tolerance).
• Use RAY_STORAGE for a cluster-wide persistent storage URI (checkpoints, Train/Tune artifacts) on durable object storage (S3/GCS/NFS).

Warning

Without external-Redis GCS fault tolerance, a head-node failure tears down the whole cluster — all actors and in-flight work are lost. Treat the head node as a single point of failure and plan accordingly (Redis HA + RayService on K8s).

9. Observability & Metrics

Note

Ray exports metrics per node via the dashboard agent. Configure Prometheus to discover all node agents (not just the head). A known issue (#63219) ships a Serve LLM Grafana dashboard with malformed PromQL — validate panel queries after import.

10. Troubleshooting (operational)

Symptom: head node restart loses the entire cluster

Cause: GCS state is in-memory. Fix: Enable GCS fault tolerance with external Redis (RAY_REDIS_ADDRESS); on K8s use RayService + a managed Redis.

Symptom: workers can't join the head

Cause: Blocked ports or wrong advertised IP. Fix: Open the worker/manager port ranges node-to-node; set --node-ip-address to the address peers can actually reach; confirm the --address='<head>:6379' value.

Symptom: heavy object spilling / ObjectStoreFullError

Cause: Object store too small or /dev/shm undersized (common in containers). Fix: Increase --object-store-memory, enlarge /dev/shm, spill to fast disk via --object-spilling-directory, and stop pinning large objects.

Symptom: GCS memory grows unbounded over a long job

Cause: The GCS KV store has no eviction for pickled functions during a job's lifetime (community issue #63666). Fix: Avoid re-registering many distinct remote functions in long-lived jobs; restart long-running drivers periodically; track the upstream issue.

Symptom: Serve controller RSS grows after autoscaling cycles

Cause: Known controller memory growth (#58815). Fix: Reduce autoscaling churn, restart the controller as a stopgap, and upgrade once a fix lands.

Symptom: autoscaler not adding nodes

Cause: No pending demand visible, provider quota/permissions, or max_workers reached. Fix: Check ray monitor <config> and ray status; verify cloud credentials/quotas and the node-type resource definitions.

11. Repository Layout (for contributors)

11.1 Build system

Ray's core is built with Bazel (C++), wrapped by a setup.py that drives the Bazel build during pip install from source. Key build env vars: RAY_BUILD_CORE, RAY_INSTALL_JAVA, RAY_INSTALL_CPP, RAY_DISABLE_EXTRA_CPP, SKIP_BAZEL_BUILD, BAZEL_PATH. Java uses Maven (java/pom.xml); the dashboard client uses react-scripts (python/ray/dashboard/client/package.json).

11.2 Public-API surface

The Python public API is the __all__ of python/ray/__init__.py (init, remote, get, put, wait, cancel, kill, method, get_actor, nodes, cluster_resources, available_resources, shutdown, get_runtime_context, LoggingConfig, ClientBuilder, …). Internal APIs live under _private/_internal and are not stability-guaranteed. The canonical set of @ray.remote / .options() keys is defined in python/ray/_common/ray_option_utils.py — that file is the source of truth when documenting or validating task/actor options.

12. Improvements & Technical Debt

Observations drawn from the source and the live issue tracker:

• Head-node SPOF by default. GCS fault tolerance (external Redis) is opt-in; many deployments run without it and are one head-node failure from total loss. Make Redis-backed GCS the documented default for production.
• GCS KV store has no eviction for pickled functions during a job's lifetime (#63666) — a slow leak for long-lived drivers that register many functions.
• Serve controller memory growth after repeated autoscaling cycles (#58815).
• Serve LLM Grafana dashboard ships malformed PromQL (#63219) — observability templates need CI validation.
• Ray Data Parquet pushdown crashes on generic UDFExpr filter predicates (#63761) and lacks unnest for expressions/with_columns (#63838).
• Fractional-GPU plumbing for Serve LLM — auto-derived VLLM_RAY_PER_WORKER_GPUS doesn't reach the vLLM engine (#63875).
• Security is opt-in and easy to misconfigure. Auth (RAY_AUTH_*) and TLS (RAY_TLS_*) exist but are off by default; the dashboard is a standing RCE vector if exposed. Ship secure-by-default templates for KubeRay/RayService.
• Operational ergonomics around /dev/shm in containers remains a common foot-gun (object-store sizing) — worth surfacing more loudly in the K8s path.

13. Quick Reference

Start / stop

ray start --head --port=6379 --dashboard-host=127.0.0.1
ray start --address='<head>:6379'
ray stop [--force] [--grace-period N]
ray status [--verbose]

Cluster launcher

ray up|down|attach|monitor <config.yaml>
ray exec <config.yaml> "<cmd>"
ray submit <config.yaml> <script.py> [--start]

Jobs

ray job submit --working-dir . -- python main.py
ray job status|logs|stop|delete <id>
ray job list

State / observability

ray list tasks|actors|nodes|jobs|objects|placement-groups [--filter ... --detail]
ray get <resource> <id>
ray summary tasks|actors|objects
ray logs cluster|actor|worker|job|task [--id ...]
ray memory ; ray timeline ; ray cluster-dump

Key ports: GCS 6379 · Dashboard/Jobs 8265 · Ray Client 10001 · Dashboard agent 52365 · Workers 10002–19999.

Key env vars: RAY_ADDRESS · RAY_REDIS_ADDRESS · RAY_STORAGE · RAY_USE_TLS + RAY_TLS_* · RAY_AUTH_MODE + RAY_AUTH_TOKEN* · RAY_enable_autoscaler_v2.