Apache Spark Administrator Guide

This guide is the operational and architectural reference for Apache Spark — for the engineers, architects, and platform operators who deploy, secure, tune, and run Spark in production. It covers the runtime architecture, the cluster deployment models (Standalone, YARN, Kubernetes), the complete configuration surface, security hardening, monitoring, and the internals a team needs to operate Spark at scale.

End-user material (the data APIs, shells, and spark-submit usage) lives in the companion User Guide.

Note

Generated from the development (master) branch, self-reporting version 5.0.0-SNAPSHOT (PySpark 5.0.0.dev0), built against Scala 2.13.18 and Java 17+. Configuration defaults and feature availability reflect this branch at the documented commit. Released lines (4.x and earlier, also actively built in this tree) may differ — verify against your release.

1. Executive Summary

Apache Spark is a distributed, general-purpose analytics engine. A Spark application consists of a driver process that runs the user program and coordinates work, and a set of executor processes that run tasks and hold cached data. A pluggable cluster manager allocates the executors.

Operational model at a glance:

• One engine, four cluster managers. The same application JAR/script runs unchanged on local, Standalone, YARN, or Kubernetes — only --master and a handful of manager-specific properties change.
• Configuration is layered. Site defaults (conf/spark-defaults.conf) < command-line --conf < programmatic SparkConf. Environment-level concerns (memory ceilings, daemon ports, log dirs) come from conf/spark-env.sh.
• Observability is built in. A live per-application Web UI (default port 4040), event logs replayed by the History Server (default 18080), and a metrics system with sinks for JMX, Prometheus, Graphite, StatsD, CSV, and console.
• Security is opt-in but comprehensive. RPC authentication (spark.authenticate), AES network encryption (spark.network.crypto.enabled), local I/O encryption (spark.io.encryption.enabled), SSL for web UIs, and Kerberos for Hadoop/YARN.

The single most important sizing decision is executor geometry: cores per executor (spark.executor.cores), memory per executor (spark.executor.memory + overhead), and executor count (spark.executor.instances or dynamic allocation). The rest of this guide explains how those interact with the scheduler, the memory manager, and the shuffle subsystem.

2. Architecture

2.1 Runtime topology

                         ┌──────────────────────────────────────┐
                         │             DRIVER PROCESS            │
                         │  ┌────────────────────────────────┐   │
   User program ───────▶ │  │ SparkSession / SparkContext    │   │
                         │  ├────────────────────────────────┤   │
                         │  │ DAGScheduler  (stages, RDD/plan)│   │
                         │  │ TaskScheduler (TaskSetManager)  │   │
                         │  │ SchedulerBackend                │   │
                         │  │ BlockManager (master)           │   │
                         │  │ Web UI  :4040                   │   │
                         │  └───────────────┬────────────────┘   │
                         └──────────────────┼────────────────────┘
                                            │ (RPC: Netty)
                  ┌─────────────────────────┼─────────────────────────┐
                  │            CLUSTER MANAGER (one of):                │
                  │   local │ Standalone Master │ YARN RM │ K8s API     │
                  └─────────────────────────┼─────────────────────────┘
                                            │ launches
              ┌─────────────────────────────┼─────────────────────────────┐
              ▼                              ▼                             ▼
     ┌─────────────────┐          ┌─────────────────┐          ┌─────────────────┐
     │   EXECUTOR 1    │          │   EXECUTOR 2    │   ...    │   EXECUTOR N    │
     │  Task threads   │          │  Task threads   │          │  Task threads   │
     │  BlockManager   │          │  BlockManager   │          │  BlockManager   │
     │  Memory (Unified)│         │  Shuffle files  │          │  Cached blocks  │
     └─────────────────┘          └─────────────────┘          └─────────────────┘

2.2 The execution pipeline

1. Logical plan. DataFrame/SQL operations build a logical plan; the RDD API builds an RDD lineage DAG.
2. Catalyst optimization (SQL/DataFrame). The Catalyst optimizer rewrites the logical plan (predicate pushdown, column pruning, join reordering with cost-based optimization when spark.sql.cbo.enabled=true) and produces a physical plan. Tungsten generates whole-stage codegen for tight, cache-friendly execution.
3. DAGScheduler splits the plan into stages at shuffle boundaries and submits TaskSets.
4. TaskSchedulerImpl + SchedulerBackend assign tasks to executors, respecting locality, the scheduling pool (FIFO/FAIR), and resource profiles. TaskSetManager handles retries, speculation, and blacklisting/excludes.
5. Executors run tasks, read/write shuffle data via their BlockManager, and report heartbeats and metrics back to the driver.
6. Adaptive Query Execution (AQE) (spark.sql.adaptive.enabled, default on) re-optimizes at runtime between stages — coalescing shuffle partitions, switching join strategies, and splitting skewed partitions using actual statistics.

2.3 Memory model (Unified Memory Manager)

Each executor's JVM heap is divided into:

• Reserved memory — fixed overhead.
• Unified region (spark.memory.fraction, default 0.6 of usable heap) — shared between execution memory (shuffles, joins, aggregations, sorts) and storage memory (cached/persisted blocks). The boundary floats; spark.memory.storageFraction (default 0.5) sets the storage region that is immune to eviction.
• User memory — the remainder, for user data structures.
• Off-heap — optional, enabled by spark.memory.offHeap.enabled with size spark.memory.offHeap.size.

On top of the heap, the cluster manager reserves overhead memory (spark.executor.memoryOverhead / spark.executor.memoryOverheadFactor, default factor 0.1) for the off-heap JVM footprint, Python/R workers, and shuffle buffers.

Important

Out-of-memory failures on YARN/Kubernetes are frequently overhead exhaustion, not heap exhaustion — especially with PySpark (each executor forks Python workers, sized by spark.executor.pyspark.memory). If containers are killed for exceeding memory limits, raise spark.executor.memoryOverhead before raising spark.executor.memory.

2.4 Shuffle subsystem

Shuffles redistribute data across partitions (after groupBy, join, repartition). Map tasks write shuffle files locally; reduce tasks fetch them. Key controls:

• spark.shuffle.manager — the shuffle implementation.
• spark.shuffle.compress, spark.shuffle.file.buffer — compression and write buffering.
• spark.shuffle.checksum.enabled / .algorithm — corruption detection.
• Push-based shuffle (spark.shuffle.push.enabled) — merges shuffle blocks on remote mergers to reduce small-block random I/O at scale.
• External Shuffle Service — lets executors be removed (dynamic allocation, decommissioning) without losing their shuffle files.

3. Component Inventory

Spark is a large multi-module Maven build (spark-parent_2.13). The principal modules:

4. System Requirements

4.1 Default ports

5. Deployment Models

5.1 Local mode

No daemons. --master local[*] runs driver and executors in one JVM. For development, CI, and small jobs.

5.2 Standalone cluster

Spark's built-in cluster manager. Start a master and workers with the sbin/ scripts.

# On the master node
./sbin/start-master.sh
# → master at spark://<host>:7077, UI at http://<host>:8080

# On each worker node (point at the master)
./sbin/start-worker.sh spark://<master-host>:7077

Or, with a conf/workers file listing worker hostnames, launch the whole cluster from the master:

./sbin/start-all.sh      # master + all workers in conf/workers
./sbin/stop-all.sh

Individual daemons: start-master.sh/stop-master.sh, start-worker.sh/stop-worker.sh, start-workers.sh/stop-workers.sh, workers.sh <cmd> (run a command on every worker), decommission-worker.sh, and the generic spark-daemon.sh / spark-daemons.sh.

Standalone configuration (set in conf/spark-env.sh):

High availability & recovery — spark.deploy.recoveryMode:

Other standalone knobs: spark.deploy.defaultCores, spark.deploy.maxExecutorRetries, spark.deploy.spreadOutApps, spark.deploy.retainedApplications, spark.deploy.retainedDrivers, spark.deploy.workerSelectionPolicy, spark.cores.max, spark.worker.timeout.

5.3 YARN

export HADOOP_CONF_DIR=/etc/hadoop/conf
./bin/spark-submit --master yarn --deploy-mode cluster \
  --num-executors 20 --executor-cores 4 --executor-memory 8g \
  --queue production --class com.example.App app.jar

Key YARN properties (resource-managers/yarn): spark.yarn.am.memory / .cores / .memoryOverhead (client-mode AM), spark.yarn.queue, spark.yarn.archive (a pre-uploaded jars archive — avoids re-uploading Spark on every submit), spark.yarn.am.nodeLabelExpression, spark.yarn.am.waitTime, and the spark.yarn.cache.* distributed-cache settings. Kerberos: --principal + --keytab (the AM renews delegation tokens).

Important

Always set spark.yarn.archive (or spark.yarn.jars) in production. Without it, Spark uploads its entire jar set to HDFS on every submission, adding latency and load.

5.4 Kubernetes

./bin/spark-submit --master k8s://https://<api-server>:6443 \
  --deploy-mode cluster --name my-job \
  --conf spark.kubernetes.container.image=myrepo/spark:5.0.0 \
  --conf spark.executor.instances=10 \
  local:///opt/app/app.jar

Build images with bin/docker-image-tool.sh. Key K8s properties (resource-managers/kubernetes):

Note

Use the local:// URI scheme for application resources baked into the image. The Kubernetes backend supports executor pod templates, PVC reuse, and graceful decommissioning for spot/preemptible nodes.

5.5 Dynamic allocation

Let Spark scale executors with workload (works on all cluster managers; requires shuffle tracking or an external shuffle service so shuffle data survives executor removal):

5.6 Executor decommissioning

Graceful removal (spot instances, scale-down) migrates shuffle/cache blocks off a doomed executor: spark.decommission.enabled, spark.executor.decommission.killInterval, spark.executor.decommission.forceKillTimeout, spark.executor.decommission.signal.

6. Configuration Reference

Configuration sources, in increasing precedence: conf/spark-defaults.conf → --properties-file → --conf / programmatic SparkConf. The core engine alone defines 530+ spark.* properties (plus the full spark.sql.*, spark.yarn.*, and spark.kubernetes.* families). The tables below cover the high-traffic operational settings; for an exhaustive list, the authoritative source is the ConfigBuilder definitions in core/src/main/scala/org/apache/spark/internal/config/ and the per-module config packages.

6.1 Application & driver/executor

6.2 Memory & storage

6.3 Shuffle & serialization

6.4 Scheduling

For FAIR scheduling, copy conf/fairscheduler.xml.template to fairscheduler.xml and define pools (weights, minShare, scheduling mode). The scheduler implementation is Pool/Schedulable/SchedulableBuilder in core/.../scheduler.

6.5 Event log & History Server

6.6 Networking & RPC

6.7 SQL & Adaptive Query Execution

Tip

For most SQL workloads, leave AQE on and tune only spark.sql.shuffle.partitions (start near total cores × 2–3 for large jobs). AQE's coalescing then right-sizes partitions automatically using runtime statistics.

7. Environment Variables (conf/spark-env.sh)

Copy conf/spark-env.sh.template to conf/spark-env.sh. This file is sourced by all Spark programs and daemons.

Read when launching locally / by drivers & executors:

Standalone daemon options:

Generic daemon options:

Other: SPARK_LAUNCHER_OPTS (launcher JVM options); native BLAS tuning MKL_NUM_THREADS=1 / OPENBLAS_NUM_THREADS=1 (recommended with MLlib + native BLAS, per SPARK-21305); beeline: SPARK_BEELINE_OPTS, SPARK_BEELINE_MEMORY (default 1G).

8. Security

Spark security is off by default and must be explicitly enabled. Harden in this order:

8.1 Authentication & encryption

Caution

Enabling spark.authenticate and spark.network.crypto.enabled is the baseline for any shared or multi-tenant cluster. Without them, RPC traffic is unauthenticated and unencrypted, and the application Web UI / submission endpoints can be abused. On Standalone/Kubernetes, generate the auth secret per-application rather than hardcoding it.

8.2 SSL / TLS for web UIs

Configure the spark.ssl.* family (keystore/truststore paths and passwords, enabled protocols and cipher suites) to serve the Web UI and History Server over HTTPS. Use a dedicated spark.ssl.ui.* namespace where supported.

8.3 Kerberos (Hadoop/YARN)

Submit with --principal <kdc-principal> --keytab <path>; the application (and YARN AM) obtains and renews delegation tokens for HDFS, Hive, HBase, and Kafka. --proxy-user enables impersonation but is mutually exclusive with --principal/--keytab.

8.4 Web UI access control & redaction

• ACLs: spark.acls.enable, spark.ui.view.acls, spark.modify.acls, plus *.groups variants and spark.admin.acls.
• spark.ui.filters — install servlet filters (e.g. authentication proxies) in front of the UI.
• Secret redaction: spark.redaction.regex scrubs matching config values from the UI, logs, and event logs. Confirm it covers your credential key patterns.

Warning

The live application Web UI (port 4040) and History Server expose configuration, environment, and SQL plans. In production, place them behind authentication (spark.ui.filters / a reverse proxy) and enable redaction. Do not expose these ports to untrusted networks.

9. Monitoring & Operations

9.1 Web UI & History Server

The live UI (driver, port 4040) shows jobs, stages, tasks, the DAG, the SQL/AQE plan, storage, executors, and environment — but disappears when the application ends. For post-hoc analysis, enable event logging and run the History Server:

./sbin/start-history-server.sh     # serves http://<host>:18080
./sbin/stop-history-server.sh

9.2 Metrics system

Spark's metrics system (configured via conf/metrics.properties, from the .template) exposes driver, executor, master, worker, and application metrics through pluggable sinks:

9.3 Logging

Logging uses Log4j2. Copy conf/log4j2.properties.template to log4j2.properties (or use log4j2-json-layout.properties.template for structured JSON logs — ideal for log aggregation). Daemon logs land in SPARK_LOG_DIR.

9.4 The Thrift JDBC/ODBC server

Expose Spark SQL to BI tools and JDBC clients:

./sbin/start-thriftserver.sh      # HiveServer2-compatible, port 10000
./bin/beeline -u jdbc:hive2://localhost:10000
./sbin/stop-thriftserver.sh

9.5 The Spark Connect server

./sbin/start-connect-server.sh    # gRPC, port 15002
./sbin/stop-connect-server.sh

Connect-server configuration lives under spark.connect.* (e.g. session/store and grpc settings in sql/connect/server).

9.6 Operational runbook

10. Performance Tuning Checklist

• [ ] Use Kryo serialization (spark.serializer=org.apache.spark.serializer.KryoSerializer).
• [ ] Keep AQE on; tune spark.sql.shuffle.partitions for very large jobs.
• [ ] Right-size executor geometry — 3–5 cores/executor is a common sweet spot; avoid one fat executor per node (GC pressure) and avoid many tiny executors (overhead).
• [ ] Set memory overhead generously for PySpark / off-heap workloads before raising heap.
• [ ] Enable dynamic allocation for bursty/multi-tenant clusters (with shuffle tracking or the external shuffle service).
• [ ] Cache (persist) only datasets reused across actions; watch the Storage tab for eviction.
• [ ] Prefer broadcast joins for small dimension tables (spark.sql.autoBroadcastJoinThreshold).
• [ ] Use push-based shuffle (spark.shuffle.push.enabled) on very large shuffles.
• [ ] Place spark.local.dir/SPARK_LOCAL_DIRS on fast local disks (NVMe), not the OS root volume.
• [ ] Pre-stage Spark jars on YARN (spark.yarn.archive) to cut submission latency.

11. Troubleshooting (Operational)

Issue: Executors killed by YARN/K8s for exceeding memory

Cause: Container memory = heap + overhead exceeded the manager's limit (often overhead, not heap). Resolution: Raise spark.executor.memoryOverhead (or the factor). For PySpark, account for spark.executor.pyspark.memory.

Issue: Job stuck / very slow on a few tasks (skew)

Cause: Data skew concentrating work on a few partitions. Resolution: Keep spark.sql.adaptive.skewJoin.enabled=true; consider salting keys or repartitioning. Enable speculation (spark.speculation) for straggler nodes.

Issue: Total size of serialized results … is bigger than spark.driver.maxResultSize

Cause: An action returned too much data to the driver. Resolution: Avoid collect() on large data; write to storage. Raise spark.driver.maxResultSize only if truly needed (0 = unlimited, risky).

Issue: Standalone master failover loses applications

Cause: spark.deploy.recoveryMode=NONE. Resolution: Use ZOOKEEPER (multi-master HA) or FILESYSTEM/ROCKSDB recovery with a durable spark.deploy.recoveryDirectory.

Issue: YARN submission rejected — no Hadoop config

Cause: HADOOP_CONF_DIR/YARN_CONF_DIR unset. Resolution: Export it (set in conf/spark-env.sh).

Issue: Shuffle fetch failures after executors are removed

Cause: Dynamic allocation/decommission removed executors holding shuffle files with no service to serve them. Resolution: Enable the external shuffle service or spark.dynamicAllocation.shuffleTracking.enabled=true.

Issue: Web UI / logs leak secrets

Cause: Credentials in config values, no redaction. Resolution: Set spark.redaction.regex to match your secret keys; restrict UI access with spark.ui.filters/ACLs.

12. Internals (Developer Deep-Dive)

For contributors and platform engineers extending Spark:

• Configuration system. Every property is a typed ConfigEntry built by ConfigBuilder("spark.x.y").doc(...).version(...).<type>Conf.createWithDefault(...) in core/.../internal/config/ and per-module config packages. The .doc() string is the canonical description and the .version(...) records when the key was introduced — the authoritative, version-stamped config reference. Negative/removed knowledge and deprecations are tracked in the same definitions.
• Scheduler. DAGScheduler (stage/DAG construction, shuffle-boundary splitting, retry of failed stages) → TaskSchedulerImpl (offer-based task assignment, locality, excludes) → SchedulerBackend (manager-specific executor management) → TaskSetManager (per-stage task lifecycle, speculation). Fair scheduling via Pool/Schedulable/SchedulableBuilder.
• Block management. BlockManager on driver and each executor handles cached blocks, broadcast variables, and shuffle data, backed by the network-common/network-shuffle Netty transport.
• Catalyst & Tungsten (sql/catalyst, sql/core). Parser → analyzer (catalog resolution) → optimizer (rule-based + cost-based) → physical planning → whole-stage codegen. AQE re-plans between materialized stages.
• Plugin & extension points. SparkPlugin/DriverPlugin/ExecutorPlugin (core/.../api/plugin), custom shuffle (shuffle/api), data source v2, custom catalogs (spark.sql.catalog.*), custom metrics sinks, and standalone recovery factories.
• Spark Connect (sql/connect). A gRPC protocol decoupling the client from the driver; the server translates protobuf-encoded relations into Spark plans. Clients exist for Python, Scala/JVM, and JDBC.
• Language bridges. PySpark (python/pyspark) and SparkR communicate with the JVM through a gateway; PySpark workers are forked per executor and tuned via PYSPARK_* / spark.executor.pyspark.memory. The pandas API on Spark lives in python/pyspark/pandas.

13. Repository Structure

spark/
├── core/             Engine: scheduler, BlockManager, RDD, Standalone deploy, shuffle
├── common/           network-common, network-shuffle, kvstore, unsafe, sketch, variant, utils, tags
├── launcher/         SparkLauncher + SparkSubmitOptionParser
├── repl/             Scala interactive shell
├── sql/
│   ├── api/          Stable public SQL API
│   ├── catalyst/     Logical plans, analyzer, optimizer, expressions, parser
│   ├── core/         DataFrame/Dataset, physical execution, data sources, AQE
│   ├── hive/         Hive catalog integration
│   ├── hive-thriftserver/  JDBC/ODBC (HiveServer2) server
│   ├── connect/      Spark Connect (gRPC server + clients)
│   └── pipelines/    Declarative Pipelines engine
├── streaming/        Structured Streaming + legacy DStreams
├── mllib/ mllib-local/  Machine learning
├── graphx/           Graph processing + algorithms
├── connector/        kafka-0-10, kinesis-asl, avro, protobuf, profiler, ganglia
├── resource-managers/
│   ├── yarn/         YARN backend
│   └── kubernetes/   Kubernetes backend + integration tests + docker
├── hadoop-cloud/     S3A/GCS/ABFS cloud object-store integration
├── python/           PySpark (incl. pandas API, pipelines, errors)
├── R/                SparkR (deprecated)
├── bin/  sbin/       User launchers / cluster daemon scripts
├── conf/             Config templates
├── examples/ data/   Sample programs and datasets
├── dev/ project/     Build, release, and dev tooling
└── docs/             Documentation site sources

14. Potential Improvements & Operational Notes

These are operator-facing observations, not defects:

• Pin firewall-sensitive ports. Driver (spark.driver.port), block manager (spark.blockManager.port), and the external shuffle service (7337) default to random/fixed ports; pin them explicitly when running behind firewalls or in locked-down networks.
• Standalone HA requires deliberate setup. The default recoveryMode=NONE means a master restart drops running applications. Choose ZOOKEEPER for true HA in any production Standalone deployment.
• SparkR is deprecated. Plan migrations off R toward PySpark/Scala/SQL; avoid building new pipelines on SparkR.
• Declarative Pipelines is new on the master branch. Treat the spark-pipelines CLI and spec format as evolving; validate against release notes before standardizing on it.
• spark.sql.ansi.enabled semantics differ by line. ANSI mode changes overflow/cast/null behavior; test workloads before flipping it cluster-wide, as it can change query results and error behavior.
• Event-log growth. Long-running apps and busy clusters accumulate event logs quickly; enable spark.eventLog.rolling.enabled and the History Server cleaner (spark.history.fs.cleaner.enabled) to bound storage.

15. Quick Reference

Cluster daemon scripts (sbin/)

Essential properties