# Keras Administrator & Developer Guide

**Keras 3.15.0** — Multi-backend deep learning framework for Python. Build models once and run them on TensorFlow, JAX, PyTorch, NumPy, or OpenVINO without changing a single line of model code.

This guide covers architecture internals, configuration, deployment patterns, performance tuning, security, and developer workflows. For installation, everyday usage, and model-building tutorials, see the **User Guide**.

---

## 1. Architecture Overview

Keras 3 is a ground-up rewrite that decouples the high-level modeling API from any single computation backend. The architecture consists of five major subsystems: **backend dispatch**, **layer/model hierarchy**, **training loop**, **serialization**, and **ops abstraction**.

### 1.1 Backend Dispatch

Backend selection happens exactly once — at `import keras` time — and is immutable for the lifetime of the process. The selection logic in `keras/src/backend/__init__.py` follows this priority:

1. `KERAS_BACKEND` environment variable (highest priority)
2. `"backend"` key in `~/.keras/keras.json`
3. Default: `"tensorflow"`

Once resolved, the backend module (`keras.src.backend.{name}`) is loaded via wildcard import, making all backend-specific implementations available through a single namespace. Each backend implements a standardized set of modules:

| Module | Purpose |
|--------|---------|
| `core.py` | Tensor creation, casting, basic operations |
| `nn.py` | Neural network primitives (conv, pool, softmax) |
| `numpy.py` | NumPy-compatible array operations |
| `math.py` | Mathematical functions |
| `image.py` | Image processing operations |
| `linalg.py` | Linear algebra |
| `random.py` | Random number generation |
| `rnn.py` | Recurrent network operations |
| `trainer.py` | Backend-specific training loop |
| `layer.py` | Backend-specific layer behavior |
| `export.py` | Model export to backend-native formats |

JAX and TensorFlow backends additionally implement `distribution_lib.py` for multi-device distributed training. NumPy and OpenVINO backends set `distribution_lib = None` — these are inference-only backends that do not support training or distribution.

> [!NOTE]
> When the active backend is anything other than TensorFlow, Keras automatically sets `TF_FORCE_GPU_ALLOW_GROWTH=true` to prevent TensorFlow (which may still be installed as a dependency) from pre-allocating all GPU memory. This is critical in environments where JAX or PyTorch needs GPU access.

### 1.2 Layer and Model Hierarchy

The class hierarchy is:

```
Operation
  └── Layer
        └── Model (also inherits Trainer)
              ├── Sequential
              └── Functional
```

**`Layer`** (`keras/src/layers/layer.py`) is the fundamental building block. It implements:

- **Deferred weight creation** via `build()` — weights are allocated on first call when input shapes are known
- **Forward pass** via `call()`
- **Automatic variable tracking** through the `Tracker` utility (`keras/src/utils/tracking.py`), which wraps Python containers (`TrackedList`, `TrackedDict`, `TrackedSet`, `TrackedOrderedDict`) to monitor attribute mutations
- **Serialization** via `get_config()` / `from_config()`
- **Quantization** via `quantize(mode)` and LoRA via `enable_lora(rank)`
- **Name scoping** via `keras.name_scope` for hierarchical naming

**`Model`** (`keras/src/models/model.py`) adds training capabilities. Its `__new__` method inspects the active backend at instantiation time and returns a backend-specific Trainer subclass (JAX Trainer, TF Trainer, Torch Trainer, NumPy Trainer, or OpenVINO Trainer). This is transparent to user code.

**Three model construction patterns:**

| Pattern | Class | How It Works |
|---------|-------|--------------|
| Subclassing | `Model` | Override `call()`, imperative style |
| Sequential | `Sequential` | Linear layer stack via `add()` / `pop()` |
| Functional | `Functional` | DAG from `Input()` → layer calls → `Model(inputs, outputs)` |

`Sequential` validates that each layer accepts a single positional argument and auto-creates an `InputLayer` when `batch_shape` is provided. `Functional` builds a directed acyclic graph and uses `_run_through_graph()` for graph traversal, supporting optional inputs.

### 1.3 Operations System

The `keras/src/ops/` package provides backend-agnostic mathematical operations. These are not thin wrappers — each op has a symbolic path (for graph tracing) and a concrete path (for eager execution):

- **`core.py`**: Control flow — `scan`, `while_loop`, `cond`, `fori_loop`, `stop_gradient`, `custom_gradient`, `vectorized_map`
- **`numpy.py`**: Full NumPy-compatible API (200+ functions)
- **`nn.py`**: Neural network operations — convolutions, pooling, activations, normalization
- **`math.py`**: Mathematical operations beyond NumPy
- **`image.py`**: Image processing (resize, crop, affine transforms)
- **`linalg.py`**: Linear algebra (solve, SVD, cholesky, etc.)
- **`einops.py`**: Einstein notation operations (`rearrange`)

The `Operation` base class (`keras/src/ops/operation.py`) tracks computation graphs via `Node` objects (`keras/src/ops/node.py`). The `Function` class (`keras/src/ops/function.py`) traces operations into a reusable computation graph.

### 1.4 Trainer System

The training loop (`keras/src/trainers/trainer.py`) implements `compile()`, `fit()`, `evaluate()`, and `predict()`. Compilation binds a loss function, optimizer, and metrics to the model via `compile_utils.py`.

Data flows through **data adapters** (`keras/src/trainers/data_adapters/`), which normalize different input formats into a consistent iterator interface:

| Adapter | Input Type |
|---------|------------|
| `ArrayDataAdapter` | NumPy arrays, lists |
| `TFDatasetAdapter` | `tf.data.Dataset` |
| `TorchDataLoaderAdapter` | `torch.utils.data.DataLoader` |
| `PyDatasetAdapter` | `keras.utils.PyDataset` (multiprocessing) |
| `GeneratorDataAdapter` | Python generators |
| `GrainDatasetAdapter` | Grain datasets (JAX ecosystem) |

The `EpochIterator` (`epoch_iterator.py`) manages epoch/step iteration, calling the appropriate data adapter and yielding batches.

### 1.5 Serialization Architecture

Keras uses a layered serialization system in `keras/src/saving/`:

- **`.keras` format** (`saving_lib.py`): A zip archive containing `config.json` (model topology + layer configs) and `model.weights.h5` (weights in HDF5). This is the primary save format.
- **Object serialization** (`serialization_lib.py`): Converts arbitrary Keras objects to/from JSON-safe dictionaries using a class registry. Custom objects must be registered via `@keras.saving.register_keras_serializable`.
- **Object registration** (`object_registration.py`): Global registry mapping class names to implementations.
- **File editor** (`file_editor.py`): Allows in-place editing of saved model files without full deserialization.
- **Orbax integration** (`orbax_util.py`): JAX checkpointing via the Orbax library.
- **Legacy** (`keras/src/legacy/saving/`): HDF5 format support for backward compatibility.

### 1.6 Export Subsystem

The export system (`keras/src/export/`) converts trained models to deployment-ready formats:

| Format | Module | Requirements |
|--------|--------|-------------|
| TF SavedModel | `saved_model.py` | TensorFlow backend or installed |
| LiteRT (TFLite) | `litert.py` | TensorFlow backend required |
| ONNX | `onnx.py`, `tf2onnx_lib.py` | `tf2onnx` package |
| OpenVINO IR | `openvino.py` | `openvino` package |
| Neptune | `neptune_model_export_archive.py` | Neptune SDK |

The `TFSMLayer` class (`tfsm_layer.py`) enables loading a TF SavedModel back as a Keras layer, enabling interop with non-Keras TensorFlow models.

### 1.7 Quantization Architecture

Keras supports post-training quantization through `keras/src/quantizers/`:

| Mode | Implementation | Description |
|------|---------------|-------------|
| `int8` | `quantizers.py` | 8-bit integer quantization |
| `int4` | `quantizers.py` | 4-bit integer quantization |
| `float8` | `quantizers.py` | 8-bit floating point |
| `awq` | `awq.py`, `awq_core.py`, `awq_config.py` | Activation-aware weight quantization |
| `gptq` | `gptq.py`, `gptq_core.py`, `gptq_config.py` | Post-training quantization (GPT-Q) |

LoRA (Low-Rank Adaptation) is supported on `Dense`, `EinsumDense`, and `Embedding` layers via `enable_lora(rank)`. LoRA is incompatible with GPTQ quantization and certain kernel constraints.

### 1.8 Distribution System

Multi-device training is implemented in `keras/src/distribution/distribution_lib.py` with backend-specific implementations:

- **JAX**: `keras/src/backend/jax/distribution_lib.py` — uses JAX's `pjit` and mesh-based sharding
- **TensorFlow**: `keras/src/backend/tensorflow/distribution_lib.py` — uses `tf.distribute` strategies

NumPy and OpenVINO backends do not support distribution.

### 1.9 Knowledge Distillation

The `keras/src/distillation/` module provides:

- `distiller.py`: `Distiller` class — a training wrapper that takes a teacher model and student model, training the student to match the teacher's outputs
- `distillation_loss.py`: Custom loss combining student task loss with teacher-student divergence

---

## 2. System Requirements

### 2.1 Python Version

Keras 3.15.0 requires Python 3.9+. The `pyproject.toml` specifies `requires-python = ">= 3.9"`.

### 2.2 Backend-Specific Dependencies

Each backend has its own requirements file. Install only the backend(s) you need:

**TensorFlow backend:**

```bash
pip install keras tensorflow
# GPU (CUDA):
pip install keras -r requirements-tensorflow-cuda.txt
# TPU:
pip install keras -r requirements-tensorflow-tpu.txt
```

On non-macOS systems, `tensorflow-cpu` is sufficient for CPU-only workloads. On macOS, use `tensorflow` (which includes Metal acceleration).

**JAX backend:**

```bash
pip install keras jax[cpu] flax
# GPU (CUDA):
pip install keras -r requirements-jax-cuda.txt
# TPU:
pip install keras -r requirements-jax-tpu.txt
```

**PyTorch backend:**

```bash
pip install keras torch
# GPU (CUDA):
pip install keras -r requirements-torch-cuda.txt
```

`torch-xla` is additionally required on non-macOS platforms for XLA compilation support.

**NumPy backend (inference only):**

```bash
pip install keras numpy
```

**OpenVINO backend (inference only):**

```bash
pip install keras openvino
```

> [!IMPORTANT]
> The NumPy and OpenVINO backends support inference only — `model.fit()` and `model.train_on_batch()` will raise errors. Use these backends strictly for `model.predict()` and `model()` calls.

### 2.3 Common Dependencies

The `requirements-common.txt` file lists shared dependencies across all backends. These are pulled in automatically when installing Keras via pip.

### 2.4 Optional Dependencies

| Package | Feature |
|---------|---------|
| `pydot` | `keras.utils.plot_model()` visualization |
| `tf2onnx` | ONNX export |
| `openvino` | OpenVINO export (separate from OpenVINO backend) |
| `ai-edge-litert` | LiteRT (TFLite) export |

---

## 3. Configuration Reference

### 3.1 Config File

Keras reads configuration from `~/.keras/keras.json` at import time. The config directory can be overridden with the `KERAS_HOME` environment variable. If the home directory is not writable, Keras falls back to `/tmp`.

Example `keras.json`:

```json
{
    "floatx": "float32",
    "epsilon": 1e-7,
    "backend": "tensorflow",
    "image_data_format": "channels_last",
    "nnx_enabled": false
}
```

### 3.2 Configuration Parameters

All configuration is managed through module-level globals in `keras/src/backend/config.py` with getter/setter pairs:

| Parameter | Default | Getter | Setter | Description |
|-----------|---------|--------|--------|-------------|
| `_FLOATX` | `"float32"` | `floatx()` | `set_floatx(value)` | Default float dtype for weights and computations |
| `_EPSILON` | `1e-7` | `epsilon()` | `set_epsilon(value)` | Small constant for numerical stability |
| `_IMAGE_DATA_FORMAT` | `"channels_last"` | `image_data_format()` | `set_image_data_format(value)` | `"channels_last"` (NHWC) or `"channels_first"` (NCHW) |
| `_BACKEND` | `"tensorflow"` | `backend()` | — (immutable after import) | Active computation backend |
| `_NNX_ENABLED` | `False` | `is_nnx_enabled()` | `set_nnx_enabled(value)` | Enable Flax NNX features for JAX backend |
| `_MAX_EPOCHS` | `None` | `max_epochs()` | `set_max_epochs(value)` | Cap on training epochs (debugging) |
| `_MAX_STEPS_PER_EPOCH` | `None` | `max_steps_per_epoch()` | `set_max_steps_per_epoch(value)` | Cap on steps per epoch (debugging) |

> [!IMPORTANT]
> The backend cannot be changed after `import keras`. To switch backends, set the environment variable *before* importing: `os.environ["KERAS_BACKEND"] = "jax"` must come before `import keras`. Alternatively, set it in the shell: `KERAS_BACKEND=jax python train.py`.

### 3.3 Environment Variables

| Variable | Purpose | Default |
|----------|---------|---------|
| `KERAS_BACKEND` | Select backend: `tensorflow`, `jax`, `torch`, `numpy`, `openvino` | `tensorflow` |
| `KERAS_HOME` | Override config directory location | `~/.keras` |
| `KERAS_MAX_EPOCHS` | Cap training epochs (useful for CI/debugging) | None (unlimited) |
| `KERAS_MAX_STEPS_PER_EPOCH` | Cap steps per epoch | None (unlimited) |
| `KERAS_NNX_ENABLED` | Enable Flax NNX integration (`"true"`) | `"false"` |
| `TF_FORCE_GPU_ALLOW_GROWTH` | Auto-set by Keras when backend ≠ tensorflow | — |

### 3.4 Runtime Configuration

Configuration can be changed at runtime through the Python API:

```python
import keras

# Change float precision
keras.config.set_floatx("float16")

# Change image data format
keras.config.set_image_data_format("channels_first")

# Change epsilon for numerical stability
keras.config.set_epsilon(1e-5)

# Limit training for debugging
keras.config.set_max_epochs(2)
keras.config.set_max_steps_per_epoch(10)

# Enable Flax NNX (JAX backend only)
keras.config.set_nnx_enabled(True)
```

### 3.5 Mixed Precision

Keras supports mixed precision training through dtype policies (`keras/src/dtype_policies/`):

```python
keras.mixed_precision.set_global_policy("mixed_float16")
```

This configures layers to use `float16` for computations while keeping `float32` for weight storage, reducing memory usage and improving throughput on GPUs with Tensor Cores.

---

## 4. Deployment & Scaling

### 4.1 Model Export for Production

Keras does not include a serving runtime — it is a library for building and training models. Deployment involves exporting to a format consumed by a serving system.

**TF SavedModel (recommended for TensorFlow Serving):**

```python
model.export("./saved_model_dir")
# Or with format specification:
model.export("./saved_model_dir", format="tf_saved_model")
```

**LiteRT / TFLite (mobile and edge):**

```python
model.export("model.tflite", format="tflite")
```

This requires the TensorFlow backend and the `ai-edge-litert` package.

**ONNX (cross-platform):**

```python
model.export("model.onnx", format="onnx")
```

Requires `tf2onnx`. The export goes through TF SavedModel as an intermediate step.

**OpenVINO IR (Intel hardware):**

```python
model.export("model_dir", format="openvino")
```

> [!TIP]
> For maximum portability, export to ONNX. For best performance on Intel hardware, export to OpenVINO IR. For the TensorFlow ecosystem (TF Serving, TFX), export to SavedModel.

### 4.2 Distributed Training

Distributed training is supported on JAX and TensorFlow backends via the distribution API:

```python
import keras
from keras.distribution import DeviceMesh, LayoutMap, ModelParallel

# Create a device mesh
devices = keras.distribution.list_devices()
mesh = DeviceMesh(shape=(2, 4), axis_names=["batch", "model"], devices=devices)

# Define how tensors map to devices
layout_map = LayoutMap(mesh)
layout_map[".*dense.*kernel"] = ("model", None)
layout_map[".*dense.*bias"] = ("model",)

# Apply distribution strategy
distribution = ModelParallel(mesh, layout_map=layout_map)
keras.distribution.set_distribution(distribution)

# Build and train model as usual
model = create_model()
model.compile(optimizer="adam", loss="mse")
model.fit(dataset)
```

### 4.3 Quantization for Deployment

Reduce model size and improve inference speed:

```python
# Post-training quantization
model.quantize("int8")  # or "int4", "float8"

# AWQ (requires calibration data)
model.quantize("awq")

# GPTQ (requires calibration data)
model.quantize("gptq")

# Save quantized model
model.save("quantized_model.keras")
```

> [!WARNING]
> Quantization is a one-way operation on the model instance. Always save your full-precision model before quantizing. The quantized model will have reduced numerical precision and may show degraded accuracy on some tasks.

### 4.4 Build System

Keras uses `pip_build.py` for wheel creation:

```bash
# Build wheel package
python3 pip_build.py
# Output: dist/keras-3.15.0-py3-none-any.whl

# Build and install locally
python3 pip_build.py --install

# Nightly build (appends dev version suffix)
python3 pip_build.py --nightly

# Release candidate
python3 pip_build.py --rc 1
```

The build process copies source to `tmp_build_dir`, excludes `_test.py` files, and runs `python3 -m build`. Package metadata is defined in `pyproject.toml`.

---

## 5. Operations

### 5.1 Pre-trained Applications

Keras ships with pre-trained model architectures in `keras/src/applications/` (VGG, ResNet, EfficientNet, MobileNet, etc.). Weights are downloaded on first use and cached in `KERAS_HOME/models/`.

```python
from keras.applications import ResNet50
model = ResNet50(weights="imagenet")
```

Weight files are fetched via `keras.utils.get_file()`, which supports SHA256/MD5 hash verification for integrity.

### 5.2 File Downloads and Caching

The `get_file()` utility (`keras/src/utils/file_utils.py`) manages file downloads:

- Default cache directory: `~/.keras/` (overridden by `KERAS_HOME`)
- Supports hash verification (SHA256, MD5) for downloaded files
- Implements path traversal protection with symlink validation

### 5.3 Callbacks for Operational Monitoring

Keras provides built-in callbacks for monitoring training:

| Callback | Purpose |
|----------|---------|
| `ModelCheckpoint` | Save model at intervals |
| `EarlyStopping` | Stop training when metric plateaus |
| `TensorBoard` | Log metrics for TensorBoard visualization |
| `CSVLogger` | Log metrics to CSV file |
| `ProgbarLogger` | Display progress bar |
| `ReduceLROnPlateau` | Reduce learning rate on plateau |
| `LearningRateScheduler` | Custom learning rate schedules |
| `RemoteMonitor` | Send metrics to a remote server |
| `BackupAndRestore` | Fault-tolerant training with checkpointing |
| `TerminateOnNaN` | Stop on NaN loss |
| `LambdaCallback` | Custom callback via lambdas |
| `SwapEMAWeights` | Swap in EMA weights for evaluation |

> [!NOTE]
> `BackupAndRestore` is essential for long-running training jobs on preemptible instances. It saves training state (epoch, step) and restores automatically on restart.

### 5.4 Debugging Training Issues

Use the epoch/step capping environment variables to rapidly iterate:

```bash
# Run only 2 epochs with 10 steps each
KERAS_MAX_EPOCHS=2 KERAS_MAX_STEPS_PER_EPOCH=10 python train.py
```

Or set programmatically:

```python
keras.config.set_max_epochs(2)
keras.config.set_max_steps_per_epoch(10)
```

> [!TIP]
> These caps are ideal for CI pipelines where you want to verify the training loop runs without errors, but don't need convergence. They also work for debugging data loading and preprocessing pipelines.

### 5.5 Model Visualization

Generate architecture diagrams (requires `pydot`):

```python
keras.utils.plot_model(
    model,
    to_file="model.png",
    show_shapes=True,
    show_layer_names=True,
    show_dtype=True,
    show_layer_activations=True,
    show_trainable=True,
)
```

The `model_to_dot()` function returns a `pydot.Dot` object for programmatic graph manipulation.

Image-specific visualization tools in `keras/src/visualization/`:

- `draw_bounding_boxes()` — overlay boxes on images
- `draw_segmentation_masks()` — overlay masks on images
- `plot_bounding_box_gallery()`, `plot_image_gallery()`, `plot_segmentation_mask_gallery()` — gallery views

---

## 6. Performance

### 6.1 Backend Selection for Performance

Backend choice has significant performance implications:

| Backend | Best For | Training | GPU | TPU |
|---------|----------|----------|-----|-----|
| TensorFlow | Production serving, TF ecosystem | Yes | Yes | Yes |
| JAX | Research, TPU workloads, functional style | Yes | Yes | Yes |
| PyTorch | Research, PyTorch ecosystem integration | Yes | Yes | No |
| NumPy | CPU inference, debugging | No | No | No |
| OpenVINO | Intel CPU/GPU/VPU inference | No | Intel only | No |

### 6.2 Mixed Precision Training

Enable `float16` computation with `float32` master weights:

```python
keras.mixed_precision.set_global_policy("mixed_float16")
```

This typically provides 2-3x speedup on GPUs with Tensor Cores (NVIDIA Volta and later) with minimal accuracy impact. For TPUs, use `"mixed_bfloat16"`.

### 6.3 Data Pipeline Performance

The data adapter system (`keras/src/trainers/data_adapters/`) abstracts input formats, but performance characteristics differ:

- **`tf.data.Dataset`** (TFDatasetAdapter): Best for TensorFlow backend. Supports prefetching, parallel mapping, and interleaving.
- **`torch.utils.data.DataLoader`** (TorchDataLoaderAdapter): Best for PyTorch backend. Supports multi-worker loading.
- **`keras.utils.PyDataset`** (PyDatasetAdapter): Backend-agnostic multiprocessing data loading. Set `workers` and `use_multiprocessing` parameters.
- **`GrainDatasetAdapter`**: For JAX ecosystem Grain datasets.

### 6.4 XLA Compilation

JAX uses XLA compilation by default. For TensorFlow, enable with:

```python
model.compile(optimizer="adam", loss="mse", jit_compile=True)
```

XLA compilation fuses operations and optimizes memory layout, often providing 20-50% speedup for computation-bound models. However, it increases initial compilation time and may not support all operations.

### 6.5 Image Data Format

The `image_data_format` setting affects convolution and pooling layer behavior:

- `"channels_last"` (NHWC): Default. Best performance on most GPUs via cuDNN.
- `"channels_first"` (NCHW): May be faster on some older NVIDIA GPUs and is the PyTorch convention.

```python
keras.config.set_image_data_format("channels_first")
```

### 6.6 Quantization Performance Impact

| Mode | Model Size Reduction | Inference Speedup | Accuracy Impact |
|------|---------------------|-------------------|-----------------|
| `int8` | ~4x | 2-4x (hardware dependent) | Low |
| `int4` | ~8x | Variable | Moderate |
| `float8` | ~2x | 1.5-2x | Very low |
| `awq` | ~4x | 2-4x | Low (activation-aware) |
| `gptq` | ~4x | 2-4x | Low (calibration-based) |

---

## 7. Security

### 7.1 Deserialization Safety

Keras implements a security boundary around model deserialization. By default, only known Keras classes can be deserialized. Two scenarios require explicit opt-in:

**Lambda layers:**

```python
# Lambda layers contain arbitrary Python code
# Loading a model with Lambda layers requires:
keras.config.enable_unsafe_deserialization()
model = keras.models.load_model("model_with_lambda.keras")
```

**Torch `nn.Module` wrappers:**

```python
# TorchModuleWrapper also carries arbitrary code
keras.config.enable_unsafe_deserialization()
model = keras.models.load_model("model_with_torch_module.keras")
```

> [!CAUTION]
> `enable_unsafe_deserialization()` allows arbitrary code execution during model loading. NEVER call this on models from untrusted sources. A malicious `.keras` file with Lambda layers can execute arbitrary Python code on your system.

### 7.2 File Download Integrity

The `get_file()` utility supports cryptographic hash verification:

```python
path = keras.utils.get_file(
    "weights.h5",
    "https://example.com/weights.h5",
    file_hash="abc123...",
    hash_algorithm="sha256"  # or "md5"
)
```

Always provide `file_hash` when downloading weights from external sources to verify integrity.

### 7.3 Path Traversal Protection

`keras/src/utils/file_utils.py` implements protections against path traversal attacks:

- Symlink validation prevents following symbolic links outside expected directories
- Path filtering rejects suspicious path components

### 7.4 Serialization Registry Validation

The `serialization_lib.py` module validates class names during deserialization against a registry of known safe classes. However, GitHub issue #22703 reports a gap where `deserialize_keras_object` accepts arbitrary `registered_name` values. Exercise caution when loading models from untrusted sources, even without `enable_unsafe_deserialization()`.

> [!WARNING]
> Do not load `.keras` model files from untrusted sources without reviewing the `config.json` inside the archive first. You can inspect it by unzipping the `.keras` file (it is a standard zip archive) and examining the class references.

### 7.5 Security Checklist for Production

1. Never call `enable_unsafe_deserialization()` in production serving pipelines
2. Always verify file hashes when downloading pre-trained weights
3. Audit `config.json` in `.keras` files before loading models from third parties
4. Pin Keras and backend versions in production `requirements.txt`
5. Use quantized/exported models (SavedModel, ONNX, OpenVINO IR) for serving — these have a smaller attack surface than `.keras` files with arbitrary Python

---

## 8. Developer Internals

### 8.1 API Generation System

The public API in `keras/api/` is **entirely auto-generated** — never edit files in this directory manually. The generation system works as follows:

1. Developers annotate classes and functions with `@keras_export("keras.path.to.symbol")` (defined in `keras/src/api_export.py`)
2. The `api_gen.py` script copies source to a build directory and runs `namex.generate_api_files()`, which reads the export decorators and generates the `keras/api/` tree
3. A `_tf_keras/` compatibility shim directory is created for backward compatibility with `tf.keras` imports
4. The pre-commit hook (`.pre-commit-config.yaml`) runs `shell/api_gen.sh` automatically on every commit, followed by `ruff` for linting/formatting

To regenerate the API manually:

```bash
bash shell/api_gen.sh
```

> [!CAUTION]
> Any manual edits to files under `keras/api/` will be silently overwritten by the next API generation run (either via pre-commit hook or manual `api_gen.sh` execution). Always edit the source files under `keras/src/` and add `@keras_export` decorators.

### 8.2 Adding a New Layer

1. Create the implementation file in the appropriate `keras/src/layers/` subdirectory
2. Implement `__init__`, `build`, `call`, `get_config`
3. Add the `@keras_export("keras.layers.YourLayer")` decorator
4. Write tests in a `*_test.py` file alongside the implementation
5. Run `bash shell/api_gen.sh` to regenerate the public API
6. The pre-commit hook will also handle this automatically

### 8.3 Adding a New Op

1. Add the symbolic class and function to the appropriate ops module (`keras/src/ops/numpy.py`, `nn.py`, etc.)
2. Implement the backend-specific version in each backend's corresponding module
3. Add `@keras_export` decorator
4. Write tests covering all backends

### 8.4 Testing

Keras uses pytest with custom markers defined in `conftest.py`:

```bash
# Run all tests
pytest

# Run tests for a specific module
pytest keras/src/layers/core/dense_test.py

# Run only tests that don't require a trainable backend
pytest -m "not requires_trainable_backend"

# Run multi-device tests
pytest -m "multi_device"
```

**Test configuration (`conftest.py`):**

- Registers pytest markers: `requires_trainable_backend`, `multi_device`
- Disables TF32 for numerical accuracy in tests (TF32 introduces small numerical differences that cause test failures)
- Handles OpenVINO-excluded tests

Tests are colocated with source — every `foo.py` has a corresponding `foo_test.py` in the same directory.

**Integration tests** (`integration_tests/`):

- Basic flow validation
- Custom training loops per backend (JAX, TF, Torch)
- Model visualization
- PyTorch export
- Dataset tests

**Benchmarks** (`benchmarks/`):

- Layer benchmarks
- Model benchmarks (BERT, image classification)
- Torch CTL (Custom Training Loop) benchmarks

### 8.5 Code Formatting

Keras uses `ruff` for both linting and formatting:

```bash
bash shell/format.sh
```

The pre-commit hook runs `ruff` automatically after API generation.

### 8.6 Key Internal Classes

**`Variable`** (`keras/src/backend/common/variables.py`): Wraps backend-specific tensor variables. Handles dtype casting, naming, and trainable/non-trainable distinction.

**`KerasTensor`** (`keras/src/backend/common/keras_tensor.py`): Symbolic tensor representation used during model building (before eager execution). Carries shape and dtype metadata for graph construction.

**`AutocastScope`**: Context manager that temporarily overrides the dtype policy, enabling mixed-precision scoping within specific model sections.

**`StatelessScope`**: Context manager for stateless execution — variables are provided as explicit arguments rather than read from the layer's internal state. Used by JAX's functional paradigm.

**`Tracker`** (`keras/src/utils/tracking.py`): Monitors attribute assignments on `Layer` objects. When you assign a `Variable`, `Layer`, `Metric`, or `Optimizer` to a layer attribute, `Tracker` automatically registers it in the appropriate collection (weights, layers, metrics, etc.).

**`Config`**: Frozen dict-like object used for layer configuration snapshots. Immutable after creation.

### 8.7 The `__new__` Pattern in Model

`Model.__new__` is a critical piece of the multi-backend architecture. When you write `model = Model(...)`, `__new__` inspects `keras.backend.backend()` and returns an instance of the appropriate backend-specific Trainer subclass. This means `type(model)` is not `Model` but rather `JAXTrainer`, `TFTrainer`, `TorchTrainer`, `NumPyTrainer`, or `OpenVINOTrainer`, depending on the active backend. All of these are transparent to user code since they share the `Model` interface.

### 8.8 Tree Utilities

`keras/src/tree/` provides a unified interface for traversing nested Python structures (lists, tuples, dicts). It wraps backend-specific implementations:

- `dmtree` (Google DeepMind's tree library)
- `optree` (JAX ecosystem)
- `torch.utils._pytree` (PyTorch)

These are used extensively in the trainer and ops systems for handling nested inputs/outputs.

---

## 9. Integration Points

### 9.1 Backend Interoperability

Keras models can interoperate with native backend code:

**JAX:**

```python
# Use JAX transformations on Keras models
import jax

@jax.jit
def predict(params, x):
    return model.stateless_call(params, {}, x)[0]

# Use JAX layers in Keras via JaxLayer/FlaxLayer
from keras.wrappers import JaxLayer, FlaxLayer
```

**PyTorch:**

```python
# Wrap a torch.nn.Module as a Keras layer
from keras.wrappers import TorchModuleWrapper
keras_layer = TorchModuleWrapper(torch_module)

# Keras models are torch.nn.Module subclasses when using torch backend
# Standard PyTorch training loops work directly
```

**TensorFlow:**

```python
# Load TF SavedModel as a Keras layer
from keras.export import TFSMLayer
layer = TFSMLayer("path/to/saved_model")

# Use tf.data.Dataset directly with model.fit()
model.fit(tf_dataset, epochs=10)
```

### 9.2 Scikit-learn Integration

`keras/src/wrappers/` provides scikit-learn compatible wrappers:

```python
from keras.wrappers import SKLearnClassifier, SKLearnRegressor

# Use with sklearn pipelines, cross-validation, grid search
from sklearn.model_selection import cross_val_score
estimator = SKLearnClassifier(build_fn=create_model)
scores = cross_val_score(estimator, X, y, cv=5)
```

### 9.3 Data Format Compatibility

Keras accepts data from multiple ecosystems without conversion:

| Input Format | Adapter | Backend Affinity |
|-------------|---------|-----------------|
| NumPy arrays | `ArrayDataAdapter` | All |
| Python lists | `ArrayDataAdapter` | All |
| `tf.data.Dataset` | `TFDatasetAdapter` | TensorFlow |
| `torch.utils.data.DataLoader` | `TorchDataLoaderAdapter` | PyTorch |
| `keras.utils.PyDataset` | `PyDatasetAdapter` | All |
| Python generators | `GeneratorDataAdapter` | All |
| Grain datasets | `GrainDatasetAdapter` | JAX |

### 9.4 Pre-trained Model Ecosystem

Keras includes pre-trained architectures in `keras/src/applications/`:

- **Classification**: VGG16/19, ResNet50/101/152, InceptionV3, EfficientNetV2, MobileNetV2/V3, DenseNet, NASNet, ConvNeXt, Xception
- **Feature extraction**: Use `include_top=False` to get feature maps

All weights are downloaded from the Keras model repository and cached in `KERAS_HOME/models/`.

---

## Troubleshooting

### Error: `RuntimeError: Backend cannot be changed after import`

**Cause:** `KERAS_BACKEND` was set or `keras.config.set_backend()` was called after `import keras`.

**Solution:**
1. Set the environment variable before any Keras import:
   ```python
   import os
   os.environ["KERAS_BACKEND"] = "jax"
   import keras
   ```
2. Or set it in the shell before running your script:
   ```bash
   KERAS_BACKEND=jax python train.py
   ```

**Prevention:** Always set backend configuration before importing Keras. Use shell-level environment variables in production scripts.

### Error: `ValueError: Unknown backend: 'xyz'`

**Cause:** Invalid value for `KERAS_BACKEND` or `backend` in `keras.json`.

**Solution:**
1. Check your `~/.keras/keras.json` file for typos
2. Valid values: `tensorflow`, `jax`, `torch`, `numpy`, `openvino`

**Prevention:** Validate backend names against the five supported options.

### Error: `NotImplementedError` during `model.fit()` with NumPy/OpenVINO backend

**Cause:** NumPy and OpenVINO are inference-only backends. Training operations are not implemented.

**Solution:**
1. Switch to a trainable backend (TensorFlow, JAX, or PyTorch) for training
2. Use NumPy/OpenVINO only for `model.predict()` or direct `model()` calls

**Prevention:** Choose a trainable backend for any workflow involving `fit()`, `train_on_batch()`, or gradient computation.

### Error: `ImportError: Lambda layers require unsafe deserialization`

**Cause:** Loading a model containing `Lambda` layers without enabling unsafe deserialization.

**Solution:**
1. If you trust the model source:
   ```python
   keras.config.enable_unsafe_deserialization()
   model = keras.models.load_model("model.keras")
   ```
2. If you do not trust the source, inspect the `.keras` file first:
   ```bash
   unzip -p model.keras config.json | python -m json.tool
   ```
   Look for `Lambda` or custom class references.

**Prevention:** Avoid `Lambda` layers in production models. Use named layers or subclassing instead.

### Error: Gradient divergence between backends (e.g., TF vs Torch)

**Cause:** Numerical differences in gradient computation across backends. GitHub issue #22695 documents this for `Conv1D + Flatten` with `channels_first`. Issue #22697 reports similar divergence in recurrent WDCG.

**Solution:**
1. Ensure `image_data_format` is consistent across backends
2. Test numerical equivalence with `channels_last` format (better cross-backend consistency)
3. Increase `epsilon` if losses involve division near zero
4. Compare outputs layer-by-layer to isolate the divergent operation

**Prevention:** Run cross-backend numerical tests for critical models. Use `channels_last` as the default format for maximum compatibility.

### Error: `MelSpectrogram` validation failure with symbolic inputs

**Cause:** GitHub issue #22535 — `fft_length` validation fails when input shapes are symbolic (during model building).

**Solution:**
1. Provide explicit `input_shape` to the layer to avoid symbolic shape inference
2. Check for updates in Keras releases that address this issue

**Prevention:** Pin to a Keras version where the fix is available. Test preprocessing layers with both symbolic and eager inputs.

### Error: `IntegerLookup` OOV error with `num_oov_indices=0`

**Cause:** GitHub issue #22521 — validation logic incorrectly handles the case where `num_oov_indices` is set to 0.

**Solution:**
1. Set `num_oov_indices=1` (default) unless you specifically need to disable OOV handling
2. Filter input data to ensure all values are in the vocabulary

**Prevention:** Test preprocessing layers with edge-case configurations before deploying.

---

## Appendix A: Repository Structure

```
keras-team/keras/
├── keras/
│   ├── api/                    # AUTO-GENERATED public API (do not edit)
│   │   ├── activations/
│   │   ├── applications/
│   │   ├── backend/
│   │   ├── callbacks/
│   │   ├── config/
│   │   ├── constraints/
│   │   ├── datasets/
│   │   ├── distillation/
│   │   ├── distribution/
│   │   ├── dtype_policies/
│   │   ├── export/
│   │   ├── initializers/
│   │   ├── layers/
│   │   ├── losses/
│   │   ├── metrics/
│   │   ├── mixed_precision/
│   │   ├── models/
│   │   ├── ops/
│   │   ├── optimizers/
│   │   ├── preprocessing/
│   │   ├── quantizers/
│   │   ├── random/
│   │   ├── regularizers/
│   │   ├── saving/
│   │   ├── tree/
│   │   ├── utils/
│   │   ├── visualization/
│   │   ├── wrappers/
│   │   └── _tf_keras/         # TF-Keras compatibility shim
│   └── src/                    # Source implementation
│       ├── version.py          # __version__ = "3.15.0"
│       ├── api_export.py       # @keras_export decorator
│       ├── backend/            # Multi-backend dispatch
│       │   ├── config.py       # Configuration globals
│       │   ├── common/         # KerasTensor, Variable, dtypes
│       │   ├── jax/
│       │   ├── tensorflow/
│       │   ├── torch/
│       │   ├── numpy/
│       │   └── openvino/
│       ├── callbacks/
│       ├── distillation/
│       ├── distribution/
│       ├── dtype_policies/
│       ├── export/
│       ├── layers/
│       │   ├── layer.py        # Base Layer class
│       │   ├── core/           # Dense, Embedding, Input
│       │   ├── convolutional/
│       │   ├── normalization/
│       │   ├── attention/
│       │   ├── rnn/
│       │   └── preprocessing/
│       ├── models/             # Sequential, Functional, Model
│       ├── ops/                # Backend-agnostic operations
│       ├── optimizers/
│       ├── quantizers/         # AWQ, GPTQ, int8/int4/float8
│       ├── saving/             # .keras format, serialization
│       ├── trainers/           # Training loop, data adapters
│       ├── utils/
│       ├── visualization/
│       └── wrappers/           # sklearn, torch wrappers
├── integration_tests/
├── benchmarks/
├── shell/                      # api_gen.sh, format.sh
├── api_gen.py
├── pip_build.py
├── conftest.py
├── pyproject.toml
└── requirements*.txt           # Backend-specific requirements
```

## Appendix B: Complete Configuration Quick Reference

```json
{
    "floatx": "float32",
    "epsilon": 1e-7,
    "backend": "tensorflow",
    "image_data_format": "channels_last",
    "nnx_enabled": false
}
```

| Setting | Values | Impact |
|---------|--------|--------|
| `floatx` | `float16`, `float32`, `float64` | Default dtype for all weights |
| `epsilon` | Any small float | Numerical stability floor |
| `backend` | `tensorflow`, `jax`, `torch`, `numpy`, `openvino` | Computation engine (immutable after import) |
| `image_data_format` | `channels_last`, `channels_first` | Tensor layout for image operations |
| `nnx_enabled` | `true`, `false` | Flax NNX features (JAX backend only) |

---

_Generated May 2026 from keras-team/keras at b800fdf. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._