Pathway Administrator & Developer Guide

1. Architecture Overview

Pathway is a Python framework for building real-time data processing pipelines, powered by a high-performance Rust computation engine rooted in differential dataflow and timely dataflow paradigms. The system is a hybrid Python+Rust architecture bridged via PyO3, designed for streaming-first incremental computation.

Layered Architecture

Rust Crate Structure

src/
├── lib.rs               # Crate root — module declarations + jemalloc setup
├── async_runtime.rs     # Tokio runtime creation (rt-multi-thread)
├── deepcopy.rs          # S3Bucket deep copy utilities
├── env.rs               # Environment variable parsing
├── fs_helpers.rs        # Directory management
├── mat_mul.rs           # N-dimensional matrix multiplication (ndarray)
├── pipe.rs              # Named pipe support
├── retry.rs             # Retry logic with backoff
├── timestamp.rs         # Timestamp handling
├── connectors/
│   ├── mod.rs           # Connector orchestration
│   ├── data_lake/       # Delta Lake, Iceberg, Arrow integration
│   ├── scanner/         # File system scanning
│   ├── aws/             # S3, DynamoDB, Kinesis
│   └── metadata/        # Connector metadata
├── engine/
│   ├── dataflow/        # Differential/timely dataflow operators
│   │   └── operators/   # Individual operator implementations
│   └── telemetry/       # OpenTelemetry integration
├── persistence/
│   └── backends/        # Filesystem, S3 backends (5 files, 17K chars)
├── external_integration/
└── python_api/          # PyO3 bindings

Important

The computation engine uses vendorized forks of differential-dataflow and timely-dataflow (located in external/). These are not upstream crates — they contain Pathway-specific modifications. Do not replace them with upstream versions.

Python Package Structure

python/pathway/
├── __init__.py          # Main namespace exports
├── __main__.py          # Entry point → pathway.cli.main
├── cli.py               # Click CLI (spawn, replay, web_dashboard, airbyte)
├── _engine_finder.py    # MagicCargoFinder for dev builds
├── internals/           # Core API (Table, Schema, expressions, joins)
├── io/                  # 35+ connector modules
├── stdlib/              # Standard library (temporal, graphs, ML, stats)
├── xpacks/              # Extension packs (LLM/RAG, SharePoint)
├── persistence/         # State persistence Python API
├── demo/                # Demo stream generators
├── debug/               # Debug utilities
└── web_dashboard/       # Monitoring dashboard (frontend + backend)

Data Flow

Input Connectors → Python API (Table operations) → Rust Engine (differential dataflow) → Output Connectors
                                                         ↕
                                                   Persistence Layer

All table operations are compiled into a dataflow graph executed by the Rust engine. The engine processes data incrementally — only changes (diffs) propagate through the graph, making it efficient for streaming workloads.

2. System Requirements

Runtime Requirements

Build Requirements (from source)

Installation

pip install -U pathway

Note

The pip package ships a pre-compiled Rust engine. Building from source requires the full Rust 1.93 toolchain and CMake for the rdkafka dependency (ssl-vendored, zstd, curl features).

Memory Allocator

Pathway uses jemalloc by default (via the jemallocator crate 0.5.4). This provides better performance for the allocation patterns in dataflow computation. To override:

• Compile with the standard-allocator Cargo feature to use the system allocator instead.

3. Configuration Reference

Environment Variables

Core Runtime

Build-time Variables

AWS/Cloud Credentials

Debugging

Licensing Configuration

import pathway as pw

# Set license key programmatically
pw.set_license_key("your-license-key-here")

# Or via environment variable
# export PATHWAY_LICENSE_KEY=your-license-key-here

Entitlement checks are performed by _check_entitlements(). For example, the "multiple-machines" entitlement requires Scale tier or above.

Persistence Configuration

import pathway as pw

# Filesystem backend
persistence_config = pw.persistence.Config(
    backend=pw.persistence.Backend.filesystem(path="/data/checkpoints")
)

# S3 backend
persistence_config = pw.persistence.Config(
    backend=pw.persistence.Backend.s3(
        bucket="my-bucket",
        root_path="pathway-state/"
    )
)

YAML Pipeline Configuration

Pathway supports YAML-based pipeline definitions loaded via pw.load_yaml():

# Custom YAML tags supported:
# !pw.schema_from_types — define schemas inline
# !pw.io.kafka.read — Kafka source
# !pw.io.postgres.write — PostgreSQL sink
# !pw.persistence.Config — persistence configuration
# !pw.persistence.Backend.filesystem — filesystem backend

# Environment variable interpolation:
kafka_brokers: $KAFKA_BROKERS

Tip

YAML configurations support environment variable interpolation using $VAR_NAME syntax. This allows the same YAML file to work across environments without modification.

Monitoring Configuration

import pathway as pw

# Set monitoring level
pw.set_monitoring_config(level=pw.MonitoringLevel.ALL)

The monitoring HTTP endpoint is exposed on the port specified by PATHWAY_MONITORING_HTTP_PORT. The web dashboard runs separately on port 8088 via pathway web_dashboard.

4. Deployment & Scaling

Single-Process Deployment

# Basic execution
python my_pipeline.py

# With persistence
PATHWAY_PERSISTENCE_MODE=... python my_pipeline.py

Multi-Process/Thread Deployment via CLI

The pathway spawn command manages multi-process and multi-thread execution:

# Multi-threaded (single process, multiple threads)
pathway spawn --threads 4 python my_pipeline.py

# Multi-process
pathway spawn --processes 4 python my_pipeline.py

# Combined
pathway spawn --processes 2 --threads 4 python my_pipeline.py

Auto-Scaling Mechanism

The spawn system supports dynamic scaling via process exit codes:

Scaling factors:

• UPSCALING_FACTOR = 2.0 — doubles worker count on upscale signal
• DOWNSCALING_FACTOR = 0.5 — halves worker count on downscale signal

Warning

The multi-machine feature (PATHWAY_ADDRESSES) requires a Scale or Enterprise license. The _check_entitlements() function enforces this at runtime. Community tier is limited to single-machine deployment.

Multi-Machine Deployment

For distributed execution across multiple hosts:

1. Set PATHWAY_ADDRESSES to a list of all participating machine addresses
2. Set PATHWAY_FIRST_PORT for inter-process communication
3. Each machine gets a unique PATHWAY_PROCESS_ID
4. Requires Scale license ("multiple-machines" entitlement)

The pathway spawn_from_env command reads configuration from PATHWAY_SPAWN_ARGS (JSON), useful in orchestrated/containerized deployments.

Cargo Build Profiles

Note

The dev profile uses opt-level = 3 (full optimization), which is unusual. This means development builds are nearly as fast as release builds, but compile times are longer.

Cargo Features

5. Operations

CLI Reference

Monitoring Stack

Pathway integrates with multiple observability systems:

OpenTelemetry (traces + metrics):

• Uses OTLP/gRPC export
• opentelemetry 0.30.0 with SDK
• opentelemetry-otlp for export
• Configured in src/engine/telemetry/

Prometheus:

• prometheus-client 0.23.1 for metrics export
• Exposes standard process and pipeline metrics

Web Dashboard:

• Built-in monitoring UI at port 8088
• Start with pathway web_dashboard
• Frontend + backend in python/pathway/web_dashboard/

HTTP Monitoring Endpoint:

• Configured via PATHWAY_MONITORING_HTTP_PORT
• Provides runtime metrics and health information

Persistence Operations

The persistence system supports:

• Recording: Capture connector data and state for replay
• Replay: Re-execute pipelines from recorded state
• Checkpointing: Periodic state snapshots for recovery
• Backends: Filesystem (local disk) and S3

State is stored in configurable paths. The PATHWAY_SNAPSHOT_ACCESS variable controls whether the system is recording new snapshots or replaying existing ones.

Running Pipelines

import pathway as pw

# Define pipeline...

# Execute (blocking)
pw.run()

# Execute all registered pipelines
pw.run_all()

Tip

Use pw.run() for single-pipeline scripts. pw.run_all() executes all pipelines that have been constructed in the current process, useful when composing multiple independent dataflows.

6. Performance

Memory Management

• Default allocator: jemalloc (via jemallocator 0.5.4)
• jemalloc provides better fragmentation behavior for dataflow workloads
• Override with standard-allocator feature if profiling shows jemalloc overhead

Engine Optimization

The Rust engine leverages:

• Differential dataflow: Only processes changes (diffs), not full datasets
• Timely dataflow: Parallel, pipelined execution of operators
• Incremental computation: Maintains operator state; re-computes only affected outputs

Concurrency Model

• Tokio (1.45.1, rt-multi-thread): Async runtime for I/O-bound connectors
• Timely dataflow workers: CPU-bound computation threads
• Separation: I/O connectors use async Tokio tasks; computation uses timely workers

Connector Performance

High-throughput connectors use native Rust implementations:

• rdkafka 0.39.0 (librdkafka via cmake-build) for Kafka
• Native S3, PostgreSQL, MongoDB clients
• Bulk operations where supported (Elasticsearch bulk API, etc.)

Known Performance Issues

• GitHub Issue #21: 1M row processing takes >10 minutes (open issue)

Caution

The PATHWAY_YOLO_RARE_WAKEUPS environment variable reduces wakeup frequency to lower CPU usage in idle periods, but may increase latency for processing new data. Only enable in scenarios where latency tolerance is high.

7. Security

TLS Configuration

The TLSSettings class (in python/pathway/internals/_io_helpers.py) provides comprehensive TLS configuration:

from pathway.internals._io_helpers import TLSSettings

tls = TLSSettings(
    mode="verify-full",         # disable|allow|prefer|require|verify-ca|verify-full
    root_cert_path="/certs/ca.pem",
    client_cert_path="/certs/client.pem",
    client_key_path="/certs/client.key",
    trust_certificates=True
)

Schema Registry Authentication

The Schema Registry connector supports multiple auth methods:

• Token-based authentication
• Username/password (Basic auth)
• Custom headers
• HTTP proxy configuration

S3/Cloud Credentials

from pathway.internals._io_helpers import AwsS3Settings

s3 = AwsS3Settings(
    access_key="...",
    secret_access_key="...",
    session_token="...",           # Optional, for temporary credentials
    endpoint="...",                # Custom endpoint (MinIO, etc.)
    path_style=True               # Path-style access (for MinIO)
)

Warning

Never hardcode credentials in pipeline code. Use environment variables (AWS_S3_ACCESS_KEY, AWS_S3_SECRET_ACCESS_KEY, AWS_PROFILE) or instance roles for cloud deployments.

License Key Security

The PATHWAY_LICENSE_KEY grants access to Scale/Enterprise features. Treat it as a secret:

• Set via environment variable, not in source code
• The pw.set_license_key() API is for interactive/notebook use only

8. Developer Internals

PyO3 Bridge Architecture

The Python↔Rust bridge uses PyO3 0.25.0 with abi3-py310 (stable ABI from Python 3.10+):

• Binding location: src/python_api/ (17K chars)
• ABI stability: Single compiled extension works across Python 3.10, 3.11, 3.12, etc.
• Dev builds: _engine_finder.py implements MagicCargoFinder to locate dev-built Rust libraries

Core Python Internals

Design Patterns

Schema-First Design

All tables require explicit schemas:

import pathway as pw

class SensorReading(pw.Schema):
    sensor_id: str
    timestamp: int
    value: float
    metadata: dict

Schemas are Python classes inheriting from pw.Schema with typed attributes. The framework enforces type safety at the dataflow boundary.

Connector Subject/Observer Pattern

Connectors follow a unified read/write interface:

# Reading (subject — produces data)
table = pw.io.kafka.read(
    rdkafka_settings={"bootstrap.servers": "localhost:9092"},
    topic="input-topic",
    schema=SensorReading
)

# Writing (observer — consumes data)
pw.io.kafka.write(
    table,
    rdkafka_settings={"bootstrap.servers": "localhost:9092"},
    topic_name="output-topic"
)

UDF Decorator Pattern

@pw.udf
def normalize(value: float, max_val: float) -> float:
    return value / max_val if max_val != 0 else 0.0

# Used in table operations
result = table.select(normalized=normalize(pw.this.value, pw.this.max_value))

AsyncTransformer Pattern

For async operations (LLM calls, HTTP requests):

from pathway.stdlib.utils import AsyncTransformer

class LLMTransformer(AsyncTransformer):
    async def invoke(self, row):
        response = await call_llm(row.prompt)
        return {"result": response}

Custom Reducers

from pathway.internals.custom_reducers import BaseCustomAccumulator

class RunningMean(BaseCustomAccumulator):
    def __init__(self):
        self.sum = 0.0
        self.count = 0

    def update(self, value):
        self.sum += value
        self.count += 1

    def compute(self):
        return self.sum / self.count if self.count > 0 else 0.0

Rust Engine Internals

The engine (src/engine/, 515K chars) implements:

• Dataflow operators: Map, filter, reduce, join, temporal operations
• Operator fusion: Combines compatible operators for efficiency
• Progress tracking: Timely dataflow frontier tracking for correctness
• Worker coordination: Multi-worker synchronization via timely infrastructure

The src/engine/dataflow/operators/ directory contains individual operator implementations that map to Python API operations.

Vendorized External Dependencies

Located in external/:

• differential-dataflow: Incremental computation framework (Pathway fork)
• timely-dataflow: Parallel dataflow computation (Pathway fork with timely/ subpath)

These forks enable Pathway-specific optimizations and fixes that are not in upstream repositories.

9. Integration Points

Connector Inventory

Input Connectors (Python pw.io.*)

Output/Sink Connectors

Most input connectors have corresponding write methods. Notable Rust-backed sinks:

• Kafka (rdkafka)
• PostgreSQL (postgres + pg_walstream for WAL streaming)
• Elasticsearch (bulk API)
• S3/Delta Lake/Iceberg

Note

PostgreSQL WAL streaming (pg_walstream 0.6.1) enables CDC (Change Data Capture) from PostgreSQL without polling. This is the recommended input method for PostgreSQL sources with real-time requirements.

Vector Database Integration

Full-Text Search

• Tantivy 0.22.1 — Embedded full-text search engine used for indexing operations within Pathway pipelines.

LLM/RAG Extension Pack

Located in python/pathway/xpacks/llm/:

This xpack enables building RAG (Retrieval-Augmented Generation) pipelines natively within Pathway's streaming framework.

Standard Library

Schema Registry Integration

Confluent Schema Registry support via schema_registry_converter 4.4.0:

from pathway.internals._io_helpers import SchemaRegistrySettings

registry = SchemaRegistrySettings(
    url="https://schema-registry:8081",
    # Auth options:
    token="...",
    # OR
    username="...",
    password="...",
    # OR
    custom_headers={"Authorization": "Bearer ..."},
    proxy="http://proxy:8080"
)

Testing Infrastructure

Test Utilities (pathway.tests.utils):

• assert_equal_tables — Compare table contents with index
• assert_equal_tables_wo_index — Compare table contents ignoring index

Integration tests require running instances of:

• PostgreSQL, MongoDB, MySQL, MSSQL
• Kafka, NATS, RabbitMQ
• S3/MinIO, DynamoDB (LocalStack)
• Milvus, Elasticsearch

Important

Integration tests connect to real services. Ensure Docker containers or equivalent test infrastructure are running before executing the integration test suite. Fixtures in conftest.py manage connection parameters.

Open Feature Requests

Tip

For production deployments requiring features listed above, check the GitHub issue tracker for current status. Several of these (SQLite output, MySQL input) have partial implementations in the codebase that may be usable with custom wrappers.

Appendix: Complete Rust Dependency Map

Core Computation

Python Interop

Async/Networking

Data Sources/Sinks

Search/Indexing

Observability

Data Formats

Memory

Appendix: Project Metadata