# Polars User Guide

Polars is an extremely fast query engine for **DataFrames**, written in Rust. It is designed to be fast, easy to use, and expressive. You write data transformations as **expressions**, and Polars compiles and runs them across all your CPU cores — eagerly for interactive work, or lazily so the query optimizer can rewrite the plan before a single row is read.

This guide is task-oriented. It takes you from installation to your first DataFrame, through every everyday operation (selecting, filtering, aggregating, joining), into the expression API that makes Polars powerful, and finally into the lazy and streaming engines that let you process datasets larger than memory. Concepts are explained as you need them; deep dives live near the end.

> [!NOTE]
> Examples in this guide use the Python API (`import polars as pl`), the most widely used Polars frontend. Polars also has first-class Rust, Node.js, R, and SQL frontends; the **Rust** equivalent is shown where it differs meaningfully. The underlying engine is identical across all of them.

---

## Table of Contents

1. [Quick Start](#1-quick-start)
2. [Installation](#2-installation)
3. [Core Ideas: Expressions, Contexts, Eager vs. Lazy](#3-core-ideas-expressions-contexts-eager-vs-lazy)
4. [Reading and Writing Data](#4-reading-and-writing-data)
5. [Selecting and Creating Columns](#5-selecting-and-creating-columns)
6. [Filtering Rows](#6-filtering-rows)
7. [Aggregating and Grouping](#7-aggregating-and-grouping)
8. [Sorting, Reshaping, and Combining](#8-sorting-reshaping-and-combining)
9. [Joining DataFrames](#9-joining-dataframes)
10. [The Expression API in Depth](#10-the-expression-api-in-depth)
11. [Working with Specific Data Types](#11-working-with-specific-data-types)
12. [The Lazy API](#12-the-lazy-api)
13. [Streaming: Larger-Than-RAM Data](#13-streaming-larger-than-ram-data)
14. [Querying with SQL](#14-querying-with-sql)
15. [Interoperability (pandas, NumPy, Arrow)](#15-interoperability-pandas-numpy-arrow)
16. [Controlling Display and Output](#16-controlling-display-and-output)
17. [How It Works](#17-how-it-works)
18. [Troubleshooting](#18-troubleshooting)
19. [Frequently Asked Questions](#19-frequently-asked-questions)
20. [Reference](#20-reference)

---

## 1. Quick Start

Install Polars and run your first query in under a minute.

=== "Python"

    ```bash
    pip install polars
    ```

=== "Rust"

    ```bash
    cargo add polars -F lazy
    ```

Create a DataFrame, transform it, and inspect the result:

```python
import polars as pl
import datetime as dt

df = pl.DataFrame(
    {
        "name": ["Alice Archer", "Ben Brown", "Chloe Cooper", "Daniel Donovan"],
        "birthdate": [
            dt.date(1997, 1, 10),
            dt.date(1985, 2, 15),
            dt.date(1983, 3, 22),
            dt.date(1981, 4, 30),
        ],
        "weight": [57.9, 72.5, 53.6, 83.1],  # (kg)
        "height": [1.56, 1.77, 1.65, 1.75],  # (m)
    }
)

print(df)
```

```text
shape: (4, 4)
┌────────────────┬────────────┬────────┬────────┐
│ name           ┆ birthdate  ┆ weight ┆ height │
│ ---            ┆ ---        ┆ ---    ┆ ---    │
│ str            ┆ date       ┆ f64    ┆ f64    │
╞════════════════╪════════════╪════════╪════════╡
│ Alice Archer   ┆ 1997-01-10 ┆ 57.9   ┆ 1.56   │
│ Ben Brown      ┆ 1985-02-15 ┆ 72.5   ┆ 1.77   │
│ Chloe Cooper   ┆ 1983-03-22 ┆ 53.6   ┆ 1.65   │
│ Daniel Donovan ┆ 1981-04-30 ┆ 83.1   ┆ 1.75   │
└────────────────┴────────────┴────────┴────────┘
```

Compute new columns with an expression:

```python
result = df.select(
    pl.col("name"),
    pl.col("birthdate").dt.year().alias("birth_year"),
    (pl.col("weight") / (pl.col("height") ** 2)).alias("bmi"),
)
print(result)
```

```text
shape: (4, 3)
┌────────────────┬────────────┬───────────┐
│ name           ┆ birth_year ┆ bmi       │
│ ---            ┆ ---        ┆ ---       │
│ str            ┆ i32        ┆ f64       │
╞════════════════╪════════════╪═══════════╡
│ Alice Archer   ┆ 1997       ┆ 23.791913 │
│ Ben Brown      ┆ 1985       ┆ 23.141498 │
│ Chloe Cooper   ┆ 1983       ┆ 19.687787 │
│ Daniel Donovan ┆ 1981       ┆ 27.134694 │
└────────────────┴────────────┴───────────┘
```

Write to disk and read it back:

```python
df.write_csv("people.csv")
df_csv = pl.read_csv("people.csv", try_parse_dates=True)
```

> [!TIP]
> The single most important idea in Polars is the **expression**. `pl.col("weight") / (pl.col("height") ** 2)` is not a value — it is a *recipe* that Polars runs against a column, in parallel, inside a context like `select`. Master expressions and the rest of the API follows naturally. See [Core Ideas](#3-core-ideas-expressions-contexts-eager-vs-lazy).

---

## 2. Installation

Polars installs through your language's package manager. The core has **zero required dependencies**; optional features are opt-in to keep the footprint small.

=== "Python"

    ```bash
    pip install polars
    ```

=== "Rust"

    ```bash
    cargo add polars -F lazy
    ```

    Or in `Cargo.toml`:

    ```toml
    [dependencies]
    polars = { version = "x", features = ["lazy"] }
    ```

Then import it:

=== "Python"

    ```python
    import polars as pl
    ```

=== "Rust"

    ```rust
    use polars::prelude::*;
    ```

### Python optional dependencies (extras)

Install extras with bracket syntax, e.g. `pip install 'polars[numpy,fsspec]'`.

| Tag | Enables |
| --- | --- |
| `all` | Every optional dependency below. |
| `gpu` | Run queries on NVIDIA GPUs (see [GPU support](#gpu-acceleration)). |
| `pandas` | Convert to/from pandas DataFrames and Series. |
| `numpy` | Convert to/from NumPy arrays. |
| `pyarrow` | Convert to/from PyArrow tables and arrays. |
| `pydantic` | Build Polars data from Pydantic models. |
| `calamine` | Read Excel via the calamine engine (fastest). |
| `openpyxl` | Read Excel via openpyxl. |
| `xlsx2csv` | Read Excel via xlsx2csv. |
| `xlsxwriter` | Write Excel via XlsxWriter. |
| `excel` | All supported Excel engines. |
| `adbc` | Read/write databases via Arrow Database Connectivity. |
| `connectorx` | Read databases via ConnectorX. |
| `sqlalchemy` | Write databases via SQLAlchemy. |
| `database` | All supported database engines. |
| `fsspec` | Read/write remote filesystems (S3, GCS, Azure, …). |
| `deltalake` | Read/write Delta Lake tables. |
| `iceberg` | Read Apache Iceberg tables. |
| `async` | Collect LazyFrames asynchronously. |
| `cloudpickle` | Serialize user-defined Python functions. |
| `graph` | Visualize LazyFrames as a graph (`.show_graph()`). |
| `plot` | Plot DataFrames through the `.plot` namespace. |
| `style` | Style DataFrames through the `.style` namespace. |
| `timezone` | Timezone support (only needed on Windows). |

> [!NOTE]
> Throughout this guide, when a feature needs an extra we say so. If you are unsure, `pip install 'polars[all]'` installs everything; trim it down later for a leaner environment.

### Big index builds (>4.3 billion rows)

By default a Polars DataFrame is limited to 2³² (~4.3 billion) rows. If you expect more, install the big-index build, which uses a 64-bit row index (2⁶⁴ ≈ 18 quintillion rows):

=== "Python"

    ```bash
    pip install polars[rt64]
    ```

=== "Rust"

    ```bash
    cargo add polars -F bigidx
    ```

> [!CAUTION]
> Only use `bigidx`/`rt64` if you actually hit the 2³² row boundary. The default build is faster and uses less memory because its data structures are more cache-efficient.

### Legacy CPUs

The default Python wheels are compiled for modern CPUs (AVX2). On an older CPU (roughly pre-2011), or on an x86-64 Python running under Rosetta on Apple Silicon, install the compatibility build:

```bash
pip install polars[rtcompat]
```

### Compiling from source (Python)

For a bleeding-edge build or maximum performance, compile against your own CPU:

1. Install the latest [Rust compiler](https://www.rust-lang.org/tools/install).
2. `pip install maturin`
3. `cd py-polars` and run one of:

| Command | Result |
| --- | --- |
| `make build` | Slow binary with debug assertions; fast to compile. |
| `make build-release` | Fast binary, minimal debug symbols; long compile. |
| `make build-nodebug-release` | Like release, no debug symbols; slightly faster compile. |
| `make build-debug-release` | Like release, full debug symbols. |
| `make build-dist-release` | Fastest binary; extreme compile times. |

> [!TIP]
> Pass `LTS_CPU=1` with any build command to target an older CPU that lacks AVX2.

### Verifying your installation

```python
import polars as pl
pl.show_versions()
```

This prints the Polars version, the index type (32- or 64-bit), and the status of every optional dependency — the first thing to check when debugging an environment problem.

---

## 3. Core Ideas: Expressions, Contexts, Eager vs. Lazy

Three ideas underpin everything in Polars. Internalize them once and the entire API becomes predictable.

### Expressions

An **expression** is a description of a computation over one or more columns. It does nothing on its own:

```python
pl.col("weight") / (pl.col("height") ** 2)
```

The line above does not produce a number. It produces an `Expr` object — an abstract recipe meaning "take `weight`, divide it by `height` squared." Polars only materializes an expression when you run it inside a **context**.

### Contexts

A **context** is a DataFrame (or LazyFrame) method that takes expressions and applies them. The four contexts you will use constantly:

| Context | What it does |
| --- | --- |
| `select` | Choose / compute columns; result has only the produced columns. |
| `with_columns` | Add or replace columns; keeps all existing columns. |
| `filter` | Keep rows where a boolean expression is true. |
| `group_by(...).agg(...)` | Split into groups, then compute per-group expressions. |

```python
# select: only the computed columns come out
df.select(pl.col("name"), pl.col("weight") * 0.95)

# with_columns: original columns kept, new ones appended
df.with_columns(bmi=pl.col("weight") / pl.col("height") ** 2)

# filter: keep matching rows
df.filter(pl.col("height") > 1.7)

# group_by + agg: one row per group
df.group_by("name").agg(pl.col("weight").mean())
```

> [!IMPORTANT]
> Because expressions are independent recipes, Polars runs the expressions inside a context **in parallel**, across all CPU cores, automatically. You get multi-threaded execution without writing any threading code.

### Eager vs. Lazy execution

Polars has two execution modes that share the same expression API:

- **Eager** — methods on a `DataFrame` run immediately, line by line. Best for interactive exploration and small data.
- **Lazy** — methods on a `LazyFrame` build up a *query plan*. Nothing runs until you call `.collect()`. This lets the **query optimizer** rewrite your query (pushing filters down to the file reader, dropping unused columns, eliminating common subexpressions) before any data is touched.

```python
# Eager: executes now
df = pl.read_csv("people.csv")
result = df.filter(pl.col("height") > 1.7)

# Lazy: builds a plan, runs only at collect()
lf = pl.scan_csv("people.csv")
result = lf.filter(pl.col("height") > 1.7).collect()
```

> [!WARNING]
> A common surprise for newcomers: a lazy query produces **no output until you call `.collect()`**. If you build a `LazyFrame` chain and "nothing happens," you almost certainly forgot the final `.collect()`. The lazy API is covered fully in [The Lazy API](#12-the-lazy-api).

> [!TIP]
> Rule of thumb: explore eagerly with `read_*`, but run production pipelines lazily with `scan_*` so the optimizer and streaming engine can help you.

---

## 4. Reading and Writing Data

Polars reads and writes the common columnar and text formats, plus cloud storage and databases. Every format has an **eager reader** (`read_*`, returns a `DataFrame`) and most also have a **lazy scanner** (`scan_*`, returns a `LazyFrame` and enables predicate/projection pushdown).

### Format support matrix

| Format | Eager read | Lazy scan | Write | Extra required |
| --- | --- | --- | --- | --- |
| CSV | `read_csv`, `read_csv_batched` | `scan_csv` | `write_csv` | — |
| Parquet | `read_parquet` | `scan_parquet` | `write_parquet`, `sink_parquet` | — |
| JSON | `read_json` | — | `write_json` | — |
| NDJSON (line-delimited) | `read_ndjson` | `scan_ndjson` | `write_ndjson` | — |
| Arrow IPC / Feather | `read_ipc`, `read_ipc_stream` | `scan_ipc` | `write_ipc` | — |
| Avro | `read_avro` | — | `write_avro` | — |
| Excel | `read_excel`, `read_ods` | — | `write_excel` | `excel` (or a specific engine) |
| Delta Lake | `read_delta` | `scan_delta` | `write_delta` | `deltalake` |
| Apache Iceberg | — | `scan_iceberg` | — | `iceberg` |
| Database (SQL) | `read_database`, `read_database_uri` | — | `write_database` | `adbc`/`connectorx`/`sqlalchemy` |
| Clipboard | `read_clipboard` | — | `write_clipboard` | — |
| Plain text lines | `read_lines` | `scan_lines` | — | — |

### CSV

```python
# Read, asking Polars to detect and parse date columns
df = pl.read_csv("people.csv", try_parse_dates=True)

# Write
df.write_csv("out.csv")

# Lazy scan (preferred for large files)
lf = pl.scan_csv("people.csv")
```

Common `read_csv` parameters: `separator`, `has_header`, `columns` (subset), `schema_overrides`, `skip_rows`, `n_rows`, `null_values`, `encoding`, `infer_schema_length`.

### Parquet

Parquet is the recommended format for analytics: columnar, compressed, and self-describing. Always prefer `scan_parquet` for large datasets so Polars can prune row groups and columns.

```python
df = pl.read_parquet("data.parquet")
lf = pl.scan_parquet("data/*.parquet")        # globs work
df.write_parquet("data.parquet", compression="zstd")
```

You can read just the schema or metadata without loading rows:

```python
schema = pl.read_parquet_schema("data.parquet")
meta = pl.read_parquet_metadata("data.parquet")
```

### Reading many files at once

`scan_*` functions accept a glob or a list of paths and concatenate them into one frame:

```python
lf = pl.scan_parquet("year=2024/**/*.parquet")   # hive-partitioned tree
df = lf.collect()
```

### Cloud storage

With the `fsspec` extra (and the relevant cloud SDK), `read_*`/`scan_*` accept `s3://`, `gs://`, and `az://` URIs. For credentials, pass a `storage_options` dict or use one of the built-in credential providers (`CredentialProviderAWS`, `CredentialProviderAzure`, `CredentialProviderGCP`).

```python
lf = pl.scan_parquet(
    "s3://my-bucket/data/*.parquet",
    storage_options={"aws_region": "us-east-1"},
)
```

### Databases

```python
# Via a SQLAlchemy/ADBC connection or URI
df = pl.read_database_uri(
    query="SELECT * FROM users WHERE active",
    uri="postgresql://user:pass@host:5432/db",
)
df.write_database("users_copy", connection="postgresql://…")
```

> [!NOTE]
> `read_database` takes an existing connection object; `read_database_uri` takes a connection string and chooses an engine (ConnectorX by default, ADBC if requested). ConnectorX is fastest for reads; SQLAlchemy and ADBC support writes.

### Excel

```python
# Requires: pip install 'polars[calamine]'  (or another engine)
df = pl.read_excel("report.xlsx", sheet_name="Q1")
df.write_excel("out.xlsx", worksheet="Summary")
```

---

## 5. Selecting and Creating Columns

### `select` — pick and compute columns

```python
df.select("name", "weight")                       # by name
df.select(pl.col("name"), pl.col("weight") * 2)   # with expressions
df.select(pl.all())                               # everything
df.select(pl.all().exclude("birthdate"))          # everything except
```

### `with_columns` — add or replace columns

```python
df.with_columns(
    birth_year=pl.col("birthdate").dt.year(),
    bmi=pl.col("weight") / (pl.col("height") ** 2),
)
```

Named keyword arguments become column names. Equivalently, use `.alias()`:

```python
df.with_columns(
    pl.col("birthdate").dt.year().alias("birth_year"),
)
```

### Expression expansion — one expression, many columns

A single expression can target several columns at once and produce several output columns. This is **expression expansion** and is unique to Polars' design:

```python
result = df.select(
    pl.col("name"),
    (pl.col("weight", "height") * 0.95).round(2).name.suffix("-5%"),
)
```

Here `pl.col("weight", "height") * 0.95` expands into two computations, and `.name.suffix("-5%")` renames each result. Use selectors (next) for pattern-based expansion.

### Selectors — choose columns by property

The `polars.selectors` module (imported as `cs`) selects columns by dtype, name pattern, or position:

```python
import polars.selectors as cs

df.select(cs.numeric())               # all numeric columns
df.select(cs.string())                # all string columns
df.select(cs.starts_with("weight"))   # name pattern
df.select(cs.numeric() - cs.first())  # set algebra on selectors
```

> [!TIP]
> Selectors compose with set operators: `cs.numeric() & cs.matches("2024")`, `cs.all() - cs.string()`. They are the cleanest way to apply one transformation to a whole family of columns.

---

## 6. Filtering Rows

`filter` keeps rows where a boolean expression is true:

```python
df.filter(pl.col("birthdate").dt.year() < 1990)
```

Multiple conditions passed as separate arguments are combined with logical **AND**:

```python
df.filter(
    pl.col("birthdate").is_between(dt.date(1982, 12, 31), dt.date(1996, 1, 1)),
    pl.col("height") > 1.7,
)
```

For OR / NOT, combine boolean expressions explicitly:

```python
df.filter((pl.col("height") > 1.8) | (pl.col("weight") < 60))
df.filter(~pl.col("name").str.starts_with("A"))
```

Useful boolean expression methods: `is_in`, `is_between`, `is_null`, `is_not_null`, `is_nan`, `is_finite`, `is_duplicated`, `is_unique`.

```python
df.filter(pl.col("name").is_in(["Ben Brown", "Alice Archer"]))
```

---

## 7. Aggregating and Grouping

### Whole-frame aggregations

Aggregation expressions used in `select` collapse each column to one value:

```python
df.select(
    pl.col("weight").mean().alias("avg_weight"),
    pl.col("height").max().alias("tallest"),
    pl.len().alias("n_rows"),
)
```

### `group_by(...).agg(...)`

Split rows into groups, then compute per-group expressions:

```python
result = df.group_by(
    (pl.col("birthdate").dt.year() // 10 * 10).alias("decade"),
    maintain_order=True,
).agg(
    pl.len().alias("sample_size"),
    pl.col("weight").mean().round(2).alias("avg_weight"),
    pl.col("height").max().alias("tallest"),
)
```

```text
shape: (3, 4)
┌────────┬─────────────┬────────────┬─────────┐
│ decade ┆ sample_size ┆ avg_weight ┆ tallest │
│ ---    ┆ ---         ┆ ---        ┆ ---     │
│ i32    ┆ u32         ┆ f64        ┆ f64     │
╞════════╪═════════════╪════════════╪═════════╡
│ 1990   ┆ 1           ┆ 57.9       ┆ 1.56    │
│ 1980   ┆ 3           ┆ 69.73      ┆ 1.77    │
└────────┴─────────────┴────────────┴─────────┘
```

> [!NOTE]
> `maintain_order=True` preserves the order in which groups first appear. It costs a little performance, so omit it when group order does not matter.

Inside `agg`, an expression that does **not** reduce to a scalar (for example `pl.col("name")`) produces a *list* per group, letting you gather group members:

```python
df.group_by("decade").agg(pl.col("name"))   # name column becomes List[str]
```

### Common aggregation expressions

`sum`, `mean`, `median`, `min`, `max`, `std`, `var`, `quantile`, `count`, `n_unique`, `first`, `last`, `len`, `arg_min`, `arg_max`. There are also horizontal (across-columns) variants: `pl.sum_horizontal`, `pl.mean_horizontal`, `pl.max_horizontal`, `pl.min_horizontal`, `pl.any_horizontal`, `pl.all_horizontal`.

### Window functions: `.over()`

Aggregate *within groups* without collapsing rows, by appending `.over()` to an expression:

```python
df.with_columns(
    avg_weight_by_decade=pl.col("weight")
        .mean()
        .over(pl.col("birthdate").dt.year() // 10),
)
```

This adds the group average back to every row — the equivalent of a SQL window function.

---

## 8. Sorting, Reshaping, and Combining

### Sorting

```python
df.sort("weight")                          # ascending
df.sort("weight", descending=True)
df.sort(["decade", "weight"], descending=[False, True])   # multi-key
```

### Selecting rows

```python
df.head(10)        # first 10 rows
df.tail(10)        # last 10 rows
df.sample(5)       # random 5 rows
df.slice(10, 5)    # 5 rows starting at offset 10
df.unique()        # drop duplicate rows
```

### Reshaping

| Operation | Method |
| --- | --- |
| Wide → long | `df.unpivot(...)` |
| Long → wide | `df.pivot(...)` |
| Transpose | `df.transpose()` |
| Explode a list column into rows | `df.explode("col")` |

```python
df.pivot(values="weight", index="decade", on="name", aggregate_function="mean")
```

### Combining frames

| Goal | Function |
| --- | --- |
| Stack rows (same schema) | `pl.concat([df1, df2])` |
| Stack rows (differing schemas) | `pl.concat([...], how="diagonal")` |
| Stack columns side by side | `pl.concat([...], how="horizontal")` |
| Union (alias of vertical concat) | `pl.union([...])` |
| Align on an index column | `pl.align_frames(...)` |
| Merge two sorted frames | `pl.merge_sorted(...)` |

---

## 9. Joining DataFrames

Combine two frames on one or more key columns:

```python
df.join(other, on="name", how="left")
df.join(other, left_on="id", right_on="user_id", how="inner")
```

### Join types

| `how=` | Keeps |
| --- | --- |
| `inner` | Rows with a match in both frames (default). |
| `left` | All left rows; nulls where no right match. |
| `right` | All right rows; nulls where no left match. |
| `full` | All rows from both frames. |
| `semi` | Left rows that *have* a match (no right columns added). |
| `anti` | Left rows that have *no* match. |
| `cross` | Cartesian product of both frames. |

### As-of joins

Join on the *nearest* key rather than an exact match — essential for time series (e.g. matching trades to the most recent quote):

```python
trades.join_asof(quotes, on="timestamp", by="symbol", strategy="backward")
```

> [!TIP]
> Both frames must be sorted on the `on` key for `join_asof`. `strategy` may be `"backward"` (default), `"forward"`, or `"nearest"`.

---

## 10. The Expression API in Depth

Expressions are built by chaining methods onto a starting point. The most common starting points are module-level functions:

| Function | Produces |
| --- | --- |
| `pl.col("a")` / `pl.col("a", "b")` | One or more columns. |
| `pl.lit(value)` | A literal constant. |
| `pl.first()` / `pl.last()` / `pl.nth(i)` | Positional column references. |
| `pl.when(cond).then(a).otherwise(b)` | Conditional (if/else) logic. |
| `pl.sum("a")`, `pl.mean("a")`, … | Shorthand aggregations. |
| `pl.concat_str([...])`, `pl.concat_list([...])` | Combine columns. |
| `pl.struct([...])` | Pack columns into a struct. |
| `pl.format("{}-{}", a, b)` | String formatting across columns. |
| `pl.arange` / `pl.int_range` / `pl.date_range` | Range generators. |

### Conditional logic with `when`/`then`/`otherwise`

```python
df.with_columns(
    category=pl.when(pl.col("bmi") < 18.5).then(pl.lit("underweight"))
              .when(pl.col("bmi") < 25).then(pl.lit("normal"))
              .otherwise(pl.lit("overweight")),
)
```

### Chaining transformations

Expressions read left to right as a pipeline:

```python
pl.col("name").str.to_lowercase().str.replace(" ", "_").alias("slug")
```

### Namespaces

Type-specific operations live under namespaces on an expression:

| Namespace | For | Examples |
| --- | --- | --- |
| `.str` | Strings | `to_uppercase`, `contains`, `replace`, `split`, `slice`, `len_chars`, `json_decode` |
| `.dt` | Dates/times | `year`, `month`, `day`, `weekday`, `truncate`, `offset_by`, `strftime` |
| `.list` | List columns | `first`, `last`, `len`, `sum`, `eval`, `gather`, `explode` |
| `.arr` | Fixed-size arrays | `sum`, `min`, `max`, `get` |
| `.struct` | Struct columns | `field`, `rename_fields`, `unnest` |
| `.cat` | Categoricals | `get_categories` |
| `.name` | Output naming | `suffix`, `prefix`, `keep`, `to_lowercase`, `map` |

```python
df.with_columns(
    first_name=pl.col("name").str.split(" ").list.first(),
    birth_month=pl.col("birthdate").dt.month(),
)
```

### Folds — efficient horizontal reduction

To reduce across many columns efficiently, use `pl.fold` (or the `*_horizontal` helpers):

```python
pl.fold(acc=pl.lit(0), function=lambda acc, x: acc + x, exprs=pl.col("*"))
```

### User-defined functions

When no built-in expression fits, drop into Python — but prefer native expressions first, since UDFs run row-by-row and lose parallelism guarantees:

```python
df.with_columns(
    pl.col("weight").map_elements(my_func, return_dtype=pl.Float64)  # per element
)
df.with_columns(
    pl.col("weight").map_batches(my_vectorized_func)                 # per Series
)
```

> [!WARNING]
> `map_elements` calls your Python function once per value and is slow. Reach for it only when a transformation truly cannot be expressed with native expressions. For compiled speed, write the UDF in Rust as a plugin (see [How It Works](#17-how-it-works)).

---

## 11. Working with Specific Data Types

Polars has a rich, Arrow-backed type system. The headline types:

| Category | Types |
| --- | --- |
| Integers | `Int8`, `Int16`, `Int32`, `Int64`, `Int128`, `UInt8`…`UInt128` |
| Floats | `Float32`, `Float64` |
| Boolean | `Boolean` |
| Strings/binary | `String` (alias `Utf8`), `Binary`, `Categorical`, `Enum` |
| Temporal | `Date`, `Datetime`, `Time`, `Duration` |
| Nested | `List`, `Array` (fixed-size), `Struct` |
| Other | `Decimal`, `Null`, `Object`, `Unknown` |

### Casting

```python
df.with_columns(pl.col("weight").cast(pl.Int32))
```

### Missing data

Polars distinguishes **null** (missing) from **NaN** (float not-a-number):

```python
df.with_columns(pl.col("weight").fill_null(0))
df.with_columns(pl.col("weight").fill_null(strategy="forward"))
df.filter(pl.col("weight").is_not_null())
df.drop_nulls()
```

### Strings

```python
df.select(
    pl.col("name").str.to_uppercase(),
    pl.col("name").str.contains("Brown"),
    pl.col("name").str.split(" ").list.len().alias("word_count"),
)
```

### Temporal

```python
df.select(
    pl.col("birthdate").dt.year(),
    pl.col("birthdate").dt.weekday(),
    pl.col("birthdate").dt.offset_by("1mo"),
    pl.col("birthdate").dt.truncate("1y"),
)
```

> [!NOTE]
> Timezone handling needs the `timezone` extra on Windows (it is built in on Linux/macOS). The `temporal` Rust feature enables Chrono interop on the Rust side.

### Categorical and Enum

Use `Categorical` for repeated string values to save memory and speed up grouping/joining. Use `Enum` when you know the full set of categories ahead of time:

```python
df.with_columns(pl.col("name").cast(pl.Categorical))
ratings = pl.Enum(["low", "medium", "high"])
```

> [!TIP]
> When working with multiple categorical columns that must share an encoding (for joins/comparisons), enable the global string cache: `pl.enable_string_cache()` or the `with pl.StringCache():` context manager.

---

## 12. The Lazy API

The lazy API is where Polars delivers its biggest wins. Instead of executing each step immediately, you build a **query plan**; Polars optimizes the whole plan, then runs it on `.collect()`.

Use the lazy API because it:

- lets the **query optimizer** rewrite your query (predicate pushdown, projection pushdown, common-subexpression elimination, slice pushdown);
- enables **streaming** of larger-than-memory datasets;
- catches **schema errors before** any data is processed.

### Starting a lazy query

From a file (best — the optimizer can reduce what is read):

```python
lf = pl.scan_csv("reddit.csv")
result = (
    lf.with_columns(pl.col("name").str.to_uppercase())
      .filter(pl.col("comment_karma") > 0)
      .collect()
)
```

From an existing DataFrame:

```python
result = df.lazy().filter(pl.col("height") > 1.7).collect()
```

> [!IMPORTANT]
> A `LazyFrame` does nothing until `.collect()`. Build the whole chain, then collect once at the end. Calling `.collect()` mid-chain defeats the optimizer.

### Inspecting the plan

```python
lf.explain()                 # optimized plan as text
lf.explain(optimized=False)  # the plan as you wrote it
lf.show_graph()              # visual graph (requires the `graph` extra)
```

Comparing the optimized and unoptimized plans is the best way to *see* the optimizer pushing your filter down to the scan.

### Running variants

| Call | Behavior |
| --- | --- |
| `lf.collect()` | Run and return a `DataFrame`. |
| `lf.collect(engine="streaming")` | Run with the streaming engine (larger-than-RAM). |
| `lf.collect(engine="gpu")` | Run on an NVIDIA GPU (requires the `gpu` extra). |
| `pl.collect_all([lf1, lf2])` | Run multiple LazyFrames together, sharing work. |
| `lf.collect_async()` | Run asynchronously (requires the `async` extra). |
| `lf.sink_parquet("out.parquet")` | Stream results straight to a file without materializing in memory. |
| `lf.fetch(n)` | Run on the first `n` rows for a quick preview. |

### Sinks — streaming output

For pipelines whose output is also large, **sinks** write results directly to disk in a streaming fashion, never holding the full result in memory:

```python
pl.scan_csv("huge.csv").filter(pl.col("x") > 0).sink_parquet("filtered.parquet")
```

Sink targets include `sink_parquet`, `sink_ipc`, `sink_csv`, and `sink_ndjson`.

### GPU acceleration

With `pip install 'polars[gpu]'`, run an optimized lazy query on an NVIDIA GPU by passing the GPU engine:

```python
result = lf.collect(engine="gpu")
```

You can tune behavior with `pl.GPUEngine(...)`. Operations not supported on the GPU fall back to CPU automatically.

---

## 13. Streaming: Larger-Than-RAM Data

If your data does not fit in memory, the streaming engine processes the query in batches:

```python
result = (
    pl.scan_parquet("huge/*.parquet")
      .filter(pl.col("amount") > 100)
      .group_by("region")
      .agg(pl.col("amount").sum())
      .collect(engine="streaming")
)
```

This can let you process a 250 GB dataset on a laptop. Besides easing memory pressure, the streaming engine is often *faster* than the in-memory engine.

> [!NOTE]
> Some operations are inherently non-streaming or not yet implemented for streaming; Polars transparently falls back to the in-memory engine for those parts of the plan. You do not need to do anything — but if you are debugging memory use, inspect the physical plan with `lf.show_graph()` to see which nodes are memory-intensive.

> [!TIP]
> Pair streaming with **sinks** (`sink_parquet`, etc.) when the *output* is also too large to hold in memory — the entire read → transform → write path then runs in bounded memory.

---

## 14. Querying with SQL

Polars includes a SQL frontend. Register frames as tables and run standard SQL against them — executed by the same optimized engine.

```python
ctx = pl.SQLContext(people=df)
result = ctx.execute(
    "SELECT name, weight FROM people WHERE height > 1.7 ORDER BY weight DESC",
    eager=True,
)
```

Or the one-shot helper, which auto-registers DataFrames/LazyFrames in scope:

```python
pl.sql("SELECT * FROM df WHERE height > 1.7", eager=True)
```

Convert a SQL scalar expression into a Polars expression:

```python
df.select(pl.sql_expr("weight / (height * height) AS bmi"))
```

> [!NOTE]
> The SQL frontend supports `SELECT`, `WHERE`, `GROUP BY`, `ORDER BY`, joins, CTEs (`WITH`), `CREATE TABLE ... AS`, and a large library of SQL functions and operators. SQL and the expression API are fully interoperable — mix whichever reads best for a given query.

---

## 15. Interoperability (pandas, NumPy, Arrow)

Polars converts to and from the rest of the Python data ecosystem. Conversions to Arrow are zero-copy where possible because Polars is built on the Arrow memory format.

### From other formats into Polars

| Function | Source |
| --- | --- |
| `pl.from_pandas(df)` | pandas DataFrame/Series (needs `pandas`). |
| `pl.from_numpy(arr)` | NumPy array (needs `numpy`). |
| `pl.from_arrow(tbl)` | PyArrow Table/Array (needs `pyarrow`). |
| `pl.from_dict(d)` / `pl.from_dicts(rows)` | Python dict / list of dicts. |
| `pl.from_records(rows)` | Row-oriented records. |
| `pl.from_torch(t)` | PyTorch tensor. |
| `pl.from_repr(s)` | A printed DataFrame repr (handy for bug reports). |
| `pl.json_normalize(data)` | Nested JSON → flat columns. |

### From Polars into other formats

```python
df.to_pandas()      # requires pandas
df.to_numpy()       # requires numpy
df.to_arrow()       # PyArrow table (zero-copy where possible)
df.to_dicts()       # list of row dicts
df.to_dict()        # dict of columns
```

> [!TIP]
> Migrating from pandas? The official user guide has a dedicated pandas-comparison section. The biggest mental shift is from in-place mutation and the index to **immutable frames + expressions** — there is no row index in Polars, and every operation returns a new frame.

---

## 16. Controlling Display and Output

The `pl.Config` object controls how DataFrames print. Set options globally, or scope them with a context manager.

```python
# Global
pl.Config.set_tbl_rows(20)
pl.Config.set_tbl_cols(-1)            # show all columns
pl.Config.set_fmt_str_lengths(100)

# Scoped — reverts on exit
with pl.Config(tbl_rows=50, tbl_cols=20):
    print(df)
```

Frequently used `Config` settings:

| Setting | Effect |
| --- | --- |
| `set_tbl_rows(n)` | Max rows printed. |
| `set_tbl_cols(n)` | Max columns printed (`-1` = all). |
| `set_tbl_width_chars(n)` | Max table width. |
| `set_fmt_str_lengths(n)` | Max characters per string cell. |
| `set_float_precision(n)` | Decimal places for floats. |
| `set_thousands_separator(True)` | Group digits in large numbers. |
| `set_tbl_hide_column_data_types(True)` | Hide the dtype row. |
| `set_ascii_tables(True)` | ASCII-only borders (for logs/terminals). |
| `set_streaming_chunk_size(n)` | Batch size for the streaming engine. |

```python
pl.Config.save_to_file("polars.config")   # persist settings
pl.Config.load_from_file("polars.config")
pl.Config.restore_defaults()
```

> [!NOTE]
> Display and behavior can also be controlled with `POLARS_*` environment variables (for example `POLARS_TABLE_WIDTH`). The full environment-variable reference lives in the **Admin Guide**.

---

## 17. How It Works

You do not need this section to use Polars, but it explains *why* the API is shaped the way it is.

### One engine, many frontends

Polars is a Rust query engine. The Python, Rust, Node.js, R, and SQL frontends are thin layers that build the same internal **logical plan**, which the same optimizer and execution engines run. Performance and semantics are therefore consistent across languages.

### Apache Arrow memory format

Data is stored column-by-column in the [Apache Arrow](https://arrow.apache.org/docs/format/Columnar.html) columnar format. Columnar storage is cache-friendly and SIMD-friendly, enables zero-copy interchange with other Arrow tools, and is the reason Polars imports in ~70 ms with zero required dependencies.

### From expression to result

1. Your expressions and contexts build a **logical plan** (an abstract description of the query).
2. The **optimizer** rewrites it — pushing filters and column projections down to the data source, eliminating common subexpressions, reordering operations.
3. A **physical plan** is produced and dispatched to the in-memory engine, the streaming engine, or the GPU engine.
4. Independent expressions run **in parallel** across CPU cores.

This is why lazy queries are faster: the optimizer sees the *whole* query before any data is read.

### Eager is lazy underneath

Even eager `DataFrame` operations are built on the same machinery — an eager call is essentially a one-step lazy query that collects immediately. That is why the expression API is identical in both modes.

### Extending Polars

- **Expression plugins** — write a custom expression in Rust, compile it, and call it from Python with near-native speed (the `pyo3-polars` crate; see the `docs/source/user-guide/plugins/` section of the project).
- **IO plugins** — register a custom data source so `scan_*`-style lazy reading works against it.

---

## 18. Troubleshooting

### Error: `ModuleNotFoundError` for an optional dependency

**Cause:** You used a feature whose backing library is an opt-in extra (e.g. `read_excel` without an Excel engine, or `to_pandas` without pandas).
**Solution:** Install the relevant extra, e.g. `pip install 'polars[excel]'` or `pip install 'polars[pandas]'`.
**Prevention:** Run `pl.show_versions()` to see which optional dependencies are present before relying on them.

### Error: `the configured CPU is not supported` / illegal instruction at import

**Cause:** The default wheels are compiled for modern CPUs (AVX2). On an older CPU, or under Rosetta, the default build cannot run.
**Solution:** Install the compatibility build: `pip install polars[rtcompat]`.
**Prevention:** On legacy hardware always start from `rtcompat`. As a last-resort diagnostic you can set `POLARS_SKIP_CPU_CHECK=1`, but the compatibility wheel is the correct fix.

### Error: `MemoryError` / `memory allocation of … bytes failed`

**Cause:** An operation tried to materialize more data than fits in RAM — for example reading a very wide Excel sheet, or a non-streaming aggregation over a huge dataset (see issue #27677 for a pathological `read_excel` case).
**Solution:** Switch to the lazy API with `scan_*` and run `collect(engine="streaming")`; restrict columns/rows at read time (`columns=`, `n_rows=`); for output, use a `sink_*` instead of `collect`.
**Prevention:** Prefer Parquet over Excel/CSV for large data, and always scan large files lazily.

### My lazy query produced no output

**Cause:** A `LazyFrame` chain only describes a query; it does not run until collected.
**Solution:** Add `.collect()` (or a `sink_*`) at the end of the chain.
**Prevention:** Remember the rule: `scan_* → transform → collect()`.

### `pl.lit(1)` and `pl.lit([1])` have different integer types

**Cause:** A scalar integer literal infers as `Int32`, while a list literal element infers as `Int64` (issue #13114).
**Solution:** Pin the dtype explicitly: `pl.lit(1, dtype=pl.Int64)`.
**Prevention:** When literal dtypes matter (e.g. before a join or comparison), always pass an explicit `dtype`.

### `ValueError` from a `Config` setting

**Cause:** A `Config.set_*` method received an out-of-range or invalid argument (the `config.py` validators raise `ValueError`).
**Solution:** Check the method's accepted values; e.g. `set_tbl_cols(-1)` means "all," but arbitrary negative values for other settings are rejected.
**Prevention:** Use `pl.Config.state()` to inspect current settings and confirm a change took effect.

### `DuplicateError` building a Schema / frame

**Cause:** Two columns share the same name.
**Solution:** Rename one with `.alias()` or `.name.suffix(...)` before the conflicting operation.
**Prevention:** When using expression expansion, add a `.name.suffix`/`.name.prefix` so expanded outputs stay unique.

### A `panic` / `PanicException` from the engine

**Cause:** An internal invariant was violated — sometimes a genuine bug (e.g. `sink_delta` on null struct values, issue #27079; `hist` on a `String` column, issue #27155).
**Solution:** Re-run with `POLARS_BACKTRACE_IN_ERR=1` (and `POLARS_PANIC_ON_ERR=1`) to capture a full backtrace, then report it with a minimal reproduction.
**Prevention:** Keep Polars up to date; many panics are fixed quickly upstream.

---

## 19. Frequently Asked Questions

**Q: How is Polars different from pandas?**
Polars is a multi-threaded, Arrow-backed query engine with a lazy optimizer and no row index. Operations return new (immutable) frames rather than mutating in place, and transformations are written as composable expressions. It is typically much faster and uses less memory; its import time is ~70 ms versus pandas' ~520 ms.

**Q: Do I have to use the lazy API?**
No — eager `DataFrame` operations work line by line and are great for exploration. But for performance and for larger-than-memory data, the lazy API (`scan_*` … `collect`) is strongly recommended.

**Q: When does my expression actually run?**
Inside a context. `pl.col("x") * 2` alone is just a recipe; it runs when placed in `select`, `with_columns`, `filter`, or `agg`. In the lazy API, the context call is itself deferred until `.collect()`.

**Q: Is there a row index like in pandas?**
No. Polars has no implicit index. If you need a positional index column, add one with `pl.int_range`/`with_row_index`.

**Q: How do I process data that doesn't fit in memory?**
Scan it lazily and collect with the streaming engine: `pl.scan_parquet(...).…​.collect(engine="streaming")`. For large outputs, stream straight to disk with `sink_parquet`/`sink_ipc`/`sink_csv`.

**Q: Can I run Polars on a GPU?**
Yes. Install `polars[gpu]` and call `lf.collect(engine="gpu")`. Unsupported operations fall back to the CPU automatically.

**Q: How do I write a custom function?**
Prefer native expressions. If you must, use `map_elements` (per value, slow) or `map_batches` (per Series, faster). For compiled speed, write an **expression plugin** in Rust via `pyo3-polars`.

**Q: How do I handle more than 4.3 billion rows?**
Install the big-index build: `pip install polars[rt64]` (Python) or enable the `bigidx` feature (Rust). Only do this if you actually exceed 2³² rows — the default build is faster.

**Q: My old CPU crashes on import. What do I do?**
Install the compatibility build compiled without AVX: `pip install polars[rtcompat]`.

**Q: Can I use SQL?**
Yes. Register frames with `pl.SQLContext` (or use `pl.sql(...)`) and run standard SQL, executed by the same optimized engine. SQL and expressions interoperate freely.

**Q: How do I see which optional features are installed?**
`pl.show_versions()` prints the version, the index type, and every optional dependency's status.

---

## 20. Reference

### Top-level constructors and functions (selected)

| Symbol | Purpose |
| --- | --- |
| `pl.DataFrame` | Eager, in-memory table. |
| `pl.LazyFrame` | Deferred query plan. |
| `pl.Series` | A single typed column. |
| `pl.Expr` | An expression (built via `pl.col`, etc.). |
| `pl.col`, `pl.lit`, `pl.when` | Expression starting points. |
| `pl.sum`, `pl.mean`, `pl.min`, `pl.max`, `pl.count`, `pl.len`, `pl.n_unique` | Aggregations. |
| `pl.concat`, `pl.union`, `pl.align_frames`, `pl.merge_sorted` | Combine frames. |
| `pl.SQLContext`, `pl.sql`, `pl.sql_expr` | SQL frontend. |
| `pl.Config` | Display/behavior configuration. |
| `pl.show_versions`, `pl.build_info`, `pl.thread_pool_size` | Environment introspection. |

### Reading and writing (selected)

`read_csv`, `read_parquet`, `read_json`, `read_ndjson`, `read_ipc`, `read_avro`, `read_excel`, `read_delta`, `read_database` / `read_database_uri`, `read_clipboard` — and the lazy scanners `scan_csv`, `scan_parquet`, `scan_ndjson`, `scan_ipc`, `scan_delta`, `scan_iceberg`, `scan_pyarrow_dataset`. Writers: `write_csv`, `write_parquet`, `write_json`, `write_ndjson`, `write_ipc`, `write_avro`, `write_excel`, `write_delta`, `write_database`, plus streaming sinks `sink_parquet`, `sink_ipc`, `sink_csv`, `sink_ndjson`.

### Expression namespaces

`.str` (strings) · `.dt` (temporal) · `.list` (lists) · `.arr` (fixed arrays) · `.struct` (structs) · `.cat` (categoricals) · `.name` (output naming).

### Selected environment variables

A few useful ones for end users (the full list is in the Admin Guide):

| Variable | Effect |
| --- | --- |
| `POLARS_TABLE_WIDTH` | Default printed table width. |
| `POLARS_SKIP_CPU_CHECK` | Skip the AVX CPU check at import (diagnostic only). |
| `POLARS_BACKTRACE_IN_ERR` | Include a Rust backtrace in error messages. |
| `POLARS_VERBOSE` | Emit verbose engine logging. |

### Official resources

- **User guide & API docs:** <https://docs.pola.rs/>
- **Python API reference:** <https://docs.pola.rs/api/python/stable/reference/index.html>
- **Rust API reference (docs.rs):** <https://docs.rs/polars/latest/polars/>
- **Community:** StackOverflow tag `python-polars`, and the project Discord.

---

_Generated June 2026 from pola-rs/polars at 1c11555. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._