# pandas User Guide

**pandas** is a fast, flexible, and expressive Python library for working with **labeled, tabular, and time-series data**. Its two core objects — the one-dimensional `Series` and the two-dimensional `DataFrame` — give you a single, consistent toolkit for loading data from almost any source, cleaning and reshaping it, grouping and aggregating it, joining it with other data, and writing the result back out.

This guide shows you how to *use* pandas: how to install it, the everyday tasks that make up real data work, and the recipes you reach for again and again. It is organized around what you want to *do* with your data — not around the library's internal module layout. If you need pandas' architecture, build system, or internals, see the **pandas Admin Guide**.

> [!NOTE]
> Throughout this guide, pandas is imported under its conventional alias and NumPy alongside it:
> ```python
> import pandas as pd
> import numpy as np
> ```
> Every example assumes these two imports are already in place.

---

## 1. Quick Start

The fastest way to understand pandas is to build a `DataFrame`, look at it, and ask it a question.

```python
import pandas as pd

# Build a DataFrame from a dict of columns
df = pd.DataFrame({
    "name": ["Ada", "Bram", "Cleo", "Dao"],
    "team": ["A", "B", "A", "B"],
    "score": [91, 72, 88, 64],
})

print(df)
```

```text
   name team  score
0   Ada    A     91
1  Bram    B     72
2  Cleo    A     88
3   Dao    B     64
```

Now answer a real question — *what is the average score per team?* — in one line:

```python
print(df.groupby("team")["score"].mean())
```

```text
team
A    89.5
B    68.0
Name: score, dtype: float64
```

That single expression captures the **split-apply-combine** pattern that pandas is built around: split the rows into groups, apply a calculation, and combine the results back into a labeled object. The rest of this guide expands on every step.

> [!TIP]
> The two objects you will use constantly are `Series` (a single labeled column) and `DataFrame` (a table of `Series` that share one row index). Almost every method exists on both, and selecting a single column from a `DataFrame` gives you a `Series`.

---

## 2. Installation

pandas ships as a pre-built binary wheel for all common platforms, so a normal install requires no compiler.

=== "pip (PyPI)"

    ```bash
    python -m pip install pandas
    ```

=== "conda (conda-forge)"

    ```bash
    conda install -c conda-forge pandas
    ```

=== "With optional extras"

    ```bash
    # Excel, Parquet, plotting, and HTML support in one go
    python -m pip install "pandas[excel,parquet,plot,html]"
    ```

Verify the install and inspect your environment:

```python
import pandas as pd

print(pd.__version__)
pd.show_versions()      # full dependency + system report, useful for bug reports
```

```text
3.0.0
INSTALLED VERSIONS
------------------
python           : 3.13.x
pandas           : 3.0.0
numpy            : 2.x
...
```

### 2.1 Requirements

pandas requires **Python 3.11 or newer** and these runtime dependencies:

| Dependency | Role | Notes |
|---|---|---|
| `numpy` | Underlying numeric arrays | `>=1.26`, `<3` |
| `python-dateutil` | Flexible date parsing | `>=2.8.2` |
| `tzdata` | IANA time-zone database | Required on **Windows** and **Emscripten** only |

> [!IMPORTANT]
> `numpy` and `python-dateutil` are *hard* dependencies — pandas refuses to import without them and raises an `ImportError` naming the missing package.

### 2.2 Optional dependencies

Many pandas features activate only when an optional package is installed. You do not need any of these to start, but a feature will raise an `ImportError` telling you exactly which package to install when you first use it.

| Feature you want | Install |
|---|---|
| Read/write **Excel** (`.xlsx`) | `openpyxl`, or `xlsxwriter` (write), `python-calamine`/`xlrd` (read), `odfpy` (`.ods`) |
| Read/write **Parquet** | `pyarrow` (recommended) or `fastparquet` |
| Read/write **HDF5** (`.h5`) | `pytables` |
| Read/write **Feather / ORC** | `pyarrow` |
| **SQL** databases | `sqlalchemy` plus a driver (`psycopg2`, `pymysql`, `adbc-driver-*`) |
| **HTML** tables (`read_html`) | `lxml`, `html5lib`, or `beautifulsoup4` |
| **XML** (`read_xml` / `to_xml`) | `lxml` |
| **Plotting** | `matplotlib` |
| **Styler** HTML/Excel export | `jinja2` |
| **Numba**-accelerated `apply` / windows | `numba` |
| Faster evaluation | `numexpr`, `bottleneck` |
| **Arrow-backed** dtypes | `pyarrow` (`>=13`) |
| Cloud storage paths (`s3://`, `gs://`) | `fsspec` + `s3fs` / `gcsfs` |
| SAS / SPSS / Stata via reader | `pyreadstat` |
| Clipboard I/O | `pyqt`/`qtpy` (Linux) |

> [!TIP]
> When in doubt, install `pyarrow` and `matplotlib`. Between them they unlock Parquet, Feather, ORC, Arrow-backed dtypes, and all charting — the features most users reach for first.

---

## 3. The Two Core Objects

### 3.1 Series — a labeled column

A `Series` is a one-dimensional array of values paired with an **index** (the labels).

```python
s = pd.Series([10, 20, 30], index=["a", "b", "c"], name="counts")
print(s)
```

```text
a    10
b    20
c    30
Name: counts, dtype: int64
```

```python
print(s["b"])        # label-based access -> 20
print(s.mean())      # 20.0
print(s * 2)         # vectorized: every element doubled
```

### 3.2 DataFrame — a labeled table

A `DataFrame` is a collection of `Series` that share a single row index. You can build one from a dict of columns, a list of rows, NumPy arrays, or by reading a file.

=== "From a dict of columns"

    ```python
    df = pd.DataFrame({
        "city": ["Phoenix", "Tucson", "Mesa"],
        "pop": [1_650_000, 543_000, 504_000],
    })
    ```

=== "From a list of row dicts"

    ```python
    df = pd.DataFrame([
        {"city": "Phoenix", "pop": 1_650_000},
        {"city": "Tucson", "pop": 543_000},
    ])
    ```

=== "From a NumPy array"

    ```python
    import numpy as np
    df = pd.DataFrame(
        np.arange(6).reshape(3, 2),
        columns=["x", "y"],
    )
    ```

### 3.3 First look at any DataFrame

When you receive an unfamiliar table, these methods orient you instantly:

| Method | What it tells you |
|---|---|
| `df.head(n)` / `df.tail(n)` | First / last `n` rows (default 5) |
| `df.shape` | `(rows, columns)` tuple |
| `df.info()` | Column names, non-null counts, dtypes, memory use |
| `df.describe()` | Summary statistics for numeric columns |
| `df.dtypes` | The dtype of each column |
| `df.columns` / `df.index` | The column and row labels |
| `df.sample(n)` | A random `n`-row sample |

```python
df.info()
```

```text
<class 'pandas.DataFrame'>
RangeIndex: 3 entries, 0 to 2
Data columns (total 2 columns):
 #   Column  Non-Null Count  Dtype
---  ------  --------------  -----
 0   city    3 non-null      object
 1   pop     3 non-null      int64
```

> [!NOTE]
> A column of text shows as dtype `object` by default. To get pandas' dedicated, more memory-efficient string type, pass `dtype="string"` when constructing, or call `df["city"] = df["city"].astype("string")`. See §11 on dtypes.

---

## 4. Reading and Writing Data (I/O)

pandas' I/O layer is one of its biggest draws: a family of `pd.read_*` functions that pull data *in* and matching `.to_*` methods that write it back *out*. Every reader returns a `DataFrame` (or `Series`); every writer is a method on the object you want to save.

### 4.1 The reader / writer matrix

| Format | Read | Write | Optional dependency |
|---|---|---|---|
| CSV / delimited text | `pd.read_csv` | `df.to_csv` | built-in |
| Fixed-width text | `pd.read_fwf` | — | built-in |
| Excel | `pd.read_excel` | `df.to_excel` | `openpyxl` / `xlsxwriter` |
| JSON | `pd.read_json` | `df.to_json` | built-in |
| HTML tables | `pd.read_html` | `df.to_html` | `lxml` / `bs4` |
| XML | `pd.read_xml` | `df.to_xml` | `lxml` |
| Parquet | `pd.read_parquet` | `df.to_parquet` | `pyarrow` |
| Feather | `pd.read_feather` | `df.to_feather` | `pyarrow` |
| ORC | `pd.read_orc` | `df.to_orc` | `pyarrow` |
| HDF5 | `pd.read_hdf` | `df.to_hdf` | `pytables` |
| Stata | `pd.read_stata` | `df.to_stata` | built-in |
| SAS | `pd.read_sas` | — | built-in / `pyreadstat` |
| SPSS | `pd.read_spss` | — | `pyreadstat` |
| Pickle | `pd.read_pickle` | `df.to_pickle` | built-in |
| Clipboard | `pd.read_clipboard` | `df.to_clipboard` | platform |
| SQL | `pd.read_sql` / `read_sql_query` / `read_sql_table` | `df.to_sql` | `sqlalchemy` + driver |
| Iceberg | `pd.read_iceberg` | `df.to_iceberg` | `pyiceberg` |

### 4.2 CSV — the workhorse

```python
df = pd.read_csv("sales.csv")
```

`read_csv` has the deepest option set of any reader. The ones you will use most:

| Parameter | Purpose | Example |
|---|---|---|
| `sep` / `delimiter` | Field separator | `sep="\t"` for TSV |
| `header` | Row number(s) to use as column names | `header=None` |
| `names` | Supply column names yourself | `names=["a","b"]` |
| `index_col` | Column to use as the row index | `index_col=0` |
| `usecols` | Read only some columns | `usecols=["date","total"]` |
| `dtype` | Force column dtypes | `dtype={"id": "int32"}` |
| `parse_dates` | Parse columns as datetimes | `parse_dates=["date"]` |
| `nrows` | Read only the first N rows | `nrows=1000` |
| `skiprows` | Skip leading rows | `skiprows=3` |
| `na_values` | Extra strings to treat as missing | `na_values=["N/A", "-"]` |
| `encoding` | File text encoding | `encoding="latin-1"` |
| `chunksize` | Iterate in chunks (for big files) | `chunksize=100_000` |
| `engine` | Parser backend | `engine="pyarrow"` |
| `compression` | De/compress on the fly | `compression="gzip"` |

> [!TIP]
> Choose the right CSV engine for the job. The valid values are `"c"` (the default, fast), `"python"` (most flexible, supports regex separators), `"pyarrow"` (multi-threaded, fastest for large files — requires `pyarrow`), and `"python-fwf"` (fixed-width). Pass it as `engine=...`.

Reading a large file without exhausting memory:

```python
total = 0
for chunk in pd.read_csv("huge.csv", chunksize=100_000):
    total += chunk["amount"].sum()
print(total)
```

Writing back out — note `index=False` to avoid dumping the row labels as a column:

```python
df.to_csv("clean.csv", index=False)
```

### 4.3 Excel

```python
# Read one sheet
df = pd.read_excel("report.xlsx", sheet_name="Q1")

# Read every sheet into a dict of DataFrames
sheets = pd.read_excel("report.xlsx", sheet_name=None)

# Write multiple DataFrames to separate sheets
with pd.ExcelWriter("out.xlsx") as writer:
    q1.to_excel(writer, sheet_name="Q1", index=False)
    q2.to_excel(writer, sheet_name="Q2", index=False)
```

> [!NOTE]
> When you give an existing workbook to `ExcelWriter`, the `if_sheet_exists` argument controls collisions. Valid values are `"error"` (the default), `"new"`, `"replace"`, and `"overlay"`.

### 4.4 Parquet, Feather, and ORC — fast columnar formats

These binary formats preserve dtypes exactly, compress well, and load far faster than CSV. They require `pyarrow`.

```python
df.to_parquet("data.parquet")              # write
df = pd.read_parquet("data.parquet")       # read

# Read just two columns from a large file — columnar formats make this cheap
df = pd.read_parquet("data.parquet", columns=["date", "total"])
```

> [!TIP]
> For any dataset you will load more than once, save it as **Parquet** rather than CSV. Round-trips are faster, files are smaller, and — unlike CSV — dtypes survive the trip.

### 4.5 JSON

```python
df = pd.read_json("records.json")

# Flatten nested JSON into a flat table
records = [{"id": 1, "user": {"name": "Ada", "city": "Phoenix"}}]
flat = pd.json_normalize(records)
#    id  user.name  user.city
# 0   1        Ada    Phoenix
```

`to_json` and `read_json` share an `orient` argument that controls the JSON shape: `"records"`, `"columns"`, `"index"`, `"split"`, `"table"`, and `"values"`.

### 4.6 SQL databases

```python
from sqlalchemy import create_engine
engine = create_engine("postgresql+psycopg2://user:pw@host/db")

# Full table
df = pd.read_sql_table("customers", engine)

# Arbitrary query (use parameters, never string-format user input)
df = pd.read_sql_query(
    "SELECT * FROM orders WHERE total > %(min)s",
    engine,
    params={"min": 100},
)

# Write a DataFrame to a table
df.to_sql("orders_clean", engine, if_exists="replace", index=False)
```

> [!WARNING]
> Build SQL with parameters (`params=...`), not Python string formatting. Interpolating user input directly into a query string exposes you to SQL injection.

---

## 5. Selecting and Indexing Data

Getting the rows and columns you want is the most common pandas task. There are three doors into a `DataFrame`, and choosing the right one prevents most beginner confusion.

### 5.1 The three accessors

| Accessor | Selects by | Use for |
|---|---|---|
| `[]` | Column name (or boolean mask) | Quick column or filter |
| `.loc[]` | **Labels** (row label, column name) | Almost everything |
| `.iloc[]` | **Integer position** (0-based) | Position-based slicing |

```python
df["score"]                 # one column -> Series
df[["name", "score"]]       # several columns -> DataFrame

df.loc[0, "score"]          # value at row label 0, column "score"
df.loc[0:2, ["name", "score"]]   # label slice (END-INCLUSIVE)
df.iloc[0:2, [0, 2]]        # position slice (END-EXCLUSIVE)
```

> [!IMPORTANT]
> `.loc` label slices **include** the endpoint (`df.loc[0:2]` returns rows 0, 1, and 2), while `.iloc` position slices **exclude** it (`df.iloc[0:2]` returns rows 0 and 1) — matching normal Python slicing. This is the single most common source of off-by-one surprises in pandas.

### 5.2 Boolean filtering (the workhorse)

Build a boolean `Series` and pass it inside `[]` or `.loc[]`:

```python
df[df["score"] >= 80]                      # rows where score >= 80
df[(df["score"] >= 80) & (df["team"] == "A")]   # AND
df[(df["team"] == "A") | (df["team"] == "B")]   # OR
df[~(df["team"] == "A")]                    # NOT
```

> [!CAUTION]
> Use `&`, `|`, and `~` — not the Python keywords `and`, `or`, `not` — for element-wise boolean logic, and **wrap each condition in parentheses**. The bitwise operators bind tighter than comparisons, so `df["a"] > 1 & df["b"] < 2` is parsed as `df["a"] > (1 & df["b"]) < 2` and raises `ValueError: The truth value of a Series is ambiguous`.

Other handy filters:

```python
df[df["team"].isin(["A", "C"])]            # membership
df[df["score"].between(70, 90)]            # inclusive range
df[df["name"].str.startswith("A")]         # string predicate
df.query("score >= 80 and team == 'A'")    # string expression form
```

### 5.3 Adding, renaming, and dropping

```python
df["passed"] = df["score"] >= 70           # new column from a vector
df = df.assign(bonus=df["score"] * 0.1)    # functional, returns a new frame

df = df.rename(columns={"score": "points"})
df = df.drop(columns=["bonus"])            # drop columns
df = df.drop(index=[0, 1])                 # drop rows by label
```

### 5.4 Setting and resetting the index

```python
df = df.set_index("name")     # promote a column to the row index
df = df.reset_index()         # push the index back to a column
```

---

## 6. Cleaning Data and Handling Missing Values

Real data is messy. pandas represents "missing" with `NaN` (for floats), `NaT` (for datetimes), and the dtype-agnostic `pd.NA`. A consistent set of methods detects and fills it.

### 6.1 Detecting missing data

```python
df.isna()                 # element-wise boolean mask of missing values
df.isna().sum()           # count of missing values per column
df.notna()                # the inverse
df["col"].isna().mean()   # fraction missing in one column
```

> [!NOTE]
> `isnull` / `notnull` are exact aliases of `isna` / `notna`. Use whichever reads better — they behave identically.

### 6.2 Dropping vs. filling

```python
df.dropna()                       # drop any row containing a missing value
df.dropna(how="all")              # drop only rows that are entirely missing
df.dropna(subset=["score"])       # consider only specific columns
df.dropna(axis=1)                 # drop columns (instead of rows)

df.fillna(0)                      # replace missing with a constant
df.fillna({"score": 0, "city": "unknown"})   # per-column fills
df["score"].ffill()               # forward-fill (carry last value forward)
df["score"].bfill()               # back-fill
df["value"].interpolate()         # numeric/temporal interpolation
```

> [!TIP]
> `how` accepts `"any"` (default — drop if *any* value is missing) or `"all"` (drop only if *every* value is missing).

### 6.3 Removing duplicates

```python
df.duplicated()                   # boolean mask of duplicate rows
df.drop_duplicates()              # keep the first occurrence of each row
df.drop_duplicates(subset=["id"], keep="last")
```

> [!NOTE]
> The `keep` argument chooses which duplicate survives: `"first"` (default), `"last"`, or `False` (drop *all* duplicated rows).

### 6.4 Replacing and converting values

```python
df["grade"] = df["grade"].replace({"A+": "A", "B-": "B"})
df["amount"] = pd.to_numeric(df["amount"], errors="coerce")   # bad -> NaN
df["when"] = pd.to_datetime(df["when"], errors="coerce")
```

> [!IMPORTANT]
> The `errors` argument on `pd.to_numeric`, `pd.to_datetime`, and `pd.to_timedelta` accepts `"raise"` (default — stop on the first bad value) or `"coerce"` (turn unparseable values into `NaN`/`NaT`). Use `"coerce"` when you expect dirty input and want to find the bad rows afterward with `.isna()`.

### 6.5 Working with text columns — the `.str` accessor

Vectorized string methods live under `.str` and operate on a whole column at once:

```python
df["name"].str.lower()
df["name"].str.strip()
df["name"].str.contains("a", case=False)
df["email"].str.split("@").str[1]          # domain part
df["phone"].str.replace(r"\D", "", regex=True)   # keep digits only
df["name"].str.len()
```

---

## 7. Transforming and Computing

### 7.1 Vectorized arithmetic

Operations apply to whole columns at once — no Python loop required, and far faster than one.

```python
df["total"] = df["price"] * df["qty"]
df["pct"]   = df["score"] / df["score"].sum() * 100
df["z"]     = (df["x"] - df["x"].mean()) / df["x"].std()
```

### 7.2 Summary statistics

Every numeric `Series` and `DataFrame` exposes the standard reductions:

| Method | Result |
|---|---|
| `.sum()` `.mean()` `.median()` | Totals and central tendency |
| `.min()` `.max()` | Extremes |
| `.std()` `.var()` | Spread |
| `.count()` | Non-missing count |
| `.nunique()` | Number of distinct values |
| `.value_counts()` | Frequency table of a column |
| `.quantile(q)` | The `q`-th quantile |
| `.cumsum()` `.cumprod()` | Running totals |
| `.describe()` | All of the above at once |

```python
df["team"].value_counts()
```

```text
A    2
B    2
Name: count, dtype: int64
```

> [!NOTE]
> By default reductions **skip** missing values (`skipna=True`). Pass `skipna=False` to make any `NaN` propagate into the result.

### 7.3 Applying your own functions

When a built-in method does not exist, map a function over the data:

```python
df["score"].map(lambda v: "pass" if v >= 70 else "fail")   # element-wise on a Series
df.apply(np.mean)                       # reduce each column
df.apply(lambda row: row["a"] + row["b"], axis=1)   # across columns, per row
```

> [!TIP]
> Reach for `apply` only when a vectorized method genuinely does not exist. `df["x"] * 2` is dramatically faster than `df["x"].apply(lambda v: v * 2)` because it runs in compiled code rather than calling Python once per row.

### 7.4 Binning continuous values

```python
pd.cut(df["age"], bins=[0, 18, 65, 120], labels=["child", "adult", "senior"])
pd.qcut(df["income"], q=4)              # four equal-sized quantile buckets
```

---

## 8. Group By: Split–Apply–Combine

`groupby` is the heart of analytical pandas. It splits rows into groups by one or more keys, applies a function to each group, and combines the answers into a labeled result.

```python
df.groupby("team")["score"].mean()                 # one stat, one column
df.groupby("team").agg({"score": "mean", "name": "count"})
df.groupby(["team", "passed"])["score"].sum()      # multiple keys -> MultiIndex
```

### 8.1 Named aggregations

To control the output column names, use `pd.NamedAgg` (or the `(column, func)` tuple form):

```python
df.groupby("team").agg(
    avg_score=pd.NamedAgg(column="score", aggfunc="mean"),
    n_players=pd.NamedAgg(column="name", aggfunc="count"),
)
```

```text
      avg_score  n_players
team
A          89.5          2
B          68.0          2
```

### 8.2 transform vs. aggregate vs. filter

| Operation | Returns | Use when |
|---|---|---|
| `.agg(...)` | One row per group | You want a summary table |
| `.transform(...)` | Same shape as input | You want to broadcast a group stat back onto every row |
| `.filter(...)` | A subset of rows | You want to keep/drop *entire groups* by a condition |

```python
# Add a column holding each row's group mean (transform keeps the original shape)
df["team_avg"] = df.groupby("team")["score"].transform("mean")

# Keep only groups with more than one member
df.groupby("team").filter(lambda g: len(g) > 1)
```

> [!TIP]
> `transform` is the tool for "compare each value to its group" — for example computing a within-group z-score or the percent of a group total. It returns a result aligned to the original rows so you can assign it straight back as a new column.

---

## 9. Combining DataFrames

### 9.1 Concatenating (stacking)

```python
pd.concat([df_jan, df_feb])                 # stack rows (same columns)
pd.concat([df_left, df_right], axis=1)      # stack columns (same index)
```

### 9.2 Merging / joining (database-style)

`pd.merge` aligns two frames on one or more key columns, just like a SQL join.

```python
pd.merge(orders, customers, on="customer_id", how="left")
pd.merge(a, b, left_on="uid", right_on="id", how="inner")
```

The `how` argument selects the join type:

| `how` | Keeps |
|---|---|
| `"inner"` | Only keys present in **both** frames (default) |
| `"left"` | All keys from the **left** frame |
| `"right"` | All keys from the **right** frame |
| `"outer"` | The **union** of keys from both |
| `"cross"` | The Cartesian product (every row paired with every row) |
| `"left_anti"` | Left rows whose key is **not** in the right frame |
| `"right_anti"` | Right rows whose key is **not** in the left frame |

> [!TIP]
> Pass `validate="one_to_one"` (or `"one_to_many"`, `"many_to_one"`) to have pandas *assert* the relationship between your keys. If the data violates it — e.g. a "unique" key that has duplicates — the merge raises immediately instead of silently producing a row explosion.

### 9.3 Specialized merges

```python
# As-of (nearest-key) merge — essential for time-series alignment
pd.merge_asof(trades, quotes, on="time", by="symbol")

# Ordered merge with optional group-wise fill
pd.merge_ordered(a, b, on="date", fill_method="ffill")
```

---

## 10. Reshaping and Pivoting

| Function / method | Turns | Into |
|---|---|---|
| `df.pivot_table(...)` | Long rows | A wide summary grid (with aggregation) |
| `df.pivot(...)` | Long rows | Wide (no aggregation; keys must be unique) |
| `pd.melt(df, ...)` | Wide columns | Long key/value rows |
| `df.stack()` / `df.unstack()` | Move levels between rows and columns | |
| `pd.crosstab(a, b)` | Two vectors | A frequency cross-tabulation |
| `pd.get_dummies(df)` | Categorical column | One-hot indicator columns |

```python
pd.pivot_table(
    df, values="score", index="team", columns="passed", aggfunc="mean"
)

pd.melt(df, id_vars=["name"], value_vars=["math", "science"],
        var_name="subject", value_name="grade")

pd.get_dummies(df["team"])        # -> columns team_A, team_B (0/1)
```

> [!NOTE]
> `pivot` requires the index/column combination to be unique and only *reshapes*; `pivot_table` *aggregates* duplicate combinations with `aggfunc` (default `"mean"`). If `pivot` raises `ValueError: Index contains duplicate entries`, switch to `pivot_table`.

---

## 11. Data Types (dtypes)

Every column has a single dtype that determines how it is stored and what operations are valid. Picking good dtypes saves memory and prevents surprises.

### 11.1 The dtype families

| Family | Examples | Notes |
|---|---|---|
| NumPy numeric | `int64`, `float64`, `bool` | Default; cannot hold missing values in integer columns |
| **Nullable** (masked) | `Int64`, `Float64`, `boolean` | Capital first letter; support `pd.NA` |
| **String** | `string` | Dedicated text type (vs. generic `object`) |
| **Categorical** | `category` | Repeated labels stored once — big memory win |
| **Datetime / timedelta** | `datetime64[ns]`, `timedelta64[ns]` | Time-series native types |
| **Period / Interval** | `period[M]`, `interval` | Spans and ranges |
| **Arrow-backed** | `int64[pyarrow]`, `string[pyarrow]` | Backed by Apache Arrow (needs `pyarrow`) |

### 11.2 Converting dtypes

```python
df["id"]     = df["id"].astype("int32")
df["team"]   = df["team"].astype("category")        # memory + speed for repeated text
df["name"]   = df["name"].astype("string")
df = df.convert_dtypes()                             # auto-pick best nullable dtypes
df = df.astype({"a": "Int64", "b": "float64"})      # several at once
```

> [!TIP]
> Convert any column with few distinct repeated values (status codes, country names, categories) to `"category"`. On large frames this can cut memory use by an order of magnitude and speeds up `groupby` on that column.

### 11.3 Nullable integers and `pd.NA`

A plain NumPy `int64` column cannot store missing values — pandas would silently promote it to `float64`. The nullable `Int64` dtype (capital *I*) fixes this:

```python
s = pd.Series([1, 2, None], dtype="Int64")
# 0       1
# 1       2
# 2    <NA>
# dtype: Int64
```

---

## 12. Working with Time Series

Time-series support is a flagship pandas feature: a rich timestamp type, calendar-aware date ranges, frequency conversion, and rolling-window statistics.

### 12.1 Timestamps and date ranges

```python
pd.Timestamp("2026-06-04 09:30")
pd.to_datetime(df["when"])                         # parse a column
pd.date_range("2026-01-01", periods=12, freq="MS") # 12 month-start dates
pd.bdate_range("2026-01-01", "2026-01-31")         # business days only
```

Common frequency aliases for `freq=`: `"D"` (day), `"B"` (business day), `"W"` (week), `"MS"`/`"ME"` (month start/end), `"QS"`/`"QE"` (quarter), `"YS"`/`"YE"` (year), `"h"`, `"min"`, `"s"`.

### 12.2 Datetime components — the `.dt` accessor

```python
df["when"].dt.year
df["when"].dt.month_name()
df["when"].dt.dayofweek            # Monday=0
df["when"].dt.normalize()          # strip the time part
df["when"].dt.tz_localize("UTC").dt.tz_convert("US/Arizona")
```

### 12.3 Resampling (time-based group by)

When the index is a datetime, `resample` groups rows into time buckets:

```python
ts = ts.set_index("when")
ts["sales"].resample("ME").sum()       # monthly totals
ts["price"].resample("W").mean()       # weekly average
```

### 12.4 Rolling and expanding windows

```python
df["roll7"]  = df["value"].rolling(window=7).mean()    # 7-row moving average
df["cumavg"] = df["value"].expanding().mean()          # cumulative average
df["ewm"]    = df["value"].ewm(span=10).mean()         # exponentially weighted
```

> [!TIP]
> For an irregular time index, give `rolling` a time-based window string such as `rolling("30D")` instead of an integer count — pandas then uses a true 30-day window regardless of how many rows fall inside it.

---

## 13. Plotting

With `matplotlib` installed, every `Series` and `DataFrame` gains a `.plot` accessor.

```python
df["score"].plot(kind="line")
df.plot(x="date", y="total", kind="line")
df["team"].value_counts().plot(kind="bar")
df.plot.scatter(x="height", y="weight")
df["score"].plot(kind="hist", bins=20)
```

Valid `kind` values: `"line"`, `"bar"`, `"barh"`, `"hist"`, `"box"`, `"kde"`/`"density"`, `"area"`, `"pie"`, `"scatter"`, and `"hexbin"`.

> [!NOTE]
> `kind="scatter"`, `"hexbin"`, and `"pie"` apply to `DataFrame` objects and need their axis columns specified. Some kinds raise `ValueError: pie requires either y column or 'subplots=True'` if you omit the column.

---

## 14. Displaying and Configuring Output

pandas truncates wide and tall output by default. Control this with the options system:

```python
pd.set_option("display.max_rows", 100)
pd.set_option("display.max_columns", 50)
pd.set_option("display.precision", 3)         # decimal places shown
pd.set_option("display.width", 120)

pd.get_option("display.max_rows")             # read a setting
pd.reset_option("display.max_rows")           # restore the default
pd.describe_option("display.max_rows")        # see docs + current value
```

Scope a setting to a single block with a context manager so it reverts automatically:

```python
with pd.option_context("display.max_rows", None):
    print(df)          # show every row, just here
```

> [!TIP]
> Frequently useful display options: `display.max_rows`, `display.max_columns`, `display.precision`, `display.float_format` (e.g. `pd.set_option("display.float_format", "{:,.2f}".format)`), `display.max_colwidth`, and `display.expand_frame_repr`.

---

## 15. Common Recipes

**Top N rows by a column**

```python
df.nlargest(5, "score")
df.nsmallest(3, "price")
```

**Conditional new column (vectorized if/else)**

```python
df["band"] = np.where(df["score"] >= 70, "pass", "fail")
```

**Multi-condition mapping**

```python
df["band"] = pd.cut(df["score"], [0, 60, 80, 100],
                    labels=["low", "mid", "high"])
```

**Sort by one or more columns**

```python
df.sort_values(["team", "score"], ascending=[True, False])
df.sort_index()
```

**Count and percentage in one table**

```python
counts = df["team"].value_counts()
pct    = df["team"].value_counts(normalize=True).mul(100).round(1)
pd.DataFrame({"count": counts, "pct": pct})
```

**Chained, readable pipeline**

```python
result = (
    df
    .query("score >= 70")
    .assign(bonus=lambda d: d["score"] * 0.1)
    .groupby("team", as_index=False)
    .agg(total=("score", "sum"))
    .sort_values("total", ascending=False)
)
```

> [!TIP]
> Wrap a multi-step transformation in parentheses and put one method per line. This "method chaining" style reads top-to-bottom like a recipe and avoids littering your code with throwaway intermediate variables.

---

## 16. Troubleshooting

### Error: `The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()`

**Cause:** You combined boolean conditions with Python's `and`/`or`, or forgot parentheses around comparisons.
**Solution:** Use `&`, `|`, `~` and wrap each condition: `df[(df.a > 1) & (df.b < 2)]`.
**Prevention:** Treat parentheses around every comparison as mandatory in filters.

### Error: `KeyError: 'columnname'`

**Cause:** The column or index label does not exist — often a stray space or a case mismatch.
**Solution:** Check `df.columns.tolist()`. Strip whitespace with `df.columns = df.columns.str.strip()`.
**Prevention:** Inspect `df.info()` right after loading so you know the exact labels.

### Error: `SettingWithCopyWarning` / changes that don't "stick"

**Cause:** Assigning into the result of a chained selection (`df[df.a > 1]["b"] = 0`) writes to a temporary copy.
**Solution:** Use a single `.loc` assignment: `df.loc[df["a"] > 1, "b"] = 0`.
**Prevention:** Whenever you both *filter* and *assign*, do it in one `.loc[rows, cols] = value` statement.

### Error: `ValueError: Index contains duplicate entries, cannot reshape`

**Cause:** `pivot` (or `unstack`) found duplicate index/column key combinations.
**Solution:** Switch to `pivot_table`, which aggregates duplicates via `aggfunc`.
**Prevention:** Confirm key uniqueness with `df.duplicated(subset=keys).any()` before pivoting.

### Error: `ImportError: Missing optional dependency 'openpyxl'` (or `pyarrow`, `lxml`, `matplotlib`, …)

**Cause:** The format/feature you used needs an optional package that is not installed.
**Solution:** `pip install openpyxl` (or whichever package the message names).
**Prevention:** Install `pyarrow` and `matplotlib` up front; they cover most optional features.

### Error: `MergeError` / unexpected row explosion after `merge`

**Cause:** A key you assumed was unique actually has duplicates, so rows multiply (a many-to-many join).
**Solution:** Add `validate="one_to_many"` (or the right relationship) to make pandas check; deduplicate the key side first.
**Prevention:** Run `df["key"].is_unique` on the side you expect to be unique before merging.

### Symptom: An integer column became `float64` after introducing missing values

**Cause:** NumPy `int64` cannot hold `NaN`, so pandas promotes the column to `float64`.
**Solution:** Use the nullable `Int64` dtype: `s.astype("Int64")`.
**Prevention:** Construct integer columns that may contain gaps with `dtype="Int64"` from the start.

### Error: `TypeError: only list-like objects are allowed to be passed to isin(), you passed a 'str'`

**Cause:** You passed a single string to `.isin()`, which expects a collection.
**Solution:** Wrap it in a list: `df[df["team"].isin(["A"])]`.
**Prevention:** Always give `.isin()` a list/set, even for one value.

### Symptom: `read_csv` is slow or runs out of memory on a huge file

**Cause:** The whole file is being read into memory at once.
**Solution:** Read in chunks with `chunksize=...`, select only needed columns with `usecols=...`, or switch to `engine="pyarrow"`.
**Prevention:** For repeated loads, convert the source to Parquet once and read that instead.

### Symptom: Dates loaded as plain strings, so `.dt` accessor fails

**Cause:** `read_csv` does not parse dates unless told to.
**Solution:** Pass `parse_dates=["col"]`, or convert afterward with `pd.to_datetime(df["col"])`.
**Prevention:** Specify `parse_dates` at read time.

---

## 17. Frequently Asked Questions

**Q: What is the difference between a `Series` and a `DataFrame`?**
A `Series` is one labeled column (1-D); a `DataFrame` is a table of `Series` sharing one row index (2-D). Selecting a single column from a `DataFrame` returns a `Series`.

**Q: When should I use `.loc` vs `.iloc`?**
Use `.loc` for label-based access (row/column *names*) and `.iloc` for integer *position*. `.loc` is what you want the vast majority of the time. Remember `.loc` slices include the endpoint; `.iloc` slices do not.

**Q: How do I avoid the `SettingWithCopyWarning`?**
Make filtered assignments in one statement with `.loc`: `df.loc[mask, "col"] = value`. Do not chain `df[mask]["col"] = value`.

**Q: My numbers turned into floats with `.0` everywhere — why?**
A NumPy integer column was promoted to `float64` because a missing value was introduced. Use the nullable `Int64` dtype to keep integers and still allow `pd.NA`.

**Q: How do I apply a function to every row?**
`df.apply(func, axis=1)`. But first check whether a vectorized expression can do the same thing far faster — `apply` calls Python once per row.

**Q: How do I get the unique values / their counts in a column?**
`df["col"].unique()` for the distinct values, `df["col"].nunique()` for the count, and `df["col"].value_counts()` for a frequency table.

**Q: How do I rename columns?**
`df.rename(columns={"old": "new"})`, or assign a fresh list to `df.columns`.

**Q: How do I select rows where a column is in a set of values?**
`df[df["col"].isin(["a", "b", "c"])]`.

**Q: Why is my filter raising "truth value of a Series is ambiguous"?**
You used `and`/`or` instead of `&`/`|`, or omitted parentheses. Write `df[(cond1) & (cond2)]`.

**Q: How do I read just part of a giant CSV?**
Use `nrows=` to cap rows, `usecols=` to limit columns, or `chunksize=` to iterate. For repeated work, convert to Parquet.

**Q: What's the fastest on-disk format for round-tripping a DataFrame?**
Parquet (via `pyarrow`). It is compact, fast, and — unlike CSV — preserves dtypes exactly.

**Q: How do I see all rows/columns instead of the truncated view?**
Raise or remove the display limits: `pd.set_option("display.max_rows", None)`, or scope it with `pd.option_context`.

**Q: Does pandas have a command-line tool?**
No. pandas is a library you import (`import pandas as pd`) and use from Python, notebooks, or scripts. It ships no end-user CLI.

**Q: Which Python versions are supported?**
Python 3.11 and newer. Free-threaded (no-GIL) CPython 3.13 is supported on an experimental basis.

---

_Generated June 2026 from pandas-dev/pandas at b5c64e2. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._