Pandas read_pickle(): Restore and Validate Trusted Pickle Files

Use read_pickle() to restore a trusted saved object, then validate its type and fields.

Only load pickle files from trusted sources. Unpickling can execute code; compression and file extensions do not establish trust.

read_pickle to read data from Python Pickle file and create a Pandas DataFrame

Show Table of Contents ↓

Read the original marks DataFrame 🔝

read_pickle() returns the saved object. Here it is a DataFrame with Ravi, Raju and Alex records; the notebook creates the trusted input itself. No download or uploaded pickle is required. See to_pickle().

import pandas as pd
import pickle
import tempfile
import sqlite3
from pathlib import Path
from io import BytesIO, StringIO
df = pd.DataFrame({"NAME": ["Ravi", "Raju", "Alex"], "ID": [1, 2, 3],
    "MATH": [30, 40, 50], "ENGLISH": [20, 30, 40]})
print(df)
with tempfile.TemporaryDirectory(dir=Path.cwd()) as folder:
    target = Path(folder) / "my_data.pkl"
    df.to_pickle(target)
    my_data = pd.read_pickle(target)
    pd.testing.assert_frame_equal(my_data, df)
    print("Restored rows:", len(my_data))

Expected output

   NAME  ID  MATH  ENGLISH
0  Ravi   1    30       20
1  Raju   2    40       30
2  Alex   3    50       40
Restored rows: 3

Read from a chosen folder 🔝

Resolve paths relative to the current working directory, or use an absolute path. A Windows raw path such as Path(r"D:\my_data") / "my_data.pkl" preserves backslashes. Check that the path points to a trusted file before unpickling.

with tempfile.TemporaryDirectory(dir=Path.cwd()) as folder:
    target = Path(folder) / "archive" / "my_data.pkl"
    target.parent.mkdir()
    df.to_pickle(target)
    restored = pd.read_pickle(target)
    pd.testing.assert_frame_equal(restored, df)
    print("Read archive/my_data.pkl")

Expected output

Read archive/my_data.pkl

Read inferred gzip compression 🔝

compression="infer" inspects recognized path extensions. Read the .pkl.gz file using the default. Do not guess compression from the .pkl suffix alone.

with tempfile.TemporaryDirectory(dir=Path.cwd()) as folder:
    target = Path(folder) / "my_data.pkl.gz"
    df.to_pickle(target)
    restored = pd.read_pickle(target)
    pd.testing.assert_frame_equal(restored, df)
    print("Gzip restored rows:", len(restored))

Expected output

Gzip restored rows: 3

Read explicit ZIP compression 🔝

Retain the original compression="zip" example. When ZIP bytes use a .pkl name, specify ZIP explicitly at read time. A ZIP or TAR archive must contain only one data file.

with tempfile.TemporaryDirectory(dir=Path.cwd()) as folder:
    target = Path(folder) / "my_data.pkl"
    df.to_pickle(target, compression="zip")
    restored = pd.read_pickle(target, compression="zip")
    pd.testing.assert_frame_equal(restored, df)
    print("Explicit ZIP restored rows:", len(restored))

Expected output

Explicit ZIP restored rows: 3

Read a binary buffer 🔝

Rewind a buffer before reading. read_pickle() reads binary pickle data, rather than a text StringIO buffer.

buffer = BytesIO()
df.to_pickle(buffer)
buffer.seek(0)
restored = pd.read_pickle(buffer, compression=None)
pd.testing.assert_frame_equal(restored, df)
print("Buffer restored rows:", len(restored))

Expected output

Buffer restored rows: 3

Check which object was stored 🔝

A pickle need not contain a DataFrame. read_pickle() can return a Series or another Python object. After loading a trusted file, check the type before applying DataFrame methods. A post-load type check is a data contract, not a security check.

series = pd.Series([30, 40, 50], index=[1, 2, 3], name="MATH")
with tempfile.TemporaryDirectory(dir=Path.cwd()) as folder:
    target = Path(folder) / "marks.pkl"
    series.to_pickle(target)
    restored = pd.read_pickle(target)
    assert isinstance(restored, pd.Series)
    pd.testing.assert_series_equal(restored, series)
    print("Stored type:", type(restored).__name__)
    print(restored)

Expected output

Stored type: Series
1    30
2    40
3    50
Name: MATH, dtype: int64

Verify index and missing-value dtypes 🔝

Read_pickle restores saved dtypes and labels in this same-environment example. Check the artifact against the expected schema rather than inferring success from a filename.

typed = pd.DataFrame({"units": pd.array([2, None], dtype="Int64"),
    "state": pd.Categorical(["ready", "review"])},
    index=pd.Index(["001", "002"], name="order_id"))
with tempfile.TemporaryDirectory(dir=Path.cwd()) as folder:
    target = Path(folder) / "orders.pkl"
    typed.to_pickle(target)
    restored = pd.read_pickle(target)
    pd.testing.assert_frame_equal(restored, typed)
    print(restored.dtypes)
    assert restored.index.tolist() == ["001", "002"]
    assert restored["units"].isna().sum() == 1

Expected output

units       Int64
state    category
dtype: object

Validate a restored sales checkpoint 🔝

This synthetic sales checkpoint contains a pending amount. Preserve the raw records and separate accepted and review rows. The pickle is created within this example and is trusted. Continue with Data Cleaning.

sales = pd.DataFrame({"id": ["001", "002", "003"], "amount": ["5.25", "20", "pending"]})
with tempfile.TemporaryDirectory(dir=Path.cwd()) as folder:
    target = Path(folder) / "sales.pkl"
    sales.to_pickle(target)
    raw = pd.read_pickle(target)
assert isinstance(raw, pd.DataFrame) and raw.columns.tolist() == ["id", "amount"]
clean = raw.copy()
clean["amount"] = pd.to_numeric(raw["amount"], errors="coerce")
valid = clean["amount"].ge(0).fillna(False)
accepted = clean.loc[valid].copy()
review = raw.loc[~valid].copy()
print("Accepted total:", accepted["amount"].sum())
print("Review IDs:", review["id"].tolist())
assert accepted["amount"].sum() == 25.25
assert len(accepted) + len(review) == len(raw)

Expected output

Accepted total: 25.25
Review IDs: ['003']

Parameters, trusted files and compatibility 🔝

Parameter	Purpose
`filepath_or_buffer`	Required path or binary readable buffer.
`compression`	infer by default; specify the format when the suffix does not identify it.
`storage_options`	Forwarded options for supported storage backends.

Trusted sources only: loading a pickle can execute code. Do not load an unknown attachment merely to inspect it. Missing file: check the working directory and source path. Wrong compression: match the writer settings. Compatibility errors: recreate the documented Python/Pandas environment or regenerate an interchange file in the producing environment. Pickle is not a permanent cross-version guarantee. Large files: read_pickle() loads an object and has no chunksize parameter. Consider chunked CSV or SQL imports for large workflows. See Python pickle and database pickling.

Exercise: restore and filter marks 🔝

Create a trusted pickle from df, restore it, then select NAME and MATH where MATH is at least 40. Verify the names Raju and Alex.

Exercise solution 🔝

Read first, verify the restored object, then filter the required fields.

with tempfile.TemporaryDirectory(dir=Path.cwd()) as folder:
    target = Path(folder) / "exercise.pkl"
    df.to_pickle(target)
    restored = pd.read_pickle(target)
assert isinstance(restored, pd.DataFrame)
answer = restored.loc[restored["MATH"].ge(40), ["NAME", "MATH"]]
print(answer)
assert answer["NAME"].tolist() == ["Raju", "Alex"]

Expected output

   NAME  MATH
1  Raju    40
2  Alex    50

Practice in Google Colab 🔝

Open in Google Colab View on GitHub

Run examples in order. The notebook creates all practice files. Save a copy to keep edits.

Reference: Pandas read_pickle documentation.

Subhendu Mohapatra

Author & Instructor at plus2net

I write and maintain practical tutorials on Python, PHP, SQL, JavaScript, HTML, jQuery, and web development at plus2net. The tutorials focus on clear explanations, working examples, and code that readers can test and adapt while learning.

About the author Watch on YouTube

Connect:

Subscribe to our YouTube Channel here