Pandas Parquet & Feather Binary Formats

Traditional text-based formats like Comma-Separated Values (CSV) and Microsoft Excel suffer from slow read/write performance, large disk storage footprints, and type loss during persistence. Apache Parquet and Feather (Apache Arrow) solve these limitations by providing binary columnar storage with strict Schema and Data Type preservation.

Choose the optimal storage format based on your operational requirements across speed, compression, and schema fidelity:

Reading & Writing Parquet Files 🔝

To use Parquet in Pandas, ensure pyarrow or fastparquet is installed. Use df.to_parquet() and pd.read_parquet():

import pandas as pd
import numpy as np

# Generate synthetic DataFrame
df = pd.DataFrame({
    "id": np.arange(1, 100001),
    "category": np.random.choice(["A", "B", "C", "D"], 100000),
    "sales": np.random.uniform(10.0, 500.0, 100000),
    "timestamp": pd.date_range(start="2026-01-01", periods=100000, freq="min")
})

# Save DataFrame to Parquet file
df.to_parquet("data.parquet", index=False)

# Read Parquet file back into DataFrame
df_loaded = pd.read_parquet("data.parquet")
print(df_loaded.info())

Compression Engines (Snappy, Gzip, Zstd) 🔝

Parquet supports several compression algorithms via the compression parameter:

• snappy (Default): Balanced speed and compression. Ideal for general-purpose analytics.
• gzip: Highest compression ratio, but slower write performance. Great for archiving.
• zstd: Modern compression engine with outstanding speed and compression ratio.
• none: Uncompressed binary output.

# Save with zstd compression for high efficiency
df.to_parquet("data_zstd.parquet", compression="zstd", index=False)

Ultra-Fast Caching with Feather 🔝

Feather is an uncompressed or lightly compressed memory-mapped format built on Apache Arrow. It provides near-instantaneous load times for local session caching:

# Export to Feather format
df.to_feather("cache.feather")

# Load instantly from Feather
df_cached = pd.read_feather("cache.feather")

Converting Sample Student Datasets to Parquet 🔝

You can collect sample student benchmark tables across various formats directly from plus2net Student Benchmark Dataset and convert them to optimized Parquet binary files:

import pandas as pd

# Download sample datasets from plus2net server inside Google Colab
# !wget https://www.plus2net.com/python/download/student.csv
# !wget https://www.plus2net.com/python/download/student.xlsx
# !wget https://www.plus2net.com/python/download/student.json
# !wget https://www.plus2net.com/python/download/student.xml
# !wget https://www.plus2net.com/python/download/my_db.db

# Load CSV format and convert to Parquet
df_student = pd.read_csv("https://www.plus2net.com/python/download/student.csv")

# Export ingested student records into Parquet binary storage
df_student.to_parquet("student.parquet", index=False)

# Verify Parquet reload
df_student_pq = pd.read_parquet("student.parquet")
print(df_student_pq.head())

Practical Real-World Use Cases 🔝

• Cloud Data Lakes & Serverless Analytics: Parquet is the native format for Amazon Web Services (AWS) Athena, Google Cloud BigQuery, and DuckDB, enabling direct SQL querying over cloud storage.
• Machine Learning Feature Stores: Persisting pre-processed training datasets in Parquet guarantees exact float precision and category types across training iterations.
• Inter-Process Communication (IPC): Passing Feather binary files between R scripts and Python pipelines avoids slow text-parsing overhead.

Common Errors & Troubleshooting 🔝

• ImportError: Unable to find a valid parquet library: Triggered when neither pyarrow nor fastparquet is installed. Fix: Run pip install pyarrow.
• ArrowInvalid: System time before epoch: Occurs when saving nanosecond timestamps. Fix: Pass coerce_timestamps="ms" or allow_truncated_timestamps=True in to_parquet().
• Cannot serialize mixed object dtypes: Occurs when a string column contains mixed non-string objects. Fix: Cast string columns using df['col'] = df['col'].astype(str) prior to export.

Frequently Asked Questions (FAQ) 🔝

Performance Benchmarking & Memory Footprint 🔝

When comparing disk sizes across formats for a dataset with 100,000 rows:

• CSV File: ~5.8 MB
• Parquet (Snappy): ~1.1 MB (81% disk savings)
• Feather (LZ4): ~1.4 MB (76% disk savings)

Exercise: Columnar Storage Conversion 🔝

Perform the following task:

1. Create a DataFrame with 50,000 synthetic transaction records containing item_id, price, and transaction_date.
2. Export the DataFrame to transactions.parquet using snappy compression.
3. Read back only the item_id and price columns using columns=['item_id', 'price'] in pd.read_parquet().
4. Verify the loaded DataFrame shape and schema types.

Exercise Solution 🔝

import pandas as pd
import numpy as np

np.random.seed(42)
df_tx = pd.DataFrame({
    "item_id": np.random.randint(1000, 9999, size=50000),
    "price": np.random.round(np.random.uniform(5.0, 250.0, size=50000), 2),
    "transaction_date": pd.date_range(start="2026-01-01", periods=50000, freq="s")
})

# 2. Export to Parquet
df_tx.to_parquet("transactions.parquet", compression="snappy", index=False)

# 3. Read selective columns
df_sub = pd.read_parquet("transactions.parquet", columns=["item_id", "price"])

# 4. Verify shape and content
print(f"Subset Shape: {df_sub.shape}")
print(df_sub.head())
assert df_sub.shape == (50000, 2)

Interactive Companion Jupyter Notebook

Explore interactive code cells, dynamically test Parquet/Feather benchmarks, convert sample student datasets, troubleshoot common errors, and run self-validating assertions.

View on GitHub

Further Reading & Related Tutorials 🔝

• Pandas to_feather() — Ultra-fast memory-mapped binary caching with Apache Arrow.
• Pandas read_csv() — Reading text-delimited flat files with parsing parameters.
• Pandas Data Ingestion & I/O Home — Master index of all Pandas input and output methods.
• Pandas Section Home — Master index of all Pandas tutorials, pillars, and guides.

Reference: Pandas IO Tools official user guide.

Subhendu Mohapatra

Author & Instructor at plus2net

I write and maintain practical tutorials on Python, PHP, SQL, JavaScript, HTML, jQuery, and web development at plus2net. The tutorials focus on clear explanations, working examples, and code that readers can test and adapt while learning.

About the author   Watch on YouTube

Connect: