Large Dataset Memory Mapping with np.memmap

When working with multi-gigabyte numerical datasets, loading entire arrays into system RAM can trigger severe MemoryError exceptions or cause heavy operating system swap disk paging. NumPy solves this memory bottleneck through np.memmap() (memory-mapped file arrays), which maps raw binary data stored on disk directly into virtual memory address space without allocating physical RAM for unread elements.

How Memory-Mapped Arrays Work 🔝

Memory mapping leverages the operating system's virtual memory manager to page binary blocks between disk storage and RAM dynamically. Only the array slices explicitly requested in code are loaded into RAM, while updates can be written back flush to disk:

Creating and Writing Binary Disk Files 🔝

Initialize a large 10,000 × 10,000 float64 binary file on disk without allocating full RAM space:

import numpy as np

filename = "large_matrix.dat"
shape = (10000, 10000)  # 100,000,000 float64 elements (~800 MB on disk)

# Create binary memmap on disk
fp = np.memmap(filename, dtype="float64", mode="w+", shape=shape)

# Write values chunk-by-chunk to keep memory footprint flat
for i in range(0, shape[0], 1000):
    fp[i:i+1000, :] = np.random.randn(1000, shape[1])

# Flush dirty memory pages to physical disk storage
fp.flush()
print("Memory-mapped array successfully written to disk!")

Reading & In-Place Matrix Updates 🔝

Re-open the binary file in read-write mode and process selective slices efficiently:

# Access disk file in read-write mode without loading whole array into RAM
new_fp = np.memmap(filename, dtype="float64", mode="r+", shape=shape)

# Compute row mean on sliced block
slice_mean = np.mean(new_fp[500:1000, :])
print(f"Mean of slice [500:1000]: {slice_mean:.4f}")

# Update values directly on disk
new_fp[0, :5] = [1.0, 2.0, 3.0, 4.0, 5.0]
new_fp.flush()
print("Updated first 5 elements directly on disk:", new_fp[0, :5])

Chunked Batch Processing & Normalization 🔝

To normalize a giant matrix (e.g. z-score scaling), process the data in row chunks so system RAM usage remains minimal:

# Re-open file for chunked normalization
data = np.memmap(filename, dtype="float64", mode="r+", shape=shape)

chunk_size = 2000
for start in range(0, shape[0], chunk_size):
    end = start + chunk_size
    chunk = data[start:end, :]
    c_mean = np.mean(chunk)
    c_std = np.std(chunk)
    # In-place normalization per chunk
    data[start:end, :] = (chunk - c_mean) / c_std

data.flush()
print("Chunked normalization finished successfully!")

Concurrent Read-Only Access & Offsets 🔝

Multiple processes can attach to the same binary file safely using read-only mode (mode='r'), or skip structured header bytes using the offset parameter:

# Open in read-only mode for safe analytical worker threads
ro_data = np.memmap(filename, dtype="float64", mode="r", shape=shape)
print("Read-only overall sum of slice:", np.sum(ro_data[:10, :10]))

# Skipping 800 bytes (e.g. 100 float64 elements header offset)
offset_data = np.memmap(filename, dtype="float64", mode="r", offset=800, shape=(1000, 1000))
print("Offset memmap view shape:", offset_data.shape)

Common Errors & Solutions 🔝

• ValueError: shape does not match size of file: Occurs when reopening a memmap file with a shape or dtype that does not align with byte length. Fix: Ensure shape count matches byte size ($N imes ext{dtype\_bytes}$).
• Unflushed Disk Writes: Memory updates stay in operating system page cache until explicitly committed. Fix: Call fp.flush() before closing process handlers.

Frequently Asked Questions (FAQ) 🔝

Practical Exercise 🔝

Create a 5000 x 5000 float32 memory-mapped array file named exercise_memmap.dat. Initialize all elements to 0.0, update the first row to 100.0, flush changes to disk, and verify matrix dimensions.

Exercise Solution 🔝

import numpy as np

fp_ex = np.memmap("exercise_memmap.dat", dtype="float32", mode="w+", shape=(5000, 5000))
fp_ex[0, :] = 100.0
fp_ex.flush()

assert fp_ex[0, 0] == 100.0
print("Exercise completed successfully!")

Further Reading & Related Tutorials 🔝

• NumPy Practical Applications Hub — Master list of NumPy performance engineering modules.
• NumPy Vectorization vs Loops — High-speed SIMD instruction benchmarks.

Interactive Companion Jupyter Notebook

Practice memory mapping multi-gigabyte binary matrices dynamically in your browser.

View on GitHub

Subhendu Mohapatra

Author & Instructor at plus2net

I write and maintain practical tutorials on Python, PHP, SQL, JavaScript, HTML, jQuery, and web development at plus2net. The tutorials focus on clear explanations, working examples, and code that readers can test and adapt while learning.

About the author   Watch on YouTube

Connect: