NumPy Structured Arrays & Record Arrays: Parse Binary Logs

import numpy as np

# Define composite binary record structure: packet_id, timestamp, signal, status
packet_dtype = np.dtype([
    ('packet_id', 'i4'),
    ('timestamp', 'f8'),
    ('signal_strength', 'f4'),
    ('status_code', 'u2')
])

# Parse raw binary payload directly into structured array memory
packets = np.frombuffer(b"\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\xf0\x3f\x00\x00\x80\x3f\x00\x00", dtype=packet_dtype)
print("Parsed Packet ID:", packets['packet_id'][0])
assert packets['packet_id'][0] == 1

Prerequisites & Required Libraries

Ensure you have NumPy version 1.20 or later installed in your Python environment. Structured arrays and record arrays are built directly into standard NumPy core (numpy.dtype, numpy.recarray), so no external third-party C libraries are required:

pip install numpy

1. Defining Composite Data Types (`np.dtype`)

In standard NumPy arrays, every element must share an identical data type (e.g., all 64-bit floats). Structured arrays solve this limitation by allowing named fields of varying dtypes mapped to fixed byte offsets in memory.

2. Parsing Raw Binary Streams (`np.frombuffer` & `np.fromfile`)

Binary network captures and system telemetry logs stream continuous byte sequences without newline separators. By passing a custom np.dtype schema to np.frombuffer() or np.fromfile(), NumPy zero-copies raw memory directly into structured columnar fields.

import numpy as np

# Define schema for financial tick data: Ticker (S4), Timestamp (f8), Price (f8), Volume (i4)
tick_dtype = np.dtype([
    ('ticker', 'S4'),
    ('timestamp', 'f8'),
    ('price', 'f8'),
    ('volume', 'i4')
])

# Construct raw binary buffer representing 2 ticks
raw_binary_stream = (
    b"AAPL" + np.float64(1625097600.0).tobytes() + np.float64(145.50).tobytes() + np.int32(500).tobytes() +
    b"GOOG" + np.float64(1625097601.0).tobytes() + np.float64(2500.25).tobytes() + np.int32(120).tobytes()
)

# Parse raw binary stream without loops or external libraries
ticks = np.frombuffer(raw_binary_stream, dtype=tick_dtype)

print("Parsed Tickers:", ticks['ticker'])
print("Prices:        ", ticks['price'])
print("Total Volume:  ", ticks['volume'].sum())

assert ticks['ticker'][0] == b"AAPL"
assert ticks['volume'].sum() == 620

3. Record Arrays (`np.recarray`) vs Structured Arrays

Standard structured arrays require dictionary-style string indexing to access fields (e.g., data['price']). Record Arrays (np.recarray) subclass structured arrays to enable convenient attribute-style access (e.g., data.price).

import numpy as np

# Define log entry structure
log_dtype = np.dtype([('sensor_id', 'u2'), ('voltage', 'f4'), ('error', '?')])

# Create a recarray directly
rec_logs = np.array([
    (101, 3.28, False),
    (102, 4.12, True),
    (103, 3.31, False)
], dtype=log_dtype).view(np.recarray)

# Access fields directly via dot notation
print("Voltages via attribute:", rec_logs.voltage)
print("Error Count:           ", rec_logs.error.sum())

assert rec_logs.voltage[0] == 3.28
assert rec_logs.error.sum() == 1

4. Relational Database-Style Querying & Filtering

Because structured arrays align fields in memory, you can execute boolean masking operations across multiple fields simultaneously without copying underlying data buffers or running Python for loops.

import numpy as np

# Network packet capture schema: Source IP, Dest Port, Packet Size (bytes), Latency (ms)
pcap_dtype = np.dtype([
    ('src_ip', 'S15'),
    ('dst_port', 'u2'),
    ('size', 'u4'),
    ('latency', 'f4')
])

packets = np.array([
    (b"192.168.1.10", 80, 1500, 12.4),
    (b"10.0.0.55", 443, 512, 125.8),
    (b"192.168.1.12", 80, 1420, 8.1),
    (b"10.0.0.55", 22, 64, 2.1)
], dtype=pcap_dtype)

# SQL Equivalent: SELECT * FROM packets WHERE dst_port = 80 AND latency < 15.0
query_mask = (packets['dst_port'] == 80) & (packets['latency'] < 15.0)
filtered_packets = packets[query_mask]

print("Filtered Packet Count:", len(filtered_packets))
print("Average Size (bytes) :", filtered_packets['size'].mean())

assert len(filtered_packets) == 2
assert filtered_packets['size'].mean() == 1460.0

5. Nested Sub-Arrays & Struct Alignment

Structured arrays support nested substructures (arrays inside fields), ideal for multi-axis sensor data (e.g., 3D accelerometer readings [x, y, z] attached to a timestamp record).

import numpy as np

# Sub-array schema for 3D coordinates
imu_dtype = np.dtype([
    ('device_id', 'u2'),
    ('accel', 'f4', (3,)),  # 3-element float array for X, Y, Z
    ('gyro', 'f4', (3,))   # 3-element float array for Pitch, Roll, Yaw
])

imu_records = np.array([
    (1, [0.01, 0.98, 9.81], [0.1, 0.0, -0.2]),
    (2, [0.05, 1.02, 9.75], [0.0, 0.2, -0.1])
], dtype=imu_dtype)

# Compute magnitude of Z-axis acceleration across all records
z_accel = imu_records['accel'][:, 2]
print("Z-Axis Accelerations:", z_accel)

assert z_accel[0] == 9.81

Common Errors & Troubleshooting

Frequently Asked Questions

Interactive Companion Jupyter Notebook

Explore interactive code cells, dynamically generate classes, test type vs isinstance comparisons, and run self-validating assertions.

View on GitHub

Further Reading & Related Tutorials: NumPy dtype Reference, NumPy File I/O, Memory Mapping (np.memmap), and NumPy Overview.

Subhendu Mohapatra

Author & Instructor at plus2net

I write and maintain practical tutorials on Python, PHP, SQL, JavaScript, HTML, jQuery, and web development at plus2net. The tutorials focus on clear explanations, working examples, and code that readers can test and adapt while learning.

About the author   Watch on YouTube

Connect: