Pandas: Merge & Fill Nulls with combine_first()

Combine two DataFrames by prioritizing non-null values from the primary object and patching missing entries (NaN) with matching values from a secondary fallback object using Pandas combine_first().

Show Table of Contents ↓

Introduction to combine_first() 🔝

The combine_first() method reconciles two DataFrames or Series by filling null values in the calling object with non-null values from the passed secondary object at matching index and column locations.

Create primary and secondary datasets 🔝

We start with a primary student DataFrame containing missing scores and a secondary backup DataFrame with full scores.

import pandas as pd
import numpy as np

df1 = pd.DataFrame({
    "MATH": [85, np.nan, 78],
    "SCIENCE": [np.nan, 92, 80]
}, index=["Ravi", "Raju", "Alex"])

df2 = pd.DataFrame({
    "MATH": [70, 90, 75],
    "SCIENCE": [88, 95, 82]
}, index=["Ravi", "Raju", "Alex"])

print(df1)
assert pd.isna(df1.loc["Raju", "MATH"])

Expected output

      MATH  SCIENCE
Ravi  85.0      NaN
Raju   NaN     92.0
Alex  78.0     80.0

Basic combine_first operation 🔝

Call df1.combine_first(df2) to retain non-null values in df1 and fill missing entries using df2.

result = df1.combine_first(df2)
print(result)
assert result.loc["Raju", "MATH"] == 90.0 and result.loc["Ravi", "SCIENCE"] == 88.0

Expected output

      MATH  SCIENCE
Ravi  85.0     88.0
Raju  90.0     92.0
Alex  78.0     80.0

Index alignment and new rows 🔝

If df2 contains additional index rows or columns not present in df1, combine_first() automatically aligns indices and appends the new entries.

df3 = pd.DataFrame({"MATH": [95], "SCIENCE": [89]}, index=["John"])
combined = df1.combine_first(df3)
print(combined)
assert "John" in combined.index

Expected output

      MATH  SCIENCE
Alex  78.0     80.0
John  95.0     89.0
Raju   NaN     92.0
Ravi  85.0      NaN

Combining Series objects 🔝

combine_first() works seamlessly on Pandas Series objects to perform prioritized element-wise null filling.

s1 = pd.Series([10, np.nan, 30], index=["a", "b", "c"])
s2 = pd.Series([15, 25, 35], index=["a", "b", "c"])
s_combined = s1.combine_first(s2)
print(s_combined)
assert s_combined["b"] == 25.0

Expected output

a    10.0
b    25.0
c    30.0
dtype: float64

Exercise: patch missing inventory scores 🔝

Create primary Series s_main = pd.Series([100, np.nan, 300], index=["item1", "item2", "item3"]) and secondary Series s_backup = pd.Series([150, 200, 350], index=["item1", "item2", "item3"]). Use combine_first() to patch the missing value and assert that item2 equals 200.

Exercise solution 🔝

Solution creating the Series objects and executing combine_first().

s_main = pd.Series([100, np.nan, 300], index=["item1", "item2", "item3"])
s_backup = pd.Series([150, 200, 350], index=["item1", "item2", "item3"])
answer = s_main.combine_first(s_backup)
print(answer)
assert answer["item2"] == 200.0

Expected output

item1    100.0
item2    200.0
item3    300.0
dtype: float64

Practice in Google Colab 🔝

Interactive Colab Notebook

Practice combine_first() examples interactively directly in your browser:

View on GitHub

Reference: Pandas combine_first() documentation.

Subhendu Mohapatra

Author & Instructor at plus2net

I write and maintain practical tutorials on Python, PHP, SQL, JavaScript, HTML, jQuery, and web development at plus2net. The tutorials focus on clear explanations, working examples, and code that readers can test and adapt while learning.

About the author Watch on YouTube

Connect:

Subscribe to our YouTube Channel here