Pandas isnull() and isna(): Find and Audit Missing Values
Use isnull() or isna() to mark missing values True and present values False. Count missing cells, select incomplete records and distinguish absent input from invalid values before choosing a cleaning policy.
True means missing, False means present 🔝
isnull() detects missing values without changing them. isna() is an equivalent name; notna()/notnull() return the inverse. Use np.nan rather than the removed NumPy spelling np.NaN.
import pandas as pd
import numpy as np
df = pd.DataFrame({"NAME": ["Ravi", "Raju", "Alex", None, "King", None], "ID": [1, 2, np.nan, 4, 5, 6], "MATH": [80, 40, 70, 70, 82, 30], "ENGLISH": [81, 70, 40, 50, np.nan, 30]})
print(df.isnull())
assert df.isnull().equals(df.isna())
assert df.isnull().equals(~df.notna())Expected output
NAME ID MATH ENGLISH
0 False False False False
1 False False False False
2 False True False False
3 True False False False
4 False False False True
5 True False False FalseCreate the student practice table 🔝
The embedded table recreates the displayed student example using consistent columns id, name, class1, mark and gender. The student workbook link provides an optional Excel route. Check imported column labels before using them; see read_excel().
def student_data():
return pd.DataFrame([
[1, "John Deo", "Four", 75, "female"],
[2, "Max Ruin", "Three", 85, "male"],
[None, "Arnold", "Three", 55, "male"],
[4, "Krish Star", "Four", 60, "female"],
[None, "John Mike", "Four", 60, "female"],
[6, "Alex John", "Four", 55, None],
[7, "My John Rob", "Five", 78, "male"],
[None, None, None, None, None],
[9, "Tes Qry", "Six", 78, "male"],
[10, None, "Four", 55, "female"],
[11, "Ronald", "Six", None, "female"],
[12, "Recky", "Six", 94, "female"]
], columns=["id", "name", "class1", "mark", "gender"])
df = student_data()
print(df)
# Optional after uploading the workbook to Colab:
# df = pd.read_excel("student-isnull.xlsx")Expected output
id name class1 mark gender
0 1.0 John Deo Four 75.0 female
1 2.0 Max Ruin Three 85.0 male
2 NaN Arnold Three 55.0 male
3 4.0 Krish Star Four 60.0 female
4 NaN John Mike Four 60.0 female
5 6.0 Alex John Four 55.0 NaN
6 7.0 My John Rob Five 78.0 male
7 NaN NaN NaN NaN NaN
8 9.0 Tes Qry Six 78.0 male
9 10.0 NaN Four 55.0 female
10 11.0 Ronald Six NaN female
11 12.0 Recky Six 94.0 femaleCheck for missing values in the table or a column 🔝
A global any() over the mask asks whether at least one missing cell exists. One column may be complete while another contains missing data. Select the fields appropriate to the question.
df = student_data()
print("Any missing cell:", df.isnull().to_numpy().any())
print("Any missing name:", df["name"].isnull().any())
print("Any missing name or mark:", df[["name", "mark"]].isnull().to_numpy().any())
assert df.isnull().to_numpy().any()Expected output
Any missing cell: True
Any missing name: True
Any missing name or mark: TrueSelect rows missing an ID or a name 🔝
Apply a column mask with loc to retain the full records for review. This does not delete or fill source values.
df = student_data()
print("Missing ID:")
print(df.loc[df["id"].isnull()])
print("Missing name:")
print(df.loc[df["name"].isnull()])
assert df.index[df["id"].isnull()].tolist() == [2, 4, 7]
assert df.index[df["name"].isnull()].tolist() == [7, 9]Expected output
Missing ID:
id name class1 mark gender
2 NaN Arnold Three 55.0 male
4 NaN John Mike Four 60.0 female
7 NaN NaN NaN NaN NaN
Missing name:
id name class1 mark gender
7 NaN NaN NaN NaN NaN
9 10.0 NaN Four 55.0 femaleCompare any missing field with all fields missing 🔝
any(axis=1) marks a row with at least one missing field. all(axis=1) marks a row where every considered field is missing. Use a deliberate set of required columns; optional fields should not automatically cause rejection.
df = student_data()
any_missing = df.isnull().any(axis=1)
all_missing = df.isnull().all(axis=1)
print("Rows with any missing field:", int(any_missing.sum()))
print("Entirely missing rows:")
print(df.loc[all_missing])
required_missing = df[["id", "name"]].isnull().any(axis=1)
print("Missing at least one required field:")
print(df.loc[required_missing])
assert any_missing.sum() == 6
assert df.index[all_missing].tolist() == [7]Expected output
Rows with any missing field: 6
Entirely missing rows:
id name class1 mark gender
7 NaN NaN NaN NaN NaN
Missing at least one required field:
id name class1 mark gender
2 NaN Arnold Three 55.0 male
4 NaN John Mike Four 60.0 female
7 NaN NaN NaN NaN NaN
9 10.0 NaN Four 55.0 femaleCount missing cells per column, row and table 🔝
Summing a Boolean mask counts True values. Column totals count missing cells, while summing any(axis=1) counts affected rows. Do not confuse those quantities.
df = student_data()
print("Missing by column:")
print(df.isnull().sum())
print("Missing by row:")
print(df.isnull().sum(axis=1))
print("Total missing cells:", df.isnull().sum().sum())
print("Rows with missing cells:", df.isnull().any(axis=1).sum())
assert df.isnull().sum().tolist() == [3, 2, 1, 2, 2]
assert df.isnull().sum().sum() == 10Expected output
Missing by column:
id 3
name 2
class1 1
mark 2
gender 2
dtype: int64
Missing by row:
0 0
1 0
2 1
3 0
4 1
5 1
6 0
7 5
8 0
9 1
10 1
11 0
dtype: int64
Total missing cells: 10
Rows with missing cells: 6Report missing percentages with a clear denominator 🔝
Divide column counts by the number of rows. For an empty table, a missing percentage is undefined; report that rather than inventing zero. A percentage describes completeness, not whether present values are valid.
df = student_data()
summary = pd.DataFrame({"missing": df.isna().sum(), "present": df.notna().sum()})
summary["missing_percent"] = summary["missing"].div(len(df)).mul(100) if len(df) else np.nan
print(summary.round(2))
assert summary["missing"].sum() + summary["present"].sum() == df.sizeExpected output
missing present missing_percent
id 3 9 25.00
name 2 10 16.67
class1 1 11 8.33
mark 2 10 16.67
gender 2 10 16.67Recognise missing sentinels without treating text as missing 🔝
None, np.nan, pd.NA and pd.NaT are detected as missing. The strings "NaN" and "None", blank strings, zero, False and infinity are not automatically missing in an in-memory object Series. File readers can recognise additional tokens according to their import settings.
values = pd.Series([None, np.nan, pd.NA, pd.NaT, "NaN", "None", "", 0, False, np.inf], dtype="object")
print(pd.DataFrame({"value": values, "missing": values.isna()}))
assert values.isna().tolist() == [True, True, True, True, False, False, False, False, False, False]Expected output
value missing
0 None True
1 NaN True
2 <NA> True
3 NaT True
4 NaN False
5 None False
6 False
7 0 False
8 False False
9 inf FalseNormalise blank text before auditing 🔝
Strip surrounding whitespace and map empty values to pd.NA only when your schema defines them as absent. Do not replace meaningful text tokens globally. See text replacement and text normalisation.
text = pd.Series(["Pen", "", " ", None, "None"], dtype="string")
trimmed = text.str.strip()
normalised = trimmed.mask(trimmed.eq(""), pd.NA)
print(pd.DataFrame({"raw": text, "normalised": normalised, "missing": normalised.isna()}))
assert normalised.isna().sum() == 3
assert normalised.iloc[4] == "None"Expected output
raw normalised missing
0 Pen Pen False
1 <NA> True
2 <NA> True
3 <NA> <NA> True
4 None None FalsePractical workflow: audit missing and invalid order data 🔝
This synthetic extract requires a non-blank product and a finite non-negative amount. Parsing can create missing values from invalid text. Track source missingness separately from parse failures instead of classifying every failure as absent input.
raw = pd.DataFrame({"order_id": [101, 102, 103, 104, 105, 106], "product": [" Pen ", "", None, "Book", "Mug", "Pencil"], "amount": ["5", "7", None, "bad", "inf", "-3"]})
product_text = raw["product"].astype("string").str.strip()
product = product_text.mask(product_text.eq(""), pd.NA)
amount_text = raw["amount"].astype("string").str.strip()
amount_text = amount_text.mask(amount_text.eq(""), pd.NA)
amount = pd.to_numeric(amount_text, errors="coerce")
finite = pd.Series(np.isfinite(amount.to_numpy(dtype=float, na_value=np.nan)), index=raw.index)
audit = pd.DataFrame({
"missing_product": product.isna(),
"missing_amount_input": amount_text.isna(),
"unparseable_amount": amount_text.notna() & amount.isna(),
"nonfinite_amount": amount.notna() & ~finite,
"negative_amount": amount.lt(0).fillna(False)
})
needs_review = audit.any(axis=1)
print(raw.join(audit))
print("Reason counts:")
print(audit.sum())
assert audit["unparseable_amount"].sum() == 1
assert needs_review.sum() == 5Expected output
order_id product ... nonfinite_amount negative_amount
0 101 Pen ... False False
1 102 ... False False
2 103 NaN ... False False
3 104 Book ... False False
4 105 Mug ... True False
5 106 Pencil ... False True
[6 rows x 8 columns]
Reason counts:
missing_product 2
missing_amount_input 1
unparseable_amount 1
nonfinite_amount 1
negative_amount 1
dtype: Int64Retain review reasons and export accepted records 🔝
Reason counts can overlap; use any(axis=1) to count affected rows. Keep the raw input and audit flags together. Filling an unknown amount with zero would change its meaning. See fillna() and dropna() for explicit policies.
review = raw.loc[needs_review].join(audit.loc[needs_review])
clean = raw.loc[~needs_review].copy()
clean["product"] = product.loc[~needs_review]
clean["amount"] = amount.loc[~needs_review]
print("Accepted records:")
print(clean)
print("Review records:")
print(review)
assert len(raw) == len(clean) + len(review)
assert clean[["product", "amount"]].notna().all().all()
assert clean["order_id"].tolist() == [101]
# Optional Colab exports:
# clean.to_csv("accepted_orders.csv", index=False)
# review.to_csv("missing_data_audit.csv", index=False)Expected output
Accepted records:
order_id product amount
0 101 Pen 5.0
Review records:
order_id product ... nonfinite_amount negative_amount
1 102 ... False False
2 103 NaN ... False False
3 104 Book ... False False
4 105 Mug ... True False
5 106 Pencil ... False True
[5 rows x 8 columns]Common questions 🔝
Does isnull() fill missing values? No, it returns a mask. Is isna() different? It is an alias. Why is a blank False? Empty text is not a missing sentinel until normalised. Can I compare a value to NaN with ==? Use isna() instead. Do index labels count? DataFrame.isna() checks data cells; inspect the index separately. Does zero mean missing? Only if your specific schema says so. Should all incomplete rows be removed? Review required fields and intended use; thresh supports a completeness requirement.
Exercise: audit two required fields 🔝
Create names Ravi, missing, Alex and marks 75, 80, missing. Count missing values per column, select rows missing either required field and retain complete rows. Expect one missing name, one missing mark and two review rows.
Exercise solution 🔝
Reduce the missing-value mask across columns to identify affected rows.
practice = pd.DataFrame({"name": ["Ravi", None, "Alex"], "mark": [75, 80, None]})
missing = practice[["name", "mark"]].isna()
review_mask = missing.any(axis=1)
print(missing.sum())
print("Review:")
print(practice.loc[review_mask])
print("Complete:")
print(practice.loc[~review_mask])
assert missing.sum().tolist() == [1, 1]
assert review_mask.sum() == 2Expected output
name 1
mark 1
dtype: int64
Review:
name mark
1 NaN 80.0
2 Alex NaN
Complete:
name mark
0 Ravi 75.0Practice in Google Colab 🔝
Open in Google Colab View on GitHub
Run the examples in order, change missing, blank and invalid inputs, then compare missing-value masks and audit reasons. Save your own copy to keep edits. All sample tables are included in the notebook.
Continue with the Data Cleaning hub, notnull()/notna(), loc selection, at access, mask() and iloc selection.
Reference: Pandas DataFrame.isna documentation.
Subhendu Mohapatra
Author & Instructor at plus2net
I write and maintain practical tutorials on Python, PHP, SQL, JavaScript, HTML, jQuery, and web development at plus2net. The tutorials focus on clear explanations, working examples, and code that readers can test and adapt while learning.
Subscribe to our YouTube Channel here
Python Video Tutorials