Pandas notnull() and notna(): Check Present Values
Use notnull() or notna() to mark non-missing values True and missing values False. Combine the mask with row selection, any(), all() or sum() to inspect completeness. Present data still need validation before use.
True means present, False means missing 🔝
notnull() returns a Boolean mask with the same shape as the source. None and NaN produce False. notna() is an equivalent name; isna()/isnull() return the inverse mask.
import pandas as pd
import numpy as np
df = pd.DataFrame({"NAME": ["Ravi", "Raju", "Alex", None, "King", None], "ID": [1, 2, np.nan, 4, 5, 6], "MATH": [80, 40, 70, 70, 82, 30], "ENGLISH": [81, 70, 40, 50, np.nan, 30]})
print(df.notnull())
assert df.notnull().equals(df.notna())
assert df.notnull().equals(~df.isna())
assert not df.notnull().loc[3, "NAME"]Expected output
NAME ID MATH ENGLISH
0 True True True True
1 True True True True
2 True False True True
3 False True True True
4 True True True False
5 False True True TrueCreate the student table for filtering 🔝
This embedded table reproduces the values displayed in the tutorial. The student workbook link provides the Excel route; see read_excel(). The notebook uses the embedded table to avoid an upload dependency.
def student_data():
return pd.DataFrame([
[1, "John Deo", "Four", 75, "female"],
[2, "Max Ruin", "Three", 85, "male"],
[None, "Arnold", "Three", 55, "male"],
[4, "Krish Star", "Four", 60, "female"],
[None, "John Mike", "Four", 60, "female"],
[6, "Alex John", "Four", 55, None],
[7, "My John Rob", "Five", 78, "male"],
[None, None, None, None, None],
[9, "Tes Qry", "Six", 78, "male"],
[10, None, "Four", 55, "female"],
[11, "Ronald", "Six", None, "female"],
[12, "Recky", "Six", 94, "female"]
], columns=["id", "name", "class1", "mark", "gender"])
df = student_data()
print(df)
# Optional after uploading the workbook to Colab:
# df = pd.read_excel("student-isnull.xlsx")Expected output
id name class1 mark gender
0 1.0 John Deo Four 75.0 female
1 2.0 Max Ruin Three 85.0 male
2 NaN Arnold Three 55.0 male
3 4.0 Krish Star Four 60.0 female
4 NaN John Mike Four 60.0 female
5 6.0 Alex John Four 55.0 NaN
6 7.0 My John Rob Five 78.0 male
7 NaN NaN NaN NaN NaN
8 9.0 Tes Qry Six 78.0 male
9 10.0 NaN Four 55.0 female
10 11.0 Ronald Six NaN female
11 12.0 Recky Six 94.0 femaleCheck whether any value is present 🔝
A global any() over the Boolean array asks whether at least one present cell exists. It does not mean every row is complete or that a value is valid. A Series check inspects one column only.
df = student_data()
print("Any present cell:", df.notnull().to_numpy().any())
print("Any name:", df["name"].notnull().any())
print("Any name or mark cell:", df[["name", "mark"]].notnull().to_numpy().any())
assert df.notnull().to_numpy().any()Expected output
Any present cell: True
Any name: True
Any name or mark cell: TrueDistinguish entirely missing, partly present and complete rows 🔝
any(axis=1) checks for at least one present field per row. all(axis=1) checks that every considered field is present. Negating any() selects entirely missing rows.
df = student_data()
has_any = df.notnull().any(axis=1)
complete = df.notnull().all(axis=1)
print("Entirely missing rows:")
print(df.loc[~has_any])
print("Rows with at least one present field:")
print(df.loc[has_any])
print("Complete rows:", int(complete.sum()))
assert df.index[~has_any].tolist() == [7]
assert complete.sum() == 6Expected output
Entirely missing rows:
id name class1 mark gender
7 NaN NaN NaN NaN NaN
Rows with at least one present field:
id name class1 mark gender
0 1.0 John Deo Four 75.0 female
1 2.0 Max Ruin Three 85.0 male
2 NaN Arnold Three 55.0 male
3 4.0 Krish Star Four 60.0 female
4 NaN John Mike Four 60.0 female
5 6.0 Alex John Four 55.0 NaN
6 7.0 My John Rob Five 78.0 male
8 9.0 Tes Qry Six 78.0 male
9 10.0 NaN Four 55.0 female
10 11.0 Ronald Six NaN female
11 12.0 Recky Six 94.0 female
Complete rows: 6Filter rows with a present ID or name 🔝
Apply a one-dimensional mask with loc to select complete rows from a particular column. The other fields may still be missing.
df = student_data()
print("Rows with an ID:")
print(df.loc[df["id"].notnull()])
print("Rows with a name:")
print(df.loc[df["name"].notnull()])
assert df["id"].notnull().sum() == 9
assert df["name"].notnull().sum() == 10Expected output
Rows with an ID:
id name class1 mark gender
0 1.0 John Deo Four 75.0 female
1 2.0 Max Ruin Three 85.0 male
3 4.0 Krish Star Four 60.0 female
5 6.0 Alex John Four 55.0 NaN
6 7.0 My John Rob Five 78.0 male
8 9.0 Tes Qry Six 78.0 male
9 10.0 NaN Four 55.0 female
10 11.0 Ronald Six NaN female
11 12.0 Recky Six 94.0 female
Rows with a name:
id name class1 mark gender
0 1.0 John Deo Four 75.0 female
1 2.0 Max Ruin Three 85.0 male
2 NaN Arnold Three 55.0 male
3 4.0 Krish Star Four 60.0 female
4 NaN John Mike Four 60.0 female
5 6.0 Alex John Four 55.0 NaN
6 7.0 My John Rob Five 78.0 male
8 9.0 Tes Qry Six 78.0 male
10 11.0 Ronald Six NaN female
11 12.0 Recky Six 94.0 femaleRequire several fields together 🔝
Combine masks with & for AND, or use all(axis=1) over required columns. Use | or any(axis=1) when at least one of the selected fields must be present. Parenthesise comparisons when mixing conditions.
df = student_data()
mask = df["id"].notnull() & df["name"].notnull()
required = df[["id", "name"]].notnull().all(axis=1)
print(df.loc[required])
assert mask.equals(required)
assert required.sum() == 8Expected output
id name class1 mark gender
0 1.0 John Deo Four 75.0 female
1 2.0 Max Ruin Three 85.0 male
3 4.0 Krish Star Four 60.0 female
5 6.0 Alex John Four 55.0 NaN
6 7.0 My John Rob Five 78.0 male
8 9.0 Tes Qry Six 78.0 male
10 11.0 Ronald Six NaN female
11 12.0 Recky Six 94.0 femaleCount present and missing cells 🔝
Summing Boolean masks counts True values. Sum once for column counts and twice for total cells. These are cell counts, not row counts. See DataFrame.count() and sum().
df = student_data()
print("Present by column:")
print(df.notnull().sum())
print("Missing by column:")
print(df.isna().sum())
print("Total present cells:", df.notnull().sum().sum())
print("Total missing cells:", df.isna().sum().sum())
assert df.notnull().sum().tolist() == [9, 10, 11, 10, 10]
assert df.notnull().sum().sum() == 50
assert df.notnull().sum().sum() + df.isna().sum().sum() == df.sizeExpected output
Present by column:
id 9
name 10
class1 11
mark 10
gender 10
dtype: int64
Missing by column:
id 3
name 2
class1 1
mark 2
gender 2
dtype: int64
Total present cells: 50
Total missing cells: 10Blank text is present until normalised 🔝
Empty strings, whitespace, zero and False are not automatically missing. Define which blanks represent absent input, then normalise only the appropriate text fields.
text = pd.Series(["Pen", "", " ", None], dtype="string")
print("Before normalising blanks:")
print(text.notnull())
trimmed = text.str.strip()
normalised = trimmed.mask(trimmed.eq(""), pd.NA)
print("After normalising blanks:")
print(normalised.notnull())
print("Zero and False are present:", pd.Series([0, False], dtype="object").notnull().tolist())
assert normalised.notnull().tolist() == [True, False, False, False]Expected output
Before normalising blanks:
0 True
1 True
2 True
3 False
dtype: bool
After normalising blanks:
0 True
1 False
2 False
3 False
dtype: bool
Zero and False are present: [True, True]Present numeric data may still be invalid 🔝
Infinity is not missing, and bad numeric text is present before parsing. For usable amounts, parse values and separately require finite, non-negative numbers. notnull() alone is not a data-validation rule.
raw_amount = pd.Series(["12.50", "bad", None, "inf", "-3"], dtype="string")
amount = pd.to_numeric(raw_amount, errors="coerce")
finite = pd.Series(np.isfinite(amount.to_numpy(dtype=float, na_value=np.nan)), index=amount.index)
usable = amount.notnull() & finite & amount.ge(0).fillna(False)
print(pd.DataFrame({"raw": raw_amount, "parsed": amount, "present": amount.notnull(), "usable": usable}))
assert usable.tolist() == [True, False, False, False, False]Expected output
raw parsed present usable
0 12.50 12.5 True True
1 bad <NA> False False
2 <NA> <NA> False False
3 inf inf True False
4 -3 -3.0 True FalsePractical workflow: validate an order extract 🔝
This synthetic rule requires a non-blank product and a finite non-negative amount. Keep rejected rows rather than replacing uncertain amounts with zero. The raw source remains available for review.
raw = pd.DataFrame({"order_id": [101, 102, 103, 104, 105], "product": [" Pen ", "", None, "Book", "Mug"], "amount": ["5", "7", "9", "bad", "inf"]})
product = raw["product"].astype("string").str.strip()
product = product.mask(product.eq(""), pd.NA)
amount = pd.to_numeric(raw["amount"], errors="coerce")
finite = pd.Series(np.isfinite(amount.to_numpy(dtype=float, na_value=np.nan)), index=raw.index)
keep = product.notnull() & amount.notnull() & finite & amount.ge(0).fillna(False)
clean = raw.loc[keep].copy()
clean["product"] = product.loc[keep]
clean["amount"] = amount.loc[keep]
review = raw.loc[~keep].copy()
print(clean)
print("Review rows:")
print(review)
assert clean["order_id"].tolist() == [101]Expected output
order_id product amount
0 101 Pen 5.0
Review rows:
order_id product amount
1 102 7
2 103 NaN 9
3 104 Book bad
4 105 Mug infValidate the result and record review reasons 🔝
Reason counts may overlap. Compare total retained and review rows separately. Filling or removing missing data is a later policy decision; see fillna(), dropna() and completeness thresholds.
print("Missing or blank products:", int(product.isna().sum()))
print("Missing or unparseable amounts:", int(amount.isna().sum()))
print("Non-finite parsed amounts:", int((amount.notna() & ~finite).sum()))
assert len(raw) == len(clean) + len(review)
assert clean[["product", "amount"]].notnull().all().all()
print(clean.to_csv(index=False).rstrip())
# Optional Colab exports:
# clean.to_csv("usable_orders.csv", index=False)
# review.to_csv("orders_for_review.csv", index=False)Expected output
Missing or blank products: 2
Missing or unparseable amounts: 1
Non-finite parsed amounts: 1
order_id,product,amount
101,Pen,5.0Common questions 🔝
Is notnull() different from notna()? They are aliases. Does it find missing values? It marks present values True; use isnull()/isna() for the missing mask. Does it modify data? No. Why is an empty string True? It is not a missing sentinel. Can I test NaN with ==? Use isna()/notna() instead. Is a row with any present field complete? No; use all() over required fields. Does notnull() mean valid? No; apply type, range and format checks separately.
Exercise: require both a name and a mark 🔝
Create names Ravi, missing, Alex and marks 75, 80, missing. Retain rows where both fields are present, and count missing cells. Expect one retained row and two missing cells.
Exercise solution 🔝
all(axis=1) over the required fields implements the AND rule.
practice = pd.DataFrame({"name": ["Ravi", None, "Alex"], "mark": [75, 80, None]})
mask = practice[["name", "mark"]].notnull().all(axis=1)
print(practice.loc[mask])
print("Missing cells:", practice.isna().sum().sum())
assert mask.tolist() == [True, False, False]
assert practice.isna().sum().sum() == 2Expected output
name mark
0 Ravi 75.0
Missing cells: 2Practice in Google Colab 🔝
Open in Google Colab View on GitHub
Run the examples in order, change missing, blank and invalid inputs, then inspect masks and review rows. Save your own copy to keep edits. All sample tables are included in the notebook.
Continue with the Data Cleaning hub, missing-value masks, loc selection, at access, mask() and iloc selection.
Reference: Pandas DataFrame.notna documentation.
Subhendu Mohapatra
Author & Instructor at plus2net
I write and maintain practical tutorials on Python, PHP, SQL, JavaScript, HTML, jQuery, and web development at plus2net. The tutorials focus on clear explanations, working examples, and code that readers can test and adapt while learning.
Subscribe to our YouTube Channel here
Python Video Tutorials