Pandas dropna(thresh): Require Enough Non-Missing Data
Use dropna(thresh=n) to require at least n non-missing values in each row or column. Specify the fields being assessed, round percentage requirements up to a whole count and review excluded data before deleting it.
Create the student table with missing values 🔝
The table uses id as its index, so id is not counted among the four data columns. Each example starts from a fresh copy. The student workbook link is available for the Excel route; see read_excel(). The embedded examples do not depend on that download.
import pandas as pd
import math
def student_data():
rows = [
[1, "John Deo", "Four", 75, "female"],
[2, "Max Ruin", "Three", 85, "male"],
[None, "Arnold", "Three", 55, "male"],
[4, "Krish Star", "Four", 60, "female"],
[None, "John Mik", "Four", 60, "female"],
[6, "Alex Jo", "Four", 55, None],
[7, "My John", "Five", 78, "male"],
[None, None, None, None, None],
[9, "Tes Qry", "Six", 78, "male"],
[10, None, "Four", 55, "female"],
[11, "Ronald", "Six", None, "female"],
[12, "Recky", "Six", 94, "female"],
[13, "Ron", None, 55, None],
[14, "King", None, None, None]
]
return pd.DataFrame(rows, columns=["id", "name", "class1", "mark", "gender"]).set_index("id")
df = student_data()
print(df)
print("Non-missing counts per column:")
print(df.count())
assert df.shape == (14, 4)Expected output
name class1 mark gender
id
1.0 John Deo Four 75.0 female
2.0 Max Ruin Three 85.0 male
NaN Arnold Three 55.0 male
4.0 Krish Star Four 60.0 female
NaN John Mik Four 60.0 female
6.0 Alex Jo Four 55.0 NaN
7.0 My John Five 78.0 male
NaN NaN NaN NaN NaN
9.0 Tes Qry Six 78.0 male
10.0 NaN Four 55.0 female
11.0 Ronald Six NaN female
12.0 Recky Six 94.0 female
13.0 Ron NaN 55.0 NaN
14.0 King NaN NaN NaN
Non-missing counts per column:
name 12
class1 11
mark 11
gender 10
dtype: int64Keep rows with at least two non-missing fields 🔝
axis=0 removes rows. thresh=2 counts present values in the data columns; it is not the maximum number of missing cells. Rows with zero or one present field are removed.
df = student_data()
result = df.dropna(axis=0, thresh=2)
print(result)
assert len(result) == 12
assert result.notna().sum(axis=1).ge(2).all()Expected output
name class1 mark gender
id
1.0 John Deo Four 75.0 female
2.0 Max Ruin Three 85.0 male
NaN Arnold Three 55.0 male
4.0 Krish Star Four 60.0 female
NaN John Mik Four 60.0 female
6.0 Alex Jo Four 55.0 NaN
7.0 My John Five 78.0 male
9.0 Tes Qry Six 78.0 male
10.0 NaN Four 55.0 female
11.0 Ronald Six NaN female
12.0 Recky Six 94.0 female
13.0 Ron NaN 55.0 NaNKeep rows with at least three non-missing fields 🔝
Raise the requirement to three. Start from the source, not the result of another filter. Missing index labels still do not affect the count.
df = student_data()
result = df.dropna(axis=0, thresh=3)
print(result)
assert len(result) == 11Expected output
name class1 mark gender
id
1.0 John Deo Four 75.0 female
2.0 Max Ruin Three 85.0 male
NaN Arnold Three 55.0 male
4.0 Krish Star Four 60.0 female
NaN John Mik Four 60.0 female
6.0 Alex Jo Four 55.0 NaN
7.0 My John Five 78.0 male
9.0 Tes Qry Six 78.0 male
10.0 NaN Four 55.0 female
11.0 Ronald Six NaN female
12.0 Recky Six 94.0 femaleKeep columns with at least eleven non-missing values 🔝
axis=1 removes columns. The threshold counts values down the rows. The original four-column table has counts 12, 11, 11 and 10, so gender is removed.
df = student_data()
result = df.dropna(axis=1, thresh=11)
print(result)
assert result.columns.tolist() == ["name", "class1", "mark"]Expected output
name class1 mark
id
1.0 John Deo Four 75.0
2.0 Max Ruin Three 85.0
NaN Arnold Three 55.0
4.0 Krish Star Four 60.0
NaN John Mik Four 60.0
6.0 Alex Jo Four 55.0
7.0 My John Five 78.0
NaN NaN NaN NaN
9.0 Tes Qry Six 78.0
10.0 NaN Four 55.0
11.0 Ronald Six NaN
12.0 Recky Six 94.0
13.0 Ron NaN 55.0
14.0 King NaN NaNRequire at least 70 percent of fields in each row 🔝
Convert the fraction to an integer requirement with math.ceil(). With four fields, 70 percent means at least three present values. Define which fields count before choosing the percentage.
df = student_data()
fraction = 0.70
required = math.ceil(df.shape[1] * fraction)
result = df.dropna(axis=0, thresh=required)
print("Required fields:", required)
print(result)
assert required == 3 and len(result) == 11Expected output
Required fields: 3
name class1 mark gender
id
1.0 John Deo Four 75.0 female
2.0 Max Ruin Three 85.0 male
NaN Arnold Three 55.0 male
4.0 Krish Star Four 60.0 female
NaN John Mik Four 60.0 female
6.0 Alex Jo Four 55.0 NaN
7.0 My John Five 78.0 male
9.0 Tes Qry Six 78.0 male
10.0 NaN Four 55.0 female
11.0 Ronald Six NaN female
12.0 Recky Six 94.0 femaleRequire at least 80 percent of rows in each column 🔝
With fourteen rows, 80 percent requires twelve non-missing values. Calculate the denominator from the intended population. Filtering rows first changes this requirement and potentially the result.
df = student_data()
required = math.ceil(len(df) * 0.80)
result = df.dropna(axis=1, thresh=required)
print("Required rows:", required)
print(result)
assert required == 12 and result.columns.tolist() == ["name"]Expected output
Required rows: 12
name
id
1.0 John Deo
2.0 Max Ruin
NaN Arnold
4.0 Krish Star
NaN John Mik
6.0 Alex Jo
7.0 My John
NaN NaN
9.0 Tes Qry
10.0 NaN
11.0 Ronald
12.0 Recky
13.0 Ron
14.0 KingCount only relevant fields with subset 🔝
For row filtering, subset specifies columns used in the count while retaining the other columns in the output. An always-present helper column should not inflate a completeness score.
df = student_data()
fields = ["name", "mark"]
result = df.dropna(subset=fields, thresh=2)
print(result)
assert len(result) == 10
assert result[fields].notna().all().all()Expected output
name class1 mark gender
id
1.0 John Deo Four 75.0 female
2.0 Max Ruin Three 85.0 male
NaN Arnold Three 55.0 male
4.0 Krish Star Four 60.0 female
NaN John Mik Four 60.0 female
6.0 Alex Jo Four 55.0 NaN
7.0 My John Five 78.0 male
9.0 Tes Qry Six 78.0 male
12.0 Recky Six 94.0 female
13.0 Ron NaN 55.0 NaNChoose a separate output or inplace modification 🔝
inplace=True modifies the DataFrame and returns None. Do not assign that return value to df. Separate output tables make it easier to compare retained and rejected records.
df = student_data()
returned = df.dropna(thresh=2, inplace=True)
print("Retained rows:", len(df))
print("Return value:", returned)
assert returned is None and len(df) == 12Expected output
Retained rows: 12
Return value: NoneDo not combine how and thresh 🔝
how specifies any/all missing-value logic, while thresh specifies a minimum present-value count. Passing both raises TypeError. Omit how when using thresh. This caught error is included to show the invalid combination safely.
df = student_data()
try:
df.dropna(how="any", thresh=2)
except TypeError as error:
print(type(error).__name__ + ": " + str(error))
print("Valid threshold result rows:", len(df.dropna(thresh=2)))Expected output
TypeError: You cannot set both the how and thresh arguments at the same time.
Valid threshold result rows: 12Practical workflow: retain sufficiently complete customer records 🔝
This synthetic rule requires a customer ID plus at least two of email, phone and city. A count across the whole table would wrongly let ID or audit columns satisfy the contact requirement. Normalise blank strings before counting.
raw = pd.DataFrame({"customer_id": [101, 102, 103, 104, None], "email": ["a@example.com", "", None, "d@example.com", "e@example.com"], "phone": ["111", "222", None, " ", "555"], "city": ["Delhi", "Pune", "Mumbai", None, "Kochi"]})
df = raw.copy()
fields = ["email", "phone", "city"]
for field in fields:
text = df[field].astype("string").str.strip()
df[field] = text.mask(text.eq(""), pd.NA)
present = df[fields].notna().sum(axis=1)
keep = df["customer_id"].notna() & present.ge(2)
clean = df.loc[keep].copy()
review = raw.loc[~keep].copy()
print(clean)
print("Review rows:")
print(review)
assert clean["customer_id"].tolist() == [101, 102]Expected output
customer_id email phone city
0 101.0 a@example.com 111 Delhi
1 102.0 <NA> 222 Pune
Review rows:
customer_id email phone city
2 103.0 NaN NaN Mumbai
3 104.0 d@example.com NaN
4 NaN e@example.com 555 KochiCheck completeness without confusing it with validity 🔝
Non-missing does not mean valid: email syntax, phone format and customer identity require separate checks. Retain rejected records for review and justify the threshold based on intended use. See email validation, non-missing counts and fillna().
candidate = df.dropna(subset=fields, thresh=2)
candidate = candidate.loc[candidate["customer_id"].notna()]
assert candidate.equals(clean)
assert clean[fields].notna().sum(axis=1).ge(2).all()
assert len(raw) == len(clean) + len(review)
print("Input:", len(raw), "Retained:", len(clean), "Review:", len(review))
print(clean.to_csv(index=False).rstrip())
# Optional Colab exports:
# clean.to_csv("complete_customer_records.csv", index=False)
# review.to_csv("incomplete_customer_records.csv", index=False)Expected output
Input: 5 Retained: 2 Review: 3
customer_id,email,phone,city
101.0,a@example.com,111,Delhi
102.0,,222,PuneCommon questions 🔝
Does thresh count missing cells? No; it counts non-missing cells. Are blanks and spaces missing? Not automatically; define and normalise them. Does the index count? No. Can a threshold require a specific field? Not by itself; combine a completeness rule with an explicit required-field check. What if thresh exceeds the number of considered fields? No rows can meet that requirement. Which percentage should I use? Choose it from the required use of the data, and inspect which records it excludes. Is it safe to remove sparse columns automatically? Review their meaning first; rare fields may still be important.
Exercise: require two of three measurements 🔝
Create rows [1, 2, missing], [1, missing, missing] and [missing, 2, 3] in columns a, b and c. Retain rows with at least two present measurements. Expect the first and third rows.
Exercise solution 🔝
The requirement applies to the three measurement columns only.
practice = pd.DataFrame({"a": [1, 1, None], "b": [2, None, 2], "c": [None, None, 3]})
answer = practice.dropna(subset=["a", "b", "c"], thresh=2)
print(answer)
assert answer.index.tolist() == [0, 2]Expected output
a b c
0 1.0 2.0 NaN
2 NaN 2.0 3.0Practice in Google Colab 🔝
Open in Google Colab View on GitHub
Run the examples in order, change thresholds, assessed fields and blank inputs, then inspect retained and rejected records. Save your own copy to keep edits. All sample tables are included in the notebook.
Continue with the Data Cleaning hub, dropna(), conditional filtering, column inspection, add_prefix() and add_suffix().
Reference: Pandas DataFrame.dropna documentation.
Subhendu Mohapatra
Author & Instructor at plus2net
I write and maintain practical tutorials on Python, PHP, SQL, JavaScript, HTML, jQuery, and web development at plus2net. The tutorials focus on clear explanations, working examples, and code that readers can test and adapt while learning.
Subscribe to our YouTube Channel here
Python Video Tutorials