Pandas dropna(): Remove Missing Rows and Columns
Use dropna() to remove rows or columns according to a missing-value rule. Decide which fields are required, choose how much known data is enough, and review excluded records before calculating results.
Create the student table 🔝
None, NaN, pd.NA and NaT are missing-value representations recognized by Pandas. Use np.nan with current NumPy. Keep the source table and compare each cleaning rule independently.
import pandas as pd
import numpy as np
import math
def student_data():
return pd.DataFrame({
"NAME": ["Ravi", "Raju", None, None, "King", "Alex"],
"ID": [1, 2, 3, np.nan, 5, 6],
"MATH": [80, 40, 70, np.nan, 82, 30],
"ENGLISH": [81, 70, 40, np.nan, np.nan, 30]
})
df = student_data()
print(df)
print("Default result:")
print(df.dropna())
assert df.dropna().index.tolist() == [0, 1, 5]Expected output
NAME ID MATH ENGLISH
0 Ravi 1.0 80.0 81.0
1 Raju 2.0 40.0 70.0
2 NaN 3.0 70.0 40.0
3 NaN NaN NaN NaN
4 King 5.0 82.0 NaN
5 Alex 6.0 30.0 30.0
Default result:
NAME ID MATH ENGLISH
0 Ravi 1.0 80.0 81.0
1 Raju 2.0 40.0 70.0
5 Alex 6.0 30.0 30.0Choose a missing-data rule 🔝
axis=0 removes rows; axis=1 removes columns. The default how="any" removes an item if any considered value is missing. how="all" removes only completely missing items. subset limits the fields considered. thresh requires a minimum number of non-missing values and cannot be combined with how. inplace=False returns a result; inplace=True modifies the source and returns None. ignore_index=True resets labels along the resulting axis. Dropping observations can change the population represented by a report; review exclusions before deciding.
Drop rows with any missing value 🔝
The default removes rows 2, 3 and 4. It checks every column, including the name. This may be too strict when a name is optional for the analysis.
df = student_data()
result = df.dropna(how="any")
print(result)
print("Removed rows:", len(df) - len(result))Expected output
NAME ID MATH ENGLISH
0 Ravi 1.0 80.0 81.0
1 Raju 2.0 40.0 70.0
5 Alex 6.0 30.0 30.0
Removed rows: 3Drop only completely missing rows 🔝
Row 3 has no known values and is removed. Partially filled records remain available for later review.
df = student_data()
result = df.dropna(how="all")
print(result)
assert result.index.tolist() == [0, 1, 2, 4, 5]Expected output
NAME ID MATH ENGLISH
0 Ravi 1.0 80.0 81.0
1 Raju 2.0 40.0 70.0
2 NaN 3.0 70.0 40.0
4 King 5.0 82.0 NaN
5 Alex 6.0 30.0 30.0Require particular columns rather than every field 🔝
With subset=["ENGLISH"], only the English score is required. To require both subjects, include both labels. A missing name does not disqualify a record when it is outside subset.
df = student_data()
print("Require English:")
print(df.dropna(subset=["ENGLISH"]))
print("Require both scores:")
print(df.dropna(subset=["MATH", "ENGLISH"]))
assert df.dropna(subset=["ENGLISH"]).index.tolist() == [0, 1, 2, 5]Expected output
Require English:
NAME ID MATH ENGLISH
0 Ravi 1.0 80.0 81.0
1 Raju 2.0 40.0 70.0
2 NaN 3.0 70.0 40.0
5 Alex 6.0 30.0 30.0
Require both scores:
NAME ID MATH ENGLISH
0 Ravi 1.0 80.0 81.0
1 Raju 2.0 40.0 70.0
2 NaN 3.0 70.0 40.0
5 Alex 6.0 30.0 30.0Drop columns with missing values 🔝
Every column in the student sample contains at least one missing value, so axis=1 with how="any" leaves zero columns. With how="all", no column is removed because each has at least one known value.
df = student_data()
print("Any missing value:")
print(df.dropna(axis=1, how="any"))
print("All missing:")
print(df.dropna(axis=1, how="all"))
assert df.dropna(axis=1, how="any").shape == (6, 0)Expected output
Any missing value:
Empty DataFrame
Columns: []
Index: [0, 1, 2, 3, 4, 5]
All missing:
NAME ID MATH ENGLISH
0 Ravi 1.0 80.0 81.0
1 Raju 2.0 40.0 70.0
2 NaN 3.0 70.0 40.0
3 NaN NaN NaN NaN
4 King 5.0 82.0 NaN
5 Alex 6.0 30.0 30.0Remove a completely missing column 🔝
The modified example makes the English column entirely missing. Removing that field is appropriate only if the resulting report no longer requires English scores.
df = student_data()
df["ENGLISH"] = np.nan
print(df.dropna(axis=1, how="all"))
assert "ENGLISH" not in df.dropna(axis=1, how="all").columnsExpected output
NAME ID MATH
0 Ravi 1.0 80.0
1 Raju 2.0 40.0
2 NaN 3.0 70.0
3 NaN NaN NaN
4 King 5.0 82.0
5 Alex 6.0 30.0Keep rows with enough known values 🔝
thresh=3 means at least three non-missing values, not three missing values. Do not pass how alongside thresh. See the dedicated threshold tutorial for further practice.
df = student_data()
df.loc[2, "NAME"] = "Alex"
df.loc[5, "NAME"] = None
df["ENGLISH"] = np.nan
result = df.dropna(thresh=3)
print(result)
assert result.index.tolist() == [0, 1, 2, 4]Expected output
NAME ID MATH ENGLISH
0 Ravi 1.0 80.0 NaN
1 Raju 2.0 40.0 NaN
2 Alex 3.0 70.0 NaN
4 King 5.0 82.0 NaNApply a threshold to selected measurements 🔝
Require at least one subject score, without counting the ID or name toward the threshold. This retains incomplete score records for a clearly labelled partial analysis.
df = student_data()
result = df.dropna(subset=["MATH", "ENGLISH"], thresh=1)
print(result)
assert result.index.tolist() == [0, 1, 2, 4, 5]Expected output
NAME ID MATH ENGLISH
0 Ravi 1.0 80.0 81.0
1 Raju 2.0 40.0 70.0
2 NaN 3.0 70.0 40.0
4 King 5.0 82.0 NaN
5 Alex 6.0 30.0 30.0Translate a completeness percentage into thresh 🔝
To require at least 75 percent known fields, round the required count upward. For rows, count the selected columns; for columns, count the rows. Percentages must be computed over the intended fields, not unrelated identifiers.
measurements = pd.DataFrame({"a": [1, 1, None], "b": [2, None, None], "c": [3, 3, None], "d": [None, None, None]})
minimum = math.ceil(0.75 * len(measurements.columns))
print("Required valid fields:", minimum)
print(measurements.dropna(thresh=minimum))
assert measurements.dropna(thresh=minimum).index.tolist() == [0]Expected output
Required valid fields: 3
a b c d
0 1.0 2.0 3.0 NoneRemove missing dates represented by NaT 🔝
Parse date text using an explicit format. A missing date stays NaT and can be removed with subset. Missing values in unrelated fields remain. Explore to_datetime() for parsing errors and date formats.
df = student_data()
df["Entry"] = pd.to_datetime(["1/1/2020", "2/1/2020", None, None, "5/1/2020", "1/2/2020"], format="%d/%m/%Y")
print(df)
print("Known entry dates:")
print(df.dropna(subset=["Entry"]))
assert df.dropna(subset=["Entry"]).index.tolist() == [0, 1, 4, 5]Expected output
NAME ID MATH ENGLISH Entry
0 Ravi 1.0 80.0 81.0 2020-01-01
1 Raju 2.0 40.0 70.0 2020-01-02
2 NaN 3.0 70.0 40.0 NaT
3 NaN NaN NaN NaN NaT
4 King 5.0 82.0 NaN 2020-01-05
5 Alex 6.0 30.0 30.0 2020-02-01
Known entry dates:
NAME ID MATH ENGLISH Entry
0 Ravi 1.0 80.0 81.0 2020-01-01
1 Raju 2.0 40.0 70.0 2020-01-02
4 King 5.0 82.0 NaN 2020-01-05
5 Alex 6.0 30.0 30.0 2020-02-01Understand inplace and return values 🔝
Never assign an in-place result back to df: it is None. Returning a new table usually makes comparisons easier. Reset the source before comparing in-place and non-in-place behavior.
df = student_data()
df["ENGLISH"] = np.nan
result = df.dropna(axis=1, how="all", inplace=False)
print("Source columns:", df.columns.tolist())
print("Result columns:", result.columns.tolist())
returned = df.dropna(axis=1, how="all", inplace=True)
print("In-place return:", returned)
print("Updated source:", df.columns.tolist())
assert returned is None
assert "ENGLISH" not in df.columnsExpected output
Source columns: ['NAME', 'ID', 'MATH', 'ENGLISH']
Result columns: ['NAME', 'ID', 'MATH']
In-place return: None
Updated source: ['NAME', 'ID', 'MATH']Reset row labels with ignore_index 🔝
By default, surviving row labels remain unchanged. ignore_index=True gives the result a new sequential row index. Preserve an order ID or other stable identifier when tracking exclusions.
df = student_data()
print("Original surviving labels:", df.dropna().index.tolist())
result = df.dropna(ignore_index=True)
print(result)
assert result.index.tolist() == [0, 1, 2]Expected output
Original surviving labels: [0, 1, 5]
NAME ID MATH ENGLISH
0 Ravi 1.0 80.0 81.0
1 Raju 2.0 40.0 70.0
2 Alex 6.0 30.0 30.0Normalize blank text before dropping missing values 🔝
Empty strings and whitespace are not automatically missing. Zero is also a valid value. Define which placeholders mean missing, normalize them and then apply the cleaning rule. See text replacement.
raw = pd.DataFrame({"customer": ["Alex", "", " ", None], "amount": [10, 0, 3, 4]})
clean = raw.copy()
clean["customer"] = clean["customer"].astype("string").str.strip().replace("", pd.NA)
print("Before normalization:", len(raw.dropna(subset=["customer"])))
print("After normalization:")
print(clean.dropna(subset=["customer"]))
assert clean["customer"].isna().sum() == 3Expected output
Before normalization: 3
After normalization:
customer amount
0 Alex 10Practical workflow: require an order ID and known amount 🔝
This synthetic sales table includes optional customer names and missing amounts. Keep an order when its identifier and amount are known; preserve a genuine zero amount. Avoid a table-wide dropna() rule when optional fields should not exclude an order.
sales = pd.DataFrame({
"order_id": [101, 102, 103, 104, None],
"customer": ["Alex", None, "Ravi", "King", "Ron"],
"amount": [12.5, 20, None, 0, 9],
"note": [None, None, "review", None, None]
})
required = ["order_id", "amount"]
missing_required = sales[required].isna().any(axis=1)
accepted = sales.dropna(subset=required).copy()
rejected = sales.loc[missing_required].copy()
print("Accepted:")
print(accepted)
print("Rejected:")
print(rejected)
assert accepted["order_id"].tolist() == [101, 102, 104]
assert len(accepted) + len(rejected) == len(sales)Expected output
Accepted:
order_id customer amount note
0 101.0 Alex 12.5 NaN
1 102.0 NaN 20.0 NaN
3 104.0 King 0.0 NaN
Rejected:
order_id customer amount note
2 103.0 Ravi NaN review
4 NaN Ron 9.0 NaNAudit exclusions before calculating totals 🔝
The accepted subtotal describes only retained orders. Report how many rows and known amounts were excluded; do not imply the retained subtotal represents all revenue. info(), isnull() and sum() help inspect completeness.
print("Rows before:", len(sales))
print("Accepted rows:", len(accepted))
print("Rejected rows:", len(rejected))
print("Accepted known subtotal:", accepted["amount"].sum(min_count=1))
print("Known amounts in rejected rows:", rejected["amount"].sum(min_count=1))
print("Missing required values:")
print(sales[required].isna().sum())
assert accepted["amount"].sum() == 32.5
# Optional Colab export:
# accepted.to_csv("accepted_sales.csv", index=False)
# rejected.to_csv("sales_for_review.csv", index=False)Expected output
Rows before: 5
Accepted rows: 3
Rejected rows: 2
Accepted known subtotal: 32.5
Known amounts in rejected rows: 9.0
Missing required values:
order_id 1
amount 1
dtype: int64Decide whether to drop, fill or retain 🔝
Dropping removes records or fields. fillna() replaces missing values, which requires a justified interpretation. Retaining a missing optional field can be preferable to either deletion or invented values. Avoid filling unknown amounts with zero merely to make a total complete.
Common questions and pitfalls 🔝
Why are blank strings retained? Normalize them to missing first. Why did every column disappear? axis=1 with how="any" removes every column that has a missing value. Why does how with thresh fail? They are alternative rules and cannot be combined. Why did df become None? You assigned the return of an in-place operation. Can I drop by missing percentage? Convert the desired known-data fraction into a minimum count with ceil(). Does dropna validate values? No; invalid text, negative amounts and duplicate identifiers require separate checks.
Exercise: keep orders with both measurements 🔝
Create IDs 1, 2, 3 and 4 with quantity 2, missing, 0 and 3 and price 5, 8, 10 and missing. Require both quantity and price, preserve the zero quantity and calculate retained revenue. Expect IDs 1 and 3 and revenue 10.
Exercise solution 🔝
subset checks the two required measurements and does not treat zero as missing.
orders = pd.DataFrame({"id": [1, 2, 3, 4], "quantity": [2, None, 0, 3], "price": [5, 8, 10, None]})
complete = orders.dropna(subset=["quantity", "price"]).copy()
complete["revenue"] = complete["quantity"] * complete["price"]
print(complete)
print("Revenue:", complete["revenue"].sum())
assert complete["id"].tolist() == [1, 3]
assert complete["revenue"].sum() == 10Expected output
id quantity price revenue
0 1 2.0 5.0 10.0
2 3 0.0 10.0 0.0
Revenue: 10.0Practice in Google Colab 🔝
Open in Google Colab View on GitHub
Run the examples in order, change missing values and compare which rows each cleaning rule retains. Save your own copy to keep edits. All sample tables are included in the notebook.
Continue with data cleaning, fillna(), conditional text filtering and mask().
Reference: Pandas DataFrame.dropna documentation.
Subhendu Mohapatra
Author & Instructor at plus2net
I write and maintain practical tutorials on Python, PHP, SQL, JavaScript, HTML, jQuery, and web development at plus2net. The tutorials focus on clear explanations, working examples, and code that readers can test and adapt while learning.
Subscribe to our YouTube Channel here
Python Video Tutorials