Pandas drop_duplicates(): Remove Repeated Rows Safely
Use drop_duplicates() to remove repeated Pandas rows. Choose the fields that define a duplicate, decide which occurrence to retain, and review removed records before calculating totals.
Create the student DataFrame 🔝
Rows 5 and 6 repeat rows 3 and 1 respectively across all columns. The two John records have different IDs and marks, so they are not identical whole rows.
import pandas as pd
def student_data():
return pd.DataFrame({
"id": [1, 2, 3, 4, 5, 4, 2],
"name": ["John", "Max", "Arnold", "Krish", "John", "Krish", "Max"],
"class1": ["Four", "Three", "Three", "Four", "Four", "Four", "Three"],
"mark": [75, 85, 55, 60, 60, 60, 85],
"gender": ["female", "male", "male", "female", "female", "female", "male"]
})
df = student_data()
print(df)Expected output
id name class1 mark gender
0 1 John Four 75 female
1 2 Max Three 85 male
2 3 Arnold Three 55 male
3 4 Krish Four 60 female
4 5 John Four 60 female
5 4 Krish Four 60 female
6 2 Max Three 85 maleChoose the duplicate rule 🔝
subset=None compares all columns; a list of labels compares only those fields. keep="first" retains the first occurrence in current row order; keep="last" retains the last; keep=False removes every member of a repeated group. False is a Boolean, not the string "False". inplace=True modifies the source and returns None. ignore_index=True renumbers the result. Row index labels are ignored when comparing duplicates.
Keep the first occurrence 🔝
Reset the source for each demonstration so the keep options are compared on the same seven rows. The default retains five rows.
df = student_data()
result = df.drop_duplicates(keep="first")
print(result)
assert result.index.tolist() == [0, 1, 2, 3, 4]Expected output
id name class1 mark gender
0 1 John Four 75 female
1 2 Max Three 85 male
2 3 Arnold Three 55 male
3 4 Krish Four 60 female
4 5 John Four 60 femaleKeep the last occurrence 🔝
The last occurrence means the last row in the current table order. It does not automatically mean the newest date or the most reliable record.
df = student_data()
result = df.drop_duplicates(keep="last")
print(result)
assert result.index.tolist() == [0, 2, 4, 5, 6]Expected output
id name class1 mark gender
0 1 John Four 75 female
2 3 Arnold Three 55 male
4 5 John Four 60 female
5 4 Krish Four 60 female
6 2 Max Three 85 maleRemove every member of a repeated group 🔝
keep=False retains only rows that occur once under the chosen comparison. It is useful when ambiguous groups should be separated for review rather than choosing one arbitrarily.
df = student_data()
result = df.drop_duplicates(keep=False)
print(result)
assert result.index.tolist() == [0, 2, 4]Expected output
id name class1 mark gender
0 1 John Four 75 female
2 3 Arnold Three 55 male
4 5 John Four 60 femaleUse selected columns as duplicate keys 🔝
Using class1, mark and gender as keys merges records even if ID and name differ. Here John with mark 60 and Krish with mark 60 share those selected values. Choose keys that identify the intended entity; matching marks alone does not prove two students are the same person.
df = student_data()
keys = ["class1", "mark", "gender"]
result = df.drop_duplicates(subset=keys)
print(result)
assert result.index.tolist() == [0, 1, 2, 3]Expected output
id name class1 mark gender
0 1 John Four 75 female
1 2 Max Three 85 male
2 3 Arnold Three 55 male
3 4 Krish Four 60 femaleUnderstand inplace and the returned result 🔝
By default, the source table is unchanged. With inplace=True, the method updates the DataFrame and returns None. Do not assign that None back to df.
df = student_data()
result = df.drop_duplicates()
print("Source rows:", len(df), "Result rows:", len(result))
returned = df.drop_duplicates(inplace=True)
print("In-place return:", returned)
print(df)
assert returned is None and len(df) == 5Expected output
Source rows: 7 Result rows: 5
In-place return: None
id name class1 mark gender
0 1 John Four 75 female
1 2 Max Three 85 male
2 3 Arnold Three 55 male
3 4 Krish Four 60 female
4 5 John Four 60 femaleReset labels with ignore_index 🔝
ignore_index=True gives the surviving rows sequential labels. Keep a stable ID column when you need to reconcile cleaned data with its source.
df = student_data()
result = df.drop_duplicates(keep="last", ignore_index=True)
print(result)
assert result.index.tolist() == list(range(5))Expected output
id name class1 mark gender
0 1 John Four 75 female
1 3 Arnold Three 55 male
2 5 John Four 60 female
3 4 Krish Four 60 female
4 2 Max Three 85 maleInspect repeated groups before removing rows 🔝
duplicated() returns a Boolean mask. keep=False flags all members of repeated groups; the default flags only extra occurrences after the first. These counts answer different questions.
df = student_data()
all_members = df.duplicated(keep=False)
extras = df.duplicated()
print(df.loc[all_members])
print("Rows in repeated groups:", all_members.sum())
print("Extra occurrences:", extras.sum())
assert all_members.sum() == 4 and extras.sum() == 2Expected output
id name class1 mark gender
1 2 Max Three 85 male
3 4 Krish Four 60 female
5 4 Krish Four 60 female
6 2 Max Three 85 male
Rows in repeated groups: 4
Extra occurrences: 2Understand why the index is not a duplicate key 🔝
Different index labels do not prevent identical data rows from being removed. Duplicate index labels alone do not imply duplicated data. Move a meaningful index value into a column if it must participate in comparison; see reset_index().
table = pd.DataFrame({"value": [10, 10, 20]}, index=["a", "b", "b"])
print("Compare values:")
print(table.drop_duplicates())
with_label = table.rename_axis("source_label").reset_index()
print("Compare label and value:")
print(with_label.drop_duplicates())
assert len(table.drop_duplicates()) == 2
assert len(with_label.drop_duplicates()) == 3Expected output
Compare values:
value
a 10
b 20
Compare label and value:
source_label value
0 a 10
1 b 10
2 b 20Normalize text keys before comparison 🔝
Whitespace and case differences are significant until you define normalization rules. Keep the original values and create a comparison key. Normalization can merge genuinely distinct identifiers; only apply it when the domain treats those variations as equivalent. See str.replace() and case conversion.
contacts = pd.DataFrame({"email": [" Alex@example.com ", "alex@example.com", "ravi@example.com"]})
contacts["email_key"] = contacts["email"].astype("string").str.strip().str.casefold()
print(contacts.drop_duplicates(subset=["email_key"]))
assert len(contacts.drop_duplicates(subset=["email_key"])) == 2Expected output
email email_key
0 Alex@example.com alex@example.com
2 ravi@example.com ravi@example.comSeparate missing keys before deduplication 🔝
Repeated missing values can participate in duplicate groups. Two unknown order IDs do not prove that two records are the same order. Separate incomplete keys for review and deduplicate only known keys when this is the policy. See dropna().
orders = pd.DataFrame({"order_id": pd.Series([101, 101, None, None], dtype="Int64"), "amount": [10, 10, 5, 8]})
print("Direct key deduplication:")
print(orders.drop_duplicates(subset=["order_id"]))
review = orders.loc[orders["order_id"].isna()].copy()
known = orders.dropna(subset=["order_id"]).drop_duplicates(subset=["order_id"])
print("Known-key rows:", len(known), "Unknown-key review rows:", len(review))
assert len(review) == 2Expected output
Direct key deduplication:
order_id amount
0 101 10
2 <NA> 5
Known-key rows: 1 Unknown-key review rows: 2Sort before keeping the latest record 🔝
Parse timestamps and sort before applying keep="last". A stable sort preserves input order for ties; here a sequence column explicitly breaks ties. Missing timestamps need a separate policy and must not silently win because of default missing-value sorting.
updates = pd.DataFrame({
"order_id": [101, 102, 101, 102],
"updated": ["2026-01-02", "2026-01-03", "2026-01-01", "2026-01-03"],
"sequence": [1, 2, 3, 4],
"amount": [12, 20, 10, 21]
})
updates["updated"] = pd.to_datetime(updates["updated"], format="%Y-%m-%d")
assert updates[["order_id", "updated", "sequence"]].notna().all().all()
ordered = updates.sort_values(["updated", "sequence"], kind="stable")
latest = ordered.drop_duplicates(subset=["order_id"], keep="last").sort_values("order_id")
print(latest)
assert latest["amount"].tolist() == [12, 21]Expected output
order_id updated sequence amount
0 101 2026-01-02 1 12
3 102 2026-01-03 4 21Practical sales workflow: choose an order-line key 🔝
An order can have multiple legitimate product lines. Deduplicating only order_id would discard valid sales. This synthetic extract uses order_id plus line_id as the key and removes only an exact repeat. Repeated keys with conflicting amounts need review rather than arbitrary deletion.
sales = pd.DataFrame({
"order_id": [101, 101, 101, 102],
"line_id": [1, 2, 1, 1],
"product": ["Notebook", "Pen", "Notebook", "Pencil"],
"amount": [12.5, 3, 12.5, 4]
})
keys = ["order_id", "line_id"]
conflicts = sales.groupby(keys)["amount"].nunique(dropna=False).gt(1)
assert not conflicts.any()
clean = sales.drop_duplicates(subset=keys).copy()
removed = sales.loc[sales.duplicated(subset=keys)].copy()
print("Retained:")
print(clean)
print("Removed repeat:")
print(removed)
assert len(clean) == 3 and len(removed) == 1Expected output
Retained:
order_id line_id product amount
0 101 1 Notebook 12.5
1 101 2 Pen 3.0
3 102 1 Pencil 4.0
Removed repeat:
order_id line_id product amount
2 101 1 Notebook 12.5Audit row counts and totals after removal 🔝
Reconcile the retained and removed rows. A smaller total after deduplication is expected only if repeated observations inflated the source. Confirm that your duplicate key reflects the data model; see sum() and export to Excel.
print("Rows before:", len(sales))
print("Rows retained:", len(clean))
print("Rows removed:", len(removed))
print("Before subtotal:", sales["amount"].sum())
print("Retained subtotal:", clean["amount"].sum())
assert len(clean) + len(removed) == len(sales)
assert clean["amount"].sum() == 19.5
assert not clean.duplicated(subset=keys).any()
# Optional Colab exports:
# clean.to_csv("unique_sales_lines.csv", index=False)
# removed.to_csv("removed_sales_repeats.csv", index=False)Expected output
Rows before: 4
Rows retained: 3
Rows removed: 1
Before subtotal: 32.0
Retained subtotal: 19.5Remove duplicate values from a Series 🔝
A Series compares its values rather than whole table rows. See Series.drop_duplicates() and Series.duplicated() for additional examples.
names = student_data()["name"]
print(names.drop_duplicates().tolist())
print("Values appearing once:", names.drop_duplicates(keep=False).tolist())Expected output
['John', 'Max', 'Arnold', 'Krish']
Values appearing once: ['Arnold']Common questions and mistakes 🔝
Why are similar rows still present? Some compared fields differ; check subset. Does last mean newest? No; sort and define timestamp tie rules first. Why did all copies disappear? keep=False removes every member of repeated groups. Does it compare the index? No. Does it remove duplicate columns? No; it removes rows. Why does a list-valued key fail? Comparison fields must be hashable; define a meaningful scalar or tuple key. Is dropping always appropriate? No; conflicting records may need correction, aggregation or review.
Exercise: keep the latest customer update 🔝
Create customer IDs 1, 2 and 1 with dates 2026-01-01, 2026-01-02 and 2026-01-03 and names Alex, Ravi and Alexander. Parse dates, sort and retain the latest row per customer. Expect Alexander for customer 1 and Ravi for customer 2.
Exercise solution 🔝
Validate known keys and dates, then sort before keeping the last row for each customer.
customers = pd.DataFrame({"customer_id": [1, 2, 1], "updated": ["2026-01-01", "2026-01-02", "2026-01-03"], "name": ["Alex", "Ravi", "Alexander"]})
customers["updated"] = pd.to_datetime(customers["updated"], format="%Y-%m-%d")
assert customers[["customer_id", "updated"]].notna().all().all()
answer = customers.sort_values("updated", kind="stable").drop_duplicates(subset=["customer_id"], keep="last").sort_values("customer_id")
print(answer)
assert answer["name"].tolist() == ["Alexander", "Ravi"]Expected output
customer_id updated name
2 1 2026-01-03 Alexander
1 2 2026-01-02 RaviPractice in Google Colab 🔝
Open in Google Colab View on GitHub
Run the examples in order, change duplicate keys and keep options, then compare retained and removed rows. Save your own copy to keep edits. All sample tables are included in the notebook.
Continue with data cleaning, duplicated(), dropna() and info().
Reference: Pandas DataFrame.drop_duplicates documentation.
Subhendu Mohapatra
Author & Instructor at plus2net
I write and maintain practical tutorials on Python, PHP, SQL, JavaScript, HTML, jQuery, and web development at plus2net. The tutorials focus on clear explanations, working examples, and code that readers can test and adapt while learning.
Subscribe to our YouTube Channel here
Python Video Tutorials