Pandas Series.drop_duplicates(): Retain Distinct Values
Use Series.drop_duplicates() to retain distinct values in a Series. Choose the first or last occurrence, or keep only values that occur once. Review missing values and preserve source records before producing a separate list.
Keep the first occurrence of each value 🔝
The default keep="first" returns a Series with one representative per distinct value. Original labels and retained order remain intact. The source is unchanged unless inplace=True is used.
import pandas as pd
s = pd.Series(["One", "Two", "Three", "Two", "Four"], name="label")
result = s.drop_duplicates()
print(result)
assert result.tolist() == ["One", "Two", "Three", "Four"]
assert result.index.tolist() == [0, 1, 2, 4]
assert len(s) == 5Expected output
0 One
1 Two
2 Three
4 Four
Name: label, dtype: strKeep the last occurrence 🔝
keep="last" retains the last occurrence of each value in the current order. It does not sort or identify the latest timestamp automatically.
s = pd.Series(["One", "Two", "Three", "Two", "Four"], name="label")
print(s.drop_duplicates(keep="last"))
assert s.drop_duplicates(keep="last").index.tolist() == [0, 2, 3, 4]Expected output
0 One
2 Three
3 Two
4 Four
Name: label, dtype: strRemove every member of repeated groups 🔝
keep=False leaves only values occurring once. This differs from keeping one representative. Use the Boolean False, not the string "False".
s = pd.Series(["One", "Two", "Three", "Two", "Four"], name="label")
singletons = s.drop_duplicates(keep=False)
print(singletons)
assert singletons.tolist() == ["One", "Three", "Four"]Expected output
0 One
2 Three
4 Four
Name: label, dtype: strUnderstand inplace and the return value 🔝
With inplace=True the Series is modified and the method returns None. Do not assign that return value back to the variable. For a reviewable workflow, returning a separate Series is often easier to compare with the source.
s = pd.Series(["One", "Two", "Three", "Two", "Four"], name="label")
returned = s.drop_duplicates(keep="last", inplace=True)
print(s)
print("Return value:", returned)
assert returned is None
assert s.index.tolist() == [0, 2, 3, 4]Expected output
0 One
2 Three
3 Two
4 Four
Name: label, dtype: str
Return value: NoneReset labels only when needed 🔝
ignore_index=True gives the result a fresh RangeIndex. Preserve original labels while reviewing source records; reset them when the output requires sequential labels.
s = pd.Series(["A", "B", "A", "C"], index=[10, 20, 30, 40], name="code")
print("Original labels:")
print(s.drop_duplicates())
print("Fresh labels:")
print(s.drop_duplicates(ignore_index=True))
assert s.drop_duplicates(ignore_index=True).index.tolist() == [0, 1, 2]Expected output
Original labels:
10 A
20 B
40 C
Name: code, dtype: str
Fresh labels:
0 A
1 B
2 C
Name: code, dtype: strInspect removed entries using duplicated() 🔝
The retained rows match the inverse duplicated() mask for the same keep policy. Capture removed entries before producing the output. See Series.duplicated().
s = pd.Series(["A", "B", "A", "C", "B"], name="code")
removed_mask = s.duplicated(keep="first")
print("Removed occurrences:")
print(s.loc[removed_mask])
result = s.drop_duplicates(keep="first")
assert result.equals(s.loc[~removed_mask])
print("Input:", len(s), "Retained:", len(result), "Removed:", int(removed_mask.sum()))Expected output
Removed occurrences:
2 A
4 B
Name: code, dtype: str
Input: 5 Retained: 3 Removed: 2Handle missing and blank values deliberately 🔝
Repeated missing entries in a nullable string Series are deduplicated. Blank strings are distinct from missing values. If missing identifiers require review, separate them before applying drop_duplicates() to known values.
s = pd.Series(["A", None, "A", None, "", ""], dtype="string")
print("Default result:")
print(s.drop_duplicates())
missing = s.loc[s.isna()]
known = s.loc[s.notna()]
print("Missing entries for review:", len(missing))
print("Known representatives:")
print(known.drop_duplicates())
assert len(s.drop_duplicates()) == 3
assert len(missing) == 2Expected output
Default result:
0 A
1 <NA>
4
dtype: string
Missing entries for review: 2
Known representatives:
0 A
4
dtype: stringCompare unique(), nunique() and drop_duplicates() 🔝
drop_duplicates() returns a Series retaining labels by default. unique() returns distinct values without source-row labels. nunique() returns a count and excludes missing values by default. Deduplication does not sort the retained values.
s = pd.Series(["B", "A", "B", "C"], dtype="string", name="code")
print(s.drop_duplicates())
print(s.unique())
print("Distinct values:", s.nunique())
print("Explicitly sorted:")
print(s.drop_duplicates().sort_values())
assert s.drop_duplicates().tolist() == ["B", "A", "C"]Expected output
0 B
1 A
3 C
Name: code, dtype: string
<StringArray>
['B', 'A', 'C']
Length: 3, dtype: string
Distinct values: 3
Explicitly sorted:
1 A
0 B
3 C
Name: code, dtype: stringAvoid assigning a shorter Series into a full column 🔝
A deduplicated Series is shorter and assignment back to a DataFrame column aligns by index, leaving other rows missing. If the goal is to retain complete records based on a column, use DataFrame.drop_duplicates(subset=...).
table = pd.DataFrame({"code": ["A", "A", "B"], "amount": [5, 7, 9]})
distinct = table["code"].drop_duplicates()
demonstration = table.copy()
demonstration["short_result"] = distinct
print(demonstration)
print("Complete retained rows:")
print(table.drop_duplicates(subset=["code"]))
assert pd.isna(demonstration.loc[1, "short_result"])Expected output
code amount short_result
0 A 5 A
1 A 7 NaN
2 B 9 B
Complete retained rows:
code amount
0 A 5
2 B 9Practical workflow: create a distinct category list 🔝
This synthetic category specification ignores surrounding spaces and case. Normalise into a separate Series, reject blank or missing inputs and create a distinct list for a selection menu. This removes duplicate labels, not transaction records. See case conversion.
raw = pd.Series([" Books ", "books", "PENS", " pens ", "Mugs", "", None], dtype="string", name="category")
normalised = raw.str.strip().str.casefold()
valid = normalised.notna() & normalised.ne("")
valid = valid.fillna(False)
known = normalised.loc[valid]
removed_mask = known.duplicated(keep="first")
categories = known.drop_duplicates()
print("Distinct categories:")
print(categories)
print("Invalid source entries:")
print(raw.loc[~valid])
assert categories.tolist() == ["books", "pens", "mugs"]Expected output
Distinct categories:
0 books
2 pens
4 mugs
Name: category, dtype: string
Invalid source entries:
5
6 <NA>
Name: category, dtype: stringKeep a mapping and validate the category output 🔝
A mapping preserves source spellings and shows which repeated labels were collapsed. The input count must equal retained representatives plus removed repeats plus invalid entries. Keep original records elsewhere if they represent purchases or customers.
mapping = pd.DataFrame({"original": raw.loc[valid], "normalised": known, "removed_repeat": removed_mask})
print(mapping)
assert len(raw) == len(categories) + int(removed_mask.sum()) + int((~valid).sum())
assert not categories.duplicated().any()
assert categories.notna().all() and categories.ne("").all()
print("Retained:", len(categories), "Repeated labels:", int(removed_mask.sum()), "Invalid:", int((~valid).sum()))
print(categories.to_frame().to_csv(index=False).rstrip())
# Optional Colab exports:
# categories.to_frame().to_csv("distinct_categories.csv", index=False)
# mapping.to_csv("category_normalisation_audit.csv", index=True)Expected output
original normalised removed_repeat
0 Books books False
1 books books True
2 PENS pens False
3 pens pens True
4 Mugs mugs False
Retained: 3 Repeated labels: 2 Invalid: 2
category
books
pens
mugsChoose identifier types before comparing values 🔝
Text identifiers and numbers are different representations. Converting all values to text is a policy decision, and numeric import may already have lost leading zeros. Keep the schema explicit; see type conversion and identifier formatting.
codes = pd.Series(["0012", "12", "0012", None], dtype="string")
print(codes.drop_duplicates())
assert codes.drop_duplicates().iloc[:2].tolist() == ["0012", "12"]Expected output
0 0012
1 12
3 <NA>
dtype: stringCommon questions 🔝
Does Series.drop_duplicates() have subset? No; use the DataFrame method for multiple fields. Why are labels not sequential? Source labels are preserved by default; use ignore_index=True deliberately. Does keep=False keep one copy? No, it removes every member of repeated groups. Does inplace=True return the cleaned Series? No, it returns None. Does keeping last mean keeping newest? Only if the current order matches that business rule. Should I remove repeated transaction categories? Usually not from transaction data; create a separate distinct category list instead.
Exercise: compare retention policies 🔝
Create values red, blue, red, green and blue with labels 10, 20, 30, 40 and 50. Return first representatives, last representatives and singleton values. Predict the retained labels before running the solution.
Exercise solution 🔝
Both representative policies retain three values, but preserve different source labels. keep=False leaves only green.
practice = pd.Series(["red", "blue", "red", "green", "blue"], index=[10, 20, 30, 40, 50], dtype="string")
first = practice.drop_duplicates(keep="first")
last = practice.drop_duplicates(keep="last")
singletons = practice.drop_duplicates(keep=False)
print("First representatives:")
print(first)
print("Last representatives:")
print(last)
print("Singletons:")
print(singletons)
assert first.index.tolist() == [10, 20, 40]
assert last.index.tolist() == [30, 40, 50]
assert singletons.tolist() == ["green"]Expected output
First representatives:
10 red
20 blue
40 green
dtype: string
Last representatives:
30 red
40 green
50 blue
dtype: string
Singletons:
40 green
dtype: stringPractice in Google Colab 🔝
Open in Google Colab View on GitHub
Run the examples in order, change keep policies, missing entries and category spellings, then compare the retained labels. Save your own copy to keep edits. All sample tables are included in the notebook.
Continue with the Data Cleaning hub, Series.duplicated(), DataFrame.duplicated() and DataFrame.drop_duplicates().
Reference: Pandas Series.drop_duplicates documentation.
Subhendu Mohapatra
Author & Instructor at plus2net
I write and maintain practical tutorials on Python, PHP, SQL, JavaScript, HTML, jQuery, and web development at plus2net. The tutorials focus on clear explanations, working examples, and code that readers can test and adapt while learning.
Subscribe to our YouTube Channel here
Python Video Tutorials