Pandas Series.duplicated(): Detect and Review Repeated Values
Use Series.duplicated(keep="first") to identify repeated values. Choose whether to mark later repetitions, earlier repetitions or every member of a repeated group. Keep missing keys and potential normalisation collisions visible for review.
Mark repetitions after the first value 🔝
Series.duplicated() returns a Boolean Series with the same index. The default keep="first" marks later repetitions True and the first occurrence False. It does not remove values.
import pandas as pd
s = pd.Series(["One", "Two", "Three", "Two", "Four"], name="label")
print(s.duplicated())
assert s.duplicated().tolist() == [False, False, False, True, False]Expected output
0 False
1 False
2 False
3 True
4 False
Name: label, dtype: boolKeep the last occurrence unmarked 🔝
keep="last" marks earlier occurrences True. First and last refer to the current order, not a timestamp or the smallest index label.
s = pd.Series(["One", "Two", "Three", "Two", "Four"], name="label")
print(s.duplicated(keep="last"))
assert s.duplicated(keep="last").tolist() == [False, True, False, False, False]Expected output
0 False
1 True
2 False
3 False
4 False
Name: label, dtype: boolMark every member of repeated groups 🔝
Use the Boolean keep=False to mark both occurrences of Two. The string "False" is not a valid keep policy.
s = pd.Series(["One", "Two", "Three", "Two", "Four"], name="label")
print(s.duplicated(keep=False))
assert s.duplicated(keep=False).tolist() == [False, True, False, True, False]Expected output
0 False
1 True
2 False
3 True
4 False
Name: label, dtype: boolDisplay repeated values 🔝
Apply the returned mask with loc. The default mask selects repetitions after the first; keep=False selects every member of repeated groups.
s = pd.Series(["One", "Two", "Three", "Two", "Four"], name="label")
print("After first:")
print(s.loc[s.duplicated()])
print("All repeated members:")
print(s.loc[s.duplicated(keep=False)])
assert s.loc[s.duplicated()].index.tolist() == [3]Expected output
After first:
3 Two
Name: label, dtype: str
All repeated members:
1 Two
3 Two
Name: label, dtype: strDistinguish representatives from singleton values 🔝
Negating the default mask retains one occurrence of every distinct value, including Two. Negating keep=False retains only values that occur once. Use Series.drop_duplicates() when directly returning representatives.
s = pd.Series(["One", "Two", "Three", "Two", "Four"], name="label")
representatives = s.loc[~s.duplicated()]
singletons = s.loc[~s.duplicated(keep=False)]
print("Representatives:")
print(representatives)
print("Singleton values:")
print(singletons)
assert len(representatives) == 4 and len(singletons) == 3Expected output
Representatives:
0 One
1 Two
2 Three
4 Four
Name: label, dtype: str
Singleton values:
0 One
2 Three
4 Four
Name: label, dtype: strCompare unique(), nunique() and value_counts() 🔝
unique() returns distinct values without the original row index. nunique() counts distinct non-missing values by default. value_counts() counts frequencies. duplicated() answers which positions repeat. See unique values.
s = pd.Series(["One", "Two", "Three", "Two", "Four"], name="label")
print(s.unique())
print("Distinct count:", s.nunique())
print(s.value_counts())
print("Repeated groups:", int(s.value_counts().gt(1).sum()))
assert s.nunique() == 4Expected output
<StringArray>
['One', 'Two', 'Three', 'Four']
Length: 4, dtype: str
Distinct count: 4
label
Two 2
One 1
Three 1
Four 1
Name: count, dtype: int64
Repeated groups: 1Report repetitions, repeated members and repeated groups 🔝
These counts differ: repetitions after the first, all rows in repeated groups and the number of repeated distinct values. Use clear labels instead of calling all three duplicate count.
s = pd.Series(["A", "A", "A", "B", "B", "C"], dtype="string")
print("Repetitions after first:", int(s.duplicated().sum()))
print("Members of repeated groups:", int(s.duplicated(keep=False).sum()))
print("Repeated distinct values:", int(s.value_counts().gt(1).sum()))
assert s.duplicated().sum() == 3
assert s.duplicated(keep=False).sum() == 5Expected output
Repetitions after first: 3
Members of repeated groups: 5
Repeated distinct values: 2Review missing values separately 🔝
Repeated missing entries in a nullable string Series are flagged as duplicates. Unknown IDs do not prove that two records identify the same entity. Keep missing keys for review; see missing-value handling.
s = pd.Series(["A", None, "A", None, "B"], dtype="string")
print(s.duplicated(keep=False))
print("Missing positions:")
print(s.loc[s.isna()])
assert s.duplicated(keep=False).tolist() == [True, True, True, True, False]
assert s.isna().sum() == 2Expected output
0 True
1 True
2 True
3 True
4 False
dtype: bool
Missing positions:
1 <NA>
3 <NA>
dtype: stringCheck values and index labels independently 🔝
Series.duplicated() compares values and ignores the index. Use index.duplicated() for repeated labels. Duplicate labels can complicate row retrieval and alignment; keep a stable source-row identifier when reviewing data.
s = pd.Series(["A", "B", "A"], index=[10, 10, 20])
print("Repeated values:", s.duplicated().tolist())
print("Repeated labels:", s.index.duplicated().tolist())
assert s.duplicated().tolist() == [False, False, True]
assert s.index.duplicated().tolist() == [False, True, False]Expected output
Repeated values: [False, False, True]
Repeated labels: [False, True, False]Practical workflow: inspect normalised customer codes 🔝
This synthetic schema treats whitespace and letter case as insignificant and allows two letters followed by two digits. Preserve the source code, create a comparison key and separate invalid or missing keys. Use case conversion according to the actual key specification.
raw = pd.DataFrame({"source_row": [1, 2, 3, 4, 5, 6], "code": [" AB12 ", "ab12", "CD34", "bad!", None, "CD34"]})
key = raw["code"].astype("string").str.strip().str.upper()
valid = key.str.fullmatch(r"[A-Z]{2}[0-9]{2}", na=False)
accepted = raw.loc[valid].copy()
accepted["comparison_key"] = key.loc[valid]
mask = accepted["comparison_key"].duplicated(keep=False)
accepted["repeated_key"] = mask
print(accepted)
print("Invalid or missing rows:")
print(raw.loc[~valid])
assert mask.sum() == 4Expected output
source_row code comparison_key repeated_key
0 1 AB12 AB12 True
1 2 ab12 AB12 True
2 3 CD34 CD34 True
5 6 CD34 CD34 True
Invalid or missing rows:
source_row code
3 4 bad!
4 5 NaNInspect the records behind each repeated value 🔝
A repeated customer code may represent a legitimate repeat transaction, a duplicate import or a key collision. This example flags candidates for review rather than deleting customer records. Use DataFrame.duplicated() to compare complete rows or several business fields.
review = accepted.loc[accepted["repeated_key"]].copy()
print(review[["source_row", "code", "comparison_key"]].sort_values("comparison_key"))
print("Repeated key frequencies:")
print(accepted["comparison_key"].value_counts())
assert review["source_row"].tolist() == [1, 2, 3, 6]Expected output
source_row code comparison_key
0 1 AB12 AB12
1 2 ab12 AB12
2 3 CD34 CD34
5 6 CD34 CD34
Repeated key frequencies:
comparison_key
AB12 2
CD34 2
Name: count, dtype: Int64Export an audit table with source references 🔝
Keep the original spelling and source-row identifier alongside the helper key. Preserve missing and invalid records separately. The example exports CSV text without writing files during practice.
assert len(raw) == len(accepted) + int((~valid).sum())
print("Valid rows:", len(accepted), "Invalid or missing:", int((~valid).sum()))
print(review.to_csv(index=False).rstrip())
# Optional Colab exports:
# review.to_csv("repeated_customer_codes.csv", index=False)
# raw.loc[~valid].to_csv("invalid_customer_codes.csv", index=False)Expected output
Valid rows: 4 Invalid or missing: 2
source_row,code,comparison_key,repeated_key
1, AB12 ,AB12,True
2,ab12,AB12,True
3,CD34,CD34,True
6,CD34,CD34,TrueCommon questions 🔝
Does Series.duplicated() have subset? No; a Series contains one value per row. Use DataFrame.duplicated(subset=...) for several fields. Does it remove data? No, it returns a mask. Are first and last sorted by index? No; they refer to current order. Why are missing IDs repeated? Duplicate detection can group repeated missing entries; review them separately. Can I ignore case? Create a normalised comparison Series only when the data specification permits it. Is every repeated value an error? No; define what one row represents before removing anything.
Exercise: count and locate repeated values 🔝
Create values A, B, A, C, B and D. Identify repetitions after the first, all members of repeated groups and singleton values. Expect counts two, four and two respectively.
Exercise solution 🔝
Use separate masks for each meaning and keep the original Series unchanged.
practice = pd.Series(["A", "B", "A", "C", "B", "D"], dtype="string")
later = practice.duplicated()
all_members = practice.duplicated(keep=False)
singletons = practice.loc[~all_members]
print("Later repetitions:")
print(practice.loc[later])
print("All repeated members:")
print(practice.loc[all_members])
print("Singleton values:")
print(singletons)
assert later.sum() == 2 and all_members.sum() == 4
assert singletons.tolist() == ["C", "D"]Expected output
Later repetitions:
2 A
4 B
dtype: string
All repeated members:
0 A
1 B
2 A
4 B
dtype: string
Singleton values:
3 C
5 D
dtype: stringPractice in Google Colab 🔝
Open in Google Colab View on GitHub
Run the examples in order, change keep policies and sample values, then compare repeated members and singleton values. Save your own copy to keep edits. All sample tables are included in the notebook.
Continue with the Data Cleaning hub, Series.drop_duplicates(), DataFrame.duplicated() and DataFrame.drop_duplicates().
Reference: Pandas Series.duplicated documentation.
Subhendu Mohapatra
Author & Instructor at plus2net
I write and maintain practical tutorials on Python, PHP, SQL, JavaScript, HTML, jQuery, and web development at plus2net. The tutorials focus on clear explanations, working examples, and code that readers can test and adapt while learning.
Subscribe to our YouTube Channel here
Python Video Tutorials