Pandas str.contains(): Filter Text and Clean Data
Use str.contains() to find text within a Pandas column and select the matching rows. Choose literal matching for ordinary search text, regex for patterns, and an explicit missing-value policy before combining conditions.
Quick choice: series.str.contains("text", case=False, regex=False, na=False) gives a case-insensitive literal-search mask. Apply it with df.loc[mask].
Create the student dataset 🔝
Keep the source table unchanged and store each selection separately. This makes it easier to compare filters without accidentally searching an already filtered table.
import pandas as pd
import re
def student_data():
return pd.DataFrame({
"id": [1, 2, 3, 4, 5, 6, 7],
"name": ["$John", "Ma51", "Arnold1", "Krish0", "Roni", "Krish", "Max"],
"class": ["Four", "Three", "#Three", "Four", "7Four", "Four,", "%Three"],
"mark": [75, 85, 55, 60, 60, 60, 85]
})
df = student_data()
print(df)Expected output
id name class mark
0 1 $John Four 75
1 2 Ma51 Three 85
2 3 Arnold1 #Three 55
3 4 Krish0 Four 60
4 5 Roni 7Four 60
5 6 Krish Four, 60
6 7 Max %Three 85Find a substring with case=False 🔝
str.contains() returns a Boolean mask. Use it with df.loc[mask] to select matching rows. Set regex=False for a literal search and na=False to treat missing text as a non-match.
df = student_data()
mask = df["name"].str.contains("ar", case=False, regex=False, na=False)
print(mask.tolist())
print(df.loc[mask])
assert df.loc[mask, "id"].tolist() == [3]Expected output
[False, False, True, False, False, False, False]
id name class mark
2 3 Arnold1 #Three 55Choose the matching options 🔝
pat is the text or pattern; case=True is the default case-sensitive mode; regex=True is the default regular-expression mode. flags accepts Python regex flags when regex is enabled. The default missing-value result depends on the Series dtype, so specify na=False explicitly for a two-valued mask. An empty literal matches every non-missing string; reject empty input when that is not the intended search.
Match the beginning or end of a name 🔝
Regex anchors restrict where a match occurs. ^[AR] means starts with A or R, while h$ means ends with h. Square brackets in a regex describe a character set, rather than a DataFrame selection.
df = student_data()
for pattern in [r"^[AR]", r"h$", r"[hn]$"]:
selected = df.loc[df["name"].str.contains(pattern, na=False)]
print(pattern, selected["name"].tolist())Expected output
^[AR] ['Arnold1', 'Roni']
h$ ['Krish']
[hn]$ ['$John', 'Krish']Exclude matches with the ~ operator 🔝
Invert the mask to keep names without ar. With na=False, inversion also keeps missing names: they were non-matches. Add notna() when missing names should be excluded as well. A Boolean inversion is usually easier to read than a negative-lookahead regex.
df = student_data()
mask = df["name"].str.contains("ar", case=False, regex=False, na=False)
print(df.loc[~mask, "name"].tolist())
# Equivalent regex for these non-missing student names:
alternative = df["name"].str.contains(r"^(?!.*ar).*$", case=False, na=False)
assert alternative.equals(~mask)Expected output
['$John', 'Ma51', 'Krish0', 'Roni', 'Krish', 'Max']Find punctuation in class labels 🔝
Define the characters that count as unwanted punctuation. Here the set includes a comma, and the hyphen is escaped so it cannot introduce a character range. A digit such as the 7 in 7Four is not punctuation under this rule.
df = student_data()
special = r"[@#&$%+*/?,\-]"
mask = df["class"].str.contains(special, na=False)
print(df.loc[mask])
assert df.loc[mask, "id"].tolist() == [3, 6, 7]Expected output
id name class mark
2 3 Arnold1 #Three 55
5 6 Krish Four, 60
6 7 Max %Three 85Find digits or the literal digit 0 🔝
Use a raw string for regex patterns such as r"\d". It matches a digit anywhere in the text. A literal 0 search selects only Krish0.
df = student_data()
print("Any digit:")
print(df.loc[df["name"].str.contains(r"\d", na=False)])
print("Literal zero:")
print(df.loc[df["name"].str.contains("0", regex=False, na=False)])Expected output
Any digit:
id name class mark
1 2 Ma51 Three 85
2 3 Arnold1 #Three 55
3 4 Krish0 Four 60
Literal zero:
id name class mark
3 4 Krish0 Four 60Combine text masks across columns 🔝
Use | for OR and & for AND. Python or and and do not combine whole Series. Both masks must refer to the same table and index.
df = student_data()
class_bad = df["class"].str.contains(special, na=False)
name_zero = df["name"].str.contains("0", regex=False, na=False)
print(df.loc[class_bad | name_zero])
assert df.loc[class_bad | name_zero, "id"].tolist() == [3, 4, 6, 7]Expected output
id name class mark
2 3 Arnold1 #Three 55
3 4 Krish0 Four 60
5 6 Krish Four, 60
6 7 Max %Three 85Remove selected rows with drop() 🔝
You can select the inverse mask directly or pass matching row labels to drop(). Neither approach changes df unless you assign the returned table or deliberately use an in-place operation. Learn more about dropping rows.
df = student_data()
zero = df["name"].str.contains("0", regex=False, na=False)
without_zero = df.drop(index=df.index[zero])
class_bad = df["class"].str.contains(special, na=False)
without_class_punctuation = df.drop(index=df.index[class_bad])
name_bad = df["name"].str.contains(special, na=False)
without_either = df.drop(index=df.index[class_bad | name_bad])
print("No zero:", without_zero["id"].tolist())
print("No class punctuation:", without_class_punctuation["id"].tolist())
print("No punctuation in either column:", without_either["id"].tolist())
assert len(df) == 7Expected output
No zero: [1, 2, 3, 5, 6, 7]
No class punctuation: [1, 2, 4, 5]
No punctuation in either column: [2, 4, 5]Exclude URLs containing a question mark 🔝
For a literal question mark, use regex=False. In regex mode, ? has special meaning and must be escaped. For index labels, convert the Index to a Series before making a mask. This simple rule detects the character; it does not parse or normalize URLs.
pages = pd.DataFrame({"Page": ["/python/", "/python/list.php?q=list", "/python/pandas.php"]})
mask = pages["Page"].str.contains("?", regex=False, na=False)
print(pages.loc[~mask])
indexed = pages.set_index("Page")
index_mask = indexed.index.to_series().str.contains("?", regex=False, na=False)
print(indexed.loc[~index_mask].index.tolist())Expected output
Page
0 /python/
2 /python/pandas.php
['/python/', '/python/pandas.php']Combine text and numeric conditions 🔝
Parenthesize comparisons before combining them. The OR result includes id 5; the AND result is empty because no student meets all three requirements.
df = student_data()
str1 = df["id"] == 5
str2 = df["name"].str.contains("Al", case=False, regex=False, na=False)
str3 = df["mark"] < 50
print("OR:", df.loc[str1 | str2 | str3, "id"].tolist())
print("AND:", df.loc[str1 & str2 & str3, "id"].tolist())Expected output
OR: [5]
AND: []Search for any or all words 🔝
For regex OR, escape each literal search term before joining with |. To require every term, combine literal masks. This avoids interpreting user text as regex and makes the AND rule explicit. Both examples search substrings: color also matches colors.
queries = pd.DataFrame({"queries": ["tkinter color picker", "tkinter grid", "pandas color chart", "Tkinter colors", None]})
conditions = ["tkinter", "color"]
pattern = "|".join(re.escape(word) for word in conditions)
any_word = queries["queries"].str.contains(pattern, case=False, na=False)
all_words = pd.Series(True, index=queries.index)
for word in conditions:
all_words &= queries["queries"].str.contains(word, case=False, regex=False, na=False)
print("Any:", queries.loc[any_word, "queries"].tolist())
print("All:", queries.loc[all_words, "queries"].tolist())
# Regex lookaheads provide another all-terms approach:
lookahead = "^" + "".join("(?=.*" + re.escape(word) + ")" for word in conditions)
assert queries["queries"].str.contains(lookahead, case=False, na=False).equals(all_words)Expected output
Any: ['tkinter color picker', 'tkinter grid', 'pandas color chart', 'Tkinter colors']
All: ['tkinter color picker', 'Tkinter colors']Handle missing and mixed-type text deliberately 🔝
Nullable string conversion preserves missing entries instead of turning them into searchable text such as None. Converting numbers to strings is a policy choice: here a numeric product label becomes searchable. If numbers indicate invalid input, validate them instead. fillna() and astype() cover related cleaning decisions.
values = pd.Series(["Notebook", None, 120, "notepad"], dtype="object")
text = values.astype("string")
mask = text.str.contains("note", case=False, regex=False, na=False)
print("Matches:", text.loc[mask].tolist())
print("Missing count:", text.isna().sum())
print("Known non-matches:", text.loc[text.notna() & ~mask].tolist())
assert mask.tolist() == [True, False, False, True]Expected output
Matches: ['Notebook', 'notepad']
Missing count: 1
Known non-matches: ['120']Search for dots, plus signs and brackets literally 🔝
Regex characters can change the meaning of a product search. A dot in regex mode matches almost any character. Use literal mode for a single search term, or re.escape() when building a regex from literal terms.
labels = pd.Series(["A.B", "AxB", "C++ guide", "[draft]", None], dtype="string")
print("Regex dot:", labels.loc[labels.str.contains("A.B", na=False)].tolist())
print("Literal dot:", labels.loc[labels.str.contains("A.B", regex=False, na=False)].tolist())
for term in ["C++", "[draft]"]:
print(term, labels.loc[labels.str.contains(term, regex=False, na=False)].tolist())Expected output
Regex dot: ['A.B', 'AxB']
Literal dot: ['A.B']
C++ ['C++ guide']
[draft] ['[draft]']Match a word rather than part of a word 🔝
Use regex word boundaries to distinguish pen from pencil. Word boundaries depend on regex word-character rules; they are not a linguistic tokenizer. For exact labels, use eq() or isin() rather than substring matching.
products = pd.Series(["pen", "pencil", "Blue pen", "pen-holder", "open"], dtype="string")
print("Substring:", products.loc[products.str.contains("pen", regex=False, na=False)].tolist())
print("Word:", products.loc[products.str.contains(r"\bpen\b", case=False, na=False)].tolist())
print("Exact:", products.loc[products.eq("pen")].tolist())Expected output
Substring: ['pen', 'pencil', 'Blue pen', 'pen-holder', 'open']
Word: ['pen', 'Blue pen', 'pen-holder']
Exact: ['pen']Practical workflow: inspect a sales extract 🔝
This synthetic extract has product variants, missing product text, test orders and mixed numeric values. Keep the source rows so exclusions can be reviewed. The example selects valid notebook orders, rather than changing product descriptions automatically.
def sales_data():
return pd.DataFrame({
"order_id": [101, 102, 103, 104, 105, 106],
"product": [" Notebook A5 ", "NOTEBOOK A4", None, "Notebook test pack", "Pen blue", "notebook A6"],
"status": ["paid", "paid", "paid", "test", "paid", "paid"],
"amount": ["12.50", "20", "9", "0", "3.50", "bad"]
})
sales = sales_data()
product = sales["product"].astype("string").str.strip()
has_notebook = product.str.contains("notebook", case=False, regex=False, na=False)
is_test = sales["status"].eq("test")
amount = pd.to_numeric(sales["amount"], errors="coerce")
selected_mask = has_notebook & ~is_test & amount.notna()
selected = sales.loc[selected_mask].copy()
selected["amount"] = amount.loc[selected_mask]
print(selected)
print("Selected orders:", len(selected))
print("Known revenue:", selected["amount"].sum())
assert selected["order_id"].tolist() == [101, 102]
assert selected["amount"].sum() == 32.5Expected output
order_id product status amount
0 101 Notebook A5 paid 12.5
1 102 NOTEBOOK A4 paid 20.0
Selected orders: 2
Known revenue: 32.5Explain exclusions and export the selected rows 🔝
Report reasons before removing data. Reason counts may overlap and must not be added to obtain the excluded-row count. The CSV is generated from selected rows; index=False avoids an extra index column. For files and spreadsheets, follow read_csv(), read_excel() and to_excel().
print("Missing product:", product.isna().sum())
print("Test status:", is_test.sum())
print("Invalid or missing amount:", amount.isna().sum())
print("Total excluded:", (~selected_mask).sum())
csv_text = selected.to_csv(index=False)
print(csv_text.rstrip())
# To save a file in Colab, uncomment:
# selected.to_csv("selected_notebook_orders.csv", index=False)Expected output
Missing product: 1
Test status: 1
Invalid or missing amount: 1
Total excluded: 4
order_id,product,status,amount
101, Notebook A5 ,paid,12.5
102,NOTEBOOK A4,paid,20.0Common questions and mistakes 🔝
Why does filtering fail with missing text? Specify na=False. Why does C++ give unexpected results? It is interpreted as regex unless literal mode is used. Why is the result empty? Check case, whitespace, the source table and the AND conditions. Does contains clean the text? It returns a mask; use str.replace() for replacements. Does contains match the whole value? No; use eq() or str.fullmatch() as appropriate. How do I count matches? Sum a non-missing Boolean mask; see str.contains().sum(). Avoid complicated or untrusted regex patterns on large text inputs; prefer literal searches where possible.
Exercise: select paid pen orders 🔝
Create a table with products Blue pen, pencil, PEN refill and a missing value; statuses paid, paid, test and paid; amounts 5, 8, 2 and 4. Select paid rows containing pen as a whole word, case-insensitively. Predict the count and revenue before reading the solution: one order and revenue 5.
Exercise solution 🔝
The boundary excludes pencil, the status excludes PEN refill, and the missing product is a non-match.
practice = pd.DataFrame({
"product": ["Blue pen", "pencil", "PEN refill", None],
"status": ["paid", "paid", "test", "paid"],
"amount": [5, 8, 2, 4]
})
mask = practice["product"].str.contains(r"\bpen\b", case=False, na=False) & practice["status"].eq("paid")
answer = practice.loc[mask]
print(answer)
print("Orders:", len(answer), "Revenue:", answer["amount"].sum())
assert len(answer) == 1 and answer["amount"].sum() == 5Expected output
product status amount
0 Blue pen paid 5
Orders: 1 Revenue: 5Practice in Google Colab 🔝
Open in Google Colab View on GitHub
Run the examples in order, change the search terms and inspect how the selected rows change. Save your own copy to keep edits. All sample tables are included in the notebook.
Continue with Pandas data cleaning, the search-query exercise, and interactive DataFrame search with Tkinter.
Reference: Pandas Series.str.contains documentation.
Subhendu Mohapatra
Author & Instructor at plus2net
I write and maintain practical tutorials on Python, PHP, SQL, JavaScript, HTML, jQuery, and web development at plus2net. The tutorials focus on clear explanations, working examples, and code that readers can test and adapt while learning.
Subscribe to our YouTube Channel here
Python Video Tutorials