Pandas apply(): Rows, Columns and Custom Cleaning Rules
Use apply() for custom row or column calculations. Understand the axis and returned shape, handle missing values explicitly and compare simple rules with built-in Pandas operations.
Choose DataFrame.apply() or Series.apply() 🔝
DataFrame.apply() passes a column to your function with axis=0 and a row with axis=1. With raw=False, each is a Series. Series.apply() with an ordinary callable processes individual values by default. DataFrame.apply() is not a general cell-by-cell mapper. Prefer built-in arithmetic, string methods and reductions when they express the rule clearly.
Sum two marks across each row 🔝
Select only the score columns. axis=1 passes each row to np.sum. The total is named MATH_NEW here to demonstrate the student example; a name such as total is clearer in a report.
import pandas as pd
import numpy as np
def student_data():
return pd.DataFrame({"NAME": ["Ravi", "Raju", "Alex", "Ron", "King", "Jack"], "ID": [1, 2, 3, 4, 5, 6], "MATH": [80, 40, 70, 70, 82, 30], "ENGLISH": [81, 70, 40, 50, 60, 30]})
df = student_data()
df["MATH_NEW"] = df[["MATH", "ENGLISH"]].apply(np.sum, axis=1)
print(df)
assert df["MATH_NEW"].tolist() == [161, 110, 110, 120, 142, 60]Expected output
NAME ID MATH ENGLISH MATH_NEW
0 Ravi 1 80 81 161
1 Raju 2 40 70 110
2 Alex 3 70 40 110
3 Ron 4 70 50 120
4 King 5 82 60 142
5 Jack 6 30 30 60Apply a function down each column 🔝
axis=0 is the default and passes whole columns. Select numeric columns to avoid trying numeric calculations on student names.
df = student_data()
result = df[["MATH", "ENGLISH"]].apply(lambda column: column.max() - column.min(), axis=0)
print(result)
assert result.to_dict() == {"MATH": 52, "ENGLISH": 51}Expected output
MATH 52
ENGLISH 51
dtype: int64Apply a lambda to a Series 🔝
Series.apply() passes each score to the lambda. For adding a constant, direct arithmetic is simpler and produces the same values.
df = student_data()
df["MATH_NEW"] = df["MATH"].apply(lambda value: value + 5)
print(df)
assert df["MATH_NEW"].equals(df["MATH"] + 5)Expected output
NAME ID MATH ENGLISH MATH_NEW
0 Ravi 1 80 81 85
1 Raju 2 40 70 45
2 Alex 3 70 40 75
3 Ron 4 70 50 75
4 King 5 82 60 87
5 Jack 6 30 30 35Use a named function and a conditional rule 🔝
A named function is useful when the rule needs explanation or reuse. Pass it directly instead of wrapping a one-argument function in a redundant lambda. Do not shadow Python built-ins such as sum.
def add_marks(value):
return value + 5
def adjust_low_mark(value):
return value + 5 if value < 50 else value
df = student_data()
df["all_plus_five"] = df["MATH"].apply(add_marks)
df["low_only"] = df["MATH"].apply(adjust_low_mark)
print(df)
assert df["low_only"].tolist() == [80, 45, 70, 70, 82, 35]Expected output
NAME ID MATH ENGLISH all_plus_five low_only
0 Ravi 1 80 81 85 80
1 Raju 2 40 70 45 45
2 Alex 3 70 40 75 70
3 Ron 4 70 50 75 70
4 King 5 82 60 87 82
5 Jack 6 30 30 35 35Combine two fields in a row function 🔝
axis=1 gives access to fields by their column labels. The named two-argument function and the lambda expression below calculate the same result. For this simple total, adding the two Series is also sufficient.
def total_marks(math, english):
return math + english
df = student_data()
df["total"] = df.apply(lambda row: total_marks(row["MATH"], row["ENGLISH"]), axis=1)
lambda_total = df.apply(lambda row: row["MATH"] + row["ENGLISH"], axis=1)
print(df)
assert df["total"].equals(lambda_total)
assert df["total"].equals(df["MATH"] + df["ENGLISH"])Expected output
NAME ID MATH ENGLISH total
0 Ravi 1 80 81 161
1 Raju 2 40 70 110
2 Alex 3 70 40 110
3 Ron 4 70 50 120
4 King 5 82 60 142
5 Jack 6 30 30 60Return Boolean results or readable labels 🔝
For this synthetic rule, a total of 120 or more is Pass. Use a conditional expression for labels. Comparison methods can create the Boolean result without apply().
df = student_data()
df["total"] = df["MATH"] + df["ENGLISH"]
df["status"] = df["total"].apply(lambda value: value >= 120)
df["label"] = df["total"].apply(lambda value: "Pass" if value >= 120 else "Fail")
print(df)
assert df["status"].equals(df["total"].ge(120))
assert df["label"].tolist() == ["Pass", "Fail", "Fail", "Pass", "Pass", "Fail"]Expected output
NAME ID MATH ENGLISH total status label
0 Ravi 1 80 81 161 True Pass
1 Raju 2 40 70 110 False Fail
2 Alex 3 70 40 110 False Fail
3 Ron 4 70 50 120 True Pass
4 King 5 82 60 142 True Pass
5 Jack 6 30 30 60 False FailPass extra arguments to the function 🔝
args is a tuple of positional arguments supplied after the row or column. Keyword arguments can be passed by name. Keep threshold choices visible rather than hiding them inside a function.
def threshold_label(row, threshold, pass_text="Pass"):
return pass_text if row["MATH"] + row["ENGLISH"] >= threshold else "Review"
df = student_data()
labels = df.apply(threshold_label, axis=1, args=(120,), pass_text="Ready")
print(labels)
assert labels.tolist() == ["Ready", "Review", "Review", "Ready", "Ready", "Review"]Expected output
0 Ready
1 Review
2 Review
3 Ready
4 Ready
5 Review
dtype: strExpand several returned values into columns 🔝
With axis=1, result_type="expand" turns list-like results into columns. Give those columns names before joining them to the source table. Return a consistent shape for every row.
df = student_data()
details = df.apply(lambda row: [row["MATH"] + row["ENGLISH"], max(row["MATH"], row["ENGLISH"])], axis=1, result_type="expand")
details.columns = ["total", "best_score"]
print(details)
assert details.shape == (6, 2)
assert details.loc[0].tolist() == [161, 81]Expected output
total best_score
0 161 81
1 110 70
2 110 70
3 120 70
4 142 82
5 60 30Use raw=True only when labels are unnecessary 🔝
raw=True passes arrays instead of labelled Series. Select the columns first and avoid string-key lookups inside such functions. result_type="broadcast" preserves the input shape, while "reduce" favours a Series result for suitable row functions. These examples use the ordinary Python engine; optional execution engines require additional compatibility checks.
scores = student_data()[["MATH", "ENGLISH"]]
totals = scores.apply(np.sum, axis=1, raw=True)
print(totals)
assert totals.tolist() == [161, 110, 110, 120, 142, 60]Expected output
0 161
1 110
2 110
3 120
4 142
5 60
dtype: int64Handle missing values explicitly 🔝
A custom comparison must account for missing values before using an if statement. Do not label unknown scores as Fail automatically. For row totals, sum() with min_count can enforce a required number of available scores.
def score_label(value):
if pd.isna(value):
return "Missing"
return "Pass" if value >= 50 else "Fail"
scores = pd.Series([80, pd.NA, 40], dtype="Int64")
print(scores.apply(score_label))
pair = pd.DataFrame({"MATH": [80, 40], "ENGLISH": [pd.NA, 50]}).astype("Int64")
print("Require both scores:")
print(pair.sum(axis=1, min_count=2))
assert scores.apply(score_label).tolist() == ["Pass", "Missing", "Fail"]
assert pd.isna(pair.sum(axis=1, min_count=2).iloc[0])Expected output
0 Pass
1 Missing
2 Fail
dtype: str
Require both scores:
0 <NA>
1 90
dtype: Int64Create review reasons for a synthetic order extract 🔝
Use built-in methods to normalise and parse fields, then a custom row function to combine readable review reasons. The rules here require a non-blank product and a non-negative numeric amount. Retain the original extract for inspection.
raw = pd.DataFrame({"order_id": [101, 102, 103, 104, 105], "product": [" Pen ", " ", "Book", None, "Pencil"], "amount": ["5.5", "-3", "bad", "8", "0"]})
cleaned = raw.copy()
product = raw["product"].astype("string").str.strip()
cleaned["product"] = product.mask(product.eq(""))
cleaned["amount"] = pd.to_numeric(raw["amount"], errors="coerce").astype("Float64")
def review_reason(row):
reasons = []
if pd.isna(row["product"]):
reasons.append("missing product")
if pd.isna(row["amount"]):
reasons.append("invalid or missing amount")
elif row["amount"] < 0:
reasons.append("negative amount")
return "; ".join(reasons) if reasons else "accepted"
cleaned["review_reason"] = cleaned.apply(review_reason, axis=1)
print(cleaned)
accepted = cleaned.loc[cleaned["review_reason"].eq("accepted")].copy()
review = raw.loc[~cleaned["review_reason"].eq("accepted")].copy()
print("Original review rows:")
print(review)
assert accepted["order_id"].tolist() == [101, 105]
assert cleaned.loc[1, "review_reason"] == "missing product; negative amount"
assert review["order_id"].tolist() == [102, 103, 104]Expected output
order_id product amount review_reason
0 101 Pen 5.5 accepted
1 102 <NA> -3.0 missing product; negative amount
2 103 Book <NA> invalid or missing amount
3 104 <NA> 8.0 missing product
4 105 Pencil 0.0 accepted
Original review rows:
order_id product amount
1 102 -3
2 103 Book bad
3 104 NaN 8Compare with built-in operations and export 🔝
A row function helps produce combined reasons; a simple acceptance mask needs no row apply. Prefer clear built-in operations for arithmetic, text replacement, conditional replacement and keeping valid values. Performance depends on the workload; measure representative data before choosing an implementation.
valid = cleaned["product"].notna() & cleaned["amount"].ge(0).fillna(False)
assert valid.tolist() == cleaned["review_reason"].eq("accepted").tolist()
print("Accepted orders:", int(valid.sum()))
print(accepted.to_csv(index=False).rstrip())
# Optional Colab export:
# accepted.to_csv("accepted_orders.csv", index=False)Expected output
Accepted orders: 2
order_id,product,amount,review_reason
101,Pen,5.5,accepted
105,Pencil,0.0,acceptedCommon questions 🔝
Does axis=1 mean each column is passed? It passes each row. axis=0 passes each column. Why is a column name unavailable? Check the selected columns and whether raw=True removed labels. Should I mutate the row inside my function? Return a value instead; mutating the passed object is unsupported. Is apply() always faster than a loop? No; choose and measure the operation appropriate for your data. Why is output unexpectedly shaped? Keep return types consistent and inspect result_type. What about an empty table? Validate required columns and define the expected empty output schema before assignment. For statistics and summaries, visit Data Analysis.
Exercise: classify a record using two fields 🔝
Create orders with quantity 2, 0 and 3, and status paid, paid and test. Use a row function to return ready only when quantity is positive and status is paid. Predict the labels and verify with a Boolean mask.
Exercise solution 🔝
This sample has no missing fields. If missing values are possible, add an explicit policy before testing the fields.
practice = pd.DataFrame({"quantity": [2, 0, 3], "status": ["paid", "paid", "test"]})
def order_label(row):
return "ready" if row["quantity"] > 0 and row["status"] == "paid" else "review"
answer = practice.apply(order_label, axis=1)
print(answer)
ready = practice["quantity"].gt(0) & practice["status"].eq("paid")
assert answer.tolist() == ["ready", "review", "review"]
assert ready.equals(answer.eq("ready"))Expected output
0 ready
1 review
2 review
dtype: strPractice in Google Colab 🔝
Open in Google Colab View on GitHub
Run the examples in order, change the custom rules and inspect totals, labels and review reasons. Save your own copy to keep edits. All sample tables are included in the notebook.
Continue with the Data Cleaning hub, sort_values(), groupby(), cut(), where(), loc, at, NumPy where(), fillna() and iloc.
Reference: Pandas DataFrame.apply documentation.
Subhendu Mohapatra
Author & Instructor at plus2net
I write and maintain practical tutorials on Python, PHP, SQL, JavaScript, HTML, jQuery, and web development at plus2net. The tutorials focus on clear explanations, working examples, and code that readers can test and adapt while learning.
Subscribe to our YouTube Channel here
Python Video Tutorials