Pandas read_html(): Select Web Tables and Clean Imported Data
Use read_html() to read HTML tables into a list of DataFrames. Choose the intended table, inspect its labels and clean imported values before analysis.
Read HTML into a list of DataFrames 🔝
read_html() returns a list, even for a single table. Select a DataFrame from that list after checking the table count. Wrap literal HTML in StringIO. This embedded table contains the familiar PHP date-function examples; the live date index is an optional source.
import pandas as pd
import tempfile
from pathlib import Path
from io import StringIO
functions = pd.DataFrame({"Function": ["Date", "createfromformat()", "checkdate()", "date()", "date_create()"],
"Description": ["PHP Date & Time object", "Change date format", "Validating date", "Required date and time in given format", "Creating date objects"]})
html_text = functions.to_html(index=False, table_id="functions")
tables = pd.read_html(StringIO(html_text), flavor="lxml")
df = tables[0]
print("Return type:", type(tables).__name__, "Tables:", len(tables))
print(df.head())
assert len(tables) == 1 and df.equals(functions)
# Optional live source; table contents and order may change:
# tables = pd.read_html("https://www.plus2net.com/php_tutorial/site_map-date.php")Expected output
Return type: list Tables: 1
Function Description
0 Date PHP Date & Time object
1 createfromformat() Change date format
2 checkdate() Validating date
3 date() Required date and time in given format
4 date_create() Creating date objectsRead a local student HTML file 🔝
Use an actual local path. index_col=0 selects the first column as row labels. A relative path uses the current working directory; raw strings are useful for Windows paths. The student sample is embedded here and converted to HTML for practice.
student_csv = 'id,name,class,mark,gender\n1,John Deo,Four,75,female\n2,Max Ruin,Three,85,male\n3,Arnold,Three,55,male\n4,Krish Star,Four,60,female\n5,John Mike,Four,60,female\n6,Alex John,Four,55,male\n7,My John Rob,Fifth,78,male\n8,Asruid,Five,85,male\n9,Tes Qry,Six,78,male\n10,Big John,Four,55,female\n11,Ronald,Six,89,female\n12,Recky,Six,94,female\n13,Kty,Seven,88,female\n14,Bigy,Seven,88,female\n15,Tade Row,Four,88,male\n16,Gimmy,Four,88,male\n17,Tumyu,Six,54,male\n18,Honny,Five,75,male\n19,Tinny,Nine,18,male\n20,Jackly,Nine,65,female\n21,Babby John,Four,69,female\n22,Reggid,Seven,55,female\n23,Herod,Eight,79,male\n24,Tiddy Now,Seven,78,male\n25,Giff Tow,Seven,88,male\n26,Crelea,Seven,79,male\n27,Big Nose,Three,81,female\n28,Rojj Base,Seven,86,female\n29,Tess Played,Seven,55,male\n30,Reppy Red,Six,79,female\n31,Marry Toeey,Four,88,male\n32,Binn Rott,Seven,90,female\n33,Kenn Rein,Six,96,female\n34,Gain Toe,Seven,69,male\n35,Rows Noump,Six,88,female\n'
students = pd.read_csv(StringIO(student_csv))
student_html = students.to_html(index=False, table_id="students")
with tempfile.TemporaryDirectory(dir=Path.cwd()) as folder:
target = Path(folder) / "student.html"
target.write_text(student_html, encoding="utf-8")
indexed = pd.read_html(target, index_col=0, encoding="utf-8", flavor="lxml")[0]
print(indexed.tail())
assert indexed.index.name == "id" and len(indexed) == len(students)
# Windows example: pd.read_html(r"C:\data\student.html", index_col=0)Expected output
name class mark gender
id
31 Marry Toeey Four 88 male
32 Binn Rott Seven 90 female
33 Kenn Rein Six 96 female
34 Gain Toe Seven 69 male
35 Rows Noump Six 88 femaleRead from an open file or HTML text 🔝
Use a context manager to close an opened file. read_html() accepts a readable file object. If you read text yourself, wrap it in StringIO before parsing. See Python file handling.
with tempfile.TemporaryDirectory(dir=Path.cwd()) as folder:
target = Path(folder) / "student.html"
target.write_text(student_html, encoding="utf-8")
with target.open(encoding="utf-8") as handle:
from_file = pd.read_html(handle, index_col=0, flavor="lxml")[0]
text = target.read_text(encoding="utf-8")
from_text = pd.read_html(StringIO(text), index_col=0, flavor="lxml")[0]
print("Same result:", from_file.equals(from_text))
assert from_file.equals(from_text)Expected output
Same result: TrueInspect multiple tables before choosing one 🔝
Table positions can change when a website adds a table. Inspect columns and shape rather than assuming that position zero always contains the intended data. The sample below embeds two topic tables with the same class attribute.
topics = pd.DataFrame({"Topic": ["Operators", "MySQL database", "Django", "Pygsheet"],
"Description": ["Expressions", "Database access", "Web framework", "Spreadsheet access"]})
page_html = (functions.to_html(index=False, table_id="tb1", classes="table table-striped")
+ topics.to_html(index=False, table_id="tb2", classes="table table-striped"))
all_tables = pd.read_html(StringIO(page_html), flavor="lxml")
for position, table in enumerate(all_tables):
print(position, table.columns.tolist(), table.shape)
print(all_tables[1])
assert len(all_tables) == 2Expected output
0 ['Function', 'Description'] (5, 2)
1 ['Topic', 'Description'] (4, 2)
Topic Description
0 Operators Expressions
1 MySQL database Database access
2 Django Web framework
3 Pygsheet Spreadsheet accessSelect a table by matching its text 🔝
match is a regular expression applied to table text. It selects whole tables, not only the matching rows. Match distinctive text, then validate the returned columns. Similar matching can be tried on the Python tutorial index.
selected = pd.read_html(StringIO(page_html), match="Pygsheet", flavor="lxml")
print(selected[0])
for term in ["Operators", "MySQL database", "Django"]:
matches = pd.read_html(StringIO(page_html), match=term, flavor="lxml")
assert len(matches) == 1 and len(matches[0]) == 4
assert selected[0].columns.tolist() == ["Topic", "Description"]Expected output
Topic Description
0 Operators Expressions
1 MySQL database Database access
2 Django Web framework
3 Pygsheet Spreadsheet accessSelect by table ID or class 🔝
attrs filters HTML table attributes. A unique ID is generally clearer than a shared class. A class filter can return several tables; check the count rather than relying on a fixed number from a live page.
by_id = pd.read_html(StringIO(page_html), attrs={"id": "tb2"}, flavor="lxml")
print(by_id[0])
by_class = pd.read_html(StringIO(page_html), attrs={"class": "dataframe table table-striped"}, flavor="lxml")
print("Matching class tables:", len(by_class))
print("Second table columns:", by_class[1].columns.tolist())
assert len(by_id) == 1 and len(by_class) == 2Expected output
Topic Description
0 Operators Expressions
1 MySQL database Database access
2 Django Web framework
3 Pygsheet Spreadsheet access
Matching class tables: 2
Second table columns: ['Topic', 'Description']Choose headings after skipping report rows 🔝
header is the parsed row position to use as column labels. skiprows removes parsed table rows; it does not refer to source-code line numbers. Here the report title is inside the table and is skipped before choosing its heading.
report = """<table><tr><td>Weekly report</td><td>Practice data</td></tr>
<tr><td>Product</td><td>Amount</td></tr>
<tr><td>Pen</td><td>5</td></tr><tr><td>Book</td><td>20</td></tr></table>"""
report_frame = pd.read_html(StringIO(report), skiprows=1, header=0, flavor="lxml")[0]
print(report_frame)
assert report_frame.columns.tolist() == ["Product", "Amount"]Expected output
Product Amount
0 Pen 5
1 Book 20Handle rowspan and colspan headings 🔝
The parser expands merged cells. Multiple heading rows can produce a MultiIndex. Inspect those labels before flattening them; careless flattening can create duplicate names. See column inspection.
merged_html = """<table><thead>
<tr><th rowspan="2">Product</th><th colspan="2">Sales</th></tr>
<tr><th>Units</th><th>Revenue</th></tr></thead>
<tbody><tr><td>Pen</td><td>2</td><td>5</td></tr></tbody></table>"""
merged = pd.read_html(StringIO(merged_html), header=[0, 1], flavor="lxml")[0]
print("Parsed labels:", merged.columns.tolist())
merged.columns = ["Product", "Sales_units", "Sales_revenue"]
print(merged)
assert merged.iloc[0]["Sales_revenue"] == 5Expected output
Parsed labels: [('Product', 'Product'), ('Sales', 'Units'), ('Sales', 'Revenue')]
Product Sales_units Sales_revenue
0 Pen 2 5Choose whether to include hidden rows 🔝
displayed_only=True excludes elements marked with display:none. Set False when those rows belong in your import. This is not a browser rendering engine; external stylesheets and JavaScript visibility may not be reproduced.
hidden_html = """<table><tr><th>Product</th><th>Amount</th></tr>
<tr><td>Pen</td><td>5</td></tr>
<tr style="display:none"><td>Book</td><td>20</td></tr></table>"""
visible = pd.read_html(StringIO(hidden_html), displayed_only=True, flavor="lxml")[0]
including_hidden = pd.read_html(StringIO(hidden_html), displayed_only=False, flavor="lxml")[0]
print("Visible rows:", len(visible), "Including hidden:", len(including_hidden))
assert len(visible) == 1 and len(including_hidden) == 2Expected output
Visible rows: 1 Including hidden: 2Parse decimal and missing-value conventions 🔝
thousands and decimal describe numeric punctuation. na_values and keep_default_na define missing markers. read_html() extracts displayed values, so currency symbols or footnotes may still need cleaning. See Data Cleaning.
regional_html = """<table><tr><th>Product</th><th>Amount</th></tr>
<tr><td>Pen</td><td>1.234,50</td></tr>
<tr><td>Book</td><td>20,00</td></tr>
<tr><td>Pencil</td><td>--</td></tr></table>"""
regional = pd.read_html(StringIO(regional_html), thousands=".", decimal=",",
na_values=["--"], keep_default_na=False, flavor="lxml")[0]
print(regional)
assert regional["Amount"].sum() == 1254.5 and regional["Amount"].isna().sum() == 1Expected output
Product Amount
0 Pen 1234.5
1 Book 20.0
2 Pencil NaNPreserve text identifiers during parsing 🔝
read_html() has no dtype parameter. Use converters for fields that must remain text, then verify the result. Converting an already inferred number to text cannot recover lost leading zeros.
id_html = """<table><tr><th>Order</th><th>Amount</th></tr>
<tr><td>001</td><td>5</td></tr><tr><td>002</td><td>20</td></tr></table>"""
identified = pd.read_html(StringIO(id_html), converters={"Order": str}, flavor="lxml")[0]
print(identified)
assert identified["Order"].tolist() == ["001", "002"]Expected output
Order Amount
0 001 5
1 002 20Extract link text and href values 🔝
extract_links="body" returns body-cell values as text/href pairs. A relative href remains relative; resolve it against the actual page URL if you need an absolute URL. Cells without a link have None as the href.
link_html = """<table><tr><th>Topic</th><th>Level</th></tr>
<tr><td><a href="pandas-read_csv.php">read_csv()</a></td><td>Beginner</td></tr></table>"""
linked = pd.read_html(StringIO(link_html), extract_links="body", flavor="lxml")[0]
print(linked)
print("Text:", linked.iloc[0]["Topic"][0], "Href:", linked.iloc[0]["Topic"][1])
assert linked.iloc[0]["Topic"][1] == "pandas-read_csv.php"Expected output
Topic Level
0 (read_csv(), pandas-read_csv.php) (Beginner, None)
Text: read_csv() Href: pandas-read_csv.phpPractical example: clean an imported sales report 🔝
This synthetic report mixes currency text, invalid amounts and dates. Preserve the raw table. Accept only rows with a non-negative numeric amount and valid ISO date, keeping all others for review. See text replacement and date conversion.
sales_html = """<table id="sales"><tr><th>Order</th><th>Date</th><th>Amount</th></tr>
<tr><td>001</td><td>2026-01-02</td><td>$5.25</td></tr>
<tr><td>002</td><td>2026-01-03</td><td>$20.00</td></tr>
<tr><td>003</td><td>bad</td><td>$8.00</td></tr>
<tr><td>004</td><td>2026-01-04</td><td>pending</td></tr></table>"""
raw = pd.read_html(StringIO(sales_html), attrs={"id": "sales"},
converters={"Order": str}, flavor="lxml")[0]
clean = raw.copy()
amount_text = raw["Amount"].astype("string").str.replace("$", "", regex=False)
clean["Amount"] = pd.to_numeric(amount_text, errors="coerce")
clean["Date"] = pd.to_datetime(raw["Date"], format="%Y-%m-%d", errors="coerce")
valid = clean["Date"].notna() & clean["Amount"].ge(0).fillna(False)
accepted = clean.loc[valid].copy()
review = raw.loc[~valid].copy()
print(accepted)
print("Review orders:", review["Order"].tolist())
print("Accepted revenue:", accepted["Amount"].sum())
assert accepted["Order"].tolist() == ["001", "002"]
assert accepted["Amount"].sum() == 25.25
assert len(accepted) + len(review) == len(raw)Expected output
Order Date Amount
0 001 2026-01-02 5.25
1 002 2026-01-03 20.0
Review orders: ['003', '004']
Accepted revenue: 25.25Check and export the accepted records 🔝
Check expected fields and IDs before exporting. CSV does not preserve every dtype, so specify identifier types when reading it back. See to_csv(), to_excel() and to_html().
csv_text = accepted.to_csv(index=False, date_format="%Y-%m-%d")
print(csv_text.rstrip())
checked = pd.read_csv(StringIO(csv_text), dtype={"Order": "string"})
assert checked["Order"].tolist() == ["001", "002"]
assert checked["Amount"].sum() == 25.25
# Optional Colab download:
# accepted.to_csv("accepted_web_sales.csv", index=False, date_format="%Y-%m-%d")
# from google.colab import files
# files.download("accepted_web_sales.csv")Expected output
Order,Date,Amount
001,2026-01-02,5.25
002,2026-01-03,20.0Common questions 🔝
Does this extract every visible grid? It targets HTML table elements, rather than arbitrary div layouts or images. Can it execute JavaScript? No; prefer a provided CSV or JSON endpoint when the table is loaded dynamically. Why is the result a list? A page may contain several tables. How do I choose one? Use attrs or match and check its fields. Why is a parser missing? Install lxml, or the Beautiful Soup/html5lib combination for that parser. Can merged cells be handled? Yes, but inspect resulting labels and repeated values. Why did a match fail? Check the actual returned HTML and table text; no matching table normally raises ValueError. Can dates be parsed? parse_dates is available; for strict validation use explicit to_datetime() after reading. Can remote pages be read? Yes, when accessible; login, blocked requests and changed layouts can require a different source. A .to_html() export makes a useful controlled round-trip source.
Exercise: select the high-mark students 🔝
Parse student_html with the id field as the index. Select name and mark for students with marks at least 90. Predict the row count before checking the solution.
Exercise solution 🔝
Select rows and fields with loc, then compare the selected count with the condition count.
exercise = pd.read_html(StringIO(student_html), index_col=0, flavor="lxml")[0]
answer = exercise.loc[exercise["mark"].ge(90), ["name", "mark"]]
print(answer)
print("Rows:", len(answer))
assert len(answer) == int(students["mark"].ge(90).sum())Expected output
name mark
id
12 Recky 94
32 Binn Rott 90
33 Kenn Rein 96
Rows: 3Practice in Google Colab 🔝
Open in Google Colab View on GitHub
Run the examples in order, change the HTML tables and parsing options, then inspect imported records and review rows. Save your own copy to keep edits. All sample tables are included in the notebook.
Reference: Pandas read_html documentation.
Subhendu Mohapatra
Author & Instructor at plus2net
I write and maintain practical tutorials on Python, PHP, SQL, JavaScript, HTML, jQuery, and web development at plus2net. The tutorials focus on clear explanations, working examples, and code that readers can test and adapt while learning.
Subscribe to our YouTube Channel here
Python Video Tutorials