Data cleaning by using Pandas methods

Data we get from different sources may not be in the form or format for direct use in our applications. We need to correct these data by using various methods available in Pandas.

thresh	Using thresh option of dropna() removing rows and columns
dropna	Delete NaN rows or columns
df.drop_duplicates	Delete Duplicate rows from DataFrame
duplicated	duplicate rows from DataFrame
Series.duplicated	duplicate value from Series
Series.drop_duplicates	Delete Duplicate data from Series
replace	Replace data
notnull	Check for Not Null and NaN data
fillna	Fill NA/NaN values
Email	Validate Email address

dtypes	Pandas Data types
select_dtypes	Subset of DataFrame based on data type

In Pandas missing data is represented by two value: None and NaN : NaN (an acronym for Not a Number)
It is generally recommended to use NaN (np.nan) instead of None for missing or undefined values in numerical columns of a Pandas DataFrame. This allows Pandas to retain the original data type (such as int), while treating the NaN values as missing data. This ensures consistent data types and enables proper handling of missing values using Pandas' functions and methods designed for numerical data.

Removing comma from a string column

df['p_view']=df['p_view'].apply(lambda x: x.replace(',',''))

Converting to integer data type

df['p_view'] = df['p_view'].astype('int')

A regex which selects only characters in UTF-8, removing the rest, for each field in the dataframe.

df.replace({r'[^\x00-\x7F]+':''}, regex=True, inplace=True)

Pandas columns() add_prefix() add_suffix()
Pandas Data Analysis

Numpy arrays Python & MySQL Python- Tutorials

Subhendu Mohapatra

Author

🎥 Join me live on YouTube

Passionate about coding and teaching, I publish practical tutorials on PHP, Python, JavaScript, SQL, and web development. My goal is to make learning simple, engaging, and project‑oriented with real examples and source code.

Subscribe to our YouTube Channel here

Cleaning Data

Subhendu Mohapatra