Python Data Cleaning Cookbook

Modern techniques and Python tools to detect and remove dirty data and extract key insights

Michael Walker 지음

Packt(GCO Science) 2020년 12월 11일 출간

작품소개

책소개

Discover how to describe your data in detail, identify data issues, and find out how to solve them using commonly used techniques and tips and tricks

▶Book Description
Getting clean data to reveal insights is essential, as directly jumping into data analysis without proper data cleaning may lead to incorrect results. This book shows you tools and techniques that you can apply to clean and handle data with Python. You'll begin by getting familiar with the shape of data by using practices that can be deployed routinely with most data sources. Then, the book teaches you how to manipulate data to get it into a useful form. You'll also learn how to filter and summarize data to gain insights and better understand what makes sense and what does not, along with discovering how to operate on data to address the issues you've identified. Moving on, you'll perform key tasks, such as handling missing values, validating errors, removing duplicate data, monitoring high volumes of data, and handling outliers and invalid dates. Next, you'll cover recipes on using supervised learning and Naive Bayes analysis to identify unexpected values and classification errors, and generate visualizations for exploratory data analysis (EDA) to visualize unexpected values. Finally, you'll build functions and classes that you can reuse without modification when you have new data.

By the end of this Python book, you'll be equipped with all the key skills that you need to clean data and diagnose problems within it.

▶What You Will Learn
-Find out how to read and analyze data from a variety of sources
-Produce summaries of the attributes of data frames, columns, and rows
-Filter data and select columns of interest that satisfy given criteria
-Address messy data issues, including working with dates and missing values
-Improve your productivity in Python pandas by using method chaining
-Use visualizations to gain additional insights and identify potential data issues
-Enhance your ability to learn wha

목차

▶TABLE of CONTENTS
-Chapter 1: Anticipating Data Cleaning Issues when Importing Tabular Data into pandas
-Chapter 2: Anticipating Data Cleaning Issues when Importing HTML and JSON into pandas
-Chapter 3: Taking the Measure of Your Data
-Chapter 4: Identifying Missing Values and Outliers in Subsets of Data
-Chapter 5: Using Visualizations for the Identification of Unexpected Values
-Chapter 6: Cleaning and Exploring Data with Series Operations
-Chapter 7: Fixing Messy Data when Aggregating
-Chapter 8: Addressing Data Issues When Combining DataFrames
-Chapter 9: Tidying and Reshaping Data
-Chapter 10: User-Defined Functions and Classes to Automate Data Cleaning

책 속으로

▶What this book covers
- Chapter 1, Anticipating Data Cleaning Issues when Importing Tabular Data into pandas, explores tools for loading CSV files, Excel files, relational database tables, SAS, SPSS, and Stata files, and R files into pandas DataFrames.

- Chapter 2, Anticipating Data Cleaning Issues when Importing HTML and JSON into pandas, discusses techniques for reading and normalizing JSON data, and for web scraping.

- Chapter 3, Taking the Measure of Your Data, introduces common techniques for navigating around a DataFrame, selecting columns and rows, and generating summary statistics.

- Chapter 4, Identifying Missing Values and Outliers in Subsets of Data, explores a wide range of strategies to identify missing values and outliers across a whole DataFrame and by selected groups.

- Chapter 5, Using Visualizations for the Identification of Unexpected Values, demonstrates the use of matplotlib and seaborn tools to visualize how key variables are distributed, including with histograms, boxplots, scatter plots, line plots, and violin plots.

- Chapter 6, Cleaning and Exploring Data with Series Operations, discusses updating pandas series with scalars, arithmetic operations, and conditional statements based on the values of one or more series.

- Chapter 7, Fixing Messy Data when Aggregating, demonstrates multiple approaches to aggregating data by group, and discusses when to choose one approach over the others.

- Chapter 8, Addressing Data Issues when Combining DataFrames, examines different strategies for concatenating and merging data, and how to anticipate common data challenges when combining data.

- Chapter 9, Tidying and Reshaping Data, introduces several strategies for de-duplicating, stacking, melting, and pivoting data.

- Chapter 10, User-Defined Functions and Classes to Automate Data Cleaning, examines how to turn many of the techniques from the first nine chapters into reusable code.

출판사 서평

▶ Preface
This book is a practical guide to data cleaning, broadly defined as all tasks necessary to prepare data for analysis. It is organized by the tasks usually completed during the data cleaning process: importing data, viewing data diagnostically, identifying outliers and unexpected values, imputing values, tidying data, and so on. Each recipe walks the reader from raw data through the completion of a specific data cleaning task.

There are already a number of very good pandas books. Unsurprisingly, there is some overlap between those texts and this one. However, the emphasis here is different. I focus as much on the why as on the how in this book.

Since pandas is still relatively new, the lessons I have learned about cleaning data have been shaped by my experiences with other tools. Before settling into my current work routine with Python and R about 8 years ago, I relied mostly on C# and T-SQL in the early 2000s, SAS and Stata in the 90s, and FORTRAN and Pascal in the 80s. Most readers of this text probably have experience with a variety of data cleaning and analysis tools. In many ways the specific tool is less significant than the data preparation task and the attributes of the data. I would have covered pretty much the same topics if I had been asked to write The SAS Data Cleaning Cookbook or The R Data Cleaning Cookbook. I just take a Python/pandas specific approach to the same data cleaning challenges that analysts have faced for decades.

I start each chapter with how to think about the particular data cleaning task at hand before discussing how to approach it with a tool from the Python ecosystem - pandas, NumPy, matplotlib, SciPy, and so on. This is reinforced in each recipe by a discussion of the implications of what we are uncovering in the data. I try to connect tool to purpose. For example, concepts like skew and kurtosis matter as much for handling outliers as does knowing how to update pandas series values.

작가정보

저자(글) Michael Walker

Michael Walker has worked as a data analyst for over 30 years at a variety of educational institutions. He has also taught data science, research methods, statistics, and computer programming to undergraduates since 2006. He generates public sector and foundation reports and conducts analyses for publication in academic journals.