Data Engineering with Scala and Spark

Build streaming and batch pipelines that process massive amounts of data using Scala

Eric Tome , Rupam Bhattacharjee , David Radford 지음

Packt(GCO Science) 2024년 01월 31일 출간

작품소개

책소개

Take your data engineering skills to the next level by learning how to utilize Scala and functional programming to create continuous and scheduled pipelines that ingest, transform, and aggregate data

▶Book Description
Most data engineers know that performance issues in a distributed computing environment can easily lead to issues impacting the overall efficiency and effectiveness of data engineering tasks. While Python remains a popular choice for data engineering due to its ease of use, Scala shines in scenarios where the performance of distributed data processing is paramount.
This book will teach you how to leverage the Scala programming language on the Spark framework and use the latest cloud technologies to build continuous and triggered data pipelines. You’ll do this by setting up a data engineering environment for local development and scalable distributed cloud deployments using data engineering best practices, test-driven development, and CI/CD. You’ll also get to grips with DataFrame API, Dataset API, and Spark SQL API and its use. Data profiling and quality in Scala will also be covered, alongside techniques for orchestrating and performance tuning your end-to-end pipelines to deliver data to your end users.
By the end of this book, you will be able to build streaming and batch data pipelines using Scala while following software engineering best practices.

▶ What You Will Learn
⦁ Set up your development environment to build pipelines in Scala
⦁ Get to grips with polymorphic functions, type parameterization, and Scala implicits
⦁ Use Spark DataFrames, Datasets, and Spark SQL with Scala
⦁ Read and write data to object stores
⦁ Profile and clean your data using Deequ
⦁ Performance tune your data pipelines using Scala

목차

▶ TABLE of CONTENTS
1. Scala Essentials for Data Engineers
2. Environment Setup
3. An Introduction to Apache Spark and Its APIs – DataFrame, Dataset, and Spark SQL
4. Working with Databases
5. Object Stores and Data Lakes
6. Understanding Data Transformation
7. Data Profiling and Data Quality
8. Test-Driven Development, Code Health, and Maintainability
9. CI/CD with GitHub
10. Data Pipeline Orchestration
11. Performance Tuning
12. Building Batch Pipelines Using Spark and Scala
13. Building Streaming Pipelines Using Spark and Scala

책 속으로

▶ What this book covers
⦁ Chapter 1, Scala Essentials for Data Engineers, introduces Scala in data engineering, recognizing its importance due to type safety, adoption by major companies such as Netflix and Airbnb, native integration with Spark, fostering a software engineering mindset, and its versatility in both objectoriented and functional programming. The chapter covers concepts such as functional programming, objects, classes, higher-order functions, polymorphism, variance, option types, collections, pattern matching, and implicits in Scala.
⦁ Chapter 2, Environment Setup, presents two data engineering pipeline development environments. The first, a cloud-based setup, offers portability and easy access but incurs costs for system maintenance. The second involves local machine utilization, requiring a setup but avoiding cloud expenses.
⦁ Chapter 3, An Introduction to Apache Spark and Its APIs – DataFrame, Dataset, and Spark SQL, focuses on Apache Spark as a leading distributed data processing framework. It emphasizes handling large data volumes across machine clusters. Topics include working with Spark, building Spark applications with Scala, and comprehending Spark’s Dataset and DataFrame APIs for effective data processing.
⦁ Chapter 4, Working with Databases, dives into relational databases’ utilization within data pipelines, emphasizing efficiency in reading from and writing to databases. It covers the Spark API and building a straightforward database library, exploring Spark’s JDBC API, loading configurations, creating an interface, and executing multiple database operations.
⦁ Chapter 5, Object Stores and Data Lakes, discusses the evolution from traditional databases to the era of data lakes and lakehouses, due to surges in data volumes. The focus will be on object stores, which are fundamental for both data lakes and lake houses.
⦁ Chapter 6, Understanding Data Transformation, goes deeper into essential Spark skills for data engineers aiming to transform data for downstream use cases. It covers advanced Spark topics such as the distinctions between transformations and actions, aggregation, grouping, joining data, utilizing window functions, and handling complex dataset types.
⦁ Chapter 7, Data Profiling and Data Quality, stresses the importance of data quality checks in preventing issues downstream. It introduces the Deequ library, an open source tool by Amazon, for defining checks, performing analysis, suggesting constraints, and storing metrics.
⦁ Chapter 8, Test-Driven Development, Code Health, and Maintainability discusses software development best practices applied to data engineering, defect identification, code consistency, and security. It introduces Test-Driven Development (TDD), unit tests, integration tests, code coverage checks, static code analysis, and the importance of linting and code style for development practices.
⦁ Chapter 9, CI/CD with GitHub, introduces Continuous Integration/Continuous Delivery (CI/CD) concepts in Scala data engineering projects using GitHub. It explains CI/CD as automated testing and deployment, aiming for rapid iteration, error reduction, and consistent quality. Chapter 10, Data Pipeline Orchestration, focuses on data pipeline orchestration, emphasizing the need for seamless task coordination and failure notification. It introduces tools such as Apache Airflow, Argo, Databricks Workflows, and Azure Data Factory.
⦁ Chapter 11, Performance Tuning, emphasizes the critical role of the Spark UI in optimizing performance. It covers topics such as the Spark UI basics, performance tuning, computing resource optimization, understanding data skewing, indexing, and partitioning.
⦁ Chapter 12, Building Batch Pipelines Using Spark and Scala, combines all of your previously learned skills to construct a batch pipeline. It stresses the significance of batch processing, leveraging Apache Spark’s distributed processing and Scala’s versatility. The topics cover a typical business use case, medallion architecture, batch data ingestion, transformation, quality checks, loading into a serving layer, and pipeline orchestration.
⦁ Chapter 13, Building Streaming Pipelines Using Spark and Scala, focuses on constructing a streaming pipeline, emphasizing real-time data ingestion using Azure Event Hubs, configured as Apache Kafka for Spark integration. It employs Spark’s Structured Streaming and Scala for efficient data handling. Topics include use case understanding, streaming data ingestion, transformation, serving layer loading, and orchestration, aiming to equip you with the skills to develop and implement similar pipelines in your organizations.

출판사 서평

▶ Preface
Every company today is a data company regardless of the industry. Innovative companies use data to analyze the past, predict what will happen, and react to what is happening now. Data engineers are some of the most critical employees at companies today. They are essential for collecting, cleaning, and maintaining trusted datasets that analysts, data scientists, and reporting tools use to provide insights.
This book will teach you to leverage the Scala programming language on the Spark framework and the latest cloud technologies to build continuous and triggered data pipelines. You will do this by setting up a data engineering environment for local development and scalable distributed cloud deployments, using data engineering best practices, test-driven development, and Continuous Integration/Continuous Delivery (CI/CD). You will also orchestrate and performance-tune your end-to-end pipelines to deliver data to your end users.

작가정보

저자(글) Eric Tome

Eric Tome has over 25 years of experience working with data. He has contributed to and led teams that ingested, cleansed, standardized, and prepared data used by business intelligence, data science, and operations teams. He has a background in mathematics and currently works as a senior solutions architect at Databricks, helping customers solve their data and AI challenges.

저자(글) Rupam Bhattacharjee

Rupam Bhattacharjee works as a lead data engineer at IBM. He has architected and developed data pipelines, processing massive structured and unstructured data using Spark and Scala for on-premises Hadoop and K8s clusters on the public cloud. He has a degree in electrical engineering.

저자(글) David Radford

David Radford has worked in big data for over 10 years, with a focus on cloud technologies. He led consulting teams for several years, completing a migration from legacy systems to modern data stacks. He holds a master’s degree in computer science and works as a senior solutions architect at Databricks.