Deep Reinforcement Learning with Python Second Edition

Master classic RL, deep RL, distributional RL, inverse RL, and more with OpenAI Gym and TensorFlow

Sudharsan Ravichandiran 지음

Packt(GCO Science) 2020년 09월 30일 출간

작품소개

책소개

An example-rich guide for beginners to start their reinforcement and deep reinforcement learning journey with state-of-the-art distinct algorithms

▶What You Will Learn
？Understand core RL concepts including the methodologies, math, and code
？Train an agent to solve Blackjack, FrozenLake, and many other problems using OpenAI Gym
？Train an agent to play Ms Pac-Man using a Deep Q Network
？Learn policy-based, value-based, and actor-critic methods
？Master the math behind DDPG, TD3, TRPO, PPO, and many others
？Explore new avenues such as the distributional RL, meta RL, and inverse RL
？Use Stable Baselines to train an agent to walk and play Atari games

▶Key Features
？Covers a vast spectrum of basic-to-advanced RL algorithms with mathematical explanations of each algorithm
？Learn how to implement algorithms with code by following examples with line-by-line explanations
？Explore the latest RL methodologies such as DDPG, PPO, and the use of expert demonstrations

▶Who This Book Is For
If you're a machine learning developer with little or no experience with neural networks interested in artificial intelligence and want to learn about reinforcement learning from scratch, this book is for you.

Basic familiarity with linear algebra, calculus, and the Python programming language is required. Some experience with TensorFlow would be a plus.

목차

▶TABLE of CONTENTS
？Chapter 1: Fundamentals of Reinforcement Learning
？Chapter 2: A Guide to the Gym Toolkit
？Chapter 3: The Bellman Equation and Dynamic Programming
？Chapter 4: Monte Carlo Methods
？Chapter 5: Understanding Temporal Difference Learning
？Chapter 6: Case Study ？ The MAB Problem
？Chapter 7: Deep Learning Foundations
？Chapter 8: A Primer on TensorFlow
？Chapter 9: Deep Q Network and Its Variants
？Chapter 10: Policy Gradient Method
？Chapter 11: Actor-Critic Methods ？ A2C and A3C
？Chapter 12: Learning DDPG, TD3, and SAC
？Chapter 13: TRPO, PPO, and ACKTR Methods
？Chapter 14: Distributional Reinforcement Learning
？Chapter 15: Imitation Learning and Inverse RL
？Chapter 16: Deep Reinforcement Learning with Stable Baselines
？Chapter 17: Reinforcement Learning Frontiers
？Appendix 1 ？ Reinforcement Learning Algorithms
？Appendix 2 ？ Assessments

책 속으로

▶What this book covers
？ Chapter 1, Fundamentals of Reinforcement Learning, helps you build a strong foundation on RL concepts. We will learn about the key elements of RL, the Markov decision process, and several important fundamental concepts such as action spaces, policies, episodes, the value function, and the Q function. At the end of the chapter, we will learn about some of the interesting applications of RL and we will also look into the key terms and terminologies frequently used in RL.

？ Chapter 2, A Guide to the Gym Toolkit, provides a complete guide to OpenAI's Gym toolkit. We will understand several interesting environments provided by Gym in detail by implementing them. We will begin our hands-on RL journey from this chapter by implementing several fundamental RL concepts using Gym.

？ Chapter 3, The Bellman Equation and Dynamic Programming, will help us understand the Bellman equation in detail with extensive math. Next, we will learn two interesting classic RL algorithms called the value and policy iteration methods, which we can use to find the optimal policy. We will also see how to implement value and policy iteration methods for solving the Frozen Lake problem.

？ Chapter 4, Monte Carlo Methods, explains the model-free method, Monte Carlo. We will learn what prediction and control tasks are, and then we will look into Monte Carlo prediction and Monte Carlo control methods in detail. Next, we will implement the Monte Carlo method to solve the blackjack game using the Gym toolkit.

？ Chapter 5, Understanding Temporal Difference Learning, deals with one of the most popular and widely used model-free methods called Temporal Difference (TD) learning. First, we will learn how the TD prediction method works in detail, and then we will explore the on-policy TD control method called SARSA and the off-policy TD control method called Q learning in detail. We will also implement TD control methods to solve the Frozen Lake problem using Gym.

？ Chapter 6, Case Study ？ The MAB Problem, explains one of the classic problems in RL called the multi-armed bandit (MAB) problem. We will start the chapter by understanding what the MAB problem is and then we will learn about several exploration strategies such as epsilon-greedy, softmax exploration, upper confidence bound, and Thompson sampling methods for solving the MAB problem in detail.

？ Chapter 7, Deep Learning Foundations, helps us to build a strong foundation on deep learning. We will start the chapter by understanding how artificial neural networks work. Then we will learn several interesting deep learning algorithms, such as recurrent neural networks, LSTM networks, convolutional neural networks, and generative adversarial networks.

？ Chapter 8, A Primer on TensorFlow, deals with one of the most popular deep learning libraries called TensorFlow. We will understand how to use TensorFlow by implementing a neural network to recognize handwritten digits. Next, we will learn to perform several math operations using TensorFlow. Later, we will learn about TensorFlow 2.0 and see how it differs from the previous TensorFlow versions.

？ Chapter 9, Deep Q Network and Its Variants, enables us to kick-start our deep RL journey. We will learn about one of the most popular deep RL algorithms called the Deep Q Network (DQN). We will understand how DQN works step by step along with the extensive math. We will also implement a DQN to play Atari games. Next, we will explore several interesting variants of DQN, called Double DQN, Dueling DQN, DQN with prioritized experience replay, and DRQN.

？ Chapter 10, Policy Gradient Method, covers policy gradient methods. We will understand how the policy gradient method works along with the detailed derivation. Next, we will learn several variance reduction methods such as policy gradient with reward-to-go and policy gradient with baseline. ...

출판사 서평

▶ Preface
With significant enhancements in the quality and quantity of algorithms in recent years, this second edition of Hands-On Reinforcement Learning with Python has been revamped into an example-rich guide to learning state-of-the-art reinforcement learning (RL) and deep RL algorithms with TensorFlow 2 and the OpenAI Gym toolkit.

In addition to exploring RL basics and foundational concepts such as Bellman equation, Markov decision processes, and dynamic programming algorithms, this second edition dives deep into the full spectrum of value-based, policy-based, and actor-critic RL methods. It explores state-of-the-art algorithms such as DQN, TRPO, PPO and ACKTR, DDPG, TD3, and SAC in depth, demystifying the underlying math and demonstrating implementations through simple code examples.

The book has several new chapters dedicated to new RL techniques, including distributional RL, imitation learning, inverse RL, and meta RL. You will learn to leverage stable baselines, an improvement of OpenAI's baseline library, to effortlessly implement popular RL algorithms. The book concludes with an overview of promising approaches such as meta-learning and imagination augmented agents in research.

By the end, you will become skilled in effectively employing RL and deep RL in your real-world projects.

작가정보

저자(글) Sudharsan Ravichandiran

Sudharsan Ravichandiran is a data scientist, researcher, best selling author, and YouTuber (search for "Sudharsan reinforcement learning"). He completed his Bachelor's in Information Technology at Anna University. His area of research focuses on practical implementations of deep learning and reinforcement learning, including Natural Language Processing and computer vision. He is an open-source contributor and loves answering questions on Stack Overflow. He also authored a best-seller, Hands-On Reinforcement Learning with Python, published by Packt Publishing.