Deep Reinforcement Learning Hands-On Second Edition

Apply modern RL methods to practical problems of chatbots, robotics, discrete optimization, web automation, and more

Maxim Lapan 지음

Packt(GCO Science) 2020년 01월 31일 출간

작품소개

책소개

▶Book Description
Deep Reinforcement Learning Hands-On, Second Edition is an updated and expanded version of the bestselling guide to the very latest reinforcement learning (RL) tools and techniques. It provides you with an introduction to the fundamentals of RL, along with the hands-on ability to code intelligent learning agents to perform a range of practical tasks.

With six new chapters devoted to a variety of up-to-the-minute developments in RL, including discrete optimization (solving the Rubik's Cube), multi-agent methods, Microsoft's TextWorld environment, advanced exploration techniques, and more, you will come away from this book with a deep understanding of the latest innovations in this emerging field.

In addition, you will gain actionable insights into such topic areas as deep Q-networks, policy gradient methods, continuous control problems, and highly scalable, non-gradient methods. You will also discover how to build a real hardware robot trained with RL for less than $100 and solve the Pong environment in just 30 minutes of training using step-by-step code optimization.

In short, Deep Reinforcement Learning Hands-On, Second Edition, is your companion to navigating the exciting complexities of RL as it helps you attain experience and knowledge through real-world examples.

▶What You Will Learn
- Understand the deep learning context of RL and implement complex deep learning models
- Evaluate RL methods including cross-entropy, DQN, actor-critic, TRPO, PPO, DDPG, D4PG, and others
- Build a practical hardware robot trained with RL methods for less than $100
- Discover Microsoft's TextWorld environment, which is an interactive fiction games platform
- Use discrete optimization in RL to solve a Rubik's Cube
- Teach your agent to play Connect 4 using AlphaGo Zero
- Explore the very latest deep RL research on topics including AI chatbots
- Discover advanced exploration techniques, including noisy networks and network distillation techniques

▶Key Features
- Second edition of the bestselling introduction to deep reinforcement learning, expanded with six new chapters
- Learn advanced exploration techniques including noisy networks, pseudo-count, and network distillation methods
- Apply RL methods to cheap hardware robotics platforms

▶Who This Book Is For
Some fluency in Python is assumed. Sound understanding of the fundamentals of deep learning will be helpful. This book is an introduction to deep RL and requires no background in RL

목차

▶TABLE of CONTENTS
1. What Is Reinforcement Learning?
2. OpenAI Gym
3. Deep Learning with PyTorch
4. The Cross-Entropy Method
5. Tabular Learning and the Bellman Equation
6. Deep Q-Networks
7. Higher-Level RL Libraries
8. DQN Extensions
9. Ways to Speed up RL
10. Stocks Trading Using RL
11. Policy Gradients ？ an Alternative
12. The Actor-Critic Method
13. Asynchronous Advantage Actor-Critic
14. Training Chatbots with RL
15. The TextWorld Environment
16. Web Navigation
17. Continuous Action Space
18. RL in Robotics
19. Trust Regions ？ PPO, TRPO, ACKTR, and SAC
20. Black-Box Optimization in RL
21. Advanced Exploration
22. Beyond Model-Free ？ Imagination
23. AlphaGo Zero
24. RL in Discrete Optimization
25. Multi-agent RL

책 속으로

▶What this book covers
- Chapter 1, What Is Reinforcement Learning?, contains an introduction to RL ideas and the main formal models.

- Chapter 2, OpenAI Gym, introduces the practical aspects of RL, using the open source library Gym.

- Chapter 3, Deep Learning with PyTorch, gives a quick overview of the PyTorch library.

- Chapter 4, The Cross-Entropy Method, introduces one of the simplest methods in RL to give you an impression of RL methods and problems.

- Chapter 5, Tabular Learning and the Bellman Equation, introduces the value-based family of RL methods.

- Chapter 6, Deep Q-Networks, describes deep Q-networks (DQNs), an extension of the basic value-based methods, allowing us to solve a complicated environment.

- Chapter 7, Higher-Level RL Libraries, describes the library PTAN, which we will use in the book to simplify the implementations of RL methods.

- Chapter 8, DQN Extensions, gives a detailed overview of a modern extension to the DQN method, to improve its stability and convergence in complex environments.

- Chapter 9, Ways to Speed up RL Methods, provides an overview of ways to make the execution of RL code faster.

- Chapter 10, Stocks Trading Using RL, is the first practical project and focuses on applying the DQN method to stock trading.

- Chapter 11, Policy Gradients―an Alternative, introduces another family of RL methods that is based on policy learning.

- Chapter 12, The Actor-Critic Method, describes one of the most widely used methods in RL.

- Chapter 13, Asynchronous Advantage Actor-Critic, extends the actor-critic method with parallel environment communication, which improves stability and convergence.

- Chapter 14, Training Chatbots with RL, is the second project and shows how to apply RL methods to natural language processing problems.

- Chapter 15, The TextWorld Environment, covers the application of RL methods to interactive fiction games.

- Chapter 16, Web Navigation, is another long project that applies RL to web page navigation using the MiniWoB set of tasks.

- Chapter 17, Continuous Action Space, describes the specifics of environments using continuous action spaces and various methods.

- Chapter 18, RL in Robotics, covers the application of RL methods to robotics problems. In this chapter, I describe the process of building and training a small hardware robot with RL methods.

- Chapter 19, Trust Regions ？ PPO, TRPO, ACKTR, and SAC, is yet another chapter about continuous action spaces describing the trust region set of methods.

- Chapter 20, Black-Box Optimization in RL, shows another set of methods that don't use gradients in their explicit form.

- Chapter 21, Advanced Exploration, covers different approaches that can be used for better exploration of the environment.

- Chapter 22, Beyond Model-Free ？ Imagination, introduces the model-based approach to RL and uses recent research results about imagination in RL.

- Chapter 23, AlphaGo Zero, describes the AlphaGo Zero method and applies it to the game Connect 4.

- Chapter 24, RL in Discrete Optimization, describes the application of RL methods to the domain of discrete optimization, using the Rubik's Cube as an environment.

- Chapter 25, Multi-agent RL, introduces a relatively new direction of RL methods for situations with multiple agents.

출판사 서평

▶ Preface
The topic of this book is reinforcement learning (RL), which is a subfield of machine learning (ML); it focuses on the general and challenging problem of learning optimal behavior in a complex environment. The learning process is driven only by the reward value and observations obtained from the environment. This model is very general and can be applied to many practical situations, from playing games to optimizing complex manufacturing processes.

Due to its flexibility and generality, the field of RL is developing very quickly and attracting lots of attention, both from researchers who are trying to improve existing methods or create new methods and from practitioners interested in solving their problems in the most efficient way.

작가정보

저자(글) Maxim Lapan

Maxim Lapan is a deep learning enthusiast and independent researcher. His background and 15 years' work expertise as a software developer and a systems architect lies from low-level Linux kernel driver development to performance optimization and design of distributed applications working on thousands of servers. With vast work experiences in big data, machine learning, and large parallel distributed HPC and non-HPC systems, he is able to explain a number of complicated concepts in simple words and vivid examples. His current areas of interest are in practical applications of deep learning, such as deep natural language processing and deep reinforcement learning. Maxim lives in Moscow, Russian Federation, with his family.