Dissecting “Reinforcement Studying” by Richard S. Sutton with customized Python implementations, Episode V
In our earlier put up, we wrapped up the introductory sequence on elementary reinforcement studying (RL) strategies by exploring Temporal-Distinction (TD) studying. TD strategies merge the strengths of Dynamic Programming (DP) and Monte Carlo (MC) strategies, leveraging their finest options to type a few of the most necessary RL algorithms, comparable to Q-learning.
Constructing on that basis, this put up delves into n-step TD studying, a flexible strategy launched in Chapter 7 of Sutton’s ebook [1]. This methodology bridges the hole between classical TD and MC strategies. Like TD, n-step strategies use bootstrapping (leveraging prior estimates), however additionally they incorporate the subsequent n rewards, providing a novel mix of short-term and long-term studying. In a future put up, we’ll generalize this idea even additional with eligibility traces.
We’ll comply with a structured strategy, beginning with the prediction drawback earlier than transferring to management. Alongside the way in which, we’ll:
- Introduce n-step Sarsa,
- Prolong it to off-policy studying,
- Discover the n-step tree backup algorithm, and
- Current a unifying perspective with n-step Q(σ).
As at all times, you will discover all accompanying code on GitHub. Let’s dive in!
