Deep Reinforcement Learning: The Actor-Critic Method

that irritating hovering drone from ? The one that learned to descend toward the platform, pass through it, and then just… hang out below it forever? Yeah, me too. I spent an entire afternoon watching it hover there, accumulating negative rewards like a slow-motion crash, and I couldn’t even be mad because technically it was doing exactly what I told it to do.

Let’s dive in. Grab some coffee, you’re going to need it. All the code can be found in my repository on my github.

What’s Actor-Critic?

REINFORCE had one elementary downside: we needed to wait. Look forward to the drone to crash. Look forward to the episode to finish. Wait to compute the total return. Then, and solely then, might we replace the coverage. One studying sign per episode. For a 150-step trajectory, that’s one replace after watching 150 actions play out.

I ran REINFORCE for 1200 iterations (6 hours on my machine) to hit 55% success charge. And the entire time I saved pondering: this feels wasteful. Why can’t I be taught throughout the episode?

Actor-Critic fixes this with a easy thought: prepare a second neural community (the “critic”) to estimate future returns for any state. Then use these estimates to replace the coverage after each single step. No extra ready for episodes to complete. Simply steady studying because the drone flies.

The consequence? 68% success charge in 600 iterations (3 hours). Half the time. Higher efficiency. Identical {hardware}.

The way it works: Two networks collaborate in real-time.

The Actor (π(a|s)): Identical coverage community from REINFORCE. Takes the present state and outputs motion possibilities. That is the community that really controls the drone.

The Critic (V(s)): New community. Takes the present state and estimates “how good is that this state?” It outputs a single worth representing anticipated future rewards. Not tied to any particular motion, simply evaluates states.

Right here’s the intelligent half: the critic offers rapid suggestions. The actor takes an motion, the atmosphere updates, and the critic instantly evaluates whether or not that moved us to a greater or worse state. The actor learns from this sign and adjusts. The critic concurrently learns to make higher predictions. Each networks enhance collectively as episodes unfold.

In code, they seem like this:

class DroneGamerBoi(nn.Module):
    """The Actor: outputs motion possibilities"""
    def __init__(self, state_dim=15):
        tremendous().__init__()
        self.community = nn.Sequential(
            nn.Linear(state_dim, 128), nn.LayerNorm(128), nn.ReLU(),
            nn.Linear(128, 128), nn.LayerNorm(128), nn.ReLU(),
            nn.Linear(128, 64), nn.LayerNorm(64), nn.ReLU(),
            nn.Linear(64, 3),  # Three impartial thrusters
            nn.Sigmoid()
        )

    def ahead(self, state):
        return self.community(state)  # Output: possibilities for every thruster


class DroneTeacherBoi(nn.Module):
    """The Critic: outputs state worth estimate"""
    def __init__(self, state_dim=15):
        tremendous().__init__()
        self.community = nn.Sequential(
            nn.Linear(state_dim, 128), nn.LayerNorm(128), nn.ReLU(),
            nn.Linear(128, 128), nn.LayerNorm(128), nn.ReLU(),
            nn.Linear(128, 64), nn.LayerNorm(64), nn.ReLU(),
            nn.Linear(64, 1)  # Single worth: V(s)
        )

    def ahead(self, state):
        return self.community(state)  # Output: scalar worth estimate

Discover the critic community is nearly an identical to the actor, besides the ultimate layer outputs a single worth (how good is that this state?) as an alternative of motion possibilities.

The Bootstrapping Trick

Okay, right here’s the place it will get intelligent. In REINFORCE, we wanted the total return to replace the coverage:

[ G_t = r_t + gamma r_{t+1} + gamma^2 r_{t+2} + cdots + gamma^{T-t} r_T ]

We needed to wait till the episode ended to know all of the rewards. However what if… we didn’t? What if we simply estimated the long run utilizing our critic community?

As an alternative of computing the precise return, we estimate it:

[ G_t approx r_t + gamma V(s_{t+1}) ]

That is referred to as bootstrapping. The critic “bootstraps” its personal worth estimate to approximate the total return. We use its prediction of “how good will the following state be?” to estimate the return proper now.

Why does this assist?

Decrease variance. We’re not ready for the precise random sequence of future rewards. We’re utilizing an estimate primarily based on what we’ve discovered about states basically. That is noisier than the bottom reality (the critic is likely to be incorrect!), but it surely’s much less noisy than any single episode end result.

On-line studying. We will replace instantly at each step. No want to complete the episode first. As quickly because the drone takes one motion, we all know the rapid reward, and we will estimate what comes subsequent, so we will be taught.

Higher pattern effectivity. In REINFORCE with 6 parallel video games, every drone learns as soon as per episode completion. In Actor-Critic with 6 parallel video games, every drone learns at each step (about 150 steps per episode). That’s 150x extra studying indicators per episode!

After all, there’s a trade-off: we introduce bias. If our critic is incorrect (and it is going to be, particularly early in coaching), our agent learns from incorrect estimates. However the critic doesn’t have to be good. It simply must be much less noisy than a single episode end result. Because the critic progressively improves, the actor learns from higher suggestions. They bootstrap one another upward. In follow, the variance discount is so highly effective that it’s price accepting the small bias.

TD Error: The New Benefit

Now we have to reply: how a lot better or worse was this motion than anticipated?

In REINFORCE, we had the benefit: precise return minus baseline. The baseline was a world common. However we will do a lot better. As an alternative of a world baseline, we use the critic’s state-specific estimate.

The TD (Temporal Distinction) error is our new benefit:

[ delta_t = r_t + gamma V(s_{t+1}) – V(s_t) ]

In plain phrases:

• (r_t + gamma V(s_{t+1})) = TD goal. The rapid reward plus our estimate of the following state’s worth.
• (V(s_t)) = Our prediction for the present state.
• (delta_t) = The distinction. Did we do higher or worse than anticipated?

If (delta_t > 0), we did higher than anticipated → reinforce that motion.

If (delta_t < 0), we did worse than anticipated → lower that motion’s likelihood.

If (delta_t approx 0), we have been spot on → motion was about common.

That is far more informative than REINFORCE’s international baseline. The sign is now state-specific. The drone in a difficult spin may get -10 reward and that’s truly fairly good (normally will get -50 there). But when it’s hovering peacefully over the platform, -10 is horrible. The critic is aware of the distinction. The TD error captures that.

Right here’s how this flows by means of the coaching loop (simplified):

# 1. Take one motion in every parallel recreation
motion = actor(state)
next_state, reward = env.step(motion)

# 2. Get worth estimates
value_current = critic(state)
value_next = critic(next_state)

# 3. Compute TD error (our benefit)
td_error = reward + gamma * value_next - value_current

# 4. Replace the critic: it ought to have predicted higher
#    The critic needs to reduce prediction error, so we use squared error.
#    The gradient then pushes the critic's predictions nearer to precise returns.
critic_loss = td_error ** 2
critic_loss.backward()
critic_optimizer.step()

# 5. Replace the actor: reinforce or discourage primarily based on TD error
#    (similar coverage gradient as REINFORCE, however with TD error as an alternative of returns)
actor_loss = -log_prob(motion) * td_error
actor_loss.backward()
actor_optimizer.step()

Discover we’re updating each networks per step, not per episode. That’s the net studying magic.

Yet one more comparability to make this crystal clear:

The second is extra exact, extra informative, and arrives a lot sooner.

Because of this Actor-Critic converged in 600 iterations on my machine whereas REINFORCE wanted 1200. Identical reward operate, similar atmosphere, similar drone. However getting suggestions after each step as an alternative of each 150 steps? That’s a 150x data benefit per iteration.

The Three Bugs: A Debugging Odyssey

Alright, I’m about to inform you about three bugs that just about broke me. Not “oops, off-by-one error” damaged. I imply the type of damaged the place you stare at coaching curves for six hours, significantly query whether or not you perceive backpropagation, debug your code 5 occasions, after which spend one other two hours studying tutorial papers to persuade your self you’re not insane.

These bugs are adequately subtle that even skilled RL practitioners need to watch out. The excellent news: when you perceive them, they turn out to be apparent. The dangerous information: you need to perceive them first, and I discovered the laborious means.

Bug #1: The Transferring Goal Drawback

The Setup

I applied Actor-Critic precisely because it appeared logical. I’ve two networks. One predicts actions, one predicts values. Easy, proper? I wrote out the TD error computation:

# Compute worth estimates
values = critic(batch_data['states'])
next_values = critic(batch_data['next_states'])

# Compute TD targets and errors
td_targets = rewards + gamma * next_values * (1 - dones)
td_errors = td_targets - values

# Critic loss
critic_loss = (td_errors ** 2).imply()

# Backward go
critic_loss.backward()

This regarded fully affordable to me. We compute what we anticipated (values), we compute what we should always have gotten (td_targets), we measure the error, and we replace. Commonplace supervised studying stuff.

The Symptom: Nothing Works

I educated for 200 iterations and the critic loss was… sitting round 500-1000 and never transferring. Not reducing, not growing, simply oscillating wildly like a sine wave. I checked my reward operate. Appeared superb. I checked the critic community. Commonplace structure, nothing bizarre. I checked the TD error values themselves. They have been bouncing round between -50 and +50, which appeared affordable given the reward scale.

However the loss refused to converge.

I spent two days on this. I added dropout, pondering perhaps overfitting. (Incorrect downside, didn’t assist.) I diminished the educational charge from 1e-3 to 1e-4, pondering perhaps the optimizer was overshooting. (Nope, simply discovered slower whereas oscillating.) I checked if my atmosphere was returning NaNs. (It wasn’t.) I even questioned if PyTorch’s autograd had a bug. (Spoiler: PyTorch is okay, I used to be the bug.)

The Breakthrough

I used to be studying the Actor-Critic chapter in Sutton & Barto (once more, for the fifth time) when one thing caught my eye. The pseudocode had a line about “computing the following worth estimate.” And I assumed: wait, after I compute next_values = critic(next_states), what occurs to these gradients throughout backprop?

After which my mind went click on. Oh no. The goal is transferring as we attempt to optimize towards it. That is referred to as the transferring goal downside.

Why This Breaks All the things

After we compute next_values = critic(next_states) with out detaching, PyTorch’s autograd flows gradients by means of BOTH V(s) and V(s’). Meaning we’re updating the prediction AND the goal concurrently—the critic chases a goal that strikes each time it updates. The gradient turns into:

[ frac{partial L}{partial theta} = 2 cdot (r + gamma V(s’) – V(s)) cdot left( gamma frac{partial V(s’)}{partial theta} – frac{partial V(s)}{partial theta} right) ]

That γ · ∂V(s')/∂θ time period is the issue—we’re telling the critic to vary the goal, not simply the prediction. The loss oscillates eternally.

The Repair (Lastly)

I wanted to deal with the TD goal as a hard and fast fixed. In PyTorch, which means detaching the gradients:

# ✅ CORRECT
values = critic(batch_data['states'])

with torch.no_grad():  # CRITICAL LINE
    next_values = critic(batch_data['next_states'])

td_targets = rewards + gamma * next_values * (1 - dones)
td_errors = td_targets - values

critic_loss = (td_errors ** 2).imply()
critic_loss.backward()

The torch.no_grad() context supervisor says: “Compute these subsequent values, however don’t bear in mind the way you computed them. For gradient functions, deal with this as a relentless.” Now through the backward go:

[ frac{partial L}{partial theta} = 2 cdot (r + gamma V(s’) – V(s)) cdot left( – frac{partial V(s)}{partial theta} right) ]

That problematic time period is gone! Now we’re solely updating V(s), the prediction, to match the mounted goal r + γV(s’). That is precisely what we would like.

The TD goal turns into what it needs to be: a mounted label, like the bottom reality in supervised studying. We’re now not making an attempt to hit a transferring goal. We’re simply making an attempt to foretell one thing steady.

I modified precisely one line. The critic loss went from oscillating chaotically round 500-1000 to reducing easily: 500 → 250 → 100 → 35 → 8 over 200 iterations. This bug is insidious as a result of the code seems to be fully affordable—however at all times detach your TD targets.

Bug #2: Gamma Too Low (Invisible Rewards)

The Setup

Alright, Bug #1 was refined. This bug is embarrassingly apparent on reflection. However you understand what? Generally the obvious errors are the simplest to overlook since you don’t count on the issue to be that straightforward.

I mounted the transferring goal bug and out of the blue the critic loss began converging. Implausible! I felt like an actual engineer for a second there. However then I ran the agent for a full coaching iteration and… nothing. Completely nothing improved. The drone would take just a few random strikes after which instantly crash into the bottom or fly off the display. No studying. No enchancment. No indicators of life.

Really, wait. The critic was studying. The loss was happening. However the drone wasn’t getting higher. That appeared backwards. Why would the critic be taught to foretell values if the agent wasn’t studying something from these values?

The Discovery

I printed the TD targets they usually have been all damaging—starting from -5 to -30. No signal of the +500 touchdown reward. Then I did the maths: with 150-step episodes and gamma=0.90:

[ 500 times 0.90^{150} approx 0.00000006 ]

The touchdown reward had been discounted into oblivion. The agent discovered to crash instantly as a result of making an attempt to land was actually invisible to the worth operate.

The low cost issue γ controls the efficient horizon (≈ 1/(1-γ)). With gamma = 0.90, that’s solely 10 steps—means too brief for 100-300 step episodes.

The repair: change gamma from 0.90 to 0.99.

The Impression

I modified gamma from 0.90 to 0.99. Identical community, similar rewards, similar all the things else.

Outcome: Iteration 5, the drone moved towards the platform. Iteration 50, it slowed when approaching. Iteration 100, first touchdown. By iteration 600, 68% success charge.

One parameter change, fully completely different agent conduct. The terminal reward went from invisible to crystal clear. All the time verify: efficient horizon (1/(1-γ)) ought to match your episode size.

Bug #3: Reward Exploits (The Arms Race)

At this level, I’d mounted each the transferring goal downside and the gamma situation. My agent was truly studying! It approached the platform, slowed down sometimes, and even landed typically. I used to be genuinely excited. Then I began watching the failures extra rigorously, and one thing bizarre occurred.

After fixing bugs #1 and #2, the agent discovered two new exploits:

Zoom-past: Speed up towards the platform at most pace, overshoot, crash distant. Internet reward: -140 (strategy rewards +60, crash penalty -200). Higher than crashing instantly (-300), however not touchdown.

Hovering: Get near the platform and vibrate in place with tiny actions (pace 0.01-0.02) to farm strategy rewards indefinitely whereas avoiding crash penalties.

Why This Occurs: The Basic Drawback

Right here’s the factor that bothered me: My reward operate might solely see the present state, not the trajectory.

The reward operate is r(s', a): given the following state and the motion I simply took, compute my reward. It has no reminiscence. It may’t inform the distinction between:

1. A drone making real progress towards touchdown: approaching from above with managed, purposeful descent
2. A drone farming the reward construction: hovering with meaningless micro-movements

Each situations may need:

• distance_to_platform < 0.3 (shut to focus on)
• pace > 0 (technically transferring)
• velocity_alignment > 0 (pointed in the proper route)

The agent isn’t dumb. It’s doing precisely what I informed it to do—maximize the scalar rewards I’m feeding it. The issue is that the rewards don’t truly encode touchdown, they encode proximity and motion. And proximity with out touchdown is exploitable.

That is the core perception of reward hacking: the agent will discover loopholes in your reward specification, not as a result of it’s intelligent, however since you under-specified the duty.

The Repair: Reward State Transitions, Not Snapshots

The repair: reward primarily based on state transitions r(s, s'), not simply present state r(s'). As an alternative of asking “Is distance < 0.3?”, ask “Did we get nearer (distance_delta > 0) AND transfer quick sufficient to imply it (pace ≥ 0.15)?”

def calc_reward(state: DroneState, prev_state: DroneState = None):
    if prev_state is just not None:
        distance_delta = prev_state.distance_to_platform - state.distance_to_platform
        pace = state.pace
        velocity_toward_platform = calculate_alignment(state)  # cosine similarity

        MIN_MEANINGFUL_SPEED = 0.15

        if pace >= MIN_MEANINGFUL_SPEED and velocity_toward_platform > 0.1:
            speed_multiplier = 1.0 + pace * 2.0
            rewards['approach'] = distance_delta * 15.0 * speed_multiplier
        elif pace < 0.05:
            rewards['hovering_penalty'] = -1.0

Key adjustments: (1) Reward distance_delta (progress), not proximity, (2) MIN_SPEED threshold blocks hovering, (3) Pace multiplier encourages decisive motion.

To make use of this, observe prev_state in your coaching loop and go it to calc_reward(next_state, prev_state).

90% of RL is reward engineering. The opposite 90% is debugging your reward engineering. Rewards are a specification of the target, and the agent will discover each loophole.

Primary Actor-Critic Outcomes

I’ve to confess, after I mounted the third bug (that velocity-magnitude-weighted reward operate) and launched a recent coaching run with all three fixes in place, I used to be skeptical. I’d spent a lot time chasing my tail with these algorithms that I half anticipated Actor-Critic to hit some new, inventive failure mode I hadn’t anticipated. However one thing shocking occurred: it simply… labored.

And I imply actually labored. Higher than REINFORCE, in reality—noticeably higher. After tons of of hours debugging REINFORCE’s reward hacking, I used to be anticipating Actor-Critic to a minimum of match its efficiency. As an alternative, it blew previous it.

Why This Beats REINFORCE (And Why That Issues):

Actor-Critic’s on-line updates create a suggestions loop that REINFORCE can’t match. Each single step, the critic whispers within the actor’s ear: “Hey, that state is nice” or “That state is dangerous.” It’s not a world baseline like REINFORCE makes use of. It’s state-specific analysis that will get higher and higher because the critic learns.

Because of this the convergence is 2x sooner. Because of this the ultimate efficiency is 13% higher. Because of this the educational curves are so clear.

And all of it hinged on three issues: detaching the TD goal, utilizing the proper low cost issue, and monitoring state transitions within the reward operate. No new algorithm methods wanted. Simply right implementation.

What’s Subsequent: Pushing Past Actor-Critic

With Actor-Critic working alright, you will have seen that the coverage is constantly touchdown the drone on the left aspect of the platform, and in addition the actions are barely jittery. To resolve this, I’m engaged on convering Proximal Coverage Optimization (PPO), which is meant to assist with this by “making the educational course of extra steady”. The great factor is, this technique has utilized by the researchers at OpenAI to coach their flagship “GPT” fashions.

References

Foundational RL Papers

1. Sutton, R. S., & Barto, A. G. (2018). Reinforcement Studying: An Introduction (2nd ed.). MIT Press.

Actor-Critic Strategies

2. Konda, V. R., & Tsitsiklis, J. N. (2000). “Actor-Critic Algorithms.” SIAM Journal on Management and Optimization, 42(4), 1143-1166.
   • Theoretical foundations of Actor-Critic with convergence proofs
3. Mnih, V., Badia, A. P., Mirza, M., et al. (2016). “Asynchronous Strategies for Deep Reinforcement Studying.” Worldwide Convention on Machine Studying.

Temporal Distinction Studying

4. Sutton, R. S. (1988). “Studying to Predict by the Strategies of Temporal Variations.” Machine Studying, 3(1), 9-44.
   • Unique TD studying paper

Earlier Posts in This Collection

5. Jumle, V. (2025). “Deep Reinforcement Studying: 0 to 100 – Coverage Gradients (REINFORCE).”

Code Repository & Implementation

6. Jumle, V. (2025). “Reinforcement Studying 101: Supply Drone Touchdown.”

All photographs on this article are both AI-generated (utilizing Gemini or Sora), personally made by me, or screenshots & plots that I made, except specified in any other case.