Deepseek has lately made fairly a buzz within the AI group, because of its spectacular efficiency at comparatively low prices. I feel this can be a good alternative to dive deeper into how Giant Language Fashions (LLMs) are skilled. On this article, we’ll give attention to the Reinforcement Studying (RL) facet of issues: we’ll cowl TRPO, PPO, and, extra lately, GRPO (don’t fear, I’ll clarify all these phrases quickly!)
I’ve aimed to maintain this text comparatively simple to learn and accessible, by minimizing the maths, so that you gained’t want a deep Reinforcement Studying background to observe alongside. Nevertheless, I’ll assume that you’ve got some familiarity with Machine Studying, Deep Studying, and a primary understanding of how LLMs work.
I hope you benefit from the article!
The three steps of LLM coaching
Earlier than diving into RL specifics, let’s briefly recap the three major phases of coaching a Giant Language Mannequin:
- Pre-training: the mannequin is skilled on a large dataset to foretell the subsequent token in a sequence based mostly on previous tokens.
- Supervised Superb-Tuning (SFT): the mannequin is then fine-tuned on extra focused information and aligned with particular directions.
- Reinforcement Studying (usually referred to as RLHF for Reinforcement Studying with Human Suggestions): that is the main focus of this text. The principle purpose is to additional refine responses’ alignments with human preferences, by permitting the mannequin to be taught instantly from suggestions.
Reinforcement Studying Fundamentals
Earlier than diving deeper, let’s briefly revisit the core concepts behind Reinforcement Studying.
RL is sort of simple to know at a excessive stage: an agent interacts with an atmosphere. The agent resides in a selected state throughout the atmosphere and might take actions to transition to different states. Every motion yields a reward from the atmosphere: that is how the atmosphere gives suggestions that guides the agent’s future actions.
Think about the next instance: a robotic (the agent) navigates (and tries to exit) a maze (the atmosphere).
- The state is the present state of affairs of the atmosphere (the robotic’s place within the maze).
- The robotic can take completely different actions: for instance, it could actually transfer ahead, flip left, or flip proper.
- Efficiently navigating in direction of the exit yields a constructive reward, whereas hitting a wall or getting caught within the maze ends in damaging rewards.
Straightforward! Now, let’s now make an analogy to how RL is used within the context of LLMs.
RL within the context of LLMs
When used throughout LLM coaching, RL is outlined by the next elements:
- The LLM itself is the agent
- Surroundings: all the things exterior to the LLM, together with person prompts, suggestions programs, and different contextual info. That is principally the framework the LLM is interacting with throughout coaching.
- Actions: these are responses to a question from the mannequin. Extra particularly: these are the tokens that the LLM decides to generate in response to a question.
- State: the present question being answered together with tokens the LLM has generated up to now (i.e., the partial responses).
- Rewards: this is a little more tough right here: not like the maze instance above, there may be often no binary reward. Within the context of LLMs, rewards often come from a separate reward mannequin, which outputs a rating for every (question, response) pair. This mannequin is skilled from human-annotated information (therefore “RLHF”) the place annotators rank completely different responses. The purpose is for higher-quality responses to obtain greater rewards.
Observe: in some instances, rewards can really get easier. For instance, in DeepSeekMath, rule-based approaches can be utilized as a result of math responses are typically extra deterministic (appropriate or flawed reply)
Coverage is the ultimate idea we’d like for now. In RL phrases, a coverage is just the technique for deciding which motion to take. Within the case of an LLM, the coverage outputs a chance distribution over potential tokens at every step: briefly, that is what the mannequin makes use of to pattern the subsequent token to generate. Concretely, the coverage is decided by the mannequin’s parameters (weights). Throughout RL coaching, we regulate these parameters so the LLM turns into extra prone to produce “higher” tokens— that’s, tokens that produce greater reward scores.
We frequently write the coverage as:
the place a is the motion (a token to generate), s the state (the question and tokens generated up to now), and θ (mannequin’s parameters).
This concept of discovering the very best coverage is the entire level of RL! Since we don’t have labeled information (like we do in supervised studying) we use rewards to regulate our coverage to take higher actions. (In LLM phrases: we regulate the parameters of our LLM to generate higher tokens.)
TRPO (Belief Area Coverage Optimization)
An analogy with supervised studying
Let’s take a fast step again to how supervised studying usually works. you could have labeled information and use a loss operate (like cross-entropy) to measure how shut your mannequin’s predictions are to the true labels.
We are able to then use algorithms like backpropagation and gradient descent to attenuate our loss operate and replace the weights θ of our mannequin.
Recall that our coverage additionally outputs chances! In that sense, it’s analogous to the mannequin’s predictions in supervised studying… We’re tempted to write down one thing like:
the place s is the present state and a is a potential motion.
A(s, a) is named the benefit operate and measures how good is the chosen motion within the present state, in comparison with a baseline. That is very very similar to the notion of labels in supervised studying however derived from rewards as an alternative of specific labeling. To simplify, we will write the benefit as:
In observe, the baseline is calculated utilizing a worth operate. It is a widespread time period in RL that I’ll clarify later. What you could know for now’s that it measures the anticipated reward we’d obtain if we proceed following the present coverage from the state s.
What’s TRPO?
TRPO (Belief Area Coverage Optimization) builds on this concept of utilizing the benefit operate however provides a crucial ingredient for stability: it constrains how far the brand new coverage can deviate from the outdated coverage at every replace step (much like what we do with batch gradient descent for instance).
- It introduces a KL divergence time period (see it as a measure of similarity) between the present and the outdated coverage:
- It additionally divides the coverage by the outdated coverage. This ratio, multiplied by the benefit operate, offers us a way of how helpful every replace is relative to the outdated coverage.
Placing all of it collectively, TRPO tries to maximize a surrogate goal (which entails the benefit and the coverage ratio) topic to a KL divergence constraint.
PPO (Proximal Coverage Optimization)
Whereas TRPO was a big development, it’s not used broadly in observe, particularly for coaching LLMs, on account of its computationally intensive gradient calculations.
As an alternative, PPO is now the popular strategy in most LLMs structure, together with ChatGPT, Gemini, and extra.
It’s really fairly much like TRPO, however as an alternative of imposing a tough constraint on the KL divergence, PPO introduces a “clipped surrogate goal” that implicitly restricts coverage updates, and vastly simplifies the optimization course of.
Here’s a breakdown of the PPO goal operate we maximize to tweak our mannequin’s parameters.
GRPO (Group Relative Coverage Optimization)
How is the worth operate often obtained?
Let’s first discuss extra in regards to the benefit and the worth capabilities I launched earlier.
In typical setups (like PPO), a worth mannequin is skilled alongside the coverage. Its purpose is to foretell the worth of every motion we take (every token generated by the mannequin), utilizing the rewards we get hold of (do not forget that the worth ought to symbolize the anticipated cumulative reward).
Right here is the way it works in observe. Take the question “What’s 2+2?” for instance. Our mannequin outputs “2+2 is 4” and receives a reward of 0.8 for that response. We then go backward and attribute discounted rewards to every prefix:
- “2+2 is 4” will get a worth of 0.8
- “2+2 is” (1 token backward) will get a worth of 0.8γ
- “2+2” (2 tokens backward) will get a worth of 0.8γ²
- and so forth.
the place γ is the low cost issue (0.9 for instance). We then use these prefixes and related values to coach the worth mannequin.
Essential observe: the worth mannequin and the reward mannequin are two various things. The reward mannequin is skilled earlier than the RL course of and makes use of pairs of (question, response) and human rating. The worth mannequin is skilled concurrently to the coverage, and goals at predicting the long run anticipated reward at every step of the technology course of.
What’s new in GRPO
Even when in observe, the reward mannequin is commonly derived from the coverage (coaching solely the “head”), we nonetheless find yourself sustaining many fashions and dealing with a number of coaching procedures (coverage, reward, worth mannequin). GRPO streamlines this by introducing a extra environment friendly technique.
Bear in mind what I stated earlier?
In PPO, we determined to make use of our worth operate because the baseline. GRPO chooses one thing else: Here’s what GRPO does: concretely, for every question, GRPO generates a bunch of responses (group of dimension G) and makes use of their rewards to calculate every response’s benefit as a z-score:
the place rᵢ is the reward of the i-th response and μ and σ are the imply and commonplace deviation of rewards in that group.
This naturally eliminates the necessity for a separate worth mannequin. This concept makes numerous sense when you consider it! It aligns with the worth operate we launched earlier than and likewise measures, in a way, an “anticipated” reward we will get hold of. Additionally, this new technique is effectively tailored to our drawback as a result of LLMs can simply generate a number of non-deterministic outputs by utilizing a low temperature (controls the randomness of tokens technology).
That is the principle thought behind GRPO: eliminating the worth mannequin.
Lastly, GRPO provides a KL divergence time period (to be precise, GRPO makes use of a easy approximation of the KL divergence to enhance the algorithm additional) instantly into its goal, evaluating the present coverage to a reference coverage (usually the post-SFT mannequin).
See the ultimate formulation beneath:
And… that’s principally it for GRPO! I hope this provides you a transparent overview of the method: it nonetheless depends on the identical foundational concepts as TRPO and PPO however introduces extra enhancements to make coaching extra environment friendly, sooner, and cheaper — key components behind DeepSeek’s success.
Conclusion
Reinforcement Studying has turn out to be a cornerstone for coaching right this moment’s Giant Language Fashions, notably by way of PPO, and extra lately GRPO. Every technique rests on the identical RL fundamentals — states, actions, rewards, and insurance policies — however provides its personal twist to stability stability, effectivity, and human alignment:
• TRPO launched strict coverage constraints through KL divergence
• PPO eased these constraints with a clipped goal
• GRPO took an additional step by eradicating the worth mannequin requirement and utilizing group-based reward normalization. In fact, DeepSeek additionally advantages from different improvements, like high-quality information and different coaching methods, however that’s for one more time!
I hope this text gave you a clearer image of how these strategies join and evolve. I imagine that Reinforcement Studying will turn out to be the principle focus in coaching LLMs to enhance their efficiency, surpassing pre-training and SFT in driving future improvements.
For those who’re thinking about diving deeper, be happy to take a look at the references beneath or discover my earlier posts.
Thanks for studying, and be happy to go away a clap and a remark!
Wish to be taught extra about Transformers or dive into the maths behind the Curse of Dimensionality? Take a look at my earlier articles:
Transformers: How Do They Transform Your Data?
Diving into the Transformers architecture and what makes them unbeatable at language taskstowardsdatascience.com
The Math Behind “The Curse of Dimensionality”
Dive into the “Curse of Dimensionality” concept and understand the math behind all the surprising phenomena that arise…towardsdatascience.com
References:
Source link
