How to Fine-Tune an SLM for Emotion Recognition

Introduction

fashions (SLMs) fine-tuned for sentiment classification infer sentiment as a single rating, capturing the general emotional tone of the textual content. For a lot of use instances, the positive-negative classification doesn’t inform the total story an organization wants. Emotion recognition fashions go additional, decomposing sentiment into emotion courses (“anger”, “approval”, “disappointment”, and so on.) and assigning chances to a set of feelings within the textual content. It’s then potential to mannequin emotional content material in datasets that an organization receives (buyer tickets, emails, brand-related discussions), and react swiftly to altering situations.

For one in every of our current initiatives, modeling feelings in on-line media, we required an emotion recognition mannequin with open weights and a versatile license, sustaining excessive transparency requirements, and, after all, benefiting from the decrease prices related to open fashions. We subjectively want European fashions, however Hugging Face didn’t supply a Mistral alternative with a developed mannequin card. One potential motive is that essentially the most detailed coaching set for emotion recognition, the 28-emotion GoEmotions dataset, is extremely class-imbalanced. Superb-tuning an SLM on a high-class-imbalance knowledge set that performs decently on the take a look at requires a deeper focus.

We handled the class-imbalance downside by a mix of three methods: (1) undersampling essentially the most represented emotional class, (2) synthetically increasing the minority courses with Nature’s 2025 ISMOTE algorithm, and (3) weighting the loss operate. With this mixture of methods, MistralSmall-3.1.GoEmotions, now launched on Hugging Face, infers most goal feelings related to our mission with F1 > 0.7.

This text explains intimately tips on how to fine-tune an open-weight SLM. We’ll additionally work out: 

• Easy methods to preprocess class-imbalanced knowledge for LLM fine-tuning with the 2025 ISMOTE algorithm.
• Easy methods to decompose sentiment into emotion classes by finetuning a Small Language Mannequin for emotion recognition in textual content knowledge.

2. Information

GoEmotions is a human-annotated dataset of 58k Reddit feedback extracted from English-language subreddits and labeled with 27 emotion categories and a “impartial” label. It’s a multi-label classification dataset through which every remark could also be labeled with a number of TRUEs for feelings (e.g., “Hitting me. That simply added one other humorous dynamic to it though I wasn’t truly attempting to hit her” is True for “amusement”, and “annoyance”).

The dataset was launched on TensorFlow Datasets below the Apache 2.0 License and accommodates 54,263 labeled texts. Here’s what it seems like:

After a fast examine, we will see a high-class imbalance within the knowledge the place the impartial class prevails:

3. Coaching set preprocessing

Our purpose is to develop a classifier to establish 15 feelings in general-language texts. Coaching on class-imbalanced knowledge can introduce bias, because the fine-tuned mannequin tends to favor the bulk class and carry out worse on the minority ones, so preprocessing is crucial.

We used a mix of strategies for the coaching set; the validation and take a look at units remained unchanged to handle class imbalance and maximize efficiency on the goal feelings (worry, disappointment, disgust, disapproval, annoyance, anger, disappointment, optimism, amusement, shock, admiration, pleasure, confusion, pleasure, love): 

• We thinned the information by randomly filtering the “impartial” rows.
• We generated artificial samples for the least-represented emotional classes utilizing ISMOTE (Improved Artificial Minority Over-sampling Approach).

The ISMOTE algorithm extends the frequent SMOTE method by (1) increasing the pattern era area and (2) enhancing sampling distribution. The synthetically generated samples then have extra sensible knowledge distributions than these produced by the unique methodology.

By lowering the bulk class and synthetically increasing the minority classes to 4000 samples, we constructed a comparatively balanced set for fine-tuning. The code for ISMOTE oversampling is here.

4. SLM Superb-tuning

Amongst Mistral’s fashions, we selected the Small class (Small-3.1-24B-Instruct-2503), which inserts our GPU and offers the multilingual capabilities we want for the classifier. The Unsloth framework makes the finetuning steps uncomplicated and quicker than with Transformers:

1. Information loading — loading preprocessed coaching set, validation, and take a look at units. We use a 60:20:20 break up.

2. Loading the base mannequin — loading the Small-3.1–24B-Instruct-2503 domestically.

3. Apply LoRA —lowering hardware requirements.

4. Multilabel wrapper with focal loss operate — updates the coach for multilabel classification. Additionally provides focal loss to weight the loss operate for a particular set of feelings, prioritizing their efficiency.

 5. Analysis metrics and coaching args— specifying the analysis metrics and hyperparameters for mannequin coaching.

6. Mannequin coaching— coach formulation and launch.

7. Analysis — evaluating the very best mannequin efficiency on the take a look at set.

4.1. Coding

Right here is the code implementation.

4.1.1. Information loading

# Loading augmented prepare, validation and take a look at units
BASE = r"augmented"

def load_split(path: str) -> Dataset:
    with open(path, encoding="utf-8") as f:
        d = json.load(f)
    return Dataset.from_dict({"input_embeds": d["X"], "labels": d["y"]})

train_dataset = load_split(f"{BASE}/prepare.json")
val_dataset   = load_split(f"{BASE}/val.json")
test_dataset  = load_split(f"{BASE}/take a look at.json")

# Formulate embedding dimension
EMBED_DIM = len(train_dataset[0]["input_embeds"])

# Return Pytorch tensors
train_dataset.set_format("torch")
val_dataset.set_format("torch")
test_dataset.set_format("torch")

4.1.2. Loading the bottom mannequin 

# Load base mannequin with Unsloth FastLanguageModel
MODEL_NAME = "unsloth/Mistral-Small-3.1-24B-Instruct-2503"

base_model, _ = FastLanguageModel.from_pretrained(
    model_name=MODEL_NAME,
    max_seq_length=2,
    load_in_4bit=True,
    dtype=torch.bfloat16,
)

4.1.3. Apply LoRA

# Aply Low-rank adaptation (LoRA) 
base_model = FastLanguageModel.get_peft_model(
    base_model,
    r=16,
    lora_alpha=32,
    lora_dropout=0,
    bias="none",
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    use_gradient_checkpointing="unsloth",
    random_state = 3407,
    use_rslora = False, 
)

4.1.4. Multilabel wrapper with focal loss operate

# Focal loss weights for preffered labels  
FOCAL_ALPHA_DEFAULT   = 0.25
FOCAL_ALPHA_PREFERRED = 0.75

PREFERRED_LABELS = {
    "worry", "disappointment", "disgust", "disapproval", "annoyance",
    "anger", "disappointment", "optimism", "amusement", "shock",
    "admiration", "pleasure", "confusion","pleasure","love"
}

FOCAL_ALPHA_PER_LABEL: checklist[float] = [
    FOCAL_ALPHA_PREFERRED if lbl in PREFERRED_LABELS else FOCAL_ALPHA_DEFAULT
    for lbl in EMOTION_LABELS
]

"Per-label weighted focal binary cross-entropy for multi-label issues"
class FocalLossWithAlpha(nn.Module):
        def __init__(self, alpha: checklist[float], gamma: float = 2.0):
        tremendous().__init__()
        self.register_buffer("alpha", torch.tensor(alpha, dtype=torch.float32))
        self.gamma = gamma
    def ahead(self, logits: torch.Tensor, targets: torch.Tensor) -> torch.Tensor:
        probs   = torch.sigmoid(logits)
        p_t     = probs * targets + (1.0 - probs) * (1.0 - targets)
        alpha_t = self.alpha * targets + (1.0 - self.alpha) * (1.0 - targets)
        focal_w = alpha_t * (1.0 - p_t) ** self.gamma
        bce     = nn.practical.binary_cross_entropy_with_logits(
            logits, targets, discount="none"
        )
        return (focal_w * bce).imply()

# Multilabel classification wrapper with focal loss class weighting
class MistralForMultiLabel(nn.Module):
    is_loaded_in_4bit = True

    def __init__(self, spine: nn.Module, num_labels: int,
                 hidden_size: int, embed_dim: int):
        tremendous().__init__()
        self.spine = spine
        _device = torch.gadget("cuda" if torch.cuda.is_available() else "cpu")
        self.projection = nn.Sequential(
            nn.Linear(embed_dim, hidden_size // 2),
            nn.GELU(),
            nn.Linear(hidden_size // 2, hidden_size),
        ).to(_device)
        self.dropout    = nn.Dropout(0.1).to(_device)
        self.classifier = nn.Linear(hidden_size, num_labels).to(_device)
        self.focal_loss = FocalLossWithAlpha(FOCAL_ALPHA_PER_LABEL).to(_device)

    def gradient_checkpointing_enable(self, gradient_checkpointing_kwargs=None):
        self.spine.gradient_checkpointing_enable(gradient_checkpointing_kwargs)

    def gradient_checkpointing_disable(self):
        self.spine.gradient_checkpointing_disable()

    def ahead(
        self,
        input_embeds: torch.Tensor,
        labels: torch.Tensor | None = None,
        **kwargs,
    ):
        B = input_embeds.dimension(0)
        projected = self.projection(input_embeds).unsqueeze(1)
        attn_mask = torch.ones(B, 1, gadget=input_embeds.gadget)

        outputs = self.spine.base_model.mannequin.mannequin(
            inputs_embeds=projected,
            attention_mask=attn_mask,
            output_hidden_states=True,
        )
        pooled = outputs.hidden_states[-1][:, 0, :]
        logits = self.classifier(self.dropout(pooled))

        loss = self.focal_loss(logits, labels.float()) if labels is just not None else None
        return {"loss": loss, "logits": logits}

4.1.5. Analysis metrics and coaching args

# Specifiy the analysis operate
def compute_metrics(eval_pred):
    logits, labels = eval_pred
    probs = torch.sigmoid(torch.tensor(logits)).numpy()
    preds = (probs >= 0.5).astype(int)
    labels = labels.astype(int)

    from sklearn.metrics import accuracy_score

    exact_accuracy  = accuracy_score(labels, preds)
    macro_f1        = f1_score(labels, preds, common="macro", zero_division=0)
    micro_f1        = f1_score(labels, preds, common="micro", zero_division=0)
    macro_precision = precision_score(labels, preds, common="macro", zero_division=0)
    macro_recall    = recall_score(labels, preds, common="macro", zero_division=0)

    per_class_f1        = f1_score(labels, preds, common=None, zero_division=0)
    per_class_recall    = recall_score(labels, preds, common=None, zero_division=0)
    per_class_precision = precision_score(labels, preds, common=None, zero_division=0)
    per_class_accuracy  = (preds == labels).imply(axis=0)

    per_class_metrics = {}
    for i, emotion in enumerate(EMOTION_LABELS):
        per_class_metrics[f"f1_{emotion}"]        = float(per_class_f1[i])
        per_class_metrics[f"recall_{emotion}"]    = float(per_class_recall[i])
        per_class_metrics[f"precision_{emotion}"] = float(per_class_precision[i])
        per_class_metrics[f"accuracy_{emotion}"]  = float(per_class_accuracy[i])

    return {
        "exact_accuracy":   exact_accuracy,
        "macro_f1":         macro_f1,
        "micro_f1":         micro_f1,
        "macro_precision":  macro_precision,
        "macro_recall":     macro_recall,
        **per_class_metrics,
    }

# Specify hyperparameters
training_args = TrainingArguments(
    output_dir=OUTPUT_DIR,            # the place checkpoints and logs are written
    eval_strategy="epoch",            # run analysis as soon as per epoch
    save_strategy="epoch",            # save checkpoint as soon as per epoch
    per_device_train_batch_size=8,    # samples per GPU per step
    per_device_eval_batch_size=16,    # bigger batch is okay — no gradients
    gradient_accumulation_steps=4,    # efficient batch = 8 × 4 = 32
    num_train_epochs=15,              # complete passes over the coaching knowledge
    learning_rate=1e-4,               # peak LR after warmup
    bf16=True,                        # bfloat16 blended precision
    optim="adamw_8bit",               # 8-bit AdamW
    warmup_ratio=0.05,                # first 5 % of steps ramp LR from 0 to peak
    lr_scheduler_type="cosine",       # cosine decay from peak LR to ~0
    logging_steps=25,                 # print loss/LR to console each 25 steps
    logging_first_step=True,          # additionally log step 1 to catch early instability
    load_best_model_at_end=True,      # restore greatest checkpoint after coaching ends
    metric_for_best_model="macro_f1", # criterion used to pick the very best checkpoint
    greater_is_better=True,           # larger macro_f1 is best in analysis
    gradient_checkpointing=False,    
    remove_unused_columns=False,      # maintain input_embeds column
    save_total_limit=15,              # maintain all checkpoints on disk to load the very best mannequin
    weight_decay=0.01,                # L2 regularisation on all trainable parameters
)

4.1.6. Mannequin coaching

# Set-up the coach for multilabel finetuning
class MultiLabelTrainer(Coach):
    def compute_loss(self, mannequin, inputs, return_outputs=False, **kwargs):
        labels = inputs.pop("labels")
        outputs = mannequin(**inputs, labels=labels)
        loss = outputs["loss"]
        return (loss, outputs) if return_outputs else loss

    def _save_checkpoint(self, mannequin, trial, metrics=None):
        tremendous()._save_checkpoint(mannequin, trial)
        ckpt_dir = self._get_output_dir(trial)
        # Save head
        torch.save({
            "projection": mannequin.projection.state_dict(),
            "classifier":  mannequin.classifier.state_dict(),
        }, os.path.be part of(ckpt_dir, "head_weights.pt"))
        # Save LoRA adapter explicitly (bypasses bitsandbytes serialization points)
        mannequin.spine.save_pretrained(os.path.be part of(ckpt_dir, "lora_adapter"))

    def _load_best_model(self):
        best_ckpt = self.state.best_model_checkpoint
        if not best_ckpt:
            return
        # Restore head
        head_path = os.path.be part of(best_ckpt, "head_weights.pt")
        if os.path.exists(head_path):
            head = torch.load(head_path, map_location="cpu")
            self.mannequin.projection.load_state_dict(head["projection"])
            self.mannequin.classifier.load_state_dict(head["classifier"])
            print(f"Head restored from: {best_ckpt}")
        else:
            print(f"WARNING: head_weights.pt not present in {best_ckpt}")
        # Restore LoRA adapter
        lora_path = os.path.be part of(best_ckpt, "lora_adapter")
        if os.path.exists(lora_path):
            from peft import PeftModel
            self.mannequin.spine.load_adapter(lora_path, adapter_name="default")
            print(f"LoRA restored from: {best_ckpt}")
        else:
            print(f"WARNING: lora_adapter/ not present in {best_ckpt}")

# Launch the coach
coach = MultiLabelTrainer(
    mannequin=mannequin,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
    compute_metrics=compute_metrics,
)

# Launch coaching
coach.prepare()

Superb-tuning for 15 epochs took 9 hours and half-hour on a machine with an NVIDIA RTX 6000 GPU and 192 GB of VRAM, with the very best mannequin loaded on the finish.

4.1.7. Mannequin analysis

Let’s present the efficiency on the take a look at dataset. The usual statistics for mannequin analysis per class are F1, Precision, and Recall. We are able to see comparatively good efficiency on the goal feelings, with F1 scores over 0.7, for many classes. Full efficiency is on the model card.

5. Abstract

Let’s now summarize the important thing factors of the article. The necessities and full code are on this repo.

• Emotion recognition modeling extends sentiment evaluation by decomposing the general sentiment rating into its emotional elements.
• MistralSmall-3.1.GoEmotions is on Hugging Face below the Apache 2.0 license. The repo additionally consists of the inference guideline.
• Deployment use instances are brand and social monitoring, and email categorization.

Petr Koráb is the founding father of Textual content Mining Tales, a Prague-based improvement & consultancy firm. Be taught extra concerning the cutting-edge NLP on our blog.

AI assertion. Some elements of the code have been reviewed by Sonnet 4.6 (Cursor). No textual content was generated utilizing AI.

Acknowledgements. The Nationwide Financial institution of Slovakia Basis supported this improvement. I thank Martin Feldkircher, Václav Jež, and Michala Moravcová for feedback and recommendations.

References

[1] Ying Li, Yali Yang, Peihua Track, Lian Duan, Rui Ren. 2025. An improved SMOTE algorithm for enhanced imbalanced knowledge classification by increasing pattern era area. Scientific Reports, 15 (23521).

[2] Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov
Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer. 2020. Multilingual Denoising Pre-training for Neural Machine Translation. Transactions of the Affiliation for Computational Linguistics, 8, pp. 726–742.