AM on a Tuesday (nicely, technically Wednesday, I suppose), when my telephone buzzed with that acquainted, dreaded PagerDuty notification.
I didn’t even must open my laptop computer to know that the daily_ingest.py script had failed. Once more.
It retains failing as a result of our knowledge supplier all the time modifications their file format with out warning. I imply, they may randomly swap from commas to pipes and even mess up the dates in a single day.
Often, the precise repair takes me nearly thirty seconds: I merely open the script, swap sep=',' for sep='|', and hit run.
I do know that was fast, however in all honesty, the true value isn’t the coding time, however slightly the interrupted sleep and the way arduous it’s to get your mind working at 2 AM.
This routine acquired me pondering: if the answer is so apparent that I can determine it out simply by glancing on the uncooked textual content, why couldn’t a mannequin do it?
We regularly hear hype about “Agentic AI” changing software program engineers, which, to me, truthfully feels considerably overblown.
However then, the concept of utilizing a small, cost-effective LLM to behave as an on-call junior developer dealing with boring pandas exceptions?
Now that gave the impression of a venture value attempting.
So, I constructed a “Self-Therapeutic” pipeline. Though it isn’t magic, it has efficiently shielded me from at the least three late-night wake-up calls this month.
And personally, something (regardless of how little) that may enhance my sleep well being is certainly a giant win for me.
Right here is the breakdown of how I did it so you possibly can construct it your self.
The Structure: A “Attempt-Heal-Retry” Loop
The core idea of that is comparatively easy. Most knowledge pipelines are fragile as a result of they assume the world is ideal, and when the enter knowledge modifications even barely, they fail.
As a substitute of accepting that crash, I designed my script to catch the exception, seize the “crime scene proof”, which is principally the traceback and the primary few traces of the file, after which go it right down to an LLM.
Fairly neat, proper?
The LLM now acts as a diagnostic device, analyzing the proof to return the right parameters, which the script then makes use of to robotically retry the operation.
To make this technique strong, I relied on three particular instruments:
- Pandas: For the precise knowledge loading (clearly).
- Pydantic: To make sure the LLM returns structured JSON slightly than conversational filler.
- Tenacity: A Python library that makes writing advanced retry logic extremely clear.
Step 1: Defining the “Repair”
The first problem with utilizing Giant Language Fashions for code technology is their tendency to hallucinate. From my expertise, when you ask for a easy parameter, you usually obtain a paragraph of conversational textual content in return.
To cease that, I leveraged structured outputs by way of Pydantic and OpenAI’s API.
This forces the mannequin to finish a strict type, performing as a filter between the messy AI reasoning and our clear Python code.
Right here is the schema I settled on, focusing strictly on the arguments that mostly trigger read_csv to fail:
from pydantic import BaseModel, Area
from typing import Non-compulsory, Literal
# We want a strict schema so the LLM does not simply yap at us.
# I am solely together with the params that really trigger crashes.
class CsvParams(BaseModel):
sep: str = Area(description="The delimiter, e.g. ',' or '|' or ';'")
encoding: str = Area(default="utf-8", description="File encoding")
header: Non-compulsory[int | str] = Area(default="infer", description="Row for col names")
# Generally the C engine chokes on regex separators, so we let the AI swap engines
engine: Literal["python", "c"] = "python"By defining this BaseModel, we’re successfully telling the LLM: “I don’t need a dialog or an evidence. I would like these 4 variables crammed out, and nothing else.”
Step 2: The Healer Perform
This operate is the center of the system, designed to run solely when issues have already gone incorrect.
Getting the immediate proper took some trial and error. And that’s as a result of initially, I solely offered the error message, which pressured the mannequin to guess blindly on the drawback.
I shortly realized that to appropriately establish points like delimiter mismatches, the mannequin wanted to truly “see” a pattern of the uncooked knowledge.
Now right here is the massive catch. You can not really learn the entire file.
When you attempt to go a 2GB CSV into the immediate, you’ll blow up your context window and apparently your pockets.
Luckily, I came upon that simply pulling the primary few traces offers the mannequin simply sufficient data to repair the issue 99% of the time.
import openai
import json
consumer = openai.OpenAI()
def ask_the_doctor(fp, error_trace):
"""
The 'On-Name Agent'. It seems on the file snippet and error,
and suggests new parameters.
"""
print(f"🔥 Crash detected on {fp}. Calling LLM...")
# Hack: Simply seize the primary 4 traces. No must learn 1GB.
# We use errors='change' so we do not crash whereas attempting to repair a crash.
strive:
with open(fp, "r", errors="change") as f:
head = "".be part of([f.readline() for _ in range(4)])
besides Exception:
head = "<>"
# Preserve the immediate easy. No want for advanced "persona" injection.
immediate = f"""
I am attempting to learn a CSV with pandas and it failed.
Error Hint: {error_trace}
Knowledge Snippet (First 4 traces):
---
{head}
---
Return the proper JSON params (sep, encoding, header, engine) to repair this.
"""
# We drive the mannequin to make use of our Pydantic schema
completion = consumer.chat.completions.create(
mannequin="gpt-4o", # gpt-4o-mini can also be advantageous right here and cheaper
messages=[{"role": "user", "content": prompt}],
capabilities=[{
"name": "propose_fix",
"description": "Extracts valid pandas parameters",
"parameters": CsvParams.model_json_schema()
}],
function_call={"identify": "propose_fix"}
)
# Parse the consequence again to a dict
args = json.hundreds(completion.selections[0].message.function_call.arguments)
print(f"💊 Prescribed repair: {args}")
return args I’m kind of glossing over the API setup right here, however you get the concept. It takes the “signs” and prescribes a “tablet” (the arguments).
Step 3: The Retry Loop (The place the Magic Occurs)
Now we have to wire this diagnostic device into our precise knowledge loader.
Previously, I wrote ugly whereas True loops with nested strive/besides blocks that had been a nightmare to learn.
Then I discovered tenacity, which lets you embellish a operate with clear retry logic.
And the very best half is that tenacity additionally lets you outline a customized “callback” that runs between makes an attempt.
That is precisely the place we inject our Healer operate.
import pandas as pd
from tenacity import retry, stop_after_attempt, retry_if_exception_type
# A grimy world dict to retailer the "repair" between retries.
# In an actual class, this might be self.state, however for a script, this works.
fix_state = {}
def apply_fix(retry_state):
# This runs proper after the crash, earlier than the following try
e = retry_state.end result.exception()
fp = retry_state.args[0]
# Ask the LLM for brand spanking new params
suggestion = ask_the_doctor(fp, str(e))
# Replace the state so the following run makes use of the suggestion
fix_state[fp] = suggestion
@retry(
cease=stop_after_attempt(3), # Give it 3 strikes
retry_if_exception_type(Exception), # Catch the whole lot (dangerous, however enjoyable)
before_sleep=apply_fix # <--- That is the hook
)
def tough_loader(fp):
# Verify if now we have a advised repair for this file, in any other case default to comma
params = fix_state.get(fp, {"sep": ","})
print(f"🔄 Making an attempt to load with: {params}")
df = pd.read_csv(fp, **params)
return dfDoes it really work?
To check this, I created a purposefully damaged file referred to as messy_data.csv. I made it pipe-delimited (|) however didn’t inform the script.
After I ran tough_loader('messy_data.csv'), the script crashed, paused for a second whereas it “thought,” after which mounted itself robotically.
It feels surprisingly satisfying to observe the code fail, diagnose itself, and recuperate with none human intervention.
The “Gotchas” (As a result of Nothing is Good)
I don’t need to oversell this resolution, as there are positively dangers concerned.
The Price
First, keep in mind that each time your pipeline breaks, you make an API name.
That is perhaps advantageous for a couple of errors, however when you’ve got a large job processing, let’s say about 100,000 recordsdata, and a nasty deployment causes all of them to interrupt without delay, you would get up to a really nasty shock in your OpenAI invoice.
When you’re working this at scale, I extremely advocate implementing a circuit breaker or switching to a neighborhood mannequin like Llama-3 by way of Ollama to maintain your prices down.
Knowledge Security
Whereas I’m solely sending the primary 4 traces of the file to the LLM, you could be very cautious about what’s in these traces. In case your knowledge accommodates Personally Identifiable Data (PII), you might be successfully sending that delicate knowledge to an exterior API.
When you work in a regulated business like healthcare or finance, please use a neighborhood mannequin.
Significantly.
Don’t ship affected person knowledge to GPT-4 simply to repair a comma error.
The “Boy Who Cried Wolf”
Lastly, there are occasions when knowledge ought to fail.
If a file is empty or corrupt, you don’t need the AI to hallucinate a method to load it anyway, probably filling your DataFrame with rubbish.
Pydantic filters the unhealthy knowledge, but it surely isn’t magic. You must watch out to not disguise actual errors that you simply really need to repair your self.
Conclusion and takeaway
You could possibly argue that utilizing an AI to repair CSVs is overkill, and technically, you is perhaps proper.
However in a area as fast-moving as knowledge science, the very best engineers aren’t those clinging to the strategies they realized 5 years in the past; they’re those always experimenting with new instruments to resolve previous issues.
Actually, this venture was only a reminder to remain versatile.
We will’t simply hold guarding our previous pipelines; now we have to maintain discovering methods to enhance them. On this business, probably the most useful talent isn’t writing code sooner; slightly, it’s having the curiosity to strive an entire new approach of working.
