Let the AI Do the Experimenting

in a state of affairs the place you could have loads of concepts on the best way to enhance your product, however no time to check all of them? I wager you could have.

What if I instructed you that you simply not must do all of it by yourself, you possibly can delegate it to AI. It might run dozens (and even lots of) of experiments for you, discard concepts that don’t work, and iterate on those that truly transfer the needle.

Sounds wonderful. And that’s precisely the thought behind autoresearch, the place an LLM operates in a loop, repeatedly experimenting, measuring influence, and iterating from there. The method sounded compelling, and plenty of of my colleagues have already seen advantages from it. So I made a decision to strive it out myself.

For this, I picked a sensible analytical activity: marketing budget optimisation with a bunch of constraints. Let’s see whether or not an autonomous loop can attain the identical outcomes as we did.

Background

Let’s begin with some background to set the context. Autoresearch was developed by Andrej Karpathy. As he wrote in his repository:

At some point, frontier AI analysis was achieved by meat computer systems in between consuming, sleeping, having different enjoyable, and synchronizing every so often utilizing sound wave interconnect within the ritual of “group assembly”. That period is lengthy gone. Analysis is now totally the area of autonomous swarms of AI brokers operating throughout compute cluster megastructures within the skies. The brokers declare that we are actually within the 10,205th era of the code base, in any case nobody might inform if that’s proper or mistaken because the “code” is now a self-modifying binary that has grown past human comprehension. This repo is the story of the way it all started. -@karpathy, March 2026.

The concept behind autoresearch is to let an LLM function by itself in an setting the place it may well repeatedly run experiments. It modifications the code, trains the mannequin, evaluates whether or not efficiency improves, after which both retains or discards every change earlier than repeating the loop. Ultimately, you come again and (hopefully) discover a higher mannequin than you began with. Utilizing this method, Andrej was capable of considerably enhance nanochat.

The unique implementation was targeted on optimising an ML mannequin. Nonetheless, simialr method might be utilized to any activity with a transparent goal (from lowering web site load time to minimising errors when scraping with Playwright). Shopify later open-sourced an extension of the unique autoresearch, pi-autoresearch. It builds on pi, a minimal open-source terminal coding harness.

It follows the same loop to the unique autoresearch, with a number of key steps:

• Outline the metric you wish to enhance, together with any constraints.
• Measure the baseline.
• Speculation testing: in every iteration, the agent proposes an concept, writes it down, and exams it. There are three potential outcomes: it doesn’t work (discard), it worsens the metric (discard), or it improves the goal (preserve it and iterate from there).
• Repeat: the loop continues till you cease it, enhancements plateau, or it reaches a predefined iteration restrict.

So the core concept is to outline a transparent goal and let the agent strive daring concepts and study from them. This method can uncover potential enhancements to your KPIs by testing concepts your crew merely by no means had the time to discover. It undoubtedly sounds fascinating, so let’s strive it out.

Process

I wish to take a look at this method on an analytical activity, since in analytical day-to-day duties we frequently have clear targets and have to iterate a number of occasions to succeed in an optimum answer. So, I went by means of all the posts I’ve written for In the direction of Knowledge Science through the years and located a activity round optimising advertising and marketing campaigns, which we mentioned within the article “Linear Optimisations in Product Analytics”.

The duty is sort of frequent. Think about you’re employed as a advertising and marketing analyst and have to plan advertising and marketing actions for the following month. Your objective is to maximise income inside a restricted advertising and marketing price range ($30M).

You could have a set of potential advertising and marketing campaigns, together with projections for every of them. For every marketing campaign, we all know the next:

• nation and advertising and marketing channel,
• marketing_spending — funding required for this exercise,
• income — anticipated income from acquired prospects over the following 12 months (our goal metric).

We even have some extra info, such because the variety of acquired customers and the variety of buyer help contacts. We are going to use these to iterate on the preliminary activity and make it progressively tougher by including further constraints.

It’s helpful to present the agent a baseline method so it has one thing to begin from. So, let’s put it collectively. One easy answer for this optimisation is to concentrate on the top-performing segments by income per greenback spent. We are able to kind all campaigns by this metric and choose those that match throughout the price range. In fact, this method is sort of naive and might undoubtedly be improved, however it offers start line. 

import pandas as pd

df = pd.read_csv('marketing_campaign_estimations.csv', sep='t')

# --- Baseline: grasping by revenue-per-dollar ---
df['revenue_per_spend'] = df.income / df.marketing_spending
df = df.sort_values('revenue_per_spend', ascending=False)
df['spend_cumulative'] = df.marketing_spending.cumsum()
selected_df = df[df.spend_cumulative <= 30_000_000]

total_spend = selected_df.marketing_spending.sum()
revenue_millions = selected_df.income.sum() / 1_000_000

assert total_spend <= 30_000_000, f"Finances violated: {total_spend}"

print(f"METRIC revenue_millions={revenue_millions:.4f}")
print(f"Segments={len(selected_df)} spend={total_spend/1e6:.2f}M")

I put this code in optimise.py within the repository. 

If we run the baseline, we see that the ensuing income is 107.9M USD, whereas the overall spend is 29.2M.

python3 optimise.py
# METRIC revenue_millions=107.9158
# Segments=48 spend=29.23M

Establishing

Earlier than transferring on to the precise experiment, we first want to put in pi_autoresearch. We begin by establishing pi itself by following the directions from pi.dev. Fortunately, it may be put in with a single command, supplying you with a pi coding harness up and operating regionally that you may already use to assist with coding duties.

npm set up -g @mariozechner/pi-coding-agent # set up pi
pi # begin pi
/login  # choose supplier and specify APIKey

Nonetheless, as talked about earlier, our objective is to strive the pi-autoresearch extension on prime of pi, so let’s set up that as nicely.

pi set up https://github.com/davebcn87/pi-autoresearch

I additionally needed some guardrails in place, so I created an autoresearch.config.json file within the root of my repo to outline the utmost variety of iterations. This helps restrict what number of iterations the agent can run and, in flip, retains token prices beneath management throughout experiments. You may as well set a per-API-key spending restrict together with your LLM supplier for even tighter management.

{
  "maxIterations": 30
}

You could find all the main points on configuration in the docs.

That’s it. The setup is finished, and we’re prepared to begin the experiment.

Experiments

Lastly, it’s time to begin utilizing the autoresearch method to determine which advertising and marketing campaigns we must always run. I’m fairly positive our preliminary method isn’t optimum, so let’s see whether or not autoresearch can enhance it. Let the journey start.

I began autoresearch by calling the talent.

/talent:autoresearch-create

After that, autoresearch tries to deduce the optimisation objective, and if it fails, it asks for extra particulars.

In my case, it merely inspected the code we carried out in optimise.py and created an autoresearch.md file summarising the duty. Right here’s what we obtained (a reasonably strong abstract, contemplating it solely noticed our baseline optimisation perform). We are able to see that it clearly outlined the metrics and constraints. I additionally appreciated that it explicitly highlighted that altering the enter knowledge isn’t allowed. That’s guardrail.

# Autoresearch: maximize advertising and marketing marketing campaign income beneath price range

## Goal
Enhance `optimise.py` so it selects a set of marketing campaign segments with **most whole income** whereas respecting the fastened advertising and marketing price range of **30,000,000**. The present implementation is a grasping heuristic: it types by revenue-per-spend, takes a cumulative prefix, and stops as soon as the following merchandise would exceed price range. Meaning it may well go away price range unused and by no means think about cheaper worthwhile gadgets later within the sorted record.

The workload is tiny (62 rows), so higher-quality combinatorial optimization methods are doubtless sensible. We must always favor precise or near-exact choice logic over fragile heuristics when the runtime stays quick.

## Metrics
- **Main**: `revenue_millions` (thousands and thousands, larger is best) - whole chosen income divided by 1,000,000
- **Secondary**:
  - `spend_millions` - whole chosen spend divided by 1,000,000
  - `budget_slack_millions` - unused price range in thousands and thousands
  - `segment_count` - variety of chosen segments

## The right way to Run
`./autoresearch.sh` - runs a fast syntax pre-check, then `optimise.py`, which should emit `METRIC title=quantity` strains.

## Information in Scope
- `optimise.py` - campaign-selection logic and metric output
- `autoresearch.sh` - benchmark harness and pre-checks
- `autoresearch.md` - session reminiscence / findings
- `autoresearch.concepts.md` - backlog for promising deferred concepts

## Off Limits
- `marketing_campaign_estimations.csv` - enter knowledge; don't edit
- Git historical past / department construction exterior the autoresearch workflow

## Constraints
- Should preserve spend `<= 30_000_000`
- Should preserve the script runnable with `python3 optimise.py`
- No dataset modifications
- Hold the answer easy and explainable until further complexity yields materially higher income
- Runtime ought to stay quick sufficient for a lot of autoresearch iterations

## What's Been Tried
- Baseline code types by `income / marketing_spending`, computes cumulative spend, and retains solely the sorted prefix beneath price range.

After defining the duty, it instantly began the loop. It might run for a while, however you continue to retain visibility. You possibly can see each its reasoning and a few key stats within the widget (akin to the present iteration, finest goal worth, and enchancment over the baseline), which is sort of useful.

Because it iterates, it additionally writes an autoresearch.jsonl file with full particulars of every experiment and the ensuing goal metric. This log could be very helpful each for reviewing what has been tried and for the mannequin itself to maintain observe of which hypotheses it has already examined.

In my case, regardless of the configured restrict of 30 iterations, it determined to cease after simply 5. The agent explored a number of completely different methods: precise knapsack optimisation, search-space pruning, and a Pareto-frontier dynamic programming method. Let’s undergo the main points:

• Iteration 1: Reproduced our baseline method. The prefix-greedy technique (income/spend) reached 107.9M, however stopped early when gadgets didn’t match, lacking higher downstream combos. No breakthrough right here, only a sanity verify of the baseline.
• Iteration 2: Precise knapsack solver. The agent switched to a branch-and-bound (0/1 knapsack) method and reached 110.16M income (+2.25M uplift), which is a transparent enchancment. A robust acquire already within the second iteration.
• Iteration 3: Dominance pruning. This iteration tried to shrink the search house by eradicating pairwise dominated segments (i.e., segments worse in each spend and income than one other). Whereas intuitive, this assumption doesn’t maintain within the 0/1 knapsack setting: a “dominating” phase could already be chosen, whereas a “dominated” one can nonetheless be helpful together with others. Because of this, this method failed and dropped to 95.9M income, and was discarded. An excellent instance of trial and error. We examined it, it didn’t work, and we instantly moved on.
• Iteration 4: Dynamic programming frontier. The agent switched to a Pareto-frontier dynamic programming method, however it achieved the identical end result as iteration 2. From an analyst perspective, that is nonetheless helpful. It confirms we’ve doubtless reached the optimum.
• Iteration 5: Integer accounting. This iteration transformed all financial values from floats to integer cents to enhance numerical stability and reproducibility, however once more produced the identical remaining worth. It is smart that the agent stopped there.

So in the long run, the optimum answer was already discovered within the second iteration and it matches the answer we present in my article with linear programming. The agent nonetheless tried a number of different concepts, however stored ending up with the identical end result and finally stopped (as an alternative of burning much more tokens).

Now we will end the analysis by operating the /talent:autoresearch-finalize command, which commits and pushes every little thing to GitHub. Because of this, it created a brand new department with a PR, saving each the modifications to the optimise.py code and the intermediate reasoning information. This manner, we will simply observe what occurred all through the method.

The agent simply solved our preliminary activity. Subsequent, let’s strive making it extra sensible by including extra constraints from the Operations crew. Assume we realised that we additionally want to make sure there are not more than 5K incremental buyer help tickets (so the Ops crew can deal with the load), and that the general buyer contact price stays under 4.2%, since that is one in all our system well being checks. This makes the issue tougher, because it provides further constraints and forces the agent to revisit the answer house and seek for a brand new optimum.

To kick this off, I merely restarted the /talent:autoresearch-create course of, offering the extra constraints.

/talent:autoresearch-create I've extra constraints for our CS contacts to make sure that our Operations
crew can deal with the demand in a wholesome means:
- The variety of extra CS contacts ≤ 5K
- Contact price (CS contacts/customers) ≤ 0.042

This time, it picked up precisely the place we left off. It already had full context from the earlier run, together with every little thing we had achieved thus far. Because of updating the duty, the agent revised the autoresearch.md file to incorporate the brand new constraints.

## Constraints
- Should preserve spend `<= 30_000_000`
- Should preserve extra CS contacts `<= 5_000`
- Should preserve contact price `<= 0.042`
- Should preserve the script runnable with `python3 optimise.py`
- No dataset modifications
- Hold the answer easy and explainable until further complexity yields materially higher income
- Runtime ought to stay quick sufficient for a lot of autoresearch iterations

It ran 8 extra iterations and converged to the next answer (once more matching what we had seen beforehand):

• Income: $109.87M,
• Finances spent: $29.9981M (beneath $30M),
• Buyer help contacts: 3,218 (beneath 5K),
• Contact price: 0.038 (beneath 0.042).

After introducing the brand new constraints, the agent reformulated the issue and switched to an exact MILP solver. It shortly discovered the optimum answer, reaching 109.87M income whereas satisfying all constraints. Many of the later iterations didn’t actually change the end result, they only cleaned issues up: eliminated fallback logic, diminished dependencies, and improved runtime. So, as soon as the issue was well-defined, the agent stopped “looking out” and began “engineering”. What’s much more fascinating is that it knew when to cease optimising and didn’t run all the best way to the 30-iteration restrict.

Lastly, I requested the agent to finalise the analysis. This time, for some cause, /talent:autoresearch-finalize didn’t push all of the modifications, so I needed to manually ask pi to create two PRs: one with clear code modifications, and another with the reasoning and supporting information. You possibly can undergo the PRs if you wish to see extra particulars about what the agent tried.

That’s all for the experiments. We obtained wonderful outcomes and was capable of see the capabilities of autoresearch. So, it’s time to wrap it up.

Abstract

That was a very fascinating experiment. The agent was capable of attain the identical optimum answer we beforehand discovered, utterly by itself. Whereas it didn’t push the end result additional (which isn’t stunning given how well-studied issues like knapsack are), it was spectacular to see how an LLM can iteratively discover options and converge to a strong final result with out handbook steerage.

I imagine this method has robust potential throughout a number of domains (from coaching ML fashions and fixing analytical duties to extra engineering-heavy issues like optimising system efficiency or loading occasions). In lots of groups, we merely don’t have the time to check all potential concepts, or we dismiss a few of them too early. An autonomous loop like this will systematically strive completely different approaches and validate them with precise metrics.

On the identical time, that is undoubtedly not a silver bullet. There shall be circumstances the place the agent finds “optimum” options that aren’t possible in follow, for instance, bettering web site loading pace at the price of breaking consumer expertise. That’s the place human supervision turns into essential: not simply to validate outcomes, however to make sure the answer is smart holistically.

From what I’ve seen, this method works finest when you could have a transparent goal, well-defined constraints, and one thing measurable to optimise. It’s a lot more durable to use it to extra ambiguous issues, like making a product extra user-friendly, the place success is much less clearly outlined.

Total, I’d undoubtedly suggest making an attempt out pi-autoresearch or related instruments by yourself issues. It’s a strong strategy to take a look at concepts you wouldn’t usually have time to discover and see what really works in follow. And there’s one thing nearly magical about your product bettering whilst you sleep.

Disclaimer: I work at Shopify, however this publish is impartial of my work there and displays my private views.