Manually tweaking prompts and agent logic is a slow, trial-and-error process that wastes developer time. Instead of guessing which change improves performance, what if your agent could optimize itself?
The Problem with Manual Agent Tuning
Most developers fall into the trap of “prompt engineering hell,” where they make small changes to instructions or tools and hope for a better result. This manual approach lacks scientific rigor and makes it nearly impossible to track whether a change actually improved the system or just shifted the error elsewhere.
The Solution: An Automated Experiment Loop
Inspired by machine learning optimization, the Autoresearch for Agents repository introduces a closed-loop system designed to automate this process. Instead of optimizing model hyperparameters in a training script, we optimize the agent implementation itself.
| Feature | Karpathy’s Autoresearch (ML) | Autoresearch for Agents |
|---|---|---|
| Target | train.py (Model code) | agent.py (Agent logic) |
| Metric | val_bpb (Lower is better) | Eval score (Higher is better) |
| Observability | Training logs | LangSmith traces |
How the Loop Functions
The system uses a coding agent to autonomously iterate through an experimentation cycle. The process follows these steps:
1. Read agent.py + results so far 2. Propose a change (prompt, tools, architecture) 3. Edit agent.py 4. git commit 5. Run evaluation: python run_eval.py 6. Parse scores from eval output 7. If improved -> keep commit; If worse -> git reset 8. Log result to results.tsv 9. Repeat forever
Implementation: Bring Your Own Everything
This framework is designed as a template, meaning you must customize the core components before starting the loop. You provide your own agent implementation (any Python code), an evaluation harness, and a dataset.
- agent.py: Your custom logic or framework (LangGraph, OpenAI SDK, etc.).
- run_eval.py: The evaluation script that scores the agent.
- dataset.json: A JSON array of inputs and expected outputs.
- program.md: Instructions for the AI coding agent to follow.
For example, your agent function simply needs to accept an input and return a dictionary that can be scored:
def run_agent(question: str) -> dict:
# Your custom logic here — RAG, multi-step, tool use, whatever
return {"response": answer, "tools_used": [...]}Observability and Results
By integrating with LangSmith, every single experiment is fully observable. You can inspect the traces to see exactly why a specific code change led to better or worse performance. This allows you to move beyond prompting and master autonomous iteration by treating agent development as a measurable, scientific process.
Ready to automate your agent’s evolution? Clone the repository and start building your first evaluation loop today!
