floatboat
Open navigation menu

Autoresearch

Autoresearch lets a coding agent propose and measure changes to a small language model. Establish a baseline on one GPU, run a candidate under the same budget, and review the result against the experiment ledger.

Compare One Autoresearch Candidate With Its Baseline

Define a single training change and record the setup before asking the agent to run the experiment.

Give the repository path, prepared GPU environment, baseline commit, one hypothesis, and the approved trial limit.

Copy the request into a new Floatboat conversation and provide the required material.

Download Floatboat

Your request for Floatboat

Preview and copy the request. It is not sent automatically from this page.

Reference: Autoresearch
https://github.com/karpathy/autoresearch

Read the original documentation and check the current environment and dependencies. Propose any new installation or tool connection before proceeding.

My task:
Read the official Autoresearch program.md. Check my GPU, prepared data, and dedicated branch. Show the baseline and one train.py candidate plan. After I approve the commands, run only the agreed trials and compare val_bpb, peak memory, and status in the local ledger.

Expected deliverable:
A run log, a results.tsv entry, and a baseline comparison with a keep, discard, or crash decision.

About Autoresearch

Andrej Karpathy’s Autoresearch is a small experiment loop for a coding agent such as Codex or Claude Code with access to a training environment. The human sets the scope in program.md. The default loop edits train.py while leaving data preparation and evaluation in prepare.py fixed.

The default comparison uses a five-minute training budget, excluding startup and compilation. Review validation bits per byte (val_bpb) alongside peak memory on the same GPU and setup. A lower score is an improvement on that run’s validation metric.

The experiment instructions use a dedicated Git branch and a local results.tsv ledger. Record the baseline, change, score, memory, and whether the candidate was kept, discarded, or crashed. The README declares MIT for the project.

What the Experiment Loop Records

Keep the setup and evaluation fixed long enough to judge one change. Review the baseline and candidate side by side before deciding whether a lower score is worth carrying forward.

  • Fixed comparison budget

    Run baseline and candidate on the same setup with the same default training budget and evaluation procedure.

  • One editable training file

    Limit the candidate change to train.py and keep the preparation and evaluation code fixed during the comparison.

  • A reviewable run ledger

    Use the log and results.tsv to retain the candidate, val_bpb, peak memory, and disposition.

Autoresearch Use Cases for Single-GPU Experiments

The same loop supports a first baseline, a targeted change, or a failure investigation. Each run should leave a log and a clear decision about what to keep.

Establish a repeatable baseline

Start from the recorded upstream commit and prepared data, then run the unchanged training script. Save the GPU model, software environment, val_bpb, and memory so the next candidate has a comparable reference.

Check one optimizer change

Give the agent a learning-rate or optimizer hypothesis and a limit of one candidate run. Compare against the baseline on the same hardware and capture the diff and outcome in the ledger.

Investigate a memory failure

If training runs out of memory, preserve the error log and label the trial as a crash. Review the proposed change and resource limit before authorizing a revised candidate.

Hand off a promising candidate

Give a reviewer the baseline and candidate commits, raw logs, and unchanged evaluation setup. Ask them to confirm the comparison and rerun a promising result before extending the experiment.

What Autoresearch Needs to Run

Check the training environment before granting the coding agent access. A short baseline run is the first useful check, with a separate limit for how many trials the agent may start.

  • GPU and Python environment

    The upstream quick start uses one NVIDIA GPU, Python 3.10 or newer, and uv. The README names H100 as the tested GPU.

  • Prepared training data

    Review and authorize dependency installation and data downloads. Use uv run prepare.py for data and tokenizer setup, then uv run train.py to check the baseline.

  • Bounded branch and run budget

    Give the agent a dedicated branch, a permitted file scope, and a maximum number of trials. Keep normal permissions and review each proposed expansion.

Related Software Projects

Compare the workflows, operating requirements, and source licenses of these projects.

Autoresearch FAQ

Which file does the default agent loop edit?
The upstream instructions constrain changes to train.py. Keep prepare.py fixed so the data and evaluation stay comparable.
What does val_bpb measure?
Validation bits per byte measures model loss on the validation setup. The upstream comparison favors a lower value.
Can I compare runs from different GPUs?
Use the same GPU and training setup for a controlled comparison. A fixed wall-clock budget can yield different amounts of work on different hardware.
Where does the run history live?
Program instructions call for the run log and a local results.tsv with the commit, val_bpb, peak memory, outcome, and short description.
What happens if a candidate fails?
Retain the failure evidence and mark the trial as a crash or discard according to the result. Compare only valid runs against the baseline.
Can I run the default setup on a Mac?
The official setup requires a single NVIDIA GPU. The README lists community ports for other platforms and documents their separate setup.