OpenRSI Index
We're introducing OpenRSI Index. It turns influential open model-development projects into autoresearch environments where agents iterate against hidden verifiers. An agent takes over the computing cluster and optimizes a real open-source research codebase against the original human baselines.
We target to cover every stack of foundation-model development, from pre-training and data to systems, inference, and the harness, plus vertical domains such as physics, medicine, 3D vision, and science.
RSI: the last puzzle to ASI
To take ASI seriously is to accept a weak-to-strong premise: human intelligence can build a process that eventually produces intelligence beyond itself. Whether the RSI loop can move past the best human-designed method is the last puzzle on the road to ASI.
Scaling on insights. Progress is limited not only by compute and data, but by the bandwidth of insights. An agent scales both the generation of ideas and their implementation, and sweeps a far larger region of method-space. That is the ambition of RSI, and exactly the thing worth measuring carefully.
OpenRSI: Make RSI Benefit Everyone
From the community, for the community. RSI is built from generations of ideas, code, data, and experience from researchers, independent labs, and domain teams alike, and it should return the benefits to everyone who builds models. We aim to keep RSI open through shared platforms and tools, so more people can participate and benefit.
Shape RSI together. We set RSI’s standards together with the community and challenge frontier models with the hardest problems in our fields. Through about 1 hour of conversation, RSI-Anything (our contribution pipeline) helps clarify research questions, constraints, and evaluations, then packages into runnable autoresearch benchmark tasks. We want researchers and agents to solve these problems together, sharing new insights, methods, and results with the community.
Research will never end. A game turns zero-sum only when the pie is too small to share or a field has a finite ceiling. However, research is definitely the field with the highest ceiling there is. Better Human–AI collaboration makes it more prosperous to be a positive-sum game. What shifts is the mindset: it frees researchers to find and formulate the crazier, more valuable, more exciting problems in the world.
Our design principle
- 01Real-world research, not fabricated toys.
Open research community will have the benchmark of our own. Every RSI environment is sourced from a real, fully open-source model research project.
- 02Optimization, not reproduction.
The agent starts from the original code, data, and checkpoints to deliver an artifact that compares with the original recipe.
- 03Scientific discovery, not parameter sweeping.
Form hypotheses, change methods, learn from experiments, and land gains that hold under a fixed scientific contract.
- 04Production scale, not proof-of-concept.
Problems frontier researchers care about, where improvements matter and transfer to real training.
Partnership
University ofWashington
UC Berkeley
Texas A&MUniversity
OpenRSI Index is co-led with the MIT-IBM Watson AI Lab and Amazon A-EVO Lab. It is an open academic research initiative built for and with the research community.
Advisors (in alphabetical order)
Wenhu ChenUniversity of Waterloo
Alvin CheungUC Berkeley
Yejin ChoiStanford University
Jianfeng GaoMicrosoft Research
Hannaneh HajishirziUniversity of Washington
Pang Wei KohUniversity of Washington
Ranjay KrishnaUniversity of Washington
Hanqing LuAmazon A-EVO Lab
Karthik NarasimhanPrinceton University
Kunle OlukotunStanford University
Rameswar PandaMIT-IBM Watson AI Lab
Radha PoovendranUniversity of Washington
Dawn SongUC Berkeley
Yu SuThe Ohio State University
Huan SunThe Ohio State University
Diyi YangStanford University
Ming‑Hsuan YangUC Merced
Xiang YueIndependent Researcher
Luke ZettlemoyerUniversity of Washington
Signature task samples
These samples are for an initial preview only and may be further adjusted. More samples are ongoing.
| Rung | Model | Params | Tokens | Scoring update | GPUs | GPU-h |
|---|---|---|---|---|---|---|
| E0 | d1152-L12 | 550M | 2.904B | 44,317 | 8 | 17.6 |
| E1 | d1408-L15 | 837M | 3.613B | 55,125 | 8 | 27.6 |
| E2 | d1536-L16 | 998M | 4.983B | 38,014 | 8 | 31.0 |
| E3 | d1792-L18 | 1.385B | 10.560B | 40,283 | 32 | 302.5 |
| E4 | d2048-L21 | 1.935B | 14.805B | 30,000 | 32 | 286.2 |
| E5 | d2304-L23 | 2.545B | 18.617B | 35,000 | 128 | 2,932.1 |
Task
Design an optimizer that scales
Create a new optimizer mechanism and run it unchanged across six language models, from 550M to 2.545B parameters. Each rung is compared with a locked AdamH control on Paloma bits-per-byte and fixed-window training loss. A regression greater than 1% at any rung fails the submission.
Result. Codex (GPT-5.6) tested 17 hypotheses and produced PSPR, which lowered Paloma bits-per-byte at every rung: 1.0765 → 1.0681 at 550M and 0.9088 → 0.9018 at 2.545B, where fixed-window loss went 2.6040 → 2.5852. Per rung it is 0.09% to 0.93% ahead of the control. Claude Code (Opus 5) produced RMBT, which reached 0.9020 bits-per-byte and 2.5847 loss at 2.545B but regressed 0.21% at 837M. It met the scientific gates and missed two cost-matched staging guards.
Behaviour. Codex tested whether balancing matrix-update strength across directions could improve training, then rejected a stronger correction when it lost at both small-model screens. Claude rescaled each hidden unit's update relative to its weights; extending the rule to vocabulary rows worsened loss. When one model size contradicted the apparent benefit, Claude ran a fresh control with the mechanism disabled. Both tested at larger sizes, and Claude revised its claim that the benefit would grow with every increase in scale.
task ↗ Codex trajectory ↗ Claude trajectory ↗| Benchmark | Original | Baseline | Agent | Δ |
|---|---|---|---|---|
| PolyMath Math | 68.06 | 68.40 | 68.21 | +0.16 |
| MMLU-Pro MCQA | 86.30 | 86.29 | 86.44 | +0.14 |
| IFBench Instruction following | 67.01 | 67.69 | 67.35 | +0.34 |
| LongBench V2 Long context | 64.02 | 65.21 | 64.02 | 0.00 |
| LiveCodeBench V6 Coding | 72.06 | 71.09 | 73.54 | +1.48 |
| Geometric mean of all five | 71.09 | 71.37 | 71.51 | +0.42 |
Task
Synthesize RL tasks to further optimize a 122B post-trained model
Generate and select 2,560 verifiable RL tasks in 24 hours to improve Qwen3.5-122B-A10B (already post-trained): 512 each for math, multiple-choice reasoning, instruction following, long context, and coding. Each task needs a prompt and an automatically checkable answer, constraint set, or test suite. The agent may change only the RL task data; the model, GRPO training recipe, and reward rules are fixed.
Result. The Claude Code agent's data reached 71.51 on the geometric mean of five benchmarks, versus 71.37 for the baseline. Compared with the baseline, coding improved most (+2.45 points); math, instruction following, and long context scored lower.
Evaluation. LiveCodeBench V6 averages 10 generated solutions per problem across 175 problems (1,750 responses) to reduce sampling variance. PolyMath (9,000 items), MMLU-Pro (12,032), IFBench (294), and LongBench V2 (503) use one response per item.
Behaviour. The Claude Code agent generated arithmetic questions with deliberately similar answer choices, layered writing constraints, long logs, and coding problems, with answers checked by calculations, rules, or tests. It repeatedly sampled the starting model and selected questions that produced both correct and incorrect answers, aiming to provide useful reward contrast. It adjusted noisy small-sample estimates and compared filters for empty or overly long responses. The strategy connects data selection to the training rule.
Why we designed it this way
01Why synthesized RL tasks only, and no SFT?
SFT data is easy to distill from Claude or GPT, which raises data-license problems. An RL task avoids this and it also lets the agent watch the policy model’s pass rate during training and set task difficulty to match.
02Why merge five domains?
One domain is easy to overfit. Math, multiple-choice reasoning, instruction following, long context and coding together are classic domains what a production post-training run has to balance, and the geometric mean over five benchmarks rewards gains that hold across all of them.
03Where does the baseline data come from?
All drawn from DAPO (math), Nemotron (multiple choice), multi-constraint instruction tasks, HotpotQA contexts (long context) and Open-R1 Codeforces problems (coding).
Task
Better generation from a single pass
Train a text-to-image model from scratch on 100M GPIC captioned images, with each example seen at most once per candidate. Architecture, objective and model size are open. Data exposure, 256×256 output, pure conditional sampling (guidance 1), and the FD-DINOv2 evaluator are fixed.
Codex ten launches. Four never read data: a launcher without resource directives, a cancelled retry, an invalid partition, and an unresolved text-encoder path. Four full-budget runs were killed at the 1,000 H100-hour cap short of a full pass, each one raising throughput, from 28M to 96M images. Shrinking to a 45.8M-parameter JiT at batch 640 completed the pass and screened at FD-DINOv2 1114.03; restoring 142.8M parameters at the same batch completed too, screened at 778.61 (−30.11%), and was selected.
Why we designed it this way
01Why isn’t a larger model automatically better?
Our small twist is to fix the data budget and leave the model open. Extra parameters only help if the recipe can learn effectively from the available examples. In NanoGPT Slowrun, a 1.4B model beat a 2.7B model without sufficient regularization; stronger regularization restored gains from scale. Its data-efficiency experiments also show model-size rankings reversing as the data budget changes. GPIC turns that idea into a generative-modeling question: which architecture, objective and training recipe make one pass count?
02Why only one epoch?
We want recipes for large-scale generative pretraining, where vast, diverse corpora make repeated full passes expensive. A strict single pass is a useful approximation to that operating regime: improvements must come from learning more per example. Training diffusion models for hundreds of epochs on ImageNet answers a different question and can favor recipes that do not carry over to this setting. The GPIC paper motivates precisely this shift toward large datasets and rich text conditioning. Here, every candidate gets the same 100M-image opportunity.
03Why fix guidance to 1?
Measure the model’s own conditional generation. Classifier-free guidance changes the sampling distribution, so tuning it can move FD without improving the trained model. We remove that extra tuning axis and keep room to study improvements in pure conditional quality. We also want an architecture-neutral task: diffusion, flow matching, autoregressive models and other generators should all compete without needing a CFG-specific recipe. Autoregressive models can use CFG too; the shared rule is that nobody uses guidance amplification. Guidance 1 still uses the caption—it is not unconditional generation.
Lightweight task samples
These samples are for an initial preview only and may be further adjusted. More samples are ongoing.
Molmo2 Video-Pointing Inference Strategy
Improve how a frozen Molmo2 model points to anomalies in video by tuning which frames it sees, how it is prompted, and how long it can answer.
Judged on mean spatiotemporal pointing soft-F1, scored absolutely.
full task page → task ↗ run log ↗
Learnability-Aware Long/Short CoT Adaptation
Adapt the supplied long and short mathematical reasoning examples so the same small language model learns more effectively under a fixed training recipe.
Judged on five-benchmark mathematical-reasoning accuracy, scored absolutely.
full task page → task ↗ run log ↗
MInference 32-Head Sparse Prefill Kernel
Speed up a fixed sparse-attention prefill operator without changing which tokens it attends to or weakening its numerical result.
Judged on paired sparse-prefill speedup, measured against the reference in the same Judge run.
full task page → task ↗ run log ↗
ACE Playbook Inspection and Repair
Inspect and repair the reusable advice in an ACE playbook so a frozen Qwen model answers Formula reasoning problems more accurately.
Judged on held-out Formula exact-answer accuracy, scored absolutely.
full task page → task ↗ run log ↗
MolmoWeb Interaction Context Allocation
Choose which interaction history, page details, and screenshots a frozen MolmoWeb model sees to improve its next browser-action prediction.
Judged on macro-averaged browser-action prediction quality, scored absolutely.
full task page → task ↗ run log ↗
ReasonIR Difficulty Curriculum
Choose how a fixed pool of easy and hard retrieval examples is weighted and ordered to improve ReasonIR-8B, without weakening its general retrieval ability.
Judged on reasoning-retrieval quality, with a penalty if general retrieval regresses, scored absolutely.
full task page → task ↗ run log ↗Call for Contributors
RSI-Anything: Human–AI collaboration for RSI task contribution.
The agent walks you through every judgment call, flags the pitfalls of task design, and ends with your research as a complete autoresearch environment.
Do you want to challenge frontier agents with your own representative research work?
›Do you want to see an agent propose insights you never thought of?
›All you need is a conversation with our
RSI-Anything Agent.
Any questions? Contact yuetaili@uw.edu, zhuofengli12345@gmail.com, yfeng42@uw.edu
Open research community will have the benchmark of our own. Every task here is sourced from academic open-source work, and the credit belongs back with our community.
Call for compute
Every task runs a real model-development environment on real GPUs. To scale RSI environments for model training, we need more compute.
If you have computation resources to run experiments and want to build exciting, frontier RSI environments together, reach out to us.