Optimizing 400 tiny networks with agents

How small can a neural network be and still be exactly right? That was the question behind the 2026 NeuroGolf Championship on Kaggle. My team finished 4th, and much of our work was done not by us but by a pipeline of AI agents, each improving one task at a time and sharing what it learned with the others.

This post walks through how that pipeline worked, then looks closely at two of the networks it produced: one that solves a puzzle with two parameters, and one that solves a puzzle with none.

The game

NeuroGolf takes its 400 puzzles from the public training set of ARC-AGI. Each puzzle shows a handful of small colored grids before and after some transformation, and leaves you to work out the rule. Here’s the first one. Look at the three examples, then guess what the last grid becomes:

Your job is to build a network, in the ONNX format, that performs the transformation. The network never sees a picture. Each grid arrives as a one-hot tensor of shape [1, 10, 30, 30]: ten 30×30 planes, one per color, with a 1 wherever that color appears. The network has to produce the output grid in the same form, exactly. Close doesn’t count. A network that gets a single cell wrong on the hidden test set scores zero for that task.

A correct network earns 25 − ln(cost) points, where the cost is its number of parameters plus the bytes of memory its intermediate tensors take up. The input and output tensors are free. Because the score is logarithmic, trimming a few bytes barely matters, but finding a different way to compute the same thing that is ten times smaller gains you more than two points. A network with a cost of 0 gets the maximum, 25.

Multiply that by 400 tasks, each with its own rule to understand, its own design to invent, and its own graph to debug, and it becomes a problem too big to solve by hand. So we built a system to do it.

The pipeline

The system had five parts: agents that improve one task at a time, a registry that remembers every model they produce, a shared catalogue of tricks, a way to check models against the real leaderboard, and a final step that assembles the submission.

This is a simplified picture. Some tasks got extra instructions or approaches we developed by hand, and over the competition we kept experimenting with which prompts, context and tools to give the agents.

One agent, one task

Each agentic loop focuses on a single task, and many loops run in parallel on different tasks. When a loop starts, its prompt includes everything the agent needs so it doesn’t have to rediscover it:

• a description of the task, with a reference implementation of the rule,
• the current best ONNX graph for the task, with a breakdown of where its cost comes from,
• a reference explaining exactly how the scorer counts cost,
• the current catalogue of tricks, which keeps growing as agents add to it.

The agent can evaluate models with a tool, and vote on or add tricks to the catalogue. It also gets nudged depending on what’s holding it back. If its progress has plateaued, it’s told to rethink the algorithm from scratch. If a model fails local checks or the leaderboard, it’s asked to repair it. If evaluation is too slow, it’s asked to make the model faster.

Keeping every model

It’s tempting to keep only the best model for each task, but the best model locally can turn out to be broken. So for every task we kept many versions, each identified by a hash of the model, along with its score, how long it took to evaluate, and its leaderboard status: unknown, correct, or broken.

Every model was validated before it went into the registry. One of those checks looked for convolution bias vectors shorter than the number of output channels, which could make the grader read memory nondeterministically.

Sharing tricks between tasks

A trick an agent discovers on one task often applies to many others, but an agent working on one task has no way of knowing what an agent on another task figured out an hour ago. We fixed that with a catalogue of reusable tricks, which we called idioms.

Agents could add new idioms, vote on existing ones, and comment on how an idiom worked (or didn’t) on their task. Some idioms were also mined semi-automatically. Idioms that were repeatedly judged wrong or misleading got disabled. To keep prompts short, each agent saw only its own task’s comment on an idiom plus a summary of everyone else’s, not the full history.

Here’s one of them:

title: ULP-collision finite-difference selector
description: Inside a terminal fp32 Einsum, use adjacent-magnitude taps [c_hi, -c_lo] and a coordinate code Q chosen so fl(c_hi*Q[r]*Q[h]) and fl(c_lo*Q[r]*Q[h]) are identical for mismatched classes but differ by a positive ULP for matched classes. A singleton dimension of a repeated runtime input can broadcast over the tap axis and keep it coupled to the source coordinates, preventing premature reduction. This realizes an exact thresholded equality selector with two tap parameters instead of a conventional multi-row coordinate table.
summary: 'Exploit fp32 rounding collisions: choose exact-bit Q codes and adjacent-magnitude signed taps so matched coordinate/class products retain a positive ULP while mismatches cancel to zero or nonpositive values; broadcast the tap axis through a singleton dimension of a repeated runtime input so source labels remain coupled until the terminal Einsum. It fits exact equality, adjacency, prefix, validity, and sparse signed routing—typically 2 taps for binary gates, 3–4 for richer relations, with 10-value color or 30-value coordinate Q vectors—and can fold taps into padded/dead Q slots for zero-memory graphs. Exact fp32 bit patterns, calibrated tap magnitudes, anchor factors, and empirically validated operand order are prerequisites: ORT reassociation, pre-summing, scalarizing/removing the tap axis, or premature reduction destroys the ULP barrier; non-separable/asymmetric relations and large multi-operand contractions are unreliable even when the isolated selector is exact. Evidence is strongly positive: 89 measured +2 confirmations versus 8 measured failures, 4 neutral probes, and 4 partial/+1 results; failures chiefly reflect contraction-order fragility or tasks whose relation is not representable by a shared Q collision code.'
votes:
  1:
    comment: A searched pair of scalar color codes, multiplied in gate/copy/output role order and contracted against the cube of the shared adjacent-float tap, implements all four valid Boolean color cases.
    model_md5: 057cd542b172612c9de268f35a18f5bf
    vote: 2
    certainty: 1.0
  3:
    comment: The four collision pairs remained exact after flattening and coordinated power-of-two rescaling; their eight coefficients could therefore double as the terminal color code.
    model_md5: 4fe72afcfa2e4151b9e95a385cd40376
    vote: 2
    certainty: 1.0
  # ... 106 more task votes/comments

It’s a good example of how far the tricks went. It uses the rounding of 32-bit floats as a logic gate: two nearly opposite constants are chosen so that, inside a single Einsum, the products for the wrong cases round to exactly zero or below, while the right cases keep a tiny positive leftover. Across tasks it gathered 89 confirmations against 8 failures, and the failures mostly came from the order of operations changing how the numbers rounded.

Checking against the leaderboard

A model that passes every local example can still fail on the hidden test set, and the only way to find out is to submit it. Submissions are limited, so we checked unknown models in groups.

This works because of how the scoring behaves: a correct model scores exactly what it scored locally, and a broken one scores exactly zero. Submit a group of untested models, and if the total comes out lower than expected, at least one of them is broken. Split the group and try again, and you quickly narrow it down to the culprit.

Picking the final submission

The last step is choosing one model per task. You’d think you’d just take the highest-scoring one each time, but that can produce a submission too slow to finish. Some operations, Einsum especially, need only a few parameters but can take hours to run through all the examples on a single CPU.

So we treated runtime as a budget. We picked the set of models that roughly maximized the total score while keeping their combined local evaluation time within the limit.

What came out

Our final submission scored 8,142.02 locally. Nine tasks hit the perfect score of 25.

The networks ended up remarkably simple. More than half of them, 224 of the 400, are a single ONNX node, and 193 of those single nodes are an Einsum. 349 networks use at least one Einsum. Across the whole submission there are 5,747 nodes. The biggest network, for task 023, has 491 nodes and still scores 18.151.

Two networks up close

Recoloring with two parameters

Task 267 shows a shape in one color and a single marker pixel in another. The output recolors the shape with the marker’s color and erases the marker. Our network does this with one Einsum node and a constant holding two numbers.

An Einsum multiplies tensors together and sums over the axes you choose, all described by a short string of letters. In this one, p is the batch axis, h and w are the rows and columns, k is the input color, and e is the output color. The last operand, pkhw, says which color each pixel has. Summing over k while keeping e decides each pixel’s new color independently, at every position.

Every other copy of the input in the equation uses its own spatial letters, so those get summed away entirely. What’s left of each copy is just how many pixels of color k or color e there are in the whole grid. That turns the problem into a question about color counts: pixels of the shape’s color should become the color that appears exactly once (the marker), the marker itself should become background, and the background should stay background.

The two numbers in T are almost exact opposites, stored as the float32 values 0xC5BCCB9A and 0x45BCCF43. Used along the q and r axes, they act as a selector. With the operands in exactly this order, float32 rounding makes the wrong color pairs cancel to zero or below, while every right one keeps a small positive leftover, which is enough for the output to pick the right color. Rearrange the equation into something mathematically identical and the rounding changes and the trick stops working. Here, the order of the operands is part of the program.

The cost is 2: the scorer counts the elements of the constant, not its bytes, so T costs exactly two parameters. Reusing the input or T inside the equation is free, and since the only node writes straight to the output, there’s no intermediate memory to charge. The score is 25 − ln(2) = 24.307.

Shifting with a pooling layer

Task 053 moves every colored pixel down by one row. Our network is a single LpPool node, with no parameters at all.

LpPool normally summarizes each window of a tensor, the way max pooling does. Here its settings turn it into something else: a way to copy rows. A vertical kernel of size 2 with dilation 3, negative padding, and a row stride of −2 makes output rows 0, 1 and 2 read input rows 2, 0 and 1. In this task the bottom row of the input is always empty, so this is exactly a shift down by one. The horizontal kernel has size 1, so columns and colors are left alone, and every row past the 3×3 grid stays empty and is ignored when the output is read.

Each window contains at most one 1, and with p = 2 the pool just returns that value unchanged. There are no constants and the node writes straight to the output, so the cost is 0 and the score is the maximum, 25.

Looking back

Networks like these are hard to arrive at directly. They come from many rounds of trying to make a graph smaller and keeping what works. Running those rounds across 400 tasks at once depended less on any single agent than on the structure around them: context that told each agent what was already known, a registry that never threw a working model away, a shared catalogue so discoveries spread, a cheap way to catch broken models, and a final step that respected the runtime limit.

This post is adapted from our team’s writeup on Kaggle.