Natural Scenes Dataset
People viewed thousands of natural images while fMRI measured activity across the visual cortex.
- 9k
- training images
- 1k
- held-out images
Microsoft Research · visual neuroscience · July 2026
Can an AI discover how the visual cortex computes?
We let agents write fixed visual models from scratch. Held-out neural predictivity decides which ideas survive.
Stimulus
Agent proposal
def f(x):
z = vision(x)
return z
z = vision(x)
return z
Neural prediction
- fixed
- visual functions
- held out
- neural reward
- parallel
- independent lineages
- none
- gradient training
The question
An encoding model is a hypothesis about computation.
Visual neuroscientists often start with a network trained for another task, then ask how well its features predict neural activity. We reverse that process. The neural data define the objective, and agents search directly for the function.
f(X)fixed visual model
→
Zrepresentation
→
ŷneural predictionOne input, two measurement scales
Natural scenes connect human fMRI and macaque electrophysiology.
Triple-N
Five macaques viewed 1,000 NSD images while dense probes recorded single-neuron responses.
- 5
- macaques
- 30k+
- responsive units
Why search architecture?
Training helps, but architecture already matters.
Even random CNN features predict more neural variance than a random projection. ImageNet training improves them further. This leaves a precise question: can search find better fixed computations without training?
Noise-normalized explained variance, averaged across each dataset’s search cohort.
Human NSD
Random projection0.10
Untrained CNN0.26
Trained CNN0.46
Macaque Triple-N
Random projection0.06
Untrained CNN0.11
Trained CNN0.29
The method
The agent edits one file. The evaluator returns one number.
Each proposal is executable code for a fixed function, f(X). The harness freezes everything else: the dataset, readout, held-out split, and scoring rule.
The model never reads neural targets and never updates weights. It can only improve by proposing a better computation.
- 01
Propose
An agent writes or revises
model.py. - 02
Score
A fixed evaluator fits the same linear readout and measures held-out predictivity.
- 03
Revise
The score returns as feedback. The agent keeps, changes, or replaces the idea.
def model(x):
gabor = quadrature_energy(x)
complex_cells = local_max(gabor)
colour = opponent_energy(x)
parts = part_gram(complex_cells)
return pool(gabor, colour, parts)Fixed constraints
Every proposal is inspectable- no checkpoint loads
- no network fetches
- no gradient training
- no neural-target leakage
The search
Many lineages explore in parallel. One honest evaluator ranks them.
Independent search lineages evolve separate model families. A conductor shares broad directions without forcing every lineage toward the current winner, preserving the diversity needed to discover different mechanisms.
Conductorperiodic advisory directions
ABC···N
Asynchronous queuefixed evaluatorsame held-out reward
Separate lineages preserve competing ideas.
No lineage is required to copy the current best model.
Every proposal faces the same held-out test.
The evaluator, data split, and readout remain fixed.
Search turns feedback into new computations.
Agents can combine, revise, or discard visual mechanisms.
The result
Search discovers useful computation without training.
The search converged on an HMAX-like energy model built from familiar cortical motifs: oriented filters, phase-invariant complex cells, normalization, colour opponency, local pooling, and part–whole statistics.
NSD · July 17 evaluation
Architecture explains early cortex especially well.
The searched model exceeds the best pretrained reference in V1, V2, V3, and hV4. Pretraining leads in the ventral aggregate. The same regional pattern remains visible in people held out from model search.
Searched, never trainedResNet-50, ViT-B/16, DINOv2, and CLIPPatterned: untrained · solid: pretrained
Quadrature energy
Pairs of Gabor filters preserve oriented structure while discarding local phase.
Complex-cell pooling
Local max pooling adds tolerance to small changes in position and scale.
Part–whole statistics
Co-occurrence features summarize which local parts appear together.
10,601 reliable voxels · NSD subject 1
A computational gradient
The fixed model explains early visual cortex best.
The searched model’s predictivity declines along the cortical hierarchy. It leads the strongest pretrained reference through hV4, but not in the ventral aggregate.
Generalization
The regional result survives held-out brains.
Across NSD and Triple-N, the latest evaluation separates people or macaques used during search from held-out subjects. TVSD repeats the test in a second macaque spiking dataset.
Three datasets · two species
Each dataset is evaluated on its own scale.
NSD and Triple-N report noise-normalized explained variance. TVSD reports mean Pearson correlation. Values should be compared within a dataset, never across datasets.
Searched model
ROISearchHeld outV10.6520.583Ventral0.3470.325
Best pretrained references: ResNet-50 in V1; DINOv2 or CLIP in ventral cortex.
Searched model
ROISearchHeld outEVC0.4420.422IT0.2330.300
EVC contains one macaque per cohort. The best pretrained reference leads in IT.
Searched model
ROIMonkey FMonkey NV10.7900.826IT0.5920.604
Monkey F is the search animal. Monkey N is held out.
What comes next
Search over functions turns model building into an empirical science.
The result is not the agent system. It is the discovered function: executable, testable, and interpretable as a hypothesis about cortical computation. We are now testing the same idea on language fMRI and intracranial recordings. Those results are preliminary and mixed, so they define a direction rather than a conclusion.