open source a connectome at the controls

A real fruit fly brain plays NES games.

The whole nervous system of an adult fruit fly, 166,700 neurons wired from an electron-microscope map, runs live on one GPU. It sees the game through its motion and object detectors, its descending neurons press the buttons, and you can grab the controller and play along.

View on GitHub Run it yourself Videos & release

The 20-second launch video, recorded from the running app. A 2-minute walkthrough of rewards and senses is on the release page.

166,700neurons, the whole central nervous system (MaleCNS)
25.6 Mconnections, mapped with an electron microscope
20 / sticks: see, think, press, every 50 ms
0ROMs included: you bring games you own
01 how it works

From the screen to the buttons

The brain itself is frozen: no training inside it. A small learner on top rewards what scores and punishes losing a life, the way food and pain teach an animal.

the game
NES emulatorruns in the server, keeps playing with the tab closed
the picture
motion & objectsturned into signals for the fly's own detector neurons
its eyes
T4/T5, LC10a, LPLC213,581 motion detectors and object detectors
the brain
166,700 neuronsa spiking simulation of the whole connectome
its output
descending neuronsthe brain's cables to the body press the pad
on top
reward learnerscore = food, lost life = pain, saved per game
this programthe fly's own wiring
02 the app

Watch it play, see inside, take the controller

The Play page: Galaga on a TV with the fly's ship tracked in green, the NES controller, the fly's eyes and buttons, its whole nervous system firing
Play. The game on a TV; green marks the ship the fly controls, pink the enemies and shots. Below: how strongly each sense drives the brain, and the descending neurons that press the pad. Right: all 166,700 neurons, amber where they fire this tick.
The Brain page: the retina image, voltage into left and right T4/T5 motion detectors, every input set this tick, spikes per cell type
Brain. Exactly what goes into the connectome each tick and the spikes that come back. Nothing on it is simulated for show.
The Rewards page: event weights with live counts and the log of what the fly just felt
Rewards. Food, kills, relief, explore, danger, pain: weights you change live, with what each paid in the last minute.
The NES controller: the left arrow glows amber because the fly's neurons press it
The controller. A button glows amber when the fly's neurons press it, green when you do. Click, touch or use the keys to play along.
The Play page on a phone

On a phone, and more

Every page works on a phone. + Add game puts a ROM you own into the library from the page. Learning is saved per game and resumes next time. Unknown games get sensible controls and the fly works out which sprite it is.

03 results

Scramble the wiring and it stalls

What we found, honestly

  • The wiring matters. The real connectome covered about 6× the ground of a degree-preserving scrambled copy.
  • Learning helps, then plateaus. In Galaga the reward learner gives about 3× the score (p = 0.003).
  • Not everything worked. A game-event reward feed and an n-step TD learner did not beat the simpler defaults; they are left in to experiment with.
  • Best at action games. Shooters, platformers, maze and racing games. Sports, RPGs and puzzles are marked "not suited": the fly needs moving things to react to.
04 run it

Run it, with games you own

Python 3.10+. An NVIDIA GPU runs 4 voting flies per axis at full speed; a CPU runs 1 (slower, still works). The brain (~260 MB) downloads itself on the first run.

git clone https://github.com/arpanguria68-ui/fly-plays-nes.git
cd fly-plays-nes
python -m venv .venv
.venv/Scripts/pip install -r requirements.txt
.venv/Scripts/pip install -r requirements-gpu.txt   # NVIDIA GPU only

# put games you own in roms/ (or use "+ Add game" on the page), then:
.venv/Scripts/python mesen/connect.py              # http://127.0.0.1:8777/

No games are included

This project does not include or link to any ROMs. Use cartridges you own, dumped with a cartridge dumper, or homebrew and public-domain games shared by their authors. Your games stay on your PC.

05 sister project
fly-ears: an approaching disc, the fly's looming neurons filling, the brain map in amber
fly-earsThe same fly brain watches your videos and hears your music, through its own photoreceptors and ears, with a 3D body and honest checks.