There is a mystery at the heart of computational neuroscience. It concerns the nature of representation and raises questions from abstract algebra. This post is about the brain, but it has implications for how we think about ML.
You probably know that “neural networks” originally took inspiration from biological networks of actual wet, squishy neurons, but that there are some pretty major differences. One of those differences is pretty fundamental, because it concerns how numbers are represented at all.
Background about Neurons
This is a neuron:
This is a recording of the activity of a neuron. It’s not a picture of the cell, like you would get from a microscope, but this represents how a neuron is seen from the perspective of other neurons. Each vertical line in the graph represents a “spike”, more formally called an action potential, which is a moment when the neuron experiences a burst of electrical activity. Each spike carries information about something, and they are the primary way in which information is represented and manipulated within the brain. For example, the movements to your muscles are conveyed by spikes.
These spikes transmit information between two neurons, occurring here:
Here’s more information about the neuron spikes, to show the timing.
If we could extend this picture to the right and the left, we would see that this neuron has similar activity, basically 24/7 until you die, although the patterns and timings will change. As you can see from the picture, these spikes occur at very specific moments in time; the action potential can last less than a millisecond.
Information, and a Story about a Rabbit
Here’s the thing about these spikes: they contain no other information. Each spike is just as you see it on this graph, a brief moment in time of electrical activity. When neuron A sends a signal to neuron B, the receiver neuron B knows these things:
- An electrical spike arrived
- It came from Neuron A
- Precisely what time the spike arrived
I’ve said that these spikes are the primary way in which the brain transmits and manipulates information. Let me give an example of what that means. Here’s a short story:
A rabbit is eating sorrel when it hears a rustling branch to its right. It turns its head and 200 milliseconds later it sees a cat leaping out from behind a bush. It reacts as fast as it can, releasing the tension in its powerful hind legs.
In this story, the rabbit experiences sensations, such as sounds and sights, and it performs movement, eating and hopping. These are all transmitted by neuronal spikes, just like everything else in the rabbit’s brain. The spikes carry precise timing information that’s inherent in the time that they arrive. An action potential arrives from the ears and it says “I hear something right now”. An action potential goes to the legs and it says “muscles to contract right now”. In a situation like this, a lot of information needs to be processed right now.
There’s also information in the timing information that comes from the relative timing of things, too. If the rabbit survived in our story, then it can remember how long it takes between hearing the cat and seeing the cat. It can learn how long it takes between sending the signal to move and feeling the air rush past its whiskers.
Zero
Here is the basic move in machine learning. First, you convert your data into numbers, for example maybe each pixel is represented by its brightness. Then, you put those numbers into a complicated system of linear algebra where the numbers get multiplied and added together a whole lot. Finally, you take the answer numbers and convert them back into whatever you care about, so you can tell whether the image contains a cat or whatever.
The brain is not exactly a pile of linear algebra, but it’s performing a similar function. The animal’s senses report information about the world, as a series of action potentials, the brain does a lot of parallel computation, and then the resulting action potentials are converted into whatever the animal cares about, like muscle movement.
The action potentials are numbers.
Just like numbers, action potentials are representative of real world values. And just like numbers, they are precise. The precision comes from their inherent timing information.
But unlike numbers, they don’t have a zero.
Action potentials have precise relative timing, but they don’t have absolute timing. There’s no global start time or Unix epoch, the rabbit’s cells don’t remember when it was born.
This means that some mathematical operations are possible, but not others. Two action potentials can be subtracted, to find their timing difference, but they can’t be added, multiplied, or divided. The differences are more like proper numbers, with a clearly defined zero, and they actually can be added, multiplied, and divided. Neurons end up working with difference information quite a lot.
The central problem there is that a difference could arise from any pair of neurons, so a quadratic blowup occurs.
Rate Coding
There’s a traditional answer to this dilemma, which is to posit that neurons use information about the frequency of spiking to approximate a scalar value. The idea is basically that if a neuron is active at 50Hz, then it’s encoding a larger number than if it were active at 10Hz. Unlike timing information, there is a clearly defined zero point here.
As is often the case in biology, both stories are true to some extent. Sometimes the firing rate matters, and sometimes the precise timing matters.
There are reasons to think that precise timing matters, and the most basic is that the timing of our thoughts and actions matters a lot. When a basketball player takes a shot on the hoop, they need to move their muscles in a particular pattern, and they need to time the shot to avoid defenders. All of that can be done with relative timing.
Ending Thoughts
The brain transmits information this way because timing information is so important for organisms that need to take action quickly in response to events in the world.
Data representation is sometimes an afterthought in machine learning, but it’s almost always important to choose a data format that’s a good fit for the domain. And sometimes this choice of data formatting has significant downstream effects on the network. Data representation is generally an important area of inquiry in neuroscience for this reason, and because it’s often difficult to determine. As I noted in Marr’s Levels, neuroscience asks what is, while ML asks what would be best. In this case, we have some insight into why neurons do what they do, and we can bring some of that insight to our models.
If you work in ML, then I hope this leaves you with questions. Is there another way to represent your data? Is there a better way to deal with timing data, especially in robotics or physical domains?