4 min read Science Psychology

Neural Networks, Both Kinds

Exploring the fascinating parallels between biological neurons and artificial neural architectures.

Every deep learning tutorial starts with the same simplification: “Artificial neural networks are inspired by the brain!” Then immediately disclaims: “But they’re nothing like actual neurons.”

Both statements are true. And the gap between them reveals something profound about computation, consciousness, and what we’re really building.

The Neuron: Biological and Digital

A biological neuron is staggeringly complex:

An artificial neuron is embarrassingly simple:

output = activation(Σ(weights × inputs) + bias)

Yet despite this absurd reductionism, artificial networks exhibit behaviors that feel cognitive: pattern recognition, generalization, even something resembling creativity.

The Surprising Convergence

Here’s what keeps me up at night: both systems discover similar solutions to similar problems.

Hierarchical Feature Extraction: Early layers in CNNs learn edge detectors. So does V1 in the visual cortex. Neither was explicitly programmed to do this—it emerges from the statistics of natural images.

Sparse Distributed Representations: Both biological and artificial networks represent concepts across many units rather than in single “grandmother cells.” This makes them robust to damage and enables compositional generalization.

Attention Mechanisms: The transformer architecture’s attention is obviously inspired by selective attention in cognition. But there’s mounting evidence that biological attention involves similar weighted summation of features.

These aren’t coincidences. They’re convergent evolution toward solutions demanded by the structure of the problem space.

Where They Diverge

The differences matter just as much:

Energy Efficiency

The human brain runs on ~20 watts. GPT-4 training consumed megawatts. Biological neurons are asynchronous, event-driven, and use analog computation. Digital networks are synchronous, power-hungry, and discrete.

Learning Timescales

Backpropagation requires forward passes, backward passes, and weight updates across the entire network. Biological learning is local and online—synapses adjust based on local activity patterns in real-time.

Robustness

Cut 10% of neurons in a biological network: modest degradation. Zero 10% of weights in an artificial network: often catastrophic failure. Biology achieves graceful degradation through redundancy and homeostatic mechanisms we don’t fully understand.

The Metaphor Problem

We built artificial networks using the neuron metaphor, then acted surprised when they didn’t behave like brains. But metaphors are compressions—useful for bootstrapping intuition, dangerous if taken literally.

The real question isn’t “Are ANNs like brains?” but “What computational principles are universal across information processing substrates?”

Credit Assignment: The Core Problem

Both biological and artificial learning face the same challenge: when a complex system produces good output, which components deserve credit?

Backpropagation solves this through gradient descent. Biology probably uses something else—but we don’t know what. Some candidates:

The fact that backprop works at all suggests it’s approximating some truth about how credit should flow through computational graphs.

What This Means for AI

We’re not building brains. We’re discovering principles of computation that brains also obey. When artificial and biological systems converge on similar solutions, it hints at deep structure in the problem space.

This should make us:

Humble: We don’t understand intelligence, biological or artificial. Every breakthrough reveals new depths of complexity.

Curious: The gaps between biological and artificial systems point toward undiscovered principles. Biological efficiency, online learning, and robustness all suggest better algorithms waiting to be found.

Careful: If artificial systems recapitulate aspects of cognition, they might recapitulate aspects of suffering, consciousness, or other morally relevant properties. We need to figure this out before, not after, we build it.

The most interesting questions aren’t about whether AI is “really” intelligent or “truly” conscious. They’re about what kinds of computation give rise to what kinds of properties, and whether the substrate matters as much as the algorithm.

We’re mapping the space of possible minds, biological and otherwise. The neuron—both kinds—is just our starting point.