Thank you to Novium, the team behind the hoverpen, for supporting PBS
2025 is … was the international year of quantum science and technology. In celebration of
the invention of our strangest true theory 100 years ago.
In 1925, quantum mechanics went from being a peculiar set of ideas
to describe some funny results from experiments,
to a full-blown theoretical framework that overturned how we think reality really works.
So today, as the centenary year approaches its end I want to take you on a little journey through
subscribe, hit the bell, and introduce yourself in the comments. We’re friendly.
at the end of the universe. There’s a link in the description. Now onto the episode
Let’s start by understanding where we stood at the beginning of 1925.
What was the scientific worldview that was about to be shattered?
So Albert Einstein has been the king of physics for a decade, ever since he published his general
theory of relativity in 1915 and toppled the four-century reign of Newtonian mechanics. GR
completely supplanted our understanding of space, time, motion and gravity. This was a revolution,
no doubt, but in a way Einstein actually reinforced the status quo rather than
overturning it. See in the Newtonian worldview, the motion of all particles could be perfectly
computed through a simple set of universal laws. And all things are made of particles, so, surely,
the universe itself and all it contains is computable. It’s all deterministic. Know the
present and you can calculate all past and all future states. Newton’s achievement gave
us a sense that the universe is knowable to a degree of completeness never before imagined.
There were some few inconsistencies remaining—the fact that
Mercury’s orbit didn’t quite obey Newton, some stuff about electromagnetism and the speed of
light. But the universe has to be axiomatically self-consistent. And so Einstein was able to
leverage these seemingly minor glitches into a full-blown scientific revolution,
in which space and time were merged and the resulting spacetime was no longer
Newton’s static, universal background, but rather a malleable, dynamic thing.
But the new picture that came out of Einstein’s relativity still shared a fundamental quality with
Newton’s worldview. It was just as deterministic. The equations of general relativity suffer no
uncertainty. The distribution of matter and energy in spacetime perfectly defines the
geometry of spacetime, which in turn completely determines the motion of said matter and energy,
forever into the past and future. Einstein also showed that simultaneity is relative,
destroying our delusion of an absolute sense of “now”. So the already-deterministic past and
future became as real as the present, at least in some interpretations,
and all of time was crystalized into an eternal block for the entirety of existence.
On the one hand the universe under Einstein remained measurable and knowable,
but what is measured and what can be known came to depend on perspective;
where you are, how fast you’re moving, the lightspeed-limited horizon of your embedded
vision. A hell of a perspective shift, but still firmly on the side of determinism.
In the early 1920s Einstein was a science rockstar. There were still a few loose
threads in the fabric of our understanding of the world. So you can imagine the hunger of a young
physicist of the era to pull at these threads and so, perhaps, pull off another revolution.
Now a big outstanding mystery, perhaps THE big one was the strange world of the atom,
and especially of the behavior of the humble electron. From Rutherford we knew
the rudimentary atomic nucleus. You’ve all seen depictions of electrons whizzing around the nuclei
like tiny solar systems—a holdover from early assumptions that the tiny universe resembles
the gigantic. But there were so many questions! Why were only certain electron orbits allowed,
and why those ones? Why didn’t electrons spiral into their nuclei? The uncanny orbit of the
electron was surely the key to a new revolution, just as the orbit of Mercury unlocked relativity.
First steps were tentative. Neils Bohr, building on Planck and Einstein’s quantization of light,
gave us an empirical model of quantized electron energy levels. Louis de Broglie,
building on that same quantization, guessed that all matter has a wave nature. Bohr then
realised that electron waves made sense of his atomic model. The allowed orbits are those that
perfectly fit an integer number of electron wave cycles—these are self-reinforcing, standing waves.
While this gave a nice story for the quantization of electron orbit, other problems remained,
perhaps the worst being that it only worked for the humble hydrogen atom.
So that’s where we stood going into 1925. Bohr’s model was still very classical-feeling,
and so the Newtonian and Einsteinian worldviews still reigned. We clung to the ambition to be
aloof masters of a computable universe. The spark that set this entire worldview ablaze
had already started—in Munich a few years earlier when a 20 year old student showed up
in the office of Arnold Sommerfeld. Sommerfeld, famous both for his physics and his mentorship,
handed the student a trial problem—to solve one of the many failings of the Bohr model,
in particular the distortions in hydrogen energy levels seen when
the stuff was in strong magnetic fields called the anomalous Zeeman effect.
So the thing about being a kid is that you’ll try anything. You don’t yet know what you’re
supposed to take as ground truth. All common sense dictated that the math should be based on integer
numbers—the whole numbers of wavecycles in Bohr’s electron orbits. But Sommerfeld’s naive student
for whatever reason tried a mathematical form involving physically irrelevant half-integers. But
that worked. The student was Werner Heisenberg, and at 20 he’d discovered the first clue to the
strange phase symmetry of the electron, a result of its yet undiscovered quantum spin.
This little project began Heisenberg’s obsession with the mysteries of the atom. Three years later,
now 1925, Heisenberg was in Gottingen studying under Max Born, but also in
close connection with the now very famous Neils Bohr, his future closest collaborator.
Surrounded by such giants of physics, Heisenberg still turned to the wisdom of Einstein. He asked
himself, what would Einstein do? In coming up with general relativity, Einstein resolved to reject
unfounded assumptions, no matter how self-evident they seemed. That meant putting aside ideas like
the universality of time and an unchanging and flat geometry of space. So Heisenberg
challenged himself to take this credo seriously and push it as far as possible. In his own words,
“The aim of quantum theory should be to describe only quantities which are observable.” Make no
assumptions about, for example, what the electrons are really doing inside the atom; instead,
build a theory entirely around the results of what we see when we do measurements on the atom.
In a cartoon view of an atom you can imagine knowing things like where the electron is or
how fast it’s moving at any point in time. In the Bohr model that translates to a radius and
energy-slash-momentum for each orbit—a pair of concrete values, each with an index to connect
them to a list of possible electron states. But Heisenberg realized that it’s not possible to
actually observe these orbits of the electrons within the atom. The only thing we can measure
is the light—the photon—that shoots out when an electron changes orbit. In fact,
even the idea of orbits is presumptuous—so we will call them electron states.
These invisible electron states couldn’t be the main players in the theory. Instead,
he committed to finding a law of motion that only describes the observables—the frequency
and intensity-slash-amplitude of the photon produced when electrons change states. The
difference seems subtle but it really changes everything. Instead of a formula to describe
electron properties for a single list of states, Heisenberg sought a formula to describe the
resulting photon properties for every combination of initial and final state of the electron.
This sort of two-index list of properties is known as a matrix, although Heisenberg didn’t know it at
the time. His job was to guess a formula relating these matrices, and so he figured out the algebra
of matrices on his own. He was disconcerted to discover something that mathematicians
already knew—that matrix algebra defies standard multiplications rules. For example,
X*Y is not always the same as Y*X. They are not commutable. This turns out to be a central feature
of quantum mechanics and ultimately led Heisenberg not only to his solution, but to the thing he’s
most famous for—his uncertainty principle. But for now, this unintuitive type of math was just
another challenge to Heisenberg’s resolution to reject prior assumptions, and so he forged ahead.
Now his theory needed to do a couple of things: to predict the photon frequencies
and intensities—the “spectra” of atoms. It also needed to be consistent with certain
known fundamentals of the universe, and the big one is the principle of energy conservation.
Heisenberg writes that he “knew only too well that my scheme stood or fell by that principle.”
So Heisenberg tested his new theory as simply as he could—for a physical system similar to a
pendulum—the anharmonic oscillator—chosen for some mathematical conveniences. If energy conservation
held in this case then his strange theory must be connected to the true machinery of nature.
This was May 1925 and Gottingham was awash in springtime pollen and Heisenberg’s own
immune system tried to kill him. He had a crippling attack of hay fever. It swelled
his face, weakened his body, and left him barely able to multiply a matrix. He begged his boss,
Max Born, for permission to travel to the one place he might find relief—a small,
barren island in the middle of the North Sea called Helgoland.
It’s an amazing place. Helgoland—sometimes also Heligoland—from the High Frisian “holy land”--
with its stunning ocean cliffs and an elegant resort town. Now the war changed everything—the
town was obliterated by ally bombs and a Nazi U-boat harbor and the entire tip of the island
destroyed in the largest non-nuke, man-made explosion in all of history. But Heisenberg
came before all of that. He spent 10 days walking the cliffs, swimming in the ocean,
and calculating his matrices. Pages upon pages of them. He talks about a mounting excitement
as the scheme remained self-consistent, of how that excitement led to many mistakes that had
to be fixed. And finally, on the night of June 9th 1925 … well, let’s let Heisenberg say it:
It was 3 o’clock in the morning before the final result of my computations lay before
me. The energy principle had held ... At first I was deeply alarmed. I had the feeling that,
through the surface of atomic phenomena, I was looking at a strangely beautiful interior,
and felt almost giddy at the thought that I now had to probe this wealth of mathematical
structures nature had so generously spread out before me. I was far too excited to sleep, and so,
as a new day dawned, I made for the southern tip of the island … and waited for the sun to rise.
So cured of hayfever but now afflicted by revelation, Heisenberg returned to Gottingen.
His friend Wolfgang Pauli encouraged him to show his work to Max Born,
who immediately recognized Heisenberg’s strange mathematics as being matrix algebra. From there,
he worked with Jordan Pasqual and then with Pauli and Paul Dirac to flesh out
matrix mechanics—-the first complete formulation of quantum mechanics.
But it was not the instant hit you might have imagined. It was a strange theory—matrices were
unfamiliar to most physicists of the era, the non-commutivity felt alien,
but perhaps worst of all the theory seemed to tell no story of what was really happening inside the
electron. The catalyzing philosophy of “only consider the observables” was uncomfortable.
But it did evolve into the Copenhagen interpretation of quantum mechanics,
developed primarily with Neils Bohr. In this interpretation,
the universe between measurements is unknowable—not just in a practical
sense because you didn’t measure, but in the sense of being truly undefined.
This is the key fracture in the old worldview of Newton and Einstein. Matrix mechanics implied
that the world is not fundamentally knowable. Perhaps not even “real” between observations
in the concrete manner that we were used to. And any attempt at knowing was limited by Heisenberg’s
uncertainty principle, which he soon realized must follow from matrix non-commutivity. There’s also
the related randomness of quantum mechanics, which is baked into the quantum laws of nature. All of
it was a violent takedown of the deterministic and observer-independent Newtonian worldview.
So physicists were eager to claw back at least some sense of realism—of a mechanism that’s
independent of observer and measurement. And a few months later, Erwin Schrodinger gave them that.
Where Heisenberg launched his chain of thought from the quantization of
energy levels, Schrodinger started with de Broglie’s idea of particles as waves of
matter. Physicists had long pondered the connection between particles and waves,
and there was even a formulation of classical mechanics—the Haminton-Jakobi equation—that
could represent particles as waves while being formally equivalent to Newton’s laws. In fact
Schrodinger just substituted the so called “action” in this Hamilton-Jakobi equation
with what we now call the wavefunction, to lead to the famous Schrodinger equation,
and in doing so he established wave mechanics as a new way to formulate quantum mechanics.
Apparently seclusion in nature is key to quantum discoveries because Schrodinger was on vacation
high in the Alps, in the Swiss town of Arosa, when he made his discovery. He’d retreated there
for the Christmas holidays with, as he reports, a stack of de Broglie’s papers. And with one of
his many girlfriends whose identity to this day remains a mystery. So, not quite as secluded as
Heisenberg. Schrodinger may have come up with the final version of the paper in December 1925 and so
you may be watching this episode at the centenary of the Schrodinger equation. Certainly by January
1926 the work was complete because by then he had returned to Zurich and submitted the paper. So,
by the start of 26, we had not one by two complete formulations of quantum mechanics.
Schrodinger’s wave mechanics was an instant hit in a way that matrix mechanics was not,
and that's thanks to the familiarity of the wave formulation versus the
then-unfamiliar matrix mechanics. Perhaps even more attractive was the fact that Schrodinger’s
picture seemed to tell the story of what was happening behind the math.
The theory described this continuous, deterministic object—the wavefunction.
It was something that you could imagined. Something that “existed” between observations.
This comfort wouldn’t last. Although the idea of waves moving through space was familiar,
Schrodinger’s formulation couldn’t tell us what these were waves of. Max Born then
showed that the wavefunction could be thought of as representing probability amplitudes,
the square of which gives the actual probability of a given measurement.
But what does a wave of probability even mean? Is it a thing that exists? Or a statistical map
of underlying activity? Or some wishy-washy epistemic entity that’s uncomfortably tied to
the observer? Schrodinger and probably many others hoped that the wavefunction would
retain some sort of realist nature, a physical existence. But no fully consistent approach to
such an interpretation emerged, nor has it yet. Regarding this probabilistic interpretation,
Schrodinger wrote “I don’t like it, and I’m sorry I ever had anything to do with it.”
In fact, applying the Schrodinger equation directly to things like the double slit
experiment quickly told us that even the more “realist” wave mechanics had the same
underlying principle as matrix mechanics—we can only know the input and output states,
and everything in between is pure potentiality.
And then, in 1927, Paul Dirac showed that the Heisenberg and Schrodinger pictures are
mathematically equivalent. They really are two representations of the same system. They predict
the same amplitudes. While wave mechanics did become the more famous due to its intuitive
advantage, the Heisenberg and Schrodinger pictures both have their roles—both are useful in certain
circumstances. But it’s arguable that Heisenberg’s picture is more general, closer to base truth.
That’s because the Schrodinger equation is inconsistent with special relativity.
Relativity treats time on the same footing as the dimensions of space,
while the Schrodinger equation assigns both space and time a primary and universal status more like
in Newtonian mechanics. Of course Schrodinger knew his equation was an approximation,
valid only at low speeds. But the principle of the equation was right, and Paul Dirac
would go on to publish a relativistic version of the wave equation in 1928,
but that’s a story for another time. In fact for a previous time, because we already covered it.
Now matrix mechanics on the other hand is perfectly consistent with special relativity.
It doesn’t treat time and space separately because it doesn’t treat space at all. At
least, space isn’t built into it. Where wave mechanics describes evolution through space,
matrix mechanics describes evolution in something called Hilbert space. That’s the space of states
of the quantum system, which can involve spacey things like position and momentum, but can also
be electron orbitals or spins or whatever. The same abstractness that makes matrix mechanics so
daunting also frees it from assumptions about the nature of space, and so it can be made
consistent with Einstein. Heisenberg’s “only the observables” led him to a more general, if less
straightforward, formulation than Schrodinger. And it’s for this reason Heisenberg’s formulation also
became the foundation for the next evolution of quantum mechanics: quantum field theory.
Following this crazy half year from June to December 1925, a mad flurry
of advances propelled the fringe idea into a full-blown description of the subatomic world.
Now I mentioned the Dirac equation, then came second quantization via the Heisenberg
picture. That gave us Quantum Field Theory, in which particles are excitations in fields,
shaped by the symmetries of nature, and that lead to the Standard Model of particle physics.
The predictive power of this model has enabled so much incredible technology. But all of this
was all founded on core principles that were set down in 1925 and have held true to this day.
The world forged by quantum mechanics—the world of the international year of quantum science
and technology—is one of transistor-driven miracles—from smartphones to satellites.
It’s one of quantum chemistry and advanced materials and nuclear power. 100 years of building
useful things from century-old epiphanies. But it’s also a world whose deepest layers are,
if anything, less settled than in the previous era. A century later, we still don’t know what
quantum mechanics means—what story to tell about the mechanisms at play beneath the observables—if
such a story can be told at all. Questions like “what exists” or “what can be known” are
no longer clearly answerable. One more Heisenberg quote: "What we observe is not nature itself but
nature exposed to our method of questioning.” There’s a boundary between external reality and
our observations of it. Quantum mechanics makes us question whether even our most powerful theories
can ever cross that boundary. It’s been a giddying era of equally profound insight and confusion.
So here’s to a century of quantum mechanics, and while we’re at it, to another year of space time.
as a Fountain Pen. If you’re looking for a Holiday gift or just want one for yourself, there’s links
in the description below and you’ll get 15% off for the next 72 hours by using the code PBS.