Is alchemy gone? To answer that, say exactly what alchemy was.
Alchemy was a complete reliance on the philosophy of the past, specifically Aristotle's philosophy, with no regard to evidence to the contrary or areas where the philosophy fails. This is difficult to do if you are paying attention, so this disregard can only be described as a willful disregard of reality. And why would anyone do that?
Now I’m going to discuss how we would look for a new law. In general, we look for a new law by the following process. First, we guess it (audience laughter), no, don’t laugh, that’s the truth. Then we compute the consequences of the guess, to see what, if this is right, if this law we guess is right, to see what it would imply and then we compare those computation results to nature or we say compare to experiment or experience, compare it directly with observations to see if it works.
If it disagrees with experiment, it’s WRONG. In that simple statement is the key to science.
It doesn’t make any difference how beautiful your guess is, it doesn’t matter how smart you are, who made the guess, or what his name is… If it disagrees with experiment, it’s wrong. That’s all there is to it.
Richard Feynmann, 1964
Ideology
We have a tendency toward ideology. We like a good, clean story that explains our life. We like knowing why the world behaves as it does, and will stick to that explanation even when the explanation fails a little. Or even a lot. As long as we have a "why," an explanation of the behavior, we hold to that explanation until we are forced to abandon it by facts outside of our explanation.
But what if those facts are just a little off? Or if the theory works most of the time, just not all the time? Will we abandon the theory just for the sake of a few odd observations? Or an unusual situation? We haven't abandoned any theory in my lifetime. And I'm not young. Why have we never in the last 60 years abandoned a theory?
Quantum Theory
There are two quantum theories, matrix mechanics and wave mechanics. Wave mechanics was adopted by chemists who needed a workable quantum theory with definite predictions to describe their molecules (see below). Physics seems to have adopted matrix mechanics to describe the behavior of all subatomic particles (of which they have many), which has made them the most popular because of a number of unreasonable predictions it makes. Those predictions are popular because they are so strange. One of those predictions is superposition, where two discernably-different states exist at the same time. What it means is the electron, flying at great speed, goes through one slit or through the other slit. The observation is that it goes through both. Matrix mechanics interprets that as the "superposition" or blending, of both possibilities. Edwin Schrödinger, inventor of the wave mechanics method, realized how silly this interpretation was and made fun of it in what is now known as the "Schrödinger's Cat" thought experiment. I talk about that here. Wave mechanics is free of ideas like collapsible superposition and the multiverse; matrix mechanics takes pride in them.
But I want to talk specifically here about matrix mechanics. And despite the efforts of the matrix-mechanicists to justify superposition with the Cat (listen to any popularization of the Cat and they will tell it as an example of how fun and real superposition is) they will leave out one utterly fundamental thing: what caused the Cat to change its state from both-alive-and-dead-at-the-same-time to alive or dead? That event is undefined in matrix mechanics. Can a machine do it? Does it require a conscious being, who can tell the difference between alive and newly-dead? Can it be done by putting something in the box without looking inside? The theory is silent. Wave collapse just isn't part of the theory. This makes matrix mechanics incomplete. And it's a huge hole in the theory when something fundamental like superpositional wave collapse (as it's called) can't be described.
An odder aspect of quantum theory appears when we examine the electron configurations of the transition metals. Scandium,. for example, is listed as [Ar]4s23d1. We tell freshman students the order doesn't matter, but then when we ionize the atom, it's the 4s electron which leaves first, meaning it has the higher energy, so [Ar]3d14s2 is more appropriate. What this means is that the quantum mechanical calculations, which put the 4s subshell below the 3d subshell, are wrong. The situation is further complicated when you look down a column in the transition metals. Periodic theory says that for all these elements to behave chemically similar they must have the same outer-shell electron configurations. But look at the column that starts with nickel; platinum and palladium act chemically similar to nickel, but each have a different outer-shell electron configuration: Ni: [Ar]3d84s2, Pd: [Kr]4d10; Pt: [Xe]4f145d96s1. This breaks the periodic law, and leaves quantum theory in a tight spot when it disagrees with reality. We chemists then have a routine of hand-waving some explanations that never satisfy us, and confuse our students. A theoretical chemist named Schwartz has the best explanation: you need to allow atoms to have multiple ground states then find the average of them to describe the physical behavior (S-G Wang and W. H. E. Schwarz, Angewandte Chemie International Edition 48, no. 3, 404, 2009). This goes against a fundamental aspect of quantum physics, that all systems are stable only at the lowest energy state. In short, the periodic table is not the consequence of quantum theory.
Another observation, that ns2 configurations measures as being more stable than ns1 configurations by about 1 eV (getting larger as n gets higher), has no explanation I've ever seen from quantum theory; it is a purely chemical phenomenon. The situation is complex enough that we can't assign an energy to a specific orbital; to calculate the electron configuration you need to calculate total energy of all possible electron configurations, then select the one configuration (or average multiple configurations) with the lowest total. This is not how quantum mechanics describes electron energies.
Molecular Orbital Theory
Molecular Orbital Theory, MO theory, is chemistry's application of wave mechanics to model the electron orbital energies of a molecule. We chemists chose the wave mechanics approach because it was so approachable. Back in the early 1950's there was a bonding theory that was very popular, the Valence Bond Theory, developed and popularized by one man, Linus Pauling. He had invented it twenty years earlier, and it was a real hit, even though some aspects of the theory weren't all that well thought out. But while Pauling was interested in defending it, it was the only theory known by most chemists. Pauling became interested in another idea, that of the beneficial effects of massive doses of vitamin C. It occupied so much of his time that he dropped work on VB theory and published a book in 1971 bragging up the virtues of vitamin C. The book and Pauling himself was trashed by the scientific and medical communities; it was a silly idea that had very little legitimate proof.
When chemists saw all this going down they turned to another application of wave mechanics to molecules, the Molecular Orbital Theory. This was a more difficult-to-apply use of wave mechanics, but examples were soon produced that demonstrated a remarkable closeness of the MO model of a molecule to the physical measurements of the same molecule. Problematically, the calculation was far more difficult to do, and required massive mainframe computers to run a calculation. Even more problematically, there were so many ways the calculation could be done that the experimenter running the calculation first needed those measurements to assess which of the many options produced a model sufficiently close to the measurements to trust. And that's the way MO calculations are still done: measure a molecule, model it so that you get the same orbital energies and properties of the molecule (like bond lengths and angles, orbital energies, vibrational and rotational frequencies, among others), then use that exact method for a similar, but unmeasured molecule, with the faith that the calculation matches reality. Matching reality is difficult, because quantum mechanical calculations don't use parameters that interest chemists, like bond distance, bond angles, and bond energies. They use electron energy and momentum. We are trying to describe a car race by measuring the frame of the competing cars. So we try our best to take the electron calculations and expand them to describe the molecule and it's properties.
Did you see the problem? The theory does not say how to do the calculation! The theory cannot, on its own, find the correct model of a molecule; only an experimenter can find one. The theory is so incomplete that it cannot by itself do anything; it requires an experimenter who uses both hands and both feet driving the calculation. And the involvement of the experimenter is very worrisome to those who wonder, does the model of a molecule match reality, or does it match what the chemist wants it to be? To me this is a far bigger problem than incompleteness. It introduces experimenter bias as a necessary aspect of the method. Yikes! Molecular orbital theory can best be described as forcing quantum theory into a molecule, a place it does not belong.
Darwinian Evolution
Darwin proposed that one species evolves into another (species being an animal which can no longer mate successfully with the species it used to be before evolving) because of environmental pressures. When the environment changes, some organisms will adapt faster and better to the new situation, and become more successful in reproduction. When enough of those changes happen, a new species is created. The process happens slowly, over time, but it is responsible for all the variety of life we know today.
A satisfactory theory of natural selection must be quantitative. In order to establish the view that natural selection is capable of accounting for the known facts of evolution we must show not only that it can cause a species to change, but that it can cause it to change at a rate which will account for present and past transmutations.
J. B. S. Haldane, “A mathematical theory of natural and artificial selection. Part I,” Transactions, Cambridge Philosophical Society 23, no. 2 (1924): 19.
Until J. B. S. Haldane started doing some math. In 1957 he published a paper which asked, then answered, a simple question: when a new change happens in an individual that gives that individual a reproductive advantage, how many generations will it require for that change to be present in every individual of that species? He actually asked the question as the negative, how many individual which do not have the advantageous mutation need to die before the mutation is present in every individual?
To understand the problem, I'll use cattle, as Haldane did in his introduction to his The Cost of Natural Selection: Ranchers want three characteristics in their cattle: lots of milk, lots of good meat, and lots of healthy calves. If they breed for one characteristic, they will need to sacrifice the other two. So what's the best approach to developing a cow with all three: one at a time, or find one cow with all three already, then improve them? Well, as the calculation verifies, and as ranchers attest, it never happens. Cattle are slow breeding, and Haldane calculated that for a population of 10,000 individuals, the number of generations to get a single mutation in all individuals is more than 300 generations. And if you want all three in one individual, it takes three times as long waiting for it to happen randomly, assuming that the development of one characteristic does not interfere with the development of the next.
It's the 300 generations part that is the problem. Given the approximately 1000 mutations that mark one species as different from a closely related species, it takes 300,000 generations for those 1000 changes to happen. It's way too long. Darwin has species being generated in a much shorter span, examples are as few as 10 to 20 generations. In the case of chimpanzees to humans, 1.5 genetic changes per generation! The number of generations are even longer when some mutations need to be optimized by mutating a second time, as seems the case in many mutations. It's a big problem. It is Haldane's Dilemma.
Only a few biologists seem interested in resolving Haldane's Dilemma. Some have chosen denial, asserting that 1000 mutations are far too many to explain evolution by lumping many mutations into a single event. That doesn't solve the problem, though it does shorten the discrepancy, because now you need mutations to happen and to survive with no environmental pressure to do so. Others have admitted to an extreme mutation rate, but that brings on a condition known as error catastrophe, when mutations are so rapid that the species can't eliminate bad mutations (by death when young) to help keep the DNA viable, and the species is simply unable to survive in any environment.
Despite Haldane’s work, a massive body of literature has accumulated asserting the primary role of natural selection in evolutionary change, often implying rates of adaptive evolution that exceed plausible limits.
Chase Nelson, https://inference-review.com/article/haldanes-dilemma
Also see Justin Fay, Gerald Wyckoff, and Chung-I Wu, “Positive and negative selection on the human genome,” Genetics 158 (2001): 1,227–34.
The best solution was proposed by Motoo Kimura in 1968: neutral evolution. In neutral evolution, mutations happen not because they give the organism any advantage to survival and propagation, but simply because they happen. This means that several mutations can happen and the individual still be indistinguishable from it's neighbors. Random genetic drift could have hundreds or thousands of mutations active in a population simultaneously (Kimura calculated the number of mutations which could survive this process at fewer than 527, a small number). This could explain the rate of mutations high enough to account for speciation in the time we observe it in nature. But it does introduce another problem: why do species form now that there is no environmental fitness advantage to the mutations? By solving one problem the underpinning of evolution is kicked out from under foot. This remains a massive problem in evolution, and there are no satisfactory answers.
The Cosmological Constant
Cosmologists are often in error, but never in doubt.
Lev Landau. quoted in Big Bang by Simon Singh (2004)
Otherwise known as Lambda, this was added as a temporary fix by Einstein to his Theory of General Relativity to help explain the expansion of the universe, or at least why a static universe was not being drawn together by gravity, and he later removed it when he showed it was zero. But it wasn't zero, and physicists put it back. They now call it "vacuum energy" and it is probably related to the "dark energy" which is accelerating the expansion of the universe. Measuring this value is very important to cosmology, probably the most important thing in cosmology. Predictions from quantum theory, which operates on the atomic scale, give it a value near 1 (measured in reduced Plank units). When measured in the universe, by observing how fast the universal expansion is accelerating, it has a value of around 10-122! This is a HUGE difference, greater in magnitude than the number of atoms in the universe, including the parts we can't see. This difference between the two measurements of the same number is the biggest number in all of science.
The history of science provides many instances of discoveries which have been made for reasons which are no longer entirely satisfactory...the discovery of the cosmological constant is such a case.
Georges Lemaitre in Albert Einstein: Philosopher-Scientist by Paul Arthur Schilpp 1949
Something has to be wrong, but cosmologists hang onto the notion that lambda is a thing, because both quantum theory and big bang theory are well-known and accepted.
Dinosaur Extinction
The dinosaurs went extinct, the story goes, when an enormous asteroid hit the earth. The debris of that asteroid, rich in iridium, is well known around the world as an orange layer less then one inch thick in most places. The layer is known as the K-T boundary. The story has dinosaurs peacefully living until the asteroid arrives. So one might reason that in the boundary layer there should be a massive number of dinosaur fossils. Nope, not one has been found in the layer. Even more oddly, not one has been found in the ten feet below the layer. I don't know how long that is, but enough dust to pack down to ten feet of rock takes a while. Why are there no dinosaurs there? Maybe we missed them all. Or maybe they weren't there any more. Maybe the extinction event was gradual, and all the dinosaurs were gone long before the asteroid arrived.
But ask a scientist what killed the dinosaurs and you get one answer only: the asteroid.
The Grand Unified Theory
The Grand Unified Theory, or GUT, is the effort to combine the two great theories of physics, quantum theory and relativity theory, into a single description of matter and space. It seems reasonable that there is one theory that describes everything. But it's never been done, and really can't be done, because each describes mass in a fundamentally different way.
Other Theories
There are other current theories which have holes in them, or unresolvable problems: In cosmology, is the universe flat, or is space curved over great distances? And why is the distribution of galaxies to homogenous and completely isotropic when there is good reason why they should not be? Why is there still no good explanation of what gravity is, only descriptions of how it behaves? In mathematics, why can no system of mathematics define itself completely? Why does it always take an outside explanation to make it complete? (This is something like assessing the truth of the sentence, "This sentence is false.") There are others, I just haven't studied them yet.
All this was summarized beautifully by Mark Henry:
The inescapable conclusion then is that scientific research tries to find answers to six basic questions:
- Q: What is a universe? A: refer to general relativity (GR).
- Q: What is a vacuum? A: refer to quantum mechanics (QM).
- Q: What is light? A: refer to electromagnetism (EM).
- Q: What is matter? A: refer to chemistry (CH).
- Q: What is information? A: refer to thermodynamics (TH).
- Q: What is life? A: refer to biology (BL).
Mark Henry in a reply to a letter from Eric Scerri at https://inference-review.com/letter/on-the-madelung-rule (an amazing document)
The top three are theories used to describe the universe and everything in it, and the bottom three are described as being wholly dependent on the three theories [chemistry and biology are not theories, and thermodynamics is a set of laws which stand alone from the other theories]. But it doesn't work that way, not here in reality. Maybe it does in the new mythology, where we believe the stories over reality.
Quantum theory is 90 years old, is incomplete, and goes unchallenged. Molecular Orbital theory is incomplete in a different way, and goes unchallenged. Evolution theory has been accepted for 150 years, has been deeply challenged, and the challenge goes ignored. Why?
The answer is, I think, the same as why alchemy existed for 2000 years and never produced gold or metallic transmutation in all that time: idealism. We love holding to an ideal to explain the world around us, to give us an answer to "Why?" When we find a Why we hold it tightly, even when we know it's not working.
Confirmation bias is a name for seeing what we want (or expect) to see. It massively colors how we interpret our observations, and it's likely how the alchemists sustained themselves intellectually. Ideology informed their senses.
Scientists today are doing exactly the same thing.
Are we still alchemists?
We certainly are. We are just like them.
An idealist is one who, on noticing that a rose smells better than a cabbage, concludes that it makes a better soup.
H.L. Mencken, A Book of Burlesques
Whenever a theory appears to you as the only possible one, take this as a sign that you have neither understood the theory nor the problem which it was intended to solve.
Karl Popper
Idealism is fine, but as it approaches reality, the costs become prohibitive.
William F. Buckley, Jr.
Idealism is the despot of thought, just as politics is the despot of will.
Mikhail Bakunin
If it disagrees with theory, it's WRONG. That's all there is to it.
Every alchemist, dreamer and lousy scientist, ever