John O. Campbell
The new book Darwin does Physics is now published! Here is an excerpt.
Edwin Jaynes, a great champion of the Bayesian revival,
spent much of his long career battling those supporting the Frequentist
interpretation of probability theory. Jaynes was convinced that probabilities
describe a state of knowledge or the degree of certainty we may
claim given the data we have (1).
We believe that to achieve a rational picture of the world it is necessary to set up another clear division of labour within theoretical physics; it is the job of the laws of physics to describe physical causation at the level of ontology, and the job of probability theory to describe human inferences at the level of epistemology.
On the other hand Frequentists understand probabilities as
simply frequencies; the proportion of times that a given outcome occurs. They
consider frequencies to be properties of nature with no strong links to inference.
Figure 1: Edwin Jaynes made the most important contributions to the development of Bayesian inference of anyone since Laplace.
Jaynes was a physicist, a field of study where the Bayesian
approach makes a lot of sense. We might say that in random encounters with the
atoms of the universe, the probability that the next one will be a hydrogen
atom is about .75. While this probability might seem merely a property of
nature it is actually calculated or inferred from a physical model which
describes the creation of atoms and which is supported by many spectral
measurements.
Physics has developed a detailed model of how atoms were
formed which is roughly:
- First hydrogen was created and then some hydrogen atom combined to become helium atoms.
- The model predicts this should have happened in the ratio of 1 helium to 3 hydrogen atoms.
- Almost all atoms more complex than helium were created in stars.
- As stars are relatively sparse in the universe only a scant number of atoms more complex than helium have been created, so the relative abundance remains at nearly 75% hydrogen and 25% helium.
When we measure the relative abundance of atoms out to the
extent of the visible universe the data supports many aspects of this physical
model. Thus .75 is a number we have inferred using a model that is supported by the data.
While it is true that a physical process is responsible for
.75 of the atoms in the universe being hydrogen the model which infers this
number was developed by physicists, it is a model created by humans and it is
here that the inference takes place.
Frequentist probability theory, on the other hand, was
developed largely by biologists. Their primary objective was to develop
mathematical tools capable of modelling the details of Darwinian evolution. The
aim of this project was to model the shift in the proportion of physical characteristics
or alleles within a population between generations. This proportion or
frequency of a given characteristic can be found out merely by counting; it does
not require a sophisticated human model. Any scientific model designed to infer
these probabilities would have to include the nebulous quality of ‘fitness’.
Worse and somewhat of an embarrassment, any equation referring to fitness used
to calculate the change in frequencies between generations is clearly in the
form of a Bayesian update.
However biologist became increasingly aware of one thing:
the model responsible for inferring the proportion of characteristics which
would be found in the next generation was not the scientific model, it was a biological
model written in DNA. This model is in the form of the genome and this model is
also involved in inference, it infers those characteristics which will contribute
to the reproductive success of the next generation. The model is supported by
evidence in the form of the reproductive success experienced by the current
generation.
Biological science developed a model of this process composed of
Darwinian Theory and the many details of micro-biology. This scientific model also
performs inference; competing hypothesis may be formed and decisions reached on
the basis of evidence and experimental data. Nevertheless, it is clear that the
scientific model is essentially a secondary model developed to describe the
primary model composed of genomes, phenotypes and their experience in the
environment.
While the secondary model fits nicely within the Bayesian
interpretation, the primary model provides some challenges. Natural
selection, the inferential system operating within the biological world, is
clearly not a form of human reasoning. It was busy inferring new life forms
long before humans existed and in fact humans themselves were inferred by this
process. In this sense the probabilities involved with the primary model are
properties of nature and may fit more comfortably within the Frequentist
interpretation.
Clearly some compromises must be made to reconcile the
Bayesian and Frequentist interpretations of probability. The one I suggest is
that probabilities be considered as components of inferential systems, not just
as tools of human inference but of all inferential systems that are found in nature.
Bayesians have always understood that
probabilities are a component of inference but they insist they are restricted
to human inferences. Frequentist recognize the role of probabilities in
non-human spheres but deny them a role in inference. Perhaps the strengths of
both these interpretations may be retained by viewing probabilities as a
component of all inferential systems whether human or non-human.
Although Jaynes never deviated from his anthropocentric interpretation
of probability he did consider some instances where inferences are performed in
the non-human realm. One case was with intelligent robots. The unifying theme
of his great text on probability is the design of a robot capable of performing
human-like inferences, making it clear that he considered it possible for
inference to be performed by non-humans (2).
He also made a brief foray into considerations of the
inferential ability of natural selection. Jaynes starts by noting that the scientific
understanding of our rich mental life had not yet discovered any general organizing
principles. He then suggests that Bayesian inference might fill this role and
that natural selection, in its evolutions of brains, would surely have hit upon
Bayesian inference as the correct mechanism for gaining knowledge of the world.
Workers note the seeming absence of any organizing principle; we wonder whether the principles of Bayesian inference might serve as a start. We would expect Natural Selection to produce such a result; after all, any reasoning format whose results conflict with Bayesian inference will place a creature at a decided survival disadvantage. Indeed, as we noted long ago (Jaynes, 1957b), to deny that we reason in a Bayesian way is to assert that we reason in a deliberately inconsistent way; we find this very hard to believe.
He opens the door, a crack, and considers ‘creatures’ rather
than only humans. Jaynes also comes tantalizingly close to recognizing natural
selection as an inferential system. He is on record as claiming that Bayesian
inference is the only mechanism capable of finding good solutions. Surely, if
pressed, as to the process used by natural selection to discover the optimum
mathematics of reasoning, he would have identified this as a form of Bayesian
inference.
In considering the challenges of designing an inferential
robot he is drawn to examine the mental processes of sensation and perception.
He concludes that the general organizing principles of these mental functions
is Bayesian inference.
The general conclusion from all these experiments is less surprising to our relativist generation than it was to the absolutist generation which made the discoveries. Seeing is not a direct apprehension of reality, as we often like to pretend. Quite the contrary: Seeing is Inference from Incomplete Information, no different in nature from the inference that we are studying here.
The information that reaches us through our eyes is grossly inadequate to determine what is “really there" before us. The failures of perception revealed by the experiments of Ames and Cantrell are not mechanical failures in the lens, retina, or optic nerve; they are the reactions of the subsequent inference process in the brain when it receives new data that are inconsistent with its prior information.
It did not escape Jaynes that the brain is a biological adaptation
which came into existence long before humans and that animal brains presumably also
invoke the principle of inference:
Presumably, a dozen other examples of human and animal perception would be found to obey a Bayesian reasoning format as its “high level" organizing principle, for the same reason.
Here he explicitly recognizes other animal’s ability to perform
inference but nowhere in his writing does he concede a non-anthropocentric definition
of inference.
Jaynes’ view of the mental processes taking place within
brains is well in accord with current understanding; they are examples of
Bayesian inference. For example vision is explained as inferring an accurate
mental model of the world from scant sensory evidence. Many researchers
describe the process of evolving such accurate mental models as involving a
Darwinian process (2).
This view of mental processes, such as vision, as Darwinian processes following the principle of Bayesian inference, leads us
to add mental processes to our list of inferential systems found in nature.
Brains accumulate knowledge in order to more accurately model the world around
them. Neuroscience or the scientific study of these mental processes and
behaviours is also an inferential system involving models and supporting
evidence. Again the science is clearly at a secondary level as its subject is composed
of a more fundamental inferential system.
This view suggests an astonishing unification of our
understanding of nature as many of the subject matters studied by science are
found to operate according to the same general inferential principles as does science.
If we consider science to be but an example of the more general
process of cultural evolution, we are led to a four tier nested hierarchy of
inferential systems: Cultural evolution entails mental evolution which entails
biological evolution which entails physical structures. Placing ‘physical
structures’, at the bottom of this hierarchy is troublesome. Are we
confident that this is really the bottom? Why not consider instead ‘physical evolution’
as an inferential system at the bottom of the hierarchy? This would have the
benefit of leaving our hierarchy open-ended.
We have no clear evidence that currently known physical
structures form the true bottom of this hierarchy and some, admittedly inconclusive
and indirect, evidence that they do not. Uncomfortable as we are with ignorance,
we tend to claim that the boundary between our knowledge and ignorance accurately maps nature; that nature extends only as far as our knowledge. This must surely be hubris. I earlier presented some
arguments which might lead us to consider ‘physical structures’ themselves as inferential
systems and will continue to develop this notions in latter sections.
Bibliography
1. Jaynes, Edwin T.
Clearing up the mysteries - the original goal. [book auth.] John Skillings. Maximum
Entropy and Bayesian Methods. s.l. :
http://bayes.wustl.edu/etj/articles/cmystery.pdf, 1989.
2. —. Probability Theory: The Logic of Science. s.l. :
University of Cambridge Press, 2003.
3. Selectionist and evolutionary approaches to brain
function: a critical appraisal. Fernando, Chrisantha, Szathmary, Eros
and Husbands, Phil. s.l. :
http://www.sussex.ac.uk/Users/philh/pubs/fncom-Fernandoetal2012.pdf, 2012,
Computational Neuroscience.
No comments:
Post a Comment