Knowledge

John O. Campbell

 

The prominent physicists David Deutsch reminds us of the power of knowledge:

Everything that is not forbidden by the laws of nature is achievable, given the right knowledge.

Undoubtedly knowledge is the essential ingredient of order. I will argue however that the preeminent position which Deutsch bestows upon knowledge does not go far enough, that the very laws of nature have themselves come into being through the accumulation of knowledge, that the laws of physics may be considered but a summary of nature’s accumulated knowledge.

Information at the basis of structure

Since atomic theory came to general acceptance near the start of the 20^th century physics has learned that atoms themselves are composed of yet more fundamental particles.  There is little reason to suppose that the quarks and leptons currently understood to be fundamental will not in turn be found to be composed of an underlying family of even more fundamental ‘preons’.

All structures appear to be composed of ever more fundamental units. At each level new structures may form because the subunits are able to interact and exchange information. Without the exchange of information between them no structures could exist; at that level such a reality would be composed entirely of isolated entities unable to detect or influence other isolated entity.

The complexity of the world we experience is due, in a primary sense, to this transfer of information; the ability of one entity to record information about another entity. The four forces of nature by which fundamental particles are understood to influence each other may be described as instances of information transfer. Indeed these forces appear to be the ultimate form of all information transfer.

It is akin to the miraculous that our universe is not a barren one where each fundamental particle is isolated and uninfluenced by anything else. In such a world there could be no bonds between matter and no entity could contain information regarding another entity. Indeed up to the level of the universe as a whole science understands all structures to be composed of interacting sub units.

Principle of incomplete information

However information transfer is highly constrained; it appears to be a universal principal, which I will name the principle of incomplete knowledge, that one entity may convey only very little information about itself to other entities[1]. This principle seems as applicable to human communications as it does to information transfer between quantum entities. Complete information concerning one entity is never available to another entity; some degree of ignorance is unavoidable.  

A general argument I will make is that knowledge is essential for the existence of many natural systems including quantum systems, biological systems, behavioural systems, and cultural systems. In this view knowledge has played a long and illustrious role in the evolution of the universe.

Towards a definition of knowledge

Surprisingly within science, our primary means of understanding the universe, the term ‘knowledge’ is used in only a vague sense and does not have a clear technical definition. This lack of an adequate definition for a phenomena playing a central role in the structure of the universe has resulted in a good deal of confusion. I will suggest a technical definition of knowledge later in this section which may help to resolve this problem.

While science has not provided a clear definition of knowledge it has developed a detailed understanding of ignorance. Ignorance is the amount of information in bits which any entity lacks in its knowledge of another and has the technical term entropy. Entropy is conceptually and mathematically well understood. 

My proposed definition of knowledge leverages our deep understanding of ignorance. Knowledge is the probability which is the mathematical inverse of entropy. This probability is essentially the chance that a random pick within the realm of ignorance will be the correct choice; our odds of being correct (knowledge) increases as ignorance decreases.

Knowledge and Bayesian inference

 The principle of incomplete knowledge requires that any knowledge must be uncertain knowledge. The field of mathematics which describes degrees of uncertainty or degrees of plausibility is Bayesian probability. It provides all the necessary mechanisms for determining the probability of knowledge from the information content of supporting evidence. Thus Bayesian probability prescribes the evolution of knowledge as evidential information is gained.

Common usage of the word knowledge usually involves an internal representation or model of external phenomena. The definition of knowledge I will develop is in close agreement with this view. In order for an internal model to be accurate it must receive information of the external phenomena and be capable of updating its representation as the phenomena changes. Thus it is necessary for the model to receive information concerning the phenomena and to possess the ability to update itself accordingly.

In contrast to knowledge, information has a fairly straight forward scientific definition. It is measured in bits which may be considered answers to ‘yes’ or ‘no’ type questions. For instance the game of twenty questions can be considered as one where the questioner receives twenty bits of information in order to identify the correct answer. Twenty bits of information is powerful, it is able to distinguish between 2²⁰ or over a million different possibilities.

Information may also take the form of a coded message that represents an entity. For instance all of the concepts dealt with by computers are represented by messages in binary code. As a simple example we might consider the sixteen binary states of four flipped coins represented by ones for heads and zeroes for tails.

0000		0001		0010		0011
0100		0101		0110		0111
1000		1001		1010		1011
1100		1101		1110		1111

These sixteen distinguishable outcomes of flipping four coins may each be identified with four bits of information. In general n distinguishable states can be distinguished or coded with log₂(n) bits of information. The probability of randomly choosing a specific state from n states is 1/n or if we let I stand for the information required to code for the state then the probability is 1/2^I or equivalently 2^-I. 

The history of information as a well-defined scientific concept has been quite brief. Claude Shannon introduced our modern conception of information in 1948. Since then it has come to be seen by many as perhaps the most fundamental concept in Science. The great physicist John Wheeler said that he had come to view ‘everything as information’ (52). This astonishing ascendance of a scientific concept from its introduction as a scientific concept to perhaps the most fundamental in science has occurred in only fifty years.

There can be difficulties when a colloquial term such as ‘information’ is adopted by science and given a precise technical definition. The technical definition may be quite different from common usage and confusion may arise.

Technical definition of information

 Dictionary.com defines information:

1.  knowledge communicated or received concerning a particular fact or circumstance; news: information concerning a crime.
2. knowledge gained through study, communication, research, instruction, etc.; factual data: His wealth of general information is amazing.

These definitions describe information in terms of knowledge but this is not the technical definition of information. Technically information is defined in terms of probability:

Where each w is one of the possible outcomes of some event and w_n is the nth possible outcome.  P(w_n) is the probability that the nth possible outcome will actually occur. In our discussion it should be assumed that the log function is to the base 2 and thus information is given it bits. 

The term on the left side of the equal sign might be paraphrased as ‘the information (I) received on learning that the outcome w_n has occurred’. The right hand side of the expression might be paraphrased as ‘the negative log function of the probability previously assigned to the possibility that the outcome w_n would occur’.

So this definition says that the information received when event occurs equals the negative log of the probability that had been previously assigned to the event happening. Information may be thought of as the amount of surprise experienced when the actual outcome is learned. 

Thus technically information is a measure of probability. If we assigned a low probability to an outcome we receive a lot of information if it does occur. If we expected it to be sunny today but it rained we received a lot of information; our plans may have to be fully revised. On the other hand if we expected rain and it did rain then we did not receive much information and not much needs to be updated. 

It is perhaps somewhat paradoxical that although information has come to be considered perhaps the most fundamental concept in science it is not simple. It requires the assignment of a probability to an outcome and in addition it requires that this probability be compared to the actual outcome, requiring a rather complicated mechanism for any physical instantiation. Thus information transfer is itself a complex phenomenon.

We might also use the above equation as a definition of probability: probability is a numerical assignment of the degree of plausibility for a given outcome. 

Bayesian interpretation of information and knowledge

In Bayesian terminology probabilities represent states of knowledge thus making a connection with information’s colloquial meaning.

Perhaps surprisingly although the term knowledge is used extensively within the Bayesian scientific literature there does not seem to be an accepted definition. In fact Jaynes uses the term to define probability itself:

In our terminology, a probability is something that we assign, in order to represent a state of knowledge.

However nowhere in his writing or in other Bayesian literature have I been able to find an in-depth description of what is meant by ‘knowledge’. Unfortunately the primary technical definition of knowledge seems to still be the one offered by Plato over two thousand years ago and still embraced by many philosophers today that knowledge is ‘justified true belief’.

The first problem with this definition is that it just refocuses our attempts at clarity onto deciphering what is meant by ‘justified true belief’. This seems to offer only a regress to other vague terms. A perhaps more serious problem is that this definition has come to be understood as referring to  human knowledge and justified true human beliefs. It does not refer to knowledge found anywhere else in nature.

Jaynes himself seems to have accepted the philosophers’ definition.

it is...the job of probability theory to describe human inferences at the level of epistemology.

I suggest that this confusion over the nature and scope of ‘knowledge’ within Bayesian thought has led to numerous difficulties in its proper application to fields such as biology where the existence of non-human knowledge is evident.

In our brief review of the proper context for knowledge we have encountered a number of related concepts including: models, information, updating models with information and probability. We can now combine these to gain an understanding of the process by which knowledge may evolve.

Returning to the definition of information as a measure of probability we should consider that the probabilities assigned to the mutually exclusive and exhaustive set of all the possible outcomes of an event must sum to 1. One and only one outcome of the model must occur. We may consider this set as a list of hypothesis; each assigned a probability that the associated outcome will occur.  This kind of complete set of hypothesis forms a model of the event. To find the correct hypothesis in the set we must gather enough information to label one true and the rest false.

A set of probabilities which sums to 1 is called a probability distribution and has many interesting mathematical properties. Perhaps foremost amongst them is entropy. Entropy is the sum of the information contained in the set of hypotheses, the information of each hypothesis weighted by its probability:

Where E is entropy, H is our model and h_n are the n hypotheses making up the model. This expression for entropy may be paraphrased as: the expected surprise that a model of the outcomes will experience when the actual outcome becomes known.

Surprise, and thus increased entropy, occurs when the model lacks predictive accuracy. The entropy of every probability distribution has a value between zero and infinity. It equals zero when the probability distribution is a certainty; one hypothesis has a probability of 1 and the rest of 0. The uniform distribution which has n members all having probability 1/n has the highest entropy of any distribution with n members. Its entropy approaches infinity as n approaches infinity.

Entropy measures what a model does not know or its uncertainty. In the case of thermodynamics entropy is the amount of uncertainty in the exact microstate of the system when we have some partial information such as temperature:

The amount of additional information that would allow us to pinpoint the actual microstate is given by the entropy of the distribution.

Definition of knowledge

As entropy measures a lack of knowledge or ignorance it is a kind of inverse of knowledge and we might expect a technical definition of knowledge could be formed in terms of entropy. A first step forward is to recognize that knowledge, like entropy, is a property of a model; it is a measure of a model’s predictive accuracy. Drawing on the relationship between information and probability we noted earlier and noting that entropy is a form of information I propose the technical definition for the knowledge of a model K(H) as:

For example the model describing a coin flip is the two member uniform probability distribution {.5, .5}. It has entropy =1 bit. There are 2¹ = 2 distinct possible outcomes about which the model is ignorant: [Heads] and [Tails]. The model’s knowledge is 2^-1  = .5 which is the probability that an arbitrary choice will produce the correct prediction.

Knowledge and ignorance

In general entropy is a measure of ignorance and ignorance is described by the uniform distribution; when nothing is known all possibilities are equally likely. The entropy of the uniform distribution which has n possibilities is log(n). Using our definition its knowledge is 2^-log(n) or more simply 1/n. But this is the probability that any of the possibilities within the space of our ignorance is the correct one. Our definition of knowledge is the probability of randomly guessing the correct possibility within the boundaries of ignorance.

The amazing implication is that knowledge amounts to a random guess within the sphere of ignorance. The only way the guess may become more likely is if the space of ignorance is reduced.

As an example let consider a model in the form of a distribution which has 16 possibilities. To begin with we have no information which would make one possibility more likely than the others so we assign the uniform distribution where each probability is 1/16. This distribution has 4 bits of entropy. Let’s say we get some evidence concerning the model and when this is applied via Bayesian updating some possibilities become more likely than others and the entropy of the new distribution is reduced to 3 bits.

The new state of knowledge is 2^-3 = 1/8. But this is the same knowledge as contained in a uniform distribution with only 8 possibilities; it is the same probability as a random guess amongst 8 possibilities. The change in certainty of the model due to the evidence is equivalent to reducing the scope of our ignorance from 16 possibilities to 8.

The amazing implication is that knowledge may be considered a random guess within a scope of ignorance. The only way for the guess to become more likely, for knowledge to increase, is for the scope of ignorance to be reduced.

 

While this definition might seem mathematically cumbersome it has some attractive properties:

1. Knowledge is a positive number between 0 and 1.
2. Knowledge increases in value as entropy or ignorance decreases and vice versa.
3. Knowledge approaches 1 when one of the model’s hypothesis approaches certainty.
4. The knowledge of the uniform distribution is especially simple; it is 1/n, the same as the probability for any particular outcome. Thus a fair six sided dice has a distribution with knowledge of 1/6 which agrees with our common sense perception of knowledge as a measure of how close we are to certainty.    
5. The knowledge of the uniform distribution approaches zero as n approaches infinity. Again in agreement with common sense; all else being equal we know the least when there are a great many possibilities and we have no information that would allow us to prefer one over any of the others.

Our definition of knowledge in terms of a probability agrees with the usual Bayesian definition of probability as a state of knowledge with one important difference; knowledge is a property of any model in nature and such models are not necessarily closely related to humans.

With this definition in hand we might next ask ‘how is a model’s knowledge increased?’ Fortunately mathematicians have shown that knowledge increase must follow a unique algorithm: the Bayesian update. On the reception of new information (I) by the model (H) the probability of each component hypothesis (h_n) making up the model must be updated according to:

Where X is the information we had prior to receiving our new information I.

This theorem demonstrates that the model composed of the updated probabilities will have the greatest accuracy possible given the data. Models which are updated according to Bayes’ theorem on the reception of new information will tend to have the greatest knowledge or predictive accuracy. There are however some important caveats to this that are explored in Appendix 1 using as an example the results of medical tests. 

We have seen that the mathematical concepts of information, probability, entropy, knowledge, Bayesian update, and models are intimately related. They are in fact but different properties of a mathematical entity called inference. Inference is the mathematical process for basing conclusions on data and for reaching the best conclusions possible in the face of incomplete information.

A clearer view of the intimate relation amongst these concepts might begin with probability.

1. Cox showed that any consistent process of assigning real numbers to degrees of plausibility would lead to the sum and product rules of probability theory; these rules may be taken as the axioms of probability theory.
2. The Bayesian update is a mere rearrangement of the terms of the product rule.
3. The Bayesian update connects new information with updated probabilities which form a probability distribution over an exhaustive and mutually exclusive set of hypothesis (H); in our terms this is a model.
4.  Entropy and knowledge are inverse functions which are properties of models.

Thus we see that these concepts are inseparable; they are defined in terms of one another and any one of them implies the others. The integrated entity which they form is an inferential agent. When we encounter any one of these concepts we should expect to encounter them all operating together as an inferential agent.

 

This claim may appear reckless. In some views probability or information are primitive concepts which are found throughout nature and do not necessarily entail the complications of inference. However we might consider that probability and information are defined in terms of one another. Probability is the assignment of a degree of plausibility of an outcome. No such assignment can be made without considering the set of alternative outcomes, in other words without considering a probability distribution over all possible outcomes. This is a model and on the reception of new information the correct probabilities entailed by the model are given by the Bayesian update. Thus my claim that probability (and the other related concepts) has no meaning other than within the context of inference.

Rather than narrowing the context for probability, this view actually is an expansion on the usual Bayesian view of probability. Bayesians have stressed that probability is related to a state of human knowledge or inference. In the view presented here the scope of probability is expanded beyond humans to the larger arena of inferential agents in general.

---

[1] The basis for the principle of incomplete information may reside in the nature of quantum information which is the basic form of all information exchange. The quantum information necessary to fully describe an entity can be divided between Holevo information which may be communicated and the information of quantum discord which may not (91). The quantity of Holevo, or classical information, is usually minute compared to quantum discord.

The Arrow of Time

John O. Campbell

Last evening my wife and I enjoyed a wonderful dinner and visited with a couple of our Science Buddies, Michael and Julia Skrigitil. I often call Michael a 'rocket scientist' because he worked on the Russian space program and due to his deep knowledge of physics.

During the course of our conversation Michael told me of some exciting new research  claiming to have solved the arrow of time problem which has long plaqued physics.

Briefly this problem arises as all the laws of physics are time reversible; they make equal sense running both forward or backward in time. However we do not observe this symmetry; many physical phenomena, the breaking of an egg for example, always appear to run forward in time. This observation is encapsulated in the second law of thermodynamics which states that the entropy or uncertainty in closed systems will tend to increase over time. This raises the unanswered question: What are the physical processes which result in the arrow of time?

Today I looked up this research and found an informative article: Times Arrow Traced to Quantum Source.

It turns out this 'new' solution was first proposed by Seth Lloyd over twenty years ago. Lloyd is brilliant. He coined one of my favorite quotes on quantum behaviour: Quantum interactions are quantum computations; what is computed is the outcome of the interaction.

The article says:

Lloyd realized that quantum uncertainty, and the way it spreads as particles become increasingly entangled, could replace human uncertainty in the old classical proofs as the true source of the arrow of time.

Entropy is a measure of the amount of uncertainty or ignorance. It is the expected value of surprise inherent to any probability distribution (sum of the products of each probability and its negative log value). Thus we should expect to find entropy in any model of reality based on probability distributions, such as quantum systems, genetics and science. In order for these models to accurately describe reality they must be consistent with one another.

The problem which science has had with entropy is, in my opinion, largely due to thinking of entropy in terms of human ignorance. In thermodynamics it is the amount of ignorance we still have of the micro-state of a system, such as a container of gas, when all we know is the value of some macro-variables such as temperature and pressure.

I think this research may be telling us that although our scientific models specify entropy or our ignorance of quantum systems this ignorance is also a property of the actual quantum reality. That is the quantum wave function of the entangled system specifies only a probability distribution for the value of any measurable quantity; a probability distribution having entropy.

The article uses the example of a cup of coffee slowly coming to an equilibrium temperature with that of the air in the room. From the view of thermodynamics the system is coming to a state of higher entropy as required by the second law of thermodynamics. That is our uncertainty of the microstate of the coffee and air system is increasing as an equilibrium state has many more possible states than does a state where the coffee and air are out of equilibrium.

The new research argues that from the view of the quantum system uncertainty is increased as the coffee's quantum wave function becomes entangle with a growing number of air molecules. As the wave function grows in the extent of the reality it encompasses it grows in uncertainty as to the exact state of the entire system.

We have been famously confused over the nature of the quantum wave function since it was first conceived a hundred years ago. Some researchers believe it has no physical embodiment while others argue that it must have an instantiation in reality. Some current research, which may be conclusive, argues that the quantum wave function must be real (see for example Lucien Hardy's paper: Are quantum states real?) . 

If we understand the quantum wave function to be embodied in an actual physical form, it must be one independent of human construction. In terms of the cup of coffee example the quantum wave function may be said to become more uncertain as the knowledge contained in its model stretches to describe not just the coffee but also a growing amount of the cooling air in its environment.

 This view points to an analogy between quantum systems and biological systems; genetics also forms a probabilistic model independent of human construction.  Clearly the object of genetics is to better understand the world in terms of the adaptations that will lead to reproductive success.

This view of nested,evolving hierarchies of knowledge mechanisms including quantum, biological and cultural systems may underline our relatedness to whole of reality. In following our cultural and scientific activities we are following an ageless quest of the universe, the quest to know itself.

If that is the case we should expect that our scientific ignorance of thermodynamic micro-states would have to be consistent with the ignorance of the quantum system itself. If it were otherwise either thermodynamics or quantum theory or both would be of little value in describing reality.