Updating Your Beliefs: Bayesian Updating

Spread the love

DISCLAIMER: Please note that the information provided on this blog is for general informational purposes only and is not intended as professional advice. This blog is part of Wiseyak’s knowledge sharing sessions; the views and opinions expressed on this blog are those of the authors and do not necessarily reflect the official policy or position of WiseYak.

Each of us holds a set of beliefs that we have formed as a result of our experiences. Every time when we need to make a decision, we tend to base it on those belief i.e. our attitude towards the world at that given time. Time is a very important dimension here because with time we update our beliefs. Although the ability to always search for the motivation to change or update self-beliefs are important for us to make a better decision yet human factors like ego, blind superstition, etc comes as a tyrannical force to prevent us from changing or updating our self-beliefs.

The idea of updating beliefs in light of new evidence have also become an important part of the scientific process, Karl Popper refers this kind of occurrence as the Idea of Falsification. For instance, our previous view regarding the form of the Earth was that it was flat, but with the advent of new scientific tools, we can today conclude that it is roughly a sphere. The point is that even if the world was thought to be flat at the time, we nevertheless built roads, railroads, and other infrastructures given that the earth was flat. Therefore, we cannot hold individuals from the past who held this notion to be true responsible; as it was still a useful belief at that given time. And later we updated our beliefs, and maybe in the future from a different dimension (if exists) Earth’s form can be different from the sphere.

Updating Beliefs about Ghost!!

(Just a toy example)

Scenario

Assume you are moving into a new house; this is how the opening scene of the majority of horror movies start. You had some strange experiences and made the decision to keep watch every night for a few days to see what would happen. We are going to start with our prior belief that we had about the ghost, incorporate the evidence we collect everyday and form new opinions about the existence of the ghost. We can reflect these prevailing notions mathematically;

Bayesian formula: Quantifying and Updating Beliefs

$P(\theta|evidence)=\frac{P(\theta)*P(evidence|\theta)}{P(evidence)}$

If you peek at the right side of our equation; this side incorporates both our existing beliefs and the likelihood of the evidence (data); $P(\theta)$ is our existing belief on $\theta$ which is called a Prior and P( $evidence|\theta )$ is our likelihood of the evidence given the prior belief. Thus the left side P( $\theta | evidence$ )is our updated belief on $\theta$ . For instance, back to our example; we can assume the parameter $\theta$ as the probability of the ghost.

Using Bayesian approach

The unknown probability of ghost( $\theta$ ) would be treated as a random variable and given a distribution, but we already have some existing beliefs on the presence of ghosts. In bayesian Statistics they are known as Prior and can be expresses as a distribution; Let’s plot some of the Prior beliefs before we discuss its underlying distributions.

Figs: Prior Distributions

The above plots are some of our prior distributions which will be explained below, the shaded region indicates that the mass of probability is in that region.

Prior from Fig 1 (first from the left)** implies that I am new to the planet or that I have no awareness of these phenomena because no one has ever told me about them. The first prior is hence neutral and any chances on probability of ghost from 0 to 1 is equally likely. However, we can do better because we all have some beliefs about the ghost; the rest of these priors suggest that we all hold certain beliefs about ghosts and their existence.

Prior from Fig 2 (second from the left)** suggests that I am more likely to believe that ghosts don’t exist; there are, however, some chances.

Similarly, Prior from Fig 3 suggests that there is a 50/50 possibility of ghost.

Prior from Fig 4 suggests that “I have greater beliefs on ghosts,” ; it’s like saying I believe in Ghost but i have not meet one.

Note: This is just a part of an imaginary exercise; if you are doing a generalizable study you would want to use priors based on some scientific theory or an existing experiment. Or any other priors depending upon the type of your study.

Bayesian Updating

Moving on with “I kind of don’t believe in ghosts”(second from the left). This suggests that the our prior belief on absence of ghosts ( $\theta$ =0) is extremely likely and existence of ghosts ( $\theta$ =1) is extremely implausible. This does not indicate that there is no chances of ghost, just because the absence of ghosts ( $\theta$ =0) is quite likely. It only suggests that the existence of ghosts ( $\theta$ =1) is exceedingly improbable (see the figure above; it still has some mass around its tail i.e towards 1).

Let’s now enter the monitoring phase and begin gathering and documenting each of our paranormal encounters. Every day is like flipping a coin (Bernoulli trials): if we experience some paranormal activity, we record 1; otherwise, we record 0. See the mathematical instructions at the end of the article.

Fig: Updating our Beliefs

We begin with our preexisting belief about the ghost; if we experience any paranormal experiences, such as discovering someone under the bed, we then shift our beliefs to the right ( $\theta$ =1); But since it is a bayesian shift we obtain the full distribution i.e the entire mass of the probability. For instance, on Day 2, the dashed distribution represents our prior distribution whereas the solid-line distribution represents our updated belief. But an intriguing fact is that we will use this updated distribution on day 2 as our prior distribution on day 3 while we monitor the data and revise our opinions.

Change In beliefs

The process of updating continues until day 9, which is the present. Using the same Bayesian formula we previously encountered, we compute our updated distribution for each day. The updated distribution are called the posterior distribution.

Fig: Updated Beliefs

The black dotted distribution reflects our initial prior belief regarding ghosts; however, the black solid line shows our current beliefs, which lean more toward a 50/50 chance of ghost. Our understanding of the ghost has been updated as a result of our experience, and based on the updated distribution (posterior distribution), it appears that it is now time to call the priests.

These systems for updating beliefs are hardwired into our brain. This is how we form an opinion on something too. The more strongly we feel about some opinion, the more difficult it is to change our minds. Even though this occurs automatically inside our brains, teaching it to our computers—is a little more tricky. You can use any programming language of your choice but the underlying mathematics are the same and are explained next.

Math Section

Although there are many methods for computing posterior distribution (updated belief), at its heart, it just uses the Bayesian formula that we had previously encountered. We could use different approaches like MCMC, analytical solutions and quadratic approximations depending upon the complexity of the problem. Here, we are computing the posterior, or the updated beliefs, using an analytical solution (Beta-Bernoulli Conjugacy).

Since everything that can be expressed mathematically can be coded into computers, we will attempt to express our beliefs through mathematical expressions.