Back to course

Markov Chain Probability

A Markov chain models a system moving between states, where the next state depends only on the current one:

The Markov property
P(Xt+1=j∣Xt=i,Xt−1,…)=P(Xt+1=j∣Xt=i)P(X_{t+1} = j \mid X_t = i, X_{t-1}, \ldots) = P(X_{t+1} = j \mid X_t = i)

The future depends on where you are and not on the path that got you there. Everything else follows from that.

This is the Markov property, and it is a strong assumption: the present state contains everything relevant about the past. It is also the assumption that makes analysis tractable, because the entire system is then described by one matrix.

Pij=P(Xt+1=j∣Xt=i)P_{ij} = P(X_{t+1} = j \mid X_t = i)

Each row of the transition matrix PP sums to 1, since the system must go somewhere.

Evolving the distribution

If πt\pi_t is the row vector of state probabilities at time tt:

The rest of this lesson is for subscribers

Unlock every lesson in Fundamentals of Probability and Statistics, and every other premium course.

Subscribe to continue

Test your knowledge

Questions are only available to subscribers.
Continue
NextBayesian Philosophy

Keep reading Fundamentals of Probability and Statistics

41 lessons in this course, and every other premium course, on one subscription.

Subscribe now
  • Every lesson in every course, with the worked examples and interactive simulators
  • Graded questions on every lesson, with explanations for the wrong answers as well as the right one
  • The trainers, timed assessments and brainteaser library that go with them