91Ó°ÊÓ

A stationary distribution is a probability distribution that remains unchanged as time progresses in a Markov chain. For a given Markov chain, if the probability of being in state \(i\) at time \(t\) is \(\pi_i\), then at any future time, the probability will still be \(\pi_i\). This means that once the chain reaches this distribution, it "settles" into it, regardless of the initial state.
Key features of a stationary distribution include:

• Doesn't change over time as the Markov chain evolves.
• Ensures the long-term behavior of the chain is predictable.
• Provides the weights for the expected long-time average of functions.

In an irreducible Markov chain, all states communicate meaning all states are part of one class, and there exists a unique stationary distribution provided that the chain is also positive recurrent.
In the context of the given exercise, the stationary distribution \(\pi\) represents the probabilities of lingering in each state over the long run, and it is vital for calculating the expected value of a function over time.

The Ergodic Theorem is a fundamental result for Markov chains that connects time averages and probabilistic averages. In the case of an irreducible and positive recurrent Markov chain, the theorem ensures that time averages of state functions converge to the expected value derived from the stationary distribution.
Let's break it down:

• **Irreducible**: Every state can be reached from every other state within a finite amount of time.
• **Positive Recurrent**: Each state is returned to infinitely often within a finite expected time.

The theorem highlights that the function's time average, when observed over a long period, will match the expected value computed using its stationary distribution. In terms of the exercise, it guarantees that the average observed behavior of the Markov chain aligns with the expected theoretical behavior dictated by \(\pi\).
This convergence happens almost surely, meaning with probability 1, this alignment between time averages and expected values is assured.

An irreducible Markov chain is a type of chain where every state can be reached from every other state within a finite number of steps. This implies strong connectivity across all states.

• There is a path of positive probability connecting any two states.
• All states belong to a single communicating class.
• An irreducible Markov chain has a unique stationary distribution if it is positive recurrent, meaning it consistently returns to states in finite expected time.

In the exercise context, the irreducibility of the Markov chain ensures that the entire state space \(S\) is navigable from any starting point, reinforcing the ability of all states to contribute to the stationary distribution.
This property is crucial for applying the Ergodic Theorem, as it guarantees a form of "fairness" where no state is neglected over time unless the Markov chain reaches null space, which doesn't happen in an irreducible positive recurrent Markov chain.

Time average convergence refers to how the average value of a bounded function applied to a Markov chain behaves as time progresses. More precisely, it reveals how the average tends toward a certain value. In the case of the Markov chain exercise, it specifies how the time average becomes equivalent to an expected value calculated with the stationary distribution.

• Convergence involves the calculated time average of the function \(u(X(s))\).
• As time \(t\) goes to infinity, the average behavior of the chain aligns with predictions based on the stationary distribution.
• Almost sure convergence indicates that this behavior will hold with probability 1, leaving very little room, if any, for deviation.

The transition from descriptive time averages to a theoretical expectation shows how theoretical models (like stationary distributions and ergodic theorems) in Markov chains apply to practical, long-term scenarios. This principle is at the heart of predicting and managing systems governed by stochastic processes.