Partially Observable Markov Decision Process (POMDP) in AI

A Partially Observable Markov Decision Process (POMDP) is a mathematical framework for decision-making under uncertainty when an agent cannot directly observe the complete state of its environment. Instead, the agent receives observations that provide partial or noisy information about the underlying state. POMDPs are used in areas such as robotics, healthcare, finance and autonomous systems.

Components

• States (S): A finite set of states representing all possible conditions the environment can be in.
• Actions (A): A finite set of actions available to the agent.
• Transition Model (T): A function T(s, a, s′) = P(s′ | s, a) that defines the probability of transitioning from state s to state s′ after taking action a.
• Observations (O): A finite set of observations that the agent can receive from the environment.
• Observation Model (Z): A function Z(s′, a, o) = P(o | s′, a) that defines the probability of receiving observation o after taking action a and reaching state s′.
• Rewards (R): A function R(s, a) that assigns a numerical reward to taking action a in state s.
• Discount Factor (γ): A factor, usually between 0 and 1, that determines how much future rewards are discounted compared with immediate rewards.

Mathematical Framework

A POMDP represents decision-making as a sequence of actions, state transitions, observations and rewards.

At each time step, the agent:

1. Maintains a belief about the current state.
2. Chooses an action based on this belief.
3. The environment transitions to a new state according to the transition model.
4. The agent receives a reward and an observation based on the new state.
5. The agent updates its belief using the new observation.

The agent does not know the exact state of the environment. Instead, it maintains a belief state, which is a probability distribution over all possible states. The belief is updated using Bayes' rule whenever a new observation is received.

The key challenge in a POMDP is that the agent does not know its exact state but has a belief or probability distribution over the possible states. This belief is updated using the Bayes' rule as new observations are made, forming a belief update rule:

b'(s') = \eta P(o \mid s', a) \sum_s T(s' \mid s, a)b(s)

Where:

• b(s) is the current belief that the environment is in state s.
• b′(s′) is the updated belief that the environment is in state s′.
• T(s′ | s, a) is the transition probability.
• P(o | s′, a) is the probability of receiving observation o in state s′ after taking action a.
• η is a normalization factor that makes the updated belief probabilities sum to 1.

Strategies for Solving

POMDPs pose significant challenges in environments where agents have incomplete information. Solving POMDPs involves optimizing decision-making strategies under uncertainty, crucial in many real-world applications. This overview highlights key strategies and methods for addressing these challenges.

Belief State Representation

In a POMDP, the agent represents its uncertainty about the environment using a belief state, which is a probability distribution over possible states. The belief is updated after each action and observation using the transition and observation models.

Solving Techniques:

1. Value Iteration: Extends value iteration to the belief space and represents the value function using a piecewise-linear and convex representation. Exact methods can become computationally expensive as the number of states and observations increases.
2. Point-Based Methods: Methods such as Point-Based Value Iteration (PBVI) and Perseus evaluate a selected set of belief states instead of the entire belief space, reducing computational cost while providing approximate solutions.
3. Approximate Methods: Methods such as QMDP and Fast Informed Bound (FIB) approximate the POMDP value function to make planning more computationally tractable.
4. Monte Carlo Methods: Methods such as Partially Observable Monte Carlo Planning (POMCP) and DESPOT use simulation and tree search to make decisions under uncertainty, particularly for large or complex POMDPs.

Example: Exploring Maze Navigation in Python

• A simple maze can be used to demonstrate the main concepts of a POMDP. In this example, an agent moves through a 5 × 5 maze and tries to reach a goal while receiving noisy observations of its position.
• The agent maintains a belief state, which is a probability distribution over all possible positions. It uses this belief state to choose actions and updates it whenever a new observation is received.
• The maze has 25 possible positions as states, four possible actions, several obstacles and a goal. The agent receives the correct position with 80% probability; otherwise, it receives an incorrect observation.

Step 1: Define the POMDP Environment

First, define the states, actions, obstacles, goal, transition model, observation model and reward function.

import random

class MazePOMDP:
    def __init__(self, maze_size, observation_accuracy=0.8):
        self.maze_size = maze_size
        self.states = [
            (x, y)
            for x in range(maze_size)
            for y in range(maze_size)
        ]
        self.actions = ["up", "down", "left", "right"]
        self.obstacles = {(1, 1), (2, 2), (3, 3)}
        self.goal = (maze_size - 1, maze_size - 1)
        self.observation_accuracy = observation_accuracy

    def transition(self, state, action):
        x, y = state

        if action == "up":
            next_state = (x - 1, y)
        elif action == "down":
            next_state = (x + 1, y)
        elif action == "left":
            next_state = (x, y - 1)
        elif action == "right":
            next_state = (x, y + 1)
        else:
            return state

        if (
            next_state[0] < 0
            or next_state[0] >= self.maze_size
            or next_state[1] < 0
            or next_state[1] >= self.maze_size
        ):
            return state

        if next_state in self.obstacles:
            return state

        return next_state

    def observation(self, actual_state):
        if random.random() < self.observation_accuracy:
            return actual_state

        possible_states = [
            state for state in self.states
            if state != actual_state
        ]

        return random.choice(possible_states)

    def observation_probability(self, observation, state):
        if observation == state:
            return self.observation_accuracy

        return (1 - self.observation_accuracy) / (len(self.states) - 1)

    def reward(self, state):
        if state == self.goal:
            return 10

        return -1

The transition() method prevents the agent from moving outside the maze or into an obstacle. The observation() method introduces noise so that the agent does not always know its exact position.

Step 2: Define the Belief-State Functions

The agent uses a belief state to represent its uncertainty about its current position.

def normalize_belief(belief):
    total = sum(belief.values())

    if total == 0:
        probability = 1 / len(belief)
        return {
            state: probability
            for state in belief
        }

    return {
        state: probability / total
        for state, probability in belief.items()
    }

def predict_belief(pomdp, belief, action):
    predicted_belief = {
        state: 0.0
        for state in pomdp.states
    }

    for state, probability in belief.items():
        next_state = pomdp.transition(
            state,
            action
        )
        predicted_belief[next_state] += probability

    return predicted_belief

def update_belief(pomdp, belief, observation):
    new_belief = {}

    for state in pomdp.states:
        observation_prob = pomdp.observation_probability(
            observation,
            state
        )

        new_belief[state] = (
            observation_prob * belief[state]
        )

    return normalize_belief(new_belief)

The belief is first predicted using the transition model and then updated using the new observation.

Step 3: Define Action Selection and Maze Visualization

For simplicity, the example uses a heuristic that selects an action that moves the most likely state closer to the goal.

def choose_action(pomdp, belief):
    most_likely_state = max(
        belief,
        key=belief.get
    )

    best_action = None
    best_distance = float("inf")

    for action in pomdp.actions:
        next_state = pomdp.transition(
            most_likely_state,
            action
        )

        distance = (
            abs(next_state[0] - pomdp.goal[0])
            + abs(next_state[1] - pomdp.goal[1])
        )

        if distance < best_distance:
            best_distance = distance
            best_action = action

    return best_action

def print_maze(pomdp, actual_state, most_likely_state):
    for i in range(pomdp.maze_size):
        for j in range(pomdp.maze_size):
            position = (i, j)

            if position == actual_state:
                print("A", end=" ")
            elif position == pomdp.goal:
                print("G", end=" ")
            elif position in pomdp.obstacles:
                print("X", end=" ")
            elif position == most_likely_state:
                print("?", end=" ")
            else:
                print(".", end=" ")

        print()

• A represents the actual position of the agent.
• G represents the goal.
• X represents an obstacle.
• ? represents the position with the highest belief probability.
• . represents other positions.

Step 4: Initialize the POMDP and Belief State

Create the maze and initialize the agent's starting position and belief state.

maze_size = 5
observation_accuracy = 0.8

pomdp = MazePOMDP(
    maze_size,
    observation_accuracy
)

actual_state = (0, 0)

belief = {
    state: 0.0
    for state in pomdp.states
}

belief[actual_state] = 1.0

num_steps = 15

The agent starts at (0, 0) and initially has complete belief that it is at that position. As it moves and receives observations, this belief is updated.

Step 5: Run the POMDP Simulation

Now, run the agent through the maze. At each step, it selects an action, transitions to a new state, receives an observation and updates its belief.

for step in range(num_steps):
    action = choose_action(
        pomdp,
        belief
    )

    actual_state = pomdp.transition(
        actual_state,
        action
    )

    observation = pomdp.observation(
        actual_state
    )

    predicted_belief = predict_belief(
        pomdp,
        belief,
        action
    )

    belief = update_belief(
        pomdp,
        predicted_belief,
        observation
    )

    most_likely_state = max(
        belief,
        key=belief.get
    )

    reward = pomdp.reward(
        actual_state
    )

    print(f"Step: {step + 1}")
    print(f"Action: {action}")
    print(f"Actual State: {actual_state}")
    print(f"Observation: {observation}")
    print(f"Most Likely State: {most_likely_state}")
    print(f"Reward: {reward}")

    print("Maze:")
    print_maze(
        pomdp,
        actual_state,
        most_likely_state
    )

    print("-" * 30)

    if actual_state == pomdp.goal:
        print("Goal reached!")
        break

Output:

• The exact observations may differ between runs because the observations are randomly generated. Even when an observation is incorrect, the agent can update its belief by combining the observation with its previous belief and the transition model.
• This demonstrates the main idea of a POMDP: the agent does not directly know the true state of the environment. Instead, it maintains a belief distribution over possible states and updates this belief as it takes actions and receives new observations.

Note: The action-selection method used here is a simple heuristic for demonstrating the POMDP concept. It is not a complete POMDP solver such as Point-Based Value Iteration (PBVI) or Partially Observable Monte Carlo Planning (POMCP).

You can download the complete source code from here.

Markov Decision Process vs. POMDP