This paper provides an overview of decision-theoretic planning (DTP) and Markov decision processes (MDPs), highlighting their connections and how they can be used to model and solve planning problems in artificial intelligence (AI). It discusses the structural assumptions and computational leverage in MDPs, emphasizing how specialized representations and algorithms can exploit these structures to achieve efficient planning. The paper surveys various types of representations and algorithms for classical and decision-theoretic planning problems, focusing on abstraction, aggregation, and decomposition techniques. It also explores the relationship between MDP solution algorithms and AI planning algorithms, demonstrating how AI methods can be applied to solve general MDPs. The paper aims to bridge the gap between MDPs and AI planning, making explicit the connections between the two fields and suggesting directions for future research.This paper provides an overview of decision-theoretic planning (DTP) and Markov decision processes (MDPs), highlighting their connections and how they can be used to model and solve planning problems in artificial intelligence (AI). It discusses the structural assumptions and computational leverage in MDPs, emphasizing how specialized representations and algorithms can exploit these structures to achieve efficient planning. The paper surveys various types of representations and algorithms for classical and decision-theoretic planning problems, focusing on abstraction, aggregation, and decomposition techniques. It also explores the relationship between MDP solution algorithms and AI planning algorithms, demonstrating how AI methods can be applied to solve general MDPs. The paper aims to bridge the gap between MDPs and AI planning, making explicit the connections between the two fields and suggesting directions for future research.