Considering Unseen States as Impossible in Factored Reinforcement Learning

Kozlova, Olga
LIP6 - Pôle IA
Université Paris 6
8, rue du capitaine Scott F-75015 Paris, France


Sigaud, Olivier
LIP6 - Pôle IA
Université Paris 6
8, rue du capitaine Scott F-75015 Paris, France
email: olivier.sigaud@lip6.fr   

Wuillemin, Pierre-Henri
LIP6 - Pôle IA
Université Paris 6
8, rue du capitaine Scott F-75015 Paris, France
email: pierre-henri.wuillemin@lip6.fr   home: www-desir.lip6.fr/~phw

Meyer, Christophe

Kozlova, Olga and Sigaud, Olivier and Wuillemin, Pierre-Henri and Meyer, Christophe (2009) "Considering Unseen States as Impossible in Factored Reinforcement Learning". In European Conference on Machine Learning, pp. 721--735, Springer.

The Factored Markov Decision Process (FMDP) framework is a standard representation for sequential decision problems under uncertainty where the state is represented as a collection of random variables. Factored Reinforcement Learning (FRL) is an Model-based Reinforcement Learning approach to FMDPs where the transition and reward functions of the problem are learned. In this paper, we show how to model in a theoretically well-founded way the problems where some combinations of state variable values may not occur, giving rise to impossible states. Furthermore, we propose a new heuristics that considers as impossible the states that have not been seen so far. We derive an algorithm whose improvement in performance with respect to the standard approach is illustrated through benchmark experiments.

@InProceedings{,
  author = {Kozlova, Olga and Sigaud, Olivier and Wuillemin, Pierre-Henri and Meyer, Christophe},
  title = {Considering Unseen States as Impossible in Factored Reinforcement Learning},
  booktitle = {European Conference on Machine Learning},
  year = {2009},
  pages = {721--735},
  publisher = {Springer}
}