EXPLORER: Exploration-guided Reasoning for Textual Reinforcement Learning: Symbolic Policy Learner

Table of Links

Abstract and 1 Introduction

2 Background

3 Symbolic Policy Learner

3.1 Learning Symbolic Policy using ILP

3.2 Exception Learning

4 Rule Generalization

4.1 Dynamic Rule Generalization

5 Experiments and Results

5.1 Dataset

5.2 Experiments

5.3 Results

6 Related Work

7 Future Work and Conclusion, Limitations, Ethics Statement, and References

3 Symbolic Policy Learner

Deep reinforcement learning (DRL) has experienced great success by learning directly from high-dimensional sensory inputs, yet it suffers from a lack of interpretability. Interpretability of an agent’s action is of utmost importance in sequential decision-making problems, as it increases the transparency of black-box-style agents; it also helps RL researchers understand the high-level behavior of the system better. To make a system interpretable, one of the most widely used approaches is learning the agent’s policies symbolically. In our work, EXPLORER learns these symbolic policies in the form of logical rules represented in the ASP. An example of such a rule is - insert(X, fridge) :- apple(X) which can be translated as “X is insertable into a fridge if X is an apple” [2]. These learned ASP rules not only provide a better understanding of the system’s functionality but can also be used to predict the agent’s next action using an ASP solver. EXPLORER learns the rules iteratively (in an online manner) and applies the rules to predict an action in collaboration with the neural module. Our results show that this approach is very effective in terms of performance and interpretability.

Partial Observability and Non-Monotonic Reasoning: EXPLORER works in a partially observable environment, where it needs to predict an action based on its prior knowledge. If EXPLORER fails, then it learns something new that will be applied in the next episode. The reasoning approach of EXPLORER is non-monotonic in nature: that is, what it believes currently may become false in the future with new evidence. We can model this using a non-monotonic logic programming paradigm that supports default rules and exception to defaults (Gelfond and Kahl, 2014). In this work, the belief of EXPLORER has been represented as an Answer Set Program in the form of default rules with exceptions. With the help of Inductive Logic Programming (ILP) (see Section 3.1) and Exception learner (see Section 3.2), these rules are learned by EXPLORER after each episode and then applied in the following episode. The agent uses an ASP solver to predict actions by utilizing the observation and the rules. Based on the outcome after applying the rules, the learned policies are updated with the exception (if needed), and new rules are learned as needed.

[2] For ease of use, we retain action names as the predicate names; however, they are interpreted normally as adjectives.