Model-Based Actor-Critic for Multi-Objective Reinforcement Learning with Dynamic Utility Functions

• Many real-world problems require a trade-off between multiple conflicting objectives. Decision-makers’ preferences over solutions to such problems are determined by their utility functions, which convert multi-objective values to scalars. In some settings, utility functions change over time, and the goal is to find methods that can efficiently adapt an agent’s policy to changes in utility. Previous work on learning with dynamic utility functions has focused on model-free methods, which often suffer from poor sample efficiency. In this work, we instead propose a model-based actor-critic, which explores with diverse utility functions through imagined rollouts within a learned world model between interactions with the real environment. An experimental evaluation on Minecart, a well-known benchmark for multi-objective reinforcement learning, shows that by learning a model of the environment the quality of the agent’s policy is improved compared to model-free algorithms.

Subject headings

NATURVETENSKAP  -- Data- och informationsvetenskap -- Datavetenskap (hsv//swe)

NATURAL SCIENCES  -- Computer and Information Sciences -- Computer Sciences (hsv//eng)

Keyword

Multiple Objectives

Reinforcement Learning

Model-Based Learning

Publication and Content Type

ref (subject category)

kon (subject category)