The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learning
Amii AI Seminar Series, University of Alberta
Abstract
Large language models (LLMs) and vision-language models (VLMs) enrich reinforcement learning (RL) with knowledge, communication, and reasoning—capabilities that standard RL lacks.
Our survey, “The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learning”, classifies the role of the foundation model (FM) into three categories: Agent, where the FM selects actions; Planner, where the FM decomposes long-horizon tasks into sub-goals; and Reward, where the FM supplies preference feedback or autonomously proposes, tests, and refines reward functions.