Physics-Informed Multiagent Reinforcement Learning for Distributed Multirobot Problems

Sebastián, Eduardo (Universidad de Zaragoza) ; Duong, Thai ; Atanasov, Nikolay ; Montijano, Eduardo (Universidad de Zaragoza) ; Sagüés, Carlos (Universidad de Zaragoza)

Resumen: The networked nature of multirobot systems presents challenges in the context of multiagent reinforcement learning. Centralized control policies do not scale with increasing numbers of robots, whereas independent control policies do not exploit the information provided by other robots, exhibiting poor performance in cooperative-competitive tasks. In this work, we propose a physics-informed reinforcement learning approach able to learn distributed multirobot control policies that are both scalable and make use of all the available information to each robot. Our approach has three key characteristics. First, it imposes a port-Hamiltonian structure on the policy representation, respecting energy conservation properties of physical robot systems and the networked nature of robot team interactions. Second, it uses self-attention to ensure a sparse policy representation able to handle time-varying information at each robot from the interaction graph. Third, we present a soft actor–critic reinforcement learning algorithm parameterized by our self-attention port-Hamiltonian control policy, which accounts for the correlation among robots during training while overcoming the need of value function factorization. Extensive simulations in different multirobot scenarios demonstrate the success of the proposed approach, surpassing previous multirobot reinforcement learning solutions in scalability, while achieving similar or superior performance (with averaged cumulative reward up to ×2 greater than the state-of-the-art with robot teams ×6 larger than the number of robots at training time). We also validate our approach on multiple real robots in the Georgia Tech Robotarium under imperfect communication, demonstrating zero-shot sim-to-real transfer and scalability across number of robots.

Idioma: Inglés

DOI: 10.1109/TRO.2025.3582836

Año: 2025

Publicado en: IEEE Transactions on Robotics 41 (2025), 4499-4517
ISSN: 1552-3098

Financiación: info:eu-repo/grantAgreement/EUR/AEI/TED2021-130224B-I00
Financiación: info:eu-repo/grantAgreement/ES/DGA/T45-23R
Financiación: info:eu-repo/grantAgreement/ES/MCIU/FPU19-05700
Financiación: info:eu-repo/grantAgreement/ES/MICINN/PID2021-125514NB-I00

Tipo y forma: Artículo (Versión definitiva)

Área (Departamento): Área Ingen.Sistemas y Automát. (Dpto. Informát.Ingenie.Sistms.)

Dataset asociado: The supplementary video is a supporting document to the article ( 10.1109/TRO.2025.3582836/mm1)