Discrete variational calculus for accelerated optimization

M. Campos, Cédric ; Mahillo, Alejandro (Universidad de Zaragoza) ; Martín de Diego, David

Resumen: Many of the new developments in machine learning are connected with gradient-based optimization methods. Recently, these methods have been studied using a variational perspective (Betancourt et al., 2018). This has opened up the possibility of introducing variational and symplectic methods using geometric integration. In particular, in this paper, we introduce variational integrators (Marsden and West, 2001) which allow us to derive different methods for optimization. Using both Hamilton’s and Lagrange-d’Alembert’s principle, we derive two families of optimization methods in one-to-one correspondence that generalize Polyak’s heavy ball (Polyak, 1964) and Nesterov’s accelerated gradient (Nesterov, 1983), the second of which mimics the behavior of the latter reducing the oscillations of classical momentum methods. However, since the systems considered are explicitly time-dependent, the preservation of symplecticity of autonomous systems occurs here solely on the fibers. Several experiments exemplify the result.

Idioma: Inglés

Año: 2023

Publicado en: JOURNAL OF MACHINE LEARNING RESEARCH 24 (2023), 1-33
ISSN: 1532-4435

Originalmente disponible en: Texto completo de la revista

Factor impacto JCR: 4.3 (2023)

Categ. JCR: AUTOMATION & CONTROL SYSTEMS rank: 21 / 84 = 0.25 (2023) - Q1 - T1
Categ. JCR: COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE rank: 54 / 197 = 0.274 (2023) - Q2 - T1

Factor impacto CITESCORE: 18.8 - Software (Q1) - Artificial Intelligence (Q1) - Control and Systems Engineering (Q1) - Statistics and Probability (Q1)

Factor impacto SCIMAGO: 2.796 - Artificial Intelligence (Q1) - Statistics and Probability (Q1) - Software (Q1) - Control and Systems Engineering (Q1)

Tipo y forma: Artículo (Versión definitiva)

Área (Departamento): Área Análisis Matemático (Dpto. Matemáticas)