Multi-label affordance mapping from egocentric vision

Mur-Labadia, Lorenzo (Universidad de Zaragoza) ; Guerrero, José J. (Universidad de Zaragoza) ; Martínez-Cantín, Rubén (Universidad de Zaragoza)

Resumen: Accurate affordance detection and segmentation with pixel precision is an important piece in many complex systems based on interactions, such as robots and assitive devices. We present a new approach to affordance perception which enables accurate multi-label segmentation. Our approach can be used to automatically extract grounded affordances from first person videos of interactions using a 3D map of the environment providing pixel level precision for the affordance location. We use this method to build the largest and most complete dataset on affordances based on the EPIC-Kitchen dataset, EPIC-Aff, which provides interaction-grounded, multi-label, metric and spatial affordance annotations. Then, we propose a new approach to affordance segmentation based on multi-label detection which enables multiple affordances to co-exists in the same space, for example if they are associated with the same object. We present several strategies of multi-label detection using several segmentation architectures. The experimental results highlight the importance of the multi-label detection. Finally, we show how our metric representation can be exploited for build a map of interaction hotspots in spatial action-centric zones and use that representation to perform a task-oriented navigation.

Idioma: Inglés

DOI: 10.1109/ICCV51070.2023.00483

Año: 2024

Publicado en: Proceedings (IEEE International Conference on Computer Vision) 2023 (2024), 5215-5226
ISSN: 1550-5499

Factor impacto SCIMAGO: 3.544 - Software - Computer Vision and Pattern Recognition

Financiación: info:eu-repo/grantAgreement/ES/DGA/T45-23R
Financiación: info:eu-repo/grantAgreement/ES/MICINN-AEI/PID2021-125209OB-I00
Financiación: info:eu-repo/grantAgreement/EUR/MICINN/TED2021-129410B-I00
Financiación: info:eu-repo/grantAgreement/EUR/MICINN/TED2021-131150B-I00

Tipo y forma: Artículo (PostPrint)

Área (Departamento): Área Ingen.Sistemas y Automát. (Dpto. Informát.Ingenie.Sistms.)