LANGUAGE-BASED EXPLAINABILITY OF ERRORS MADE BY COMPUTER VISION MODELS

Abstract: a system includes: a model configured to determine uncertainties of results of a task based on images, respectively; a classification module configured to selectively classify the images into a first category or a second category based on the uncertainties, respectively; an embedding module configured to: determine first embeddings based on the images using an embedding function; determine second embeddings for textual explanations of errors, respectively, using the embedding function; a clustering module configured to: cluster first ones of the first embeddings for images classified in the first category into first clusters; cluster second ones of the first embeddings for images classified in the second category into second clusters; and an explanation module configured to: determine similarities between each of the first and second clusters and each of the textual explanations; and determine k of the textual explanations for one of the second clusters based on the similarities.

Inventor(s): Gabriela Csurka Khedari, Tyler Hayes, Diane Larlus, Riccardo Volpi

CPC Classification: G06N3/08 (Learning methods)

Search for rejections for patent application number 20250190777