What Is the Intent behind Online Image and Video Postings?

Images and videos are posted in large quantities daily on Facebook’s platform. Serge Belongie, Cornell Tech/Computer Science, is working with the social media giant to investigate the problem of recognizing the intent behind an image or video posting. Is it malicious? Fake news? Polarizing? Hateful?

Monitoring and filtering out harmful content manually is not scalable and therefore not practical for companies like Facebook, given the sheer amount of data posted on their sites. Belongie, with researchers at Facebook, is defining the taxonomy of intents, as well as building an ontological knowledge graph of these intents. For example, what are the subjects, predicates, and objects involved in each intent category? The goal is to provide a knowledge graph representative of the many intents on Facebook’s platform.

Belongie is also applying fine-grained image detection methods, previously used in such applications as Visipedia and iNaturalist, to Facebook’s ID forgery detection. The ID forgery detection problem is extremely fine-grained, involving the ability to tell apart minute details such as a curly comma versus a straight comma and the subtle differences between fonts. The collaborative team is investigating how different physics-based deep learning can be used to infer the material on IDs and how to extend these fine-grained capabilities to video.

Cornell Researchers

Funding Received

$1.77 Million spanning 3 years

Sponsored by