Let’s Make a Cinephile Search Tool!

The second data challenge hosted by the Data Competence Center HERMES focuses on improving access to cultural audiovisual data. The goal is to develop a multimodal search tool that enables searching through videos and metadata from the following European institutions:

Deutsches Filminstitut und Filmmuseum (German Film Institute and Film Museum)
Das Bundesarchiv (Federal Archives of Germany)
Nederlands Instituut voor Beeld en Geluid (Netherlands Institute for Sound and Vision)

The Challenge
Film and media scholars increasingly work with large and complex digital datasets related to moving images. In addition to examining metadata—such as crew members, production dates, and locations—they also conduct visual and audio analyses. The exponential growth of both public and private audiovisual archives makes it nearly impossible to manually browse through all available videos when looking for specific visual or sound patterns.

Yet tools for multimodal search in these data collections remain scarce. No commonly available application currently allows users to search local or institutional video collections for clips showing, for instance, a woman operating machinery in a historical film within a specific national context. Nor is it possible to upload an image and retrieve all scenes with a similar visual composition.

This is where you come in. The aim of this challenge is to create exactly such a comprehensive multimodal search tool.

Your Mission
Develop a search system that can perform at least two of the following tasks (the more, the better):

• Search by metadata (creators, date, location)
• Shot type detection (e.g., wide shot, close-up)
• Transition detection (e.g., cuts)
• Object, person, action, and location recognition in scenes
• Scene search based on an uploaded image (visual similarity)
• Bonus: Combine all features into a chatbot that can answer complex queries, such as:
“Find a scene from the 1940s in Germany where a woman is operating a machine in a medium close-up.”
“Find a film where a shot of a soldier transitions into a close-up of his face.”

Who Can Participate?
Students, PhD candidates, academics, and researchers in the fields of film studies, digital humanities, and data science. Teams should combine both technical and domain expertise. During the kickoff event, there will be an opportunity to form interdisciplinary teams.

Timeline:

May 25, 2025 – Kickoff (Zoom)
By May 31, 2025 – Team registration
June–August 2025 – Development phase
By August 31, 2025 – Submission deadline 
December 1, 2025 – Results announcement
March 2026 – Final presentation, Philipps University in Marburg (Depending on your location and team size, all or part of your travel expenses will be covered)

More information about the project, dataset, and registration here.​​