This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.

Search for Publication

Year(s) from:  to 
Keywords (separated by spaces):

Query-adaptive Video Summarization via Quality-aware Relevance Estimation

Arun Balajee Vasudevan, Michael Gygli, Anna Volokitin, Luc Van Gool
ACM Multimedia 2017
Mountain View, CA, USA, October 2017


Although the problem of automatic video summarization has recently received a lot of attention, the problem of creating a video summary that also highlights elements relevant to a search query has been less studied. We address this problem by posing query-relevant summarization as a video frame subset selection problem, which lets us optimise for summaries which are simultaneously diverse, representative of the entire video, and relevant to a text query. We quantify relevance by measuring the distance between frames and queries in a common textual-visual semantic embedding space induced by a neural network. In addition, we extend the model to capture query-independent properties, such as frame quality. We compare our method against previous state of the art on textual-visual embeddings for thumbnail selection and show that our model outperforms them on relevance prediction. Furthermore, we introduce a new dataset, annotated with diversity and query-specific relevance labels. On this dataset, we train and test our complete model for video summarization and show that it outperforms standard baselines such as Maximal Marginal Relevance.

Download in pdf format
  author = {Arun Balajee Vasudevan and Michael Gygli and Anna Volokitin and Luc Van Gool},
  title = {Query-adaptive Video Summarization via Quality-aware Relevance Estimation},
  booktitle = {ACM Multimedia 2017},
  year = {2017},
  month = {October},
  keywords = {}