Large Language Models (LLMs) are massive neural network architectures trained on vast amounts of data while being tasked to classify or generate text. These models have acquired a deep understanding of the intrinsic language structure as well as the world itself by analyzing web-scraped documents word after word, sentence after sentence for a long machine-time. Although LLMs have been around for several years and are popular among the scientific community since their advent in 2018 starting with the BERT model1, they have recently increased in scale and have reached impressive levels of human-like text understanding and generation capabilities. Additionally, they have been released to the general public through web services such as OpenAI’s chatGPT and Google’s Bard, and have generated a lot of buzz around Artificial Intelligence (AI), its tremendous capabilities as well as potential hazards that may arise by its use. Moreover, extended versions of them - called multimodal LLMs - that are able to also process visual content and answer questions related to it, are now available.

In the era of smartphones and social media, where everyone is able to capture the moment and share it online in seconds, knowing the origin of an image could make the difference between early debunking of a fabricated story and the continued spread of a false message. In this blog post, we will take a look at the task of identifying the source of an image, a key problem in the field of image provenance analysis. Also, we will present our findings on how most of the big digital platforms modify the popular JPEG standard and how we use these traces for identifying the source of images that circulate online without any prior knowledge or assumption.

In this post, we explain the basics behind our paper “DnS: Distill-and-Select for Efficient and Accurate Video Indexing and Retrieval”, which has been accepted for publication in the International Journal of Computer Vision (IJCV). Following our prior work in ViSiL, we realised that high-performing video retrieval systems require a significant amount of compute and storage resources. More precisely, our network achieved state-of-the-art performance on several video retrieval problems; however, it needed substantial time for similarity calculation and memory space for storing the video features.

Last year, we had the opportunity to conduct a desk research study in an effort to map the landscape of online disinformation in Europe. Our research was guided by three research questions: What are the measures and journalistic authorities in Europe that supervise and monitor the ethical application of journalism? Are there patterns of false information spreading as a means to serve the agenda and interests of certain political groups? Are there certain political groups that benefit as a result of given disinformation narratives? Are there specific and tangible policies that have been proposed by European and international organisations that could help tackle online disinformation? To make the study of the first two questions more manageable, we focused on five European countries: France, Germany, Greece, Italy and Spain, and on three topics that are commonly associated with disinformation: COVID-19, immigration, and climate change. The resulting report is available here and we believe it could be valuable for informing a number of stakeholders, including professionals in the media industry and policy makers. In the following, we try to present the key insights that we managed to draw from this report.

In this post we explain the basics behind our paper “Leveraging EfficientNet and Contrastive Learning for Accurate Global-scale Location Estimation” that has been accepted for publication at this year’s International Conference on Multimedia Retrieval (ICMR 2021). The problem that we tackle in this work is the following: “Given an input image from anywhere in the world, estimate the coordinates of its geographic location, in terms of latitude and longitude, based solely on the visual information of the image.”

In this post, we explain the basics behind our paper “Operation-wise Attention Network for Tampering Localization Fusion”, which has been accepted for publication at this year’s Content-Based Multimedia Indexing conference (CBMI 2021). The main motivation of this approach is to combine the outcomes of multiple image forensics algorithms and to provide a fused tampering localization map, which requires no expert knowledge and is easier to interpret by end users. Our fusion framework includes a set of five individual tampering localization methods for splicing localization on JPEG images:

The MediaEval Multimedia Evaluation benchmark was founded in 2008 as VideoCLEF and in 2011 became an independent benchmarking initiative. Each year it offers tasks that are related to multimedia retrieval, analysis, and exploration. This year MediaEval offered 12 tasks: Emotional Mario: Believable AI agents in video games Emotions and Themes in Music FakeNews: Corona virus and 5G conspiracy Flood-related Multimedia Insight for Wellbeing: Multimodal personal health lifelog data analysis Medico NewsImages: The role of images in online news No-Audio Multimodal Speech Detection Task Pixel Privacy: Quality Camouflage for Social Images Predicting Media Memorability Scene Change: Fun faux photos Sports Video Classification A COVID-19-related task could not be missing as the misinformation around the coronavirus has been of great concern. The ‘FakeNews: Corona virus and 5G conspiracy’ task was introduced and was related to misinformation claims that the construction of the 5G network and the associated electromagnetic radiation triggers the SARS-CoV-2 virus.

Towards the end of 2020, I had the opportunity to give a talk on Deepfakes in the 60th AI4EU WebCafe. With deepfakes being already with us for more than two years now, we have already witnessed several real-world cases of deepfakes with both positive and negative impact, while their use has recently been commoditized by popular apps such as ZAO and Reface. At the same time, it seems that there is an exponentially growing interest in the area from the academic community as indicated by the quickly growing number of papers related to the topic.

WeVerify is an EU granted project that aims to address the complex content verification challenges through a participatory verification approach, open-source algorithms, low-overhead human-in-the-loop machine learning and intuitive visualizations. The MeVer team leads the ‘Cross-modal Disinformation Detection and Content Verification’ work package and in collaboration with the University of Sheffield (USFD) have developed tools that will bring together features from text, images and videos to tackle the challenge of cross-modal disinformation detection and content verification.

In this post, we explain the basics behind our paper “Audio-based Near-Duplicate Video Retrieval with Audio Similarity Learning,” which has been accepted for publication at this year’s International Conference on Pattern Recognition (ICPR 2020). So, let’s consider the audio signals of two arbitrary videos as the ones displayed below. The left video serves as a query and the right as a target match.