The MediaEval Multimedia Evaluation benchmark was founded in 2008 as VideoCLEF and in 2011 became an independent benchmarking initiative. Each year it offers tasks that are related to multimedia retrieval, analysis, and exploration.

This year MediaEval offered 12 tasks:

• Emotional Mario: Believable AI agents in video games
• Emotions and Themes in Music
• FakeNews: Corona virus and 5G conspiracy
• Flood-related Multimedia
• Insight for Wellbeing: Multimodal personal health lifelog data analysis
• Medico
• NewsImages: The role of images in online news
• No-Audio Multimodal Speech Detection Task
• Pixel Privacy: Quality Camouflage for Social Images
• Predicting Media Memorability
• Scene Change: Fun faux photos
• Sports Video Classification

A COVID-19-related task could not be missing as the misinformation around the coronavirus has been of great concern. The ‘FakeNews: Corona virus and 5G conspiracy’ task was introduced and was related to misinformation claims that the construction of the 5G network and the associated electromagnetic radiation triggers the SARS-CoV-2 virus.

Two subtasks were formed and the participants could participate in either one or both of them. The first subtask, NLP-Based Fake News Detection, encouraged participants to build a binary classifier that can predict whether a Tweet contains the disinformation described above or whether it only accidentally contains the two buzzwords. The second subtask, Structure-Based Fake News Detection, referred to the subgraph of the Twitter follower network, which contains the nodes (Twitter users) who shared the respective tweet aiming to find disinformation spreaders.

17 teams submitted their approaches for the Text-based subtask and the winning team achieved a score of 0.603 of Matthews Correlation Coefficient (MCC), which was the official evaluation metric of the task. With regards to the Structure-Based subtask, nine teams participated with the dominant one achieving a score of 0.409, also measured using the MCC score.

A variety of methods were submitted, ranging from simple text-based approaches using the frequency of words, extraction of character and word-based 𝑛-grams from the text of the tweets as features for classification models to more complex approaches of ensemble learning incorporating deep learning-based text and image classifiers, bidirectional LSTM and Transformers and additionally an approach was submitted generating artificial examples to investigate the use of generative models in order to artificially augment and balance the datasets. The best performing approaches are based on extracting embeddings from transformer models and training neural networks. For instance, the authors of the second top approach (no participation paper was submitted by the team that achieved the best score) compared two transformer models and observed that the BERT transformer pre-trained on COVID tweets performs better than the vanilla version. The RoBERTa large model was preferred by the top third approach because its tokenizer is expected to be more suited for Twitter content.

This was our first attempt at this problem, leveraging an off-the-shelf solution to the problem (BERT) without investing a lot of development and computational resources. Our approach achieved an MCC score of 0.413, which was approximately the median performance among the 17 participating teams. The main challenge in building our solution arose as a result of the scarcity of training data to build accurate and robust classifiers. Additionally, the fact that BERT transformer pre-trained on COVID tweets performed better than the vanilla version indicates that domain-specific data are always beneficial. To address data scarcity, we intend to experiment with data augmentation approaches. Furthermore, we aim to investigate more transformer models and ensemble learning techniques to further increase classifier accuracy.

The event took place online on December 14th where we presented our approach ‘MeVer team tackling Corona virus and 5G conspiracy using ensemble classification based on BERT’. The presentation is available online at slideshare and YouTube.