<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Papers | MeVer</title>
    <link>https://mever.gr/tag/papers/</link>
      <atom:link href="https://mever.gr/tag/papers/index.xml" rel="self" type="application/rss+xml" />
    <description>Papers</description>
    <generator>Wowchemy (https://wowchemy.com)</generator><language>en-us</language><lastBuildDate>Thu, 12 Jun 2025 00:00:00 +0000</lastBuildDate>
    <image>
      <url>https://mever.gr/media/logo_hu_ecbb58bff06221f9.png</url>
      <title>Papers</title>
      <link>https://mever.gr/tag/papers/</link>
    </image>
    
    <item>
      <title>Any-Resolution AI-Generated Image Detection by Spectral Learning</title>
      <link>https://mever.gr/post/any-resolution-ai-generated-image-detection-by-spectral-learning/</link>
      <pubDate>Thu, 12 Jun 2025 00:00:00 +0000</pubDate>
      <guid>https://mever.gr/post/any-resolution-ai-generated-image-detection-by-spectral-learning/</guid>
      <description>&lt;p&gt;AI-generated visual content already plays &lt;a href=&#34;https://home-affairs.ec.europa.eu/networks/eu-knowledge-hub-prevention-radicalisation/welcome-package/learning-resources/generative-ai-and-german-far-right-narratives-tactics-and-digital-strategies_en&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;a prominent role in spreading disinformation across Europe&lt;/a&gt;. As the decrease in the cost of computational power enables sophisticated open-source models to be crafted and adapted with minimal effort, safeguards implemented by major AI providers, while helpful, cannot suffice for a robust long-term solution. Therefore, a major research question is how AI can also be employed to robustly distinguish AI-generated imagery from authentic one.&lt;/p&gt;
&lt;p&gt;While several approaches for detecting AI-generated images have emerged, their common issue is the inability to maintain consistent detection performance on images generated by models not considered during their training. Consequently, keeping them effective requires maintaining an elaborate training dataset, which, apart from needing significant human labor, could also be completely intractable, due to the countless available generative models. Key to this limitation is the popular belief that one could detect generated imagery by identifying a set of inconsistencies that are common for all generative models. However, to date, the image forensics community has yet to discover any universal theoretical limitation that would hold for any generative model. Instead, we constantly see weaknesses that were common in some early models, like the inability to generate clear text, to be fixed by newer ones, like in the case of &lt;a href=&#34;https://cloud.google.com/blog/products/ai-machine-learning/announcing-veo-3-imagen-4-and-lyria-2-on-vertex-ai&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Google’s Imagen&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;To tackle this fundamental issue, we took a different path in our latest work. Instead of searching for general inconsistencies among the generated images, we opted to model the attributes of real images. We assume that the latter are not directly affected by the introduction of any single generative model, but primarily by long-term technological advances. Therefore, their distribution constitutes a more invariant pattern for the task of AI-generated image detection. In particular, we opted for the spectral distribution of real images, as recent literature has shown that the differences among real and AI-generated images are exceptionally visible in the spectral domain.&lt;/p&gt;
&lt;p&gt;Given a model of the spectral attributes of real images, it would be possible to detect the generated as the ones that do not conform to this learned model, i.e. as out-of-distribution samples. However, it was neither clear how such a spectral model could be learned, nor how to detect its out-of-distribution samples. Therefore, we had to introduce both in our Spectral AI-Generated Image (SPAI) detection framework.&lt;/p&gt;
&lt;p&gt;In SPAI, we employ a self-supervised setup to model the spectral distribution of real images. To achieve this, we use the pre-text task of frequency reconstruction. Then, to detect its out-of-distribution samples we introduce three learnable components. First, we hypothesize that the learned spectral model will better reconstruct the frequencies of the real images, i.e. its in-distribution samples. So, we introduce the concept of spectral reconstruction similarity to measure this reconstruction error. Next, we assume that the spectral information present in an image defines the importance of reconstruction errors for different frequency ranges. Thus, we craft the spectral context vector to capture the spectral context of the image. Finally, by introducing the spectral context attention mechanism we enable the efficient processing of any-resolution images without further pre-processing. This significantly increases the ability of a detector to capture subtle inconsistencies, crucial for classifying high-fidelity imagery. An overview of SPAI’s architecture is presented in Figure 2, while an in-depth technical analysis is available on our CVPR 2025 &lt;a href=&#34;https://arxiv.org/abs/2411.19417&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;paper&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/spai_1.png&#34; alt=&#34;spai_1&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Figure 2: We learn a model of the spectral distribution of real images under a self-supervised setup using masked spectral learning. Then, we use the spectral reconstruction similarity to measure the divergence from this learned distribution and detect AI-generated images as out-of-distribution samples of this model. Spectral context vector captures the spectral context under which the spectral reconstruction similarity values are computed, while spectral context attention enables the processing of any-resolution images for capturing subtle spectral inconsistencies.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;To evaluate the efficacy of SPAI, we composed a comprehensive benchmark of 13 recent generative approaches as well as real images originating from DSLR and smartphone cameras and the Web. As we see in Table 1, while previous detectors exhibit very high performance on specific GenAI models, they catastrophically fail on others. Instead, SPAI achieves consistently high detection performance across all the considered image generators.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/spai_2.png&#34; alt=&#34;spai_2&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Table 1: Comparison against state-of-the-art. SPAI generalizes across all the considered generative approaches, even on ones producing imagery of extreme fidelity.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;In Figure 3, we see that SPAI is capable of distinguishing among AI-generated and real images independently of the depicted topic or visual aesthetics, highlighting its capability to not overfit specific image semantics.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/spai_3.png&#34; alt=&#34;spai_3&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Figure 3: SPAI is able to distinguish AI-generated from real images that depict a diverse set of topics and incorporate different levels of visual fidelity and aesthetics.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Moreover, the spectral context attention provides a built-in mechanism to explain which image regions were the most important for the final decision. For example, we see on Figure 4 that not only SPAI could correctly classify this Stable Diffusion 3 image but was also capable of  identifying the 6-fingers inconsistency as important for this decision.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/spai_4.png&#34; alt=&#34;spai_4&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Figure 4: Qualitative analysis of spectral context attention. A cool-warm overlay has been applied on each patch. Red color indicates significant patches for deciding whether the image is AI-generated (high attention values), while blue color indicates irrelevant patches (low attention values).&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;On the other hand, while SPAI significantly improves over the previous state-of-the-art approaches, we believe that it would be premature for any approach to claim fully solving this extremely complex task. For example, generated images frequently circulate in the form of derivative images, such as in screenshots, external photos of a screen or even printed material. In these cases, the intermediate medium significantly distorts the image signal and limits the performance of any detector solely relying on the image signal. Therefore, further research is necessary on combining spectral learning with contextual understanding.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/spai_5a.png&#34; alt=&#34;spai_5a&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/spai_5b.png&#34; alt=&#34;spai_5b&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Figure 5: Top: Accurately detected AI-generated images shared online. Spectral context attention correctly identifies artifacts in the hand and the texture of the jacket (left), as well as on the face and the ear (right). Bottom: Derivative AI-generated images failed to be detected. The distortion introduced by the intermediate medium makes indistinguishable the spectral artifacts that were previously captured correctly.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;As we remain committed to open research, we open-source SPAI, to enable AI practitioners and researchers to build upon it towards creating a sustainable AI-driven ecosystem of digital media.&lt;a href=&#34;https://github.com/mever-team/spai&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Check out our project page&lt;/a&gt; for the source code, trained weights and data of SPAI.&lt;/p&gt;
&lt;p&gt;Our work was performed in close collaboration with the VIS lab of the University of Amsterdam and is included in the program of the &lt;a href=&#34;https://cvpr.thecvf.com/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2025&lt;/a&gt; (June 11th-15th 2025 in Nashville, USA). For those not attending, you can check out our &lt;a href=&#34;https://www.youtube.com/watch?v=ZA0Aa0o623I&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;online presentation&lt;/a&gt; and contact us for any further questions.&lt;/p&gt;
&lt;p&gt;We would like to thank the &lt;a href=&#34;https://elias-ai.eu/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;ELIAS&lt;/a&gt; and &lt;a href=&#34;https://www.veraai.eu/home&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;vera.ai&lt;/a&gt; Horizon Europe projects for actively supporting this effort, as well as &lt;a href=&#34;https://grnet.gr/en/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;GRNET&lt;/a&gt; for providing access to the required computational resources.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Reducing Inference Energy Consumption Using Dual Complementary CNNs</title>
      <link>https://mever.gr/post/reducing-inference-energy-consumption-using-dual-complementary-cnns/</link>
      <pubDate>Tue, 10 Dec 2024 00:00:00 +0000</pubDate>
      <guid>https://mever.gr/post/reducing-inference-energy-consumption-using-dual-complementary-cnns/</guid>
      <description>&lt;p&gt;In this post we present our dual CNN methodology, that reduces inference energy requirements of CNNs in image classification tasks on low-resource devices. Further details about our approach and associated experiments are included  in our paper &lt;a href=&#34;https://www.sciencedirect.com/science/article/pii/S0167739X24005703?dgcid=author&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Reducing inference energy consumption using dual complementary CNNs&lt;/a&gt;, which has been accepted for publication in the journal of Future Generation Computer Systems.&lt;/p&gt;
&lt;h2 id=&#34;introduction&#34;&gt;Introduction&lt;/h2&gt;
&lt;p&gt;Deep neural networks (DNNs) play a critical role in solving complex problems in smart environments, with growing applications in IoT, smart homes, and edge computing. However, deploying large DNNs on edge devices poses significant challenges due to their high energy demands, which can quickly deplete battery life and shorten device energy autonomy. Existing DNN compression techniques aim to address these issues by creating smaller models with fewer parameters and reduced computational requirements. While these methods enhance efficiency, they often come with trade-offs, including potential loss of accuracy and significant computational costs for re-training and fine-tuning. To address these limitations we propose a methodology that uses two small complementary CNNs, that combined they increased their predictive performance while still keeping energy consumption low.&lt;/p&gt;
&lt;h2 id=&#34;dual-complementary-cnn-architecture&#34;&gt;Dual complementary CNN architecture&lt;/h2&gt;
&lt;p&gt;Our architecture consists of a pair of small convolutional neural networks (CNNs) that exhibit a relatively high complementarity score from a given pool of pretrained models. The selection process ensures that the two selected CNN models have the smallest size possible while together covering most of the prediction cases in a given dataset. This setup is further enhanced by the use of a caching mechanism that remembers previous similar classifications and skips the CNN inference if the same input is detected, leading to further energy reduction.
For each input, the process begins with the first CNN generating a prediction and calculating a confidence score derived from its logits vector. An output logits vector of an artificial neural network represents the raw, unnormalized scores from the network&amp;rsquo;s final layer, where each value corresponds to a class in the dataset and indicates the relative likelihood of the input belonging to that class. This confidence score is then evaluated against a predefined threshold, which determines whether the second CNN needs to be activated. If the score falls below the threshold, the second CNN is invoked to refine the prediction. When both CNNs are utilized, their confidence scores are compared, and the final prediction is selected based on the higher confidence level. Additionally, with the memory component for every input, a unique fingerprint is generated and used as a key to access a hash table. If the hash table contains a stored classification for the fingerprint, the cached result is returned immediately, bypassing the need to invoke the CNNs leading to further energy reduction.&lt;/p&gt;
&lt;h2 id=&#34;the-concept-of-complementarity&#34;&gt;The Concept of Complementarity&lt;/h2&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/dual_cnn_complementarity-idea-demonstration.png&#34; alt=&#34;complementarity_idea&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Complementarity based on the predictions of two CNN models.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;The concept of complementarity between two CNN models is illustrated using the above figure where the dataset is represented by a gray box, and the models’ correct predictions are depicted as blue and green circles. Complementarity is defined as the symmetric difference between the correct predictions of the two models, emphasizing their ability to complement each other&amp;rsquo;s strengths. Four scenarios highlight how complementarity impacts prediction performance and resource efficiency.&lt;/p&gt;
&lt;p&gt;In the first scenario (a), both models correctly predict much of the dataset but overlap significantly, leading to low complementarity, redundancy, and higher energy consumption. The second scenario (b) shows high complementarity and extensive prediction coverage, but the larger model&amp;rsquo;s energy demands make the configuration inefficient. The third example (c) illustrates perfect complementarity with no overlap but limited prediction coverage, resulting in suboptimal accuracy. The fourth scenario (d) demonstrates the ideal balance, where models of similar size achieve high complementarity and cover most of the dataset with minimal overlap, optimizing both performance and resource utilization.&lt;/p&gt;
&lt;h2 id=&#34;results&#34;&gt;Results&lt;/h2&gt;
&lt;p&gt;We conducted our experiments on four datasets: &lt;a href=&#34;https://www.cs.toronto.edu/~kriz/cifar.html&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;CIFAR-10&lt;/a&gt;, &lt;a href=&#34;https://image-net.org/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;ImageNet&lt;/a&gt;, &lt;a href=&#34;https://www.kaggle.com/datasets/puneet6060/intel-image-classification&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Intel Image Classification&lt;/a&gt;, and &lt;a href=&#34;https://www.kaggle.com/datasets/zalando-research/fashionmnist&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;FashionMNIST&lt;/a&gt;. For each dataset, our architecture setups (CI3, IM3, IN3, and FM3, respectively) demonstrated reduced energy consumption while maintaining high accuracy. In contrast, other model compression techniques either achieved better performance at the cost of significantly higher energy consumption or reduced energy usage with substantial drops in accuracy. For more details on the other comparison methods you can read the paper &lt;a href=&#34;https://www.sciencedirect.com/science/article/pii/S0167739X24005703?dgcid=author&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;here&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/dual_cnn_efficiency_quad.png&#34; alt=&#34;efficiency&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Current and Energy consumption measurements of tested configurations for (a) CIFAR-10, (b) ImageNet, (c) Intel and (d) FashionMNIST datasets.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/dual_cnn_performance_quad.png&#34; alt=&#34;performance&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Performance of tested configurations for (a) CIFAR-10, (b) ImageNet, (c) Intel and (d) FashionMNIST datasets.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;When utilizing the optional memory component, we observe significantly lower energy consumption in environments with repeated similar inputs. In the following figure, our implementations without the memory component are represented as CI3, IM3, IN3, and FM3, while those with the memory component (using two different methods) are labeled as CI4 and CI5, IM4 and IM5, IN4 and IN5, and FM4 and FM5. Notably, increasing the number of duplicated samples results in little to no additional energy consumption.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/dual_cnn_memory_quad.png&#34; alt=&#34;memory&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Energy consumption comparisons using memory component with duplicated samples.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;We also make available the code to reproduce our experiments on &lt;a href=&#34;https://github.com/michaelkinnas/Reducing-Inference-Energy-Consumption-Using-Two-Complementary-CNNs&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;this&lt;/a&gt; GitHub repository.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Leveraging Representations from Intermediate Encoder-Blocks for Synthetic Image Detection</title>
      <link>https://mever.gr/post/representations-from-intermediate-encoder-blocks/</link>
      <pubDate>Wed, 10 Jul 2024 00:00:00 +0000</pubDate>
      <guid>https://mever.gr/post/representations-from-intermediate-encoder-blocks/</guid>
      <description>&lt;p&gt;In this post we present the essential parts of our method RINE, described in the paper &lt;em&gt;&lt;a href=&#34;https://arxiv.org/abs/2402.19091&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Leveraging Representations from Intermediate Encoder-Blocks for Synthetic Image Detection&lt;/a&gt;&lt;/em&gt;, which has been accepted by the &lt;a href=&#34;https://eccv2024.ecva.net/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;European Conference on Computer Vision (ECCV 2024)&lt;/a&gt;.&lt;/p&gt;
&lt;h1 id=&#34;motivation&#34;&gt;Motivation&lt;/h1&gt;
&lt;p&gt;Recent research on Synthetic Image Detection (SID) has led to strong evidence on the advantages of representations extracted by foundation models, exhibiting exceptional generalization capabilities on GAN and Diffusion model generated data &lt;a href=&#34;https://arxiv.org/abs/2302.10174&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;(Ojha et al. 2023)&lt;/a&gt;. Motivated by this success, we hypothesize that further performance gains are possible by leveraging representations from intermediate layers, which carry low-level visual information, in addition to representations from the final layer that primarily carry high-level semantic information.&lt;/p&gt;
&lt;h1 id=&#34;method&#34;&gt;Method&lt;/h1&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/rine-architecture.png&#34; alt=&#34;architecture&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;The RINE architecture. A batch of images is processed by CLIP&amp;rsquo;s image encoder. The concatenation of intermediate CLS tokens is first projected and then multiplied with the blocks&amp;rsquo; scores, estimated by the Trainable Importance Estimator (TIE) module. Summation across the second dimension results in one feature vector per image. Finally, after the second projection and the consequent classification head modules, two loss functions are computed. Binary cross-entropy directly optimizes SID, while the contrastive loss assists the training by forming a dense feature vector cluster per class. Figure by Christos Koutlis&lt;/em&gt;&lt;/p&gt;
&lt;h2 id=&#34;feature-extraction&#34;&gt;Feature Extraction&lt;/h2&gt;
&lt;p&gt;After the standard image pre-processing (patch extraction and projection, positional encoding), CLIP&amp;rsquo;s Visual Transformer processes the input tokens $\mathbf{Z}_0\in\mathbb{R}^{b\times (p+1)\times d}$ as below:&lt;/p&gt;
&lt;p&gt;$$
\tilde{\mathbf{Z}_l}=\text{MSA}(\text{LN}(\mathbf{Z}_{l-1}))+\mathbf{Z}_{l-1}
$$&lt;/p&gt;
&lt;p&gt;$$
\mathbf{Z}_l=\text{MLP}(\text{LN}(\tilde{\mathbf{Z}_l}))+\tilde{\mathbf{Z}_l}
$$&lt;/p&gt;
&lt;p&gt;with $l$ denoting the block&amp;rsquo;s index.&lt;/p&gt;
&lt;p&gt;Then, we define the Representations from Intermediate Encoder-blocks (RINE) $\mathbf{K}$, as the concatenation of &lt;code&gt;CLS&lt;/code&gt; tokens from the corresponding $n$ blocks:&lt;/p&gt;
&lt;p&gt;$$
\mathbf{K}=\oplus\lbrace\mathbf{Z}_l^{\text{[0]}}\rbrace_{l=1}^n\in\mathbb{R}^{b\times n\times d}
$$&lt;/p&gt;
&lt;p&gt;where $\oplus$ denotes concatenation, and $\mathbf{Z}_l^{\text{[0]}}\in\mathbb{R}^{b\times 1\times d}$ denotes the &lt;code&gt;CLS&lt;/code&gt; token from the output of block $l$. We keep CLIP, which produces these features, frozen during training and use $\mathbf{K}$ to construct discriminative features for the SID task.&lt;/p&gt;
&lt;h2 id=&#34;learning&#34;&gt;Learning&lt;/h2&gt;
&lt;p&gt;A projection network $\mathfrak{Q}_1$ with linear mappings, &lt;code&gt;ReLU&lt;/code&gt; activations, and &lt;code&gt;Dropout&lt;/code&gt; regulatizations, processes the input representations $\mathbf{K}$. Its output is then adjusted, with the use of a Trainable Importance Estimator (TIE) module, in order to reflect the importance of each Transformer block to the SID task:&lt;/p&gt;
&lt;p&gt;$$
\tilde{\mathbf{K}}^{(ik)}=\sum_l^n\mathcal{S}(\mathbf{A})^{(lk)}\cdot\mathbf{K}_q^{(ilk)}
$$&lt;/p&gt;
&lt;p&gt;Another projection network $\mathfrak{Q}_2$ is then applied before the classification head that estimates the probability $\hat{y}_i$ of image $i$ to be fake.&lt;/p&gt;
&lt;p&gt;Finally, we use a combination of binary cross-entropy and supervised contrastive loss to optimize our model:&lt;/p&gt;
&lt;p&gt;$$
\mathfrak{L}_{CE}=-\sum_{i=1}^by_i\text{log}\hat{y}_i+(1-y_i)\cdot\text{log}(1-\hat{y}_i)
$$&lt;/p&gt;
&lt;p&gt;$$
\mathfrak{L}_{Cont.}=-\sum_{i=1}^b\frac{1}{G(i)}\sum_{g\in G(i)}\text{log}\frac{\text{exp}(\mathbf{z}_i\cdot \mathbf{z}_g/\tau)}{\sum_{a\in A(i)}\text{exp}(\mathbf{z}_i\cdot \mathbf{z}_a/\tau)}
$$&lt;/p&gt;
&lt;p&gt;$$
\mathfrak{L}=\mathfrak{L}_{CE}+\xi\cdot\mathfrak{L}_{Cont.}
$$&lt;/p&gt;
&lt;h1 id=&#34;results&#34;&gt;Results&lt;/h1&gt;
&lt;p&gt;In the below Table, we present the performance of our method versus the competing ones in terms of accuracy. Our 1-class model outperforms all state-of-the-art methods irrespective of training class number. On average, we surpass the state-of-the-art by +9.4% with the 1-class model, by +6.8% with the 2-class model, and by +10.6% with the 4-class model. We obtain the best score in 14 out of 20 test datasets, and simultaneously the first and second best performance in 10 of them. The biggest performance gain is on the SAN dataset (+16.9%).&lt;/p&gt;
&lt;p&gt;Additionally, in the &lt;a href=&#34;https://arxiv.org/abs/2402.19091&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;paper&lt;/a&gt; you can find ablations that support our architectural choices, further experiments regarding robustness to perturbations, the effects of training duration, and training set size, the fact that considering more ProGAN object classes during training does not reduce generalization, and even more&amp;hellip;&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/rine-results-acc.png&#34; alt=&#34;accuracy&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Accuracy (ACC) scores of baselines and our model across 20 test datasets. The second column (# cl.) presents the number of used training classes. Best performance is denoted with bold and second to best with underline. Our method yields +10.6% average accuracy compared to the state-of-the-art.&lt;/em&gt;&lt;/p&gt;
&lt;h1 id=&#34;code&#34;&gt;Code&lt;/h1&gt;
&lt;p&gt;Our code and model checkpoints are publicly available here &lt;a href=&#34;https://github.com/mever-team/rine&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;https://github.com/mever-team/rine&lt;/a&gt;&lt;/p&gt;
&lt;h1 id=&#34;references&#34;&gt;References&lt;/h1&gt;
&lt;p&gt;Ojha, U., Li, Y., &amp;amp; Lee, Y. J. (2023). Towards universal fake image detectors that generalize across generative models. In &lt;em&gt;Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition&lt;/em&gt; (pp. 24480-24489).&lt;/p&gt;
&lt;p&gt;Wang, S. Y., Wang, O., Zhang, R., Owens, A., &amp;amp; Efros, A. A. (2020). CNN-generated images are surprisingly easy to spot&amp;hellip; for now. In &lt;em&gt;Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition&lt;/em&gt; (pp. 8695-8704).&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Filtering external evidence for realistic training and evaluation of Automated Fact-Checking</title>
      <link>https://mever.gr/post/filtering-external-evidence-for-realistic-automated-fact-checking/</link>
      <pubDate>Wed, 03 Jul 2024 00:00:00 +0000</pubDate>
      <guid>https://mever.gr/post/filtering-external-evidence-for-realistic-automated-fact-checking/</guid>
      <description>&lt;p&gt;In this post, we explain the basics behind our paper &lt;a href=&#34;https://dl.acm.org/doi/abs/10.1145/3643491.3660278&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Credible, Unreliable or Leaked?: Evidence verification for enhanced automated fact-checking&lt;/a&gt; by Zacharias Chrysidis, Stefanos-Iordanis Papadopoulos, Symeon Papadopoulos and Panagiotis C. Petrantonakis, which has been presented at ICMR’s 3rd ACM International Workshop on Multimedia AI against Disinformation.&lt;/p&gt;
&lt;h2 id=&#34;automated-fact-checking-afc&#34;&gt;Automated Fact-Checking (AFC)&lt;/h2&gt;
&lt;p&gt;The Information Age, especially after the explosion of online platforms and social media, has led to a surge in new forms of mis- and disinformation, making it increasingly difficult for people to trust what they see and read online. To combat this, many fact-checking organizations, including &lt;a href=&#34;https://www.snopes.com/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Snopes&lt;/a&gt;, &lt;a href=&#34;https://www.politifact.com/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;PolitiFact&lt;/a&gt; as well as &lt;a href=&#34;https://www.reuters.com/fact-check/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Reuters&lt;/a&gt; and &lt;a href=&#34;https://factcheck.afp.com/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;AFP&lt;/a&gt; fact-checks, have emerged, dedicated to verifying claims in news articles and social media posts. Nevertheless, the manual process of fact-checking is time-consuming and can&amp;rsquo;t always keep pace with the rapid spread of mis- and disinformation. This is where the field of Automated Fact-Checking (AFC) comes in. In recent years, researchers have been trying to leverage advances in deep learning, large language models, computer vision, and multimodal learning to develop tools to assist the work of professional fact-checkers. AFC systems aim to automate key parts of the fact-checking process, such as detecting check-worthy claims, retrieving relevant evidence from the web, and cross-examining them against the claim (&lt;a href=&#34;https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00454/109469/A-Survey-on-Automated-Fact-Checking&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Guo, et al., 2022&lt;/a&gt;). Since fact-checking rarely relies solely on examining internal contradictions in claims, AFC systems often require the retrieval of external information from the web, knowledge databases, or performing inverse image searches to support or refute a claim, as shown below.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/evidence-retrieval-example.png&#34; alt=&#34;evidence_retrieval_example&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;An example of a claim, evidence collection and retrieval. Claim and verdict source: &lt;a href=&#34;https://www.snopes.com/fact-check/electric-bus-fire-kenya/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Snopes&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;
&lt;h2 id=&#34;challenges-of-training-automated-fact-checking-afc-systems&#34;&gt;Challenges of training Automated Fact-Checking (AFC) systems&lt;/h2&gt;
&lt;p&gt;Nevertheless, in order to develop and train AFC systems that can be applied to real-world cases, the collected evidence should follow certain criteria, such as “[the evidence in datasets] must be (1) sufficient to refute the claim and (2) not leaked from existing fact-checking articles” as defined by &lt;a href=&#34;https://aclanthology.org/2022.emnlp-main.397/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Glockner et al., 2022&lt;/a&gt;. The problem of leaked evidence in AFC systems arises when information from previously fact-checked articles is used during training. This can lead the model to overly rely on high-quality information curated by professional fact-checkers, making it less effective at handling new and emerging misinformation that has not yet been fact-checked –which is supposedly the task at hand. Additionally, the external information retrieved by the model must be credible to avoid feeding it with unreliable and false data. For example, as shown in the banner of this article, the retrieved evidence “Did President Obama Ban Muslims from Entering the United States in 2011?” is sourced from Snopes, thus, using the full article would allow leaked information into the AFC system. Conversely, feeding “MORE HYPOCRISY: Obama Banned all Iraqi Refugees for 6 Months in 2011 – Liberals SAID NOTHING” into the AFC system would introduce unreliable and false information. These problems are under-researched but crucial for achieving realistic and effective fact-checking.&lt;/p&gt;
&lt;h2 id=&#34;credible-unreliable-or-leaked-credule-dataset-assisting-the-detection-of-leaked-and-unreliable-evidence&#34;&gt;“CREDible, Unreliable, or LEaked” (CREDULE) dataset assisting the detection of leaked and unreliable evidence&lt;/h2&gt;
&lt;p&gt;To address these challenges, we developed the “CREDible, Unreliable, or LEaked” (CREDULE) dataset by modifying, merging, and extending previous datasets such as &lt;a href=&#34;https://www.copenlu.com/publication/2019_emnlp_augenstein/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;MultiFC&lt;/a&gt;, &lt;a href=&#34;https://www.kaggle.com/datasets/rmisra/politifact-fact-check-dataset&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;PolitiFact&lt;/a&gt;, &lt;a href=&#34;https://github.com/neemakot/Health-Fact-Checking&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;PUBHEALTH&lt;/a&gt;, &lt;a href=&#34;https://github.com/MELALab/nela-gt&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;NELA-GT&lt;/a&gt;, &lt;a href=&#34;https://github.com/several27/FakeNewsCorpus&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Fake News Corpus&lt;/a&gt;, and &lt;a href=&#34;https://www.kaggle.com/datasets/mrisdal/fake-news&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Getting Real About Fake News&lt;/a&gt;. The CREDULE dataset consists of 91,632 samples from 2016 to 2022, equally distributed across three classes: “Credible,” “Unreliable,” and “Fact-checked” (Leaked). Our criteria ensure balanced distributions not only across these classes but also across years, title length, and topics. The dataset includes both short texts (titles) and long texts (full articles), along with meta-data such as date, domain, URL, topic, and credibility scores. The credibility and bias scores were collected from the &lt;a href=&#34;https://mediabiasfactcheck.com/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Media Bias / Fact Check&lt;/a&gt; organization.&lt;/p&gt;
&lt;p&gt;After creating the CREDULE dataset, we developed the EVidence VERification Network (EVVER-Net), a neural network designed to detect leaked (fact-checked) and unreliable evidence during the evidence retrieval process. EVVER-Net can be integrated into an AFC pipeline to ensure that only credible and non-leaked information is used during training. The model leverages the power of large pre-trained Transformer-based text encoders, such as DeBERTa, CLIP, and T5 for short texts, and Long T5 and Longformer for long texts. It also integrates information from credibility and bias scores. EVVER-Net demonstrated impressive performance, achieving up to 89.0% accuracy without credibility scores and 94.4% with credibility scores.&lt;/p&gt;
&lt;h2 id=&#34;examining-the-evidence-of-widely-used-afc-datasets&#34;&gt;Examining the evidence of widely used AFC datasets&lt;/h2&gt;
&lt;p&gt;Finally, we applied the EVVER-Net model to the evidence provided by several widely used AFC datasets, including &lt;a href=&#34;https://github.com/Tariq60/LIAR-PLUS&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;LIAR-Plus&lt;/a&gt;, &lt;a href=&#34;https://github.com/VT-NLP/Mocheg&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;MOCHEG&lt;/a&gt;, &lt;a href=&#34;https://github.com/Shreyashm16/Factify&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;FACTIFY&lt;/a&gt;, &lt;a href=&#34;https://github.com/g-luo/news_clippings&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;NewsCLIPpings&lt;/a&gt;, and &lt;a href=&#34;https://github.com/stevejpapad/image-text-verification&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;VERITE&lt;/a&gt;. EVVER-Net identified up to 98.5% of items in LIAR-Plus, 95.0% in the “Refute” class of FACTIFY, and 83.6% in MOCHEG as leaked evidence. For instance, in the FACTIFY dataset, the claim &amp;ldquo;Microsoft bought Sony for 121 billion&amp;rdquo; was accompanied by the evidence: &amp;ldquo;A satire article claiming that Microsoft has bought Sony for 130 billion has gone viral […] the original article in a Spanish website was intended to be a prank. [&amp;hellip;]&amp;rdquo; which is sourced from a previously fact-checked article and directly provides the model with the answer curated by a professional fact-checker. This limits the model&amp;rsquo;s ability to detect and verify new claims that have not yet been fact-checked.&lt;/p&gt;
&lt;p&gt;On the other hand, the NewsCLIPpings dataset had the lowest rate of leaked evidence at 14.1%, followed by &lt;a href=&#34;https://mever.gr/post/overcoming-unimodal-bias-in-multimodal-misinformation-detection/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;our own VERITE&lt;/a&gt; at 35.3%. For instance, a claim from VERITE: “An image shows a large crowd gathered on the Rio de Janeiro seafront to celebrate a Mass delivered by Pope Francis in 2013” is accompanied by evidence “Pope Francis wraps up Brazil trip with Mass for 3 million”, which is sourced by CBS News and is correctly identified as credible by EVVER-Net. Another claim, “A photograph captured by NASA’s Mars Curiosity Rover on May 7, 2022, showed an artificial portal nearby.” is accompanied by evidence “Does This NASA Photo Show a &amp;lsquo;Portal&amp;rsquo; and &amp;lsquo;Wall&amp;rsquo; on Mars?” sourced by  Snopes. The title by itself does not provide any leaked information into the model, but if the full article were to be fed into the model during training, it would reinforce the problem of leaked evidence.&lt;/p&gt;
&lt;p&gt;While NewsCLIPpings and VERITE display significant improvements over other datasets in the field, there is still room for enhancement, as 20% of the evidence in these datasets was estimated to be non-credible. By utilizing EVVER-Net or similar architectures, the credibility of the evidence in NewsCLIPpings and VERITE could be further improved, helping to avoid the issues of unreliable and leaked evidence and overall, building and evaluating AFC pipelines in a more realistic manner.&lt;/p&gt;
&lt;h2 id=&#34;references&#34;&gt;References&lt;/h2&gt;
&lt;p&gt;Guo, Z., Schlichtkrull, M., &amp;amp; Vlachos, A. (2022). A survey on automated fact-checking. Transactions of the Association for Computational Linguistics, 10, 178-206.&lt;/p&gt;
&lt;p&gt;Glockner, M., Hou, Y., &amp;amp; Gurevych, I. (2022, December). Missing Counter-Evidence Renders NLP Fact-Checking Unrealistic for Misinformation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (pp. 5916-5936).&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>SIDBench - A Python Framework for Reliably Assessing Synthetic Image Detection Methods</title>
      <link>https://mever.gr/post/sidbench-assessing-synthetic-image-detection-methods/</link>
      <pubDate>Fri, 21 Jun 2024 00:00:00 +0000</pubDate>
      <guid>https://mever.gr/post/sidbench-assessing-synthetic-image-detection-methods/</guid>
      <description>&lt;h2 id=&#34;the-problem&#34;&gt;The problem&lt;/h2&gt;
&lt;p&gt;Generative AI is making it easier to create incredibly realistic synthetic images. Detecting these fake images is challenging, and several methods have been developed to tackle this issue. However, there&amp;rsquo;s often a big difference between how well these methods work in experiments and how they perform in real life.&lt;/p&gt;
&lt;p&gt;To help bridge this gap, our recent work &lt;a href=&#34;https://arxiv.org/abs/2404.18552&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;&amp;ldquo;SIDBench: A Python Framework for Reliably Assessing Synthetic Image Detection&amp;rdquo;&lt;/a&gt; introduces a new benchmarking framework that combines various state-of-the-art detection models. These models were chosen for their different features and network designs to cover a wide range of techniques. The framework uses the latest, highly realistic datasets and studies how common online image transformations, like JPEG compression, affect detection accuracy. This tool is available &lt;a href=&#34;https://github.com/mever-team/sidbench&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;online&lt;/a&gt; and is designed to be easily updated with new datasets and detection models, making it a flexible and valuable resource for improving synthetic image detection.&lt;/p&gt;
&lt;h2 id=&#34;the-sidbench-framework&#34;&gt;The SIDBench Framework&lt;/h2&gt;
&lt;p&gt;To map the current landscape of Synthetic Image Detection (SID) methods, we reviewed recent literature and identified 11 state-of-the-art models since 2020. These models were evaluated based on two criteria: the type of input used (raw images or extracted features like frequency patterns and noise) and the architecture of the model. Popular architectures include ResNet50 pre-trained on ImageNet and ViT-L/14 pre-trained with &lt;a href=&#34;https://arxiv.org/abs/2103.00020&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;CLIP&lt;/a&gt;.&lt;/p&gt;
&lt;h2 id=&#34;main-findings-from-sidbench&#34;&gt;Main Findings from SIDBench&lt;/h2&gt;
&lt;h3 id=&#34;performance-across-different-models&#34;&gt;Performance Across Different Models&lt;/h3&gt;
&lt;p&gt;Our benchmark study verified that almost all models trained on GAN-generated images generalize well across different GANs. However, their performance drops significantly when applied to Diffusion Models (DMs), such as Guided Diffusion, Latent Diffusion, and Glide. Notably, some models like RINE and PatchCraft perform surprisingly well on DMs.&lt;/p&gt;
&lt;h3 id=&#34;calibration-for-improved-accuracy&#34;&gt;Calibration for Improved Accuracy&lt;/h3&gt;
&lt;p&gt;We discovered that accuracy could be further enhanced by calibrating the threshold used for classifying images. This suggests there is room for further improvement of performance.&lt;/p&gt;
&lt;h3 id=&#34;real-world-performance-evaluation&#34;&gt;Real-World Performance Evaluation&lt;/h3&gt;
&lt;p&gt;To assess real-world applicability, we evaluated ProGAN-trained models using the &lt;a href=&#34;https://zenodo.org/records/10066460&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Synthbuster&lt;/a&gt; dataset. Results showed a significant drop in performance across all models, highlighting the challenge of maintaining accuracy outside experimental conditions, with high-resolution, high quality images. The following table presents the overall accuracy for all proGAN-trained detectors.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/synthbuster-acc-all-models.png&#34; alt=&#34;synthbuster_results&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Accuracy (ACC) scores of integrated models for the &lt;a href=&#34;https://zenodo.org/records/10066460&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Synthbuster&lt;/a&gt; dataset&lt;/em&gt;&lt;/p&gt;
&lt;h3 id=&#34;impact-of-training-datasets&#34;&gt;Impact of Training Datasets&lt;/h3&gt;
&lt;p&gt;To explore the impact of training datasets, we included models trained on sources other than ProGAN. For instance, &lt;a href=&#34;https://arxiv.org/abs/2402.19091&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;RINE&lt;/a&gt;, trained on Latent Diffusion, showed improved detection of DM images while retaining good performance on GAN-based images. Additionally, models trained on DM images yielded better results on challenging subsets of Synthbuster, such as images generated with the challenging Midjourney v5 algorithm.&lt;/p&gt;
&lt;h3 id=&#34;handling-image-preprocessing-resizing-and-cropping&#34;&gt;Handling Image Preprocessing (Resizing and Cropping)&lt;/h3&gt;
&lt;p&gt;When using pre-trained models, images can be either cropped to a predefined size or resized. Each method has its trade-offs: cropping can lose important information outside the cropped area, while resizing may introduce distortion, affecting performance. For example, resizing alters frequency components, which is crucial for models like FreqDetect that rely on frequency artifacts. Our findings indicate that, on average, performance remains relatively consistent regardless of the preprocessing method. However, the standard deviation in performance suggests that resizing versus cropping impacts each model differently across the Synthbuster datasets.&lt;/p&gt;
&lt;h3 id=&#34;influence-of-image-transformations&#34;&gt;Influence of Image Transformations&lt;/h3&gt;
&lt;p&gt;Image transformations like Gaussian blurring and JPEG re-compression negatively affect the performance of SID models. Our tests revealed that Gaussian blurring and JPEG re-compression generally reduce accuracy (ACC), with varying impacts on true positive rates (TPR) and true negative rates (TNR). Gaussian blurring tends to increase TPR at the expense of TNR, causing models to detect more images as fakes, including some real ones. However, when combined with JPEG re-compression, this behavior changes, indicating a need for further investigation. JPEG compression often impacts TNR less than TPR, except in some models where it leads to a complete inability to detect fake images.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/transformation-impact.png&#34; alt=&#34;transformation_impact&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;ACC, TPR and TNR across detection models for different image transformations. Metrics have been averaged across all evaluation datasets.&lt;/em&gt;&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Mitigating Viewer Impact From Disturbing Imagery Using AI Filters A User-Study</title>
      <link>https://mever.gr/post/mitigating-viewer-impact-from-disturbing-imagery-using-ai-filters-a-user-study/</link>
      <pubDate>Wed, 19 Jun 2024 00:00:00 +0000</pubDate>
      <guid>https://mever.gr/post/mitigating-viewer-impact-from-disturbing-imagery-using-ai-filters-a-user-study/</guid>
      <description>&lt;p&gt;In today&amp;rsquo;s digital age, professionals such as journalists and human rights investigators often encounter distressing and graphic imagery as part of their work. The emotional toll of viewing such content can be profound, leading to secondary trauma and affecting mental well-being. Traditional methods like blurring have been used to shield viewers from explicit details, but they often compromise image interpretability.&lt;/p&gt;
&lt;h2 id=&#34;exploring-the-capabilities-of-ai-filters&#34;&gt;Exploring The Capabilities of AI filters&lt;/h2&gt;
&lt;p&gt;Our recent paper &lt;a href=&#34;https://www.tandfonline.com/doi/full/10.1080/10447318.2024.2313890&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;&amp;ldquo;Mitigating Viewer Impact From Disturbing Imagery Using AI Filters A User-Study&amp;rdquo;&lt;/a&gt; explores the effectiveness of Artificial Intelligence (AI) filters on mitigating the emotional impact of disturbing imagery while preserving crucial image information.&lt;/p&gt;
&lt;p&gt;AI has revolutionized various fields, including image processing. Style transfer techniques, which alter image appearance based on artistic styles, offer a promising alternative to traditional blurring. Unlike blurring, which can obscure details crucial for professional analysis, AI filters can maintain image integrity while reducing negative emotional responses significantly. Our study, involving 107 participants primarily from journalism and human rights fields, assessed the performance of five filter styles: Blurring, Partial Blurring, Drawing, Colored Drawing, and Painting.&lt;/p&gt;
&lt;h2 id=&#34;study-findings-the-power-of-ai-filters&#34;&gt;Study Findings: The Power of AI Filters&lt;/h2&gt;
&lt;p&gt;Participants reported a remarkable -30.38% reduction in negative emotional impact when viewing images filtered with the AI-based Drawing style. Moreover, this style retained an impressive 97.19% of image interpretability, far surpassing the 6.54% retained by traditional Blurring techniques. Furthermore, professionals highlighted the potential of AI filters as a preparatory step before viewing original images. This approach could mitigate immediate emotional distress while allowing for detailed inspection when necessary. Participants also suggested enhancements and identified workflow stages where AI filters could be seamlessly integrated.&lt;/p&gt;
&lt;p&gt;To delve deeper into our findings and explore the future of AI in content moderation, read our full study &lt;a href=&#34;https://www.tandfonline.com/doi/full/10.1080/10447318.2024.2313890&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;here&lt;/a&gt;.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Unveiling AI Bias in Facial Analysis: Meet FaceX</title>
      <link>https://mever.gr/post/facex-understanding-face-attribute-classifiers-through-summary-model-explanations/</link>
      <pubDate>Wed, 19 Jun 2024 00:00:00 +0000</pubDate>
      <guid>https://mever.gr/post/facex-understanding-face-attribute-classifiers-through-summary-model-explanations/</guid>
      <description>&lt;p&gt;In today&amp;rsquo;s world, AI systems are integral to many applications, from unlocking smartphones to recognizing emotions. While these advancements bring numerous benefits, they also raise ethical and societal concerns, particularly regarding AI bias. Facial analysis technologies, in particular, have been scrutinized for their potential biases, which can disproportionately affect certain groups of people. This is where our new tool, FaceX, comes into play.&lt;/p&gt;
&lt;h2 id=&#34;what-is-facex&#34;&gt;What is FaceX?&lt;/h2&gt;
&lt;p&gt;FaceX is a novel method designed to uncover biases in facial attribute classifiers. While existing instance-level XAI methods require labor-intensive manual inspections to draw general conclusions about model behavior, FaceX addresses this limitation by aggregating model activations at the region level, making it easier to visualize and understand biases. In particular, it summarizes model behavior across 19 predefined facial regions, such as hair, ears, and skin, providing a more comprehensive understanding of where and why a model focuses on specific facial areas.&lt;/p&gt;
&lt;h2 id=&#34;how-does-facex-work&#34;&gt;How Does FaceX Work?&lt;/h2&gt;
&lt;p&gt;FaceX computes a region-level aggregation of model instance-level attributions, summarizing the model’s output with respect to each region of interest. Then, spatial explanations, offered through a heatmap visualization over an abstract face prototype, provide in-depth understanding of the weight of each facial region (or accessory) on the model decision. Additionally, FaceX visualizes the high-impact image patches for each region, revealing not only where the model focuses but also helping the human analyst understand why certain features are influential. This dual approach of spatial explanation (understanding where the model focuses) and appearance-oriented insights (understanding the impact of specific image patches) sets FaceX apart as a powerful tool for identifying biases in facial analysis systems and acts as a comprehensive lens, allowing practitioners, researchers, and developers to scrutinize the entire spectrum of model behavior.&lt;/p&gt;
&lt;p&gt;Want to dive deeper? Check out the &lt;a href=&#34;https://dl.acm.org/doi/abs/10.1145/3652583.3658007&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;paper&lt;/a&gt; and &lt;a href=&#34;https://github.com/gsarridis/faceX&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;run&lt;/a&gt; FaceX to explain the behaviour of your models.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>SDFD: Building a Versatile Synthetic Face Image Dataset with Diverse Attributes</title>
      <link>https://mever.gr/post/sdfd_building-a-versatile-synthetic-face-image-dataset-with-diverse-attributes/</link>
      <pubDate>Fri, 31 May 2024 00:00:00 +0000</pubDate>
      <guid>https://mever.gr/post/sdfd_building-a-versatile-synthetic-face-image-dataset-with-diverse-attributes/</guid>
      <description>&lt;p&gt;Artificial Intelligence (AI) systems today are incredibly advanced, leveraging vast amounts of data to perform complex tasks with high accuracy. These systems require huge datasets to train effectively. The quality and quantity of data significantly impact their performance, enabling them to recognize patterns, make predictions, and improve decision-making processes. As AI continues to evolve, the demand for large, diverse, and high-quality datasets increases especially, for image-based systems dealing with demographic attribute prediction that often face considerable bias and discrimination issues. Many current face image datasets primarily focus on representativeness in terms of demographic factors such as age, gender, and skin tone, overlooking other crucial facial attributes like hairstyle and accessories. This narrow focus limits the diversity of the data and consequently the robustness of AI systems trained on them.&lt;/p&gt;
&lt;p&gt;In our recent work &lt;a href=&#34;https://arxiv.org/abs/2404.17255&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;“SDFD: Building a Versatile Synthetic Face Image Dataset with Diverse Attributes”&lt;/a&gt; that we presented in the &lt;a href=&#34;https://responsiblefaceimageprocessing.github.io/fg2024/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;1st International Workshop on Responsible Face Image Processing&lt;/a&gt;, we tried to address the former limitations of existing datasets, by proposing a methodology for generating synthetic face image datasets that capture a broader spectrum of facial diversity. Specifically, our approach integrates a state-of-the-art text-to-image generation model with a systematic prompt formulation strategy, encompassing not only demographics and biometrics but also non-permanent traits like make-up, hairstyle, and accessories. These lead to a comprehensive dataset of high-quality realistic images, which can be used as an evaluation set in face analysis systems.&lt;/p&gt;
&lt;p&gt;The first step in our process was to compile a list of terms that will represent the attributes that one would like to be depicted in the target face images. These terms should be carefully picked and then filtered to eliminate specific words or phrases (see Table 1). After completing the basic attribute list, combinations of them are created to generate the text prompt. Prompts are used as input text to the generative model, describing the image that should be constructed. The more exact the description, the closer the resulting image will be to what was envisioned. Besides, negative terms are also important in the prompt process since they specify what should not be included in the generated image.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/sdfd_table.png&#34; alt=&#34;Table 1&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;
&lt;em&gt;Table 1. Terms used in prompts. Options are presented in alphabetical order.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Figure 1 shows some example face image datasets created by our methodology along with their input prompts.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/sdfd_featured.png&#34; alt=&#34;Figure 1&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;
&lt;em&gt;Fig. 1. Example generated images and their corresponding prompts without including universal prompt terms:a. Black, baby boy, short hair, 4K, Fujifilm XT3, b. Pacific Islander-Middle Eastern woman, having face tattoo, headband,having wrinkles, 8K, Nikon Z9, c. Pacific Islander, stressed, wearing headscarf, girl, black eyes, wearing lipstick, 8K, d. White, androgynous person, pink hair, blue eyes, Fujifilm XT3, e. White, girl, headband, Ultra HD, Nikon Z9, f. Southeast Asian-White, man, having piercing, dreadlocks, having wrinkles, Ultra HD, Canon Eos 5D.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;We opted for Stable Diffusion v2.1 [1] and generated 1000 face images, which constitute the first version of our Stabe Diffusion Face-image Dataset (SDFD).&lt;/p&gt;
&lt;p&gt;We compared SDFD with the existing FairFace [2] and LFW [3] datasets, which are established face images that comprise real images and are used in several fairness-aware research works. Our comparative study demonstrated that SDFD was an equally or even more challenging dataset compared to the existing ones, while also being considerably smaller in size.&lt;/p&gt;
&lt;p&gt;In the future, we intend to extend the number of images in our dataset to include less well-represented facial attributes such as scars, braces, and even disabilities - that we were unable to include in the current dataset version due to limitations in the text-to-image model. Besides, we intend to explore different prompting strategies and generative models. Finally, we would also like to extend SDFD with multiple images for the same identity to make it appropriate for other tasks such as face verification.&lt;/p&gt;
&lt;p&gt;SDFD is publicly available on &lt;a href=&#34;https://github.com/gebaltso/SDFD/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Github&lt;/a&gt; and the corresponding paper on &lt;a href=&#34;https://arxiv.org/abs/2404.17255&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;arXiv&lt;/a&gt;.&lt;/p&gt;
&lt;h1 id=&#34;references&#34;&gt;References&lt;/h1&gt;
&lt;p&gt;[1] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., &amp;amp; Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10684-10695).&lt;/p&gt;
&lt;p&gt;[2] Karkkainen, K., &amp;amp; Joo, J. (2021). Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision (pp. 1548-1558).&lt;/p&gt;
&lt;p&gt;[3] Huang, G. B., Mattar, M., Berg, T., &amp;amp; Learned-Miller, E. (2008, October). Labeled faces in the wild: A database for studying face recognition in unconstrained environments. In Workshop on faces in &amp;lsquo;Real-Life&amp;rsquo; Images: detection, alignment, and recognition.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>AI Fairness Definition Guide</title>
      <link>https://mever.gr/post/ai-fairness-definition-guide/</link>
      <pubDate>Mon, 13 May 2024 00:00:00 +0000</pubDate>
      <guid>https://mever.gr/post/ai-fairness-definition-guide/</guid>
      <description>&lt;p&gt;In the context of the Horizon Europe MAMMOth project, we developed the &amp;ldquo;AI Fairness Definition Guide&amp;rdquo; to help those creating AI (such as researchers, developers, and product owners) understand how to define fairness in the social context of their created systems by working with stakeholders and experts from other disciplines. The guide presents a workflow for gathering fairness concerns of affected stakeholders and using them to derive corresponding formalisms and practices under a combined computer, social, and legal science viewpoint.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/fairness_guide.png&#34; alt=&#34;retrieval_system&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Overview of interactions between different actors for defining what constitutes fair AI in an examined context. Figure by Emmanouil Krasanakis&lt;/em&gt;&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>FRCSyn-onGoing:Benchmarking and comprehensive evaluation of real and synthetic data to improve face recognition systems</title>
      <link>https://mever.gr/post/frcsyn_ongoing_benchmarking-and-comprehensive-evaluation-of-real-and-synthetic-data-to-improve-face-recognition-systems/</link>
      <pubDate>Fri, 03 May 2024 00:00:00 +0000</pubDate>
      <guid>https://mever.gr/post/frcsyn_ongoing_benchmarking-and-comprehensive-evaluation-of-real-and-synthetic-data-to-improve-face-recognition-systems/</guid>
      <description>&lt;p&gt;This post provides an overview of the paper &lt;a href=&#34;https://www.sciencedirect.com/science/article/pii/S1566253524001003&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;&amp;ldquo;FRCSyn-onGoing: Benchmarking and comprehensive evaluation of real and synthetic data to improve face recognition systems&amp;rdquo;&lt;/a&gt; , which has been published in the &lt;a href=&#34;https://www.sciencedirect.com/journal/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Information Fusion&lt;/a&gt;information-fusion journal. This has been a collaborative effort led by the organizers of the FRCSyn-onGoing challenge and several of the participating teams, such as ours.&lt;/p&gt;
&lt;p&gt;The utilization of synthetic data in face recognition systems offers several advantages over real-world datasets. Synthetic datasets provide a solution to privacy concerns associated with collecting real data that potentially include personal information without obtaining the consent by the affected individuals, a case that is quite common in image datasets sourced from the Internet. They also enable the generation of large datasets, which is especially important after the discontinuation of established databases due to privacy issues and regulations like the General Data Protection Regulation (GDPR) [27]. Synthetic data can be tailored to specific characteristics, such as demographic groups and age, without the need for additional human efforts, unlike real-world databases that may lack diversity in representation or suffer from a variety of sampling biases.&lt;/p&gt;
&lt;p&gt;In particular, the paper focuses on advancing face recognition technology by evaluating the effectiveness of using synthetic data alongside real data. This initiative involves a collaborative effort from researchers worldwide to address the limitations and challenges faced by current face recognition systems. By combining real and synthetic data, the paper aims to enhance the performance and accuracy of face recognition technology. The article summarizes the findings from the Face Recognition Challenge in the Era of Synthetic Data (FRCSyn) held at WACV 2024.&lt;/p&gt;
&lt;p&gt;One of the key aspects of the paper is the comprehensive evaluation of Face Recognition (FR) systems across different demographic groups and databases. Researchers have identified notable findings, such as lower performance on demographic groups representing the Asian population compared to others. However, training face recognition systems with synthetic data generated by frameworks like DCFace [1] and GANDiffFace [2] (see Figure 1) has led to promising results, particularly in addressing demographic diversity within test populations.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img alt=&#34;Figure 1&#34; srcset=&#34;
               /post/frcsyn_ongoing_benchmarking-and-comprehensive-evaluation-of-real-and-synthetic-data-to-improve-face-recognition-systems/featured_hu_d4f3b7b69ff09ca8.webp 400w,
               /post/frcsyn_ongoing_benchmarking-and-comprehensive-evaluation-of-real-and-synthetic-data-to-improve-face-recognition-systems/featured_hu_8d7a593e7d1434ab.webp 760w,
               /post/frcsyn_ongoing_benchmarking-and-comprehensive-evaluation-of-real-and-synthetic-data-to-improve-face-recognition-systems/featured_hu_fd1937081eeb92c6.webp 1200w&#34;
               src=&#34;https://mever.gr/post/frcsyn_ongoing_benchmarking-and-comprehensive-evaluation-of-real-and-synthetic-data-to-improve-face-recognition-systems/featured_hu_d4f3b7b69ff09ca8.webp&#34;
               width=&#34;760&#34;
               height=&#34;185&#34;
               loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;
&lt;em&gt;Figure 1: Examples of synthetic identities (one for each row) and their intra-class variations provided by two generative frameworks:(a) DCFace [1] and (b) GANDiffFa[2].&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Teams participating in the challenge, among which the MeVer team, proposed new approaches to reduce demographic bias and improve the overall performance of FR systems. By using ResNet backbones, ArcFace, and AdaFace loss functions, with several integrating multiple networks and advanced data augmentation techniques, researchers aimed to create less biased features and enhance the learning process for FR systems. Standardized benchmarks and metrics to evaluate FR performance in realistic scenarios, are also presented. By considering factors such as pose variations, aging, occlusions, and diverse demographic groups, researchers provide a more comprehensive assessment of the capabilities of these systems.&lt;/p&gt;
&lt;p&gt;Through the FRCSyn-onGoing challenge, researchers are paving the way for the future of FR technology by exploring the potential of synthetic data and domain generalization techniques. The collaboration between industry and academia in this initiative highlights the importance of addressing demographic bias and improving the fairness and accuracy of face recognition systems. By continuing to explore the benefits of synthetic data and innovative approaches, the paper aims to drive advancements in FR technology and contribute to rendering it more trustworthy and reliable.&lt;/p&gt;
&lt;h1 id=&#34;references&#34;&gt;References&lt;/h1&gt;
&lt;p&gt;[1] M. Kim, F. Liu, A. Jain, X. Liu, DCFace: Synthetic face generation with dual condition diffusion model, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.&lt;/p&gt;
&lt;p&gt;[2] P. Melzi, C. Rathgeb, R. Tolosana, R. Vera-Rodriguez, D. Lawatsch, F. Domin, M. Schaubert, GANDiffFace: Controllable generation of synthetic datasets for face recognition with realistic variations, in: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 2023.&lt;/p&gt;
&lt;p&gt;[3] P. Voigt, A. Von dem Bussche, The EU general data protection regulation (GDPR), in: A Practical Guide, first ed., Vol. 10, (3152676) Springer International Publishing, Cham, 2017, pp. 10–5555.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Overcoming Unimodal Bias in Multimodal Misinformation Detection</title>
      <link>https://mever.gr/post/overcoming-unimodal-bias-in-multimodal-misinformation-detection/</link>
      <pubDate>Wed, 10 Jan 2024 00:00:00 +0000</pubDate>
      <guid>https://mever.gr/post/overcoming-unimodal-bias-in-multimodal-misinformation-detection/</guid>
      <description>&lt;p&gt;In this post, we explain the basics behind our paper  &lt;a href=&#34;https://link.springer.com/article/10.1007/s13735-023-00312-6&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;“VERITE: a robust benchmark for multimodal misinformation detection accounting for unimodal bias”&lt;/a&gt;, by Stefanos-Iordanis Papadopoulos, Christos Koutlis, Symeon Papadopoulos and Panagiotis C. Petrantonakis, which has been published in the International Journal of Multimedia Information Retrieval (IJMIR).&lt;/p&gt;
&lt;p&gt;Given the rampant spread of misinformation, the role of fact-checkers becomes increasingly important and their task more challenging given the sheer volume of content generated and shared daily across social media platforms. Studies reveal that multimedia content, with its visual appeal, tends to capture attention more effectively and enjoys broader dissemination compared to plain text (&lt;a href=&#34;https://journals.sagepub.com/doi/full/10.1177/0022243719881113&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Li &amp;amp; Xie, 2019&lt;/a&gt;) and the inclusion of images can notably amplify the persuasiveness of false statements (&lt;a href=&#34;https://link.springer.com/article/10.3758/s13423-012-0292-0&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Newman et al., 2012&lt;/a&gt;). It is for that reason that Multimodal Misinformation (MM) is very concerning. MM involves false or misleading information disseminated through diverse “modalities” of communication, including text, images, audio, and video. Scenarios often unfold where an image is removed from its authentic context, or its accompanying caption distorts critical elements, such as the event, date, location, or the identity of the depicted person. For instance, in the above figure (a) we observe an image where the grounds of a musical festival are covered in garbage, accompanied by the claim that it was taken in June 2022 “after Greta Thunberg’s environmentalist speech”. However, the image &lt;a href=&#34;https://www.snopes.com/fact-check/glastonbury-greta-thunberg/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;was removed from its authentic context&lt;/a&gt;, since it was actually taken in 2015, not 2022.&lt;/p&gt;
&lt;p&gt;In response, researchers have been developing a range of AI-based methods for detecting MM in recent years. Typically, AI-based approaches for MM detection require two things: 1) a sufficiently large training dataset, and 2) deep learning model(s) capable of analyzing and recognizing patterns across both visual and textual modalities. Yet, the process of collecting and annotating a large dataset is cumbersome, time-consuming, and resource-intensive. To circumvent this problem, researchers have been experimenting with the creation of algorithmically created or “synthetic” misinformation. This synthetic data aims to mimic, to a certain extent, the characteristics of real-world misinformation, facilitating the training of deep learning models. Subsequently, to assess the models&amp;rsquo; performance and their ability to generalize to authentic instances of misinformation, researchers evaluate them using annotated datasets featuring real-world examples of MM (&lt;a href=&#34;https://dl.acm.org/doi/fullHtml/10.1145/3592572.3592842&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Papadopoulos et al., 2023&lt;/a&gt;).&lt;/p&gt;
&lt;p&gt;In our recent &lt;a href=&#34;https://link.springer.com/article/10.1007/s13735-023-00312-6&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;research paper&lt;/a&gt;, we uncovered the existence of unimodal bias within widely used evaluation benchmarks for MM detection, specifically &lt;a href=&#34;https://github.com/MKLab-ITI/image-verification-corpus&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;VMU Twitter&lt;/a&gt; (MediaEval2016)  and &lt;a href=&#34;https://github.com/shivangi-aneja/COSMOS&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;COSMOS&lt;/a&gt;. The term “unimodal bias” denotes the phenomenon where a model relying solely on one modality (text-only or image-only) appears to outperform its multimodal counterpart in what is supposedly a multimodal task. This observation raised a few critical questions: what are the underlying reasons for unimodal bias? Can we reliably assess progress in the field of MM detection with these datasets? If not, how can we create a new evaluation benchmark that effectively addresses the challenge of unimodal bias?&lt;/p&gt;
&lt;p&gt;First and foremost, we posit that a key contributor to unimodal bias is what we term “asymmetric multimodal misinformation” (Asymmetric-MM). We introduce the term Asymmetric-MM to emphasize cases where a single “dominant” modality plays a pivotal role in disseminating misinformation, while other modalities exert minimal or no influence. Illustratively, in the above figure (c) a claim about “deceased people turning up to vote” is accompanied by an image that is thematically related to the claim, which primarily serves a cosmetic purpose. By concentrating on the dominant modality, a robust text-only detector would effectively identify misinformation, making the contribution of the other modality negligible in the detection process.&lt;/p&gt;
&lt;p&gt;Furthermore, we introduce the concept of “modality balancing” as a proactive measure to counteract unimodal bias. In this framework, all images and captions are included twice during the evaluation process – once in their truthful pair and once in their misleading pair. The underlying rationale is to necessitate that a model consistently considers both modalities and their interrelation when discerning between truthfulness and misinformation. By adopting this approach, we aim to prevent text-only and image-only detectors from achieving seemingly high accuracy solely by focusing on one modality.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/verite_collection_process.png&#34; alt=&#34;verite_collection_process&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Data collection, filtering and refinement process for creating VERITE. Figure by Stefanos-Iordanis Papadopoulos&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;By leveraging modality balancing, excluding Asymmetric-MM and establishing rigorous criteria, we curated data from reputable fact-checking sources, namely snopes.com and reuters.com/fact-check, and created the VERITE benchmark, which stands for “VERification of Image-TExt pairs”. The above figure illustrates the data collection process for creating VERITE. Starting from articles centered around “Miscaptioned Images” and adhering to a set of inclusion criteria, we identified the authentic claim related to the image (depicted by continuous blue lines) and juxtaposed it with the false “miscaptioned” caption (MC) (illustrated by intermittent red lines). Subsequently, we formulated a query associated with the truthful claim to retrieve a related but out-of-context (OOC) image (depicted by dotted yellow lines). Thus, VERITE consists of three categories: Truthful, Out-of-context, and Miscaptioned image-caption pairs and can be used both for multiclass and binary classification tasks. We collected 260 articles from Snopes and 78 from Reuters that met our criteria resulting in 338 Truthful, 338 Miscaptioned and 324 Out-of-Context pairs.&lt;/p&gt;
&lt;p&gt;We undertook a comprehensive comparative analysis based on a Transformer-based architecture trained on various established training datasets as well as a novel method of our own for algorithmically created training data. Our findings demonstrate that VERITE successfully mitigates unimodal bias, making it a robust evaluation benchmark and more reliable as a means of assessing progress in the field of MM detection. Our proposed method, which is a state-of-the-art deep learning method, reached up to 72.7% accuracy for “out-of-context” pairs, 63.9% for “miscaptioned images” and 52.1% overall accuracy for the multiclass classification task. This leaves ample room for further experimentation and improvement for the task of MM detection and we have provided a series of different routes that future research could explore. VERITE is publicly available on &lt;a href=&#34;https://zenodo.org/records/10474450&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;zenodo&lt;/a&gt; and &lt;a href=&#34;https://github.com/stevejpapad/image-text-verification&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;github&lt;/a&gt;.&lt;/p&gt;
&lt;h2 id=&#34;references&#34;&gt;References&lt;/h2&gt;
&lt;p&gt;[1] Li Y, Xie Y (2020) Is a picture worth a thousand words? an empirical study of image content and social media engagement. J Mark Res 57(1):1–19.&lt;/p&gt;
&lt;p&gt;[2] Newman EJ, Garry M, Bernstein DM et al (2012) Nonprobative photographs (or words) inflate truthiness. Psychon Bull Rev 19:969–974.&lt;/p&gt;
&lt;p&gt;[3] Papadopoulos, S. I., Koutlis, C., Papadopoulos, S., &amp;amp; Petrantonakis, P. (2023, June). Synthetic Misinformers: Generating and Combating Multimodal Misinformation. In Proceedings of the 2nd ACM International Workshop on Multimedia AI against Disinformation (pp. 36-44).&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Better, Faster and Lighter Video Retrieval</title>
      <link>https://mever.gr/post/better-faster-and-lighter-video-retrieval/</link>
      <pubDate>Tue, 26 Jul 2022 00:00:00 +0000</pubDate>
      <guid>https://mever.gr/post/better-faster-and-lighter-video-retrieval/</guid>
      <description>&lt;p&gt;In this post, we explain the basics behind our paper  &lt;a href=&#34;https://arxiv.org/abs/2106.13266&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;“DnS: Distill-and-Select for Efficient and Accurate Video Indexing and Retrieval”&lt;/a&gt;, which has been accepted for publication in the &lt;a href=&#34;https://www.springer.com/journal/11263&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;International Journal of Computer Vision&lt;/a&gt; (IJCV).&lt;/p&gt;
&lt;p&gt;Following our prior work in ViSiL, we realised that high-performing video retrieval systems require a significant amount of compute and storage resources. More precisely, our network achieved state-of-the-art performance on several video retrieval problems; however, it needed substantial time for similarity calculation and memory space for storing the video features.&lt;/p&gt;
&lt;p&gt;To overcome this limitation, we have developed a reranking method called Distill-and-Select (DnS) that offers an excellent trade-off between performance and efficiency, leveraging Knowledge Distillation and devising a Selection Mechanism.&lt;/p&gt;
&lt;p&gt;Our reranking system consists of three main networks: (i) a coarse-grained student () that provides very fast retrieval speed but with low retrieval performance, (ii) a fine-grained student () that has high retrieval performance but with high computational cost, and (iii) a selector network () that routes the similarity calculation of the video pairs and provides a balance between performance and time efficiency.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/retrieval_overview-1024x412.png&#34; alt=&#34;retrieval_system&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Overview of our DnS reranking video retrieval system. Figure by Giorgos Kordopatis-Zilos&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;As displayed in the above figure, the processing can be split into two phases, i.e., Indexing and Retrieval. At indexing time, the feature representations of all videos in the database are extracted and stored in the video index. Each of our three networks extracts its corresponding representation, which is necessary for the retrieval phase. We use the notation  for each network  to denote the indexing process. At retrieval time, all possible query-target video pairs are sent by the selector network to the coarse-grained student so that their similarity is rapidly estimated. Considering the results and the self-similarity of the two videos, the SN takes a binary decision on whether the calculated coarse similarity needs to be refined by the fine-grained student, and for the small percentage of videos that this is needed, the fine-grained network calculates their similarity. Similarly, we use the notation  for each network  to denote the retrieval process.&lt;/p&gt;
&lt;p&gt;In what follows, we will go through the basics regarding the training of the networks and the architectures employed for the teacher, students and selector networks.&lt;/p&gt;
&lt;h2 id=&#34;knowledge-distillation&#34;&gt;Knowledge Distillation&lt;/h2&gt;
&lt;p&gt;Our goal here is to build a training process in order to train our student networks to mimic the behaviour of a well-performing teacher network and generate similarity scores that are close to the teacher’s. In that way,  we can train several networks on different retrieval performance-computational efficiency trade-offs.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/Untitled-Project2-6.gif&#34; alt=&#34;training_process&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Illustration of the training process of the student networks. Figure by Giorgos Kordopatis-Zilos&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;The training process is straightforward and displayed in the above figure. First, a video pair is selected by a large unlabeled dataset. Then, the teacher calculates the similarity between the two videos and generates their similarity score . Next, the student calculates its own similarity score  between the two videos, and the loss is derived based on the difference between the scores generated by the teacher and the student as in:&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/svg.svg&#34; alt=&#34;code&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;
&lt;/p&gt;
&lt;p&gt;Note that the loss is defined on the output of the teacher, which allows us to use any arbitrary unlabelled dataset for the training of our students.&lt;/p&gt;
&lt;p&gt;Additionally, in this setting, the selection of the training pairs is crucial. Since it is very time-consuming to apply the teacher network to every pair of videos in the dataset ( complexity) and randomly selecting videos would result in mostly pairs with low similarity scores, we have devised a pair selection scheme to select pairs of interest.&lt;/p&gt;
&lt;p&gt;Since our previous ViSiL network achieves state-of-the-art performance, we use it as a teacher to our framework. We have designed three students, two fine-grained and one coarse-grained variant, each providing different benefits. The fine-grained students are both using the ViSiL architecture. The first fine-grained student simply introduces more trainable parameters with a more complex attention scheme, leading to better performance with similar computational and storage requirements to the teacher. It is called fine-grained attention student (). The second fine-grained student optimises a binarization function that hashes features into a Hamming space and has very low storage space requirements for indexing with little performance sacrifice. It is called fine-grained binarization student (). Keep in mind that in our final DnS system, only one of the two fine-grained students is deployed as the  network mentioned above. The third coarse-grained student learns to aggregate the region-level feature vectors based on a Transformer+NetVLAD network architecture in order to generate global video-level representations. It needs considerably fewer resources for indexing and retrieval but at a notable performance loss. It is called coarse-grained student ().&lt;/p&gt;
&lt;h2 id=&#34;selection-mechanism&#34;&gt;Selection Mechanism&lt;/h2&gt;
&lt;p&gt;This is a reranking process based on a selector network that judges whether the similarities of video pairs calculated by a coarse-grained student  are reliable or need to be refined by a fine-grained student .&lt;/p&gt;
&lt;p&gt;Typically, the similarity between two videos estimated by a fine-grained student leads to better retrieval scores than the one estimated by the coarse-grained student. However, for some video pairs, the difference between them is small, which has a negligible effect on the ranking and on whether the video will be retrieved. So, we use a selector network trained to distinguish between those video pairs and pairs of videos that exhibit large similarity differences.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/Untitled-Project-3.gif&#34; alt=&#34;selector_networks&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Illustration of the training process of the selector networks. Figure by Giorgos Kordopatis-Zilos&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;The selector network is trained as a binary classifier with Binary Cross-Entropy (BCE) used as a loss function. We first derive video pairs from an unlabeled dataset, and their  and  similarities are calculated based on the coarse- and fine-grained students, respectively. Then, we obtain binary labels by setting a threshold for the difference between the two similarity scores:&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/svg%20%281%29.svg&#34; alt=&#34;code&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;
&lt;/p&gt;
&lt;p&gt;The selector network gets as input the similarity calculated by the coarse-grained student and the videos of the pair in question. First, it processes each video independently in order to capture their self-similarities, using a modified ViSiL head. Then, these self-similarity scores are combined with the coarse-grained similarity through an MLP network to derive the confidence score , which is the final output of the selector.&lt;/p&gt;
&lt;p&gt;As stated above, the selector network is trained with the BCE loss calculated based on its confidence score and the binary label derived from the label function.&lt;/p&gt;
&lt;p&gt;During retrieval, the selector takes a binary decision based on its confidence score. If it is closer to 1, then the fine-grained student is invoked; otherwise, the similarity of the coarse-grained student is considered for the ranking.&lt;/p&gt;
&lt;h2 id=&#34;experimental-results&#34;&gt;Experimental results&lt;/h2&gt;
&lt;p&gt;The figure below displays the performance of our proposed DnS framework and its variants against several state-of-the-art methods in terms of retrieval performance based on mean Average Precision (mAP) on the DSVR task of &lt;a href=&#34;https://arxiv.org/abs/1809.04094&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;FIVR-200K&lt;/a&gt; and efficiency based on the required time per query and storage space per video. Coarse-grained methods are in blue, fine-grained in red, and re-ranking in orange.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/bitmap-720x773.png&#34; alt=&#34;bitmap&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;
&lt;/p&gt;
&lt;p&gt;Our approaches achieve the best performance, requiring very little compute resources, especially in comparison to the ViSiL teacher network, which was the previous state-of-the-art. In particular, when the fine-grained binarization student is used with 5% of the dataset sent for reranking, our DnS method achieves 55 times faster retrieval with 230 times less storage space, with a minor drop of 0.03 in mAP compared to the ViSiL.&lt;/p&gt;
&lt;p&gt;Additionally, DnS achieved very competitive performance on several other video retrieval datasets, i.e., the rest &lt;a href=&#34;https://ndd.iti.gr/fivr/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;FIVR-200K tasks&lt;/a&gt;, &lt;a href=&#34;http://vireo.cs.cityu.edu.hk/webvideo/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;CC_WEB_VIDEO&lt;/a&gt;, &lt;a href=&#34;http://pascal.inrialpes.fr/data2/evve/index.html&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;EVVE&lt;/a&gt;, and &lt;a href=&#34;https://svdbase.github.io/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;SVD&lt;/a&gt;. For more details regarding the network architectures, training of the models, and comprehensive experimental results, we point readers to our DnS paper. Also, the implementation of DnS is publicly available.&lt;/p&gt;
&lt;p&gt;Based on the developed approach, we have developed a web service, a short demo of which is available in the video below. Please contact us if you are interested in trying it out.&lt;/p&gt;
&lt;figure&gt;
  &lt;div class=&#34;&#34; style=&#34;height:400px&#34;&gt;
    &lt;iframe title=&#34;Near-duplicate Detection on Reaction Videos&#34; width=&#34;1170&#34; height=&#34;658&#34; src=&#34;https://www.youtube.com/embed/nPeC0_tGgnk?feature=oembed&#34; frameborder=&#34;0&#34; allow=&#34;accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture&#34; allowfullscreen=&#34;&#34; style=&#34;width: 100%;height: 100%;&#34;&gt;&lt;/iframe&gt;
  &lt;/div&gt;
&lt;/figure&gt;
&lt;h2 id=&#34;references&#34;&gt;References&lt;/h2&gt;
&lt;p&gt;[1] Kordopatis-Zilos G., TzelepisPapadopoulos S., Kompatsiaris, I, and Patras I. (2022). &lt;a href=&#34;https://arxiv.org/abs/2106.13266&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;DnS: Distill-and-Select for Efficient and Accurate Video Indexing and Retrieval&lt;/a&gt;. arXiv preprint arXiv:2106.13266.&lt;/p&gt;
&lt;p&gt;[2] Kordopatis-Zilos G., Papadopoulos S., Patras I., and Kompatsiaris, Y. (2019). ViSiL: &lt;a href=&#34;https://openaccess.thecvf.com/content_ICCV_2019/papers/Kordopatis-Zilos_ViSiL_Fine-Grained_Spatio-Temporal_Video_Similarity_Learning_ICCV_2019_paper.pdf&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Fine-grained Spatio-Temporal Video Similarity Learning&lt;/a&gt;. In International Conference on Computer Vision.&lt;/p&gt;
&lt;p&gt;[3] Kordopatis-Zilos G., Papadopoulos S., Patras I., and Kompatsiaris, Y. (2019). &lt;a href=&#34;https://arxiv.org/abs/1809.04094&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;FIVR: Fine-grained Incident Video Retrieval&lt;/a&gt;. IEEE Transactions on Multimedia, 2019.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Online Disinformation in Europe</title>
      <link>https://mever.gr/post/online_disinformation_in_europe/</link>
      <pubDate>Tue, 24 May 2022 00:00:00 +0000</pubDate>
      <guid>https://mever.gr/post/online_disinformation_in_europe/</guid>
      <description>&lt;p&gt;Last year, we had the opportunity to conduct a desk research study in an effort to map the landscape of online disinformation in Europe. Our research was guided by three research questions:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;What are the measures and journalistic authorities in Europe that supervise and monitor the ethical application of journalism?&lt;/li&gt;
&lt;li&gt;Are there patterns of false information spreading as a means to serve the agenda and interests of certain political groups? Are there certain political groups that benefit as a result of given disinformation narratives?&lt;/li&gt;
&lt;li&gt;Are there specific and tangible policies that have been proposed by European and international organisations that could help tackle online disinformation?&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;To make the study of the first two questions more manageable, we focused on five European countries: France, Germany, Greece, Italy and Spain, and on three topics that are commonly associated with disinformation: COVID-19, immigration, and climate change. The resulting report is available here and we believe it could be valuable for informing a number of stakeholders, including professionals in the media industry and policy makers. In the following, we try to present the key insights that we managed to draw from this report.&lt;/p&gt;
&lt;p&gt;&lt;em&gt;What are the measures and journalistic authorities in Europe that supervise and monitor the ethical application of journalism?&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;The primary organizations overseeing the application of ethical standards in journalism in Europe and worldwide are the &lt;a href=&#34;https://erga-online.eu/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;European Regulators Group for Audiovisual Media Services&lt;/a&gt; (ERGA), the &lt;a href=&#34;https://europeanjournalists.org/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;European Federation of Journalists&lt;/a&gt; (EFJ) and the &lt;a href=&#34;https://www.poynter.org/ifcn/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;International Fact-Checking Network&lt;/a&gt; (IFCN). European countries follow their own national codes of ethical conduct, though there are certain commonalities across the conducts (cf. figure below) and truthful reporting is among these common principles.An additional important reference document in Europe is the &lt;a href=&#34;https://digital-strategy.ec.europa.eu/en/policies/code-practice-disinformation&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Code of Practice on Disinformation&lt;/a&gt;, which includes relevant commitments on behalf of social media platforms and internet advertising companies. Finally, our study points out that complying with ethical standards is increasingly challenging for news organizations and journalists due to the 24/7 news lifecycle, the reliance on Internet advertising as a primary source of income and several other factors that affect media independence and plurality. Yet, failure to comply with ethical standards is found as a reason for the reduced trust of citizens in the media.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/codes_ethics.png&#34; alt=&#34;codes_ethics&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Summary of Codes of Ethics for EFJ and five studied European countries. Picture taken from &lt;a href=&#34;https://mever.gr/media/disinformation-study-v1.9.pdf&#34;&gt;Online Disinformation in Europe Report&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Are there patterns of false information spreading as a means to serve the agenda and interests of certain political groups? Are there certain political groups that benefit as a result of given disinformation narratives?&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Among the three topics of focus for our study, disinformation was prevalent with respect to COVID-19 and immigration and much less pronounced with respect to climate change. At an EU and national level, we found evidence that COVID-19 related disinformation more often originates or is disseminated by far-right wing parties and politicians, while at an international level disinformation campaigns targeting European citizens appear to mainly originate from Russia and China and primarily target Germany and Italy. On the topic of immigration, we could identify several disinformation activities, featuring anti-immigrant narratives and sentiments, racist and xenophobic attitudes that were aligned with the agenda and ideology of far-right and rightwing parties, and sometimes pro-Kremlin media amplify the messages of far-right politicians (e.g. in Germany); however, there is no evidence of any kind of cooperation or coordination between them. On the topic of climate change, there seems to be a shift from climate change denialism to skepticism, but in general it appears that European countries are not fertile ground for climate change disinformation.&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Are there specific and tangible policies that have been proposed by European and international organisations that could help tackle online disinformation?&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;There are already a number of reports that recommend concrete policies and measures with a view to mitigating the effects of disinformation and limiting its root factors. Our analysis of existing recommendations highlights that the phenomenon of disinformation cannot be addressed with fragmented, one-dimensional or simply regulatory policies. It calls for a multi-dimensional, multi-faceted, multi-stakeholder policy framework that assigns fair responsibility to and requires decisive action from all relevant stakeholders. In particular, we present a six-dimensional policy framework (cf. figure below) that could be a useful reference for discussions among policy makers and other stakeholders. The recommended policies are organized in the following six dimensions: a) enhancement of the transparency of the digital media ecosystem; b) cultivation of media literacy and digital skills in different groups of citizens; c) empowerment of different groups of stakeholders, including platform users, citizens, and journalists; d) strengthening media independence and pluralism; e) promotion of ethical conduct in media, journalism and platforms; and f) support of independent research on monitoring the disinformation phenomenon and building services and tools for countering online disinformation.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/dis_framework.png&#34; alt=&#34;dis_framework&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Overview of six-dimensional policy framework to tackle disinformation. &lt;a href=&#34;https://mever.gr/media/disinformation-study-v1.9.pdf&#34;&gt;Picture taken from Online Disinformation in Europe Report&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;The report was co-authored and researched by Ms. Georgia Pantalona, Dr. Filareti Tsalakanidou, Dr. Evangelia Kartsounidou, Dr. Symeon Papadopoulos, Dr. Nikos Sarris and Dr. Yiannis Kompatsiaris. The study was commissioned by the &lt;a href=&#34;https://left.eu/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Left group in the European Parliament&lt;/a&gt; – GUE/NGL. To ensure the objectivity and transparency of the research, our research team conducted the study in a fully autonomous way without any intervention by the EP members. We made every possible effort to avoid expressing directly or indirectly any personal opinion or stance by the authors in this document and therefore we limited ourselves to analysing, synthesizing, summarizing and interpreting conclusions and evidence from primary sources. Last but not least, the researchers involved in the execution of the study have no direct or indirect association or affiliation with any political party or fraction. In addition, they have no further benefit or dependence on the content and conclusions of this study.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Geolocating images in the globe with high accuracy</title>
      <link>https://mever.gr/post/geolocating-images-in-the-globe-with-high-accuracy/</link>
      <pubDate>Fri, 21 May 2021 00:00:00 +0000</pubDate>
      <guid>https://mever.gr/post/geolocating-images-in-the-globe-with-high-accuracy/</guid>
      <description>&lt;p&gt;In this post we explain the basics behind our paper  &lt;a href=&#34;https://arxiv.org/abs/2105.07645&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;“Leveraging EfficientNet and Contrastive Learning for Accurate Global-scale Location Estimation”&lt;/a&gt; that has been accepted for publication at this year’s &lt;a href=&#34;https://icmr2021.org/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;International Conference on Multimedia Retrieval&lt;/a&gt; (ICMR 2021).&lt;/p&gt;
&lt;p&gt;The problem that we tackle in this work is the following: “Given an input image from anywhere in the world, estimate the coordinates of its geographic location, in terms of latitude and longitude, based solely on the visual information of the image.”&lt;/p&gt;
&lt;p&gt;Typically, there are two types of approaches that try to solve the problem either as a &lt;a href=&#34;https://arxiv.org/abs/1602.05314&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;classification&lt;/a&gt; problem or a &lt;a href=&#34;http://graphics.cs.cmu.edu/projects/im2gps/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;retrieval&lt;/a&gt; one. The former partitions the earth’s surface into a grid of cells and then trains a classifier per cell to assign input images to it. The latter searches in a large-scale background database of geotagged images to retrieve similar ones based on a given image query and then aggregates them for estimating its location.&lt;/p&gt;
&lt;p&gt;However, both formulations have limitations. The major drawback of classification approaches stems from the division of the earth into large geographic areas, which results in coarse estimations. On the other hand, retrieval-based approaches are more error-prone and need significantly more computational resources during inference.&lt;/p&gt;
&lt;p&gt;This work aims to address these limitations and achieve high geolocation accuracy in all granularity scales. To this end, we build on the strong aspects of both classification and retrieval approaches based on our prior work on &lt;a href=&#34;https://ieeexplore.ieee.org/document/7998610&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;text-based geolocation&lt;/a&gt;. More precisely, we develop a Search within Cell (SwC) scheme, which consists of two modules, i.e., a classification and a retrieval one. An overview of our system is displayed below.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/method_overview.png&#34; alt=&#34;SwC&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Overview of the proposed Search within Cell (SwC) scheme. Figure by Giorgos Kordopatis-Zilos&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;For the classification module, we first partition the earth’s surface into a grid of cells. Then, we employ &lt;a href=&#34;https://arxiv.org/abs/1905.11946&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;EfficientNet&lt;/a&gt;, a state-of-the-art architecture, in order to map input images to a specific cell in the grid. We build the model based on a probabilistic approach of the geolocation task that uses a &lt;a href=&#34;https://ecmlpkdd2019.org/downloads/paper/63.pdf&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Mixture of von Mises-Fisher distributions (MvMF)&lt;/a&gt;. The network is trained by minimizing the log-likelihood loss with a large corpus of geotagged images.&lt;/p&gt;
&lt;p&gt;For the retrieval module, we first extract features for our images from the backbone CNN trained for the classification module, and then we feed them to a Residual Retrieval Module (RRM) that learns to map images to an embedding space where the images from the same location are closer to each other than the rest. We train our module based on the &lt;a href=&#34;https://arxiv.org/abs/2002.05709&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;contrastive learning paradigm&lt;/a&gt;. Once we have trained our RRM module and extracted embeddings for our images, we can compute similarities and rank images from a database given a query.&lt;/p&gt;
&lt;p&gt;Once we have both our modules trained, we employ a hybrid scheme to combine them, which we call Search within Cell (SwC):&lt;/p&gt;
&lt;p&gt;-We perform classification to derive the cell with the largest probability.
-We retrieve the top-most similar images of the background collection, with the constraint that the retrieved images fall within the borders of the estimated cell.
-To enhance the robustness of the results, we develop a density-based spatial clustering scheme based on the DBSCAN algorithm, and we derive our final estimation, averaging the locations of the images that belong to the largest cluster.&lt;/p&gt;
&lt;p&gt;We evaluate our approach on four benchmark datasets. Our method exhibits very competitive performance, significantly improving the state-of-the-art in many granularity ranges. In the well-known Im2GPS3K, we manage to geolocate 15% of the dataset images within 1km of their true location, and 30% of them within 25km. We also evaluate our method with various configurations to gain insight into its behaviour. For more details, we point readers to our &lt;a href=&#34;https://arxiv.org/pdf/2105.07645.pdf&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;paper&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Based on the developed approach, we have developed a web service. Please contact us if you are interested to try it out.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/pasted%20image%200.png&#34; alt=&#34;screenshot&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Screenshot from our visual location estimation web service.&lt;/em&gt;&lt;/p&gt;
&lt;h2 id=&#34;references&#34;&gt;References&lt;/h2&gt;
&lt;p&gt;[1] Weyand, T., Kostrikov, I., &amp;amp; Philbin, J. (2016). Planet-photo geolocation with convolutional neural networks. In European Conference on Computer Vision (pp. 37-55).&lt;/p&gt;
&lt;p&gt;[2] Hays, J., &amp;amp; Efros, A. A. (2008). Im2gps: estimating geographic information from a single image. In 2008 ieee conference on computer vision and pattern recognition (pp. 1-8).&lt;/p&gt;
&lt;p&gt;[3] Kordopatis-Zilos, G., Papadopoulos, S., &amp;amp; Kompatsiaris, I. (2017). Geotagging text content with language models and feature mining. Proceedings of the IEEE, 105(10), 1971-1986.&lt;/p&gt;
&lt;p&gt;[4] Tan, M., &amp;amp; Le, Q. (2019, May). Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning (pp. 6105-6114). PMLR.&lt;/p&gt;
&lt;p&gt;[5] Izbicki, M., Papalexakis, E. E., &amp;amp; Tsotras, V. J. (2019). Exploiting the earth’s spherical geometry to geolocate images. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (pp. 3-19). Springer, Cham.&lt;/p&gt;
&lt;p&gt;[6] Chen, T., Kornblith, S., Norouzi, M., &amp;amp; Hinton, G. (2020). A simple framework for contrastive learning of visual representations. In International conference on machine learning (pp. 1597-1607). PMLR.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Operation-wise Attention Network for Tampering Localization Fusion</title>
      <link>https://mever.gr/post/operation-wise-attention-network-for-tampering-localization-fusion/</link>
      <pubDate>Thu, 13 May 2021 00:00:00 +0000</pubDate>
      <guid>https://mever.gr/post/operation-wise-attention-network-for-tampering-localization-fusion/</guid>
      <description>&lt;p&gt;In this post, we explain the basics behind our paper  &lt;a href=&#34;https://arxiv.org/abs/2105.05515&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;“Operation-wise Attention Network for Tampering Localization Fusion”&lt;/a&gt;, which has been accepted for publication at this year’s &lt;a href=&#34;https://cbmi2021.univ-lille.fr/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Content-Based Multimedia Indexing&lt;/a&gt; conference (CBMI 2021).&lt;/p&gt;
&lt;p&gt;The main motivation of this approach is to combine the outcomes of multiple image forensics algorithms and to provide a fused tampering localization map, which requires no expert knowledge and is easier to interpret by end users. Our fusion framework includes a set of five individual tampering localization methods for splicing localization on JPEG images:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;ADQ1 [1] and DCT [2] that both base their detection on analysis of the JPEG compression, in the transform domain.&lt;/li&gt;
&lt;li&gt;BLK [3] and CAGI [4] that base their detection on analysis of the JPEG compression in the spatial domain.&lt;/li&gt;
&lt;li&gt;Splicebuster [5], a noise-based detector selected as a complementary method due to its high reported performance and good interpretability of its produced outputs.
To fuse the localization signals of the above algorithms, we adapt an architecture that was initially proposed for the image restoration task [6]. This architecture (OwAF) performs multiple operations in parallel, weighted by an attention mechanism to enable the selection of proper operations depending on the input signals. This weighting process can be very beneficial for cases where the input signal is very diverse, as in our case where the output signals of multiple image forensics algorithms are combined.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/Fusion.png&#34; alt=&#34;attention_layer&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;The architecture of adapted operation-wise attention layer, comprising an attention and operation layer, a concatenation operation, and 1 × 1 convolution&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;This architecture is suitable for our problem because it uses attention to capture important features by examining which operations are the most beneficial, depending on the input signal. Another important aspect of this architecture is that it can learn to attend low-level features, which is important for the fusion task, as semantic or high-level representations are often not relevant or useful for the problem. The network includes operations like, dilated convolutions, separable convolutions, and pooling with various kernel sizes and rates, which are weighted by an attention mechanism. To adapt this architecture to our needs, we replaced dilated convolutions with simple convolutions as the former are better suited for the image restoration task and exhibit lower validation performance in preliminary experiments.&lt;/p&gt;
&lt;p&gt;For the model training in the fusion task, we generate a set of tampering localization outputs by applying the forensics algorithms for each image in the dataset. For this work, we use the &lt;a href=&#34;https://defactodataset.github.io/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;DEFACTO dataset&lt;/a&gt;, which contains more than 150,000 images with synthetic forgeries and their corresponding ground truth masks. We randomly compile 15,000 images for training, 1000 for test, and 1000 for validation. For every input image in each set, we produce a set of tampering localization maps obtained by the selected detection methods. We evaluate the proposed deep learning models against the DEFACTO test dataset and two image forgery localization datasets, namely CASIA V2 and IFS-TC Image Forensics Challenge dataset. In the reported results besides the individual algorithms we also compared our method with another proposed architecture (Eff-B4-Unet) and another fusion framework (Iakovidou et al [7])&lt;/p&gt;
&lt;p&gt;The OwAF model outperforms the Eff-B4-Unet in every evaluation metric for the DEFACTO test dataset. Evaluation results for individual algorithms are very low when compared to the fusion approaches. For the case of CASIA v2 we can observe that manipulations can be localized better by the individual algorithms. Regarding the fusion methods evaluation, our approach outperforms the competing fusion framework from Iakovidou et al. Regarding results on the First IFS-TC Image Forensics Challenge dataset, we observe a significant decrease in the performance of the individual algorithms with the exception of the Spicebuster algorithm, which even outperformed both fusion approaches.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/eval_results.png&#34; alt=&#34;visual_comparisons&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Successful examples of tampering localization outputs from the OwAF model and visual comparisons with the individual methods and Iakovidou et al.; for the first four rows, images are taken from the CASIA v2 dataset, for the last four rows from the First IFS-TC Image Forensics Challenge dataset.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;From our experiments, it is evident that the main challenge of the proposed approach stems from overfitting to the training data.  Although OwAF can  outperform  individual  forensics algorithms  or  other  fusion  models,  we  observe  a  lack  of generalization to unseen manipulations. Namely, we get poor predictions for datasets that have different types of manipulations compared to those that appeared in the training dataset. Additionally, the low  evaluation  performance  of  individual models  is  an indication  that  the  forgery  localization problem  is  very  challenging  and  it is  even harder  to design a general fusion solution for images in the wild.&lt;/p&gt;
&lt;h2 id=&#34;references&#34;&gt;References&lt;/h2&gt;
&lt;p&gt;[1] Z. Lin, J. He, X. Tang, and C.-K. Tang, “Fast, automatic and fine-grained tampered jpeg image detection via dct coefficient analysis,”Pattern Recognition, 2009.
[2] S. Ye, Q. Sun, and E.-C. Chang, “Detecting digital image forgeries by measuring inconsistencies of blocking artifact,” in 2007 IEEE International Conference on Multimedia and Expo, 2007.
[3] W. Li, Y. Yuan, and N. Yu, “Passive detection of doctored jpeg image via block artifact grid extraction,”Signal Processing, 2009.
[4] C. Iakovidou, M. Zampoglou, S. Papadopoulos, and Y. Kompatsiaris,“Content-aware detection of jpeg grid inconsistencies for intuitive image forensics,” Journal of Visual Communication and Image Representation, 2018.
[5] D. Cozzolino, G. Poggi, and L. Verdoliva, “Splicebuster: A new blindimage splicing detector,” in 2015 IEEE International Workshop onInformation Forensics and Security (WIFS), 2015.
[6] M. Suganuma, X. Liu, and T. Okatani, “Attention-based adaptive se-lection of operations for image restoration in the presence of unknown combined distortions,” in Proceedings of the IEEE/CVF Conference onComputer Vision and Pattern Recognition, 2019.
[7] C. Iakovidou, S. Papadopoulos, and Y. Kompatsiaris, “Knowledge-based fusion for image tampering localization,” inIFIP International Confer-ence on Artificial Intelligence Applications and Innovations, 2020.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Video similarity based on audio</title>
      <link>https://mever.gr/post/video-similarity-baed-on-audio/</link>
      <pubDate>Tue, 24 Nov 2020 00:00:00 +0000</pubDate>
      <guid>https://mever.gr/post/video-similarity-baed-on-audio/</guid>
      <description>&lt;p&gt;In this post, we explain the basics behind our paper  &lt;a href=&#34;https://arxiv.org/abs/2010.08737&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;“Audio-based Near-Duplicate Video Retrieval with Audio Similarity Learning,”&lt;/a&gt; which has been accepted for publication at this year’s &lt;a href=&#34;https://www.micc.unifi.it/icpr2020/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;International Conference on Pattern Recognition&lt;/a&gt; (ICPR 2020).&lt;/p&gt;
&lt;p&gt;So, let’s consider the audio signals of two arbitrary videos as the ones displayed below. The left video serves as a query and the right as a target match.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/audio_examples.png&#34; alt=&#34;signals&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Sampled keyframes and audio signals of a query and a target video. Figure by Giorgos Kordopatis-Zilos&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;To calculate the similarity between the two videos, we have to extract feature descriptors for the audio signals of the videos and then calculate the similarity between them as the final similarity score of the video pair. In this work, we have designed two processes that implement these functionalities: (i) a feature extraction scheme based on transfer learning from a pre-trained Convolutional Neural Network (CNN), and (ii) a similarity calculation process based on video similarity learning.&lt;/p&gt;
&lt;h2 id=&#34;feature-extraction&#34;&gt;Feature extraction&lt;/h2&gt;
&lt;p&gt;Let’s begin with feature extraction. We employ the &lt;a href=&#34;https://arxiv.org/abs/1711.01369&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;pre-trained CNN network designed for transfer learning&lt;/a&gt;. The network is trained on a large-scale dataset, namely &lt;a href=&#34;https://research.google.com/audioset/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;AudioSet&lt;/a&gt;, consisting of approximately 2.1 million weakly-labeled videos from YouTube with 527 audio event classes.&lt;/p&gt;
&lt;p&gt;To extract features, we first generate the &lt;a href=&#34;https://research.google.com/audioset/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Mel-filtered spectrogram&lt;/a&gt; from the audio of the videos. The audio signals are resampled at 44.1kHz sampling frequency. For the spectrogram generation, we use 128 Mel-bands and a window size of 23ms (1024 samples at 44.1kHz) with an overlap of 11.5ms  (512 hop size).  The generated spectrograms are divided into overlapping time frames of 2 seconds with t seconds time step.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/spectrogram-1-768x274.png&#34; alt=&#34;spectrofram&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Illustration of spectrogram extraction from the audio signal. Figure by Giorgos Kordopatis-Zilos&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Then, the generated spectrogram frames are fed to a feature extraction CNN  that consists of seven blocks. Each block contains two convolutional layers, with batch normalization and ReLU activation,  followed by a max-pooling layer. We extract a compact audio representation for each spectrogram frame by applying &lt;a href=&#34;https://mever.gr/wp-content/uploads/2020/03/Kordopatis-Zilos2017_Chapter_Near-DuplicateVideoRetrievalBy.pdf&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Maximum Activation of Convolutions (MAC) on the activations of the intermediate convolutional layers&lt;/a&gt;. In that way, we generate a vector representation  for each audio frame. To improve the discriminative capabilities of the audio descriptors, we then apply PCA whitening and an attention-based scheme for the decorrelation and weighting of the extracted feature vectors respectively.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/feature_extraction.png&#34; alt=&#34;spectrofram_feature extraction&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Feature extraction from the spectrograms of the audio signals. Figure by Giorgos Kordopatis-Zilos&lt;/em&gt;&lt;/p&gt;
&lt;h2 id=&#34;similarity-calculation&#34;&gt;Similarity calculation&lt;/h2&gt;
&lt;p&gt;To measure the similarity between the two compared videos, we employ the &lt;a href=&#34;https://arxiv.org/abs/1908.07410&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;video similarity learning&lt;/a&gt; scheme for the robust and accurate similarity calculation. More precisely, having extracted the audio representation of the two videos, we can now calculate the similarity between all the descriptor pairs of the two videos. In our example, let’s consider that our videos q, p have X, Y audio frames, respectively. We calculate the similarity between the feature vectors of the corresponding video descriptors by applying the dot product. In that way, we generate a pairwise similarity matrix , that contains the similarities between all vectors of the two videos.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/pairwise_matrix.png&#34; alt=&#34;pairwise_matrix&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Calculation of the pairwise similarity matrix between audio descriptors of two videos. Figure by Giorgos Kordopatis-Zilos&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Then, to calculate the similarity between the two videos, we provide the refined pairwise similarity matrix to a CNN network, which we call AuSiL. The network captures the temporal similarity structures existing within the content of the similarity matrix, and it is capable of learning robust patterns of within-video similarities. In our examples, the network can detect temporal patterns and assign high similarity scores in the corresponding segments, i.e., the diagonal part existing in the center of the similarity matrix.  At the same time, the noise in the input matrix, introduced by the similarity calculation process,  has been significantly reduced in the output.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/video_similarity.png&#34; alt=&#34;video_similarity&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Calculation of video similarity. Figure by Giorgos Kordopatis-Zilos&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;To calculate the final video similarity, we apply the hard tanh activation function on the values of the network output, which clips values within range [-1, 1]. Finally, we apply Chamfer Similarity to derive a single value, which is considered as the final similarity between the two videos.&lt;/p&gt;
&lt;h2 id=&#34;experimental-results&#34;&gt;Experimental results&lt;/h2&gt;
&lt;p&gt;For the evaluation of the proposed approach, we employ two datasets compiled for fine-grained incident and near-duplicate video retrieval, i.e., &lt;a href=&#34;https://ndd.iti.gr/fivr/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;FIVR-200K&lt;/a&gt; and &lt;a href=&#34;https://svdbase.github.io/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;SVD&lt;/a&gt;. We have manually annotated the videos in the dataset according to their audio duplicity with the set of query videos. Also, we evaluate the robustness of our approach to audio speed transformations by artificially generating audio duplicates.&lt;/p&gt;
&lt;p&gt;In the following table, we compare the retrieval performance of AuSiL against &lt;a href=&#34;https://github.com/worldveil/dejavu&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;Dejavu&lt;/a&gt;, a publicly available Shazam-like system. The performance is measured based on mean Average Precision (mAP) on the two annotated datasets with two different settings, i.e., the original version and the artificially generated with speed transformation. AuSiL outperforms Dejavu by a considerable margin on three out of four runs. Dejavu achieves marginally better results on the original version of the FIVR-200K. It is evident that our approach is very robust against speed transformation, unlike the competing method.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/pasted%20image%200.png&#34; alt=&#34;mAP_comparison&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;mAP comparison of the proposed approach Dejavu, a publicly available  Shazam-like system. Superscript T indicates the runs with audio speed transformations.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;For more details regarding the architecture and training of the model, but also for comprehensive experimental results, feel free to have a look at the &lt;a href=&#34;https://arxiv.org/abs/2010.08737&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;AuSiL paper&lt;/a&gt;. The implementation of AuSiL is &lt;a href=&#34;https://github.com/mever-team/ausil&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;publicly available&lt;/a&gt;.&lt;/p&gt;
&lt;figure&gt;
  &lt;div class=&#34;&#34; style=&#34;height:400px&#34;&gt;
    &lt;iframe title=&#34;Audio-based Near-duplicate Video Rertrieval with Audio Similarity Learning &#34; width=&#34;1170&#34; height=&#34;658&#34; src=&#34;https://www.youtube.com/embed/MMHCYS7v-vw&#34; frameborder=&#34;0&#34; allow=&#34;accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture&#34; allowfullscreen=&#34;&#34; style=&#34;width: 100%;height: 100%;&#34;&gt;&lt;/iframe&gt;
  &lt;/div&gt;
&lt;/figure&gt;
&lt;h2 id=&#34;references&#34;&gt;References&lt;/h2&gt;
&lt;p&gt;[1] Avgoustinakis, P., Kordopatis-Zilos, G., Papadopoulos, S., Symeonidis, A. L., &amp;amp; Kompatsiaris, I. (2020). Audio-based Near-Duplicate Video Retrieval with Audio Similarity Learning. arXiv preprint &lt;a href=&#34;https://arxiv.org/abs/2010.08737&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;arXiv:2010.08737&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;[2] Kumar, A., Khadkevich, M., &amp;amp; Fügen, C. (2018). Knowledge transfer from weakly labeled audio using convolutional neural network for sound events and scenes. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 326-330).&lt;/p&gt;
&lt;p&gt;[3] Kordopatis-Zilos G., Papadopoulos S., Patras I., &amp;amp; Kompatsiaris, Y. (2017). Near-duplicate video retrieval by aggregating intermediate cnn layers. In international conference on Multimedia Modeling.&lt;/p&gt;
&lt;p&gt;[4] Kordopatis-Zilos, G., Papadopoulos, S., Patras, I., &amp;amp; Kompatsiaris, I. (2019). ViSiL: Fine-grained spatio-temporal video similarity learning. In Proceedings of the IEEE International Conference on Computer Vision.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>MeVer participation in Deepfake detection challenge</title>
      <link>https://mever.gr/post/mever-participation-in-deepfake-detection-challenge/</link>
      <pubDate>Tue, 16 Jun 2020 00:00:00 +0000</pubDate>
      <guid>https://mever.gr/post/mever-participation-in-deepfake-detection-challenge/</guid>
      <description>&lt;p&gt;It has been almost 2 months since the final deadline for the &lt;a href=&#34;https://www.kaggle.com/c/deepfake-detection-challenge&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;challenge&lt;/a&gt; on the Kaggle platform. Competition organizers have just finalized the standings (13th of June 2020) in the &lt;a href=&#34;https://www.kaggle.com/c/deepfake-detection-challenge/leaderboard&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;private leaderboard&lt;/a&gt;. A Kaggle staff member mentioned in a &lt;a href=&#34;https://www.kaggle.com/c/deepfake-detection-challenge/discussion/152987&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;discussion&lt;/a&gt; that competition organizers took their time to validate winning submissions and ensure that they comply with the &lt;a href=&#34;https://www.kaggle.com/c/deepfake-detection-challenge/rules&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;competition rules&lt;/a&gt;. This process resulted in the disqualification of the top-performing team due to the usage of external data without proper license. This caused a lot of disturbance among the Kaggle community mainly because the competition rules were &lt;a href=&#34;https://www.kaggle.com/c/deepfake-detection-challenge/discussion/157983&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;vague&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Performance evaluation on the challenge was measured with two leaderboards. The public leaderboard evaluation was performed using videos that share common characteristics with the &lt;a href=&#34;https://www.kaggle.com/c/deepfake-detection-challenge/data&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;training data&lt;/a&gt; that were provided by the competition and was available to participants throughout the challenge. On the other hand, the private leaderboard evaluation was performed on organic videos and various Deepfake manipulations that do not necessarily exist on training data and was revealed after the competition deadline&lt;/p&gt;
&lt;p&gt;After the final rerankings, our team (which &lt;a href=&#34;https://mever.gr/2020/03/10/mever-meets-deeptrace/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;joined forces&lt;/a&gt; with &lt;a href=&#34;https://deeptracelabs.com/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;DeepTrace&lt;/a&gt;) stands at the 115-th position on the private leaderboard, among 2,265 teams. The achieved Log Loss evaluation score is 0.51485. Note that our team stood at the 50-th position in the public leaderboard with a Log Loss score of 0.29502. It is apparent that our approach performs well on videos that originate from the same distribution as the training data and has relatively limited generalization ability. This observation applies to the large majority of submissions. For example, the winning team scored 0.20336 in terms of public validation score and only 0.42798 in the private leaderboard. Additionally, there were cases of severe overfitting on the training distributions. For example, the best submission in the public leaderboard scored 0.19207 in terms of Log Loss error, but in the private evaluation, the error was 0.57468 leading to the 904-th position in the private leaderboard.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/detection_approach.png&#34; alt=&#34;detection_approach&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Figure 1: Proposed preprocessing step. For more details check our &lt;a href=&#34;https://arxiv.org/pdf/2006.07084.pdf&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;arxiv paper&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;To deal with the DeepFake detection challenge, we focused on preprocessing the training data and used a deep learning approach that combined multiple architectures. The preprocessing step mainly involved dataset cleaning and generally increased the performance of our initial baseline models. More specifically, we used the &lt;a href=&#34;https://github.com/timesler/facenet-pytorch&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;MTCNN&lt;/a&gt; face detector to extract the bounding box information of all faces in video frames. Using the facial information we devised a similarity-based approach, which is shown in Figure 1,  to cluster similar faces and remove false detections. Our preprocessing methodology is described in more detail &lt;a href=&#34;https://arxiv.org/pdf/2006.07084.pdf&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;here&lt;/a&gt;. Following this methodology, we managed to remove noisy detections and generate distinct face clusters from all frames of a video. Furthermore, we removed DeepFake videos that were very similar to original ones by calculating the face similarity among them and further balanced the dataset by sampling more frames from original videos. In practice, our final dataset consisted of real and fake face images. Real face images were sampled from approximately 20,000 real videos by uniformly extracting 16 frames per video. Fake face images were sampled from 80,000 fake videos by uniformly extracting 4 frames per video. Then, we resized the images to 300×300 and applied various augmentations such as horizontal and vertical flipping, random cropping, rotation, image compression, Gaussian and motion blurring, and brightness, saturation and contrast transformations.&lt;/p&gt;
&lt;p&gt;We initially experimented with various deep learning architectures such as XceptionNet, MesoNet, InceptionResnet, etc. However, we noticed that EfficientNet outperformed all other architectures by a large margin. We decided to train EfficientNet-B3 and EfficientNet-B4 and although the latter scored better in the public leaderboard, a simple average of both architectures provided even better evaluation scores. For the inference pipeline, we uniformly extracted 40 frames per video and applied the preprocessing methodology we devised for the training data. Each detected face in a video could contain up to 40 face frames for inference. To get the aggregated prediction for each face, we averaged the individual predictions of all frames and models. Finally, we output the maximum prediction among the detected faces in a video. To exploit the temporal nature of videos, we also experimented with the I3D model. We used the I3D model just like the EfficientNet models, meaning that we made predictions at frame level with the difference that the I3D model used as input 10 consecutive frames instead of 1. By adding I3D to our ensemble and averaging individual frame and model predictions we managed to further increase the performance in the public leaderboard. This was our final model that finished in the 115-th position on the private leaderboard.&lt;/p&gt;
&lt;p&gt;With the help of DeepTrace, we also experimented with an audio architecture but this did not appear to improve the performance of the model. Additionally, we tried various other techniques from the related bibliography (residual filters, optical flow, etc.) but none of them seemed to lead to further improvements for this task.&lt;/p&gt;
&lt;p&gt;Although not every top-performing approach has been released so far, there are a lot of approaches among the top 50 that have been discussed online. Some common practices included the following:&lt;/p&gt;
&lt;p&gt;The EfficientNet architecture and its variants were admittedly the best for this task. Most approaches combined EfficientNet architectures (B3-B7) and some of them were trained with different seeds. ResNeXT was another architecture used by a top-performing solution combined with 3D architectures such as I3D, 3D ResNet34, MC3 &amp;amp; R2+1D. Input image resolution varied from 256 to 380 among these approaches. Also, several approaches increased the margin of the detected facial bounding box. In our methodology, we used an additional margin of 20% but other works proposed a higher proportion. To improve generalization apart from strong augmentations like the ones we employed,  some approaches proposed domain-specific augmentations like half face removal horizontally or vertically or landmark  (eyes, nose, or mouth) removal. Additionally, the solution that was ranked third on the private leaderboard, reported that &lt;a href=&#34;https://arxiv.org/abs/1710.09412&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;mixup&lt;/a&gt; augmentation further increased the generalization of the detection model. Finally, almost all approaches calculated frame-level predictions and aggregated them for final prediction. Finally, label smoothing seemed to improve the performance of some architectures.&lt;/p&gt;
&lt;p&gt;Overall the competition was a valuable experience and we learned a lot along the process. We would also like to thank the DeepTrace team for the cooperation and exchange of knowledge. From our side, the main observation from the outcomes of the competition is that the Deepfake detection problem is far from solved and the generalization capabilities of modern deep learning approaches are still limited. We have already deployed our approach as a DeepFake detection API within the &lt;a href=&#34;https://weverify.eu/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;WeVerify&lt;/a&gt; platform to test the authenticity of online media and we will focus on further improving it in the future.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>How similar are two videos?</title>
      <link>https://mever.gr/post/how-similar-are-two-videos/</link>
      <pubDate>Thu, 22 Aug 2019 00:00:00 +0000</pubDate>
      <guid>https://mever.gr/post/how-similar-are-two-videos/</guid>
      <description>&lt;p&gt;In this post we explain the basics behind our paper with title “ViSiL: Fine-grained Spatio-Temporal Video Similarity Learning” [1] which was accepted for an oral presentation at this year’s &lt;a href=&#34;https://iccv2019.thecvf.com/&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;International Conference on Computer Vision&lt;/a&gt; (ICCV 2019).&lt;/p&gt;
&lt;p&gt;Let’s consider two arbitrary videos, such as the ones displayed below. Also, we consider that the right video serves as query and the left as a candidate match. For the better understanding of the method, we select two near-duplicate videos.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/frames-1.gif&#34; alt=&#34;near_duplicate_videos&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Two arbitrary near-duplicate videos. Figure by Giorgos Kordopatis-Zilos&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;To calculate the similarity between the two videos, we can extract global feature descriptors for the frames of the video [2, 3], and then calculate the frame-to-frame similarity matrix from the dot products between each frame pair, as displayed below.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/sim_maps-2.gif&#34; alt=&#34;sim_maps&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Frame-to-frame similarity matrix between two videos. Figure by Giorgos Kordopatis-Zilos&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;The video-to-video similarity score can be measured using Chamfer Similarity (CS), which we empirically found to capture well the similarity between videos. This is calculated as the average similarity of the most similar frames in the candidate video for each frame of the query. The mathematical formula is:&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/svg.svg&#34; alt=&#34;chamfer_similarity&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;
&lt;/p&gt;
&lt;p&gt;where,  is the frame-to-frame similarity matrix, and  and  are the number of frames of each video. CS is robust against the different lengths of the candidate videos since it is calculated according to the query. For example, consider two candidates that contain the query in their entirety, and the second one is much longer video than the first one. Due to CS, they will have approximately the same video similarity to the query, which is handy for ranking videos for content-based video retrieval.&lt;/p&gt;
&lt;p&gt;However, this scheme has several issues. First, the feature extraction process aggregates all frame information in a single vector representation resulting in a frame-to-frame similarity matrix with a lot of noise. Second, the CS function completely disregards the temporal structure of the frame-to-frame similarity matrix. Hence, we have to design a similarity calculation process that is able to introduce the least possible noise, and at the same time capture and model the temporal structure for the calculation of the video similarity.&lt;/p&gt;
&lt;p&gt;To cope with these limitations, we introduce ViSiL, a Video Similarity Learning architecture that considers fine-grained spatio-temporal relations to calculate  similarity between pairs of videos. ViSiL is trained to calculate video-to-video similarity from refined frame-to-frame similarity matrices, so as to consider both intra- and inter-frame relations.&lt;/p&gt;
&lt;h2 id=&#34;feature-extraction&#34;&gt;Feature extraction&lt;/h2&gt;
&lt;p&gt;Let’s begin with feature extraction. Instead of aggregating the entire frame in a vector representation, we feed frames to a CNN and extract frame feature maps , based on Regional Maximum Activation of Convolution (R-MAC) [2] from the intermediate convolutional layers [3]. Also, we can consider that the extracted feature maps consist of 















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/svg%20N2.svg&#34; alt=&#34;N2&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;
 region vectors 















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/svg%20reRc.svg&#34; alt=&#34;reRc&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;
.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/feature_extraction-1024x224.png&#34; alt=&#34;feature_extraction&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Illustration of feature extraction scheme. Figure by Giorgos Kordopatis-Zilos&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;To improve the discriminative capabilities of the frame descriptors, we apply PCA whitening on the region vectors in order to decorrelate them. The following figure illustrates the 2D projection of a sample of region vectors before and after whitening. We can observe that after whitening, the vector distribution has a spherical form which facilitates the similarity calculation step.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/pca-720x333.png&#34; alt=&#34;pca&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;2D projection of sampled region vectors before (left) and after (right) PCA whitening. Figure by Giorgos Kordopatis-Zilos&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Additionally, for more accurate similarity calculation, we want to focus on the information-rich part of the frames, and ignore the regions that contain no meaningful object (e.g. solid dark regions). To do so, we weigh frame regions based on their saliency via a visual attention mechanism over regions. This relies on an internal context vector  that is applied on each region vector independently to generate the attention weights. The attention results on an arbitrary video are displayed below. In this example, the network apply high weights on the frame regions of the concert stage, and low ones on the completely dark regions.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/attention.gif&#34; alt=&#34;attention&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Frame weights of an arbitrary video calculated based on the attention mechanism. Figure by Giorgos Kordopatis-Zilos&lt;/em&gt;&lt;/p&gt;
&lt;h2 id=&#34;frame-to-frame-similarity&#34;&gt;Frame-to-frame similarity&lt;/h2&gt;
&lt;p&gt;Having extracted the region vectors of the frames of two videos, we can now calculate the frame-to-frame similarity between all the frame pairs of the two videos. To do so, for two video frames, we calculate the dot product between every pair of region vectors which creates a region-to-region similarity matrix. Then, we apply CS on the output similarity matrix to compute the frame-to-frame similarity for the given frame pair. The following figure depicts the process for the frame-to-frame similarity calculation.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/frame_similarity.png&#34; alt=&#34;frame_similarity&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Frame-to-frame similarity calculation between two video frames. Figure by Giorgos Kordopatis-Zilos&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;This process leverages the geometric information captured by region vectors and provides some degree of spatial invariance. More specifically, the CNN extracts features that correspond to mid-level visual structures, such as object parts, and combined with CS, that by design disregards the global structure of the region-to-region matrix, constitutes a robust similarity calculation process against spatial transformations, e.g. spatial shift. This can be perceived as a trade-off between the preservation of the frame structure and invariance to spatial transformations.&lt;/p&gt;
&lt;p&gt;The following figure illustrates the frame-to-frame similarity matrix from the initial calculation scheme and the refined one based on our method. It is evident that a lot of noise has been suppressed and the diagonal parts in the refined matrix are more clear than in the initial one. This leads to more accurate video-to-video similarity calculation.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/f2f_matrix-720x328.png&#34; alt=&#34;f2f_matrix&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Comparison between the initial (left) and refined (right) frame-to-frame similarity matrix. Figure by Giorgos Kordopatis-Zilos&lt;/em&gt;&lt;/p&gt;
&lt;h2 id=&#34;video-to-video-similarity&#34;&gt;Video-to-video similarity&lt;/h2&gt;
&lt;p&gt;Finally, to calculate the similarity between two videos, we provide the refined frame-to-frame similarity matrix to a CNN network in order to capture and highlight temporal similarity structures. The network is capable of learning robust patterns of within-video similarities at segment level, and therefore further refine the similarity matrix. Below are displayed the input frame-to-frame similarity matrix and the ViSiL output.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/visil_output.png&#34; alt=&#34;video_similarity&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;Illustration of input frame-to-frame similarity matrix and the ViSiL output. Figure by Giorgos Kordopatis-Zilos&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;To calculate the final video-to-video similarity, we apply the hard tanh activation function on the values of the network output, which clips values within range [-1, 1], and then we apply CS to derive a single value which is considered the similarity between the two videos.&lt;/p&gt;
&lt;p&gt;Similar to the frame-to-frame similarity calculation, this process is a trade-off between  respecting video-level structure and being invariant to some temporal differences. As a result, different temporal similarity structures in the frame-to-frame similarity matrix can be captured, e.g. strong diagonals or diagonal parts (i.e. contained sequences).&lt;/p&gt;
&lt;h2 id=&#34;experimental-results&#34;&gt;Experimental results&lt;/h2&gt;
&lt;p&gt;In the following table, we can find the results of the initial baseline scheme (iMAC) and four different versions of ViSiL in terms of mean Average Precision (mAP) on FIVR-5K (a subset of the recently published FIVR-200K video dataset [4]). ViSiLf, which is the version with only the presented frame-to-frame similarity and CS for video-to-video similarity, significantly outperforms the baseline. Also, the utilization of whitening and attention mechanism further improves performance. ViSiLv, which is the version with all the presented refinements, achieves the best performance with large margin.&lt;/p&gt;
&lt;p&gt;















&lt;figure  &gt;
  &lt;div class=&#34;d-flex justify-content-center&#34;&gt;
    &lt;div class=&#34;w-100&#34; &gt;&lt;img src=&#34;https://mever.gr/media/pasted%20image%200.png&#34; alt=&#34;mAP&#34; loading=&#34;lazy&#34; data-zoomable /&gt;&lt;/div&gt;
  &lt;/div&gt;&lt;/figure&gt;

&lt;em&gt;mAP comparison of initial baseline scheme (iMAC) and four version of the ViSiL approach on FIVR-5K. W and A stand for whitening and attention mechanism respectively.
Table by Giorgos Kordopatis-Zilos&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Additionally, ViSiL achieved state-of-the-art performance on a number of retrieval problems, i.e. Near-Duplicate Video Retrieval (NDVR), Fine-grained Incident Video Retrieval (FIVR), Event Video Retrieval (EVR), and Action Video Retrieval (AVR). For more details regarding the architecture and training of the model, and for comprehensive experimental results on the four retrieval problems, feel free to have a look at the ViSiL paper [1]. Also, the implementation of ViSiL is &lt;a href=&#34;https://github.com/MKLab-ITI/visil&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;publicly available&lt;/a&gt;.&lt;/p&gt;
&lt;h2 id=&#34;references&#34;&gt;References&lt;/h2&gt;
&lt;p&gt;[1] Kordopatis-Zilos G., Papadopoulos S., Patras I., and Kompatsiaris, Y. (2019). ViSiL: Fine-grained Spatio-Temporal Video Similarity Learning. arXiv preprint &lt;a href=&#34;https://arxiv.org/abs/1908.07410&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;arXiv:1908.07410&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;[2] Tolias G., Sicre R., and Jégou H. (2015). Particular object retrieval with integral max-pooling of CNN activations. arXiv preprint &lt;a href=&#34;https://arxiv.org/abs/1511.05879&#34; target=&#34;_blank&#34; rel=&#34;noopener&#34;&gt;arXiv:1511.05879&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;[3] Kordopatis-Zilos G., Papadopoulos S., Patras I., and Kompatsiaris, Y. (2017). Near-Duplicate Video Retrieval by Aggregating Intermediate CNN Layers. In International conference on Multimedia Modeling.&lt;/p&gt;
&lt;p&gt;[4] Kordopatis-Zilos G., Papadopoulos S., Patras I., and Kompatsiaris, Y. (2019). FIVR: Fine-grained Incident Video Retrieval. IEEE Transactions on Multimedia.&lt;/p&gt;
</description>
    </item>
    
  </channel>
</rss>
