In general, there are multiple tasks in media forensic applications. For example, manipulation detection and localization, Generative Adversarial Network (GAN) detection, image splice detection and localization, event verification, camera verification, and provenance history analysis etc.

The OpenMFC initially focuses on manipulation detection and deepfake tasks. In future, challenges may be expanded with community interest. The OpenMFC 2022 has following three task categories: Manipulation Detection (MD), Deepfakes Detection (DD), and Steganography Detection (StegD).

• MD: Manipulation Detection
• DD: Deepfakes Detection
• StegD: Steganography Detection

A brief summary of each category and their tasks are described below. In the summary, the evaluation media is described in the following way: A ‘base’ indicates original media with high provenance, while a ‘probe’ indicates a test media. A ‘donor’ indicates another media whose content was donated into the base media and generated the probe media. For a full description of the evaluation tasks, please refer to the OpenMFC 2022 Evaluation Plan [Download Link].

• Manipulation Detection (MD)

  The objective for Manipulation Detection (MD) is to detect if a probe has been manipulated, and if so, to spatially localize the edits. Manipulation in this context is defined as deliberate modifications of media (e.g., splicing and cloning etc.) and localization is encouraged but not required for OpenMFC.

  The MD task includes three tasks, namely,

  • Image_MD (or IMD): Image Manipulation Detection

  The Image Manipulation Detection task is to detect if the image has been manipulated, and then to spatially localize the manipulated region. For detection, the IMD system provides a confidence score for all probe (i.e., a test image) with higher numbers indicating the image is more likely to have been manipulated. The target probes (i.e., probes that should be detected as manipulated) included potentially any image manipulations while the non-target probes (i.e., probes not containing image manipulations) include only high provenance images that are known to be original. Systems are required to process and report a confidence score for every probe.

  For the localization part of the task, the system provides an image bit-plane mask (either binary or greyscale) that indicates the manipulated pixels. Only local manipula­tions (e.g., clone) require a mask output while global manipulations (e.g., blur) affecting the entire image do not require a mask.

  • ImageSplice_MD (or ISMD): Image Splice Manipulation Detection

  The new task, Image Splice Manipulation Detection, is added in the OpenMFC 2022 to support entry-level public participants. The ISMD is designed for 'splice' manipulation operation only. The testing dataset is a small-size dataset (2K images), which contains either original images without any manipulation, or spliced images. The ISMD task will detect if a probe image has been spliced.

  • Video_MD (or VMD): Video Manipulation Detection

  The Video Manipulation Detection (VMD) task is to detect if the video has been manipulated. In this task, the localization of spatial/temporal-spatial manipulated regions is not addressed. For detection, the VMD system provides a confidence score for all probes (i.e, a test video) with higher numbers indicating the video is more likely to have been manipulated. For VMD, target probes (i.e., probes that should be detected as manipulated) included potentially any video manipulations while the non-target probes (i.e., probes not containing video manipulations) include only high provenance videos that are known to be original. Systems are required to process and report a confidence score for every probe.

• Deepfakes Detection (DD)

  With recent advances in DeepFakes techniques and GAN (Generative Adversarial Network), imagery producers are able to generate realistic fake objects in media. The objective for Deepfakes Detection (DD) is to detect if a probe has been Deepfakes or GAN manipulated.

  The DD task includes two tasks based on testing media type, namely,

  • Image_DD (or IDD): Image Deepfakes Detection
  The Image Deepfakes Detection task evaluates if a system can detect Deepfaked images (e.g. a face video is manipulated by a Deepfakes tool, then deepfaked face image frames are extracted as image files etc.) or GAN-manipulated images (e.g. created by a GAN model, locally/globally modified by a GAN filter/operation, etc.) specifically while not detecting other forms of manipulations. In the testing datasets, the non-target probes are high provenance images (or cropped high provenance images) and the target images are the ones manipulated with Deepfakes or GAN techniques.
  • Video_DD (or VDD): Video Deepfakes Detection
  The Video Deepfakes Detection task evaluates if a system can detect Deepfaked videos. The target probes include Deepfaked videos while the non-target probes include high provenance original videos or video clips.
• Steganography Detection (StegD)
  • StegD: Steganography Detection
  The Steganography Detection task evaluates if a probe is a stego image, which contains the hidden message either in pixel values or in optimally selected coefficients. For each StegD trial, which consists of a single probe image, the StegD system must render a confidence score with higher numbers indicating the probe image is more likely to be a stego image.

For detection performance assessment, system performance is measured by Area Under Curve (AUC) which is the primary metric and the Correct Detection Rate at a False Alarm Rate of 5% (CDR@FAR = 0.05) from the Receiver Operating Characteristic (ROC) as shown Figure (a) below. This applies to both image and video tasks.

For the image localization performance assessment, the Optimum Matthews Correlation Coefficient (MCC) is the primary metric. The optimum MCC is calculated using an ideal mask-specific threshold found by computing metric scores over all pixel thresholds. Figure (b) below shows a visualization of the different mask regions used for mask image evaluations.

• TP (True Positive) is an overlap area (green) of the reference mask and system output mask as manipulated at the pixel threshold.
• FN (False Negative) is the reference mask indicates as manipulated, but the system did not detect it as manipulated at the threshold (red).
• FP (False Positive) is the reference mask indicates not-manipulated, but the system detected it as manipulated at the threshold (orange).
• TN (True Negative) is the reference mask indicates not-manipulated, and the system also detects it as not-manipulated at the threshold (white).
• NS (No-Score) is the region of the reference mask not scored, the result of the dilation and erosion operations (purple).

• If the denominator is zero, then we set MCC_o = 0.
• If MCC_o = 1, there is a perfect correlation between the reference and system output masks.
• If MCC_o = 0, there is no correlation between the reference mask and the system output mask.
• If MCC_o = -1, there is perfect anti-correlation.

Figure 1. Detection System Performance Metrics: Receiver Operating Characteristic (ROC) and Area Under the Curve (AUC)

Figure 2. Localization System Performance Metrics: Optimum Matthews Correlation Coefficient (MCC)

Figure 3. An Example of Localization System Evaluation Report