Deepfake Detection CNN

Overview

The proliferation of highly realistic deepfakes poses a significant challenge to digital media integrity. This project aimed to develop an effective deepfake detection model. Initially, the goal was to improve upon the ~70% accuracy of a CNN-based model by Karandikar et al. (2020), which utilized the VGG16 architecture.

The project involved working with datasets like FaceForensics++ (first-generation deepfakes) and Celeb-DF (v2) (more advanced deepfakes). While the VGG16-based approach showed promise (achieving 90% validation on FaceForensics++ alone and 75.69% on a combined dataset), it faced challenges with overfitting and complexity when applied to the harder, combined dataset.

This led to the development of a lighter, custom Convolutional Neural Network (CNN) from scratch. This custom CNN, with approximately 6.5 million parameters, ultimately achieved an 75% validation accuracy on the challenging combined dataset, outperforming the VGG16-based approach on the same data. The project began after an initial exploration of DefakeHop++, which was pivoted away from due to its limited use of deep learning techniques.

Technical Approach & Model Architectures

The project explored two main CNN-based approaches for deepfake detection after an initial pivot:

• VGG16-based Model with Custom Head:
  • Utilized transfer learning, leveraging the pretrained convolutional layers of VGG16.
  • A custom classification head was added, differing from the original Karandikar et al. paper by using a binary classification output with a sigmoid activation and a revised layer ordering to potentially reduce overfitting.
  • Data preprocessing involved face detection (libface detector), cropping with a 30% margin, and alignment (cv2). Frames were selected to ensure balanced classes.
  • This model had ~14.7M parameters from VGG16 and ~1.7M from the custom head.
• Custom CNN Model (from scratch):
  • Developed to address the complexity and overfitting issues encountered with the VGG16-based model.
  • Totaled approximately 6.5 million parameters.
  • Architecture included:
    • Conv2D (32 filters, 3x3) -> BatchNormalization -> MaxPool2D
    • Conv2D (64 filters, 3x3) -> BatchNormalization -> MaxPool2D
    • Conv2D (64 filters, 3x3) -> BatchNormalization -> MaxPool2D
    • Flatten
    • Dense (64, LeakyReLU, L2 regularizer) -> Dropout(0.5) -> BatchNormalization
    • Dense (64, LeakyReLU, L2 regularizer) -> Dropout(0.5) -> BatchNormalization
    • Dense (1, sigmoid, L2 regularizer)
  • This model achieved the highest validation accuracy (75%) on the combined dataset.

Key Results

The original Karandikar et al. (2020) paper reported ~70% accuracy with their VGG16-based model. Both models developed in this project surpassed that on comparable tasks.

Challenges & Learnings

Several challenges were encountered and overcome during development:

• Overfitting: This was the primary challenge, especially with the complex VGG16 model on the combined dataset, leading to a persistent gap between training and validation accuracies. Various techniques like regularization, dropout, and hyperparameter tuning were explored.
• Data Augmentation: Attempts to use data augmentation (e.g., noise, brightness adjustment) yielded mixed results, sometimes worsening overfitting. Transformative augmentations (rotation, zoom) interfered with the facial alignment preprocessing.
• Preprocessing Imperfections: The face detection algorithm occasionally misidentified non-face objects (e.g., buttons, posters) as faces, potentially impacting model training. Manually checking nearly 100,000 frames was infeasible.
• Hardware Constraints: Training was performed on an M2 MacBook Pro. Limited computational resources and disk space made it challenging to effectively parallelize preprocessing/training and handle the large datasets required for optimal fine-tuning of VGG16.
• Initial Project Pivot: The project initially aimed to implement DefakeHop++, but pivoted to a CNN-based approach after discovering DefakeHop++ lacked substantial deep learning components.

Future Enhancements

Potential improvements and future directions for the model include:

• Implementing a more fine-tuned transfer learning approach for VGG16, involving unfreezing more layers and training with a very low learning rate, potentially with a larger dataset if hardware permits.
• Further developing and refining the lightweight custom CNN architecture, which showed promising results in balancing performance and complexity.