Key takeaways
- Deepfakes are made with deep learning models that learn faces, voices and scenes from large amounts of example data.
- Classic face swaps use an autoencoder: one shared encoder and one decoder per person, so one person's expressions can be rendered with another person's face.
- GANs produced the first photorealistic fake faces. Diffusion models now power most AI image generators and AI video tools.
- Face reenactment and lip-sync change what a real person appears to do or say. Voice cloning supplies the audio.
- Every technique leaves statistical traces in the pixels, and some AI tools also add invisible watermarks and content credentials. Detection reads both.
Contents
The building blocks
All deepfake techniques use deep learning: neural networks that learn patterns from examples rather than following hand-written rules. The examples are the raw material. A face-swap model learns from photos and video frames of faces; a voice model learns from recordings; a general image or video generator learns from very large collections of captioned images and clips.
Three families of models do most of the work:
| Model family | What it does | Typical use |
|---|---|---|
| Autoencoder | Compresses a face into a compact description and rebuilds it | Classic face swaps |
| Generative adversarial network (GAN) | Two networks compete until generated images pass as real | Faces of people who do not exist, reenactment, early lip-sync |
| Diffusion model | Turns random noise into an image or video, step by step, guided by a prompt | Today's image generators, AI edits and text-to-video |
Face swaps
A face swap replaces one person's face with another's in a photo or video while keeping the rest of the scene.
The autoencoder approach
The approach behind the original 2017 deepfakes, and the open-source tools that followed, is an autoencoder with a twist:
- An encoder compresses a face image into a compact description: the pose, the expression and the lighting.
- A decoder rebuilds a face from that description.
- The model is trained with one shared encoder and two decoders, one for person A and one for person B. Because the encoder is shared, it learns a description that works for both faces.
To swap, frames of person A go through the shared encoder and come out through person B's decoder. The result is B's face making A's expressions, which is then blended back into the original frame. The FaceForensics++ benchmark describes this method and includes videos made with it, alongside other manipulation methods, to train and test detectors.
Swaps with image models
Today's general-purpose image models can also produce a face swap from a written instruction, with no training on the person at all. The example below was made by Sightova with OpenAI's GPT-Image 2.5, using two AI-generated men so that no real person was involved.
What face swaps leave behind: blending seams around the face, skin tone and lighting that do not quite match the scene, a face that is sharper or softer than its surroundings, and in video, flicker from frame to frame.
GANs: faces of people who do not exist
A generative adversarial network trains two networks against each other. The generator makes images from random noise. The discriminator sees both generated images and real photos and tries to tell them apart. Each round, the generator learns from what gave its fakes away, until the discriminator can no longer tell the difference.
NVIDIA's StyleGAN, published in late 2018, generated portraits of people who do not exist at a quality that became the standard source of fake profile photos. In a 2022 study, people identified faces made by its successor, StyleGAN2, correctly 48.2% of the time, close to the 50% of a coin toss.
What GANs leave behind: researchers showed that GAN images carry regular patterns from the way the networks enlarge images, visible when the image is analysed by frequency rather than by eye. A 2020 study found that a detector trained on one GAN could recognise images from many others because of these shared traces.
Face reenactment
Face reenactment keeps a person's identity and changes what they do. The expressions and head movements of a "driver" are transferred onto the target.
- Face2Face (2016) transferred a source actor's facial expressions onto a target video in real time.
- The First Order Motion Model (2019) animated a single still photo from a driving video, using keypoints that track how the face moves.
Reenactment is how a single profile photo can be turned into a talking head, or how a public figure can appear to react to something they never saw.
What reenactment leaves behind: warping around the edges of the face and the hairline, an inside of the mouth that lacks detail, and head movements that do not match the body or background.
Lip-sync
Lip-sync models redraw only the mouth region of a genuine video so that it matches new audio. Wav2Lip (2020) showed that this could work on almost any video of a speaking face. Because the rest of the footage is real, a lip-synced clip of a press conference or an earnings call keeps all the context that makes it believable.
What lip-sync leaves behind: a mouth area that is softer than the rest of the face, teeth that blur or change shape, and mouth shapes that do not quite match the sounds.
Voice cloning
Voice cloning models learn a person's voice from recordings and then speak any text in it. Microsoft's VALL-E research model (2023) could imitate a voice from a three-second sample. Cloned voices were used in the AI-generated robocalls imitating President Biden in January 2024, and they are combined with lip-sync and face swaps in video-call fraud.
Diffusion models: today's image generators
A diffusion model is trained by adding noise to real images and learning to remove it. To generate, it starts from pure noise and removes a little at each step, steered by a text prompt, until an image appears. Latent diffusion (2022) runs this process on a compressed version of the image, which made high-resolution generation fast enough for consumer use.
Diffusion models and closely related techniques power most AI image generators in use today. The same models edit real photos: a region is masked and regenerated to match a prompt. This is how a real photo gets a new face, a removed object or added damage, and why AI edits are now as important to detect as fully generated images.
What diffusion images leave behind: statistical traces in the pixels that detection models learn to recognise even when the image looks flawless. Google and OpenAI also add invisible watermarks to images from their tools (SynthID and OpenAI's watermark) and attach signed C2PA content credentials, as our test of their newest models shows.
AI video
Text-to-video models extend diffusion and related techniques to moving images. Google's Veo 3 (May 2025) and OpenAI's Sora 2 (September 2025) generate realistic scenes with synchronised speech, sound effects and ambient noise. Sora 2 also launched with an app that lets users place their own likeness into generated scenes.
The misuse risk comes from realistic footage of events that never happened, and from clips that are re-uploaded without their labels.
What AI video leaves behind: objects and text that change from frame to frame, physics that does not hold, and the same per-frame pixel traces as AI images. Watermarks and content credentials are lost when a clip is re-encoded or screen-recorded.
What each technique changes and leaves behind
| Technique | What it changes | Typical traces |
|---|---|---|
| Face swap (autoencoder) | Who is shown | Blending seams, lighting and skin-tone mismatch, frame-to-frame flicker |
| GAN faces | The whole face or image | Regular patterns from upsampling, odd backgrounds and accessories |
| Face reenactment | Expressions and head movement | Warping at face edges and hairline, missing mouth detail |
| Lip-sync | The mouth and the words | Soft mouth region, unstable teeth, mismatched mouth shapes |
| Voice cloning | The voice | Unnatural pacing and breathing, artefacts between words |
| Diffusion images and AI edits | All or part of an image | Statistical pixel traces; watermarks and C2PA credentials from some tools |
| AI video | The whole clip | Inconsistent objects and physics; per-frame pixel traces |
How Sightova detects AI-generated images
Sightova reads the traces that image generation and AI editing leave, and the marks some tools add, in three independent checks:
- Pixel-level detection. A detection model trained on images from current generators and editing tools returns an AI probability and a verdict. It reads the pixels, so it keeps working when metadata has been stripped.
- Invisible watermark detection. Finds the watermarks Google (SynthID) and OpenAI hide in their AI images. A watermark that is found names the source.
- Content credentials and metadata. Reads C2PA manifests and EXIF data, which can state the tool that made the image.
On Starter plans and above, a heatmap shows where in the image the model sees generation or editing. On the face swap shown above, the checks returned an AI probability of 94.93%, the OpenAI watermark at 100% and valid C2PA credentials signed by OpenAI. The full test is in face swaps without a real victim.
Images can be checked in the dashboard or through the REST API. Our testing methodology reports how the detection model performs on public benchmarks. AI-generated video is checked frame by frame with AI Video Detection, on the Starter plan and above.
Detect AI-generated images with Sightova
Scan in the dashboard or build it into your upload flow with the API.
Frequently asked questions
How are deepfakes made?
Deepfakes are made with deep learning models trained on images, video or audio. Classic face swaps use autoencoders, the first realistic fake faces came from GANs, and most current AI images and videos come from diffusion models. Face reenactment, lip-sync and voice cloning change what a real person appears to do or say.
What is the difference between a GAN and a diffusion model?
A GAN trains a generator against a discriminator until its images pass as real, and produces an image in a single pass. A diffusion model learns to remove noise and generates an image by starting from noise and refining it over many steps, guided by a prompt.
What is face reenactment?
Face reenactment transfers one person's expressions and head movements onto another person's face. The identity stays the same, but the target appears to do things they never did.
Is it legal to make a deepfake?
It depends on the content and the consent of the person shown. Sexual deepfakes of real people without consent are a crime in the US, the UK and South Korea, and EU member states must criminalise them by June 2027. Deepfakes used for fraud fall under fraud law. From 2 August 2026, the EU AI Act requires deepfakes to be disclosed.
Can deepfakes be detected?
Yes. Each technique leaves statistical traces in the pixels that detection models can learn to recognise, and some AI tools add invisible watermarks and content credentials. Sightova checks images for all three.
