Testing Methodology
A detailed look at how we evaluate, benchmark, and continuously validate every model in the Sightova detection pipeline.
Our Commitment to Transparency
As generative AI continues to evolve at an unprecedented pace, the tools designed to detect it must be held to an equally high standard. At Sightova, we believe that publishing our testing methodology is not optional—it is a responsibility. Users, enterprises, and researchers who rely on our platform deserve complete clarity on how we measure performance, where our models excel, and where known limitations exist.
This document covers the full Sightova detection pipeline: four independent models that run in parallel on every submitted image, each targeting a distinct dimension of authenticity analysis.
Model V1 — Real vs. AI Classification
The foundation of the Sightova pipeline is a Vision Transformer (ViT-Base) trained specifically for binary classification: determining whether an image is an authentic photograph or an AI-generated composition. The model processes every input as a 224×224 patch grid and learns subtle statistical patterns—compression artifacts, frequency-domain anomalies, and pixel-level noise distributions—that differ between camera-captured and synthetically rendered images.
Architecture
We apply an elevated confidence threshold of 85% before labelling an image as AI-generated. This deliberate decision prioritises minimising false positives—incorrectly accusing an authentic photograph—over aggressively flagging borderline cases. Images that fall between 50% and 85% AI probability are reported with their raw scores but not classified as AI-generated.
Confidence Bands
Model V2 — AI Generator Identification
When V1 confirms that an image is AI-generated, Model V2 activates as a follow-up classifier. It is a separate ViT-Base model fine-tuned on a large-scale proprietary dataset spanning 14 distinct generator families. The model outputs a probability distribution across all classes, enabling users to see not only the most likely source but also the relative confidence for each alternative.
V2 is intentionally decoupled from V1. It is never executed on images classified as authentic, because the generator classifier was trained exclusively on synthetic samples and would produce meaningless outputs on real photographs.
Supported Generator Classes
Training Summary
Model V3 — Pixel-Level Heatmap Segmentation
Beyond binary classification, Sightova provides pixel-level forensic analysis through a U-Net segmentation model with a ResNet-50 encoder backbone. This model processes the image at 256×256 resolution and generates a probability map where each pixel receives a score from 0.0 (unchanged / authentic) to 1.0 (AI-edited or synthetically generated).
The resulting heatmap is overlaid directly on the original image in our interface, rendering AI-manipulated regions in red and untouched areas in green. This enables users to pinpoint exactly which regions of a photograph have been altered—critical for forensic investigators, journalists verifying source material, and compliance teams reviewing submitted documentation.
Architecture
The heatmap also produces aggregate statistics: the percentage of total pixels classified as edited and the peak confidence value across the entire map. These metrics give a quantitative summary alongside the visual overlay.
NSFW Content Safety Classifier
Every image submitted to Sightova is simultaneously evaluated by a dedicated content safety model. This ViT-based binary classifier determines whether an image contains not-safe-for-work material, allowing platforms that integrate our API to enforce content policies automatically and flag inappropriate submissions before they reach human reviewers.
The NSFW model runs in parallel with V1, V2, and V3—it does not gate or depend on the other classifiers. It shares the same ViT-Base architecture and 224×224 input preprocessing as V1, ensuring consistent inference performance.
Evaluation Methodology
Data Separation
All evaluation datasets are strictly isolated from training data. No image used during model training appears in any benchmark or test set. We maintain a rigorous data lineage process to prevent leakage between training, validation, and evaluation splits. Evaluation images are collected from new sources and generated with the latest publicly available AI tools after model training has concluded.
Dataset Construction
Each evaluation dataset is constructed with two balanced categories:
- Authentic images — Camera-captured photographs sourced from verified public archives, stock libraries, and manually created test sets. These images have not been processed through any generative AI tool.
- AI-generated images — Synthetically produced images from the most current generator models including DALL·E 3, Midjourney v7, Stable Diffusion XL, Flux, GPT-Image, Google Imagen, and others. We continuously update these sets as new generators emerge.
Image Requirements
Evaluation is conducted exclusively on images meeting these specifications:
Evaluation Metrics
We evaluate each model using metrics appropriate to its task:
V1 — Binary Classification
- True Negative Rate (TNR) — Percentage of authentic images correctly classified as real. Our primary focus: minimising false positives.
- True Positive Rate (TPR) — Percentage of AI-generated images correctly flagged. Measures detection sensitivity.
- Overall Accuracy — Balanced measure across both classes.
V2 — Generator Identification
- F1 Macro — Harmonic mean of precision and recall averaged equally across all 14 generator classes, preventing dominant classes from skewing results.
- Top-1 Accuracy — Frequency with which the model's highest-confidence prediction matches the true generator.
V3 — Pixel Heatmap
- Pixel-level True Positive Rate — For AI-generated images, the proportion of pixels correctly identified as synthetic must exceed 90% for the image to count as a successful detection.
- Pixel-level False Positive Rate — For authentic images, fewer than 5% of pixels should be incorrectly flagged as AI-edited.
Continuous Dataset Refreshes
Generative AI models are updated on a near-weekly cadence. A benchmark created in January can be obsolete by March. To address this, we operate a rolling evaluation programme:
- New evaluation images are generated with the latest publicly released AI models within days of their launch.
- Authentic image sets are expanded with fresh camera captures across diverse devices, lighting conditions, and subject matter.
- Every model release triggers a full re-evaluation cycle against the updated dataset before deployment to production.
Error Analysis
Every misclassification identified during evaluation is logged, categorised, and reviewed through a structured root-cause analysis process. Errors are grouped by type—false positive, false negative, generator misattribution—and further segmented by image characteristics such as resolution, compression level, content type, and source generator.
This systematic approach allows our research team to identify recurring failure patterns and prioritise targeted improvements. For example, if a disproportionate number of false negatives originate from a single generator family, we can augment the training data for that class specifically rather than retraining the entire model.
Error analysis findings directly inform the next training cycle, creating a tight feedback loop between evaluation and improvement.
Known Limitations
No detection system achieves perfect accuracy in every scenario. We publish our known limitations to set honest expectations and guide responsible adoption of our technology.
Traditional Photo Edits
Our models are trained to detect generative AI manipulations. Manual edits made with traditional software—cropping, colour correction, brightness adjustments, manual compositing—are outside the scope of the current pipeline.
Extreme Compression and Resizing
Heavy JPEG compression or aggressive downscaling can destroy the subtle statistical fingerprints that our models rely on. Performance may degrade on highly compressed images below 512×512 pixels.
Emerging Generators
A new generator released after our most recent training cycle may temporarily evade detection until we incorporate samples from that model into the next update. Our rolling evaluation programme is designed to minimise this window.
Adversarial Attacks
Deliberately crafted adversarial perturbations—images specifically engineered to fool detection models—remain an active area of research across the industry. While our models demonstrate resilience against common attack vectors, determined adversaries with knowledge of the model architecture may achieve temporary evasion.
Generator Confidence Threshold
When V2 reports a top generator confidence below 40%, the identification should be considered uncertain. In these cases, we recommend treating the generator label as indicative rather than definitive.
Runtime Pipeline
When an image is submitted via the Sightova dashboard or API, the following steps execute in a single request:
Preprocessing
The image is validated (format, size, dimensions), saved to a temporary location, and prepared for inference.
V1 Classification
The ViT binary classifier produces a real/AI probability pair and a confidence band (high, medium, low).
V2 Generator ID (conditional)
If V1 determines the image is AI-generated (≥ 85% probability), the generator classifier runs and outputs scores for all 14 classes.
V3 Heatmap Segmentation
The U-Net model generates a pixel-level probability map, identifies edited regions, and computes aggregate statistics (edited %, peak confidence).
NSFW Classification
The safety classifier runs in parallel and returns a normal/nsfw probability pair.
Metadata Extraction
EXIF, C2PA provenance, geolocation, and file metadata are extracted concurrently with model inference.
Response Assembly
All results are merged into a single JSON response containing detection verdict, generator scores, heatmap data, NSFW status, and full metadata.
Try it yourself
Upload any image to the Sightova dashboard and see every model in action—real-time classification, generator identification, pixel heatmap, and full metadata extraction.