Testing AI image watermarks: C2PA and SynthId

Testing AI image watermarks: C2PA and SynthID

C2PA and SynthID are two technologies used to identify AI-generated images. C2PA (Content Credentials) is pretty simple, as it just adds signed metadata to a file. SynthID, on the other hand, is a different beast. It adds a watermark directly into the pixels of images.

I decided to experiment with these to understand more.

Test 1: Plain rectangle

Prompt:

Let's inspect the image:

c2patool ai-generated-images/pure-white-blank-canvas1.png --info

The important part here is "One manifest". Let's extract the manifest and search for interesting fields.

It is clear that this image has C2PA metadata.

Stripping is pretty easy with magick. Let's inspect again, after stripping.

At this point, the metadata is gone.

{
  "type": "c2pa",
  "outcome": "not_detected"
}

Did we change the image, though? Let's decode these files into raw RGBA pixels and check the hash.

magick rectangle-original.png rgba:- | shasum -a 256
magick rectangle-stripped.png rgba:- | shasum -a 256

// Both output ef994f090d047a181138e41a0e44bdec7cc22e4982bda529eecc2240c21c3cb2

They both output the same hash, so the images are identical.

Interestingly enough, OpenAI does not seem to add SynthID in simple shapes like these. Let's generate something better.

Test 2: Apple

I used the prompt "Draw a red apple, realistic studio photo, on a plain white background":

Let's check for provenance signals.

We could either use the web's version or just call the API.

I stripped the metadata and ran the check again.

{
  "object": "content_provenance_check",
  "results": [
    {
      "type": "c2pa",
      "generated_at": null,
      "issuer": null,
      "model": null,
      "outcome": "not_detected",
      "validation_state": "not_present"
    },
    {
      "type": "SynthID",
      "generated_at": null,
      "model": null,
      "outcome": "detected"
    }
  ]
}

This is expected, as removing metadata does not remove SynthID.

What happens with the crop?

magick Pure-White-Canvas.png -gravity center -crop 512x512+0+0 +repage ai_crop_512.png

SynthID survives.

JPEG conversion?

magick apple-stripped.png -quality 75 apple-q75.jpg

SynthID survives.

Maybe a diffusion attack, then?

...
pipe = StableDiffusionImg2ImgPipeline.from_pretrained(
        model_id, 
        torch_dtype=torch.float16 if device == "mps" else torch.float32
    )
prompt = "A high-quality, detailed, photorealistic macro photograph of a single glossy red apple with a brown stem, natural yellow streaks, isolated on a plain white background with a soft shadow"
    image = pipe(
        prompt=prompt, 
        image=init_image, 
        strength=0.25, 
        guidance_scale=7.5
    ).images[0]
  ...

That is, convert the original image to numbers (latents), add some noise, then denoise based on a prompt.

But this is cheating. I am essentially regenerating the image (from the original), not removing SynthID. This type of diffusion reconstruction will not exactly preserve the original image (texture, color, edges, lighting, etc.).

Guess I am back to analyzing pixels, and there is no better candidate for this than a nearly white image.

Test 3: Plain white background

Prompt:

How about we compare this plain white image with a plain white image (that I can create locally)?

Let's analyze the AI image for spatial patterns or uniformities.

import numpy as np
import matplotlib.pyplot as plt
from PIL import Image

# Compute a windowed, 0 mean 2D FFT spectrum
def fft_spectrum(channel):
    centered = channel - channel.mean(dtype=np.float64) # zero mean
    h, w = centered.shape
    window = np.outer(
        np.hanning(h),
        np.hanning(w),
    )
    windowed = centered * window
    fft_raw = np.fft.fft2(windowed)
    fft_shifted = np.fft.fftshift(fft_raw)
    magnitude = np.log1p(np.abs(fft_shifted))
    return magnitude

# analyze spatial variation
# load image, calculate mean RGB independently, subtract mean color from every pixel and compute FFT spectra
def analyze_zero_mean_residual(
    image_path,
    output_path="zero_mean_analysis.png",
    amp_factor=50,
):
    img = Image.open(image_path).convert("RGB")
    ai = np.array(img, dtype=np.float64)
    height, width, channels = ai.shape

    print("--- Image ---")
    print(f"Path: {image_path}")
    print(f"Dimensions: {width} x {height}")
    print(f"Shape: {ai.shape}")
    print("--- Raw Pixel Statistics ---")
    print(f"Overall min:  {ai.min():.0f}")
    print(f"Overall max:  {ai.max():.0f}")
    print(f"Overall mean: {ai.mean(dtype=np.float64):.6f}")

    unique_values = np.unique(ai.astype(np.uint8))
    print(f"Unique channel values: {len(unique_values)}")

    if len(unique_values) <= 30:
        print(f"Values: {unique_values}")
    else:
        print(
            f"Value range: "
            f"{unique_values[0]} ... {unique_values[-1]}"
        )

    mean_rgb = ai.mean(
        axis=(0, 1),
        dtype=np.float64,
    )

    print("--- Mean RGB ---")
    print(f"R = {mean_rgb[0]:.9f}")
    print(f"G = {mean_rgb[1]:.9f}")
    print(f"B = {mean_rgb[2]:.9f}")

    channel_min = ai.min(axis=(0, 1))
    channel_max = ai.max(axis=(0, 1))

    print("--- Channel Ranges ---")
    print(
        f"R: {channel_min[0]:.0f} to {channel_max[0]:.0f}"
    )
    print(
        f"G: {channel_min[1]:.0f} to {channel_max[1]:.0f}"
    )
    print(
        f"B: {channel_min[2]:.0f} to {channel_max[2]:.0f}"
    )

    if np.any(mean_rgb < channel_min) or np.any(mean_rgb > channel_max):
        raise RuntimeError(
            "Calculated RGB mean is outside the observed pixel range."
            f"Mean: {mean_rgb}"
            f"Min:  {channel_min}"
            f"Max:  {channel_max}"
        )

    residual = ai - mean_rgb
    residual_channel_means = residual.mean(
        axis=(0, 1),
        dtype=np.float64,
    )

    print("--- Residual ---")
    print(f"Min:  {residual.min():.9f}")
    print(f"Max:  {residual.max():.9f}")
    print(
        f"Overall mean: "
        f"{residual.mean(dtype=np.float64):.12f}"
    )

    print("--- Residual Mean By Channel ---")
    print(f"R = {residual_channel_means[0]:.12f}")
    print(f"G = {residual_channel_means[1]:.12f}")
    print(f"B = {residual_channel_means[2]:.12f}")

    if not np.allclose(
        residual_channel_means,
        0.0,
        atol=1e-10,
    ):
        raise RuntimeError(
            "Residual was not centered correctly."
            f"Residual means: {residual_channel_means}"
        )
    channel_names = ["Red", "Green", "Blue"]

    print("--- Per-channel Residual Statistics ---")

    for index, name in enumerate(channel_names):
        channel = residual[:, :, index]

        print(
            f"{name:>5} | "
            f"mean={channel.mean(dtype=np.float64): .12f} "
            f"std={channel.std(dtype=np.float64): .9f} "
            f"min={channel.min(): .9f} "
            f"max={channel.max(): .9f}"
        )

    # Overall RMS residual
    rms = np.sqrt(
        np.mean(
            residual ** 2,
            dtype=np.float64,
        )
    )
    print(f"Overall residual RMS: {rms:.9f}")


    red_fft = fft_spectrum( residual[:, :, 0] )
    green_fft = fft_spectrum( residual[:, :, 1] )
    blue_fft = fft_spectrum( residual[:, :, 2] )
    all_fft_values = np.concatenate([ red_fft.ravel(), green_fft.ravel(), blue_fft.ravel(), ])
    fft_vmin = np.percentile( all_fft_values, 1.0, )
    fft_vmax = np.percentile( all_fft_values, 99.5, )

    print("--- Shared FFT Display Scale ---")
    print(f"vmin = {fft_vmin:.9f}")
    print(f"vmax = {fft_vmax:.9f}")


    # signed residual visualization
    residual_vis = np.clip( 128.0 + residual * amp_factor, 0, 255, ).astype(np.uint8)

    # flat mean-color reference for visualization
    flat_rgb_uint8 = np.clip( np.round(mean_rgb), 0, 255, ).astype(np.uint8)
    flat_vis = np.empty_like( ai, dtype=np.uint8, )
    flat_vis[:] = flat_rgb_uint8

    # plot
    fig, axes = plt.subplots( 2, 3, figsize=(18, 11), constrained_layout=True, )

    # Original image
    axes[0, 0].set_title( "Original AI image" )

    axes[0, 0].imshow(img)
    axes[0, 0].axis("off")

    # Mean-matched flat reference
    axes[0, 1].set_title( "Mean-matched flat reference" f"Display RGB {tuple(flat_rgb_uint8.tolist())}" )

    axes[0, 1].imshow(flat_vis)
    axes[0, 1].axis("off")


    axes[0, 2].set_title( f"Signed residual ×{amp_factor}" "128 = zero difference" )
    axes[0, 2].imshow( residual_vis )
    axes[0, 2].axis("off")

    fft_image = axes[1, 0].imshow( red_fft, cmap="magma", vmin=fft_vmin, vmax=fft_vmax, ) # red
    axes[1, 0].set_title( "FFT: red residual" )
    axes[1, 0].axis("off")
    axes[1, 1].imshow( green_fft, cmap="magma", vmin=fft_vmin, vmax=fft_vmax, ) # green
    axes[1, 1].set_title( "FFT: green residual" )
    axes[1, 1].axis("off")
    axes[1, 2].imshow( blue_fft, cmap="magma", vmin=fft_vmin, vmax=fft_vmax, )
    axes[1, 2].set_title( "FFT: blue residual" )

    axes[1, 2].axis("off")

    # Shared colorbar
    fig.colorbar( fft_image, ax=axes[1, :], fraction=0.025, pad=0.02, label="log(1 + FFT magnitude)", )

    # All done!!!
    plt.savefig( output_path, dpi=200, bbox_inches="tight", )
    plt.close(fig)
    print(f"Saved: {output_path}")


if __name__ == "__main__":
    analyze_zero_mean_residual(
        image_path="ai-generated-images/pure-white-blank-canvas1.png",
        output_path="zero_mean_analysis.png",
        amp_factor=50,
    )

There is a lot of data to unpack, but we are mostly interested in mean RGB values and the residual standard deviation. Green seems to have a lot more residual energy, but we can't conclude that it is carrying the watermark.

We can use Pillow to generate a-like image. We will get the exact value distribution per channel and generate a random image with the same histogram.

random_control = np.empty_like(source, dtype=np.uint8)
channel_names = ["Red", "Green", "Blue"]
for i, channel_name in enumerate(channel_names):
    channel = source[:, :, i]
    unique_vals, counts = np.unique(channel, return_counts=True)
    probs = counts / counts.sum()
    random_channel = rng.choice( unique_vals, size=(h, w), p=probs, ).astype(np.uint8) random_control[:, :, i] = random_channel
Image.fromarray(random_control).save(output_path)

Checking this newly generated image through the provenance API shows "SynthID not detected". So, just having the matching RGB distributions is not enough. The AI image contains spatial information that matters.

We will do one final spectral comparison here.

...
for i, channel_name in enumerate(channel_names):
    ai_mag, ai_log, ai_power = fft_data( ai_residual[:, :, i] )
    control_mag, control_log, control_power = fft_data( control_residual[:, :, i] )
    ai_fft.append(ai_log)
    control_fft.append(control_log)
    ai_entropy = spectral_entropy(ai_power)
    control_entropy = spectral_entropy(control_power)
    ai_peak = peak_prominence(ai_power)
    control_peak = peak_prominence(control_power)
    corr = spectrum_correlation( ai_log, control_log, )

    metrics.append({
        "channel": channel_name,
        "ai_entropy": ai_entropy,
        "control_entropy": control_entropy,
        "ai_peak": ai_peak,
        "control_peak": control_peak,
        "correlation": corr,
    })

    radial_ai.append( radial_power_spectrum(ai_power) )
    radial_control.append( radial_power_spectrum(control_power) )
...

This is an interesting result.

The AI image and control are identical but very different spectrally.

Spectral entropy:
AI image: ~0.889-0.900
Control image: ~0.970
Peak prominence:
AI image: ~716-1078
Control image: ~19-25
Spectrum correlation:
Zero

Running it a few more times showed:

Metric	Run 1	Run 2	Run 3	Controls
Red entropy	0.9000	0.9011	0.8895	~0.9704
Green entropy	0.8893	0.8818	0.8702	~0.9704
Blue entropy	0.8970	0.8896	0.8817	~0.9704
Red peak	715.7	554.5	485.2	~21–25
Green peak	1077.8	884.5	633.6	~19–21
Blue peak	777.5	718.3	811.3	~19–24

In these samples, SynthID images show much lower spectral entropy than their matched random controls.

I am now curious to see how much shaving off or limiting high-frequency spikes affects the image's appearance. We can convert each color channel into the frequency domain (via a 2D Fast Fourier Transform) and shift the zero-frequency component to the center. Then we iterate over some magnitude thresholds, cap the ones that exceed and reconstruct the image back.

import numpy as np
from PIL import Image

THRESHOLDS = [300, 500, 750, 1000, 1500, 2000, 2500, 3000]

def evaluate_threshold(arr, threshold, dc_radius=15):
    h, w, c = arr.shape
    reconstructed = np.zeros_like(arr)

    y, x = np.ogrid[:h, :w]
    cy, cx = h // 2, w // 2
    dc_mask = ((x - cx) ** 2 + (y - cy) ** 2) <= dc_radius ** 2

    per_channel_affected = []
    max_imag_residue = 0.0

    for i in range(c):
        channel = arr[:, :, i]

        fft_raw = np.fft.fft2(channel)
        fft_shifted = np.fft.fftshift(fft_raw)

        magnitude = np.abs(fft_shifted)
        phase = np.angle(fft_shifted)

        capped_mask = (magnitude > threshold) & (~dc_mask)
        affected_count = np.sum(capped_mask)
        affected_fraction = (affected_count / magnitude.size) * 100.0
        per_channel_affected.append(affected_fraction)

        magnitude[capped_mask] = threshold
        fft_capped = magnitude * np.exp(1j * phase)

        inverse = np.fft.ifft2(np.fft.ifftshift(fft_capped))

        max_imag_residue = max(
            max_imag_residue,
            np.max(np.abs(inverse.imag))
        )

        reconstructed[:, :, i] = inverse.real

    overall_affected_fraction = float(np.mean(per_channel_affected))

    # Float-domain error
    diff_float = reconstructed - arr
    rmse_float = np.sqrt(np.mean(diff_float ** 2))
    mae_float = np.mean(np.abs(diff_float))
    max_abs_float = np.max(np.abs(diff_float))

    # 8-bit / saved-image error
    arr_8bit = np.clip(np.round(arr), 0, 255).astype(np.uint8)
    recon_8bit = np.clip(np.round(reconstructed), 0, 255).astype(np.uint8)

    diff_8bit = arr_8bit.astype(np.float64) - recon_8bit.astype(np.float64)
    rmse_8bit = np.sqrt(np.mean(diff_8bit ** 2))

    pixel_changed_mask = np.any(arr_8bit != recon_8bit, axis=2)
    pixels_changed_pct = (np.sum(pixel_changed_mask) / (h * w)) * 100.0

    subpixels_changed_pct = (
        np.sum(arr_8bit != recon_8bit) / arr_8bit.size
    ) * 100.0

    return {
        "threshold": threshold,
        "affected_overall_pct": overall_affected_fraction,
        "affected_red_pct": per_channel_affected[0],
        "affected_green_pct": per_channel_affected[1],
        "affected_blue_pct": per_channel_affected[2],
        "rmse_float": rmse_float,
        "mae_float": mae_float,
        "max_abs_float": max_abs_float,
        "rmse_8bit": rmse_8bit,
        "pixels_changed_pct": pixels_changed_pct,
        "subpixels_changed_pct": subpixels_changed_pct,
        "max_imag_residue": max_imag_residue,
    }


def run_sweep(input_path, dc_radius=15):
    print("--- Running Threshold Sweep ---")
    print(f"Image: {input_path}")
    print(f"DC protection radius: {dc_radius}")

    img = Image.open(input_path).convert("RGB")
    arr = np.array(img, dtype=np.float64)

    print(
        f"{'Threshold':<10} | "
        f"{'FFT Avg':<9} | "
        f"{'R%':<7} | "
        f"{'G%':<7} | "
        f"{'B%':<7} | "
        f"{'Float RMSE':<11} | "
        f"{'8-bit RMSE':<11} | "
        f"{'RGB Px':<9} | "
        f"{'Subpx':<9} | "
        f"{'Max Imag'}"
    )
    print("-" * 120)

    for t in THRESHOLDS:
        result = evaluate_threshold(arr, t, dc_radius=dc_radius)

        print(
            f"{result['threshold']:<10} | "
            f"{result['affected_overall_pct']:>7.2f}% | "
            f"{result['affected_red_pct']:>6.2f}% | "
            f"{result['affected_green_pct']:>6.2f}% | "
            f"{result['affected_blue_pct']:>6.2f}% | "
            f"{result['rmse_float']:>10.4f} | "
            f"{result['rmse_8bit']:>10.4f} | "
            f"{result['pixels_changed_pct']:>7.2f}% | "
            f"{result['subpixels_changed_pct']:>7.2f}% | "
            f"{result['max_imag_residue']:.2e}"
        )

if __name__ == "__main__":
    run_sweep(
        "ai-generated-images/pure-white-blank-canvas1.png",
        dc_radius=15,
    )

This result ruled out my naive but tempting shortcut.

At 1500, I am changing about 5% of FFT bins, but only ~0.5% of RGB pixels survive quantization. But at 2000, I am changing about 2.6% of the FFT bins, but the 8-bit image is unchanged. This probably means threshold sweeps aren't a good way to target recurring peaks.

The strong FFT peaks are interesting, but blindly capping them was not a good way to isolate whatever the detector was seeing.

Final words

I started this experiment thinking the watermark was simple stuff. But the data decided to consume my evening. The takeaway is that the AI-generated images contained small pixel variations, and they do not behave like random noise. When I compared them with random controls, the AI-generated ones showed lower spectral entropy and near-zero spectrum correlations with our controls. And my attempt to suppress the high-energy FFT coefficients didn't turn out the way I hoped.

So I didn't find the SynthID frequency (yet). Maybe this needs further investigation.