IC-133GPT-4, Claude 3, and Gemini 1.0 Pro do not exhibit detectable watermarks from the red-green, fixed-sampling, or cache-augmented families under black-box statistical tests
Thibaud Gloaguen, Nikola Jovanović, Robin Staab, Martin Vechev
The authors applied their three watermark detection tests (red-green logit-bias test, fixed-sampling diversity test, and cache-augmented distribution-shift test) to the public APIs of GPT-4, Claude 3, and Gemini 1.0 Pro. For every model-test combination, the null hypothesis (no watermark of that family is present) was not rejected at the 95% confidence level. The authors conclude that they cannot confirm the presence of a watermark on any of these three deployments. This is a null result: the absence of a detectable signal does not prove the absence of a watermark, but it does indicate that if a watermark is present, it is not from one of the three families tested or is implemented in a way that evades these specific statistical signatures.
The authors note they cannot conclude on the presence of a watermark; the tests are restricted to three scheme families and make assumptions (symmetric error terms, perfect sampling) that may not hold for all models. A watermark from a novel family or one with theoretical undetectability guarantees would not be detected.