Omit vs Google Cloud DLP: The Full Benchmark

This is the reference page behind our Google Cloud DLP comparison: every scored cell, both corpora, both tiers, both Google configurations, in one place. The blog posts tell the story; this page is the working. Nothing here is narrative, so start with the numbers.

Method

Two corpora. Benchmark v1 is synthetic, 34 structured identifier types with a fixed shape and often a checksum: VAT numbers, national IDs, passports, bank codes. The ai4privacy corpus is 1,500 documents of ordinary prose in English, German, French and Italian: emails, notes, records, the kind of text a person actually pastes into something. The metric is strict F2 on both: recall-weighted, and a detection only counts when the span and the type both match exactly. Google Cloud DLP v2 was run at its POSSIBLE likelihood threshold. On the structured corpus it was scored twice, once on its default infoType configuration and once handed the exact infoType for every entity in the corpus, which is close to a best case for it. On the prose corpus it was scored once, with an explicit infoType list rather than its unrestricted default, because an unrestricted scan on ordinary text is likely to return infoTypes our label map never anticipated. We have not measured Google's unrestricted default on either corpus, and defaults are usually worse, not better.

Structured identifiers, all 34 types

Ordinary prose, all 14 comparable types

Why the Accuracy tier's structured-ID score reads lower

On the structured corpus the Accuracy tier covers more of the text than the Fast tier, 96.80% of gold spans against 92.73%, but scores a lower strict F2. That is the metric, not the redaction. Strict F2 needs an exact type match, and the transformer often returns a broader-but-true label instead of the specific one our gold data expects.

What we found and fixed on the way here

The first prose measurement against Google, on 13 Aug 2026, found Google beating both our tiers outright on nine of fourteen entity types. Chasing that down turned up real defects, not benchmark tuning: a URL match was swallowing the sentence's closing period, which strict scoring counts as a total miss; IMEI numbers printed with dashes were not recognised at all; a validated VIN was losing an overlap fight it should always win, because a confident but wrong general-purpose guess was allowed to outscore an exact-format match on the same span, a bug URLs shared. Fixing those closed most of the gap. Two entities were still behind on 17 Aug: BIC_SWIFT, because we deliberately kept its confidence low to avoid flagging an ordinary capitalised word as a bank code, and a separate registration bug that left four languages (Spanish, Italian, Polish, Finnish) with no financial-identifier detection at all, in the shipped app, not only the benchmark. Both are fixed now: BIC_SWIFT moved from 0.2470 to 0.9955, and the fourth language, Italian, moved from 0.0 to 1.0 once the detector was actually wired up for it.

Corrections to our own published numbers

Cost

What this does not measure

Both corpora are ours, both benchmarks are small, and nobody independent has audited either. The structured corpus's Google numbers are several weeks older than ours, because rerunning them costs money and the predictions are stable, but the service may have moved since. The prose corpus covers four languages, not the seven we ship. And we are not publishing the detector inventory, the models, the weights or the thresholds behind any of this: a published description of how detection works is a published description of how to evade it. The numbers are real; the mechanism stays private.

Every file behind this page is in the product repository: the raw scored cells for both tiers on both corpora, the frozen Google cells, and the scripts that produced all of it.