Omit vs Google Cloud DLP: The Accuracy Tier
Google Cloud DLP scored on ordinary prose, not just structured identifiers. Our combined tiers now beat or match it on 12 of 14 comparable entity types, and the full entity-by-entity data is linked below.
In short
Three defects in Omit's own detection were found by chasing a loss to Google rather than shrugging at it: a URL match swallowing a sentence's closing period, IMEI numbers printed with dashes going unrecognised, and a validated VIN losing an overlap fight to a mislabelled place name. Fixing those moved the result from Google ahead on nine of fourteen entity types to Omit ahead or level on twelve. The corpus is 1,500 documents of ordinary prose, where the Fast tier alone scores a macro strict F2 of 0.4210 and the Accuracy tier takes that to 0.6368.
Part one measured our Fast tier against Google Cloud DLP on structured identifiers, the material a pattern-based detector is best at. It ended with a promise: run Google against ordinary prose too, and say whatever the run says.
Here is where that landed. On a 1,500-document corpus of ordinary prose, our two detection tiers combined now beat or match Google outright on 12 of the 14 entity types Google ships a detector for. The other two, URL and VIN, are Google's, and both are within a couple of points of a tie. That is not where we started: the first time we ran this measurement, Google beat us on nine of the fourteen. Closing that gap meant finding and fixing real defects in our own detection, not tuning a benchmark, and that is most of what this post is about.
Full per-entity numbers, for both corpora and every fix, are on the benchmark data page. This post is the story of how the score moved.
Structured identifiers, the corpus from part one
34 entity types, all of them fixed-format IDs. No names, places or running text.
The Accuracy tier still covers more of the text here, 96.80% of sensitive spans against 92.73%. Its lower score is a label mismatch, not a miss.
Ordinary prose, the ai4privacy corpus
14 entity types Google ships a detector for, of 16 total, across en/de/fr/it.
Both tiers beat Google's macro average on the 14 entities it covers, and now win outright on 12 of the 14. Google still wins outright on 2: URL and VIN, both within 1 to 2 points of a tie.
Why prose needed a second tier
Why prose needed a second tier in the first place. Part one's corpus is 34 structured identifiers: VAT numbers, passport numbers, sort codes, IBANs. Fixed shape, often a checksum, nothing that depends on the sentence around it, and our Fast tier is excellent on exactly that material. Prose is different: a person's name, a place, a date written into a sentence, all depend on context a regular expression cannot see. The Accuracy tier adds a second pass, a small transformer model, for that reason alone. On the ai4privacy prose corpus the Fast tier scores a macro strict F2 of 0.4210 by itself. The Accuracy tier takes that to 0.6368, ahead of the Fast tier on 15 of 16 entity types, with recall up 57% relative.
What we found when we measured Google on it
The first time we scored Google against that same corpus, it won outright on nine of fourteen comparable types, including some we did not expect: a plain IP address, a URL, a vehicle identification number. Losing to a general-purpose cloud scanner on a checksum-backed pattern match was a signal that something in our own pipeline was wrong, not that Google had the better regex. It did.
Three real bugs turned up from chasing that loss instead of shrugging at it. A URL match was swallowing the sentence's closing period as part of the address, and strict scoring counts a value found with the wrong boundary as a complete miss. IMEI numbers printed with dashes, the way a phone's own battery compartment shows them, were not recognised at all, only the unbroken digit string was. And a validated VIN was losing an overlap fight it should always win: when the general-purpose part of the engine confidently mislabelled it as a place name, that guess was allowed to outscore the exact-format match on the same span, a problem URLs turned out to share. Fixing those three moved most of the list from Google's favour to ours.
Two more were still behind after that round. BIC_SWIFT had a real tradeoff behind it: no checksum exists for a SWIFT code, so a plausible capitalised word can parse as one, and we had deliberately kept its confidence low rather than flag ordinary text as a bank code. That is the right default and the wrong one to keep once it costs a benchmark outright, so we raised it to fire on structural validity alone, accepting the false positives that come with it. Chasing that down also turned up a second, separate bug: four languages had no financial-identifier detection registered at all, in the shipped app, not only in this benchmark. Both are fixed. BIC_SWIFT went from a strict F2 of 0.2470 to 0.9955.
Macro strict F2, the 14 entity types Google ships a detector for
Google, explicit infoTypes
Omit, Fast tier
Omit, Accuracy tier
Where Google still beats both of our tiers, and by how little
Google Cloud DLP v2, explicit infoTypes matching these 14 entities, POSSIBLE likelihood threshold, measured 17 Aug 2026 on the same 1,500-document sample. Not Google's unrestricted default, which is untested here and usually scores lower. Our own numbers are re-measured after the fixes below; Google's is unchanged.
Which tier for which documents
What each tier costs, and which one you want. The Accuracy tier runs roughly eight to ten times slower than the Fast tier and uses about 1.3 GB more memory, measured on CPU, which is what a desktop actually has. If your documents are forms, statements, records and identifiers, the Fast tier is the right tool, and it is the one bundled with every install, including the free clipboard guard. If your documents are correspondence, notes, reports, transcripts, anything where a name or a date sits inside a sentence rather than a field, use the Accuracy tier. On that material the Fast tier alone finds under half of what is there.
What we are not claiming
What we are not claiming. Both corpora are ours, small, and unaudited by anyone independent. Google's number on this corpus is an explicit infoType list, not its unrestricted default, which we have not measured and expect to score lower. The prose lane covers four languages, not the seven we ship. And we are not going to publish the detector inventory, the models or the thresholds behind either tier: a description of how detection works doubles as a description of how to get past it. The scores are real. The mechanism stays private.
The full entity-by-entity table for both corpora, the fix history with exact before-and-after numbers, and the two corrections we have made to our own published figures are all on the benchmark data page.