Tokenization vs Masking vs Encryption

Three techniques get offered as the answer to the same question, and they are not interchangeable. Masking hides a value. Encryption scrambles it. Tokenization replaces it with a stand-in and files the original somewhere else. The choice really turns on one question: can the original come back, and who is holding the thing that brings it back.

Masking overwrites part of a value with a fixed character and leaves the rest visible. A card number becomes **** **** **** 4242. It is the simplest of the three because there is nothing to manage: no key, no store, no second system. In Omit it takes three settings, how many characters stay visible, which character replaces the rest, and whether the visible ones are counted from the start or the end, which is what a rule asking for the last four digits requires. In database products the same word covers both static masking, which rewrites a copy of the data, and dynamic masking, which rewrites query results for unprivileged users. Either way the removed characters were never written down, so nobody can bring them back, including whoever applied the mask.

Encryption transforms the value into ciphertext with a key, and the same key transforms it back. It is reversible by design, which is the point. What decides where it can be used is that ciphertext does not keep the original's shape. Encrypting a nine-character reference number produces a binary blob, and everything downstream expecting nine characters breaks, which is why encryption belongs around data at rest and in transit rather than inside a document somebody has to read. Whoever holds the key sees everything the key covers, so a key compromise is dataset-wide rather than per record.

Tokenization replaces the value with a token that has no mathematical relationship to it at all, and stores the mapping in a separate protected place. The token is a reference, not a transformation. A document says PERSON-K3M9Q and the name sits in the vault. It is reversible, but only through the vault, and only by someone who can open it.

The difference between tokenization and encryption is where the secret lives. With encryption the ciphertext contains the data, and the key is the only thing between an attacker and it. With tokenization the token contains nothing. There is no computation you can perform on a token that yields the original, because the relationship was never mathematical. Someone holding every token in a document holds zero values. That is why the payments industry moved card data to tokens rather than encrypting it in place: it takes the value out of the systems handling the document, and systems that never hold the value fall out of audit scope.

What you buy with that is a vault, and the vault becomes the thing that must not be lost. In Omit it is a single SQLite file under Documents/Omit/vault/vault.db, deliberately shared by all five products so a document tokenized in one can be re-identified in another, with no server involved anywhere. If that file goes, every token that ever pointed into it is permanently opaque. That is a property to plan a backup around, not a footnote.

Masking and tokenization look identical in the finished document and have opposite consequences. Both leave a readable placeholder. Masking discards, tokenization files. So the question is whether anyone will ever legitimately need the original back from this artifact. If no, mask, and enjoy having no key to steal and no vault to run. If yes, tokenize, and accept that the document is now exactly as safe as the vault. One further difference matters in practice: masking deliberately preserves a residual chosen by rule, the last four digits or the year of a date, while tokenization deliberately preserves nothing, because a token you could derive the value from would just be encryption with extra steps.

Pseudonymization is the legal word for this territory and tokenization is the engineering one. GDPR Article 4(5) describes processing personal data so it can no longer be attributed to a person without additional information kept separately and protected. Tokenization is one implementation of that, the one where the additional information is a vault. The consequence people skip is that pseudonymized data is still personal data with every obligation attached, because reversible by design means in scope by definition. In Omit the PSEUDONYMIZE operator with a vault attached mints a vault token; the same operator with no vault produces a visibly synthetic stand-in instead, in the form <PERSON_4C1A9F2B>, stable within a run and reversible by nobody including us, because nothing was written down.

Anonymization sits outside all three, because it is a claim about an outcome rather than a technique. Masking the name column while leaving date of birth, postcode and appointment date achieves nothing under the GDPR's actual test, which asks whether a person can still be singled out by any means reasonably likely to be used, including joining your data to somebody else's. Recital 26 puts genuinely anonymous data outside the regulation entirely, which is why the bar is set where it is. Tokenized data never clears it, by construction, because the vault exists.

The token itself is worth opening up, because most systems mint tokens from a counter and a counter leaks. A document whose last token is PERSON-14 has told you it named fourteen people, and a counter cannot be predicted, so a preview and the committed run disagree. Omit derives the token instead: an HMAC-SHA256 under a per-project key over the entity type and the normalized value, rendered in Crockford base32, an alphabet that already omits I, L, O and U so a token survives being read aloud. On a collision the key extends by a character rather than falling back to a counter, because two values collapsing onto one token would silently merge two people.

The per-project key matters as much as the derivation. It is an HMAC of the vault's data key with the project identifier rather than the data key itself, so the same person in two projects gets two unrelated tokens. Skip that and the token becomes a stable cross-project identifier, which is the thing a project boundary exists to prevent.

Inside the vault the value is stored encrypted rather than stored. Each row is AES-256-GCM, with the schema version, project, entity type, token and field name bound in as associated data, so a row cannot be lifted and replayed under a different token. The data key is sealed at rest by DPAPI on Windows, scoped to the user and the machine, and the recovery path wraps that same key with Argon2id under a passphrase, which is also how a vault moves to a new machine.

Two behaviours are what an auditor asks about. Checking whether a value has been seen before never decrypts anything: the lookup is a keyed tag derived from the data key, so the vault answers "same person as last time" without opening a row. And re-identification commits its audit row in the same transaction that returns the value, so no path through the code discloses a value without recording it. Re-identify is a licensed capability enforced at the dispatcher rather than in the interface, so hiding a button is not what stops it.

Choosing is easier working backwards from the artifact than forwards from the technique. If the output will be published and must never come back, mask or remove outright, and prefer removing, because a mask that keeps four digits does keep four digits. If a human will need the real value again, tokenize and treat the vault as an asset with a backup policy. If the whole thing has to come back and nobody reads it in between, encrypt, and accept the key as your perimeter. Most real documents want a mix: mask the card, tokenize the name, keep the case number untouched. That is why Omit settles this per entity type rather than offering one setting for the file.