data-masking-techniques-reference
Pure-reference catalog of data-masking techniques and de-identification privacy models. Enumerates the seven canonical masking operators (substitution, shuffling, number/date variance, encryption, hashing, nulling, masking-out / character-scrambling) plus tokenisation, redaction, format-preserving encryption, and Microsoft Presidio's six built-in operators. Distinguishes reversible techniques (pseudonymisation candidates per GDPR Art. 4(5)) from irreversible techniques (anonymisation candidates), and maps them to NIST SP 800-188 privacy models - k-anonymity, l-diversity, t-closeness, differential privacy (deep model definitions in references/). Cites ISO/IEC 20889:2018 for the standard taxonomy. Use to pick the right masking operator per field type and risk level.
Install with skills.sh (any agent)
npx skills add testland/qa --skill data-masking-techniques-referencedata-masking-techniques-reference
Overview
Masking is the act of transforming a real value into a substitute that breaks the link to the original subject while preserving testable properties (format, distribution, referential integrity). Which technique is correct depends on three things: whether the result must be reversible, whether the field is referentially shared across tables, and what privacy model the dataset must satisfy.
This skill is the pure reference that the pipeline builder (pii-masking-pipeline-builder) and leak-detection audits draw from to choose operators per field.
When to use
How to use this reference
The seven canonical masking techniques
Drawing from the Wikipedia data-masking taxonomy (en.wikipedia.org/wiki/Data_masking (opens in new window)) and ISO/IEC 20889:2018 (cite by stable ID; standard text behind paywall):
1. Substitution
Replace the real value with an authentic-looking value from a lookup table - "John Smith" → "Maria Garcia."
2. Shuffling
Randomly rearrange values within a column - salaries column gets shuffled, each row keeps a real salary but no longer the right person's salary.
3. Number / date variance
Apply a bounded random offset: salary ± 10 %, dates ± 120 days (Wikipedia data-masking page).
4. Encryption
Apply a cryptographic algorithm with a key. Two sub-variants:
5. Hashing
Apply a one-way hash (SHA-256 / SHA-512) with optional salt.
6. Nulling out / deletion
Replace the value with NULL or remove the column entirely.
7. Masking-out / character scrambling
Show partial value - credit card "**** **** **** 1234," email "j***@example.com."
Additional techniques
Tokenisation
Replace the real value with a token (random opaque string) and store the real-value → token map in a separate, access-controlled vault.
Redaction
Remove the value entirely (no placeholder, no length signal).
Synthetic substitution
Replace with a synthetically generated value preserving distribution / format (faker-synthetic-data; synthea-healthcare-data for health records).
Microsoft Presidio anonymizer operators
Per presidio.dataprivacystack.org/anonymizer (opens in new window), the Presidio Anonymizer engine supports six built-in operators:
| Operator | Parameters | Reversible | Maps to canonical technique |
|---|---|---|---|
replace | new_value (defaults to <entity_type>) | No (random) / Yes (deterministic substitution) | #1 Substitution |
redact | - | No | Redaction |
mask | chars_to_mask, masking_char, from_end | No | #7 Masking-out |
hash | hash_type (sha256 / sha512), salt | No (one-way) | #5 Hashing |
encrypt | key | Yes (with key) | #4 Encryption |
custom | lambda | Depends on lambda | (caller-defined) |
Invocation: engine.anonymize(text=, analyzer_results=, operators={"PERSON": OperatorConfig("replace", {"new_value": "BIP"})}).
OperatorConfig constructor signature: OperatorConfig(operator_name, params={}) (Presidio docs).
Reversible vs irreversible - pseudonymisation vs anonymisation
GDPR Art. 4(5) defines pseudonymisation as "processing of personal data in such a manner that the personal data can no longer be attributed to a specific data subject without the use of additional information, provided that such additional information is kept separately" (gdpr-info.eu/art-4-gdpr/ (opens in new window)).
| Technique | Pseudonymisation? | Anonymisation? |
|---|---|---|
| Deterministic substitution (same input → same output) | ✓ | - |
| Random substitution | - | ✓ |
| Shuffling | - | ✓ (when distribution-only) |
| Number / date variance | - | ✓ if variance ≥ identifying granularity |
| General encryption (key kept) | ✓ | - |
| FPE (key kept) | ✓ | - |
| Salted hashing (salt kept separately) | ✓ | - |
| Unsalted hashing of low-entropy field | ✗ (re-identifiable by enumeration) | ✗ |
| Nulling | - | ✓ |
| Masking-out (partial) | depends on revealed chars | depends |
| Tokenisation (vault kept) | ✓ | - |
| Tokenisation + vault destroyed | - | ✓ |
| Redaction | - | ✓ |
| Synthetic substitution | - | ✓ |
Implication: A "masking pipeline" output that uses reversible techniques is still personal data under GDPR - it remains in scope. Only fully irreversible output is out of GDPR scope per Recital 26.
Privacy models - NIST SP 800-188
NIST SP 800-188:2023 formalises statistical privacy models that sit above the per-field operators - pick one for the whole dataset's disclosure risk once quasi-identifiers remain after masking:
Full definitions, achievement methods, weaknesses, and ε / k guidance (with NIST + primary-source citations): references/privacy-models.md.
Picking a technique per field
| Field characteristic | Recommended technique | Privacy model layer |
|---|---|---|
| Must round-trip for authorised consumer (payment processing) | Tokenisation (vault) or FPE | none (reversible) |
| Must join across tables, opaque value OK | Deterministic substitution / salted hashing | k-anonymity on quasi-identifiers |
| Free-text PII inside a log line | Redaction or replace-with-<TYPE> (Presidio analyzer + anonymizer) | - |
| Continuous numeric for analytics | Number variance | t-closeness if sensitive attribute |
| Categorical demographic (race, etc.) for analytics | Generalisation + l-diversity | l-diversity |
| Statistical query release | Differential privacy mechanism | DP |
| Demo / training, no analytics utility needed | Synthetic substitution (Faker / Synthea) | n/a (no real data) |
Worked example - masking a non-prod customers table
An analytics team needs a non-prod copy of a customers table. Walk each field through the steps in "How to use this reference":
| Field | Need | Operator | Scope outcome |
|---|---|---|---|
customer_id (FK, joined across tables) | Opaque but joinable | Deterministic substitution / salted hashing (#1 / #5) | Pseudonymised - reversible via key |
full_name | No analytics value | Random substitution (Faker) | Anonymised |
email | Support must recognise own value | Masking-out j***@example.com (#7) | Partial - depends on revealed chars |
national_id (SSN) | No analytics value; enumerable format | Nulling out (#6) - never unsalted hashing | Anonymised |
date_of_birth | Age band useful | Generalise to a band (age 47 → "40 - 50") | Anonymised (k-anonymity input) |
salary | Distribution useful | Number variance ± 10 % (#3) | Anonymised - t-closeness if sensitive |
auth_token | No analytics value | Nulling out / deletion (#6) | Anonymised |
Resulting scope: because customer_id uses a reversible deterministic map, the output is pseudonymised - still personal data under GDPR Recital 26. To move the dataset out of scope, destroy the substitution key so customer_id can no longer be re-linked. The remaining quasi-identifiers (date_of_birth band, salary bracket) then need a dataset privacy model - see references/privacy-models.md.
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Unsalted hashing of SSN | SSN format is enumerable (~10⁹); attacker rebuilds the mapping table in minutes. | Salt + key per tenant; or tokenise via vault. |
| FPE for an analytics dataset | Format preservation lets a join attack with another dataset recover identity. | Use random substitution for analytics datasets that don't need format round-trip. |
| "GDPR-compliant" pseudonymisation claim | GDPR pseudonymised data is still personal data - Article 4(5) is explicit. | Either mark output pseudonymised (in scope) or fully anonymise (out of scope). |
| k = 2 anonymity | Re-identification probability is 50 % for the equivalence class. | k ≥ 5 typical; k = 10+ for high-risk datasets. |
| Shuffling a rare-value column | Outliers identify themselves regardless of position. | Combine shuffling with generalisation or suppression of outliers. |
| Number variance ± 1 % on salaries | The variance is smaller than the precision needed to identify; effectively no masking. | Variance must exceed the identifying granularity - ± 10 % minimum for salary. |
| Tokenisation without vault access controls | The vault becomes the single point of failure. | Strict access control + audit logging + separate key custody. |
| Differential privacy with ε = 100 | Useless budget; no privacy guarantee. | ε ≤ 1 typical for strong privacy; ε ≤ 10 for relaxed cases. |
Limitations
References
Privacy models - NIST SP 800-188
View source (opens in new window)Privacy models - NIST SP 800-188
Deep reference for data-masking-techniques-reference SKILL.md. Consult after per-field operators are chosen, when quasi-identifiers remain and the whole dataset's disclosure risk must be bounded.
NIST SP 800-188:2023 ("De-Identifying Government Datasets", csrc.nist.gov/pubs/sp/800/188/final (opens in new window)) formalises statistical privacy models layered above the per-field techniques.
k-anonymity
A dataset is k-anonymous if every record is indistinguishable from at least k - 1 other records when projected on the quasi-identifiers (Sweeney 2002, cited in NIST 800-188).
l-diversity
Strengthens k-anonymity by requiring at least l well-represented values of the sensitive attribute within each equivalence class (Machanavajjhala et al. 2007).
t-closeness
Strengthens l-diversity by requiring the distribution of the sensitive attribute in each equivalence class be close (within t, by Earth Mover's Distance) to the distribution in the overall dataset (Li et al. 2007).
Differential privacy
A formal mathematical guarantee: the probability of any output changes by at most a multiplicative factor (e^ε) when a single record is added/removed. ε (epsilon) is the privacy budget - lower ε = stronger privacy.
Picking the model
References
Related skills
faker-synthetic-data
Substitutes realistic replacement values for PII that a masking or de-identification step removed, nulled, or redacted, so a non-production dataset stays usable. Covers building an injective substitution map that keeps a shared identifier consistent everywhere it appears so joins survive; choosing deterministic (seeded) over random substitution, and the re-identification risk a shared or committed seed reintroduces, since a generator seed is a reproducibility control and not a cryptographic key; preserving field shape where a downstream system validates it, including check-digit values such as payment-card and national-ID numbers plus phone and postal formats; and why a value that merely looks realistic is not yet safe, leaving a residual re-identification measurement over the remaining quasi-identifiers. Use when a masking pipeline has nulled or dropped PII columns and the dataset now needs replacement values that keep cross-table joins intact.
k-anonymity-verifier
Verifies that a masked dataset satisfies k-anonymity, l-diversity, and t-closeness by computing equivalence classes over chosen quasi-identifiers and reporting re-identification risk. Covers quasi-identifier selection heuristics, threshold guidance, pycanon API (k_anonymity / l_diversity / t_closeness / report), ARX Java API and GUI workflow, SmartNoise for differential-privacy comparison, and CI-gate integration. Distinct from data-masking-techniques-reference (which catalogs masking operators but defers k-anonymity measurement to dedicated tooling) and from presidio-pii-detection (which detects PII spans but offers no equivalence-class analysis). Use when you need to confirm whether a masked dataset meets a stated k, l, or t threshold before promoting it to a non-production environment.
pii-categories-reference
Pure-reference catalog of personally identifiable information (PII) categories across GDPR, CCPA/CPRA, NIST SP 800-122, and HIPAA. Defines what counts as personal data under each regime, enumerates the explicit identifiers each regulator lists (GDPR Art. 4(1) and Art. 9 special categories; CPRA sensitive personal information; NIST direct-identifier vs linkable distinction; HIPAA Safe Harbor 18 identifiers), and maps overlapping fields across jurisdictions so a masking pipeline knows which regulator's rules apply. Use as the authoritative source when authoring or reviewing masking rules, classifying a dataset's risk level, or scoping which fields a PII detector must catch.
pii-masking-pipeline-builder
Build-an-X workflow that produces a PII masking pipeline spec from a source-data inventory. Walks the author through (1) classifying each field against pii-categories-reference, (2) picking a masking operator from data-masking-techniques-reference, (3) deciding pseudonymisation (reversible, in GDPR scope) vs anonymisation (irreversible, out of scope), (4) ordering the pipeline (detect → operator → audit), and (5) emitting a deployable config for Presidio + Faker + Synthea wrappers. Output is a YAML pipeline spec plus a per-field rationale table. Use after classifying a dataset's PII risk; this is the workflow that translates classification into runnable masking config.
presidio-pii-detection
Author and run Microsoft Presidio PII detection - wraps presidio-analyzer (PII detector) + presidio-anonymizer (replace/redact/mask/hash/encrypt operators) for scanning datasets, log streams, and free-text fields. Covers AnalyzerEngine + AnonymizerEngine setup, built-in recognizers (PERSON, EMAIL_ADDRESS, CREDIT_CARD, US_SSN, IBAN_CODE, country-specific IDs across US/UK/Spain/Italy/Poland/Singapore/Australia/India and more), custom PatternRecognizer authoring, score thresholds, and CI gating. Use when scanning *existing* data for PII (vs synthesising fresh fixtures with synthetic-pii-generator).
synthea-healthcare-data
Author and run Synthea (MITRE's open-source synthetic patient population simulator) to produce HIPAA-safe synthetic medical records for testing health IT systems. Covers Gradle build, population-size and state-specific generation, FHIR R4 / STU3 / DSTU2 / C-CDA / CSV / CPCDS output formats, disease-module customisation, and the lifecycle-simulation approach (birth-through-death patient journeys with realistic demographics). Use when testing FHIR servers, EHR integrations, claims processing, or any health IT system that needs realistic patient records without HIPAA exposure (distinct from faker-synthetic-data which is generic; this is health-domain-specific).
test-data-governance-reference
Pure-reference catalog of test-data lifecycle governance: retention schedules for test datasets, cross-environment data-sharing agreements, deletion of test data containing real PII, refresh cadence, access controls, and the legal basis for each policy under GDPR Art. 5 storage limitation and NIST SP 800-122. Use when defining a data-steward role for test environments, authoring a retention policy for a test database, scoping a data-sharing agreement before promoting a dataset from production to staging, or determining the deletion timeline for any test fixture that contains live personal data.