Skip to main content
Compliance Engine

Compliance the engine enforces

Most stacks bolt compliance on at the edges — a scrubber before ingest, a filter in the application, a policy document for everything else. Lucenia enforces it inside the retrieval engine, on the way in and on the way out, so the guarantee does not depend on every client remembering to ask for it.

Talk to us Read the docs

Two enforcement points, one policy

Redaction at ingest protects what is stored. Redaction at query time protects what is returned. You need both, because the same document is legitimately readable by one person and not by another.

Enforced at ingest

A compliance processor scans fields as documents are indexed, applies the profile you name, and stamps what it found. Sensitive values are handled before they ever reach a segment.

Enforced at query time

A response processor redacts field text on the way out, per request, against the caller’s verified identity. The same document returns different text to different people.

Fail-closed by default

Exemption is granted by security role or backend role. A request that arrives without a verified identity is redacted, not exempted — you opt in to the looser behaviour, never out of the safe one.

Auditable, not implied

Every scanned document carries a _compliance record: the profile applied, how many values were found, how many were redacted, and a count per entity type. Your audit trail is data, not a log line.

One document, two readers

Exemption is resolved from the verified security identity on the request — not from a header the caller sets, and not from application code. A cleared analyst sees the field as written. Everyone else sees the same result set with the sensitive spans replaced by their entity type, so the finding is still there even when the value is not.

  • Exempt by security role or backend role
  • Applies to the fields you name, per search pipeline
  • No second index, no shadow copy, no per-clearance duplication
clearedexempt role
Contact analyst j.rivera@agency.gov from 10.4.22.9 re: case file.
unclearedredacted at query time
Contact analyst <EMAIL_ADDRESS> from <IP_ADDRESS> re: case file.

Name a profile, not a regex

Profiles ship with the engine and map to the frameworks your auditors already recognise. Point a pipeline at one and the matching entity set is enforced for you.

GDPRgdpr
CCPAccpa
HIPAAhipaa
PCI-DSSpci_dss
SOC 2soc2
FedRAMPfedramp
ISO 27001iso_27001
NIST 800-53nist_800_53
PII (baseline)pii_basic

The full framework, preset, and policy catalog is queryable at runtime through the compliance REST API, so what you build against is the deployment’s real capability rather than a number on a slide.

Entities the engine knows

Detection is built in, jurisdiction-aware, and validated — card numbers check against Luhn, IBANs against mod-97, national identifiers against their own checksum rules, so a plausible string is not treated as a real one.

EMAIL_ADDRESSPHONE_NUMBERUS_SSNUS_ITINUS_PASSPORTDATE_OF_BIRTHMEDICAL_RECORD_NUMBERMEDICAL_LICENSECREDIT_CARDIBAN_CODEUS_BANK_ROUTINGCRYPTOIP_ADDRESSAWS_ACCESS_KEYAPI_KEYUK_NINOIN_AADHAARAU_ABNES_NIFSG_NRIC_FIN…and more

Wired in a pipeline, not in your app

Both enforcement points are pipeline configuration. Nothing in your query path changes, and no client can opt itself out.

1 — Scan and stamp at ingest
PUT _ingest/pipeline/analyst-compliance
{
  "processors": [
    {
      "compliance": {
        "fields": ["analyst_note"],
        "profile": "nist_800_53"
      }
    }
  ]
}
2 — Redact per identity at query time
PUT _search/pipeline/geo-compliance
{
  "response_processors": [
    {
      "compliance_redact": {
        "fields": ["analyst_note"],
        "profile": "nist_800_53",
        "exempt_backend_roles": ["cui-cleared"]
      }
    }
  ]
}

A caller holding an exempt role sees the field as written. Everyone else gets the redacted form — and a request that arrives with no verified identity is redacted too, becauseexempt_when_no_identity defaults to false.

Write your own policy

A built-in profile is a starting point, not a ceiling. Define a named policy once, store it in the cluster, and reference it from every pipeline that needs it — a base profile refined by per-entity redaction modes, exclusions, and your own detectors.

Create a named policy
PUT /_plugins/_compliance/policies/clinical-notes
{
  "profile": "hipaa",
  "audit": false,
  "hash_salt": "rotate-me",

  "overrides": {
    "credit_card": "partial",
    "email_address": "hash",
    "date_of_birth": "tokenize"
  },

  "exclude": ["ip_address"],

  "detectors": [
    {
      "name": "trial_subject_id",
      "pattern": "\\bSUBJ-[0-9]{4}-[A-Z]{2}\\b",
      "category": "PHI",
      "mode": "tokenize",
      "max_matches": 200
    },
    {
      "name": "internal_dea",
      "pattern": "\\b[A-Z]{2}[0-9]{7}\\b",
      "category": "PHI",
      "mode": "mask",
      "validator": "dea"
    }
  ]
}
profileThe base to inherit — or none for a detector-only policy that enforces nothing but your own rules.
overridesPer-entity redaction mode, so one policy can hash an email and only partially mask a card.
excludeTurn off a built-in entity your corpus legitimately needs in the clear.
detectorsYour own patterns, with an optional checksumvalidator (luhn, iban_mod97, dea, uk_nhs, au_abn and others) so a match has to be structurally valid, not merely shaped right.
auditForces every entity to tag — detect and record without changing a byte. The safe way to measure a policy before you enforce it.

Redaction modes

maskReplace with a typed placeholder — <EMAIL_ADDRESS>. Unambiguous and non-reversible.
partialKeep the last four characters — ************1234. For reconcilable identifiers.
hashDeterministic salted hash. Redacted data stays join-able for analytics.
tokenizeStable per-document pseudonym — <PERSON_1>. Preserves referential structure.
removeDrop the span entirely.
tagDetect and record without altering the text. What audit mode forces everywhere.
Use it by name
{
  "compliance": {
    "fields": ["note"],
    "policy": "clinical-notes"
  }
}
Manage and inspect
GET    /_plugins/_compliance/policies
GET    /_plugins/_compliance/policies/clinical-notes
DELETE /_plugins/_compliance/policies/clinical-notes

GET    /_plugins/_compliance/frameworks
GET    /_plugins/_compliance/presets

Evidence, per document

Every scanned document carries what the engine found. That record is a normal field — queryable, aggregatable, and exportable — so “show me every record where PHI was detected last quarter” is a search, not a spreadsheet exercise.

  • Which profile or policy was applied
  • How many values were found and how many were redacted
  • A count per entity type
_compliance
"_compliance": {
  "profile": "nist_800_53",
  "total": 4,
  "redacted": 3,
  "counts": {
    "EMAIL_ADDRESS": 2,
    "IP_ADDRESS": 1,
    "US_SSN": 1
  }
}

See it against your data

Bring a corpus and a clearance model. We will wire the pipelines and show you the same query returning different text to different roles.

Talk to us Explore the platform