AI Privacy Filter🔗
Starting with Lightrun SaaS version 1.98, AI Privacy Filter provides context-aware detection and redaction of sensitive text in Dynamic Logs and Snapshots. It uses OpenAI's open-weight Privacy Filter model, described in the official announcement, within Lightrun's service; it does not send action data to the OpenAI API.
AI Privacy Filter complements Lightrun's existing pattern-based PII redaction. Customer-configured redaction rules continue to run, while the AI model provides an additional layer that can recognize sensitive text from its surrounding context.
The AI in the feature name refers to how sensitive text is detected, not to who can view the action results. When enabled, the filter applies to supported Dynamic Log and Snapshot results whether the action was created manually or through Lightrun MCP.
For MCP-created Snapshots, detected sensitive text is redacted before results are returned through MCP. The AI assistant receives the redacted result rather than the original candidate text.
Important
AI Privacy Filter is disabled by default. A Company Manager must enable it for the organization.
Data protected by AI Privacy Filter🔗
When enabled, AI Privacy Filter processes supported text captured by every eligible action across the organization:
- Dynamic Logs: The formatted log message is redacted before Lightrun stores and shares it.
- Snapshots: Candidate text values, names, and capture status details are redacted before the completed Snapshot is stored and made available.
Redaction is asynchronous. Dynamic Logs and Snapshots may take longer to appear while Lightrun waits for the Privacy Filter service.
The model detects the following categories:
| Category | Examples |
|---|---|
| Person | Names and other private person references |
| Address | Private physical addresses |
| Private email addresses | |
| Phone | Private telephone numbers |
| URL | Private URLs |
| Date | Private dates |
| Account number | Credit card, bank account, and similar account numbers |
| Secret | Passwords, API keys, and other credentials |
Detected spans are replaced with <REDACTED>. If a value cannot be processed safely, Lightrun stores <PII_REDACTION_FAILED> in place of that value rather than storing the original candidate text.
Relationship to pattern-based PII redaction🔗
AI Privacy Filter does not replace your existing PII redaction configuration. Lightrun applies customer-configured patterns before it sends eligible text to the AI Privacy Filter.
Use both methods as complementary controls:
| Method | Best suited for |
|---|---|
| Pattern-based PII redaction | Organization-specific field names, identifiers, formats, and deterministic policies |
| AI Privacy Filter | Context-aware detection across the model's supported sensitive-data categories |
For information about creating deterministic rules, see Configure PII Redaction.
Enable AI Privacy Filter🔗
Before you begin:
- Ensure that you have Company Manager permissions.
To enable AI Privacy Filter:
- Log in to the Lightrun Management Portal.
- Click Settings.
- From the left navigation, select Service configuration under Compliance.
- Under the action-related settings, turn on AI Privacy Filter.
- Save the service configuration.
The setting applies across the organization. Turning it off stops new action data from using AI Privacy Filter, but customer-configured PII redaction rules continue to apply.
Reported model performance🔗
OpenAI reports the following results on the PII-Masking-300k benchmark:
| Evaluation | F1 score | Precision | Recall | Labeled PII not detected |
|---|---|---|---|---|
| Original benchmark | 96% | 94.04% | 98.04% | 1.96% |
| Corrected benchmark | 97.43% | 96.79% | 98.08% | 1.92% |
The corrected benchmark accounts for annotation issues OpenAI identified during evaluation. Labeled PII not detected is calculated as 100% - recall; it represents missed labeled PII in that benchmark, not a guaranteed miss rate for production data.
For evaluation details, see the OpenAI Privacy Filter announcement and model card.
Limitations🔗
AI Privacy Filter is a data-minimization aid. It is not an anonymization tool, a compliance certification, or a replacement for your organization's privacy review.
Consider the following limitations:
- The model can miss sensitive information or redact non-sensitive text.
- Detection is limited to the model's trained categories and decision boundaries.
- Performance can vary for non-English text, non-Latin scripts, regional naming conventions, and data that differs from the model's training distribution.
- Organization-specific identifiers and secret formats may require explicit pattern-based rules.
Test both AI and pattern-based redaction against representative application data before relying on the configuration in production. For additional guidance, see PII Redaction best practices.
For technical details about the underlying model, see the model repository and model card.