Skip to content

Instantly share code, notes, and snippets.

@burnsie7

burnsie7/log_redaction_examples.md

Created March 11, 2020 15:58
Show Gist options
  • Star 4 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
  • Save burnsie7/6c0ce97a507dfe485e28ba8f7899e008 to your computer and use it in GitHub Desktop.
Save burnsie7/6c0ce97a507dfe485e28ba8f7899e008 to your computer and use it in GitHub Desktop.
Download ZIP
PII Log Redaction
Raw
log_redaction_examples.md

Log Processing Rules

log_processing_rules:

Generic String: "sensitive-info"

  - type: exclude_at_match
    name: exclude_sensitive_info
    pattern: (?:sensitive\-info)

my_key=value

- type: mask_sequences
  name: redact_key_match_letters_numbers_spaces_unders
  replace_placeholder: "my_key=[VALUE REDACTED]"
  pattern: (?:my_key=[A-Za-z0-9\s_]*[A-Za-z0-9][A-Za-z0-9\s_])

my_key=value

- type: mask_sequences
  name: redact_key_match_to_period
  replace_placeholder: "my_key=[VALUE REDACTED]"
  pattern: (?:my_key=[^.])

SSN (Basic)

  - type: mask_sequences
    name: social_security_number_basic
    pattern: (?:\d{3}-?\d{2}-?\d{4})
    replace_placeholder: "[SSN REDACTED]"

Email Address

  - type: mask_sequences
    name: RFC_5322_email
    pattern: (?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])
    replace_placeholder: "[EMAIL REDACTED]"

Credit Card (Visa, MC, AMEX, DINERS, DISCOVER, JCB)

- type: mask_sequences
  name: visa_mc_amex_diners_discover_jcb_credit_card
  replace_placeholder: "[CREDIT CARD REDACTED]"
  pattern: (?:4[0-9]{12}(?:[0-9]{3})?|[25][1-7][0-9]{14}|6(?:011|5[0-9][0-9])[0-9]{12}|3[47][0-9]{13}|3(?:0[0-5]|[68][0-9])[0-9]{11}|(?:2131|1800|35\d{3})\d{11})

Postal Codes

- type: mask_sequences
  name: postal_codes
  replace_placeholder: "[POSTAL CODE REDACTED]"
  pattern: (?:\d{5}-\d{4}|\d{5}|[A-Z]\d[A-Z] \d[A-Z]\d)

Filename

- type: mask_sequences
  name: redact_filename_to_period
  replace_placeholder: "filename=[VALUE REDACTED]"
  pattern: (?:filename=[^.]+)

Example Config

- type: mask_sequences
  name: Example_config_redaction
  replace_placeholder: "ExampleConfig([REDACTED, REDACTED]"
  pattern: (?:ExampleConfig\([^\)]+)

On Example

- type: mask_sequences
  name: on_Example_redaction
  replace_placeholder: "on Example [Example REDACTED]"
  pattern: (?:on Example\s?[^\s]+)

Class A IP Address

- type: mask_sequences
  name: simple_ip_address
  replace_placeholder: "[IP REDACTED]"
  pattern: (?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)

(^10.)|

Tests

export LOGFILE=test.log

Basic String

echo 'this is sensitive-info yes?' >> $LOGFILE echo 'this is sensitive-infooryeno yes?' >> $LOGFILE echo 'this is 234sdfssensitive-infooryeno yes?' >> $LOGFILE

SSN

echo 'Testing a SSN 123-12-1232' >> $LOGFILE echo 'Testing a SSN surrouned by numbers 1123-12-123212' >> $LOGFILE echo 'Testing a SSN surrouned by special characters *1123-12-123212*' >> $LOGFILE

Email Address

echo 'Testing an email hello@domain.com' >> $LOGFILE echo 'Testing another email adf6787687@asdfsa.hy' >> $LOGFILE

Credit Cards

Visa

echo 'Testing Visa 4532795200169807' >> $LOGFILE echo 'Testing Visa 4556545631646571' >> $LOGFILE echo 'Testing Visa 4916196240977893' >> $LOGFILE echo 'Testing Visa 4539196185605084' >> $LOGFILE echo 'Testing Visa 4485412145416148' >> $LOGFILE

Mastercard

echo 'Testing Mastercard 5203626370595536' >> $LOGFILE echo 'Testing Mastercard 5117102640130160' >> $LOGFILE echo 'Testing Mastercard 5294478290306127' >> $LOGFILE echo 'Testing Mastercard 5588538758184343' >> $LOGFILE echo 'Testing Mastercard 5582276204809835' >> $LOGFILE

Discover

echo 'Testing Discover 6011981818590569' >> $LOGFILE echo 'Testing Discover 6011586647143173' >> $LOGFILE echo 'Testing Discover 6011599165325148' >> $LOGFILE echo 'Testing Discover 6011686138502562' >> $LOGFILE echo 'Testing Discover 6011898889747391' >> $LOGFILE

Amex

echo 'Testing Amex 371200017910123' >> $LOGFILE echo 'Testing Amex 378135646755254' >> $LOGFILE echo 'Testing Amex 342829672720827' >> $LOGFILE echo 'Testing Amex 340210881809804' >> $LOGFILE echo 'Testing Amex 345683203743734' >> $LOGFILE

Postal Codes

echo 'Testing Postal Code 44240' >> $LOGFILE echo 'Testing Postal Code 44240-5555' >> $LOGFILE echo 'Testing Postal Code G3H 6A3' >> $LOGFILE

Filenames (key=value) up to period

echo 'this filename filename=abc-adf-asdf.txt should be redacted' >> $LOGFILE echo 'this filename filename=My File Name.file should be redacted' >> $LOGFILE

ExampleConfig(,)

echo 'Please redact this ExampleConfig(Rhino, Disappointed) or else!' >> $LOGFILE echo 'Please redact this ExampleConfig(99sdajj--a+++,234kjbi$$657rfu) or else!' >> $LOGFILE

OnExample

echo 'I am checking to see if on Example ABC123 was redacted.' >> $LOGFILE echo 'I am checking to see if on ExampleNOSPACES was redacted.' >> $LOGFILE

Simple IP Address

echo 'Can you ping 10.0.0.45?' >> $LOGFILE echo 'Can you ping 66.234.23.96?' >> $LOGFILE

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment