jeffvestal

from faker import Faker
import random
import datetime

# Create an instance of the Faker class
faker = Faker()

# Define the number of logs
num_logs = 1000000

jeffvestal

Product vector / knn search examples

sentence embedding model - all-mpnet-base-v2 Test dataset - Kaggle

Dataset loaded as is

I used the data uploader in kibana

GET fashion_clothing_products-ori/_search

jeffvestal

MOVED TO -> https://github.com/jeffvestal/pii_redaction

Main Components

• NER model for Location/Org/People identification
• Regex patterns in a script processor toidentify pii with standard structures.

Ingest Pipeline

PUT _ingest/pipeline/pii_script-redact

jeffvestal

Click Here for Example Jupyter Notebooks

Short Link to this gist - ela.st/operationalize-nlp

NER - Named Entity Recognition

NER models can be used two ways in elasticsearch:

1. On ingest, new documents can be run through an ingest pipeline, which sends one of the fields through the model, and the resulting entity fields are indexed along with the original document. This is useful later when you want to be able to quickly filter on an entity, or run aggregations, etc.
2. Strings can be submitted directly to a model using the _infer endpoint. The string will be processed by the model and the response message will include any identified entities. Same output as #1, but this is done adhoc, and the results are not stored.

jeffvestal

will update shortly

jeffvestal

jeffvestal

# Preview and create transform
# PUT _transform/churn
POST _transform/_preview
{
  "source": {
    "index": [
      "churn_demo-calls",
      "churn_demo-customers"
    ]
  },