Skip to content

Instantly share code, notes, and snippets.

@rupeshtiwari

rupeshtiwari/ism example.md

Last active March 21, 2024 06:34
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save rupeshtiwari/31c667ef8569e175d95c31c246187523 to your computer and use it in GitHub Desktop.
Save rupeshtiwari/31c667ef8569e175d95c31c246187523 to your computer and use it in GitHub Desktop.
Download ZIP
ism example,opensearch ism policy,
Raw
ism example.md

Times will be longer because ISM takes some minutes between checking status and applying settings. It may take about 10 minutes to start.

Prerequisite

Create opensearch domain, make sure to enable dedicated master nodes and enable ultrawarm and cold storage

image

Step 1: Create Index Template and Datastream

PUT _index_template/email_conversations_data_template
{
  "index_patterns": ["email_conversations*"],
  "data_stream": {
  "timestamp_field": {
        "name": "date_time"
      }
  },
  "template": {
    "settings": {
      "number_of_shards": 1
    },
    "mappings": {
      "properties": {
        "message_id": {
          "type": "keyword"
        },
        "date_time": {
          "type": "date"
        },
        "from": {
          "type": "keyword"
        },
        "to": {
          "type": "keyword"
        },
        "cc": {
          "type": "keyword"
        },
        "bcc": {
          "type": "keyword"
        },
        "subject": {
          "type": "text"
        },
        "body": {
          "type": "text"
        },
        "attachments": {
          "type": "nested",
          "properties": {
            "file_name": {
              "type": "keyword"
            },
            "content_type": {
              "type": "keyword"
            }
          }
        },
        "labels": {
          "type": "keyword"
        },
        "thread_id": {
          "type": "keyword"
        },
        "status": {
          "type": "keyword"
        }
      }
    },
     "aliases": {
      "company_email_records": {}
    }
  },
  "priority": 200
}

image

output should be:

{
  "acknowledged": true
}

Index Management>Templates

image

Step 2: Create ISM Policy

Run below script to create ISM policy.

PUT _plugins/_ism/policies/rollover_then_transition
{
  "policy": {
    "description": "Roll over after 1 min, move to warm after 5 minutes, then to cold after 2 days",
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [
          {
            "rollover": {
              "min_index_age": "1m"
            }
          }
        ],
        "transitions": [
          {
            "state_name": "warm",
            "conditions": {
              "min_rollover_age": "5m"
            }
          }
        ]
      },
      {
        "name": "warm",
        "actions": [],
        "transitions": []
      },
      {
        "name": "cold",
        "actions": [],
        "transitions": []
      }
    ],
    "ism_template": {
      "index_patterns": ["email_conversations*"],
      "priority": 200
    }
  }
}

image

Step 3: Bulk Indexing

Run below script on dev tools to do below:

  • Create email_conversations index
  • Create email_conversations data streams.
  • Create email_conversations_data_template index template
  • Create company_email_records alias
  • Insert 11 email documents

📝 Alias, Index Template and Data Stream must have different unique names

After indexing in data stream 1 backed index (.ds-email_conversations-000001) will be created with documents and ISM is running in this to check if age is 1min or more then rollover this index.

POST _bulk
{ "create" : { "_index" : "email_conversations" } }
{ "message_id": "1", "date_time": "2023-11-14T08:00:00Z", "from": "alice@example.com", "to": ["bob@example.com"], "subject": "Project Update", "body": "The project is on track for the deadline.", "status": "unread" }
{ "create" : { "_index" : "email_conversations" } }
{ "message_id": "2", "date_time": "2023-11-14T08:10:00Z", "from": "carol@example.com", "to": ["dave@example.com"], "subject": "Budget Review", "body": "Please review the attached budget report.", "status": "unread" }
{ "create" : { "_index" : "email_conversations" } }
{ "message_id": "3", "date_time": "2023-11-14T08:20:00Z", "from": "eve@example.com", "to": ["frank@example.com"], "subject": "New Design Mockups", "body": "Attached are the new mockups for the design.", "status": "unread" }
{ "create" : { "_index" : "email_conversations" } }
{ "message_id": "4", "date_time": "2023-11-14T08:30:00Z", "from": "grace@example.com", "to": ["henry@example.com"], "subject": "Weekly Sync", "body": "Let's schedule our weekly sync to discuss the project's progress.", "status": "unread" }
{ "create" : { "_index" : "email_conversations" } }
{ "message_id": "5", "date_time": "2023-11-14T08:40:00Z", "from": "ida@example.com", "to": ["jack@example.com"], "subject": "Client Feedback", "body": "Here's the client feedback on the recent submission.", "status": "unread" }
{ "create" : { "_index" : "email_conversations" } }
{ "message_id": "6", "date_time": "2023-11-14T08:50:00Z", "from": "jill@example.com", "to": ["kevin@example.com"], "subject": "Conference Call", "body": "Reminder about the conference call later today.", "status": "unread" }

Confirm alias created

image

Confirm Data Stream created

image

Confirm 1 Backed Index inside datastream created immediately

image

Go to Index Management>Indices>Policy managed indices

The policy is now assigned and starts managing the indexes. On the Managed Indices page, you can observe the Job status as Running and Action info is empty.

image

The ISM policy takes some time to activate. If you click the policy name you will see the following message:

image

After few more mins you should see the action info is "Successfully Initialized policy:rollover_then_transition"

image

Click on action info link to see the message

image

Wait for 5 mins and you should see rollover happened and 2nd backed index created.

The rollover is finished, so now we have two indices:

image

image

Let's wait another 5 mins and notice rollover index .ds-email_conversations-000001 will be soon transitioning to warm

image

Check the Action info is saying transitionig to warm

image

Next it will evaluate transition condition for warm state to initiate warm transition for .ds-email_conversations-000001

image

image

Finally, .ds-email_conversations-000001 is transitioned to warm node

and replica shards are removed from this backed index

Cleanup

# 1. Delete Data Stream
DELETE _data_stream/email_conversations

# 2. Delete Index Template
DELETE _index_template/email_conversations_data_template

# 3. Delete indices
DELETE /email_conversations*

# 4. Delete ISM Policy
DELETE _plugins/_ism/policies/rollover_then_transition

Apendix

Index State Manager Job Interval 5 mins

GET /_cluster/settings?include_defaults=true&pretty

Check value of: opendistro.index_state_management.job_interval

image

Checking indices managed by ISM Policy and their state by index name email_conversations

GET /_plugins/_ism/explain/email_conversations

image

ISM policy states When an ISM policy is attached to an index, the index enters an "Initializing" state. From the "Initializing" state, the index moves into the "Default" state, which is defined in the policy. This "Initializing" operation, and every subsequent operation, can take 30 to 48 minutes. ISM uses this time to perform policy actions, and then checks for any conditions and transitions the index into different states. A random jitter of 0-60% is also added to make sure that there aren't surges of activity coming from all indices at the same time.

Note: For a rollover operation, an index is "complete" after the index rolls over, transitions into a "warm" state, and the replica count is updated.

If you're using an ISM policy and the index isn't properly migrating, then check the status of the ISM.

To check the status of the migration for particular index, use the following syntax:

GET _ultrawarm/migration/<put_index_name_here>/_status

To get a summary migration of all indices, use the following syntax:

GET _ultrawarm/migration/_status?

References

  1. Video: on ISM policy migrating data from hot to ultrawarm to cold: https://www.youtube.com/watch?v=b9n2kvNRI6E
  2. Blog: Automatic ISM for Amazon OpenSearch Service: https://aws.amazon.com/blogs/big-data/automating-index-state-management-for-amazon-opensearch-service/
  3. Document: Transitions in ISM Policy documentation: https://opensearch.org/docs/latest/im-plugin/ism/policies/#transitions
  4. ISM job interval 5 mins: https://opensearch.org/docs/latest/im-plugin/ism/settings/
  5. ISM Policy OpenSearch document: https://opensearch.org/docs/1.2/im-plugin/ism/api/#update-policy
  6. ISM repost: https://repost.aws/knowledge-center/opensearch-low-storage-ism
  7. ISM blog: https://opster.com/guides/opensearch/opensearch-data-architecture/setting-up-hot-warm-architecture-for-ism/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment