Skip to content

Instantly share code, notes, and snippets.

@m18h

m18h/elasticsearch-bible.md

Last active September 26, 2021 15:50
Download ZIP

Revisions

  1. m18h revised this gist Sep 26, 2021. 1 changed file with 22 additions and 0 deletions.
    22 changes: 22 additions & 0 deletions elasticsearch-bible.md
    View file
    Original file line number Diff line number Diff line change
    @@ -182,3 +182,25 @@ curl -X PUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -H "Co
    curl -X PUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -H "Content-Type: application/json" \
    -d '{"indices.query.bool.max_clause_count" : "100000"}'
    ```


    ## Security

    ### Create API Key
    ```bash
    POST /_security/api_key
    {
    "name": "<key-name>",
    "role_descriptors": {
    "<role-name>": {
    "cluster": ["all"],
    "index": [
    {
    "names": ["<index-name>"],
    "privileges": ["read"]
    }
    ]
    }
    }
    }
    ```
  2. m18h revised this gist Feb 4, 2021. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion elasticsearch-bible.md
    View file
    Original file line number Diff line number Diff line change
    @@ -180,5 +180,5 @@ curl -X PUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -H "Co

    ```bash
    curl -X PUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -H "Content-Type: application/json" \
    -d '{"index.query.bool.max_clause_count" : "100000"}'
    -d '{"indices.query.bool.max_clause_count" : "100000"}'
    ```
  3. m18h revised this gist Feb 4, 2021. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion elasticsearch-bible.md
    View file
    Original file line number Diff line number Diff line change
    @@ -179,6 +179,6 @@ curl -X PUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -H "Co
    ### Set max clause count

    ```bash
    curl -X PUT 'http://localhost:9200/_all/_settings?preserve_existing=true'-H "Content-Type: application/json" \
    curl -X PUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -H "Content-Type: application/json" \
    -d '{"index.query.bool.max_clause_count" : "100000"}'
    ```
  4. m18h revised this gist Feb 4, 2021. 1 changed file with 4 additions and 6 deletions.
    10 changes: 4 additions & 6 deletions elasticsearch-bible.md
    View file
    Original file line number Diff line number Diff line change
    @@ -172,15 +172,13 @@ curl -X POST http://localhost:9200/logs/_update_by_query?wait_for_completion=tru
    ### Set results window

    ```bash
    curl -X PUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{
    "index.max_result_window" : "100000"
    }'
    curl -X PUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -H "Content-Type: application/json" \
    -d '{"index.max_result_window" : "100000"}'
    ```

    ### Set max clause count

    ```bash
    curl -X PUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{
    "index.query.bool.max_clause_count" : "100000"
    }'
    curl -X PUT 'http://localhost:9200/_all/_settings?preserve_existing=true'-H "Content-Type: application/json" \
    -d '{"index.query.bool.max_clause_count" : "100000"}'
    ```
  5. m18h revised this gist Feb 4, 2021. 1 changed file with 18 additions and 0 deletions.
    18 changes: 18 additions & 0 deletions elasticsearch-bible.md
    View file
    Original file line number Diff line number Diff line change
    @@ -166,3 +166,21 @@ curl -X POST http://localhost:9200/logs/_update_by_query?wait_for_completion=tru
    curl -X POST http://localhost:9200/logs/_update_by_query?wait_for_completion=true -H "Content-Type: application/json" \
    -d {"script":{"source":"ctx._source.remove('new_field')"}}
    ```

    ## Update Indice Settings

    ### Set results window

    ```bash
    curl -X PUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{
    "index.max_result_window" : "100000"
    }'
    ```

    ### Set max clause count

    ```bash
    curl -X PUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{
    "index.query.bool.max_clause_count" : "100000"
    }'
    ```
  6. m18h revised this gist Jan 28, 2021. 1 changed file with 3 additions and 0 deletions.
    3 changes: 3 additions & 0 deletions elasticsearch-bible.md
    View file
    Original file line number Diff line number Diff line change
    @@ -139,6 +139,9 @@ curl -X DELETE http://localhost:9200/logs
    curl -X POST http://localhost:9200/logs-temp/_clone/logs

    curl -X DELETE http://localhost:9200/logs-temp

    curl -X PUT http://localhost:9200/logs/_settings -H "Content-Type: application/json" \
    -d '{"settings":{"index.blocks.write":false}}'
    ```

    ## Update By Query
  7. m18h revised this gist Jan 28, 2021. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions elasticsearch-bible.md
    View file
    Original file line number Diff line number Diff line change
    @@ -131,11 +131,11 @@ DELETE /audit-logs
    curl -X POST http://localhost:9200/_reindex?wait_for_completion=true -H "Content-Type: application/json" \
    -d '{"source":{"index":"logs"},"dest":{"index":"logs-temp"}}'

    curl -X DELETE http://localhost:9200/logs

    curl -X PUT http://localhost:9200/logs-temp/_settings -H "Content-Type: application/json" \
    -d '{"settings":{"index.blocks.write":true}}'

    curl -X DELETE http://localhost:9200/logs

    curl -X POST http://localhost:9200/logs-temp/_clone/logs

    curl -X DELETE http://localhost:9200/logs-temp
  8. m18h revised this gist Jan 20, 2021. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion elasticsearch-bible.md
    View file
    Original file line number Diff line number Diff line change
    @@ -136,7 +136,7 @@ curl -X DELETE http://localhost:9200/logs
    curl -X PUT http://localhost:9200/logs-temp/_settings -H "Content-Type: application/json" \
    -d '{"settings":{"index.blocks.write":true}}'

    curl -X POST http://localhost:9200/logs-temp/_clone/addresses-staging
    curl -X POST http://localhost:9200/logs-temp/_clone/logs

    curl -X DELETE http://localhost:9200/logs-temp
    ```
  9. m18h revised this gist Jan 15, 2021. 1 changed file with 16 additions and 0 deletions.
    16 changes: 16 additions & 0 deletions elasticsearch-bible.md
    View file
    Original file line number Diff line number Diff line change
    @@ -143,7 +143,23 @@ curl -X DELETE http://localhost:9200/logs-temp

    ## Update By Query

    ### Updating multiple fields (including objects)

    ```bash
    curl -X POST http://localhost:9200/logs/_update_by_query?wait_for_completion=true -H "Content-Type: application/json" \
    -d '{"script":{"source":"ctx._source.result=true;ctx._source.created_boy=params.user;","lang":"painless","params":{"user":{"id":"1","name":"User 1"}}}}'
    ```

    ### Add field

    ```bash
    curl -X POST http://localhost:9200/logs/_update_by_query?wait_for_completion=true -H "Content-Type: application/json" \
    -d {"script":{"source":"ctx._source.new_field='value_of_new_field'"}}
    ```

    ### Remove field

    ```bash
    curl -X POST http://localhost:9200/logs/_update_by_query?wait_for_completion=true -H "Content-Type: application/json" \
    -d {"script":{"source":"ctx._source.remove('new_field')"}}
    ```
  10. m18h revised this gist Jan 15, 2021. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions elasticsearch-bible.md
    View file
    Original file line number Diff line number Diff line change
    @@ -129,12 +129,12 @@ DELETE /audit-logs
    ```bash
    curl -X POST http://localhost:9200/_reindex?wait_for_completion=true -H "Content-Type: application/json" \
    -d '{"source": {"index": "logs"}, "dest": {"index": "logs-temp"}}'
    -d '{"source":{"index":"logs"},"dest":{"index":"logs-temp"}}'

    curl -X DELETE http://localhost:9200/logs

    curl -X PUT http://localhost:9200/logs-temp/_settings -H "Content-Type: application/json" \
    -d '{"settings": {"index.blocks.write": true}}'
    -d '{"settings":{"index.blocks.write":true}}'

    curl -X POST http://localhost:9200/logs-temp/_clone/addresses-staging

  11. m18h revised this gist Jan 15, 2021. 1 changed file with 7 additions and 0 deletions.
    7 changes: 7 additions & 0 deletions elasticsearch-bible.md
    View file
    Original file line number Diff line number Diff line change
    @@ -140,3 +140,10 @@ curl -X POST http://localhost:9200/logs-temp/_clone/addresses-staging

    curl -X DELETE http://localhost:9200/logs-temp
    ```

    ## Update By Query

    ```bash
    curl -X POST http://localhost:9200/logs/_update_by_query?wait_for_completion=true -H "Content-Type: application/json" \
    -d '{"script":{"source":"ctx._source.result=true;ctx._source.created_boy=params.user;","lang":"painless","params":{"user":{"id":"1","name":"User 1"}}}}'
    ```
  12. m18h revised this gist Jan 10, 2021. 1 changed file with 18 additions and 0 deletions.
    18 changes: 18 additions & 0 deletions elasticsearch-bible.md
    View file
    Original file line number Diff line number Diff line change
    @@ -122,3 +122,21 @@ PUT /logs/_settings
    ```bash
    DELETE /audit-logs
    ```

    ## Update Mappings

    > First create temp index
    ```bash
    curl -X POST http://localhost:9200/_reindex?wait_for_completion=true -H "Content-Type: application/json" \
    -d '{"source": {"index": "logs"}, "dest": {"index": "logs-temp"}}'

    curl -X DELETE http://localhost:9200/logs

    curl -X PUT http://localhost:9200/logs-temp/_settings -H "Content-Type: application/json" \
    -d '{"settings": {"index.blocks.write": true}}'

    curl -X POST http://localhost:9200/logs-temp/_clone/addresses-staging

    curl -X DELETE http://localhost:9200/logs-temp
    ```
  13. m18h revised this gist Jan 7, 2021. 1 changed file with 21 additions and 14 deletions.
    35 changes: 21 additions & 14 deletions elasticsearch-bible.md
    View file
    Original file line number Diff line number Diff line change
    @@ -61,6 +61,27 @@ curl -XPUT "localhost:9200/_snapshot/logs/$SNAPSHOT?wait_for_completion=true"
    GET /_snapshot/logs/_all?pretty
    ```

    ### 2. Close index

    ```bash
    POST /logs/_close
    ```

    ### 3. Restore index

    ```bash
    POST /_snapshot/<repo_name>/<snapshot_name>/_restore?wait_for_completion=true
    {
    "indices": "logs"
    }
    ```

    ### 4. Open index

    ```bash
    POST /logs/_open
    ```

    ## Clone index

    ### 1. Disable writes on old index
    @@ -99,19 +120,5 @@ PUT /logs/_settings
    ## Delete Index

    ```bash
    GET /_snapshot/_all?pretty

    PUT /_snapshot/audit-logs
    {
    "type": "fs",
    "settings": {
    "location": "/etc/elasticsearch/audit-logs",
    "compress": true
    }
    }

    GET /_snapshot/audit-logs/_all?pretty


    DELETE /audit-logs
    ```
  14. m18h revised this gist Jan 7, 2021. 1 changed file with 83 additions and 10 deletions.
    93 changes: 83 additions & 10 deletions elasticsearch-bible.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,44 +1,117 @@
    # Elasticsearch Bible

    ## Backup & Restore
    > For ease of use, use Kibana for running API requests
    ## Stuff
    ## Backup

    ### 1. Create snapshot dir

    ```bash
    GET /_snapshot/_all?pretty
    mkdir -p /snapshots
    ```

    PUT /_snapshot/audit-logs
    ### 2. Set owner of snapshot dir

    ```bash
    chown -R elasticsearch. /snapshots
    ```

    ### 3. Set snapshot dir in config

    ```bash
    cat >> /etc/elasticsearch/elasticsearch.yml << EOF
    path.repo: ["/snapshots"]
    EOF
    ```

    ### 4. Restart elasticsearch service

    ```bash
    systemctl restart elasticsearch
    ```

    ### 5. Set up snapshot repo

    ```bash
    PUT /_snapshot/logs
    {
    "type": "fs",
    "settings": {
    "location": "/etc/elasticsearch/audit-logs",
    "location": "/snapshots",
    "compress": true
    }
    }
    ```

    GET /_snapshot/audit-logs/_all?pretty
    ### 4. Create script to backup

    ```bash
    #!/bin/bash
    SNAPSHOT=`date +%Y%m%d-%H%M%S`
    curl -XPUT "localhost:9200/_snapshot/logs/$SNAPSHOT?wait_for_completion=true"
    ```

    ### 5. Execute backup script

    DELETE /audit-logs
    ## Restore

    ### 1. View all snapshots

    ```bash
    GET /_snapshot/logs/_all?pretty
    ```

    PUT /audit-logs/_settings
    ## Clone index

    ### 1. Disable writes on old index

    ```bash
    PUT /logs/_settings
    {
    "settings": {
    "index.blocks.write": true
    }
    }
    ```

    POST /audit-logs/_clone/audit-logs-new
    ### 2. Clone index

    ```bash
    POST /logs/_clone/logs-new
    {
    "settings": {
    "index.number_of_shards": 5
    }
    }
    ```

    PUT /audit-logs/_settings
    ### 3. Re-enable writes on old index

    ```bash
    PUT /logs/_settings
    {
    "settings": {
    "index.blocks.write": false
    }
    }
    ```

    ## Delete Index

    ```bash
    GET /_snapshot/_all?pretty

    PUT /_snapshot/audit-logs
    {
    "type": "fs",
    "settings": {
    "location": "/etc/elasticsearch/audit-logs",
    "compress": true
    }
    }

    GET /_snapshot/audit-logs/_all?pretty


    DELETE /audit-logs
    ```
  15. m18h revised this gist Jan 5, 2021. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion elasticsearch-bible.md
    View file
    Original file line number Diff line number Diff line change
    @@ -4,7 +4,7 @@

    ## Stuff

    ```
    ```bash
    GET /_snapshot/_all?pretty

    PUT /_snapshot/audit-logs
  16. m18h revised this gist Jan 5, 2021. 1 changed file with 6 additions and 35 deletions.
    41 changes: 6 additions & 35 deletions elasticsearch-bible.md
    View file
    Original file line number Diff line number Diff line change
    @@ -7,64 +7,35 @@
    ```
    GET /_snapshot/_all?pretty
    PUT /_snapshot/addresses-prod
    {
    "type": "fs",
    "settings": {
    "location": "/etc/elasticsearch/oratio/prod",
    "compress": true
    }
    }
    PUT /_snapshot/addresses-staging
    {
    "type": "fs",
    "settings": {
    "location": "/etc/elasticsearch/oratio/staging",
    "compress": true
    }
    }
    PUT /_snapshot/street-picks
    {
    "type": "fs",
    "settings": {
    "location": "/etc/elasticsearch/oratio/street-picks",
    "compress": true
    }
    }
    PUT /_snapshot/audit-logs
    {
    "type": "fs",
    "settings": {
    "location": "/etc/elasticsearch/oratio/audit-logs",
    "location": "/etc/elasticsearch/audit-logs",
    "compress": true
    }
    }
    GET /_snapshot/addresses-staging/_all?pretty
    GET /_snapshot/addresses-prod/_all?pretty
    GET /_snapshot/audit-logs/_all?pretty
    DELETE /addresses-prod
    DELETE /audit-logs
    PUT /addresses-staging/_settings
    PUT /audit-logs/_settings
    {
    "settings": {
    "index.blocks.write": true
    }
    }
    POST /addresses-staging/_clone/addresses-prod
    POST /audit-logs/_clone/audit-logs-new
    {
    "settings": {
    "index.number_of_shards": 5
    }
    }
    PUT /addresses-staging/_settings
    PUT /audit-logs/_settings
    {
    "settings": {
    "index.blocks.write": false
  17. m18h created this gist Jan 5, 2021.
    73 changes: 73 additions & 0 deletions elasticsearch-bible.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,73 @@
    # Elasticsearch Bible

    ## Backup & Restore

    ## Stuff

    ```
    GET /_snapshot/_all?pretty
    PUT /_snapshot/addresses-prod
    {
    "type": "fs",
    "settings": {
    "location": "/etc/elasticsearch/oratio/prod",
    "compress": true
    }
    }
    PUT /_snapshot/addresses-staging
    {
    "type": "fs",
    "settings": {
    "location": "/etc/elasticsearch/oratio/staging",
    "compress": true
    }
    }
    PUT /_snapshot/street-picks
    {
    "type": "fs",
    "settings": {
    "location": "/etc/elasticsearch/oratio/street-picks",
    "compress": true
    }
    }
    PUT /_snapshot/audit-logs
    {
    "type": "fs",
    "settings": {
    "location": "/etc/elasticsearch/oratio/audit-logs",
    "compress": true
    }
    }
    GET /_snapshot/addresses-staging/_all?pretty
    GET /_snapshot/addresses-prod/_all?pretty
    DELETE /addresses-prod
    PUT /addresses-staging/_settings
    {
    "settings": {
    "index.blocks.write": true
    }
    }
    POST /addresses-staging/_clone/addresses-prod
    {
    "settings": {
    "index.number_of_shards": 5
    }
    }
    PUT /addresses-staging/_settings
    {
    "settings": {
    "index.blocks.write": false
    }
    }
    ```