Skip to content

Instantly share code, notes, and snippets.

@rafaltrojanowski

rafaltrojanowski/gist:a669f23721d0c4a2648220ff458be17a

Created January 10, 2022 13:00
Show Gist options
  • Save rafaltrojanowski/a669f23721d0c4a2648220ff458be17a to your computer and use it in GitHub Desktop.
Save rafaltrojanowski/a669f23721d0c4a2648220ff458be17a to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
Local fix of type:
index = "answers"
{:ok, response} = Domain.Env.get(:elasticsearch_url) |> URI.merge("/#{index}/_mapping?pretty") |> URI.to_string() |> Elastix.HTTP.get()
body = response.body
Map.keys(body["answers"]["mappings"]["properties"]["label_ids_with_keywords"]["properties"]) |> Enum.count
@rafaltrojanowski
Copy link
Author

IEx.configure(inspect: [limit: :infinity])

@rafaltrojanowski
Copy link
Author 

Map.keys(body["answers"]["mappings"]["properties"]["label_ids_with_keywords"]["properties"])
["id", "8", "36", "107", "44", "100", "78", "71", "142", "120", "98", "64", "3",
 "163", "93", "204", "148", "121", "155", "149", "23", "99", "225", "127", "86",
 "one", "92", "50", "106", "119", "79", "65", "141", "keywords", "72", "29",
 "96", "189", "30", "113", "134", "152", "135", "22", "162", "128", "37", "51",
 "16", "114", "9", "85", "174", "156"]

@rafaltrojanowski
Copy link
Author 

1. Create a new index with the correct mapping info where the field type is changed to your desired type.
2. Re-index the data from your old index to the new index.
3. Remove the old index.

@rafaltrojanowski
Copy link
Author

Logger.configure(level: :info)

@rafaltrojanowski
Copy link
Author

rafaltrojanowski commented Jan 10, 2022 •
edited
Loading

Update mapping:

curl -X PUT "localhost:9200/answers_v1/_mapping?pretty" -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "properties": {
      "id": {
        "type": "integer"
      },
      "score": {
        "type": "byte",
        "null_value": 0
      },
      "stripped_comment": {
        "type": "text",
        "analyzer": "standard"
      },
      "comment": {
        "type": "text",
        "analyzer": "standard",
        "search_analyzer": "standard",
        "term_vector": "with_positions_offsets"
      },
      "comment_ar_AE": {
        "analyzer": "arabic",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_bg_BG": {
        "analyzer": "bulgarian",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_bn_BD": {
        "analyzer": "bengali",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_ca_ES": {
        "analyzer": "catalan",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_cs_CZ": {
        "analyzer": "czech",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_da_DK": {
        "analyzer": "danish",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_de_DE": {
        "analyzer": "german",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_el_GR": {
        "analyzer": "greek",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_en_US": {
        "analyzer": "english",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_es_AR": {
        "analyzer": "spanish",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_es_ES": {
        "analyzer": "spanish",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_eu_ES": {
        "analyzer": "basque",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_fr_FR": {
        "analyzer": "french",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_ga_IE": {
        "analyzer": "irish",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_gl_ES": {
        "analyzer": "galician",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_hi_IN": {
        "analyzer": "hindi",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_hr_HR": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_hu_HU": {
        "analyzer": "hungarian",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_hy_AM": {
        "analyzer": "armenian",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_id_ID": {
        "analyzer": "indonesian",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_it_IT": {
        "analyzer": "italian",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_ja_JP": {
        "analyzer": "kuromoji",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_ko_KR": {
        "analyzer": "nori",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_lt_LT": {
        "analyzer": "lithuanian",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_lv_LV": {
        "analyzer": "latvian",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_ms_MY": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_nl_NL": {
        "analyzer": "dutch",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_pl_PL": {
        "analyzer": "polish",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_pt_BR": {
        "analyzer": "brazilian",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_pt_PT": {
        "analyzer": "portuguese",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_ro_RO": {
        "analyzer": "romanian",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_ru_RU": {
        "analyzer": "russian",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_sk_SK": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_sv_SE": {
        "analyzer": "swedish",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_th_TH": {
        "analyzer": "thai",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_tr_TR": {
        "analyzer": "turkish",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_uk_UA": {
        "analyzer": "ukrainian",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_vi_VN": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_zh_Hans": {
        "analyzer": "smartcn",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_af_ZA": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_sq_AL": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_am_ET": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_az_AZ": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_be_BY": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_bs_BA": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_ceb_PH": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_co_FR": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_eo_EO": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_et_EE": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_fy_NL": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_ka_GE": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_gu_IN": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_ht_HT": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_ha_GH": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_haw_US": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_he_IL": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_hmn_HMN": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_is_IS": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_ig_NG": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_jv_ID": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_kn_IN": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_kk_KZ": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_km_KH": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_ku_IQ": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_ky_KG": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_lo_LA": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_la_LA": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_lb_LU": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_mk_MK": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_ma_MG": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_mg_MG": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_ml_IN": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_mt_MT": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_mi_MI": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_mr_IN": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_mn_MN": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_ne_NP": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_ny_ZW": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_ps_AR": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_pa_IN": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_sm_SM": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_gd_GB": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_sr_SR": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_sn_ZW": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_si_LK": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_sd_IN": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_sl_SI": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_so_SO": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_su_ID": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_sw_KE": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_tl_PH": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_tg_TJ": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_ta_IN": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_te_IN": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_ur_PK": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_uz_AF": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_cy_GB": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_xh_ZA": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_yi": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_yo_NG": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "comment_zu_ZA": {
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "term_vector": "with_positions_offsets",
        "type": "text"
      },
      "translated_comment": {
        "type": "text",
        "analyzer": "standard"
      },
      "score_type": {
        "type": "keyword",
        "null_value": "detractor"
      },
      "has_comment": {
        "type": "boolean",
        "null_value": false
      },
      "public_hash_id": {
        "type": "keyword"
      },
      "properties": {
        "type": "nested",
        "properties": {
          "id": {
            "type": "integer"
          },
          "name": {
            "type": "keyword"
          },
          "value": {
            "type": "keyword"
          }
        }
      },
      "label_ids": {
        "type": "keyword"
      },
      "label_ids_with_keywords": {
        "type": "object"
      },
      "recipient_id": {
        "type": "integer"
      },
      "recipient_full_name": {
        "type": "text"
      },
      "recipient_identity": {
        "type": "text",
        "analyzer": "standard",
        "search_analyzer": "lowercase_analyzer",
        "fields": {
          "email": {
            "type": "text",
            "analyzer": "email_analyzer",
            "search_analyzer": "email_analyzer"
          }
        }
      },
      "recipient_identity_type": {
        "type": "keyword"
      },
      "recipient_upload_method": {
        "type": "keyword"
      },
      "sentiment": {
        "type": "keyword"
      },
      "sentiment_per_label": {
        "type": "nested",
        "properties": {
          "id": {
            "type": "integer"
          },
          "name": {
            "type": "keyword"
          },
          "sentiment": {
            "type": "keyword"
          }
        }
      },
      "survey_id": {
        "type": "integer"
      },
      "survey_recipient_id": {
        "type": "integer"
      },
      "language": {
        "type": "keyword"
      },
      "language_id": {
        "type": "integer"
      },
      "translated_language_id": {
        "type": "integer"
      },
      "inserted_at": {
        "type": "date"
      },
      "updated_at": {
        "type": "date"
      },
      "additional_answers": {
        "type": "nested",
        "properties": {
          "id": {
            "type": "text"
          },
          "additional_question_id": {
            "type": "integer"
          },
          "additional_question_public_hash_id": {
            "type": "keyword"
          },
          "answer": {
            "type": "keyword"
          },
          "inserted_at": {
            "type": "date"
          },
          "position": {
            "type": "integer"
          },
          "question": {
            "type": "text"
          },
          "score_system_id": {
            "type": "integer"
          },
          "has_answer": {
            "type": "boolean"
          }
        }
      }
    }
  }
}
'

@rafaltrojanowski
Copy link
Author

can use Elastic API to to most of the operations, but not sure if all are supported thought (I'd like to get all indexes to check those for a specific organizations) like this:

curl -X GET "localhost:9200/_cat/indices/*?v&s=index&pretty"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment