Skip to content

Instantly share code, notes, and snippets.

@jinnabaalu
Last active June 6, 2023 13:40
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jinnabaalu/d630c37ef1f87cfcfa622c3a4e77d78c to your computer and use it in GitHub Desktop.
Save jinnabaalu/d630c37ef1f87cfcfa622c3a4e77d78c to your computer and use it in GitHub Desktop.
Jenkins supported time zones depend on the Java runtime

The Efficient Mappings in Elasticsearch Index

Implicit/Dynamic Mappings vs Explicit Mappings

Implicit/Dynamic Mapping: Indexing the document without defining the mappings, new fields will be added to the top level and nested. Helps to index data faster, the optimized way of indexing will help indexing perform better with the following.

  • Enable date_detection and set the dynamic_date_formats
  • Numeric Detection, "numeric_detection": true
  • Customer Analyzers need to be tested before going production using Analyzer API, standard analyzers are supported out of the box for natural languages.
  • Coerce, indexing fails if the dirty values are getting indexed. Example if you are trying to save a number with "10" it will be either saved as floating point or a string but not integer, to make it strict we use coerce at index level of field level
  • Ignore indexing a field, if the string length more than the required number of characters use ignore_above. Default is 250
  • Ignore indexing a field, if the document contains wrong data with "ignore_malformed": true

Dynamic mappings will give bad results when indexing a document unaware. This behavior can be disabled, both at the document and at the object level, by setting the dynamic parameter to false (to ignore new fields) or to strict (to throw an exception if an unknown field is encountered). Default is true.

Explicit Mappings: Defining the mappings before indexing the document, all new fields will be added using PUT MAPPING API

Text vs Keyword

Data Type, text is used to allow full text search like resume content, of email content. keyword is used to allow exact value match, like email, phone number.

Index Per Document

In prior version of the elasticsearch, every index can have multiple types per index, which leads to the ambiguity between same name in different mapping types are backed by the same Lucene field internally.

Indexing the documents of different mapping types, with data related to one types stored as a record other mapping types may be left empty, which leads to spread data thinly dispersed or scattered documents.

The better option always having index per document, will resolve the above. Single responsibility of the index is to maintain the single document type / mapping type per index.

Prevent Mapping Explosion - Limit Settings

Unlimited number of mapping fields leads to the mapping explosion, which results in Out of Memory exception. Update the index mappings to prevent the document which causes the mapping explosion and limiting the fields as bellow.

Limit Total Fields

Set the total number of fields with index.mapping.total_fields.limit - default is 1000

Define Max Field Depth

  • All fields at root, depth=1
  • If there is any object mapping to a filed, depth=2

Set the index.mapping.depth.limit value to set the limit, default value is 20

Define Max Nested Field Limit

Nested fields are used when an array of object need to be indexed or queried, best practice to define the number of nested mappings. Set the index.mapping.nested_fields.limit value, default is 50.

Define Max Nested Objects Limit

A single document can have maximum of of 10000 nested objects by default across all the nested objects. Set the nested index.mapping.nested_objects.limit to limit the nested objects.

Above all the limits will help us prevent the mapping explosion, we can limit or extend the limit as required, but observability is required if we are exceeding the default limits

Query-time Mappings

Boost : To boost the specific filed in the document we add the boost mapping to the field to give more weight

The supported time zones depend on the Java runtime Jenkins is running on. The list of supported time zone IDs on this instance is:

  • Africa/Abidjan
  • Africa/Accra
  • Africa/Addis_Ababa
  • Africa/Algiers
  • Africa/Asmara
  • Africa/Asmera
  • Africa/Bamako
  • Africa/Bangui
  • Africa/Banjul
  • Africa/Bissau
  • Africa/Blantyre
  • Africa/Brazzaville
  • Africa/Bujumbura
  • Africa/Cairo
  • Africa/Casablanca
  • Africa/Ceuta
  • Africa/Conakry
  • Africa/Dakar
  • Africa/Dar_es_Salaam
  • Africa/Djibouti
  • Africa/Douala
  • Africa/El_Aaiun
  • Africa/Freetown
  • Africa/Gaborone
  • Africa/Harare
  • Africa/Johannesburg
  • Africa/Juba
  • Africa/Kampala
  • Africa/Khartoum
  • Africa/Kigali
  • Africa/Kinshasa
  • Africa/Lagos
  • Africa/Libreville
  • Africa/Lome
  • Africa/Luanda
  • Africa/Lubumbashi
  • Africa/Lusaka
  • Africa/Malabo
  • Africa/Maputo
  • Africa/Maseru
  • Africa/Mbabane
  • Africa/Mogadishu
  • Africa/Monrovia
  • Africa/Nairobi
  • Africa/Ndjamena
  • Africa/Niamey
  • Africa/Nouakchott
  • Africa/Ouagadougou
  • Africa/Porto-Novo
  • Africa/Sao_Tome
  • Africa/Timbuktu
  • Africa/Tripoli
  • Africa/Tunis
  • Africa/Windhoek
  • America/Adak
  • America/Anchorage
  • America/Anguilla
  • America/Antigua
  • America/Araguaina
  • America/Argentina/Buenos_Aires
  • America/Argentina/Catamarca
  • America/Argentina/ComodRivadavia
  • America/Argentina/Cordoba
  • America/Argentina/Jujuy
  • America/Argentina/La_Rioja
  • America/Argentina/Mendoza
  • America/Argentina/Rio_Gallegos
  • America/Argentina/Salta
  • America/Argentina/San_Juan
  • America/Argentina/San_Luis
  • America/Argentina/Tucuman
  • America/Argentina/Ushuaia
  • America/Aruba
  • America/Asuncion
  • America/Atikokan
  • America/Atka
  • America/Bahia
  • America/Bahia_Banderas
  • America/Barbados
  • America/Belem
  • America/Belize
  • America/Blanc-Sablon
  • America/Boa_Vista
  • America/Bogota
  • America/Boise
  • America/Buenos_Aires
  • America/Cambridge_Bay
  • America/Campo_Grande
  • America/Cancun
  • America/Caracas
  • America/Catamarca
  • America/Cayenne
  • America/Cayman
  • America/Chicago
  • America/Chihuahua
  • America/Coral_Harbour
  • America/Cordoba
  • America/Costa_Rica
  • America/Creston
  • America/Cuiaba
  • America/Curacao
  • America/Danmarkshavn
  • America/Dawson
  • America/Dawson_Creek
  • America/Denver
  • America/Detroit
  • America/Dominica
  • America/Edmonton
  • America/Eirunepe
  • America/El_Salvador
  • America/Ensenada
  • America/Fort_Nelson
  • America/Fort_Wayne
  • America/Fortaleza
  • America/Glace_Bay
  • America/Godthab
  • America/Goose_Bay
  • America/Grand_Turk
  • America/Grenada
  • America/Guadeloupe
  • America/Guatemala
  • America/Guayaquil
  • America/Guyana
  • America/Halifax
  • America/Havana
  • America/Hermosillo
  • America/Indiana/Indianapolis
  • America/Indiana/Knox
  • America/Indiana/Marengo
  • America/Indiana/Petersburg
  • America/Indiana/Tell_City
  • America/Indiana/Vevay
  • America/Indiana/Vincennes
  • America/Indiana/Winamac
  • America/Indianapolis
  • America/Inuvik
  • America/Iqaluit
  • America/Jamaica
  • America/Jujuy
  • America/Juneau
  • America/Kentucky/Louisville
  • America/Kentucky/Monticello
  • America/Knox_IN
  • America/Kralendijk
  • America/La_Paz
  • America/Lima
  • America/Los_Angeles
  • America/Louisville
  • America/Lower_Princes
  • America/Maceio
  • America/Managua
  • America/Manaus
  • America/Marigot
  • America/Martinique
  • America/Matamoros
  • America/Mazatlan
  • America/Mendoza
  • America/Menominee
  • America/Merida
  • America/Metlakatla
  • America/Mexico_City
  • America/Miquelon
  • America/Moncton
  • America/Monterrey
  • America/Montevideo
  • America/Montreal
  • America/Montserrat
  • America/Nassau
  • America/New_York
  • America/Nipigon
  • America/Nome
  • America/Noronha
  • America/North_Dakota/Beulah
  • America/North_Dakota/Center
  • America/North_Dakota/New_Salem
  • America/Ojinaga
  • America/Panama
  • America/Pangnirtung
  • America/Paramaribo
  • America/Phoenix
  • America/Port-au-Prince
  • America/Port_of_Spain
  • America/Porto_Acre
  • America/Porto_Velho
  • America/Puerto_Rico
  • America/Punta_Arenas
  • America/Rainy_River
  • America/Rankin_Inlet
  • America/Recife
  • America/Regina
  • America/Resolute
  • America/Rio_Branco
  • America/Rosario
  • America/Santa_Isabel
  • America/Santarem
  • America/Santiago
  • America/Santo_Domingo
  • America/Sao_Paulo
  • America/Scoresbysund
  • America/Shiprock
  • America/Sitka
  • America/St_Barthelemy
  • America/St_Johns
  • America/St_Kitts
  • America/St_Lucia
  • America/St_Thomas
  • America/St_Vincent
  • America/Swift_Current
  • America/Tegucigalpa
  • America/Thule
  • America/Thunder_Bay
  • America/Tijuana
  • America/Toronto
  • America/Tortola
  • America/Vancouver
  • America/Virgin
  • America/Whitehorse
  • America/Winnipeg
  • America/Yakutat
  • America/Yellowknife
  • Antarctica/Casey
  • Antarctica/Davis
  • Antarctica/DumontDUrville
  • Antarctica/Macquarie
  • Antarctica/Mawson
  • Antarctica/McMurdo
  • Antarctica/Palmer
  • Antarctica/Rothera
  • Antarctica/South_Pole
  • Antarctica/Syowa
  • Antarctica/Troll
  • Antarctica/Vostok
  • Arctic/Longyearbyen
  • Asia/Aden
  • Asia/Almaty
  • Asia/Amman
  • Asia/Anadyr
  • Asia/Aqtau
  • Asia/Aqtobe
  • Asia/Ashgabat
  • Asia/Ashkhabad
  • Asia/Atyrau
  • Asia/Baghdad
  • Asia/Bahrain
  • Asia/Baku
  • Asia/Bangkok
  • Asia/Barnaul
  • Asia/Beirut
  • Asia/Bishkek
  • Asia/Brunei
  • Asia/Calcutta
  • Asia/Chita
  • Asia/Choibalsan
  • Asia/Chongqing
  • Asia/Chungking
  • Asia/Colombo
  • Asia/Dacca
  • Asia/Damascus
  • Asia/Dhaka
  • Asia/Dili
  • Asia/Dubai
  • Asia/Dushanbe
  • Asia/Famagusta
  • Asia/Gaza
  • Asia/Harbin
  • Asia/Hebron
  • Asia/Ho_Chi_Minh
  • Asia/Hong_Kong
  • Asia/Hovd
  • Asia/Irkutsk
  • Asia/Istanbul
  • Asia/Jakarta
  • Asia/Jayapura
  • Asia/Jerusalem
  • Asia/Kabul
  • Asia/Kamchatka
  • Asia/Karachi
  • Asia/Kashgar
  • Asia/Kathmandu
  • Asia/Katmandu
  • Asia/Khandyga
  • Asia/Kolkata
  • Asia/Krasnoyarsk
  • Asia/Kuala_Lumpur
  • Asia/Kuching
  • Asia/Kuwait
  • Asia/Macao
  • Asia/Macau
  • Asia/Magadan
  • Asia/Makassar
  • Asia/Manila
  • Asia/Muscat
  • Asia/Nicosia
  • Asia/Novokuznetsk
  • Asia/Novosibirsk
  • Asia/Omsk
  • Asia/Oral
  • Asia/Phnom_Penh
  • Asia/Pontianak
  • Asia/Pyongyang
  • Asia/Qatar
  • Asia/Qyzylorda
  • Asia/Rangoon
  • Asia/Riyadh
  • Asia/Saigon
  • Asia/Sakhalin
  • Asia/Samarkand
  • Asia/Seoul
  • Asia/Shanghai
  • Asia/Singapore
  • Asia/Srednekolymsk
  • Asia/Taipei
  • Asia/Tashkent
  • Asia/Tbilisi
  • Asia/Tehran
  • Asia/Tel_Aviv
  • Asia/Thimbu
  • Asia/Thimphu
  • Asia/Tokyo
  • Asia/Tomsk
  • Asia/Ujung_Pandang
  • Asia/Ulaanbaatar
  • Asia/Ulan_Bator
  • Asia/Urumqi
  • Asia/Ust-Nera
  • Asia/Vientiane
  • Asia/Vladivostok
  • Asia/Yakutsk
  • Asia/Yangon
  • Asia/Yekaterinburg
  • Asia/Yerevan
  • Atlantic/Azores
  • Atlantic/Bermuda
  • Atlantic/Canary
  • Atlantic/Cape_Verde
  • Atlantic/Faeroe
  • Atlantic/Faroe
  • Atlantic/Jan_Mayen
  • Atlantic/Madeira
  • Atlantic/Reykjavik
  • Atlantic/South_Georgia
  • Atlantic/St_Helena
  • Atlantic/Stanley
  • Australia/ACT
  • Australia/Adelaide
  • Australia/Brisbane
  • Australia/Broken_Hill
  • Australia/Canberra
  • Australia/Currie
  • Australia/Darwin
  • Australia/Eucla
  • Australia/Hobart
  • Australia/LHI
  • Australia/Lindeman
  • Australia/Lord_Howe
  • Australia/Melbourne
  • Australia/NSW
  • Australia/North
  • Australia/Perth
  • Australia/Queensland
  • Australia/South
  • Australia/Sydney
  • Australia/Tasmania
  • Australia/Victoria
  • Australia/West
  • Australia/Yancowinna
  • Brazil/Acre
  • Brazil/DeNoronha
  • Brazil/East
  • Brazil/West
  • CET
  • CST6CDT
  • Canada/Atlantic
  • Canada/Central
  • Canada/Eastern
  • Canada/Mountain
  • Canada/Newfoundland
  • Canada/Pacific
  • Canada/Saskatchewan
  • Canada/Yukon
  • Chile/Continental
  • Chile/EasterIsland
  • Cuba
  • EET
  • EST5EDT
  • Egypt
  • Eire
  • Etc/GMT
  • Etc/GMT+0
  • Etc/GMT+1
  • Etc/GMT+10
  • Etc/GMT+11
  • Etc/GMT+12
  • Etc/GMT+2
  • Etc/GMT+3
  • Etc/GMT+4
  • Etc/GMT+5
  • Etc/GMT+6
  • Etc/GMT+7
  • Etc/GMT+8
  • Etc/GMT+9
  • Etc/GMT-0
  • Etc/GMT-1
  • Etc/GMT-10
  • Etc/GMT-11
  • Etc/GMT-12
  • Etc/GMT-13
  • Etc/GMT-14
  • Etc/GMT-2
  • Etc/GMT-3
  • Etc/GMT-4
  • Etc/GMT-5
  • Etc/GMT-6
  • Etc/GMT-7
  • Etc/GMT-8
  • Etc/GMT-9
  • Etc/GMT0
  • Etc/Greenwich
  • Etc/UCT
  • Etc/UTC
  • Etc/Universal
  • Etc/Zulu
  • Europe/Amsterdam
  • Europe/Andorra
  • Europe/Astrakhan
  • Europe/Athens
  • Europe/Belfast
  • Europe/Belgrade
  • Europe/Berlin
  • Europe/Bratislava
  • Europe/Brussels
  • Europe/Bucharest
  • Europe/Budapest
  • Europe/Busingen
  • Europe/Chisinau
  • Europe/Copenhagen
  • Europe/Dublin
  • Europe/Gibraltar
  • Europe/Guernsey
  • Europe/Helsinki
  • Europe/Isle_of_Man
  • Europe/Istanbul
  • Europe/Jersey
  • Europe/Kaliningrad
  • Europe/Kiev
  • Europe/Kirov
  • Europe/Lisbon
  • Europe/Ljubljana
  • Europe/London
  • Europe/Luxembourg
  • Europe/Madrid
  • Europe/Malta
  • Europe/Mariehamn
  • Europe/Minsk
  • Europe/Monaco
  • Europe/Moscow
  • Europe/Nicosia
  • Europe/Oslo
  • Europe/Paris
  • Europe/Podgorica
  • Europe/Prague
  • Europe/Riga
  • Europe/Rome
  • Europe/Samara
  • Europe/San_Marino
  • Europe/Sarajevo
  • Europe/Saratov
  • Europe/Simferopol
  • Europe/Skopje
  • Europe/Sofia
  • Europe/Stockholm
  • Europe/Tallinn
  • Europe/Tirane
  • Europe/Tiraspol
  • Europe/Ulyanovsk
  • Europe/Uzhgorod
  • Europe/Vaduz
  • Europe/Vatican
  • Europe/Vienna
  • Europe/Vilnius
  • Europe/Volgograd
  • Europe/Warsaw
  • Europe/Zagreb
  • Europe/Zaporozhye
  • Europe/Zurich
  • GB
  • GB-Eire
  • GMT
  • GMT0
  • Greenwich
  • Hongkong
  • Iceland
  • Indian/Antananarivo
  • Indian/Chagos
  • Indian/Christmas
  • Indian/Cocos
  • Indian/Comoro
  • Indian/Kerguelen
  • Indian/Mahe
  • Indian/Maldives
  • Indian/Mauritius
  • Indian/Mayotte
  • Indian/Reunion
  • Iran
  • Israel
  • Jamaica
  • Japan
  • Kwajalein
  • Libya
  • MET
  • MST7MDT
  • Mexico/BajaNorte
  • Mexico/BajaSur
  • Mexico/General
  • NZ
  • NZ-CHAT
  • Navajo
  • PRC
  • PST8PDT
  • Pacific/Apia
  • Pacific/Auckland
  • Pacific/Bougainville
  • Pacific/Chatham
  • Pacific/Chuuk
  • Pacific/Easter
  • Pacific/Efate
  • Pacific/Enderbury
  • Pacific/Fakaofo
  • Pacific/Fiji
  • Pacific/Funafuti
  • Pacific/Galapagos
  • Pacific/Gambier
  • Pacific/Guadalcanal
  • Pacific/Guam
  • Pacific/Honolulu
  • Pacific/Johnston
  • Pacific/Kiritimati
  • Pacific/Kosrae
  • Pacific/Kwajalein
  • Pacific/Majuro
  • Pacific/Marquesas
  • Pacific/Midway
  • Pacific/Nauru
  • Pacific/Niue
  • Pacific/Norfolk
  • Pacific/Noumea
  • Pacific/Pago_Pago
  • Pacific/Palau
  • Pacific/Pitcairn
  • Pacific/Pohnpei
  • Pacific/Ponape
  • Pacific/Port_Moresby
  • Pacific/Rarotonga
  • Pacific/Saipan
  • Pacific/Samoa
  • Pacific/Tahiti
  • Pacific/Tarawa
  • Pacific/Tongatapu
  • Pacific/Truk
  • Pacific/Wake
  • Pacific/Wallis
  • Pacific/Yap
  • Poland
  • Portugal
  • ROK
  • Singapore
  • SystemV/AST4
  • SystemV/AST4ADT
  • SystemV/CST6
  • SystemV/CST6CDT
  • SystemV/EST5
  • SystemV/EST5EDT
  • SystemV/HST10
  • SystemV/MST7
  • SystemV/MST7MDT
  • SystemV/PST8
  • SystemV/PST8PDT
  • SystemV/YST9
  • SystemV/YST9YDT
  • Turkey
  • UCT
  • US/Alaska
  • US/Aleutian
  • US/Arizona
  • US/Central
  • US/East-Indiana
  • US/Eastern
  • US/Hawaii
  • US/Indiana-Starke
  • US/Michigan
  • US/Mountain
  • US/Pacific
  • US/Pacific-New
  • US/Samoa
  • UTC
  • Universal
  • W-SU
  • WET
  • Zulu
  • EST
  • HST
  • MST
  • ACT
  • AET
  • AGT
  • ART
  • AST
  • BET
  • BST
  • CAT
  • CNT
  • CST
  • CTT
  • EAT
  • ECT
  • IET
  • IST
  • JST
  • MIT
  • NET
  • NST
  • PLT
  • PNT
  • PRT
  • PST
  • SST
  • VST
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment