Created
August 12, 2023 01:11
-
-
Save maazghani/27b14e4b64f11415a1c0feb9bd32dd3d to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "cells": [ | |
| { | |
| "attachments": {}, | |
| "cell_type": "markdown", | |
| "id": "35b09b92-70b9-48a5-b164-c2ae540dddd2", | |
| "metadata": { | |
| "noteable": { | |
| "cell_type": "markdown" | |
| } | |
| }, | |
| "source": [ | |
| "# Elasticsearch Runbook\n", | |
| "This runbook provides a systematic guide to monitor and troubleshoot Elasticsearch clusters. It includes scripts and explanations for various checks to ensure the health and performance of the cluster. The runbook is designed to be executed directly from this notebook, assuming network access, authentication, and credentials have been pre-configured in the kernel runners." | |
| ] | |
| }, | |
| { | |
| "attachments": {}, | |
| "cell_type": "markdown", | |
| "id": "bbedc078-2d09-41f1-946f-68df16cff466", | |
| "metadata": { | |
| "noteable": { | |
| "cell_type": "markdown" | |
| } | |
| }, | |
| "source": [ | |
| "## Checking Cluster Health\n", | |
| "This section focuses on checking the overall health of the Elasticsearch cluster. It retrieves the cluster's settings, which can provide insights into its current configuration and any modifications made post-deployment. Regularly monitoring these settings can help in identifying any misconfigurations or changes that might impact the cluster's performance or stability." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "id": "4855645f-a016-48c4-af4d-ee8a7bc56200", | |
| "metadata": { | |
| "ExecuteTime": { | |
| "end_time": "2023-08-11T16:18:56.418648+00:00", | |
| "start_time": "2023-08-11T16:18:55.564203+00:00" | |
| }, | |
| "noteable": { | |
| "cell_type": "code" | |
| }, | |
| "vscode": { | |
| "languageId": "python" | |
| } | |
| }, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "['{\"persistent\":{},\"transient\":{},\"defaults\":{\"cluster\":{\"max_voting_config_exclusions\":\"10\",\"auto_shrink_voting_configuration\":\"true\",\"discovery_configuration_check\":{\"interval\":\"30000ms\"},\"election\":{\"duration\":\"500ms\",\"initial_timeout\":\"100ms\",\"max_timeout\":\"10s\",\"back_off_time\":\"100ms\",\"strategy\":\"supports_voting_only\"},\"no_master_block\":\"write\",\"persistent_tasks\":{\"allocation\":{\"enable\":\"all\",\"recheck_interval\":\"30s\"}},\"blocks\":{\"read_only_allow_delete\":\"false\",\"read_only\":\"false\"},\"remote\":{\"initial_connect_timeout\":\"30s\",\"node\":{\"attr\":\"\"},\"connections_per_cluster\":\"3\"},\"follower_lag\":{\"timeout\":\"90000ms\"},\"routing\":{\"use_adaptive_replica_selection\":\"true\",\"rebalance\":{\"enable\":\"all\"},\"allocation\":{\"enforce_default_tier_preference\":\"true\",\"node_concurrent_incoming_recoveries\":\"2\",\"node_initial_primaries_recoveries\":\"4\",\"desired_balance\":{\"progress_log_interval\":\"1m\",\"undesired_allocations\":{\"log_interval\":\"1h\",\"threshold\":\"0.1\"}},\"same_shard\":{\"host\":\"false\"},\"total_shards_per_node\":\"-1\",\"type\":\"desired_balance\",\"disk\":{\"threshold_enabled\":\"true\",\"reroute_interval\":\"60s\",\"watermark\":{\"flood_stage.frozen.max_headroom\":\"20GB\",\"flood_stage\":\"95%\",\"high\":\"90%\",\"low\":\"85%\",\"flood_stage.frozen\":\"95%\",\"flood_stage.max_headroom\":\"100GB\",\"low.max_headroom\":\"200GB\",\"enable_for_single_data_node\":\"true\",\"high.max_headroom\":\"150GB\"}},\"awareness\":{\"attributes\":[]},\"balance\":{\"disk_usage\":\"2.0E-11\",\"index\":\"0.55\",\"threshold\":\"1.0\",\"shard\":\"0.45\",\"write_load\":\"10.0\"},\"enable\":\"all\",\"node_concurrent_outgoing_recoveries\":\"2\",\"allow_rebalance\":\"indices_all_active\",\"cluster_concurrent_rebalance\":\"2\",\"node_concurrent_recoveries\":\"2\"}},\"indices\":{\"tombstones\":{\"size\":\"500\"},\"close\":{\"enable\":\"true\"}},\"join_validation\":{\"cache_timeout\":\"60s\"},\"max_shards_per_node.frozen\":\"3000\",\"nodes\":{\"reconnect_interval\":\"10s\"},\"service\":{\"master_service_starvation_logging_threshold\":\"5m\",\"slow_master_task_logging_threshold\":\"10s\",\"slow_task_logging_threshold\":\"30s\"},\"publish\":{\"timeout\":\"30000ms\",\"info_timeout\":\"10000ms\"},\"name\":\"codelibs-es\",\"fault_detection\":{\"leader_check\":{\"interval\":\"1000ms\",\"timeout\":\"10000ms\",\"retry_count\":\"3\"},\"follower_check\":{\"interval\":\"1000ms\",\"timeout\":\"10000ms\",\"retry_count\":\"3\"}},\"max_shards_per_node\":\"1000\",\"initial_master_nodes\":[\"es01\",\"es02\",\"es03\"],\"deprecation_indexing\":{\"enabled\":\"true\",\"x_opaque_id_used\":{\"enabled\":\"true\"}},\"snapshot\":{\"info\":{\"max_concurrent_fetches\":\"5\"}},\"info\":{\"update\":{\"interval\":\"30s\",\"timeout\":\"15s\"}}},\"stack\":{\"templates\":{\"enabled\":\"true\"}},\"time_series\":{\"poll_interval\":\"5m\"},\"readiness\":{\"port\":\"-1\"},\"logger\":{\"level\":\"INFO\"},\"bootstrap\":{\"memory_lock\":\"true\",\"ctrlhandler\":\"true\"},\"health_node\":{\"transport_action_timeout\":\"5s\"},\"ingest\":{\"user_agent\":{\"cache_size\":\"1000\"},\"geoip\":{\"cache_size\":\"1000\",\"downloader\":{\"endpoint\":\"https://geoip.elastic.co/v1/database\",\"poll\":{\"interval\":\"3d\"},\"eager\":{\"download\":\"false\"},\"enabled\":\"true\"}},\"grok\":{\"watchdog\":{\"max_execution_time\":\"1s\",\"interval\":\"1s\"}}},\"network\":{\"host\":[\"0.0.0.0\"],\"tcp\":{\"reuse_address\":\"true\",\"keep_count\":\"-1\",\"keep_interval\":\"-1\",\"no_delay\":\"true\",\"keep_alive\":\"true\",\"receive_buffer_size\":\"-1b\",\"keep_idle\":\"-1\",\"send_buffer_size\":\"-1b\"},\"bind_host\":[\"0.0.0.0\"],\"server\":\"true\",\"breaker\":{\"inflight_requests\":{\"limit\":\"100%\",\"overhead\":\"2.0\"}},\"publish_host\":[\"0.0.0.0\"]},\"searchable_snapshots\":{\"blob_cache\":{\"periodic_cleanup\":{\"interval\":\"1h\",\"batch_size\":\"100\",\"pit_keep_alive\":\"10m\",\"retention_period\":\"1h\"}}},\"path\":{\"data\":[],\"logs\":\"/usr/share/elasticsearch/logs\",\"shared_data\":\"\",\"home\":\"/usr/share/elasticsearch\",\"repo\":[]},\"search\":{\"default_search_timeout\":\"-1\",\"max_open_scroll_context\":\"500\",\"max_buckets\":\"65536\",\"max_async_search_response_size\":\"10mb\",\"keep_alive_interval\":\"1m\",\"max_keep_alive\":\"24h\",\"highlight\":{\"term_vector_multi_value\":\"true\"},\"default_allow_partial_results\":\"true\",\"low_level_cancellation\":\"true\",\"allow_expensive_queries\":\"true\",\"check_ccs_compatibility\":\"false\",\"default_keep_alive\":\"5m\",\"aggs\":{\"rewrite_to_filter_by_filter\":\"true\",\"tdigest_execution_hint\":\"DEFAULT\"}},\"security\":{\"manager\":{\"filter_bad_defaults\":\"true\"}},\"ccr\":{\"wait_for_metadata_timeout\":\"60s\",\"indices\":{\"recovery\":{\"recovery_activity_timeout\":\"60s\",\"chunk_size\":\"1mb\",\"internal_action_timeout\":\"60s\",\"max_bytes_per_sec\":\"40mb\",\"max_concurrent_file_chunks\":\"5\"}},\"auto_follow\":{\"wait_for_metadata_timeout\":\"60s\"}},\"repositories\":{\"fs\":{\"chunk_size\":\"9223372036854775807b\",\"location\":\"\"},\"url\":{\"supported_protocols\":[\"http\",\"https\",\"ftp\",\"file\",\"jar\"],\"allowed_urls\":[],\"url\":\"http:\"}},\"action\":{\"auto_create_index\":\"true\",\"search\":{\"pre_filter_shard_size\":{\"default\":\"128\"},\"shard_count\":{\"limit\":\"9223372036854775807\"}},\"destructive_requires_name\":\"true\"},\"client\":{\"type\":\"node\"},\"enrich\":{\"max_force_merge_attempts\":\"3\",\"cleanup_period\":\"15m\",\"fetch_size\":\"10000\",\"cache_size\":\"1000\",\"coordinator_proxy\":{\"max_concurrent_requests\":\"8\",\"max_lookups_per_request\":\"128\",\"queue_capacity\":\"1024\"},\"max_concurrent_policy_executions\":\"50\"},\"xpack\":{\"watcher\":{\"execution\":{\"scroll\":{\"size\":\"0\",\"timeout\":\"\"},\"default_throttle_period\":\"5s\"},\"internal\":{\"ops\":{\"bulk\":{\"default_timeout\":\"\"},\"index\":{\"default_timeout\":\"\"},\"search\":{\"default_timeout\":\"\"}}},\"thread_pool\":{\"queue_size\":\"1000\",\"size\":\"1\"},\"index\":{\"rest\":{\"direct_access\":\"\"}},\"use_ilm_index_management\":\"true\",\"trigger\":{\"schedule\":{\"ticker\":{\"tick_interval\":\"500ms\"}}},\"enabled\":\"true\",\"input\":{\"search\":{\"default_timeout\":\"\"}},\"encrypt_sensitive_data\":\"false\",\"transform\":{\"search\":{\"default_timeout\":\"\"}},\"stop\":{\"timeout\":\"30s\"},\"watch\":{\"scroll\":{\"size\":\"0\"}},\"bulk\":{\"concurrent_requests\":\"0\",\"flush_interval\":\"1s\",\"size\":\"1mb\",\"actions\":\"1\"},\"actions\":{\"bulk\":{\"default_timeout\":\"\"},\"index\":{\"default_timeout\":\"\"}}},\"eql\":{\"enabled\":\"true\"},\"ent_search\":{\"enabled\":\"true\"},\"monitoring\":{\"migration\":{\"decommission_alerts\":\"false\"},\"collection\":{\"cluster\":{\"stats\":{\"timeout\":\"10s\"}},\"node\":{\"stats\":{\"timeout\":\"10s\"}},\"indices\":[],\"ccr\":{\"stats\":{\"timeout\":\"10s\"}},\"enrich\":{\"stats\":{\"timeout\":\"10s\"}},\"index\":{\"stats\":{\"timeout\":\"10s\"},\"recovery\":{\"active_only\":\"false\",\"timeout\":\"10s\"}},\"interval\":\"10s\",\"enabled\":\"false\",\"ml\":{\"job\":{\"stats\":{\"timeout\":\"10s\"}}}},\"history\":{\"duration\":\"168h\"},\"elasticsearch\":{\"collection\":{\"enabled\":\"true\"}},\"templates\":{\"enabled\":\"true\"}},\"graph\":{\"enabled\":\"true\"},\"searchable\":{\"snapshot\":{\"allocate_on_rolling_restart\":\"false\",\"cache\":{\"range_size\":\"32mb\",\"sync\":{\"max_files\":\"10000\",\"interval\":\"60s\",\"shutdown_timeout\":\"10s\"},\"recovery_range_size\":\"128kb\"},\"shared_cache\":{\"recovery_range_size\":\"128kb\",\"region_size\":\"16mb\",\"size\":\"0\",\"min_time_delta\":\"60s\",\"decay\":{\"interval\":\"60s\"},\"size.max_headroom\":\"-1\",\"range_size\":\"16mb\",\"max_freq\":\"100\"}}},\"rollup\":{\"task_thread_pool\":{\"queue_size\":\"-1\",\"size\":\"1\"}},\"searchable_snapshots\":{\"cache_fetch_async_thread_pool\":{\"core\":\"0\",\"max\":\"48\",\"keep_alive\":\"30s\"},\"cache_prewarming_thread_pool\":{\"core\":\"0\",\"max\":\"16\",\"keep_alive\":\"30s\"}},\"downsample\":{\"thread_pool\":{\"queue_size\":\"256\",\"size\":\"2\"}},\"license\":{\"upload\":{\"types\":[\"standard\",\"gold\",\"platinum\",\"enterprise\",\"trial\"]},\"self_generated\":{\"type\":\"basic\"}},\"notification\":{\"pagerduty\":{\"default_account\":\"\"},\"webhook\":{\"additional_token_enabled\":\"false\"},\"email\":{\"account\":{\"domain_allowlist\":[\"*\"]},\"default_account\":\"\",\"html\":{\"sanitization\":{\"allow\":[\"body\",\"head\",\"_tables\",\"_links\",\"_blocks\",\"_formatting\",\"img:embedded\"],\"disallow\":[],\"enabled\":\"true\"}}},\"reporting\":{\"retries\":\"40\",\"warning\":{\"enabled\":\"true\"},\"interval\":\"15s\"},\"jira\":{\"default_account\":\"\"},\"slack\":{\"default_account\":\"\"}},\"security\":{\"operator_privileges\":{\"enabled\":\"false\"},\"dls_fls\":{\"enabled\":\"true\"},\"dls\":{\"bitset\":{\"cache\":{\"size\":\"10%\",\"ttl\":\"2h\"}}},\"transport\":{\"filter\":{\"allow\":[],\"deny\":[],\"enabled\":\"true\"},\"ssl\":{\"enabled\":\"false\"}},\"ssl\":{\"diagnose\":{\"trust\":\"true\"}},\"enabled\":\"false\",\"enrollment\":{\"enabled\":\"false\"},\"filter\":{\"always_allow_bound_address\":\"true\"},\"encryption\":{\"algorithm\":\"AES/CTR/NoPadding\"},\"audit\":{\"enabled\":\"false\",\"logfile\":{\"emit_cluster_name\":\"false\",\"emit_node_id\":\"true\",\"emit_node_name\":\"false\",\"emit_node_host_address\":\"false\",\"emit_cluster_uuid\":\"true\",\"emit_node_host_name\":\"false\",\"events\":{\"emit_request_body\":\"false\",\"include\":[\"ACCESS_DENIED\",\"ACCESS_GRANTED\",\"ANONYMOUS_ACCESS_DENIED\",\"AUTHENTICATION_FAILED\",\"CONNECTION_DENIED\",\"TAMPERED_REQUEST\",\"RUN_AS_DENIED\",\"RUN_AS_GRANTED\",\"SECURITY_CONFIG_CHANGE\"],\"exclude\":[]}}},\"authc\":{\"password_hashing\":{\"algorithm\":\"BCRYPT\"},\"success_cache\":{\"size\":\"10000\",\"enabled\":\"true\",\"expire_after_access\":\"1h\"},\"api_key\":{\"doc_cache\":{\"ttl\":\"5m\"},\"cache\":{\"hash_algo\":\"ssha256\",\"max_keys\":\"25000\",\"ttl\":\"24h\"},\"delete\":{\"interval\":\"24h\",\"retention_period\":\"7d\",\"timeout\":\"-1\"},\"enabled\":\"true\",\"hashing\":{\"algorithm\":\"PBKDF2\"}},\"anonymous\":{\"authz_exception\":\"true\",\"roles\":[],\"username\":\"_anonymous\"},\"run_as\":{\"enabled\":\"true\"},\"reserved_realm\":{\"enabled\":\"true\"},\"service_token\":{\"cache\":{\"hash_algo\":\"ssha256\",\"max_tokens\":\"100000\",\"ttl\":\"20m\"}},\"token\":{\"delete\":{\"interval\":\"30m\",\"timeout\":\"-1\"},\"enabled\":\"false\",\"timeout\":\"20m\"}},\"autoconfiguration\":{\"enabled\":\"true\"},\"fips_mode\":{\"enabled\":\"false\"},\"encryption_key\":{\"length\":\"128\",\"algorithm\":\"AES\"},\"http\":{\"filter\":{\"allow\":[],\"deny\":[],\"enabled\":\"true\"},\"ssl\":{\"enabled\":\"false\"}},\"automata\":{\"max_determinized_states\":\"100000\",\"cache\":{\"size\":\"10000\",\"ttl\":\"48h\",\"enabled\":\"true\"}},\"user\":null,\"authz\":{\"timer\":{\"indices\":{\"enabled\":\"false\",\"threshold\":{\"warn\":\"200ms\",\"debug\":\"20ms\",\"info\":\"100ms\"}}},\"store\":{\"privileges\":{\"cache\":{\"ttl\":\"24h\",\"max_size\":\"10000\"}},\"roles\":{\"has_privileges\":{\"cache\":{\"max_size\":\"1000\"}},\"cache\":{\"max_size\":\"10000\"},\"negative_lookup_cache\":{\"max_size\":\"10000\"},\"field_permissions\":{\"cache\":{\"max_size_in_bytes\":\"104857600\"}}}}}},\"transform\":{\"num_transform_failure_retries\":\"10\",\"transform_scheduler_frequency\":\"1s\"},\"ccr\":{\"enabled\":\"true\",\"ccr_thread_pool\":{\"queue_size\":\"100\",\"size\":\"32\"}},\"idp\":{\"privileges\":{\"application\":\"\",\"cache\":{\"size\":\"100\",\"ttl\":\"90m\"}},\"metadata\":{\"signing\":{\"keystore\":{\"alias\":\"\"}}},\"slo_endpoint\":{\"post\":\"https:\",\"redirect\":\"https:\"},\"defaults\":{\"nameid_format\":\"urn:oasis:names:tc:SAML:2.0:nameid-format:transient\",\"authn_expiry\":\"5m\"},\"allowed_nameid_formats\":[\"urn:oasis:names:tc:SAML:2.0:nameid-format:transient\"],\"contact\":{\"given_name\":\"\",\"email\":\"\",\"surname\":\"\"},\"organization\":{\"display_name\":\"\",\"name\":\"\",\"url\":\"http:\"},\"sso_endpoint\":{\"post\":\"https:\",\"redirect\":\"https:\"},\"entity_id\":\"\",\"signing\":{\"keystore\":{\"alias\":\"\"}},\"sp\":{\"cache\":{\"size\":\"1000\",\"ttl\":\"60m\"},\"wildcard\":{\"path\":\"wildcard_services.json\"}},\"enabled\":\"false\"},\"profiling\":{\"enabled\":\"true\",\"query\":{\"stacktrace\":{\"max_slices\":\"16\"},\"details\":{\"max_slices\":\"16\"},\"realtime\":\"true\"},\"templates\":{\"enabled\":\"false\"}},\"http\":{\"tcp\":{\"keep_alive\":\"true\"},\"default_connection_timeout\":\"10s\",\"proxy\":{\"host\":\"\",\"scheme\":\"\",\"port\":\"0\"},\"connection_pool_ttl\":\"-1\",\"max_response_size\":\"10mb\",\"whitelist\":[\"*\"],\"default_read_timeout\":\"10s\"},\"autoscaling\":{\"memory\":{\"monitor\":{\"timeout\":\"15s\"}}},\"applications\":{\"behavioral_analytics\":{\"ingest\":{\"bulk_processor\":{\"max_events_per_bulk\":\"500\",\"flush_delay\":\"10s\",\"max_bytes_in_flight\":\"5%\",\"max_number_of_retries\":\"1\"}}}},\"ml\":{\"utility_thread_pool\":{\"core\":\"1\",\"max\":\"2048\",\"keep_alive\":\"10m\"},\"max_anomaly_records\":\"500\",\"enable_config_migration\":\"true\",\"max_open_jobs\":\"512\",\"delayed_data_check_freq\":\"15m\",\"min_disk_space_off_heap\":\"5gb\",\"model_repository\":\"https://ml-models.elastic.co\",\"use_auto_machine_memory_percent\":\"false\",\"inference_model\":{\"cache_size\":\"40%\",\"time_to_live\":\"5m\"},\"nightly_maintenance_requests_per_second\":\"-1.0\",\"node_concurrent_job_allocations\":\"2\",\"max_model_memory_limit\":\"0b\",\"enabled\":\"true\",\"max_lazy_ml_nodes\":\"0\",\"max_ml_node_size\":\"0b\",\"max_machine_memory_percent\":\"30\",\"persist_results_max_retries\":\"20\",\"autodetect_process\":\"true\",\"datafeed_thread_pool\":{\"core\":\"1\",\"max\":\"512\",\"keep_alive\":\"1m\"},\"max_inference_processors\":\"50\",\"native_inference_comms_thread_pool\":{\"core\":\"3\",\"max\":\"348\",\"keep_alive\":\"1m\"},\"process_connect_timeout\":\"10s\",\"job_comms_thread_pool\":{\"core\":\"4\",\"max\":\"2048\",\"keep_alive\":\"1m\"}}},\"rest\":{\"action\":{\"multi\":{\"allow_explicit_index\":\"true\"}}},\"cache\":{\"recycler\":{\"page\":{\"limit\":{\"heap\":\"10%\"},\"type\":\"CONCURRENT\",\"weight\":{\"longs\":\"1.0\",\"ints\":\"1.0\",\"bytes\":\"1.0\",\"objects\":\"0.1\"}}}},\"tracing\":{\"apm\":{\"sanitize_field_names\":[\"password\",\"passwd\",\"pwd\",\"secret\",\"*key\",\"*token*\",\"*session*\",\"*credit*\",\"*card*\",\"*auth*\",\"*principal*\",\"set-cookie\"],\"enabled\":\"false\",\"names\":{\"include\":[],\"exclude\":[]}}},\"async_search\":{\"index_cleanup_interval\":\"1h\"},\"reindex\":{\"remote\":{\"whitelist\":[]}},\"resource\":{\"reload\":{\"enabled\":\"true\",\"interval\":{\"low\":\"60s\",\"high\":\"5s\",\"medium\":\"30s\"}}},\"thread_pool\":{\"force_merge\":{\"queue_size\":\"-1\",\"size\":\"2\"},\"search_coordination\":{\"queue_size\":\"1000\",\"size\":\"5\"},\"snapshot_meta\":{\"core\":\"1\",\"max\":\"48\",\"keep_alive\":\"30s\"},\"fetch_shard_started\":{\"core\":\"1\",\"max\":\"32\",\"keep_alive\":\"5m\"},\"estimated_time_interval.warn_threshold\":\"5s\",\"scheduler\":{\"warn_threshold\":\"5s\"},\"cluster_coordination\":{\"queue_size\":\"-1\",\"size\":\"1\"},\"search\":{\"queue_size\":\"1000\",\"size\":\"25\"},\"fetch_shard_store\":{\"core\":\"1\",\"max\":\"32\",\"keep_alive\":\"5m\"},\"flush\":{\"core\":\"1\",\"max\":\"5\",\"keep_alive\":\"5m\"},\"vectortile\":{\"queue_size\":\"-1\",\"size\":\"1\"},\"get\":{\"queue_size\":\"1000\",\"size\":\"25\"},\"system_read\":{\"queue_size\":\"2000\",\"size\":\"5\"},\"system_critical_read\":{\"queue_size\":\"2000\",\"size\":\"5\"},\"estimated_time_interval\":\"200ms\",\"write\":{\"queue_size\":\"10000\",\"size\":\"16\"},\"system_critical_write\":{\"queue_size\":\"1500\",\"size\":\"5\"},\"refresh\":{\"core\":\"1\",\"max\":\"8\",\"keep_alive\":\"5m\"},\"repository_azure\":{\"core\":\"0\",\"max\":\"5\",\"keep_alive\":\"30s\"},\"system_write\":{\"queue_size\":\"1000\",\"size\":\"5\"},\"generic\":{\"core\":\"4\",\"max\":\"128\",\"keep_alive\":\"30s\"},\"warmer\":{\"core\":\"1\",\"max\":\"5\",\"keep_alive\":\"5m\"},\"auto_complete\":{\"queue_size\":\"100\",\"size\":\"4\"},\"azure_event_loop\":{\"core\":\"0\",\"max\":\"1\",\"keep_alive\":\"30s\"},\"profiling\":{\"core\":\"0\",\"max\":\"1\",\"keep_alive\":\"30m\"},\"management\":{\"core\":\"1\",\"max\":\"5\",\"keep_alive\":\"5m\"},\"analyze\":{\"queue_size\":\"16\",\"size\":\"1\"},\"snapshot\":{\"core\":\"1\",\"max\":\"5\",\"keep_alive\":\"5m\"},\"search_throttled\":{\"queue_size\":\"100\",\"size\":\"1\"}},\"health\":{\"node\":{\"enabled\":\"true\"},\"master_history\":{\"has_master_lookup_timeframe\":\"30s\",\"identity_changes_threshold\":\"4\",\"no_master_transitions_threshold\":\"4\"},\"ilm\":{\"max_time_on_action\":\"1d\",\"max_time_on_step\":\"1d\",\"max_retries_per_step\":\"100\"},\"reporting\":{\"local\":{\"monitor\":{\"interval\":\"30s\"}}}},\"index\":{\"codec\":\"default\",\"recovery\":{\"type\":\"\"},\"store\":{\"type\":\"\",\"fs\":{\"fs_lock\":\"native\"},\"preload\":[],\"snapshot\":{\"uncached_chunk_size\":\"-1b\",\"cache\":{\"excluded_file_types\":[]}}}},\"monitor\":{\"jvm\":{\"gc\":{\"enabled\":\"true\",\"overhead\":{\"warn\":\"50\",\"debug\":\"10\",\"info\":\"25\"},\"refresh_interval\":\"1s\"},\"refresh_interval\":\"1s\"},\"process\":{\"refresh_interval\":\"1s\"},\"os\":{\"refresh_interval\":\"1s\"},\"fs\":{\"health\":{\"enabled\":\"true\",\"refresh_interval\":\"120s\",\"slow_path_logging_threshold\":\"5s\"},\"refresh_interval\":\"1s\"}},\"runtime_fields\":{\"grok\":{\"watchdog\":{\"max_execution_time\":\"1s\",\"interval\":\"1s\"}}},\"cluster_state\":{\"document_page_size\":\"1mb\"},\"transport\":{\"tcp\":{\"reuse_address\":\"true\",\"keep_count\":\"-1\",\"keep_interval\":\"-1\",\"no_delay\":\"true\",\"keep_alive\":\"true\",\"receive_buffer_size\":\"-1b\",\"keep_idle\":\"-1\",\"send_buffer_size\":\"-1b\"},\"bind_host\":[],\"connect_timeout\":\"30s\",\"compress\":\"INDEXING_DATA\",\"ping_schedule\":\"-1\",\"connections_per_node\":{\"recovery\":\"2\",\"state\":\"1\",\"bulk\":\"3\",\"reg\":\"6\",\"ping\":\"1\"},\"tracer\":{\"include\":[],\"exclude\":[\"internal:coordination/fault_detection/*\"]},\"type\":\"\",\"slow_operation_logging_threshold\":\"5s\",\"type.default\":\"netty4\",\"rst_on_close\":\"false\",\"port\":\"9300-9399\",\"compression_scheme\":\"LZ4\",\"host\":[],\"publish_port\":\"-1\",\"publish_host\":[],\"netty\":{\"receive_predictor_size\":\"64kb\",\"receive_predictor_max\":\"64kb\",\"worker_count\":\"16\",\"receive_predictor_min\":\"64kb\",\"boss_count\":\"1\"}},\"deprecation\":{\"skip_deprecated_settings\":[]},\"script\":{\"allowed_contexts\":[],\"max_compilations_rate\":\"150/5m\",\"cache\":{\"max_size\":\"3000\",\"expire\":\"0ms\"},\"painless\":{\"regex\":{\"enabled\":\"limited\",\"limit-factor\":\"6\"}},\"max_size_in_bytes\":\"65535\",\"allowed_types\":[],\"disable_max_compilations_rate\":\"false\"},\"indexing_pressure\":{\"memory\":{\"limit\":\"10%\"}},\"node\":{\"bandwidth\":{\"recovery\":{\"disk\":{\"write\":\"-1\",\"read\":\"-1\"},\"factor\":{\"write\":\"0.4\",\"read\":\"0.4\"},\"operator\":{\"factor.read\":\"0.4\",\"factor.write\":\"0.4\",\"factor\":\"0.4\",\"factor.max_overcommit\":\"100.0\"},\"network\":\"-1\"}},\"enable_lucene_segment_infos_trace\":\"false\",\"roles\":[\"master\"],\"_internal\":{\"default_refresh_interval\":\"1s\"},\"name\":\"es01\",\"external_id\":\"\",\"id\":{\"seed\":\"0\"},\"processors\":\"16.0\",\"store\":{\"allow_mmap\":\"true\"},\"attr\":{\"xpack\":{\"installed\":\"true\"}},\"portsfile\":\"false\"},\"indices\":{\"replication\":{\"retry_timeout\":\"60s\",\"initial_retry_backoff_bound\":\"50ms\"},\"cache\":{\"cleanup_interval\":\"1m\"},\"mapping\":{\"dynamic_timeout\":\"30s\",\"max_in_flight_updates\":\"10\"},\"memory\":{\"interval\":\"5s\",\"max_index_buffer_size\":\"-1\",\"shard_inactive_time\":\"5m\",\"index_buffer_size\":\"10%\",\"min_index_buffer_size\":\"48mb\"},\"breaker\":{\"request\":{\"limit\":\"60%\",\"type\":\"memory\",\"overhead\":\"1.0\"},\"total\":{\"limit\":\"95%\",\"use_real_memory\":\"true\"},\"fielddata\":{\"limit\":\"40%\",\"type\":\"memory\",\"overhead\":\"1.03\"},\"type\":\"hierarchy\"},\"write_ack_delay_interval\":\"0ms\",\"query\":{\"bool\":{\"max_nested_depth\":\"30\",\"max_clause_count\":\"4096\"},\"query_string\":{\"analyze_wildcard\":\"false\",\"allowLeadingWildcard\":\"true\"}},\"id_field_data\":{\"enabled\":\"false\"},\"recovery\":{\"internal_action_retry_timeout\":\"1m\",\"recovery_activity_timeout\":\"1800000ms\",\"retry_delay_network\":\"5s\",\"internal_action_timeout\":\"15m\",\"max_concurrent_snapshot_file_downloads_per_node\":\"25\",\"retry_delay_state_sync\":\"500ms\",\"max_concurrent_snapshot_file_downloads\":\"5\",\"internal_action_long_timeout\":\"1800000ms\",\"max_concurrent_operations\":\"1\",\"use_snapshots\":\"true\",\"max_bytes_per_sec\":\"40mb\",\"max_concurrent_file_chunks\":\"2\"},\"requests\":{\"cache\":{\"size\":\"1%\",\"expire\":\"0ms\"}},\"store\":{\"delete\":{\"shard\":{\"timeout\":\"30s\"}},\"shard_lock_retry\":{\"interval\":\"1s\",\"timeout\":\"1m\"}},\"analysis\":{\"hunspell\":{\"dictionary\":{\"ignore_case\":\"false\",\"lazy\":\"false\"}}},\"queries\":{\"cache\":{\"count\":\"10000\",\"size\":\"10%\",\"all_segments\":\"false\"}},\"lifecycle\":{\"poll_interval\":\"10m\",\"rollover\":{\"only_if_has_documents\":\"true\"},\"step\":{\"master_timeout\":\"30s\"},\"history_index_enabled\":\"true\"},\"write_ack_delay_randomness_bound\":\"70ms\",\"fielddata\":{\"cache\":{\"size\":\"-1b\"}}},\"master_history\":{\"max_age\":\"30m\"},\"plugin\":{\"mandatory\":[]},\"ingest_node\":{\"transport_action_timeout\":\"20s\"},\"slm\":{\"health\":{\"failed_snapshot_warn_threshold\":\"5\"},\"minimum_interval\":\"15m\",\"retention_schedule\":\"0 30 1 * * ?\",\"retention_duration\":\"1h\",\"history_index_enabled\":\"true\"},\"discovery\":{\"seed_hosts\":[\"es01\",\"es02\",\"es03\"],\"unconfigured_bootstrap_timeout\":\"3s\",\"request_peers_timeout\":\"3000ms\",\"initial_state_timeout\":\"30s\",\"cluster_formation_warning_timeout\":\"10000ms\",\"seed_providers\":[],\"type\":\"multi-node\",\"seed_resolver\":{\"max_concurrent_resolvers\":\"10\",\"timeout\":\"5s\"},\"find_peers_interval\":\"1000ms\",\"probe\":{\"connect_timeout\":\"30s\",\"handshake_timeout\":\"30s\"}},\"http\":{\"cors\":{\"max-age\":\"1728000\",\"allow-origin\":\"\",\"allow-headers\":\"X-Requested-With,Content-Type,Content-Length,Authorization,Accept,User-Agent,X-Elastic-Client-Meta\",\"allow-credentials\":\"false\",\"allow-methods\":\"OPTIONS,HEAD,GET,POST,PUT,DELETE\",\"enabled\":\"false\"},\"max_chunk_size\":\"8kb\",\"compression_level\":\"3\",\"max_initial_line_length\":\"4kb\",\"shutdown_grace_period\":\"0ms\",\"type\":\"\",\"pipelining\":{\"max_events\":\"10000\"},\"type.default\":\"netty4\",\"host\":[],\"publish_port\":\"-1\",\"read_timeout\":\"0ms\",\"max_content_length\":\"100mb\",\"netty\":{\"receive_predictor_size\":\"64kb\",\"max_composite_buffer_components\":\"69905\",\"worker_count\":\"0\"},\"tcp\":{\"reuse_address\":\"true\",\"keep_count\":\"-1\",\"keep_interval\":\"-1\",\"no_delay\":\"true\",\"keep_alive\":\"true\",\"receive_buffer_size\":\"-1b\",\"keep_idle\":\"-1\",\"send_buffer_size\":\"-1b\"},\"bind_host\":[],\"client_stats\":{\"enabled\":\"true\",\"closed_channels\":{\"max_age\":\"5m\",\"max_count\":\"10000\"}},\"reset_cookies\":\"false\",\"max_warning_header_count\":\"-1\",\"tracer\":{\"include\":[],\"exclude\":[]},\"max_warning_header_size\":\"-1b\",\"detailed_errors\":{\"enabled\":\"true\"},\"port\":\"9200-9300\",\"max_header_size\":\"16kb\",\"compression\":\"true\",\"publish_host\":[]},\"write_load_forecaster\":{\"max_index_age\":\"7d\"},\"gateway\":{\"recover_after_data_nodes\":\"-1\",\"expected_data_nodes\":\"-1\",\"write_dangling_indices_info\":\"true\",\"slow_write_logging_threshold\":\"10s\",\"recover_after_time\":\"0ms\"},\"snapshot\":{\"refresh_repo_uuid_on_restore\":\"true\",\"max_concurrent_operations\":\"1000\"}}}']\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "# Script to check cluster health by retrieving cluster settings\n", | |
| "ES7_URL_WITH_CREDS = 'https://fuzzy-system-46j556jrg7c7jpr-9200.app.github.dev'\n", | |
| "response = !curl -sSL -X GET \"{ES7_URL_WITH_CREDS}/_cluster/settings?include_defaults=true\"\n", | |
| "print(response)" | |
| ] | |
| }, | |
| { | |
| "attachments": {}, | |
| "cell_type": "markdown", | |
| "id": "31227ba5-4be2-4cf6-9aba-c78db54dbba6", | |
| "metadata": { | |
| "noteable": { | |
| "cell_type": "markdown" | |
| } | |
| }, | |
| "source": [ | |
| "## Checking Cluster Shard Allocation\n", | |
| "This section is dedicated to checking the shard allocation within the Elasticsearch cluster. Shards are the basic units of storage and search in Elasticsearch, and their allocation determines how data is distributed across the nodes of the cluster. Monitoring shard allocation is crucial for ensuring data redundancy, load balancing, and overall cluster performance. The script below retrieves the current shard allocation status, which can be used to identify any unassigned shards or imbalances in shard distribution." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "id": "0ab57afd-7eda-4bc9-b518-64134c103ad6", | |
| "metadata": { | |
| "noteable": { | |
| "cell_type": "code" | |
| }, | |
| "vscode": { | |
| "languageId": "python" | |
| } | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "# Script to check cluster shard allocation\n", | |
| "response = !curl -sSL \"{ES7_URL_WITH_CREDS}/_cat/shards?v\" | sort\n", | |
| "print(response)" | |
| ] | |
| }, | |
| { | |
| "attachments": {}, | |
| "cell_type": "markdown", | |
| "id": "7d74fc81-b96f-482b-b36e-a8aee5d3ffcf", | |
| "metadata": { | |
| "noteable": { | |
| "cell_type": "markdown" | |
| } | |
| }, | |
| "source": [ | |
| "## Checking Cluster Nodes Health\n", | |
| "This section is dedicated to checking the health of the nodes within the Elasticsearch cluster. Nodes are the individual instances of Elasticsearch that hold data and participate in the cluster's indexing and search capabilities. Monitoring the health and allocation of nodes is essential for ensuring the cluster's resilience, performance, and scalability. The script below retrieves the current node allocation status, providing insights into which nodes are available, how many shards each node holds, and the disk space usage." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "id": "68db22da-c406-4ba4-909b-229036ad12f2", | |
| "metadata": { | |
| "ExecuteTime": { | |
| "end_time": "2023-08-11T16:20:36.206101+00:00", | |
| "start_time": "2023-08-11T16:20:35.391353+00:00" | |
| }, | |
| "noteable": { | |
| "cell_type": "code" | |
| }, | |
| "vscode": { | |
| "languageId": "python" | |
| } | |
| }, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "['shards disk.indices disk.used disk.avail disk.total disk.percent host ip node', ' 19 2.6mb 33gb 92.7gb 125.8gb 26 172.22.0.7 172.22.0.7 es05', ' 19 2.6mb 33gb 92.7gb 125.8gb 26 172.22.0.6 172.22.0.6 es04']\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "# Script to check cluster nodes health by retrieving node allocation\n", | |
| "response = !curl -sSL \"{ES7_URL_WITH_CREDS}/_cat/allocation?v\"\n", | |
| "print(response)" | |
| ] | |
| } | |
| ], | |
| "metadata": { | |
| "kernel_info": { | |
| "name": "python3" | |
| }, | |
| "kernelspec": { | |
| "display_name": "Python 3.9", | |
| "language": "python", | |
| "name": "python3" | |
| }, | |
| "noteable": { | |
| "last_delta_id": "68f3d60e-a548-416f-b40a-7cf2573d7d87", | |
| "last_transaction_id": "a91114a5-b907-419b-9c22-4350d14c3721" | |
| }, | |
| "noteable-chatgpt": { | |
| "create_notebook": { | |
| "openai_conversation_id": "d43e4a58-13df-5d3b-8e64-1b49ea725cac", | |
| "openai_ephemeral_user_id": "5df9c841-3988-5bb2-9b96-f785101b3c84", | |
| "openai_subdivision1_iso_code": "US-WA" | |
| } | |
| }, | |
| "nteract": { | |
| "version": "noteable@2.9.0" | |
| }, | |
| "selected_hardware_size": "small" | |
| }, | |
| "nbformat": 4, | |
| "nbformat_minor": 5 | |
| } |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment