Skip to content

Instantly share code, notes, and snippets.

@DumboJetEngine

DumboJetEngine/opensearch.log

Created July 17, 2024 11:14
Show Gist options
  • Save DumboJetEngine/8cdeaf9159e9af14831a352b1f5a3128 to your computer and use it in GitHub Desktop.
Save DumboJetEngine/8cdeaf9159e9af14831a352b1f5a3128 to your computer and use it in GitHub Desktop.
Download ZIP
OpenSearch fails to start
Raw
opensearch.log
[2024-07-17T13:57:12,381][INFO ][o.o.n.Node ] [] version[2.9.0], pid[34828], build[zip/1164221ee2b8ba3560f0ff492309867beea28433/2023-07-18T21:29:14.278655500Z], OS[Windows 11/10.0/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.7/17.0.7+7]
[2024-07-17T13:57:12,389][INFO ][o.o.n.Node ] [PC123] JVM home [D:\opensearch-2.9.0\jdk], using bundled JDK [true]
[2024-07-17T13:57:12,389][INFO ][o.o.n.Node ] [PC123] JVM arguments [-Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=C:\Users\Ghost\AppData\Local\Temp\opensearch, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Dopensearch, -Dopensearch.path.home=D:\opensearch-2.9.0, -Dopensearch.path.conf=D:\opensearch-2.9.0\config, -Dopensearch.distribution.type=zip, -Dopensearch.bundled_jdk=true]
[2024-07-17T13:57:26,767][INFO ][o.o.s.s.t.SSLConfig ] [PC123] SSL dual mode is disabled
[2024-07-17T13:57:26,768][INFO ][o.o.s.OpenSearchSecurityPlugin] [PC123] OpenSearch Config path is D:\opensearch-2.9.0\config
[2024-07-17T13:57:27,147][INFO ][o.o.s.s.DefaultSecurityKeyStore] [PC123] JVM supports TLSv1.3
[2024-07-17T13:57:27,153][INFO ][o.o.s.s.DefaultSecurityKeyStore] [PC123] Config directory is D:\opensearch-2.9.0\config/, from there the key- and truststore files are resolved relatively
[2024-07-17T13:57:27,927][INFO ][o.o.s.s.DefaultSecurityKeyStore] [PC123] TLS Transport Client Provider : JDK
[2024-07-17T13:57:27,927][INFO ][o.o.s.s.DefaultSecurityKeyStore] [PC123] TLS Transport Server Provider : JDK
[2024-07-17T13:57:27,928][INFO ][o.o.s.s.DefaultSecurityKeyStore] [PC123] TLS HTTP Provider : JDK
[2024-07-17T13:57:27,928][INFO ][o.o.s.s.DefaultSecurityKeyStore] [PC123] Enabled TLS protocols for transport layer : [TLSv1.2]
[2024-07-17T13:57:27,928][INFO ][o.o.s.s.DefaultSecurityKeyStore] [PC123] Enabled TLS protocols for HTTP layer : [TLSv1.2]
[2024-07-17T13:57:28,002][INFO ][o.o.s.OpenSearchSecurityPlugin] [PC123] Clustername: opensearch
[2024-07-17T13:57:57,199][INFO ][o.o.i.r.ReindexPlugin ] [PC123] ReindexPlugin reloadSPI called
[2024-07-17T13:57:57,202][INFO ][o.o.i.r.ReindexPlugin ] [PC123] Unable to find any implementation for RemoteReindexExtension
[2024-07-17T13:57:57,316][INFO ][o.o.j.JobSchedulerPlugin ] [PC123] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2024-07-17T13:57:57,344][INFO ][o.o.j.JobSchedulerPlugin ] [PC123] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2024-07-17T13:57:57,346][INFO ][o.o.j.JobSchedulerPlugin ] [PC123] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2024-07-17T13:57:57,371][INFO ][o.o.j.JobSchedulerPlugin ] [PC123] Loaded scheduler extension: observability, index: .opensearch-observability-job
[2024-07-17T13:57:57,381][INFO ][o.o.p.PluginsService ] [PC123] loaded module [aggs-matrix-stats]
[2024-07-17T13:57:57,381][INFO ][o.o.p.PluginsService ] [PC123] loaded module [analysis-common]
[2024-07-17T13:57:57,381][INFO ][o.o.p.PluginsService ] [PC123] loaded module [geo]
[2024-07-17T13:57:57,382][INFO ][o.o.p.PluginsService ] [PC123] loaded module [ingest-common]
[2024-07-17T13:57:57,382][INFO ][o.o.p.PluginsService ] [PC123] loaded module [ingest-geoip]
[2024-07-17T13:57:57,382][INFO ][o.o.p.PluginsService ] [PC123] loaded module [ingest-user-agent]
[2024-07-17T13:57:57,382][INFO ][o.o.p.PluginsService ] [PC123] loaded module [lang-expression]
[2024-07-17T13:57:57,383][INFO ][o.o.p.PluginsService ] [PC123] loaded module [lang-mustache]
[2024-07-17T13:57:57,383][INFO ][o.o.p.PluginsService ] [PC123] loaded module [lang-painless]
[2024-07-17T13:57:57,383][INFO ][o.o.p.PluginsService ] [PC123] loaded module [mapper-extras]
[2024-07-17T13:57:57,384][INFO ][o.o.p.PluginsService ] [PC123] loaded module [opensearch-dashboards]
[2024-07-17T13:57:57,384][INFO ][o.o.p.PluginsService ] [PC123] loaded module [parent-join]
[2024-07-17T13:57:57,384][INFO ][o.o.p.PluginsService ] [PC123] loaded module [percolator]
[2024-07-17T13:57:57,385][INFO ][o.o.p.PluginsService ] [PC123] loaded module [rank-eval]
[2024-07-17T13:57:57,385][INFO ][o.o.p.PluginsService ] [PC123] loaded module [reindex]
[2024-07-17T13:57:57,385][INFO ][o.o.p.PluginsService ] [PC123] loaded module [repository-url]
[2024-07-17T13:57:57,385][INFO ][o.o.p.PluginsService ] [PC123] loaded module [search-pipeline-common]
[2024-07-17T13:57:57,385][INFO ][o.o.p.PluginsService ] [PC123] loaded module [transport-netty4]
[2024-07-17T13:57:57,386][INFO ][o.o.p.PluginsService ] [PC123] loaded plugin [opensearch-alerting]
[2024-07-17T13:57:57,386][INFO ][o.o.p.PluginsService ] [PC123] loaded plugin [opensearch-anomaly-detection]
[2024-07-17T13:57:57,386][INFO ][o.o.p.PluginsService ] [PC123] loaded plugin [opensearch-asynchronous-search]
[2024-07-17T13:57:57,386][INFO ][o.o.p.PluginsService ] [PC123] loaded plugin [opensearch-cross-cluster-replication]
[2024-07-17T13:57:57,388][INFO ][o.o.p.PluginsService ] [PC123] loaded plugin [opensearch-geospatial]
[2024-07-17T13:57:57,388][INFO ][o.o.p.PluginsService ] [PC123] loaded plugin [opensearch-index-management]
[2024-07-17T13:57:57,388][INFO ][o.o.p.PluginsService ] [PC123] loaded plugin [opensearch-job-scheduler]
[2024-07-17T13:57:57,389][INFO ][o.o.p.PluginsService ] [PC123] loaded plugin [opensearch-knn]
[2024-07-17T13:57:57,389][INFO ][o.o.p.PluginsService ] [PC123] loaded plugin [opensearch-ml]
[2024-07-17T13:57:57,389][INFO ][o.o.p.PluginsService ] [PC123] loaded plugin [opensearch-neural-search]
[2024-07-17T13:57:57,389][INFO ][o.o.p.PluginsService ] [PC123] loaded plugin [opensearch-notifications]
[2024-07-17T13:57:57,390][INFO ][o.o.p.PluginsService ] [PC123] loaded plugin [opensearch-notifications-core]
[2024-07-17T13:57:57,390][INFO ][o.o.p.PluginsService ] [PC123] loaded plugin [opensearch-observability]
[2024-07-17T13:57:57,390][INFO ][o.o.p.PluginsService ] [PC123] loaded plugin [opensearch-reports-scheduler]
[2024-07-17T13:57:57,390][INFO ][o.o.p.PluginsService ] [PC123] loaded plugin [opensearch-security]
[2024-07-17T13:57:57,391][INFO ][o.o.p.PluginsService ] [PC123] loaded plugin [opensearch-security-analytics]
[2024-07-17T13:57:57,391][INFO ][o.o.p.PluginsService ] [PC123] loaded plugin [opensearch-sql]
[2024-07-17T13:57:57,478][INFO ][o.o.s.OpenSearchSecurityPlugin] [PC123] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[2024-07-17T13:57:57,494][INFO ][o.o.e.ExtensionsManager ] [PC123] ExtensionsManager initialized
[2024-07-17T13:57:57,568][INFO ][o.o.e.NodeEnvironment ] [PC123] using [1] data paths, mounts [[Expansion (D:)]], net usable_space [5.5tb], net total_space [7.2tb], types [NTFS]
[2024-07-17T13:57:57,569][INFO ][o.o.e.NodeEnvironment ] [PC123] heap size [1gb], compressed ordinary object pointers [true]
[2024-07-17T13:57:57,908][INFO ][o.o.n.Node ] [PC123] node name [PC123], node ID [CxtRxAvFSTetihtxZgKYvg], cluster name [opensearch], roles [ingest, remote_cluster_client, data, cluster_manager]
[2024-07-17T13:58:02,834][WARN ][o.o.s.c.Salt ] [PC123] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2024-07-17T13:58:02,885][INFO ][o.o.s.a.i.AuditLogImpl ] [PC123] Message routing enabled: true
[2024-07-17T13:58:02,973][INFO ][o.o.s.f.SecurityFilter ] [PC123] <NONE> indices are made immutable.
[2024-07-17T13:58:03,386][INFO ][o.o.a.b.ADCircuitBreakerService] [PC123] Registered memory breaker.
[2024-07-17T13:58:03,952][INFO ][o.o.m.b.MLCircuitBreakerService] [PC123] Registered ML memory breaker.
[2024-07-17T13:58:03,953][INFO ][o.o.m.b.MLCircuitBreakerService] [PC123] Registered ML disk breaker.
[2024-07-17T13:58:03,954][INFO ][o.o.m.b.MLCircuitBreakerService] [PC123] Registered ML native memory breaker.
[2024-07-17T13:58:04,110][INFO ][o.r.Reflections ] [PC123] Reflections took 44 ms to scan 1 urls, producing 17 keys and 43 values
[2024-07-17T13:58:04,209][WARN ][o.o.s.p.SQLPlugin ] [PC123] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information
[2024-07-17T13:58:04,909][INFO ][o.o.t.NettyAllocator ] [PC123] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]
[2024-07-17T13:58:05,056][INFO ][o.o.d.DiscoveryModule ] [PC123] using discovery type [zen] and seed hosts providers [settings]
[2024-07-17T13:58:05,605][WARN ][o.o.g.DanglingIndicesState] [PC123] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2024-07-17T13:58:06,155][INFO ][o.o.n.Node ] [PC123] initialized
[2024-07-17T13:58:06,156][INFO ][o.o.n.Node ] [PC123] starting ...
[2024-07-17T13:58:06,196][INFO ][o.o.s.l.BuiltinLogTypeLoader] [PC123] Loaded [windows_logtype.json] log type
[2024-07-17T13:58:06,198][INFO ][o.o.s.l.BuiltinLogTypeLoader] [PC123] Loaded [vpcflow_logtype.json] log type
[2024-07-17T13:58:06,199][INFO ][o.o.s.l.BuiltinLogTypeLoader] [PC123] Loaded [test_windows_logtype.json] log type
[2024-07-17T13:58:06,200][INFO ][o.o.s.l.BuiltinLogTypeLoader] [PC123] Loaded [s3_logtype.json] log type
[2024-07-17T13:58:06,201][INFO ][o.o.s.l.BuiltinLogTypeLoader] [PC123] Loaded [others_web_logtype.json] log type
[2024-07-17T13:58:06,202][INFO ][o.o.s.l.BuiltinLogTypeLoader] [PC123] Loaded [others_proxy_logtype.json] log type
[2024-07-17T13:58:06,202][INFO ][o.o.s.l.BuiltinLogTypeLoader] [PC123] Loaded [others_macos_logtype.json] log type
[2024-07-17T13:58:06,203][INFO ][o.o.s.l.BuiltinLogTypeLoader] [PC123] Loaded [others_compliance_logtype.json] log type
[2024-07-17T13:58:06,203][INFO ][o.o.s.l.BuiltinLogTypeLoader] [PC123] Loaded [others_cloud_logtype.json] log type
[2024-07-17T13:58:06,205][INFO ][o.o.s.l.BuiltinLogTypeLoader] [PC123] Loaded [others_apt_logtype.json] log type
[2024-07-17T13:58:06,205][INFO ][o.o.s.l.BuiltinLogTypeLoader] [PC123] Loaded [others_application_logtype.json] log type
[2024-07-17T13:58:06,206][INFO ][o.o.s.l.BuiltinLogTypeLoader] [PC123] Loaded [okta_logtype.json] log type
[2024-07-17T13:58:06,207][INFO ][o.o.s.l.BuiltinLogTypeLoader] [PC123] Loaded [network_logtype.json] log type
[2024-07-17T13:58:06,208][INFO ][o.o.s.l.BuiltinLogTypeLoader] [PC123] Loaded [netflow_logtype.json] log type
[2024-07-17T13:58:06,209][INFO ][o.o.s.l.BuiltinLogTypeLoader] [PC123] Loaded [m365_logtype.json] log type
[2024-07-17T13:58:06,210][INFO ][o.o.s.l.BuiltinLogTypeLoader] [PC123] Loaded [linux_logtype.json] log type
[2024-07-17T13:58:06,210][INFO ][o.o.s.l.BuiltinLogTypeLoader] [PC123] Loaded [gworkspace_logtype.json] log type
[2024-07-17T13:58:06,211][INFO ][o.o.s.l.BuiltinLogTypeLoader] [PC123] Loaded [github_logtype.json] log type
[2024-07-17T13:58:06,212][INFO ][o.o.s.l.BuiltinLogTypeLoader] [PC123] Loaded [dns_logtype.json] log type
[2024-07-17T13:58:06,213][INFO ][o.o.s.l.BuiltinLogTypeLoader] [PC123] Loaded [cloudtrail_logtype.json] log type
[2024-07-17T13:58:06,214][INFO ][o.o.s.l.BuiltinLogTypeLoader] [PC123] Loaded [azure_logtype.json] log type
[2024-07-17T13:58:06,214][INFO ][o.o.s.l.BuiltinLogTypeLoader] [PC123] Loaded [apache_access_logtype.json] log type
[2024-07-17T13:58:06,215][INFO ][o.o.s.l.BuiltinLogTypeLoader] [PC123] Loaded [ad_ldap_logtype.json] log type
[2024-07-17T13:58:06,471][INFO ][o.o.t.TransportService ] [PC123] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2024-07-17T13:58:06,473][INFO ][o.o.t.TransportService ] [PC123] Remote clusters initialized successfully.
[2024-07-17T13:58:07,375][WARN ][o.o.b.BootstrapChecks ] [PC123] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_cluster_manager_nodes / cluster.initial_master_nodes] must be configured
[2024-07-17T13:58:07,377][INFO ][o.o.c.c.Coordinator ] [PC123] cluster UUID [BIx66hXnSmiyDENIq45nWA]
[2024-07-17T13:58:07,388][INFO ][o.o.c.c.ClusterBootstrapService] [PC123] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing cluster-manager is discovered
[2024-07-17T13:58:07,572][INFO ][o.o.c.s.MasterService ] [PC123] elected-as-cluster-manager ([1] nodes joined)[{PC123}{CxtRxAvFSTetihtxZgKYvg}{2SPPYg7_T-atqDmkdhspXQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true} elect leader, _BECOME_CLUSTER_MANAGER_TASK_, _FINISH_ELECTION_], term: 16, version: 325, delta: cluster-manager node changed {previous [], current [{PC123}{CxtRxAvFSTetihtxZgKYvg}{2SPPYg7_T-atqDmkdhspXQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}]}
[2024-07-17T13:58:07,907][INFO ][o.o.c.s.ClusterApplierService] [PC123] cluster-manager node changed {previous [], current [{PC123}{CxtRxAvFSTetihtxZgKYvg}{2SPPYg7_T-atqDmkdhspXQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}]}, term: 16, version: 325, reason: Publication{term=16, version=325}
[2024-07-17T13:58:07,914][INFO ][o.o.a.c.ADClusterEventListener] [PC123] Cluster is not recovered yet.
[2024-07-17T13:58:07,919][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [PC123] Detected cluster change event for destination migration
[2024-07-17T13:58:07,936][INFO ][o.o.i.i.ManagedIndexCoordinator] [PC123] Cache cluster manager node onClusterManager time: 1721213887935
[2024-07-17T13:58:07,948][INFO ][o.o.d.PeerFinder ] [PC123] setting findPeersInterval to [1s] as node commission status = [true] for local node [{PC123}{CxtRxAvFSTetihtxZgKYvg}{2SPPYg7_T-atqDmkdhspXQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}]
[2024-07-17T13:58:07,954][INFO ][o.o.h.AbstractHttpServerTransport] [PC123] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[2024-07-17T13:58:07,954][INFO ][o.o.n.Node ] [PC123] started
[2024-07-17T13:58:07,954][INFO ][o.o.s.OpenSearchSecurityPlugin] [PC123] Node started
[2024-07-17T13:58:07,955][INFO ][o.o.s.c.ConfigurationRepository] [PC123] Will attempt to create index .opendistro_security and default configs if they are absent
[2024-07-17T13:58:07,955][INFO ][o.o.s.c.ConfigurationRepository] [PC123] Background init thread started. Install default config?: true
[2024-07-17T13:58:07,956][INFO ][o.o.s.OpenSearchSecurityPlugin] [PC123] 0 OpenSearch Security modules loaded so far: []
[2024-07-17T13:58:07,956][INFO ][o.o.s.c.ConfigurationRepository] [PC123] Wait for cluster to be available ...
[2024-07-17T13:58:08,144][INFO ][o.o.c.s.ClusterSettings ] [PC123] updating [plugins.index_state_management.template_migration.control] from [0] to [-1]
[2024-07-17T13:58:08,149][INFO ][o.o.a.c.HashRing ] [PC123] Node added: [CxtRxAvFSTetihtxZgKYvg]
[2024-07-17T13:58:08,150][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [PC123] Detected cluster change event for destination migration
[2024-07-17T13:58:08,152][INFO ][o.o.a.c.HashRing ] [PC123] Add data node to AD version hash ring: CxtRxAvFSTetihtxZgKYvg
[2024-07-17T13:58:08,155][INFO ][o.o.a.c.HashRing ] [PC123] All nodes with known AD version: {CxtRxAvFSTetihtxZgKYvg=ADNodeInfo{version=2.9.0, isEligibleDataNode=true}}
[2024-07-17T13:58:08,155][INFO ][o.o.a.c.HashRing ] [PC123] Rebuild AD hash ring for realtime AD with cooldown, nodeChangeEvents size 0
[2024-07-17T13:58:08,155][INFO ][o.o.a.c.HashRing ] [PC123] Build AD version hash ring successfully
[2024-07-17T13:58:08,156][INFO ][o.o.a.c.ADDataMigrator ] [PC123] Start migrating AD data
[2024-07-17T13:58:08,156][INFO ][o.o.a.c.ADDataMigrator ] [PC123] AD job index doesn't exist, no need to migrate
[2024-07-17T13:58:08,157][INFO ][o.o.a.c.ADClusterEventListener] [PC123] Init AD version hash ring successfully
[2024-07-17T13:58:08,173][ERROR][o.o.b.Bootstrap ] [PC123] Exception
org.opensearch.cluster.block.ClusterBlockException: index [.opensearch-observability] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];
at org.opensearch.cluster.block.ClusterBlocks.indicesBlockedException(ClusterBlocks.java:243) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.block.ClusterBlocks.indexBlockedException(ClusterBlocks.java:221) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.admin.indices.create.TransportCreateIndexAction.checkBlock(TransportCreateIndexAction.java:95) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.admin.indices.create.TransportCreateIndexAction.checkBlock(TransportCreateIndexAction.java:56) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction.doStart(TransportClusterManagerNodeAction.java:237) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$2.onNewClusterState(TransportClusterManagerNodeAction.java:336) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onNewClusterState(ClusterStateObserver.java:380) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.clusterChanged(ClusterStateObserver.java:230) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.service.ClusterApplierService.callClusterStateListener(ClusterApplierService.java:625) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:613) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:577) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:484) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:186) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245) ~[opensearch-2.9.0.jar:2.9.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
[2024-07-17T13:58:08,170][ERROR][o.o.s.l.LogTypeService ] [PC123] Failed creating LOG_TYPE_INDEX
org.opensearch.cluster.block.ClusterBlockException: index [.opensearch-sap-log-types-config] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];
at org.opensearch.cluster.block.ClusterBlocks.indicesBlockedException(ClusterBlocks.java:243) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.block.ClusterBlocks.indexBlockedException(ClusterBlocks.java:221) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.admin.indices.create.TransportCreateIndexAction.checkBlock(TransportCreateIndexAction.java:95) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.admin.indices.create.TransportCreateIndexAction.checkBlock(TransportCreateIndexAction.java:56) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction.doStart(TransportClusterManagerNodeAction.java:237) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$2.onNewClusterState(TransportClusterManagerNodeAction.java:336) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onNewClusterState(ClusterStateObserver.java:380) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.clusterChanged(ClusterStateObserver.java:230) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.service.ClusterApplierService.callClusterStateListener(ClusterApplierService.java:625) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:613) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:577) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:484) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:186) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245) [opensearch-2.9.0.jar:2.9.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
[2024-07-17T13:58:08,170][ERROR][o.o.s.u.SecurityAnalyticsException] [PC123] Security Analytics error:
org.opensearch.cluster.block.ClusterBlockException: index [.opensearch-sap-pre-packaged-rules-config] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];
at org.opensearch.cluster.block.ClusterBlocks.indicesBlockedException(ClusterBlocks.java:243) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.block.ClusterBlocks.indexBlockedException(ClusterBlocks.java:221) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.admin.indices.create.TransportCreateIndexAction.checkBlock(TransportCreateIndexAction.java:95) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.admin.indices.create.TransportCreateIndexAction.checkBlock(TransportCreateIndexAction.java:56) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction.doStart(TransportClusterManagerNodeAction.java:237) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$2.onNewClusterState(TransportClusterManagerNodeAction.java:336) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onNewClusterState(ClusterStateObserver.java:380) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.clusterChanged(ClusterStateObserver.java:230) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.service.ClusterApplierService.callClusterStateListener(ClusterApplierService.java:625) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:613) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:577) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:484) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:186) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245) ~[opensearch-2.9.0.jar:2.9.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
[2024-07-17T13:58:08,185][WARN ][o.o.s.SecurityAnalyticsPlugin] [PC123] Failed to initialize LogType config index and builtin log types
[2024-07-17T13:58:08,187][INFO ][o.o.g.GatewayService ] [PC123] recovered [10] indices into cluster_state
[2024-07-17T13:58:08,185][WARN ][o.o.s.SecurityAnalyticsPlugin] [PC123] Failed initializing prepackaged rules
org.opensearch.securityanalytics.util.SecurityAnalyticsException: index [.opensearch-sap-pre-packaged-rules-config] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];
at org.opensearch.securityanalytics.util.SecurityAnalyticsException.wrap(SecurityAnalyticsException.java:51) ~[?:?]
at org.opensearch.securityanalytics.transport.TransportSearchRuleAction$AsyncSearchRulesAction.lambda$finishHim$0(TransportSearchRuleAction.java:239) ~[?:?]
at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:73) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:88) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908) [opensearch-2.9.0.jar:2.9.0]
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.9.0.jar:2.9.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.Exception: org.opensearch.cluster.block.ClusterBlockException: index [.opensearch-sap-pre-packaged-rules-config] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];
... 9 more
[2024-07-17T13:58:08,190][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [PC123] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: ClusterBlockException[index [.opensearch-observability] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.9.0.jar:2.9.0]
at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.9.0.jar:2.9.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.9.0.jar:2.9.0]
Caused by: org.opensearch.cluster.block.ClusterBlockException: index [.opensearch-observability] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];
at org.opensearch.cluster.block.ClusterBlocks.indicesBlockedException(ClusterBlocks.java:243) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.block.ClusterBlocks.indexBlockedException(ClusterBlocks.java:221) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.admin.indices.create.TransportCreateIndexAction.checkBlock(TransportCreateIndexAction.java:95) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.admin.indices.create.TransportCreateIndexAction.checkBlock(TransportCreateIndexAction.java:56) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction.doStart(TransportClusterManagerNodeAction.java:237) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$2.onNewClusterState(TransportClusterManagerNodeAction.java:336) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onNewClusterState(ClusterStateObserver.java:380) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.clusterChanged(ClusterStateObserver.java:230) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.service.ClusterApplierService.callClusterStateListener(ClusterApplierService.java:625) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:613) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:577) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:484) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:186) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245) ~[opensearch-2.9.0.jar:2.9.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
[2024-07-17T13:58:08,923][INFO ][o.o.p.PluginsService ] [PC123] PluginService:onIndexModule index:[documents/94E1WoxfSref8d7RCuWdYw]
[2024-07-17T13:58:08,953][INFO ][o.o.s.l.LogTypeService ] [PC123] Loading builtin types!
[2024-07-17T13:58:08,956][INFO ][o.o.s.l.LogTypeService ] [PC123] Indexing [418] fieldMappingDocs from logTypes: 23
[2024-07-17T13:58:08,968][ERROR][o.o.s.c.ConfigurationRepository] [PC123] Cannot apply default config (this is maybe not an error!)
org.opensearch.cluster.block.ClusterBlockException: index [.opendistro_security] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];
at org.opensearch.cluster.block.ClusterBlocks.indicesBlockedException(ClusterBlocks.java:243) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.cluster.block.ClusterBlocks.indexBlockedException(ClusterBlocks.java:221) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.admin.indices.create.TransportCreateIndexAction.checkBlock(TransportCreateIndexAction.java:95) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.admin.indices.create.TransportCreateIndexAction.checkBlock(TransportCreateIndexAction.java:56) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction.doStart(TransportClusterManagerNodeAction.java:237) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction.tryAction(TransportClusterManagerNodeAction.java:206) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.support.RetryableAction$1.doRun(RetryableAction.java:137) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:343) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.support.RetryableAction.run(RetryableAction.java:115) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction.doExecute(TransportClusterManagerNodeAction.java:167) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction.doExecute(TransportClusterManagerNodeAction.java:79) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) ~[?:?]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.indexmanagement.controlcenter.notification.filter.IndexOperationActionFilter.apply(IndexOperationActionFilter.kt:39) ~[?:?]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:290) ~[?:?]
at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) ~[?:?]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:463) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1528) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.client.support.AbstractClient$IndicesAdmin.create(AbstractClient.java:1622) ~[opensearch-2.9.0.jar:2.9.0]
at org.opensearch.security.configuration.ConfigurationRepository.createSecurityIndexIfAbsent(ConfigurationRepository.java:268) ~[opensearch-security-2.9.0.0.jar:2.9.0.0]
at org.opensearch.security.configuration.ConfigurationRepository.lambda$new$0(ConfigurationRepository.java:143) [opensearch-security-2.9.0.0.jar:2.9.0.0]
at java.lang.Thread.run(Thread.java:833) [?:?]
[2024-07-17T13:58:08,971][INFO ][o.o.s.l.LogTypeService ] [PC123] Loading builtin types!
[2024-07-17T13:58:08,973][INFO ][o.o.s.l.LogTypeService ] [PC123] Indexing [418] fieldMappingDocs from logTypes: 23
[2024-07-17T13:58:08,980][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [PC123] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@27795f5b] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-07-17T13:58:08,980][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [PC123] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@27795f5b] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-07-17T13:58:08,980][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [PC123] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@27795f5b] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-07-17T13:58:08,980][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [PC123] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@27795f5b] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-07-17T13:58:08,981][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [PC123] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@27795f5b] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-07-17T13:58:08,981][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [PC123] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@27795f5b] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-07-17T13:58:08,981][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [PC123] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@27795f5b] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-07-17T13:58:08,981][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [PC123] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@27795f5b] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-07-17T13:58:08,982][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [PC123] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@27795f5b] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-07-17T13:58:08,982][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [PC123] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@27795f5b] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-07-17T13:58:09,154][INFO ][o.o.p.PluginsService ] [PC123] PluginService:onIndexModule index:[.opendistro_security/G82nN-VcS0qYde9k6Qs4fQ]
[2024-07-17T13:58:09,161][INFO ][o.o.p.PluginsService ] [PC123] PluginService:onIndexModule index:[security-auditlog-2024.07.16/p97JlS5rQE6vX4OfOo3h8g]
[2024-07-17T13:58:09,597][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [PC123] Detected cluster change event for destination migration
[2024-07-17T13:58:11,907][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [PC123] Detected cluster change event for destination migration
[2024-07-17T13:58:13,310][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [PC123] Detected cluster change event for destination migration
[2024-07-17T13:58:15,524][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [PC123] Detected cluster change event for destination migration
[2024-07-17T13:58:18,408][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [PC123] Detected cluster change event for destination migration
[2024-07-17T13:58:19,619][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [PC123] Detected cluster change event for destination migration
[2024-07-17T13:58:21,343][INFO ][o.o.p.PluginsService ] [PC123] PluginService:onIndexModule index:[security-auditlog-2024.07.15/ziQTtyXRTPmy4z-i1b7WPg]
[2024-07-17T13:58:21,353][INFO ][o.o.p.PluginsService ] [PC123] PluginService:onIndexModule index:[.kibana_92668751_admin_1/K0V3aLK-S9-dDa5kHn6e_Q]
[2024-07-17T13:58:21,678][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [PC123] Detected cluster change event for destination migration
[2024-07-17T13:58:22,551][INFO ][stdout ] [PC123] [FINE] No subscribers registered for event class org.opensearch.security.securityconf.DynamicConfigFactory$NodesDnModelImpl
[2024-07-17T13:58:22,552][INFO ][stdout ] [PC123] [FINE] No subscribers registered for event class org.greenrobot.eventbus.NoSubscriberEvent
[2024-07-17T13:58:22,553][INFO ][o.o.s.a.i.AuditLogImpl ] [PC123] Auditing on REST API is enabled.
[2024-07-17T13:58:22,553][INFO ][o.o.s.a.i.AuditLogImpl ] [PC123] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing.
[2024-07-17T13:58:22,553][INFO ][o.o.s.a.i.AuditLogImpl ] [PC123] Auditing on Transport API is enabled.
[2024-07-17T13:58:22,553][INFO ][o.o.s.a.i.AuditLogImpl ] [PC123] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing.
[2024-07-17T13:58:22,553][INFO ][o.o.s.a.i.AuditLogImpl ] [PC123] Auditing of request body is enabled.
[2024-07-17T13:58:22,555][INFO ][o.o.s.a.i.AuditLogImpl ] [PC123] Bulk requests resolution is disabled during request auditing.
[2024-07-17T13:58:22,555][INFO ][o.o.s.a.i.AuditLogImpl ] [PC123] Index resolution is enabled during request auditing.
[2024-07-17T13:58:22,555][INFO ][o.o.s.a.i.AuditLogImpl ] [PC123] Sensitive headers auditing is enabled.
[2024-07-17T13:58:22,555][INFO ][o.o.s.a.i.AuditLogImpl ] [PC123] Auditing requests from kibanaserver users is disabled.
[2024-07-17T13:58:22,622][WARN ][o.o.s.a.r.AuditMessageRouter] [PC123] No endpoint configured for categories [BAD_HEADERS, FAILED_LOGIN, MISSING_PRIVILEGES, GRANTED_PRIVILEGES, OPENDISTRO_SECURITY_INDEX_ATTEMPT, SSL_EXCEPTION, AUTHENTICATED, INDEX_EVENT, COMPLIANCE_DOC_READ, COMPLIANCE_DOC_WRITE, COMPLIANCE_EXTERNAL_CONFIG, COMPLIANCE_INTERNAL_CONFIG_READ, COMPLIANCE_INTERNAL_CONFIG_WRITE], using default endpoint
[2024-07-17T13:58:22,623][INFO ][o.o.s.a.i.AuditLogImpl ] [PC123] Auditing of external configuration is disabled.
[2024-07-17T13:58:22,623][INFO ][o.o.s.a.i.AuditLogImpl ] [PC123] Auditing of internal configuration is enabled.
[2024-07-17T13:58:22,624][INFO ][o.o.s.a.i.AuditLogImpl ] [PC123] Auditing only metadata information for read request is enabled.
[2024-07-17T13:58:22,624][INFO ][o.o.s.a.i.AuditLogImpl ] [PC123] Auditing will watch {} for read requests.
[2024-07-17T13:58:22,624][INFO ][o.o.s.a.i.AuditLogImpl ] [PC123] Auditing read operation requests from kibanaserver users is disabled.
[2024-07-17T13:58:22,624][INFO ][o.o.s.a.i.AuditLogImpl ] [PC123] Auditing only metadata information for write request is enabled.
[2024-07-17T13:58:22,624][INFO ][o.o.s.a.i.AuditLogImpl ] [PC123] Auditing diffs for write requests is disabled.
[2024-07-17T13:58:22,625][INFO ][o.o.s.a.i.AuditLogImpl ] [PC123] Auditing write operation requests from kibanaserver users is disabled.
[2024-07-17T13:58:22,625][INFO ][o.o.s.a.i.AuditLogImpl ] [PC123] Auditing will watch <NONE> for write requests.
[2024-07-17T13:58:22,625][INFO ][o.o.s.a.i.AuditLogImpl ] [PC123] .opendistro_security is used as internal security index.
[2024-07-17T13:58:22,625][INFO ][o.o.s.a.i.AuditLogImpl ] [PC123] Internal index used for posting audit logs is null
[2024-07-17T13:58:22,626][INFO ][o.o.s.c.ConfigurationRepository] [PC123] Hot-reloading of audit configuration is enabled
[2024-07-17T13:58:22,626][INFO ][o.o.s.c.ConfigurationRepository] [PC123] Node 'PC123' initialized
[2024-07-17T13:58:22,627][INFO ][o.o.n.Node ] [PC123] stopping ...
[2024-07-17T13:58:22,628][INFO ][o.o.s.a.r.AuditMessageRouter] [PC123] Closing AuditMessageRouter
[2024-07-17T13:58:22,628][INFO ][o.o.s.a.s.SinkProvider ] [PC123] Closing InternalOpenSearchSink
[2024-07-17T13:58:22,628][INFO ][o.o.s.a.s.SinkProvider ] [PC123] Closing DebugSink
[2024-07-17T13:58:23,641][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [PC123] Detected cluster change event for destination migration
[2024-07-17T13:58:24,184][INFO ][o.o.n.Node ] [PC123] stopped
[2024-07-17T13:58:24,185][INFO ][o.o.n.Node ] [PC123] closing ...
[2024-07-17T13:58:24,193][INFO ][o.o.s.a.i.AuditLogImpl ] [PC123] Closing AuditLogImpl
[2024-07-17T13:58:24,197][INFO ][o.o.n.Node ] [PC123] closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment