Skip to content

Instantly share code, notes, and snippets.

@lukas-vlcek

lukas-vlcek/gist:a61aea97ebe4c2bf22651c3694e76c71

Last active August 9, 2019 12:03
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save lukas-vlcek/a61aea97ebe4c2bf22651c3694e76c71 to your computer and use it in GitHub Desktop.
Save lukas-vlcek/a61aea97ebe4c2bf22651c3694e76c71 to your computer and use it in GitHub Desktop.
Download ZIP
Dots in fields problem when upgrading from ES2 to ES5
Raw
gistfile1.txt
# Start fresh ES 2.4.6 with dots in field names enabled
$ ES_JAVA_OPTS="-Dmapper.allow_dots_in_name=true" ./bin/elasticsearch
# Index two documents with conflisting mappings (conflisting from the ES5 perspective)
# See: https://www.elastic.co/guide/en/elasticsearch/reference/2.4/dots-in-names.html#_upgrading_fields_with_dots_to_5_x
$ curl -X POST localhost:9200/test/type -d '{"foo.bar":1}'
{"_index":"test","_type":"type","_id":"AWx2N0zkYS6E_SuIWEKX","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true}
$ curl -X POST localhost:9200/test/type2 -d '{"foo":1}'
{"_index":"test","_type":"type2","_id":"AWx2N2wfYS6E_SuIWEKY","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true}
$ curl -X POST localhost:9200/_flush
{"_shards":{"total":10,"successful":5,"failed":0}}
# We can see both documents are indexed and as sitting in the index side by side
$ curl http://localhost:9200/_search?q=1&pretty
{
"took": 6,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 0.30685282,
"hits": [
{
"_index": "test",
"_type": "type",
"_id": "AWx2N0zkYS6E_SuIWEKX",
"_score": 0.30685282,
"_source": {
"foo.bar": 1
}
},
{
"_index": "test",
"_type": "type2",
"_id": "AWx2N2wfYS6E_SuIWEKY",
"_score": 0.30685282,
"_source": {
"foo": 1
}
}
]
}
}
# Stop ES 2.4.6
# Start ES 5.6.13 (default ES 5, not special config options this time)
$ ./bin/elasticsearch
[2019-08-09T13:51:31,116][INFO ][o.e.n.Node ] [] initializing ...
[2019-08-09T13:51:31,199][INFO ][o.e.e.NodeEnvironment ] [Jqd0Cgi] using [1] data paths, mounts [[/ (/dev/disk1s1)]], net usable_space [312.5gb], net total_space [465.6gb], spins? [unknown], types [apfs]
[2019-08-09T13:51:31,200][INFO ][o.e.e.NodeEnvironment ] [Jqd0Cgi] heap size [1.9gb], compressed ordinary object pointers [true]
[2019-08-09T13:51:31,212][INFO ][o.e.n.Node ] node name [Jqd0Cgi] derived from node ID [Jqd0CgiKTmuy5myLPeD4Rg]; set [node.name] to override
[2019-08-09T13:51:31,213][INFO ][o.e.n.Node ] version[5.6.13], pid[27948], build[4d5320b/2018-10-30T19:05:08.237Z], OS[Mac OS X/10.14.6/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_202-ea/25.202-b03]
[2019-08-09T13:51:31,213][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/Users/lvlcek/projects/elasticsearch-distros/elasticsearch-5.6.13]
[2019-08-09T13:51:31,854][INFO ][o.e.p.PluginsService ] [Jqd0Cgi] loaded module [aggs-matrix-stats]
[2019-08-09T13:51:31,854][INFO ][o.e.p.PluginsService ] [Jqd0Cgi] loaded module [ingest-common]
[2019-08-09T13:51:31,855][INFO ][o.e.p.PluginsService ] [Jqd0Cgi] loaded module [lang-expression]
[2019-08-09T13:51:31,855][INFO ][o.e.p.PluginsService ] [Jqd0Cgi] loaded module [lang-groovy]
[2019-08-09T13:51:31,855][INFO ][o.e.p.PluginsService ] [Jqd0Cgi] loaded module [lang-mustache]
[2019-08-09T13:51:31,855][INFO ][o.e.p.PluginsService ] [Jqd0Cgi] loaded module [lang-painless]
[2019-08-09T13:51:31,855][INFO ][o.e.p.PluginsService ] [Jqd0Cgi] loaded module [parent-join]
[2019-08-09T13:51:31,855][INFO ][o.e.p.PluginsService ] [Jqd0Cgi] loaded module [percolator]
[2019-08-09T13:51:31,855][INFO ][o.e.p.PluginsService ] [Jqd0Cgi] loaded module [reindex]
[2019-08-09T13:51:31,855][INFO ][o.e.p.PluginsService ] [Jqd0Cgi] loaded module [transport-netty3]
[2019-08-09T13:51:31,855][INFO ][o.e.p.PluginsService ] [Jqd0Cgi] loaded module [transport-netty4]
[2019-08-09T13:51:31,856][INFO ][o.e.p.PluginsService ] [Jqd0Cgi] no plugins loaded
[2019-08-09T13:51:33,301][INFO ][o.e.d.DiscoveryModule ] [Jqd0Cgi] using discovery type [zen]
[2019-08-09T13:51:33,641][INFO ][o.e.c.u.IndexFolderUpgrader] [test/pX_LLKqZQW2Jbxlavy3ngA] upgrading [/Users/lvlcek/projects/elasticsearch-distros/elasticsearch-5.6.13/data/elasticsearch/nodes/0/indices/test] to new naming convention
[2019-08-09T13:51:33,642][INFO ][o.e.c.u.IndexFolderUpgrader] [test/pX_LLKqZQW2Jbxlavy3ngA] moved from [/Users/lvlcek/projects/elasticsearch-distros/elasticsearch-5.6.13/data/elasticsearch/nodes/0/indices/test] to [/Users/lvlcek/projects/elasticsearch-distros/elasticsearch-5.6.13/data/elasticsearch/nodes/0/indices/pX_LLKqZQW2Jbxlavy3ngA]
[2019-08-09T13:51:33,708][ERROR][o.e.g.GatewayMetaState ] [Jqd0Cgi] failed to read local state, exiting...
java.lang.IllegalStateException: unable to upgrade the mappings for the index [[test/pX_LLKqZQW2Jbxlavy3ngA]]
at org.elasticsearch.cluster.metadata.MetaDataIndexUpgradeService.checkMappingsCompatibility(MetaDataIndexUpgradeService.java:199) ~[elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.cluster.metadata.MetaDataIndexUpgradeService.upgradeIndexMetaData(MetaDataIndexUpgradeService.java:100) ~[elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:249) ~[elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:93) [elasticsearch-5.6.13.jar:5.6.13]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [?:1.8.0_202-ea]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [?:1.8.0_202-ea]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [?:1.8.0_202-ea]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) [?:1.8.0_202-ea]
at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:49) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:116) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:825) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:50) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:116) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:825) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:50) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:191) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:183) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:818) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:183) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:173) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:161) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:96) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:96) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:42) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.node.Node.<init>(Node.java:499) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.node.Node.<init>(Node.java:245) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:233) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.6.13.jar:5.6.13]
Caused by: java.lang.IllegalArgumentException: [foo] is defined as an object in mapping [type] but this name is already used for a field in other types
at org.elasticsearch.index.mapper.MapperService.checkFieldUniqueness(MapperService.java:570) ~[elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:394) ~[elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:336) ~[elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:285) ~[elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:264) ~[elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.cluster.metadata.MetaDataIndexUpgradeService.checkMappingsCompatibility(MetaDataIndexUpgradeService.java:195) ~[elasticsearch-5.6.13.jar:5.6.13]
... 46 more
[2019-08-09T13:51:33,721][ERROR][o.e.g.GatewayMetaState ] [Jqd0Cgi] failed to read local state, exiting...]
[...]
[2019-08-09T13:51:33,776][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: org.elasticsearch.common.inject.CreationException: Guice creation errors:
1) Error injecting constructor, java.lang.IllegalStateException: unable to upgrade the mappings for the index [[test/pX_LLKqZQW2Jbxlavy3ngA]]
at org.elasticsearch.gateway.GatewayMetaState.<init>(Unknown Source)
while locating org.elasticsearch.gateway.GatewayMetaState
for parameter 4 at org.elasticsearch.gateway.GatewayService.<init>(Unknown Source)
while locating org.elasticsearch.gateway.GatewayService
Caused by: java.lang.IllegalStateException: unable to upgrade the mappings for the index [[test/pX_LLKqZQW2Jbxlavy3ngA]]
at org.elasticsearch.cluster.metadata.MetaDataIndexUpgradeService.checkMappingsCompatibility(MetaDataIndexUpgradeService.java:199)
at org.elasticsearch.cluster.metadata.MetaDataIndexUpgradeService.upgradeIndexMetaData(MetaDataIndexUpgradeService.java:100)
at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:249)
at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:93)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:49)
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:116)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:825)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:50)
at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:116)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:825)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:50)
at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:191)
at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:183)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:818)
at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:183)
at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:173)
at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:161)
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:96)
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:96)
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:42)
at org.elasticsearch.node.Node.<init>(Node.java:499)
at org.elasticsearch.node.Node.<init>(Node.java:245)
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:233)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132)
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84)
Caused by: java.lang.IllegalArgumentException: [foo] is defined as an object in mapping [type] but this name is already used for a field in other types
at org.elasticsearch.index.mapper.MapperService.checkFieldUniqueness(MapperService.java:570)
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:394)
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:336)
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:285)
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:264)
at org.elasticsearch.cluster.metadata.MetaDataIndexUpgradeService.checkMappingsCompatibility(MetaDataIndexUpgradeService.java:195)
... 46 more
4 errors
at org.elasticsearch.common.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:361) ~[elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:164) ~[elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:96) ~[elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:96) ~[elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70) ~[elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:42) ~[elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.node.Node.<init>(Node.java:499) ~[elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.node.Node.<init>(Node.java:245) ~[elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:233) ~[elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233) ~[elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.13.jar:5.6.13]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.13.jar:5.6.13]
... 6 more
And the ES node does not start.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment