Skip to content

Instantly share code, notes, and snippets.

@matthuhiggins
Last active July 30, 2019 17:19
Show Gist options
  • Save matthuhiggins/16b5d29c2ea0bb4a9e21f64268b6865f to your computer and use it in GitHub Desktop.
Save matthuhiggins/16b5d29c2ea0bb4a9e21f64268b6865f to your computer and use it in GitHub Desktop.
Multiple tokens in a synonym analyzer within a multiplexer
curl -X DELETE "localhost:9200/companies"
curl -X PUT "localhost:9200/companies" -H 'Content-Type: application/json' -d'
{
"settings": {
"analysis": {
"analyzer": {
"company_name_analyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": ["company_name_filter"]
}
},
"filter": {
"company_name_filter": {
"type": "multiplexer",
"filters": ["company_name_abbreviations"]
},
"company_name_abbreviations": {
"type": "synonym",
"synonyms": ["co-op => cooperative", "corp => corporation"]
}
}
}
}
}'
# This will not work. Returns ["co op"]
curl -X GET "localhost:9200/companies/_analyze" -H 'Content-Type: application/json' -d'
{
"analyzer": "company_name_analyzer",
"text": "co-op"
}'
# This will work. Returns ["corporation"]
curl -X GET "localhost:9200/companies/_analyze" -H 'Content-Type: application/json' -d'
{
"analyzer": "company_name_analyzer",
"text": "corp"
}'
@matthuhiggins
Copy link
Author

This is fixed by removing the "multiplexer" wrapper and directly setting "filters": "company_name_abbreviations" to the analyzer.

This example is distilled to show the bug. My full use case requires a multiplexer.

@romseygeek
Copy link

This is annoying, it's because SynonymGraphFilter can't handle incoming graphs yet, which is a lucene-level issue. I do plan on working on a fix for this at some point, but it's not a trivial problem unfortunately.

@matthuhiggins
Copy link
Author

Bummer, thanks.

@matthuhiggins
Copy link
Author

Apologies for bringing this up again. The nuanced problem above is blocking an ES 7 upgrade. I am unsure what you meant by the graph filter not handling incoming graphs, as my problem is exposing what this warning describes:

multi-word synonym token filters will not function normally when they are declared in the filters array because they read ahead internally which is unsupported by the multiplexer

Is this the same problem as not supporting an incoming graph?

Currently, my best option is to pre-process the text in my code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment