This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
package net.cloudburo.avro.registry; | |
import com.google.gson.JsonObject; | |
import org.apache.avro.Schema; | |
import net.cloudburo.elasticsearch.ESPersistencyManager; | |
public class ElasticSearchSchemaRegistry extends SchemaRegistry { | |
private static final String esIndex = "avroschema"; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
version: '2' | |
services: | |
db: | |
image: mariadb | |
restart: always | |
volumes: | |
- c:/Users/talfc/DockerVolumes/nextcloud/mysql:/var/lib/mysql | |
environment: | |
- MYSQL_ROOT_PASSWORD=gugus |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Challenge | Examples | Strategy | |
---|---|---|---|
Phonetic Similarity | Klaus <-> Claus | double_metaphone | |
Missing Spaces&Hyphens | PeterLow, Anne Meier-King | normalize_unicode_to_ascii | |
Missing Components | Peter James Low <-> Peter Low | <no solution - manual check> | |
Titles & Honorifics | Dr. Peter Low, Ms. Anne Meier | __calculate_name_matching | |
Out of Order Components | Peter James Low <-> Low Peter James | sort_words | |
Nicknames | Wiiliam <-> Will <-> Bill | <no solution - manual check> | |
Multiple Languages | José Müller <-> Jose Müller | normalize_unicode_to_ascii | |
Initials | James Earl Smith <-> J.E. Smith | __calculate_name_matching | |
Typos | Jamse Earl Smith | double_metaphone |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
public String persist(byte[] msg, String index, String type, String id) throws IOException { | |
if (checkForAvroSingleObjectEncoding(msg)) { | |
String jsonDoc = convertAvroBinaryToJSON(msg); | |
esPersistencyManager.createUpdateDocument(index,type,jsonDoc,id); | |
return jsonDoc; | |
} | |
else { | |
logger.error("Received message wasn't Avro Single Object encoded"); | |
throw new IOException("Received message wasn't Avro Single Object encoded"); | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
public String persist(byte[] msg, String index, String type, String id) throws IOException { | |
if (checkForAvroSingleObjectEncoding(msg)) { | |
String jsonDoc = convertAvroBinaryToJSON(msg); | |
esPersistencyManager.createUpdateDocument(index,type,jsonDoc,id); | |
return jsonDoc; | |
} | |
else { | |
logger.error("Received message wasn't Avro Single Object encoded"); | |
throw new IOException("Received message wasn't Avro Single Object encoded"); | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
private String convertAvroBinaryToJSON(byte[] msg) throws IOException { | |
long msgFingerprint = getAvroFingerprint(msg); | |
Schema mgsSchema = registry.getSchema(msgFingerprint); | |
byte[] payload = extractPayload(msg); | |
DatumReader<Object> reader = null; | |
if (msgFingerprint == this.backendComponentSchemaFingerprint) | |
reader = new GenericDatumReader<>(mgsSchema); | |
else | |
reader = new GenericDatumReader<>(mgsSchema,this.backendComponentSchema); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{ "namespace": "business.model.strategy", | |
"type": "record", | |
"doc" : "Schema for the Coporate Business Model Strategy Part", | |
"name": "BusinessModelStrategy", | |
"fields": [ | |
{ "name": "avro_fingerprint", "type": "long"}, | |
{ "name": "index_ipid", "type": "string"}, | |
{ "name": "last_update_timestamp", "type": "long", "logical-type":"time-micros"}, | |
{ "name": "last_update_loginid", "type": "string"}, | |
{ "name": "client_ipid", "type": "string"}, |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
public JsonObject readDocumentByIdAsObject(String index, String id) throws IOException{ | |
return esClient.execute(new Get.Builder(index, id).build()).getJsonObject(); | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
public void createUpdateDocument(String index, String type, String docJson, String id) throws IOException { | |
Index esIndex; | |
if (id == null) { | |
esIndex = new Index.Builder(docJson).index(index).type(type).build(); | |
} else { | |
esIndex = new Index.Builder(docJson).index(index).type(type).id(id).build(); | |
} | |
JestResult jestResult = esClient.execute(esIndex); | |
if(jestResult.isSucceeded()) { | |
logger.info("Document persisted"); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{ "namespace": "business.model.strategy", | |
"type": "record", | |
"doc" : "A simple Avro example schema", | |
"name": "Person", | |
"fields": [ | |
{ "name": "personid", "type": "long", | |
"name": "lastname", "type": "string", | |
"name": "surname", "type": [ "null", { "type": "string" }],"default": null} | |
] | |
} |
NewerOlder