Last active
May 22, 2020 14:15
-
-
Save paulsonnentag/201b8a13cba8ba91e384240cf26c63f1 to your computer and use it in GitHub Desktop.
Scala example Avro schema modifcation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
// ... | |
// creating a new schema with the fields of the old schema added plus the new fields | |
val schema = // ... the schema of the input data | |
var newSchema = SchemaBuilder | |
.builder(schema.getNamespace) | |
.record(schema.getName) | |
.fields() | |
// create new schema with existing fields from schemas and new fields which are created through transforms | |
val fields = schema.getFields ++ getNewFields(schema, transforms) | |
fields | |
.foldLeft(newSchema)((newSchema, field: Schema.Field) => { | |
newSchema | |
.name(field.name) | |
.`type`(field.schema()) | |
.noDefault() | |
// TODO: find way to differentiate between explicitly set null defaults and fields which have no default | |
//.withDefault(field.defaultValue()) | |
}) | |
newSchema.endRecord() | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
// ... | |
// create new fields like this | |
new Schema.Field( | |
"addedField", | |
Schema.createUnion(List( | |
Schema.create(Schema.Type.STRING), | |
Schema.create(Schema.Type.NULL) | |
)), | |
null, | |
null | |
) |
btw, would it be possible to access particular fields by name and add default value as null and update the Schema Type
I'm sorry, I no longer have the full source code. I don't even remember the exact context when I wrote this code. I hope you find a solution.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi Paul
I am trying to handle avro schema evolution, I am thinking the example you wrote might help.
If possible could you please share the entire code of this. thanks in advance.