Skip to content

Instantly share code, notes, and snippets.

@paulsonnentag
Last active May 22, 2020 14:15
Show Gist options
  • Save paulsonnentag/201b8a13cba8ba91e384240cf26c63f1 to your computer and use it in GitHub Desktop.
Save paulsonnentag/201b8a13cba8ba91e384240cf26c63f1 to your computer and use it in GitHub Desktop.
Scala example Avro schema modifcation
// ...
// creating a new schema with the fields of the old schema added plus the new fields
val schema = // ... the schema of the input data
var newSchema = SchemaBuilder
.builder(schema.getNamespace)
.record(schema.getName)
.fields()
// create new schema with existing fields from schemas and new fields which are created through transforms
val fields = schema.getFields ++ getNewFields(schema, transforms)
fields
.foldLeft(newSchema)((newSchema, field: Schema.Field) => {
newSchema
.name(field.name)
.`type`(field.schema())
.noDefault()
// TODO: find way to differentiate between explicitly set null defaults and fields which have no default
//.withDefault(field.defaultValue())
})
newSchema.endRecord()
}
// ...
// create new fields like this
new Schema.Field(
"addedField",
Schema.createUnion(List(
Schema.create(Schema.Type.STRING),
Schema.create(Schema.Type.NULL)
)),
null,
null
)
@nimnp6583
Copy link

Hi Paul
I am trying to handle avro schema evolution, I am thinking the example you wrote might help.
If possible could you please share the entire code of this. thanks in advance.

@nimnp6583
Copy link

btw, would it be possible to access particular fields by name and add default value as null and update the Schema Type

@paulsonnentag
Copy link
Author

I'm sorry, I no longer have the full source code. I don't even remember the exact context when I wrote this code. I hope you find a solution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment