Skip to content

Instantly share code, notes, and snippets.

View rafaelkyrdan's full-sized avatar
🏠
Working from home

Rafael rafaelkyrdan

🏠
Working from home
View GitHub Profile
@rafaelkyrdan
rafaelkyrdan / DataSourceV2
Created August 17, 2020 16:09
How to migrate DataSourceV2 into Spark 3.0.0
// 1.
// Cannot find what should I use instead of ReadSupport, ReadSupport, DataSourceReader
// if instead of ReadSupport we have to use now Scan then what happened to method createReader?
class DefaultSource extends ReadSupport {
override def createReader(options: DataSourceOptions): DataSourceReader = new GeneratingReader()
}
class GeneratingReader() extends DataSourceReader {
override def readSchema(): StructType = {...}