Skip to content

Instantly share code, notes, and snippets.

View jackzhangsir's full-sized avatar

icanfly520 jackzhangsir

View GitHub Profile
@rafaelkyrdan
rafaelkyrdan / DataSourceV2
Created August 17, 2020 16:09
How to migrate DataSourceV2 into Spark 3.0.0
// 1.
// Cannot find what should I use instead of ReadSupport, ReadSupport, DataSourceReader
// if instead of ReadSupport we have to use now Scan then what happened to method createReader?
class DefaultSource extends ReadSupport {
override def createReader(options: DataSourceOptions): DataSourceReader = new GeneratingReader()
}
class GeneratingReader() extends DataSourceReader {
override def readSchema(): StructType = {...}
@tailnode
tailnode / trimJson.go
Created April 13, 2017 06:38
golang 将复杂 struct 输出为 JSON 时过滤掉不需要字段(不修改原 struct)
package main
import (
"encoding/json"
"fmt"
)
type T1 struct {
Field1 string `json:"field_1"`
Field2 []int `json:"field_2"`
@superalsrk
superalsrk / PathFilter.java
Last active March 2, 2023 08:17
PathFilter.java
package com.miaozhen.dataeye.input;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.PathFilter;
/**