Serhii Sokolenko datancoffee
howarddierking
/ CsvShredder.java
Created
December 6, 2017 17:20
Sample ParDo transformation for shredding CSV files into a map
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package com.howarddierking.demo; | |
| import org.apache.beam.sdk.io.FileSystems; | |
| import org.apache.beam.sdk.io.fs.ResourceId; | |
| import org.apache.beam.sdk.transforms.DoFn; | |
| import org.apache.commons.csv.CSVFormat; | |
| import org.apache.commons.csv.CSVParser; | |
| import org.apache.commons.csv.CSVRecord; | |
| import org.apache.commons.io.FilenameUtils; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #standardSQL | |
| #Top two Hacker News articles by day | |
| WITH TitlesAndScores AS ( | |
| SELECT | |
| ARRAY_AGG(STRUCT(title, | |
| score)) AS titles, | |
| EXTRACT(DATE | |
| FROM | |
| time_ts) AS date | |
| FROM |