This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
PCollection<TableRow> bqTableRows = rows.apply( | |
"HashMap to TableRow", | |
ParDo.of(new HashMapToTableRowFn()) | |
).setCoder(TableRowJsonCoder.of()); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AppOptions options = PipelineOptionsFactory.fromArgs(args).withValidation().as(AppOptions.class); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
PipelineOptionsFactory.register(AppOptions.class); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Pipeline pipeline = Pipeline.create(options); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
PGSimpleDataSource pgDataSource = getPostgresDataSource(options); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dependencies { | |
// Apache Beam | |
compile group: 'org.apache.beam', name: 'beam-sdks-java-core', version:'2.9.0' // Core Java SDK | |
compile group: 'org.apache.beam', name: 'beam-sdks-java-io-google-cloud-platform', version:'2.9.0' // Google Cloud IO | |
compile group: 'org.apache.beam', name: 'beam-sdks-java-io-jdbc', version:'2.9.0' // JDBC IO | |
runtime group: 'org.apache.beam', name: 'beam-runners-direct-java', version:'2.9.0' // Beam Direct Runner | |
// Dataflow | |
compile group: 'com.google.cloud.dataflow', name: 'google-cloud-dataflow-java-sdk-all', version: '2.5.0' // Dataflow Java SDK | |
// Logging | |
runtime group: 'org.slf4j', name: 'slf4j-jdk14', version:'1.7.25' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
package com.sandboxws.beam.coders; | |
import com.google.gson.Gson; | |
import java.io.IOException; | |
import java.io.InputStream; | |
import java.io.OutputStream; | |
import java.util.HashMap; | |
import org.apache.beam.sdk.coders.AtomicCoder; | |
import org.apache.beam.sdk.coders.StringUtf8Coder; | |
import org.apache.beam.sdk.values.TypeDescriptor; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
package com.sandboxws.chinook.bigquery.schema; | |
import com.google.api.services.bigquery.model.TableSchema; | |
import com.sandboxws.bigquery.TableSchemaBuilder; | |
/** | |
* Album BigQuery table schema. | |
* | |
* @author Ahmed El.Hussaini | |
*/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
package com.sandboxws.chinook; | |
import java.sql.ResultSet; | |
import java.sql.ResultSetMetaData; | |
import java.sql.SQLException; | |
import java.text.SimpleDateFormat; | |
import java.util.Date; | |
import java.util.HashMap; | |
import java.util.TimeZone; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
package com.sandboxws.bigquery; | |
import com.google.api.services.bigquery.model.TableFieldSchema; | |
import com.google.api.services.bigquery.model.TableSchema; | |
import java.util.ArrayList; | |
import java.util.List; | |
/** | |
* BigQuery table schema builder. | |
* |