This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import com.bouncex.ingestion.ProductFeedOptions; | |
import com.google.cloud.bigtable.beam.CloudBigtableIO; | |
import com.google.cloud.bigtable.beam.CloudBigtableScanConfiguration; | |
import com.google.cloud.bigtable.beam.CloudBigtableTableConfiguration; | |
import com.google.cloud.bigtable.data.v2.models.Row; | |
import org.apache.beam.sdk.Pipeline; | |
import org.apache.beam.sdk.io.BoundedSource; | |
import org.apache.beam.sdk.io.FileSystems; | |
import org.apache.beam.sdk.io.gcp.bigtable.BigtableIO; | |
import org.apache.beam.sdk.options.PipelineOptions; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
PIPELINE STEPS | |
- read most recent product feed data from GCS (stream A) | |
- read second most recent product feed data from GCS IF IT EXISTS (stream B) | |
- if only stream A | |
- write to new stream with the price drop data -> write to Bigtable | |
- write to new stream with product item data -> write to cloudSQL | |
- flatten stream A and stream B -> stream C | |
- stream C -> groupby SKU | |
- for each item, get diff and: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import com.bouncex.ingestion.Product; | |
import org.apache.beam.sdk.io.FileSystems; | |
import org.apache.beam.sdk.io.fs.ResourceId; | |
import org.apache.commons.compress.utils.IOUtils; | |
import java.io.IOException; | |
import java.io.InputStream; | |
import java.io.OutputStream; | |
import java.net.URL; | |
import java.nio.channels.Channels; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
### Keybase proof | |
I hereby claim: | |
* I am muhang on github. | |
* I am muhang (https://keybase.io/muhang) on keybase. | |
* I have a public key ASD6sslOoJzmttSw5Iu52PET43___VYb4H7m9pS0im_FYgo | |
To claim this, I am signing this object: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
contentSecurityPolicy: { | |
'default-src': "'none'", | |
'script-src': "'self' 'unsafe-eval' 'unsafe-inline' use.typekit.net https://script.crazyegg.com https://js.stripe.com http://cdn.ravenjs.com http://www.google-analytics.com https://maps.googleapis.com https://mts1.googleapis.com http://www.googleadservices.com/pagead/conversion_async.js", | |
'font-src': "'self' data://* use.typekit.net https://csi.gstatic.com https://fonts.gstatic.com", | |
'connect-src': "'self' http://localhost:9000 https://api-sandbox.billforward.net/v1/tokenization/pre-auth https://api-sandbox.billforward.net/v1/tokenization/auth-capture https://app.getsentry.com http://roost.freshdesk.com/helpdesk/tickets.json", | |
'img-src': "'self' p.typekit.net https://s3-us-east-1.amazonaws.com//AR-logos https://s3.amazonaws.com https://go.goroost.com data: app.getsentry.com http://www.google-analytics.com https://maps.gstatic.com https://csi.gstatic.com https://maps.googleapis.com https://mts1.googleapis.com https://mts0.googl |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
set nocompatible " be iMproved, required | |
" Pathogen installs | |
" | |
" Functionality | |
" ------------- | |
" | |
" ag.vim Smart code search with :Ag using the_silver_searcher | |
" YouCompleteMe Intelligent autocompletion -- requires setup | |
" emmet-vim Emmet syntax expansion with <Ctrl-y> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
require 'formula' | |
class Vim < Formula | |
homepage 'http://www.vim.org/' | |
url 'https://vim.googlecode.com/hg/', :revision => 'ca3db36a6ed8' | |
version '7.4.553' | |
def features; %w(tiny small normal big huge) end | |
def interp; %w(lua mzscheme perl python python3 tcl ruby) end |