Skip to content

Instantly share code, notes, and snippets.

@muhang
muhang / vim.rb
Last active August 29, 2015 14:21 — forked from mgrouchy/vim.rb
require 'formula'
class Vim < Formula
homepage 'http://www.vim.org/'
url 'https://vim.googlecode.com/hg/', :revision => 'ca3db36a6ed8'
version '7.4.553'
def features; %w(tiny small normal big huge) end
def interp; %w(lua mzscheme perl python python3 tcl ruby) end
@muhang
muhang / vimrc
Created December 10, 2015 02:43
vimrc
set nocompatible " be iMproved, required
" Pathogen installs
"
" Functionality
" -------------
"
" ag.vim Smart code search with :Ag using the_silver_searcher
" YouCompleteMe Intelligent autocompletion -- requires setup
" emmet-vim Emmet syntax expansion with <Ctrl-y>
contentSecurityPolicy: {
'default-src': "'none'",
'script-src': "'self' 'unsafe-eval' 'unsafe-inline' use.typekit.net https://script.crazyegg.com https://js.stripe.com http://cdn.ravenjs.com http://www.google-analytics.com https://maps.googleapis.com https://mts1.googleapis.com http://www.googleadservices.com/pagead/conversion_async.js",
'font-src': "'self' data://* use.typekit.net https://csi.gstatic.com https://fonts.gstatic.com",
'connect-src': "'self' http://localhost:9000 https://api-sandbox.billforward.net/v1/tokenization/pre-auth https://api-sandbox.billforward.net/v1/tokenization/auth-capture https://app.getsentry.com http://roost.freshdesk.com/helpdesk/tickets.json",
'img-src': "'self' p.typekit.net https://s3-us-east-1.amazonaws.com//AR-logos https://s3.amazonaws.com https://go.goroost.com data: app.getsentry.com http://www.google-analytics.com https://maps.gstatic.com https://csi.gstatic.com https://maps.googleapis.com https://mts1.googleapis.com https://mts0.googl
### Keybase proof
I hereby claim:
* I am muhang on github.
* I am muhang (https://keybase.io/muhang) on keybase.
* I have a public key ASD6sslOoJzmttSw5Iu52PET43___VYb4H7m9pS0im_FYgo
To claim this, I am signing this object:
import com.bouncex.ingestion.Product;
import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.io.fs.ResourceId;
import org.apache.commons.compress.utils.IOUtils;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URL;
import java.nio.channels.Channels;
PIPELINE STEPS
- read most recent product feed data from GCS (stream A)
- read second most recent product feed data from GCS IF IT EXISTS (stream B)
- if only stream A
- write to new stream with the price drop data -> write to Bigtable
- write to new stream with product item data -> write to cloudSQL
- flatten stream A and stream B -> stream C
- stream C -> groupby SKU
- for each item, get diff and:
import com.bouncex.ingestion.ProductFeedOptions;
import com.google.cloud.bigtable.beam.CloudBigtableIO;
import com.google.cloud.bigtable.beam.CloudBigtableScanConfiguration;
import com.google.cloud.bigtable.beam.CloudBigtableTableConfiguration;
import com.google.cloud.bigtable.data.v2.models.Row;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.io.BoundedSource;
import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.io.gcp.bigtable.BigtableIO;
import org.apache.beam.sdk.options.PipelineOptions;