Skip to content

Instantly share code, notes, and snippets.

View mwebler's full-sized avatar

Matheus Webler mwebler

10 followers · 9 following
  • Shopify
  • Calgary
View GitHub Profile
@mwebler
mwebler / logical_expression_query.py
Created July 20, 2018 04:51
Apache Beam Python: Google Datastore Query with composite filter
import apache_beam as beam
from apache_beam.io.gcp.datastore.v1.datastoreio import ReadFromDatastore
from google.cloud.proto.datastore.v1 import query_pb2
from googledatastore import helper as datastore_helper
from googledatastore import PropertyFilter
p = beam.Pipeline(options=pipeline_options)
@mwebler
mwebler / query_by_property.py
Created July 20, 2018 04:49
Apache Beam Python: Google Datastore Query with Property Filter
import apache_beam as beam
from apache_beam.io.gcp.datastore.v1.datastoreio import ReadFromDatastore
from google.cloud.proto.datastore.v1 import query_pb2
from googledatastore import helper as datastore_helper
from googledatastore import PropertyFilter
p = beam.Pipeline(options=pipeline_options)
@mwebler
mwebler / query_by_kind.py
Last active July 20, 2018 04:51
Apache Beam Python: Google Datastore Querying by kind
import apache_beam as beam
from apache_beam.io.gcp.datastore.v1.datastoreio import ReadFromDatastore
from google.cloud.proto.datastore.v1 import query_pb2
p = beam.Pipeline(options=pipeline_options)
# Create a query and filter by kind
query = query_pb2.Query()
@mwebler
mwebler / firebase-functions-debug.md
Last active May 17, 2018 03:14
Debug firebase http functions v1.0 locally with VS Code

You will need to have google cloud functions emulator modules installed:

npm install -g @google-cloud/functions-emulator

Then you will need to get your firebase project configuration. For that, you need to go to your firebase functions project folder, log in to your project and run the setup:web command:

firebase login
firebase setup:we
@mwebler
mwebler / json_file_source.py
Created January 9, 2018 23:54
Google datastore json filesource for big files
import apache_beam as beam
import re
import json
class JsonFileSource(beam.io.filebasedsource.FileBasedSource):
"""A Beam source for JSON that emits all top-level json objects.
Note that the beginning of a top-level json object is assumed to begin on a
new line with no leading spaces, possibly preceded by a square bracket ([)
@mwebler
mwebler / GoogleDatastoreCount.py
Created December 12, 2017 21:42
Iterate and count entities from google cloud datastore
# Imports the Google Cloud client library
from google.cloud import datastore
# Instantiates a client
client = datastore.Client(project='PROJECT')
query = client.query(kind='KIND')
query_iter = query.fetch()
total = sum(1 for _ in query_iter)
@mwebler
mwebler / CloudSQLSource.py
Created December 11, 2017 23:44
Google Dataflow source for Cloud SQL
import threading
import mysql.connector
import apache_beam as beam
class CloudSQLSource(beam.io.iobase.BoundedSource):
def __init__(self, node):
sql = mysql.connector.connect(user='', password='',
@mwebler
mwebler / index.js
Created June 1, 2017 13:29
yeoman-environment-error
const yeoman = require('yeoman-environment');
const env = yeoman.createEnv();
env.lookup(() => {
env.run('generator', {'skip-install': true, 'force':true}, err => {
console.log('done');
env.run('generator', {'skip-install': true, 'force':true}, err => {
console.log('done');
});
});
@mwebler
mwebler / index.html
Created March 4, 2017 00:35
Simple initial HTML file for scaffolding/bootstrapping
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>HTML Init</title>
<meta name="description" content="HTML scaffold">
<meta name="author" content="HTML Init">
</head>
@mwebler
mwebler / python_resources.md
Created January 21, 2014 15:27 — forked from jookyboi/python_resources.md
Python-related modules and guides.

Packages

  • lxml - Pythonic binding for the C libraries libxml2 and libxslt.
  • boto - Python interface to Amazon Web Services
  • Django - Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design.
  • Fabric - Library and command-line tool for streamlining the use of SSH for application deployment or systems administration task.
  • PyMongo - Tools for working with MongoDB, and is the recommended way to work with MongoDB from Python.
  • Celery - Task queue to distribute work across threads or machines.
  • pytz - pytz brings the Olson tz database into Python. This library allows accurate and cross platform timezone calculations using Python 2.4 or higher.

Guides