Skip to content

Instantly share code, notes, and snippets.

👩‍🚀
you are on a branch yet to be born

Brendan O'Brien b5

👩‍🚀
you are on a branch yet to be born
Block or report user

Report or block b5

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
View logs_readme.md

Log

Qri is a version control system (VCS) for datasets. Since most VCS's provide some system for collaborating, we aim to do the same. Collaboration within the context of a version control system means coordinating who has done what. Providing tools to syncronize collaborator's actions is a prerequisite for working together effectively.

The characteristics we want:

  • provide a foundation for about version histories.
  • trak & map to human names
  • Decentralized.
  • Offline first. Accomodate sync lags that may be years in length
  • as-small-as-possible storage footprint, size must be calculatable
View download.py
load("http.star", "http")
load("bsoup.star", "bsoup")
baseUrl = "http://www.lawhelp.org"
def download(ctx):
soup, rows = fetch_page(baseUrl + '/dc/find-legal-help/directory/')
pages = page_links(soup)
@b5
b5 / gov.noaa.esrl-walk.job.json
Last active Apr 24, 2019
github.com/qri-io/walk job
View gov.noaa.esrl-walk.job.json
{
"Seeds": [
"https://esrl.noaa.gov/"
],
"SeedsPath": "",
"Crawl": true,
"Domains": [
"https://esrl.noaa.gov/"
],
"IgnorePatterns": null,
@b5
b5 / gov.epa.yosemite-walk.job.json
Created Apr 24, 2019
github.com/qri-io/walk job
View gov.epa.yosemite-walk.job.json
{
"Seeds": [
"https://yosemite.epa.gov"
],
"SeedsPath": "",
"Crawl": true,
"Domains": [
"https://yosemite.epa.gov"
],
"IgnorePatterns": null,
@b5
b5 / labels.json
Created Feb 27, 2019
qri github labels
View labels.json
[
{
"name": "bug",
"color": "ee0701"
},
{
"name": "chore",
"color": "14272D"
},
{
@b5
b5 / prefix-postfix.go
Created Feb 25, 2019
Top Down & Bottom Up Tree Traversal
View prefix-postfix.go
package main
import "fmt"
// Node is a named element of a tree witha any number of children
type Node struct {
Name string
Children []Node
}
@b5
b5 / geo.py
Created Jan 22, 2019
Starlark Geospatial sample
View geo.py
# this is a Qri starlark transform script, use it with qri: https://qri.io
load("http.star", "http")
load("geo.star", "geo")
def download(ctx):
# Download list of 311 complaints, currently capped to 10000 responses for testing purposes
complaints = http.get("https://data.cityofnewyork.us/resource/fhrw-4uyv.json?$offset=0&$limit=10000")
# Download the New York Borough Boundaries
boros = http.get("http://data.beta.nyc//dataset/68c0332f-c3bb-4a78-a0c1-32af515892d6/resource/7c164faa-4458-4ff2-9ef0-09db00b509ef/download/42c737fd496f4d6683bba25fb0e86e1dnycboroughboundaries.geojson")
@b5
b5 / config.json
Created Dec 17, 2018
walk test config
View config.json
{
"Badger": {
"Dir": "badger",
"ValueDir": "badger",
"SyncWrites": true,
"TableLoadingMode": 1,
"ValueLogLoadingMode": 2,
"NumVersionsToKeep": 1,
"MaxTableSize": 67108864,
"LevelSizeMultiplier": 10,
@b5
b5 / skylark_transformations_tutorial.md
Created Jun 11, 2018
Qri Skylark Transformations Tutorial
View skylark_transformations_tutorial.md

Qri ("query") is about datasets. A transformion is a repeatable script for generating a dataset. Skylark is a scripting langauge from Google that feels a lot like python. This package implements skylark as a transformation syntax. Skylark tranformations are about as close as one can get to the full power of a programming language as a transformation syntax. Often you need this degree of control to generate a dataset.

Typical examples of a skylark transformation include:

  • combining paginated calls to an API into a single dataset
  • downloading unstructured structured data from the internet to extract
  • re-shaping raw input data before saving a dataset

We're excited about skylark for a few reasons:

  • python syntax - many people working in data science these days write python, we like that, skylark likes that. dope.
  • deterministic subset of python - unlike python, skylark removes properties that reduce introspection into code behaviour
@b5
b5 / cr_to_crlf_replacer.go
Created Feb 27, 2018
Dealing with Solo Carriage Returns in csv.Reader
View cr_to_crlf_replacer.go
package main
import (
"bufio"
"bytes"
"fmt"
"io"
"encoding/csv"
)
You can’t perform that action at this time.