Skip to content

Instantly share code, notes, and snippets.

@ajnavarro

ajnavarro/test.ipynb

Created November 22, 2017 11:25
Show Gist options
  • Save ajnavarro/5d4cd23befe9f763f5153fb0f7dc3a4f to your computer and use it in GitHub Desktop.
Save ajnavarro/5d4cd23befe9f763f5153fb0f7dc3a4f to your computer and use it in GitHub Desktop.
Download ZIP
Test notebook example
Display the source blob
Display the rendered blob
Raw
test.ipynb
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"from sourced.engine import Engine\n",
"from pyspark.sql import SparkSession\n",
"from pyspark.sql.functions import *\n",
"\n",
"spark = SparkSession.builder\\\n",
".master(\"local[*]\").appName(\"Examples\")\\\n",
".getOrCreate()\n",
"\n",
"engine = Engine(spark, \"/repositories\")"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"+--------------------+--------------------+-------+\n",
"| id| urls|is_fork|\n",
"+--------------------+--------------------+-------+\n",
"|github.com/soimor...|[https://github.c...| false|\n",
"|github.com/donnem...|[https://github.c...| false|\n",
"|github.com/chrisa...|[https://github.c...| false|\n",
"|github.com/geekan...|[https://github.c...| false|\n",
"|github.com/mailpi...|[https://github.c...| false|\n",
"|github.com/maxme/...|[https://github.c...| false|\n",
"|github.com/pybee/...|[https://github.c...| false|\n",
"|github.com/redece...|[https://github.c...| false|\n",
"|github.com/openst...|[https://github.c...| false|\n",
"|github.com/justqu...|[https://github.c...| false|\n",
"|github.com/Reacti...|[https://github.c...| false|\n",
"|github.com/minima...|[https://github.c...| false|\n",
"|github.com/Kinto/...|[https://github.c...| false|\n",
"|github.com/jazzba...|[https://github.c...| false|\n",
"|github.com/ricequ...|[https://github.c...| false|\n",
"|github.com/jdf/pr...|[https://github.c...| false|\n",
"|github.com/lennyl...|[https://github.c...| false|\n",
"|github.com/scipy/...|[https://github.c...| false|\n",
"|github.com/GetBli...|[https://github.c...| false|\n",
"|github.com/phusio...|[https://github.c...| false|\n",
"|github.com/Capgem...|[https://github.c...| false|\n",
"|github.com/sqlmap...|[https://github.c...| false|\n",
"|github.com/deepmi...|[https://github.c...| false|\n",
"|github.com/apenwa...|[https://github.c...| false|\n",
"|github.com/banga/...|[https://github.c...| false|\n",
"|github.com/borgba...|[https://github.c...| false|\n",
"|github.com/USArmy...|[https://github.c...| false|\n",
"|github.com/jarun/...|[https://github.c...| false|\n",
"|github.com/cloude...|[https://github.c...| false|\n",
"|github.com/Netfli...|[https://github.c...| false|\n",
"|github.com/qiyebo...|[https://github.c...| false|\n",
"|github.com/imwils...|[https://github.c...| false|\n",
"|github.com/wavefo...|[https://github.c...| false|\n",
"|github.com/OmkarP...|[https://github.c...| false|\n",
"|github.com/devtty...|[https://github.c...| false|\n",
"|github.com/lincol...|[https://github.c...| false|\n",
"|github.com/randy3...|[https://github.c...| false|\n",
"|github.com/lisa-l...|[https://github.c...| false|\n",
"|github.com/Kozea/...|[https://github.c...| false|\n",
"|github.com/hack4i...|[https://github.c...| false|\n",
"|github.com/DonJay...|[https://github.c...| false|\n",
"|github.com/fcholl...|[https://github.c...| false|\n",
"|github.com/Kozea/...|[https://github.c...| false|\n",
"|github.com/Roches...|[https://github.c...| false|\n",
"|github.com/guille...|[https://github.c...| false|\n",
"|github.com/jcjohn...|[https://github.c...| false|\n",
"|github.com/mitsuh...|[https://github.c...| false|\n",
"|github.com/elasti...|[https://github.c...| false|\n",
"|github.com/alexjc...|[https://github.c...| false|\n",
"|github.com/django...|[https://github.c...| false|\n",
"|github.com/JakeWh...|[https://github.c...| false|\n",
"|github.com/entrop...|[https://github.c...| false|\n",
"|github.com/rpicar...|[https://github.c...| false|\n",
"|github.com/coala/...|[https://github.c...| false|\n",
"|github.com/yandex...|[https://github.c...| false|\n",
"|github.com/Conchy...|[https://github.c...| false|\n",
"|github.com/oliver...|[https://github.c...| false|\n",
"|github.com/pwniee...|[https://github.c...| false|\n",
"|github.com/fail2b...|[https://github.c...| false|\n",
"|github.com/lauris...|[https://github.c...| false|\n",
"|github.com/fxsjy/...|[https://github.c...| false|\n",
"|github.com/emre/s...|[https://github.c...| false|\n",
"|github.com/marshm...|[https://github.c...| false|\n",
"|github.com/nument...|[https://github.c...| false|\n",
"|github.com/sindre...|[https://github.c...| false|\n",
"|github.com/n1nj4s...|[https://github.c...| false|\n",
"|github.com/awesto...|[https://github.c...| false|\n",
"|github.com/seb-m/...|[https://github.c...| false|\n",
"|github.com/gitpyt...|[https://github.c...| false|\n",
"|github.com/getsen...|[https://github.c...| false|\n",
"|github.com/sublim...|[https://github.c...| false|\n",
"|github.com/LiuLan...|[https://github.c...| false|\n",
"|github.com/Yelp/e...|[https://github.c...| false|\n",
"|github.com/tensor...|[https://github.c...| false|\n",
"|github.com/facele...|[https://github.c...| false|\n",
"|github.com/lk-gei...|[https://github.c...| false|\n",
"|github.com/eliben...|[https://github.c...| false|\n",
"|github.com/billry...|[https://github.c...| false|\n",
"|github.com/pandas...|[https://github.c...| false|\n",
"|github.com/rushte...|[https://github.c...| false|\n",
"|github.com/edx/ed...|[https://github.c...| false|\n",
"|github.com/mail-i...|[https://github.c...| false|\n",
"|github.com/google...|[https://github.c...| false|\n",
"|github.com/ternjs...|[https://github.c...| false|\n",
"|github.com/home-a...|[https://github.c...| false|\n",
"|github.com/eudico...|[https://github.c...| false|\n",
"|github.com/sdg-mi...|[https://github.c...| false|\n",
"|github.com/disqus...|[https://github.c...| false|\n",
"|github.com/gleitz...|[https://github.c...| false|\n",
"|github.com/sherji...|[https://github.c...| false|\n",
"|github.com/jarun/...|[https://github.c...| false|\n",
"|github.com/ageitg...|[https://github.c...| false|\n",
"|github.com/Yelp/p...|[https://github.c...| false|\n",
"|github.com/martin...|[https://github.c...| false|\n",
"|github.com/rubik/...|[https://github.c...| false|\n",
"|github.com/realpy...|[https://github.c...| false|\n",
"|github.com/mahmou...|[https://github.c...| false|\n",
"|github.com/DEAP/deap|[https://github.c...| false|\n",
"|github.com/warner...|[https://github.c...| false|\n",
"|github.com/python...|[https://github.c...| false|\n",
"|github.com/xiyouM...|[https://github.c...| false|\n",
"|github.com/entrop...|[https://github.c...| false|\n",
"|github.com/tangqi...|[https://github.c...| false|\n",
"|github.com/shiden...|[https://github.c...| false|\n",
"|github.com/miguel...|[https://github.c...| false|\n",
"|github.com/google...|[https://github.c...| false|\n",
"|github.com/flask-...|[https://github.c...| false|\n",
"|github.com/gongji...|[https://github.c...| false|\n",
"|github.com/hypero...|[https://github.c...| false|\n",
"|github.com/Fantom...|[https://github.c...| false|\n",
"|github.com/redodo...|[https://github.c...| false|\n",
"|github.com/espres...|[https://github.c...| false|\n",
"|github.com/django...|[https://github.c...| false|\n",
"|github.com/yhat/ggpy|[https://github.c...| false|\n",
"|github.com/jisaac...|[https://github.c...| false|\n",
"|github.com/pypa/s...|[https://github.c...| false|\n",
"|github.com/ethere...|[https://github.c...| false|\n",
"|github.com/shazow...|[https://github.c...| false|\n",
"|github.com/laiwei...|[https://github.c...| false|\n",
"|github.com/trentm...|[https://github.c...| false|\n",
"|github.com/getred...|[https://github.c...| false|\n",
"|github.com/aws/ch...|[https://github.c...| false|\n",
"|github.com/jtrile...|[https://github.c...| false|\n",
"|github.com/golemf...|[https://github.c...| false|\n",
"|github.com/reorx/...|[https://github.c...| false|\n",
"|github.com/zedsha...|[https://github.c...| false|\n",
"|github.com/instab...|[https://github.c...| false|\n",
"|github.com/swdunl...|[https://github.c...| false|\n",
"|github.com/mniels...|[https://github.c...| false|\n",
"| github.com/bup/bup|[https://github.c...| false|\n",
"|github.com/samuel...|[https://github.c...| false|\n",
"|github.com/mozill...|[https://github.c...| false|\n",
"|github.com/hypero...|[https://github.c...| false|\n",
"|github.com/common...|[https://github.c...| false|\n",
"|github.com/larsen...|[https://github.c...| false|\n",
"|github.com/snare/...|[https://github.c...| false|\n",
"|github.com/davids...|[https://github.c...| false|\n",
"|github.com/ponyor...|[https://github.c...| false|\n",
"|github.com/somera...|[https://github.c...| false|\n",
"|github.com/mkdocs...|[https://github.c...| false|\n",
"|github.com/spulec...|[https://github.c...| false|\n",
"|github.com/aponxi...|[https://github.c...| false|\n",
"|github.com/mathic...|[https://github.c...| false|\n",
"|github.com/KeepSa...|[https://github.c...| false|\n",
"|github.com/hack4i...|[https://github.c...| false|\n",
"|github.com/crista...|[https://github.c...| false|\n",
"|github.com/ckreib...|[https://github.c...| false|\n",
"|github.com/python...|[https://github.c...| false|\n",
"|github.com/fzliu/...|[https://github.c...| false|\n",
"|github.com/nitely...|[https://github.c...| false|\n",
"|github.com/kennet...|[https://github.c...| false|\n",
"|github.com/Keyboa...|[https://github.c...| false|\n",
"|github.com/beetbo...|[https://github.c...| false|\n",
"|github.com/kennet...|[https://github.c...| false|\n",
"|github.com/wepe/M...|[https://github.c...| false|\n",
"|github.com/django...|[https://github.c...| false|\n",
"|github.com/michel...|[https://github.c...| false|\n",
"|github.com/Netfli...|[https://github.c...| false|\n",
"|github.com/DanMcI...|[https://github.c...| false|\n",
"|github.com/RasaHQ...|[https://github.c...| false|\n",
"|github.com/Shangt...|[https://github.c...| false|\n",
"|github.com/sebast...|[https://github.c...| false|\n",
"|github.com/Shangt...|[https://github.c...| false|\n",
"|github.com/twilio...|[https://github.c...| false|\n",
"|github.com/soheil...|[https://github.c...| false|\n",
"|github.com/TheAlg...|[https://github.c...| false|\n",
"|github.com/common...|[https://github.c...| false|\n",
"|github.com/fail2b...|[https://github.c...| false|\n",
"|github.com/agiliq...|[https://github.c...| false|\n",
"|github.com/scipy/...|[https://github.c...| false|\n",
"|github.com/yandex...|[https://github.c...| false|\n",
"|github.com/vmware...|[https://github.c...| false|\n",
"|github.com/deepmi...|[https://github.c...| false|\n",
"|github.com/django...|[https://github.c...| false|\n",
"|github.com/fail2b...|[https://github.c...| false|\n",
"|github.com/bhrigu...|[https://github.c...| false|\n",
"|github.com/worldv...|[https://github.c...| false|\n",
"|github.com/jeanph...|[https://github.c...| false|\n",
"|github.com/matthe...|[https://github.c...| false|\n",
"|github.com/entrop...|[https://github.c...| false|\n",
"|github.com/treeio...|[https://github.c...| false|\n",
"|github.com/aio-li...|[https://github.c...| false|\n",
"|github.com/fabric...|[https://github.c...| false|\n",
"|github.com/thearn...|[https://github.c...| false|\n",
"|github.com/yahoo/...|[https://github.c...| false|\n",
"|github.com/eBay/b...|[https://github.c...| false|\n",
"|github.com/coala/...|[https://github.c...| false|\n",
"|github.com/spadgo...|[https://github.c...| false|\n",
"|github.com/joke2k...|[https://github.c...| false|\n",
"|github.com/Tobler...|[https://github.c...| false|\n",
"|github.com/jorgeb...|[https://github.c...| false|\n",
"|github.com/kovidg...|[https://github.c...| false|\n",
"|github.com/pallet...|[https://github.c...| false|\n",
"|github.com/sightm...|[https://github.c...| false|\n",
"|github.com/Julian...|[https://github.c...| false|\n",
"|github.com/dz0ny/...|[https://github.c...| false|\n",
"|github.com/facebo...|[https://github.c...| false|\n",
"|github.com/pyinvo...|[https://github.c...| false|\n",
"|github.com/jtrile...|[https://github.c...| false|\n",
"|github.com/google...|[https://github.c...| false|\n",
"+--------------------+--------------------+-------+\n",
"only showing top 200 rows\n",
"\n"
]
}
],
"source": [
"engine.repositories.show(200)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment