Skip to content

Instantly share code, notes, and snippets.


Antonio Piccolboni piccolbo

View GitHub Profile

Putting wings on the Elephant


HBase is used widely at Facebook and one of the biggest usecase is Facebook Messages. With a billion users there are a lot of reliability and performance challenges on both HBase and HDFS. HDFS was originally designed for a batch processing system like MapReduce/Hive. A realtime usecase like Facebook Messages where the p99 latency can`t be more than a couple hundreds of milliseconds poses a lot of challenges for HDFS. In this talk we will share the work the HDFS team at Facebook has done to support a realtime usecase like Facebook Messages : (1) Using system calls to tune performance; (2) Inline checksums to reduce iops by 40%; (3) Reducing the p99 for read and write latencies by about 10x; (4) Tools used to determine root cause of outliers. We will discuss the details of each technique, the challenges we faced, lessons learned and results showing the impact of each improvement.

speaker: Pritam Damania

Real-Time Market Basket Analysis for Retail with

piccolbo / My atom data science
Last active Aug 24, 2019
A list of atom extensions I use for data science.
View My atom data science

Code execution

  • Hydrogen: allows to send any code selection to a kernel and visualize the results in-text, plus watches and more
  • hydrogen-launcher: launch terminal or iPython

The git bundle

Some of this may be superseeded by native git integration:

  • git-blame: find who wrote that cryptic code
piccolbo /
Last active Feb 23, 2022 — forked from audreyfeldroy/
My PyPI Release Checklist 2 (now with bumpversion)
  • merge any development branch you need to merge
  • git checkout master
  • run test
make install-dev
make test
  • when test pass git push
  • Update HISTORY.rst
  • Check readthedocs to make sure docs are OK
piccolbo /
Created Oct 15, 2017
keybase identification

Keybase proof

I hereby claim:

  • I am piccolbo on github.
  • I am piccolbo ( on keybase.
  • I have a public key ASATO-Kj3cWENOHAPB5OgNFMlc4xEUtScX1L0-Er8tYX-Ao

To claim this, I am signing this object:

View emr_spark_thrift_on_yarn
#on cluster
thrift /spark/sbin/ --master yarn-client
#ssh tunnel, direct 10000 to unused 8157
ssh -i ~/caserta-1.pem -N -L
#see this for JDBC config on client
piccolbo /
Last active May 2, 2016
Fully remote business

Criteria: most employess remote all the time. No office space allocated or available. No relocation required. Low travel requirement.

Company Notes
Automattic 4 weeks/year travel all employees
Open Knowledge Foundation Time zone restricted -3:1, different contracts UK/non UK
piccolbo /
Last active Jun 23, 2018
Dplyr backends: the ultimate collection

Dplyr is a well known R package to work on structured data, either in memory or in DB and, more recently, in cluster. The in memory implementations have in general capabilities that are not found in the others, so the notion of backend is used with a bit of a poetic license. Even the different DB and cluster backends differ in subtle ways. But it sure is better than writing SQL directly! Here I provide a list of backends with links to the packages that implement them when necessary. I've done my best to provide links to active projects, but I am not endorsing any of them. Do your own testing. Enjoy and please contribute any corrections or additions, in the comments.

Backend Package
data.frame builtin
data.table builtin
arrays builtin
SQLite builtin
PostgreSQL/Redshift builtin
View names scope.Rmd
ff = function(){}
names(ff) = "abc"
# Error in names(ff) = "abc" : names() applied to a non-vector
#[1] FALSE
names(mtcars) = LETTERS[1:11]
# [1] "A" "B" "C" "D" "E" "F" "G" "H" "I" "J" "K"
View release
  • merge into master
  • update version #
  • update date
  • update Rd help()
  • push master
  • Repeat until tests pass
    • test local and debug
    • test remote and debug
    • test additional platforms
  • apply necessary fixes
piccolbo /
Last active Aug 29, 2015
Vectorized grouped ops in plyrmr

Goal is to expose the vectorize group feature of rmr2 in a plyrmr way


  1. Operations should encapsulate the knowledge of whether they can handle multiple groups. vectorized.reduce should be set accordingly.
  2. vectorized.reduce should be propagated along a pipe when possible. Rules TBD
  3. A repertoire of vectorized reduce ops should be made available, and adding more should be easy (no C++)
  4. Wordcount is our guiding app here.