| #Efficient fuzzy match of two data frames by one common column | |
| library(dplyr) | |
| library(fuzzyjoin) | |
| library(stringdist) | |
| eff_fuzzy_match<-function(data_frame_A, | |
| data_frame_B, | |
| by_what, | |
| choose_p = 0.1, | |
| choose_max_dist = 0.4, |
This configuration worked for me, hope it helps
It is based on: https://becominghuman.ai/deep-learning-gaming-build-with-nvidia-titan-xp-and-macbook-pro-with-thunderbolt2-5ceee7167f8b
and on: https://stackoverflow.com/questions/44744737/tensorflow-mac-os-gpu-support
The dplyr package in R makes data wrangling significantly easier.
The beauty of dplyr is that, by design, the options available are limited.
Specifically, a set of key verbs form the core of the package.
Using these verbs you can solve a wide range of data problems effectively in a shorter timeframe.
Whilse transitioning to Python I have greatly missed the ease with which I can think through and solve problems using dplyr in R.
The purpose of this document is to demonstrate how to execute the key dplyr verbs when manipulating data using Python (with the pandas package).
dplyr is organised around six key verbs:
This is a list inspired by some of our current or potential lines of work at the World Bank Innovation Labs. The “Innovations in Big Data Analytics” program helps to strengthen the World Bank capabilities to effectively use big data in its operational and strategic work.
We are always looking for great Data Scientists. If you can solve any of these [using open software], you'll be heads down helping us from day one. Email us to brunosanchez@worldbank.org
(This list is updated frequently).
We are building an open stack to process nightly data from satellite and query light output from all known villages. Currently we are doing 20 years of nightly data for 600,000 villages in India.
Some useful links relating to performing Google Earth Engine development / project work.
| lmOut <- function(res, file="test.csv", ndigit=3, writecsv=T) { | |
| # If summary has not been run on the model then run summary | |
| if (length(grep("summary", class(res)))==0) res <- summary(res) | |
| co <- res$coefficients | |
| nvar <- nrow(co) | |
| ncol <- ncol(co) | |
| f <- res$fstatistic | |
| formatter <- function(x) format(round(x,ndigit),nsmall=ndigit) | |