pdeffebach

New DataFramesMeta.jl behavior

Goal

The goal of this document is to outline proposed new behavior for DataFramesMeta and it's integration with the new DataFrames.jl "piping" functions, e.g. transform, select, combine, and filter.

Limitations of current implementation

1. The most glaring limitation of the current behavior of DataFramesMeta is

pdeffebach

module MWEMissings

using Missings, InteractiveUtils

export TwoVectors, mapreduce_impl_bad, mapreduce_impl_good

struct TwoVectors{V}
    x::V
    others::V
end

pdeffebach

using BenchmarkTools, Missings

function make_x_y(N, p)
    x = map(1:N) do _
      rand() < p ? missing : rand()
    end
    y = map(1:N) do _
        rand() < p ? missing : rand()
    end
    sx = skipmissing(x)

pdeffebach

# Question 5
module Q5
using LinearAlgebra, Distributions, DataFrames

function make_data(T, N, β, σ)
	T = map(1:T*N) do i 
		x = rand(Normal(0, sqrt(σ)))
		u = rand(Normal(0, abs(x)))
		y = x * β + u
		x, y

pdeffebach

using DataFrames, DataFramesMeta, BenchmarkTools

function scalarfun(g::GroupedDataFrame)
    @transform(g, t = :b[1])
end

function vectorfun(g::GroupedDataFrame)
    @transform(g, t = :b .- 1.0)
end