tl;dr: When calculating the average time it takes to complete a HIT, it may be more appropriate to use the geometric mean or the median instead of the arithmetic mean. You may otherwise spend considerably more money than necessary (in our case 50% more).
(defun tmalsburg-rmarkdown-render () | |
"Compiles the current RMarkdown file to PDF and shows output of | |
the compiler process in a separate window." | |
(interactive) | |
(let* ((buf (get-buffer-create "*rmarkdown-render*")) | |
(temp-window (or (get-buffer-window buf) | |
(split-window-below -10))) | |
(command "Rscript -e 'library(rmarkdown); render(\"%s\", output_format=\"%s\")'") | |
(command (format command (buffer-file-name) "pdf_document"))) | |
(set-window-buffer temp-window buf) |
Ibex does Latin squares in a way that can potentially have serious unintended consequences in the form of spurious effects.
The problem: When you submit an experiment to Amazon Mechanical Turk, a lot of workers will immediately jump at it but the rate of participation quickly decays (the distribution over time often looks like an exponential decay). For every participant, Ibex selects a stimulus list based on an internal counter and this counter is incremented when a participant submits their results. Unfortunately, this means that the initial wave of participants all work on the same list of the Latin square and this list will therefore be strongly overrepresented. This can lead to strong spurious effects that are not due to the experimental manipulation but due to between-item differences. This is an easy-to-miss problem and I would not be surprised if some published results obtained with Ibex were false because of this prob
library(lme4) | |
head(sleepstudy) | |
summary(sleepstudy) | |
# Relabel subjects: | |
d <- sleepstudy | |
d$Subject <- factor(rep(1:18, each=10)) | |
# Fit model: |