If you have a command line that can generate a result, and you want to test whether successive results are statistically indepenent of each other, you can use the following:
seq <N_TESTS> \
| parallel -n0 <COMMAND...> \
| uniq -c \
| awk '{ NR%2 ? n1+=$1 : n2+=$1 } END { print NR, n1, n2 }' \
| python -c 'from statistics import NormalDist as N; import math; n_runs, n1, n2 = map(int, input().split()); n = n1+n2; μ=1.0*(2*n1*n2)/n + 1; σ=math.sqrt(1.0*(μ-1)*(μ-2)/(n-1)); Z = (n_runs-μ)/σ; p = N().cdf(-abs(Z)); p_two_tailed = p*2; print(); print(*list(vars().items())[-9:], sep="\n")'(To explain: uniq -c counts unique lines within runs -- it only gets you overall unique counts if you sort the data first.)
p_two_tailed in this case represents the "probability of getting the observed results if they are statistically independent of each other". You can only conclude that the results are not independent if the p-value is small e.g. less than 0.05 or 0.01.
Note that you may need to either:
- add extra quoting to <COMMAND...> because
parallelinterprets the args as a shell script by default, or - pass the
-q/--quoteflag toparallelif your shell command consists of a single program with args
You can also pass additional args (e.g. --progress) to parallel as long as they only output to stderr, not stdout.
$ seq 500 \
> | parallel -n0 echo '$((RANDOM%2))' \
> | uniq -c \
> | awk '{ NR%2 ? n1+=$1 : n2+=$1 } END { print NR, n1, n2 }' \
> | python -c 'from statistics import NormalDist as N; import math; n_runs, n1, n2 = map(int, input().split()); n = n1+n2; μ=1.0*(2*n1*n2)/n + 1; σ=math.sqrt(1.0*(μ-1)*(μ-2)/(n-1)); Z = (n_runs-μ)/σ; p = N().cdf(-abs(Z)); p_two_tailed = p*2; print(); print(*list(vars().items())[-9:], sep="\n")'
('n_runs', 258)
('n1', 242)
('n2', 258)
('n', 500)
('μ', 250.744)
('σ', 11.157671366941461)
('Z', 0.6503149054468893)
('p', 0.25774441546058247)
('p_two_tailed', 0.5154888309211649)