Skip to content

Instantly share code, notes, and snippets.

Model AGIEval GPT4All TruthfulQA Bigbench Average
Mistral-7B-Instruct-v0.2 38.94 71.67 66.88 42.22 54.93

AGIEval

Task Version Metric Value Stderr
agieval_aqua_rat 0 acc 23.23 ± 2.65
acc_norm 22.83 ± 2.64
agieval_logiqa_en 0 acc 36.25 ± 1.89

Mistral-7B-Instruct-v0.2 - Lighteval

Task Version Metric Value Stderr
all qem 0.0000 ± 0.0000
truthfulqa_mc1 0.5288 ± 0.0175
truthfulqa_mc2 0.6827 ± 0.0152
leaderboard|gsm8k|0 0 qem 0.0000 ± 0.0000
leaderboard|truthfulqa:mc|0 0 truthfulqa_mc1 0.5288 ± 0.0175
truthfulqa_mc2 0.6827 ± 0.0152
#Fixes notorious `dockerd: failed to start daemon: Devices cgroup isn't mounted` error when trying to start docker
#in a privileged concourse container.
#From: https://github.com/concourse/docker-image-resource/blob/master/assets/common.sh#L1-L40
#See also: https://github.com/concourse/concourse/issues/324
sanitize_cgroups() {
if [ -e /sys/fs/cgroup/cgroup.controllers ]; then
return
fi
mkdir -p /sys/fs/cgroup
@idoru
idoru / pr.md
Created July 13, 2022 13:52 — forked from piscisaureus/pr.md
Checkout github pull requests locally

Locate the section for your github remote in the .git/config file. It looks like this:

[remote "origin"]
	fetch = +refs/heads/*:refs/remotes/origin/*
	url = git@github.com:joyent/node.git

Now add the line fetch = +refs/pull/*/head:refs/remotes/origin/pr/* to this section. Obviously, change the github url to match your project's URL. It ends up looking like this:

@idoru
idoru / gist:5470286
Last active December 16, 2015 17:29
Clang with ARC bug: Incorrect substitution for pointer template parameters. This bug report is suitable to be posted to http://bugreport.apple.com
Summary:
When ARC is enabled, C++ candidate templates' pointer template parameters are not correctly substituted for objective-C object pointers.
Steps to reproduce:
Compile the following main.mm file, with and without ARC:
#import <Foundation/Foundation.h>
template <typename T>
@idoru
idoru / gist:3102360
Created July 13, 2012 02:41
Clang with ARC bug: Incorrect Template substitution failure
Summary:
When ARC is enabled, C++ candidate templates are ignored due to an incorrectly diagnosed substitution failure.
Steps to Reproduce:
Compile the following main.mm file, with and without ARC:
#import <Foundation/Foundation.h>
template <typename T>
@idoru
idoru / SwapToSpec.applescript
Created July 8, 2012 03:18
XCode behavior script for quickly opening corresponding Cedar spec and vice-versa
#!/usr/bin/osascript
(*
What does this script do?
This script lets you quickly open the Cedar spec for the corresponding class header/implementation that is currently open in Xcode. If the spec is open, it opens the implementation instead.
How do I use it?
Save this script somewhere.
Start Xcode and go to Xcode > Behaviors > Edit Behaviors…
@idoru
idoru / .zshrc
Created July 25, 2011 01:39
.zsh with oh-my-zsh + personalized prompt customization
# Path to your oh-my-zsh configuration.
ZSH=$HOME/.oh-my-zsh
# Set name of the theme to load.
# Look in ~/.oh-my-zsh/themes/
# Optionally, if you set this to "random", it'll load a random theme each
# time that oh-my-zsh is loaded.
ZSH_THEME="robbyrussell"
# Set to this to use case-sensitive completion
@idoru
idoru / aerobuild.sh
Created June 19, 2011 21:33
bump version, build & archive, tag git version and publish iOS app builds to testflight
#!/bin/sh
#
# aerobuild.sh
#
# What does this do?
# ==================
# - Bumps build numbers in your project using agvtool
# - Builds your app and packages it for testing/ad-hoc distribution
# - Commits build number changes to git
@idoru
idoru / gist:1014467
Created June 8, 2011 14:01
Applescript for an Automator service that switches between specs and implementations
on run {input, parameters}
tell application "Xcode"
set my_workspace to active workspace document
set workspace_projects to projects of my_workspace
set currentFullPath to ""
set projectDir to ""
set reverseDocumentList to (reverse of (source documents as list))