Release announcement in the Google group
Raise bug reports as issues in this git repo.
The new Ibex Farm was released on December 31 2020.
The site should look pretty familiar. I've tried to minimize changes to existing workflows. You should find that your account and all of your existing data has been imported automatically (but see the 'import caveats' section below).
The only change you really need to know about is that results are no longer
appended to a results file in the results directory. Instead, results can be
downloaded on the results page for a given experiment, which is linked to near
the top of the experiment's page.
All existing results have been imported for download using the new method.
If you were a user of the somewhat esoteric git import feature, note that this has been removed and replaced with the feature described in the 'uploading experiment folders' section below.
I expect there to be some teething problems. My aim is for the new Ibex Farm to become fully stable over the next few months. You might want to check the 'known issues' section at the bottom.
For further details and info on new features, read on.
All accounts from the original Ibex Farm were imported on 18:35 UTC Dec 31 2020. This import was a one-off event, and there is no connection between the old and new sites. Any changes you make on one site will have no effect on the other.
On the new Ibex Farm, usernames are normalized to all lower case. (The login form is case insensitive, so you can continue to type your username however you're used to typing it.)
There were ~50 pairs of accounts with names that differed only by case. If one member of the pair was all lower case, this account will be accessible and the other will not. If neither member of the pair was all lower case, neither account will be accessible. This is something I can easily fix manually just by renaming the account that you need to access.
To resolve an issue with logging in to an imported account, follow these steps:
- Check that you can log in to the account on the original Ibex Farm.
- For verification, create an experiment in your original Ibex Farm account called "IMPORT_ISSUE".
- Email me (a.d.drummond@gmail.com) the username of the account you want to access. I'll then email you the new username and you'll be able to log in with your existing password.
- If you don't like the name I picked, you can click on your username to go to a form which allows you to rename your account.
The original Ibex Farm was written in Perl and Python and stored data directly on the filesystem. The new Ibex Farm is a complete rewrite in Elixir using Phoenix and LiveView, with front-end code written in Typescript. Experiment data is stored in S3 and in a PostgreSQL database (on AWS RDS). Transitory data, such as session data, is stored in Redis (on AWS ElastiCache). The app is deployed via Terraform to AWS Elastic Beanstalk using a Docker container.
Upshots of the above:
- Your data is now much safer. Everything is stored redundantly and automatically backed up by Amazon.
- Ibex Farm is more robust and scalable (🤞). The new code shifts a lot of work that used to be done on the server onto the client, and Elastic Beanstalk automatically increases the available resources when the server is under heavy load.
- Elastic Beanstalk's rolling deployments make it possible (usually) to deploy a new version of the site without users experiencing any downtime.
- There is finally a sane code base to build on for future development. The original Ibex Farm was a quick hack that ended up having a far longer lifespan than I anticipated.
- Although I've designed the code around AWS's infrastructure, it's not tied to AWS. For smaller deployments, it would be easy to modify the Dockerized development environment (which uses MinIO to stand in for S3).
I have a policy of keeping full backups for 30 days. If you permanently delete an experiment, or delete your account, all data should be irrecoverably deleted after approximately this period of time.
I've set up RDS and S3 with full point-in-time restore capability, meaning that within the 30 day backup window I can recover the state of the storage at any given point in time.
The original Ibex Farm, and Ibex itself, are released under an open source BSD license. Nothing has changed here, and you can still do approximately whatever you like with the original Ibex Farm code. In particular, you are perfectly free to use it to set up your own Ibex Farm instance.
The code for the new Ibex Farm is not currently released under any license, open source or otherwise. I am evaluating licensing options.
When you create an experiment, a number of files from Ibex are automatically
added to it (e.g. Form.js). The code in these files is (as before) covered
by the BSD license of the original Ibex. Libraries used by
Ibex experiments (e.g. jQuery) are covered by their own licenses.
All experiments on the old Ibex Farm were public. On the new Ibex Farm, experiments start out as private by default (i.e. visible only to the experiment owner). All of your imported experiments have been set to private.
You can alter the public/private status of an experiment at any time.
As before, you can set a password for an experiment. However, this is now implemented using a normal password prompt rather than the archaic HTTP auth mechanism used by the original Ibex Farm.
If the experiment is public and accessed by someone other than the experiment owner, they'll be prompted for the password. Passwords can be set for private experiments too, but this has no discernable effect until the experiment is made public.
Experiment passwords have been imported from the old Ibex Farm. It is no longer necessary to enter a username to access the experiment, just the password.
At some point I may add an option to localize the password prompt. As a stop-gap I've designed the form not to use any text at all.
The new Ibex Farm is smart enough not to prompt for experiment passwords if you're logged in as the experiment owner. To verify that password protection is working as you anticipate, access the experiment via your browser's 'incognito' or 'private browsing' mode.
Results can now be downloaded as CSV or Excel spreadsheets as well as in the original Ibex format. Look for a link to the experiments results page just under the experiment title.
If you download results in spreadsheet format, a column is included indicating whether the results come from someone who was logged in as the owner of the experiment. This can be useful for automatically excluding results that derive from test runs.
This information is unfortunately not available for results imported from the old Ibex Farm.
Any results files from existing experiments have been imported exactly as-is
for archival purposes. They will not be updated as new results come in.
Newly-created experiments do not have a results directory at all.
You can now copy experiments. Copying an experiment does not copy its results. At present the only way to delete an experiment's results is to permanently delete the experiment.
If you want to delete an experiment's results but keep the experiment files, you can copy the experiment then permanently delete the original.
If you upload a .csv or .xlsx spreadsheet to the chunk_includes directory,
it will automatically be converted to a JSON representation of the spreadsheet.
You can then access this data via the CHUNKS_DICT variable (see next section).
Each spreadsheet is a dictionary. The keys of the dictionary are worksheet names. The values are arrays of rows, where each row is an array of cells. In the case of a CSV file, each cell is a string. For an Excel file, cells are either numbers or strings depending on the type of the underlying Excel cell.
You now have the option of declaring items as a function, async function or promise.
If items is an ordinary (i.e. non-async) function with two arguments, it is called with an object as its first argument (see next section) and a callback as its second argument. The callback should be called with a single argument. Promise.resolve is then applied to this argument to obtain an items array.
If items is an ordinary function with other than two arguments, it is called with an object as its first argument (see next section), and Promise.resolve is applied to its return value to obtain an items array.
If item is an async function then it is called with an object as its first argument (see next section) and Promise.resolve is applied to its return value to obtain an items array.
If items is not a function then Promise.resolve is applied
directly to items.
What all of this means in practice is that items can be any of the following (not an exhaustive list):
- An items array (as in the original Ibex).
- A promise that resolves to an items array.
- An ordinary function that takes two arguments and calls its second argument with an items array.
- An ordinary function that takes other than two arguments and returns an items array.
- An async function that returns an items array.
- An async function that returns a promise resolving to an items array.
- An ordinary function that takes two arguments and calls its second argument with a promise that resolves to an items array.
- An ordinary function that takes other than two arguments and returns a promise that resolves to an items array.
If items is a function (or a promise constructed from a callback function), then within the function,
you can access __counter_value_from_server__ and CHUNKS_DICT. For example,
if you have uploaded a spreadsheet called foo.xlsx to the chunk_includes
dir, you can access the corresponding JSON data via CHUNKS_DICT['foo.xlsx'].
You can drag a folder onto the folder icon next to an experiment to automatically upload all and only the new/modified files.
The folder can have any name and should have the same structure as the folder you get if you download a zip archive of the experiment and then extract it. For example:
<folder with any name>/
|
|--> chunk_includes/
| |
| |--> <my html file>.html
|
|--> data_includes/
| |
| |--> <my data file>.js
All subfolders are optional. Files are never removed from the experiment through this process, only added/updated.
To prevent certain files from being uploaded, you can add an .ibexignore file to the root folder.
The syntax for this file is the same as for .gitignore.
It's parsed using the ignore module.
The editor for a file now opens in a separate tab rather than in a modal dialog on the same page.
Experiment files (e.g. example_data.js) are stored as objects in
S3. The key for an object is a nonsense value
unrelated to the user-facing filename. If you click the link for a file on the
experiment page, the page downloads the file from S3 in the background and then
uses some tricks to get the browser to "download" the file again with the
expected filename. If, however, you right click on the link and paste the URL
into another tab, the file will download with a nonsense name derived from the
S3 key.
Experiments on the new Ibex Farm load through a complex process that involves a lot of dynamic downloading of and execution of Javascript. I use sourceUrl to link back to the original source files. You'll need to make sure that source maps are enabled in your browser dev tools to make use of this info.
To get the best debugging functionality, I recommend taking the following steps:
- Use Chrome.
- Make sure you are logged in as the experiment owner.
- Open the dev tools and then refresh the experiment page. Refreshing is important because Chrome won't always interpret sourceURL information unless the browser is already in dev move when the page loads.
If you do this, you should reliably get source position information for Javascript errors in your data file.
The new Ibex Farm uses a system of push notifications to update experiment pages almost immediately following certain events. For example, if results are submitted for an experiment, the page for the experiment will promptly update with the new results count (without requiring a refresh).
If your account has an associated email address, you can now reset your password via email — just like on a real website! (There's an 'I forgot my password' link at the bottom of the sign in page.)
If you know the username, enter the username. If you enter an email address and you have multiple accounts under that address, you'll currently only be able to reset one of those accounts.
Ibex Experiments should continue to work on ancient browsers (even Internet Explorer 8!) I assume that users of the Ibex Farm itself have access to an up-to-date browser. I test on up to date versions of Chrome, Firefox and Safari.
In the old Ibex Farm, the Ibex version for a given experiment was shown in its entry in the list of experiments. For experiments imported from the old Ibex Farm, the version can now be seen on the experiment page.
- For experiments with very large sets of results (typically over 1000), you may find that it is not possible to download results in Excel format. Other formats should work fine, though you may have to wait a while.
This feature is experimental. Feel free to play around with it, but don't rely on it just yet.
Ibex is now able to track how many participants are actively in the process of completing an experiment for each counter value. This makes it possible to automatically balance the number of participants who complete each condition. This tracking is enabled by having experiments send regular pings to the server as a participant progresses.
To make use of this functionality, you must define items as a function (either ordinary or async) in your data file. If ibex is an ordinary function it is called with an ibex object as its first argument and a callback as its second argument. The following example code (written in ES5 for compatibility with older browsers) illustrates the use of the feature:
function items(ibex, callback) {
var NUMBER_OF_CONDITIONS = 2;
ibex.getCounterInfo(NUMBER_OF_CONDITIONS, function (info) {
ibex.overrideCounter(chooseCounter(info)); // does not change counter on server
ibex.initPing(); // initPing must be be called after overrideCounter
callback([
// ...
// Contents of a normal Ibex items array go here.
// For readability, you might want to define the array as
// a variable outside of this function.
// ...
]);
});
}
function chooseCounter(info) {
// Estimate of the probability that someone will complete
// the experiment given that they've started it.
var P_COMPLETE = 0.9;
var nPerCondition = new Array(info.completed.length);
// Compute expected completions for experiments currently in progress.
for (var i = 0; i < info.completed.length; ++i)
nPerCondition[i] = info.completed[i] + (info.inProgress[i] * P_COMPLETE);
// Compute the mean expected completions per condition.
var total = 0;
for (var i = 0; i < nPerCondition.length; ++i)
total += nPerCondition[i];
var mean = total/info.completed.length;
// Find the condition with the least expected completions.
var min = +Infinity;
var minI = 0;
var maxDeviation = 0;
for (var i = 0; i < nPerCondition.length; ++i) {
if (nPerCondition[i] < min) {
min = nPerCondition[i];
minI = i;
}
maxDeviation = Math.max(Math.abs(nPerCondition[i] - mean), maxDeviation);
}
// If expected completions for each condition are close, choose a
// condition at random. This avoids assigning everyone to the same
// condition in a situation where many people begin the experiment
// at almost exactly the same time.
if (maxDeviation < 0.05 * mean)
return Math.floor(Math.random() * info.completed.length);
return minI;
}If an experiment is public then pings from the logged-in owner of the experiment will be ignored. This means that you can play around with your own experiments without influencing the assignment of participants to conditions.
A similar principle applies for results. If an experiment is public then ibex.getCounterInfo will ignore results from the experiment owner. Conversely, if an experiment is private, ibex.getCounterInfo will only look at results from the experiment owner.
Each ping and each set of results is associated with a named ping group. The default ping group is called default. The name of a ping group can be any string of 50 or fewer characters.
You can append ?ping_group=PING_GROUP to the URL for an experiment to set the ping group for an experiment to something other than default. The information obtained from ibex.getCounterInfo is then based only on results and pings for the selected ping group.
The example code above makes use of an estimate of the probability that a participant will complete an experiment given that they have started it. To get optimal results, you will need an accurate estimate of this probability for your participants.
Some relevant data for making an estimate is available on the ping state page of an experiment. You can get to this by clicking 'View ping state' on the page for the experiment (just below the link to the results page).
The ibex object passed to items has a getCounterInfo(nConditions, callback) method. This method calls callback with a dictionary containing the keys completed, inProgress, and pings. Each field of the dictionary is an array of counts for each condition. In the case of completed and inProgress, the count is the number of users. The interpretation of the counts for pings will be discussed shortly (this data is not used in the example code above).
The absolute values of the counts mentioned in the preceding paragraph are randomized to avoid leaking information about how many participants have completed the experiment, or how many participants are currently in progress. The ratio of the counts between different conditions is meaningful, but the absolute values are not.
Inside the callback passed to getCounterInfo you may call ibex.overrideCounter(counterValue) to set the counter value for the experiment. This method does not update the counter value on the server (which is irrelevant when using dynamic counter assignment). After calling overrideCounter, call ibex.initPing() to start sending pings to the server.
A participant's progress through an experiment is tracked as follows. When ibex.initPing is called, the server is pinged to indicate that a participant is progressing through the experiment. Upon any further interaction with the experiment (key presses, mouse clicks, touches, etc.), the server is pinged again if the last ping occured more than 5 second ago. If the participant completes the experiment normally, or closes the browser, or if no ping has been received for a while, then the server removes its record of the relevant participant progressing through the experiment.
If you prefer to use promises, you can use ibex.getCounterInfoPromise, which returns a promise rather than taking a callback argument.
Dynamic counter assignment cannot currently be used if the number of conditions is greater than 16.
The object obtained by getCounterInfo has a pings property that is not used in the example code above. Like inProgress and completed, this is an array of counts per condition. The counts in this case (modulo randomization) are the total number of pings received per condition from participants that are still active. This information could potentially be useful for estimating how near to completion (on average) the participants for a given condition are.