(The following is from an email that was received on the email@example.com mailing list...)
Well, I finally managed to sit down and code the entire prototype. This has been refined a couple times and tested via the Church play-space (https://probmods.org/play-space.html). The parameters and set-up can probably use some tweaking to make the cooperation more robust (ie: more probable/numerous as a fraction of the histogram used for testing), but oh well.
The paper follows the Goodman & Tenenbaum approach of "commonsense reasoning as conditional simulation (in probabilistic generative models)". Each agent simulates:
- The other agent simulating it,
- Simulating the other agent playing naively.
So the recursion levels are: 0 - Naive play 1 - Observing your opponent's naive play 2 - Observing your opponent's observation of your naive play
Since there are only two moves available in the game, there can only be a total of two levels of Sicilian Reasoning.
Really solving the general problem here is coming up with an inference procedure capable of noticing that the two agents will, each either playing naively or observing the other one's naive play, each make the same move, and then conditioning on that.
We'd like a fully general theory that lets us not need to explicitly condition on the sameness of the two moves to get cooperation. My hypothesis is that if we treated naive play and our opponent's move as two separate random variables and ran a causal induction algorithm on them, we would then come up with a smarter model which stochastically guarantees the sameness, and we could invert that distribution to play wisely -- the general principle being that a wise agent treats its own actions as further inputs to causal modeling and notices when its actions, placed in their context, display a meaningful (ie: utility-affecting) pattern.
אלי סנש firstname.lastname@example.org