Skip to content

Instantly share code, notes, and snippets.

@Equim-chan
Last active Dec 26, 2022
Embed
What would you like to do?
Should I release the trained model of my mahjong AI?
简体中文

更新

2022-08-19

结论:鉴于在线版本 https://mjai.ekyu.moe 已经基本实现了我所期望自己的 AI 去提供的功能,目前没有公开模型参数的计划。

2022-04-13

已经开源在了 https://github.com/Equim-chan/Mortal

我是否要发布训练好的麻将 AI 模型?

麻将社群,我需要你们的帮助。

背景

我是 akochan-reviewer 的作者,这是一个基于 critter 的 akochan 制作的麻将学习工具

我制作麻将牌谱检讨工具的初衷是希望它能成为一个 7x24 小时在线的教练来教导人类玩家,尤其是那些没法接触到高质量教学资源或者教练的人。这样的工具已经在国际象棋和围棋这样的游戏里存在了,而麻将也应当有一个。当我看到有人说他们从 akochan-reviewer 学到了什么的时候,我是挺欣慰的,即便 akochan 时常做出反常的行为。

akochan 举止怪异是一个我也无法修复的已知问题。出于对 akochan 存在的种种限制、不足、以及糟糕的性能的不满,我决定自己做一个全新的 AI。

费劲将近一年的心血之后,我从零开始完成了一个强度足够的麻将 AI 的 1.0 版本。它很强,根据它和 akochan 对战的统计数据可以得出它的水平已经超过 akochan,也包括了其他我所能拿到的 baseline,它可能和初版的 Suphx 强度相当。

这个 AI 兼容 akochan 的协议,也就是 mjai,所以我的 akochan-reviewer 能很轻易地支持以它为后端。

在接下来的几个月,我准备要将源代码公开至 GitHub,但我在犹豫是否要公开训练好的模型。

如果你感兴趣的话,这里有几张新 AI 的 1.0 版本(旧版本,现已被新版本击败)之间自我对战的牌谱:

理由

公开模型的理由

1. 有始有终

如果只有代码没有预先训练好的模型,对于一个深度学习项目来说总显得有点不太完整。如果模型也发布了的话,其他的研究员就无需自己训练模型了,节省了大量的时间和金钱。

2. 在本地运行的能力

正如我所做的 akochan-reviewer 一样,用户可以轻易地下载到预先构建好的版本,然后在本地跑。目前,我正在维护的 akochan-reviewer 的在线版本只有有限的功能,也无法检讨雀魂牌谱,而本地版本却有所有的功能,并且能读取下载好的雀魂牌谱,甚至还能输入任意自定义的牌谱。

3. 方便衍生作品

也许其他人比我有更好的方式去做麻将牌谱检讨机。发布模型的话,他们就可以轻易地发布或部署他们自己的检讨机或者其他正规的衍生作品,只要把模型嵌入。

4. 就算不发布,开挂的人自己也会训练模型

然后他们还能以此为理由卖得更贵。

也许在我把源代码公开的那一刻,我就已经输给了做挂的人。如果他们不惜一切代价都要搞的话,说不定我自己免费公开模型让他们难以获利还更好,尽管这理由听着挺蠢。

不公开模型的理由

开挂的人一定会利用它

这是我最大的担忧。

首先有个坏消息,这样的不当利用 已经 发生在了 akochan 上。一些脚本小子已经售卖基于 akochan 的雀魂 bot 有些时间了。尽管他们在描述里没提到 akochan 的名字,但有很多线索指向他们在用 akochan 或者 akochan 的改版。

我看到的时候心情很复杂。我对做挂卖挂的行为感到气愤,也为他们在卖偷来的东西感到失望。但突然间又感到一丝庆幸,还好他们没打上 akochan 的标签,不然吃瓜群众可能会误解是 akochan 开发者的授意。最后我只能感到悲哀,因为即便他们没有售卖、散布、制作外挂,终有一天也会冒出来另一群人在做类似的事,只要 AI 能够从公共途径获取到。

我并不反对制作天凤或者雀魂的个室 bot,相反,我觉得他们非常实用,无论是想玩但找不齐人,或者只是单纯好奇想和 AI 在自己熟悉的平台上比试比试。

可惜,人的恶是无法被阻止的。能被不当利用的东西一定会被不当利用。如果我发布了训练好的模型,我实在找不到什么方法能去避免它被利用去打段位战。更糟糕的是,我可能还会被受害者指责“在帮人做外挂”。很难想象会有多糟糕。

其他类似的项目是如何处理的?

国际象棋的 stockfish

stockfish 是个很有名的开源国际象棋 AI。如果你玩国际象棋,辛苦把你所知道的分享一下。

  • stockfish 有把在线象棋平台毁了吗?如果是的话是多大程度的影响?
  • 在线象棋平台如何避免人们使用 stockfish 来作弊?
  • 在线象棋的玩家对此怎么看?

NLP 的 GPT-3

这个不是游戏 AI,但它在现在这个上下文里算是个不错的例子。

OpenAI 的 GPT-3 的预训练模型,从来没有被公开发布过。根据我所听到的故事,是因为模型实在太强,太容易被滥用。当然也可能是因为商业价值太高。

最近,OpenAI 宣布了一系列的由 GPT-3 模型支持的在线服务。我觉得我也可以走类似的路子,提供些在线服务,比如牌谱检讨和去个室玩。这也是 NAGA 正在做的事情。

可能的解决方案

1. 不发布模型

最直接,但这很大程度上限制了这个项目能对麻将社群做的贡献。

2. 不发布模型,但是提供在线服务

听起来不错,但是开发和维护在线服务需要时间和钱,而且如果我哪天丧失了兴趣就没了。我可能需要做一些需要捐赠才能解锁的会员功能来支撑服务运行下去。但是做成收费的话又会丧失其之于 NAGA 的优势。

3. 发布模型,但是要付费

盗版的运作方式同样能放在这。同时,这也只让做外挂的人更难搞,并不是完全阻止。

… 更多?

请在下方评论。


如果你有任何想法,请在下方留言。谢谢。

Equim.

English

Updates

2022-08-19

Conclusion: considering the online version https://mjai.ekyu.moe has already provided what I expected my AI to provide, currently I have no plan to release the trained model.

2022-04-13

The source code is now avilable on GitHub at https://github.com/Equim-chan/Mortal.

Should I release the trained model of my mahjong AI?

Dear mahojng community, I need your help.

Background

I am the author of akochan-reviewer, a mahjong learning tool powered by critter's akochan.

My original intention for building mahjong reviewer was that it could serve as a 7x24 available coach that guides human players, especially for those without access to high-quality tutorial or coach resource. Such tools already exist in other games like chess and go, and mahjong should also have one. I am really happy to see people telling me they learn something from akochan-reviewer, even though akochan acts weird from time to time.

Akochan acts weird, this is one of the known issues, and I can't fix it as well. Out of dissatisfaction with the limitations, weaknesses and poor performance of akochan, I decided to make a brand new one myself.

After almost a year of hard work, I managed to build the 1.0 version of a powerful mahjong AI from scratch. It is very strong. Statistics from test plays conclude it is stronger than akochan and any other available baseline I have, and it's probably as strong as the early version of Suphx.

The AI is compatible with akochan's protocol, namely mjai, therefore it can serve as a backend for my akochan-reviewer very easily.

In the next several months, I am going to release the source code on GitHub, but I am hesitant to release the trained model.

If you are interested, here are some games the new AI (version 1.0, weaker than the current version) played with each others:

Reasons

To release the model

1. Make the research complete

It feels a bit incomplete for a deep learning project to only release the code without a trained model. If the model is released, other researchers won't have to reproduce the model on their own, which save a huge amount of time and money.

2. Ability of running locally

Just like akochan-reviewer I made, users can easily download a prebuilt all-in-one package and run it locally. At the moment, the online reviewer service I maintain has limited features and no access to mjsoul logs, whereas the local version has all the features and can read downloaded mjsoul logs and even any hand-crafted log.

3. Easier for derived works

Maybe somebody else have a better idea of making a mahjong reviewer than me. By releasing the trained model, they can easily release or deploy their own reviewer or other legit derived works bundled with the AI.

4. Cheaters are going to train their own model anyways

And they will even sell it higher for such reason.

I may have already lost to cheaters by the time I release the code to the public. If they're going to do it at any cost, it might be even better for me to release the model for free to make them harder to profit, although the idea sounds a bit dumb.

Not to release the model

Cheaters will definitely exploit it

This is the biggest concern.

First of all I have some bad news, such exploit is already happening on akochan. Some script kiddies in the wild have been selling akochan bot for mjsoul for some time. Although they don't mention of the name "akochan" in the description, there are some indications that they are using akochan or a modified version of it.

I had very mixed feelings when I saw it. I was angry for the conduct of making and selling cheats itself, and also for them selling things they stole from others. But then I happened to feel a bit thankful for them not having tagged it with the name "akochan", which will be worse as people may think it is affiliated with akochan devs. And in the end, I was just sad. Even if those guys had not sold, distributed, or even developed the cheat, as long as the AI is publicly available, another group of people will eventually get their hands on it.

I am not against making tenhou or mjsoul bots that play in private lobbies, instead, I think they are very handy and useful when you want to play but can't find enough friends, or you are just curious and simply want to challenge the AI on the platform you are familiar with.

Unfortunately, there is no way to stop people from being evil. Things that can be exploited will be exploited. If I release the trained model, I don't see a way to prevent it from being exploited to play in ranked lobbies. Even worse, I may become the one to be blamed by some victims for "assisting cheaters". I honestly can't imagine how horrible it will be.

How do other similar projects handle it?

stockfish for chess

Stockfish is a well-known open source chess AI. If you are a chess player, please share information you know about it.

  • Has and how has stockfish destroyed online chess?
  • How do online chess platforms keep people from cheating with stockfish?
  • How do online chess players think about it?

GPT-3 for NLP

This is not a game AI, but it serves as a good example in this context.

The trained model of GPT-3, by OpenAI, were never released to the public, according to the story I heard, because the model is so powerful that it can't be kept from being exploited. Of course it is also possible that it is because the commercial value of it is too high.

Recently, OpenAI announced a suite of online services backed by GPT-3. I believe I can go the similar way by not releasing the weights and instead offering online reviewer and private lobby bot services. Offering online services is also what NAGA is doing.

Possible solutions

1. Do not release the model at all

The straightforward way, but it greatly limits the project's ability to benefit the mahjong community.

2. Do not release the model, but offer an online service for review and bot for private lobby

Sounds good, but it takes time and money for me to develop and maintain the service, and it probably won't last forever if someday I lose my interest. I may need to make some features premium that require donations to unlock to keep the service alive. But if it becomes pay-to-use, it will lose its advantage over NAGA.

3. Release the model, but behind a paywall

We all know how piracy works and the same can apply to this. It only makes it harder for cheaters but does not completely make it impossible.

... more?

Please comment down bellow.


If you have any idea, please comment down bellow. Thank you.

Equim.

@luooooob
Copy link

我提供一个做决策的角度:

核心的问题是,你究竟是想要

一个 7x24 可用的教练来指导人类玩家,特别是对于那些无法获得高质量教程或教练资源的人

还是

一个非常酷的ai项目,用来验证和提高自己的代码水平和工程能力

如果你的目标是前者,那就应该只提供在线服务,naga收费明显过高了而且支付方式能劝退一堆人,市场还是很大的。但是市场的前提是不能把市场送人,什么扩展能力都可以给但是必须通过在线服务提供,绝对不能提供源码或二进制这种不受控制的东西。还能顺便防止有人用这个ai作弊

如果你的目标是后者的话,那确实应该全程开源,所有细节都展示出来,没啥需要遮掩的。公开源码了不公开模型没必要,甚至数据集也可以公开。我今天来就是要卷死各位或者被各位卷死,什么开挂代练没工夫管那些,也不需要考虑长期怎么办,比较简单粗暴。

@Equim-chan
Copy link
Author

@luooooob 我觉得这两个说法都偏极端了一些。前者并不是就不能开源了,因为开源能提升透明度,还能方便其他人学习、改善这个项目;后者也不是必须要公开模型,甚至都不一定需要开源,就像 AlphaGo 之类的作品一样,不开源对影响力没有太大影响。

但是我想提的一点是,麻将 AI 需要依赖玩家的生态存活。麻将的圈子相比围棋、象棋本来就不大,如果 AI 滥用作弊的情形变得更加普遍了,很多玩家可能会离开这个游戏,圈子会进一步萎缩。到时候 AI 可能都成了口诛笔伐的对象,更难发展下去了。

@luooooob
Copy link

luooooob commented Jul 5, 2022

@luooooob 我觉得这两个说法都偏极端了一些。前者并不是就不能开源了,因为开源能提升透明度,还能方便其他人学习、改善这个项目;后者也不是必须要公开模型,甚至都不一定需要开源,就像 AlphaGo 之类的作品一样,不开源对影响力没有太大影响。

但是我想提的一点是,麻将 AI 需要依赖玩家的生态存活。麻将的圈子相比围棋、象棋本来就不大,如果 AI 滥用作弊的情形变得更加普遍了,很多玩家可能会离开这个游戏,圈子会进一步萎缩。到时候 AI 可能都成了口诛笔伐的对象,更难发展下去了。

其实后面只是气话,我是希望能有个牌谱分析网站成功地运营起来的。一旦引擎泄漏出去,网站肯定经营不起来了,各种抄袭和作弊的负面消息也很难让你继续有动力研究下去,大概率就渐渐弃坑了。目前看来能持续下去的方法也只有商业化,其他方案对这个项目都是有害的。虽然商业化的路不好走,但是还是希望能多多考虑

@Zeks
Copy link

Zeks commented Sep 5, 2022

Now that the site has been purged by tenhou everyone regrets the decision to never release the model.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment