I made a superhuman Generals.io agent with self-play RL [P]

Hi everyone,

I trained a self-play RL agent for Generals.io that reached superhuman-level and ranked #1 on the human 1v1 leaderboard.

It began as my master's thesis where the goal was to beat a prior algorithm based agent. We succeeded using behavior cloning, RL fine-tuning and reward shaping, but the agent was still consistently beaten by the top players.

So I gave it a round two and fixed the largest bottlenecks:

• Reimplemented the whole pipeline in JAX (from NumPy/Torch)
• Used Vision Transformer instead of the CNN

Both are a result of the same idea: to invest in scaling rather than human priors and ad-hoc patches.

The blog is written as a guide for anyone building something similar — the dead ends, the decisions, and the intuitions and tricks I picked up along the way.

It's all open source, including the fast JAX simulator — handy on its own if you want an imperfect-information RTS env to play with.

Links

- Guide: https://kam.mff.cuni.cz/~straka/blog/generals.html

- Simulator (JAX): https://github.com/strakam/generals-bots

- Agent: https://github.com/strakam/AverageJoe

I hope you find the blogpost entertaining!

Feedback and questions welcome 🤗.