Teaching Bots to Experiment
The bots on my Hollowtide server started out following rulebooks I wrote. Now they're moving to something better: a score for real progress, experiments on private test servers, and alarms for wins that look too good. It's a small, hands-on version of a real problem in AI research.
Why the server has bots
Hollowtide is my fan revival of Asheron's Call, with a new client and a live server I run. An MMO with a small population feels empty, and a server that only real players exercise finds its bugs the hard way. So the server also runs bots: characters that log in and play the game the way a person would. They level up, gear up, hunt and travel. They're part of how each release gets tested before players see it, and they keep the world from feeling empty If you ask one, it will tell you it's a bot.
This page is about how they decide what to do, and why I'm changing that from the ground up.
Version one: the rulebook
The first bots played from documents I wrote: a leveling guide, rules for gear, rules for how to play safely. AI coding agents turned those documents into behavior, and it worked. The bots played like a careful new player who had read the wiki.
Then the same loop kept repeating. I'd watch the bots, notice something a real player would never do, write a new rule, and ship it. Each fix was fine. Together they showed the problem:
- They could only be as good as my notes. Every trick had to come from me, so the bots never knew anything I didn't.
- Spoon-feeding doesn't scale. Each new rule touched the others, and the edge cases grew faster than I could write them down.
- Rules go stale. When the world changes with new content or a rebalance, a written strategy keeps doing yesterday's best thing.
- They could never surprise me. A veteran player finds things the designer didn't plan for. A bot that only follows instructions can't.
The rulebook had a ceiling, and the ceiling was me.
Version two: a score, experiments and alarms
Instead of telling the bots how to play, the new design tells them what good looks like and lets them find out how. It has three parts:
- Try a variation
- Play it on a test server
- Score it
- Check the alarms
- Keep what wins
A score for real progress. This is the hardest part, and it's a design problem, not a coding one. The score has to measure what a player would call progress, not something that's easy to inflate. Get it wrong and you've built a machine for making the wrong number go up.
Experiments on private test servers. Bots try variations on a strategy in copies of the world that no player is on, side by side with the current approach. A change is kept only when it measurably wins, over enough runs that it isn't luck. Nothing reaches the live server until it has won somewhere safe first.
Alarms for suspicious wins. A win that's too big, too sudden, or comes from something no player would think to do doesn't get kept automatically. It gets flagged and held until a person looks at it.
| Rulebook | Experiments | |
| Where strategy comes from | What I already know | What measurably works |
| When the world changes | I rewrite the rules | The next experiment notices |
| My job | Write every trick down | Define progress, review surprises |
| Can it surprise me? | No | Yes, which is the point and the risk |
The part that's really an AI alignment problem
AI researchers have a name for what happens when you give an optimizer a score: specification gaming, or reward hacking. The system finds the cheapest way to raise the number, and the cheapest way is often one nobody intended. The classic example is a boat-racing agent that learned to spin in circles collecting points instead of finishing the race. It wasn't broken. It did exactly what the score asked.
On a game server, that has a very concrete meaning. A system clever enough to find a better way to level is also clever enough to find a hole in the wall. You don't get one without the risk of the other. That's why the alarms exist, and why a surprising win is treated as a question, not a victory.
The guardrail: clever is welcome, exploits are reported
The line is simple to say and matters a lot:
- Clever play within the game's rules is welcome. An unexpected but fair strategy is exactly what this is for. It might even teach me something about my own game.
- Real bugs and exploits get reported and fixed. They never count as a win, and they're never used on the live server.
It matters for three reasons. It's a shared world, and a bot using a bug would be cheating against real people. It keeps the score honest, because an exploit that can't be kept isn't worth chasing. And it turns the bots' cleverness into something useful: anything that finds a hole becomes the server's most persistent QA tester. A found exploit is a bug report, not a strategy.
Results
The experiments are running now. Once they've had a few weeks, this is where the results go: how the bots' progress compares with the rulebook version, the most surprising fair strategy they found, and what the alarms caught and how it was handled.
| Progress vs. the rulebook | Coming after the first runs |
|---|---|
| Most surprising fair strategy | Coming after the first runs |
| Suspicious wins flagged | Coming after the first runs |
| Bugs found and fixed | Coming after the first runs |
What I'm taking from it
- Rules capture what you know. Experiments find what you don't. Both have a place, but only one can grow past its author.
- The score is the design. Most of the real thinking goes into what counts as progress and what explicitly doesn't.
- Plan for the system to be cleverer than you. Build the alarm before you need it.
- Keep a person on the decisions that matter. What counts, what gets kept and what ships are still mine, the same way every Hollowtide release goes out on my go.
I'll update this page as the bots learn things I didn't teach them.
On purpose, this page stays high level. How the bots play, where and in detail is theirs to work out, and it's a shared world that players should discover for themselves. Hollowtide is a fan project, not affiliated with or endorsed by Turbine, WB Games or the original game's owners.