Reinforcement Learning for Runescape

I. A question with 63 waves

Can an agent learn an encounter without being handed the strategy?

Fight Caves has a wonderfully compact job description: survive 63 waves of enemies, defeat TzTok-Jad, and earn a fire cape. The implementation details are less forgiving. You need to choose targets, manage resources, move around obstacles, and change protection prayers before an attack becomes a very expensive mistake.

In reinforcement learning, the agent learns by interacting with an environment. Here, the environment is my simulator. Each tick, the agent receives an observation, chooses an action, and gets a reward after the world advances. Across many attempts, training adjusts its policy—the rule it uses to choose actions—to increase future reward. The reward tells it how an outcome was scored, not which move it should have made.

That question turned into three connected projects: a game engine fast enough to generate experience, a training system that could learn from it, and a viewer that let me inspect what either of them was doing. The viewer became especially useful when “learning to win” looked suspiciously like “learning to stand there.”

The agent I’ll follow throughout this article comes from one training run, which I’ll call the main run. It completed 9,570 of 10,800 full caves: 88.61%, with a Twisted bow and no food or Prayer potions. A later search over training settings produced an agent that completed 93.40%. Those numbers are encouraging. The interesting part is what had to become correct before they meant anything.

II. Build a world worth learning

The simulator defines the problem the agent actually solves.

Fight Caves is a combat encounter in Old School RuneScape. Its waves introduce enemies separately, then combine them. Protection prayers defend against different attack styles, but consume a limited resource. Positioning changes which enemies can reach you, whether they have line of sight, and whether you can attack them.

The simulated arena is a 64 × 64 tile grid with 15 possible spawn patterns, called rotations. Each game tick represents 0.6 seconds. Training advances those ticks as fast as the computer allows; it doesn’t wait for the clock on the wall.

In the main run, the player starts with strong equipment: a Twisted bow, Dragon arrows, Masori armour, and 99 HP, Prayer, Defence, and Ranged. There are no sharks or Prayer-potion doses. Natural HP regeneration remains active. These conditions matter: better equipment shortens fights and changes how many mistakes the player can survive.

One engine, several ways to use it

The project began as a prototype in Kotlin. I later rewrote the simulation in C, then wrote tools that read the game’s own data files to produce terrain, objects, collision maps, and visual assets. Raylib, a small graphics library, draws the viewer.

The useful architectural decision was to keep gameplay in one place. Drawing a projectile and resolving its damage are different jobs. If the viewer and trainer implement combat independently, a convincing replay can hide a different training environment. One Jad was quite enough; I didn’t need two implementations disagreeing about how to kill me.

Correctness needed a different set of tools. Restarting from the same random seed and replaying the same actions makes a failure repeatable. Comparing fingerprints (hashes) of the game state shows the exact tick where two runs stop matching. Viewer overlays expose collision, line of sight, routes, footprints, and pending attacks. An agent willing to repeat a bad interaction millions of times is a remarkably persistent test case.

III. Give the agent eyes and hands

What the policy can observe and choose is part of the experiment.

A policy maps observations to action probabilities. This one starts with randomly initialized weights. It doesn’t know that a large red creature on wave 63 deserves special attention. It doesn’t see a large red creature at all. Jad arrives as numbers, which is a fairly undignified entrance for a final boss.

A 320-number view of the cave

Each decision receives 320 floating-point values. Structured observations let me study combat decisions without first solving image recognition or mouse control.

The simulation can hold 16 enemies, but the policy sees the eight nearest. Slots are sorted by distance to each enemy’s footprint, then by spawn index. Empty slots contain zeros. A slot is a place in the current observation, not a permanent enemy identity, so the engine resolves target selections before movement can reorder the list.

Three decisions per tick

The network has three action heads: separate outputs for movement, target selection, and Prayer. It samples one choice from each.

An action mask removes illegal choices before sampling. Those same legality bits are included in the observation.

Targets persist, so “no new selection” can still lead to an attack. A shot that fires consumes that tick’s movement opportunity. Prayer flicking is an explicit action: the engine handles the off/on transition, and the policy chooses when to request it. Eating, drinking, gear switching, and spell selection are outside this policy’s action space.

A deadline before the damage

Timing is where a seemingly small implementation choice changes the task. In this simulator, a non-melee Jad attack launched at tick T locks its protection decision at the boundary T + 2, before that tick’s action. The action at T + 1 can still change the protection used for the hit. Impact occurs at T + 3 or later.

IV. Turn experience into a policy

Try an action, observe the consequence, update, repeat.

One way to write the objective is a discounted return:

Finishing the cave is the goal, but a learner that almost never reaches the end needs useful feedback along the way. The reward design will determine what counts as useful.

The learner uses Proximal Policy Optimization (PPO). A critic estimates future reward; an advantage estimate measures how an outcome compares with that expectation. The policy update encourages actions with positive estimated advantages and discourages those with negative ones. PPO clips its probability-ratio objective to reduce the incentive for overly large changes in a single update.

PufferLib also learns from each batch of experience more than once, and applies a correction called V-trace when the updated policy differs from the policy that collected it. An entropy term encourages action diversity. These tools help with learning; none substitutes for a sensible environment.

Memory and parallel caves

The policy has three recurrent layers of a lightweight type called MinGRU, each 512 units wide, with 2,541,056 parameters in total. Recurrence gives the network a memory to carry between decisions. The actor produces action probabilities, the critic predicts value, and an optimizer called Muon applies the updates to the parameters.

# Conceptual flow, not the PufferLib API.
for rollout in training:
    memory = zero_state(num_agents)
    for tick in range(256):
        obs, masks = caves.observe()
        logits, value, memory = policy(obs, memory)
        actions = sample_masked(logits, masks)
        rewards, done = caves.step(actions)
        rollout.store(obs, actions, rewards, done, value)
        memory[done] = 0
    learner.update(rollout)

The sketch shows how the main run trained: the policy’s memory resets at the start of every 256-step rollout and whenever an episode ends. Final evaluation keeps the memory across rollouts and resets it only between episodes. That difference belongs in a run’s records; changing it silently changes the experiment.

learning_rate = 0.00123960734
momentum = 0.991584122
ent_coef = 0.00026363763
replay_ratio = 2.20972681
clip_coef = 0.0250000004
gamma = 0.9991261141073255
gae_lambda = 0.9

V. Reward progress that stays made

The agent optimizes the feedback it receives, including the parts I got wrong.

Survival can be a dead end

An early run of 500 million steps reached mean wave 27.6 and completed zero caves in 10,174 episodes. Later, I tried a 200,000-tick time limit per episode with a budget of 2 billion steps. Mean episode length reached 199,939 ticks. Mean wave stayed near 30. There were still no completions.

I had made room for an exceptionally long unsuccessful cave. The extra survival time was not becoming progress.

Rewarding protection introduced another problem. In one failed run, the correct-prayer component contributed 86.87 reward per episode, while progress contributed 2.92 before total-reward clipping. The policy spent about 83% of its time without a target. I was paying it well to defend a fight it wasn’t finishing. The policy had found a steady job. I had meant to offer a quest.

Later, I trained two otherwise identical runs for 750 million steps each: one kept the remaining reward for using the correct protection prayer against dangerous attacks, and one removed it. Removing it improved completion from 88.02% to 92.74%: the extra payment wasn’t helping.

Count the work still required

The reward design I kept gives dense feedback for reducing the work left in the cave. “Dense” means feedback arrives along the way, rather than only at the final kill.

Gross damage counts each hit as another accomplishment. Remaining-work accounting also records when that accomplishment has been undone. If damage and healing cancel, the progress component should cancel too.

Let W be remaining work and S be work at the start of the wave, measured in the engine’s internal tenths of HP. Progress through the wave is q; overall cave progress is C:

The progress component is 0.001 × ΔC × 63 × S. Within a wave, that works out to 0.001 times the net work removed. When a new wave starts, the comparison starts over, so the newly spawned enemies are not charged as negative progress.

Required work includes the children a large Tz-Kek will spawn. Splitting therefore doesn’t create surprise work. On the final wave, it tracks Jad’s HP. Killing a healer is not required to finish the cave, but healing Jad increases the work remaining.

def progress_reward(before_hp, after_hp):
    # Ten internal units equal one game HP.
    work_removed = (before_hp - after_hp) * 10
    return 0.001 * work_removed

damage = progress_reward(100, 90)
healing = progress_reward(90, 100) - 0.005
print(f"{damage:+.3f}, {healing:+.3f}, {damage + healing:+.3f}")

+0.100, -0.105, -0.005

Damage and healing cancel in the progress reward, and each healing event adds its own small cost. The example leaves out the other reward terms.

Other terms penalize damage taken, Prayer spent, time, invalid actions, and sustained inactivity. Completion earns +1 and death earns −1. The trainer clips the combined reward to [−1, +1]. Gross damage, individual kills, wave clears, and correct prayers earn no direct reward in the main run.

Check the rules before tuning the penalty

Some stalls were simulation problems. Some enemies stopped chasing when they were technically in attack range but couldn’t see the player or reach them for a melee hit. Fixing that movement bug, with everything else unchanged, raised the mean wave from 40.82 to 49.17. No caves were completed yet, but the fight was less broken.

Accuracy corrections can lower the score, too. Adding a missing Magic attack bonus to Ket-Zek and Jad lowered an otherwise identical run from 92.74% to 91.49% completion. I would rather measure the harder, more accurate task than preserve a nicer number by leaving a combat rule wrong.

VI. What the agent learned

A strong trained agent and a dependable training recipe are different results.

The main run used PufferLib 5.0’s native trainer, which is written in C and CUDA. Its settings came from a hyperparameter search: 90 training runs, each trying different training settings, of which this one scored highest. It trained for 499,122,176 steps in 434.08 seconds on one NVIDIA RTX 5070 Ti graphics card, not counting the final evaluation. That is the cost of this run. It does not include building the simulator or searching for the recipe. The GPU got the short assignment.

The training log records one average for each quarter of training, plus a final evaluation. In the first two quarters, the agent progresses through ordinary waves. In the third, it begins reaching Jad but almost never finishes. By the fourth, it reaches him in most episodes, and completion has increased sharply.

Start at wave 1 and count outcomes

Final evaluation reloads the checkpoint and samples actions from the masked policy. It starts full caves at wave 1, keeps the policy’s memory across rollouts, and resets it between episodes. Evaluation plays caves in large batches: it asked for at least 10,000 episodes and finished with 10,800.

Mean episode length was 4,930.82 ticks: about 49.31 minutes of simulated game time, averaged over successes and failures. Accelerated simulation makes it possible to collect that experience without waiting 49 minutes per cave.

Train again with a different seed

A random seed is the number that fixes a run’s random choices. Evaluating one checkpoint across many caves tests those weights. Training again from scratch tests the recipe: a new seed changes the starting weights and the experience the agent meets while learning.

Hyperparameter searches helped substantially, but the successful settings act together. Learning rate, entropy, clipping, replay, and advantage estimation all affect the updates. A hyperparameter search can identify a useful combination without proving which individual change caused the improvement.

VII. Beyond the Fight Caves

Reuse the engineering; test whether the learning transfers.

I want to take this approach to more RuneScape encounters and content. Fight Caves is a useful starting point because it combines a long sequence of decisions with a clear terminal objective. Other encounters can test different kinds of planning, resource use, and coordination.

The next step within this environment is to understand the seeds that trained poorly. I want to compare their HP and Prayer when they reach Jad, their protection timing, their target choices, and how they handle healers. I also want to test how much the policy depends on exact timing features, recurrence, and fixed equipment.

Reusing a simulator framework saves engineering time. Reusing learned weights is a separate claim. A useful first transfer experiment would implement one bounded encounter, then compare training from scratch with a Fight Caves initialization under the same budget and across several seeds.

That is the curiosity I want to keep following: which decisions become reusable skills, and which depend on the particular world I built? For now, I can load a checkpoint, start at wave 1, and watch it finish the cave repeatedly. Getting there required learning how to ask the question as much as learning how to answer it.

VIII. Code and experiment notes

The results in this article come from one version of the code: commit 9ad3633d of my fork of PufferLib, which adds Fight Caves to PufferLib 5.0. The main run and the 93.40% run were both trained with it. The earlier experiments described along the way ran on older versions, and the project has kept changing since, so newer code may not reproduce these numbers.

1. My PufferLib fork at that commit (branch fight-caves-5.0): the code behind these results. Start with its Fight Caves guide, or go straight to the simulation and policy interface, the training adapter, and the network and optimizer.
2. Fight Caves RL: the project’s original repository, built on PufferLib 4.0, with the viewer, earlier experiments, and implementation notes. It describes an older version of the project, trained under different conditions.
3. Training dashboards on Weights & Biases, an experiment-tracking site: the main run and the retrained 93.40% run. Some details may require access.
4. Proximal Policy Optimization Algorithms, Schulman et al. (2017), for the clipped policy objective.
5. Results data: the main run’s metrics, the four-seed retraining results, and the record of the exact 93.40% retraining, in one JSON file.

Reproduce the results

These steps train the main run again from scratch and then evaluate it. Trained checkpoints aren’t published, so training is how you get the agent.

What you need: Linux and an NVIDIA graphics card; the trainer does not run without one. The build also needs NVIDIA’s CUDA toolkit, NCCL, the Clang compiler with OpenMP, ccache, and Python 3. PufferTank 5.0, PufferLib’s ready-made Docker environment, includes all of them.

1. Download the code. The “5.0” in the branch name, fight-caves-5.0, is the PufferLib version Fight Caves is integrated with. The branch holds the exact code behind these results, commit 9ad3633d, with the main run’s settings saved in config/fight_caves.ini.

   git clone --depth 1 --branch fight-caves-5.0 \
     https://github.com/jordanbailey00/PufferLib.git fight-caves
   cd fight-caves

2. Build. This downloads the game’s maps and graphics, then compiles everything into one program, ./puffer.

   ./build.sh fight_caves

3. Train and evaluate. Training stops after 499,122,176 steps, then evaluates the agent on at least 10,000 caves. In the final summary, jad_kill_rate is the share of caves completed: 0.886 means 88.6%. The full results are saved in logs/fight_caves/.

   ./puffer train

4. Watch it play. This opens the viewer with the agent you just trained, so it needs a display. Press 1, 2, 4, or 0 for 1×, 2×, 4×, or 10× speed, and Q to quit. If the viewer is slow inside PufferTank, run unset __GLX_VENDOR_LIBRARY_NAME first.

   ./puffer eval latest

To train the 93.40% run instead, use fight-caves-best-run in step 1. Everything else is the same.