fight-caves-rl

1. Defining the problem

The Fight Caves is a useful long-horizon control problem because success depends on several kinds of reasoning at once. The agent must preserve limited food and prayer points, recognize which incoming attack is dangerous, switch protection prayers before a hit is locked in, choose targets, move around collision, exploit line of sight, and avoid spending thousands of ticks on a locally safe but globally useless behavior. A mistake on wave 60 can erase the value of every correct decision made before it.

“From scratch” has a precise meaning here. Training begins from randomly initialized policy weights. There is no behavior-cloning dataset, no expert trajectory, and no hard-coded phase controller that takes over for Jad. Curriculum and reward shaping can change which states the policy encounters and which outcomes are reinforced, but the policy still has to infer the action sequence from interaction.

2. System architecture

The system has three runtime layers. fc-core is a pure C state machine containing the arena, waves, NPC behavior, collision, combat, observations, masks, and rewards. The PufferLib adapter exposes thousands of independent core states to PPO with contiguous buffers. The Raylib viewer reads the same state and writes the same actions, adding interpolation and diagnostics without owning game rules.

Keeping rendering outside the simulator makes the training path headless and reproducible. Keeping the human and policy paths on one action interface provides a stronger check: if the viewer mutates state directly, it can appear correct while the agent is training against different rules. Here, a mouse click becomes the same movement-head values consumed by a training episode.

3. The deterministic simulator

An episode uses a 64-by-64 collision arena, one player, at most 16 active NPCs, 63 wave definitions, and 15 official spawn rotations. Each state owns its random generator and all mutable timers. Resetting two environments with the same seed and replaying the same action vectors should produce identical state transitions.

Tick ordering is part of the environment contract. Prayer is instantaneous and therefore processed before queued attacks resolve. Eating and drinking have cooldowns. Movement can alter distance and line of sight before NPC decisions. Pending hits carry the information needed to resolve later, rather than looking up a possibly changed attack style at impact time.

void fc_tick(FcState* state, const int actions[FC_NUM_ACTION_HEADS]) {
    int prayer_active_at_tick_start =
        (state->player.prayer != PRAYER_NONE);

    clear_per_tick_flags(state);
    process_player_actions(state, actions);
    decrement_player_timers(&state->player);
    fc_prayer_drain_tick(&state->player, prayer_active_at_tick_start);

    for (int i = 0; i < FC_MAX_NPCS; i++)
        fc_npc_tick(state, i);

    fc_resolve_player_pending_hits(state);
    for (int i = 0; i < FC_MAX_NPCS; i++)
        if (state->npcs[i].active)
            fc_resolve_npc_pending_hits(state, i);

    check_terminal(state);
    state->tick++;
}

Fixed-size arrays replace per-tick heap allocation. That design trades some unused capacity for predictable memory access and removes allocator contention when 4,096 environments advance in parallel. It also makes state inspection straightforward: the viewer and test tools can copy or compare a state without reconstructing an object graph.

4. Observation and action design

The policy receives 122 normalized float values. Seventeen describe the player: health, prayer, position, timers, active protection prayer, remaining supplies, incoming-hit counts, and selected target. Eight deterministic NPC slots contribute 12 values each, including position, health, distance, attack telegraph, timer, line of sight, pending style, and impact time. Nine values describe the wave and recent events.

/* The policy sees player state, eight NPC slots, and wave metadata. */
#define FC_OBS_PLAYER_SIZE      17
#define FC_OBS_NPC_STRIDE       12
#define FC_OBS_NPC_SLOTS         8
#define FC_OBS_NPC_TOTAL        (FC_OBS_NPC_STRIDE * FC_OBS_NPC_SLOTS)
#define FC_OBS_META_SIZE         9
#define FC_POLICY_OBS_SIZE      (FC_OBS_PLAYER_SIZE + FC_OBS_NPC_TOTAL + FC_OBS_META_SIZE)

/* Reward features are adjacent in memory, but hidden from the policy. */
#define FC_REWARD_FEATURES      19
#define FC_TOTAL_OBS            (FC_POLICY_OBS_SIZE + FC_REWARD_FEATURES)

/* The simulator has two viewer pathfinding heads; training uses the first five. */
#define FC_NUM_ACTION_HEADS      7
#define FC_ACT_SIZES { FC_MOVE_DIM, FC_ATTACK_DIM, FC_PRAYER_DIM,                        FC_EAT_DIM, FC_DRINK_DIM,                        FC_MOVE_TARGET_X_DIM, FC_MOVE_TARGET_Y_DIM }

When more than eight NPCs are alive, the observation contains the eight closest. Slots are sorted first by Chebyshev distance and then by spawn index. That tie-breaker is not cosmetic: unstable slot ordering makes the meaning of “attack slot 3” change between otherwise identical runs. Overflow NPCs remain active in the simulator even though they cannot be directly selected until they enter the visible set.

Training uses five categorical action heads. Movement has 17 choices: idle, eight one-tile directions, and eight two-tile run directions. Attack selects none or one of eight NPC slots. Prayer chooses no change, off, protect magic, protect ranged, or protect melee. Eat and drink heads control consumables. The core also defines target-tile heads for human click-to-move, but the training adapter disables those higher-level actions so PPO has to choose low-level movement.

5. Reward design and the behaviors it created

The core emits 19 reward features separately from the 122 policy inputs. The trainer converts those features into a scalar using configuration weights. This prevents reward-only facts from leaking into the policy while keeping experiments configurable: I can change the cost of wasted food or a Jad-heal event without recompiling observation logic.

FcRewardBreakdown out;
memset(&out, 0, sizeof(out));
fc_write_reward_features(state, out.raw);

/* A large mistake costs more than several small scratches. */
{
    float dmg_frac = out.raw[FC_RWD_DAMAGE_TAKEN];
    out.damage_taken =
        dmg_frac * dmg_frac * 70.0f * params->w_damage_taken;
}

if (out.raw[FC_RWD_WAVE_CLEAR] > 0.0f) {
    int cleared_wave = state->current_wave - 1;
    if (cleared_wave < 1) cleared_wave = 1;
    out.wave_clear = params->w_wave_clear * (float)cleared_wave;
}

The published v35 recipe rewards damage dealt, kills, wave clears, correct protection, kiting, safe attacks, and Jad completion. It penalizes damage, death, incorrect prayer, wasted resources, invalid actions, unnecessary prayer, and very long stalls. Damage cost is quadratic in the fraction of health lost, so one catastrophic hit is worse than several small errors with the same total damage. Wave-clear reward scales with wave number because a late clear represents more progress and a rarer state.

6. Training at scale

The policy is a three-layer MinGRU with a 256-unit hidden state trained using PPO. Recurrence matters because the current vector does not encode every past event: the network benefits from remembering target changes, previous movement, prayer transitions, and the sequence leading into a wave. PPO collects experience from 4,096 environments across two buffers and updates the policy in batches.

[vec]
total_agents = 4096
num_buffers = 2

[train]
total_timesteps = 3_000_000_000
learning_rate = 0.0009
gamma = 0.996327
gae_lambda = 0.96405
clip_coef = 0.1783
ent_coef = 0.02423
vf_coef = 1
max_grad_norm = 0.25

[policy]
hidden_size = 256
num_layers = 3

The configuration shown above is abbreviated, but it captures the scale and principal optimizer values. The published sweep selected a learning rate of 9e-4, discount factor of 0.996327, GAE lambda of 0.96405, entropy coefficient of 0.02423, clipping coefficient of 0.1783, and gradient-norm cap of 0.25. At approximately 1.9 million environment steps per second, a three-billion-step run completes in roughly 26 minutes on the measured RTX 5070 Ti system.

High throughput changes the experiment loop. A configuration can be tested to late-game behavior in minutes, and broad sweeps become practical. It also makes silent correctness errors more expensive: a missing collision file or wrong terminal observation can produce billions of invalid transitions before a curve makes the fault obvious.

7. Integration bugs that changed the result

Several of the most consequential problems were outside the neural network. They are worth documenting because a reinforcement-learning result is only as valid as the transition data that reaches the optimizer.

• A curriculum helper advanced the environment by repeatedly calling the normal step function. Directly setting the intended wave removed unintended state transitions before the episode began.
• A collision asset was not found on one launch path, so training silently used an open arena. Path discovery and startup checks were tightened; removing permissive fallbacks entirely remains a useful hardening step.
• Shared static BFS scratch memory caused failures when thousands of environments ran concurrently. Moving scratch state out of the shared global path restored environment isolation.
• The adapter initially risked writing the first observation of the reset episode over the terminal observation. Writing reward and terminal state before reset preserved the transition that PPO actually needs.
• The 3D path exposed reversed triangle winding, coordinate mismatches, duplicate render passes, and terrain-color errors. These did not change headless training, but they made policy inspection misleading until fixed.

/* Step the simulation and derive this transition's reward. */
fc_step(&env->state, actions);
env->rewards[0] = fc_puffer_compute_reward(env);
env->ep_length++;

/* The agent must see the terminal state observation,
 * not the next episode's reset observation. */
fc_puffer_write_obs(env);

if (fc_is_terminal(&env->state)) {
    env->terminals[0] = 1.0f;
    /* Episode metrics are recorded here; reset follows externally. */
}

The terminal-observation issue is a small ordering bug with large statistical consequences. If the value function is trained on the reset state while the reward belongs to the death or completion state, the target joins two unrelated episodes. The run may still improve, which makes this class of error harder to detect than a crash.

8. Experiment progression

The result came from a sequence of measured revisions rather than one final configuration. Early versions established cold-start progress to wave 60. Later versions learned Jad, corrected observation gaps, removed score-inflating reward, and tested PPO settings. The table records milestone configurations from the repository's run history.

The v29.1 ablation is a useful example of why removal experiments matter. A prioritized sampling feature added compute and complexity, but setting its priority coefficient to zero preserved or improved the metric that mattered. The simpler training path was faster and easier to interpret.

9. Published result and late-training degradation

v35.1 exceeded 90% Jad kills for a substantial portion of training, roughly 1.1 to 1.6 billion steps. It later degraded, reaching about 62% near three billion steps under the constant 9e-4 learning rate. For deployment I therefore preserve peak evaluation checkpoints instead of assuming the final optimizer state is best.

This degradation remains an open optimization problem. Candidate tests include learning-rate decay after the first stable peak, stronger checkpoint selection, lower late-stage entropy, and multi-seed comparison. Reporting only the best point would hide this behavior; reporting only the final point would hide the policy the run actually discovered.

10. Loadout experiments on the development branch

A later development branch applies the same training setup to nine equipment loadouts. These runs are exploratory and are not the v35.1 published result. Each sweep trains to 1.5 billion steps with 4,096 environments, two buffers, and the 256-unit, three-layer MinGRU.

The crossbow result separates “reach Jad” from “kill Jad”: it traverses the cave reliably but lacks enough effective terminal-phase control or damage to convert. One no-supplies seed with the nominal best bow collapsed completely while another loadout remained strong. That is not evidence that the equipment is inherently worse. It is a reason to repeat seeds and inspect configuration interactions before making a claim.

This sweep also exposed stale native builds as an experiment-integrity risk. The development tooling now records a backend stamp and forces rebuilds when core inputs change, reducing the chance that a run name and its compiled simulator disagree.

11. What the project established

The central lesson is that environment engineering and learning are inseparable. The largest gains did not come from making the network arbitrarily larger. They came from repairing missing information, removing misleading incentives, testing simplifications, preserving terminal transitions, and measuring the outcome directly. The trained policy is the visible result, but the durable artifact is a fast and inspectable experimental system for asking why it works.