chore(training): Track training artifacts in git, add idempotent Linux setup/run scripts, and commit run01 results

.gitignore

@@ -1,8 +1,7 @@
 .DS_Store
 
-# RL training artifacts (training/ code is committed; outputs are not)
+# RL training: checkpoints and logs ARE committed (training results must
+# survive any single machine); only the env and scratch files are not.
 training/.venv/
-training/logs/
-training/checkpoints/
 training/smoke_run.log
 training/__pycache__/

TRAINING_LINUX.md

@@ -1,44 +1,44 @@
 # Training on the Linux / RTX 3090 box
 
-Remote-training workflow: run long training sessions on the Linux machine,
+Remote-training workflow: run long training sessions on the Linux machine and
-watch the dashboard from any machine on the network, and ship the trained
+watch the dashboard from any machine on the network. **All training artifacts
-model back to the Mac mini automatically when the run finishes. General
+(checkpoints, TensorBoard logs, exported bots) are committed to git** — no
-training concepts and the export/evaluate workflow live in
+result ever depends on a single machine, and moving models between the box
-[TRAINING.md](TRAINING.md) — this doc is only what differs on the Linux box.
+and the Mac is just `git pull`. General training concepts and the
+export/evaluate workflow live in [TRAINING.md](TRAINING.md) — this doc is
+only what differs on the Linux box.
+
+Both workflows below are wrapped in idempotent scripts in `training/` —
+re-running either is always safe.
 
 ## One-time setup
 
+GitHub auth first (git-over-HTTPS no longer accepts account passwords, so
+clone over SSH — this key also lets `run_training.sh` push results):
+
 ```bash
-# GitHub auth (one-time): git-over-HTTPS no longer accepts account passwords,
-# so clone over SSH. Generate a key, then add the printed public key at
-# github.com/settings/keys → "New SSH key".
 ssh-keygen -t ed25519        # accept the defaults
-cat ~/.ssh/id_ed25519.pub
+cat ~/.ssh/id_ed25519.pub    # add at github.com/settings/keys → "New SSH key"
 
 cd ~/ai-training
 git clone git@github.com:jcreek/CosmicClash.git
 # (submodules are editor tooling only — training doesn't need them)
 
-# Godot 4.7.1 Linux binary
+~/ai-training/CosmicClash/training/setup_linux.sh
-mkdir -p ~/ai-training/godot && cd ~/ai-training/godot
-wget https://github.com/godotengine/godot/releases/download/4.7.1-stable/Godot_v4.7.1-stable_linux.x86_64.zip
-unzip Godot_v4.7.1-stable_linux.x86_64.zip
-echo 'export GODOT_BIN=~/ai-training/godot/Godot_v4.7.1-stable_linux.x86_64' >> ~/.bashrc && source ~/.bashrc
-
-# Python env
-cd ~/ai-training/CosmicClash/training
-python3 -m venv .venv
-.venv/bin/pip install -r requirements.txt
-
-# CUDA sanity check — should print True
-.venv/bin/python -c "import torch; print(torch.cuda.is_available())"
-
-# Headless smoke test — the game must boot without rendering
-$GODOT_BIN --headless --path ../Game res://scenes/free_play.tscn --quit-after 300
 ```
 
-If the CUDA check prints `False`, reinstall torch from the CUDA index
+`setup_linux.sh` is safe to re-run any time (after a Godot upgrade, a broken
-(`pip install torch --index-url https://download.pytorch.org/whl/cu121`).
+venv, a fresh clone — it checks each step before acting). It:
+
+- downloads the Godot 4.7.1 Linux binary to `~/ai-training/godot/` if missing
+  (override the location by exporting `GODOT_BIN`);
+- creates `training/.venv` if missing and installs requirements;
+- verifies CUDA torch, reinstalling from the CUDA wheel index if the box got
+  a CPU-only build;
+- runs the Godot import pass (fresh clones have no `.godot/` cache, so
+  `class_name` scripts aren't registered until the project imports once);
+- finishes with the headless smoke test — the game must boot without
+  rendering.
 
 ## Maximising throughput
 
@@ -60,22 +60,27 @@ Tune by watching `time/fps`: run a 2-minute smoke run per setting and keep
 the best. Reference: 6 instances × speedup 16 ≈ 1,385 steps/s on an M4 Mac
 mini — a 20M-step run in ~4 h. Doubling fps halves that.
 
-## Start a run
+## Run training
+
+`run_training.sh <experiment> [train.py args...]` wraps the whole cycle:
+`git pull` → train → export the policy JSON → commit and push checkpoints,
+logs, and the exported bot. Run it inside `tmux`/`screen` so an SSH
+disconnect doesn't kill training. Ctrl-C is safe: the trainer writes
+`final.zip` on the way out, and the script still exports, commits, and
+pushes what it has.
 
 Fresh run:
 
 ```bash
 cd ~/ai-training/CosmicClash/training
-.venv/bin/python train.py --experiment run03 --timesteps 20000000 \
+./run_training.sh run03 --timesteps 20000000 --n-parallel 14 --speedup 24
-    --n-parallel 14 --speedup 24
 ```
 
 Resuming a previous policy (continues its timestep counter; `--timesteps` is
 *additional* steps). Lessons from run01/run02 hard-coded into flags:
 
 ```bash
-.venv/bin/python train.py --experiment run03 --timesteps 20000000 \
+./run_training.sh run03 --timesteps 20000000 --n-parallel 14 --speedup 24 \
-    --n-parallel 14 --speedup 24 \
     --resume checkpoints/run02/final.zip --ent-coef 0.001 --reset-std 0.3
 ```
 
@@ -85,32 +90,21 @@ Resuming a previous policy (continues its timestep counter; `--timesteps` is
   ~1.0 — check it 30–45 min in before committing to a long run.
 - `--reset-std` — on resume, restores exploration a collapsed checkpoint lost.
 
-Run inside `tmux`/`screen` so an SSH disconnect doesn't kill training.
+## Results travel via git
-Ctrl-C is safe: `final.zip` is written on the way out.
 
-## Auto-copy the result to the Mac mini when training finishes
+`run_training.sh` commits and pushes everything a run produces:
 
-One-time: enable **System Settings → General → Sharing → Remote Login** on
+- `training/checkpoints/<exp>/` — periodic checkpoints + `final.zip`, for
-the Mac mini, and `ssh-copy-id jcreek@Joshs-Mac-mini.local` from the Linux
+  future `--resume`, evaluation, and difficulty tiers (an early checkpoint
-box so rsync runs unattended.
+  *is* an easy bot);
+- `training/logs/` — TensorBoard history;
+- `Game/bots/<exp>.json` — the exported policy, immediately playable (point
+  Match or Spectate mode at `res://bots/<exp>.json`);
+- `training/eval_history.json` — if evaluations ran.
 
-Chain export + copy onto the training command (`;` not `&&`, so the copy
+On the Mac (or anywhere), collecting the results is just `git pull`. A
-still happens after a Ctrl-C — `final.zip` exists either way):
+20M-step run adds roughly 40 MB of checkpoints — acceptable growth for the
-
+guarantee that training is never lost with a machine.
-```bash
-EXP=run03
-.venv/bin/python train.py --experiment $EXP --timesteps 20000000 --n-parallel 14 --speedup 24 ; \
-.venv/bin/python export_policy.py checkpoints/$EXP/final.zip ../Game/bots/$EXP.json && \
-rsync -av checkpoints/$EXP/final.zip \
-    jcreek@Joshs-Mac-mini.local:~/Documents/repos/GitHub/CosmicClash/training/checkpoints/$EXP/ && \
-rsync -av ../Game/bots/$EXP.json \
-    jcreek@Joshs-Mac-mini.local:~/Documents/repos/GitHub/CosmicClash/Game/bots/
-```
-
-That lands both the raw checkpoint (for future `--resume` / evaluation on the
-Mac) and the exported JSON policy (immediately playable — point Match or
-Spectate mode at `res://bots/<exp>.json`). Add a third rsync of `logs/` if
-you also want the TensorBoard history archived on the Mac.
 
 ## Dashboard over the network